2024-04-21T22:50:31.670 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-21T22:50:31.670 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} --archive /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667584 --verbose --name teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667584/orig.config.yaml 2024-04-21T22:50:31.738 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-21T22:50:31.836 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667584 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7667584' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 1 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 820a2f8ced66152ed3d765c31444b8076ea28c6b subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 820a2f8ced66152ed3d765c31444b8076ea28c6b kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 676 sha1: 820a2f8ced66152ed3d765c31444b8076ea28c6b sleep_before_teardown: 0 subset: 46/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi044.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZOvhnCoO7RuFOartC1IN/OgpYX71LoFas7Yf+L8iyezIJEJJHq9lkcX4y7PFhh+Cl+y29sQRuC/d2gEfYhkcc= smithi055.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKSVAq56VARQO+y2yeuZ/vYCWSxTekgJKkLSDcXTkdmzYvXXVZJGLVvqpiR/TTZlKTDEVRNMCAkJv/SD0t7xD5c= smithi069.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNZ2ARyEPbkEtkRZnOsqMYXK/A5Ks0sUyozuovJ0cl0Nx8nokcMxbrshyyLbO8SudTx87tx1rfIYFL2CIQupYJQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-21_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-21T22:50:31.836 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa; will attempt to use it 2024-04-21T22:50:31.837 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks 2024-04-21T22:50:31.837 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-21T22:50:31.839 INFO:teuthology.task.internal:Checking packages... 2024-04-21T22:50:31.862 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '820a2f8ced66152ed3d765c31444b8076ea28c6b' 2024-04-21T22:50:31.862 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-21T22:50:31.862 INFO:teuthology.packaging:ref: None 2024-04-21T22:50:31.862 INFO:teuthology.packaging:tag: None 2024-04-21T22:50:31.862 INFO:teuthology.packaging:branch: squid 2024-04-21T22:50:31.862 INFO:teuthology.packaging:sha1: 820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T22:50:31.863 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-21T22:50:32.155 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2407.g430e09df 2024-04-21T22:50:32.157 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-21T22:50:32.168 INFO:teuthology.task.internal:no buildpackages task found 2024-04-21T22:50:32.168 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-21T22:50:32.210 INFO:teuthology.task.internal:Saving configuration 2024-04-21T22:50:32.232 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-21T22:50:32.240 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T22:50:32.267 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi044.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667584', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 22:45:08.998255', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZOvhnCoO7RuFOartC1IN/OgpYX71LoFas7Yf+L8iyezIJEJJHq9lkcX4y7PFhh+Cl+y29sQRuC/d2gEfYhkcc='} 2024-04-21T22:50:32.290 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667584', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 22:45:09.000616', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKSVAq56VARQO+y2yeuZ/vYCWSxTekgJKkLSDcXTkdmzYvXXVZJGLVvqpiR/TTZlKTDEVRNMCAkJv/SD0t7xD5c='} 2024-04-21T22:50:32.313 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667584', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 22:45:08.999772', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNZ2ARyEPbkEtkRZnOsqMYXK/A5Ks0sUyozuovJ0cl0Nx8nokcMxbrshyyLbO8SudTx87tx1rfIYFL2CIQupYJQ='} 2024-04-21T22:50:32.313 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-21T22:50:32.319 INFO:teuthology.task.internal:roles: ubuntu@smithi044.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-21T22:50:32.319 INFO:teuthology.task.internal:roles: ubuntu@smithi055.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-21T22:50:32.319 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-21T22:50:32.319 INFO:teuthology.run_tasks:Running task console_log... 2024-04-21T22:50:32.432 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2ed08e3160>, signals=[15]) 2024-04-21T22:50:32.432 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-21T22:50:32.439 INFO:teuthology.task.internal:Opening connections... 2024-04-21T22:50:32.439 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi044.front.sepia.ceph.com 2024-04-21T22:50:32.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T22:50:32.518 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi055.front.sepia.ceph.com 2024-04-21T22:50:32.519 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T22:50:32.595 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi069.front.sepia.ceph.com 2024-04-21T22:50:32.596 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T22:50:32.670 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-21T22:50:32.676 DEBUG:teuthology.orchestra.run.smithi044:> uname -m 2024-04-21T22:50:32.695 INFO:teuthology.orchestra.run.smithi044.stdout:x86_64 2024-04-21T22:50:32.695 DEBUG:teuthology.orchestra.run.smithi044:> cat /etc/os-release 2024-04-21T22:50:32.753 INFO:teuthology.orchestra.run.smithi044.stdout:NAME="CentOS Stream" 2024-04-21T22:50:32.753 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION="9" 2024-04-21T22:50:32.753 INFO:teuthology.orchestra.run.smithi044.stdout:ID="centos" 2024-04-21T22:50:32.753 INFO:teuthology.orchestra.run.smithi044.stdout:ID_LIKE="rhel fedora" 2024-04-21T22:50:32.753 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION_ID="9" 2024-04-21T22:50:32.753 INFO:teuthology.orchestra.run.smithi044.stdout:PLATFORM_ID="platform:el9" 2024-04-21T22:50:32.753 INFO:teuthology.orchestra.run.smithi044.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T22:50:32.753 INFO:teuthology.orchestra.run.smithi044.stdout:ANSI_COLOR="0;31" 2024-04-21T22:50:32.753 INFO:teuthology.orchestra.run.smithi044.stdout:LOGO="fedora-logo-icon" 2024-04-21T22:50:32.753 INFO:teuthology.orchestra.run.smithi044.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T22:50:32.754 INFO:teuthology.orchestra.run.smithi044.stdout:HOME_URL="https://centos.org/" 2024-04-21T22:50:32.754 INFO:teuthology.orchestra.run.smithi044.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T22:50:32.754 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T22:50:32.754 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T22:50:32.754 INFO:teuthology.lock.ops:Updating smithi044.front.sepia.ceph.com on lock server 2024-04-21T22:50:32.778 DEBUG:teuthology.orchestra.run.smithi055:> uname -m 2024-04-21T22:50:32.797 INFO:teuthology.orchestra.run.smithi055.stdout:x86_64 2024-04-21T22:50:32.797 DEBUG:teuthology.orchestra.run.smithi055:> cat /etc/os-release 2024-04-21T22:50:32.855 INFO:teuthology.orchestra.run.smithi055.stdout:NAME="CentOS Stream" 2024-04-21T22:50:32.855 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION="9" 2024-04-21T22:50:32.855 INFO:teuthology.orchestra.run.smithi055.stdout:ID="centos" 2024-04-21T22:50:32.855 INFO:teuthology.orchestra.run.smithi055.stdout:ID_LIKE="rhel fedora" 2024-04-21T22:50:32.855 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION_ID="9" 2024-04-21T22:50:32.855 INFO:teuthology.orchestra.run.smithi055.stdout:PLATFORM_ID="platform:el9" 2024-04-21T22:50:32.855 INFO:teuthology.orchestra.run.smithi055.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T22:50:32.855 INFO:teuthology.orchestra.run.smithi055.stdout:ANSI_COLOR="0;31" 2024-04-21T22:50:32.855 INFO:teuthology.orchestra.run.smithi055.stdout:LOGO="fedora-logo-icon" 2024-04-21T22:50:32.855 INFO:teuthology.orchestra.run.smithi055.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T22:50:32.856 INFO:teuthology.orchestra.run.smithi055.stdout:HOME_URL="https://centos.org/" 2024-04-21T22:50:32.856 INFO:teuthology.orchestra.run.smithi055.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T22:50:32.856 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T22:50:32.856 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T22:50:32.856 INFO:teuthology.lock.ops:Updating smithi055.front.sepia.ceph.com on lock server 2024-04-21T22:50:32.878 DEBUG:teuthology.orchestra.run.smithi069:> uname -m 2024-04-21T22:50:32.896 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64 2024-04-21T22:50:32.897 DEBUG:teuthology.orchestra.run.smithi069:> cat /etc/os-release 2024-04-21T22:50:32.953 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="CentOS Stream" 2024-04-21T22:50:32.953 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="9" 2024-04-21T22:50:32.953 INFO:teuthology.orchestra.run.smithi069.stdout:ID="centos" 2024-04-21T22:50:32.953 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE="rhel fedora" 2024-04-21T22:50:32.954 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="9" 2024-04-21T22:50:32.954 INFO:teuthology.orchestra.run.smithi069.stdout:PLATFORM_ID="platform:el9" 2024-04-21T22:50:32.954 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T22:50:32.954 INFO:teuthology.orchestra.run.smithi069.stdout:ANSI_COLOR="0;31" 2024-04-21T22:50:32.954 INFO:teuthology.orchestra.run.smithi069.stdout:LOGO="fedora-logo-icon" 2024-04-21T22:50:32.954 INFO:teuthology.orchestra.run.smithi069.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T22:50:32.954 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://centos.org/" 2024-04-21T22:50:32.954 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T22:50:32.954 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T22:50:32.954 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T22:50:32.955 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2024-04-21T22:50:32.978 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-21T22:50:32.988 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-21T22:50:33.002 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-21T22:50:33.002 DEBUG:teuthology.orchestra.run.smithi044:> test '!' -e /home/ubuntu/cephtest 2024-04-21T22:50:33.005 DEBUG:teuthology.orchestra.run.smithi055:> test '!' -e /home/ubuntu/cephtest 2024-04-21T22:50:33.007 DEBUG:teuthology.orchestra.run.smithi069:> test '!' -e /home/ubuntu/cephtest 2024-04-21T22:50:33.023 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-21T22:50:33.029 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-21T22:50:33.029 DEBUG:teuthology.orchestra.run.smithi044:> test -z $(ls -A /var/lib/ceph) 2024-04-21T22:50:33.062 DEBUG:teuthology.orchestra.run.smithi055:> test -z $(ls -A /var/lib/ceph) 2024-04-21T22:50:33.065 DEBUG:teuthology.orchestra.run.smithi069:> test -z $(ls -A /var/lib/ceph) 2024-04-21T22:50:33.172 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-21T22:50:33.249 INFO:teuthology.run_tasks:Running task kernel... 2024-04-21T22:50:33.264 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-04-21T22:50:33.265 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-21T22:50:33.265 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-21T22:50:33.265 DEBUG:teuthology.orchestra.run.smithi044:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T22:50:33.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T22:50:33.282 DEBUG:teuthology.orchestra.run.smithi044:> uname -r 2024-04-21T22:50:33.340 INFO:teuthology.orchestra.run.smithi044.stdout:5.14.0-437.el9.x86_64 2024-04-21T22:50:33.340 INFO:teuthology.task.kernel:Running kernel on smithi044: 5.14.0-437.el9.x86_64 2024-04-21T22:50:33.341 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum install -y kernel 2024-04-21T22:50:35.662 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-21T22:50:38.014 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-04-21T22:50:42.297 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-21T22:50:43.740 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 16 kB 00:00 2024-04-21T22:51:02.842 INFO:teuthology.orchestra.run.smithi044.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 21 MB 00:18 2024-04-21T22:51:08.206 INFO:teuthology.orchestra.run.smithi044.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-21T22:51:10.196 INFO:teuthology.orchestra.run.smithi044.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-21T22:51:10.196 INFO:teuthology.orchestra.run.smithi044.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-21T22:51:10.292 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-21T22:51:10.293 INFO:teuthology.orchestra.run.smithi044.stdout:Nothing to do. 2024-04-21T22:51:10.293 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-21T22:51:10.419 DEBUG:teuthology.orchestra.run.smithi044:> echo no | sudo yum reinstall kernel || true 2024-04-21T22:51:10.941 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:00:02 ago on Sun 21 Apr 2024 10:51:08 PM UTC. 2024-04-21T22:51:11.064 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-21T22:51:11.075 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-21T22:51:11.075 INFO:teuthology.orchestra.run.smithi044.stdout: Package Architecture Version Repository Size 2024-04-21T22:51:11.075 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-21T22:51:11.075 INFO:teuthology.orchestra.run.smithi044.stdout:Reinstalling: 2024-04-21T22:51:11.075 INFO:teuthology.orchestra.run.smithi044.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-21T22:51:11.075 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:51:11.075 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-21T22:51:11.076 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-21T22:51:11.076 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:51:11.076 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 255 k 2024-04-21T22:51:11.076 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 0 2024-04-21T22:51:11.076 INFO:teuthology.orchestra.run.smithi044.stderr:Operation aborted. 2024-04-21T22:51:11.141 INFO:teuthology.orchestra.run.smithi044.stdout:Is this ok [y/N]: 2024-04-21T22:51:11.141 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum reinstall -y kernel || true 2024-04-21T22:51:11.654 INFO:teuthology.orchestra.run.smithi044.stdout:Last metadata expiration check: 0:00:03 ago on Sun 21 Apr 2024 10:51:08 PM UTC. 2024-04-21T22:51:11.775 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-21T22:51:11.775 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-21T22:51:11.776 INFO:teuthology.orchestra.run.smithi044.stdout: Package Architecture Version Repository Size 2024-04-21T22:51:11.776 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-21T22:51:11.776 INFO:teuthology.orchestra.run.smithi044.stdout:Reinstalling: 2024-04-21T22:51:11.776 INFO:teuthology.orchestra.run.smithi044.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-21T22:51:11.776 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:51:11.776 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-21T22:51:11.776 INFO:teuthology.orchestra.run.smithi044.stdout:================================================================================ 2024-04-21T22:51:11.776 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:51:11.776 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 255 k 2024-04-21T22:51:11.776 INFO:teuthology.orchestra.run.smithi044.stdout:Installed size: 0 2024-04-21T22:51:11.776 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-04-21T22:51:12.275 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.1 MB/s | 255 kB 00:00 2024-04-21T22:51:12.275 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-21T22:51:12.275 INFO:teuthology.orchestra.run.smithi044.stdout:Total 510 kB/s | 255 kB 00:00 2024-04-21T22:51:12.293 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-04-21T22:51:12.480 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-04-21T22:51:12.481 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-04-21T22:51:12.486 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-04-21T22:51:12.486 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-04-21T22:51:12.632 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-04-21T22:51:12.718 INFO:teuthology.orchestra.run.smithi044.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-21T22:51:12.718 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T22:51:13.332 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T22:51:13.332 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-21T22:51:14.042 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T22:51:14.042 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:51:14.042 INFO:teuthology.orchestra.run.smithi044.stdout:Reinstalled: 2024-04-21T22:51:14.042 INFO:teuthology.orchestra.run.smithi044.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-21T22:51:14.042 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:51:14.042 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-21T22:51:14.225 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q kernel | sort -rV | head -n 1 2024-04-21T22:51:14.274 INFO:teuthology.orchestra.run.smithi044.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-21T22:51:14.274 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-21T22:51:14.274 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-21T22:51:14.275 DEBUG:teuthology.parallel:result is None 2024-04-21T22:51:14.275 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-21T22:51:14.284 INFO:teuthology.task.internal:Creating test directory... 2024-04-21T22:51:14.284 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T22:51:14.287 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T22:51:14.289 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T22:51:14.308 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-21T22:51:14.322 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-21T22:51:14.340 INFO:teuthology.task.internal:Creating archive directory... 2024-04-21T22:51:14.340 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T22:51:14.343 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T22:51:14.348 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T22:51:14.390 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-21T22:51:14.398 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-21T22:51:14.398 DEBUG:teuthology.orchestra.run.smithi055:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T22:51:14.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T22:51:14.433 DEBUG:teuthology.orchestra.run.smithi069:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T22:51:14.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T22:51:14.450 DEBUG:teuthology.orchestra.run.smithi044:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T22:51:14.452 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T22:51:14.476 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T22:51:14.480 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T22:51:14.499 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T22:51:14.524 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T22:51:14.528 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T22:51:14.539 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T22:51:14.548 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T22:51:14.550 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-21T22:51:14.557 INFO:teuthology.task.internal:Configuring sudo... 2024-04-21T22:51:14.557 DEBUG:teuthology.orchestra.run.smithi044:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T22:51:14.559 DEBUG:teuthology.orchestra.run.smithi055:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T22:51:14.562 DEBUG:teuthology.orchestra.run.smithi069:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T22:51:14.618 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-21T22:51:14.627 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-21T22:51:14.627 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T22:51:14.630 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T22:51:14.633 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T22:51:14.675 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T22:51:14.729 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T22:51:14.773 DEBUG:teuthology.orchestra.run.smithi044:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T22:51:14.857 DEBUG:teuthology.orchestra.run.smithi044:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T22:51:14.895 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T22:51:14.895 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T22:51:14.959 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T22:51:15.005 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T22:51:15.057 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T22:51:15.143 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T22:51:15.186 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-21T22:51:15.186 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T22:51:15.254 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T22:51:15.299 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T22:51:15.350 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T22:51:15.435 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T22:51:15.478 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-21T22:51:15.478 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T22:51:15.546 DEBUG:teuthology.orchestra.run.smithi044:> sudo service rsyslog restart 2024-04-21T22:51:15.548 DEBUG:teuthology.orchestra.run.smithi055:> sudo service rsyslog restart 2024-04-21T22:51:15.551 DEBUG:teuthology.orchestra.run.smithi069:> sudo service rsyslog restart 2024-04-21T22:51:15.582 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T22:51:15.586 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T22:51:15.626 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T22:51:16.005 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-21T22:51:16.014 INFO:teuthology.task.internal:Starting timer... 2024-04-21T22:51:16.014 INFO:teuthology.run_tasks:Running task pcp... 2024-04-21T22:51:16.034 INFO:teuthology.run_tasks:Running task selinux... 2024-04-21T22:51:16.058 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-21T22:51:16.058 DEBUG:teuthology.orchestra.run.smithi044:> sudo service auditd rotate 2024-04-21T22:51:16.118 INFO:teuthology.orchestra.run.smithi044.stdout:Rotating logs: 2024-04-21T22:51:16.119 DEBUG:teuthology.orchestra.run.smithi055:> sudo service auditd rotate 2024-04-21T22:51:16.184 INFO:teuthology.orchestra.run.smithi055.stdout:Rotating logs: 2024-04-21T22:51:16.186 DEBUG:teuthology.orchestra.run.smithi069:> sudo service auditd rotate 2024-04-21T22:51:16.262 INFO:teuthology.orchestra.run.smithi069.stdout:Rotating logs: 2024-04-21T22:51:16.264 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-21T22:51:16.264 DEBUG:teuthology.orchestra.run.smithi044:> /usr/sbin/getenforce 2024-04-21T22:51:16.289 INFO:teuthology.orchestra.run.smithi044.stdout:Permissive 2024-04-21T22:51:16.289 DEBUG:teuthology.orchestra.run.smithi055:> /usr/sbin/getenforce 2024-04-21T22:51:16.324 INFO:teuthology.orchestra.run.smithi055.stdout:Permissive 2024-04-21T22:51:16.325 DEBUG:teuthology.orchestra.run.smithi069:> /usr/sbin/getenforce 2024-04-21T22:51:16.353 INFO:teuthology.orchestra.run.smithi069.stdout:Permissive 2024-04-21T22:51:16.354 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi044.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi055.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi069.front.sepia.ceph.com': 'permissive'} 2024-04-21T22:51:16.354 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T22:51:16.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T22:51:16.379 DEBUG:teuthology.orchestra.run.smithi055:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T22:51:16.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T22:51:16.407 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T22:51:16.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T22:51:16.434 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-21T22:51:16.434 DEBUG:teuthology.orchestra.run.smithi044:> sudo /usr/sbin/setenforce permissive 2024-04-21T22:51:16.460 DEBUG:teuthology.orchestra.run.smithi055:> sudo /usr/sbin/setenforce permissive 2024-04-21T22:51:16.488 DEBUG:teuthology.orchestra.run.smithi069:> sudo /usr/sbin/setenforce permissive 2024-04-21T22:51:16.522 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-21T22:51:16.532 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-21T22:51:16.550 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-21T22:51:16.552 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi044.front.sepia.ceph.com,smithi055.front.sepia.ceph.com,smithi069.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-21T22:55:22.301 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi044.front.sepia.ceph.com'), Remote(name='ubuntu@smithi055.front.sepia.ceph.com'), Remote(name='ubuntu@smithi069.front.sepia.ceph.com')] 2024-04-21T22:55:22.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-04-21T22:55:22.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi044.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T22:55:22.388 DEBUG:teuthology.orchestra.run.smithi044:> true 2024-04-21T22:55:22.472 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi044.front.sepia.ceph.com' 2024-04-21T22:55:22.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-04-21T22:55:22.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T22:55:22.555 DEBUG:teuthology.orchestra.run.smithi055:> true 2024-04-21T22:55:22.645 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2024-04-21T22:55:22.645 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-21T22:55:22.646 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T22:55:22.725 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-04-21T22:55:22.816 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-21T22:55:22.816 INFO:teuthology.run_tasks:Running task clock... 2024-04-21T22:55:22.828 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-21T22:55:22.828 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T22:55:22.829 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T22:55:22.831 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T22:55:22.831 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T22:55:22.834 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T22:55:22.835 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T22:55:22.862 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T22:55:22.864 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T22:55:22.878 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T22:55:22.880 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T22:55:22.887 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T22:55:22.902 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T22:55:22.903 INFO:teuthology.orchestra.run.smithi044.stderr:sudo: ntpd: command not found 2024-04-21T22:55:22.905 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: ntpd: command not found 2024-04-21T22:55:22.915 INFO:teuthology.orchestra.run.smithi044.stdout:506 Cannot talk to daemon 2024-04-21T22:55:22.917 INFO:teuthology.orchestra.run.smithi055.stdout:506 Cannot talk to daemon 2024-04-21T22:55:22.926 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: ntpd: command not found 2024-04-21T22:55:22.929 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T22:55:22.932 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T22:55:22.937 INFO:teuthology.orchestra.run.smithi069.stdout:506 Cannot talk to daemon 2024-04-21T22:55:22.944 INFO:teuthology.orchestra.run.smithi044.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T22:55:22.946 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T22:55:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T22:55:22.964 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T22:55:23.002 INFO:teuthology.orchestra.run.smithi044.stderr:bash: line 1: ntpq: command not found 2024-04-21T22:55:23.005 INFO:teuthology.orchestra.run.smithi044.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T22:55:23.005 INFO:teuthology.orchestra.run.smithi044.stdout:=============================================================================== 2024-04-21T22:55:23.005 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T22:55:23.005 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T22:55:23.006 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T22:55:23.006 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T22:55:23.013 INFO:teuthology.orchestra.run.smithi055.stderr:bash: line 1: ntpq: command not found 2024-04-21T22:55:23.017 INFO:teuthology.orchestra.run.smithi055.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T22:55:23.017 INFO:teuthology.orchestra.run.smithi055.stdout:=============================================================================== 2024-04-21T22:55:23.017 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T22:55:23.017 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T22:55:23.017 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T22:55:23.017 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T22:55:23.023 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-04-21T22:55:23.027 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T22:55:23.027 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-04-21T22:55:23.027 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T22:55:23.028 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T22:55:23.028 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T22:55:23.028 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T22:55:23.028 INFO:teuthology.run_tasks:Running task install... 2024-04-21T22:55:23.036 DEBUG:teuthology.task.install:project ceph 2024-04-21T22:55:23.036 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b'}} 2024-04-21T22:55:23.037 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b'} 2024-04-21T22:55:23.037 INFO:teuthology.task.install:Using flavor: default 2024-04-21T22:55:23.043 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-21T22:55:23.043 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-21T22:55:23.043 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'tag': None, 'wait_for_package': False} 2024-04-21T22:55:23.044 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T22:55:23.046 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'tag': None, 'wait_for_package': False} 2024-04-21T22:55:23.046 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T22:55:23.047 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'tag': None, 'wait_for_package': False} 2024-04-21T22:55:23.047 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T22:55:23.196 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/ 2024-04-21T22:55:23.196 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2405.g820a2f8c 2024-04-21T22:55:23.263 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/ 2024-04-21T22:55:23.263 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2405.g820a2f8c 2024-04-21T22:55:23.289 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/ 2024-04-21T22:55:23.290 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2405.g820a2f8c 2024-04-21T22:55:23.312 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T22:55:23.313 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T22:55:23.313 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T22:55:23.339 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T22:55:23.339 DEBUG:teuthology.orchestra.run.smithi044:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/820a2f8ced66152ed3d765c31444b8076ea28c6b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T22:55:23.405 DEBUG:teuthology.orchestra.run.smithi044:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T22:55:23.433 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T22:55:23.433 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-21T22:55:23.433 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T22:55:23.461 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T22:55:23.462 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-21T22:55:23.462 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T22:55:23.463 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T22:55:23.463 DEBUG:teuthology.orchestra.run.smithi055:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/820a2f8ced66152ed3d765c31444b8076ea28c6b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T22:55:23.481 DEBUG:teuthology.orchestra.run.smithi044:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T22:55:23.488 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T22:55:23.488 DEBUG:teuthology.orchestra.run.smithi069:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/820a2f8ced66152ed3d765c31444b8076ea28c6b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T22:55:23.532 DEBUG:teuthology.orchestra.run.smithi055:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T22:55:23.545 INFO:teuthology.orchestra.run.smithi044.stdout:check_obsoletes = 1 2024-04-21T22:55:23.547 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum clean all 2024-04-21T22:55:23.552 DEBUG:teuthology.orchestra.run.smithi069:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T22:55:23.611 DEBUG:teuthology.orchestra.run.smithi055:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T22:55:23.626 DEBUG:teuthology.orchestra.run.smithi069:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T22:55:23.679 INFO:teuthology.orchestra.run.smithi055.stdout:check_obsoletes = 1 2024-04-21T22:55:23.681 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum clean all 2024-04-21T22:55:23.688 INFO:teuthology.orchestra.run.smithi069.stdout:check_obsoletes = 1 2024-04-21T22:55:23.690 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum clean all 2024-04-21T22:55:23.783 INFO:teuthology.orchestra.run.smithi044.stdout:48 files removed 2024-04-21T22:55:23.811 DEBUG:teuthology.orchestra.run.smithi044:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T22:55:23.912 INFO:teuthology.orchestra.run.smithi069.stdout:48 files removed 2024-04-21T22:55:23.921 INFO:teuthology.orchestra.run.smithi055.stdout:48 files removed 2024-04-21T22:55:23.938 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T22:55:23.949 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T22:55:24.439 INFO:teuthology.orchestra.run.smithi044.stdout:ceph packages for x86_64 233 kB/s | 83 kB 00:00 2024-04-21T22:55:24.610 INFO:teuthology.orchestra.run.smithi055.stdout:ceph packages for x86_64 215 kB/s | 83 kB 00:00 2024-04-21T22:55:24.611 INFO:teuthology.orchestra.run.smithi069.stdout:ceph packages for x86_64 202 kB/s | 83 kB 00:00 2024-04-21T22:55:24.840 INFO:teuthology.orchestra.run.smithi044.stdout:ceph noarch packages 32 kB/s | 11 kB 00:00 2024-04-21T22:55:24.970 INFO:teuthology.orchestra.run.smithi069.stdout:ceph noarch packages 37 kB/s | 11 kB 00:00 2024-04-21T22:55:24.977 INFO:teuthology.orchestra.run.smithi055.stdout:ceph noarch packages 36 kB/s | 11 kB 00:00 2024-04-21T22:55:25.199 INFO:teuthology.orchestra.run.smithi044.stdout:ceph source packages 5.7 kB/s | 1.8 kB 00:00 2024-04-21T22:55:25.320 INFO:teuthology.orchestra.run.smithi069.stdout:ceph source packages 5.9 kB/s | 1.8 kB 00:00 2024-04-21T22:55:25.411 INFO:teuthology.orchestra.run.smithi055.stdout:ceph source packages 4.6 kB/s | 1.8 kB 00:00 2024-04-21T22:55:25.766 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-21T22:55:28.020 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.0 MB 00:02 2024-04-21T22:55:28.117 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-04-21T22:55:28.661 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.0 MB 00:03 2024-04-21T22:55:30.529 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-21T22:55:32.246 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 6.0 MB/s | 19 MB 00:03 2024-04-21T22:55:32.351 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-21T22:55:33.802 INFO:teuthology.orchestra.run.smithi044.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 16 kB 00:00 2024-04-21T22:55:34.832 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-21T22:55:36.539 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-04-21T22:55:37.993 INFO:teuthology.orchestra.run.smithi055.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-21T22:55:39.805 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.2 MB 00:03 2024-04-21T22:55:41.372 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-04-21T22:55:43.034 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-21T22:55:43.274 INFO:teuthology.orchestra.run.smithi055.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-21T22:55:45.484 INFO:teuthology.orchestra.run.smithi055.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T22:55:45.485 INFO:teuthology.orchestra.run.smithi055.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T22:55:45.496 INFO:teuthology.orchestra.run.smithi055.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T22:55:45.497 INFO:teuthology.orchestra.run.smithi055.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T22:55:45.497 INFO:teuthology.orchestra.run.smithi055.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T22:55:45.498 INFO:teuthology.orchestra.run.smithi055.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T22:55:45.500 INFO:teuthology.orchestra.run.smithi055.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T22:55:45.502 INFO:teuthology.orchestra.run.smithi055.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T22:55:45.503 INFO:teuthology.orchestra.run.smithi055.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T22:55:45.567 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2024-04-21T22:55:45.585 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================= 2024-04-21T22:55:45.586 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2024-04-21T22:55:45.586 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================= 2024-04-21T22:55:45.586 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2024-04-21T22:55:45.586 INFO:teuthology.orchestra.run.smithi055.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T22:55:45.586 INFO:teuthology.orchestra.run.smithi055.stdout: ceph x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 6.6 k 2024-04-21T22:55:45.586 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.4 M 2024-04-21T22:55:45.586 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 889 k 2024-04-21T22:55:45.586 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 145 k 2024-04-21T22:55:45.586 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 1.5 M 2024-04-21T22:55:45.586 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 141 k 2024-04-21T22:55:45.586 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 3.5 M 2024-04-21T22:55:45.587 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 7.4 M 2024-04-21T22:55:45.587 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-rook noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 49 k 2024-04-21T22:55:45.587 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 10 M 2024-04-21T22:55:45.587 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 48 M 2024-04-21T22:55:45.587 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-volume noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 264 k 2024-04-21T22:55:45.587 INFO:teuthology.orchestra.run.smithi055.stdout: cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 762 k 2024-04-21T22:55:45.587 INFO:teuthology.orchestra.run.smithi055.stdout: cephfs-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 222 k 2024-04-21T22:55:45.587 INFO:teuthology.orchestra.run.smithi055.stdout: cephfs-shell x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 27 k 2024-04-21T22:55:45.587 INFO:teuthology.orchestra.run.smithi055.stdout: cephfs-top noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 18 k 2024-04-21T22:55:45.587 INFO:teuthology.orchestra.run.smithi055.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T22:55:45.587 INFO:teuthology.orchestra.run.smithi055.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T22:55:45.587 INFO:teuthology.orchestra.run.smithi055.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T22:55:45.587 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 32 k 2024-04-21T22:55:45.588 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 745 k 2024-04-21T22:55:45.588 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 127 k 2024-04-21T22:55:45.588 INFO:teuthology.orchestra.run.smithi055.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T22:55:45.588 INFO:teuthology.orchestra.run.smithi055.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T22:55:45.588 INFO:teuthology.orchestra.run.smithi055.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T22:55:45.588 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T22:55:45.588 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T22:55:45.588 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T22:55:45.588 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 161 k 2024-04-21T22:55:45.588 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 321 k 2024-04-21T22:55:45.588 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 298 k 2024-04-21T22:55:45.589 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 99 k 2024-04-21T22:55:45.589 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 86 k 2024-04-21T22:55:45.589 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-21T22:55:45.589 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 171 k 2024-04-21T22:55:45.589 INFO:teuthology.orchestra.run.smithi055.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T22:55:45.589 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrading: 2024-04-21T22:55:45.589 INFO:teuthology.orchestra.run.smithi055.stdout: librados2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.4 M 2024-04-21T22:55:45.589 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-21T22:55:45.589 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2024-04-21T22:55:45.589 INFO:teuthology.orchestra.run.smithi055.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T22:55:45.589 INFO:teuthology.orchestra.run.smithi055.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T22:55:45.589 INFO:teuthology.orchestra.run.smithi055.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T22:55:45.589 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 21 M 2024-04-21T22:55:45.590 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 32 k 2024-04-21T22:55:45.590 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 2.4 M 2024-04-21T22:55:45.590 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 249 k 2024-04-21T22:55:45.590 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 4.7 M 2024-04-21T22:55:45.590 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 17 M 2024-04-21T22:55:45.590 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 15 k 2024-04-21T22:55:45.590 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-selinux x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 25 k 2024-04-21T22:55:45.590 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T22:55:45.590 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T22:55:45.590 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T22:55:45.590 INFO:teuthology.orchestra.run.smithi055.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T22:55:45.591 INFO:teuthology.orchestra.run.smithi055.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T22:55:45.591 INFO:teuthology.orchestra.run.smithi055.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T22:55:45.591 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T22:55:45.591 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T22:55:45.591 INFO:teuthology.orchestra.run.smithi055.stdout: libcephsqlite x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 164 k 2024-04-21T22:55:45.591 INFO:teuthology.orchestra.run.smithi055.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T22:55:45.591 INFO:teuthology.orchestra.run.smithi055.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T22:55:45.591 INFO:teuthology.orchestra.run.smithi055.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T22:55:45.591 INFO:teuthology.orchestra.run.smithi055.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T22:55:45.591 INFO:teuthology.orchestra.run.smithi055.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T22:55:45.591 INFO:teuthology.orchestra.run.smithi055.stdout: libradosstriper1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 501 k 2024-04-21T22:55:45.591 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T22:55:45.591 INFO:teuthology.orchestra.run.smithi055.stdout: librgw2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.3 M 2024-04-21T22:55:45.592 INFO:teuthology.orchestra.run.smithi055.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T22:55:45.592 INFO:teuthology.orchestra.run.smithi055.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T22:55:45.592 INFO:teuthology.orchestra.run.smithi055.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T22:55:45.592 INFO:teuthology.orchestra.run.smithi055.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T22:55:45.592 INFO:teuthology.orchestra.run.smithi055.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T22:55:45.592 INFO:teuthology.orchestra.run.smithi055.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T22:55:45.592 INFO:teuthology.orchestra.run.smithi055.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T22:55:45.592 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T22:55:45.592 INFO:teuthology.orchestra.run.smithi055.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T22:55:45.592 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T22:55:45.592 INFO:teuthology.orchestra.run.smithi055.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T22:55:45.592 INFO:teuthology.orchestra.run.smithi055.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T22:55:45.592 INFO:teuthology.orchestra.run.smithi055.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T22:55:45.592 INFO:teuthology.orchestra.run.smithi055.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T22:55:45.593 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T22:55:45.593 INFO:teuthology.orchestra.run.smithi055.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T22:55:45.593 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T22:55:45.593 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 45 k 2024-04-21T22:55:45.593 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 133 k 2024-04-21T22:55:45.593 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T22:55:45.593 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T22:55:45.593 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T22:55:45.593 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T22:55:45.593 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T22:55:45.593 INFO:teuthology.orchestra.run.smithi055.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T22:55:45.593 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T22:55:45.593 INFO:teuthology.orchestra.run.smithi055.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T22:55:45.593 INFO:teuthology.orchestra.run.smithi055.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T22:55:45.594 INFO:teuthology.orchestra.run.smithi055.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T22:55:45.594 INFO:teuthology.orchestra.run.smithi055.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T22:55:45.594 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T22:55:45.594 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T22:55:45.594 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T22:55:45.594 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T22:55:45.594 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T22:55:45.594 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T22:55:45.594 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T22:55:45.594 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T22:55:45.594 INFO:teuthology.orchestra.run.smithi055.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T22:55:45.594 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T22:55:45.594 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T22:55:45.595 INFO:teuthology.orchestra.run.smithi055.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T22:55:45.595 INFO:teuthology.orchestra.run.smithi055.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T22:55:45.595 INFO:teuthology.orchestra.run.smithi055.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T22:55:45.595 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T22:55:45.595 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T22:55:45.595 INFO:teuthology.orchestra.run.smithi055.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T22:55:45.595 INFO:teuthology.orchestra.run.smithi055.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T22:55:45.595 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T22:55:45.595 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T22:55:45.595 INFO:teuthology.orchestra.run.smithi055.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T22:55:45.595 INFO:teuthology.orchestra.run.smithi055.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T22:55:45.595 INFO:teuthology.orchestra.run.smithi055.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T22:55:45.595 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T22:55:45.595 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T22:55:45.595 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T22:55:45.596 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T22:55:45.596 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T22:55:45.596 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T22:55:45.596 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T22:55:45.596 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T22:55:45.596 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T22:55:45.596 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T22:55:45.596 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T22:55:45.596 INFO:teuthology.orchestra.run.smithi055.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T22:55:45.596 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T22:55:45.596 INFO:teuthology.orchestra.run.smithi055.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T22:55:45.596 INFO:teuthology.orchestra.run.smithi055.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T22:55:45.596 INFO:teuthology.orchestra.run.smithi055.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T22:55:45.596 INFO:teuthology.orchestra.run.smithi055.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T22:55:45.597 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T22:55:45.597 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T22:55:45.597 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T22:55:45.597 INFO:teuthology.orchestra.run.smithi055.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T22:55:45.597 INFO:teuthology.orchestra.run.smithi055.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T22:55:45.597 INFO:teuthology.orchestra.run.smithi055.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T22:55:45.597 INFO:teuthology.orchestra.run.smithi055.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T22:55:45.597 INFO:teuthology.orchestra.run.smithi055.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T22:55:45.597 INFO:teuthology.orchestra.run.smithi055.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T22:55:45.597 INFO:teuthology.orchestra.run.smithi055.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T22:55:45.597 INFO:teuthology.orchestra.run.smithi055.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T22:55:45.597 INFO:teuthology.orchestra.run.smithi055.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T22:55:45.597 INFO:teuthology.orchestra.run.smithi055.stdout:Installing weak dependencies: 2024-04-21T22:55:45.597 INFO:teuthology.orchestra.run.smithi055.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T22:55:45.598 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-21T22:55:45.598 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2024-04-21T22:55:45.598 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================= 2024-04-21T22:55:45.598 INFO:teuthology.orchestra.run.smithi055.stdout:Install 148 Packages 2024-04-21T22:55:45.598 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrade 2 Packages 2024-04-21T22:55:45.598 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-21T22:55:45.598 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 216 M 2024-04-21T22:55:45.598 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2024-04-21T22:55:46.409 INFO:teuthology.orchestra.run.smithi055.stdout:(1/150): ceph-19.0.0-2405.g820a2f8c.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-04-21T22:55:46.634 INFO:teuthology.orchestra.run.smithi055.stdout:(2/150): ceph-fuse-19.0.0-2405.g820a2f8c.el9.x8 3.9 MB/s | 889 kB 00:00 2024-04-21T22:55:46.701 INFO:teuthology.orchestra.run.smithi055.stdout:(3/150): ceph-immutable-object-cache-19.0.0-240 2.1 MB/s | 145 kB 00:00 2024-04-21T22:55:46.801 INFO:teuthology.orchestra.run.smithi055.stdout:(4/150): ceph-base-19.0.0-2405.g820a2f8c.el9.x8 9.4 MB/s | 5.4 MB 00:00 2024-04-21T22:55:46.935 INFO:teuthology.orchestra.run.smithi055.stdout:(5/150): ceph-mds-19.0.0-2405.g820a2f8c.el9.x86 10 MB/s | 2.4 MB 00:00 2024-04-21T22:55:46.969 INFO:teuthology.orchestra.run.smithi055.stdout:(6/150): ceph-mgr-19.0.0-2405.g820a2f8c.el9.x86 9.0 MB/s | 1.5 MB 00:00 2024-04-21T22:55:47.385 INFO:teuthology.orchestra.run.smithi055.stdout:(7/150): ceph-mon-19.0.0-2405.g820a2f8c.el9.x86 10 MB/s | 4.7 MB 00:00 2024-04-21T22:55:48.090 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-21T22:55:48.269 INFO:teuthology.orchestra.run.smithi055.stdout:(8/150): ceph-osd-19.0.0-2405.g820a2f8c.el9.x86 13 MB/s | 17 MB 00:01 2024-04-21T22:55:48.402 INFO:teuthology.orchestra.run.smithi055.stdout:(9/150): ceph-radosgw-19.0.0-2405.g820a2f8c.el9 10 MB/s | 10 MB 00:01 2024-04-21T22:55:48.430 INFO:teuthology.orchestra.run.smithi055.stdout:(10/150): ceph-selinux-19.0.0-2405.g820a2f8c.el 157 kB/s | 25 kB 00:00 2024-04-21T22:55:48.488 INFO:teuthology.orchestra.run.smithi055.stdout:(11/150): cephfs-mirror-19.0.0-2405.g820a2f8c.e 3.7 MB/s | 222 kB 00:00 2024-04-21T22:55:48.547 INFO:teuthology.orchestra.run.smithi055.stdout:(12/150): cephfs-shell-19.0.0-2405.g820a2f8c.el 462 kB/s | 27 kB 00:00 2024-04-21T22:55:48.614 INFO:teuthology.orchestra.run.smithi055.stdout:(13/150): libcephfs-devel-19.0.0-2405.g820a2f8c 477 kB/s | 32 kB 00:00 2024-04-21T22:55:48.723 INFO:teuthology.orchestra.run.smithi055.stdout:(14/150): libcephfs2-19.0.0-2405.g820a2f8c.el9. 6.7 MB/s | 745 kB 00:00 2024-04-21T22:55:48.798 INFO:teuthology.orchestra.run.smithi055.stdout:(15/150): libcephsqlite-19.0.0-2405.g820a2f8c.e 2.1 MB/s | 164 kB 00:00 2024-04-21T22:55:48.881 INFO:teuthology.orchestra.run.smithi055.stdout:(16/150): librados-devel-19.0.0-2405.g820a2f8c. 1.5 MB/s | 127 kB 00:00 2024-04-21T22:55:48.974 INFO:teuthology.orchestra.run.smithi055.stdout:(17/150): libradosstriper1-19.0.0-2405.g820a2f8 5.3 MB/s | 501 kB 00:00 2024-04-21T22:55:49.566 INFO:teuthology.orchestra.run.smithi055.stdout:(18/150): librgw2-19.0.0-2405.g820a2f8c.el9.x86 9.0 MB/s | 5.3 MB 00:00 2024-04-21T22:55:49.783 INFO:teuthology.orchestra.run.smithi055.stdout:(19/150): ceph-common-19.0.0-2405.g820a2f8c.el9 5.9 MB/s | 21 MB 00:03 2024-04-21T22:55:49.808 INFO:teuthology.orchestra.run.smithi055.stdout:(20/150): python3-ceph-argparse-19.0.0-2405.g82 187 kB/s | 45 kB 00:00 2024-04-21T22:55:49.858 INFO:teuthology.orchestra.run.smithi055.stdout:(21/150): python3-ceph-common-19.0.0-2405.g820a 1.7 MB/s | 133 kB 00:00 2024-04-21T22:55:49.884 INFO:teuthology.orchestra.run.smithi055.stdout:(22/150): python3-cephfs-19.0.0-2405.g820a2f8c. 2.1 MB/s | 161 kB 00:00 2024-04-21T22:55:49.942 INFO:teuthology.orchestra.run.smithi055.stdout:(23/150): python3-rados-19.0.0-2405.g820a2f8c.e 3.8 MB/s | 321 kB 00:00 2024-04-21T22:55:49.968 INFO:teuthology.orchestra.run.smithi055.stdout:(24/150): python3-rbd-19.0.0-2405.g820a2f8c.el9 3.5 MB/s | 298 kB 00:00 2024-04-21T22:55:50.018 INFO:teuthology.orchestra.run.smithi055.stdout:(25/150): python3-rgw-19.0.0-2405.g820a2f8c.el9 1.3 MB/s | 99 kB 00:00 2024-04-21T22:55:50.043 INFO:teuthology.orchestra.run.smithi055.stdout:(26/150): rbd-fuse-19.0.0-2405.g820a2f8c.el9.x8 1.1 MB/s | 86 kB 00:00 2024-04-21T22:55:50.119 INFO:teuthology.orchestra.run.smithi055.stdout:(27/150): rbd-nbd-19.0.0-2405.g820a2f8c.el9.x86 2.2 MB/s | 171 kB 00:00 2024-04-21T22:55:50.188 INFO:teuthology.orchestra.run.smithi055.stdout:(28/150): ceph-grafana-dashboards-19.0.0-2405.g 477 kB/s | 32 kB 00:00 2024-04-21T22:55:50.242 INFO:teuthology.orchestra.run.smithi069.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T22:55:50.243 INFO:teuthology.orchestra.run.smithi069.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T22:55:50.253 INFO:teuthology.orchestra.run.smithi055.stdout:(29/150): ceph-mgr-cephadm-19.0.0-2405.g820a2f8 2.1 MB/s | 141 kB 00:00 2024-04-21T22:55:50.253 INFO:teuthology.orchestra.run.smithi069.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T22:55:50.254 INFO:teuthology.orchestra.run.smithi069.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T22:55:50.255 INFO:teuthology.orchestra.run.smithi069.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T22:55:50.255 INFO:teuthology.orchestra.run.smithi069.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T22:55:50.257 INFO:teuthology.orchestra.run.smithi069.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T22:55:50.259 INFO:teuthology.orchestra.run.smithi069.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T22:55:50.260 INFO:teuthology.orchestra.run.smithi069.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T22:55:50.318 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-21T22:55:50.336 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2024-04-21T22:55:50.336 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-04-21T22:55:50.336 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2024-04-21T22:55:50.337 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-21T22:55:50.337 INFO:teuthology.orchestra.run.smithi069.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T22:55:50.337 INFO:teuthology.orchestra.run.smithi069.stdout: ceph x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 6.6 k 2024-04-21T22:55:50.337 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.4 M 2024-04-21T22:55:50.337 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 889 k 2024-04-21T22:55:50.337 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 145 k 2024-04-21T22:55:50.337 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 1.5 M 2024-04-21T22:55:50.337 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 141 k 2024-04-21T22:55:50.337 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 3.5 M 2024-04-21T22:55:50.337 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 7.4 M 2024-04-21T22:55:50.337 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 49 k 2024-04-21T22:55:50.337 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 10 M 2024-04-21T22:55:50.337 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 48 M 2024-04-21T22:55:50.338 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 264 k 2024-04-21T22:55:50.338 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 762 k 2024-04-21T22:55:50.338 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 222 k 2024-04-21T22:55:50.338 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-shell x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 27 k 2024-04-21T22:55:50.338 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-top noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 18 k 2024-04-21T22:55:50.338 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T22:55:50.338 INFO:teuthology.orchestra.run.smithi069.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T22:55:50.338 INFO:teuthology.orchestra.run.smithi069.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T22:55:50.338 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 32 k 2024-04-21T22:55:50.338 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 745 k 2024-04-21T22:55:50.338 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 127 k 2024-04-21T22:55:50.338 INFO:teuthology.orchestra.run.smithi069.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T22:55:50.338 INFO:teuthology.orchestra.run.smithi069.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T22:55:50.339 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T22:55:50.339 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T22:55:50.339 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T22:55:50.339 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T22:55:50.339 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 161 k 2024-04-21T22:55:50.339 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 321 k 2024-04-21T22:55:50.339 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 298 k 2024-04-21T22:55:50.340 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 99 k 2024-04-21T22:55:50.340 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 86 k 2024-04-21T22:55:50.340 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-21T22:55:50.340 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 171 k 2024-04-21T22:55:50.340 INFO:teuthology.orchestra.run.smithi069.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T22:55:50.340 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrading: 2024-04-21T22:55:50.340 INFO:teuthology.orchestra.run.smithi069.stdout: librados2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.4 M 2024-04-21T22:55:50.340 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-21T22:55:50.341 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-04-21T22:55:50.341 INFO:teuthology.orchestra.run.smithi069.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T22:55:50.341 INFO:teuthology.orchestra.run.smithi069.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T22:55:50.341 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T22:55:50.341 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 21 M 2024-04-21T22:55:50.341 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 32 k 2024-04-21T22:55:50.342 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 2.4 M 2024-04-21T22:55:50.342 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 249 k 2024-04-21T22:55:50.342 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 4.7 M 2024-04-21T22:55:50.342 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 17 M 2024-04-21T22:55:50.343 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 15 k 2024-04-21T22:55:50.343 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 25 k 2024-04-21T22:55:50.343 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T22:55:50.343 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T22:55:50.343 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T22:55:50.343 INFO:teuthology.orchestra.run.smithi069.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T22:55:50.344 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T22:55:50.344 INFO:teuthology.orchestra.run.smithi069.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T22:55:50.344 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T22:55:50.344 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T22:55:50.344 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 164 k 2024-04-21T22:55:50.344 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T22:55:50.344 INFO:teuthology.orchestra.run.smithi069.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T22:55:50.344 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T22:55:50.345 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T22:55:50.345 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T22:55:50.345 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 501 k 2024-04-21T22:55:50.345 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T22:55:50.345 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.3 M 2024-04-21T22:55:50.345 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T22:55:50.345 INFO:teuthology.orchestra.run.smithi069.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T22:55:50.345 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T22:55:50.346 INFO:teuthology.orchestra.run.smithi069.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T22:55:50.346 INFO:teuthology.orchestra.run.smithi069.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T22:55:50.346 INFO:teuthology.orchestra.run.smithi069.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T22:55:50.346 INFO:teuthology.orchestra.run.smithi069.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T22:55:50.346 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T22:55:50.346 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T22:55:50.346 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T22:55:50.346 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T22:55:50.347 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T22:55:50.347 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T22:55:50.347 INFO:teuthology.orchestra.run.smithi069.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T22:55:50.347 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T22:55:50.347 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T22:55:50.347 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T22:55:50.347 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 45 k 2024-04-21T22:55:50.347 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 133 k 2024-04-21T22:55:50.348 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T22:55:50.348 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T22:55:50.348 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T22:55:50.348 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T22:55:50.348 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T22:55:50.348 INFO:teuthology.orchestra.run.smithi069.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T22:55:50.348 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T22:55:50.348 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T22:55:50.348 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T22:55:50.348 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T22:55:50.348 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T22:55:50.349 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T22:55:50.349 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T22:55:50.349 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T22:55:50.349 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T22:55:50.349 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T22:55:50.349 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T22:55:50.349 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T22:55:50.349 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T22:55:50.349 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T22:55:50.349 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T22:55:50.349 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T22:55:50.349 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T22:55:50.349 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T22:55:50.350 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T22:55:50.350 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T22:55:50.350 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T22:55:50.350 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T22:55:50.350 INFO:teuthology.orchestra.run.smithi069.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T22:55:50.350 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T22:55:50.350 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T22:55:50.350 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T22:55:50.350 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T22:55:50.350 INFO:teuthology.orchestra.run.smithi069.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T22:55:50.350 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T22:55:50.350 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T22:55:50.350 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T22:55:50.351 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T22:55:50.351 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T22:55:50.351 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T22:55:50.351 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T22:55:50.351 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T22:55:50.351 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T22:55:50.351 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T22:55:50.351 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T22:55:50.351 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T22:55:50.351 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T22:55:50.351 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T22:55:50.351 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T22:55:50.351 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T22:55:50.352 INFO:teuthology.orchestra.run.smithi069.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T22:55:50.352 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T22:55:50.352 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T22:55:50.352 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T22:55:50.352 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T22:55:50.352 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T22:55:50.352 INFO:teuthology.orchestra.run.smithi069.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T22:55:50.352 INFO:teuthology.orchestra.run.smithi069.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T22:55:50.352 INFO:teuthology.orchestra.run.smithi069.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T22:55:50.352 INFO:teuthology.orchestra.run.smithi069.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T22:55:50.352 INFO:teuthology.orchestra.run.smithi069.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T22:55:50.352 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T22:55:50.352 INFO:teuthology.orchestra.run.smithi069.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T22:55:50.353 INFO:teuthology.orchestra.run.smithi069.stdout:Installing weak dependencies: 2024-04-21T22:55:50.353 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T22:55:50.353 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-21T22:55:50.353 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-21T22:55:50.353 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2024-04-21T22:55:50.353 INFO:teuthology.orchestra.run.smithi069.stdout:Install 148 Packages 2024-04-21T22:55:50.353 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrade 2 Packages 2024-04-21T22:55:50.353 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-21T22:55:50.354 INFO:teuthology.orchestra.run.smithi055.stdout:(30/150): rbd-mirror-19.0.0-2405.g820a2f8c.el9. 9.5 MB/s | 3.1 MB 00:00 2024-04-21T22:55:50.354 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 216 M 2024-04-21T22:55:50.354 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-21T22:55:50.637 INFO:teuthology.orchestra.run.smithi055.stdout:(31/150): ceph-mgr-dashboard-19.0.0-2405.g820a2 9.2 MB/s | 3.5 MB 00:00 2024-04-21T22:55:50.703 INFO:teuthology.orchestra.run.smithi055.stdout:(32/150): ceph-mgr-modules-core-19.0.0-2405.g82 3.7 MB/s | 249 kB 00:00 2024-04-21T22:55:50.770 INFO:teuthology.orchestra.run.smithi055.stdout:(33/150): ceph-mgr-rook-19.0.0-2405.g820a2f8c.e 743 kB/s | 49 kB 00:00 2024-04-21T22:55:50.838 INFO:teuthology.orchestra.run.smithi055.stdout:(34/150): ceph-prometheus-alerts-19.0.0-2405.g8 230 kB/s | 15 kB 00:00 2024-04-21T22:55:50.921 INFO:teuthology.orchestra.run.smithi055.stdout:(35/150): ceph-volume-19.0.0-2405.g820a2f8c.el9 3.1 MB/s | 264 kB 00:00 2024-04-21T22:55:51.054 INFO:teuthology.orchestra.run.smithi055.stdout:(36/150): cephadm-19.0.0-2405.g820a2f8c.el9.noa 5.6 MB/s | 762 kB 00:00 2024-04-21T22:55:51.149 INFO:teuthology.orchestra.run.smithi069.stdout:(1/150): ceph-19.0.0-2405.g820a2f8c.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-04-21T22:55:51.155 INFO:teuthology.orchestra.run.smithi055.stdout:(37/150): ceph-mgr-diskprediction-local-19.0.0- 9.1 MB/s | 7.4 MB 00:00 2024-04-21T22:55:51.180 INFO:teuthology.orchestra.run.smithi055.stdout:(38/150): cephfs-top-19.0.0-2405.g820a2f8c.el9. 145 kB/s | 18 kB 00:00 2024-04-21T22:55:51.247 INFO:teuthology.orchestra.run.smithi055.stdout:(39/150): fuse-2.9.9-15.el9.x86_64.rpm 877 kB/s | 80 kB 00:00 2024-04-21T22:55:51.322 INFO:teuthology.orchestra.run.smithi055.stdout:(40/150): libgfortran-11.4.1-3.el9.x86_64.rpm 5.5 MB/s | 803 kB 00:00 2024-04-21T22:55:51.422 INFO:teuthology.orchestra.run.smithi055.stdout:(41/150): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-04-21T22:55:51.473 INFO:teuthology.orchestra.run.smithi055.stdout:(42/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-04-21T22:55:51.532 INFO:teuthology.orchestra.run.smithi055.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-21T22:55:51.590 INFO:teuthology.orchestra.run.smithi055.stdout:(44/150): python3-cryptography-36.0.1-4.el9.x86 7.3 MB/s | 1.2 MB 00:00 2024-04-21T22:55:51.632 INFO:teuthology.orchestra.run.smithi055.stdout:(45/150): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 135 kB 00:00 2024-04-21T22:55:51.675 INFO:teuthology.orchestra.run.smithi055.stdout:(46/150): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-04-21T22:55:51.724 INFO:teuthology.orchestra.run.smithi055.stdout:(47/150): python3-urllib3-1.26.5-5.el9.noarch.r 2.3 MB/s | 215 kB 00:00 2024-04-21T22:55:51.824 INFO:teuthology.orchestra.run.smithi069.stdout:(2/150): ceph-base-19.0.0-2405.g820a2f8c.el9.x8 5.5 MB/s | 5.4 MB 00:00 2024-04-21T22:55:51.841 INFO:teuthology.orchestra.run.smithi055.stdout:(48/150): automake-1.16.2-8.el9.noarch.rpm 4.0 MB/s | 689 kB 00:00 2024-04-21T22:55:51.858 INFO:teuthology.orchestra.run.smithi069.stdout:(3/150): ceph-fuse-19.0.0-2405.g820a2f8c.el9.x8 1.2 MB/s | 889 kB 00:00 2024-04-21T22:55:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:(4/150): ceph-immutable-object-cache-19.0.0-240 2.4 MB/s | 145 kB 00:00 2024-04-21T22:55:52.008 INFO:teuthology.orchestra.run.smithi069.stdout:(5/150): ceph-mgr-19.0.0-2405.g820a2f8c.el9.x86 12 MB/s | 1.5 MB 00:00 2024-04-21T22:55:52.275 INFO:teuthology.orchestra.run.smithi055.stdout:(49/150): ceph-test-19.0.0-2405.g820a2f8c.el9.x 12 MB/s | 48 MB 00:03 2024-04-21T22:55:52.292 INFO:teuthology.orchestra.run.smithi055.stdout:(50/150): boost-program-options-1.75.0-8.el9.x8 237 kB/s | 107 kB 00:00 2024-04-21T22:55:52.334 INFO:teuthology.orchestra.run.smithi069.stdout:(6/150): ceph-mon-19.0.0-2405.g820a2f8c.el9.x86 14 MB/s | 4.7 MB 00:00 2024-04-21T22:55:52.342 INFO:teuthology.orchestra.run.smithi055.stdout:(51/150): bison-3.7.4-5.el9.x86_64.rpm 1.5 MB/s | 943 kB 00:00 2024-04-21T22:55:52.359 INFO:teuthology.orchestra.run.smithi055.stdout:(52/150): elfutils-libelf-devel-0.190-2.el9.x86 258 kB/s | 21 kB 00:00 2024-04-21T22:55:52.376 INFO:teuthology.orchestra.run.smithi055.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 980 kB/s | 33 kB 00:00 2024-04-21T22:55:52.401 INFO:teuthology.orchestra.run.smithi055.stdout:(54/150): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-04-21T22:55:52.418 INFO:teuthology.orchestra.run.smithi055.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 435 kB/s | 18 kB 00:00 2024-04-21T22:55:52.435 INFO:teuthology.orchestra.run.smithi055.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 332 kB/s | 11 kB 00:00 2024-04-21T22:55:52.452 INFO:teuthology.orchestra.run.smithi055.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 160 kB 00:00 2024-04-21T22:55:52.469 INFO:teuthology.orchestra.run.smithi055.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-04-21T22:55:52.511 INFO:teuthology.orchestra.run.smithi055.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-04-21T22:55:52.544 INFO:teuthology.orchestra.run.smithi055.stdout:(60/150): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 582 kB 00:00 2024-04-21T22:55:52.570 INFO:teuthology.orchestra.run.smithi055.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 435 kB/s | 25 kB 00:00 2024-04-21T22:55:52.595 INFO:teuthology.orchestra.run.smithi055.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-04-21T22:55:52.620 INFO:teuthology.orchestra.run.smithi055.stdout:(63/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-21T22:55:52.645 INFO:teuthology.orchestra.run.smithi055.stdout:(64/150): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-21T22:55:52.671 INFO:teuthology.orchestra.run.smithi055.stdout:(65/150): openblas-0.3.21-2.el9.x86_64.rpm 704 kB/s | 35 kB 00:00 2024-04-21T22:55:52.702 INFO:teuthology.orchestra.run.smithi069.stdout:(7/150): ceph-mds-19.0.0-2405.g820a2f8c.el9.x86 2.8 MB/s | 2.4 MB 00:00 2024-04-21T22:55:52.730 INFO:teuthology.orchestra.run.smithi055.stdout:(66/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 8.1 MB/s | 3.0 MB 00:00 2024-04-21T22:55:52.780 INFO:teuthology.orchestra.run.smithi044.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 21 MB 00:18 2024-04-21T22:55:52.822 INFO:teuthology.orchestra.run.smithi055.stdout:(67/150): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-04-21T22:55:52.864 INFO:teuthology.orchestra.run.smithi055.stdout:(68/150): postgresql-13.14-1.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-21T22:55:52.897 INFO:teuthology.orchestra.run.smithi055.stdout:(69/150): postgresql-contrib-13.14-1.el9.x86_64 12 MB/s | 911 kB 00:00 2024-04-21T22:55:52.923 INFO:teuthology.orchestra.run.smithi055.stdout:(70/150): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-21T22:55:52.982 INFO:teuthology.orchestra.run.smithi055.stdout:(71/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 13 MB/s | 4.1 MB 00:00 2024-04-21T22:55:53.023 INFO:teuthology.orchestra.run.smithi055.stdout:(72/150): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-04-21T22:55:53.048 INFO:teuthology.orchestra.run.smithi055.stdout:(73/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-21T22:55:53.090 INFO:teuthology.orchestra.run.smithi055.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 5.7 MB/s | 244 kB 00:00 2024-04-21T22:55:53.182 INFO:teuthology.orchestra.run.smithi055.stdout:(75/150): postgresql-server-13.14-1.el9.x86_64. 20 MB/s | 5.8 MB 00:00 2024-04-21T22:55:53.207 INFO:teuthology.orchestra.run.smithi055.stdout:(76/150): python3-jinja2-2.11.3-5.el9.noarch.rp 2.1 MB/s | 248 kB 00:00 2024-04-21T22:55:53.233 INFO:teuthology.orchestra.run.smithi055.stdout:(77/150): python3-jsonpatch-1.21-16.el9.noarch. 512 kB/s | 26 kB 00:00 2024-04-21T22:55:53.258 INFO:teuthology.orchestra.run.smithi055.stdout:(78/150): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-04-21T22:55:53.283 INFO:teuthology.orchestra.run.smithi055.stdout:(79/150): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-04-21T22:55:53.312 INFO:teuthology.orchestra.run.smithi055.stdout:(80/150): python3-markupsafe-1.1.1-12.el9.x86_6 693 kB/s | 35 kB 00:00 2024-04-21T22:55:53.359 INFO:teuthology.orchestra.run.smithi069.stdout:(8/150): ceph-osd-19.0.0-2405.g820a2f8c.el9.x86 16 MB/s | 17 MB 00:01 2024-04-21T22:55:53.392 INFO:teuthology.orchestra.run.smithi055.stdout:(81/150): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-04-21T22:55:53.417 INFO:teuthology.orchestra.run.smithi055.stdout:(82/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-04-21T22:55:53.435 INFO:teuthology.orchestra.run.smithi069.stdout:(9/150): ceph-selinux-19.0.0-2405.g820a2f8c.el9 337 kB/s | 25 kB 00:00 2024-04-21T22:55:53.443 INFO:teuthology.orchestra.run.smithi055.stdout:(83/150): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-04-21T22:55:53.460 INFO:teuthology.orchestra.run.smithi055.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 1.8 MB/s | 77 kB 00:00 2024-04-21T22:55:53.485 INFO:teuthology.orchestra.run.smithi055.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-04-21T22:55:53.514 INFO:teuthology.orchestra.run.smithi055.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-04-21T22:55:53.536 INFO:teuthology.orchestra.run.smithi055.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-21T22:55:53.686 INFO:teuthology.orchestra.run.smithi055.stdout:(88/150): python3-pyasn1-modules-0.4.8-6.el9.no 1.6 MB/s | 279 kB 00:00 2024-04-21T22:55:53.703 INFO:teuthology.orchestra.run.smithi055.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 306 kB/s | 51 kB 00:00 2024-04-21T22:55:53.719 INFO:teuthology.orchestra.run.smithi055.stdout:(90/150): python3-requests-oauthlib-1.3.0-12.el 1.6 MB/s | 54 kB 00:00 2024-04-21T22:55:53.754 INFO:teuthology.orchestra.run.smithi055.stdout:(91/150): python3-toml-0.10.2-6.el9.noarch.rpm 1.2 MB/s | 42 kB 00:00 2024-04-21T22:55:53.829 INFO:teuthology.orchestra.run.smithi055.stdout:(92/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.4 MB/s | 5.1 MB 00:00 2024-04-21T22:55:53.845 INFO:teuthology.orchestra.run.smithi055.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 482 kB/s | 44 kB 00:00 2024-04-21T22:55:53.871 INFO:teuthology.orchestra.run.smithi055.stdout:(94/150): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-04-21T22:55:53.888 INFO:teuthology.orchestra.run.smithi055.stdout:(95/150): qatlib-service-24.02.0-1.el9.x86_64.r 855 kB/s | 36 kB 00:00 2024-04-21T22:55:53.905 INFO:teuthology.orchestra.run.smithi055.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-04-21T22:55:53.930 INFO:teuthology.orchestra.run.smithi055.stdout:(97/150): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 305 kB 00:00 2024-04-21T22:55:53.955 INFO:teuthology.orchestra.run.smithi055.stdout:(98/150): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-21T22:55:53.981 INFO:teuthology.orchestra.run.smithi055.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-21T22:55:54.006 INFO:teuthology.orchestra.run.smithi055.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-21T22:55:54.031 INFO:teuthology.orchestra.run.smithi055.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 891 kB/s | 45 kB 00:00 2024-04-21T22:55:54.248 INFO:teuthology.orchestra.run.smithi055.stdout:(102/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-04-21T22:55:54.590 INFO:teuthology.orchestra.run.smithi055.stdout:(103/150): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-04-21T22:55:54.674 INFO:teuthology.orchestra.run.smithi055.stdout:(104/150): gperftools-libs-2.9.1-2.el9.x86_64.r 3.6 MB/s | 309 kB 00:00 2024-04-21T22:55:54.724 INFO:teuthology.orchestra.run.smithi055.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 390 kB/s | 19 kB 00:00 2024-04-21T22:55:54.885 INFO:teuthology.orchestra.run.smithi069.stdout:(10/150): ceph-common-19.0.0-2405.g820a2f8c.el9 5.2 MB/s | 21 MB 00:04 2024-04-21T22:55:54.924 INFO:teuthology.orchestra.run.smithi055.stdout:(106/150): protobuf-compiler-3.14.0-13.el9.x86_ 967 kB/s | 863 kB 00:00 2024-04-21T22:55:54.952 INFO:teuthology.orchestra.run.smithi069.stdout:(11/150): cephfs-mirror-19.0.0-2405.g820a2f8c.e 3.3 MB/s | 222 kB 00:00 2024-04-21T22:55:54.991 INFO:teuthology.orchestra.run.smithi055.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 397 kB/s | 26 kB 00:00 2024-04-21T22:55:55.035 INFO:teuthology.orchestra.run.smithi069.stdout:(12/150): cephfs-shell-19.0.0-2405.g820a2f8c.el 325 kB/s | 27 kB 00:00 2024-04-21T22:55:55.083 INFO:teuthology.orchestra.run.smithi055.stdout:(108/150): liboath-2.6.7-2.el9.x86_64.rpm 535 kB/s | 49 kB 00:00 2024-04-21T22:55:55.144 INFO:teuthology.orchestra.run.smithi069.stdout:(13/150): libcephfs-devel-19.0.0-2405.g820a2f8c 293 kB/s | 32 kB 00:00 2024-04-21T22:55:55.150 INFO:teuthology.orchestra.run.smithi055.stdout:(109/150): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-04-21T22:55:55.175 INFO:teuthology.orchestra.run.smithi055.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 738 kB/s | 67 kB 00:00 2024-04-21T22:55:55.226 INFO:teuthology.orchestra.run.smithi055.stdout:(111/150): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-04-21T22:55:55.309 INFO:teuthology.orchestra.run.smithi055.stdout:(112/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 838 kB 00:00 2024-04-21T22:55:55.327 INFO:teuthology.orchestra.run.smithi069.stdout:(14/150): libcephfs2-19.0.0-2405.g820a2f8c.el9. 4.0 MB/s | 745 kB 00:00 2024-04-21T22:55:55.335 INFO:teuthology.orchestra.run.smithi055.stdout:(113/150): python3-asyncssh-2.13.2-5.el9.noarch 4.9 MB/s | 548 kB 00:00 2024-04-21T22:55:55.360 INFO:teuthology.orchestra.run.smithi055.stdout:(114/150): lua-devel-5.4.4-4.el9.x86_64.rpm 16 kB/s | 22 kB 00:01 2024-04-21T22:55:55.410 INFO:teuthology.orchestra.run.smithi055.stdout:(115/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 437 kB/s | 43 kB 00:00 2024-04-21T22:55:55.428 INFO:teuthology.orchestra.run.smithi069.stdout:(15/150): libcephsqlite-19.0.0-2405.g820a2f8c.e 1.6 MB/s | 164 kB 00:00 2024-04-21T22:55:55.462 INFO:teuthology.orchestra.run.smithi055.stdout:(116/150): python3-cachetools-4.2.4-1.el9.noarc 320 kB/s | 32 kB 00:00 2024-04-21T22:55:55.462 INFO:teuthology.orchestra.run.smithi055.stdout:(117/150): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-04-21T22:55:55.494 INFO:teuthology.orchestra.run.smithi055.stdout:(118/150): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-04-21T22:55:55.536 INFO:teuthology.orchestra.run.smithi055.stdout:(119/150): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-04-21T22:55:55.553 INFO:teuthology.orchestra.run.smithi069.stdout:(16/150): ceph-radosgw-19.0.0-2405.g820a2f8c.el 3.6 MB/s | 10 MB 00:02 2024-04-21T22:55:55.570 INFO:teuthology.orchestra.run.smithi055.stdout:(120/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-04-21T22:55:55.578 INFO:teuthology.orchestra.run.smithi069.stdout:(17/150): librados-devel-19.0.0-2405.g820a2f8c. 846 kB/s | 127 kB 00:00 2024-04-21T22:55:55.595 INFO:teuthology.orchestra.run.smithi055.stdout:(121/150): python3-colorama-0.4.4-12.el9.noarch 351 kB/s | 35 kB 00:00 2024-04-21T22:55:55.620 INFO:teuthology.orchestra.run.smithi055.stdout:(122/150): python3-google-auth-2.28.2-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-04-21T22:55:55.679 INFO:teuthology.orchestra.run.smithi055.stdout:(123/150): python3-jaraco-8.2.1-3.el9.noarch.rp 182 kB/s | 11 kB 00:00 2024-04-21T22:55:55.721 INFO:teuthology.orchestra.run.smithi055.stdout:(124/150): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-04-21T22:55:55.763 INFO:teuthology.orchestra.run.smithi055.stdout:(125/150): python3-jaraco-classes-3.2.1-5.el9.n 214 kB/s | 18 kB 00:00 2024-04-21T22:55:55.804 INFO:teuthology.orchestra.run.smithi069.stdout:(18/150): libradosstriper1-19.0.0-2405.g820a2f8 2.0 MB/s | 501 kB 00:00 2024-04-21T22:55:55.813 INFO:teuthology.orchestra.run.smithi055.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 253 kB/s | 23 kB 00:00 2024-04-21T22:55:55.899 INFO:teuthology.orchestra.run.smithi055.stdout:(127/150): python3-grpcio-1.46.7-10.el9.x86_64. 6.2 MB/s | 2.0 MB 00:00 2024-04-21T22:55:55.922 INFO:teuthology.orchestra.run.smithi055.stdout:(128/150): python3-jaraco-functools-3.5.0-2.el9 123 kB/s | 19 kB 00:00 2024-04-21T22:55:55.937 INFO:teuthology.orchestra.run.smithi069.stdout:(19/150): python3-ceph-argparse-19.0.0-2405.g82 339 kB/s | 45 kB 00:00 2024-04-21T22:55:55.947 INFO:teuthology.orchestra.run.smithi055.stdout:(129/150): python3-jaraco-text-3.2.0-6.el9.noar 147 kB/s | 20 kB 00:00 2024-04-21T22:55:56.054 INFO:teuthology.orchestra.run.smithi069.stdout:(20/150): python3-ceph-common-19.0.0-2405.g820a 1.1 MB/s | 133 kB 00:00 2024-04-21T22:55:56.056 INFO:teuthology.orchestra.run.smithi055.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 347 kB/s | 46 kB 00:00 2024-04-21T22:55:56.098 INFO:teuthology.orchestra.run.smithi055.stdout:(131/150): python3-kubernetes-26.1.0-2.el9.noar 5.8 MB/s | 1.0 MB 00:00 2024-04-21T22:55:56.115 INFO:teuthology.orchestra.run.smithi055.stdout:(132/150): python3-more-itertools-8.12.0-2.el9. 471 kB/s | 79 kB 00:00 2024-04-21T22:55:56.132 INFO:teuthology.orchestra.run.smithi055.stdout:(133/150): python3-natsort-7.1.1-5.el9.noarch.r 773 kB/s | 58 kB 00:00 2024-04-21T22:55:56.165 INFO:teuthology.orchestra.run.smithi055.stdout:(134/150): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-21T22:55:56.199 INFO:teuthology.orchestra.run.smithi055.stdout:(135/150): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2024-04-21T22:55:56.213 INFO:teuthology.orchestra.run.smithi069.stdout:(21/150): librgw2-19.0.0-2405.g820a2f8c.el9.x86 8.4 MB/s | 5.3 MB 00:00 2024-04-21T22:55:56.239 INFO:teuthology.orchestra.run.smithi069.stdout:(22/150): python3-cephfs-19.0.0-2405.g820a2f8c. 877 kB/s | 161 kB 00:00 2024-04-21T22:55:56.283 INFO:teuthology.orchestra.run.smithi055.stdout:(136/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 597 kB/s | 90 kB 00:00 2024-04-21T22:55:56.299 INFO:teuthology.orchestra.run.smithi055.stdout:(137/150): python3-pyperclip-1.8.2-2.el9.noarch 211 kB/s | 28 kB 00:00 2024-04-21T22:55:56.316 INFO:teuthology.orchestra.run.smithi055.stdout:(138/150): python3-repoze-lru-0.7-16.el9.noarch 263 kB/s | 31 kB 00:00 2024-04-21T22:55:56.342 INFO:teuthology.orchestra.run.smithi055.stdout:(139/150): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-04-21T22:55:56.400 INFO:teuthology.orchestra.run.smithi055.stdout:(140/150): python3-rsa-4.9-2.el9.noarch.rpm 712 kB/s | 59 kB 00:00 2024-04-21T22:55:56.417 INFO:teuthology.orchestra.run.smithi055.stdout:(141/150): python3-tempora-5.0.0-2.el9.noarch.r 358 kB/s | 36 kB 00:00 2024-04-21T22:55:56.434 INFO:teuthology.orchestra.run.smithi055.stdout:(142/150): python3-typing-extensions-4.4.0-2.el 554 kB/s | 51 kB 00:00 2024-04-21T22:55:56.459 INFO:teuthology.orchestra.run.smithi055.stdout:(143/150): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-04-21T22:55:56.484 INFO:teuthology.orchestra.run.smithi055.stdout:(144/150): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-04-21T22:55:56.510 INFO:teuthology.orchestra.run.smithi055.stdout:(145/150): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-04-21T22:55:56.535 INFO:teuthology.orchestra.run.smithi055.stdout:(146/150): python3-zc-lockfile-2.0-10.el9.noarc 269 kB/s | 20 kB 00:00 2024-04-21T22:55:56.569 INFO:teuthology.orchestra.run.smithi055.stdout:(147/150): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-04-21T22:55:56.664 INFO:teuthology.orchestra.run.smithi069.stdout:(23/150): ceph-test-19.0.0-2405.g820a2f8c.el9.x 15 MB/s | 48 MB 00:03 2024-04-21T22:55:56.689 INFO:teuthology.orchestra.run.smithi069.stdout:(24/150): python3-rbd-19.0.0-2405.g820a2f8c.el9 661 kB/s | 298 kB 00:00 2024-04-21T22:55:56.711 INFO:teuthology.orchestra.run.smithi055.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-04-21T22:55:56.714 INFO:teuthology.orchestra.run.smithi069.stdout:(25/150): python3-rados-19.0.0-2405.g820a2f8c.e 641 kB/s | 321 kB 00:00 2024-04-21T22:55:56.731 INFO:teuthology.orchestra.run.smithi069.stdout:(26/150): python3-rgw-19.0.0-2405.g820a2f8c.el9 1.5 MB/s | 99 kB 00:00 2024-04-21T22:55:56.756 INFO:teuthology.orchestra.run.smithi069.stdout:(27/150): rbd-fuse-19.0.0-2405.g820a2f8c.el9.x8 1.2 MB/s | 86 kB 00:00 2024-04-21T22:55:56.832 INFO:teuthology.orchestra.run.smithi069.stdout:(28/150): rbd-nbd-19.0.0-2405.g820a2f8c.el9.x86 1.7 MB/s | 171 kB 00:00 2024-04-21T22:55:56.866 INFO:teuthology.orchestra.run.smithi069.stdout:(29/150): ceph-grafana-dashboards-19.0.0-2405.g 292 kB/s | 32 kB 00:00 2024-04-21T22:55:56.949 INFO:teuthology.orchestra.run.smithi069.stdout:(30/150): ceph-mgr-cephadm-19.0.0-2405.g820a2f8 1.2 MB/s | 141 kB 00:00 2024-04-21T22:55:56.960 INFO:teuthology.orchestra.run.smithi055.stdout:(149/150): librados2-19.0.0-2405.g820a2f8c.el9. 8.0 MB/s | 3.4 MB 00:00 2024-04-21T22:55:57.094 INFO:teuthology.orchestra.run.smithi055.stdout:(150/150): librbd1-19.0.0-2405.g820a2f8c.el9.x8 6.0 MB/s | 3.1 MB 00:00 2024-04-21T22:55:57.098 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2024-04-21T22:55:57.099 INFO:teuthology.orchestra.run.smithi055.stdout:Total 19 MB/s | 216 MB 00:11 2024-04-21T22:55:57.116 INFO:teuthology.orchestra.run.smithi069.stdout:(31/150): rbd-mirror-19.0.0-2405.g820a2f8c.el9. 7.7 MB/s | 3.1 MB 00:00 2024-04-21T22:55:57.225 INFO:teuthology.orchestra.run.smithi069.stdout:(32/150): ceph-mgr-modules-core-19.0.0-2405.g82 2.3 MB/s | 249 kB 00:00 2024-04-21T22:55:57.316 INFO:teuthology.orchestra.run.smithi069.stdout:(33/150): ceph-mgr-rook-19.0.0-2405.g820a2f8c.e 539 kB/s | 49 kB 00:00 2024-04-21T22:55:57.419 INFO:teuthology.orchestra.run.smithi069.stdout:(34/150): ceph-mgr-diskprediction-local-19.0.0- 16 MB/s | 7.4 MB 00:00 2024-04-21T22:55:57.442 INFO:teuthology.orchestra.run.smithi069.stdout:(35/150): ceph-prometheus-alerts-19.0.0-2405.g8 122 kB/s | 15 kB 00:00 2024-04-21T22:55:57.484 INFO:teuthology.orchestra.run.smithi069.stdout:(36/150): ceph-volume-19.0.0-2405.g820a2f8c.el9 3.9 MB/s | 264 kB 00:00 2024-04-21T22:55:57.526 INFO:teuthology.orchestra.run.smithi069.stdout:(37/150): cephadm-19.0.0-2405.g820a2f8c.el9.noa 8.9 MB/s | 762 kB 00:00 2024-04-21T22:55:57.551 INFO:teuthology.orchestra.run.smithi069.stdout:(38/150): cephfs-top-19.0.0-2405.g820a2f8c.el9. 273 kB/s | 18 kB 00:00 2024-04-21T22:55:57.702 INFO:teuthology.orchestra.run.smithi069.stdout:(39/150): fuse-2.9.9-15.el9.x86_64.rpm 459 kB/s | 80 kB 00:00 2024-04-21T22:55:57.794 INFO:teuthology.orchestra.run.smithi069.stdout:(40/150): ceph-mgr-dashboard-19.0.0-2405.g820a2 3.8 MB/s | 3.5 MB 00:00 2024-04-21T22:55:57.819 INFO:teuthology.orchestra.run.smithi069.stdout:(41/150): libquadmath-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 195 kB 00:00 2024-04-21T22:55:57.852 INFO:teuthology.orchestra.run.smithi069.stdout:(42/150): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-04-21T22:55:57.903 INFO:teuthology.orchestra.run.smithi069.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-21T22:55:57.936 INFO:teuthology.orchestra.run.smithi069.stdout:(44/150): python3-cryptography-36.0.1-4.el9.x86 10 MB/s | 1.2 MB 00:00 2024-04-21T22:55:57.961 INFO:teuthology.orchestra.run.smithi069.stdout:(45/150): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-04-21T22:55:57.987 INFO:teuthology.orchestra.run.smithi069.stdout:(46/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-04-21T22:55:58.012 INFO:teuthology.orchestra.run.smithi069.stdout:(47/150): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-04-21T22:55:58.038 INFO:teuthology.orchestra.run.smithi069.stdout:(48/150): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 215 kB 00:00 2024-04-21T22:55:58.102 INFO:teuthology.orchestra.run.smithi044.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-21T22:55:58.146 INFO:teuthology.orchestra.run.smithi069.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 986 kB/s | 107 kB 00:00 2024-04-21T22:55:58.188 INFO:teuthology.orchestra.run.smithi069.stdout:(50/150): automake-1.16.2-8.el9.noarch.rpm 3.4 MB/s | 689 kB 00:00 2024-04-21T22:55:58.190 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2024-04-21T22:55:58.213 INFO:teuthology.orchestra.run.smithi069.stdout:(51/150): elfutils-libelf-devel-0.190-2.el9.x86 322 kB/s | 21 kB 00:00 2024-04-21T22:55:58.247 INFO:teuthology.orchestra.run.smithi069.stdout:(52/150): bison-3.7.4-5.el9.x86_64.rpm 3.9 MB/s | 943 kB 00:00 2024-04-21T22:55:58.272 INFO:teuthology.orchestra.run.smithi069.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 558 kB/s | 33 kB 00:00 2024-04-21T22:55:58.305 INFO:teuthology.orchestra.run.smithi069.stdout:(54/150): flex-2.6.4-9.el9.x86_64.rpm 2.6 MB/s | 313 kB 00:00 2024-04-21T22:55:58.331 INFO:teuthology.orchestra.run.smithi069.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 312 kB/s | 18 kB 00:00 2024-04-21T22:55:58.331 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2024-04-21T22:55:58.331 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2024-04-21T22:55:58.356 INFO:teuthology.orchestra.run.smithi069.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-04-21T22:55:58.373 INFO:teuthology.orchestra.run.smithi069.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 160 kB 00:00 2024-04-21T22:55:58.398 INFO:teuthology.orchestra.run.smithi069.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-04-21T22:55:58.443 INFO:teuthology.orchestra.run.smithi069.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 662 kB 00:00 2024-04-21T22:55:58.490 INFO:teuthology.orchestra.run.smithi069.stdout:(60/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-04-21T22:55:58.516 INFO:teuthology.orchestra.run.smithi069.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 339 kB/s | 25 kB 00:00 2024-04-21T22:55:58.541 INFO:teuthology.orchestra.run.smithi069.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-04-21T22:55:58.566 INFO:teuthology.orchestra.run.smithi069.stdout:(63/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-21T22:55:58.600 INFO:teuthology.orchestra.run.smithi069.stdout:(64/150): libtool-2.4.6-45.el9.x86_64.rpm 2.8 MB/s | 582 kB 00:00 2024-04-21T22:55:58.625 INFO:teuthology.orchestra.run.smithi069.stdout:(65/150): openblas-0.3.21-2.el9.x86_64.rpm 603 kB/s | 35 kB 00:00 2024-04-21T22:55:58.642 INFO:teuthology.orchestra.run.smithi069.stdout:(66/150): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-04-21T22:55:58.709 INFO:teuthology.orchestra.run.smithi069.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-04-21T22:55:58.776 INFO:teuthology.orchestra.run.smithi069.stdout:(68/150): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-04-21T22:55:58.835 INFO:teuthology.orchestra.run.smithi069.stdout:(69/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 20 MB/s | 4.1 MB 00:00 2024-04-21T22:55:58.860 INFO:teuthology.orchestra.run.smithi069.stdout:(70/150): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-04-21T22:55:58.894 INFO:teuthology.orchestra.run.smithi069.stdout:(71/150): postgresql-contrib-13.14-1.el9.x86_64 4.8 MB/s | 911 kB 00:00 2024-04-21T22:55:58.927 INFO:teuthology.orchestra.run.smithi069.stdout:(72/150): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-04-21T22:55:58.952 INFO:teuthology.orchestra.run.smithi069.stdout:(73/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-21T22:55:58.994 INFO:teuthology.orchestra.run.smithi069.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 5.8 MB/s | 244 kB 00:00 2024-04-21T22:55:59.078 INFO:teuthology.orchestra.run.smithi069.stdout:(75/150): postgresql-server-13.14-1.el9.x86_64. 24 MB/s | 5.8 MB 00:00 2024-04-21T22:55:59.103 INFO:teuthology.orchestra.run.smithi069.stdout:(76/150): python3-jinja2-2.11.3-5.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-04-21T22:55:59.128 INFO:teuthology.orchestra.run.smithi069.stdout:(77/150): python3-jsonpatch-1.21-16.el9.noarch. 513 kB/s | 26 kB 00:00 2024-04-21T22:55:59.153 INFO:teuthology.orchestra.run.smithi069.stdout:(78/150): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-04-21T22:55:59.187 INFO:teuthology.orchestra.run.smithi069.stdout:(79/150): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-21T22:55:59.212 INFO:teuthology.orchestra.run.smithi069.stdout:(80/150): python3-markupsafe-1.1.1-12.el9.x86_6 593 kB/s | 35 kB 00:00 2024-04-21T22:55:59.288 INFO:teuthology.orchestra.run.smithi069.stdout:(81/150): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-04-21T22:55:59.313 INFO:teuthology.orchestra.run.smithi069.stdout:(82/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.4 MB/s | 250 kB 00:00 2024-04-21T22:55:59.338 INFO:teuthology.orchestra.run.smithi069.stdout:(83/150): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-04-21T22:55:59.363 INFO:teuthology.orchestra.run.smithi069.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-21T22:55:59.389 INFO:teuthology.orchestra.run.smithi069.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 840 kB/s | 42 kB 00:00 2024-04-21T22:55:59.414 INFO:teuthology.orchestra.run.smithi069.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-04-21T22:55:59.439 INFO:teuthology.orchestra.run.smithi069.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-21T22:55:59.464 INFO:teuthology.orchestra.run.smithi069.stdout:(88/150): python3-pyasn1-modules-0.4.8-6.el9.no 5.5 MB/s | 279 kB 00:00 2024-04-21T22:55:59.490 INFO:teuthology.orchestra.run.smithi069.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-21T22:55:59.515 INFO:teuthology.orchestra.run.smithi069.stdout:(90/150): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-21T22:55:59.590 INFO:teuthology.orchestra.run.smithi069.stdout:(91/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 5.1 MB 00:00 2024-04-21T22:55:59.615 INFO:teuthology.orchestra.run.smithi069.stdout:(92/150): python3-toml-0.10.2-6.el9.noarch.rpm 416 kB/s | 42 kB 00:00 2024-04-21T22:55:59.641 INFO:teuthology.orchestra.run.smithi069.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 885 kB/s | 44 kB 00:00 2024-04-21T22:55:59.666 INFO:teuthology.orchestra.run.smithi069.stdout:(94/150): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-21T22:55:59.691 INFO:teuthology.orchestra.run.smithi069.stdout:(95/150): qatlib-service-24.02.0-1.el9.x86_64.r 716 kB/s | 36 kB 00:00 2024-04-21T22:55:59.717 INFO:teuthology.orchestra.run.smithi069.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-04-21T22:55:59.742 INFO:teuthology.orchestra.run.smithi069.stdout:(97/150): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-04-21T22:55:59.768 INFO:teuthology.orchestra.run.smithi069.stdout:(98/150): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-21T22:55:59.784 INFO:teuthology.orchestra.run.smithi069.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.2 MB/s | 94 kB 00:00 2024-04-21T22:55:59.810 INFO:teuthology.orchestra.run.smithi069.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-04-21T22:55:59.835 INFO:teuthology.orchestra.run.smithi069.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 889 kB/s | 45 kB 00:00 2024-04-21T22:56:00.077 INFO:teuthology.orchestra.run.smithi069.stdout:(102/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-04-21T22:56:00.102 INFO:teuthology.orchestra.run.smithi069.stdout:(103/150): lua-devel-5.4.4-4.el9.x86_64.rpm 76 kB/s | 22 kB 00:00 2024-04-21T22:56:00.149 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2024-04-21T22:56:00.150 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2024-04-21T22:56:00.320 INFO:teuthology.orchestra.run.smithi044.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T22:56:00.321 INFO:teuthology.orchestra.run.smithi044.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T22:56:00.333 INFO:teuthology.orchestra.run.smithi044.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T22:56:00.333 INFO:teuthology.orchestra.run.smithi044.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T22:56:00.334 INFO:teuthology.orchestra.run.smithi044.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T22:56:00.335 INFO:teuthology.orchestra.run.smithi044.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T22:56:00.337 INFO:teuthology.orchestra.run.smithi044.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T22:56:00.339 INFO:teuthology.orchestra.run.smithi044.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T22:56:00.340 INFO:teuthology.orchestra.run.smithi044.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T22:56:00.344 INFO:teuthology.orchestra.run.smithi069.stdout:(104/150): protobuf-compiler-3.14.0-13.el9.x86_ 1.7 MB/s | 863 kB 00:00 2024-04-21T22:56:00.370 INFO:teuthology.orchestra.run.smithi069.stdout:(105/150): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-04-21T22:56:00.394 INFO:teuthology.orchestra.run.smithi069.stdout:(106/150): abseil-cpp-20211102.0-3.el9.x86_64.r 1.7 MB/s | 548 kB 00:00 2024-04-21T22:56:00.404 INFO:teuthology.orchestra.run.smithi044.stdout:Dependencies resolved. 2024-04-21T22:56:00.414 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================= 2024-04-21T22:56:00.414 INFO:teuthology.orchestra.run.smithi044.stdout: Package Arch Version Repository Size 2024-04-21T22:56:00.414 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================= 2024-04-21T22:56:00.414 INFO:teuthology.orchestra.run.smithi044.stdout:Installing: 2024-04-21T22:56:00.414 INFO:teuthology.orchestra.run.smithi044.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T22:56:00.414 INFO:teuthology.orchestra.run.smithi044.stdout: ceph x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 6.6 k 2024-04-21T22:56:00.414 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-base x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.4 M 2024-04-21T22:56:00.414 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 889 k 2024-04-21T22:56:00.414 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 145 k 2024-04-21T22:56:00.414 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 1.5 M 2024-04-21T22:56:00.414 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 141 k 2024-04-21T22:56:00.414 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 3.5 M 2024-04-21T22:56:00.415 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 7.4 M 2024-04-21T22:56:00.415 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-rook noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 49 k 2024-04-21T22:56:00.415 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-radosgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 10 M 2024-04-21T22:56:00.415 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-test x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 48 M 2024-04-21T22:56:00.415 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-volume noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 264 k 2024-04-21T22:56:00.415 INFO:teuthology.orchestra.run.smithi044.stdout: cephadm noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 762 k 2024-04-21T22:56:00.415 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 222 k 2024-04-21T22:56:00.415 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-shell x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 27 k 2024-04-21T22:56:00.415 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-top noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 18 k 2024-04-21T22:56:00.415 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T22:56:00.415 INFO:teuthology.orchestra.run.smithi044.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T22:56:00.415 INFO:teuthology.orchestra.run.smithi044.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T22:56:00.416 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 32 k 2024-04-21T22:56:00.416 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 745 k 2024-04-21T22:56:00.416 INFO:teuthology.orchestra.run.smithi044.stdout: librados-devel x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 127 k 2024-04-21T22:56:00.416 INFO:teuthology.orchestra.run.smithi044.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T22:56:00.416 INFO:teuthology.orchestra.run.smithi044.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T22:56:00.416 INFO:teuthology.orchestra.run.smithi044.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T22:56:00.416 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T22:56:00.416 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T22:56:00.416 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T22:56:00.416 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cephfs x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 161 k 2024-04-21T22:56:00.416 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rados x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 321 k 2024-04-21T22:56:00.416 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 298 k 2024-04-21T22:56:00.416 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rgw x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 99 k 2024-04-21T22:56:00.416 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-fuse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 86 k 2024-04-21T22:56:00.417 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-mirror x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-21T22:56:00.417 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-nbd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 171 k 2024-04-21T22:56:00.417 INFO:teuthology.orchestra.run.smithi044.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T22:56:00.417 INFO:teuthology.orchestra.run.smithi044.stdout:Upgrading: 2024-04-21T22:56:00.417 INFO:teuthology.orchestra.run.smithi044.stdout: librados2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.4 M 2024-04-21T22:56:00.417 INFO:teuthology.orchestra.run.smithi044.stdout: librbd1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 3.1 M 2024-04-21T22:56:00.417 INFO:teuthology.orchestra.run.smithi044.stdout:Installing dependencies: 2024-04-21T22:56:00.417 INFO:teuthology.orchestra.run.smithi044.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T22:56:00.417 INFO:teuthology.orchestra.run.smithi044.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T22:56:00.418 INFO:teuthology.orchestra.run.smithi044.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T22:56:00.418 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 21 M 2024-04-21T22:56:00.418 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 32 k 2024-04-21T22:56:00.418 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mds x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 2.4 M 2024-04-21T22:56:00.418 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 249 k 2024-04-21T22:56:00.418 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mon x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 4.7 M 2024-04-21T22:56:00.418 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-osd x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 17 M 2024-04-21T22:56:00.418 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2405.g820a2f8c.el9 ceph-noarch 15 k 2024-04-21T22:56:00.418 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-selinux x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 25 k 2024-04-21T22:56:00.418 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T22:56:00.418 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T22:56:00.418 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T22:56:00.419 INFO:teuthology.orchestra.run.smithi044.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T22:56:00.419 INFO:teuthology.orchestra.run.smithi044.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T22:56:00.419 INFO:teuthology.orchestra.run.smithi044.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T22:56:00.419 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T22:56:00.419 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T22:56:00.419 INFO:teuthology.orchestra.run.smithi044.stdout: libcephsqlite x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 164 k 2024-04-21T22:56:00.419 INFO:teuthology.orchestra.run.smithi044.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T22:56:00.419 INFO:teuthology.orchestra.run.smithi044.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T22:56:00.419 INFO:teuthology.orchestra.run.smithi044.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T22:56:00.419 INFO:teuthology.orchestra.run.smithi044.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T22:56:00.419 INFO:teuthology.orchestra.run.smithi044.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T22:56:00.419 INFO:teuthology.orchestra.run.smithi044.stdout: libradosstriper1 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 501 k 2024-04-21T22:56:00.420 INFO:teuthology.orchestra.run.smithi044.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T22:56:00.420 INFO:teuthology.orchestra.run.smithi044.stdout: librgw2 x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 5.3 M 2024-04-21T22:56:00.420 INFO:teuthology.orchestra.run.smithi044.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T22:56:00.420 INFO:teuthology.orchestra.run.smithi044.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T22:56:00.420 INFO:teuthology.orchestra.run.smithi044.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T22:56:00.420 INFO:teuthology.orchestra.run.smithi044.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T22:56:00.420 INFO:teuthology.orchestra.run.smithi044.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T22:56:00.420 INFO:teuthology.orchestra.run.smithi044.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T22:56:00.420 INFO:teuthology.orchestra.run.smithi044.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T22:56:00.420 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T22:56:00.420 INFO:teuthology.orchestra.run.smithi044.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T22:56:00.420 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T22:56:00.420 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T22:56:00.421 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T22:56:00.421 INFO:teuthology.orchestra.run.smithi044.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T22:56:00.421 INFO:teuthology.orchestra.run.smithi044.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T22:56:00.421 INFO:teuthology.orchestra.run.smithi044.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T22:56:00.421 INFO:teuthology.orchestra.run.smithi044.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T22:56:00.421 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T22:56:00.421 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-argparse x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 45 k 2024-04-21T22:56:00.421 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-common x86_64 2:19.0.0-2405.g820a2f8c.el9 ceph 133 k 2024-04-21T22:56:00.421 INFO:teuthology.orchestra.run.smithi044.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T22:56:00.421 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T22:56:00.421 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T22:56:00.421 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T22:56:00.421 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T22:56:00.421 INFO:teuthology.orchestra.run.smithi044.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T22:56:00.422 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T22:56:00.422 INFO:teuthology.orchestra.run.smithi044.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T22:56:00.422 INFO:teuthology.orchestra.run.smithi044.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T22:56:00.422 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T22:56:00.422 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T22:56:00.422 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T22:56:00.422 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T22:56:00.422 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T22:56:00.422 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T22:56:00.422 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T22:56:00.422 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T22:56:00.422 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T22:56:00.422 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T22:56:00.422 INFO:teuthology.orchestra.run.smithi044.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T22:56:00.423 INFO:teuthology.orchestra.run.smithi044.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T22:56:00.423 INFO:teuthology.orchestra.run.smithi044.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T22:56:00.423 INFO:teuthology.orchestra.run.smithi044.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T22:56:00.423 INFO:teuthology.orchestra.run.smithi044.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T22:56:00.423 INFO:teuthology.orchestra.run.smithi044.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T22:56:00.423 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T22:56:00.423 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T22:56:00.423 INFO:teuthology.orchestra.run.smithi044.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T22:56:00.423 INFO:teuthology.orchestra.run.smithi044.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T22:56:00.423 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T22:56:00.423 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T22:56:00.423 INFO:teuthology.orchestra.run.smithi044.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T22:56:00.423 INFO:teuthology.orchestra.run.smithi044.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T22:56:00.423 INFO:teuthology.orchestra.run.smithi044.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T22:56:00.424 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T22:56:00.424 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T22:56:00.424 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T22:56:00.424 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T22:56:00.424 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T22:56:00.424 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T22:56:00.424 INFO:teuthology.orchestra.run.smithi044.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T22:56:00.424 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T22:56:00.424 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T22:56:00.424 INFO:teuthology.orchestra.run.smithi044.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T22:56:00.424 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T22:56:00.424 INFO:teuthology.orchestra.run.smithi044.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T22:56:00.424 INFO:teuthology.orchestra.run.smithi044.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T22:56:00.424 INFO:teuthology.orchestra.run.smithi044.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T22:56:00.424 INFO:teuthology.orchestra.run.smithi044.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T22:56:00.425 INFO:teuthology.orchestra.run.smithi044.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T22:56:00.425 INFO:teuthology.orchestra.run.smithi044.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T22:56:00.425 INFO:teuthology.orchestra.run.smithi044.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T22:56:00.425 INFO:teuthology.orchestra.run.smithi044.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T22:56:00.425 INFO:teuthology.orchestra.run.smithi044.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T22:56:00.425 INFO:teuthology.orchestra.run.smithi044.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T22:56:00.425 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T22:56:00.425 INFO:teuthology.orchestra.run.smithi044.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T22:56:00.425 INFO:teuthology.orchestra.run.smithi044.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T22:56:00.425 INFO:teuthology.orchestra.run.smithi044.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T22:56:00.425 INFO:teuthology.orchestra.run.smithi044.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T22:56:00.425 INFO:teuthology.orchestra.run.smithi044.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T22:56:00.425 INFO:teuthology.orchestra.run.smithi044.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T22:56:00.425 INFO:teuthology.orchestra.run.smithi044.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T22:56:00.426 INFO:teuthology.orchestra.run.smithi044.stdout:Installing weak dependencies: 2024-04-21T22:56:00.426 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T22:56:00.426 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:56:00.426 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction Summary 2024-04-21T22:56:00.426 INFO:teuthology.orchestra.run.smithi044.stdout:======================================================================================= 2024-04-21T22:56:00.426 INFO:teuthology.orchestra.run.smithi044.stdout:Install 148 Packages 2024-04-21T22:56:00.426 INFO:teuthology.orchestra.run.smithi044.stdout:Upgrade 2 Packages 2024-04-21T22:56:00.426 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:56:00.427 INFO:teuthology.orchestra.run.smithi069.stdout:(107/150): grpc-data-1.46.7-10.el9.noarch.rpm 392 kB/s | 19 kB 00:00 2024-04-21T22:56:00.427 INFO:teuthology.orchestra.run.smithi044.stdout:Total download size: 216 M 2024-04-21T22:56:00.427 INFO:teuthology.orchestra.run.smithi044.stdout:Downloading Packages: 2024-04-21T22:56:00.445 INFO:teuthology.orchestra.run.smithi069.stdout:(108/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 522 kB/s | 26 kB 00:00 2024-04-21T22:56:00.470 INFO:teuthology.orchestra.run.smithi069.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 967 kB/s | 49 kB 00:00 2024-04-21T22:56:00.495 INFO:teuthology.orchestra.run.smithi069.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-21T22:56:00.521 INFO:teuthology.orchestra.run.smithi069.stdout:(111/150): luarocks-3.9.2-1.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-04-21T22:56:00.579 INFO:teuthology.orchestra.run.smithi069.stdout:(112/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-04-21T22:56:00.605 INFO:teuthology.orchestra.run.smithi069.stdout:(113/150): python3-asyncssh-2.13.2-5.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-04-21T22:56:00.630 INFO:teuthology.orchestra.run.smithi069.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 866 kB/s | 43 kB 00:00 2024-04-21T22:56:00.647 INFO:teuthology.orchestra.run.smithi069.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 772 kB/s | 32 kB 00:00 2024-04-21T22:56:00.672 INFO:teuthology.orchestra.run.smithi069.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 356 kB/s | 15 kB 00:00 2024-04-21T22:56:00.697 INFO:teuthology.orchestra.run.smithi069.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-04-21T22:56:00.723 INFO:teuthology.orchestra.run.smithi069.stdout:(118/150): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-04-21T22:56:00.748 INFO:teuthology.orchestra.run.smithi069.stdout:(119/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-04-21T22:56:00.773 INFO:teuthology.orchestra.run.smithi069.stdout:(120/150): python3-colorama-0.4.4-12.el9.noarch 703 kB/s | 35 kB 00:00 2024-04-21T22:56:00.799 INFO:teuthology.orchestra.run.smithi069.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 4.0 MB/s | 203 kB 00:00 2024-04-21T22:56:00.874 INFO:teuthology.orchestra.run.smithi069.stdout:(122/150): libarrow-9.0.0-10.el9.x86_64.rpm 8.8 MB/s | 4.4 MB 00:00 2024-04-21T22:56:00.899 INFO:teuthology.orchestra.run.smithi069.stdout:(123/150): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-04-21T22:56:00.924 INFO:teuthology.orchestra.run.smithi069.stdout:(124/150): python3-jaraco-8.2.1-3.el9.noarch.rp 214 kB/s | 11 kB 00:00 2024-04-21T22:56:00.950 INFO:teuthology.orchestra.run.smithi069.stdout:(125/150): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-04-21T22:56:00.975 INFO:teuthology.orchestra.run.smithi069.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-04-21T22:56:01.002 INFO:teuthology.orchestra.run.smithi069.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 373 kB/s | 19 kB 00:00 2024-04-21T22:56:01.028 INFO:teuthology.orchestra.run.smithi069.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 375 kB/s | 20 kB 00:00 2024-04-21T22:56:01.078 INFO:teuthology.orchestra.run.smithi069.stdout:(129/150): python3-grpcio-1.46.7-10.el9.x86_64. 6.7 MB/s | 2.0 MB 00:00 2024-04-21T22:56:01.103 INFO:teuthology.orchestra.run.smithi069.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 616 kB/s | 46 kB 00:00 2024-04-21T22:56:01.128 INFO:teuthology.orchestra.run.smithi069.stdout:(131/150): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-04-21T22:56:01.154 INFO:teuthology.orchestra.run.smithi069.stdout:(132/150): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-04-21T22:56:01.196 INFO:teuthology.orchestra.run.smithi069.stdout:(133/150): python3-kubernetes-26.1.0-2.el9.noar 5.3 MB/s | 1.0 MB 00:00 2024-04-21T22:56:01.221 INFO:teuthology.orchestra.run.smithi069.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 246 kB/s | 16 kB 00:00 2024-04-21T22:56:01.246 INFO:teuthology.orchestra.run.smithi069.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-04-21T22:56:01.271 INFO:teuthology.orchestra.run.smithi069.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-04-21T22:56:01.297 INFO:teuthology.orchestra.run.smithi069.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 614 kB/s | 31 kB 00:00 2024-04-21T22:56:01.322 INFO:teuthology.orchestra.run.smithi069.stdout:(138/150): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-04-21T22:56:01.347 INFO:teuthology.orchestra.run.smithi069.stdout:(139/150): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-04-21T22:56:01.372 INFO:teuthology.orchestra.run.smithi069.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-04-21T22:56:01.398 INFO:teuthology.orchestra.run.smithi069.stdout:(141/150): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-04-21T22:56:01.423 INFO:teuthology.orchestra.run.smithi069.stdout:(142/150): python3-typing-extensions-4.4.0-2.el 675 kB/s | 51 kB 00:00 2024-04-21T22:56:01.448 INFO:teuthology.orchestra.run.smithi069.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-04-21T22:56:01.473 INFO:teuthology.orchestra.run.smithi069.stdout:(144/150): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-21T22:56:01.495 INFO:teuthology.orchestra.run.smithi044.stdout:(1/150): ceph-19.0.0-2405.g820a2f8c.el9.x86_64. 36 kB/s | 6.6 kB 00:00 2024-04-21T22:56:01.499 INFO:teuthology.orchestra.run.smithi069.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-04-21T22:56:01.524 INFO:teuthology.orchestra.run.smithi069.stdout:(146/150): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-04-21T22:56:01.558 INFO:teuthology.orchestra.run.smithi069.stdout:(147/150): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-04-21T22:56:01.599 INFO:teuthology.orchestra.run.smithi069.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-21T22:56:02.228 INFO:teuthology.orchestra.run.smithi044.stdout:(2/150): ceph-fuse-19.0.0-2405.g820a2f8c.el9.x8 1.2 MB/s | 889 kB 00:00 2024-04-21T22:56:02.404 INFO:teuthology.orchestra.run.smithi044.stdout:(3/150): ceph-immutable-object-cache-19.0.0-240 828 kB/s | 145 kB 00:00 2024-04-21T22:56:03.141 INFO:teuthology.orchestra.run.smithi069.stdout:(149/150): librados2-19.0.0-2405.g820a2f8c.el9. 2.1 MB/s | 3.4 MB 00:01 2024-04-21T22:56:03.208 INFO:teuthology.orchestra.run.smithi069.stdout:(150/150): librbd1-19.0.0-2405.g820a2f8c.el9.x8 1.9 MB/s | 3.1 MB 00:01 2024-04-21T22:56:03.211 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-21T22:56:03.211 INFO:teuthology.orchestra.run.smithi069.stdout:Total 17 MB/s | 216 MB 00:12 2024-04-21T22:56:03.229 INFO:teuthology.orchestra.run.smithi044.stdout:(4/150): ceph-mds-19.0.0-2405.g820a2f8c.el9.x86 2.9 MB/s | 2.4 MB 00:00 2024-04-21T22:56:03.529 INFO:teuthology.orchestra.run.smithi044.stdout:(5/150): ceph-mgr-19.0.0-2405.g820a2f8c.el9.x86 5.0 MB/s | 1.5 MB 00:00 2024-04-21T22:56:04.261 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-21T22:56:04.387 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-21T22:56:04.387 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-21T22:56:04.388 INFO:teuthology.orchestra.run.smithi044.stdout:(6/150): ceph-base-19.0.0-2405.g820a2f8c.el9.x8 1.7 MB/s | 5.4 MB 00:03 2024-04-21T22:56:04.505 INFO:teuthology.orchestra.run.smithi044.stdout:(7/150): ceph-mon-19.0.0-2405.g820a2f8c.el9.x86 4.8 MB/s | 4.7 MB 00:00 2024-04-21T22:56:05.799 INFO:teuthology.orchestra.run.smithi044.stdout:(8/150): ceph-radosgw-19.0.0-2405.g820a2f8c.el9 7.9 MB/s | 10 MB 00:01 2024-04-21T22:56:05.872 INFO:teuthology.orchestra.run.smithi044.stdout:(9/150): ceph-selinux-19.0.0-2405.g820a2f8c.el9 337 kB/s | 25 kB 00:00 2024-04-21T22:56:06.081 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-21T22:56:06.081 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-21T22:56:06.111 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2024-04-21T22:56:06.189 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T22:56:06.273 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T22:56:06.500 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T22:56:06.564 INFO:teuthology.orchestra.run.smithi044.stdout:(10/150): ceph-common-19.0.0-2405.g820a2f8c.el9 4.0 MB/s | 21 MB 00:05 2024-04-21T22:56:06.697 INFO:teuthology.orchestra.run.smithi044.stdout:(11/150): cephfs-mirror-19.0.0-2405.g820a2f8c.e 1.6 MB/s | 222 kB 00:00 2024-04-21T22:56:06.772 INFO:teuthology.orchestra.run.smithi044.stdout:(12/150): cephfs-shell-19.0.0-2405.g820a2f8c.el 361 kB/s | 27 kB 00:00 2024-04-21T22:56:06.850 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T22:56:06.856 INFO:teuthology.orchestra.run.smithi044.stdout:(13/150): libcephfs-devel-19.0.0-2405.g820a2f8c 380 kB/s | 32 kB 00:00 2024-04-21T22:56:06.956 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-21T22:56:07.064 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-21T22:56:07.090 INFO:teuthology.orchestra.run.smithi044.stdout:(14/150): libcephfs2-19.0.0-2405.g820a2f8c.el9. 3.1 MB/s | 745 kB 00:00 2024-04-21T22:56:07.091 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T22:56:07.167 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T22:56:07.173 INFO:teuthology.orchestra.run.smithi044.stdout:(15/150): libcephsqlite-19.0.0-2405.g820a2f8c.e 1.9 MB/s | 164 kB 00:00 2024-04-21T22:56:07.209 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-21T22:56:07.238 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T22:56:07.257 INFO:teuthology.orchestra.run.smithi044.stdout:(16/150): librados-devel-19.0.0-2405.g820a2f8c. 1.5 MB/s | 127 kB 00:00 2024-04-21T22:56:07.262 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T22:56:07.399 INFO:teuthology.orchestra.run.smithi044.stdout:(17/150): libradosstriper1-19.0.0-2405.g820a2f8 3.5 MB/s | 501 kB 00:00 2024-04-21T22:56:07.424 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T22:56:07.477 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T22:56:07.504 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T22:56:07.593 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T22:56:07.622 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-21T22:56:07.793 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-21T22:56:07.815 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T22:56:07.841 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T22:56:08.025 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T22:56:08.056 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T22:56:08.170 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T22:56:08.273 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T22:56:08.305 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T22:56:08.357 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T22:56:08.434 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T22:56:08.482 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T22:56:08.526 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T22:56:08.558 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T22:56:08.585 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T22:56:08.661 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T22:56:08.718 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T22:56:08.746 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T22:56:08.757 INFO:teuthology.orchestra.run.smithi044.stdout:(18/150): librgw2-19.0.0-2405.g820a2f8c.el9.x86 3.9 MB/s | 5.3 MB 00:01 2024-04-21T22:56:08.791 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T22:56:08.870 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T22:56:08.901 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 30/152 2024-04-21T22:56:08.924 INFO:teuthology.orchestra.run.smithi044.stdout:(19/150): ceph-osd-19.0.0-2405.g820a2f8c.el9.x8 3.7 MB/s | 17 MB 00:04 2024-04-21T22:56:08.945 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 31/152 2024-04-21T22:56:08.950 INFO:teuthology.orchestra.run.smithi044.stdout:(20/150): python3-ceph-argparse-19.0.0-2405.g82 235 kB/s | 45 kB 00:00 2024-04-21T22:56:08.973 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 32/152 2024-04-21T22:56:09.017 INFO:teuthology.orchestra.run.smithi044.stdout:(21/150): python3-ceph-common-19.0.0-2405.g820a 1.4 MB/s | 133 kB 00:00 2024-04-21T22:56:09.034 INFO:teuthology.orchestra.run.smithi044.stdout:(22/150): python3-cephfs-19.0.0-2405.g820a2f8c. 1.9 MB/s | 161 kB 00:00 2024-04-21T22:56:09.047 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T22:56:09.134 INFO:teuthology.orchestra.run.smithi044.stdout:(23/150): python3-rbd-19.0.0-2405.g820a2f8c.el9 2.9 MB/s | 298 kB 00:00 2024-04-21T22:56:09.201 INFO:teuthology.orchestra.run.smithi044.stdout:(24/150): python3-rados-19.0.0-2405.g820a2f8c.e 1.7 MB/s | 321 kB 00:00 2024-04-21T22:56:09.223 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T22:56:09.226 INFO:teuthology.orchestra.run.smithi044.stdout:(25/150): python3-rgw-19.0.0-2405.g820a2f8c.el9 1.1 MB/s | 99 kB 00:00 2024-04-21T22:56:09.300 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T22:56:09.302 INFO:teuthology.orchestra.run.smithi044.stdout:(26/150): rbd-fuse-19.0.0-2405.g820a2f8c.el9.x8 854 kB/s | 86 kB 00:00 2024-04-21T22:56:09.402 INFO:teuthology.orchestra.run.smithi044.stdout:(27/150): rbd-nbd-19.0.0-2405.g820a2f8c.el9.x86 1.7 MB/s | 171 kB 00:00 2024-04-21T22:56:09.456 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T22:56:09.509 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T22:56:09.511 INFO:teuthology.orchestra.run.smithi044.stdout:(28/150): ceph-grafana-dashboards-19.0.0-2405.g 293 kB/s | 32 kB 00:00 2024-04-21T22:56:09.628 INFO:teuthology.orchestra.run.smithi044.stdout:(29/150): ceph-mgr-cephadm-19.0.0-2405.g820a2f8 1.2 MB/s | 141 kB 00:00 2024-04-21T22:56:10.003 INFO:teuthology.orchestra.run.smithi044.stdout:(30/150): rbd-mirror-19.0.0-2405.g820a2f8c.el9. 4.0 MB/s | 3.1 MB 00:00 2024-04-21T22:56:10.083 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T22:56:10.165 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T22:56:10.207 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T22:56:10.251 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T22:56:10.289 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 42/152 2024-04-21T22:56:10.322 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T22:56:10.379 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T22:56:10.464 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T22:56:10.498 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T22:56:10.543 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T22:56:10.562 INFO:teuthology.orchestra.run.smithi044.stdout:(31/150): ceph-mgr-dashboard-19.0.0-2405.g820a2 3.8 MB/s | 3.5 MB 00:00 2024-04-21T22:56:10.591 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T22:56:10.625 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T22:56:10.645 INFO:teuthology.orchestra.run.smithi044.stdout:(32/150): ceph-mgr-modules-core-19.0.0-2405.g82 2.9 MB/s | 249 kB 00:00 2024-04-21T22:56:10.669 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T22:56:10.711 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T22:56:10.721 INFO:teuthology.orchestra.run.smithi044.stdout:(33/150): ceph-mgr-rook-19.0.0-2405.g820a2f8c.e 659 kB/s | 49 kB 00:00 2024-04-21T22:56:10.796 INFO:teuthology.orchestra.run.smithi044.stdout:(34/150): ceph-prometheus-alerts-19.0.0-2405.g8 204 kB/s | 15 kB 00:00 2024-04-21T22:56:10.856 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T22:56:10.879 INFO:teuthology.orchestra.run.smithi044.stdout:(35/150): ceph-volume-19.0.0-2405.g820a2f8c.el9 3.1 MB/s | 264 kB 00:00 2024-04-21T22:56:10.900 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T22:56:10.930 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T22:56:10.972 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T22:56:11.071 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T22:56:11.104 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T22:56:11.145 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T22:56:11.178 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T22:56:11.203 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T22:56:11.274 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T22:56:11.412 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T22:56:11.452 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T22:56:11.454 INFO:teuthology.orchestra.run.smithi044.stdout:(36/150): ceph-test-19.0.0-2405.g820a2f8c.el9.x 8.6 MB/s | 48 MB 00:05 2024-04-21T22:56:11.480 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T22:56:11.497 INFO:teuthology.orchestra.run.smithi044.stdout:(37/150): cephadm-19.0.0-2405.g820a2f8c.el9.noa 1.2 MB/s | 762 kB 00:00 2024-04-21T22:56:11.515 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T22:56:11.522 INFO:teuthology.orchestra.run.smithi044.stdout:(38/150): cephfs-top-19.0.0-2405.g820a2f8c.el9. 272 kB/s | 18 kB 00:00 2024-04-21T22:56:11.559 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T22:56:11.597 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T22:56:11.614 INFO:teuthology.orchestra.run.smithi044.stdout:(39/150): ceph-mgr-diskprediction-local-19.0.0- 4.6 MB/s | 7.4 MB 00:01 2024-04-21T22:56:11.636 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T22:56:11.666 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T22:56:11.701 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T22:56:11.796 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T22:56:11.831 INFO:teuthology.orchestra.run.smithi044.stdout:(40/150): fuse-2.9.9-15.el9.x86_64.rpm 241 kB/s | 80 kB 00:00 2024-04-21T22:56:11.856 INFO:teuthology.orchestra.run.smithi044.stdout:(41/150): libquadmath-11.4.1-3.el9.x86_64.rpm 808 kB/s | 195 kB 00:00 2024-04-21T22:56:11.889 INFO:teuthology.orchestra.run.smithi044.stdout:(42/150): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-04-21T22:56:11.927 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T22:56:11.954 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-21T22:56:11.956 INFO:teuthology.orchestra.run.smithi044.stdout:(43/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 253 kB 00:00 2024-04-21T22:56:11.960 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T22:56:11.982 INFO:teuthology.orchestra.run.smithi044.stdout:(44/150): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 106 kB 00:00 2024-04-21T22:56:12.017 INFO:teuthology.orchestra.run.smithi044.stdout:(45/150): python3-cryptography-36.0.1-4.el9.x86 7.6 MB/s | 1.2 MB 00:00 2024-04-21T22:56:12.034 INFO:teuthology.orchestra.run.smithi044.stdout:(46/150): python3-pycparser-2.20-6.el9.noarch.r 1.7 MB/s | 135 kB 00:00 2024-04-21T22:56:12.055 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T22:56:12.059 INFO:teuthology.orchestra.run.smithi044.stdout:(47/150): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-04-21T22:56:12.085 INFO:teuthology.orchestra.run.smithi044.stdout:(48/150): python3-urllib3-1.26.5-5.el9.noarch.r 3.1 MB/s | 215 kB 00:00 2024-04-21T22:56:12.094 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T22:56:12.108 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T22:56:12.138 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T22:56:12.177 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T22:56:12.187 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T22:56:12.210 INFO:teuthology.orchestra.run.smithi044.stdout:(49/150): automake-1.16.2-8.el9.noarch.rpm 3.8 MB/s | 689 kB 00:00 2024-04-21T22:56:12.220 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T22:56:12.235 INFO:teuthology.orchestra.run.smithi044.stdout:(50/150): boost-program-options-1.75.0-8.el9.x8 712 kB/s | 107 kB 00:00 2024-04-21T22:56:12.260 INFO:teuthology.orchestra.run.smithi044.stdout:(51/150): elfutils-libelf-devel-0.190-2.el9.x86 428 kB/s | 21 kB 00:00 2024-04-21T22:56:12.298 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T22:56:12.303 INFO:teuthology.orchestra.run.smithi044.stdout:(52/150): bison-3.7.4-5.el9.x86_64.rpm 3.8 MB/s | 943 kB 00:00 2024-04-21T22:56:12.328 INFO:teuthology.orchestra.run.smithi044.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 487 kB/s | 33 kB 00:00 2024-04-21T22:56:12.350 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T22:56:12.353 INFO:teuthology.orchestra.run.smithi044.stdout:(54/150): flex-2.6.4-9.el9.x86_64.rpm 2.6 MB/s | 313 kB 00:00 2024-04-21T22:56:12.378 INFO:teuthology.orchestra.run.smithi044.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 363 kB/s | 18 kB 00:00 2024-04-21T22:56:12.404 INFO:teuthology.orchestra.run.smithi044.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 221 kB/s | 11 kB 00:00 2024-04-21T22:56:12.429 INFO:teuthology.orchestra.run.smithi044.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-21T22:56:12.454 INFO:teuthology.orchestra.run.smithi044.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 904 kB/s | 45 kB 00:00 2024-04-21T22:56:12.521 INFO:teuthology.orchestra.run.smithi044.stdout:(59/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 14 MB/s | 3.0 MB 00:00 2024-04-21T22:56:12.546 INFO:teuthology.orchestra.run.smithi044.stdout:(60/150): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-04-21T22:56:12.561 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T22:56:12.580 INFO:teuthology.orchestra.run.smithi044.stdout:(61/150): libtool-2.4.6-45.el9.x86_64.rpm 4.5 MB/s | 582 kB 00:00 2024-04-21T22:56:12.605 INFO:teuthology.orchestra.run.smithi044.stdout:(62/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 436 kB/s | 25 kB 00:00 2024-04-21T22:56:12.630 INFO:teuthology.orchestra.run.smithi044.stdout:(63/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 568 kB/s | 47 kB 00:00 2024-04-21T22:56:12.656 INFO:teuthology.orchestra.run.smithi044.stdout:(64/150): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-21T22:56:12.678 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-21T22:56:12.681 INFO:teuthology.orchestra.run.smithi044.stdout:(65/150): openblas-0.3.21-2.el9.x86_64.rpm 702 kB/s | 35 kB 00:00 2024-04-21T22:56:12.706 INFO:teuthology.orchestra.run.smithi044.stdout:(66/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-04-21T22:56:12.800 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-21T22:56:12.829 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T22:56:12.840 INFO:teuthology.orchestra.run.smithi044.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-21T22:56:12.921 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T22:56:12.923 INFO:teuthology.orchestra.run.smithi044.stdout:(68/150): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 911 kB 00:00 2024-04-21T22:56:12.962 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-21T22:56:12.990 INFO:teuthology.orchestra.run.smithi044.stdout:(69/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 13 MB/s | 4.1 MB 00:00 2024-04-21T22:56:12.991 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T22:56:13.017 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T22:56:13.049 INFO:teuthology.orchestra.run.smithi044.stdout:(70/150): openblas-openmp-0.3.21-2.el9.x86_64.r 12 MB/s | 4.7 MB 00:00 2024-04-21T22:56:13.074 INFO:teuthology.orchestra.run.smithi044.stdout:(71/150): postgresql-private-libs-13.14-1.el9.x 887 kB/s | 133 kB 00:00 2024-04-21T22:56:13.116 INFO:teuthology.orchestra.run.smithi044.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 2.1 MB/s | 87 kB 00:00 2024-04-21T22:56:13.136 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T22:56:13.166 INFO:teuthology.orchestra.run.smithi044.stdout:(73/150): protobuf-3.14.0-13.el9.x86_64.rpm 8.6 MB/s | 1.0 MB 00:00 2024-04-21T22:56:13.169 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T22:56:13.187 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T22:56:13.217 INFO:teuthology.orchestra.run.smithi044.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 4.8 MB/s | 244 kB 00:00 2024-04-21T22:56:13.221 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T22:56:13.222 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T22:56:13.251 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T22:56:13.267 INFO:teuthology.orchestra.run.smithi044.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-04-21T22:56:13.300 INFO:teuthology.orchestra.run.smithi044.stdout:(76/150): python3-jsonpatch-1.21-16.el9.noarch. 771 kB/s | 26 kB 00:00 2024-04-21T22:56:13.328 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T22:56:13.343 INFO:teuthology.orchestra.run.smithi044.stdout:(77/150): python3-jsonpointer-2.0-4.el9.noarch. 456 kB/s | 19 kB 00:00 2024-04-21T22:56:13.353 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T22:56:13.377 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-21T22:56:13.379 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T22:56:13.418 INFO:teuthology.orchestra.run.smithi044.stdout:(78/150): postgresql-server-13.14-1.el9.x86_64. 14 MB/s | 5.8 MB 00:00 2024-04-21T22:56:13.422 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-21T22:56:13.443 INFO:teuthology.orchestra.run.smithi044.stdout:(79/150): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-04-21T22:56:13.445 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T22:56:13.460 INFO:teuthology.orchestra.run.smithi044.stdout:(80/150): python3-markupsafe-1.1.1-12.el9.x86_6 833 kB/s | 35 kB 00:00 2024-04-21T22:56:13.469 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T22:56:13.470 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T22:56:13.506 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T22:56:13.544 INFO:teuthology.orchestra.run.smithi044.stdout:(81/150): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-04-21T22:56:13.569 INFO:teuthology.orchestra.run.smithi044.stdout:(82/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-04-21T22:56:13.595 INFO:teuthology.orchestra.run.smithi044.stdout:(83/150): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-04-21T22:56:13.620 INFO:teuthology.orchestra.run.smithi044.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-21T22:56:13.645 INFO:teuthology.orchestra.run.smithi044.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 844 kB/s | 42 kB 00:00 2024-04-21T22:56:13.670 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T22:56:13.670 INFO:teuthology.orchestra.run.smithi044.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-04-21T22:56:13.683 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T22:56:13.712 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T22:56:13.712 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T22:56:13.720 INFO:teuthology.orchestra.run.smithi044.stdout:(87/150): python3-pyasn1-modules-0.4.8-6.el9.no 5.5 MB/s | 279 kB 00:00 2024-04-21T22:56:13.746 INFO:teuthology.orchestra.run.smithi044.stdout:(88/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 159 kB 00:00 2024-04-21T22:56:13.771 INFO:teuthology.orchestra.run.smithi044.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-21T22:56:13.792 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T22:56:13.796 INFO:teuthology.orchestra.run.smithi044.stdout:(90/150): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-21T22:56:13.818 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T22:56:13.871 INFO:teuthology.orchestra.run.smithi044.stdout:(91/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-04-21T22:56:13.897 INFO:teuthology.orchestra.run.smithi044.stdout:(92/150): python3-toml-0.10.2-6.el9.noarch.rpm 415 kB/s | 42 kB 00:00 2024-04-21T22:56:13.915 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T22:56:13.939 INFO:teuthology.orchestra.run.smithi044.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 659 kB/s | 44 kB 00:00 2024-04-21T22:56:13.951 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T22:56:13.964 INFO:teuthology.orchestra.run.smithi044.stdout:(94/150): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-04-21T22:56:13.986 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T22:56:13.989 INFO:teuthology.orchestra.run.smithi044.stdout:(95/150): qatlib-service-24.02.0-1.el9.x86_64.r 715 kB/s | 36 kB 00:00 2024-04-21T22:56:14.015 INFO:teuthology.orchestra.run.smithi044.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 946 kB/s | 47 kB 00:00 2024-04-21T22:56:14.040 INFO:teuthology.orchestra.run.smithi044.stdout:(97/150): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-04-21T22:56:14.059 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T22:56:14.090 INFO:teuthology.orchestra.run.smithi044.stdout:(98/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-21T22:56:14.094 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T22:56:14.115 INFO:teuthology.orchestra.run.smithi044.stdout:(99/150): uuid-1.6.2-55.el9.x86_64.rpm 576 kB/s | 58 kB 00:00 2024-04-21T22:56:14.137 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T22:56:14.141 INFO:teuthology.orchestra.run.smithi044.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-21T22:56:14.166 INFO:teuthology.orchestra.run.smithi044.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 891 kB/s | 45 kB 00:00 2024-04-21T22:56:14.170 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T22:56:14.198 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T22:56:14.225 INFO:teuthology.orchestra.run.smithi044.stdout:(102/150): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-04-21T22:56:14.260 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T22:56:14.312 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T22:56:14.342 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T22:56:14.386 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T22:56:14.409 INFO:teuthology.orchestra.run.smithi044.stdout:(103/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-04-21T22:56:14.462 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T22:56:14.483 INFO:teuthology.orchestra.run.smithi044.stdout:(104/150): protobuf-compiler-3.14.0-13.el9.x86_ 2.7 MB/s | 863 kB 00:00 2024-04-21T22:56:14.491 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T22:56:14.496 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 30/152 2024-04-21T22:56:14.540 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 31/152 2024-04-21T22:56:14.569 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 32/152 2024-04-21T22:56:14.608 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T22:56:14.640 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T22:56:14.692 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T22:56:14.759 INFO:teuthology.orchestra.run.smithi044.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 79 kB/s | 19 kB 00:00 2024-04-21T22:56:14.776 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T22:56:14.817 INFO:teuthology.orchestra.run.smithi044.stdout:(106/150): abseil-cpp-20211102.0-3.el9.x86_64.r 925 kB/s | 548 kB 00:00 2024-04-21T22:56:14.851 INFO:teuthology.orchestra.run.smithi044.stdout:(107/150): gperftools-libs-2.9.1-2.el9.x86_64.r 747 kB/s | 309 kB 00:00 2024-04-21T22:56:14.885 INFO:teuthology.orchestra.run.smithi044.stdout:(108/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 388 kB/s | 26 kB 00:00 2024-04-21T22:56:14.910 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T22:56:14.927 INFO:teuthology.orchestra.run.smithi044.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 646 kB/s | 49 kB 00:00 2024-04-21T22:56:14.960 INFO:teuthology.orchestra.run.smithi044.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 897 kB/s | 67 kB 00:00 2024-04-21T22:56:14.962 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T22:56:15.013 INFO:teuthology.orchestra.run.smithi044.stdout:(111/150): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-04-21T22:56:15.094 INFO:teuthology.orchestra.run.smithi044.stdout:(112/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 838 kB 00:00 2024-04-21T22:56:15.161 INFO:teuthology.orchestra.run.smithi044.stdout:(113/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 652 kB/s | 43 kB 00:00 2024-04-21T22:56:15.186 INFO:teuthology.orchestra.run.smithi044.stdout:(114/150): python3-asyncssh-2.13.2-5.el9.noarch 3.1 MB/s | 548 kB 00:00 2024-04-21T22:56:15.254 INFO:teuthology.orchestra.run.smithi044.stdout:(115/150): libarrow-9.0.0-10.el9.x86_64.rpm 8.9 MB/s | 4.4 MB 00:00 2024-04-21T22:56:15.299 INFO:teuthology.orchestra.run.smithi044.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 134 kB/s | 15 kB 00:00 2024-04-21T22:56:15.321 INFO:teuthology.orchestra.run.smithi044.stdout:(117/150): python3-cachetools-4.2.4-1.el9.noarc 202 kB/s | 32 kB 00:00 2024-04-21T22:56:15.346 INFO:teuthology.orchestra.run.smithi044.stdout:(118/150): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-04-21T22:56:15.379 INFO:teuthology.orchestra.run.smithi044.stdout:(119/150): python3-cherrypy-18.6.1-2.el9.noarch 4.4 MB/s | 358 kB 00:00 2024-04-21T22:56:15.405 INFO:teuthology.orchestra.run.smithi044.stdout:(120/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-04-21T22:56:15.422 INFO:teuthology.orchestra.run.smithi044.stdout:(121/150): python3-colorama-0.4.4-12.el9.noarch 467 kB/s | 35 kB 00:00 2024-04-21T22:56:15.455 INFO:teuthology.orchestra.run.smithi044.stdout:(122/150): python3-google-auth-2.28.2-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-04-21T22:56:15.486 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T22:56:15.497 INFO:teuthology.orchestra.run.smithi044.stdout:(123/150): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-04-21T22:56:15.514 INFO:teuthology.orchestra.run.smithi044.stdout:(124/150): python3-jaraco-8.2.1-3.el9.noarch.rp 182 kB/s | 11 kB 00:00 2024-04-21T22:56:15.564 INFO:teuthology.orchestra.run.smithi044.stdout:(125/150): python3-grpcio-1.46.7-10.el9.x86_64. 13 MB/s | 2.0 MB 00:00 2024-04-21T22:56:15.573 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T22:56:15.590 INFO:teuthology.orchestra.run.smithi044.stdout:(126/150): python3-jaraco-classes-3.2.1-5.el9.n 191 kB/s | 18 kB 00:00 2024-04-21T22:56:15.602 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T22:56:15.615 INFO:teuthology.orchestra.run.smithi044.stdout:(127/150): python3-jaraco-collections-3.0.0-8.e 230 kB/s | 23 kB 00:00 2024-04-21T22:56:15.640 INFO:teuthology.orchestra.run.smithi044.stdout:(128/150): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-04-21T22:56:15.644 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T22:56:15.682 INFO:teuthology.orchestra.run.smithi044.stdout:(129/150): python3-jaraco-text-3.2.0-6.el9.noar 295 kB/s | 20 kB 00:00 2024-04-21T22:56:15.682 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 42/152 2024-04-21T22:56:15.707 INFO:teuthology.orchestra.run.smithi044.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 692 kB/s | 46 kB 00:00 2024-04-21T22:56:15.715 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T22:56:15.741 INFO:teuthology.orchestra.run.smithi044.stdout:(131/150): python3-kubernetes-26.1.0-2.el9.noar 8.1 MB/s | 1.0 MB 00:00 2024-04-21T22:56:15.766 INFO:teuthology.orchestra.run.smithi044.stdout:(132/150): python3-more-itertools-8.12.0-2.el9. 939 kB/s | 79 kB 00:00 2024-04-21T22:56:15.771 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T22:56:15.791 INFO:teuthology.orchestra.run.smithi044.stdout:(133/150): python3-natsort-7.1.1-5.el9.noarch.r 689 kB/s | 58 kB 00:00 2024-04-21T22:56:15.817 INFO:teuthology.orchestra.run.smithi044.stdout:(134/150): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-21T22:56:15.842 INFO:teuthology.orchestra.run.smithi044.stdout:(135/150): python3-portend-3.1.0-2.el9.noarch.r 217 kB/s | 16 kB 00:00 2024-04-21T22:56:15.844 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T22:56:15.867 INFO:teuthology.orchestra.run.smithi044.stdout:(136/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-21T22:56:15.875 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T22:56:15.893 INFO:teuthology.orchestra.run.smithi044.stdout:(137/150): python3-pyperclip-1.8.2-2.el9.noarch 373 kB/s | 28 kB 00:00 2024-04-21T22:56:15.903 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T22:56:15.918 INFO:teuthology.orchestra.run.smithi044.stdout:(138/150): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-04-21T22:56:15.943 INFO:teuthology.orchestra.run.smithi044.stdout:(139/150): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-21T22:56:15.949 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T22:56:15.950 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T22:56:15.968 INFO:teuthology.orchestra.run.smithi044.stdout:(140/150): python3-rsa-4.9-2.el9.noarch.rpm 783 kB/s | 59 kB 00:00 2024-04-21T22:56:15.986 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T22:56:15.994 INFO:teuthology.orchestra.run.smithi044.stdout:(141/150): python3-tempora-5.0.0-2.el9.noarch.r 473 kB/s | 36 kB 00:00 2024-04-21T22:56:16.019 INFO:teuthology.orchestra.run.smithi044.stdout:(142/150): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-04-21T22:56:16.024 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T22:56:16.029 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T22:56:16.044 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T22:56:16.045 INFO:teuthology.orchestra.run.smithi044.stdout:(143/150): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-21T22:56:16.069 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T22:56:16.069 INFO:teuthology.orchestra.run.smithi044.stdout:(144/150): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-21T22:56:16.071 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T22:56:16.095 INFO:teuthology.orchestra.run.smithi044.stdout:(145/150): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-04-21T22:56:16.120 INFO:teuthology.orchestra.run.smithi044.stdout:(146/150): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-04-21T22:56:16.145 INFO:teuthology.orchestra.run.smithi044.stdout:(147/150): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-21T22:56:16.209 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T22:56:16.221 INFO:teuthology.orchestra.run.smithi044.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-04-21T22:56:16.252 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T22:56:16.284 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T22:56:16.331 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T22:56:16.406 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T22:56:16.425 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T22:56:16.449 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-21T22:56:16.465 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T22:56:16.505 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T22:56:16.524 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-21T22:56:16.530 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T22:56:16.555 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T22:56:16.562 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 98/152 2024-04-21T22:56:16.590 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T22:56:16.629 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T22:56:16.756 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T22:56:16.787 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T22:56:16.815 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T22:56:16.837 INFO:teuthology.orchestra.run.smithi044.stdout:(149/150): librbd1-19.0.0-2405.g820a2f8c.el9.x8 4.7 MB/s | 3.1 MB 00:00 2024-04-21T22:56:16.849 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T22:56:16.902 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T22:56:16.940 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T22:56:16.971 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T22:56:17.002 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T22:56:17.036 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T22:56:17.086 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T22:56:17.087 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T22:56:17.118 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-21T22:56:17.160 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T22:56:17.190 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-21T22:56:17.219 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 102/152 2024-04-21T22:56:17.229 INFO:teuthology.orchestra.run.smithi044.stdout:(150/150): librados2-19.0.0-2405.g820a2f8c.el9. 3.1 MB/s | 3.4 MB 00:01 2024-04-21T22:56:17.238 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-21T22:56:17.238 INFO:teuthology.orchestra.run.smithi044.stdout:Total 13 MB/s | 216 MB 00:16 2024-04-21T22:56:17.312 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T22:56:17.458 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T22:56:17.500 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T22:56:17.531 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T22:56:17.578 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T22:56:17.677 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T22:56:17.801 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T22:56:18.325 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction check 2024-04-21T22:56:18.464 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction check succeeded. 2024-04-21T22:56:18.464 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction test 2024-04-21T22:56:18.530 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T22:56:18.580 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T22:56:18.614 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T22:56:18.718 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T22:56:18.772 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T22:56:18.860 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T22:56:18.899 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T22:56:19.075 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T22:56:19.097 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T22:56:19.174 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T22:56:19.631 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T22:56:19.666 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T22:56:19.826 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T22:56:19.945 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T22:56:20.249 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T22:56:20.274 INFO:teuthology.orchestra.run.smithi044.stdout:Transaction test succeeded. 2024-04-21T22:56:20.274 INFO:teuthology.orchestra.run.smithi044.stdout:Running transaction 2024-04-21T22:56:20.286 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-21T22:56:20.367 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-21T22:56:20.367 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T22:56:20.367 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T22:56:20.367 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-21T22:56:20.404 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-21T22:56:21.178 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T22:56:21.250 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T22:56:21.278 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T22:56:21.311 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T22:56:21.622 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T22:56:21.682 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-21T22:56:21.774 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-21T22:56:21.804 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 98/152 2024-04-21T22:56:21.840 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T22:56:22.309 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T22:56:22.335 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-21T22:56:22.406 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-21T22:56:22.428 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 102/152 2024-04-21T22:56:24.739 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T22:56:24.775 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T22:56:25.318 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T22:56:25.353 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-21T22:56:25.446 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-21T22:56:25.446 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T22:56:25.446 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T22:56:25.446 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-21T22:56:25.479 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-21T22:56:26.185 INFO:teuthology.orchestra.run.smithi044.stdout: Preparing : 1/1 2024-04-21T22:56:26.308 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T22:56:26.424 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T22:56:26.469 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T22:56:26.816 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T22:56:26.925 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-21T22:56:27.058 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 5/152 2024-04-21T22:56:27.084 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T22:56:27.177 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T22:56:27.219 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-21T22:56:27.247 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T22:56:27.273 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T22:56:27.439 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T22:56:27.495 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T22:56:27.523 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T22:56:27.629 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T22:56:27.657 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-21T22:56:27.703 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 13/152 2024-04-21T22:56:27.725 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T22:56:27.751 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T22:56:27.954 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T22:56:27.983 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T22:56:28.095 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T22:56:28.201 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T22:56:28.249 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T22:56:28.284 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T22:56:28.361 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T22:56:28.400 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T22:56:28.444 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T22:56:28.476 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T22:56:28.504 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T22:56:28.577 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T22:56:28.628 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T22:56:28.656 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T22:56:28.701 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T22:56:28.781 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T22:56:28.811 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 30/152 2024-04-21T22:56:28.855 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 31/152 2024-04-21T22:56:28.883 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 32/152 2024-04-21T22:56:28.958 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T22:56:29.007 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T22:56:29.086 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T22:56:29.241 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T22:56:29.303 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T22:56:29.865 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T22:56:29.950 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T22:56:29.983 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T22:56:30.027 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T22:56:30.065 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 42/152 2024-04-21T22:56:30.097 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T22:56:30.155 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T22:56:30.229 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T22:56:30.265 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T22:56:30.302 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T22:56:30.350 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T22:56:30.384 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T22:56:30.429 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T22:56:30.479 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T22:56:30.626 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T22:56:30.684 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T22:56:30.722 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T22:56:30.764 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T22:56:30.863 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T22:56:30.905 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T22:56:30.946 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T22:56:30.980 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T22:56:31.004 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T22:56:31.083 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T22:56:31.223 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T22:56:31.261 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T22:56:31.289 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T22:56:31.323 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T22:56:31.368 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T22:56:31.406 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T22:56:31.446 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T22:56:31.475 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T22:56:31.510 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T22:56:31.543 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T22:56:31.585 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T22:56:31.619 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T22:56:31.710 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T22:56:31.758 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T22:56:31.851 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T22:56:31.930 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T22:56:32.041 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T22:56:32.092 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T22:56:32.533 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-21T22:56:32.533 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /sys 2024-04-21T22:56:32.533 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /proc 2024-04-21T22:56:32.533 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /mnt 2024-04-21T22:56:32.533 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /var/tmp 2024-04-21T22:56:32.533 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /home 2024-04-21T22:56:32.533 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /root 2024-04-21T22:56:32.533 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /tmp 2024-04-21T22:56:32.533 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-21T22:56:32.708 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-21T22:56:32.724 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-21T22:56:32.725 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:32.725 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T22:56:32.725 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-21T22:56:32.878 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T22:56:32.930 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T22:56:32.963 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T22:56:33.068 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T22:56:33.123 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T22:56:33.214 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T22:56:33.249 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T22:56:33.433 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T22:56:33.462 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T22:56:33.543 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T22:56:34.242 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T22:56:34.261 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-21T22:56:34.289 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-21T22:56:34.289 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:34.289 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T22:56:34.289 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T22:56:34.289 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T22:56:34.289 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-21T22:56:34.376 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T22:56:34.520 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-21T22:56:34.544 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-21T22:56:34.545 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:34.545 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T22:56:34.545 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T22:56:34.545 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T22:56:34.545 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-21T22:56:35.021 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-21T22:56:35.045 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-21T22:56:35.046 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:35.046 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T22:56:35.046 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T22:56:35.046 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T22:56:35.046 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-21T22:56:35.160 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T22:56:35.219 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T22:56:35.282 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T22:56:35.480 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T22:56:35.531 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T22:56:35.611 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T22:56:35.701 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T22:56:35.842 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T22:56:35.889 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T22:56:35.909 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T22:56:35.921 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T22:56:35.948 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T22:56:36.000 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T22:56:36.028 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T22:56:36.061 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T22:56:36.388 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T22:56:36.427 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-21T22:56:36.440 INFO:teuthology.orchestra.run.smithi044.stdout: Upgrading : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-21T22:56:36.455 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-21T22:56:36.479 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 122/152 2024-04-21T22:56:36.532 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 97/152 2024-04-21T22:56:36.534 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 123/152 2024-04-21T22:56:36.561 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-21T22:56:36.562 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 98/152 2024-04-21T22:56:36.598 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T22:56:36.942 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-21T22:56:36.943 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /sys 2024-04-21T22:56:36.943 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /proc 2024-04-21T22:56:36.943 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /mnt 2024-04-21T22:56:36.943 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /var/tmp 2024-04-21T22:56:36.943 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /home 2024-04-21T22:56:36.943 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /root 2024-04-21T22:56:36.943 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /tmp 2024-04-21T22:56:36.944 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-21T22:56:37.094 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T22:56:37.107 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-21T22:56:37.118 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-21T22:56:37.122 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-21T22:56:37.122 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:37.123 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T22:56:37.123 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-21T22:56:37.190 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 101/152 2024-04-21T22:56:37.220 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 102/152 2024-04-21T22:56:37.653 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-21T22:56:37.689 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-21T22:56:38.594 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-21T22:56:40.357 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T22:56:40.357 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T22:56:40.357 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 103/152 2024-04-21T22:56:40.357 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-21T22:56:40.358 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 104/152 2024-04-21T22:56:40.358 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T22:56:40.358 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T22:56:40.358 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:56:40.358 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-21T22:56:40.360 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-21T22:56:40.360 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-21T22:56:40.360 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-21T22:56:40.360 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 127/152 2024-04-21T22:56:40.360 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-21T22:56:40.360 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-21T22:56:40.360 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:40.361 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T22:56:40.361 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T22:56:40.361 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T22:56:40.361 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-21T22:56:40.361 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-21T22:56:40.361 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-21T22:56:40.361 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 130/152 2024-04-21T22:56:40.361 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 131/152 2024-04-21T22:56:40.361 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T22:56:40.361 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-21T22:56:40.361 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-21T22:56:40.361 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:40.361 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T22:56:40.362 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T22:56:40.362 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T22:56:40.362 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-21T22:56:40.362 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-21T22:56:40.362 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-21T22:56:40.362 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:40.362 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T22:56:40.362 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-21T22:56:40.362 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-21T22:56:40.363 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:40.363 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T22:56:40.363 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T22:56:40.363 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T22:56:40.363 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-21T22:56:40.363 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-21T22:56:40.363 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-21T22:56:40.363 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:40.363 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T22:56:40.363 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T22:56:40.363 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T22:56:40.363 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-21T22:56:40.363 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-21T22:56:40.363 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-21T22:56:40.364 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:40.364 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T22:56:40.364 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T22:56:40.364 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T22:56:40.364 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-21T22:56:40.364 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T22:56:40.364 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T22:56:40.364 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T22:56:40.364 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T22:56:40.364 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T22:56:40.364 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T22:56:40.364 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T22:56:40.364 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T22:56:40.364 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T22:56:40.365 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T22:56:40.365 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T22:56:40.412 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-21T22:56:40.436 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-21T22:56:40.436 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:40.436 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T22:56:40.437 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T22:56:40.437 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T22:56:40.437 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-21T22:56:40.614 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-21T22:56:40.647 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-21T22:56:40.676 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 122/152 2024-04-21T22:56:40.738 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 123/152 2024-04-21T22:56:40.755 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-21T22:56:40.768 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-21T22:56:40.779 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-21T22:56:40.779 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:40.779 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T22:56:40.780 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T22:56:40.780 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T22:56:40.780 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-21T22:56:41.794 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-21T22:56:41.837 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-21T22:56:42.802 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-21T22:56:42.828 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-21T22:56:42.928 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-21T22:56:43.091 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 127/152 2024-04-21T22:56:43.114 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-21T22:56:43.137 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-21T22:56:43.137 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:43.137 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T22:56:43.137 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T22:56:43.137 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T22:56:43.137 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-21T22:56:43.190 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-21T22:56:43.201 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-21T22:56:43.324 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 130/152 2024-04-21T22:56:43.371 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 131/152 2024-04-21T22:56:44.266 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T22:56:44.300 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-21T22:56:44.323 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-21T22:56:44.323 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:44.323 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T22:56:44.323 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T22:56:44.323 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T22:56:44.323 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-21T22:56:44.368 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-21T22:56:44.389 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-21T22:56:44.389 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:44.389 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T22:56:44.390 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-21T22:56:44.435 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-21T22:56:44.458 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-21T22:56:44.458 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:44.458 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T22:56:44.458 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T22:56:44.458 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T22:56:44.458 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-21T22:56:44.761 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-21T22:56:44.783 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-21T22:56:44.783 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:44.783 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T22:56:44.783 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T22:56:44.783 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T22:56:44.783 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-21T22:56:45.534 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 137/152 2024-04-21T22:56:45.575 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 138/152 2024-04-21T22:56:45.616 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 139/152 2024-04-21T22:56:45.648 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T22:56:45.675 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T22:56:45.870 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 142/152 2024-04-21T22:56:45.913 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T22:56:45.943 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 144/152 2024-04-21T22:56:46.520 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T22:56:46.557 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T22:56:46.578 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T22:56:47.440 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 146/152 2024-04-21T22:56:47.528 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T22:56:47.571 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T22:56:47.655 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T22:56:47.692 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T22:56:47.692 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T22:56:47.783 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T22:56:47.783 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T22:56:49.288 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 137/152 2024-04-21T22:56:49.330 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 138/152 2024-04-21T22:56:49.371 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 139/152 2024-04-21T22:56:49.402 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T22:56:49.430 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T22:56:49.601 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 142/152 2024-04-21T22:56:49.651 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T22:56:49.681 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 144/152 2024-04-21T22:56:50.253 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T22:56:50.295 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T22:56:50.315 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T22:56:51.088 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 146/152 2024-04-21T22:56:51.165 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T22:56:51.206 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T22:56:51.286 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T22:56:51.328 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T22:56:51.328 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T22:56:51.412 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T22:56:51.412 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T22:56:52.360 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 105/152 2024-04-21T22:56:52.361 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /sys 2024-04-21T22:56:52.361 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /proc 2024-04-21T22:56:52.361 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /mnt 2024-04-21T22:56:52.361 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /var/tmp 2024-04-21T22:56:52.361 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /home 2024-04-21T22:56:52.361 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /root 2024-04-21T22:56:52.361 INFO:teuthology.orchestra.run.smithi044.stdout:skipping the directory /tmp 2024-04-21T22:56:52.361 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:56:52.542 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-21T22:56:52.558 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 106/152 2024-04-21T22:56:52.558 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:52.558 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T22:56:52.559 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:56:53.678 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T22:56:53.678 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 1/152 2024-04-21T22:56:53.678 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2/152 2024-04-21T22:56:53.678 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 3/152 2024-04-21T22:56:53.678 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 4/152 2024-04-21T22:56:53.678 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 5/152 2024-04-21T22:56:53.678 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T22:56:53.678 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-21T22:56:53.679 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 8/152 2024-04-21T22:56:53.679 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 9/152 2024-04-21T22:56:53.679 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 10/152 2024-04-21T22:56:53.679 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 11/152 2024-04-21T22:56:53.679 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T22:56:53.679 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 13/152 2024-04-21T22:56:53.679 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 14/152 2024-04-21T22:56:53.679 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 15/152 2024-04-21T22:56:53.679 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 16/152 2024-04-21T22:56:53.679 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 17/152 2024-04-21T22:56:53.679 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 18/152 2024-04-21T22:56:53.679 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 19/152 2024-04-21T22:56:53.679 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 20/152 2024-04-21T22:56:53.680 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 21/152 2024-04-21T22:56:53.680 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 22/152 2024-04-21T22:56:53.680 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 23/152 2024-04-21T22:56:53.680 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 24/152 2024-04-21T22:56:53.680 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 25/152 2024-04-21T22:56:53.680 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 26/152 2024-04-21T22:56:53.680 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 27/152 2024-04-21T22:56:53.681 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 28/152 2024-04-21T22:56:53.681 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 29/152 2024-04-21T22:56:53.681 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 30/152 2024-04-21T22:56:53.681 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 31/152 2024-04-21T22:56:53.681 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 32/152 2024-04-21T22:56:53.681 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 33/152 2024-04-21T22:56:53.681 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 34/152 2024-04-21T22:56:53.681 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 35/152 2024-04-21T22:56:53.681 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 36/152 2024-04-21T22:56:53.681 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 37/152 2024-04-21T22:56:53.681 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 38/152 2024-04-21T22:56:53.681 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 39/152 2024-04-21T22:56:53.681 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T22:56:53.682 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T22:56:53.682 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T22:56:53.682 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T22:56:53.682 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T22:56:53.682 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T22:56:53.682 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T22:56:53.682 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T22:56:53.683 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T22:56:53.683 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T22:56:53.683 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T22:56:53.683 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T22:56:53.683 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T22:56:53.683 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T22:56:53.683 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T22:56:53.683 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T22:56:53.683 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T22:56:53.683 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T22:56:53.683 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T22:56:53.683 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T22:56:53.683 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T22:56:53.683 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T22:56:53.684 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T22:56:53.684 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T22:56:53.684 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T22:56:53.684 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T22:56:53.684 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T22:56:53.684 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T22:56:53.684 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T22:56:53.684 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T22:56:53.684 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T22:56:53.684 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T22:56:53.684 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T22:56:53.684 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T22:56:53.684 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T22:56:53.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T22:56:53.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T22:56:53.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T22:56:53.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T22:56:53.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T22:56:53.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T22:56:53.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T22:56:53.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T22:56:53.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T22:56:53.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T22:56:53.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T22:56:53.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T22:56:53.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T22:56:53.685 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T22:56:53.686 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T22:56:53.686 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T22:56:53.686 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T22:56:53.686 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T22:56:53.686 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T22:56:53.686 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T22:56:53.686 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T22:56:53.686 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T22:56:53.686 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T22:56:53.686 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T22:56:53.686 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T22:56:53.687 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T22:56:53.687 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T22:56:53.687 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T22:56:53.687 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T22:56:53.687 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T22:56:53.687 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T22:56:53.687 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T22:56:53.687 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T22:56:53.687 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T22:56:53.687 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T22:56:53.687 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T22:56:53.687 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T22:56:53.688 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T22:56:53.688 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T22:56:53.688 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T22:56:53.688 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T22:56:53.688 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T22:56:53.688 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T22:56:53.688 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T22:56:53.688 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T22:56:53.688 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T22:56:53.688 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T22:56:53.688 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T22:56:53.688 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T22:56:53.689 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T22:56:53.689 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T22:56:53.689 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T22:56:53.689 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T22:56:53.689 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T22:56:53.689 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T22:56:53.689 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T22:56:53.689 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T22:56:53.689 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T22:56:53.689 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T22:56:53.690 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T22:56:53.690 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T22:56:53.690 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T22:56:53.690 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T22:56:53.690 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T22:56:53.690 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T22:56:53.690 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T22:56:53.690 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T22:56:53.690 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T22:56:53.691 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T22:56:53.691 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T22:56:53.691 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T22:56:53.691 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T22:56:53.691 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T22:56:53.691 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T22:56:53.691 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 149/152 2024-04-21T22:56:53.691 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T22:56:53.692 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 151/152 2024-04-21T22:56:54.095 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-21T22:56:54.122 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 107/152 2024-04-21T22:56:54.122 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:54.123 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T22:56:54.123 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T22:56:54.123 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T22:56:54.123 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:56:54.363 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-21T22:56:54.386 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 108/152 2024-04-21T22:56:54.386 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:54.386 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T22:56:54.387 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T22:56:54.387 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T22:56:54.387 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:56:54.855 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-21T22:56:54.878 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 109/152 2024-04-21T22:56:54.878 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:54.878 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T22:56:54.879 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T22:56:54.879 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T22:56:54.879 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:56:54.985 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T22:56:55.044 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T22:56:55.107 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T22:56:55.311 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T22:56:55.356 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T22:56:55.437 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T22:56:55.525 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T22:56:55.678 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T22:56:55.723 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T22:56:55.754 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T22:56:55.782 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T22:56:55.787 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T22:56:55.787 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-21T22:56:55.787 INFO:teuthology.orchestra.run.smithi055.stdout:Upgraded: 2024-04-21T22:56:55.787 INFO:teuthology.orchestra.run.smithi055.stdout: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.787 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.787 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2024-04-21T22:56:55.787 INFO:teuthology.orchestra.run.smithi055.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T22:56:55.787 INFO:teuthology.orchestra.run.smithi055.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T22:56:55.788 INFO:teuthology.orchestra.run.smithi055.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T22:56:55.788 INFO:teuthology.orchestra.run.smithi055.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T22:56:55.788 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.788 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.788 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.788 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.788 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:56:55.788 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.788 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.788 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.788 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:56:55.788 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:56:55.788 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:56:55.788 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:56:55.788 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:56:55.789 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.789 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.789 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:56:55.789 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.789 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.789 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.789 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:56:55.789 INFO:teuthology.orchestra.run.smithi055.stdout: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:56:55.789 INFO:teuthology.orchestra.run.smithi055.stdout: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.789 INFO:teuthology.orchestra.run.smithi055.stdout: cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.789 INFO:teuthology.orchestra.run.smithi055.stdout: cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:56:55.789 INFO:teuthology.orchestra.run.smithi055.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T22:56:55.789 INFO:teuthology.orchestra.run.smithi055.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T22:56:55.789 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T22:56:55.790 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T22:56:55.790 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T22:56:55.790 INFO:teuthology.orchestra.run.smithi055.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T22:56:55.790 INFO:teuthology.orchestra.run.smithi055.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T22:56:55.790 INFO:teuthology.orchestra.run.smithi055.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T22:56:55.790 INFO:teuthology.orchestra.run.smithi055.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T22:56:55.790 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T22:56:55.790 INFO:teuthology.orchestra.run.smithi055.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T22:56:55.790 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.790 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.790 INFO:teuthology.orchestra.run.smithi055.stdout: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.790 INFO:teuthology.orchestra.run.smithi055.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T22:56:55.790 INFO:teuthology.orchestra.run.smithi055.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T22:56:55.790 INFO:teuthology.orchestra.run.smithi055.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T22:56:55.791 INFO:teuthology.orchestra.run.smithi055.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T22:56:55.791 INFO:teuthology.orchestra.run.smithi055.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T22:56:55.791 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.791 INFO:teuthology.orchestra.run.smithi055.stdout: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.791 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T22:56:55.791 INFO:teuthology.orchestra.run.smithi055.stdout: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.791 INFO:teuthology.orchestra.run.smithi055.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T22:56:55.791 INFO:teuthology.orchestra.run.smithi055.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T22:56:55.791 INFO:teuthology.orchestra.run.smithi055.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T22:56:55.791 INFO:teuthology.orchestra.run.smithi055.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T22:56:55.791 INFO:teuthology.orchestra.run.smithi055.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T22:56:55.791 INFO:teuthology.orchestra.run.smithi055.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T22:56:55.791 INFO:teuthology.orchestra.run.smithi055.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T22:56:55.791 INFO:teuthology.orchestra.run.smithi055.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T22:56:55.792 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T22:56:55.792 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T22:56:55.792 INFO:teuthology.orchestra.run.smithi055.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T22:56:55.792 INFO:teuthology.orchestra.run.smithi055.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T22:56:55.792 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T22:56:55.792 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T22:56:55.792 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T22:56:55.792 INFO:teuthology.orchestra.run.smithi055.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T22:56:55.792 INFO:teuthology.orchestra.run.smithi055.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T22:56:55.792 INFO:teuthology.orchestra.run.smithi055.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T22:56:55.792 INFO:teuthology.orchestra.run.smithi055.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T22:56:55.792 INFO:teuthology.orchestra.run.smithi055.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T22:56:55.792 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T22:56:55.792 INFO:teuthology.orchestra.run.smithi055.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T22:56:55.793 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T22:56:55.793 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.793 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.793 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.793 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T22:56:55.793 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T22:56:55.793 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T22:56:55.793 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T22:56:55.793 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T22:56:55.793 INFO:teuthology.orchestra.run.smithi055.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T22:56:55.793 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T22:56:55.793 INFO:teuthology.orchestra.run.smithi055.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T22:56:55.793 INFO:teuthology.orchestra.run.smithi055.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T22:56:55.793 INFO:teuthology.orchestra.run.smithi055.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T22:56:55.793 INFO:teuthology.orchestra.run.smithi055.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T22:56:55.794 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T22:56:55.794 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T22:56:55.794 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T22:56:55.794 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T22:56:55.794 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T22:56:55.794 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T22:56:55.794 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T22:56:55.794 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T22:56:55.794 INFO:teuthology.orchestra.run.smithi055.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T22:56:55.794 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T22:56:55.794 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T22:56:55.794 INFO:teuthology.orchestra.run.smithi055.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T22:56:55.794 INFO:teuthology.orchestra.run.smithi055.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T22:56:55.795 INFO:teuthology.orchestra.run.smithi055.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T22:56:55.795 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T22:56:55.795 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T22:56:55.795 INFO:teuthology.orchestra.run.smithi055.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T22:56:55.795 INFO:teuthology.orchestra.run.smithi055.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T22:56:55.795 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T22:56:55.795 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T22:56:55.795 INFO:teuthology.orchestra.run.smithi055.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T22:56:55.795 INFO:teuthology.orchestra.run.smithi055.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T22:56:55.795 INFO:teuthology.orchestra.run.smithi055.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T22:56:55.795 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T22:56:55.795 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T22:56:55.795 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T22:56:55.795 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T22:56:55.795 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T22:56:55.796 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T22:56:55.796 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.796 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.796 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T22:56:55.796 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T22:56:55.796 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T22:56:55.796 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.796 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T22:56:55.796 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T22:56:55.796 INFO:teuthology.orchestra.run.smithi055.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T22:56:55.796 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T22:56:55.796 INFO:teuthology.orchestra.run.smithi055.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T22:56:55.796 INFO:teuthology.orchestra.run.smithi055.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T22:56:55.796 INFO:teuthology.orchestra.run.smithi055.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T22:56:55.796 INFO:teuthology.orchestra.run.smithi055.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T22:56:55.797 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T22:56:55.797 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T22:56:55.797 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T22:56:55.797 INFO:teuthology.orchestra.run.smithi055.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T22:56:55.797 INFO:teuthology.orchestra.run.smithi055.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T22:56:55.797 INFO:teuthology.orchestra.run.smithi055.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T22:56:55.797 INFO:teuthology.orchestra.run.smithi055.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T22:56:55.797 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.797 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.797 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:55.797 INFO:teuthology.orchestra.run.smithi055.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T22:56:55.797 INFO:teuthology.orchestra.run.smithi055.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T22:56:55.797 INFO:teuthology.orchestra.run.smithi055.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T22:56:55.797 INFO:teuthology.orchestra.run.smithi055.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T22:56:55.797 INFO:teuthology.orchestra.run.smithi055.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T22:56:55.798 INFO:teuthology.orchestra.run.smithi055.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T22:56:55.798 INFO:teuthology.orchestra.run.smithi055.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T22:56:55.798 INFO:teuthology.orchestra.run.smithi055.stdout: 2024-04-21T22:56:55.798 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2024-04-21T22:56:56.135 DEBUG:teuthology.parallel:result is None 2024-04-21T22:56:56.253 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-21T22:56:56.281 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 121/152 2024-04-21T22:56:56.312 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 122/152 2024-04-21T22:56:56.377 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 123/152 2024-04-21T22:56:56.403 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-21T22:56:57.161 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T22:56:57.161 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 1/152 2024-04-21T22:56:57.161 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2/152 2024-04-21T22:56:57.162 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 3/152 2024-04-21T22:56:57.162 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 4/152 2024-04-21T22:56:57.162 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 5/152 2024-04-21T22:56:57.162 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T22:56:57.162 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-21T22:56:57.162 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 8/152 2024-04-21T22:56:57.162 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 9/152 2024-04-21T22:56:57.162 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 10/152 2024-04-21T22:56:57.162 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 11/152 2024-04-21T22:56:57.162 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T22:56:57.162 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 13/152 2024-04-21T22:56:57.163 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 14/152 2024-04-21T22:56:57.163 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 15/152 2024-04-21T22:56:57.163 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 16/152 2024-04-21T22:56:57.163 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 17/152 2024-04-21T22:56:57.163 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 18/152 2024-04-21T22:56:57.163 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 19/152 2024-04-21T22:56:57.163 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 20/152 2024-04-21T22:56:57.163 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 21/152 2024-04-21T22:56:57.163 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 22/152 2024-04-21T22:56:57.163 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 23/152 2024-04-21T22:56:57.163 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 24/152 2024-04-21T22:56:57.164 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 25/152 2024-04-21T22:56:57.164 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 26/152 2024-04-21T22:56:57.164 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 27/152 2024-04-21T22:56:57.164 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 28/152 2024-04-21T22:56:57.164 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 29/152 2024-04-21T22:56:57.164 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 30/152 2024-04-21T22:56:57.164 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 31/152 2024-04-21T22:56:57.164 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 32/152 2024-04-21T22:56:57.164 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 33/152 2024-04-21T22:56:57.164 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 34/152 2024-04-21T22:56:57.164 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 35/152 2024-04-21T22:56:57.164 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 36/152 2024-04-21T22:56:57.164 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 37/152 2024-04-21T22:56:57.164 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 38/152 2024-04-21T22:56:57.165 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 39/152 2024-04-21T22:56:57.165 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T22:56:57.165 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T22:56:57.165 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T22:56:57.165 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T22:56:57.165 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T22:56:57.165 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T22:56:57.165 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T22:56:57.165 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T22:56:57.165 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T22:56:57.165 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T22:56:57.165 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T22:56:57.165 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T22:56:57.165 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T22:56:57.165 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T22:56:57.166 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T22:56:57.166 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T22:56:57.166 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T22:56:57.166 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T22:56:57.166 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T22:56:57.166 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T22:56:57.166 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T22:56:57.167 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T22:56:57.167 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T22:56:57.167 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T22:56:57.167 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T22:56:57.167 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T22:56:57.167 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T22:56:57.167 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T22:56:57.167 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T22:56:57.167 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T22:56:57.167 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T22:56:57.167 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T22:56:57.167 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T22:56:57.167 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T22:56:57.167 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T22:56:57.168 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T22:56:57.168 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T22:56:57.168 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T22:56:57.168 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T22:56:57.168 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T22:56:57.168 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T22:56:57.168 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T22:56:57.168 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T22:56:57.168 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T22:56:57.168 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T22:56:57.168 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T22:56:57.169 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T22:56:57.169 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T22:56:57.169 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T22:56:57.169 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T22:56:57.169 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T22:56:57.169 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T22:56:57.169 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T22:56:57.169 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T22:56:57.169 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T22:56:57.169 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T22:56:57.169 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T22:56:57.170 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T22:56:57.170 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T22:56:57.170 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T22:56:57.170 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T22:56:57.170 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T22:56:57.170 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T22:56:57.170 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T22:56:57.171 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T22:56:57.171 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T22:56:57.171 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T22:56:57.171 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T22:56:57.171 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T22:56:57.171 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T22:56:57.171 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T22:56:57.171 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T22:56:57.171 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T22:56:57.171 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T22:56:57.171 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T22:56:57.171 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T22:56:57.171 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T22:56:57.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T22:56:57.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T22:56:57.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T22:56:57.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T22:56:57.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T22:56:57.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T22:56:57.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T22:56:57.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T22:56:57.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T22:56:57.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T22:56:57.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T22:56:57.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T22:56:57.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T22:56:57.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T22:56:57.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T22:56:57.173 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T22:56:57.173 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T22:56:57.173 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T22:56:57.173 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T22:56:57.173 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T22:56:57.173 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T22:56:57.173 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T22:56:57.173 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T22:56:57.173 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T22:56:57.173 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T22:56:57.173 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T22:56:57.173 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T22:56:57.173 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T22:56:57.173 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T22:56:57.174 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T22:56:57.174 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T22:56:57.174 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T22:56:57.174 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 149/152 2024-04-21T22:56:57.174 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T22:56:57.174 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 151/152 2024-04-21T22:56:57.491 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 124/152 2024-04-21T22:56:57.530 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-21T22:56:58.545 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 125/152 2024-04-21T22:56:58.572 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-21T22:56:58.679 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 126/152 2024-04-21T22:56:58.859 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 127/152 2024-04-21T22:56:58.891 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-21T22:56:58.915 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 128/152 2024-04-21T22:56:58.915 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:56:58.915 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T22:56:58.915 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T22:56:58.915 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T22:56:58.916 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:56:58.967 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-21T22:56:58.978 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 129/152 2024-04-21T22:56:59.081 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 130/152 2024-04-21T22:56:59.132 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 131/152 2024-04-21T22:56:59.250 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T22:56:59.250 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-21T22:56:59.250 INFO:teuthology.orchestra.run.smithi069.stdout:Upgraded: 2024-04-21T22:56:59.250 INFO:teuthology.orchestra.run.smithi069.stdout: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.251 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.251 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-21T22:56:59.251 INFO:teuthology.orchestra.run.smithi069.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T22:56:59.251 INFO:teuthology.orchestra.run.smithi069.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T22:56:59.251 INFO:teuthology.orchestra.run.smithi069.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T22:56:59.251 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T22:56:59.251 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.251 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.251 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.251 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.251 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:56:59.251 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.251 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.252 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.252 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:56:59.252 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:56:59.252 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:56:59.252 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:56:59.252 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:56:59.252 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.252 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.252 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:56:59.252 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.252 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.252 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.252 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:56:59.253 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:56:59.253 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.253 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.253 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:56:59.253 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T22:56:59.253 INFO:teuthology.orchestra.run.smithi069.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T22:56:59.253 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T22:56:59.253 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T22:56:59.253 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T22:56:59.253 INFO:teuthology.orchestra.run.smithi069.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T22:56:59.253 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T22:56:59.253 INFO:teuthology.orchestra.run.smithi069.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T22:56:59.253 INFO:teuthology.orchestra.run.smithi069.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T22:56:59.253 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T22:56:59.254 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T22:56:59.254 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.254 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.254 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.254 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T22:56:59.254 INFO:teuthology.orchestra.run.smithi069.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T22:56:59.254 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T22:56:59.254 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T22:56:59.254 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T22:56:59.254 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.254 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.254 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T22:56:59.254 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.255 INFO:teuthology.orchestra.run.smithi069.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T22:56:59.255 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T22:56:59.255 INFO:teuthology.orchestra.run.smithi069.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T22:56:59.255 INFO:teuthology.orchestra.run.smithi069.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T22:56:59.255 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T22:56:59.255 INFO:teuthology.orchestra.run.smithi069.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T22:56:59.255 INFO:teuthology.orchestra.run.smithi069.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T22:56:59.255 INFO:teuthology.orchestra.run.smithi069.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T22:56:59.255 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T22:56:59.255 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T22:56:59.255 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T22:56:59.255 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T22:56:59.255 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T22:56:59.255 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T22:56:59.256 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T22:56:59.256 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T22:56:59.256 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T22:56:59.256 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T22:56:59.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T22:56:59.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T22:56:59.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T22:56:59.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T22:56:59.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T22:56:59.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T22:56:59.257 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T22:56:59.257 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T22:56:59.257 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T22:56:59.257 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T22:56:59.257 INFO:teuthology.orchestra.run.smithi069.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T22:56:59.257 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T22:56:59.257 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T22:56:59.257 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T22:56:59.258 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T22:56:59.258 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T22:56:59.258 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T22:56:59.258 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T22:56:59.258 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T22:56:59.258 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T22:56:59.258 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T22:56:59.258 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T22:56:59.258 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T22:56:59.258 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T22:56:59.258 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T22:56:59.258 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T22:56:59.258 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T22:56:59.259 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T22:56:59.259 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T22:56:59.259 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T22:56:59.259 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T22:56:59.259 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T22:56:59.259 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T22:56:59.259 INFO:teuthology.orchestra.run.smithi069.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T22:56:59.259 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T22:56:59.259 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T22:56:59.259 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T22:56:59.259 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T22:56:59.259 INFO:teuthology.orchestra.run.smithi069.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T22:56:59.259 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T22:56:59.259 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T22:56:59.260 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T22:56:59.260 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T22:56:59.260 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T22:56:59.260 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T22:56:59.260 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.260 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.260 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T22:56:59.260 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T22:56:59.260 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T22:56:59.260 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.260 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T22:56:59.260 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T22:56:59.260 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T22:56:59.260 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T22:56:59.261 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T22:56:59.261 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T22:56:59.261 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T22:56:59.261 INFO:teuthology.orchestra.run.smithi069.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T22:56:59.261 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T22:56:59.261 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T22:56:59.261 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T22:56:59.261 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T22:56:59.261 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T22:56:59.261 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T22:56:59.261 INFO:teuthology.orchestra.run.smithi069.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T22:56:59.261 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.261 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.261 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:56:59.262 INFO:teuthology.orchestra.run.smithi069.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T22:56:59.262 INFO:teuthology.orchestra.run.smithi069.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T22:56:59.262 INFO:teuthology.orchestra.run.smithi069.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T22:56:59.262 INFO:teuthology.orchestra.run.smithi069.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T22:56:59.262 INFO:teuthology.orchestra.run.smithi069.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T22:56:59.262 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T22:56:59.262 INFO:teuthology.orchestra.run.smithi069.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T22:56:59.262 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-21T22:56:59.262 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-21T22:56:59.604 DEBUG:teuthology.parallel:result is None 2024-04-21T22:57:00.085 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T22:57:00.111 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-21T22:57:00.135 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 133/152 2024-04-21T22:57:00.135 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:57:00.135 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T22:57:00.135 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T22:57:00.135 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T22:57:00.135 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:57:00.178 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-21T22:57:00.202 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2405.g820a2 134/152 2024-04-21T22:57:00.202 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:57:00.202 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T22:57:00.202 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:57:00.254 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-21T22:57:00.277 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 135/152 2024-04-21T22:57:00.278 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:57:00.278 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T22:57:00.278 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T22:57:00.278 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T22:57:00.278 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:57:00.596 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-21T22:57:00.619 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 136/152 2024-04-21T22:57:00.620 INFO:teuthology.orchestra.run.smithi044.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T22:57:00.620 INFO:teuthology.orchestra.run.smithi044.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T22:57:00.620 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T22:57:00.620 INFO:teuthology.orchestra.run.smithi044.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T22:57:00.620 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:57:05.375 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 137/152 2024-04-21T22:57:05.442 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 138/152 2024-04-21T22:57:05.482 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 139/152 2024-04-21T22:57:05.514 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T22:57:05.541 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T22:57:05.734 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 142/152 2024-04-21T22:57:05.779 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T22:57:05.809 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 144/152 2024-04-21T22:57:06.427 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T22:57:06.465 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T22:57:06.486 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T22:57:07.332 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 146/152 2024-04-21T22:57:07.419 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T22:57:07.470 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T22:57:07.553 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T22:57:07.581 INFO:teuthology.orchestra.run.smithi044.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T22:57:07.582 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T22:57:07.674 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T22:57:07.674 INFO:teuthology.orchestra.run.smithi044.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T22:57:13.420 INFO:teuthology.orchestra.run.smithi044.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T22:57:13.421 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 1/152 2024-04-21T22:57:13.421 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2/152 2024-04-21T22:57:13.421 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 3/152 2024-04-21T22:57:13.421 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 4/152 2024-04-21T22:57:13.421 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2405.g820a2 5/152 2024-04-21T22:57:13.421 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 6/152 2024-04-21T22:57:13.421 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 7/152 2024-04-21T22:57:13.421 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 8/152 2024-04-21T22:57:13.421 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 9/152 2024-04-21T22:57:13.421 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 10/152 2024-04-21T22:57:13.421 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 11/152 2024-04-21T22:57:13.421 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 12/152 2024-04-21T22:57:13.421 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 13/152 2024-04-21T22:57:13.421 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 14/152 2024-04-21T22:57:13.422 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_ 15/152 2024-04-21T22:57:13.422 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 16/152 2024-04-21T22:57:13.422 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 17/152 2024-04-21T22:57:13.422 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_6 18/152 2024-04-21T22:57:13.422 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86 19/152 2024-04-21T22:57:13.422 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 20/152 2024-04-21T22:57:13.422 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el 21/152 2024-04-21T22:57:13.422 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9. 22/152 2024-04-21T22:57:13.422 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_6 23/152 2024-04-21T22:57:13.422 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 24/152 2024-04-21T22:57:13.422 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 25/152 2024-04-21T22:57:13.422 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 26/152 2024-04-21T22:57:13.422 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 27/152 2024-04-21T22:57:13.423 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 28/152 2024-04-21T22:57:13.423 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 29/152 2024-04-21T22:57:13.423 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c. 30/152 2024-04-21T22:57:13.423 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noa 31/152 2024-04-21T22:57:13.423 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.n 32/152 2024-04-21T22:57:13.423 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2405.g820 33/152 2024-04-21T22:57:13.423 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el 34/152 2024-04-21T22:57:13.423 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 35/152 2024-04-21T22:57:13.423 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.e 36/152 2024-04-21T22:57:13.423 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 37/152 2024-04-21T22:57:13.423 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 38/152 2024-04-21T22:57:13.423 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 39/152 2024-04-21T22:57:13.423 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T22:57:13.424 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T22:57:13.424 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T22:57:13.424 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T22:57:13.424 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T22:57:13.424 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T22:57:13.424 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T22:57:13.424 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T22:57:13.424 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T22:57:13.424 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T22:57:13.424 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T22:57:13.424 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T22:57:13.424 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T22:57:13.424 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T22:57:13.424 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T22:57:13.424 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T22:57:13.425 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T22:57:13.425 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T22:57:13.425 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T22:57:13.425 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T22:57:13.425 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T22:57:13.425 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T22:57:13.425 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T22:57:13.425 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T22:57:13.425 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T22:57:13.425 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T22:57:13.425 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T22:57:13.425 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T22:57:13.426 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T22:57:13.426 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T22:57:13.426 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T22:57:13.426 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T22:57:13.426 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T22:57:13.426 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T22:57:13.427 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T22:57:13.427 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T22:57:13.427 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T22:57:13.427 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T22:57:13.427 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T22:57:13.427 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T22:57:13.428 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T22:57:13.428 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T22:57:13.428 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T22:57:13.428 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T22:57:13.428 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T22:57:13.428 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T22:57:13.428 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T22:57:13.429 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T22:57:13.429 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T22:57:13.429 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T22:57:13.429 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T22:57:13.429 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T22:57:13.429 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T22:57:13.430 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T22:57:13.430 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T22:57:13.430 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T22:57:13.430 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T22:57:13.430 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T22:57:13.430 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T22:57:13.430 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T22:57:13.431 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T22:57:13.431 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T22:57:13.431 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T22:57:13.431 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T22:57:13.431 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T22:57:13.431 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T22:57:13.432 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T22:57:13.432 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T22:57:13.432 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T22:57:13.432 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T22:57:13.432 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T22:57:13.432 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T22:57:13.433 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T22:57:13.433 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T22:57:13.433 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T22:57:13.433 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T22:57:13.433 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T22:57:13.433 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T22:57:13.434 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T22:57:13.434 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T22:57:13.434 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T22:57:13.434 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T22:57:13.434 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T22:57:13.435 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T22:57:13.435 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T22:57:13.435 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T22:57:13.435 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T22:57:13.435 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T22:57:13.435 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T22:57:13.435 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T22:57:13.435 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T22:57:13.435 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T22:57:13.435 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T22:57:13.436 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T22:57:13.436 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T22:57:13.436 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T22:57:13.436 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T22:57:13.436 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T22:57:13.436 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T22:57:13.437 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T22:57:13.437 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T22:57:13.437 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T22:57:13.437 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T22:57:13.437 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T22:57:13.437 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T22:57:13.437 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T22:57:13.437 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T22:57:13.437 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T22:57:13.438 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T22:57:13.438 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 149/152 2024-04-21T22:57:13.438 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T22:57:13.438 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 151/152 2024-04-21T22:57:15.614 INFO:teuthology.orchestra.run.smithi044.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T22:57:15.614 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:57:15.614 INFO:teuthology.orchestra.run.smithi044.stdout:Upgraded: 2024-04-21T22:57:15.614 INFO:teuthology.orchestra.run.smithi044.stdout: librados2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.614 INFO:teuthology.orchestra.run.smithi044.stdout: librbd1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.614 INFO:teuthology.orchestra.run.smithi044.stdout:Installed: 2024-04-21T22:57:15.614 INFO:teuthology.orchestra.run.smithi044.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T22:57:15.614 INFO:teuthology.orchestra.run.smithi044.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T22:57:15.614 INFO:teuthology.orchestra.run.smithi044.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T22:57:15.615 INFO:teuthology.orchestra.run.smithi044.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T22:57:15.615 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.615 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-base-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.615 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.615 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.615 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-grafana-dashboards-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:57:15.615 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-immutable-object-cache-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.615 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mds-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.615 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.615 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:57:15.615 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-dashboard-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:57:15.615 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:57:15.615 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-modules-core-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:57:15.615 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mgr-rook-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:57:15.616 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-mon-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.616 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-osd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.616 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-prometheus-alerts-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:57:15.616 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-radosgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.616 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-selinux-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.616 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-test-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.616 INFO:teuthology.orchestra.run.smithi044.stdout: ceph-volume-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:57:15.616 INFO:teuthology.orchestra.run.smithi044.stdout: cephadm-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:57:15.616 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.616 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-shell-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.616 INFO:teuthology.orchestra.run.smithi044.stdout: cephfs-top-2:19.0.0-2405.g820a2f8c.el9.noarch 2024-04-21T22:57:15.616 INFO:teuthology.orchestra.run.smithi044.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T22:57:15.616 INFO:teuthology.orchestra.run.smithi044.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T22:57:15.617 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T22:57:15.617 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T22:57:15.617 INFO:teuthology.orchestra.run.smithi044.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T22:57:15.617 INFO:teuthology.orchestra.run.smithi044.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T22:57:15.617 INFO:teuthology.orchestra.run.smithi044.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T22:57:15.617 INFO:teuthology.orchestra.run.smithi044.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T22:57:15.617 INFO:teuthology.orchestra.run.smithi044.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T22:57:15.617 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T22:57:15.617 INFO:teuthology.orchestra.run.smithi044.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T22:57:15.617 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.617 INFO:teuthology.orchestra.run.smithi044.stdout: libcephfs2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.617 INFO:teuthology.orchestra.run.smithi044.stdout: libcephsqlite-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.617 INFO:teuthology.orchestra.run.smithi044.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T22:57:15.617 INFO:teuthology.orchestra.run.smithi044.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T22:57:15.618 INFO:teuthology.orchestra.run.smithi044.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T22:57:15.618 INFO:teuthology.orchestra.run.smithi044.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T22:57:15.618 INFO:teuthology.orchestra.run.smithi044.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T22:57:15.618 INFO:teuthology.orchestra.run.smithi044.stdout: librados-devel-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.618 INFO:teuthology.orchestra.run.smithi044.stdout: libradosstriper1-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.618 INFO:teuthology.orchestra.run.smithi044.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T22:57:15.618 INFO:teuthology.orchestra.run.smithi044.stdout: librgw2-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.618 INFO:teuthology.orchestra.run.smithi044.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T22:57:15.618 INFO:teuthology.orchestra.run.smithi044.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T22:57:15.618 INFO:teuthology.orchestra.run.smithi044.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T22:57:15.618 INFO:teuthology.orchestra.run.smithi044.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T22:57:15.618 INFO:teuthology.orchestra.run.smithi044.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T22:57:15.618 INFO:teuthology.orchestra.run.smithi044.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T22:57:15.619 INFO:teuthology.orchestra.run.smithi044.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T22:57:15.619 INFO:teuthology.orchestra.run.smithi044.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T22:57:15.619 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T22:57:15.619 INFO:teuthology.orchestra.run.smithi044.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T22:57:15.619 INFO:teuthology.orchestra.run.smithi044.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T22:57:15.619 INFO:teuthology.orchestra.run.smithi044.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T22:57:15.619 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T22:57:15.619 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T22:57:15.619 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T22:57:15.619 INFO:teuthology.orchestra.run.smithi044.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T22:57:15.619 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T22:57:15.619 INFO:teuthology.orchestra.run.smithi044.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T22:57:15.619 INFO:teuthology.orchestra.run.smithi044.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T22:57:15.619 INFO:teuthology.orchestra.run.smithi044.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T22:57:15.620 INFO:teuthology.orchestra.run.smithi044.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T22:57:15.620 INFO:teuthology.orchestra.run.smithi044.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T22:57:15.620 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T22:57:15.620 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-argparse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.620 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ceph-common-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.620 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cephfs-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.620 INFO:teuthology.orchestra.run.smithi044.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T22:57:15.620 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T22:57:15.620 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T22:57:15.620 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T22:57:15.620 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T22:57:15.620 INFO:teuthology.orchestra.run.smithi044.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T22:57:15.620 INFO:teuthology.orchestra.run.smithi044.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T22:57:15.620 INFO:teuthology.orchestra.run.smithi044.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T22:57:15.621 INFO:teuthology.orchestra.run.smithi044.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T22:57:15.621 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T22:57:15.621 INFO:teuthology.orchestra.run.smithi044.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T22:57:15.621 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T22:57:15.621 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T22:57:15.621 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T22:57:15.621 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T22:57:15.621 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T22:57:15.621 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T22:57:15.621 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T22:57:15.621 INFO:teuthology.orchestra.run.smithi044.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T22:57:15.621 INFO:teuthology.orchestra.run.smithi044.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T22:57:15.621 INFO:teuthology.orchestra.run.smithi044.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T22:57:15.622 INFO:teuthology.orchestra.run.smithi044.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T22:57:15.622 INFO:teuthology.orchestra.run.smithi044.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T22:57:15.622 INFO:teuthology.orchestra.run.smithi044.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T22:57:15.622 INFO:teuthology.orchestra.run.smithi044.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T22:57:15.622 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T22:57:15.622 INFO:teuthology.orchestra.run.smithi044.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T22:57:15.622 INFO:teuthology.orchestra.run.smithi044.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T22:57:15.622 INFO:teuthology.orchestra.run.smithi044.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T22:57:15.622 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T22:57:15.622 INFO:teuthology.orchestra.run.smithi044.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T22:57:15.622 INFO:teuthology.orchestra.run.smithi044.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T22:57:15.622 INFO:teuthology.orchestra.run.smithi044.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T22:57:15.623 INFO:teuthology.orchestra.run.smithi044.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T22:57:15.623 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T22:57:15.623 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T22:57:15.623 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T22:57:15.623 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T22:57:15.623 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T22:57:15.623 INFO:teuthology.orchestra.run.smithi044.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T22:57:15.623 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rados-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.623 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.623 INFO:teuthology.orchestra.run.smithi044.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T22:57:15.623 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T22:57:15.623 INFO:teuthology.orchestra.run.smithi044.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T22:57:15.623 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rgw-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.624 INFO:teuthology.orchestra.run.smithi044.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T22:57:15.624 INFO:teuthology.orchestra.run.smithi044.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T22:57:15.624 INFO:teuthology.orchestra.run.smithi044.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T22:57:15.624 INFO:teuthology.orchestra.run.smithi044.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T22:57:15.624 INFO:teuthology.orchestra.run.smithi044.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T22:57:15.624 INFO:teuthology.orchestra.run.smithi044.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T22:57:15.624 INFO:teuthology.orchestra.run.smithi044.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T22:57:15.624 INFO:teuthology.orchestra.run.smithi044.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T22:57:15.624 INFO:teuthology.orchestra.run.smithi044.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T22:57:15.624 INFO:teuthology.orchestra.run.smithi044.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T22:57:15.624 INFO:teuthology.orchestra.run.smithi044.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T22:57:15.624 INFO:teuthology.orchestra.run.smithi044.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T22:57:15.624 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T22:57:15.624 INFO:teuthology.orchestra.run.smithi044.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T22:57:15.625 INFO:teuthology.orchestra.run.smithi044.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T22:57:15.625 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-fuse-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.625 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-mirror-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.625 INFO:teuthology.orchestra.run.smithi044.stdout: rbd-nbd-2:19.0.0-2405.g820a2f8c.el9.x86_64 2024-04-21T22:57:15.625 INFO:teuthology.orchestra.run.smithi044.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T22:57:15.625 INFO:teuthology.orchestra.run.smithi044.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T22:57:15.625 INFO:teuthology.orchestra.run.smithi044.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T22:57:15.625 INFO:teuthology.orchestra.run.smithi044.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T22:57:15.625 INFO:teuthology.orchestra.run.smithi044.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T22:57:15.625 INFO:teuthology.orchestra.run.smithi044.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T22:57:15.626 INFO:teuthology.orchestra.run.smithi044.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T22:57:15.626 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:57:15.626 INFO:teuthology.orchestra.run.smithi044.stdout:Complete! 2024-04-21T22:57:15.989 DEBUG:teuthology.parallel:result is None 2024-04-21T22:57:15.990 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T22:57:16.209 DEBUG:teuthology.orchestra.run.smithi044:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T22:57:16.230 INFO:teuthology.orchestra.run.smithi044.stdout:19.0.0-2405.g820a2f8c.el9 2024-04-21T22:57:16.231 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2405.g820a2f8c.el9 2024-04-21T22:57:16.231 INFO:teuthology.task.install:The correct ceph version 19.0.0-2405.g820a2f8c is installed. 2024-04-21T22:57:16.233 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T22:57:16.496 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T22:57:16.523 INFO:teuthology.orchestra.run.smithi055.stdout:19.0.0-2405.g820a2f8c.el9 2024-04-21T22:57:16.523 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2405.g820a2f8c.el9 2024-04-21T22:57:16.523 INFO:teuthology.task.install:The correct ceph version 19.0.0-2405.g820a2f8c is installed. 2024-04-21T22:57:16.525 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T22:57:16.662 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T22:57:16.689 INFO:teuthology.orchestra.run.smithi069.stdout:19.0.0-2405.g820a2f8c.el9 2024-04-21T22:57:16.690 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2405.g820a2f8c.el9 2024-04-21T22:57:16.690 INFO:teuthology.task.install:The correct ceph version 19.0.0-2405.g820a2f8c is installed. 2024-04-21T22:57:16.692 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-21T22:57:16.692 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T22:57:16.692 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T22:57:16.720 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-21T22:57:16.720 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T22:57:16.752 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-21T22:57:16.752 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T22:57:16.781 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-21T22:57:16.781 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T22:57:16.781 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T22:57:16.806 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T22:57:16.871 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-21T22:57:16.871 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T22:57:16.900 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T22:57:16.967 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-21T22:57:16.967 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T22:57:16.995 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T22:57:17.062 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-21T22:57:17.062 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T22:57:17.062 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T22:57:17.089 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T22:57:17.153 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-21T22:57:17.153 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T22:57:17.179 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T22:57:17.243 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-21T22:57:17.243 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T22:57:17.270 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T22:57:17.337 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-21T22:57:17.337 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T22:57:17.337 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T22:57:17.362 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T22:57:17.427 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-21T22:57:17.427 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T22:57:17.453 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T22:57:17.518 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-21T22:57:17.519 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T22:57:17.546 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T22:57:17.615 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-21T22:57:17.709 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 1, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'subvols': {'create': 2}} 2024-04-21T22:57:17.709 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T22:57:17.710 INFO:tasks.cephadm:Cluster fsid is 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T22:57:17.710 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-21T22:57:17.710 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.44', 'mon.b': '172.21.15.55', 'mon.c': '172.21.15.69'} 2024-04-21T22:57:17.710 INFO:tasks.cephadm:First mon is mon.a on smithi044 2024-04-21T22:57:17.710 INFO:tasks.cephadm:First mgr is x 2024-04-21T22:57:17.710 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-21T22:57:17.710 DEBUG:teuthology.orchestra.run.smithi044:> sudo hostname $(hostname -s) 2024-04-21T22:57:17.735 DEBUG:teuthology.orchestra.run.smithi055:> sudo hostname $(hostname -s) 2024-04-21T22:57:17.761 DEBUG:teuthology.orchestra.run.smithi069:> sudo hostname $(hostname -s) 2024-04-21T22:57:17.789 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-21T22:57:17.789 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T22:57:17.933 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '820a2f8ced66152ed3d765c31444b8076ea28c6b', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54945/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2405-g820a2f8c', 'node_name': '172.21.4.69+confusa07', 'job_name': 'ceph-dev-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2405.g820a2f8c'}, 'url': 'https://2.chacra.ceph.com/r/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-20 17:52:04.986240', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/flavors/default/', 'archs': ['x86_64', 'source', 'arm64'], 'distro': 'centos'}] 2024-04-21T22:57:18.302 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref squid, sha1 820a2f8ced66152ed3d765c31444b8076ea28c6b from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T22:57:18.303 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/x86_64/flavors/default/cephadm 2024-04-21T22:57:18.303 DEBUG:teuthology.orchestra.run.smithi044:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T22:57:19.061 INFO:teuthology.orchestra.run.smithi044.stdout:-rw-r--r--. 1 ubuntu ubuntu 781418 Apr 21 22:57 /home/ubuntu/cephtest/cephadm 2024-04-21T22:57:19.062 DEBUG:teuthology.orchestra.run.smithi055:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T22:57:19.867 INFO:teuthology.orchestra.run.smithi055.stdout:-rw-r--r--. 1 ubuntu ubuntu 781418 Apr 21 22:57 /home/ubuntu/cephtest/cephadm 2024-04-21T22:57:19.868 DEBUG:teuthology.orchestra.run.smithi069:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/820a2f8ced66152ed3d765c31444b8076ea28c6b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T22:57:20.517 INFO:teuthology.orchestra.run.smithi069.stdout:-rw-r--r--. 1 ubuntu ubuntu 781418 Apr 21 22:57 /home/ubuntu/cephtest/cephadm 2024-04-21T22:57:20.517 DEBUG:teuthology.orchestra.run.smithi044:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T22:57:20.537 DEBUG:teuthology.orchestra.run.smithi055:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T22:57:20.556 DEBUG:teuthology.orchestra.run.smithi069:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T22:57:20.591 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b on all hosts... 2024-04-21T22:57:20.592 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b pull 2024-04-21T22:57:20.594 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b pull 2024-04-21T22:57:20.599 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b pull 2024-04-21T22:57:22.104 INFO:teuthology.orchestra.run.smithi044.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b... 2024-04-21T22:57:22.181 INFO:teuthology.orchestra.run.smithi055.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b... 2024-04-21T22:57:22.258 INFO:teuthology.orchestra.run.smithi069.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b... 2024-04-21T22:57:55.475 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-21T22:57:55.475 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-21T22:57:55.476 INFO:teuthology.orchestra.run.smithi069.stdout: "image_id": "6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32", 2024-04-21T22:57:55.476 INFO:teuthology.orchestra.run.smithi069.stdout: "repo_digests": [ 2024-04-21T22:57:55.476 INFO:teuthology.orchestra.run.smithi069.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9" 2024-04-21T22:57:55.476 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-21T22:57:55.476 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-21T22:58:13.267 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T22:58:13.268 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-21T22:58:13.268 INFO:teuthology.orchestra.run.smithi044.stdout: "image_id": "6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32", 2024-04-21T22:58:13.268 INFO:teuthology.orchestra.run.smithi044.stdout: "repo_digests": [ 2024-04-21T22:58:13.268 INFO:teuthology.orchestra.run.smithi044.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9" 2024-04-21T22:58:13.268 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-21T22:58:13.268 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T22:58:13.748 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-04-21T22:58:13.748 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-21T22:58:13.748 INFO:teuthology.orchestra.run.smithi055.stdout: "image_id": "6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32", 2024-04-21T22:58:13.748 INFO:teuthology.orchestra.run.smithi055.stdout: "repo_digests": [ 2024-04-21T22:58:13.748 INFO:teuthology.orchestra.run.smithi055.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9" 2024-04-21T22:58:13.748 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2024-04-21T22:58:13.748 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-04-21T22:58:13.776 DEBUG:teuthology.orchestra.run.smithi044:> sudo mkdir -p /etc/ceph 2024-04-21T22:58:13.808 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /etc/ceph 2024-04-21T22:58:13.837 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph 2024-04-21T22:58:13.869 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 777 /etc/ceph 2024-04-21T22:58:13.897 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 777 /etc/ceph 2024-04-21T22:58:13.923 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 777 /etc/ceph 2024-04-21T22:58:13.951 INFO:tasks.cephadm:Writing seed config... 2024-04-21T22:58:13.952 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-21T22:58:13.952 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-21T22:58:13.952 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-21T22:58:13.952 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-21T22:58:13.952 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-21T22:58:13.953 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-21T22:58:13.953 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-21T22:58:13.953 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-21T22:58:13.953 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-21T22:58:13.953 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-21T22:58:13.953 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-21T22:58:13.953 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-21T22:58:13.953 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-21T22:58:13.953 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-21T22:58:13.953 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-21T22:58:13.953 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-21T22:58:13.953 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-21T22:58:13.953 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-21T22:58:13.953 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-21T22:58:13.954 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-21T22:58:13.954 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-21T22:58:13.954 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-21T22:58:13.954 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-21T22:58:13.954 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-21T22:58:13.954 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-21T22:58:13.954 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-21T22:58:13.954 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-21T22:58:13.954 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-21T22:58:13.954 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-21T22:58:13.954 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-21T22:58:13.954 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-21T22:58:13.954 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-21T22:58:13.954 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-21T22:58:13.954 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-21T22:58:13.955 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-21T22:58:13.955 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-21T22:58:13.955 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-21T22:58:13.955 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-21T22:58:13.955 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-21T22:58:13.955 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-21T22:58:13.955 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-21T22:58:13.955 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-21T22:58:13.955 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-21T22:58:13.955 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-21T22:58:13.955 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-21T22:58:13.956 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T22:58:13.956 DEBUG:teuthology.orchestra.run.smithi044:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-21T22:58:13.974 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 80909b94-0032-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-21T22:58:13.974 DEBUG:teuthology.orchestra.run.smithi044:mon.a> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mon.a.service 2024-04-21T22:58:14.017 DEBUG:teuthology.orchestra.run.smithi044:mgr.x> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mgr.x.service 2024-04-21T22:58:14.060 INFO:tasks.cephadm:Bootstrapping... 2024-04-21T22:58:14.060 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b -v bootstrap --fsid 80909b94-0032-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.44 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-21T22:58:14.262 INFO:teuthology.orchestra.run.smithi044.stdout:-------------------------------------------------------------------------------- 2024-04-21T22:58:14.262 INFO:teuthology.orchestra.run.smithi044.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b', '-v', 'bootstrap', '--fsid', '80909b94-0032-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.44', '--skip-admin-label'] 2024-04-21T22:58:14.262 INFO:teuthology.orchestra.run.smithi044.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-21T22:58:14.262 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying podman|docker is present... 2024-04-21T22:58:14.290 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-21T22:58:14.290 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying lvm2 is present... 2024-04-21T22:58:14.290 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying time synchronization is in place... 2024-04-21T22:58:14.296 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-21T22:58:14.296 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-21T22:58:14.302 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-21T22:58:14.303 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-04-21T22:58:14.309 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout enabled 2024-04-21T22:58:14.315 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout active 2024-04-21T22:58:14.315 INFO:teuthology.orchestra.run.smithi044.stdout:Unit chronyd.service is enabled and running 2024-04-21T22:58:14.315 INFO:teuthology.orchestra.run.smithi044.stdout:Repeating the final host check... 2024-04-21T22:58:14.340 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-21T22:58:14.340 INFO:teuthology.orchestra.run.smithi044.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-21T22:58:14.340 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl is present 2024-04-21T22:58:14.340 INFO:teuthology.orchestra.run.smithi044.stdout:lvcreate is present 2024-04-21T22:58:14.346 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-21T22:58:14.346 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-21T22:58:14.352 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-21T22:58:14.353 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-04-21T22:58:14.359 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout enabled 2024-04-21T22:58:14.365 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout active 2024-04-21T22:58:14.365 INFO:teuthology.orchestra.run.smithi044.stdout:Unit chronyd.service is enabled and running 2024-04-21T22:58:14.365 INFO:teuthology.orchestra.run.smithi044.stdout:Host looks OK 2024-04-21T22:58:14.365 INFO:teuthology.orchestra.run.smithi044.stdout:Cluster fsid: 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T22:58:14.365 INFO:teuthology.orchestra.run.smithi044.stdout:Acquiring lock 140674860639136 on /run/cephadm/80909b94-0032-11ef-bc93-c7b262605968.lock 2024-04-21T22:58:14.366 INFO:teuthology.orchestra.run.smithi044.stdout:Lock 140674860639136 acquired on /run/cephadm/80909b94-0032-11ef-bc93-c7b262605968.lock 2024-04-21T22:58:14.366 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying IP 172.21.15.44 port 3300 ... 2024-04-21T22:58:14.366 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying IP 172.21.15.44 port 6789 ... 2024-04-21T22:58:14.366 INFO:teuthology.orchestra.run.smithi044.stdout:Base mon IP(s) is [172.21.15.44:3300, 172.21.15.44:6789], mon addrv is [v2:172.21.15.44:3300,v1:172.21.15.44:6789] 2024-04-21T22:58:14.369 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.44 metric 100 2024-04-21T22:58:14.370 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.44 metric 100 2024-04-21T22:58:14.372 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-21T22:58:14.373 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-21T22:58:14.373 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-21T22:58:14.375 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-21T22:58:14.375 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-21T22:58:14.375 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-21T22:58:14.375 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-21T22:58:14.375 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-21T22:58:14.375 INFO:teuthology.orchestra.run.smithi044.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-21T22:58:14.375 INFO:teuthology.orchestra.run.smithi044.stdout:Mon IP `172.21.15.44` is in CIDR network `172.21.0.0/20` 2024-04-21T22:58:14.376 INFO:teuthology.orchestra.run.smithi044.stdout:Mon IP `172.21.15.44` is in CIDR network `172.21.0.0/20` 2024-04-21T22:58:14.376 INFO:teuthology.orchestra.run.smithi044.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-21T22:58:14.376 INFO:teuthology.orchestra.run.smithi044.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-21T22:58:14.377 INFO:teuthology.orchestra.run.smithi044.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b... 2024-04-21T22:58:15.070 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stdout 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 2024-04-21T22:58:15.070 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b... 2024-04-21T22:58:15.070 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Getting image source signatures 2024-04-21T22:58:15.070 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Copying blob sha256:4706d97462bc6e0886c6ef3c0b1ac67e3e6a7e63a606e29958996d1fc15dafd8 2024-04-21T22:58:15.070 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-21T22:58:15.070 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Copying config sha256:6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 2024-04-21T22:58:15.070 INFO:teuthology.orchestra.run.smithi044.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-21T22:58:15.999 INFO:teuthology.orchestra.run.smithi044.stdout:ceph: stdout ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev) 2024-04-21T22:58:15.999 INFO:teuthology.orchestra.run.smithi044.stdout:Ceph version: ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev) 2024-04-21T22:58:15.999 INFO:teuthology.orchestra.run.smithi044.stdout:Extracting ceph user uid/gid from container image... 2024-04-21T22:58:16.964 INFO:teuthology.orchestra.run.smithi044.stdout:stat: stdout 167 167 2024-04-21T22:58:16.964 INFO:teuthology.orchestra.run.smithi044.stdout:Creating initial keys... 2024-04-21T22:58:17.868 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph-authtool: stdout AQAJmiVm9zfZDxAAkwR/kNWIWGjZvlOTt0hMSg== 2024-04-21T22:58:18.745 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph-authtool: stdout AQAKmiVm3GFzCxAAx3IV4AnmchzpfqBFeYE6TA== 2024-04-21T22:58:19.635 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph-authtool: stdout AQALmiVmlnHkAxAA1rk7BeYCnKsIpPD56gdRyQ== 2024-04-21T22:58:19.636 INFO:teuthology.orchestra.run.smithi044.stdout:Creating initial monmap... 2024-04-21T22:58:20.619 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-21T22:58:20.620 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-21T22:58:20.620 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T22:58:20.620 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-21T22:58:20.620 INFO:teuthology.orchestra.run.smithi044.stdout:monmaptool for a [v2:172.21.15.44:3300,v1:172.21.15.44:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-21T22:58:20.621 INFO:teuthology.orchestra.run.smithi044.stdout:setting min_mon_release = quincy 2024-04-21T22:58:20.621 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: set fsid to 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T22:58:20.621 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-21T22:58:20.621 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:58:20.621 INFO:teuthology.orchestra.run.smithi044.stdout:Creating mon... 2024-04-21T22:58:21.675 INFO:teuthology.orchestra.run.smithi044.stdout:create mon.a on 2024-04-21T22:58:21.919 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-21T22:58:22.063 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-21T22:58:22.269 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-80909b94-0032-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-80909b94-0032-11ef-bc93-c7b262605968.target. 2024-04-21T22:58:22.269 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-80909b94-0032-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-80909b94-0032-11ef-bc93-c7b262605968.target. 2024-04-21T22:58:22.534 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-80909b94-0032-11ef-bc93-c7b262605968@mon.a 2024-04-21T22:58:22.534 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Failed to reset failed state of unit ceph-80909b94-0032-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-80909b94-0032-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-21T22:58:22.714 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-80909b94-0032-11ef-bc93-c7b262605968.target.wants/ceph-80909b94-0032-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-80909b94-0032-11ef-bc93-c7b262605968@.service. 2024-04-21T22:58:23.078 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:22 smithi044 podman[27549]: 2024-04-21 22:58:22.975414582 +0000 UTC m=+0.127513485 container create 9b65439df42aff16bb06fc8b2b298150b7ebfb1e0f1ce9095f1bf185e83cb05a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-21T22:58:23.234 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T22:58:23.234 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout disabled 2024-04-21T22:58:23.241 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T22:58:23.241 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-04-21T22:58:23.241 INFO:teuthology.orchestra.run.smithi044.stdout:firewalld.service is not enabled 2024-04-21T22:58:23.241 INFO:teuthology.orchestra.run.smithi044.stdout:Not possible to enable service . firewalld.service is not available 2024-04-21T22:58:23.241 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mon to start... 2024-04-21T22:58:23.242 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mon... 2024-04-21T22:58:23.336 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:23 smithi044 podman[27549]: 2024-04-21 22:58:23.152497437 +0000 UTC m=+0.304596333 container init 9b65439df42aff16bb06fc8b2b298150b7ebfb1e0f1ce9095f1bf185e83cb05a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T22:58:23.336 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:23 smithi044 podman[27549]: 2024-04-21 22:58:23.155711117 +0000 UTC m=+0.307810020 container start 9b65439df42aff16bb06fc8b2b298150b7ebfb1e0f1ce9095f1bf185e83cb05a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True) 2024-04-21T22:58:23.337 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:23 smithi044 ceph-mon[27567]: mkfs 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T22:58:23.337 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:23 smithi044 ceph-mon[27567]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-21T22:58:23.337 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:23 smithi044 bash[27549]: 9b65439df42aff16bb06fc8b2b298150b7ebfb1e0f1ce9095f1bf185e83cb05a 2024-04-21T22:58:23.337 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:23 smithi044 systemd[1]: Started Ceph mon.a for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T22:58:24.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:24 smithi044 ceph-mon[27567]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-21T22:58:24.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:24 smithi044 ceph-mon[27567]: monmap epoch 1 2024-04-21T22:58:24.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:24 smithi044 ceph-mon[27567]: fsid 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T22:58:24.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:24 smithi044 ceph-mon[27567]: last_changed 2024-04-21T22:58:20.078074+0000 2024-04-21T22:58:24.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:24 smithi044 ceph-mon[27567]: created 2024-04-21T22:58:20.078074+0000 2024-04-21T22:58:24.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:24 smithi044 ceph-mon[27567]: min_mon_release 19 (squid) 2024-04-21T22:58:24.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:24 smithi044 ceph-mon[27567]: election_strategy: 1 2024-04-21T22:58:24.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:24 smithi044 ceph-mon[27567]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-04-21T22:58:24.314 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:24 smithi044 ceph-mon[27567]: fsmap 2024-04-21T22:58:24.315 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:24 smithi044 ceph-mon[27567]: osdmap e1: 0 total, 0 up, 0 in 2024-04-21T22:58:24.315 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:24 smithi044 ceph-mon[27567]: mgrmap e1: no daemons active 2024-04-21T22:58:24.315 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:24 smithi044 ceph-mon[27567]: from='client.? 172.21.15.44:0/3771114489' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-21T22:58:24.321 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout cluster: 2024-04-21T22:58:24.322 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout id: 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T22:58:24.322 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-21T22:58:24.322 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-21T22:58:24.322 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout services: 2024-04-21T22:58:24.322 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.517961s) 2024-04-21T22:58:24.322 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-21T22:58:24.322 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-21T22:58:24.322 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-21T22:58:24.322 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout data: 2024-04-21T22:58:24.322 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-21T22:58:24.322 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-21T22:58:24.323 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-21T22:58:24.323 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout pgs: 2024-04-21T22:58:24.323 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-21T22:58:24.323 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.631+0000 7f9f301a0700 1 Processor -- start 2024-04-21T22:58:24.323 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.631+0000 7f9f301a0700 1 -- start start 2024-04-21T22:58:24.323 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.631+0000 7f9f301a0700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f28107c10 0x7f9f28108030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:24.323 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.631+0000 7f9f301a0700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9f28108600 con 0x7f9f28107c10 2024-04-21T22:58:24.323 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.632+0000 7f9f2df3c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f28107c10 0x7f9f28108030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:24.323 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.632+0000 7f9f2df3c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f28107c10 0x7f9f28108030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39554/0 (socket says 172.21.15.44:39554) 2024-04-21T22:58:24.323 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.632+0000 7f9f2df3c700 1 -- 172.21.15.44:0/1061814401 learned_addr learned my addr 172.21.15.44:0/1061814401 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:24.323 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.632+0000 7f9f2df3c700 1 -- 172.21.15.44:0/1061814401 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f28108d70 con 0x7f9f28107c10 2024-04-21T22:58:24.323 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.632+0000 7f9f2df3c700 1 --2- 172.21.15.44:0/1061814401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f28107c10 0x7f9f28108030 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9f180098a0 tx=0x7f9f18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ef668ff8b1066e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:24.324 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.633+0000 7f9f2cf3a700 1 -- 172.21.15.44:0/1061814401 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f18004160 con 0x7f9f28107c10 2024-04-21T22:58:24.324 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.633+0000 7f9f2cf3a700 1 -- 172.21.15.44:0/1061814401 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f18004300 con 0x7f9f28107c10 2024-04-21T22:58:24.324 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.633+0000 7f9f2cf3a700 1 -- 172.21.15.44:0/1061814401 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f180036a0 con 0x7f9f28107c10 2024-04-21T22:58:24.324 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.633+0000 7f9f301a0700 1 -- 172.21.15.44:0/1061814401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f28107c10 msgr2=0x7f9f28108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:24.324 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.633+0000 7f9f301a0700 1 --2- 172.21.15.44:0/1061814401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f28107c10 0x7f9f28108030 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9f180098a0 tx=0x7f9f18009bb0 comp rx=0 tx=0).stop 2024-04-21T22:58:24.324 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.633+0000 7f9f301a0700 1 -- 172.21.15.44:0/1061814401 shutdown_connections 2024-04-21T22:58:24.324 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.633+0000 7f9f301a0700 1 --2- 172.21.15.44:0/1061814401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f28107c10 0x7f9f28108030 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:24.324 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.633+0000 7f9f301a0700 1 -- 172.21.15.44:0/1061814401 >> 172.21.15.44:0/1061814401 conn(0x7f9f28103230 msgr2=0x7f9f28105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:24.324 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.633+0000 7f9f301a0700 1 -- 172.21.15.44:0/1061814401 shutdown_connections 2024-04-21T22:58:24.324 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.633+0000 7f9f301a0700 1 -- 172.21.15.44:0/1061814401 wait complete. 2024-04-21T22:58:24.324 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.634+0000 7f9f301a0700 1 Processor -- start 2024-04-21T22:58:24.324 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.634+0000 7f9f301a0700 1 -- start start 2024-04-21T22:58:24.324 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.634+0000 7f9f301a0700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f28107c10 0x7f9f2819b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:24.324 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.634+0000 7f9f301a0700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9f281092a0 con 0x7f9f28107c10 2024-04-21T22:58:24.325 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.634+0000 7f9f2df3c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f28107c10 0x7f9f2819b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:24.325 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.634+0000 7f9f2df3c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f28107c10 0x7f9f2819b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39570/0 (socket says 172.21.15.44:39570) 2024-04-21T22:58:24.325 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.634+0000 7f9f2df3c700 1 -- 172.21.15.44:0/3771114489 learned_addr learned my addr 172.21.15.44:0/3771114489 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:24.325 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.635+0000 7f9f2df3c700 1 -- 172.21.15.44:0/3771114489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f2819b810 con 0x7f9f28107c10 2024-04-21T22:58:24.325 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.635+0000 7f9f2df3c700 1 --2- 172.21.15.44:0/3771114489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f28107c10 0x7f9f2819b2d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9f18000c00 tx=0x7f9f18004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:24.325 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.635+0000 7f9f1effd700 1 -- 172.21.15.44:0/3771114489 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f18003db0 con 0x7f9f28107c10 2024-04-21T22:58:24.325 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.635+0000 7f9f1effd700 1 -- 172.21.15.44:0/3771114489 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f18011630 con 0x7f9f28107c10 2024-04-21T22:58:24.325 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.635+0000 7f9f301a0700 1 -- 172.21.15.44:0/3771114489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f2819baa0 con 0x7f9f28107c10 2024-04-21T22:58:24.325 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.635+0000 7f9f1effd700 1 -- 172.21.15.44:0/3771114489 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f18011950 con 0x7f9f28107c10 2024-04-21T22:58:24.325 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.635+0000 7f9f301a0700 1 -- 172.21.15.44:0/3771114489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f2819e790 con 0x7f9f28107c10 2024-04-21T22:58:24.325 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.635+0000 7f9f1effd700 1 -- 172.21.15.44:0/3771114489 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9f1801a460 con 0x7f9f28107c10 2024-04-21T22:58:24.325 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.635+0000 7f9f1effd700 1 -- 172.21.15.44:0/3771114489 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9f1801b780 con 0x7f9f28107c10 2024-04-21T22:58:24.325 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.636+0000 7f9f301a0700 1 -- 172.21.15.44:0/3771114489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f28063890 con 0x7f9f28107c10 2024-04-21T22:58:24.326 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.638+0000 7f9f1effd700 1 -- 172.21.15.44:0/3771114489 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9f18031020 con 0x7f9f28107c10 2024-04-21T22:58:24.326 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.711+0000 7f9f301a0700 1 -- 172.21.15.44:0/3771114489 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f9f2802dd80 con 0x7f9f28107c10 2024-04-21T22:58:24.326 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.712+0000 7f9f1effd700 1 -- 172.21.15.44:0/3771114489 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f9f1801b9d0 con 0x7f9f28107c10 2024-04-21T22:58:24.326 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.712+0000 7f9f301a0700 1 -- 172.21.15.44:0/3771114489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f28107c10 msgr2=0x7f9f2819b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:24.326 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.712+0000 7f9f301a0700 1 --2- 172.21.15.44:0/3771114489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f28107c10 0x7f9f2819b2d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9f18000c00 tx=0x7f9f18004750 comp rx=0 tx=0).stop 2024-04-21T22:58:24.326 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.712+0000 7f9f301a0700 1 -- 172.21.15.44:0/3771114489 shutdown_connections 2024-04-21T22:58:24.326 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.712+0000 7f9f301a0700 1 --2- 172.21.15.44:0/3771114489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9f28107c10 0x7f9f2819b2d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:24.326 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.712+0000 7f9f301a0700 1 -- 172.21.15.44:0/3771114489 >> 172.21.15.44:0/3771114489 conn(0x7f9f28103230 msgr2=0x7f9f28191f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:24.326 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.713+0000 7f9f301a0700 1 -- 172.21.15.44:0/3771114489 shutdown_connections 2024-04-21T22:58:24.326 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:23.713+0000 7f9f301a0700 1 -- 172.21.15.44:0/3771114489 wait complete. 2024-04-21T22:58:24.326 INFO:teuthology.orchestra.run.smithi044.stdout:mon is available 2024-04-21T22:58:24.326 INFO:teuthology.orchestra.run.smithi044.stdout:Assimilating anything we can from ceph.conf... 2024-04-21T22:58:25.363 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-21T22:58:25.363 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [global] 2024-04-21T22:58:25.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout fsid = 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T22:58:25.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.44:3300,v1:172.21.15.44:6789] 2024-04-21T22:58:25.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-21T22:58:25.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-21T22:58:25.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-21T22:58:25.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-21T22:58:25.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-21T22:58:25.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-21T22:58:25.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-21T22:58:25.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-21T22:58:25.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [osd] 2024-04-21T22:58:25.364 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-21T22:58:25.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-21T22:58:25.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-21T22:58:25.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-21T22:58:25.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-21T22:58:25.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.729+0000 7fc37ab77700 1 Processor -- start 2024-04-21T22:58:25.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.729+0000 7fc37ab77700 1 -- start start 2024-04-21T22:58:25.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.729+0000 7fc37ab77700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc374107c10 0x7fc374108030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:25.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.729+0000 7fc37ab77700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc374108600 con 0x7fc374107c10 2024-04-21T22:58:25.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.729+0000 7fc378913700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc374107c10 0x7fc374108030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:25.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.729+0000 7fc378913700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc374107c10 0x7fc374108030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39580/0 (socket says 172.21.15.44:39580) 2024-04-21T22:58:25.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.729+0000 7fc378913700 1 -- 172.21.15.44:0/1311598533 learned_addr learned my addr 172.21.15.44:0/1311598533 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:25.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.730+0000 7fc378913700 1 -- 172.21.15.44:0/1311598533 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc374108d70 con 0x7fc374107c10 2024-04-21T22:58:25.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.730+0000 7fc378913700 1 --2- 172.21.15.44:0/1311598533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc374107c10 0x7fc374108030 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc3680098a0 tx=0x7fc368009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f101f8a1acf1a2e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:25.365 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.730+0000 7fc3737fe700 1 -- 172.21.15.44:0/1311598533 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc368004160 con 0x7fc374107c10 2024-04-21T22:58:25.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.730+0000 7fc3737fe700 1 -- 172.21.15.44:0/1311598533 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc368004300 con 0x7fc374107c10 2024-04-21T22:58:25.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.730+0000 7fc3737fe700 1 -- 172.21.15.44:0/1311598533 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3680036a0 con 0x7fc374107c10 2024-04-21T22:58:25.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.730+0000 7fc37ab77700 1 -- 172.21.15.44:0/1311598533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc374107c10 msgr2=0x7fc374108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:25.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.730+0000 7fc37ab77700 1 --2- 172.21.15.44:0/1311598533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc374107c10 0x7fc374108030 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc3680098a0 tx=0x7fc368009bb0 comp rx=0 tx=0).stop 2024-04-21T22:58:25.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.730+0000 7fc37ab77700 1 -- 172.21.15.44:0/1311598533 shutdown_connections 2024-04-21T22:58:25.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.731+0000 7fc37ab77700 1 --2- 172.21.15.44:0/1311598533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc374107c10 0x7fc374108030 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:25.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.731+0000 7fc37ab77700 1 -- 172.21.15.44:0/1311598533 >> 172.21.15.44:0/1311598533 conn(0x7fc374103230 msgr2=0x7fc374105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:25.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.731+0000 7fc37ab77700 1 -- 172.21.15.44:0/1311598533 shutdown_connections 2024-04-21T22:58:25.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.731+0000 7fc37ab77700 1 -- 172.21.15.44:0/1311598533 wait complete. 2024-04-21T22:58:25.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.731+0000 7fc37ab77700 1 Processor -- start 2024-04-21T22:58:25.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.731+0000 7fc37ab77700 1 -- start start 2024-04-21T22:58:25.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.731+0000 7fc37ab77700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc374107c10 0x7fc37419b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:25.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.731+0000 7fc37ab77700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc3741092a0 con 0x7fc374107c10 2024-04-21T22:58:25.366 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.732+0000 7fc378913700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc374107c10 0x7fc37419b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:25.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.732+0000 7fc378913700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc374107c10 0x7fc37419b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39594/0 (socket says 172.21.15.44:39594) 2024-04-21T22:58:25.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.732+0000 7fc378913700 1 -- 172.21.15.44:0/730082842 learned_addr learned my addr 172.21.15.44:0/730082842 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:25.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.732+0000 7fc378913700 1 -- 172.21.15.44:0/730082842 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc37419b810 con 0x7fc374107c10 2024-04-21T22:58:25.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.732+0000 7fc378913700 1 --2- 172.21.15.44:0/730082842 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc374107c10 0x7fc37419b2d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc368000c00 tx=0x7fc368004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:25.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.732+0000 7fc371ffb700 1 -- 172.21.15.44:0/730082842 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc368003db0 con 0x7fc374107c10 2024-04-21T22:58:25.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.732+0000 7fc371ffb700 1 -- 172.21.15.44:0/730082842 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc368011630 con 0x7fc374107c10 2024-04-21T22:58:25.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.732+0000 7fc37ab77700 1 -- 172.21.15.44:0/730082842 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc37419baa0 con 0x7fc374107c10 2024-04-21T22:58:25.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.732+0000 7fc371ffb700 1 -- 172.21.15.44:0/730082842 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc368011950 con 0x7fc374107c10 2024-04-21T22:58:25.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.732+0000 7fc37ab77700 1 -- 172.21.15.44:0/730082842 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc37419e790 con 0x7fc374107c10 2024-04-21T22:58:25.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.733+0000 7fc371ffb700 1 -- 172.21.15.44:0/730082842 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc36801a460 con 0x7fc374107c10 2024-04-21T22:58:25.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.733+0000 7fc371ffb700 1 -- 172.21.15.44:0/730082842 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc36801b780 con 0x7fc374107c10 2024-04-21T22:58:25.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.733+0000 7fc37ab77700 1 -- 172.21.15.44:0/730082842 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc374063890 con 0x7fc374107c10 2024-04-21T22:58:25.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.736+0000 7fc371ffb700 1 -- 172.21.15.44:0/730082842 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc36801b980 con 0x7fc374107c10 2024-04-21T22:58:25.367 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.806+0000 7fc37ab77700 1 -- 172.21.15.44:0/730082842 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fc37419bf80 con 0x7fc374107c10 2024-04-21T22:58:25.368 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.809+0000 7fc371ffb700 1 -- 172.21.15.44:0/730082842 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7fc37419bf80 con 0x7fc374107c10 2024-04-21T22:58:25.368 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.810+0000 7fc371ffb700 1 -- 172.21.15.44:0/730082842 <== mon.0 v2:172.21.15.44:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc36801ada0 con 0x7fc374107c10 2024-04-21T22:58:25.368 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.811+0000 7fc37ab77700 1 -- 172.21.15.44:0/730082842 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc374107c10 msgr2=0x7fc37419b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:25.368 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.811+0000 7fc37ab77700 1 --2- 172.21.15.44:0/730082842 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc374107c10 0x7fc37419b2d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc368000c00 tx=0x7fc368004750 comp rx=0 tx=0).stop 2024-04-21T22:58:25.368 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.811+0000 7fc37ab77700 1 -- 172.21.15.44:0/730082842 shutdown_connections 2024-04-21T22:58:25.368 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.811+0000 7fc37ab77700 1 --2- 172.21.15.44:0/730082842 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc374107c10 0x7fc37419b2d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:25.368 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.811+0000 7fc37ab77700 1 -- 172.21.15.44:0/730082842 >> 172.21.15.44:0/730082842 conn(0x7fc374103230 msgr2=0x7fc374191f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:25.368 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.811+0000 7fc37ab77700 1 -- 172.21.15.44:0/730082842 shutdown_connections 2024-04-21T22:58:25.368 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:24.811+0000 7fc37ab77700 1 -- 172.21.15.44:0/730082842 wait complete. 2024-04-21T22:58:25.368 INFO:teuthology.orchestra.run.smithi044.stdout:Generating new minimal ceph.conf... 2024-04-21T22:58:25.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:25 smithi044 ceph-mon[27567]: from='client.? 172.21.15.44:0/730082842' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-21T22:58:25.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:25 smithi044 ceph-mon[27567]: from='client.? 172.21.15.44:0/730082842' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-21T22:58:26.421 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:26 smithi044 ceph-mon[27567]: from='client.? 172.21.15.44:0/3173962706' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T22:58:26.424 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.763+0000 7f68af967700 1 Processor -- start 2024-04-21T22:58:26.429 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.763+0000 7f68af967700 1 -- start start 2024-04-21T22:58:26.429 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.763+0000 7f68af967700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8107c10 0x7f68a8108030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:26.429 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.763+0000 7f68af967700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f68a8108600 con 0x7f68a8107c10 2024-04-21T22:58:26.430 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.763+0000 7f68ad703700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8107c10 0x7f68a8108030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:26.430 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.763+0000 7f68ad703700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8107c10 0x7f68a8108030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39602/0 (socket says 172.21.15.44:39602) 2024-04-21T22:58:26.430 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.763+0000 7f68ad703700 1 -- 172.21.15.44:0/1633275964 learned_addr learned my addr 172.21.15.44:0/1633275964 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:26.430 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.764+0000 7f68ad703700 1 -- 172.21.15.44:0/1633275964 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a8108d70 con 0x7f68a8107c10 2024-04-21T22:58:26.430 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.764+0000 7f68ad703700 1 --2- 172.21.15.44:0/1633275964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8107c10 0x7f68a8108030 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f68a40098a0 tx=0x7f68a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c0b52ece6c2b20 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:26.430 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.764+0000 7f689ffff700 1 -- 172.21.15.44:0/1633275964 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68a4004160 con 0x7f68a8107c10 2024-04-21T22:58:26.430 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.764+0000 7f689ffff700 1 -- 172.21.15.44:0/1633275964 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f68a4004300 con 0x7f68a8107c10 2024-04-21T22:58:26.430 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.764+0000 7f689ffff700 1 -- 172.21.15.44:0/1633275964 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68a4011b50 con 0x7f68a8107c10 2024-04-21T22:58:26.430 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.765+0000 7f68af967700 1 -- 172.21.15.44:0/1633275964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8107c10 msgr2=0x7f68a8108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:26.430 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.765+0000 7f68af967700 1 --2- 172.21.15.44:0/1633275964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8107c10 0x7f68a8108030 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f68a40098a0 tx=0x7f68a4009bb0 comp rx=0 tx=0).stop 2024-04-21T22:58:26.430 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.765+0000 7f68af967700 1 -- 172.21.15.44:0/1633275964 shutdown_connections 2024-04-21T22:58:26.430 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.765+0000 7f68af967700 1 --2- 172.21.15.44:0/1633275964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8107c10 0x7f68a8108030 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:26.431 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.765+0000 7f68af967700 1 -- 172.21.15.44:0/1633275964 >> 172.21.15.44:0/1633275964 conn(0x7f68a8103230 msgr2=0x7f68a8105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:26.431 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.765+0000 7f68af967700 1 -- 172.21.15.44:0/1633275964 shutdown_connections 2024-04-21T22:58:26.431 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.765+0000 7f68af967700 1 -- 172.21.15.44:0/1633275964 wait complete. 2024-04-21T22:58:26.431 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.766+0000 7f68af967700 1 Processor -- start 2024-04-21T22:58:26.431 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.766+0000 7f68af967700 1 -- start start 2024-04-21T22:58:26.431 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.766+0000 7f68af967700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8107c10 0x7f68a807db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:26.431 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.766+0000 7f68af967700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f68a810c120 con 0x7f68a8107c10 2024-04-21T22:58:26.431 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.766+0000 7f68ad703700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8107c10 0x7f68a807db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:26.431 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.766+0000 7f68ad703700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8107c10 0x7f68a807db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39604/0 (socket says 172.21.15.44:39604) 2024-04-21T22:58:26.431 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.766+0000 7f68ad703700 1 -- 172.21.15.44:0/3173962706 learned_addr learned my addr 172.21.15.44:0/3173962706 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:26.431 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.766+0000 7f68ad703700 1 -- 172.21.15.44:0/3173962706 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a807e0d0 con 0x7f68a8107c10 2024-04-21T22:58:26.431 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.766+0000 7f68ad703700 1 --2- 172.21.15.44:0/3173962706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8107c10 0x7f68a807db90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f68a4009fb0 tx=0x7f68a4011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:26.431 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.766+0000 7f689e7fc700 1 -- 172.21.15.44:0/3173962706 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68a40038b0 con 0x7f68a8107c10 2024-04-21T22:58:26.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.767+0000 7f689e7fc700 1 -- 172.21.15.44:0/3173962706 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f68a4011e50 con 0x7f68a8107c10 2024-04-21T22:58:26.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.767+0000 7f68af967700 1 -- 172.21.15.44:0/3173962706 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68a807c2c0 con 0x7f68a8107c10 2024-04-21T22:58:26.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.767+0000 7f689e7fc700 1 -- 172.21.15.44:0/3173962706 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68a4019550 con 0x7f68a8107c10 2024-04-21T22:58:26.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.767+0000 7f68af967700 1 -- 172.21.15.44:0/3173962706 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68a807c800 con 0x7f68a8107c10 2024-04-21T22:58:26.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.768+0000 7f689e7fc700 1 -- 172.21.15.44:0/3173962706 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f68a4019be0 con 0x7f68a8107c10 2024-04-21T22:58:26.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.768+0000 7f689e7fc700 1 -- 172.21.15.44:0/3173962706 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f68a4024730 con 0x7f68a8107c10 2024-04-21T22:58:26.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.768+0000 7f68af967700 1 -- 172.21.15.44:0/3173962706 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68a8063890 con 0x7f68a8107c10 2024-04-21T22:58:26.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.770+0000 7f689e7fc700 1 -- 172.21.15.44:0/3173962706 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f68a4003a50 con 0x7f68a8107c10 2024-04-21T22:58:26.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.841+0000 7f68af967700 1 -- 172.21.15.44:0/3173962706 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f68a807cb40 con 0x7f68a8107c10 2024-04-21T22:58:26.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.841+0000 7f689e7fc700 1 -- 172.21.15.44:0/3173962706 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f68a4024980 con 0x7f68a8107c10 2024-04-21T22:58:26.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.842+0000 7f68af967700 1 -- 172.21.15.44:0/3173962706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8107c10 msgr2=0x7f68a807db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:26.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.842+0000 7f68af967700 1 --2- 172.21.15.44:0/3173962706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8107c10 0x7f68a807db90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f68a4009fb0 tx=0x7f68a4011bf0 comp rx=0 tx=0).stop 2024-04-21T22:58:26.432 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.842+0000 7f68af967700 1 -- 172.21.15.44:0/3173962706 shutdown_connections 2024-04-21T22:58:26.433 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.842+0000 7f68af967700 1 --2- 172.21.15.44:0/3173962706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8107c10 0x7f68a807db90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:26.433 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.842+0000 7f68af967700 1 -- 172.21.15.44:0/3173962706 >> 172.21.15.44:0/3173962706 conn(0x7f68a8103230 msgr2=0x7f68a8103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:26.433 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.842+0000 7f68af967700 1 -- 172.21.15.44:0/3173962706 shutdown_connections 2024-04-21T22:58:26.433 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:25.842+0000 7f68af967700 1 -- 172.21.15.44:0/3173962706 wait complete. 2024-04-21T22:58:26.433 INFO:teuthology.orchestra.run.smithi044.stdout:Restarting the monitor... 2024-04-21T22:58:26.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:26 smithi044 systemd[1]: Stopping Ceph mon.a for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T22:58:27.058 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:26 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-a[27563]: 2024-04-21T22:58:26.699+0000 7f0ca6a86700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T22:58:27.058 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:26 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-a[27563]: 2024-04-21T22:58:26.699+0000 7f0ca6a86700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-21T22:58:27.058 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:26 smithi044 podman[27810]: 2024-04-21 22:58:26.802464059 +0000 UTC m=+0.198594459 container died 9b65439df42aff16bb06fc8b2b298150b7ebfb1e0f1ce9095f1bf185e83cb05a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-21T22:58:27.417 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 podman[27810]: 2024-04-21 22:58:27.158857155 +0000 UTC m=+0.554987554 container cleanup 9b65439df42aff16bb06fc8b2b298150b7ebfb1e0f1ce9095f1bf185e83cb05a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T22:58:27.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 bash[27810]: ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-a 2024-04-21T22:58:27.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 podman[27824]: 2024-04-21 22:58:27.417945248 +0000 UTC m=+0.613313477 container remove 9b65439df42aff16bb06fc8b2b298150b7ebfb1e0f1ce9095f1bf185e83cb05a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-21T22:58:27.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 systemd[1]: ceph-80909b94-0032-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-21T22:58:27.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 systemd[1]: Stopped Ceph mon.a for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T22:58:27.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 systemd[1]: Starting Ceph mon.a for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T22:58:28.007 INFO:teuthology.orchestra.run.smithi044.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-21T22:58:28.035 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 podman[27916]: 2024-04-21 22:58:27.65973112 +0000 UTC m=+0.018501945 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T22:58:28.035 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 podman[27916]: 2024-04-21 22:58:27.76871272 +0000 UTC m=+0.127483535 container create 50bff5b2e0ac0929af5754e15288de6003b952f9fe5ba91c6344ad3d1779c457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS) 2024-04-21T22:58:28.035 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 podman[27916]: 2024-04-21 22:58:27.937515188 +0000 UTC m=+0.296286004 container init 50bff5b2e0ac0929af5754e15288de6003b952f9fe5ba91c6344ad3d1779c457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8) 2024-04-21T22:58:28.035 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 podman[27916]: 2024-04-21 22:58:27.940470993 +0000 UTC m=+0.299241810 container start 50bff5b2e0ac0929af5754e15288de6003b952f9fe5ba91c6344ad3d1779c457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-21T22:58:28.035 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: set uid:gid to 167:167 (ceph:ceph) 2024-04-21T22:58:28.035 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev), process ceph-mon, pid 2 2024-04-21T22:58:28.035 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: pidfile_write: ignore empty --pid-file 2024-04-21T22:58:28.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: load: jerasure load: lrc 2024-04-21T22:58:28.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: RocksDB version: 7.9.2 2024-04-21T22:58:28.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Git sha 0 2024-04-21T22:58:28.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Compile date 2024-04-20 15:04:30 2024-04-21T22:58:28.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: DB SUMMARY 2024-04-21T22:58:28.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: DB Session ID: W13MZZ5KNH6XADL06BGH 2024-04-21T22:58:28.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: CURRENT file: CURRENT 2024-04-21T22:58:28.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: IDENTITY file: IDENTITY 2024-04-21T22:58:28.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-21T22:58:28.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-21T22:58:28.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104176 ; 2024-04-21T22:58:28.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.error_if_exists: 0 2024-04-21T22:58:28.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.create_if_missing: 0 2024-04-21T22:58:28.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.paranoid_checks: 1 2024-04-21T22:58:28.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-21T22:58:28.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-21T22:58:28.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-21T22:58:28.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.env: 0x556b334d3c60 2024-04-21T22:58:28.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.fs: PosixFileSystem 2024-04-21T22:58:28.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.info_log: 0x556b35a73aa0 2024-04-21T22:58:28.038 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_file_opening_threads: 16 2024-04-21T22:58:28.038 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.statistics: (nil) 2024-04-21T22:58:28.038 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.use_fsync: 0 2024-04-21T22:58:28.038 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_log_file_size: 0 2024-04-21T22:58:28.038 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-21T22:58:28.038 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-21T22:58:28.038 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.keep_log_file_num: 1000 2024-04-21T22:58:28.038 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.recycle_log_file_num: 0 2024-04-21T22:58:28.038 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.allow_fallocate: 1 2024-04-21T22:58:28.038 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.allow_mmap_reads: 0 2024-04-21T22:58:28.038 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.allow_mmap_writes: 0 2024-04-21T22:58:28.039 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.use_direct_reads: 0 2024-04-21T22:58:28.039 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-21T22:58:28.039 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.create_missing_column_families: 0 2024-04-21T22:58:28.039 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.db_log_dir: 2024-04-21T22:58:28.039 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.wal_dir: 2024-04-21T22:58:28.039 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-21T22:58:28.039 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-21T22:58:28.039 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-21T22:58:28.039 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-21T22:58:28.039 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-21T22:58:28.039 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-21T22:58:28.039 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.advise_random_on_open: 1 2024-04-21T22:58:28.040 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.db_write_buffer_size: 0 2024-04-21T22:58:28.040 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.write_buffer_manager: 0x556b35a461e0 2024-04-21T22:58:28.040 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-21T22:58:28.040 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-21T22:58:28.040 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-21T22:58:28.040 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.rate_limiter: (nil) 2024-04-21T22:58:28.040 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-21T22:58:28.040 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.wal_recovery_mode: 2 2024-04-21T22:58:28.042 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.enable_thread_tracking: 0 2024-04-21T22:58:28.042 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.enable_pipelined_write: 0 2024-04-21T22:58:28.042 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.unordered_write: 0 2024-04-21T22:58:28.042 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-21T22:58:28.042 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-21T22:58:28.042 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-21T22:58:28.042 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-21T22:58:28.042 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.row_cache: None 2024-04-21T22:58:28.043 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.wal_filter: None 2024-04-21T22:58:28.043 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-21T22:58:28.043 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.allow_ingest_behind: 0 2024-04-21T22:58:28.043 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.two_write_queues: 0 2024-04-21T22:58:28.043 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.manual_wal_flush: 0 2024-04-21T22:58:28.043 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.wal_compression: 0 2024-04-21T22:58:28.043 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.atomic_flush: 0 2024-04-21T22:58:28.043 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-21T22:58:28.043 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-21T22:58:28.043 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-21T22:58:28.043 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.log_readahead_size: 0 2024-04-21T22:58:28.043 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-21T22:58:28.044 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.best_efforts_recovery: 0 2024-04-21T22:58:28.044 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-21T22:58:28.044 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-21T22:58:28.044 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.allow_data_in_errors: 0 2024-04-21T22:58:28.044 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.db_host_id: __hostname__ 2024-04-21T22:58:28.044 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-21T22:58:28.044 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_background_jobs: 2 2024-04-21T22:58:28.044 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_background_compactions: -1 2024-04-21T22:58:28.044 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_subcompactions: 1 2024-04-21T22:58:28.044 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-21T22:58:28.044 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-21T22:58:28.045 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-21T22:58:28.045 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_total_wal_size: 0 2024-04-21T22:58:28.045 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-21T22:58:28.045 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-21T22:58:28.045 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-21T22:58:28.045 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-21T22:58:28.045 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_open_files: -1 2024-04-21T22:58:28.045 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.bytes_per_sync: 0 2024-04-21T22:58:28.045 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-21T22:58:28.045 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-21T22:58:28.045 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compaction_readahead_size: 0 2024-04-21T22:58:28.045 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_background_flushes: -1 2024-04-21T22:58:28.046 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Compression algorithms supported: 2024-04-21T22:58:28.046 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-21T22:58:28.046 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: kZSTD supported: 0 2024-04-21T22:58:28.046 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: kXpressCompression supported: 0 2024-04-21T22:58:28.046 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: kLZ4HCCompression supported: 1 2024-04-21T22:58:28.046 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: kZlibCompression supported: 1 2024-04-21T22:58:28.046 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: kSnappyCompression supported: 1 2024-04-21T22:58:28.046 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: kLZ4Compression supported: 1 2024-04-21T22:58:28.046 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: kBZip2Compression supported: 0 2024-04-21T22:58:28.046 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-21T22:58:28.046 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-21T22:58:28.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-21T22:58:28.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-21T22:58:28.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-21T22:58:28.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.merge_operator: 2024-04-21T22:58:28.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compaction_filter: None 2024-04-21T22:58:28.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compaction_filter_factory: None 2024-04-21T22:58:28.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.sst_partitioner_factory: None 2024-04-21T22:58:28.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-21T22:58:28.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-21T22:58:28.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556b35a73ba0) 2024-04-21T22:58:28.047 INFO:journalctl@ceph.mon.a.smithi044.stdout: cache_index_and_filter_blocks: 1 2024-04-21T22:58:28.047 INFO:journalctl@ceph.mon.a.smithi044.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-21T22:58:28.048 INFO:journalctl@ceph.mon.a.smithi044.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-21T22:58:28.048 INFO:journalctl@ceph.mon.a.smithi044.stdout: pin_top_level_index_and_filter: 1 2024-04-21T22:58:28.048 INFO:journalctl@ceph.mon.a.smithi044.stdout: index_type: 0 2024-04-21T22:58:28.048 INFO:journalctl@ceph.mon.a.smithi044.stdout: data_block_index_type: 0 2024-04-21T22:58:28.048 INFO:journalctl@ceph.mon.a.smithi044.stdout: index_shortening: 1 2024-04-21T22:58:28.048 INFO:journalctl@ceph.mon.a.smithi044.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-21T22:58:28.048 INFO:journalctl@ceph.mon.a.smithi044.stdout: checksum: 4 2024-04-21T22:58:28.048 INFO:journalctl@ceph.mon.a.smithi044.stdout: no_block_cache: 0 2024-04-21T22:58:28.048 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache: 0x556b35a77090 2024-04-21T22:58:28.048 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache_name: BinnedLRUCache 2024-04-21T22:58:28.048 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache_options: 2024-04-21T22:58:28.048 INFO:journalctl@ceph.mon.a.smithi044.stdout: capacity : 536870912 2024-04-21T22:58:28.048 INFO:journalctl@ceph.mon.a.smithi044.stdout: num_shard_bits : 4 2024-04-21T22:58:28.049 INFO:journalctl@ceph.mon.a.smithi044.stdout: strict_capacity_limit : 0 2024-04-21T22:58:28.049 INFO:journalctl@ceph.mon.a.smithi044.stdout: high_pri_pool_ratio: 0.000 2024-04-21T22:58:28.049 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_cache_compressed: (nil) 2024-04-21T22:58:28.049 INFO:journalctl@ceph.mon.a.smithi044.stdout: persistent_cache: (nil) 2024-04-21T22:58:28.049 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_size: 4096 2024-04-21T22:58:28.049 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_size_deviation: 10 2024-04-21T22:58:28.049 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_restart_interval: 16 2024-04-21T22:58:28.049 INFO:journalctl@ceph.mon.a.smithi044.stdout: index_block_restart_interval: 1 2024-04-21T22:58:28.050 INFO:journalctl@ceph.mon.a.smithi044.stdout: metadata_block_size: 4096 2024-04-21T22:58:28.050 INFO:journalctl@ceph.mon.a.smithi044.stdout: partition_filters: 0 2024-04-21T22:58:28.050 INFO:journalctl@ceph.mon.a.smithi044.stdout: use_delta_encoding: 1 2024-04-21T22:58:28.050 INFO:journalctl@ceph.mon.a.smithi044.stdout: filter_policy: bloomfilter 2024-04-21T22:58:28.050 INFO:journalctl@ceph.mon.a.smithi044.stdout: whole_key_filtering: 1 2024-04-21T22:58:28.050 INFO:journalctl@ceph.mon.a.smithi044.stdout: verify_compression: 0 2024-04-21T22:58:28.050 INFO:journalctl@ceph.mon.a.smithi044.stdout: read_amp_bytes_per_bit: 0 2024-04-21T22:58:28.050 INFO:journalctl@ceph.mon.a.smithi044.stdout: format_version: 5 2024-04-21T22:58:28.050 INFO:journalctl@ceph.mon.a.smithi044.stdout: enable_index_compression: 1 2024-04-21T22:58:28.050 INFO:journalctl@ceph.mon.a.smithi044.stdout: block_align: 0 2024-04-21T22:58:28.051 INFO:journalctl@ceph.mon.a.smithi044.stdout: max_auto_readahead_size: 262144 2024-04-21T22:58:28.051 INFO:journalctl@ceph.mon.a.smithi044.stdout: prepopulate_block_cache: 0 2024-04-21T22:58:28.051 INFO:journalctl@ceph.mon.a.smithi044.stdout: initial_auto_readahead_size: 8192 2024-04-21T22:58:28.051 INFO:journalctl@ceph.mon.a.smithi044.stdout: num_file_reads_for_auto_readahead: 2 2024-04-21T22:58:28.051 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.write_buffer_size: 33554432 2024-04-21T22:58:28.051 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_write_buffer_number: 2 2024-04-21T22:58:28.051 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compression: NoCompression 2024-04-21T22:58:28.051 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.bottommost_compression: Disabled 2024-04-21T22:58:28.051 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.prefix_extractor: nullptr 2024-04-21T22:58:28.051 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-21T22:58:28.051 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.num_levels: 7 2024-04-21T22:58:28.051 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-21T22:58:28.051 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-21T22:58:28.052 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-21T22:58:28.052 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-21T22:58:28.052 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-21T22:58:28.052 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-21T22:58:28.052 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-21T22:58:28.052 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-21T22:58:28.052 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-21T22:58:28.052 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-21T22:58:28.052 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-21T22:58:28.052 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-21T22:58:28.052 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-21T22:58:28.052 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compression_opts.level: 32767 2024-04-21T22:58:28.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compression_opts.strategy: 0 2024-04-21T22:58:28.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-21T22:58:28.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-21T22:58:28.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-21T22:58:28.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-21T22:58:28.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compression_opts.enabled: false 2024-04-21T22:58:28.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-21T22:58:28.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-21T22:58:28.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-21T22:58:28.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-21T22:58:28.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.target_file_size_base: 67108864 2024-04-21T22:58:28.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-21T22:58:28.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-21T22:58:28.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-21T22:58:28.054 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-21T22:58:28.054 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-21T22:58:28.054 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-21T22:58:28.054 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-21T22:58:28.054 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-21T22:58:28.054 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-21T22:58:28.054 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-21T22:58:28.054 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-21T22:58:28.054 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-21T22:58:28.054 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-21T22:58:28.054 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-21T22:58:28.055 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.arena_block_size: 1048576 2024-04-21T22:58:28.055 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-21T22:58:28.055 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-21T22:58:28.055 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.disable_auto_compactions: 0 2024-04-21T22:58:28.055 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-21T22:58:28.055 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-21T22:58:28.055 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-21T22:58:28.055 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-21T22:58:28.055 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-21T22:58:28.055 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-21T22:58:28.055 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-21T22:58:28.055 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-21T22:58:28.056 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-21T22:58:28.056 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-21T22:58:28.056 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.table_properties_collectors: 2024-04-21T22:58:28.056 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.inplace_update_support: 0 2024-04-21T22:58:28.056 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-21T22:58:28.056 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-21T22:58:28.056 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-21T22:58:28.056 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-21T22:58:28.056 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.bloom_locality: 0 2024-04-21T22:58:28.056 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.max_successive_merges: 0 2024-04-21T22:58:28.056 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-21T22:58:28.056 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.paranoid_file_checks: 0 2024-04-21T22:58:28.057 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.force_consistency_checks: 1 2024-04-21T22:58:28.057 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.report_bg_io_stats: 0 2024-04-21T22:58:28.057 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.ttl: 2592000 2024-04-21T22:58:28.057 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-21T22:58:28.057 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-21T22:58:28.057 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-21T22:58:28.057 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.enable_blob_files: false 2024-04-21T22:58:28.057 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.min_blob_size: 0 2024-04-21T22:58:28.057 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.blob_file_size: 268435456 2024-04-21T22:58:28.058 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-21T22:58:28.058 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-21T22:58:28.058 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-21T22:58:28.058 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-21T22:58:28.058 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-21T22:58:28.058 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.blob_file_starting_level: 0 2024-04-21T22:58:28.058 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-21T22:58:28.058 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-21T22:58:28.058 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-21T22:58:28.058 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 28e659cf-6397-4161-a2e8-65340504cdcd 2024-04-21T22:58:28.059 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713740307970959, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-21T22:58:28.059 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-21T22:58:28.059 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713740307972026, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99381, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 97301, "index_size": 298, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14034, "raw_average_key_size": 49, "raw_value_size": 89637, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 283, "num_filter_entries": 283, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713740307, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "28e659cf-6397-4161-a2e8-65340504cdcd", "db_session_id": "W13MZZ5KNH6XADL06BGH", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-21T22:58:28.059 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713740307972125, "job": 1, "event": "recovery_finished"} 2024-04-21T22:58:28.059 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-21T22:58:28.059 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-21T22:58:28.059 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556b35b68000 2024-04-21T22:58:28.059 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: DB pointer 0x556b35b52000 2024-04-21T22:58:28.059 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-21T22:58:28.059 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-21T22:58:28.059 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** DB Stats ** 2024-04-21T22:58:28.059 INFO:journalctl@ceph.mon.a.smithi044.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T22:58:28.059 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-21T22:58:28.059 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T22:58:28.060 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T22:58:28.060 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-21T22:58:28.060 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T22:58:28.060 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T22:58:28.060 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-04-21T22:58:28.060 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** Compaction Stats [default] ** 2024-04-21T22:58:28.060 INFO:journalctl@ceph.mon.a.smithi044.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T22:58:28.060 INFO:journalctl@ceph.mon.a.smithi044.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-21T22:58:28.060 INFO:journalctl@ceph.mon.a.smithi044.stdout: L0 2/0 98.89 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T22:58:28.060 INFO:journalctl@ceph.mon.a.smithi044.stdout: Sum 2/0 98.89 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T22:58:28.060 INFO:journalctl@ceph.mon.a.smithi044.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T22:58:28.060 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-04-21T22:58:28.060 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** Compaction Stats [default] ** 2024-04-21T22:58:28.060 INFO:journalctl@ceph.mon.a.smithi044.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T22:58:28.061 INFO:journalctl@ceph.mon.a.smithi044.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-21T22:58:28.061 INFO:journalctl@ceph.mon.a.smithi044.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 145.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T22:58:28.061 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-04-21T22:58:28.061 INFO:journalctl@ceph.mon.a.smithi044.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-21T22:58:28.061 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-04-21T22:58:28.061 INFO:journalctl@ceph.mon.a.smithi044.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T22:58:28.061 INFO:journalctl@ceph.mon.a.smithi044.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-21T22:58:28.061 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-21T22:58:28.061 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-21T22:58:28.061 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-21T22:58:28.061 INFO:journalctl@ceph.mon.a.smithi044.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-21T22:58:28.061 INFO:journalctl@ceph.mon.a.smithi044.stdout: Cumulative compaction: 0.00 GB write, 27.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T22:58:28.061 INFO:journalctl@ceph.mon.a.smithi044.stdout: Interval compaction: 0.00 GB write, 27.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T22:58:28.061 INFO:journalctl@ceph.mon.a.smithi044.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-21T22:58:28.061 INFO:journalctl@ceph.mon.a.smithi044.stdout: Block cache BinnedLRUCache@0x556b35a77090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-21T22:58:28.062 INFO:journalctl@ceph.mon.a.smithi044.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-21T22:58:28.062 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-04-21T22:58:28.062 INFO:journalctl@ceph.mon.a.smithi044.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-21T22:58:28.062 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: starting mon.a rank 0 at public addrs [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] at bind addrs [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T22:58:28.062 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: mon.a@-1(???) e1 preinit fsid 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T22:58:28.062 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: mon.a@-1(???).mds e1 new map 2024-04-21T22:58:28.062 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: mon.a@-1(???).mds e1 print_map 2024-04-21T22:58:28.062 INFO:journalctl@ceph.mon.a.smithi044.stdout: e1 2024-04-21T22:58:28.062 INFO:journalctl@ceph.mon.a.smithi044.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T22:58:28.062 INFO:journalctl@ceph.mon.a.smithi044.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T22:58:28.062 INFO:journalctl@ceph.mon.a.smithi044.stdout: legacy client fscid: -1 2024-04-21T22:58:28.062 INFO:journalctl@ceph.mon.a.smithi044.stdout: 2024-04-21T22:58:28.062 INFO:journalctl@ceph.mon.a.smithi044.stdout: No filesystems configured 2024-04-21T22:58:28.063 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-21T22:58:28.063 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T22:58:28.063 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T22:58:28.063 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T22:58:28.063 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:27 smithi044 ceph-mon[27934]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-21T22:58:28.063 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:28 smithi044 bash[27916]: 50bff5b2e0ac0929af5754e15288de6003b952f9fe5ba91c6344ad3d1779c457 2024-04-21T22:58:28.063 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:28 smithi044 systemd[1]: Started Ceph mon.a for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T22:58:28.063 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:28 smithi044 ceph-mon[27934]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-21T22:58:28.063 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:28 smithi044 ceph-mon[27934]: monmap epoch 1 2024-04-21T22:58:28.063 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:28 smithi044 ceph-mon[27934]: fsid 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T22:58:28.063 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:28 smithi044 ceph-mon[27934]: last_changed 2024-04-21T22:58:20.078074+0000 2024-04-21T22:58:28.063 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:28 smithi044 ceph-mon[27934]: created 2024-04-21T22:58:20.078074+0000 2024-04-21T22:58:28.063 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:28 smithi044 ceph-mon[27934]: min_mon_release 19 (squid) 2024-04-21T22:58:28.063 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:28 smithi044 ceph-mon[27934]: election_strategy: 1 2024-04-21T22:58:28.063 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:28 smithi044 ceph-mon[27934]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-04-21T22:58:28.064 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:28 smithi044 ceph-mon[27934]: fsmap 2024-04-21T22:58:28.064 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:28 smithi044 ceph-mon[27934]: osdmap e1: 0 total, 0 up, 0 in 2024-04-21T22:58:28.064 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:28 smithi044 ceph-mon[27934]: mgrmap e1: no daemons active 2024-04-21T22:58:29.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.394+0000 7f49ab9eb700 1 Processor -- start 2024-04-21T22:58:29.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.394+0000 7f49ab9eb700 1 -- start start 2024-04-21T22:58:29.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.394+0000 7f49ab9eb700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49a41064e0 0x7f49a4106900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:29.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.394+0000 7f49ab9eb700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f49a407be30 con 0x7f49a41064e0 2024-04-21T22:58:29.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.394+0000 7f49a9787700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49a41064e0 0x7f49a4106900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:29.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.394+0000 7f49a9787700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49a41064e0 0x7f49a4106900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39612/0 (socket says 172.21.15.44:39612) 2024-04-21T22:58:29.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.394+0000 7f49a9787700 1 -- 172.21.15.44:0/2226009438 learned_addr learned my addr 172.21.15.44:0/2226009438 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:29.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.395+0000 7f49a9787700 1 -- 172.21.15.44:0/2226009438 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a4106e40 con 0x7f49a41064e0 2024-04-21T22:58:29.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.395+0000 7f49a9787700 1 --2- 172.21.15.44:0/2226009438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49a41064e0 0x7f49a4106900 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f49a00098a0 tx=0x7f49a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff0021afc74aa494 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:29.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.395+0000 7f499bfff700 1 -- 172.21.15.44:0/2226009438 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49a0004160 con 0x7f49a41064e0 2024-04-21T22:58:29.067 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.395+0000 7f499bfff700 1 -- 172.21.15.44:0/2226009438 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f49a0004300 con 0x7f49a41064e0 2024-04-21T22:58:29.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.395+0000 7f499bfff700 1 -- 172.21.15.44:0/2226009438 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49a0011b50 con 0x7f49a41064e0 2024-04-21T22:58:29.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.396+0000 7f49ab9eb700 1 -- 172.21.15.44:0/2226009438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49a41064e0 msgr2=0x7f49a4106900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:29.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.396+0000 7f49ab9eb700 1 --2- 172.21.15.44:0/2226009438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49a41064e0 0x7f49a4106900 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f49a00098a0 tx=0x7f49a0009bb0 comp rx=0 tx=0).stop 2024-04-21T22:58:29.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.396+0000 7f49ab9eb700 1 -- 172.21.15.44:0/2226009438 shutdown_connections 2024-04-21T22:58:29.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.396+0000 7f49ab9eb700 1 --2- 172.21.15.44:0/2226009438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49a41064e0 0x7f49a4106900 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:29.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.396+0000 7f49ab9eb700 1 -- 172.21.15.44:0/2226009438 >> 172.21.15.44:0/2226009438 conn(0x7f49a41020a0 msgr2=0x7f49a41044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:29.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.396+0000 7f49ab9eb700 1 -- 172.21.15.44:0/2226009438 shutdown_connections 2024-04-21T22:58:29.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.396+0000 7f49ab9eb700 1 -- 172.21.15.44:0/2226009438 wait complete. 2024-04-21T22:58:29.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.397+0000 7f49ab9eb700 1 Processor -- start 2024-04-21T22:58:29.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.397+0000 7f49ab9eb700 1 -- start start 2024-04-21T22:58:29.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.397+0000 7f49ab9eb700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49a41064e0 0x7f49a419dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:29.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.397+0000 7f49ab9eb700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f49a4107ed0 con 0x7f49a41064e0 2024-04-21T22:58:29.068 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.397+0000 7f49a9787700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49a41064e0 0x7f49a419dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:29.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.397+0000 7f49a9787700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49a41064e0 0x7f49a419dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39618/0 (socket says 172.21.15.44:39618) 2024-04-21T22:58:29.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.397+0000 7f49a9787700 1 -- 172.21.15.44:0/162227159 learned_addr learned my addr 172.21.15.44:0/162227159 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:29.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.397+0000 7f49a9787700 1 -- 172.21.15.44:0/162227159 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a419e2e0 con 0x7f49a41064e0 2024-04-21T22:58:29.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.397+0000 7f49a9787700 1 --2- 172.21.15.44:0/162227159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49a41064e0 0x7f49a419dda0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009fb0 tx=0x7f49a0011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:29.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.397+0000 7f499a7fc700 1 -- 172.21.15.44:0/162227159 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49a00038b0 con 0x7f49a41064e0 2024-04-21T22:58:29.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.398+0000 7f499a7fc700 1 -- 172.21.15.44:0/162227159 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f49a0011e50 con 0x7f49a41064e0 2024-04-21T22:58:29.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.398+0000 7f49ab9eb700 1 -- 172.21.15.44:0/162227159 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49a419e570 con 0x7f49a41064e0 2024-04-21T22:58:29.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.398+0000 7f499a7fc700 1 -- 172.21.15.44:0/162227159 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49a0019420 con 0x7f49a41064e0 2024-04-21T22:58:29.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.398+0000 7f49ab9eb700 1 -- 172.21.15.44:0/162227159 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49a41a1260 con 0x7f49a41064e0 2024-04-21T22:58:29.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.398+0000 7f499a7fc700 1 -- 172.21.15.44:0/162227159 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f49a0019ad0 con 0x7f49a41064e0 2024-04-21T22:58:29.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.398+0000 7f499a7fc700 1 -- 172.21.15.44:0/162227159 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f49a0024730 con 0x7f49a41064e0 2024-04-21T22:58:29.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.398+0000 7f49ab9eb700 1 -- 172.21.15.44:0/162227159 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49a4063890 con 0x7f49a41064e0 2024-04-21T22:58:29.069 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.401+0000 7f499a7fc700 1 -- 172.21.15.44:0/162227159 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f49a0043dd0 con 0x7f49a41064e0 2024-04-21T22:58:29.070 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.470+0000 7f49ab9eb700 1 -- 172.21.15.44:0/162227159 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f49a402ddb0 con 0x7f49a41064e0 2024-04-21T22:58:29.070 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.475+0000 7f499a7fc700 1 -- 172.21.15.44:0/162227159 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f49a0032030 con 0x7f49a41064e0 2024-04-21T22:58:29.070 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.476+0000 7f49ab9eb700 1 -- 172.21.15.44:0/162227159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49a41064e0 msgr2=0x7f49a419dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:29.070 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.476+0000 7f49ab9eb700 1 --2- 172.21.15.44:0/162227159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49a41064e0 0x7f49a419dda0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009fb0 tx=0x7f49a0011bf0 comp rx=0 tx=0).stop 2024-04-21T22:58:29.070 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.476+0000 7f49ab9eb700 1 -- 172.21.15.44:0/162227159 shutdown_connections 2024-04-21T22:58:29.070 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.476+0000 7f49ab9eb700 1 --2- 172.21.15.44:0/162227159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f49a41064e0 0x7f49a419dda0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:29.070 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.476+0000 7f49ab9eb700 1 -- 172.21.15.44:0/162227159 >> 172.21.15.44:0/162227159 conn(0x7f49a41020a0 msgr2=0x7f49a4194a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:29.070 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.476+0000 7f49ab9eb700 1 -- 172.21.15.44:0/162227159 shutdown_connections 2024-04-21T22:58:29.070 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:28.476+0000 7f49ab9eb700 1 -- 172.21.15.44:0/162227159 wait complete. 2024-04-21T22:58:29.070 INFO:teuthology.orchestra.run.smithi044.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-21T22:58:29.084 INFO:teuthology.orchestra.run.smithi044.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-21T22:58:29.084 INFO:teuthology.orchestra.run.smithi044.stdout:Creating mgr... 2024-04-21T22:58:29.084 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-21T22:58:29.085 INFO:teuthology.orchestra.run.smithi044.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-21T22:58:29.322 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-80909b94-0032-11ef-bc93-c7b262605968@mgr.x 2024-04-21T22:58:29.323 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Failed to reset failed state of unit ceph-80909b94-0032-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-80909b94-0032-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-21T22:58:29.481 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-80909b94-0032-11ef-bc93-c7b262605968.target.wants/ceph-80909b94-0032-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-80909b94-0032-11ef-bc93-c7b262605968@.service. 2024-04-21T22:58:29.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:29 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/162227159' entity='client.admin' 2024-04-21T22:58:29.686 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:29 smithi044 systemd[1]: Starting Ceph mgr.x for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T22:58:29.963 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:29 smithi044 podman[28149]: 2024-04-21 22:58:29.703905462 +0000 UTC m=+0.100038134 container create 4543c9b91d4a44095f76e2eef0a4cc748039ca7ff6988fed59f02a205f68c34b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=) 2024-04-21T22:58:29.964 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:29 smithi044 podman[28149]: 2024-04-21 22:58:29.622109109 +0000 UTC m=+0.018241767 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b 2024-04-21T22:58:29.964 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:29 smithi044 podman[28149]: 2024-04-21 22:58:29.897657839 +0000 UTC m=+0.293790472 container init 4543c9b91d4a44095f76e2eef0a4cc748039ca7ff6988fed59f02a205f68c34b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-21T22:58:29.964 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:29 smithi044 podman[28149]: 2024-04-21 22:58:29.900172333 +0000 UTC m=+0.296304962 container start 4543c9b91d4a44095f76e2eef0a4cc748039ca7ff6988fed59f02a205f68c34b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-21T22:58:29.964 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:29 smithi044 bash[28149]: 4543c9b91d4a44095f76e2eef0a4cc748039ca7ff6988fed59f02a205f68c34b 2024-04-21T22:58:29.978 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T22:58:29.978 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout disabled 2024-04-21T22:58:29.985 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T22:58:29.985 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-04-21T22:58:29.985 INFO:teuthology.orchestra.run.smithi044.stdout:firewalld.service is not enabled 2024-04-21T22:58:29.985 INFO:teuthology.orchestra.run.smithi044.stdout:Not possible to enable service . firewalld.service is not available 2024-04-21T22:58:29.994 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T22:58:29.994 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout disabled 2024-04-21T22:58:30.001 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T22:58:30.001 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-04-21T22:58:30.001 INFO:teuthology.orchestra.run.smithi044.stdout:firewalld.service is not enabled 2024-04-21T22:58:30.001 INFO:teuthology.orchestra.run.smithi044.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-21T22:58:30.001 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mgr to start... 2024-04-21T22:58:30.002 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mgr... 2024-04-21T22:58:30.256 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:29 smithi044 systemd[1]: Started Ceph mgr.x for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T22:58:30.256 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:30 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:30.063+0000 7fef710b9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T22:58:30.256 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:30 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:30.256+0000 7fef710b9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T22:58:30.559 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:30 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:30.364+0000 7fef710b9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T22:58:30.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:30 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3753378095' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T22:58:30.936 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:30 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:30.699+0000 7fef710b9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T22:58:30.936 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:30 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:30.787+0000 7fef710b9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T22:58:31.139 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-21T22:58:31.142 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-04-21T22:58:31.142 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsid": "80909b94-0032-11ef-bc93-c7b262605968", 2024-04-21T22:58:31.143 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T22:58:31.143 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T22:58:31.143 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T22:58:31.143 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T22:58:31.143 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:31.143 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T22:58:31.143 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T22:58:31.143 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 0 2024-04-21T22:58:31.143 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-21T22:58:31.143 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T22:58:31.143 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T22:58:31.143 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-21T22:58:31.144 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-21T22:58:31.144 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T22:58:31.144 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T22:58:31.144 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T22:58:31.144 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T22:58:31.144 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:31.144 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T22:58:31.144 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T22:58:31.144 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T22:58:31.144 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T22:58:31.144 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T22:58:31.152 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T22:58:31.152 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T22:58:31.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T22:58:31.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:31.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T22:58:31.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T22:58:31.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T22:58:31.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T22:58:31.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T22:58:31.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T22:58:31.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T22:58:31.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T22:58:31.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T22:58:31.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:31.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T22:58:31.153 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T22:58:31.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T22:58:31.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T22:58:31.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:31.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T22:58:31.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-21T22:58:31.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T22:58:31.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T22:58:31.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T22:58:31.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T22:58:31.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T22:58:31.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-21T22:58:31.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T22:58:31.154 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:31.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T22:58:31.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T22:58:31.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T22:58:23.196647+0000", 2024-04-21T22:58:31.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T22:58:31.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:31.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T22:58:31.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-04-21T22:58:31.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.479+0000 7ff0a6d1e700 1 Processor -- start 2024-04-21T22:58:31.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.479+0000 7ff0a6d1e700 1 -- start start 2024-04-21T22:58:31.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.479+0000 7ff0a6d1e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0a0109190 0x7ff0a01095b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:31.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.479+0000 7ff0a6d1e700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a0109b80 con 0x7ff0a0109190 2024-04-21T22:58:31.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.480+0000 7ff0a4aba700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0a0109190 0x7ff0a01095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:31.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.480+0000 7ff0a4aba700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0a0109190 0x7ff0a01095b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39642/0 (socket says 172.21.15.44:39642) 2024-04-21T22:58:31.155 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.480+0000 7ff0a4aba700 1 -- 172.21.15.44:0/376961873 learned_addr learned my addr 172.21.15.44:0/376961873 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:31.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.480+0000 7ff0a4aba700 1 -- 172.21.15.44:0/376961873 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a0109d00 con 0x7ff0a0109190 2024-04-21T22:58:31.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.480+0000 7ff0a4aba700 1 --2- 172.21.15.44:0/376961873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0a0109190 0x7ff0a01095b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff0940098a0 tx=0x7ff094009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44024742d320ed14 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:31.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.480+0000 7ff09f7fe700 1 -- 172.21.15.44:0/376961873 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff094004160 con 0x7ff0a0109190 2024-04-21T22:58:31.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.480+0000 7ff09f7fe700 1 -- 172.21.15.44:0/376961873 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff094004300 con 0x7ff0a0109190 2024-04-21T22:58:31.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.480+0000 7ff09f7fe700 1 -- 172.21.15.44:0/376961873 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff094011b50 con 0x7ff0a0109190 2024-04-21T22:58:31.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.481+0000 7ff0a6d1e700 1 -- 172.21.15.44:0/376961873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0a0109190 msgr2=0x7ff0a01095b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:31.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.481+0000 7ff0a6d1e700 1 --2- 172.21.15.44:0/376961873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0a0109190 0x7ff0a01095b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff0940098a0 tx=0x7ff094009bb0 comp rx=0 tx=0).stop 2024-04-21T22:58:31.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.481+0000 7ff0a6d1e700 1 -- 172.21.15.44:0/376961873 shutdown_connections 2024-04-21T22:58:31.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.481+0000 7ff0a6d1e700 1 --2- 172.21.15.44:0/376961873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0a0109190 0x7ff0a01095b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:31.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.481+0000 7ff0a6d1e700 1 -- 172.21.15.44:0/376961873 >> 172.21.15.44:0/376961873 conn(0x7ff0a0101770 msgr2=0x7ff0a0103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:31.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.481+0000 7ff0a6d1e700 1 -- 172.21.15.44:0/376961873 shutdown_connections 2024-04-21T22:58:31.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.481+0000 7ff0a6d1e700 1 -- 172.21.15.44:0/376961873 wait complete. 2024-04-21T22:58:31.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.482+0000 7ff0a6d1e700 1 Processor -- start 2024-04-21T22:58:31.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.482+0000 7ff0a6d1e700 1 -- start start 2024-04-21T22:58:31.156 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.482+0000 7ff0a6d1e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0a0109190 0x7ff0a01087b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:31.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.482+0000 7ff0a6d1e700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a0106a00 con 0x7ff0a0109190 2024-04-21T22:58:31.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.482+0000 7ff0a4aba700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0a0109190 0x7ff0a01087b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:31.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.482+0000 7ff0a4aba700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0a0109190 0x7ff0a01087b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39656/0 (socket says 172.21.15.44:39656) 2024-04-21T22:58:31.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.482+0000 7ff0a4aba700 1 -- 172.21.15.44:0/3753378095 learned_addr learned my addr 172.21.15.44:0/3753378095 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:31.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.483+0000 7ff0a4aba700 1 -- 172.21.15.44:0/3753378095 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a0108cf0 con 0x7ff0a0109190 2024-04-21T22:58:31.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.483+0000 7ff0a4aba700 1 --2- 172.21.15.44:0/3753378095 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0a0109190 0x7ff0a01087b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff094000c00 tx=0x7ff094011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:31.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.483+0000 7ff09dffb700 1 -- 172.21.15.44:0/3753378095 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff094003b20 con 0x7ff0a0109190 2024-04-21T22:58:31.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.483+0000 7ff09dffb700 1 -- 172.21.15.44:0/3753378095 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff094003cc0 con 0x7ff0a0109190 2024-04-21T22:58:31.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.483+0000 7ff0a6d1e700 1 -- 172.21.15.44:0/3753378095 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0a0106ee0 con 0x7ff0a0109190 2024-04-21T22:58:31.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.483+0000 7ff09dffb700 1 -- 172.21.15.44:0/3753378095 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff094024910 con 0x7ff0a0109190 2024-04-21T22:58:31.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.483+0000 7ff0a6d1e700 1 -- 172.21.15.44:0/3753378095 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0a0107420 con 0x7ff0a0109190 2024-04-21T22:58:31.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.483+0000 7ff09dffb700 1 -- 172.21.15.44:0/3753378095 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff094022020 con 0x7ff0a0109190 2024-04-21T22:58:31.157 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.483+0000 7ff09dffb700 1 -- 172.21.15.44:0/3753378095 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff094011e00 con 0x7ff0a0109190 2024-04-21T22:58:31.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.484+0000 7ff0a6d1e700 1 -- 172.21.15.44:0/3753378095 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0a0063890 con 0x7ff0a0109190 2024-04-21T22:58:31.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.486+0000 7ff09dffb700 1 -- 172.21.15.44:0/3753378095 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff09401acd0 con 0x7ff0a0109190 2024-04-21T22:58:31.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.559+0000 7ff0a6d1e700 1 -- 172.21.15.44:0/3753378095 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff0a01079b0 con 0x7ff0a0109190 2024-04-21T22:58:31.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.560+0000 7ff09dffb700 1 -- 172.21.15.44:0/3753378095 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7ff09402d3f0 con 0x7ff0a0109190 2024-04-21T22:58:31.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.561+0000 7ff0a6d1e700 1 -- 172.21.15.44:0/3753378095 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0a0109190 msgr2=0x7ff0a01087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:31.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.561+0000 7ff0a6d1e700 1 --2- 172.21.15.44:0/3753378095 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0a0109190 0x7ff0a01087b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff094000c00 tx=0x7ff094011c10 comp rx=0 tx=0).stop 2024-04-21T22:58:31.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.561+0000 7ff0a6d1e700 1 -- 172.21.15.44:0/3753378095 shutdown_connections 2024-04-21T22:58:31.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.561+0000 7ff0a6d1e700 1 --2- 172.21.15.44:0/3753378095 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff0a0109190 0x7ff0a01087b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:31.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.561+0000 7ff0a6d1e700 1 -- 172.21.15.44:0/3753378095 >> 172.21.15.44:0/3753378095 conn(0x7ff0a0101770 msgr2=0x7ff0a0102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:31.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.561+0000 7ff0a6d1e700 1 -- 172.21.15.44:0/3753378095 shutdown_connections 2024-04-21T22:58:31.158 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:30.561+0000 7ff0a6d1e700 1 -- 172.21.15.44:0/3753378095 wait complete. 2024-04-21T22:58:31.158 INFO:teuthology.orchestra.run.smithi044.stdout:mgr not available, waiting (1/15)... 2024-04-21T22:58:32.418 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:32 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:32.065+0000 7fef710b9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T22:58:32.418 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:32 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:32.241+0000 7fef710b9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T22:58:32.418 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:32 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:32.418+0000 7fef710b9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T22:58:33.038 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:32 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:32.716+0000 7fef710b9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T22:58:33.038 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:32 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:32.828+0000 7fef710b9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T22:58:33.038 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:33 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:33.039+0000 7fef710b9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T22:58:33.352 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:33 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:33.248+0000 7fef710b9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T22:58:33.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:33 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3518812519' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T22:58:33.936 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:33 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:33.895+0000 7fef710b9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T22:58:34.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-21T22:58:34.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-04-21T22:58:34.296 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsid": "80909b94-0032-11ef-bc93-c7b262605968", 2024-04-21T22:58:34.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T22:58:34.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T22:58:34.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T22:58:34.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T22:58:34.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:34.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T22:58:34.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T22:58:34.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 0 2024-04-21T22:58:34.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-21T22:58:34.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T22:58:34.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T22:58:34.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-21T22:58:34.297 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-21T22:58:34.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T22:58:34.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T22:58:34.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T22:58:34.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T22:58:34.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:34.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T22:58:34.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T22:58:34.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T22:58:34.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T22:58:34.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T22:58:34.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T22:58:34.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T22:58:34.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T22:58:34.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:34.298 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T22:58:34.299 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T22:58:34.299 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T22:58:34.299 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T22:58:34.299 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T22:58:34.299 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T22:58:34.299 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T22:58:34.299 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T22:58:34.299 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T22:58:34.299 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:34.299 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T22:58:34.299 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T22:58:34.299 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T22:58:34.299 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T22:58:34.299 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:34.300 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T22:58:34.300 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-21T22:58:34.300 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T22:58:34.300 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T22:58:34.300 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T22:58:34.300 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T22:58:34.300 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T22:58:34.300 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-21T22:58:34.300 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T22:58:34.300 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:34.300 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T22:58:34.300 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T22:58:34.300 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T22:58:23.196647+0000", 2024-04-21T22:58:34.300 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T22:58:34.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:34.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T22:58:34.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-04-21T22:58:34.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.574+0000 7f1d2122f700 1 Processor -- start 2024-04-21T22:58:34.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.574+0000 7f1d2122f700 1 -- start start 2024-04-21T22:58:34.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.575+0000 7f1d2122f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d1c1085f0 0x7f1d1c10a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:34.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.575+0000 7f1d2122f700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1c07bed0 con 0x7f1d1c1085f0 2024-04-21T22:58:34.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.575+0000 7f1d1ad9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d1c1085f0 0x7f1d1c10a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:34.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.575+0000 7f1d1ad9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d1c1085f0 0x7f1d1c10a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35458/0 (socket says 172.21.15.44:35458) 2024-04-21T22:58:34.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.575+0000 7f1d1ad9d700 1 -- 172.21.15.44:0/510444019 learned_addr learned my addr 172.21.15.44:0/510444019 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:34.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.575+0000 7f1d1ad9d700 1 -- 172.21.15.44:0/510444019 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d1c10af20 con 0x7f1d1c1085f0 2024-04-21T22:58:34.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.575+0000 7f1d1ad9d700 1 --2- 172.21.15.44:0/510444019 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d1c1085f0 0x7f1d1c10a9e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c0098a0 tx=0x7f1d0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df0d293335e6b8ac server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:34.301 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.576+0000 7f1d19d9b700 1 -- 172.21.15.44:0/510444019 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d0c004160 con 0x7f1d1c1085f0 2024-04-21T22:58:34.302 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.576+0000 7f1d19d9b700 1 -- 172.21.15.44:0/510444019 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1d0c004300 con 0x7f1d1c1085f0 2024-04-21T22:58:34.302 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.576+0000 7f1d19d9b700 1 -- 172.21.15.44:0/510444019 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d0c011b50 con 0x7f1d1c1085f0 2024-04-21T22:58:34.302 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.576+0000 7f1d2122f700 1 -- 172.21.15.44:0/510444019 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d1c1085f0 msgr2=0x7f1d1c10a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:34.302 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.576+0000 7f1d2122f700 1 --2- 172.21.15.44:0/510444019 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d1c1085f0 0x7f1d1c10a9e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c0098a0 tx=0x7f1d0c009bb0 comp rx=0 tx=0).stop 2024-04-21T22:58:34.302 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.576+0000 7f1d2122f700 1 -- 172.21.15.44:0/510444019 shutdown_connections 2024-04-21T22:58:34.302 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.576+0000 7f1d2122f700 1 --2- 172.21.15.44:0/510444019 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d1c1085f0 0x7f1d1c10a9e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:34.302 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.576+0000 7f1d2122f700 1 -- 172.21.15.44:0/510444019 >> 172.21.15.44:0/510444019 conn(0x7f1d1c102000 msgr2=0x7f1d1c104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:34.302 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.577+0000 7f1d2122f700 1 -- 172.21.15.44:0/510444019 shutdown_connections 2024-04-21T22:58:34.302 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.577+0000 7f1d2122f700 1 -- 172.21.15.44:0/510444019 wait complete. 2024-04-21T22:58:34.302 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.577+0000 7f1d2122f700 1 Processor -- start 2024-04-21T22:58:34.302 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.578+0000 7f1d2122f700 1 -- start start 2024-04-21T22:58:34.302 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.578+0000 7f1d2122f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d1c1085f0 0x7f1d1c107e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:34.302 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.578+0000 7f1d2122f700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1c10bf50 con 0x7f1d1c1085f0 2024-04-21T22:58:34.303 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.578+0000 7f1d1ad9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d1c1085f0 0x7f1d1c107e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:34.303 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.578+0000 7f1d1ad9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d1c1085f0 0x7f1d1c107e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35462/0 (socket says 172.21.15.44:35462) 2024-04-21T22:58:34.303 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.578+0000 7f1d1ad9d700 1 -- 172.21.15.44:0/3518812519 learned_addr learned my addr 172.21.15.44:0/3518812519 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:34.303 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.578+0000 7f1d1ad9d700 1 -- 172.21.15.44:0/3518812519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d1c1083b0 con 0x7f1d1c1085f0 2024-04-21T22:58:34.303 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.578+0000 7f1d1ad9d700 1 --2- 172.21.15.44:0/3518812519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d1c1085f0 0x7f1d1c107e70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c011670 tx=0x7f1d0c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:34.303 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.579+0000 7f1d03fff700 1 -- 172.21.15.44:0/3518812519 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d0c0039c0 con 0x7f1d1c1085f0 2024-04-21T22:58:34.303 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.579+0000 7f1d2122f700 1 -- 172.21.15.44:0/3518812519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d1c1065a0 con 0x7f1d1c1085f0 2024-04-21T22:58:34.303 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.579+0000 7f1d03fff700 1 -- 172.21.15.44:0/3518812519 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1d0c011e50 con 0x7f1d1c1085f0 2024-04-21T22:58:34.303 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.579+0000 7f1d03fff700 1 -- 172.21.15.44:0/3518812519 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d0c01a5c0 con 0x7f1d1c1085f0 2024-04-21T22:58:34.303 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.579+0000 7f1d2122f700 1 -- 172.21.15.44:0/3518812519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d1c106ae0 con 0x7f1d1c1085f0 2024-04-21T22:58:34.303 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.579+0000 7f1d03fff700 1 -- 172.21.15.44:0/3518812519 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1d0c01ac00 con 0x7f1d1c1085f0 2024-04-21T22:58:34.303 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.579+0000 7f1d03fff700 1 -- 172.21.15.44:0/3518812519 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1d0c019700 con 0x7f1d1c1085f0 2024-04-21T22:58:34.303 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.580+0000 7f1d2122f700 1 -- 172.21.15.44:0/3518812519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d1c063890 con 0x7f1d1c1085f0 2024-04-21T22:58:34.303 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.582+0000 7f1d03fff700 1 -- 172.21.15.44:0/3518812519 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f1d0c003b60 con 0x7f1d1c1085f0 2024-04-21T22:58:34.304 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.656+0000 7f1d2122f700 1 -- 172.21.15.44:0/3518812519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f1d1c107010 con 0x7f1d1c1085f0 2024-04-21T22:58:34.304 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.656+0000 7f1d03fff700 1 -- 172.21.15.44:0/3518812519 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f1d0c0199a0 con 0x7f1d1c1085f0 2024-04-21T22:58:34.304 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.657+0000 7f1d2122f700 1 -- 172.21.15.44:0/3518812519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d1c1085f0 msgr2=0x7f1d1c107e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:34.304 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.657+0000 7f1d2122f700 1 --2- 172.21.15.44:0/3518812519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d1c1085f0 0x7f1d1c107e70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c011670 tx=0x7f1d0c011b60 comp rx=0 tx=0).stop 2024-04-21T22:58:34.304 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.657+0000 7f1d2122f700 1 -- 172.21.15.44:0/3518812519 shutdown_connections 2024-04-21T22:58:34.304 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.657+0000 7f1d2122f700 1 --2- 172.21.15.44:0/3518812519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d1c1085f0 0x7f1d1c107e70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:34.304 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.657+0000 7f1d2122f700 1 -- 172.21.15.44:0/3518812519 >> 172.21.15.44:0/3518812519 conn(0x7f1d1c102000 msgr2=0x7f1d1c102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:34.304 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.658+0000 7f1d2122f700 1 -- 172.21.15.44:0/3518812519 shutdown_connections 2024-04-21T22:58:34.304 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:33.658+0000 7f1d2122f700 1 -- 172.21.15.44:0/3518812519 wait complete. 2024-04-21T22:58:34.304 INFO:teuthology.orchestra.run.smithi044.stdout:mgr not available, waiting (2/15)... 2024-04-21T22:58:34.642 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:34 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:34.334+0000 7fef710b9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T22:58:34.642 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:34 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:34.434+0000 7fef710b9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T22:58:34.642 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:34 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:34.643+0000 7fef710b9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T22:58:35.538 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:35 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:35.193+0000 7fef710b9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T22:58:35.538 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:35 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:35.288+0000 7fef710b9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T22:58:35.895 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:35 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:35.539+0000 7fef710b9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T22:58:35.895 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:35 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:35.678+0000 7fef710b9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T22:58:35.895 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:35 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:35.776+0000 7fef710b9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T22:58:35.896 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:35 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:35.896+0000 7fef710b9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T22:58:36.185 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:35 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:35.990+0000 7fef710b9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T22:58:36.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:36 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2668203195' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T22:58:36.936 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:36 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:36.854+0000 7fef710b9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T22:58:37.334 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:37 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:37.077+0000 7fef710b9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T22:58:37.347 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-21T22:58:37.347 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-04-21T22:58:37.347 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsid": "80909b94-0032-11ef-bc93-c7b262605968", 2024-04-21T22:58:37.347 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T22:58:37.347 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T22:58:37.347 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T22:58:37.347 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T22:58:37.347 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:37.347 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T22:58:37.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T22:58:37.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 0 2024-04-21T22:58:37.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-21T22:58:37.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T22:58:37.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T22:58:37.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-21T22:58:37.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-21T22:58:37.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T22:58:37.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T22:58:37.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T22:58:37.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T22:58:37.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:37.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T22:58:37.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T22:58:37.348 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T22:58:37.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T22:58:37.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T22:58:37.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T22:58:37.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T22:58:37.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T22:58:37.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:37.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T22:58:37.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T22:58:37.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T22:58:37.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T22:58:37.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T22:58:37.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T22:58:37.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T22:58:37.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T22:58:37.349 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T22:58:37.350 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:37.350 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T22:58:37.350 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T22:58:37.350 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T22:58:37.350 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T22:58:37.350 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:37.350 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T22:58:37.350 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-21T22:58:37.350 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T22:58:37.350 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T22:58:37.350 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T22:58:37.350 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T22:58:37.350 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T22:58:37.351 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-21T22:58:37.351 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T22:58:37.351 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:37.351 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T22:58:37.351 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T22:58:37.351 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T22:58:23.196647+0000", 2024-04-21T22:58:37.351 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T22:58:37.351 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:37.351 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T22:58:37.351 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-04-21T22:58:37.351 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.696+0000 7ff57f0ac700 1 Processor -- start 2024-04-21T22:58:37.351 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.696+0000 7ff57f0ac700 1 -- start start 2024-04-21T22:58:37.351 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.696+0000 7ff57f0ac700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff578107bc0 0x7ff578107fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:37.351 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.697+0000 7ff57f0ac700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff5781085b0 con 0x7ff578107bc0 2024-04-21T22:58:37.352 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.697+0000 7ff57ce48700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff578107bc0 0x7ff578107fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:37.352 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.697+0000 7ff57ce48700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff578107bc0 0x7ff578107fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35464/0 (socket says 172.21.15.44:35464) 2024-04-21T22:58:37.352 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.697+0000 7ff57ce48700 1 -- 172.21.15.44:0/3429862075 learned_addr learned my addr 172.21.15.44:0/3429862075 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:37.352 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.697+0000 7ff57ce48700 1 -- 172.21.15.44:0/3429862075 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff578108d70 con 0x7ff578107bc0 2024-04-21T22:58:37.352 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.697+0000 7ff57ce48700 1 --2- 172.21.15.44:0/3429862075 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff578107bc0 0x7ff578107fe0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff56c0098a0 tx=0x7ff56c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ad03af0ee5b0be1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:37.352 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.697+0000 7ff5777fe700 1 -- 172.21.15.44:0/3429862075 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff56c004160 con 0x7ff578107bc0 2024-04-21T22:58:37.352 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.697+0000 7ff5777fe700 1 -- 172.21.15.44:0/3429862075 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff56c004300 con 0x7ff578107bc0 2024-04-21T22:58:37.352 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.697+0000 7ff5777fe700 1 -- 172.21.15.44:0/3429862075 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff56c011b50 con 0x7ff578107bc0 2024-04-21T22:58:37.352 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.698+0000 7ff57f0ac700 1 -- 172.21.15.44:0/3429862075 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff578107bc0 msgr2=0x7ff578107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:37.352 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.698+0000 7ff57f0ac700 1 --2- 172.21.15.44:0/3429862075 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff578107bc0 0x7ff578107fe0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff56c0098a0 tx=0x7ff56c009bb0 comp rx=0 tx=0).stop 2024-04-21T22:58:37.352 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.698+0000 7ff57f0ac700 1 -- 172.21.15.44:0/3429862075 shutdown_connections 2024-04-21T22:58:37.352 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.698+0000 7ff57f0ac700 1 --2- 172.21.15.44:0/3429862075 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff578107bc0 0x7ff578107fe0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:37.352 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.698+0000 7ff57f0ac700 1 -- 172.21.15.44:0/3429862075 >> 172.21.15.44:0/3429862075 conn(0x7ff578103120 msgr2=0x7ff5781055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:37.352 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.698+0000 7ff57f0ac700 1 -- 172.21.15.44:0/3429862075 shutdown_connections 2024-04-21T22:58:37.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.698+0000 7ff57f0ac700 1 -- 172.21.15.44:0/3429862075 wait complete. 2024-04-21T22:58:37.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.699+0000 7ff57f0ac700 1 Processor -- start 2024-04-21T22:58:37.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.699+0000 7ff57f0ac700 1 -- start start 2024-04-21T22:58:37.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.699+0000 7ff57f0ac700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff578107bc0 0x7ff57807db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:37.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.699+0000 7ff57f0ac700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff57810c120 con 0x7ff578107bc0 2024-04-21T22:58:37.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.699+0000 7ff57ce48700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff578107bc0 0x7ff57807db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:37.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.699+0000 7ff57ce48700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff578107bc0 0x7ff57807db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35478/0 (socket says 172.21.15.44:35478) 2024-04-21T22:58:37.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.699+0000 7ff57ce48700 1 -- 172.21.15.44:0/2668203195 learned_addr learned my addr 172.21.15.44:0/2668203195 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:37.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.699+0000 7ff57ce48700 1 -- 172.21.15.44:0/2668203195 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff57807e0d0 con 0x7ff578107bc0 2024-04-21T22:58:37.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.700+0000 7ff57ce48700 1 --2- 172.21.15.44:0/2668203195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff578107bc0 0x7ff57807db90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff56c009fb0 tx=0x7ff56c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:37.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.700+0000 7ff575ffb700 1 -- 172.21.15.44:0/2668203195 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff56c0038b0 con 0x7ff578107bc0 2024-04-21T22:58:37.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.700+0000 7ff57f0ac700 1 -- 172.21.15.44:0/2668203195 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff57807c2c0 con 0x7ff578107bc0 2024-04-21T22:58:37.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.700+0000 7ff575ffb700 1 -- 172.21.15.44:0/2668203195 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff56c011e50 con 0x7ff578107bc0 2024-04-21T22:58:37.353 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.700+0000 7ff575ffb700 1 -- 172.21.15.44:0/2668203195 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff56c019550 con 0x7ff578107bc0 2024-04-21T22:58:37.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.700+0000 7ff57f0ac700 1 -- 172.21.15.44:0/2668203195 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff57807c800 con 0x7ff578107bc0 2024-04-21T22:58:37.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.700+0000 7ff575ffb700 1 -- 172.21.15.44:0/2668203195 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff56c019be0 con 0x7ff578107bc0 2024-04-21T22:58:37.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.700+0000 7ff575ffb700 1 -- 172.21.15.44:0/2668203195 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff56c024730 con 0x7ff578107bc0 2024-04-21T22:58:37.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.701+0000 7ff57f0ac700 1 -- 172.21.15.44:0/2668203195 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff578063890 con 0x7ff578107bc0 2024-04-21T22:58:37.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.703+0000 7ff575ffb700 1 -- 172.21.15.44:0/2668203195 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff56c003a50 con 0x7ff578107bc0 2024-04-21T22:58:37.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.777+0000 7ff57f0ac700 1 -- 172.21.15.44:0/2668203195 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff57807ca80 con 0x7ff578107bc0 2024-04-21T22:58:37.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.778+0000 7ff575ffb700 1 -- 172.21.15.44:0/2668203195 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7ff56c0249d0 con 0x7ff578107bc0 2024-04-21T22:58:37.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.779+0000 7ff57f0ac700 1 -- 172.21.15.44:0/2668203195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff578107bc0 msgr2=0x7ff57807db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:37.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.779+0000 7ff57f0ac700 1 --2- 172.21.15.44:0/2668203195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff578107bc0 0x7ff57807db90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff56c009fb0 tx=0x7ff56c011bf0 comp rx=0 tx=0).stop 2024-04-21T22:58:37.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.779+0000 7ff57f0ac700 1 -- 172.21.15.44:0/2668203195 shutdown_connections 2024-04-21T22:58:37.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.779+0000 7ff57f0ac700 1 --2- 172.21.15.44:0/2668203195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff578107bc0 0x7ff57807db90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:37.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.779+0000 7ff57f0ac700 1 -- 172.21.15.44:0/2668203195 >> 172.21.15.44:0/2668203195 conn(0x7ff578103120 msgr2=0x7ff578103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:37.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.779+0000 7ff57f0ac700 1 -- 172.21.15.44:0/2668203195 shutdown_connections 2024-04-21T22:58:37.354 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:36.779+0000 7ff57f0ac700 1 -- 172.21.15.44:0/2668203195 wait complete. 2024-04-21T22:58:37.354 INFO:teuthology.orchestra.run.smithi044.stdout:mgr not available, waiting (3/15)... 2024-04-21T22:58:38.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:37 smithi044 ceph-mon[27934]: Activating manager daemon x 2024-04-21T22:58:38.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:37 smithi044 ceph-mon[27934]: mgrmap e2: x(active, starting, since 0.0030892s) 2024-04-21T22:58:38.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:37 smithi044 ceph-mon[27934]: from='mgr.14100 172.21.15.44:0/2736331096' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T22:58:38.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:37 smithi044 ceph-mon[27934]: from='mgr.14100 172.21.15.44:0/2736331096' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T22:58:38.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:37 smithi044 ceph-mon[27934]: from='mgr.14100 172.21.15.44:0/2736331096' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T22:58:38.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:37 smithi044 ceph-mon[27934]: from='mgr.14100 172.21.15.44:0/2736331096' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T22:58:38.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:37 smithi044 ceph-mon[27934]: from='mgr.14100 172.21.15.44:0/2736331096' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T22:58:38.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:37 smithi044 ceph-mon[27934]: Manager daemon x is now available 2024-04-21T22:58:38.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:37 smithi044 ceph-mon[27934]: from='mgr.14100 172.21.15.44:0/2736331096' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T22:58:38.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:37 smithi044 ceph-mon[27934]: from='mgr.14100 172.21.15.44:0/2736331096' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T22:58:38.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:37 smithi044 ceph-mon[27934]: from='mgr.14100 172.21.15.44:0/2736331096' entity='mgr.x' 2024-04-21T22:58:38.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:37 smithi044 ceph-mon[27934]: from='mgr.14100 172.21.15.44:0/2736331096' entity='mgr.x' 2024-04-21T22:58:38.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:37 smithi044 ceph-mon[27934]: from='mgr.14100 172.21.15.44:0/2736331096' entity='mgr.x' 2024-04-21T22:58:39.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:39 smithi044 ceph-mon[27934]: mgrmap e3: x(active, since 1.00971s) 2024-04-21T22:58:40.394 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:40 smithi044 ceph-mon[27934]: mgrmap e4: x(active, since 2s) 2024-04-21T22:58:42.711 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:42 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2290684287' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T22:58:47.509 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-21T22:58:47.509 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-04-21T22:58:47.509 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsid": "80909b94-0032-11ef-bc93-c7b262605968", 2024-04-21T22:58:47.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T22:58:47.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T22:58:47.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T22:58:47.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T22:58:47.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:47.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T22:58:47.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T22:58:47.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 0 2024-04-21T22:58:47.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-21T22:58:47.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T22:58:47.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T22:58:47.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-21T22:58:47.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-21T22:58:47.510 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T22:58:47.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T22:58:47.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T22:58:47.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T22:58:47.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:47.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T22:58:47.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T22:58:47.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T22:58:47.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T22:58:47.511 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T22:58:47.520 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T22:58:47.520 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T22:58:47.521 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T22:58:47.521 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:47.521 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T22:58:47.521 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T22:58:47.521 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T22:58:47.521 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T22:58:47.521 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T22:58:47.521 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T22:58:47.521 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T22:58:47.521 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T22:58:47.521 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T22:58:47.521 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:47.521 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T22:58:47.522 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T22:58:47.522 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T22:58:47.522 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T22:58:47.522 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:47.522 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T22:58:47.522 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T22:58:47.522 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T22:58:47.522 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T22:58:47.522 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T22:58:47.522 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T22:58:47.522 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T22:58:47.522 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ], 2024-04-21T22:58:47.522 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T22:58:47.522 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:47.523 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T22:58:47.523 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T22:58:47.523 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T22:58:23.196647+0000", 2024-04-21T22:58:47.523 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T22:58:47.523 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout }, 2024-04-21T22:58:47.523 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T22:58:47.523 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-04-21T22:58:47.523 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.189+0000 7f6fb3072700 1 Processor -- start 2024-04-21T22:58:47.523 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.189+0000 7f6fb3072700 1 -- start start 2024-04-21T22:58:47.523 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.189+0000 7f6fb3072700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fac07de20 0x7f6fac07c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:47.523 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.189+0000 7f6fb3072700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac07c860 con 0x7f6fac07de20 2024-04-21T22:58:47.523 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.189+0000 7f6fb0e0e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fac07de20 0x7f6fac07c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:47.523 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.189+0000 7f6fb0e0e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fac07de20 0x7f6fac07c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43124/0 (socket says 172.21.15.44:43124) 2024-04-21T22:58:47.523 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.189+0000 7f6fb0e0e700 1 -- 172.21.15.44:0/3940960407 learned_addr learned my addr 172.21.15.44:0/3940960407 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:47.524 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.190+0000 7f6fb0e0e700 1 -- 172.21.15.44:0/3940960407 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fac07c9e0 con 0x7f6fac07de20 2024-04-21T22:58:47.524 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.190+0000 7f6fb0e0e700 1 --2- 172.21.15.44:0/3940960407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fac07de20 0x7f6fac07c320 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6fa00098a0 tx=0x7f6fa0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=698327401d00ccf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:47.524 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.190+0000 7f6fab7fe700 1 -- 172.21.15.44:0/3940960407 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6fa0004160 con 0x7f6fac07de20 2024-04-21T22:58:47.524 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.190+0000 7f6fab7fe700 1 -- 172.21.15.44:0/3940960407 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6fa0004300 con 0x7f6fac07de20 2024-04-21T22:58:47.524 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.190+0000 7f6fab7fe700 1 -- 172.21.15.44:0/3940960407 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6fa0011b50 con 0x7f6fac07de20 2024-04-21T22:58:47.524 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.191+0000 7f6fb3072700 1 -- 172.21.15.44:0/3940960407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fac07de20 msgr2=0x7f6fac07c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:47.524 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.191+0000 7f6fb3072700 1 --2- 172.21.15.44:0/3940960407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fac07de20 0x7f6fac07c320 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6fa00098a0 tx=0x7f6fa0009bb0 comp rx=0 tx=0).stop 2024-04-21T22:58:47.524 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.191+0000 7f6fb3072700 1 -- 172.21.15.44:0/3940960407 shutdown_connections 2024-04-21T22:58:47.524 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.191+0000 7f6fb3072700 1 --2- 172.21.15.44:0/3940960407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fac07de20 0x7f6fac07c320 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:47.524 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.191+0000 7f6fb3072700 1 -- 172.21.15.44:0/3940960407 >> 172.21.15.44:0/3940960407 conn(0x7f6fac103230 msgr2=0x7f6fac105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:47.524 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.191+0000 7f6fb3072700 1 -- 172.21.15.44:0/3940960407 shutdown_connections 2024-04-21T22:58:47.524 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.191+0000 7f6fb3072700 1 -- 172.21.15.44:0/3940960407 wait complete. 2024-04-21T22:58:47.524 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.191+0000 7f6fb3072700 1 Processor -- start 2024-04-21T22:58:47.524 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.191+0000 7f6fb3072700 1 -- start start 2024-04-21T22:58:47.525 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.191+0000 7f6fb3072700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fac07de20 0x7f6fac19dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:47.525 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.191+0000 7f6fb3072700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac10e1c0 con 0x7f6fac07de20 2024-04-21T22:58:47.525 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.192+0000 7f6fb0e0e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fac07de20 0x7f6fac19dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:47.525 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.192+0000 7f6fb0e0e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fac07de20 0x7f6fac19dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43138/0 (socket says 172.21.15.44:43138) 2024-04-21T22:58:47.525 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.192+0000 7f6fb0e0e700 1 -- 172.21.15.44:0/2290684287 learned_addr learned my addr 172.21.15.44:0/2290684287 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:47.525 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.192+0000 7f6fb0e0e700 1 -- 172.21.15.44:0/2290684287 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fac19e2e0 con 0x7f6fac07de20 2024-04-21T22:58:47.525 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.192+0000 7f6fb0e0e700 1 --2- 172.21.15.44:0/2290684287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fac07de20 0x7f6fac19dda0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0009fb0 tx=0x7f6fa0011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:47.525 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.192+0000 7f6fa9ffb700 1 -- 172.21.15.44:0/2290684287 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6fa00038b0 con 0x7f6fac07de20 2024-04-21T22:58:47.525 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.192+0000 7f6fa9ffb700 1 -- 172.21.15.44:0/2290684287 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6fa0011e50 con 0x7f6fac07de20 2024-04-21T22:58:47.525 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.192+0000 7f6fb3072700 1 -- 172.21.15.44:0/2290684287 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fac19e570 con 0x7f6fac07de20 2024-04-21T22:58:47.525 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.192+0000 7f6fa9ffb700 1 -- 172.21.15.44:0/2290684287 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6fa0019420 con 0x7f6fac07de20 2024-04-21T22:58:47.525 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.192+0000 7f6fb3072700 1 -- 172.21.15.44:0/2290684287 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fac1a1220 con 0x7f6fac07de20 2024-04-21T22:58:47.525 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.193+0000 7f6fa9ffb700 1 -- 172.21.15.44:0/2290684287 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f6fa001a430 con 0x7f6fac07de20 2024-04-21T22:58:47.525 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.193+0000 7f6fb3072700 1 -- 172.21.15.44:0/2290684287 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fac063890 con 0x7f6fac07de20 2024-04-21T22:58:47.526 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.193+0000 7f6fa9ffb700 1 --2- 172.21.15.44:0/2290684287 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f6f9c039000 0x7f6f9c03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:47.526 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.193+0000 7f6fa9ffb700 1 -- 172.21.15.44:0/2290684287 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6fa004d150 con 0x7f6fac07de20 2024-04-21T22:58:47.526 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.194+0000 7f6fabfff700 1 --2- 172.21.15.44:0/2290684287 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f6f9c039000 0x7f6f9c03b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:47.526 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.194+0000 7f6fabfff700 1 --2- 172.21.15.44:0/2290684287 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f6f9c039000 0x7f6f9c03b4c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6f98006fd0 tx=0x7f6f98006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:47.526 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.199+0000 7f6fa9ffb700 1 -- 172.21.15.44:0/2290684287 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6fa002cc80 con 0x7f6fac07de20 2024-04-21T22:58:47.526 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.490+0000 7f6fb3072700 1 -- 172.21.15.44:0/2290684287 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f6fac1a1440 con 0x7f6fac07de20 2024-04-21T22:58:47.526 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.491+0000 7f6fa9ffb700 1 -- 172.21.15.44:0/2290684287 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f6fac1a1440 con 0x7f6fac07de20 2024-04-21T22:58:47.526 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.493+0000 7f6fb3072700 1 -- 172.21.15.44:0/2290684287 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f6f9c039000 msgr2=0x7f6f9c03b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:47.526 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.493+0000 7f6fb3072700 1 --2- 172.21.15.44:0/2290684287 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f6f9c039000 0x7f6f9c03b4c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6f98006fd0 tx=0x7f6f98006e40 comp rx=0 tx=0).stop 2024-04-21T22:58:47.526 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.493+0000 7f6fb3072700 1 -- 172.21.15.44:0/2290684287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fac07de20 msgr2=0x7f6fac19dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:47.526 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.493+0000 7f6fb3072700 1 --2- 172.21.15.44:0/2290684287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fac07de20 0x7f6fac19dda0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0009fb0 tx=0x7f6fa0011bf0 comp rx=0 tx=0).stop 2024-04-21T22:58:47.526 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.493+0000 7f6fb3072700 1 -- 172.21.15.44:0/2290684287 shutdown_connections 2024-04-21T22:58:47.526 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.493+0000 7f6fb3072700 1 --2- 172.21.15.44:0/2290684287 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f6f9c039000 0x7f6f9c03b4c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:47.527 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.493+0000 7f6fb3072700 1 --2- 172.21.15.44:0/2290684287 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6fac07de20 0x7f6fac19dda0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:47.527 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.493+0000 7f6fb3072700 1 -- 172.21.15.44:0/2290684287 >> 172.21.15.44:0/2290684287 conn(0x7f6fac103230 msgr2=0x7f6fac103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:47.527 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.493+0000 7f6fb3072700 1 -- 172.21.15.44:0/2290684287 shutdown_connections 2024-04-21T22:58:47.527 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:42.493+0000 7f6fb3072700 1 -- 172.21.15.44:0/2290684287 wait complete. 2024-04-21T22:58:47.527 INFO:teuthology.orchestra.run.smithi044.stdout:mgr is available 2024-04-21T22:58:48.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:48 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/4166962933' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-21T22:58:48.695 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-21T22:58:48.695 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [global] 2024-04-21T22:58:48.695 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout fsid = 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T22:58:48.695 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-21T22:58:48.695 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-21T22:58:48.695 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-21T22:58:48.695 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-21T22:58:48.695 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-21T22:58:48.695 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-21T22:58:48.696 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-21T22:58:48.696 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 2024-04-21T22:58:48.696 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout [osd] 2024-04-21T22:58:48.696 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-21T22:58:48.696 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-21T22:58:48.696 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-21T22:58:48.696 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-21T22:58:48.696 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-21T22:58:48.696 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.906+0000 7fe808b58700 1 Processor -- start 2024-04-21T22:58:48.696 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.906+0000 7fe808b58700 1 -- start start 2024-04-21T22:58:48.696 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.906+0000 7fe808b58700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe80407dea0 0x7fe80407c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:48.696 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.906+0000 7fe808b58700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe80407c8d0 con 0x7fe80407dea0 2024-04-21T22:58:48.696 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.907+0000 7fe80259c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe80407dea0 0x7fe80407c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:48.696 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.907+0000 7fe80259c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe80407dea0 0x7fe80407c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43154/0 (socket says 172.21.15.44:43154) 2024-04-21T22:58:48.697 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.907+0000 7fe80259c700 1 -- 172.21.15.44:0/1974937740 learned_addr learned my addr 172.21.15.44:0/1974937740 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:48.697 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.907+0000 7fe80259c700 1 -- 172.21.15.44:0/1974937740 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe80407ca50 con 0x7fe80407dea0 2024-04-21T22:58:48.697 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.907+0000 7fe80259c700 1 --2- 172.21.15.44:0/1974937740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe80407dea0 0x7fe80407c300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4009e80 tx=0x7fe7f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b2ee06a98656688 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:48.697 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.907+0000 7fe80159a700 1 -- 172.21.15.44:0/1974937740 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7f4017070 con 0x7fe80407dea0 2024-04-21T22:58:48.697 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.907+0000 7fe80159a700 1 -- 172.21.15.44:0/1974937740 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe7f400b8c0 con 0x7fe80407dea0 2024-04-21T22:58:48.697 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.907+0000 7fe80159a700 1 -- 172.21.15.44:0/1974937740 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7f40045c0 con 0x7fe80407dea0 2024-04-21T22:58:48.697 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.908+0000 7fe808b58700 1 -- 172.21.15.44:0/1974937740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe80407dea0 msgr2=0x7fe80407c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:48.697 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.908+0000 7fe808b58700 1 --2- 172.21.15.44:0/1974937740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe80407dea0 0x7fe80407c300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4009e80 tx=0x7fe7f400b330 comp rx=0 tx=0).stop 2024-04-21T22:58:48.697 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.908+0000 7fe808b58700 1 -- 172.21.15.44:0/1974937740 shutdown_connections 2024-04-21T22:58:48.697 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.908+0000 7fe808b58700 1 --2- 172.21.15.44:0/1974937740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe80407dea0 0x7fe80407c300 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:48.697 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.908+0000 7fe808b58700 1 -- 172.21.15.44:0/1974937740 >> 172.21.15.44:0/1974937740 conn(0x7fe804103100 msgr2=0x7fe804105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:48.697 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.908+0000 7fe808b58700 1 -- 172.21.15.44:0/1974937740 shutdown_connections 2024-04-21T22:58:48.697 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.908+0000 7fe808b58700 1 -- 172.21.15.44:0/1974937740 wait complete. 2024-04-21T22:58:48.697 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.909+0000 7fe808b58700 1 Processor -- start 2024-04-21T22:58:48.698 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.909+0000 7fe808b58700 1 -- start start 2024-04-21T22:58:48.698 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.909+0000 7fe808b58700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe80407dea0 0x7fe80419dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:48.698 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.909+0000 7fe808b58700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe80410e1b0 con 0x7fe80407dea0 2024-04-21T22:58:48.698 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.909+0000 7fe80259c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe80407dea0 0x7fe80419dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:48.698 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.909+0000 7fe80259c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe80407dea0 0x7fe80419dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43156/0 (socket says 172.21.15.44:43156) 2024-04-21T22:58:48.698 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.909+0000 7fe80259c700 1 -- 172.21.15.44:0/4166962933 learned_addr learned my addr 172.21.15.44:0/4166962933 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:48.698 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.909+0000 7fe80259c700 1 -- 172.21.15.44:0/4166962933 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe80419e240 con 0x7fe80407dea0 2024-04-21T22:58:48.698 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.910+0000 7fe80259c700 1 --2- 172.21.15.44:0/4166962933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe80407dea0 0x7fe80419dd00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4004350 tx=0x7fe7f400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:48.698 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.910+0000 7fe7fb7fe700 1 -- 172.21.15.44:0/4166962933 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7f4020020 con 0x7fe80407dea0 2024-04-21T22:58:48.698 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.910+0000 7fe808b58700 1 -- 172.21.15.44:0/4166962933 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe80419e4d0 con 0x7fe80407dea0 2024-04-21T22:58:48.698 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.910+0000 7fe7fb7fe700 1 -- 172.21.15.44:0/4166962933 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe7f4003c10 con 0x7fe80407dea0 2024-04-21T22:58:48.698 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.910+0000 7fe7fb7fe700 1 -- 172.21.15.44:0/4166962933 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7f4017040 con 0x7fe80407dea0 2024-04-21T22:58:48.698 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.910+0000 7fe808b58700 1 -- 172.21.15.44:0/4166962933 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8041a11b0 con 0x7fe80407dea0 2024-04-21T22:58:48.699 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.911+0000 7fe7fb7fe700 1 -- 172.21.15.44:0/4166962933 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7fe7f4023020 con 0x7fe80407dea0 2024-04-21T22:58:48.699 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.911+0000 7fe808b58700 1 -- 172.21.15.44:0/4166962933 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe804063890 con 0x7fe80407dea0 2024-04-21T22:58:48.699 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.911+0000 7fe7fb7fe700 1 --2- 172.21.15.44:0/4166962933 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7fe7f0039000 0x7fe7f003b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:48.699 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.911+0000 7fe7fb7fe700 1 -- 172.21.15.44:0/4166962933 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe7f400f050 con 0x7fe80407dea0 2024-04-21T22:58:48.699 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.911+0000 7fe801d9b700 1 --2- 172.21.15.44:0/4166962933 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7fe7f0039000 0x7fe7f003b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:48.699 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.911+0000 7fe801d9b700 1 --2- 172.21.15.44:0/4166962933 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7fe7f0039000 0x7fe7f003b4c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec006fd0 tx=0x7fe7ec006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:48.699 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:47.916+0000 7fe7fb7fe700 1 -- 172.21.15.44:0/4166962933 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe7f404ee20 con 0x7fe80407dea0 2024-04-21T22:58:48.699 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:48.120+0000 7fe808b58700 1 -- 172.21.15.44:0/4166962933 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fe8041a1380 con 0x7fe80407dea0 2024-04-21T22:58:48.699 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:48.121+0000 7fe7fb7fe700 1 -- 172.21.15.44:0/4166962933 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7fe8041a1380 con 0x7fe80407dea0 2024-04-21T22:58:48.699 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:48.124+0000 7fe808b58700 1 -- 172.21.15.44:0/4166962933 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7fe7f0039000 msgr2=0x7fe7f003b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:48.699 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:48.124+0000 7fe808b58700 1 --2- 172.21.15.44:0/4166962933 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7fe7f0039000 0x7fe7f003b4c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec006fd0 tx=0x7fe7ec006e40 comp rx=0 tx=0).stop 2024-04-21T22:58:48.699 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:48.124+0000 7fe808b58700 1 -- 172.21.15.44:0/4166962933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe80407dea0 msgr2=0x7fe80419dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:48.700 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:48.124+0000 7fe808b58700 1 --2- 172.21.15.44:0/4166962933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe80407dea0 0x7fe80419dd00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4004350 tx=0x7fe7f400bd50 comp rx=0 tx=0).stop 2024-04-21T22:58:48.700 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:48.124+0000 7fe808b58700 1 -- 172.21.15.44:0/4166962933 shutdown_connections 2024-04-21T22:58:48.700 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:48.124+0000 7fe808b58700 1 --2- 172.21.15.44:0/4166962933 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7fe7f0039000 0x7fe7f003b4c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:48.700 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:48.124+0000 7fe808b58700 1 --2- 172.21.15.44:0/4166962933 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe80407dea0 0x7fe80419dd00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:48.700 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:48.124+0000 7fe808b58700 1 -- 172.21.15.44:0/4166962933 >> 172.21.15.44:0/4166962933 conn(0x7fe804103100 msgr2=0x7fe804103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:48.700 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:48.124+0000 7fe808b58700 1 -- 172.21.15.44:0/4166962933 shutdown_connections 2024-04-21T22:58:48.700 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:48.124+0000 7fe808b58700 1 -- 172.21.15.44:0/4166962933 wait complete. 2024-04-21T22:58:48.700 INFO:teuthology.orchestra.run.smithi044.stdout:Enabling cephadm module... 2024-04-21T22:58:49.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:49 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/63322231' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-21T22:58:49.686 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:49 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: ignoring --setuser ceph since I am not root 2024-04-21T22:58:49.686 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:49 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: ignoring --setgroup ceph since I am not root 2024-04-21T22:58:49.686 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:49 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:49.582+0000 7f6394a27200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T22:58:50.073 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:49 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:49.773+0000 7f6394a27200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T22:58:50.073 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:49 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:49.879+0000 7f6394a27200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T22:58:50.081 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.085+0000 7f0206b8f700 1 Processor -- start 2024-04-21T22:58:50.081 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.085+0000 7f0206b8f700 1 -- start start 2024-04-21T22:58:50.081 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.085+0000 7f0206b8f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0200108630 0x7f020010aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:50.082 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.085+0000 7f0206b8f700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f020007be90 con 0x7f0200108630 2024-04-21T22:58:50.082 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.086+0000 7f020492b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0200108630 0x7f020010aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:50.082 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.086+0000 7f020492b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0200108630 0x7f020010aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43166/0 (socket says 172.21.15.44:43166) 2024-04-21T22:58:50.082 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.086+0000 7f020492b700 1 -- 172.21.15.44:0/2274811401 learned_addr learned my addr 172.21.15.44:0/2274811401 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:50.082 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.086+0000 7f020492b700 1 -- 172.21.15.44:0/2274811401 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f020010af60 con 0x7f0200108630 2024-04-21T22:58:50.082 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.086+0000 7f020492b700 1 --2- 172.21.15.44:0/2274811401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0200108630 0x7f020010aa20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f01f40098a0 tx=0x7f01f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1bdc75cdde2a955 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:50.082 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.086+0000 7f01ff7fe700 1 -- 172.21.15.44:0/2274811401 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01f4004160 con 0x7f0200108630 2024-04-21T22:58:50.082 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.086+0000 7f01ff7fe700 1 -- 172.21.15.44:0/2274811401 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f01f4004300 con 0x7f0200108630 2024-04-21T22:58:50.082 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.086+0000 7f01ff7fe700 1 -- 172.21.15.44:0/2274811401 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01f4011b50 con 0x7f0200108630 2024-04-21T22:58:50.082 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.087+0000 7f0206b8f700 1 -- 172.21.15.44:0/2274811401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0200108630 msgr2=0x7f020010aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:50.083 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.087+0000 7f0206b8f700 1 --2- 172.21.15.44:0/2274811401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0200108630 0x7f020010aa20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f01f40098a0 tx=0x7f01f4009bb0 comp rx=0 tx=0).stop 2024-04-21T22:58:50.083 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.087+0000 7f0206b8f700 1 -- 172.21.15.44:0/2274811401 shutdown_connections 2024-04-21T22:58:50.083 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.087+0000 7f0206b8f700 1 --2- 172.21.15.44:0/2274811401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0200108630 0x7f020010aa20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:50.083 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.087+0000 7f0206b8f700 1 -- 172.21.15.44:0/2274811401 >> 172.21.15.44:0/2274811401 conn(0x7f0200102000 msgr2=0x7f0200104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:50.083 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.087+0000 7f0206b8f700 1 -- 172.21.15.44:0/2274811401 shutdown_connections 2024-04-21T22:58:50.083 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.087+0000 7f0206b8f700 1 -- 172.21.15.44:0/2274811401 wait complete. 2024-04-21T22:58:50.083 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.088+0000 7f0206b8f700 1 Processor -- start 2024-04-21T22:58:50.083 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.088+0000 7f0206b8f700 1 -- start start 2024-04-21T22:58:50.083 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.088+0000 7f0206b8f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0200108630 0x7f020019bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:50.083 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.088+0000 7f0206b8f700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f020010bf80 con 0x7f0200108630 2024-04-21T22:58:50.083 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.088+0000 7f020492b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0200108630 0x7f020019bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:50.083 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.088+0000 7f020492b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0200108630 0x7f020019bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43178/0 (socket says 172.21.15.44:43178) 2024-04-21T22:58:50.083 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.088+0000 7f020492b700 1 -- 172.21.15.44:0/63322231 learned_addr learned my addr 172.21.15.44:0/63322231 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:50.083 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.088+0000 7f020492b700 1 -- 172.21.15.44:0/63322231 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f020019c070 con 0x7f0200108630 2024-04-21T22:58:50.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.088+0000 7f020492b700 1 --2- 172.21.15.44:0/63322231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0200108630 0x7f020019bb30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f01f4011b60 tx=0x7f01f4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:50.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.088+0000 7f01fdffb700 1 -- 172.21.15.44:0/63322231 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01f4003b20 con 0x7f0200108630 2024-04-21T22:58:50.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.089+0000 7f01fdffb700 1 -- 172.21.15.44:0/63322231 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f01f4003cc0 con 0x7f0200108630 2024-04-21T22:58:50.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.089+0000 7f0206b8f700 1 -- 172.21.15.44:0/63322231 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f020019c300 con 0x7f0200108630 2024-04-21T22:58:50.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.089+0000 7f01fdffb700 1 -- 172.21.15.44:0/63322231 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01f4024840 con 0x7f0200108630 2024-04-21T22:58:50.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.089+0000 7f0206b8f700 1 -- 172.21.15.44:0/63322231 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f020019eff0 con 0x7f0200108630 2024-04-21T22:58:50.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.090+0000 7f01fdffb700 1 -- 172.21.15.44:0/63322231 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f01f4022020 con 0x7f0200108630 2024-04-21T22:58:50.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.090+0000 7f0206b8f700 1 -- 172.21.15.44:0/63322231 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0200063890 con 0x7f0200108630 2024-04-21T22:58:50.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.090+0000 7f01fdffb700 1 --2- 172.21.15.44:0/63322231 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f01f0039050 0x7f01f003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:50.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.090+0000 7f01fdffb700 1 -- 172.21.15.44:0/63322231 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f01f4011e00 con 0x7f0200108630 2024-04-21T22:58:50.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.090+0000 7f01fffff700 1 --2- 172.21.15.44:0/63322231 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f01f0039050 0x7f01f003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:50.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.091+0000 7f01fffff700 1 --2- 172.21.15.44:0/63322231 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f01f0039050 0x7f01f003b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f01ec006fd0 tx=0x7f01ec006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:50.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.095+0000 7f01fdffb700 1 -- 172.21.15.44:0/63322231 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f01f400bde0 con 0x7f0200108630 2024-04-21T22:58:50.084 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.360+0000 7f0206b8f700 1 -- 172.21.15.44:0/63322231 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f020019f1c0 con 0x7f0200108630 2024-04-21T22:58:50.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.411+0000 7f01fdffb700 1 -- 172.21.15.44:0/63322231 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f020019f1c0 con 0x7f0200108630 2024-04-21T22:58:50.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.413+0000 7f01fdffb700 1 -- 172.21.15.44:0/63322231 <== mon.0 v2:172.21.15.44:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f01f402d770 con 0x7f0200108630 2024-04-21T22:58:50.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.414+0000 7f0206b8f700 1 -- 172.21.15.44:0/63322231 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f01f0039050 msgr2=0x7f01f003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:50.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.414+0000 7f0206b8f700 1 --2- 172.21.15.44:0/63322231 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f01f0039050 0x7f01f003b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f01ec006fd0 tx=0x7f01ec006e40 comp rx=0 tx=0).stop 2024-04-21T22:58:50.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.414+0000 7f0206b8f700 1 -- 172.21.15.44:0/63322231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0200108630 msgr2=0x7f020019bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:50.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.414+0000 7f0206b8f700 1 --2- 172.21.15.44:0/63322231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0200108630 0x7f020019bb30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f01f4011b60 tx=0x7f01f4011c40 comp rx=0 tx=0).stop 2024-04-21T22:58:50.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.414+0000 7f0206b8f700 1 -- 172.21.15.44:0/63322231 shutdown_connections 2024-04-21T22:58:50.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.414+0000 7f0206b8f700 1 --2- 172.21.15.44:0/63322231 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f01f0039050 0x7f01f003b510 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:50.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.414+0000 7f0206b8f700 1 --2- 172.21.15.44:0/63322231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0200108630 0x7f020019bb30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:50.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.414+0000 7f0206b8f700 1 -- 172.21.15.44:0/63322231 >> 172.21.15.44:0/63322231 conn(0x7f0200102000 msgr2=0x7f0200102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:50.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.414+0000 7f0206b8f700 1 -- 172.21.15.44:0/63322231 shutdown_connections 2024-04-21T22:58:50.085 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:49.414+0000 7f0206b8f700 1 -- 172.21.15.44:0/63322231 wait complete. 2024-04-21T22:58:50.334 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:50.205+0000 7f6394a27200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T22:58:50.334 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:50.293+0000 7f6394a27200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T22:58:50.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:50 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/63322231' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-21T22:58:50.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:50 smithi044 ceph-mon[27934]: mgrmap e5: x(active, since 12s) 2024-04-21T22:58:51.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:51 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1552235401' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-21T22:58:51.686 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:51 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:51.565+0000 7f6394a27200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T22:58:52.096 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:51 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:51.741+0000 7f6394a27200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T22:58:52.096 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:51 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:51.917+0000 7f6394a27200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T22:58:52.105 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-04-21T22:58:52.105 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-21T22:58:52.105 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T22:58:52.105 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-21T22:58:52.105 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-21T22:58:52.105 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-04-21T22:58:52.105 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.512+0000 7f2cf9de3700 1 Processor -- start 2024-04-21T22:58:52.105 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.512+0000 7f2cf9de3700 1 -- start start 2024-04-21T22:58:52.105 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.512+0000 7f2cf9de3700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cf4108630 0x7f2cf410aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:52.105 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.512+0000 7f2cf9de3700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf407be90 con 0x7f2cf4108630 2024-04-21T22:58:52.105 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.512+0000 7f2cf37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cf4108630 0x7f2cf410aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:52.106 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.512+0000 7f2cf37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cf4108630 0x7f2cf410aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43198/0 (socket says 172.21.15.44:43198) 2024-04-21T22:58:52.106 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.512+0000 7f2cf37fe700 1 -- 172.21.15.44:0/3799496844 learned_addr learned my addr 172.21.15.44:0/3799496844 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:52.106 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.512+0000 7f2cf37fe700 1 -- 172.21.15.44:0/3799496844 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf410af60 con 0x7f2cf4108630 2024-04-21T22:58:52.106 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.513+0000 7f2cf37fe700 1 --2- 172.21.15.44:0/3799496844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cf4108630 0x7f2cf410aa20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ce40098a0 tx=0x7f2ce4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f27ca88b624be5b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:52.106 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.513+0000 7f2cf27fc700 1 -- 172.21.15.44:0/3799496844 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ce4004160 con 0x7f2cf4108630 2024-04-21T22:58:52.106 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.513+0000 7f2cf27fc700 1 -- 172.21.15.44:0/3799496844 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2ce4004300 con 0x7f2cf4108630 2024-04-21T22:58:52.106 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.513+0000 7f2cf27fc700 1 -- 172.21.15.44:0/3799496844 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ce4011b50 con 0x7f2cf4108630 2024-04-21T22:58:52.106 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.513+0000 7f2cf9de3700 1 -- 172.21.15.44:0/3799496844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cf4108630 msgr2=0x7f2cf410aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:52.106 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.513+0000 7f2cf9de3700 1 --2- 172.21.15.44:0/3799496844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cf4108630 0x7f2cf410aa20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ce40098a0 tx=0x7f2ce4009bb0 comp rx=0 tx=0).stop 2024-04-21T22:58:52.106 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.513+0000 7f2cf9de3700 1 -- 172.21.15.44:0/3799496844 shutdown_connections 2024-04-21T22:58:52.106 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.513+0000 7f2cf9de3700 1 --2- 172.21.15.44:0/3799496844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cf4108630 0x7f2cf410aa20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:52.106 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.513+0000 7f2cf9de3700 1 -- 172.21.15.44:0/3799496844 >> 172.21.15.44:0/3799496844 conn(0x7f2cf4102000 msgr2=0x7f2cf4104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:52.106 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.514+0000 7f2cf9de3700 1 -- 172.21.15.44:0/3799496844 shutdown_connections 2024-04-21T22:58:52.106 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.514+0000 7f2cf9de3700 1 -- 172.21.15.44:0/3799496844 wait complete. 2024-04-21T22:58:52.107 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.514+0000 7f2cf9de3700 1 Processor -- start 2024-04-21T22:58:52.107 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.514+0000 7f2cf9de3700 1 -- start start 2024-04-21T22:58:52.107 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.515+0000 7f2cf9de3700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cf4108630 0x7f2cf419bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:52.107 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.515+0000 7f2cf9de3700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf410bf80 con 0x7f2cf4108630 2024-04-21T22:58:52.107 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.515+0000 7f2cf37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cf4108630 0x7f2cf419bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:52.107 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.515+0000 7f2cf37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cf4108630 0x7f2cf419bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43214/0 (socket says 172.21.15.44:43214) 2024-04-21T22:58:52.107 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.515+0000 7f2cf37fe700 1 -- 172.21.15.44:0/1552235401 learned_addr learned my addr 172.21.15.44:0/1552235401 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:52.107 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.515+0000 7f2cf37fe700 1 -- 172.21.15.44:0/1552235401 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf419c070 con 0x7f2cf4108630 2024-04-21T22:58:52.107 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.515+0000 7f2cf37fe700 1 --2- 172.21.15.44:0/1552235401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cf4108630 0x7f2cf419bb30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4011670 tx=0x7f2ce4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:52.107 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.515+0000 7f2cf0ff9700 1 -- 172.21.15.44:0/1552235401 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ce40039c0 con 0x7f2cf4108630 2024-04-21T22:58:52.107 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.515+0000 7f2cf0ff9700 1 -- 172.21.15.44:0/1552235401 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2ce4011e50 con 0x7f2cf4108630 2024-04-21T22:58:52.107 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.515+0000 7f2cf9de3700 1 -- 172.21.15.44:0/1552235401 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf419c300 con 0x7f2cf4108630 2024-04-21T22:58:52.107 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.515+0000 7f2cf9de3700 1 -- 172.21.15.44:0/1552235401 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf419eff0 con 0x7f2cf4108630 2024-04-21T22:58:52.108 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.516+0000 7f2cf0ff9700 1 -- 172.21.15.44:0/1552235401 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ce401a5c0 con 0x7f2cf4108630 2024-04-21T22:58:52.108 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.517+0000 7f2cf0ff9700 1 -- 172.21.15.44:0/1552235401 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f2ce401a8c0 con 0x7f2cf4108630 2024-04-21T22:58:52.108 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.517+0000 7f2cf0ff9700 1 --2- 172.21.15.44:0/1552235401 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f2cdc039000 0x7f2cdc03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:52.108 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.517+0000 7f2cf0ff9700 1 -- 172.21.15.44:0/1552235401 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2ce404d2d0 con 0x7f2cf4108630 2024-04-21T22:58:52.108 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.517+0000 7f2cf2ffd700 1 -- 172.21.15.44:0/1552235401 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f2cdc039000 msgr2=0x7f2cdc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2900331072 2024-04-21T22:58:52.108 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.517+0000 7f2cf2ffd700 1 --2- 172.21.15.44:0/1552235401 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f2cdc039000 0x7f2cdc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T22:58:52.108 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.517+0000 7f2cf9de3700 1 -- 172.21.15.44:0/1552235401 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cf4063890 con 0x7f2cf4108630 2024-04-21T22:58:52.108 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.522+0000 7f2cf0ff9700 1 -- 172.21.15.44:0/1552235401 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2ce4019db0 con 0x7f2cf4108630 2024-04-21T22:58:52.108 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.718+0000 7f2cf2ffd700 1 -- 172.21.15.44:0/1552235401 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f2cdc039000 msgr2=0x7f2cdc03b4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2900331072 2024-04-21T22:58:52.108 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.718+0000 7f2cf2ffd700 1 --2- 172.21.15.44:0/1552235401 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f2cdc039000 0x7f2cdc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T22:58:52.108 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.836+0000 7f2cf9de3700 1 -- 172.21.15.44:0/1552235401 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f2cf419f1c0 con 0x7f2cf4108630 2024-04-21T22:58:52.108 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.836+0000 7f2cf0ff9700 1 -- 172.21.15.44:0/1552235401 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f2cf419f1c0 con 0x7f2cf4108630 2024-04-21T22:58:52.108 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.838+0000 7f2cf9de3700 1 -- 172.21.15.44:0/1552235401 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f2cdc039000 msgr2=0x7f2cdc03b4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T22:58:52.108 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.838+0000 7f2cf9de3700 1 --2- 172.21.15.44:0/1552235401 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f2cdc039000 0x7f2cdc03b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:52.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.838+0000 7f2cf9de3700 1 -- 172.21.15.44:0/1552235401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cf4108630 msgr2=0x7f2cf419bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:52.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.838+0000 7f2cf9de3700 1 --2- 172.21.15.44:0/1552235401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cf4108630 0x7f2cf419bb30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4011670 tx=0x7f2ce4011b60 comp rx=0 tx=0).stop 2024-04-21T22:58:52.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.838+0000 7f2cf9de3700 1 -- 172.21.15.44:0/1552235401 shutdown_connections 2024-04-21T22:58:52.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.838+0000 7f2cf9de3700 1 --2- 172.21.15.44:0/1552235401 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f2cdc039000 0x7f2cdc03b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:52.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.838+0000 7f2cf9de3700 1 --2- 172.21.15.44:0/1552235401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2cf4108630 0x7f2cf419bb30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:52.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.838+0000 7f2cf9de3700 1 -- 172.21.15.44:0/1552235401 >> 172.21.15.44:0/1552235401 conn(0x7f2cf4102000 msgr2=0x7f2cf4102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:52.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.839+0000 7f2cf9de3700 1 -- 172.21.15.44:0/1552235401 shutdown_connections 2024-04-21T22:58:52.109 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:50.839+0000 7f2cf9de3700 1 -- 172.21.15.44:0/1552235401 wait complete. 2024-04-21T22:58:52.109 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for the mgr to restart... 2024-04-21T22:58:52.109 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mgr epoch 5... 2024-04-21T22:58:52.347 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:52 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:52.218+0000 7f6394a27200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T22:58:52.347 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:52 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:52.315+0000 7f6394a27200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T22:58:52.685 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:52 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:52.532+0000 7f6394a27200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T22:58:53.135 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:52 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:52.731+0000 7f6394a27200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T22:58:53.435 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:53 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:53.358+0000 7f6394a27200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T22:58:54.104 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:53 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:53.797+0000 7f6394a27200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T22:58:54.104 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:53 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:53.896+0000 7f6394a27200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T22:58:54.104 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:54 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:54.103+0000 7f6394a27200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T22:58:54.935 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:54 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:54.645+0000 7f6394a27200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T22:58:54.936 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:54 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:54.740+0000 7f6394a27200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T22:58:55.350 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:54 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:54.993+0000 7f6394a27200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T22:58:55.350 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:55 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:55.133+0000 7f6394a27200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T22:58:55.350 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:55 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:55.231+0000 7f6394a27200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T22:58:55.351 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:55 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:55.350+0000 7f6394a27200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T22:58:55.685 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:55 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:55.444+0000 7f6394a27200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T22:58:56.561 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:56 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:56.284+0000 7f6394a27200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T22:58:56.562 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:58:56 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:58:56.508+0000 7f6394a27200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T22:58:56.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:56 smithi044 ceph-mon[27934]: Active manager daemon x restarted 2024-04-21T22:58:56.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:56 smithi044 ceph-mon[27934]: Activating manager daemon x 2024-04-21T22:58:56.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:56 smithi044 ceph-mon[27934]: osdmap e2: 0 total, 0 up, 0 in 2024-04-21T22:58:56.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:56 smithi044 ceph-mon[27934]: mgrmap e6: x(active, starting, since 0.0033483s) 2024-04-21T22:58:56.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:56 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T22:58:56.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:56 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T22:58:56.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:56 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T22:58:56.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:56 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T22:58:56.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:56 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T22:58:56.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:56 smithi044 ceph-mon[27934]: Manager daemon x is now available 2024-04-21T22:58:56.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:56 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' 2024-04-21T22:58:56.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:56 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' 2024-04-21T22:58:56.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:56 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T22:58:56.937 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:56 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T22:58:56.937 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:56 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T22:58:57.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:57 smithi044 ceph-mon[27934]: Found migration_current of "None". Setting to last migration. 2024-04-21T22:58:57.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:57 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T22:58:57.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:57 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' 2024-04-21T22:58:57.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:57 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' 2024-04-21T22:58:57.836 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:57 smithi044 ceph-mon[27934]: mgrmap e7: x(active, since 1.00559s) 2024-04-21T22:58:58.114 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-04-21T22:58:58.114 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-21T22:58:58.114 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-21T22:58:58.114 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-04-21T22:58:58.114 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.528+0000 7f9de5cfd700 1 Processor -- start 2024-04-21T22:58:58.114 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.528+0000 7f9de5cfd700 1 -- start start 2024-04-21T22:58:58.114 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.529+0000 7f9de5cfd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9de0107c10 0x7f9de0108030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:58.114 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.529+0000 7f9de5cfd700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9de0108600 con 0x7f9de0107c10 2024-04-21T22:58:58.114 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.529+0000 7f9ddf7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9de0107c10 0x7f9de0108030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:58.114 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.529+0000 7f9ddf7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9de0107c10 0x7f9de0108030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33168/0 (socket says 172.21.15.44:33168) 2024-04-21T22:58:58.115 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.529+0000 7f9ddf7fe700 1 -- 172.21.15.44:0/2567709826 learned_addr learned my addr 172.21.15.44:0/2567709826 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:58.115 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.529+0000 7f9ddf7fe700 1 -- 172.21.15.44:0/2567709826 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de0108dc0 con 0x7f9de0107c10 2024-04-21T22:58:58.115 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.529+0000 7f9ddf7fe700 1 --2- 172.21.15.44:0/2567709826 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9de0107c10 0x7f9de0108030 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9dd00098a0 tx=0x7f9dd0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39f67491dee7b78 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:58.115 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.529+0000 7f9dde7fc700 1 -- 172.21.15.44:0/2567709826 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9dd0004160 con 0x7f9de0107c10 2024-04-21T22:58:58.115 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.529+0000 7f9dde7fc700 1 -- 172.21.15.44:0/2567709826 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9dd0004300 con 0x7f9de0107c10 2024-04-21T22:58:58.115 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.529+0000 7f9dde7fc700 1 -- 172.21.15.44:0/2567709826 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9dd0011b50 con 0x7f9de0107c10 2024-04-21T22:58:58.115 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.530+0000 7f9de5cfd700 1 -- 172.21.15.44:0/2567709826 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9de0107c10 msgr2=0x7f9de0108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:58.115 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.530+0000 7f9de5cfd700 1 --2- 172.21.15.44:0/2567709826 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9de0107c10 0x7f9de0108030 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9dd00098a0 tx=0x7f9dd0009bb0 comp rx=0 tx=0).stop 2024-04-21T22:58:58.115 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.530+0000 7f9de5cfd700 1 -- 172.21.15.44:0/2567709826 shutdown_connections 2024-04-21T22:58:58.115 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.530+0000 7f9de5cfd700 1 --2- 172.21.15.44:0/2567709826 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9de0107c10 0x7f9de0108030 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:58.115 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.530+0000 7f9de5cfd700 1 -- 172.21.15.44:0/2567709826 >> 172.21.15.44:0/2567709826 conn(0x7f9de0103190 msgr2=0x7f9de01055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:58.115 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.530+0000 7f9de5cfd700 1 -- 172.21.15.44:0/2567709826 shutdown_connections 2024-04-21T22:58:58.115 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.530+0000 7f9de5cfd700 1 -- 172.21.15.44:0/2567709826 wait complete. 2024-04-21T22:58:58.115 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.531+0000 7f9de5cfd700 1 Processor -- start 2024-04-21T22:58:58.116 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.531+0000 7f9de5cfd700 1 -- start start 2024-04-21T22:58:58.116 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.531+0000 7f9de5cfd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9de0107c10 0x7f9de007db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:58.116 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.531+0000 7f9de5cfd700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9de010c170 con 0x7f9de0107c10 2024-04-21T22:58:58.116 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.531+0000 7f9ddf7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9de0107c10 0x7f9de007db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:58.116 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.531+0000 7f9ddf7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9de0107c10 0x7f9de007db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33182/0 (socket says 172.21.15.44:33182) 2024-04-21T22:58:58.116 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.531+0000 7f9ddf7fe700 1 -- 172.21.15.44:0/2338277813 learned_addr learned my addr 172.21.15.44:0/2338277813 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:58.116 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.532+0000 7f9ddf7fe700 1 -- 172.21.15.44:0/2338277813 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de007e0d0 con 0x7f9de0107c10 2024-04-21T22:58:58.116 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.532+0000 7f9ddf7fe700 1 --2- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9de0107c10 0x7f9de007db90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0011850 tx=0x7f9dd0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:58.116 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.532+0000 7f9ddcff9700 1 -- 172.21.15.44:0/2338277813 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9dd0003af0 con 0x7f9de0107c10 2024-04-21T22:58:58.116 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.532+0000 7f9ddcff9700 1 -- 172.21.15.44:0/2338277813 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9dd0020070 con 0x7f9de0107c10 2024-04-21T22:58:58.116 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.532+0000 7f9de5cfd700 1 -- 172.21.15.44:0/2338277813 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9de007c2c0 con 0x7f9de0107c10 2024-04-21T22:58:58.116 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.532+0000 7f9ddcff9700 1 -- 172.21.15.44:0/2338277813 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9dd00195a0 con 0x7f9de0107c10 2024-04-21T22:58:58.116 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.532+0000 7f9de5cfd700 1 -- 172.21.15.44:0/2338277813 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9de007c800 con 0x7f9de0107c10 2024-04-21T22:58:58.116 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.533+0000 7f9ddcff9700 1 -- 172.21.15.44:0/2338277813 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f9dd0019850 con 0x7f9de0107c10 2024-04-21T22:58:58.117 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.533+0000 7f9ddcff9700 1 --2- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f9dc80390a0 0x7f9dc803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:58.117 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.533+0000 7f9ddcff9700 1 -- 172.21.15.44:0/2338277813 --> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9dc803bc70 con 0x7f9dc80390a0 2024-04-21T22:58:58.117 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.533+0000 7f9ddcff9700 1 -- 172.21.15.44:0/2338277813 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9dd0011e00 con 0x7f9de0107c10 2024-04-21T22:58:58.117 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.533+0000 7f9ddeffd700 1 -- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f9dc80390a0 msgr2=0x7f9dc803b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2900331072 2024-04-21T22:58:58.117 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.533+0000 7f9ddeffd700 1 --2- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f9dc80390a0 0x7f9dc803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T22:58:58.117 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.734+0000 7f9ddeffd700 1 -- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f9dc80390a0 msgr2=0x7f9dc803b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2900331072 2024-04-21T22:58:58.117 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:52.734+0000 7f9ddeffd700 1 --2- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f9dc80390a0 0x7f9dc803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T22:58:58.117 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:53.135+0000 7f9ddeffd700 1 -- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f9dc80390a0 msgr2=0x7f9dc803b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2900331072 2024-04-21T22:58:58.117 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:53.135+0000 7f9ddeffd700 1 --2- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f9dc80390a0 0x7f9dc803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-21T22:58:58.117 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:53.936+0000 7f9ddeffd700 1 -- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f9dc80390a0 msgr2=0x7f9dc803b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2900331072 2024-04-21T22:58:58.117 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:53.936+0000 7f9ddeffd700 1 --2- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f9dc80390a0 0x7f9dc803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-21T22:58:58.117 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:55.537+0000 7f9ddeffd700 1 -- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f9dc80390a0 msgr2=0x7f9dc803b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/2900331072 2024-04-21T22:58:58.117 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:55.537+0000 7f9ddeffd700 1 --2- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f9dc80390a0 0x7f9dc803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-21T22:58:58.118 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:56.512+0000 7f9ddcff9700 1 -- 172.21.15.44:0/2338277813 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7f9dd002bd00 con 0x7f9de0107c10 2024-04-21T22:58:58.118 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:56.512+0000 7f9ddcff9700 1 -- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f9dc80390a0 msgr2=0x7f9dc803b560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T22:58:58.118 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:56.512+0000 7f9ddcff9700 1 --2- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f9dc80390a0 0x7f9dc803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:58.118 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:57.513+0000 7f9ddcff9700 1 -- 172.21.15.44:0/2338277813 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f9dd004e9c0 con 0x7f9de0107c10 2024-04-21T22:58:58.118 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:57.513+0000 7f9ddcff9700 1 --2- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f9dc803c9d0 0x7f9dc803edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:58.118 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:57.513+0000 7f9ddcff9700 1 -- 172.21.15.44:0/2338277813 --> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9dc803bc70 con 0x7f9dc803c9d0 2024-04-21T22:58:58.118 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:57.513+0000 7f9ddeffd700 1 --2- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f9dc803c9d0 0x7f9dc803edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:58.118 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:57.514+0000 7f9ddeffd700 1 --2- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f9dc803c9d0 0x7f9dc803edc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4003a10 tx=0x7f9dd40092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:58.118 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:57.514+0000 7f9ddcff9700 1 -- 172.21.15.44:0/2338277813 <== mgr.14120 v2:172.21.15.44:6800/4238737803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f9dc803bc70 con 0x7f9dc803c9d0 2024-04-21T22:58:58.118 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:57.526+0000 7f9de5cfd700 1 -- 172.21.15.44:0/2338277813 --> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f9de0063890 con 0x7f9dc803c9d0 2024-04-21T22:58:58.118 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:57.526+0000 7f9ddcff9700 1 -- 172.21.15.44:0/2338277813 <== mgr.14120 v2:172.21.15.44:6800/4238737803 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f9de0063890 con 0x7f9dc803c9d0 2024-04-21T22:58:58.118 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:57.526+0000 7f9de5cfd700 1 -- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f9dc803c9d0 msgr2=0x7f9dc803edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:58.118 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:57.526+0000 7f9de5cfd700 1 --2- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f9dc803c9d0 0x7f9dc803edc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4003a10 tx=0x7f9dd40092b0 comp rx=0 tx=0).stop 2024-04-21T22:58:58.119 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:57.526+0000 7f9de5cfd700 1 -- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9de0107c10 msgr2=0x7f9de007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:58.119 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:57.526+0000 7f9de5cfd700 1 --2- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9de0107c10 0x7f9de007db90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0011850 tx=0x7f9dd0011c10 comp rx=0 tx=0).stop 2024-04-21T22:58:58.119 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:57.526+0000 7f9de5cfd700 1 -- 172.21.15.44:0/2338277813 shutdown_connections 2024-04-21T22:58:58.119 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:57.526+0000 7f9de5cfd700 1 --2- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9de0107c10 0x7f9de007db90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:58.119 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:57.526+0000 7f9de5cfd700 1 --2- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f9dc803c9d0 0x7f9dc803edc0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:58.119 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:57.526+0000 7f9de5cfd700 1 --2- 172.21.15.44:0/2338277813 >> [v2:172.21.15.44:6800/2900331072,v1:172.21.15.44:6801/2900331072] conn(0x7f9dc80390a0 0x7f9dc803b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:58.119 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:57.526+0000 7f9de5cfd700 1 -- 172.21.15.44:0/2338277813 >> 172.21.15.44:0/2338277813 conn(0x7f9de0103190 msgr2=0x7f9de0103e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:58.119 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:57.527+0000 7f9de5cfd700 1 -- 172.21.15.44:0/2338277813 shutdown_connections 2024-04-21T22:58:58.119 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:57.527+0000 7f9de5cfd700 1 -- 172.21.15.44:0/2338277813 wait complete. 2024-04-21T22:58:58.119 INFO:teuthology.orchestra.run.smithi044.stdout:mgr epoch 5 is available 2024-04-21T22:58:58.119 INFO:teuthology.orchestra.run.smithi044.stdout:Setting orchestrator backend to cephadm... 2024-04-21T22:58:58.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:58 smithi044 ceph-mon[27934]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-21T22:58:58.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:58 smithi044 ceph-mon[27934]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-21T22:58:58.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:58 smithi044 ceph-mon[27934]: [21/Apr/2024:22:58:57] ENGINE Bus STARTING 2024-04-21T22:58:58.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:58 smithi044 ceph-mon[27934]: [21/Apr/2024:22:58:57] ENGINE Serving on https://172.21.15.44:7150 2024-04-21T22:58:58.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:58 smithi044 ceph-mon[27934]: [21/Apr/2024:22:58:57] ENGINE Serving on http://172.21.15.44:8765 2024-04-21T22:58:58.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:58 smithi044 ceph-mon[27934]: [21/Apr/2024:22:58:57] ENGINE Bus STARTED 2024-04-21T22:58:58.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:58 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T22:58:59.409 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.544+0000 7f65ca8d0700 1 Processor -- start 2024-04-21T22:58:59.409 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.544+0000 7f65ca8d0700 1 -- start start 2024-04-21T22:58:59.409 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.544+0000 7f65ca8d0700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65c4106fa0 0x7f65c41073c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:59.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.544+0000 7f65ca8d0700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f65c4107990 con 0x7f65c4106fa0 2024-04-21T22:58:59.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.544+0000 7f65c3fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65c4106fa0 0x7f65c41073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:59.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.545+0000 7f65c3fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65c4106fa0 0x7f65c41073c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33254/0 (socket says 172.21.15.44:33254) 2024-04-21T22:58:59.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.545+0000 7f65c3fff700 1 -- 172.21.15.44:0/2991798673 learned_addr learned my addr 172.21.15.44:0/2991798673 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:59.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.545+0000 7f65c3fff700 1 -- 172.21.15.44:0/2991798673 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c4107cc0 con 0x7f65c4106fa0 2024-04-21T22:58:59.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.545+0000 7f65c3fff700 1 --2- 172.21.15.44:0/2991798673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65c4106fa0 0x7f65c41073c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f65b40098a0 tx=0x7f65b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce9bc5186ee5e700 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:59.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.545+0000 7f65c37fe700 1 -- 172.21.15.44:0/2991798673 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65b4004160 con 0x7f65c4106fa0 2024-04-21T22:58:59.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.545+0000 7f65c37fe700 1 -- 172.21.15.44:0/2991798673 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f65b4004300 con 0x7f65c4106fa0 2024-04-21T22:58:59.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.545+0000 7f65c37fe700 1 -- 172.21.15.44:0/2991798673 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65b4011b50 con 0x7f65c4106fa0 2024-04-21T22:58:59.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.546+0000 7f65ca8d0700 1 -- 172.21.15.44:0/2991798673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65c4106fa0 msgr2=0x7f65c41073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:59.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.546+0000 7f65ca8d0700 1 --2- 172.21.15.44:0/2991798673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65c4106fa0 0x7f65c41073c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f65b40098a0 tx=0x7f65b4009bb0 comp rx=0 tx=0).stop 2024-04-21T22:58:59.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.546+0000 7f65ca8d0700 1 -- 172.21.15.44:0/2991798673 shutdown_connections 2024-04-21T22:58:59.410 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.546+0000 7f65ca8d0700 1 --2- 172.21.15.44:0/2991798673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65c4106fa0 0x7f65c41073c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:59.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.546+0000 7f65ca8d0700 1 -- 172.21.15.44:0/2991798673 >> 172.21.15.44:0/2991798673 conn(0x7f65c4101820 msgr2=0x7f65c4103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:59.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.546+0000 7f65ca8d0700 1 -- 172.21.15.44:0/2991798673 shutdown_connections 2024-04-21T22:58:59.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.546+0000 7f65ca8d0700 1 -- 172.21.15.44:0/2991798673 wait complete. 2024-04-21T22:58:59.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.546+0000 7f65ca8d0700 1 Processor -- start 2024-04-21T22:58:59.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.547+0000 7f65ca8d0700 1 -- start start 2024-04-21T22:58:59.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.547+0000 7f65ca8d0700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65c4106fa0 0x7f65c419aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:59.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.547+0000 7f65ca8d0700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f65c410b070 con 0x7f65c4106fa0 2024-04-21T22:58:59.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.547+0000 7f65c3fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65c4106fa0 0x7f65c419aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:59.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.547+0000 7f65c3fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65c4106fa0 0x7f65c419aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33262/0 (socket says 172.21.15.44:33262) 2024-04-21T22:58:59.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.547+0000 7f65c3fff700 1 -- 172.21.15.44:0/2646594440 learned_addr learned my addr 172.21.15.44:0/2646594440 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:58:59.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.547+0000 7f65c3fff700 1 -- 172.21.15.44:0/2646594440 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c419afb0 con 0x7f65c4106fa0 2024-04-21T22:58:59.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.547+0000 7f65c3fff700 1 --2- 172.21.15.44:0/2646594440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65c4106fa0 0x7f65c419aa70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f65b4011b60 tx=0x7f65b4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:59.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.547+0000 7f65c1ffb700 1 -- 172.21.15.44:0/2646594440 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65b4003970 con 0x7f65c4106fa0 2024-04-21T22:58:59.411 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.547+0000 7f65c1ffb700 1 -- 172.21.15.44:0/2646594440 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f65b4003b10 con 0x7f65c4106fa0 2024-04-21T22:58:59.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.547+0000 7f65ca8d0700 1 -- 172.21.15.44:0/2646594440 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65c419b240 con 0x7f65c4106fa0 2024-04-21T22:58:59.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.548+0000 7f65c1ffb700 1 -- 172.21.15.44:0/2646594440 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65b4024610 con 0x7f65c4106fa0 2024-04-21T22:58:59.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.548+0000 7f65ca8d0700 1 -- 172.21.15.44:0/2646594440 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65c419df30 con 0x7f65c4106fa0 2024-04-21T22:58:59.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.548+0000 7f65c1ffb700 1 -- 172.21.15.44:0/2646594440 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f65b40248c0 con 0x7f65c4106fa0 2024-04-21T22:58:59.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.548+0000 7f65ca8d0700 1 -- 172.21.15.44:0/2646594440 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65c4063890 con 0x7f65c4106fa0 2024-04-21T22:58:59.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.549+0000 7f65c1ffb700 1 --2- 172.21.15.44:0/2646594440 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f65b0038f80 0x7f65b003b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:58:59.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.549+0000 7f65c1ffb700 1 -- 172.21.15.44:0/2646594440 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f65b4011e00 con 0x7f65c4106fa0 2024-04-21T22:58:59.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.549+0000 7f65bbfff700 1 --2- 172.21.15.44:0/2646594440 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f65b0038f80 0x7f65b003b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:58:59.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.549+0000 7f65bbfff700 1 --2- 172.21.15.44:0/2646594440 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f65b0038f80 0x7f65b003b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f65ac006fd0 tx=0x7f65ac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:58:59.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.554+0000 7f65c1ffb700 1 -- 172.21.15.44:0/2646594440 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f65b402d910 con 0x7f65c4106fa0 2024-04-21T22:58:59.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.564+0000 7f65c1ffb700 1 -- 172.21.15.44:0/2646594440 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f65b402d3f0 con 0x7f65c4106fa0 2024-04-21T22:58:59.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.777+0000 7f65ca8d0700 1 -- 172.21.15.44:0/2646594440 --> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f65c4102af0 con 0x7f65b0038f80 2024-04-21T22:58:59.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.784+0000 7f65c1ffb700 1 -- 172.21.15.44:0/2646594440 <== mgr.14120 v2:172.21.15.44:6800/4238737803 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f65c4102af0 con 0x7f65b0038f80 2024-04-21T22:58:59.412 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.786+0000 7f65ca8d0700 1 -- 172.21.15.44:0/2646594440 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f65b0038f80 msgr2=0x7f65b003b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:59.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.786+0000 7f65ca8d0700 1 --2- 172.21.15.44:0/2646594440 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f65b0038f80 0x7f65b003b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f65ac006fd0 tx=0x7f65ac006e40 comp rx=0 tx=0).stop 2024-04-21T22:58:59.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.786+0000 7f65ca8d0700 1 -- 172.21.15.44:0/2646594440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65c4106fa0 msgr2=0x7f65c419aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:58:59.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.786+0000 7f65ca8d0700 1 --2- 172.21.15.44:0/2646594440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65c4106fa0 0x7f65c419aa70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f65b4011b60 tx=0x7f65b4011c40 comp rx=0 tx=0).stop 2024-04-21T22:58:59.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.786+0000 7f65ca8d0700 1 -- 172.21.15.44:0/2646594440 shutdown_connections 2024-04-21T22:58:59.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.786+0000 7f65ca8d0700 1 --2- 172.21.15.44:0/2646594440 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f65b0038f80 0x7f65b003b440 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:59.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.786+0000 7f65ca8d0700 1 --2- 172.21.15.44:0/2646594440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f65c4106fa0 0x7f65c419aa70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:58:59.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.786+0000 7f65ca8d0700 1 -- 172.21.15.44:0/2646594440 >> 172.21.15.44:0/2646594440 conn(0x7f65c4101820 msgr2=0x7f65c41023e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:58:59.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.787+0000 7f65ca8d0700 1 -- 172.21.15.44:0/2646594440 shutdown_connections 2024-04-21T22:58:59.413 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:58.787+0000 7f65ca8d0700 1 -- 172.21.15.44:0/2646594440 wait complete. 2024-04-21T22:58:59.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:59 smithi044 ceph-mon[27934]: mgrmap e8: x(active, since 2s) 2024-04-21T22:58:59.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:59 smithi044 ceph-mon[27934]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T22:58:59.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:59 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' 2024-04-21T22:58:59.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:58:59 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T22:59:00.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-21T22:59:00.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.835+0000 7fac2ee59700 1 Processor -- start 2024-04-21T22:59:00.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.835+0000 7fac2ee59700 1 -- start start 2024-04-21T22:59:00.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.835+0000 7fac2ee59700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac28108650 0x7fac2810aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:00.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.835+0000 7fac2ee59700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fac2807bed0 con 0x7fac28108650 2024-04-21T22:59:00.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.835+0000 7fac2cbf5700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac28108650 0x7fac2810aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:00.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.835+0000 7fac2cbf5700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac28108650 0x7fac2810aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33272/0 (socket says 172.21.15.44:33272) 2024-04-21T22:59:00.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.835+0000 7fac2cbf5700 1 -- 172.21.15.44:0/2314000654 learned_addr learned my addr 172.21.15.44:0/2314000654 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:00.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.836+0000 7fac2cbf5700 1 -- 172.21.15.44:0/2314000654 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2810af80 con 0x7fac28108650 2024-04-21T22:59:00.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.836+0000 7fac2cbf5700 1 --2- 172.21.15.44:0/2314000654 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac28108650 0x7fac2810aa40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0098a0 tx=0x7fac1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff38c57d537c26b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:00.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.836+0000 7fac277fe700 1 -- 172.21.15.44:0/2314000654 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac1c004160 con 0x7fac28108650 2024-04-21T22:59:00.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.836+0000 7fac277fe700 1 -- 172.21.15.44:0/2314000654 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fac1c004300 con 0x7fac28108650 2024-04-21T22:59:00.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.836+0000 7fac277fe700 1 -- 172.21.15.44:0/2314000654 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac1c011b50 con 0x7fac28108650 2024-04-21T22:59:00.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.837+0000 7fac2ee59700 1 -- 172.21.15.44:0/2314000654 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac28108650 msgr2=0x7fac2810aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:00.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.837+0000 7fac2ee59700 1 --2- 172.21.15.44:0/2314000654 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac28108650 0x7fac2810aa40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0098a0 tx=0x7fac1c009bb0 comp rx=0 tx=0).stop 2024-04-21T22:59:00.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.837+0000 7fac2ee59700 1 -- 172.21.15.44:0/2314000654 shutdown_connections 2024-04-21T22:59:00.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.837+0000 7fac2ee59700 1 --2- 172.21.15.44:0/2314000654 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac28108650 0x7fac2810aa40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:00.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.837+0000 7fac2ee59700 1 -- 172.21.15.44:0/2314000654 >> 172.21.15.44:0/2314000654 conn(0x7fac281020a0 msgr2=0x7fac281044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:00.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.837+0000 7fac2ee59700 1 -- 172.21.15.44:0/2314000654 shutdown_connections 2024-04-21T22:59:00.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.837+0000 7fac2ee59700 1 -- 172.21.15.44:0/2314000654 wait complete. 2024-04-21T22:59:00.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.837+0000 7fac2ee59700 1 Processor -- start 2024-04-21T22:59:00.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.837+0000 7fac2ee59700 1 -- start start 2024-04-21T22:59:00.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.837+0000 7fac2ee59700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac28108650 0x7fac28107ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:00.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.838+0000 7fac2ee59700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fac2810bfb0 con 0x7fac28108650 2024-04-21T22:59:00.569 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.838+0000 7fac2cbf5700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac28108650 0x7fac28107ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:00.569 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.838+0000 7fac2cbf5700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac28108650 0x7fac28107ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:33282/0 (socket says 172.21.15.44:33282) 2024-04-21T22:59:00.569 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.838+0000 7fac2cbf5700 1 -- 172.21.15.44:0/3035530542 learned_addr learned my addr 172.21.15.44:0/3035530542 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:00.569 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.838+0000 7fac2cbf5700 1 -- 172.21.15.44:0/3035530542 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac28108410 con 0x7fac28108650 2024-04-21T22:59:00.569 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.838+0000 7fac2cbf5700 1 --2- 172.21.15.44:0/3035530542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac28108650 0x7fac28107ed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fac1c009fb0 tx=0x7fac1c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:00.569 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.838+0000 7fac25ffb700 1 -- 172.21.15.44:0/3035530542 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac1c0038b0 con 0x7fac28108650 2024-04-21T22:59:00.569 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.838+0000 7fac25ffb700 1 -- 172.21.15.44:0/3035530542 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fac1c011e50 con 0x7fac28108650 2024-04-21T22:59:00.569 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.838+0000 7fac2ee59700 1 -- 172.21.15.44:0/3035530542 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac28106600 con 0x7fac28108650 2024-04-21T22:59:00.569 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.838+0000 7fac25ffb700 1 -- 172.21.15.44:0/3035530542 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac1c019420 con 0x7fac28108650 2024-04-21T22:59:00.569 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.838+0000 7fac2ee59700 1 -- 172.21.15.44:0/3035530542 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac28106b40 con 0x7fac28108650 2024-04-21T22:59:00.569 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.839+0000 7fac25ffb700 1 -- 172.21.15.44:0/3035530542 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fac1c019720 con 0x7fac28108650 2024-04-21T22:59:00.569 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.839+0000 7fac2ee59700 1 -- 172.21.15.44:0/3035530542 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac28063890 con 0x7fac28108650 2024-04-21T22:59:00.569 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.840+0000 7fac25ffb700 1 --2- 172.21.15.44:0/3035530542 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fac18039000 0x7fac1803b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:00.570 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.840+0000 7fac25ffb700 1 -- 172.21.15.44:0/3035530542 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fac1c04d4a0 con 0x7fac28108650 2024-04-21T22:59:00.570 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.840+0000 7fac27fff700 1 --2- 172.21.15.44:0/3035530542 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fac18039000 0x7fac1803b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:00.570 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.840+0000 7fac27fff700 1 --2- 172.21.15.44:0/3035530542 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fac18039000 0x7fac1803b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fac14006fd0 tx=0x7fac14006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:00.570 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:58:59.845+0000 7fac25ffb700 1 -- 172.21.15.44:0/3035530542 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fac1c00bc70 con 0x7fac28108650 2024-04-21T22:59:00.570 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.051+0000 7fac2ee59700 1 -- 172.21.15.44:0/3035530542 --> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fac28103310 con 0x7fac18039000 2024-04-21T22:59:00.570 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.052+0000 7fac25ffb700 1 -- 172.21.15.44:0/3035530542 <== mgr.14120 v2:172.21.15.44:6800/4238737803 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fac28103310 con 0x7fac18039000 2024-04-21T22:59:00.570 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.054+0000 7fac2ee59700 1 -- 172.21.15.44:0/3035530542 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fac18039000 msgr2=0x7fac1803b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:00.570 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.054+0000 7fac2ee59700 1 --2- 172.21.15.44:0/3035530542 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fac18039000 0x7fac1803b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fac14006fd0 tx=0x7fac14006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:00.570 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.054+0000 7fac2ee59700 1 -- 172.21.15.44:0/3035530542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac28108650 msgr2=0x7fac28107ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:00.570 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.054+0000 7fac2ee59700 1 --2- 172.21.15.44:0/3035530542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac28108650 0x7fac28107ed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fac1c009fb0 tx=0x7fac1c011bf0 comp rx=0 tx=0).stop 2024-04-21T22:59:00.570 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.054+0000 7fac2ee59700 1 -- 172.21.15.44:0/3035530542 shutdown_connections 2024-04-21T22:59:00.570 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.054+0000 7fac2ee59700 1 --2- 172.21.15.44:0/3035530542 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fac18039000 0x7fac1803b4c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:00.570 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.054+0000 7fac2ee59700 1 --2- 172.21.15.44:0/3035530542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac28108650 0x7fac28107ed0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:00.571 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.054+0000 7fac2ee59700 1 -- 172.21.15.44:0/3035530542 >> 172.21.15.44:0/3035530542 conn(0x7fac281020a0 msgr2=0x7fac28102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:00.571 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.054+0000 7fac2ee59700 1 -- 172.21.15.44:0/3035530542 shutdown_connections 2024-04-21T22:59:00.571 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.054+0000 7fac2ee59700 1 -- 172.21.15.44:0/3035530542 wait complete. 2024-04-21T22:59:00.571 INFO:teuthology.orchestra.run.smithi044.stdout:Generating ssh key... 2024-04-21T22:59:00.843 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:00 smithi044 ceph-mon[27934]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T22:59:01.436 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: Generating public/private rsa key pair. 2024-04-21T22:59:01.436 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: Your identification has been saved in /tmp/tmp2o80lafc/key. 2024-04-21T22:59:01.436 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: Your public key has been saved in /tmp/tmp2o80lafc/key.pub. 2024-04-21T22:59:01.437 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: The key fingerprint is: 2024-04-21T22:59:01.437 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: SHA256:XVmaspZg0KRQ1kvRA9/hOheu+locxroJ5hRSEV96U2k ceph-80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T22:59:01.437 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: The key's randomart image is: 2024-04-21T22:59:01.437 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: +---[RSA 3072]----+ 2024-04-21T22:59:01.437 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: | ..*=++. o.. | 2024-04-21T22:59:01.437 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: | o =++o+E* | 2024-04-21T22:59:01.438 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: | o.=.=oO | 2024-04-21T22:59:01.438 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: | . ..= O . | 2024-04-21T22:59:01.438 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: | . . S @ o | 2024-04-21T22:59:01.438 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: | . . = = | 2024-04-21T22:59:01.438 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: | + . + | 2024-04-21T22:59:01.438 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: | + . = | 2024-04-21T22:59:01.439 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: | . =o. | 2024-04-21T22:59:01.439 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: +----[SHA256]-----+ 2024-04-21T22:59:01.838 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.949+0000 7f793018f700 1 Processor -- start 2024-04-21T22:59:01.839 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.950+0000 7f793018f700 1 -- start start 2024-04-21T22:59:01.839 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.950+0000 7f793018f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f792807dde0 0x7f792807c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:01.839 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.950+0000 7f793018f700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f792807c820 con 0x7f792807dde0 2024-04-21T22:59:01.839 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.950+0000 7f792df2b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f792807dde0 0x7f792807c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:01.839 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.950+0000 7f792df2b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f792807dde0 0x7f792807c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48914/0 (socket says 172.21.15.44:48914) 2024-04-21T22:59:01.839 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.950+0000 7f792df2b700 1 -- 172.21.15.44:0/740295090 learned_addr learned my addr 172.21.15.44:0/740295090 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:01.839 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.950+0000 7f792df2b700 1 -- 172.21.15.44:0/740295090 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f792807c9a0 con 0x7f792807dde0 2024-04-21T22:59:01.839 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.951+0000 7f792df2b700 1 --2- 172.21.15.44:0/740295090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f792807dde0 0x7f792807c2e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f791c0098a0 tx=0x7f791c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa778f97150d99b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:01.839 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.951+0000 7f792cf29700 1 -- 172.21.15.44:0/740295090 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f791c004160 con 0x7f792807dde0 2024-04-21T22:59:01.839 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.951+0000 7f792cf29700 1 -- 172.21.15.44:0/740295090 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f791c004300 con 0x7f792807dde0 2024-04-21T22:59:01.839 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.951+0000 7f792cf29700 1 -- 172.21.15.44:0/740295090 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f791c011b50 con 0x7f792807dde0 2024-04-21T22:59:01.839 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.951+0000 7f793018f700 1 -- 172.21.15.44:0/740295090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f792807dde0 msgr2=0x7f792807c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:01.839 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.951+0000 7f793018f700 1 --2- 172.21.15.44:0/740295090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f792807dde0 0x7f792807c2e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f791c0098a0 tx=0x7f791c009bb0 comp rx=0 tx=0).stop 2024-04-21T22:59:01.840 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.951+0000 7f793018f700 1 -- 172.21.15.44:0/740295090 shutdown_connections 2024-04-21T22:59:01.840 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.951+0000 7f793018f700 1 --2- 172.21.15.44:0/740295090 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f792807dde0 0x7f792807c2e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:01.840 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.951+0000 7f793018f700 1 -- 172.21.15.44:0/740295090 >> 172.21.15.44:0/740295090 conn(0x7f7928103230 msgr2=0x7f7928105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:01.840 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.951+0000 7f793018f700 1 -- 172.21.15.44:0/740295090 shutdown_connections 2024-04-21T22:59:01.840 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.952+0000 7f793018f700 1 -- 172.21.15.44:0/740295090 wait complete. 2024-04-21T22:59:01.840 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.952+0000 7f793018f700 1 Processor -- start 2024-04-21T22:59:01.840 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.952+0000 7f793018f700 1 -- start start 2024-04-21T22:59:01.840 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.952+0000 7f793018f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f792807dde0 0x7f792819dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:01.840 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.952+0000 7f793018f700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f792810e1a0 con 0x7f792807dde0 2024-04-21T22:59:01.841 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.952+0000 7f792df2b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f792807dde0 0x7f792819dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:01.841 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.952+0000 7f792df2b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f792807dde0 0x7f792819dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48916/0 (socket says 172.21.15.44:48916) 2024-04-21T22:59:01.841 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.952+0000 7f792df2b700 1 -- 172.21.15.44:0/1658909124 learned_addr learned my addr 172.21.15.44:0/1658909124 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:01.841 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.953+0000 7f792df2b700 1 -- 172.21.15.44:0/1658909124 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f792819e2a0 con 0x7f792807dde0 2024-04-21T22:59:01.841 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.953+0000 7f792df2b700 1 --2- 172.21.15.44:0/1658909124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f792807dde0 0x7f792819dd60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f791c009fb0 tx=0x7f791c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:01.841 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.953+0000 7f791affd700 1 -- 172.21.15.44:0/1658909124 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f791c0038b0 con 0x7f792807dde0 2024-04-21T22:59:01.841 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.953+0000 7f791affd700 1 -- 172.21.15.44:0/1658909124 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f791c011e50 con 0x7f792807dde0 2024-04-21T22:59:01.841 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.953+0000 7f791affd700 1 -- 172.21.15.44:0/1658909124 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f791c019420 con 0x7f792807dde0 2024-04-21T22:59:01.841 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.953+0000 7f793018f700 1 -- 172.21.15.44:0/1658909124 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f792819e530 con 0x7f792807dde0 2024-04-21T22:59:01.841 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.953+0000 7f793018f700 1 -- 172.21.15.44:0/1658909124 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79281a1220 con 0x7f792807dde0 2024-04-21T22:59:01.841 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.954+0000 7f791affd700 1 -- 172.21.15.44:0/1658909124 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f791c003a50 con 0x7f792807dde0 2024-04-21T22:59:01.841 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.954+0000 7f793018f700 1 -- 172.21.15.44:0/1658909124 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7928063890 con 0x7f792807dde0 2024-04-21T22:59:01.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.954+0000 7f791affd700 1 --2- 172.21.15.44:0/1658909124 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f7914039050 0x7f791403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:01.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.954+0000 7f791affd700 1 -- 172.21.15.44:0/1658909124 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f791c04d280 con 0x7f792807dde0 2024-04-21T22:59:01.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.955+0000 7f792d72a700 1 --2- 172.21.15.44:0/1658909124 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f7914039050 0x7f791403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:01.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.955+0000 7f792d72a700 1 --2- 172.21.15.44:0/1658909124 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f7914039050 0x7f791403b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7924006fd0 tx=0x7f7924006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:01.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:00.959+0000 7f791affd700 1 -- 172.21.15.44:0/1658909124 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f791c00fda0 con 0x7f792807dde0 2024-04-21T22:59:01.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:01.169+0000 7f793018f700 1 -- 172.21.15.44:0/1658909124 --> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f79281044d0 con 0x7f7914039050 2024-04-21T22:59:01.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:01.272+0000 7f791affd700 1 -- 172.21.15.44:0/1658909124 <== mgr.14120 v2:172.21.15.44:6800/4238737803 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f79281044d0 con 0x7f7914039050 2024-04-21T22:59:01.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:01.274+0000 7f793018f700 1 -- 172.21.15.44:0/1658909124 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f7914039050 msgr2=0x7f791403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:01.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:01.274+0000 7f793018f700 1 --2- 172.21.15.44:0/1658909124 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f7914039050 0x7f791403b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7924006fd0 tx=0x7f7924006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:01.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:01.274+0000 7f793018f700 1 -- 172.21.15.44:0/1658909124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f792807dde0 msgr2=0x7f792819dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:01.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:01.274+0000 7f793018f700 1 --2- 172.21.15.44:0/1658909124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f792807dde0 0x7f792819dd60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f791c009fb0 tx=0x7f791c011bf0 comp rx=0 tx=0).stop 2024-04-21T22:59:01.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:01.274+0000 7f793018f700 1 -- 172.21.15.44:0/1658909124 shutdown_connections 2024-04-21T22:59:01.842 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:01.274+0000 7f793018f700 1 --2- 172.21.15.44:0/1658909124 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f7914039050 0x7f791403b510 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:01.843 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:01.274+0000 7f793018f700 1 --2- 172.21.15.44:0/1658909124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f792807dde0 0x7f792819dd60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:01.843 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:01.274+0000 7f793018f700 1 -- 172.21.15.44:0/1658909124 >> 172.21.15.44:0/1658909124 conn(0x7f7928103230 msgr2=0x7f7928103dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:01.843 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:01.274+0000 7f793018f700 1 -- 172.21.15.44:0/1658909124 shutdown_connections 2024-04-21T22:59:01.843 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:01.274+0000 7f793018f700 1 -- 172.21.15.44:0/1658909124 wait complete. 2024-04-21T22:59:02.457 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:02 smithi044 ceph-mon[27934]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T22:59:02.458 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:02 smithi044 ceph-mon[27934]: Generating ssh key... 2024-04-21T22:59:02.458 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:02 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' 2024-04-21T22:59:02.458 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:02 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' 2024-04-21T22:59:03.036 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1eyvRUMqjzHmTxxtKqHv1QnXHFSZULaQWUilTDG0pvw0WIzMHY5dFLuJvbKbhkdYT7e5Xp+UI+O2IfTcg7dW8XZ6BQELGbhRPTBdzSwj1FISSVuRUntCLuS74H9EsGzRq7NfNzJJu4oRUkLnUHp9nfhS09M+WXSvgljI542QtKqytpPMmbDz9GZuHd3J7YF3ejl4nJ0ZPcd22sQcn1qVKUs/Tid1idFPUvLdup6QbjrXKbmcO/TgCRywoNTbWClFyifT/3NXgVuWiZfpjNzmoAXF2EW11rRlmF+Db9LE3b9aCWc5gXv13xjw/PygTCk2yvAF73hXMKLkDBRqg1gn2ici0PyfslaFs+CJv9NPUJnMy9j5wP2B80gPWmCTwtonvjWzA/zBQc3gbns8zbWSoWQ0w6xzYpz5lOHuwLXIY2sJ28GOU5MX1M2eTdPT92rc7Jl+sONnLqVX10imgyz3aJKSenZ6On2++c4NrXdm03Y1QJMaVVg0ZZOGEGTfS/sU= ceph-80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T22:59:03.036 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.240+0000 7ff204a6c700 1 Processor -- start 2024-04-21T22:59:03.036 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.240+0000 7ff204a6c700 1 -- start start 2024-04-21T22:59:03.036 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.241+0000 7ff204a6c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff200106ae0 0x7ff200106f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:03.036 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.241+0000 7ff204a6c700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff2001074d0 con 0x7ff200106ae0 2024-04-21T22:59:03.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.241+0000 7ff1fe59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff200106ae0 0x7ff200106f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:03.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.241+0000 7ff1fe59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff200106ae0 0x7ff200106f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48928/0 (socket says 172.21.15.44:48928) 2024-04-21T22:59:03.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.241+0000 7ff1fe59c700 1 -- 172.21.15.44:0/512958708 learned_addr learned my addr 172.21.15.44:0/512958708 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:03.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.241+0000 7ff1fe59c700 1 -- 172.21.15.44:0/512958708 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff200107ce0 con 0x7ff200106ae0 2024-04-21T22:59:03.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.241+0000 7ff1fe59c700 1 --2- 172.21.15.44:0/512958708 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff200106ae0 0x7ff200106f00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00098a0 tx=0x7ff1f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a14047c0814ef1bd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:03.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.241+0000 7ff1fd59a700 1 -- 172.21.15.44:0/512958708 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1f0004160 con 0x7ff200106ae0 2024-04-21T22:59:03.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.241+0000 7ff1fd59a700 1 -- 172.21.15.44:0/512958708 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff1f0004300 con 0x7ff200106ae0 2024-04-21T22:59:03.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.241+0000 7ff1fd59a700 1 -- 172.21.15.44:0/512958708 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1f0011b50 con 0x7ff200106ae0 2024-04-21T22:59:03.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.242+0000 7ff204a6c700 1 -- 172.21.15.44:0/512958708 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff200106ae0 msgr2=0x7ff200106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:03.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.242+0000 7ff204a6c700 1 --2- 172.21.15.44:0/512958708 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff200106ae0 0x7ff200106f00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00098a0 tx=0x7ff1f0009bb0 comp rx=0 tx=0).stop 2024-04-21T22:59:03.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.242+0000 7ff204a6c700 1 -- 172.21.15.44:0/512958708 shutdown_connections 2024-04-21T22:59:03.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.242+0000 7ff204a6c700 1 --2- 172.21.15.44:0/512958708 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff200106ae0 0x7ff200106f00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:03.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.242+0000 7ff204a6c700 1 -- 172.21.15.44:0/512958708 >> 172.21.15.44:0/512958708 conn(0x7ff2001020a0 msgr2=0x7ff2001044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:03.037 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.242+0000 7ff204a6c700 1 -- 172.21.15.44:0/512958708 shutdown_connections 2024-04-21T22:59:03.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.242+0000 7ff204a6c700 1 -- 172.21.15.44:0/512958708 wait complete. 2024-04-21T22:59:03.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.243+0000 7ff204a6c700 1 Processor -- start 2024-04-21T22:59:03.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.243+0000 7ff204a6c700 1 -- start start 2024-04-21T22:59:03.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.243+0000 7ff204a6c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff200106ae0 0x7ff2001999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:03.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.243+0000 7ff204a6c700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff20007aff0 con 0x7ff200106ae0 2024-04-21T22:59:03.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.243+0000 7ff1fe59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff200106ae0 0x7ff2001999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:03.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.243+0000 7ff1fe59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff200106ae0 0x7ff2001999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48934/0 (socket says 172.21.15.44:48934) 2024-04-21T22:59:03.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.243+0000 7ff1fe59c700 1 -- 172.21.15.44:0/2213282696 learned_addr learned my addr 172.21.15.44:0/2213282696 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:03.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.244+0000 7ff1fe59c700 1 -- 172.21.15.44:0/2213282696 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff200199f20 con 0x7ff200106ae0 2024-04-21T22:59:03.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.244+0000 7ff1fe59c700 1 --2- 172.21.15.44:0/2213282696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff200106ae0 0x7ff2001999e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0011850 tx=0x7ff1f0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:03.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.244+0000 7ff1f77fe700 1 -- 172.21.15.44:0/2213282696 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1f0003af0 con 0x7ff200106ae0 2024-04-21T22:59:03.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.244+0000 7ff1f77fe700 1 -- 172.21.15.44:0/2213282696 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff1f0020070 con 0x7ff200106ae0 2024-04-21T22:59:03.038 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.244+0000 7ff204a6c700 1 -- 172.21.15.44:0/2213282696 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff20019a1b0 con 0x7ff200106ae0 2024-04-21T22:59:03.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.244+0000 7ff1f77fe700 1 -- 172.21.15.44:0/2213282696 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1f00195a0 con 0x7ff200106ae0 2024-04-21T22:59:03.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.244+0000 7ff204a6c700 1 -- 172.21.15.44:0/2213282696 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff20019cea0 con 0x7ff200106ae0 2024-04-21T22:59:03.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.245+0000 7ff1f77fe700 1 -- 172.21.15.44:0/2213282696 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7ff1f0019860 con 0x7ff200106ae0 2024-04-21T22:59:03.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.245+0000 7ff1f77fe700 1 --2- 172.21.15.44:0/2213282696 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7ff1ec03b2b0 0x7ff1ec03d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:03.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.245+0000 7ff1f77fe700 1 -- 172.21.15.44:0/2213282696 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff1f0011e00 con 0x7ff200106ae0 2024-04-21T22:59:03.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.245+0000 7ff1fdd9b700 1 --2- 172.21.15.44:0/2213282696 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7ff1ec03b2b0 0x7ff1ec03d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:03.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.245+0000 7ff204a6c700 1 -- 172.21.15.44:0/2213282696 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff200063890 con 0x7ff200106ae0 2024-04-21T22:59:03.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.246+0000 7ff1fdd9b700 1 --2- 172.21.15.44:0/2213282696 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7ff1ec03b2b0 0x7ff1ec03d770 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8006fd0 tx=0x7ff1e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:03.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.250+0000 7ff1f77fe700 1 -- 172.21.15.44:0/2213282696 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff1f0003c90 con 0x7ff200106ae0 2024-04-21T22:59:03.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.457+0000 7ff204a6c700 1 -- 172.21.15.44:0/2213282696 --> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7ff200062670 con 0x7ff1ec03b2b0 2024-04-21T22:59:03.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.458+0000 7ff1f77fe700 1 -- 172.21.15.44:0/2213282696 <== mgr.14120 v2:172.21.15.44:6800/4238737803 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7ff200062670 con 0x7ff1ec03b2b0 2024-04-21T22:59:03.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.460+0000 7ff204a6c700 1 -- 172.21.15.44:0/2213282696 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7ff1ec03b2b0 msgr2=0x7ff1ec03d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:03.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.460+0000 7ff204a6c700 1 --2- 172.21.15.44:0/2213282696 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7ff1ec03b2b0 0x7ff1ec03d770 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8006fd0 tx=0x7ff1e8006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:03.039 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.460+0000 7ff204a6c700 1 -- 172.21.15.44:0/2213282696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff200106ae0 msgr2=0x7ff2001999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:03.040 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.460+0000 7ff204a6c700 1 --2- 172.21.15.44:0/2213282696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff200106ae0 0x7ff2001999e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0011850 tx=0x7ff1f0011c10 comp rx=0 tx=0).stop 2024-04-21T22:59:03.040 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.460+0000 7ff204a6c700 1 -- 172.21.15.44:0/2213282696 shutdown_connections 2024-04-21T22:59:03.040 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.460+0000 7ff204a6c700 1 --2- 172.21.15.44:0/2213282696 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7ff1ec03b2b0 0x7ff1ec03d770 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:03.040 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.460+0000 7ff204a6c700 1 --2- 172.21.15.44:0/2213282696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff200106ae0 0x7ff2001999e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:03.040 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.460+0000 7ff204a6c700 1 -- 172.21.15.44:0/2213282696 >> 172.21.15.44:0/2213282696 conn(0x7ff2001020a0 msgr2=0x7ff200190000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:03.040 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.460+0000 7ff204a6c700 1 -- 172.21.15.44:0/2213282696 shutdown_connections 2024-04-21T22:59:03.040 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:02.460+0000 7ff204a6c700 1 -- 172.21.15.44:0/2213282696 wait complete. 2024-04-21T22:59:03.040 INFO:teuthology.orchestra.run.smithi044.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-21T22:59:03.040 INFO:teuthology.orchestra.run.smithi044.stdout:Adding key to root@localhost authorized_keys... 2024-04-21T22:59:03.040 INFO:teuthology.orchestra.run.smithi044.stdout:Adding host smithi044... 2024-04-21T22:59:03.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:03 smithi044 ceph-mon[27934]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T22:59:04.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:04 smithi044 ceph-mon[27934]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi044", "addr": "172.21.15.44", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T22:59:04.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:04 smithi044 ceph-mon[27934]: Deploying cephadm binary to smithi044 2024-04-21T22:59:08.535 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Added host 'smithi044' with addr '172.21.15.44' 2024-04-21T22:59:08.535 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.486+0000 7fb9a274d700 1 Processor -- start 2024-04-21T22:59:08.535 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.486+0000 7fb9a274d700 1 -- start start 2024-04-21T22:59:08.535 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.486+0000 7fb9a274d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb99c106ae0 0x7fb99c106f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:08.535 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.486+0000 7fb9a274d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c1074d0 con 0x7fb99c106ae0 2024-04-21T22:59:08.536 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.486+0000 7fb99bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb99c106ae0 0x7fb99c106f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:08.536 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.487+0000 7fb99bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb99c106ae0 0x7fb99c106f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48946/0 (socket says 172.21.15.44:48946) 2024-04-21T22:59:08.536 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.487+0000 7fb99bfff700 1 -- 172.21.15.44:0/2733300631 learned_addr learned my addr 172.21.15.44:0/2733300631 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:08.536 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.487+0000 7fb99bfff700 1 -- 172.21.15.44:0/2733300631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb99c107ce0 con 0x7fb99c106ae0 2024-04-21T22:59:08.536 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.487+0000 7fb99bfff700 1 --2- 172.21.15.44:0/2733300631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb99c106ae0 0x7fb99c106f00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb98c0098a0 tx=0x7fb98c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b3a35cda862be6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:08.536 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.487+0000 7fb99affd700 1 -- 172.21.15.44:0/2733300631 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb98c004160 con 0x7fb99c106ae0 2024-04-21T22:59:08.536 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.487+0000 7fb99affd700 1 -- 172.21.15.44:0/2733300631 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb98c004300 con 0x7fb99c106ae0 2024-04-21T22:59:08.536 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.487+0000 7fb99affd700 1 -- 172.21.15.44:0/2733300631 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb98c011b50 con 0x7fb99c106ae0 2024-04-21T22:59:08.536 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.488+0000 7fb9a274d700 1 -- 172.21.15.44:0/2733300631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb99c106ae0 msgr2=0x7fb99c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:08.536 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.488+0000 7fb9a274d700 1 --2- 172.21.15.44:0/2733300631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb99c106ae0 0x7fb99c106f00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb98c0098a0 tx=0x7fb98c009bb0 comp rx=0 tx=0).stop 2024-04-21T22:59:08.536 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.488+0000 7fb9a274d700 1 -- 172.21.15.44:0/2733300631 shutdown_connections 2024-04-21T22:59:08.536 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.488+0000 7fb9a274d700 1 --2- 172.21.15.44:0/2733300631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb99c106ae0 0x7fb99c106f00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:08.536 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.488+0000 7fb9a274d700 1 -- 172.21.15.44:0/2733300631 >> 172.21.15.44:0/2733300631 conn(0x7fb99c1020a0 msgr2=0x7fb99c1044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:08.537 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.488+0000 7fb9a274d700 1 -- 172.21.15.44:0/2733300631 shutdown_connections 2024-04-21T22:59:08.537 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.488+0000 7fb9a274d700 1 -- 172.21.15.44:0/2733300631 wait complete. 2024-04-21T22:59:08.537 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.488+0000 7fb9a274d700 1 Processor -- start 2024-04-21T22:59:08.537 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.488+0000 7fb9a274d700 1 -- start start 2024-04-21T22:59:08.537 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.489+0000 7fb9a274d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb99c106ae0 0x7fb99c191c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:08.537 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.489+0000 7fb9a274d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c10b090 con 0x7fb99c106ae0 2024-04-21T22:59:08.537 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.489+0000 7fb99bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb99c106ae0 0x7fb99c191c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:08.537 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.489+0000 7fb99bfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb99c106ae0 0x7fb99c191c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48962/0 (socket says 172.21.15.44:48962) 2024-04-21T22:59:08.537 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.489+0000 7fb99bfff700 1 -- 172.21.15.44:0/2977678078 learned_addr learned my addr 172.21.15.44:0/2977678078 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:08.537 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.489+0000 7fb99bfff700 1 -- 172.21.15.44:0/2977678078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb99c1921d0 con 0x7fb99c106ae0 2024-04-21T22:59:08.537 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.489+0000 7fb99bfff700 1 --2- 172.21.15.44:0/2977678078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb99c106ae0 0x7fb99c191c90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb98c011850 tx=0x7fb98c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:08.537 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.489+0000 7fb9997fa700 1 -- 172.21.15.44:0/2977678078 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb98c003af0 con 0x7fb99c106ae0 2024-04-21T22:59:08.537 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.490+0000 7fb9997fa700 1 -- 172.21.15.44:0/2977678078 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb98c020070 con 0x7fb99c106ae0 2024-04-21T22:59:08.537 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.490+0000 7fb9a274d700 1 -- 172.21.15.44:0/2977678078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb99c1903c0 con 0x7fb99c106ae0 2024-04-21T22:59:08.537 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.490+0000 7fb9997fa700 1 -- 172.21.15.44:0/2977678078 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb98c0195a0 con 0x7fb99c106ae0 2024-04-21T22:59:08.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.490+0000 7fb9a274d700 1 -- 172.21.15.44:0/2977678078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb99c190900 con 0x7fb99c106ae0 2024-04-21T22:59:08.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.490+0000 7fb9997fa700 1 -- 172.21.15.44:0/2977678078 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fb98c0198a0 con 0x7fb99c106ae0 2024-04-21T22:59:08.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.490+0000 7fb9a274d700 1 -- 172.21.15.44:0/2977678078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb99c063890 con 0x7fb99c106ae0 2024-04-21T22:59:08.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.491+0000 7fb9997fa700 1 --2- 172.21.15.44:0/2977678078 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fb984039050 0x7fb98403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:08.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.491+0000 7fb9997fa700 1 -- 172.21.15.44:0/2977678078 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb98c011e00 con 0x7fb99c106ae0 2024-04-21T22:59:08.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.491+0000 7fb99b7fe700 1 --2- 172.21.15.44:0/2977678078 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fb984039050 0x7fb98403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:08.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.491+0000 7fb99b7fe700 1 --2- 172.21.15.44:0/2977678078 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fb984039050 0x7fb98403b510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb990006fd0 tx=0x7fb990006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:08.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.496+0000 7fb9997fa700 1 -- 172.21.15.44:0/2977678078 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb98c023370 con 0x7fb99c106ae0 2024-04-21T22:59:08.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:03.707+0000 7fb9a274d700 1 -- 172.21.15.44:0/2977678078 --> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi044", "addr": "172.21.15.44", "target": ["mon-mgr", ""]}) -- 0x7fb99c195a80 con 0x7fb984039050 2024-04-21T22:59:08.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:07.979+0000 7fb9997fa700 1 -- 172.21.15.44:0/2977678078 <== mgr.14120 v2:172.21.15.44:6800/4238737803 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fb99c195a80 con 0x7fb984039050 2024-04-21T22:59:08.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:07.981+0000 7fb9a274d700 1 -- 172.21.15.44:0/2977678078 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fb984039050 msgr2=0x7fb98403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:08.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:07.981+0000 7fb9a274d700 1 --2- 172.21.15.44:0/2977678078 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fb984039050 0x7fb98403b510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb990006fd0 tx=0x7fb990006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:08.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:07.981+0000 7fb9a274d700 1 -- 172.21.15.44:0/2977678078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb99c106ae0 msgr2=0x7fb99c191c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:08.539 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:07.981+0000 7fb9a274d700 1 --2- 172.21.15.44:0/2977678078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb99c106ae0 0x7fb99c191c90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb98c011850 tx=0x7fb98c011c10 comp rx=0 tx=0).stop 2024-04-21T22:59:08.539 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:07.981+0000 7fb9a274d700 1 -- 172.21.15.44:0/2977678078 shutdown_connections 2024-04-21T22:59:08.539 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:07.981+0000 7fb9a274d700 1 --2- 172.21.15.44:0/2977678078 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fb984039050 0x7fb98403b510 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:08.539 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:07.981+0000 7fb9a274d700 1 --2- 172.21.15.44:0/2977678078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb99c106ae0 0x7fb99c191c90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:08.539 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:07.981+0000 7fb9a274d700 1 -- 172.21.15.44:0/2977678078 >> 172.21.15.44:0/2977678078 conn(0x7fb99c1020a0 msgr2=0x7fb99c102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:08.539 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:07.981+0000 7fb9a274d700 1 -- 172.21.15.44:0/2977678078 shutdown_connections 2024-04-21T22:59:08.539 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:07.981+0000 7fb9a274d700 1 -- 172.21.15.44:0/2977678078 wait complete. 2024-04-21T22:59:08.539 INFO:teuthology.orchestra.run.smithi044.stdout:Deploying unmanaged mon service... 2024-04-21T22:59:09.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:08 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' 2024-04-21T22:59:09.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:08 smithi044 ceph-mon[27934]: Added host smithi044 2024-04-21T22:59:09.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:08 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T22:59:09.927 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-21T22:59:09.927 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.992+0000 7fd6aa95d700 1 Processor -- start 2024-04-21T22:59:09.927 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.992+0000 7fd6aa95d700 1 -- start start 2024-04-21T22:59:09.927 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.992+0000 7fd6aa95d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6a4106b40 0x7fd6a4106f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:09.927 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.992+0000 7fd6aa95d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a4107530 con 0x7fd6a4106b40 2024-04-21T22:59:09.928 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.993+0000 7fd6a3fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6a4106b40 0x7fd6a4106f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:09.928 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.993+0000 7fd6a3fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6a4106b40 0x7fd6a4106f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48968/0 (socket says 172.21.15.44:48968) 2024-04-21T22:59:09.928 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.993+0000 7fd6a3fff700 1 -- 172.21.15.44:0/3225763275 learned_addr learned my addr 172.21.15.44:0/3225763275 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:09.928 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.993+0000 7fd6a3fff700 1 -- 172.21.15.44:0/3225763275 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6a4107d40 con 0x7fd6a4106b40 2024-04-21T22:59:09.928 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.993+0000 7fd6a3fff700 1 --2- 172.21.15.44:0/3225763275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6a4106b40 0x7fd6a4106f60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd6940098a0 tx=0x7fd694009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86050bf85297b2b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:09.928 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.993+0000 7fd6a2ffd700 1 -- 172.21.15.44:0/3225763275 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd694004160 con 0x7fd6a4106b40 2024-04-21T22:59:09.928 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.993+0000 7fd6a2ffd700 1 -- 172.21.15.44:0/3225763275 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd694004300 con 0x7fd6a4106b40 2024-04-21T22:59:09.928 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.993+0000 7fd6a2ffd700 1 -- 172.21.15.44:0/3225763275 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd694011b50 con 0x7fd6a4106b40 2024-04-21T22:59:09.928 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.994+0000 7fd6aa95d700 1 -- 172.21.15.44:0/3225763275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6a4106b40 msgr2=0x7fd6a4106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:09.928 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.994+0000 7fd6aa95d700 1 --2- 172.21.15.44:0/3225763275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6a4106b40 0x7fd6a4106f60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd6940098a0 tx=0x7fd694009bb0 comp rx=0 tx=0).stop 2024-04-21T22:59:09.928 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.994+0000 7fd6aa95d700 1 -- 172.21.15.44:0/3225763275 shutdown_connections 2024-04-21T22:59:09.928 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.994+0000 7fd6aa95d700 1 --2- 172.21.15.44:0/3225763275 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6a4106b40 0x7fd6a4106f60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:09.928 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.994+0000 7fd6aa95d700 1 -- 172.21.15.44:0/3225763275 >> 172.21.15.44:0/3225763275 conn(0x7fd6a4102120 msgr2=0x7fd6a4104540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:09.929 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.994+0000 7fd6aa95d700 1 -- 172.21.15.44:0/3225763275 shutdown_connections 2024-04-21T22:59:09.929 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.994+0000 7fd6aa95d700 1 -- 172.21.15.44:0/3225763275 wait complete. 2024-04-21T22:59:09.929 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.995+0000 7fd6aa95d700 1 Processor -- start 2024-04-21T22:59:09.929 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.995+0000 7fd6aa95d700 1 -- start start 2024-04-21T22:59:09.929 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.995+0000 7fd6aa95d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6a4106b40 0x7fd6a419ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:09.929 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.995+0000 7fd6aa95d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a410b0f0 con 0x7fd6a4106b40 2024-04-21T22:59:09.929 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.995+0000 7fd6a3fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6a4106b40 0x7fd6a419ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:09.929 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.995+0000 7fd6a3fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6a4106b40 0x7fd6a419ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48972/0 (socket says 172.21.15.44:48972) 2024-04-21T22:59:09.929 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.995+0000 7fd6a3fff700 1 -- 172.21.15.44:0/852982093 learned_addr learned my addr 172.21.15.44:0/852982093 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:09.929 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.995+0000 7fd6a3fff700 1 -- 172.21.15.44:0/852982093 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6a419e330 con 0x7fd6a4106b40 2024-04-21T22:59:09.929 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.996+0000 7fd6a3fff700 1 --2- 172.21.15.44:0/852982093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6a4106b40 0x7fd6a419ddf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd694011670 tx=0x7fd694011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:09.929 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.996+0000 7fd6a17fa700 1 -- 172.21.15.44:0/852982093 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6940039c0 con 0x7fd6a4106b40 2024-04-21T22:59:09.929 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.996+0000 7fd6a17fa700 1 -- 172.21.15.44:0/852982093 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd694011e50 con 0x7fd6a4106b40 2024-04-21T22:59:09.929 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.996+0000 7fd6aa95d700 1 -- 172.21.15.44:0/852982093 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6a419e5c0 con 0x7fd6a4106b40 2024-04-21T22:59:09.930 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.996+0000 7fd6a17fa700 1 -- 172.21.15.44:0/852982093 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd69401a5c0 con 0x7fd6a4106b40 2024-04-21T22:59:09.930 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.996+0000 7fd6aa95d700 1 -- 172.21.15.44:0/852982093 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6a41a12b0 con 0x7fd6a4106b40 2024-04-21T22:59:09.930 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.997+0000 7fd6a17fa700 1 -- 172.21.15.44:0/852982093 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fd69401a870 con 0x7fd6a4106b40 2024-04-21T22:59:09.930 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.997+0000 7fd6a17fa700 1 --2- 172.21.15.44:0/852982093 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fd68c039050 0x7fd68c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:09.930 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.997+0000 7fd6aa95d700 1 -- 172.21.15.44:0/852982093 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6a4063890 con 0x7fd6a4106b40 2024-04-21T22:59:09.930 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.997+0000 7fd6a17fa700 1 -- 172.21.15.44:0/852982093 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd69404d3f0 con 0x7fd6a4106b40 2024-04-21T22:59:09.930 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.997+0000 7fd6a37fe700 1 --2- 172.21.15.44:0/852982093 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fd68c039050 0x7fd68c03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:09.930 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:08.997+0000 7fd6a37fe700 1 --2- 172.21.15.44:0/852982093 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fd68c039050 0x7fd68c03b510 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd698006fd0 tx=0x7fd698006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:09.930 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:09.002+0000 7fd6a17fa700 1 -- 172.21.15.44:0/852982093 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd694019c70 con 0x7fd6a4106b40 2024-04-21T22:59:09.930 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:09.226+0000 7fd6aa95d700 1 -- 172.21.15.44:0/852982093 --> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fd6a40008d0 con 0x7fd68c039050 2024-04-21T22:59:09.930 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:09.230+0000 7fd6a17fa700 1 -- 172.21.15.44:0/852982093 <== mgr.14120 v2:172.21.15.44:6800/4238737803 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fd6a40008d0 con 0x7fd68c039050 2024-04-21T22:59:09.930 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:09.232+0000 7fd6aa95d700 1 -- 172.21.15.44:0/852982093 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fd68c039050 msgr2=0x7fd68c03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:09.930 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:09.233+0000 7fd6aa95d700 1 --2- 172.21.15.44:0/852982093 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fd68c039050 0x7fd68c03b510 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd698006fd0 tx=0x7fd698006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:09.931 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:09.233+0000 7fd6aa95d700 1 -- 172.21.15.44:0/852982093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6a4106b40 msgr2=0x7fd6a419ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:09.931 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:09.233+0000 7fd6aa95d700 1 --2- 172.21.15.44:0/852982093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6a4106b40 0x7fd6a419ddf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd694011670 tx=0x7fd694011b60 comp rx=0 tx=0).stop 2024-04-21T22:59:09.931 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:09.233+0000 7fd6aa95d700 1 -- 172.21.15.44:0/852982093 shutdown_connections 2024-04-21T22:59:09.931 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:09.233+0000 7fd6aa95d700 1 --2- 172.21.15.44:0/852982093 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fd68c039050 0x7fd68c03b510 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:09.931 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:09.233+0000 7fd6aa95d700 1 --2- 172.21.15.44:0/852982093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6a4106b40 0x7fd6a419ddf0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:09.931 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:09.233+0000 7fd6aa95d700 1 -- 172.21.15.44:0/852982093 >> 172.21.15.44:0/852982093 conn(0x7fd6a4102120 msgr2=0x7fd6a4103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:09.931 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:09.233+0000 7fd6aa95d700 1 -- 172.21.15.44:0/852982093 shutdown_connections 2024-04-21T22:59:09.931 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:09.233+0000 7fd6aa95d700 1 -- 172.21.15.44:0/852982093 wait complete. 2024-04-21T22:59:09.931 INFO:teuthology.orchestra.run.smithi044.stdout:Deploying unmanaged mgr service... 2024-04-21T22:59:10.481 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:10 smithi044 ceph-mon[27934]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T22:59:10.481 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:10 smithi044 ceph-mon[27934]: Saving service mon spec with placement count:5 2024-04-21T22:59:10.481 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:10 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' 2024-04-21T22:59:11.898 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:11 smithi044 ceph-mon[27934]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T22:59:11.898 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:11 smithi044 ceph-mon[27934]: Saving service mgr spec with placement count:2 2024-04-21T22:59:11.898 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:11 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' 2024-04-21T22:59:11.898 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:11 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' 2024-04-21T22:59:11.906 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-21T22:59:11.906 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.346+0000 7f0f3da7c700 1 Processor -- start 2024-04-21T22:59:11.907 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.346+0000 7f0f3da7c700 1 -- start start 2024-04-21T22:59:11.907 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.346+0000 7f0f3da7c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f38106ae0 0x7f0f38106f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:11.907 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.346+0000 7f0f3da7c700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0f381074d0 con 0x7f0f38106ae0 2024-04-21T22:59:11.907 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.346+0000 7f0f377fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f38106ae0 0x7f0f38106f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:11.907 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.346+0000 7f0f377fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f38106ae0 0x7f0f38106f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48978/0 (socket says 172.21.15.44:48978) 2024-04-21T22:59:11.907 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.346+0000 7f0f377fe700 1 -- 172.21.15.44:0/1537798888 learned_addr learned my addr 172.21.15.44:0/1537798888 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:11.907 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.346+0000 7f0f377fe700 1 -- 172.21.15.44:0/1537798888 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f38107ce0 con 0x7f0f38106ae0 2024-04-21T22:59:11.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.346+0000 7f0f377fe700 1 --2- 172.21.15.44:0/1537798888 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f38106ae0 0x7f0f38106f00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0f280098a0 tx=0x7f0f28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b44bd3deeb7c852 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:11.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.347+0000 7f0f367fc700 1 -- 172.21.15.44:0/1537798888 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f28004160 con 0x7f0f38106ae0 2024-04-21T22:59:11.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.347+0000 7f0f367fc700 1 -- 172.21.15.44:0/1537798888 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0f28004300 con 0x7f0f38106ae0 2024-04-21T22:59:11.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.347+0000 7f0f367fc700 1 -- 172.21.15.44:0/1537798888 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f28011b50 con 0x7f0f38106ae0 2024-04-21T22:59:11.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.347+0000 7f0f3da7c700 1 -- 172.21.15.44:0/1537798888 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f38106ae0 msgr2=0x7f0f38106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:11.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.347+0000 7f0f3da7c700 1 --2- 172.21.15.44:0/1537798888 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f38106ae0 0x7f0f38106f00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0f280098a0 tx=0x7f0f28009bb0 comp rx=0 tx=0).stop 2024-04-21T22:59:11.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.347+0000 7f0f3da7c700 1 -- 172.21.15.44:0/1537798888 shutdown_connections 2024-04-21T22:59:11.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.347+0000 7f0f3da7c700 1 --2- 172.21.15.44:0/1537798888 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f38106ae0 0x7f0f38106f00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:11.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.347+0000 7f0f3da7c700 1 -- 172.21.15.44:0/1537798888 >> 172.21.15.44:0/1537798888 conn(0x7f0f381020a0 msgr2=0x7f0f381044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:11.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.347+0000 7f0f3da7c700 1 -- 172.21.15.44:0/1537798888 shutdown_connections 2024-04-21T22:59:11.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.347+0000 7f0f3da7c700 1 -- 172.21.15.44:0/1537798888 wait complete. 2024-04-21T22:59:11.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.348+0000 7f0f3da7c700 1 Processor -- start 2024-04-21T22:59:11.908 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.348+0000 7f0f3da7c700 1 -- start start 2024-04-21T22:59:11.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.348+0000 7f0f3da7c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f38106ae0 0x7f0f38191c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:11.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.348+0000 7f0f3da7c700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3810b090 con 0x7f0f38106ae0 2024-04-21T22:59:11.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.348+0000 7f0f377fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f38106ae0 0x7f0f38191c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:11.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.348+0000 7f0f377fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f38106ae0 0x7f0f38191c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48988/0 (socket says 172.21.15.44:48988) 2024-04-21T22:59:11.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.348+0000 7f0f377fe700 1 -- 172.21.15.44:0/3699037560 learned_addr learned my addr 172.21.15.44:0/3699037560 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:11.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.349+0000 7f0f377fe700 1 -- 172.21.15.44:0/3699037560 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f381921d0 con 0x7f0f38106ae0 2024-04-21T22:59:11.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.349+0000 7f0f377fe700 1 --2- 172.21.15.44:0/3699037560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f38106ae0 0x7f0f38191c90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0f28009fb0 tx=0x7f0f28011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:11.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.349+0000 7f0f34ff9700 1 -- 172.21.15.44:0/3699037560 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f280038b0 con 0x7f0f38106ae0 2024-04-21T22:59:11.909 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.349+0000 7f0f34ff9700 1 -- 172.21.15.44:0/3699037560 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0f28011e50 con 0x7f0f38106ae0 2024-04-21T22:59:11.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.349+0000 7f0f3da7c700 1 -- 172.21.15.44:0/3699037560 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f381903c0 con 0x7f0f38106ae0 2024-04-21T22:59:11.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.349+0000 7f0f34ff9700 1 -- 172.21.15.44:0/3699037560 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f28019550 con 0x7f0f38106ae0 2024-04-21T22:59:11.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.349+0000 7f0f3da7c700 1 -- 172.21.15.44:0/3699037560 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f38190900 con 0x7f0f38106ae0 2024-04-21T22:59:11.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.350+0000 7f0f34ff9700 1 -- 172.21.15.44:0/3699037560 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f0f28019850 con 0x7f0f38106ae0 2024-04-21T22:59:11.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.350+0000 7f0f3da7c700 1 -- 172.21.15.44:0/3699037560 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f38063890 con 0x7f0f38106ae0 2024-04-21T22:59:11.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.350+0000 7f0f34ff9700 1 --2- 172.21.15.44:0/3699037560 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f0f20039000 0x7f0f2003b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:11.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.350+0000 7f0f34ff9700 1 -- 172.21.15.44:0/3699037560 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0f2804d420 con 0x7f0f38106ae0 2024-04-21T22:59:11.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.350+0000 7f0f36ffd700 1 --2- 172.21.15.44:0/3699037560 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f0f20039000 0x7f0f2003b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:11.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.351+0000 7f0f36ffd700 1 --2- 172.21.15.44:0/3699037560 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f0f20039000 0x7f0f2003b4c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c006fd0 tx=0x7f0f2c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:11.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.355+0000 7f0f34ff9700 1 -- 172.21.15.44:0/3699037560 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0f2802d960 con 0x7f0f38106ae0 2024-04-21T22:59:11.910 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.581+0000 7f0f3da7c700 1 -- 172.21.15.44:0/3699037560 --> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f0f38195aa0 con 0x7f0f20039000 2024-04-21T22:59:11.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.587+0000 7f0f34ff9700 1 -- 172.21.15.44:0/3699037560 <== mgr.14120 v2:172.21.15.44:6800/4238737803 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0f38195aa0 con 0x7f0f20039000 2024-04-21T22:59:11.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.589+0000 7f0f3da7c700 1 -- 172.21.15.44:0/3699037560 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f0f20039000 msgr2=0x7f0f2003b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:11.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.589+0000 7f0f3da7c700 1 --2- 172.21.15.44:0/3699037560 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f0f20039000 0x7f0f2003b4c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c006fd0 tx=0x7f0f2c006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:11.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.589+0000 7f0f3da7c700 1 -- 172.21.15.44:0/3699037560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f38106ae0 msgr2=0x7f0f38191c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:11.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.589+0000 7f0f3da7c700 1 --2- 172.21.15.44:0/3699037560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f38106ae0 0x7f0f38191c90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0f28009fb0 tx=0x7f0f28011bf0 comp rx=0 tx=0).stop 2024-04-21T22:59:11.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.589+0000 7f0f3da7c700 1 -- 172.21.15.44:0/3699037560 shutdown_connections 2024-04-21T22:59:11.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.589+0000 7f0f3da7c700 1 --2- 172.21.15.44:0/3699037560 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f0f20039000 0x7f0f2003b4c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:11.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.589+0000 7f0f3da7c700 1 --2- 172.21.15.44:0/3699037560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0f38106ae0 0x7f0f38191c90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:11.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.589+0000 7f0f3da7c700 1 -- 172.21.15.44:0/3699037560 >> 172.21.15.44:0/3699037560 conn(0x7f0f381020a0 msgr2=0x7f0f38102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:11.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.589+0000 7f0f3da7c700 1 -- 172.21.15.44:0/3699037560 shutdown_connections 2024-04-21T22:59:11.911 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:10.589+0000 7f0f3da7c700 1 -- 172.21.15.44:0/3699037560 wait complete. 2024-04-21T22:59:13.187 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.361+0000 7f3c00fb8700 1 Processor -- start 2024-04-21T22:59:13.187 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.361+0000 7f3c00fb8700 1 -- start start 2024-04-21T22:59:13.187 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.361+0000 7f3c00fb8700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bfc106440 0x7f3bfc108870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:13.187 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.361+0000 7f3c00fb8700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3bfc07bed0 con 0x7f3bfc106440 2024-04-21T22:59:13.187 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.361+0000 7f3bfa59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bfc106440 0x7f3bfc108870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:13.187 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.361+0000 7f3bfa59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bfc106440 0x7f3bfc108870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48594/0 (socket says 172.21.15.44:48594) 2024-04-21T22:59:13.187 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.361+0000 7f3bfa59c700 1 -- 172.21.15.44:0/1725181363 learned_addr learned my addr 172.21.15.44:0/1725181363 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:13.187 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.362+0000 7f3bfa59c700 1 -- 172.21.15.44:0/1725181363 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bfc108db0 con 0x7f3bfc106440 2024-04-21T22:59:13.187 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.362+0000 7f3bfa59c700 1 --2- 172.21.15.44:0/1725181363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bfc106440 0x7f3bfc108870 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0098a0 tx=0x7f3bec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3820ad91f2449f13 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:13.187 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.362+0000 7f3bf959a700 1 -- 172.21.15.44:0/1725181363 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3bec004160 con 0x7f3bfc106440 2024-04-21T22:59:13.187 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.362+0000 7f3bf959a700 1 -- 172.21.15.44:0/1725181363 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bec004300 con 0x7f3bfc106440 2024-04-21T22:59:13.188 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.362+0000 7f3bf959a700 1 -- 172.21.15.44:0/1725181363 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3bec011b20 con 0x7f3bfc106440 2024-04-21T22:59:13.188 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.363+0000 7f3c00fb8700 1 -- 172.21.15.44:0/1725181363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bfc106440 msgr2=0x7f3bfc108870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:13.188 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.363+0000 7f3c00fb8700 1 --2- 172.21.15.44:0/1725181363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bfc106440 0x7f3bfc108870 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0098a0 tx=0x7f3bec009bb0 comp rx=0 tx=0).stop 2024-04-21T22:59:13.188 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.363+0000 7f3c00fb8700 1 -- 172.21.15.44:0/1725181363 shutdown_connections 2024-04-21T22:59:13.188 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.363+0000 7f3c00fb8700 1 --2- 172.21.15.44:0/1725181363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bfc106440 0x7f3bfc108870 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:13.188 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.363+0000 7f3c00fb8700 1 -- 172.21.15.44:0/1725181363 >> 172.21.15.44:0/1725181363 conn(0x7f3bfc1020a0 msgr2=0x7f3bfc1044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:13.188 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.363+0000 7f3c00fb8700 1 -- 172.21.15.44:0/1725181363 shutdown_connections 2024-04-21T22:59:13.188 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.363+0000 7f3c00fb8700 1 -- 172.21.15.44:0/1725181363 wait complete. 2024-04-21T22:59:13.188 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.364+0000 7f3c00fb8700 1 Processor -- start 2024-04-21T22:59:13.188 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.364+0000 7f3c00fb8700 1 -- start start 2024-04-21T22:59:13.188 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.364+0000 7f3c00fb8700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bfc106440 0x7f3bfc1a2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:13.188 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.364+0000 7f3c00fb8700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3bfc109e80 con 0x7f3bfc106440 2024-04-21T22:59:13.188 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.364+0000 7f3bfa59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bfc106440 0x7f3bfc1a2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:13.189 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.364+0000 7f3bfa59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bfc106440 0x7f3bfc1a2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48600/0 (socket says 172.21.15.44:48600) 2024-04-21T22:59:13.189 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.364+0000 7f3bfa59c700 1 -- 172.21.15.44:0/2792591479 learned_addr learned my addr 172.21.15.44:0/2792591479 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:13.189 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.364+0000 7f3bfa59c700 1 -- 172.21.15.44:0/2792591479 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bfc1a26b0 con 0x7f3bfc106440 2024-04-21T22:59:13.189 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.365+0000 7f3bfa59c700 1 --2- 172.21.15.44:0/2792591479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bfc106440 0x7f3bfc1a2170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0116a0 tx=0x7f3bec011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:13.189 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.365+0000 7f3beb7fe700 1 -- 172.21.15.44:0/2792591479 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3bec003a40 con 0x7f3bfc106440 2024-04-21T22:59:13.189 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.365+0000 7f3c00fb8700 1 -- 172.21.15.44:0/2792591479 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bfc1a2940 con 0x7f3bfc106440 2024-04-21T22:59:13.189 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.365+0000 7f3beb7fe700 1 -- 172.21.15.44:0/2792591479 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bec003be0 con 0x7f3bfc106440 2024-04-21T22:59:13.189 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.365+0000 7f3beb7fe700 1 -- 172.21.15.44:0/2792591479 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3bec01a750 con 0x7f3bfc106440 2024-04-21T22:59:13.189 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.365+0000 7f3c00fb8700 1 -- 172.21.15.44:0/2792591479 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bfc1a2da0 con 0x7f3bfc106440 2024-04-21T22:59:13.189 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.366+0000 7f3beb7fe700 1 -- 172.21.15.44:0/2792591479 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f3bec024430 con 0x7f3bfc106440 2024-04-21T22:59:13.189 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.366+0000 7f3c00fb8700 1 -- 172.21.15.44:0/2792591479 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bfc063890 con 0x7f3bfc106440 2024-04-21T22:59:13.189 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.366+0000 7f3beb7fe700 1 --2- 172.21.15.44:0/2792591479 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f3be4039050 0x7f3be403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:13.189 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.367+0000 7f3beb7fe700 1 -- 172.21.15.44:0/2792591479 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3bec04d5e0 con 0x7f3bfc106440 2024-04-21T22:59:13.190 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.367+0000 7f3bf9d9b700 1 --2- 172.21.15.44:0/2792591479 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f3be4039050 0x7f3be403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:13.190 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.367+0000 7f3bf9d9b700 1 --2- 172.21.15.44:0/2792591479 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f3be4039050 0x7f3be403b510 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0006fd0 tx=0x7f3bf0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:13.190 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.372+0000 7f3beb7fe700 1 -- 172.21.15.44:0/2792591479 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3bec011e20 con 0x7f3bfc106440 2024-04-21T22:59:13.190 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.584+0000 7f3c00fb8700 1 -- 172.21.15.44:0/2792591479 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f3bfc062e40 con 0x7f3bfc106440 2024-04-21T22:59:13.190 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.586+0000 7f3beb7fe700 1 -- 172.21.15.44:0/2792591479 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f3bec0247d0 con 0x7f3bfc106440 2024-04-21T22:59:13.190 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.589+0000 7f3c00fb8700 1 -- 172.21.15.44:0/2792591479 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f3be4039050 msgr2=0x7f3be403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:13.190 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.589+0000 7f3c00fb8700 1 --2- 172.21.15.44:0/2792591479 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f3be4039050 0x7f3be403b510 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0006fd0 tx=0x7f3bf0006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:13.190 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.589+0000 7f3c00fb8700 1 -- 172.21.15.44:0/2792591479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bfc106440 msgr2=0x7f3bfc1a2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:13.190 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.589+0000 7f3c00fb8700 1 --2- 172.21.15.44:0/2792591479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bfc106440 0x7f3bfc1a2170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0116a0 tx=0x7f3bec011ba0 comp rx=0 tx=0).stop 2024-04-21T22:59:13.190 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.589+0000 7f3c00fb8700 1 -- 172.21.15.44:0/2792591479 shutdown_connections 2024-04-21T22:59:13.190 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.589+0000 7f3c00fb8700 1 --2- 172.21.15.44:0/2792591479 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f3be4039050 0x7f3be403b510 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:13.190 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.589+0000 7f3c00fb8700 1 --2- 172.21.15.44:0/2792591479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bfc106440 0x7f3bfc1a2170 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:13.190 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.589+0000 7f3c00fb8700 1 -- 172.21.15.44:0/2792591479 >> 172.21.15.44:0/2792591479 conn(0x7f3bfc1020a0 msgr2=0x7f3bfc104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:13.190 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.589+0000 7f3c00fb8700 1 -- 172.21.15.44:0/2792591479 shutdown_connections 2024-04-21T22:59:13.191 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:12.589+0000 7f3c00fb8700 1 -- 172.21.15.44:0/2792591479 wait complete. 2024-04-21T22:59:13.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:13 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' 2024-04-21T22:59:13.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:13 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2792591479' entity='client.admin' 2024-04-21T22:59:14.453 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.584+0000 7fce0f2d3700 1 Processor -- start 2024-04-21T22:59:14.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.585+0000 7fce0f2d3700 1 -- start start 2024-04-21T22:59:14.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.585+0000 7fce0f2d3700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce08108630 0x7fce0810aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:14.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.585+0000 7fce0f2d3700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fce0807be90 con 0x7fce08108630 2024-04-21T22:59:14.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.585+0000 7fce0d06f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce08108630 0x7fce0810aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:14.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.585+0000 7fce0d06f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce08108630 0x7fce0810aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48608/0 (socket says 172.21.15.44:48608) 2024-04-21T22:59:14.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.585+0000 7fce0d06f700 1 -- 172.21.15.44:0/3584208377 learned_addr learned my addr 172.21.15.44:0/3584208377 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:14.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.586+0000 7fce0d06f700 1 -- 172.21.15.44:0/3584208377 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce0810af60 con 0x7fce08108630 2024-04-21T22:59:14.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.586+0000 7fce0d06f700 1 --2- 172.21.15.44:0/3584208377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce08108630 0x7fce0810aa20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fce040098a0 tx=0x7fce04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3680e896b8493bdb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:14.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.586+0000 7fcdfbfff700 1 -- 172.21.15.44:0/3584208377 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce04004160 con 0x7fce08108630 2024-04-21T22:59:14.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.586+0000 7fcdfbfff700 1 -- 172.21.15.44:0/3584208377 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce04004300 con 0x7fce08108630 2024-04-21T22:59:14.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.586+0000 7fcdfbfff700 1 -- 172.21.15.44:0/3584208377 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce04011b20 con 0x7fce08108630 2024-04-21T22:59:14.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.587+0000 7fce0f2d3700 1 -- 172.21.15.44:0/3584208377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce08108630 msgr2=0x7fce0810aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:14.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.587+0000 7fce0f2d3700 1 --2- 172.21.15.44:0/3584208377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce08108630 0x7fce0810aa20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fce040098a0 tx=0x7fce04009bb0 comp rx=0 tx=0).stop 2024-04-21T22:59:14.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.587+0000 7fce0f2d3700 1 -- 172.21.15.44:0/3584208377 shutdown_connections 2024-04-21T22:59:14.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.587+0000 7fce0f2d3700 1 --2- 172.21.15.44:0/3584208377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce08108630 0x7fce0810aa20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:14.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.587+0000 7fce0f2d3700 1 -- 172.21.15.44:0/3584208377 >> 172.21.15.44:0/3584208377 conn(0x7fce08102000 msgr2=0x7fce08104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:14.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.587+0000 7fce0f2d3700 1 -- 172.21.15.44:0/3584208377 shutdown_connections 2024-04-21T22:59:14.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.587+0000 7fce0f2d3700 1 -- 172.21.15.44:0/3584208377 wait complete. 2024-04-21T22:59:14.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.587+0000 7fce0f2d3700 1 Processor -- start 2024-04-21T22:59:14.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.588+0000 7fce0f2d3700 1 -- start start 2024-04-21T22:59:14.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.588+0000 7fce0f2d3700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce08108630 0x7fce0819bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:14.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.588+0000 7fce0f2d3700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fce0810bfc0 con 0x7fce08108630 2024-04-21T22:59:14.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.588+0000 7fce0d06f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce08108630 0x7fce0819bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:14.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.588+0000 7fce0d06f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce08108630 0x7fce0819bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48614/0 (socket says 172.21.15.44:48614) 2024-04-21T22:59:14.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.588+0000 7fce0d06f700 1 -- 172.21.15.44:0/713686477 learned_addr learned my addr 172.21.15.44:0/713686477 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:14.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.588+0000 7fce0d06f700 1 -- 172.21.15.44:0/713686477 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce0819c020 con 0x7fce08108630 2024-04-21T22:59:14.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.588+0000 7fce0d06f700 1 --2- 172.21.15.44:0/713686477 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce08108630 0x7fce0819bae0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fce040117e0 tx=0x7fce04011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:14.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.589+0000 7fcdfa7fc700 1 -- 172.21.15.44:0/713686477 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce04003ba0 con 0x7fce08108630 2024-04-21T22:59:14.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.589+0000 7fcdfa7fc700 1 -- 172.21.15.44:0/713686477 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce0401a910 con 0x7fce08108630 2024-04-21T22:59:14.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.589+0000 7fce0f2d3700 1 -- 172.21.15.44:0/713686477 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce0819c2b0 con 0x7fce08108630 2024-04-21T22:59:14.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.589+0000 7fcdfa7fc700 1 -- 172.21.15.44:0/713686477 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce040248f0 con 0x7fce08108630 2024-04-21T22:59:14.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.589+0000 7fce0f2d3700 1 -- 172.21.15.44:0/713686477 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce0819c710 con 0x7fce08108630 2024-04-21T22:59:14.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.589+0000 7fcdfa7fc700 1 -- 172.21.15.44:0/713686477 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fce04022020 con 0x7fce08108630 2024-04-21T22:59:14.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.590+0000 7fcdfa7fc700 1 --2- 172.21.15.44:0/713686477 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fcdf4039050 0x7fcdf403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:14.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.590+0000 7fcdfa7fc700 1 -- 172.21.15.44:0/713686477 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fce0404e900 con 0x7fce08108630 2024-04-21T22:59:14.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.590+0000 7fce0c86e700 1 --2- 172.21.15.44:0/713686477 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fcdf4039050 0x7fcdf403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:14.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.590+0000 7fce0f2d3700 1 -- 172.21.15.44:0/713686477 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce08063890 con 0x7fce08108630 2024-04-21T22:59:14.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.590+0000 7fce0c86e700 1 --2- 172.21.15.44:0/713686477 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fcdf4039050 0x7fcdf403b510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc006fd0 tx=0x7fcdfc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:14.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.596+0000 7fcdfa7fc700 1 -- 172.21.15.44:0/713686477 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fce04020c10 con 0x7fce08108630 2024-04-21T22:59:14.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.812+0000 7fce0f2d3700 1 -- 172.21.15.44:0/713686477 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fce0819f040 con 0x7fce08108630 2024-04-21T22:59:14.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.815+0000 7fcdfa7fc700 1 -- 172.21.15.44:0/713686477 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fce0404e660 con 0x7fce08108630 2024-04-21T22:59:14.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.817+0000 7fce0f2d3700 1 -- 172.21.15.44:0/713686477 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fcdf4039050 msgr2=0x7fcdf403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:14.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.817+0000 7fce0f2d3700 1 --2- 172.21.15.44:0/713686477 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fcdf4039050 0x7fcdf403b510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc006fd0 tx=0x7fcdfc006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:14.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.817+0000 7fce0f2d3700 1 -- 172.21.15.44:0/713686477 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce08108630 msgr2=0x7fce0819bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:14.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.817+0000 7fce0f2d3700 1 --2- 172.21.15.44:0/713686477 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce08108630 0x7fce0819bae0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fce040117e0 tx=0x7fce04011ba0 comp rx=0 tx=0).stop 2024-04-21T22:59:14.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.817+0000 7fce0f2d3700 1 -- 172.21.15.44:0/713686477 shutdown_connections 2024-04-21T22:59:14.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.817+0000 7fce0f2d3700 1 --2- 172.21.15.44:0/713686477 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fcdf4039050 0x7fcdf403b510 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:14.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.817+0000 7fce0f2d3700 1 --2- 172.21.15.44:0/713686477 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fce08108630 0x7fce0819bae0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:14.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.817+0000 7fce0f2d3700 1 -- 172.21.15.44:0/713686477 >> 172.21.15.44:0/713686477 conn(0x7fce08102000 msgr2=0x7fce08102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:14.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.817+0000 7fce0f2d3700 1 -- 172.21.15.44:0/713686477 shutdown_connections 2024-04-21T22:59:14.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:13.817+0000 7fce0f2d3700 1 -- 172.21.15.44:0/713686477 wait complete. 2024-04-21T22:59:14.457 INFO:teuthology.orchestra.run.smithi044.stdout:Enabling the dashboard module... 2024-04-21T22:59:14.848 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:14 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/713686477' entity='client.admin' 2024-04-21T22:59:16.145 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:15 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1086113985' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-21T22:59:16.145 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:15 smithi044 ceph-mon[27934]: from='mgr.14120 172.21.15.44:0/1465201400' entity='mgr.x' 2024-04-21T22:59:16.145 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:15 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: ignoring --setuser ceph since I am not root 2024-04-21T22:59:16.145 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:15 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: ignoring --setgroup ceph since I am not root 2024-04-21T22:59:16.146 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:16 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:16.009+0000 7f3968fdc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T22:59:16.435 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:16 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:16.201+0000 7f3968fdc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T22:59:16.436 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:16 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:16.307+0000 7f3968fdc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T22:59:16.453 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.876+0000 7f3e9d570700 1 Processor -- start 2024-04-21T22:59:16.453 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.876+0000 7f3e9d570700 1 -- start start 2024-04-21T22:59:16.453 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.877+0000 7f3e9d570700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e98106460 0x7f3e98106880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:16.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.877+0000 7f3e9d570700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9807be50 con 0x7f3e98106460 2024-04-21T22:59:16.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.877+0000 7f3e96ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e98106460 0x7f3e98106880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:16.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.877+0000 7f3e96ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e98106460 0x7f3e98106880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48624/0 (socket says 172.21.15.44:48624) 2024-04-21T22:59:16.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.877+0000 7f3e96ffd700 1 -- 172.21.15.44:0/1574500242 learned_addr learned my addr 172.21.15.44:0/1574500242 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:16.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.877+0000 7f3e96ffd700 1 -- 172.21.15.44:0/1574500242 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e98106dc0 con 0x7f3e98106460 2024-04-21T22:59:16.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.877+0000 7f3e96ffd700 1 --2- 172.21.15.44:0/1574500242 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e98106460 0x7f3e98106880 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3e880098a0 tx=0x7f3e88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef707f0747895669 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:16.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.877+0000 7f3e95ffb700 1 -- 172.21.15.44:0/1574500242 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e88004160 con 0x7f3e98106460 2024-04-21T22:59:16.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.877+0000 7f3e95ffb700 1 -- 172.21.15.44:0/1574500242 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e88004300 con 0x7f3e98106460 2024-04-21T22:59:16.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.878+0000 7f3e95ffb700 1 -- 172.21.15.44:0/1574500242 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e88011b20 con 0x7f3e98106460 2024-04-21T22:59:16.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.878+0000 7f3e9d570700 1 -- 172.21.15.44:0/1574500242 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e98106460 msgr2=0x7f3e98106880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:16.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.878+0000 7f3e9d570700 1 --2- 172.21.15.44:0/1574500242 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e98106460 0x7f3e98106880 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3e880098a0 tx=0x7f3e88009bb0 comp rx=0 tx=0).stop 2024-04-21T22:59:16.454 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.878+0000 7f3e9d570700 1 -- 172.21.15.44:0/1574500242 shutdown_connections 2024-04-21T22:59:16.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.878+0000 7f3e9d570700 1 --2- 172.21.15.44:0/1574500242 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e98106460 0x7f3e98106880 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:16.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.878+0000 7f3e9d570700 1 -- 172.21.15.44:0/1574500242 >> 172.21.15.44:0/1574500242 conn(0x7f3e98102000 msgr2=0x7f3e98104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:16.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.878+0000 7f3e9d570700 1 -- 172.21.15.44:0/1574500242 shutdown_connections 2024-04-21T22:59:16.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.878+0000 7f3e9d570700 1 -- 172.21.15.44:0/1574500242 wait complete. 2024-04-21T22:59:16.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.879+0000 7f3e9d570700 1 Processor -- start 2024-04-21T22:59:16.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.879+0000 7f3e9d570700 1 -- start start 2024-04-21T22:59:16.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.879+0000 7f3e9d570700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e98106460 0x7f3e9819dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:16.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.879+0000 7f3e9d570700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3e98107e90 con 0x7f3e98106460 2024-04-21T22:59:16.455 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.879+0000 7f3e96ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e98106460 0x7f3e9819dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:16.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.879+0000 7f3e96ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e98106460 0x7f3e9819dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48628/0 (socket says 172.21.15.44:48628) 2024-04-21T22:59:16.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.879+0000 7f3e96ffd700 1 -- 172.21.15.44:0/1086113985 learned_addr learned my addr 172.21.15.44:0/1086113985 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:16.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.879+0000 7f3e96ffd700 1 -- 172.21.15.44:0/1086113985 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9819e220 con 0x7f3e98106460 2024-04-21T22:59:16.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.880+0000 7f3e96ffd700 1 --2- 172.21.15.44:0/1086113985 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e98106460 0x7f3e9819dce0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3e880117e0 tx=0x7f3e88011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:16.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.880+0000 7f3e7ffff700 1 -- 172.21.15.44:0/1086113985 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e88003ba0 con 0x7f3e98106460 2024-04-21T22:59:16.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.880+0000 7f3e7ffff700 1 -- 172.21.15.44:0/1086113985 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e8801a910 con 0x7f3e98106460 2024-04-21T22:59:16.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.880+0000 7f3e9d570700 1 -- 172.21.15.44:0/1086113985 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e9819e4b0 con 0x7f3e98106460 2024-04-21T22:59:16.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.880+0000 7f3e7ffff700 1 -- 172.21.15.44:0/1086113985 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e880248f0 con 0x7f3e98106460 2024-04-21T22:59:16.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.880+0000 7f3e9d570700 1 -- 172.21.15.44:0/1086113985 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e9819e910 con 0x7f3e98106460 2024-04-21T22:59:16.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.881+0000 7f3e7ffff700 1 -- 172.21.15.44:0/1086113985 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f3e88022020 con 0x7f3e98106460 2024-04-21T22:59:16.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.881+0000 7f3e9d570700 1 -- 172.21.15.44:0/1086113985 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e98063890 con 0x7f3e98106460 2024-04-21T22:59:16.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.881+0000 7f3e7ffff700 1 --2- 172.21.15.44:0/1086113985 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f3e80039050 0x7f3e8003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:16.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.881+0000 7f3e7ffff700 1 -- 172.21.15.44:0/1086113985 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3e8804d8b0 con 0x7f3e98106460 2024-04-21T22:59:16.456 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.881+0000 7f3e967fc700 1 --2- 172.21.15.44:0/1086113985 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f3e80039050 0x7f3e8003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:16.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.882+0000 7f3e967fc700 1 --2- 172.21.15.44:0/1086113985 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f3e80039050 0x7f3e8003b510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c006fd0 tx=0x7f3e8c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:16.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:14.886+0000 7f3e7ffff700 1 -- 172.21.15.44:0/1086113985 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3e8800f2e0 con 0x7f3e98106460 2024-04-21T22:59:16.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:15.158+0000 7f3e9d570700 1 -- 172.21.15.44:0/1086113985 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f3e98062e40 con 0x7f3e98106460 2024-04-21T22:59:16.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:15.818+0000 7f3e7ffff700 1 -- 172.21.15.44:0/1086113985 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f3e8802d3f0 con 0x7f3e98106460 2024-04-21T22:59:16.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:15.821+0000 7f3e7ffff700 1 -- 172.21.15.44:0/1086113985 <== mon.0 v2:172.21.15.44:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f3e8804fd10 con 0x7f3e98106460 2024-04-21T22:59:16.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:15.821+0000 7f3e9d570700 1 -- 172.21.15.44:0/1086113985 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f3e80039050 msgr2=0x7f3e8003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:16.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:15.821+0000 7f3e9d570700 1 --2- 172.21.15.44:0/1086113985 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f3e80039050 0x7f3e8003b510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c006fd0 tx=0x7f3e8c006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:16.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:15.821+0000 7f3e9d570700 1 -- 172.21.15.44:0/1086113985 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e98106460 msgr2=0x7f3e9819dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:16.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:15.821+0000 7f3e9d570700 1 --2- 172.21.15.44:0/1086113985 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e98106460 0x7f3e9819dce0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3e880117e0 tx=0x7f3e88011ba0 comp rx=0 tx=0).stop 2024-04-21T22:59:16.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:15.821+0000 7f3e9d570700 1 -- 172.21.15.44:0/1086113985 shutdown_connections 2024-04-21T22:59:16.457 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:15.821+0000 7f3e9d570700 1 --2- 172.21.15.44:0/1086113985 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f3e80039050 0x7f3e8003b510 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:16.458 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:15.821+0000 7f3e9d570700 1 --2- 172.21.15.44:0/1086113985 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3e98106460 0x7f3e9819dce0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:16.458 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:15.821+0000 7f3e9d570700 1 -- 172.21.15.44:0/1086113985 >> 172.21.15.44:0/1086113985 conn(0x7f3e98102000 msgr2=0x7f3e98194920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:16.458 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:15.822+0000 7f3e9d570700 1 -- 172.21.15.44:0/1086113985 shutdown_connections 2024-04-21T22:59:16.458 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:15.822+0000 7f3e9d570700 1 -- 172.21.15.44:0/1086113985 wait complete. 2024-04-21T22:59:16.723 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:16 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:16.632+0000 7f3968fdc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T22:59:16.723 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:16 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:16.723+0000 7f3968fdc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T22:59:17.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:16 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1086113985' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-21T22:59:17.077 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:16 smithi044 ceph-mon[27934]: mgrmap e9: x(active, since 19s) 2024-04-21T22:59:17.760 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-04-21T22:59:17.760 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-21T22:59:17.760 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T22:59:17.760 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-21T22:59:17.760 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-21T22:59:17.760 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-04-21T22:59:17.760 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.870+0000 7fb44c595700 1 Processor -- start 2024-04-21T22:59:17.761 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.870+0000 7fb44c595700 1 -- start start 2024-04-21T22:59:17.761 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.871+0000 7fb44c595700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb44407dea0 0x7fb44407c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:17.761 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.871+0000 7fb44c595700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb44407c8d0 con 0x7fb44407dea0 2024-04-21T22:59:17.761 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.871+0000 7fb44a331700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb44407dea0 0x7fb44407c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:17.761 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.871+0000 7fb44a331700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb44407dea0 0x7fb44407c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48666/0 (socket says 172.21.15.44:48666) 2024-04-21T22:59:17.761 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.871+0000 7fb44a331700 1 -- 172.21.15.44:0/3831047942 learned_addr learned my addr 172.21.15.44:0/3831047942 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:17.761 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.871+0000 7fb44a331700 1 -- 172.21.15.44:0/3831047942 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb44407ca50 con 0x7fb44407dea0 2024-04-21T22:59:17.761 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.872+0000 7fb44a331700 1 --2- 172.21.15.44:0/3831047942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb44407dea0 0x7fb44407c300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb4400098a0 tx=0x7fb440009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a73a7ce544f972a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:17.761 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.872+0000 7fb44932f700 1 -- 172.21.15.44:0/3831047942 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb440004160 con 0x7fb44407dea0 2024-04-21T22:59:17.761 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.872+0000 7fb44932f700 1 -- 172.21.15.44:0/3831047942 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb440004300 con 0x7fb44407dea0 2024-04-21T22:59:17.761 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.872+0000 7fb44932f700 1 -- 172.21.15.44:0/3831047942 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb440011b20 con 0x7fb44407dea0 2024-04-21T22:59:17.761 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.872+0000 7fb44c595700 1 -- 172.21.15.44:0/3831047942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb44407dea0 msgr2=0x7fb44407c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:17.761 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.872+0000 7fb44c595700 1 --2- 172.21.15.44:0/3831047942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb44407dea0 0x7fb44407c300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb4400098a0 tx=0x7fb440009bb0 comp rx=0 tx=0).stop 2024-04-21T22:59:17.762 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.873+0000 7fb44c595700 1 -- 172.21.15.44:0/3831047942 shutdown_connections 2024-04-21T22:59:17.762 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.873+0000 7fb44c595700 1 --2- 172.21.15.44:0/3831047942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb44407dea0 0x7fb44407c300 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:17.762 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.873+0000 7fb44c595700 1 -- 172.21.15.44:0/3831047942 >> 172.21.15.44:0/3831047942 conn(0x7fb4441031a0 msgr2=0x7fb4441055e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:17.762 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.873+0000 7fb44c595700 1 -- 172.21.15.44:0/3831047942 shutdown_connections 2024-04-21T22:59:17.762 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.873+0000 7fb44c595700 1 -- 172.21.15.44:0/3831047942 wait complete. 2024-04-21T22:59:17.762 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.873+0000 7fb44c595700 1 Processor -- start 2024-04-21T22:59:17.762 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.873+0000 7fb44c595700 1 -- start start 2024-04-21T22:59:17.762 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.873+0000 7fb44c595700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb44407dea0 0x7fb44419dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:17.762 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.873+0000 7fb44c595700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb44410e280 con 0x7fb44407dea0 2024-04-21T22:59:17.762 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.874+0000 7fb44a331700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb44407dea0 0x7fb44419dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:17.762 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.874+0000 7fb44a331700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb44407dea0 0x7fb44419dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48674/0 (socket says 172.21.15.44:48674) 2024-04-21T22:59:17.762 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.874+0000 7fb44a331700 1 -- 172.21.15.44:0/2727930693 learned_addr learned my addr 172.21.15.44:0/2727930693 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:17.762 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.874+0000 7fb44a331700 1 -- 172.21.15.44:0/2727930693 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb44419e1f0 con 0x7fb44407dea0 2024-04-21T22:59:17.762 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.874+0000 7fb44a331700 1 --2- 172.21.15.44:0/2727930693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb44407dea0 0x7fb44419dcb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb4400117e0 tx=0x7fb440011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:17.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.874+0000 7fb43b7fe700 1 -- 172.21.15.44:0/2727930693 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb440003ba0 con 0x7fb44407dea0 2024-04-21T22:59:17.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.874+0000 7fb44c595700 1 -- 172.21.15.44:0/2727930693 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb44419e480 con 0x7fb44407dea0 2024-04-21T22:59:17.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.874+0000 7fb43b7fe700 1 -- 172.21.15.44:0/2727930693 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb44001a910 con 0x7fb44407dea0 2024-04-21T22:59:17.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.874+0000 7fb43b7fe700 1 -- 172.21.15.44:0/2727930693 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb4400248f0 con 0x7fb44407dea0 2024-04-21T22:59:17.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.874+0000 7fb44c595700 1 -- 172.21.15.44:0/2727930693 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb44419e8e0 con 0x7fb44407dea0 2024-04-21T22:59:17.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.875+0000 7fb43b7fe700 1 -- 172.21.15.44:0/2727930693 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7fb440022020 con 0x7fb44407dea0 2024-04-21T22:59:17.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.876+0000 7fb44c595700 1 -- 172.21.15.44:0/2727930693 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb444063890 con 0x7fb44407dea0 2024-04-21T22:59:17.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.876+0000 7fb43b7fe700 1 --2- 172.21.15.44:0/2727930693 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fb4300390a0 0x7fb43003b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:17.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.876+0000 7fb43b7fe700 1 -- 172.21.15.44:0/2727930693 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb44004da70 con 0x7fb44407dea0 2024-04-21T22:59:17.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.876+0000 7fb449b30700 1 -- 172.21.15.44:0/2727930693 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fb4300390a0 msgr2=0x7fb43003b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4238737803 2024-04-21T22:59:17.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.876+0000 7fb449b30700 1 --2- 172.21.15.44:0/2727930693 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fb4300390a0 0x7fb43003b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T22:59:17.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:16.881+0000 7fb43b7fe700 1 -- 172.21.15.44:0/2727930693 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb44002cdc0 con 0x7fb44407dea0 2024-04-21T22:59:17.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:17.077+0000 7fb449b30700 1 -- 172.21.15.44:0/2727930693 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fb4300390a0 msgr2=0x7fb43003b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4238737803 2024-04-21T22:59:17.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:17.077+0000 7fb449b30700 1 --2- 172.21.15.44:0/2727930693 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fb4300390a0 0x7fb43003b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T22:59:17.763 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:17.226+0000 7fb44c595700 1 -- 172.21.15.44:0/2727930693 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fb4441a1720 con 0x7fb44407dea0 2024-04-21T22:59:17.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:17.226+0000 7fb43b7fe700 1 -- 172.21.15.44:0/2727930693 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fb44002dc40 con 0x7fb44407dea0 2024-04-21T22:59:17.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:17.230+0000 7fb44c595700 1 -- 172.21.15.44:0/2727930693 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fb4300390a0 msgr2=0x7fb43003b560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T22:59:17.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:17.230+0000 7fb44c595700 1 --2- 172.21.15.44:0/2727930693 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fb4300390a0 0x7fb43003b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:17.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:17.230+0000 7fb44c595700 1 -- 172.21.15.44:0/2727930693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb44407dea0 msgr2=0x7fb44419dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:17.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:17.230+0000 7fb44c595700 1 --2- 172.21.15.44:0/2727930693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb44407dea0 0x7fb44419dcb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb4400117e0 tx=0x7fb440011ba0 comp rx=0 tx=0).stop 2024-04-21T22:59:17.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:17.230+0000 7fb44c595700 1 -- 172.21.15.44:0/2727930693 shutdown_connections 2024-04-21T22:59:17.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:17.230+0000 7fb44c595700 1 --2- 172.21.15.44:0/2727930693 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7fb4300390a0 0x7fb43003b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:17.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:17.230+0000 7fb44c595700 1 --2- 172.21.15.44:0/2727930693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb44407dea0 0x7fb44419dcb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:17.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:17.230+0000 7fb44c595700 1 -- 172.21.15.44:0/2727930693 >> 172.21.15.44:0/2727930693 conn(0x7fb4441031a0 msgr2=0x7fb444103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:17.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:17.230+0000 7fb44c595700 1 -- 172.21.15.44:0/2727930693 shutdown_connections 2024-04-21T22:59:17.764 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:17.230+0000 7fb44c595700 1 -- 172.21.15.44:0/2727930693 wait complete. 2024-04-21T22:59:17.764 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for the mgr to restart... 2024-04-21T22:59:17.764 INFO:teuthology.orchestra.run.smithi044.stdout:Waiting for mgr epoch 9... 2024-04-21T22:59:18.095 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:17 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2727930693' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-21T22:59:18.096 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:18 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:18.006+0000 7f3968fdc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T22:59:18.368 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:18 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:18.189+0000 7f3968fdc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T22:59:18.368 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:18 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:18.367+0000 7f3968fdc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T22:59:18.935 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:18 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:18.666+0000 7f3968fdc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T22:59:18.936 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:18 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:18.762+0000 7f3968fdc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T22:59:19.435 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:18 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:18.972+0000 7f3968fdc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T22:59:19.436 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:19 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:19.171+0000 7f3968fdc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T22:59:19.938 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:19 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:19.796+0000 7f3968fdc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T22:59:20.544 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:20 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:20.235+0000 7f3968fdc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T22:59:20.545 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:20 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:20.335+0000 7f3968fdc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T22:59:20.545 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:20 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:20.544+0000 7f3968fdc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T22:59:21.435 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:21 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:21.090+0000 7f3968fdc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T22:59:21.436 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:21 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:21.186+0000 7f3968fdc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T22:59:21.803 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:21 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:21.441+0000 7f3968fdc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T22:59:21.803 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:21 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:21.581+0000 7f3968fdc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T22:59:21.803 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:21 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:21.681+0000 7f3968fdc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T22:59:21.804 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:21 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:21.802+0000 7f3968fdc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T22:59:22.185 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:21 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:21.898+0000 7f3968fdc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T22:59:23.020 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:22 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:22.741+0000 7f3968fdc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T22:59:23.020 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 22:59:22 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T22:59:22.967+0000 7f3968fdc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T22:59:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:23 smithi044 ceph-mon[27934]: Active manager daemon x restarted 2024-04-21T22:59:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:23 smithi044 ceph-mon[27934]: Activating manager daemon x 2024-04-21T22:59:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:23 smithi044 ceph-mon[27934]: osdmap e3: 0 total, 0 up, 0 in 2024-04-21T22:59:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:23 smithi044 ceph-mon[27934]: mgrmap e10: x(active, starting, since 0.00329496s) 2024-04-21T22:59:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T22:59:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T22:59:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T22:59:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T22:59:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T22:59:23.437 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:23 smithi044 ceph-mon[27934]: Manager daemon x is now available 2024-04-21T22:59:23.437 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T22:59:23.437 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T22:59:24.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T22:59:24.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:24 smithi044 ceph-mon[27934]: [21/Apr/2024:22:59:23] ENGINE Bus STARTING 2024-04-21T22:59:24.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:24 smithi044 ceph-mon[27934]: [21/Apr/2024:22:59:23] ENGINE Serving on http://172.21.15.44:8765 2024-04-21T22:59:24.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:24 smithi044 ceph-mon[27934]: mgrmap e11: x(active, since 1.00626s) 2024-04-21T22:59:24.503 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout { 2024-04-21T22:59:24.503 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-21T22:59:24.503 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-21T22:59:24.503 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout } 2024-04-21T22:59:24.503 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.199+0000 7f433b3a3700 1 Processor -- start 2024-04-21T22:59:24.503 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.199+0000 7f433b3a3700 1 -- start start 2024-04-21T22:59:24.503 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.199+0000 7f433b3a3700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4334106a80 0x7f4334106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:24.503 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.199+0000 7f433b3a3700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4334107470 con 0x7f4334106a80 2024-04-21T22:59:24.504 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.199+0000 7f433913f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4334106a80 0x7f4334106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:24.504 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.199+0000 7f433913f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4334106a80 0x7f4334106ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48690/0 (socket says 172.21.15.44:48690) 2024-04-21T22:59:24.504 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.199+0000 7f433913f700 1 -- 172.21.15.44:0/1575038316 learned_addr learned my addr 172.21.15.44:0/1575038316 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:24.504 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.200+0000 7f433913f700 1 -- 172.21.15.44:0/1575038316 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4334107c80 con 0x7f4334106a80 2024-04-21T22:59:24.504 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.200+0000 7f433913f700 1 --2- 172.21.15.44:0/1575038316 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4334106a80 0x7f4334106ea0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f43300098a0 tx=0x7f4330009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b770381e732a9dab server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:24.504 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.200+0000 7f432bfff700 1 -- 172.21.15.44:0/1575038316 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4330004160 con 0x7f4334106a80 2024-04-21T22:59:24.504 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.200+0000 7f432bfff700 1 -- 172.21.15.44:0/1575038316 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4330004300 con 0x7f4334106a80 2024-04-21T22:59:24.504 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.200+0000 7f432bfff700 1 -- 172.21.15.44:0/1575038316 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4330011b20 con 0x7f4334106a80 2024-04-21T22:59:24.504 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.201+0000 7f433b3a3700 1 -- 172.21.15.44:0/1575038316 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4334106a80 msgr2=0x7f4334106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:24.504 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.201+0000 7f433b3a3700 1 --2- 172.21.15.44:0/1575038316 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4334106a80 0x7f4334106ea0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f43300098a0 tx=0x7f4330009bb0 comp rx=0 tx=0).stop 2024-04-21T22:59:24.504 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.201+0000 7f433b3a3700 1 -- 172.21.15.44:0/1575038316 shutdown_connections 2024-04-21T22:59:24.504 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.201+0000 7f433b3a3700 1 --2- 172.21.15.44:0/1575038316 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4334106a80 0x7f4334106ea0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:24.504 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.201+0000 7f433b3a3700 1 -- 172.21.15.44:0/1575038316 >> 172.21.15.44:0/1575038316 conn(0x7f4334102000 msgr2=0x7f4334104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:24.504 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.201+0000 7f433b3a3700 1 -- 172.21.15.44:0/1575038316 shutdown_connections 2024-04-21T22:59:24.505 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.201+0000 7f433b3a3700 1 -- 172.21.15.44:0/1575038316 wait complete. 2024-04-21T22:59:24.505 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.201+0000 7f433b3a3700 1 Processor -- start 2024-04-21T22:59:24.505 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.201+0000 7f433b3a3700 1 -- start start 2024-04-21T22:59:24.505 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.202+0000 7f433b3a3700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4334106a80 0x7f433419dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:24.505 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.202+0000 7f433b3a3700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f433410b030 con 0x7f4334106a80 2024-04-21T22:59:24.505 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.202+0000 7f433913f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4334106a80 0x7f433419dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:24.505 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.202+0000 7f433913f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4334106a80 0x7f433419dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48700/0 (socket says 172.21.15.44:48700) 2024-04-21T22:59:24.505 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.202+0000 7f433913f700 1 -- 172.21.15.44:0/2093568682 learned_addr learned my addr 172.21.15.44:0/2093568682 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:24.505 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.202+0000 7f433913f700 1 -- 172.21.15.44:0/2093568682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f433419e280 con 0x7f4334106a80 2024-04-21T22:59:24.505 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.202+0000 7f433913f700 1 --2- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4334106a80 0x7f433419dd40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f43300117e0 tx=0x7f4330011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:24.505 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.202+0000 7f432a7fc700 1 -- 172.21.15.44:0/2093568682 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4330003bf0 con 0x7f4334106a80 2024-04-21T22:59:24.505 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.202+0000 7f432a7fc700 1 -- 172.21.15.44:0/2093568682 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f433001a910 con 0x7f4334106a80 2024-04-21T22:59:24.505 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.202+0000 7f432a7fc700 1 -- 172.21.15.44:0/2093568682 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43300249a0 con 0x7f4334106a80 2024-04-21T22:59:24.505 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.202+0000 7f433b3a3700 1 -- 172.21.15.44:0/2093568682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f433419e510 con 0x7f4334106a80 2024-04-21T22:59:24.506 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.203+0000 7f433b3a3700 1 -- 172.21.15.44:0/2093568682 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f433419e970 con 0x7f4334106a80 2024-04-21T22:59:24.506 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.203+0000 7f432a7fc700 1 -- 172.21.15.44:0/2093568682 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f4330022020 con 0x7f4334106a80 2024-04-21T22:59:24.506 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.204+0000 7f432a7fc700 1 --2- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f43200390b0 0x7f432003b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:24.506 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.204+0000 7f432a7fc700 1 -- 172.21.15.44:0/2093568682 --> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f432003bc80 con 0x7f43200390b0 2024-04-21T22:59:24.506 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.204+0000 7f432a7fc700 1 -- 172.21.15.44:0/2093568682 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f433004d3e0 con 0x7f4334106a80 2024-04-21T22:59:24.506 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.204+0000 7f433893e700 1 -- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f43200390b0 msgr2=0x7f432003b570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4238737803 2024-04-21T22:59:24.506 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.204+0000 7f433893e700 1 --2- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f43200390b0 0x7f432003b570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T22:59:24.506 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.405+0000 7f433893e700 1 -- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f43200390b0 msgr2=0x7f432003b570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4238737803 2024-04-21T22:59:24.506 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.405+0000 7f433893e700 1 --2- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f43200390b0 0x7f432003b570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T22:59:24.506 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.806+0000 7f433893e700 1 -- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f43200390b0 msgr2=0x7f432003b570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4238737803 2024-04-21T22:59:24.506 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:18.806+0000 7f433893e700 1 --2- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f43200390b0 0x7f432003b570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-21T22:59:24.506 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:19.607+0000 7f433893e700 1 -- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f43200390b0 msgr2=0x7f432003b570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4238737803 2024-04-21T22:59:24.506 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:19.607+0000 7f433893e700 1 --2- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f43200390b0 0x7f432003b570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-21T22:59:24.507 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:21.208+0000 7f433893e700 1 -- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f43200390b0 msgr2=0x7f432003b570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/4238737803 2024-04-21T22:59:24.507 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:21.208+0000 7f433893e700 1 --2- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f43200390b0 0x7f432003b570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-21T22:59:24.507 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:22.970+0000 7f432a7fc700 1 -- 172.21.15.44:0/2093568682 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7f4330016ca0 con 0x7f4334106a80 2024-04-21T22:59:24.507 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:22.970+0000 7f432a7fc700 1 -- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f43200390b0 msgr2=0x7f432003b570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T22:59:24.507 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:22.970+0000 7f432a7fc700 1 --2- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f43200390b0 0x7f432003b570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:24.507 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:23.972+0000 7f432a7fc700 1 -- 172.21.15.44:0/2093568682 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f4330016920 con 0x7f4334106a80 2024-04-21T22:59:24.507 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:23.972+0000 7f432a7fc700 1 --2- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f432003cb70 0x7f432003ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:24.507 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:23.972+0000 7f432a7fc700 1 -- 172.21.15.44:0/2093568682 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f432003bc80 con 0x7f432003cb70 2024-04-21T22:59:24.507 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:23.972+0000 7f433893e700 1 --2- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f432003cb70 0x7f432003ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:24.507 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:23.973+0000 7f433893e700 1 --2- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f432003cb70 0x7f432003ef60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4324003a10 tx=0x7f4324009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:24.507 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:23.974+0000 7f432a7fc700 1 -- 172.21.15.44:0/2093568682 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f432003bc80 con 0x7f432003cb70 2024-04-21T22:59:24.507 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:23.980+0000 7f433b3a3700 1 -- 172.21.15.44:0/2093568682 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f4334063890 con 0x7f432003cb70 2024-04-21T22:59:24.507 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:23.981+0000 7f432a7fc700 1 -- 172.21.15.44:0/2093568682 <== mgr.14152 v2:172.21.15.44:6800/1417504446 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f4334063890 con 0x7f432003cb70 2024-04-21T22:59:24.507 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:23.981+0000 7f433b3a3700 1 -- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f432003cb70 msgr2=0x7f432003ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:24.508 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:23.981+0000 7f433b3a3700 1 --2- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f432003cb70 0x7f432003ef60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4324003a10 tx=0x7f4324009270 comp rx=0 tx=0).stop 2024-04-21T22:59:24.508 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:23.981+0000 7f433b3a3700 1 -- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4334106a80 msgr2=0x7f433419dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:24.508 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:23.981+0000 7f433b3a3700 1 --2- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4334106a80 0x7f433419dd40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f43300117e0 tx=0x7f4330011c10 comp rx=0 tx=0).stop 2024-04-21T22:59:24.508 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:23.981+0000 7f433b3a3700 1 -- 172.21.15.44:0/2093568682 shutdown_connections 2024-04-21T22:59:24.508 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:23.981+0000 7f433b3a3700 1 --2- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f432003cb70 0x7f432003ef60 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:24.508 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:23.981+0000 7f433b3a3700 1 --2- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4334106a80 0x7f433419dd40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:24.508 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:23.981+0000 7f433b3a3700 1 --2- 172.21.15.44:0/2093568682 >> [v2:172.21.15.44:6800/4238737803,v1:172.21.15.44:6801/4238737803] conn(0x7f43200390b0 0x7f432003b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:24.508 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:23.981+0000 7f433b3a3700 1 -- 172.21.15.44:0/2093568682 >> 172.21.15.44:0/2093568682 conn(0x7f4334102000 msgr2=0x7f4334102ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:24.508 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:23.981+0000 7f433b3a3700 1 -- 172.21.15.44:0/2093568682 shutdown_connections 2024-04-21T22:59:24.508 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:23.981+0000 7f433b3a3700 1 -- 172.21.15.44:0/2093568682 wait complete. 2024-04-21T22:59:24.508 INFO:teuthology.orchestra.run.smithi044.stdout:mgr epoch 9 is available 2024-04-21T22:59:24.508 INFO:teuthology.orchestra.run.smithi044.stdout:Generating a dashboard self-signed certificate... 2024-04-21T22:59:25.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:25 smithi044 ceph-mon[27934]: [21/Apr/2024:22:59:23] ENGINE Serving on https://172.21.15.44:7150 2024-04-21T22:59:25.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:25 smithi044 ceph-mon[27934]: [21/Apr/2024:22:59:23] ENGINE Bus STARTED 2024-04-21T22:59:25.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:25 smithi044 ceph-mon[27934]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-21T22:59:25.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:25 smithi044 ceph-mon[27934]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-21T22:59:26.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:26 smithi044 ceph-mon[27934]: mgrmap e12: x(active, since 2s) 2024-04-21T22:59:26.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:26 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:26.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:26 smithi044 ceph-mon[27934]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T22:59:26.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:26 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:26.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:26 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:26.225 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-21T22:59:26.225 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.227+0000 7f19dcc5f700 1 Processor -- start 2024-04-21T22:59:26.225 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.227+0000 7f19dcc5f700 1 -- start start 2024-04-21T22:59:26.225 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.227+0000 7f19dcc5f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8107bc0 0x7f19d8107fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:26.226 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.227+0000 7f19dcc5f700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f19d81085b0 con 0x7f19d8107bc0 2024-04-21T22:59:26.226 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.227+0000 7f19d659c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8107bc0 0x7f19d8107fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:26.226 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.227+0000 7f19d659c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8107bc0 0x7f19d8107fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60610/0 (socket says 172.21.15.44:60610) 2024-04-21T22:59:26.226 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.227+0000 7f19d659c700 1 -- 172.21.15.44:0/1186661108 learned_addr learned my addr 172.21.15.44:0/1186661108 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:26.226 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.228+0000 7f19d659c700 1 -- 172.21.15.44:0/1186661108 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d8108d70 con 0x7f19d8107bc0 2024-04-21T22:59:26.226 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.228+0000 7f19d659c700 1 --2- 172.21.15.44:0/1186661108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8107bc0 0x7f19d8107fe0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f19c80098a0 tx=0x7f19c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ddc1b470395fa46 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:26.226 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.228+0000 7f19d559a700 1 -- 172.21.15.44:0/1186661108 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f19c8004160 con 0x7f19d8107bc0 2024-04-21T22:59:26.226 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.228+0000 7f19d559a700 1 -- 172.21.15.44:0/1186661108 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19c8004300 con 0x7f19d8107bc0 2024-04-21T22:59:26.226 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.228+0000 7f19d559a700 1 -- 172.21.15.44:0/1186661108 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f19c8011b20 con 0x7f19d8107bc0 2024-04-21T22:59:26.226 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.229+0000 7f19dcc5f700 1 -- 172.21.15.44:0/1186661108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8107bc0 msgr2=0x7f19d8107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:26.226 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.229+0000 7f19dcc5f700 1 --2- 172.21.15.44:0/1186661108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8107bc0 0x7f19d8107fe0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f19c80098a0 tx=0x7f19c8009bb0 comp rx=0 tx=0).stop 2024-04-21T22:59:26.227 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.229+0000 7f19dcc5f700 1 -- 172.21.15.44:0/1186661108 shutdown_connections 2024-04-21T22:59:26.227 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.229+0000 7f19dcc5f700 1 --2- 172.21.15.44:0/1186661108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8107bc0 0x7f19d8107fe0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:26.227 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.229+0000 7f19dcc5f700 1 -- 172.21.15.44:0/1186661108 >> 172.21.15.44:0/1186661108 conn(0x7f19d8103120 msgr2=0x7f19d81055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:26.227 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.229+0000 7f19dcc5f700 1 -- 172.21.15.44:0/1186661108 shutdown_connections 2024-04-21T22:59:26.227 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.229+0000 7f19dcc5f700 1 -- 172.21.15.44:0/1186661108 wait complete. 2024-04-21T22:59:26.227 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.230+0000 7f19dcc5f700 1 Processor -- start 2024-04-21T22:59:26.227 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.230+0000 7f19dcc5f700 1 -- start start 2024-04-21T22:59:26.227 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.230+0000 7f19dcc5f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8107bc0 0x7f19d807db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:26.227 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.230+0000 7f19dcc5f700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f19d810c120 con 0x7f19d8107bc0 2024-04-21T22:59:26.227 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.230+0000 7f19d659c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8107bc0 0x7f19d807db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:26.227 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.230+0000 7f19d659c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8107bc0 0x7f19d807db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60616/0 (socket says 172.21.15.44:60616) 2024-04-21T22:59:26.227 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.230+0000 7f19d659c700 1 -- 172.21.15.44:0/3248230701 learned_addr learned my addr 172.21.15.44:0/3248230701 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:26.228 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.230+0000 7f19d659c700 1 -- 172.21.15.44:0/3248230701 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d807e0d0 con 0x7f19d8107bc0 2024-04-21T22:59:26.228 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.230+0000 7f19d659c700 1 --2- 172.21.15.44:0/3248230701 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8107bc0 0x7f19d807db90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f19c80116a0 tx=0x7f19c8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:26.228 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.231+0000 7f19c77fe700 1 -- 172.21.15.44:0/3248230701 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f19c8003a40 con 0x7f19d8107bc0 2024-04-21T22:59:26.228 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.231+0000 7f19c77fe700 1 -- 172.21.15.44:0/3248230701 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19c8003be0 con 0x7f19d8107bc0 2024-04-21T22:59:26.228 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.231+0000 7f19dcc5f700 1 -- 172.21.15.44:0/3248230701 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19d807c2c0 con 0x7f19d8107bc0 2024-04-21T22:59:26.228 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.231+0000 7f19c77fe700 1 -- 172.21.15.44:0/3248230701 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f19c801a720 con 0x7f19d8107bc0 2024-04-21T22:59:26.228 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.231+0000 7f19dcc5f700 1 -- 172.21.15.44:0/3248230701 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19d807c780 con 0x7f19d8107bc0 2024-04-21T22:59:26.228 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.231+0000 7f19c77fe700 1 -- 172.21.15.44:0/3248230701 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f19c801aa20 con 0x7f19d8107bc0 2024-04-21T22:59:26.228 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.232+0000 7f19c77fe700 1 --2- 172.21.15.44:0/3248230701 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19c0039050 0x7f19c003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:26.228 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.232+0000 7f19c77fe700 1 -- 172.21.15.44:0/3248230701 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f19c804d800 con 0x7f19d8107bc0 2024-04-21T22:59:26.228 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.232+0000 7f19d5d9b700 1 --2- 172.21.15.44:0/3248230701 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19c0039050 0x7f19c003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:26.228 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.232+0000 7f19dcc5f700 1 -- 172.21.15.44:0/3248230701 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19d8063890 con 0x7f19d8107bc0 2024-04-21T22:59:26.228 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.232+0000 7f19d5d9b700 1 --2- 172.21.15.44:0/3248230701 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19c0039050 0x7f19c003b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f19cc006fd0 tx=0x7f19cc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:26.228 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.237+0000 7f19c77fe700 1 -- 172.21.15.44:0/3248230701 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f19c8011e20 con 0x7f19d8107bc0 2024-04-21T22:59:26.229 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.453+0000 7f19dcc5f700 1 -- 172.21.15.44:0/3248230701 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f19d807ce70 con 0x7f19c0039050 2024-04-21T22:59:26.229 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.559+0000 7f19c77fe700 1 -- 172.21.15.44:0/3248230701 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f19d807ce70 con 0x7f19c0039050 2024-04-21T22:59:26.229 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.561+0000 7f19dcc5f700 1 -- 172.21.15.44:0/3248230701 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19c0039050 msgr2=0x7f19c003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:26.229 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.561+0000 7f19dcc5f700 1 --2- 172.21.15.44:0/3248230701 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19c0039050 0x7f19c003b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f19cc006fd0 tx=0x7f19cc006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:26.229 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.561+0000 7f19dcc5f700 1 -- 172.21.15.44:0/3248230701 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8107bc0 msgr2=0x7f19d807db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:26.229 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.562+0000 7f19dcc5f700 1 --2- 172.21.15.44:0/3248230701 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8107bc0 0x7f19d807db90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f19c80116a0 tx=0x7f19c8011ba0 comp rx=0 tx=0).stop 2024-04-21T22:59:26.229 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.562+0000 7f19dcc5f700 1 -- 172.21.15.44:0/3248230701 shutdown_connections 2024-04-21T22:59:26.229 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.562+0000 7f19dcc5f700 1 --2- 172.21.15.44:0/3248230701 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19c0039050 0x7f19c003b510 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:26.229 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.562+0000 7f19dcc5f700 1 --2- 172.21.15.44:0/3248230701 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19d8107bc0 0x7f19d807db90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:26.229 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.562+0000 7f19dcc5f700 1 -- 172.21.15.44:0/3248230701 >> 172.21.15.44:0/3248230701 conn(0x7f19d8103120 msgr2=0x7f19d8103ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:26.229 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.562+0000 7f19dcc5f700 1 -- 172.21.15.44:0/3248230701 shutdown_connections 2024-04-21T22:59:26.229 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:25.562+0000 7f19dcc5f700 1 -- 172.21.15.44:0/3248230701 wait complete. 2024-04-21T22:59:26.229 INFO:teuthology.orchestra.run.smithi044.stdout:Creating initial admin user... 2024-04-21T22:59:27.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:27.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:28.530 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:28 smithi044 ceph-mon[27934]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T22:59:28.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$LY4EWYOi6//ba26.vskJWOL237AMjmHRl5YmjAz5fKpCSTBarcfGO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713740367, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-21T22:59:28.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.652+0000 7f92d822f700 1 Processor -- start 2024-04-21T22:59:28.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.652+0000 7f92d822f700 1 -- start start 2024-04-21T22:59:28.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.652+0000 7f92d822f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92d0106460 0x7f92d0106880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:28.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.652+0000 7f92d822f700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f92d007be50 con 0x7f92d0106460 2024-04-21T22:59:28.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.652+0000 7f92d5fcb700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92d0106460 0x7f92d0106880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:28.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.652+0000 7f92d5fcb700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92d0106460 0x7f92d0106880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60624/0 (socket says 172.21.15.44:60624) 2024-04-21T22:59:28.538 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.653+0000 7f92d5fcb700 1 -- 172.21.15.44:0/3671064204 learned_addr learned my addr 172.21.15.44:0/3671064204 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:28.539 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.653+0000 7f92d5fcb700 1 -- 172.21.15.44:0/3671064204 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92d0106dc0 con 0x7f92d0106460 2024-04-21T22:59:28.539 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.653+0000 7f92d5fcb700 1 --2- 172.21.15.44:0/3671064204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92d0106460 0x7f92d0106880 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f92cc0098a0 tx=0x7f92cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35fd9aede1974d7a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:28.539 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.653+0000 7f92d4fc9700 1 -- 172.21.15.44:0/3671064204 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92cc004160 con 0x7f92d0106460 2024-04-21T22:59:28.539 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.653+0000 7f92d4fc9700 1 -- 172.21.15.44:0/3671064204 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92cc004300 con 0x7f92d0106460 2024-04-21T22:59:28.539 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.653+0000 7f92d4fc9700 1 -- 172.21.15.44:0/3671064204 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92cc011b20 con 0x7f92d0106460 2024-04-21T22:59:28.539 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.654+0000 7f92d822f700 1 -- 172.21.15.44:0/3671064204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92d0106460 msgr2=0x7f92d0106880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:28.539 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.654+0000 7f92d822f700 1 --2- 172.21.15.44:0/3671064204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92d0106460 0x7f92d0106880 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f92cc0098a0 tx=0x7f92cc009bb0 comp rx=0 tx=0).stop 2024-04-21T22:59:28.539 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.654+0000 7f92d822f700 1 -- 172.21.15.44:0/3671064204 shutdown_connections 2024-04-21T22:59:28.539 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.654+0000 7f92d822f700 1 --2- 172.21.15.44:0/3671064204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92d0106460 0x7f92d0106880 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:28.539 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.654+0000 7f92d822f700 1 -- 172.21.15.44:0/3671064204 >> 172.21.15.44:0/3671064204 conn(0x7f92d0102000 msgr2=0x7f92d0104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:28.539 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.654+0000 7f92d822f700 1 -- 172.21.15.44:0/3671064204 shutdown_connections 2024-04-21T22:59:28.539 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.654+0000 7f92d822f700 1 -- 172.21.15.44:0/3671064204 wait complete. 2024-04-21T22:59:28.540 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.655+0000 7f92d822f700 1 Processor -- start 2024-04-21T22:59:28.540 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.655+0000 7f92d822f700 1 -- start start 2024-04-21T22:59:28.540 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.655+0000 7f92d822f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92d0106460 0x7f92d019dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:28.540 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.655+0000 7f92d822f700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f92d0107e90 con 0x7f92d0106460 2024-04-21T22:59:28.540 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.655+0000 7f92d5fcb700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92d0106460 0x7f92d019dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:28.540 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.655+0000 7f92d5fcb700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92d0106460 0x7f92d019dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60636/0 (socket says 172.21.15.44:60636) 2024-04-21T22:59:28.540 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.655+0000 7f92d5fcb700 1 -- 172.21.15.44:0/722146050 learned_addr learned my addr 172.21.15.44:0/722146050 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:28.540 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.655+0000 7f92d5fcb700 1 -- 172.21.15.44:0/722146050 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92d019e1f0 con 0x7f92d0106460 2024-04-21T22:59:28.540 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.656+0000 7f92d5fcb700 1 --2- 172.21.15.44:0/722146050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92d0106460 0x7f92d019dcb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f92cc0117e0 tx=0x7f92cc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:28.541 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.656+0000 7f92c2ffd700 1 -- 172.21.15.44:0/722146050 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92cc003bf0 con 0x7f92d0106460 2024-04-21T22:59:28.541 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.656+0000 7f92c2ffd700 1 -- 172.21.15.44:0/722146050 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92cc01a910 con 0x7f92d0106460 2024-04-21T22:59:28.541 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.656+0000 7f92d822f700 1 -- 172.21.15.44:0/722146050 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92d019e480 con 0x7f92d0106460 2024-04-21T22:59:28.541 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.656+0000 7f92c2ffd700 1 -- 172.21.15.44:0/722146050 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92cc024970 con 0x7f92d0106460 2024-04-21T22:59:28.541 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.656+0000 7f92d822f700 1 -- 172.21.15.44:0/722146050 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92d019e8e0 con 0x7f92d0106460 2024-04-21T22:59:28.541 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.657+0000 7f92d822f700 1 -- 172.21.15.44:0/722146050 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92d0063890 con 0x7f92d0106460 2024-04-21T22:59:28.541 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.657+0000 7f92c2ffd700 1 -- 172.21.15.44:0/722146050 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f92cc022020 con 0x7f92d0106460 2024-04-21T22:59:28.541 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.657+0000 7f92c2ffd700 1 --2- 172.21.15.44:0/722146050 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f92bc0390b0 0x7f92bc03b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:28.541 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.657+0000 7f92c2ffd700 1 -- 172.21.15.44:0/722146050 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f92cc04d5e0 con 0x7f92d0106460 2024-04-21T22:59:28.541 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.657+0000 7f92d57ca700 1 --2- 172.21.15.44:0/722146050 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f92bc0390b0 0x7f92bc03b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:28.541 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.658+0000 7f92d57ca700 1 --2- 172.21.15.44:0/722146050 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f92bc0390b0 0x7f92bc03b570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f92c4006fd0 tx=0x7f92c4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:28.541 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.662+0000 7f92c2ffd700 1 -- 172.21.15.44:0/722146050 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f92cc011e20 con 0x7f92d0106460 2024-04-21T22:59:28.541 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:26.900+0000 7f92d822f700 1 -- 172.21.15.44:0/722146050 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f92d01977e0 con 0x7f92bc0390b0 2024-04-21T22:59:28.541 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:27.147+0000 7f92c2ffd700 1 -- 172.21.15.44:0/722146050 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f92d01977e0 con 0x7f92bc0390b0 2024-04-21T22:59:28.542 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:27.149+0000 7f92d822f700 1 -- 172.21.15.44:0/722146050 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f92bc0390b0 msgr2=0x7f92bc03b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:28.542 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:27.149+0000 7f92d822f700 1 --2- 172.21.15.44:0/722146050 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f92bc0390b0 0x7f92bc03b570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f92c4006fd0 tx=0x7f92c4006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:28.542 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:27.149+0000 7f92d822f700 1 -- 172.21.15.44:0/722146050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92d0106460 msgr2=0x7f92d019dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:28.542 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:27.149+0000 7f92d822f700 1 --2- 172.21.15.44:0/722146050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92d0106460 0x7f92d019dcb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f92cc0117e0 tx=0x7f92cc011c10 comp rx=0 tx=0).stop 2024-04-21T22:59:28.542 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:27.149+0000 7f92d822f700 1 -- 172.21.15.44:0/722146050 shutdown_connections 2024-04-21T22:59:28.542 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:27.149+0000 7f92d822f700 1 --2- 172.21.15.44:0/722146050 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f92bc0390b0 0x7f92bc03b570 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:28.542 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:27.149+0000 7f92d822f700 1 --2- 172.21.15.44:0/722146050 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92d0106460 0x7f92d019dcb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:28.542 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:27.149+0000 7f92d822f700 1 -- 172.21.15.44:0/722146050 >> 172.21.15.44:0/722146050 conn(0x7f92d0102000 msgr2=0x7f92d0194920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:28.542 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:27.149+0000 7f92d822f700 1 -- 172.21.15.44:0/722146050 shutdown_connections 2024-04-21T22:59:28.542 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:27.149+0000 7f92d822f700 1 -- 172.21.15.44:0/722146050 wait complete. 2024-04-21T22:59:28.542 INFO:teuthology.orchestra.run.smithi044.stdout:Fetching dashboard port number... 2024-04-21T22:59:29.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:29 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3727703934' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-21T22:59:29.702 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stdout 8443 2024-04-21T22:59:29.702 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.944+0000 7f57ab5d8700 1 Processor -- start 2024-04-21T22:59:29.702 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.944+0000 7f57ab5d8700 1 -- start start 2024-04-21T22:59:29.702 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.944+0000 7f57ab5d8700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57a4107c10 0x7f57a4108030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:29.702 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.944+0000 7f57ab5d8700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f57a4108600 con 0x7f57a4107c10 2024-04-21T22:59:29.703 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.945+0000 7f57a9374700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57a4107c10 0x7f57a4108030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:29.703 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.945+0000 7f57a9374700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57a4107c10 0x7f57a4108030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60640/0 (socket says 172.21.15.44:60640) 2024-04-21T22:59:29.703 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.945+0000 7f57a9374700 1 -- 172.21.15.44:0/792357715 learned_addr learned my addr 172.21.15.44:0/792357715 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:29.703 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.945+0000 7f57a9374700 1 -- 172.21.15.44:0/792357715 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57a4108d70 con 0x7f57a4107c10 2024-04-21T22:59:29.703 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.945+0000 7f57a9374700 1 --2- 172.21.15.44:0/792357715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57a4107c10 0x7f57a4108030 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f57a0009e80 tx=0x7f57a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2723d26f250a4c88 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:29.703 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.945+0000 7f579bfff700 1 -- 172.21.15.44:0/792357715 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57a0017070 con 0x7f57a4107c10 2024-04-21T22:59:29.703 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.945+0000 7f579bfff700 1 -- 172.21.15.44:0/792357715 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57a000b8c0 con 0x7f57a4107c10 2024-04-21T22:59:29.703 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.945+0000 7f579bfff700 1 -- 172.21.15.44:0/792357715 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57a00045c0 con 0x7f57a4107c10 2024-04-21T22:59:29.703 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.946+0000 7f57ab5d8700 1 -- 172.21.15.44:0/792357715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57a4107c10 msgr2=0x7f57a4108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:29.703 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.946+0000 7f57ab5d8700 1 --2- 172.21.15.44:0/792357715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57a4107c10 0x7f57a4108030 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f57a0009e80 tx=0x7f57a000b330 comp rx=0 tx=0).stop 2024-04-21T22:59:29.703 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.946+0000 7f57ab5d8700 1 -- 172.21.15.44:0/792357715 shutdown_connections 2024-04-21T22:59:29.703 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.946+0000 7f57ab5d8700 1 --2- 172.21.15.44:0/792357715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57a4107c10 0x7f57a4108030 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:29.703 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.946+0000 7f57ab5d8700 1 -- 172.21.15.44:0/792357715 >> 172.21.15.44:0/792357715 conn(0x7f57a4103230 msgr2=0x7f57a4105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:29.704 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.946+0000 7f57ab5d8700 1 -- 172.21.15.44:0/792357715 shutdown_connections 2024-04-21T22:59:29.704 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.946+0000 7f57ab5d8700 1 -- 172.21.15.44:0/792357715 wait complete. 2024-04-21T22:59:29.704 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.946+0000 7f57ab5d8700 1 Processor -- start 2024-04-21T22:59:29.704 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.947+0000 7f57ab5d8700 1 -- start start 2024-04-21T22:59:29.704 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.947+0000 7f57ab5d8700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57a4107c10 0x7f57a407db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:29.704 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.947+0000 7f57ab5d8700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f57a410c120 con 0x7f57a4107c10 2024-04-21T22:59:29.704 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.947+0000 7f57a9374700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57a4107c10 0x7f57a407db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:29.704 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.947+0000 7f57a9374700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57a4107c10 0x7f57a407db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60650/0 (socket says 172.21.15.44:60650) 2024-04-21T22:59:29.704 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.947+0000 7f57a9374700 1 -- 172.21.15.44:0/3727703934 learned_addr learned my addr 172.21.15.44:0/3727703934 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:29.704 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.947+0000 7f57a9374700 1 -- 172.21.15.44:0/3727703934 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57a407e0d0 con 0x7f57a4107c10 2024-04-21T22:59:29.704 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.947+0000 7f57a9374700 1 --2- 172.21.15.44:0/3727703934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57a4107c10 0x7f57a407db90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f57a0003fa0 tx=0x7f57a0004080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:29.704 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.947+0000 7f579a7fc700 1 -- 172.21.15.44:0/3727703934 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57a0020020 con 0x7f57a4107c10 2024-04-21T22:59:29.704 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.947+0000 7f579a7fc700 1 -- 172.21.15.44:0/3727703934 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57a000bba0 con 0x7f57a4107c10 2024-04-21T22:59:29.705 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.947+0000 7f57ab5d8700 1 -- 172.21.15.44:0/3727703934 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57a407c2c0 con 0x7f57a4107c10 2024-04-21T22:59:29.705 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.947+0000 7f579a7fc700 1 -- 172.21.15.44:0/3727703934 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57a0017040 con 0x7f57a4107c10 2024-04-21T22:59:29.705 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.947+0000 7f57ab5d8700 1 -- 172.21.15.44:0/3727703934 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57a407c800 con 0x7f57a4107c10 2024-04-21T22:59:29.705 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.948+0000 7f579a7fc700 1 -- 172.21.15.44:0/3727703934 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f57a0023020 con 0x7f57a4107c10 2024-04-21T22:59:29.705 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.948+0000 7f579a7fc700 1 --2- 172.21.15.44:0/3727703934 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f57900390b0 0x7f579003b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:29.705 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.949+0000 7f579a7fc700 1 -- 172.21.15.44:0/3727703934 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f57a000f050 con 0x7f57a4107c10 2024-04-21T22:59:29.705 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.949+0000 7f57ab5d8700 1 -- 172.21.15.44:0/3727703934 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57a4063890 con 0x7f57a4107c10 2024-04-21T22:59:29.705 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.949+0000 7f57a8b73700 1 --2- 172.21.15.44:0/3727703934 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f57900390b0 0x7f579003b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:29.705 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.949+0000 7f57a8b73700 1 --2- 172.21.15.44:0/3727703934 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f57900390b0 0x7f579003b570 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5794006fd0 tx=0x7f5794006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:29.705 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:28.954+0000 7f579a7fc700 1 -- 172.21.15.44:0/3727703934 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f57a002a020 con 0x7f57a4107c10 2024-04-21T22:59:29.705 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:29.160+0000 7f57ab5d8700 1 -- 172.21.15.44:0/3727703934 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f57a407d0a0 con 0x7f57a4107c10 2024-04-21T22:59:29.705 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:29.160+0000 7f579a7fc700 1 -- 172.21.15.44:0/3727703934 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f57a002d350 con 0x7f57a4107c10 2024-04-21T22:59:29.705 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:29.162+0000 7f57ab5d8700 1 -- 172.21.15.44:0/3727703934 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f57900390b0 msgr2=0x7f579003b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:29.706 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:29.162+0000 7f57ab5d8700 1 --2- 172.21.15.44:0/3727703934 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f57900390b0 0x7f579003b570 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5794006fd0 tx=0x7f5794006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:29.706 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:29.162+0000 7f57ab5d8700 1 -- 172.21.15.44:0/3727703934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57a4107c10 msgr2=0x7f57a407db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:29.706 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:29.162+0000 7f57ab5d8700 1 --2- 172.21.15.44:0/3727703934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57a4107c10 0x7f57a407db90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f57a0003fa0 tx=0x7f57a0004080 comp rx=0 tx=0).stop 2024-04-21T22:59:29.706 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:29.162+0000 7f57ab5d8700 1 -- 172.21.15.44:0/3727703934 shutdown_connections 2024-04-21T22:59:29.706 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:29.162+0000 7f57ab5d8700 1 --2- 172.21.15.44:0/3727703934 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f57900390b0 0x7f579003b570 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:29.706 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:29.162+0000 7f57ab5d8700 1 --2- 172.21.15.44:0/3727703934 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f57a4107c10 0x7f57a407db90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:29.706 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:29.162+0000 7f57ab5d8700 1 -- 172.21.15.44:0/3727703934 >> 172.21.15.44:0/3727703934 conn(0x7f57a4103230 msgr2=0x7f57a4103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:29.706 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:29.163+0000 7f57ab5d8700 1 -- 172.21.15.44:0/3727703934 shutdown_connections 2024-04-21T22:59:29.706 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:29.163+0000 7f57ab5d8700 1 -- 172.21.15.44:0/3727703934 wait complete. 2024-04-21T22:59:29.709 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T22:59:29.709 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout disabled 2024-04-21T22:59:29.717 INFO:teuthology.orchestra.run.smithi044.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T22:59:29.717 INFO:teuthology.orchestra.run.smithi044.stdout:systemctl: stdout inactive 2024-04-21T22:59:29.717 INFO:teuthology.orchestra.run.smithi044.stdout:firewalld.service is not enabled 2024-04-21T22:59:29.717 INFO:teuthology.orchestra.run.smithi044.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-21T22:59:29.719 INFO:teuthology.orchestra.run.smithi044.stdout:Ceph Dashboard is now available at: 2024-04-21T22:59:29.719 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:59:29.719 INFO:teuthology.orchestra.run.smithi044.stdout: URL: https://smithi044:8443/ 2024-04-21T22:59:29.719 INFO:teuthology.orchestra.run.smithi044.stdout: User: admin 2024-04-21T22:59:29.719 INFO:teuthology.orchestra.run.smithi044.stdout: Password: ezihdct14v 2024-04-21T22:59:29.720 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:59:29.720 INFO:teuthology.orchestra.run.smithi044.stdout:Saving cluster configuration to /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config directory 2024-04-21T22:59:29.720 INFO:teuthology.orchestra.run.smithi044.stdout:Enabling autotune for osd_memory_target 2024-04-21T22:59:30.373 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:30 smithi044 ceph-mon[27934]: mgrmap e13: x(active, since 6s) 2024-04-21T22:59:31.120 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.145+0000 7fe28768d700 1 Processor -- start 2024-04-21T22:59:31.125 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.146+0000 7fe28768d700 1 -- start start 2024-04-21T22:59:31.125 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.146+0000 7fe28768d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe28007de20 0x7fe28007c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:31.125 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.146+0000 7fe28768d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe28007c860 con 0x7fe28007de20 2024-04-21T22:59:31.125 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.146+0000 7fe285429700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe28007de20 0x7fe28007c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:31.125 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.146+0000 7fe285429700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe28007de20 0x7fe28007c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60666/0 (socket says 172.21.15.44:60666) 2024-04-21T22:59:31.125 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.146+0000 7fe285429700 1 -- 172.21.15.44:0/3209131277 learned_addr learned my addr 172.21.15.44:0/3209131277 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:31.126 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.146+0000 7fe285429700 1 -- 172.21.15.44:0/3209131277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28007c9e0 con 0x7fe28007de20 2024-04-21T22:59:31.126 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.147+0000 7fe285429700 1 --2- 172.21.15.44:0/3209131277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe28007de20 0x7fe28007c320 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe27c0098a0 tx=0x7fe27c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcacbe30ef45e27b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:31.126 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.147+0000 7fe277fff700 1 -- 172.21.15.44:0/3209131277 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe27c004160 con 0x7fe28007de20 2024-04-21T22:59:31.126 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.147+0000 7fe277fff700 1 -- 172.21.15.44:0/3209131277 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe27c004300 con 0x7fe28007de20 2024-04-21T22:59:31.126 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.147+0000 7fe277fff700 1 -- 172.21.15.44:0/3209131277 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe27c011b20 con 0x7fe28007de20 2024-04-21T22:59:31.126 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.147+0000 7fe28768d700 1 -- 172.21.15.44:0/3209131277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe28007de20 msgr2=0x7fe28007c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:31.126 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.147+0000 7fe28768d700 1 --2- 172.21.15.44:0/3209131277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe28007de20 0x7fe28007c320 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe27c0098a0 tx=0x7fe27c009bb0 comp rx=0 tx=0).stop 2024-04-21T22:59:31.126 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.147+0000 7fe28768d700 1 -- 172.21.15.44:0/3209131277 shutdown_connections 2024-04-21T22:59:31.126 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.147+0000 7fe28768d700 1 --2- 172.21.15.44:0/3209131277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe28007de20 0x7fe28007c320 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:31.126 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.147+0000 7fe28768d700 1 -- 172.21.15.44:0/3209131277 >> 172.21.15.44:0/3209131277 conn(0x7fe280103230 msgr2=0x7fe280105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:31.126 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.147+0000 7fe28768d700 1 -- 172.21.15.44:0/3209131277 shutdown_connections 2024-04-21T22:59:31.126 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.148+0000 7fe28768d700 1 -- 172.21.15.44:0/3209131277 wait complete. 2024-04-21T22:59:31.126 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.148+0000 7fe28768d700 1 Processor -- start 2024-04-21T22:59:31.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.148+0000 7fe28768d700 1 -- start start 2024-04-21T22:59:31.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.148+0000 7fe28768d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe28007de20 0x7fe28019dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:31.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.148+0000 7fe28768d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe28010e1b0 con 0x7fe28007de20 2024-04-21T22:59:31.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.148+0000 7fe285429700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe28007de20 0x7fe28019dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:31.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.148+0000 7fe285429700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe28007de20 0x7fe28019dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60678/0 (socket says 172.21.15.44:60678) 2024-04-21T22:59:31.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.148+0000 7fe285429700 1 -- 172.21.15.44:0/3079616687 learned_addr learned my addr 172.21.15.44:0/3079616687 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:31.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.149+0000 7fe285429700 1 -- 172.21.15.44:0/3079616687 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28019e230 con 0x7fe28007de20 2024-04-21T22:59:31.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.149+0000 7fe285429700 1 --2- 172.21.15.44:0/3079616687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe28007de20 0x7fe28019dcf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe27c011b60 tx=0x7fe27c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:31.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.149+0000 7fe2767fc700 1 -- 172.21.15.44:0/3079616687 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe27c003b60 con 0x7fe28007de20 2024-04-21T22:59:31.127 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.149+0000 7fe2767fc700 1 -- 172.21.15.44:0/3079616687 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe27c01a910 con 0x7fe28007de20 2024-04-21T22:59:31.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.149+0000 7fe28768d700 1 -- 172.21.15.44:0/3079616687 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe28019e4c0 con 0x7fe28007de20 2024-04-21T22:59:31.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.149+0000 7fe2767fc700 1 -- 172.21.15.44:0/3079616687 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe27c024930 con 0x7fe28007de20 2024-04-21T22:59:31.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.149+0000 7fe28768d700 1 -- 172.21.15.44:0/3079616687 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe28019e920 con 0x7fe28007de20 2024-04-21T22:59:31.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.150+0000 7fe2767fc700 1 -- 172.21.15.44:0/3079616687 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fe27c022020 con 0x7fe28007de20 2024-04-21T22:59:31.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.150+0000 7fe28768d700 1 -- 172.21.15.44:0/3079616687 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe280063890 con 0x7fe28007de20 2024-04-21T22:59:31.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.150+0000 7fe2767fc700 1 --2- 172.21.15.44:0/3079616687 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe26c039140 0x7fe26c03b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:31.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.150+0000 7fe2767fc700 1 -- 172.21.15.44:0/3079616687 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe27c04d530 con 0x7fe28007de20 2024-04-21T22:59:31.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.150+0000 7fe284c28700 1 --2- 172.21.15.44:0/3079616687 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe26c039140 0x7fe26c03b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:31.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.151+0000 7fe284c28700 1 --2- 172.21.15.44:0/3079616687 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe26c039140 0x7fe26c03b600 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe270006fd0 tx=0x7fe270006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:31.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.155+0000 7fe2767fc700 1 -- 172.21.15.44:0/3079616687 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe27c019c70 con 0x7fe28007de20 2024-04-21T22:59:31.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.360+0000 7fe28768d700 1 -- 172.21.15.44:0/3079616687 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fe2801a1440 con 0x7fe28007de20 2024-04-21T22:59:31.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.361+0000 7fe2767fc700 1 -- 172.21.15.44:0/3079616687 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fe27c02d910 con 0x7fe28007de20 2024-04-21T22:59:31.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.362+0000 7fe28768d700 1 -- 172.21.15.44:0/3079616687 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe26c039140 msgr2=0x7fe26c03b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:31.128 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.362+0000 7fe28768d700 1 --2- 172.21.15.44:0/3079616687 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe26c039140 0x7fe26c03b600 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe270006fd0 tx=0x7fe270006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:31.129 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.362+0000 7fe28768d700 1 -- 172.21.15.44:0/3079616687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe28007de20 msgr2=0x7fe28019dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:31.129 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.362+0000 7fe28768d700 1 --2- 172.21.15.44:0/3079616687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe28007de20 0x7fe28019dcf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe27c011b60 tx=0x7fe27c011c40 comp rx=0 tx=0).stop 2024-04-21T22:59:31.129 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.363+0000 7fe28768d700 1 -- 172.21.15.44:0/3079616687 shutdown_connections 2024-04-21T22:59:31.129 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.363+0000 7fe28768d700 1 --2- 172.21.15.44:0/3079616687 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe26c039140 0x7fe26c03b600 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:31.129 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.363+0000 7fe28768d700 1 --2- 172.21.15.44:0/3079616687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe28007de20 0x7fe28019dcf0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:31.129 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.363+0000 7fe28768d700 1 -- 172.21.15.44:0/3079616687 >> 172.21.15.44:0/3079616687 conn(0x7fe280103230 msgr2=0x7fe280103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:31.129 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.363+0000 7fe28768d700 1 -- 172.21.15.44:0/3079616687 shutdown_connections 2024-04-21T22:59:31.129 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:30.363+0000 7fe28768d700 1 -- 172.21.15.44:0/3079616687 wait complete. 2024-04-21T22:59:32.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.543+0000 7fc9686cd700 1 Processor -- start 2024-04-21T22:59:32.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.544+0000 7fc9686cd700 1 -- start start 2024-04-21T22:59:32.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.544+0000 7fc9686cd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9601085f0 0x7fc96010a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:32.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.544+0000 7fc9686cd700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc96007bed0 con 0x7fc9601085f0 2024-04-21T22:59:32.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.544+0000 7fc966469700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9601085f0 0x7fc96010a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:32.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.544+0000 7fc966469700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9601085f0 0x7fc96010a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52306/0 (socket says 172.21.15.44:52306) 2024-04-21T22:59:32.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.544+0000 7fc966469700 1 -- 172.21.15.44:0/4006596534 learned_addr learned my addr 172.21.15.44:0/4006596534 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:32.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.544+0000 7fc966469700 1 -- 172.21.15.44:0/4006596534 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc96010af20 con 0x7fc9601085f0 2024-04-21T22:59:32.565 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.545+0000 7fc966469700 1 --2- 172.21.15.44:0/4006596534 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9601085f0 0x7fc96010a9e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc95c0098a0 tx=0x7fc95c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54025216f8aedc6b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:32.566 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.545+0000 7fc965467700 1 -- 172.21.15.44:0/4006596534 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc95c004160 con 0x7fc9601085f0 2024-04-21T22:59:32.566 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.545+0000 7fc965467700 1 -- 172.21.15.44:0/4006596534 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc95c004300 con 0x7fc9601085f0 2024-04-21T22:59:32.566 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.545+0000 7fc965467700 1 -- 172.21.15.44:0/4006596534 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc95c011b20 con 0x7fc9601085f0 2024-04-21T22:59:32.566 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.545+0000 7fc9686cd700 1 -- 172.21.15.44:0/4006596534 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9601085f0 msgr2=0x7fc96010a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:32.566 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.545+0000 7fc9686cd700 1 --2- 172.21.15.44:0/4006596534 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9601085f0 0x7fc96010a9e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc95c0098a0 tx=0x7fc95c009bb0 comp rx=0 tx=0).stop 2024-04-21T22:59:32.566 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.545+0000 7fc9686cd700 1 -- 172.21.15.44:0/4006596534 shutdown_connections 2024-04-21T22:59:32.566 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.545+0000 7fc9686cd700 1 --2- 172.21.15.44:0/4006596534 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9601085f0 0x7fc96010a9e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:32.566 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.545+0000 7fc9686cd700 1 -- 172.21.15.44:0/4006596534 >> 172.21.15.44:0/4006596534 conn(0x7fc960102000 msgr2=0x7fc960104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:32.566 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.546+0000 7fc9686cd700 1 -- 172.21.15.44:0/4006596534 shutdown_connections 2024-04-21T22:59:32.566 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.546+0000 7fc9686cd700 1 -- 172.21.15.44:0/4006596534 wait complete. 2024-04-21T22:59:32.566 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.546+0000 7fc9686cd700 1 Processor -- start 2024-04-21T22:59:32.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.546+0000 7fc9686cd700 1 -- start start 2024-04-21T22:59:32.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.546+0000 7fc9686cd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9601085f0 0x7fc96019bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:32.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.546+0000 7fc9686cd700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc96010bf90 con 0x7fc9601085f0 2024-04-21T22:59:32.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.546+0000 7fc966469700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9601085f0 0x7fc96019bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:32.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.547+0000 7fc966469700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9601085f0 0x7fc96019bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52308/0 (socket says 172.21.15.44:52308) 2024-04-21T22:59:32.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.547+0000 7fc966469700 1 -- 172.21.15.44:0/4104505976 learned_addr learned my addr 172.21.15.44:0/4104505976 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:32.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.547+0000 7fc966469700 1 -- 172.21.15.44:0/4104505976 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc96019bff0 con 0x7fc9601085f0 2024-04-21T22:59:32.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.547+0000 7fc966469700 1 --2- 172.21.15.44:0/4104505976 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9601085f0 0x7fc96019bab0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc95c0117e0 tx=0x7fc95c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:32.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.547+0000 7fc9577fe700 1 -- 172.21.15.44:0/4104505976 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc95c003ba0 con 0x7fc9601085f0 2024-04-21T22:59:32.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.547+0000 7fc9577fe700 1 -- 172.21.15.44:0/4104505976 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc95c01a910 con 0x7fc9601085f0 2024-04-21T22:59:32.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.547+0000 7fc9686cd700 1 -- 172.21.15.44:0/4104505976 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc96019c280 con 0x7fc9601085f0 2024-04-21T22:59:32.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.547+0000 7fc9577fe700 1 -- 172.21.15.44:0/4104505976 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc95c0248f0 con 0x7fc9601085f0 2024-04-21T22:59:32.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.547+0000 7fc9686cd700 1 -- 172.21.15.44:0/4104505976 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc96019c6e0 con 0x7fc9601085f0 2024-04-21T22:59:32.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.548+0000 7fc9577fe700 1 -- 172.21.15.44:0/4104505976 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fc95c022020 con 0x7fc9601085f0 2024-04-21T22:59:32.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.548+0000 7fc9686cd700 1 -- 172.21.15.44:0/4104505976 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc960063890 con 0x7fc9601085f0 2024-04-21T22:59:32.567 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.549+0000 7fc9577fe700 1 --2- 172.21.15.44:0/4104505976 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc94c039140 0x7fc94c03b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:32.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.549+0000 7fc9577fe700 1 -- 172.21.15.44:0/4104505976 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc95c04da90 con 0x7fc9601085f0 2024-04-21T22:59:32.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.549+0000 7fc965c68700 1 --2- 172.21.15.44:0/4104505976 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc94c039140 0x7fc94c03b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:32.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.549+0000 7fc965c68700 1 --2- 172.21.15.44:0/4104505976 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc94c039140 0x7fc94c03b600 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc950006fd0 tx=0x7fc950006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:32.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.554+0000 7fc9577fe700 1 -- 172.21.15.44:0/4104505976 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc95c0212c0 con 0x7fc9601085f0 2024-04-21T22:59:32.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.874+0000 7fc9686cd700 1 -- 172.21.15.44:0/4104505976 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fc96019f170 con 0x7fc9601085f0 2024-04-21T22:59:32.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.876+0000 7fc9577fe700 1 -- 172.21.15.44:0/4104505976 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7fc95c02d3f0 con 0x7fc9601085f0 2024-04-21T22:59:32.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.879+0000 7fc9686cd700 1 -- 172.21.15.44:0/4104505976 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc94c039140 msgr2=0x7fc94c03b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:32.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.879+0000 7fc9686cd700 1 --2- 172.21.15.44:0/4104505976 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc94c039140 0x7fc94c03b600 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc950006fd0 tx=0x7fc950006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:32.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.879+0000 7fc9686cd700 1 -- 172.21.15.44:0/4104505976 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9601085f0 msgr2=0x7fc96019bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:32.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.879+0000 7fc9686cd700 1 --2- 172.21.15.44:0/4104505976 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9601085f0 0x7fc96019bab0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc95c0117e0 tx=0x7fc95c011ba0 comp rx=0 tx=0).stop 2024-04-21T22:59:32.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.879+0000 7fc9686cd700 1 -- 172.21.15.44:0/4104505976 shutdown_connections 2024-04-21T22:59:32.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.879+0000 7fc9686cd700 1 --2- 172.21.15.44:0/4104505976 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc94c039140 0x7fc94c03b600 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:32.568 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.879+0000 7fc9686cd700 1 --2- 172.21.15.44:0/4104505976 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc9601085f0 0x7fc96019bab0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:32.569 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.879+0000 7fc9686cd700 1 -- 172.21.15.44:0/4104505976 >> 172.21.15.44:0/4104505976 conn(0x7fc960102000 msgr2=0x7fc960102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:32.569 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.879+0000 7fc9686cd700 1 -- 172.21.15.44:0/4104505976 shutdown_connections 2024-04-21T22:59:32.569 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr 2024-04-21T22:59:31.879+0000 7fc9686cd700 1 -- 172.21.15.44:0/4104505976 wait complete. 2024-04-21T22:59:32.569 INFO:teuthology.orchestra.run.smithi044.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-21T22:59:32.569 INFO:teuthology.orchestra.run.smithi044.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-21T22:59:32.569 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:59:32.569 INFO:teuthology.orchestra.run.smithi044.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-21T22:59:32.569 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:59:32.569 INFO:teuthology.orchestra.run.smithi044.stdout:Or, if you are only running a single cluster on this host: 2024-04-21T22:59:32.569 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:59:32.569 INFO:teuthology.orchestra.run.smithi044.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-21T22:59:32.569 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:59:32.569 INFO:teuthology.orchestra.run.smithi044.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-21T22:59:32.570 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:59:32.570 INFO:teuthology.orchestra.run.smithi044.stdout: ceph telemetry on 2024-04-21T22:59:32.570 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:59:32.570 INFO:teuthology.orchestra.run.smithi044.stdout:For more information see: 2024-04-21T22:59:32.570 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:59:32.570 INFO:teuthology.orchestra.run.smithi044.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-21T22:59:32.570 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:59:32.570 INFO:teuthology.orchestra.run.smithi044.stdout:Bootstrap complete. 2024-04-21T22:59:32.609 INFO:tasks.cephadm:Fetching config... 2024-04-21T22:59:32.609 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T22:59:32.609 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-21T22:59:32.626 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-21T22:59:32.626 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T22:59:32.626 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-21T22:59:32.684 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-21T22:59:32.684 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T22:59:32.684 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-21T22:59:32.752 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-21T22:59:32.752 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T22:59:32.752 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-21T22:59:32.808 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-21T22:59:32.809 DEBUG:teuthology.orchestra.run.smithi044:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1eyvRUMqjzHmTxxtKqHv1QnXHFSZULaQWUilTDG0pvw0WIzMHY5dFLuJvbKbhkdYT7e5Xp+UI+O2IfTcg7dW8XZ6BQELGbhRPTBdzSwj1FISSVuRUntCLuS74H9EsGzRq7NfNzJJu4oRUkLnUHp9nfhS09M+WXSvgljI542QtKqytpPMmbDz9GZuHd3J7YF3ejl4nJ0ZPcd22sQcn1qVKUs/Tid1idFPUvLdup6QbjrXKbmcO/TgCRywoNTbWClFyifT/3NXgVuWiZfpjNzmoAXF2EW11rRlmF+Db9LE3b9aCWc5gXv13xjw/PygTCk2yvAF73hXMKLkDBRqg1gn2ici0PyfslaFs+CJv9NPUJnMy9j5wP2B80gPWmCTwtonvjWzA/zBQc3gbns8zbWSoWQ0w6xzYpz5lOHuwLXIY2sJ28GOU5MX1M2eTdPT92rc7Jl+sONnLqVX10imgyz3aJKSenZ6On2++c4NrXdm03Y1QJMaVVg0ZZOGEGTfS/sU= ceph-80909b94-0032-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T22:59:32.888 INFO:teuthology.orchestra.run.smithi044.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1eyvRUMqjzHmTxxtKqHv1QnXHFSZULaQWUilTDG0pvw0WIzMHY5dFLuJvbKbhkdYT7e5Xp+UI+O2IfTcg7dW8XZ6BQELGbhRPTBdzSwj1FISSVuRUntCLuS74H9EsGzRq7NfNzJJu4oRUkLnUHp9nfhS09M+WXSvgljI542QtKqytpPMmbDz9GZuHd3J7YF3ejl4nJ0ZPcd22sQcn1qVKUs/Tid1idFPUvLdup6QbjrXKbmcO/TgCRywoNTbWClFyifT/3NXgVuWiZfpjNzmoAXF2EW11rRlmF+Db9LE3b9aCWc5gXv13xjw/PygTCk2yvAF73hXMKLkDBRqg1gn2ici0PyfslaFs+CJv9NPUJnMy9j5wP2B80gPWmCTwtonvjWzA/zBQc3gbns8zbWSoWQ0w6xzYpz5lOHuwLXIY2sJ28GOU5MX1M2eTdPT92rc7Jl+sONnLqVX10imgyz3aJKSenZ6On2++c4NrXdm03Y1QJMaVVg0ZZOGEGTfS/sU= ceph-80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T22:59:32.901 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1eyvRUMqjzHmTxxtKqHv1QnXHFSZULaQWUilTDG0pvw0WIzMHY5dFLuJvbKbhkdYT7e5Xp+UI+O2IfTcg7dW8XZ6BQELGbhRPTBdzSwj1FISSVuRUntCLuS74H9EsGzRq7NfNzJJu4oRUkLnUHp9nfhS09M+WXSvgljI542QtKqytpPMmbDz9GZuHd3J7YF3ejl4nJ0ZPcd22sQcn1qVKUs/Tid1idFPUvLdup6QbjrXKbmcO/TgCRywoNTbWClFyifT/3NXgVuWiZfpjNzmoAXF2EW11rRlmF+Db9LE3b9aCWc5gXv13xjw/PygTCk2yvAF73hXMKLkDBRqg1gn2ici0PyfslaFs+CJv9NPUJnMy9j5wP2B80gPWmCTwtonvjWzA/zBQc3gbns8zbWSoWQ0w6xzYpz5lOHuwLXIY2sJ28GOU5MX1M2eTdPT92rc7Jl+sONnLqVX10imgyz3aJKSenZ6On2++c4NrXdm03Y1QJMaVVg0ZZOGEGTfS/sU= ceph-80909b94-0032-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T22:59:32.941 INFO:teuthology.orchestra.run.smithi055.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1eyvRUMqjzHmTxxtKqHv1QnXHFSZULaQWUilTDG0pvw0WIzMHY5dFLuJvbKbhkdYT7e5Xp+UI+O2IfTcg7dW8XZ6BQELGbhRPTBdzSwj1FISSVuRUntCLuS74H9EsGzRq7NfNzJJu4oRUkLnUHp9nfhS09M+WXSvgljI542QtKqytpPMmbDz9GZuHd3J7YF3ejl4nJ0ZPcd22sQcn1qVKUs/Tid1idFPUvLdup6QbjrXKbmcO/TgCRywoNTbWClFyifT/3NXgVuWiZfpjNzmoAXF2EW11rRlmF+Db9LE3b9aCWc5gXv13xjw/PygTCk2yvAF73hXMKLkDBRqg1gn2ici0PyfslaFs+CJv9NPUJnMy9j5wP2B80gPWmCTwtonvjWzA/zBQc3gbns8zbWSoWQ0w6xzYpz5lOHuwLXIY2sJ28GOU5MX1M2eTdPT92rc7Jl+sONnLqVX10imgyz3aJKSenZ6On2++c4NrXdm03Y1QJMaVVg0ZZOGEGTfS/sU= ceph-80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T22:59:32.954 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1eyvRUMqjzHmTxxtKqHv1QnXHFSZULaQWUilTDG0pvw0WIzMHY5dFLuJvbKbhkdYT7e5Xp+UI+O2IfTcg7dW8XZ6BQELGbhRPTBdzSwj1FISSVuRUntCLuS74H9EsGzRq7NfNzJJu4oRUkLnUHp9nfhS09M+WXSvgljI542QtKqytpPMmbDz9GZuHd3J7YF3ejl4nJ0ZPcd22sQcn1qVKUs/Tid1idFPUvLdup6QbjrXKbmcO/TgCRywoNTbWClFyifT/3NXgVuWiZfpjNzmoAXF2EW11rRlmF+Db9LE3b9aCWc5gXv13xjw/PygTCk2yvAF73hXMKLkDBRqg1gn2ici0PyfslaFs+CJv9NPUJnMy9j5wP2B80gPWmCTwtonvjWzA/zBQc3gbns8zbWSoWQ0w6xzYpz5lOHuwLXIY2sJ28GOU5MX1M2eTdPT92rc7Jl+sONnLqVX10imgyz3aJKSenZ6On2++c4NrXdm03Y1QJMaVVg0ZZOGEGTfS/sU= ceph-80909b94-0032-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T22:59:32.994 INFO:teuthology.orchestra.run.smithi069.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1eyvRUMqjzHmTxxtKqHv1QnXHFSZULaQWUilTDG0pvw0WIzMHY5dFLuJvbKbhkdYT7e5Xp+UI+O2IfTcg7dW8XZ6BQELGbhRPTBdzSwj1FISSVuRUntCLuS74H9EsGzRq7NfNzJJu4oRUkLnUHp9nfhS09M+WXSvgljI542QtKqytpPMmbDz9GZuHd3J7YF3ejl4nJ0ZPcd22sQcn1qVKUs/Tid1idFPUvLdup6QbjrXKbmcO/TgCRywoNTbWClFyifT/3NXgVuWiZfpjNzmoAXF2EW11rRlmF+Db9LE3b9aCWc5gXv13xjw/PygTCk2yvAF73hXMKLkDBRqg1gn2ici0PyfslaFs+CJv9NPUJnMy9j5wP2B80gPWmCTwtonvjWzA/zBQc3gbns8zbWSoWQ0w6xzYpz5lOHuwLXIY2sJ28GOU5MX1M2eTdPT92rc7Jl+sONnLqVX10imgyz3aJKSenZ6On2++c4NrXdm03Y1QJMaVVg0ZZOGEGTfS/sU= ceph-80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T22:59:33.006 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-21T22:59:33.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:32 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/4104505976' entity='client.admin' 2024-04-21T22:59:33.255 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T22:59:35.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.442+0000 7fef9312f700 1 -- 172.21.15.44:0/995383617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef8c100b90 msgr2=0x7fef8c100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:35.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.442+0000 7fef9312f700 1 --2- 172.21.15.44:0/995383617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef8c100b90 0x7fef8c100fb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009910 tx=0x7fef7c009c20 comp rx=0 tx=0).stop 2024-04-21T22:59:35.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.442+0000 7fef9312f700 1 -- 172.21.15.44:0/995383617 shutdown_connections 2024-04-21T22:59:35.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.442+0000 7fef9312f700 1 --2- 172.21.15.44:0/995383617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef8c100b90 0x7fef8c100fb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:35.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.442+0000 7fef9312f700 1 -- 172.21.15.44:0/995383617 >> 172.21.15.44:0/995383617 conn(0x7fef8c0fc130 msgr2=0x7fef8c0fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:35.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.442+0000 7fef9312f700 1 -- 172.21.15.44:0/995383617 shutdown_connections 2024-04-21T22:59:35.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.443+0000 7fef9312f700 1 -- 172.21.15.44:0/995383617 wait complete. 2024-04-21T22:59:35.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.443+0000 7fef9312f700 1 Processor -- start 2024-04-21T22:59:35.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.443+0000 7fef9312f700 1 -- start start 2024-04-21T22:59:35.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.443+0000 7fef9312f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef8c100b90 0x7fef8c0fdff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:35.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.443+0000 7fef9312f700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c106c20 con 0x7fef8c100b90 2024-04-21T22:59:35.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.444+0000 7fef90ecb700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef8c100b90 0x7fef8c0fdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:35.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.444+0000 7fef90ecb700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef8c100b90 0x7fef8c0fdff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52326/0 (socket says 172.21.15.44:52326) 2024-04-21T22:59:35.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.444+0000 7fef90ecb700 1 -- 172.21.15.44:0/685657191 learned_addr learned my addr 172.21.15.44:0/685657191 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:35.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.444+0000 7fef90ecb700 1 -- 172.21.15.44:0/685657191 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef8c0fe530 con 0x7fef8c100b90 2024-04-21T22:59:35.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.444+0000 7fef90ecb700 1 --2- 172.21.15.44:0/685657191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef8c100b90 0x7fef8c0fdff0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fef7c005850 tx=0x7fef7c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:35.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.444+0000 7fef89ffb700 1 -- 172.21.15.44:0/685657191 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef7c026070 con 0x7fef8c100b90 2024-04-21T22:59:35.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.444+0000 7fef89ffb700 1 -- 172.21.15.44:0/685657191 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef7c0044a0 con 0x7fef8c100b90 2024-04-21T22:59:35.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.444+0000 7fef9312f700 1 -- 172.21.15.44:0/685657191 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef8c0fc720 con 0x7fef8c100b90 2024-04-21T22:59:35.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.444+0000 7fef89ffb700 1 -- 172.21.15.44:0/685657191 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef7c003ab0 con 0x7fef8c100b90 2024-04-21T22:59:35.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.444+0000 7fef9312f700 1 -- 172.21.15.44:0/685657191 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef8c0fcc60 con 0x7fef8c100b90 2024-04-21T22:59:35.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.445+0000 7fef89ffb700 1 -- 172.21.15.44:0/685657191 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fef7c029070 con 0x7fef8c100b90 2024-04-21T22:59:35.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.445+0000 7fef9312f700 1 -- 172.21.15.44:0/685657191 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef8c1073a0 con 0x7fef8c100b90 2024-04-21T22:59:35.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.446+0000 7fef89ffb700 1 --2- 172.21.15.44:0/685657191 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fef74039150 0x7fef7403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:35.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.446+0000 7fef89ffb700 1 -- 172.21.15.44:0/685657191 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fef7c01e070 con 0x7fef8c100b90 2024-04-21T22:59:35.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.446+0000 7fef8bfff700 1 --2- 172.21.15.44:0/685657191 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fef74039150 0x7fef7403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:35.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.446+0000 7fef8bfff700 1 --2- 172.21.15.44:0/685657191 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fef74039150 0x7fef7403b610 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fef80006fd0 tx=0x7fef80006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:35.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.451+0000 7fef89ffb700 1 -- 172.21.15.44:0/685657191 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fef7c013070 con 0x7fef8c100b90 2024-04-21T22:59:35.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.655+0000 7fef9312f700 1 -- 172.21.15.44:0/685657191 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fef8c190920 con 0x7fef8c100b90 2024-04-21T22:59:35.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.657+0000 7fef89ffb700 1 -- 172.21.15.44:0/685657191 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fef7c030020 con 0x7fef8c100b90 2024-04-21T22:59:35.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.660+0000 7fef9312f700 1 -- 172.21.15.44:0/685657191 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fef74039150 msgr2=0x7fef7403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:35.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.660+0000 7fef9312f700 1 --2- 172.21.15.44:0/685657191 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fef74039150 0x7fef7403b610 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fef80006fd0 tx=0x7fef80006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:35.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.660+0000 7fef9312f700 1 -- 172.21.15.44:0/685657191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef8c100b90 msgr2=0x7fef8c0fdff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:35.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.660+0000 7fef9312f700 1 --2- 172.21.15.44:0/685657191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef8c100b90 0x7fef8c0fdff0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fef7c005850 tx=0x7fef7c005030 comp rx=0 tx=0).stop 2024-04-21T22:59:35.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.660+0000 7fef9312f700 1 -- 172.21.15.44:0/685657191 shutdown_connections 2024-04-21T22:59:35.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.660+0000 7fef9312f700 1 --2- 172.21.15.44:0/685657191 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fef74039150 0x7fef7403b610 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:35.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.660+0000 7fef9312f700 1 --2- 172.21.15.44:0/685657191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fef8c100b90 0x7fef8c0fdff0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:35.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.660+0000 7fef9312f700 1 -- 172.21.15.44:0/685657191 >> 172.21.15.44:0/685657191 conn(0x7fef8c0fc130 msgr2=0x7fef8c069a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:35.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.660+0000 7fef9312f700 1 -- 172.21.15.44:0/685657191 shutdown_connections 2024-04-21T22:59:35.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:35.660+0000 7fef9312f700 1 -- 172.21.15.44:0/685657191 wait complete. 2024-04-21T22:59:36.209 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-21T22:59:36.209 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-21T22:59:36.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:36 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:36.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:36 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:36.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:36 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi044", "name": "osd_memory_target"}]: dispatch 2024-04-21T22:59:36.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:36 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:36.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:36 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T22:59:36.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:36 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:36.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:36 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/685657191' entity='client.admin' 2024-04-21T22:59:36.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:36 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T22:59:36.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:36 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:36.454 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T22:59:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.842+0000 7faf02324700 1 -- 172.21.15.44:0/2869434597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc100be0 msgr2=0x7faefc101000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.842+0000 7faf02324700 1 --2- 172.21.15.44:0/2869434597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc100be0 0x7faefc101000 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faeec009910 tx=0x7faeec009c20 comp rx=0 tx=0).stop 2024-04-21T22:59:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.842+0000 7faf02324700 1 -- 172.21.15.44:0/2869434597 shutdown_connections 2024-04-21T22:59:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.842+0000 7faf02324700 1 --2- 172.21.15.44:0/2869434597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc100be0 0x7faefc101000 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.842+0000 7faf02324700 1 -- 172.21.15.44:0/2869434597 >> 172.21.15.44:0/2869434597 conn(0x7faefc0fc140 msgr2=0x7faefc0fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.843+0000 7faf02324700 1 -- 172.21.15.44:0/2869434597 shutdown_connections 2024-04-21T22:59:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.843+0000 7faf02324700 1 -- 172.21.15.44:0/2869434597 wait complete. 2024-04-21T22:59:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.843+0000 7faf02324700 1 Processor -- start 2024-04-21T22:59:37.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.843+0000 7faf02324700 1 -- start start 2024-04-21T22:59:37.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.844+0000 7faf02324700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc100be0 0x7faefc06ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:37.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.844+0000 7faf02324700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faefc106c70 con 0x7faefc100be0 2024-04-21T22:59:37.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.844+0000 7faefbfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc100be0 0x7faefc06ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:37.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.844+0000 7faefbfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc100be0 0x7faefc06ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52338/0 (socket says 172.21.15.44:52338) 2024-04-21T22:59:37.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.844+0000 7faefbfff700 1 -- 172.21.15.44:0/3972158621 learned_addr learned my addr 172.21.15.44:0/3972158621 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:37.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.844+0000 7faefbfff700 1 -- 172.21.15.44:0/3972158621 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faefc19a0e0 con 0x7faefc100be0 2024-04-21T22:59:37.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.844+0000 7faefbfff700 1 --2- 172.21.15.44:0/3972158621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc100be0 0x7faefc06ab40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faeec006010 tx=0x7faeec005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:37.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.844+0000 7faef97fa700 1 -- 172.21.15.44:0/3972158621 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faeec027070 con 0x7faefc100be0 2024-04-21T22:59:37.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.844+0000 7faef97fa700 1 -- 172.21.15.44:0/3972158621 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faeec00be40 con 0x7faefc100be0 2024-04-21T22:59:37.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.844+0000 7faef97fa700 1 -- 172.21.15.44:0/3972158621 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faeec00f460 con 0x7faefc100be0 2024-04-21T22:59:37.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.844+0000 7faf02324700 1 -- 172.21.15.44:0/3972158621 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faefc19a2b0 con 0x7faefc100be0 2024-04-21T22:59:37.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.844+0000 7faf02324700 1 -- 172.21.15.44:0/3972158621 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faefc19a6d0 con 0x7faefc100be0 2024-04-21T22:59:37.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.845+0000 7faef97fa700 1 -- 172.21.15.44:0/3972158621 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7faeec02a070 con 0x7faefc100be0 2024-04-21T22:59:37.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.845+0000 7faf02324700 1 -- 172.21.15.44:0/3972158621 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faefc1073f0 con 0x7faefc100be0 2024-04-21T22:59:37.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.845+0000 7faef97fa700 1 --2- 172.21.15.44:0/3972158621 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7faee4039150 0x7faee403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:37.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.845+0000 7faef97fa700 1 -- 172.21.15.44:0/3972158621 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7faeec01e030 con 0x7faefc100be0 2024-04-21T22:59:37.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.846+0000 7faefb7fe700 1 --2- 172.21.15.44:0/3972158621 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7faee4039150 0x7faee403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:37.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.846+0000 7faefb7fe700 1 --2- 172.21.15.44:0/3972158621 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7faee4039150 0x7faee403b610 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faef0006fd0 tx=0x7faef0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:37.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:37.850+0000 7faef97fa700 1 -- 172.21.15.44:0/3972158621 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7faeec013070 con 0x7faefc100be0 2024-04-21T22:59:38.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:38.064+0000 7faf02324700 1 -- 172.21.15.44:0/3972158621 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7faefc02dca0 con 0x7faee4039150 2024-04-21T22:59:38.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:38.067+0000 7faef97fa700 1 -- 172.21.15.44:0/3972158621 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7faefc02dca0 con 0x7faee4039150 2024-04-21T22:59:38.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:38.069+0000 7faf02324700 1 -- 172.21.15.44:0/3972158621 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7faee4039150 msgr2=0x7faee403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:38.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:38.069+0000 7faf02324700 1 --2- 172.21.15.44:0/3972158621 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7faee4039150 0x7faee403b610 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faef0006fd0 tx=0x7faef0006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:38.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:38.069+0000 7faf02324700 1 -- 172.21.15.44:0/3972158621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc100be0 msgr2=0x7faefc06ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:38.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:38.069+0000 7faf02324700 1 --2- 172.21.15.44:0/3972158621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc100be0 0x7faefc06ab40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faeec006010 tx=0x7faeec005050 comp rx=0 tx=0).stop 2024-04-21T22:59:38.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:38.069+0000 7faf02324700 1 -- 172.21.15.44:0/3972158621 shutdown_connections 2024-04-21T22:59:38.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:38.069+0000 7faf02324700 1 --2- 172.21.15.44:0/3972158621 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7faee4039150 0x7faee403b610 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:38.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:38.069+0000 7faf02324700 1 --2- 172.21.15.44:0/3972158621 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faefc100be0 0x7faefc06ab40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:38.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:38.069+0000 7faf02324700 1 -- 172.21.15.44:0/3972158621 >> 172.21.15.44:0/3972158621 conn(0x7faefc0fc140 msgr2=0x7faefc0fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:38.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:38.069+0000 7faf02324700 1 -- 172.21.15.44:0/3972158621 shutdown_connections 2024-04-21T22:59:38.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:38.069+0000 7faf02324700 1 -- 172.21.15.44:0/3972158621 wait complete. 2024-04-21T22:59:38.824 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi055 2024-04-21T22:59:38.824 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-21T22:59:38.824 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/etc/ceph/ceph.conf 2024-04-21T22:59:38.845 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-21T22:59:38.845 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T22:59:38.903 INFO:tasks.cephadm:Adding host smithi055 to orchestrator... 2024-04-21T22:59:38.904 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph orch host add smithi055 2024-04-21T22:59:39.152 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T22:59:39.350 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:39 smithi044 ceph-mon[27934]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T22:59:39.350 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:39.350 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T22:59:39.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T22:59:39.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T22:59:39.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:39 smithi044 ceph-mon[27934]: Updating smithi044:/etc/ceph/ceph.conf 2024-04-21T22:59:39.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:39 smithi044 ceph-mon[27934]: Updating smithi044:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T22:59:39.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:39 smithi044 ceph-mon[27934]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2024-04-21T22:59:39.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:39 smithi044 ceph-mon[27934]: Updating smithi044:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T22:59:39.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:39.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:39.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:40.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.612+0000 7f996ebec700 1 -- 172.21.15.44:0/3158378979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9968101ed0 msgr2=0x7f99681022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:40.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.612+0000 7f996ebec700 1 --2- 172.21.15.44:0/3158378979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9968101ed0 0x7f99681022b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f995c009910 tx=0x7f995c009c20 comp rx=0 tx=0).stop 2024-04-21T22:59:40.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.612+0000 7f996ebec700 1 -- 172.21.15.44:0/3158378979 shutdown_connections 2024-04-21T22:59:40.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.612+0000 7f996ebec700 1 --2- 172.21.15.44:0/3158378979 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9968101ed0 0x7f99681022b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:40.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.612+0000 7f996ebec700 1 -- 172.21.15.44:0/3158378979 >> 172.21.15.44:0/3158378979 conn(0x7f99680fd760 msgr2=0x7f99680ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:40.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.612+0000 7f996ebec700 1 -- 172.21.15.44:0/3158378979 shutdown_connections 2024-04-21T22:59:40.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.612+0000 7f996ebec700 1 -- 172.21.15.44:0/3158378979 wait complete. 2024-04-21T22:59:40.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.612+0000 7f996ebec700 1 Processor -- start 2024-04-21T22:59:40.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.612+0000 7f996ebec700 1 -- start start 2024-04-21T22:59:40.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.613+0000 7f996ebec700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9968101ed0 0x7f9968074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:40.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.613+0000 7f996ebec700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f996810c2a0 con 0x7f9968101ed0 2024-04-21T22:59:40.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.613+0000 7f996c988700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9968101ed0 0x7f9968074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:40.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.613+0000 7f996c988700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9968101ed0 0x7f9968074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56118/0 (socket says 172.21.15.44:56118) 2024-04-21T22:59:40.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.613+0000 7f996c988700 1 -- 172.21.15.44:0/3189575569 learned_addr learned my addr 172.21.15.44:0/3189575569 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:40.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.613+0000 7f996c988700 1 -- 172.21.15.44:0/3189575569 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9968076420 con 0x7f9968101ed0 2024-04-21T22:59:40.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.613+0000 7f996c988700 1 --2- 172.21.15.44:0/3189575569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9968101ed0 0x7f9968074530 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f995c005850 tx=0x7f995c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:40.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.613+0000 7f9965ffb700 1 -- 172.21.15.44:0/3189575569 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f995c026070 con 0x7f9968101ed0 2024-04-21T22:59:40.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.613+0000 7f996ebec700 1 -- 172.21.15.44:0/3189575569 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9968074ad0 con 0x7f9968101ed0 2024-04-21T22:59:40.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.613+0000 7f9965ffb700 1 -- 172.21.15.44:0/3189575569 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f995c00bc30 con 0x7f9968101ed0 2024-04-21T22:59:40.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.613+0000 7f9965ffb700 1 -- 172.21.15.44:0/3189575569 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f995c003a20 con 0x7f9968101ed0 2024-04-21T22:59:40.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.614+0000 7f996ebec700 1 -- 172.21.15.44:0/3189575569 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9968074f90 con 0x7f9968101ed0 2024-04-21T22:59:40.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.614+0000 7f9965ffb700 1 -- 172.21.15.44:0/3189575569 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f995c029070 con 0x7f9968101ed0 2024-04-21T22:59:40.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.614+0000 7f996ebec700 1 -- 172.21.15.44:0/3189575569 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99681033b0 con 0x7f9968101ed0 2024-04-21T22:59:40.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.614+0000 7f9965ffb700 1 --2- 172.21.15.44:0/3189575569 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f99580390f0 0x7f995803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:40.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.615+0000 7f9965ffb700 1 -- 172.21.15.44:0/3189575569 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f995c01e070 con 0x7f9968101ed0 2024-04-21T22:59:40.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.615+0000 7f9967fff700 1 --2- 172.21.15.44:0/3189575569 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f99580390f0 0x7f995803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:40.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.615+0000 7f9967fff700 1 --2- 172.21.15.44:0/3189575569 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f99580390f0 0x7f995803b5b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9954006fd0 tx=0x7f9954006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:40.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.620+0000 7f9965ffb700 1 -- 172.21.15.44:0/3189575569 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f995c013070 con 0x7f9968101ed0 2024-04-21T22:59:40.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:40.832+0000 7f996ebec700 1 -- 172.21.15.44:0/3189575569 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi055", "target": ["mon-mgr", ""]}) -- 0x7f9968075220 con 0x7f99580390f0 2024-04-21T22:59:41.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:40 smithi044 ceph-mon[27934]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi055", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T22:59:42.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:41 smithi044 ceph-mon[27934]: Deploying cephadm binary to smithi055 2024-04-21T22:59:44.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:43 smithi044 ceph-mon[27934]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T22:59:45.052 INFO:teuthology.orchestra.run.smithi044.stdout:Added host 'smithi055' with addr '172.21.15.55' 2024-04-21T22:59:45.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:45.051+0000 7f9965ffb700 1 -- 172.21.15.44:0/3189575569 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f9968075220 con 0x7f99580390f0 2024-04-21T22:59:45.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:45.054+0000 7f996ebec700 1 -- 172.21.15.44:0/3189575569 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f99580390f0 msgr2=0x7f995803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:45.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:45.054+0000 7f996ebec700 1 --2- 172.21.15.44:0/3189575569 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f99580390f0 0x7f995803b5b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9954006fd0 tx=0x7f9954006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:45.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:45.054+0000 7f996ebec700 1 -- 172.21.15.44:0/3189575569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9968101ed0 msgr2=0x7f9968074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:45.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:45.054+0000 7f996ebec700 1 --2- 172.21.15.44:0/3189575569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9968101ed0 0x7f9968074530 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f995c005850 tx=0x7f995c005030 comp rx=0 tx=0).stop 2024-04-21T22:59:45.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:45.054+0000 7f996ebec700 1 -- 172.21.15.44:0/3189575569 shutdown_connections 2024-04-21T22:59:45.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:45.054+0000 7f996ebec700 1 --2- 172.21.15.44:0/3189575569 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f99580390f0 0x7f995803b5b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:45.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:45.054+0000 7f996ebec700 1 --2- 172.21.15.44:0/3189575569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9968101ed0 0x7f9968074530 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:45.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:45.054+0000 7f996ebec700 1 -- 172.21.15.44:0/3189575569 >> 172.21.15.44:0/3189575569 conn(0x7f99680fd760 msgr2=0x7f9968104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:45.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:45.055+0000 7f996ebec700 1 -- 172.21.15.44:0/3189575569 shutdown_connections 2024-04-21T22:59:45.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:45.055+0000 7f996ebec700 1 -- 172.21.15.44:0/3189575569 wait complete. 2024-04-21T22:59:45.595 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-21T22:59:45.845 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T22:59:46.270 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:46 smithi044 ceph-mon[27934]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T22:59:46.270 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:46 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:46.270 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:46 smithi044 ceph-mon[27934]: Added host smithi055 2024-04-21T22:59:46.270 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:46 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T22:59:46.271 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:46 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:47.222 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:47 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:47.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.323+0000 7efcbdcee700 1 -- 172.21.15.44:0/2439599769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efcb8106c00 msgr2=0x7efcb8106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:47.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.323+0000 7efcbdcee700 1 --2- 172.21.15.44:0/2439599769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efcb8106c00 0x7efcb8106fe0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efca8009910 tx=0x7efca8009c20 comp rx=0 tx=0).stop 2024-04-21T22:59:47.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.323+0000 7efcbdcee700 1 -- 172.21.15.44:0/2439599769 shutdown_connections 2024-04-21T22:59:47.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.323+0000 7efcbdcee700 1 --2- 172.21.15.44:0/2439599769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efcb8106c00 0x7efcb8106fe0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:47.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.323+0000 7efcbdcee700 1 -- 172.21.15.44:0/2439599769 >> 172.21.15.44:0/2439599769 conn(0x7efcb8076690 msgr2=0x7efcb8076aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:47.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.323+0000 7efcbdcee700 1 -- 172.21.15.44:0/2439599769 shutdown_connections 2024-04-21T22:59:47.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.323+0000 7efcbdcee700 1 -- 172.21.15.44:0/2439599769 wait complete. 2024-04-21T22:59:47.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.324+0000 7efcbdcee700 1 Processor -- start 2024-04-21T22:59:47.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.324+0000 7efcbdcee700 1 -- start start 2024-04-21T22:59:47.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.324+0000 7efcbdcee700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efcb8106c00 0x7efcb81985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:47.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.324+0000 7efcbdcee700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7efcb810b1c0 con 0x7efcb8106c00 2024-04-21T22:59:47.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.324+0000 7efcb77fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efcb8106c00 0x7efcb81985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:47.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.324+0000 7efcb77fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efcb8106c00 0x7efcb81985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56130/0 (socket says 172.21.15.44:56130) 2024-04-21T22:59:47.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.324+0000 7efcb77fe700 1 -- 172.21.15.44:0/4136312103 learned_addr learned my addr 172.21.15.44:0/4136312103 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:47.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.325+0000 7efcb77fe700 1 -- 172.21.15.44:0/4136312103 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb8198af0 con 0x7efcb8106c00 2024-04-21T22:59:47.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.325+0000 7efcb77fe700 1 --2- 172.21.15.44:0/4136312103 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efcb8106c00 0x7efcb81985b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efca8005850 tx=0x7efca8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:47.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.325+0000 7efcb4ff9700 1 -- 172.21.15.44:0/4136312103 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efca8026070 con 0x7efcb8106c00 2024-04-21T22:59:47.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.325+0000 7efcb4ff9700 1 -- 172.21.15.44:0/4136312103 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efca800bc30 con 0x7efcb8106c00 2024-04-21T22:59:47.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.325+0000 7efcb4ff9700 1 -- 172.21.15.44:0/4136312103 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efca8003a20 con 0x7efcb8106c00 2024-04-21T22:59:47.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.325+0000 7efcbdcee700 1 -- 172.21.15.44:0/4136312103 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcb8198d80 con 0x7efcb8106c00 2024-04-21T22:59:47.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.325+0000 7efcbdcee700 1 -- 172.21.15.44:0/4136312103 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcb81991e0 con 0x7efcb8106c00 2024-04-21T22:59:47.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.326+0000 7efcbdcee700 1 -- 172.21.15.44:0/4136312103 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcb8102200 con 0x7efcb8106c00 2024-04-21T22:59:47.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.326+0000 7efcb4ff9700 1 -- 172.21.15.44:0/4136312103 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7efca8029070 con 0x7efcb8106c00 2024-04-21T22:59:47.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.326+0000 7efcb4ff9700 1 --2- 172.21.15.44:0/4136312103 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7efca0039150 0x7efca003b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:47.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.326+0000 7efcb4ff9700 1 -- 172.21.15.44:0/4136312103 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7efca801e070 con 0x7efcb8106c00 2024-04-21T22:59:47.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.327+0000 7efcb6ffd700 1 --2- 172.21.15.44:0/4136312103 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7efca0039150 0x7efca003b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:47.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.327+0000 7efcb6ffd700 1 --2- 172.21.15.44:0/4136312103 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7efca0039150 0x7efca003b610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efcac006fd0 tx=0x7efcac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:47.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.331+0000 7efcb4ff9700 1 -- 172.21.15.44:0/4136312103 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7efca8013070 con 0x7efcb8106c00 2024-04-21T22:59:47.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.541+0000 7efcbdcee700 1 -- 172.21.15.44:0/4136312103 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efcb8063890 con 0x7efca0039150 2024-04-21T22:59:47.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.542+0000 7efcb4ff9700 1 -- 172.21.15.44:0/4136312103 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7efcb8063890 con 0x7efca0039150 2024-04-21T22:59:47.542 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:59:47.542 INFO:teuthology.orchestra.run.smithi044.stdout:[{"addr": "172.21.15.44", "hostname": "smithi044", "labels": [], "status": ""}, {"addr": "172.21.15.55", "hostname": "smithi055", "labels": [], "status": ""}] 2024-04-21T22:59:47.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.544+0000 7efcbdcee700 1 -- 172.21.15.44:0/4136312103 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7efca0039150 msgr2=0x7efca003b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:47.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.544+0000 7efcbdcee700 1 --2- 172.21.15.44:0/4136312103 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7efca0039150 0x7efca003b610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efcac006fd0 tx=0x7efcac006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:47.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.544+0000 7efcbdcee700 1 -- 172.21.15.44:0/4136312103 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efcb8106c00 msgr2=0x7efcb81985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:47.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.544+0000 7efcbdcee700 1 --2- 172.21.15.44:0/4136312103 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efcb8106c00 0x7efcb81985b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efca8005850 tx=0x7efca8005030 comp rx=0 tx=0).stop 2024-04-21T22:59:47.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.544+0000 7efcbdcee700 1 -- 172.21.15.44:0/4136312103 shutdown_connections 2024-04-21T22:59:47.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.544+0000 7efcbdcee700 1 --2- 172.21.15.44:0/4136312103 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7efca0039150 0x7efca003b610 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:47.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.544+0000 7efcbdcee700 1 --2- 172.21.15.44:0/4136312103 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efcb8106c00 0x7efcb81985b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:47.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.544+0000 7efcbdcee700 1 -- 172.21.15.44:0/4136312103 >> 172.21.15.44:0/4136312103 conn(0x7efcb8076690 msgr2=0x7efcb80fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:47.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.545+0000 7efcbdcee700 1 -- 172.21.15.44:0/4136312103 shutdown_connections 2024-04-21T22:59:47.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:47.545+0000 7efcbdcee700 1 -- 172.21.15.44:0/4136312103 wait complete. 2024-04-21T22:59:48.159 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi069 2024-04-21T22:59:48.159 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-21T22:59:48.159 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/etc/ceph/ceph.conf 2024-04-21T22:59:48.177 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-21T22:59:48.177 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T22:59:48.235 INFO:tasks.cephadm:Adding host smithi069 to orchestrator... 2024-04-21T22:59:48.235 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph orch host add smithi069 2024-04-21T22:59:48.476 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T22:59:48.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:48 smithi044 ceph-mon[27934]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T22:59:48.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:48 smithi044 ceph-mon[27934]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T22:59:48.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:48 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:49.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.769+0000 7ff3351cf700 1 -- 172.21.15.44:0/4241974324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff330101ed0 msgr2=0x7ff3301022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:49.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.769+0000 7ff3351cf700 1 --2- 172.21.15.44:0/4241974324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff330101ed0 0x7ff3301022b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff320009910 tx=0x7ff320009c20 comp rx=0 tx=0).stop 2024-04-21T22:59:49.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.769+0000 7ff3351cf700 1 -- 172.21.15.44:0/4241974324 shutdown_connections 2024-04-21T22:59:49.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.769+0000 7ff3351cf700 1 --2- 172.21.15.44:0/4241974324 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff330101ed0 0x7ff3301022b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:49.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.769+0000 7ff3351cf700 1 -- 172.21.15.44:0/4241974324 >> 172.21.15.44:0/4241974324 conn(0x7ff3300fd760 msgr2=0x7ff3300ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:49.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.769+0000 7ff3351cf700 1 -- 172.21.15.44:0/4241974324 shutdown_connections 2024-04-21T22:59:49.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.769+0000 7ff3351cf700 1 -- 172.21.15.44:0/4241974324 wait complete. 2024-04-21T22:59:49.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.770+0000 7ff3351cf700 1 Processor -- start 2024-04-21T22:59:49.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.770+0000 7ff3351cf700 1 -- start start 2024-04-21T22:59:49.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.770+0000 7ff3351cf700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff330101ed0 0x7ff33019d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:49.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.770+0000 7ff3351cf700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff33010c2a0 con 0x7ff330101ed0 2024-04-21T22:59:49.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.771+0000 7ff32ed9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff330101ed0 0x7ff33019d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:49.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.771+0000 7ff32ed9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff330101ed0 0x7ff33019d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56142/0 (socket says 172.21.15.44:56142) 2024-04-21T22:59:49.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.771+0000 7ff32ed9d700 1 -- 172.21.15.44:0/2816575922 learned_addr learned my addr 172.21.15.44:0/2816575922 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:49.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.771+0000 7ff32ed9d700 1 -- 172.21.15.44:0/2816575922 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff33019d950 con 0x7ff330101ed0 2024-04-21T22:59:49.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.771+0000 7ff32ed9d700 1 --2- 172.21.15.44:0/2816575922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff330101ed0 0x7ff33019d410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff320006010 tx=0x7ff320005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:49.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.771+0000 7ff317fff700 1 -- 172.21.15.44:0/2816575922 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff320026070 con 0x7ff330101ed0 2024-04-21T22:59:49.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.771+0000 7ff3351cf700 1 -- 172.21.15.44:0/2816575922 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff33019dc40 con 0x7ff330101ed0 2024-04-21T22:59:49.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.771+0000 7ff317fff700 1 -- 172.21.15.44:0/2816575922 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff32000bbd0 con 0x7ff330101ed0 2024-04-21T22:59:49.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.771+0000 7ff317fff700 1 -- 172.21.15.44:0/2816575922 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff320003910 con 0x7ff330101ed0 2024-04-21T22:59:49.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.771+0000 7ff3351cf700 1 -- 172.21.15.44:0/2816575922 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3301975d0 con 0x7ff330101ed0 2024-04-21T22:59:49.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.772+0000 7ff317fff700 1 -- 172.21.15.44:0/2816575922 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7ff320029070 con 0x7ff330101ed0 2024-04-21T22:59:49.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.772+0000 7ff3351cf700 1 -- 172.21.15.44:0/2816575922 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3301033b0 con 0x7ff330101ed0 2024-04-21T22:59:49.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.772+0000 7ff317fff700 1 --2- 172.21.15.44:0/2816575922 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff3180390f0 0x7ff31803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:49.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.772+0000 7ff317fff700 1 -- 172.21.15.44:0/2816575922 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff32001e070 con 0x7ff330101ed0 2024-04-21T22:59:49.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.773+0000 7ff32e59c700 1 --2- 172.21.15.44:0/2816575922 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff3180390f0 0x7ff31803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:49.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.773+0000 7ff32e59c700 1 --2- 172.21.15.44:0/2816575922 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff3180390f0 0x7ff31803b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff324006fd0 tx=0x7ff324006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:49.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.777+0000 7ff317fff700 1 -- 172.21.15.44:0/2816575922 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff320013070 con 0x7ff330101ed0 2024-04-21T22:59:49.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:49.987+0000 7ff3351cf700 1 -- 172.21.15.44:0/2816575922 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi069", "target": ["mon-mgr", ""]}) -- 0x7ff330198b00 con 0x7ff3180390f0 2024-04-21T22:59:51.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:50 smithi044 ceph-mon[27934]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T22:59:52.129 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:51 smithi044 ceph-mon[27934]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi069", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T22:59:52.129 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:51 smithi044 ceph-mon[27934]: Deploying cephadm binary to smithi069 2024-04-21T22:59:52.129 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:51 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:52.129 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:51 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:52.129 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:51 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:52.129 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:51 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:52.129 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:51 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi055", "name": "osd_memory_target"}]: dispatch 2024-04-21T22:59:52.129 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:51 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T22:59:52.130 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:51 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T22:59:52.130 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:51 smithi044 ceph-mon[27934]: Updating smithi055:/etc/ceph/ceph.conf 2024-04-21T22:59:53.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:52 smithi044 ceph-mon[27934]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T22:59:53.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:52 smithi044 ceph-mon[27934]: Updating smithi055:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T22:59:54.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:54.185+0000 7ff317fff700 1 -- 172.21.15.44:0/2816575922 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7ff330198b00 con 0x7ff3180390f0 2024-04-21T22:59:54.186 INFO:teuthology.orchestra.run.smithi044.stdout:Added host 'smithi069' with addr '172.21.15.69' 2024-04-21T22:59:54.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:54.187+0000 7ff3351cf700 1 -- 172.21.15.44:0/2816575922 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff3180390f0 msgr2=0x7ff31803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:54.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:54.187+0000 7ff3351cf700 1 --2- 172.21.15.44:0/2816575922 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff3180390f0 0x7ff31803b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff324006fd0 tx=0x7ff324006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:54.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:54.187+0000 7ff3351cf700 1 -- 172.21.15.44:0/2816575922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff330101ed0 msgr2=0x7ff33019d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:54.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:54.187+0000 7ff3351cf700 1 --2- 172.21.15.44:0/2816575922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff330101ed0 0x7ff33019d410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff320006010 tx=0x7ff320005030 comp rx=0 tx=0).stop 2024-04-21T22:59:54.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:54.188+0000 7ff3351cf700 1 -- 172.21.15.44:0/2816575922 shutdown_connections 2024-04-21T22:59:54.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:54.188+0000 7ff3351cf700 1 --2- 172.21.15.44:0/2816575922 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff3180390f0 0x7ff31803b5b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:54.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:54.188+0000 7ff3351cf700 1 --2- 172.21.15.44:0/2816575922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff330101ed0 0x7ff33019d410 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:54.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:54.188+0000 7ff3351cf700 1 -- 172.21.15.44:0/2816575922 >> 172.21.15.44:0/2816575922 conn(0x7ff3300fd760 msgr2=0x7ff330104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:54.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:54.188+0000 7ff3351cf700 1 -- 172.21.15.44:0/2816575922 shutdown_connections 2024-04-21T22:59:54.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:54.188+0000 7ff3351cf700 1 -- 172.21.15.44:0/2816575922 wait complete. 2024-04-21T22:59:54.755 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-21T22:59:54.997 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T22:59:55.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:54 smithi044 ceph-mon[27934]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T22:59:55.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:54 smithi044 ceph-mon[27934]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2024-04-21T22:59:55.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:55.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:55.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:55.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:55.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T22:59:55.120 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:56.124 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:55 smithi044 ceph-mon[27934]: Updating smithi055:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T22:59:56.124 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:55 smithi044 ceph-mon[27934]: Added host smithi069 2024-04-21T22:59:56.124 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:55 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:56.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.473+0000 7fb83f37d700 1 -- 172.21.15.44:0/3338091773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb838101ea0 msgr2=0x7fb838102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:56.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.473+0000 7fb83f37d700 1 --2- 172.21.15.44:0/3338091773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb838101ea0 0x7fb838102280 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb834009910 tx=0x7fb834009c20 comp rx=0 tx=0).stop 2024-04-21T22:59:56.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.473+0000 7fb83f37d700 1 -- 172.21.15.44:0/3338091773 shutdown_connections 2024-04-21T22:59:56.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.473+0000 7fb83f37d700 1 --2- 172.21.15.44:0/3338091773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb838101ea0 0x7fb838102280 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:56.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.473+0000 7fb83f37d700 1 -- 172.21.15.44:0/3338091773 >> 172.21.15.44:0/3338091773 conn(0x7fb8380fd770 msgr2=0x7fb8380ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:56.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.473+0000 7fb83f37d700 1 -- 172.21.15.44:0/3338091773 shutdown_connections 2024-04-21T22:59:56.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.473+0000 7fb83f37d700 1 -- 172.21.15.44:0/3338091773 wait complete. 2024-04-21T22:59:56.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.474+0000 7fb83f37d700 1 Processor -- start 2024-04-21T22:59:56.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.474+0000 7fb83f37d700 1 -- start start 2024-04-21T22:59:56.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.474+0000 7fb83f37d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb838101ea0 0x7fb83819d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:56.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.474+0000 7fb83f37d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb83810c270 con 0x7fb838101ea0 2024-04-21T22:59:56.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.474+0000 7fb83d119700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb838101ea0 0x7fb83819d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:56.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.474+0000 7fb83d119700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb838101ea0 0x7fb83819d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56718/0 (socket says 172.21.15.44:56718) 2024-04-21T22:59:56.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.474+0000 7fb83d119700 1 -- 172.21.15.44:0/259035461 learned_addr learned my addr 172.21.15.44:0/259035461 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:56.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.474+0000 7fb83d119700 1 -- 172.21.15.44:0/259035461 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb83819dab0 con 0x7fb838101ea0 2024-04-21T22:59:56.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.474+0000 7fb83d119700 1 --2- 172.21.15.44:0/259035461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb838101ea0 0x7fb83819d570 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb834005850 tx=0x7fb834005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:56.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.474+0000 7fb82e7fc700 1 -- 172.21.15.44:0/259035461 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb834026070 con 0x7fb838101ea0 2024-04-21T22:59:56.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.475+0000 7fb82e7fc700 1 -- 172.21.15.44:0/259035461 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb83400bbd0 con 0x7fb838101ea0 2024-04-21T22:59:56.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.475+0000 7fb83f37d700 1 -- 172.21.15.44:0/259035461 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb83819dda0 con 0x7fb838101ea0 2024-04-21T22:59:56.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.475+0000 7fb82e7fc700 1 -- 172.21.15.44:0/259035461 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb834003970 con 0x7fb838101ea0 2024-04-21T22:59:56.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.475+0000 7fb83f37d700 1 -- 172.21.15.44:0/259035461 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb838197810 con 0x7fb838101ea0 2024-04-21T22:59:56.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.475+0000 7fb82e7fc700 1 -- 172.21.15.44:0/259035461 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fb834029070 con 0x7fb838101ea0 2024-04-21T22:59:56.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.475+0000 7fb83f37d700 1 -- 172.21.15.44:0/259035461 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb838103380 con 0x7fb838101ea0 2024-04-21T22:59:56.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.476+0000 7fb82e7fc700 1 --2- 172.21.15.44:0/259035461 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb824039150 0x7fb82403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:56.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.476+0000 7fb82e7fc700 1 -- 172.21.15.44:0/259035461 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb83401e070 con 0x7fb838101ea0 2024-04-21T22:59:56.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.476+0000 7fb83c918700 1 --2- 172.21.15.44:0/259035461 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb824039150 0x7fb82403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:56.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.476+0000 7fb83c918700 1 --2- 172.21.15.44:0/259035461 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb824039150 0x7fb82403b610 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb828006fd0 tx=0x7fb828006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:56.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.481+0000 7fb82e7fc700 1 -- 172.21.15.44:0/259035461 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb834051070 con 0x7fb838101ea0 2024-04-21T22:59:56.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.691+0000 7fb83f37d700 1 -- 172.21.15.44:0/259035461 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb838198b70 con 0x7fb824039150 2024-04-21T22:59:56.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.692+0000 7fb82e7fc700 1 -- 172.21.15.44:0/259035461 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+238 (secure 0 0 0) 0x7fb838198b70 con 0x7fb824039150 2024-04-21T22:59:56.692 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T22:59:56.692 INFO:teuthology.orchestra.run.smithi044.stdout:[{"addr": "172.21.15.44", "hostname": "smithi044", "labels": [], "status": ""}, {"addr": "172.21.15.55", "hostname": "smithi055", "labels": [], "status": ""}, {"addr": "172.21.15.69", "hostname": "smithi069", "labels": [], "status": ""}] 2024-04-21T22:59:56.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.694+0000 7fb83f37d700 1 -- 172.21.15.44:0/259035461 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb824039150 msgr2=0x7fb82403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:56.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.694+0000 7fb83f37d700 1 --2- 172.21.15.44:0/259035461 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb824039150 0x7fb82403b610 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb828006fd0 tx=0x7fb828006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:56.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.694+0000 7fb83f37d700 1 -- 172.21.15.44:0/259035461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb838101ea0 msgr2=0x7fb83819d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:56.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.694+0000 7fb83f37d700 1 --2- 172.21.15.44:0/259035461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb838101ea0 0x7fb83819d570 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb834005850 tx=0x7fb834005030 comp rx=0 tx=0).stop 2024-04-21T22:59:56.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.694+0000 7fb83f37d700 1 -- 172.21.15.44:0/259035461 shutdown_connections 2024-04-21T22:59:56.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.694+0000 7fb83f37d700 1 --2- 172.21.15.44:0/259035461 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb824039150 0x7fb82403b610 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:56.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.694+0000 7fb83f37d700 1 --2- 172.21.15.44:0/259035461 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb838101ea0 0x7fb83819d570 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:56.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.694+0000 7fb83f37d700 1 -- 172.21.15.44:0/259035461 >> 172.21.15.44:0/259035461 conn(0x7fb8380fd770 msgr2=0x7fb838104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:56.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.694+0000 7fb83f37d700 1 -- 172.21.15.44:0/259035461 shutdown_connections 2024-04-21T22:59:56.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:56.694+0000 7fb83f37d700 1 -- 172.21.15.44:0/259035461 wait complete. 2024-04-21T22:59:57.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:56 smithi044 ceph-mon[27934]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T22:59:57.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:56 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T22:59:57.292 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-21T22:59:57.292 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-21T22:59:57.532 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T22:59:57.926 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:57 smithi044 ceph-mon[27934]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T22:59:59.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.002+0000 7f3323824700 1 -- 172.21.15.44:0/1786361824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f331c106bf0 msgr2=0x7f331c106fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:59.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.002+0000 7f3323824700 1 --2- 172.21.15.44:0/1786361824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f331c106bf0 0x7f331c106fd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3318009910 tx=0x7f3318009c20 comp rx=0 tx=0).stop 2024-04-21T22:59:59.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.002+0000 7f3323824700 1 -- 172.21.15.44:0/1786361824 shutdown_connections 2024-04-21T22:59:59.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.002+0000 7f3323824700 1 --2- 172.21.15.44:0/1786361824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f331c106bf0 0x7f331c106fd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:59.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.002+0000 7f3323824700 1 -- 172.21.15.44:0/1786361824 >> 172.21.15.44:0/1786361824 conn(0x7f331c0fc630 msgr2=0x7f331c0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:59.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.002+0000 7f3323824700 1 -- 172.21.15.44:0/1786361824 shutdown_connections 2024-04-21T22:59:59.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.002+0000 7f3323824700 1 -- 172.21.15.44:0/1786361824 wait complete. 2024-04-21T22:59:59.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.003+0000 7f3323824700 1 Processor -- start 2024-04-21T22:59:59.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.003+0000 7f3323824700 1 -- start start 2024-04-21T22:59:59.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.003+0000 7f3323824700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f331c106bf0 0x7f331c0fcae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:59.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.003+0000 7f3323824700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f331c10b1b0 con 0x7f331c106bf0 2024-04-21T22:59:59.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.003+0000 7f33215c0700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f331c106bf0 0x7f331c0fcae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:59.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.003+0000 7f33215c0700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f331c106bf0 0x7f331c0fcae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56724/0 (socket says 172.21.15.44:56724) 2024-04-21T22:59:59.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.003+0000 7f33215c0700 1 -- 172.21.15.44:0/2151677110 learned_addr learned my addr 172.21.15.44:0/2151677110 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T22:59:59.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.003+0000 7f33215c0700 1 -- 172.21.15.44:0/2151677110 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f331c0fe9d0 con 0x7f331c106bf0 2024-04-21T22:59:59.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.003+0000 7f33215c0700 1 --2- 172.21.15.44:0/2151677110 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f331c106bf0 0x7f331c0fcae0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3318005850 tx=0x7f3318005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.003+0000 7f33127fc700 1 -- 172.21.15.44:0/2151677110 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3318026070 con 0x7f331c106bf0 2024-04-21T22:59:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.003+0000 7f33127fc700 1 -- 172.21.15.44:0/2151677110 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f331800bc30 con 0x7f331c106bf0 2024-04-21T22:59:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.004+0000 7f33127fc700 1 -- 172.21.15.44:0/2151677110 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3318003a20 con 0x7f331c106bf0 2024-04-21T22:59:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.004+0000 7f3323824700 1 -- 172.21.15.44:0/2151677110 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f331c0fd020 con 0x7f331c106bf0 2024-04-21T22:59:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.004+0000 7f3323824700 1 -- 172.21.15.44:0/2151677110 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f331c0fd4e0 con 0x7f331c106bf0 2024-04-21T22:59:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.005+0000 7f33127fc700 1 -- 172.21.15.44:0/2151677110 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f3318029070 con 0x7f331c106bf0 2024-04-21T22:59:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.005+0000 7f3323824700 1 -- 172.21.15.44:0/2151677110 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f331c1021f0 con 0x7f331c106bf0 2024-04-21T22:59:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.005+0000 7f33127fc700 1 --2- 172.21.15.44:0/2151677110 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f3308039150 0x7f330803b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T22:59:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.005+0000 7f33127fc700 1 -- 172.21.15.44:0/2151677110 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f331801e070 con 0x7f331c106bf0 2024-04-21T22:59:59.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.005+0000 7f3320dbf700 1 --2- 172.21.15.44:0/2151677110 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f3308039150 0x7f330803b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T22:59:59.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.005+0000 7f3320dbf700 1 --2- 172.21.15.44:0/2151677110 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f3308039150 0x7f330803b610 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f330c006fd0 tx=0x7f330c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T22:59:59.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.010+0000 7f33127fc700 1 -- 172.21.15.44:0/2151677110 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3318050030 con 0x7f331c106bf0 2024-04-21T22:59:59.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:58 smithi044 ceph-mon[27934]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T22:59:59.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.221+0000 7f3323824700 1 -- 172.21.15.44:0/2151677110 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f331c0fe010 con 0x7f331c106bf0 2024-04-21T22:59:59.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.861+0000 7f33127fc700 1 -- 172.21.15.44:0/2151677110 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f3318030090 con 0x7f331c106bf0 2024-04-21T22:59:59.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.864+0000 7f3323824700 1 -- 172.21.15.44:0/2151677110 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f3308039150 msgr2=0x7f330803b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:59.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.864+0000 7f3323824700 1 --2- 172.21.15.44:0/2151677110 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f3308039150 0x7f330803b610 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f330c006fd0 tx=0x7f330c006e40 comp rx=0 tx=0).stop 2024-04-21T22:59:59.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.864+0000 7f3323824700 1 -- 172.21.15.44:0/2151677110 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f331c106bf0 msgr2=0x7f331c0fcae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T22:59:59.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.864+0000 7f3323824700 1 --2- 172.21.15.44:0/2151677110 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f331c106bf0 0x7f331c0fcae0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3318005850 tx=0x7f3318005030 comp rx=0 tx=0).stop 2024-04-21T22:59:59.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.864+0000 7f3323824700 1 -- 172.21.15.44:0/2151677110 shutdown_connections 2024-04-21T22:59:59.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.864+0000 7f3323824700 1 --2- 172.21.15.44:0/2151677110 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f3308039150 0x7f330803b610 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:59.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.864+0000 7f3323824700 1 --2- 172.21.15.44:0/2151677110 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f331c106bf0 0x7f331c0fcae0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T22:59:59.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.864+0000 7f3323824700 1 -- 172.21.15.44:0/2151677110 >> 172.21.15.44:0/2151677110 conn(0x7f331c0fc630 msgr2=0x7f331c069ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T22:59:59.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.864+0000 7f3323824700 1 -- 172.21.15.44:0/2151677110 shutdown_connections 2024-04-21T22:59:59.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T22:59:59.864+0000 7f3323824700 1 -- 172.21.15.44:0/2151677110 wait complete. 2024-04-21T22:59:59.866 INFO:teuthology.orchestra.run.smithi044.stderr:adjusted tunables profile to default 2024-04-21T23:00:00.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 22:59:59 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2151677110' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-21T23:00:00.532 INFO:tasks.cephadm:Adding mon.a on smithi044 2024-04-21T23:00:00.532 INFO:tasks.cephadm:Adding mon.b on smithi055 2024-04-21T23:00:00.532 INFO:tasks.cephadm:Adding mon.c on smithi069 2024-04-21T23:00:00.532 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi044:172.21.15.44=a;smithi055:172.21.15.55=b;smithi069:172.21.15.69=c' 2024-04-21T23:00:00.764 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:00:01.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:00 smithi044 ceph-mon[27934]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:01.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:00 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2151677110' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-21T23:00:01.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:00 smithi044 ceph-mon[27934]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:00:01.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:00 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:01.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:00 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:01.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:00 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:01.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:00 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:01.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:00 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi069", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:00:01.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:00 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:01.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:00 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:01.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:00 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:01.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:00 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:01.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:00 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:02.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.139+0000 7f08dec12700 1 -- 172.21.15.69:0/850408915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08d8100da0 msgr2=0x7f08d8101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:02.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.139+0000 7f08dec12700 1 --2- 172.21.15.69:0/850408915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08d8100da0 0x7f08d8101180 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f08cc009910 tx=0x7f08cc009c20 comp rx=0 tx=0).stop 2024-04-21T23:00:02.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.139+0000 7f08dec12700 1 -- 172.21.15.69:0/850408915 shutdown_connections 2024-04-21T23:00:02.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.139+0000 7f08dec12700 1 --2- 172.21.15.69:0/850408915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08d8100da0 0x7f08d8101180 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:02.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.139+0000 7f08dec12700 1 -- 172.21.15.69:0/850408915 >> 172.21.15.69:0/850408915 conn(0x7f08d80fc650 msgr2=0x7f08d80fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:00:02.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.139+0000 7f08dec12700 1 -- 172.21.15.69:0/850408915 shutdown_connections 2024-04-21T23:00:02.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.139+0000 7f08dec12700 1 -- 172.21.15.69:0/850408915 wait complete. 2024-04-21T23:00:02.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.140+0000 7f08dec12700 1 Processor -- start 2024-04-21T23:00:02.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.140+0000 7f08dec12700 1 -- start start 2024-04-21T23:00:02.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.140+0000 7f08dec12700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08d8100da0 0x7f08d8102840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:02.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.140+0000 7f08dec12700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f08d810b170 con 0x7f08d8100da0 2024-04-21T23:00:02.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.141+0000 7f08dc9ae700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08d8100da0 0x7f08d8102840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:02.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.141+0000 7f08dc9ae700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08d8100da0 0x7f08d8102840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.69:58476/0 (socket says 172.21.15.69:58476) 2024-04-21T23:00:02.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.141+0000 7f08dc9ae700 1 -- 172.21.15.69:0/4186063373 learned_addr learned my addr 172.21.15.69:0/4186063373 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-21T23:00:02.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.141+0000 7f08dc9ae700 1 -- 172.21.15.69:0/4186063373 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08d8104730 con 0x7f08d8100da0 2024-04-21T23:00:02.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.141+0000 7f08dc9ae700 1 --2- 172.21.15.69:0/4186063373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08d8100da0 0x7f08d8102840 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f08cc005850 tx=0x7f08cc005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:00:02.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.141+0000 7f08d5ffb700 1 -- 172.21.15.69:0/4186063373 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08cc026070 con 0x7f08d8100da0 2024-04-21T23:00:02.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.141+0000 7f08d5ffb700 1 -- 172.21.15.69:0/4186063373 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08cc00bc30 con 0x7f08d8100da0 2024-04-21T23:00:02.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.141+0000 7f08dec12700 1 -- 172.21.15.69:0/4186063373 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08d8102de0 con 0x7f08d8100da0 2024-04-21T23:00:02.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.141+0000 7f08d5ffb700 1 -- 172.21.15.69:0/4186063373 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08cc003a20 con 0x7f08d8100da0 2024-04-21T23:00:02.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.141+0000 7f08dec12700 1 -- 172.21.15.69:0/4186063373 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08d81032a0 con 0x7f08d8100da0 2024-04-21T23:00:02.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.142+0000 7f08dec12700 1 -- 172.21.15.69:0/4186063373 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08d8102280 con 0x7f08d8100da0 2024-04-21T23:00:02.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.143+0000 7f08d5ffb700 1 -- 172.21.15.69:0/4186063373 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f08cc029070 con 0x7f08d8100da0 2024-04-21T23:00:02.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.143+0000 7f08d5ffb700 1 --2- 172.21.15.69:0/4186063373 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f08c80390f0 0x7f08c803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:02.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.143+0000 7f08d5ffb700 1 -- 172.21.15.69:0/4186063373 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f08cc01e070 con 0x7f08d8100da0 2024-04-21T23:00:02.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.143+0000 7f08d7fff700 1 --2- 172.21.15.69:0/4186063373 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f08c80390f0 0x7f08c803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:02.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.143+0000 7f08d7fff700 1 --2- 172.21.15.69:0/4186063373 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f08c80390f0 0x7f08c803b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f08c4006fd0 tx=0x7f08c4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:00:02.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.148+0000 7f08d5ffb700 1 -- 172.21.15.69:0/4186063373 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f08cc013070 con 0x7f08d8100da0 2024-04-21T23:00:02.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:01 smithi044 ceph-mon[27934]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-21T23:00:02.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:01 smithi044 ceph-mon[27934]: Updating smithi069:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:00:02.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:01 smithi044 ceph-mon[27934]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:00:02.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:01 smithi044 ceph-mon[27934]: Updating smithi069:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:00:02.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.354+0000 7f08dec12700 1 -- 172.21.15.69:0/4186063373 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi044:172.21.15.44=a;smithi055:172.21.15.55=b;smithi069:172.21.15.69=c", "target": ["mon-mgr", ""]}) -- 0x7f08d8103d70 con 0x7f08c80390f0 2024-04-21T23:00:02.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.361+0000 7f08d5ffb700 1 -- 172.21.15.69:0/4186063373 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f08d8103d70 con 0x7f08c80390f0 2024-04-21T23:00:02.360 INFO:teuthology.orchestra.run.smithi069.stdout:Scheduled mon update... 2024-04-21T23:00:02.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.363+0000 7f08dec12700 1 -- 172.21.15.69:0/4186063373 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f08c80390f0 msgr2=0x7f08c803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.363+0000 7f08dec12700 1 --2- 172.21.15.69:0/4186063373 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f08c80390f0 0x7f08c803b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f08c4006fd0 tx=0x7f08c4006e40 comp rx=0 tx=0).stop 2024-04-21T23:00:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.363+0000 7f08dec12700 1 -- 172.21.15.69:0/4186063373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08d8100da0 msgr2=0x7f08d8102840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.363+0000 7f08dec12700 1 --2- 172.21.15.69:0/4186063373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08d8100da0 0x7f08d8102840 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f08cc005850 tx=0x7f08cc005030 comp rx=0 tx=0).stop 2024-04-21T23:00:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.363+0000 7f08dec12700 1 -- 172.21.15.69:0/4186063373 shutdown_connections 2024-04-21T23:00:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.363+0000 7f08dec12700 1 --2- 172.21.15.69:0/4186063373 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f08c80390f0 0x7f08c803b5b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.363+0000 7f08dec12700 1 --2- 172.21.15.69:0/4186063373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f08d8100da0 0x7f08d8102840 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.363+0000 7f08dec12700 1 -- 172.21.15.69:0/4186063373 >> 172.21.15.69:0/4186063373 conn(0x7f08d80fc650 msgr2=0x7f08d806a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:00:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.363+0000 7f08dec12700 1 -- 172.21.15.69:0/4186063373 shutdown_connections 2024-04-21T23:00:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:02.363+0000 7f08dec12700 1 -- 172.21.15.69:0/4186063373 wait complete. 2024-04-21T23:00:02.912 DEBUG:teuthology.orchestra.run.smithi055:mon.b> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mon.b.service 2024-04-21T23:00:02.915 DEBUG:teuthology.orchestra.run.smithi069:mon.c> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mon.c.service 2024-04-21T23:00:02.917 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T23:00:02.917 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T23:00:03.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:02 smithi044 ceph-mon[27934]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:03.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:03.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:03.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:03.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:03.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:03.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:00:03.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:04.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:03 smithi044 ceph-mon[27934]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi044:172.21.15.44=a;smithi055:172.21.15.55=b;smithi069:172.21.15.69=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:00:04.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:03 smithi044 ceph-mon[27934]: Saving service mon spec with placement smithi044:172.21.15.44=a;smithi055:172.21.15.55=b;smithi069:172.21.15.69=c;count:3 2024-04-21T23:00:04.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:03 smithi044 ceph-mon[27934]: Deploying daemon mon.c on smithi069 2024-04-21T23:00:04.683 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:00:05.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:04 smithi044 ceph-mon[27934]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:07.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.115+0000 7ffbce44e700 1 -- 172.21.15.69:0/4093796373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbc8100da0 msgr2=0x7ffbc8101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:07.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.115+0000 7ffbce44e700 1 --2- 172.21.15.69:0/4093796373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbc8100da0 0x7ffbc8101180 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009910 tx=0x7ffbb8009c20 comp rx=0 tx=0).stop 2024-04-21T23:00:07.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.115+0000 7ffbce44e700 1 -- 172.21.15.69:0/4093796373 shutdown_connections 2024-04-21T23:00:07.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.115+0000 7ffbce44e700 1 --2- 172.21.15.69:0/4093796373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbc8100da0 0x7ffbc8101180 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:07.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.115+0000 7ffbce44e700 1 -- 172.21.15.69:0/4093796373 >> 172.21.15.69:0/4093796373 conn(0x7ffbc80fc650 msgr2=0x7ffbc80fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:00:07.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.115+0000 7ffbce44e700 1 -- 172.21.15.69:0/4093796373 shutdown_connections 2024-04-21T23:00:07.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.115+0000 7ffbce44e700 1 -- 172.21.15.69:0/4093796373 wait complete. 2024-04-21T23:00:07.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.116+0000 7ffbce44e700 1 Processor -- start 2024-04-21T23:00:07.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.116+0000 7ffbce44e700 1 -- start start 2024-04-21T23:00:07.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.116+0000 7ffbce44e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbc8100da0 0x7ffbc8196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:07.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.116+0000 7ffbce44e700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc810b170 con 0x7ffbc8100da0 2024-04-21T23:00:07.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.117+0000 7ffbc7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbc8100da0 0x7ffbc8196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:07.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.117+0000 7ffbc7fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbc8100da0 0x7ffbc8196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.69:58494/0 (socket says 172.21.15.69:58494) 2024-04-21T23:00:07.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.117+0000 7ffbc7fff700 1 -- 172.21.15.69:0/2860100910 learned_addr learned my addr 172.21.15.69:0/2860100910 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-21T23:00:07.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.117+0000 7ffbc7fff700 1 -- 172.21.15.69:0/2860100910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbc8196bb0 con 0x7ffbc8100da0 2024-04-21T23:00:07.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.117+0000 7ffbc7fff700 1 --2- 172.21.15.69:0/2860100910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbc8100da0 0x7ffbc8196670 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8005850 tx=0x7ffbb8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:00:07.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.117+0000 7ffbc57fa700 1 -- 172.21.15.69:0/2860100910 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffbb8026070 con 0x7ffbc8100da0 2024-04-21T23:00:07.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.117+0000 7ffbce44e700 1 -- 172.21.15.69:0/2860100910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbc8196ea0 con 0x7ffbc8100da0 2024-04-21T23:00:07.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.117+0000 7ffbc57fa700 1 -- 172.21.15.69:0/2860100910 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbb800bbd0 con 0x7ffbc8100da0 2024-04-21T23:00:07.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.117+0000 7ffbc57fa700 1 -- 172.21.15.69:0/2860100910 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffbb8003970 con 0x7ffbc8100da0 2024-04-21T23:00:07.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.117+0000 7ffbce44e700 1 -- 172.21.15.69:0/2860100910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbc819af40 con 0x7ffbc8100da0 2024-04-21T23:00:07.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.118+0000 7ffbce44e700 1 -- 172.21.15.69:0/2860100910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbc8102280 con 0x7ffbc8100da0 2024-04-21T23:00:07.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.118+0000 7ffbc57fa700 1 -- 172.21.15.69:0/2860100910 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7ffbb8029070 con 0x7ffbc8100da0 2024-04-21T23:00:07.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.118+0000 7ffbc57fa700 1 --2- 172.21.15.69:0/2860100910 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ffbb00390f0 0x7ffbb003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:07.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.118+0000 7ffbc57fa700 1 -- 172.21.15.69:0/2860100910 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ffbb801e070 con 0x7ffbc8100da0 2024-04-21T23:00:07.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.119+0000 7ffbc77fe700 1 --2- 172.21.15.69:0/2860100910 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ffbb00390f0 0x7ffbb003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:07.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.119+0000 7ffbc77fe700 1 --2- 172.21.15.69:0/2860100910 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ffbb00390f0 0x7ffbb003b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc006fd0 tx=0x7ffbbc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:00:07.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.124+0000 7ffbc57fa700 1 -- 172.21.15.69:0/2860100910 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ffbb8013070 con 0x7ffbc8100da0 2024-04-21T23:00:07.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:06 smithi044 ceph-mon[27934]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:07.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.266+0000 7ffbc57fa700 1 -- 172.21.15.69:0/2860100910 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7ffbb800e3e0 con 0x7ffbc8100da0 2024-04-21T23:00:07.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:07.409+0000 7ffbce44e700 1 -- 172.21.15.69:0/2860100910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7ffbc81979d0 con 0x7ffbc8100da0 2024-04-21T23:00:12.281 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-21T23:00:12.282 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":2,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","modified":"2024-04-21T23:00:07.263785Z","created":"2024-04-21T22:58:20.078074Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3300","nonce":0},{"type":"v1","addr":"172.21.15.69:6789","nonce":0}]},"addr":"172.21.15.69:6789/0","public_addr":"172.21.15.69:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-21T23:00:12.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:12.281+0000 7ffbc57fa700 1 -- 172.21.15.69:0/2860100910 <== mon.0 v2:172.21.15.44:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1011 (secure 0 0 0) 0x7ffbb8030020 con 0x7ffbc8100da0 2024-04-21T23:00:12.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:12.284+0000 7ffbce44e700 1 -- 172.21.15.69:0/2860100910 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ffbb00390f0 msgr2=0x7ffbb003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:12.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:12.284+0000 7ffbce44e700 1 --2- 172.21.15.69:0/2860100910 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ffbb00390f0 0x7ffbb003b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc006fd0 tx=0x7ffbbc006e40 comp rx=0 tx=0).stop 2024-04-21T23:00:12.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:12.284+0000 7ffbce44e700 1 -- 172.21.15.69:0/2860100910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbc8100da0 msgr2=0x7ffbc8196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:12.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:12.284+0000 7ffbce44e700 1 --2- 172.21.15.69:0/2860100910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbc8100da0 0x7ffbc8196670 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8005850 tx=0x7ffbb8005030 comp rx=0 tx=0).stop 2024-04-21T23:00:12.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:12.284+0000 7ffbce44e700 1 -- 172.21.15.69:0/2860100910 shutdown_connections 2024-04-21T23:00:12.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:12.284+0000 7ffbce44e700 1 --2- 172.21.15.69:0/2860100910 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ffbb00390f0 0x7ffbb003b5b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:12.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:12.284+0000 7ffbce44e700 1 --2- 172.21.15.69:0/2860100910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffbc8100da0 0x7ffbc8196670 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:12.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:12.284+0000 7ffbce44e700 1 -- 172.21.15.69:0/2860100910 >> 172.21.15.69:0/2860100910 conn(0x7ffbc80fc650 msgr2=0x7ffbc8103b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:00:12.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:12.285+0000 7ffbce44e700 1 -- 172.21.15.69:0/2860100910 shutdown_connections 2024-04-21T23:00:12.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:12.285+0000 7ffbce44e700 1 -- 172.21.15.69:0/2860100910 wait complete. 2024-04-21T23:00:12.285 INFO:teuthology.orchestra.run.smithi069.stderr:dumped monmap epoch 2 2024-04-21T23:00:12.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:12 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:12.591 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:12.592 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:00:12.592 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: mon.a calling monitor election 2024-04-21T23:00:12.592 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:00:12.592 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: from='client.? 172.21.15.69:0/2860100910' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T23:00:12.592 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:00:12.592 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:12.592 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:00:12.592 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: mon.c calling monitor election 2024-04-21T23:00:12.592 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:00:12.592 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:12.592 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:00:12.592 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:00:12.592 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T23:00:12.592 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: monmap epoch 2 2024-04-21T23:00:12.593 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: fsid 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T23:00:12.593 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: last_changed 2024-04-21T23:00:07.263785+0000 2024-04-21T23:00:12.593 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: created 2024-04-21T22:58:20.078074+0000 2024-04-21T23:00:12.593 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: min_mon_release 19 (squid) 2024-04-21T23:00:12.593 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: election_strategy: 1 2024-04-21T23:00:12.593 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-04-21T23:00:12.593 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: 1: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.c 2024-04-21T23:00:12.593 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: fsmap 2024-04-21T23:00:12.593 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:00:12.593 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: mgrmap e13: x(active, since 49s) 2024-04-21T23:00:12.593 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: overall HEALTH_OK 2024-04-21T23:00:12.593 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:12.593 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:12.593 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:12.593 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:00:12.593 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:13.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:13 smithi044 ceph-mon[27934]: Deploying daemon mon.b on smithi055 2024-04-21T23:00:13.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:13 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:00:13.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:13 smithi069 ceph-mon[31816]: Deploying daemon mon.b on smithi055 2024-04-21T23:00:13.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:13 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:00:13.835 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T23:00:13.835 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T23:00:14.084 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:00:14.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:14 smithi069 ceph-mon[31816]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:14.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:14 smithi044 ceph-mon[27934]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:15.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.399+0000 7fc1d8bdb700 1 -- 172.21.15.69:0/1186464528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1d4101fa0 msgr2=0x7fc1d4102380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:15.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.399+0000 7fc1d8bdb700 1 --2- 172.21.15.69:0/1186464528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1d4101fa0 0x7fc1d4102380 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4009910 tx=0x7fc1c4009c20 comp rx=0 tx=0).stop 2024-04-21T23:00:15.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.399+0000 7fc1d8bdb700 1 -- 172.21.15.69:0/1186464528 shutdown_connections 2024-04-21T23:00:15.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.399+0000 7fc1d8bdb700 1 --2- 172.21.15.69:0/1186464528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1d4101fa0 0x7fc1d4102380 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:15.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.399+0000 7fc1d8bdb700 1 -- 172.21.15.69:0/1186464528 >> 172.21.15.69:0/1186464528 conn(0x7fc1d40fd830 msgr2=0x7fc1d40ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:00:15.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.399+0000 7fc1d8bdb700 1 -- 172.21.15.69:0/1186464528 shutdown_connections 2024-04-21T23:00:15.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.399+0000 7fc1d8bdb700 1 -- 172.21.15.69:0/1186464528 wait complete. 2024-04-21T23:00:15.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.400+0000 7fc1d8bdb700 1 Processor -- start 2024-04-21T23:00:15.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.400+0000 7fc1d8bdb700 1 -- start start 2024-04-21T23:00:15.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.400+0000 7fc1d8bdb700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1d4101fa0 0x7fc1d4074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:15.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.400+0000 7fc1d8bdb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1d4074a70 0x7fc1d4074ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:15.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.400+0000 7fc1d8bdb700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d410c370 con 0x7fc1d4101fa0 2024-04-21T23:00:15.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.400+0000 7fc1d8bdb700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d410c1f0 con 0x7fc1d4074a70 2024-04-21T23:00:15.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.400+0000 7fc1d1d9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1d4074a70 0x7fc1d4074ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:15.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.400+0000 7fc1d1d9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1d4074a70 0x7fc1d4074ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40924/0 (socket says 172.21.15.69:40924) 2024-04-21T23:00:15.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.400+0000 7fc1d1d9b700 1 -- 172.21.15.69:0/3523504157 learned_addr learned my addr 172.21.15.69:0/3523504157 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-21T23:00:15.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.400+0000 7fc1d1d9b700 1 -- 172.21.15.69:0/3523504157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1d4074a70 msgr2=0x7fc1d4074ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-21T23:00:15.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.401+0000 7fc1d1d9b700 1 -- 172.21.15.69:0/3523504157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1d4074a70 msgr2=0x7fc1d4074ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-21T23:00:15.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.401+0000 7fc1d1d9b700 1 --2- 172.21.15.69:0/3523504157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1d4074a70 0x7fc1d4074ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-21T23:00:15.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.401+0000 7fc1d259c700 1 --2- 172.21.15.69:0/3523504157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1d4101fa0 0x7fc1d4074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:15.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.401+0000 7fc1d1d9b700 1 --2- 172.21.15.69:0/3523504157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1d4074a70 0x7fc1d4074ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:00:15.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.401+0000 7fc1d259c700 1 -- 172.21.15.69:0/3523504157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1d4074a70 msgr2=0x7fc1d4074ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:00:15.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.401+0000 7fc1d259c700 1 --2- 172.21.15.69:0/3523504157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1d4074a70 0x7fc1d4074ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:15.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.401+0000 7fc1d259c700 1 -- 172.21.15.69:0/3523504157 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1d4076450 con 0x7fc1d4101fa0 2024-04-21T23:00:15.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.401+0000 7fc1d259c700 1 --2- 172.21.15.69:0/3523504157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1d4101fa0 0x7fc1d4074530 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4014000 tx=0x7fc1c4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:00:15.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.401+0000 7fc1c37fe700 1 -- 172.21.15.69:0/3523504157 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fc1c4005dc0 con 0x7fc1d4101fa0 2024-04-21T23:00:15.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.401+0000 7fc1c37fe700 1 -- 172.21.15.69:0/3523504157 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1c400be40 con 0x7fc1d4101fa0 2024-04-21T23:00:15.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.401+0000 7fc1d8bdb700 1 -- 172.21.15.69:0/3523504157 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1d41a3cc0 con 0x7fc1d4101fa0 2024-04-21T23:00:15.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.401+0000 7fc1c37fe700 1 -- 172.21.15.69:0/3523504157 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fc1c400f460 con 0x7fc1d4101fa0 2024-04-21T23:00:15.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.401+0000 7fc1d8bdb700 1 -- 172.21.15.69:0/3523504157 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1d41a41b0 con 0x7fc1d4101fa0 2024-04-21T23:00:15.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.402+0000 7fc1d8bdb700 1 -- 172.21.15.69:0/3523504157 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1d4103480 con 0x7fc1d4101fa0 2024-04-21T23:00:15.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.402+0000 7fc1c37fe700 1 -- 172.21.15.69:0/3523504157 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fc1c402a070 con 0x7fc1d4101fa0 2024-04-21T23:00:15.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.402+0000 7fc1c37fe700 1 --2- 172.21.15.69:0/3523504157 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc1bc038e90 0x7fc1bc03b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:15.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.403+0000 7fc1c37fe700 1 -- 172.21.15.69:0/3523504157 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc1c401e030 con 0x7fc1d4101fa0 2024-04-21T23:00:15.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.403+0000 7fc1d1d9b700 1 --2- 172.21.15.69:0/3523504157 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc1bc038e90 0x7fc1bc03b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:15.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.403+0000 7fc1d1d9b700 1 --2- 172.21.15.69:0/3523504157 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc1bc038e90 0x7fc1bc03b350 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc1c8009180 tx=0x7fc1c8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:00:15.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.408+0000 7fc1c37fe700 1 -- 172.21.15.69:0/3523504157 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc1c4013070 con 0x7fc1d4101fa0 2024-04-21T23:00:15.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.682+0000 7fc1d8bdb700 1 -- 172.21.15.69:0/3523504157 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fc1d41a44d0 con 0x7fc1d4101fa0 2024-04-21T23:00:15.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.683+0000 7fc1c37fe700 1 -- 172.21.15.69:0/3523504157 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1011 (secure 0 0 0) 0x7fc1c4031020 con 0x7fc1d4101fa0 2024-04-21T23:00:15.683 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-21T23:00:15.683 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":2,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","modified":"2024-04-21T23:00:07.263785Z","created":"2024-04-21T22:58:20.078074Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3300","nonce":0},{"type":"v1","addr":"172.21.15.69:6789","nonce":0}]},"addr":"172.21.15.69:6789/0","public_addr":"172.21.15.69:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-21T23:00:15.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.685+0000 7fc1d8bdb700 1 -- 172.21.15.69:0/3523504157 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc1bc038e90 msgr2=0x7fc1bc03b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:15.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.685+0000 7fc1d8bdb700 1 --2- 172.21.15.69:0/3523504157 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc1bc038e90 0x7fc1bc03b350 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc1c8009180 tx=0x7fc1c8008040 comp rx=0 tx=0).stop 2024-04-21T23:00:15.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.685+0000 7fc1d8bdb700 1 -- 172.21.15.69:0/3523504157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1d4101fa0 msgr2=0x7fc1d4074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:15.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.685+0000 7fc1d8bdb700 1 --2- 172.21.15.69:0/3523504157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1d4101fa0 0x7fc1d4074530 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4014000 tx=0x7fc1c4005030 comp rx=0 tx=0).stop 2024-04-21T23:00:15.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.685+0000 7fc1d8bdb700 1 -- 172.21.15.69:0/3523504157 shutdown_connections 2024-04-21T23:00:15.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.685+0000 7fc1d8bdb700 1 --2- 172.21.15.69:0/3523504157 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc1bc038e90 0x7fc1bc03b350 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:15.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.685+0000 7fc1d8bdb700 1 --2- 172.21.15.69:0/3523504157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc1d4101fa0 0x7fc1d4074530 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:15.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.685+0000 7fc1d8bdb700 1 --2- 172.21.15.69:0/3523504157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1d4074a70 0x7fc1d4074ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:15.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.685+0000 7fc1d8bdb700 1 -- 172.21.15.69:0/3523504157 >> 172.21.15.69:0/3523504157 conn(0x7fc1d40fd830 msgr2=0x7fc1d4103970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:00:15.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.685+0000 7fc1d8bdb700 1 -- 172.21.15.69:0/3523504157 shutdown_connections 2024-04-21T23:00:15.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:15.685+0000 7fc1d8bdb700 1 -- 172.21.15.69:0/3523504157 wait complete. 2024-04-21T23:00:15.686 INFO:teuthology.orchestra.run.smithi069.stderr:dumped monmap epoch 2 2024-04-21T23:00:16.118 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: mon.b@-1(synchronizing).mds e1 new map 2024-04-21T23:00:16.118 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-21T23:00:16.119 INFO:journalctl@ceph.mon.b.smithi055.stdout: e1 2024-04-21T23:00:16.119 INFO:journalctl@ceph.mon.b.smithi055.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T23:00:16.119 INFO:journalctl@ceph.mon.b.smithi055.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T23:00:16.119 INFO:journalctl@ceph.mon.b.smithi055.stdout: legacy client fscid: -1 2024-04-21T23:00:16.119 INFO:journalctl@ceph.mon.b.smithi055.stdout: 2024-04-21T23:00:16.119 INFO:journalctl@ceph.mon.b.smithi055.stdout: No filesystems configured 2024-04-21T23:00:16.119 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-21T23:00:16.119 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-21T23:00:16.119 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-21T23:00:16.119 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-21T23:00:16.119 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-21T23:00:16.119 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-21T23:00:16.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-21T23:00:16.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T23:00:16.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T23:00:16.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T23:00:16.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:16.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi069", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:00:16.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: Deploying cephadm binary to smithi069 2024-04-21T23:00:16.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi055", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:00:16.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:16.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:16.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: Updating smithi055:/etc/ceph/ceph.conf 2024-04-21T23:00:16.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:16.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: Updating smithi055:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:00:16.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:16.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:00:16.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:16.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: Updating smithi055:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:00:16.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: Added host smithi069 2024-04-21T23:00:16.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:16.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:00:16.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:16.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2151677110' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-21T23:00:16.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:16.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2151677110' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-21T23:00:16.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:00:16.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi069", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:00:16.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:16.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:16.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-21T23:00:16.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: Updating smithi069:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:00:16.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:00:16.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: Updating smithi069:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:00:16.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:16.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:16.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:16.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:16.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:00:16.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:16.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi044:172.21.15.44=a;smithi055:172.21.15.55=b;smithi069:172.21.15.69=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:00:16.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: Saving service mon spec with placement smithi044:172.21.15.44=a;smithi055:172.21.15.55=b;smithi069:172.21.15.69=c;count:3 2024-04-21T23:00:16.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: Deploying daemon mon.c on smithi069 2024-04-21T23:00:16.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:16.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:16.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:16.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:00:16.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: mon.a calling monitor election 2024-04-21T23:00:16.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:00:16.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='client.? 172.21.15.69:0/2860100910' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T23:00:16.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:00:16.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:16.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:00:16.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: mon.c calling monitor election 2024-04-21T23:00:16.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:00:16.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:16.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:00:16.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:00:16.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T23:00:16.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: monmap epoch 2 2024-04-21T23:00:16.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: fsid 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T23:00:16.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: last_changed 2024-04-21T23:00:07.263785+0000 2024-04-21T23:00:16.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: created 2024-04-21T22:58:20.078074+0000 2024-04-21T23:00:16.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: min_mon_release 19 (squid) 2024-04-21T23:00:16.471 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: election_strategy: 1 2024-04-21T23:00:16.471 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-04-21T23:00:16.471 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: 1: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.c 2024-04-21T23:00:16.471 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: fsmap 2024-04-21T23:00:16.471 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:00:16.471 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: mgrmap e13: x(active, since 49s) 2024-04-21T23:00:16.471 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: overall HEALTH_OK 2024-04-21T23:00:16.471 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.471 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.471 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:16.471 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:00:16.471 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:16.471 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: Deploying daemon mon.b on smithi055 2024-04-21T23:00:16.471 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:00:16.471 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:16.471 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 ceph-mon[31117]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-21T23:00:16.471 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 bash[31096]: 64497f39ca46e5aa21c3be74ca775bbc2c582391f8631d5330f423de239a6779 2024-04-21T23:00:16.472 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:16 smithi055 systemd[1]: Started Ceph mon.b for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:00:17.245 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T23:00:17.245 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T23:00:17.505 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:00:21.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: mon.c calling monitor election 2024-04-21T23:00:21.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:00:21.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: mon.a calling monitor election 2024-04-21T23:00:21.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:21.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:00:21.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:21.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:21.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:21.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:21.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:21.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:21.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:21.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T23:00:21.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: monmap epoch 3 2024-04-21T23:00:21.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: fsid 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T23:00:21.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: last_changed 2024-04-21T23:00:16.126145+0000 2024-04-21T23:00:21.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: created 2024-04-21T22:58:20.078074+0000 2024-04-21T23:00:21.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: min_mon_release 19 (squid) 2024-04-21T23:00:21.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: election_strategy: 1 2024-04-21T23:00:21.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-04-21T23:00:21.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: 1: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.c 2024-04-21T23:00:21.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: 2: [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon.b 2024-04-21T23:00:21.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: fsmap 2024-04-21T23:00:21.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:00:21.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: mgrmap e13: x(active, since 58s) 2024-04-21T23:00:21.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: overall HEALTH_OK 2024-04-21T23:00:21.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:21.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:21.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:21.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:21 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:21.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: mon.c calling monitor election 2024-04-21T23:00:21.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:00:21.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: mon.a calling monitor election 2024-04-21T23:00:21.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:21.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:00:21.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:21.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:21.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:21.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:21.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:21.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:21.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:21.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T23:00:21.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: monmap epoch 3 2024-04-21T23:00:21.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: fsid 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T23:00:21.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: last_changed 2024-04-21T23:00:16.126145+0000 2024-04-21T23:00:21.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: created 2024-04-21T22:58:20.078074+0000 2024-04-21T23:00:21.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: min_mon_release 19 (squid) 2024-04-21T23:00:21.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: election_strategy: 1 2024-04-21T23:00:21.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-04-21T23:00:21.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: 1: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.c 2024-04-21T23:00:21.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: 2: [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon.b 2024-04-21T23:00:21.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: fsmap 2024-04-21T23:00:21.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:00:21.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: mgrmap e13: x(active, since 58s) 2024-04-21T23:00:21.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: overall HEALTH_OK 2024-04-21T23:00:21.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:21.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:21.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:21.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:21 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:22.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.052+0000 7fd3b56e0700 1 -- 172.21.15.69:0/964688420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3980035c0 msgr2=0x7fd398005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:22.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.053+0000 7fd3b56e0700 1 --2- 172.21.15.69:0/964688420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3980035c0 0x7fd398005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0069c90 tx=0x7fd3a0009e80 comp rx=0 tx=0).stop 2024-04-21T23:00:22.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.053+0000 7fd3b56e0700 1 -- 172.21.15.69:0/964688420 shutdown_connections 2024-04-21T23:00:22.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.053+0000 7fd3b56e0700 1 --2- 172.21.15.69:0/964688420 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3980035c0 0x7fd398005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:22.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.053+0000 7fd3b56e0700 1 -- 172.21.15.69:0/964688420 >> 172.21.15.69:0/964688420 conn(0x7fd3b00fc710 msgr2=0x7fd3b00feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:00:22.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.053+0000 7fd3b56e0700 1 -- 172.21.15.69:0/964688420 shutdown_connections 2024-04-21T23:00:22.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.053+0000 7fd3b56e0700 1 -- 172.21.15.69:0/964688420 wait complete. 2024-04-21T23:00:22.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.053+0000 7fd3b56e0700 1 Processor -- start 2024-04-21T23:00:22.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3b56e0700 1 -- start start 2024-04-21T23:00:22.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3b56e0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3980035c0 0x7fd3b019d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:22.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3b56e0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3b019dbe0 0x7fd3b0197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:22.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3b56e0700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3b0197d60 0x7fd3b0198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:22.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3b56e0700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b0109d70 con 0x7fd3b0197d60 2024-04-21T23:00:22.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3b56e0700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b0109bf0 con 0x7fd3b019dbe0 2024-04-21T23:00:22.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3b56e0700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b0109ef0 con 0x7fd3980035c0 2024-04-21T23:00:22.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3aeffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3980035c0 0x7fd3b019d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:22.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3aeffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3980035c0 0x7fd3b019d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37778/0 (socket says 172.21.15.69:37778) 2024-04-21T23:00:22.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3aeffd700 1 -- 172.21.15.69:0/3556903405 learned_addr learned my addr 172.21.15.69:0/3556903405 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-21T23:00:22.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3af7fe700 1 --2- 172.21.15.69:0/3556903405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3b0197d60 0x7fd3b0198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:22.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3aeffd700 1 -- 172.21.15.69:0/3556903405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3980035c0 msgr2=0x7fd3b019d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-21T23:00:22.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3aeffd700 1 -- 172.21.15.69:0/3556903405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3980035c0 msgr2=0x7fd3b019d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-21T23:00:22.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3aeffd700 1 --2- 172.21.15.69:0/3556903405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3980035c0 0x7fd3b019d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-21T23:00:22.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3aeffd700 1 --2- 172.21.15.69:0/3556903405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3980035c0 0x7fd3b019d6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:00:22.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3ae7fc700 1 --2- 172.21.15.69:0/3556903405 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3b019dbe0 0x7fd3b0197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:22.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3af7fe700 1 -- 172.21.15.69:0/3556903405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3980035c0 msgr2=0x7fd3b019d6a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:00:22.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3af7fe700 1 --2- 172.21.15.69:0/3556903405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3980035c0 0x7fd3b019d6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:22.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3af7fe700 1 -- 172.21.15.69:0/3556903405 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3b019dbe0 msgr2=0x7fd3b0197820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:22.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3af7fe700 1 --2- 172.21.15.69:0/3556903405 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3b019dbe0 0x7fd3b0197820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:22.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.054+0000 7fd3af7fe700 1 -- 172.21.15.69:0/3556903405 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b01a4500 con 0x7fd3b0197d60 2024-04-21T23:00:22.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.055+0000 7fd3af7fe700 1 --2- 172.21.15.69:0/3556903405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3b0197d60 0x7fd3b0198210 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8006fd0 tx=0x7fd3a800b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:00:22.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.055+0000 7fd397fff700 1 -- 172.21.15.69:0/3556903405 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8015020 con 0x7fd3b0197d60 2024-04-21T23:00:22.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.055+0000 7fd3b56e0700 1 -- 172.21.15.69:0/3556903405 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3b01a4790 con 0x7fd3b0197d60 2024-04-21T23:00:22.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.055+0000 7fd3b56e0700 1 -- 172.21.15.69:0/3556903405 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3b01a4ca0 con 0x7fd3b0197d60 2024-04-21T23:00:22.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.055+0000 7fd397fff700 1 -- 172.21.15.69:0/3556903405 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3a8011d10 con 0x7fd3b0197d60 2024-04-21T23:00:22.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.055+0000 7fd397fff700 1 -- 172.21.15.69:0/3556903405 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a801ae00 con 0x7fd3b0197d60 2024-04-21T23:00:22.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.056+0000 7fd3b56e0700 1 -- 172.21.15.69:0/3556903405 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3b006a490 con 0x7fd3b0197d60 2024-04-21T23:00:22.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.056+0000 7fd397fff700 1 -- 172.21.15.69:0/3556903405 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fd3a8018020 con 0x7fd3b0197d60 2024-04-21T23:00:22.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.056+0000 7fd397fff700 1 --2- 172.21.15.69:0/3556903405 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd39c039580 0x7fd39c03ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:22.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.056+0000 7fd397fff700 1 -- 172.21.15.69:0/3556903405 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd3a804bb60 con 0x7fd3b0197d60 2024-04-21T23:00:22.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.056+0000 7fd3aeffd700 1 --2- 172.21.15.69:0/3556903405 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd39c039580 0x7fd39c03ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:22.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.057+0000 7fd3aeffd700 1 --2- 172.21.15.69:0/3556903405 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd39c039580 0x7fd39c03ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009160 tx=0x7fd3a00090f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:00:22.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.061+0000 7fd397fff700 1 -- 172.21.15.69:0/3556903405 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd3a801f020 con 0x7fd3b0197d60 2024-04-21T23:00:22.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.347+0000 7fd3b56e0700 1 -- 172.21.15.69:0/3556903405 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fd3b002db30 con 0x7fd3b0197d60 2024-04-21T23:00:22.347 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:22 smithi069 ceph-mon[31816]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:22.347 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:22 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:22.347 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:22 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:22.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.348+0000 7fd397fff700 1 -- 172.21.15.69:0/3556903405 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1275 (secure 0 0 0) 0x7fd3a8017ae0 con 0x7fd3b0197d60 2024-04-21T23:00:22.348 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-21T23:00:22.348 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":3,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","modified":"2024-04-21T23:00:16.126145Z","created":"2024-04-21T22:58:20.078074Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:3300","nonce":0},{"type":"v1","addr":"172.21.15.44:6789","nonce":0}]},"addr":"172.21.15.44:6789/0","public_addr":"172.21.15.44:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3300","nonce":0},{"type":"v1","addr":"172.21.15.69:6789","nonce":0}]},"addr":"172.21.15.69:6789/0","public_addr":"172.21.15.69:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-21T23:00:22.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.350+0000 7fd3b56e0700 1 -- 172.21.15.69:0/3556903405 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd39c039580 msgr2=0x7fd39c03ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:22.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.350+0000 7fd3b56e0700 1 --2- 172.21.15.69:0/3556903405 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd39c039580 0x7fd39c03ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009160 tx=0x7fd3a00090f0 comp rx=0 tx=0).stop 2024-04-21T23:00:22.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.350+0000 7fd3b56e0700 1 -- 172.21.15.69:0/3556903405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3b0197d60 msgr2=0x7fd3b0198210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:22.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.350+0000 7fd3b56e0700 1 --2- 172.21.15.69:0/3556903405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3b0197d60 0x7fd3b0198210 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8006fd0 tx=0x7fd3a800b7e0 comp rx=0 tx=0).stop 2024-04-21T23:00:22.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.350+0000 7fd3b56e0700 1 -- 172.21.15.69:0/3556903405 shutdown_connections 2024-04-21T23:00:22.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.350+0000 7fd3b56e0700 1 --2- 172.21.15.69:0/3556903405 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd39c039580 0x7fd39c03ba40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:22.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.350+0000 7fd3b56e0700 1 --2- 172.21.15.69:0/3556903405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3b0197d60 0x7fd3b0198210 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:22.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.350+0000 7fd3b56e0700 1 --2- 172.21.15.69:0/3556903405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3980035c0 0x7fd3b019d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:22.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.350+0000 7fd3b56e0700 1 --2- 172.21.15.69:0/3556903405 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3b019dbe0 0x7fd3b0197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:22.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.350+0000 7fd3b56e0700 1 -- 172.21.15.69:0/3556903405 >> 172.21.15.69:0/3556903405 conn(0x7fd3b00fc710 msgr2=0x7fd3b0103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:00:22.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.350+0000 7fd3b56e0700 1 -- 172.21.15.69:0/3556903405 shutdown_connections 2024-04-21T23:00:22.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:22.350+0000 7fd3b56e0700 1 -- 172.21.15.69:0/3556903405 wait complete. 2024-04-21T23:00:22.353 INFO:teuthology.orchestra.run.smithi069.stderr:dumped monmap epoch 3 2024-04-21T23:00:22.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:22 smithi044 ceph-mon[27934]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:22.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:22 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:22.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:22 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:23.067 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-21T23:00:23.068 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-21T23:00:23.327 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:00:23.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: mon.c calling monitor election 2024-04-21T23:00:23.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:00:23.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: mon.a calling monitor election 2024-04-21T23:00:23.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:23.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:00:23.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:23.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:23.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:23.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:23.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:23.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:23.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:23.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T23:00:23.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: monmap epoch 3 2024-04-21T23:00:23.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: fsid 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T23:00:23.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: last_changed 2024-04-21T23:00:16.126145+0000 2024-04-21T23:00:23.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: created 2024-04-21T22:58:20.078074+0000 2024-04-21T23:00:23.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: min_mon_release 19 (squid) 2024-04-21T23:00:23.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: election_strategy: 1 2024-04-21T23:00:23.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-04-21T23:00:23.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: 1: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.c 2024-04-21T23:00:23.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: 2: [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon.b 2024-04-21T23:00:23.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: fsmap 2024-04-21T23:00:23.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:00:23.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: mgrmap e13: x(active, since 58s) 2024-04-21T23:00:23.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: overall HEALTH_OK 2024-04-21T23:00:23.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:23.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:23.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:23.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:23.471 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:23.471 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:23.471 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:24.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: mon.b calling monitor election 2024-04-21T23:00:24.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:24.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: mon.b calling monitor election 2024-04-21T23:00:24.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: mon.a calling monitor election 2024-04-21T23:00:24.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: mon.c calling monitor election 2024-04-21T23:00:24.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T23:00:24.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: monmap epoch 3 2024-04-21T23:00:24.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: fsid 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T23:00:24.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: last_changed 2024-04-21T23:00:16.126145+0000 2024-04-21T23:00:24.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: created 2024-04-21T22:58:20.078074+0000 2024-04-21T23:00:24.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: min_mon_release 19 (squid) 2024-04-21T23:00:24.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: election_strategy: 1 2024-04-21T23:00:24.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-04-21T23:00:24.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: 1: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.c 2024-04-21T23:00:24.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: 2: [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon.b 2024-04-21T23:00:24.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: fsmap 2024-04-21T23:00:24.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:00:24.188 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: mgrmap e13: x(active, since 60s) 2024-04-21T23:00:24.188 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: overall HEALTH_OK 2024-04-21T23:00:24.188 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.188 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.188 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.188 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.188 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.188 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.188 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.188 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.188 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.188 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.188 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.188 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T23:00:24.189 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:00:24.189 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:00:24.189 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:24.189 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:24 smithi044 ceph-mon[27934]: Reconfiguring daemon mon.a on smithi044 2024-04-21T23:00:24.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: mon.b calling monitor election 2024-04-21T23:00:24.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:24.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: mon.b calling monitor election 2024-04-21T23:00:24.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: mon.a calling monitor election 2024-04-21T23:00:24.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: mon.c calling monitor election 2024-04-21T23:00:24.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T23:00:24.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: monmap epoch 3 2024-04-21T23:00:24.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: fsid 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T23:00:24.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: last_changed 2024-04-21T23:00:16.126145+0000 2024-04-21T23:00:24.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: created 2024-04-21T22:58:20.078074+0000 2024-04-21T23:00:24.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: min_mon_release 19 (squid) 2024-04-21T23:00:24.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: election_strategy: 1 2024-04-21T23:00:24.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-04-21T23:00:24.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: 1: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.c 2024-04-21T23:00:24.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: 2: [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon.b 2024-04-21T23:00:24.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: fsmap 2024-04-21T23:00:24.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:00:24.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: mgrmap e13: x(active, since 60s) 2024-04-21T23:00:24.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: overall HEALTH_OK 2024-04-21T23:00:24.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T23:00:24.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:00:24.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:00:24.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:24.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:24 smithi069 ceph-mon[31816]: Reconfiguring daemon mon.a on smithi044 2024-04-21T23:00:24.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: mon.b calling monitor election 2024-04-21T23:00:24.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:24.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: mon.b calling monitor election 2024-04-21T23:00:24.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: mon.a calling monitor election 2024-04-21T23:00:24.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: mon.c calling monitor election 2024-04-21T23:00:24.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T23:00:24.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: monmap epoch 3 2024-04-21T23:00:24.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: fsid 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T23:00:24.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: last_changed 2024-04-21T23:00:16.126145+0000 2024-04-21T23:00:24.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: created 2024-04-21T22:58:20.078074+0000 2024-04-21T23:00:24.468 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: min_mon_release 19 (squid) 2024-04-21T23:00:24.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: election_strategy: 1 2024-04-21T23:00:24.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: 0: [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] mon.a 2024-04-21T23:00:24.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: 1: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.c 2024-04-21T23:00:24.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: 2: [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon.b 2024-04-21T23:00:24.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: fsmap 2024-04-21T23:00:24.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T23:00:24.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: mgrmap e13: x(active, since 60s) 2024-04-21T23:00:24.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: overall HEALTH_OK 2024-04-21T23:00:24.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.469 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:24.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T23:00:24.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:00:24.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:00:24.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:24.470 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:24 smithi055 ceph-mon[31117]: Reconfiguring daemon mon.a on smithi044 2024-04-21T23:00:25.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:25 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:25.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:25 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:25.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:25 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:00:25.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.684+0000 7fa95721b700 1 -- 172.21.15.44:0/1889932202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa950100de0 msgr2=0x7fa9501011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:25.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.684+0000 7fa95721b700 1 --2- 172.21.15.44:0/1889932202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa950100de0 0x7fa9501011c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa944009910 tx=0x7fa944009c20 comp rx=0 tx=0).stop 2024-04-21T23:00:25.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.684+0000 7fa95721b700 1 -- 172.21.15.44:0/1889932202 shutdown_connections 2024-04-21T23:00:25.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.684+0000 7fa95721b700 1 --2- 172.21.15.44:0/1889932202 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa950100de0 0x7fa9501011c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:25.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.685+0000 7fa95721b700 1 -- 172.21.15.44:0/1889932202 >> 172.21.15.44:0/1889932202 conn(0x7fa9500fc690 msgr2=0x7fa9500feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:00:25.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.685+0000 7fa95721b700 1 -- 172.21.15.44:0/1889932202 shutdown_connections 2024-04-21T23:00:25.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.685+0000 7fa95721b700 1 -- 172.21.15.44:0/1889932202 wait complete. 2024-04-21T23:00:25.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.685+0000 7fa95721b700 1 Processor -- start 2024-04-21T23:00:25.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.685+0000 7fa95721b700 1 -- start start 2024-04-21T23:00:25.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.685+0000 7fa95721b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa950100de0 0x7fa950102880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:25.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.685+0000 7fa95721b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa950102dc0 0x7fa950103240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:25.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.685+0000 7fa95721b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9501a3c80 0x7fa9501a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:25.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.685+0000 7fa95721b700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa95010b170 con 0x7fa950102dc0 2024-04-21T23:00:25.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.685+0000 7fa95721b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa95010aff0 con 0x7fa950100de0 2024-04-21T23:00:25.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.685+0000 7fa95721b700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa95010b2f0 con 0x7fa9501a3c80 2024-04-21T23:00:25.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.686+0000 7fa94ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa950102dc0 0x7fa950103240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:25.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.686+0000 7fa94ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa950102dc0 0x7fa950103240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36160/0 (socket says 172.21.15.44:36160) 2024-04-21T23:00:25.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.686+0000 7fa94ffff700 1 -- 172.21.15.44:0/4096900547 learned_addr learned my addr 172.21.15.44:0/4096900547 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:00:25.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.686+0000 7fa954fb7700 1 --2- 172.21.15.44:0/4096900547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa950100de0 0x7fa950102880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:25.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.686+0000 7fa9557b8700 1 --2- 172.21.15.44:0/4096900547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9501a3c80 0x7fa9501a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:25.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.686+0000 7fa94ffff700 1 -- 172.21.15.44:0/4096900547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9501a3c80 msgr2=0x7fa9501a6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:25.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.686+0000 7fa94ffff700 1 --2- 172.21.15.44:0/4096900547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9501a3c80 0x7fa9501a6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:25.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.686+0000 7fa94ffff700 1 -- 172.21.15.44:0/4096900547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa950100de0 msgr2=0x7fa950102880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:25.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.686+0000 7fa94ffff700 1 --2- 172.21.15.44:0/4096900547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa950100de0 0x7fa950102880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:25.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.686+0000 7fa94ffff700 1 -- 172.21.15.44:0/4096900547 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9501a66e0 con 0x7fa950102dc0 2024-04-21T23:00:25.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.686+0000 7fa9557b8700 1 --2- 172.21.15.44:0/4096900547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9501a3c80 0x7fa9501a6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:00:25.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.686+0000 7fa94ffff700 1 --2- 172.21.15.44:0/4096900547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa950102dc0 0x7fa950103240 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa93c00b780 tx=0x7fa93c00ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:00:25.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.686+0000 7fa94dffb700 1 -- 172.21.15.44:0/4096900547 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c004230 con 0x7fa950102dc0 2024-04-21T23:00:25.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.686+0000 7fa95721b700 1 -- 172.21.15.44:0/4096900547 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9501a69d0 con 0x7fa950102dc0 2024-04-21T23:00:25.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.686+0000 7fa94dffb700 1 -- 172.21.15.44:0/4096900547 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa93c0043d0 con 0x7fa950102dc0 2024-04-21T23:00:25.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.686+0000 7fa94dffb700 1 -- 172.21.15.44:0/4096900547 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c011c90 con 0x7fa950102dc0 2024-04-21T23:00:25.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.687+0000 7fa95721b700 1 -- 172.21.15.44:0/4096900547 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9501a6f10 con 0x7fa950102dc0 2024-04-21T23:00:25.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.687+0000 7fa94dffb700 1 -- 172.21.15.44:0/4096900547 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fa93c01a400 con 0x7fa950102dc0 2024-04-21T23:00:25.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.687+0000 7fa94dffb700 1 --2- 172.21.15.44:0/4096900547 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa940039580 0x7fa94003ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:25.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.687+0000 7fa95721b700 1 -- 172.21.15.44:0/4096900547 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa950063890 con 0x7fa950102dc0 2024-04-21T23:00:25.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.688+0000 7fa94dffb700 1 -- 172.21.15.44:0/4096900547 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fa93c04c070 con 0x7fa950102dc0 2024-04-21T23:00:25.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.688+0000 7fa954fb7700 1 --2- 172.21.15.44:0/4096900547 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa940039580 0x7fa94003ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:25.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.688+0000 7fa954fb7700 1 --2- 172.21.15.44:0/4096900547 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa940039580 0x7fa94003ba40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa944009750 tx=0x7fa94400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:00:25.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.693+0000 7fa94dffb700 1 -- 172.21.15.44:0/4096900547 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa93c018750 con 0x7fa950102dc0 2024-04-21T23:00:25.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.897+0000 7fa95721b700 1 -- 172.21.15.44:0/4096900547 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fa9501a7220 con 0x7fa950102dc0 2024-04-21T23:00:25.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.898+0000 7fa94dffb700 1 -- 172.21.15.44:0/4096900547 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+271 (secure 0 0 0) 0x7fa93c003ca0 con 0x7fa950102dc0 2024-04-21T23:00:25.898 INFO:teuthology.orchestra.run.smithi044.stdout:# minimal ceph.conf for 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T23:00:25.898 INFO:teuthology.orchestra.run.smithi044.stdout:[global] 2024-04-21T23:00:25.898 INFO:teuthology.orchestra.run.smithi044.stdout: fsid = 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T23:00:25.898 INFO:teuthology.orchestra.run.smithi044.stdout: mon_host = [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-04-21T23:00:25.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.900+0000 7fa95721b700 1 -- 172.21.15.44:0/4096900547 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa940039580 msgr2=0x7fa94003ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:25.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.900+0000 7fa95721b700 1 --2- 172.21.15.44:0/4096900547 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa940039580 0x7fa94003ba40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa944009750 tx=0x7fa94400b540 comp rx=0 tx=0).stop 2024-04-21T23:00:25.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.900+0000 7fa95721b700 1 -- 172.21.15.44:0/4096900547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa950102dc0 msgr2=0x7fa950103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:25.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.900+0000 7fa95721b700 1 --2- 172.21.15.44:0/4096900547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa950102dc0 0x7fa950103240 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa93c00b780 tx=0x7fa93c00ba90 comp rx=0 tx=0).stop 2024-04-21T23:00:25.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.900+0000 7fa95721b700 1 -- 172.21.15.44:0/4096900547 shutdown_connections 2024-04-21T23:00:25.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.900+0000 7fa95721b700 1 --2- 172.21.15.44:0/4096900547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9501a3c80 0x7fa9501a6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:25.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.900+0000 7fa95721b700 1 --2- 172.21.15.44:0/4096900547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa950100de0 0x7fa950102880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:25.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.900+0000 7fa95721b700 1 --2- 172.21.15.44:0/4096900547 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa940039580 0x7fa94003ba40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:25.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.900+0000 7fa95721b700 1 --2- 172.21.15.44:0/4096900547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa950102dc0 0x7fa950103240 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:25.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.900+0000 7fa95721b700 1 -- 172.21.15.44:0/4096900547 >> 172.21.15.44:0/4096900547 conn(0x7fa9500fc690 msgr2=0x7fa95006a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:00:25.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.900+0000 7fa95721b700 1 -- 172.21.15.44:0/4096900547 shutdown_connections 2024-04-21T23:00:25.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:25.900+0000 7fa95721b700 1 -- 172.21.15.44:0/4096900547 wait complete. 2024-04-21T23:00:26.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:26 smithi055 ceph-mon[31117]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:26.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:26 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:26.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:26 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:26.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:26 smithi055 ceph-mon[31117]: Reconfiguring mon.b (monmap changed)... 2024-04-21T23:00:26.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:26 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:00:26.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:26 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:00:26.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:26 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:26.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:26 smithi055 ceph-mon[31117]: Reconfiguring daemon mon.b on smithi055 2024-04-21T23:00:26.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:26 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/4096900547' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:26.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:26 smithi044 ceph-mon[27934]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:26.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:26 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:26.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:26 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:26.478 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:26 smithi044 ceph-mon[27934]: Reconfiguring mon.b (monmap changed)... 2024-04-21T23:00:26.479 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:26 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:00:26.479 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:26 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:00:26.479 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:26 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:26.479 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:26 smithi044 ceph-mon[27934]: Reconfiguring daemon mon.b on smithi055 2024-04-21T23:00:26.479 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:26 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/4096900547' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:26.480 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-21T23:00:26.480 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T23:00:26.480 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T23:00:26.510 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T23:00:26.510 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:00:26.575 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-21T23:00:26.575 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T23:00:26.602 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-21T23:00:26.603 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:00:26.667 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-21T23:00:26.667 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T23:00:26.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:26 smithi069 ceph-mon[31816]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:26.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:26 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:26.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:26 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:26.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:26 smithi069 ceph-mon[31816]: Reconfiguring mon.b (monmap changed)... 2024-04-21T23:00:26.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:26 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:00:26.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:26 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:00:26.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:26 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:26.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:26 smithi069 ceph-mon[31816]: Reconfiguring daemon mon.b on smithi055 2024-04-21T23:00:26.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:26 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/4096900547' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:26.698 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-21T23:00:26.698 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:00:26.765 INFO:tasks.cephadm:Adding mgr.x on smithi044 2024-04-21T23:00:26.765 INFO:tasks.cephadm:Adding mgr.y on smithi055 2024-04-21T23:00:26.765 INFO:tasks.cephadm:Adding mgr.z on smithi069 2024-04-21T23:00:26.765 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi044=x;smithi055=y;smithi069=z' 2024-04-21T23:00:27.038 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:00:28.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:28 smithi069 ceph-mon[31816]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:28.362 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:28 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:28.363 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:28 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:28.363 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:28 smithi069 ceph-mon[31816]: Reconfiguring mon.c (monmap changed)... 2024-04-21T23:00:28.363 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:28 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:00:28.363 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:28 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:00:28.363 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:28 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:28.363 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:28 smithi069 ceph-mon[31816]: Reconfiguring daemon mon.c on smithi069 2024-04-21T23:00:28.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:28 smithi044 ceph-mon[27934]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:28.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:28 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:28.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:28 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:28.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:28 smithi044 ceph-mon[27934]: Reconfiguring mon.c (monmap changed)... 2024-04-21T23:00:28.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:28 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:00:28.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:28 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:00:28.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:28 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:28.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:28 smithi044 ceph-mon[27934]: Reconfiguring daemon mon.c on smithi069 2024-04-21T23:00:28.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:28 smithi055 ceph-mon[31117]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:28.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:28 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:28.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:28 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:28.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:28 smithi055 ceph-mon[31117]: Reconfiguring mon.c (monmap changed)... 2024-04-21T23:00:28.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:28 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T23:00:28.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:28 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T23:00:28.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:28 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:28.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:28 smithi055 ceph-mon[31117]: Reconfiguring daemon mon.c on smithi069 2024-04-21T23:00:29.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.471+0000 7f5e0423a700 1 -- 172.21.15.69:0/1114369573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dfc100610 msgr2=0x7f5dfc100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:29.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.471+0000 7f5e0423a700 1 --2- 172.21.15.69:0/1114369573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dfc100610 0x7f5dfc100a30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5df8009990 tx=0x7f5df8009ca0 comp rx=0 tx=0).stop 2024-04-21T23:00:29.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.471+0000 7f5e0423a700 1 -- 172.21.15.69:0/1114369573 shutdown_connections 2024-04-21T23:00:29.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.471+0000 7f5e0423a700 1 --2- 172.21.15.69:0/1114369573 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5dfc104030 0x7f5dfc10a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:29.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.471+0000 7f5e0423a700 1 --2- 172.21.15.69:0/1114369573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dfc100610 0x7f5dfc100a30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:29.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.471+0000 7f5e0423a700 1 --2- 172.21.15.69:0/1114369573 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5dfc103690 0x7f5dfc103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:29.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.471+0000 7f5e0423a700 1 -- 172.21.15.69:0/1114369573 >> 172.21.15.69:0/1114369573 conn(0x7f5dfc0fbf70 msgr2=0x7f5dfc0fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:00:29.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.471+0000 7f5e0423a700 1 -- 172.21.15.69:0/1114369573 shutdown_connections 2024-04-21T23:00:29.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.471+0000 7f5e0423a700 1 -- 172.21.15.69:0/1114369573 wait complete. 2024-04-21T23:00:29.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.471+0000 7f5e0423a700 1 Processor -- start 2024-04-21T23:00:29.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.471+0000 7f5e0423a700 1 -- start start 2024-04-21T23:00:29.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.472+0000 7f5e0423a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5dfc100610 0x7f5dfc198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:29.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.472+0000 7f5e0423a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5dfc103690 0x7f5dfc198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:29.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.472+0000 7f5e0423a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dfc104030 0x7f5dfc19fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:29.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.472+0000 7f5e0423a700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc10df80 con 0x7f5dfc103690 2024-04-21T23:00:29.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.472+0000 7f5e0423a700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc10de00 con 0x7f5dfc100610 2024-04-21T23:00:29.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.472+0000 7f5e0423a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc10e100 con 0x7f5dfc104030 2024-04-21T23:00:29.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.472+0000 7f5e027d7700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dfc104030 0x7f5dfc19fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:29.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.472+0000 7f5e017d5700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5dfc103690 0x7f5dfc198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:29.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.472+0000 7f5e027d7700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dfc104030 0x7f5dfc19fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57664/0 (socket says 172.21.15.69:57664) 2024-04-21T23:00:29.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.472+0000 7f5e027d7700 1 -- 172.21.15.69:0/3163200450 learned_addr learned my addr 172.21.15.69:0/3163200450 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-21T23:00:29.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.472+0000 7f5e017d5700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5dfc103690 0x7f5dfc198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.69:53620/0 (socket says 172.21.15.69:53620) 2024-04-21T23:00:29.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.472+0000 7f5e01fd6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5dfc100610 0x7f5dfc198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:29.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.472+0000 7f5e017d5700 1 -- 172.21.15.69:0/3163200450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dfc104030 msgr2=0x7f5dfc19fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:29.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.472+0000 7f5e017d5700 1 --2- 172.21.15.69:0/3163200450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dfc104030 0x7f5dfc19fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:29.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.472+0000 7f5e017d5700 1 -- 172.21.15.69:0/3163200450 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5dfc100610 msgr2=0x7f5dfc198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:29.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.472+0000 7f5e017d5700 1 --2- 172.21.15.69:0/3163200450 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5dfc100610 0x7f5dfc198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:29.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.472+0000 7f5e017d5700 1 -- 172.21.15.69:0/3163200450 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dfc1a01c0 con 0x7f5dfc103690 2024-04-21T23:00:29.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.472+0000 7f5e027d7700 1 --2- 172.21.15.69:0/3163200450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dfc104030 0x7f5dfc19fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:00:29.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.472+0000 7f5e017d5700 1 --2- 172.21.15.69:0/3163200450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5dfc103690 0x7f5dfc198aa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00ea20 tx=0x7f5dec00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:00:29.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.472+0000 7f5df2ffd700 1 -- 172.21.15.69:0/3163200450 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec00cbd0 con 0x7f5dfc103690 2024-04-21T23:00:29.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.473+0000 7f5df2ffd700 1 -- 172.21.15.69:0/3163200450 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dec00ee60 con 0x7f5dfc103690 2024-04-21T23:00:29.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.473+0000 7f5e0423a700 1 -- 172.21.15.69:0/3163200450 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dfc1014e0 con 0x7f5dfc103690 2024-04-21T23:00:29.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.473+0000 7f5df2ffd700 1 -- 172.21.15.69:0/3163200450 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec018680 con 0x7f5dfc103690 2024-04-21T23:00:29.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.473+0000 7f5e0423a700 1 -- 172.21.15.69:0/3163200450 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dfc101a20 con 0x7f5dfc103690 2024-04-21T23:00:29.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.473+0000 7f5df2ffd700 1 -- 172.21.15.69:0/3163200450 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f5dec018820 con 0x7f5dfc103690 2024-04-21T23:00:29.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.473+0000 7f5df2ffd700 1 --2- 172.21.15.69:0/3163200450 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5de8039330 0x7f5de803b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:29.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.474+0000 7f5df2ffd700 1 -- 172.21.15.69:0/3163200450 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5dec04c430 con 0x7f5dfc103690 2024-04-21T23:00:29.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.474+0000 7f5e0423a700 1 -- 172.21.15.69:0/3163200450 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dfc068f20 con 0x7f5dfc103690 2024-04-21T23:00:29.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.474+0000 7f5e01fd6700 1 --2- 172.21.15.69:0/3163200450 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5de8039330 0x7f5de803b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:29.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.474+0000 7f5e01fd6700 1 --2- 172.21.15.69:0/3163200450 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5de8039330 0x7f5de803b7f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5df8000c00 tx=0x7f5df8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:00:29.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.479+0000 7f5df2ffd700 1 -- 172.21.15.69:0/3163200450 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5dec00fc70 con 0x7f5dfc103690 2024-04-21T23:00:29.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.685+0000 7f5e0423a700 1 -- 172.21.15.69:0/3163200450 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi044=x;smithi055=y;smithi069=z", "target": ["mon-mgr", ""]}) -- 0x7f5dfc02dfa0 con 0x7f5de8039330 2024-04-21T23:00:29.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.691+0000 7f5df2ffd700 1 -- 172.21.15.69:0/3163200450 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f5dfc02dfa0 con 0x7f5de8039330 2024-04-21T23:00:29.691 INFO:teuthology.orchestra.run.smithi069.stdout:Scheduled mgr update... 2024-04-21T23:00:29.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.693+0000 7f5e0423a700 1 -- 172.21.15.69:0/3163200450 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5de8039330 msgr2=0x7f5de803b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:29.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.693+0000 7f5e0423a700 1 --2- 172.21.15.69:0/3163200450 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5de8039330 0x7f5de803b7f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5df8000c00 tx=0x7f5df8005fb0 comp rx=0 tx=0).stop 2024-04-21T23:00:29.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.693+0000 7f5e0423a700 1 -- 172.21.15.69:0/3163200450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5dfc103690 msgr2=0x7f5dfc198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:29.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.693+0000 7f5e0423a700 1 --2- 172.21.15.69:0/3163200450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5dfc103690 0x7f5dfc198aa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00ea20 tx=0x7f5dec00ed30 comp rx=0 tx=0).stop 2024-04-21T23:00:29.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.693+0000 7f5e0423a700 1 -- 172.21.15.69:0/3163200450 shutdown_connections 2024-04-21T23:00:29.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.693+0000 7f5e0423a700 1 --2- 172.21.15.69:0/3163200450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5dfc104030 0x7f5dfc19fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:29.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.693+0000 7f5e0423a700 1 --2- 172.21.15.69:0/3163200450 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5dfc100610 0x7f5dfc198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:29.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.693+0000 7f5e0423a700 1 --2- 172.21.15.69:0/3163200450 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5de8039330 0x7f5de803b7f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:29.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.693+0000 7f5e0423a700 1 --2- 172.21.15.69:0/3163200450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5dfc103690 0x7f5dfc198aa0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:29.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.693+0000 7f5e0423a700 1 -- 172.21.15.69:0/3163200450 >> 172.21.15.69:0/3163200450 conn(0x7f5dfc0fbf70 msgr2=0x7f5dfc106930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:00:29.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.693+0000 7f5e0423a700 1 -- 172.21.15.69:0/3163200450 shutdown_connections 2024-04-21T23:00:29.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:00:29.693+0000 7f5e0423a700 1 -- 172.21.15.69:0/3163200450 wait complete. 2024-04-21T23:00:30.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:29 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:30.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:29 smithi069 ceph-mon[31816]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:30.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:29 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:30.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:29 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:30.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:29 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:30.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:29 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:30.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:29 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:30.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:29 smithi069 ceph-mon[31816]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi044=x;smithi055=y;smithi069=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:00:30.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:29 smithi069 ceph-mon[31816]: Saving service mgr spec with placement smithi044=x;smithi055=y;smithi069=z;count:3 2024-04-21T23:00:30.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:29 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:30.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:29 smithi069 ceph-mon[31816]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T23:00:30.255 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:29 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:00:30.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:29 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:00:30.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:29 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:30.256 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:29 smithi069 ceph-mon[31816]: Reconfiguring daemon mgr.x on smithi044 2024-04-21T23:00:30.281 DEBUG:teuthology.orchestra.run.smithi055:mgr.y> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mgr.y.service 2024-04-21T23:00:30.284 DEBUG:teuthology.orchestra.run.smithi069:mgr.z> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mgr.z.service 2024-04-21T23:00:30.286 INFO:tasks.cephadm:Deploying OSDs... 2024-04-21T23:00:30.286 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T23:00:30.287 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T23:00:30.304 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T23:00:30.304 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_1 2024-04-21T23:00:30.307 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:29 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:30.308 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:29 smithi055 ceph-mon[31117]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:30.308 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:29 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:30.308 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:29 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:30.309 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:29 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:30.309 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:29 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:30.309 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:29 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:30.309 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:29 smithi055 ceph-mon[31117]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi044=x;smithi055=y;smithi069=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:00:30.309 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:29 smithi055 ceph-mon[31117]: Saving service mgr spec with placement smithi044=x;smithi055=y;smithi069=z;count:3 2024-04-21T23:00:30.309 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:29 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:30.310 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:29 smithi055 ceph-mon[31117]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T23:00:30.310 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:29 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:00:30.310 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:29 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:00:30.310 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:29 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:30.310 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:29 smithi055 ceph-mon[31117]: Reconfiguring daemon mgr.x on smithi044 2024-04-21T23:00:30.359 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T23:00:30.359 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:00:30.359 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-21T23:00:30.359 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:00:30.359 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:00:30.360 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-04-21 22:59:33.607149208 +0000 2024-04-21T23:00:30.360 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-04-21 22:54:31.837450176 +0000 2024-04-21T23:00:30.360 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-04-21 22:54:31.837450176 +0000 2024-04-21T23:00:30.360 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-04-21 22:54:31.837450176 +0000 2024-04-21T23:00:30.360 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T23:00:30.375 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:29 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:30.375 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:29 smithi044 ceph-mon[27934]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:30.375 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:29 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:30.375 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:29 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:30.376 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:29 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:30.376 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:29 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:30.376 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:29 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:30.376 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:29 smithi044 ceph-mon[27934]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi044=x;smithi055=y;smithi069=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:00:30.376 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:29 smithi044 ceph-mon[27934]: Saving service mgr spec with placement smithi044=x;smithi055=y;smithi069=z;count:3 2024-04-21T23:00:30.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:29 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:30.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:29 smithi044 ceph-mon[27934]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T23:00:30.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:29 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:00:30.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:29 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:00:30.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:29 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:30.377 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:29 smithi044 ceph-mon[27934]: Reconfiguring daemon mgr.x on smithi044 2024-04-21T23:00:30.401 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-04-21T23:00:30.401 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-04-21T23:00:30.401 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000206974 s, 2.5 MB/s 2024-04-21T23:00:30.402 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T23:00:30.460 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_2 2024-04-21T23:00:30.517 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T23:00:30.517 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:00:30.517 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-21T23:00:30.517 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:00:30.517 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:00:30.517 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-04-21 22:59:33.644148591 +0000 2024-04-21T23:00:30.518 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-04-21 22:54:32.209443968 +0000 2024-04-21T23:00:30.518 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-04-21 22:54:32.209443968 +0000 2024-04-21T23:00:30.518 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-04-21 22:54:32.209443968 +0000 2024-04-21T23:00:30.518 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T23:00:30.580 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-04-21T23:00:30.581 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-04-21T23:00:30.581 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000186401 s, 2.7 MB/s 2024-04-21T23:00:30.582 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T23:00:30.638 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_3 2024-04-21T23:00:30.695 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T23:00:30.695 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:00:30.695 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-21T23:00:30.695 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:00:30.696 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:00:30.696 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-04-21 22:59:33.674148091 +0000 2024-04-21T23:00:30.696 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-04-21 22:54:32.560438110 +0000 2024-04-21T23:00:30.696 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-04-21 22:54:32.560438110 +0000 2024-04-21T23:00:30.696 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-04-21 22:54:32.560438110 +0000 2024-04-21T23:00:30.696 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T23:00:30.759 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-04-21T23:00:30.759 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-04-21T23:00:30.759 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000292134 s, 1.8 MB/s 2024-04-21T23:00:30.760 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T23:00:30.817 DEBUG:teuthology.orchestra.run.smithi044:> stat /dev/vg_nvme/lv_4 2024-04-21T23:00:30.872 INFO:teuthology.orchestra.run.smithi044.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T23:00:30.872 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:00:30.872 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-21T23:00:30.872 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:00:30.872 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:00:30.872 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-04-21 22:59:33.703147608 +0000 2024-04-21T23:00:30.873 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-04-21 22:54:32.906432336 +0000 2024-04-21T23:00:30.873 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-04-21 22:54:32.906432336 +0000 2024-04-21T23:00:30.873 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-04-21 22:54:32.906432336 +0000 2024-04-21T23:00:30.873 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T23:00:30.935 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records in 2024-04-21T23:00:30.935 INFO:teuthology.orchestra.run.smithi044.stderr:1+0 records out 2024-04-21T23:00:30.935 INFO:teuthology.orchestra.run.smithi044.stderr:512 bytes copied, 0.000212867 s, 2.4 MB/s 2024-04-21T23:00:30.936 DEBUG:teuthology.orchestra.run.smithi044:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T23:00:30.994 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2024-04-21T23:00:30.995 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T23:00:31.014 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T23:00:31.014 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_1 2024-04-21T23:00:31.074 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T23:00:31.074 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:00:31.074 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-21T23:00:31.075 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:00:31.075 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:00:31.075 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-04-21 23:00:15.530664004 +0000 2024-04-21T23:00:31.075 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-04-21 22:55:03.088444108 +0000 2024-04-21T23:00:31.075 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-04-21 22:55:03.088444108 +0000 2024-04-21T23:00:31.075 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-04-21 22:55:03.088444108 +0000 2024-04-21T23:00:31.076 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T23:00:31.142 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-04-21T23:00:31.142 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-04-21T23:00:31.142 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000320631 s, 1.6 MB/s 2024-04-21T23:00:31.144 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T23:00:31.203 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_2 2024-04-21T23:00:31.260 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T23:00:31.260 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:00:31.260 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-21T23:00:31.260 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:00:31.261 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:00:31.261 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-04-21 23:00:15.531663985 +0000 2024-04-21T23:00:31.261 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-04-21 22:55:03.468436789 +0000 2024-04-21T23:00:31.261 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-04-21 22:55:03.468436789 +0000 2024-04-21T23:00:31.261 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-04-21 22:55:03.468436789 +0000 2024-04-21T23:00:31.261 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T23:00:31.327 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-04-21T23:00:31.327 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-04-21T23:00:31.327 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000270883 s, 1.9 MB/s 2024-04-21T23:00:31.328 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T23:00:31.386 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_3 2024-04-21T23:00:31.443 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T23:00:31.443 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:00:31.443 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-21T23:00:31.444 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:00:31.444 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:00:31.444 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-04-21 23:00:15.531663985 +0000 2024-04-21T23:00:31.444 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-04-21 22:55:03.856429316 +0000 2024-04-21T23:00:31.444 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-04-21 22:55:03.856429316 +0000 2024-04-21T23:00:31.444 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-04-21 22:55:03.856429316 +0000 2024-04-21T23:00:31.444 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T23:00:31.509 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-04-21T23:00:31.509 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-04-21T23:00:31.509 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000365075 s, 1.4 MB/s 2024-04-21T23:00:31.511 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T23:00:31.568 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_4 2024-04-21T23:00:31.625 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T23:00:31.625 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:00:31.625 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-21T23:00:31.625 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:00:31.626 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:00:31.626 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2024-04-21 23:00:15.532663965 +0000 2024-04-21T23:00:31.626 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2024-04-21 22:55:04.215422402 +0000 2024-04-21T23:00:31.626 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2024-04-21 22:55:04.215422402 +0000 2024-04-21T23:00:31.626 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2024-04-21 22:55:04.215422402 +0000 2024-04-21T23:00:31.626 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T23:00:31.690 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2024-04-21T23:00:31.690 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2024-04-21T23:00:31.691 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000269195 s, 1.9 MB/s 2024-04-21T23:00:31.692 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T23:00:31.749 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-21T23:00:31.749 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T23:00:31.764 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T23:00:31.764 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_1 2024-04-21T23:00:31.820 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T23:00:31.820 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:00:31.820 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-21T23:00:31.820 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:00:31.820 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:00:31.820 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-21 23:00:06.027630113 +0000 2024-04-21T23:00:31.820 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-21 22:54:43.818856633 +0000 2024-04-21T23:00:31.820 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-21 22:54:43.818856633 +0000 2024-04-21T23:00:31.820 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-04-21 22:54:43.818856633 +0000 2024-04-21T23:00:31.821 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T23:00:31.883 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-21T23:00:31.883 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-21T23:00:31.883 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000245888 s, 2.1 MB/s 2024-04-21T23:00:31.884 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T23:00:31.940 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_2 2024-04-21T23:00:31.995 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T23:00:31.995 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:00:31.996 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-21T23:00:31.996 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:00:31.996 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:00:31.996 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-21 23:00:06.027630113 +0000 2024-04-21T23:00:31.996 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-21 22:54:44.147849282 +0000 2024-04-21T23:00:31.996 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-21 22:54:44.147849282 +0000 2024-04-21T23:00:31.996 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-04-21 22:54:44.147849282 +0000 2024-04-21T23:00:31.997 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T23:00:32.058 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-21T23:00:32.058 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-21T23:00:32.058 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000217462 s, 2.4 MB/s 2024-04-21T23:00:32.059 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T23:00:32.115 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_3 2024-04-21T23:00:32.171 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T23:00:32.171 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:00:32.171 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-21T23:00:32.171 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:00:32.171 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:00:32.171 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-21 23:00:06.028630091 +0000 2024-04-21T23:00:32.171 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-21 22:54:44.493841552 +0000 2024-04-21T23:00:32.171 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-21 22:54:44.493841552 +0000 2024-04-21T23:00:32.171 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-04-21 22:54:44.493841552 +0000 2024-04-21T23:00:32.171 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T23:00:32.234 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-21T23:00:32.234 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-21T23:00:32.234 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000199811 s, 2.6 MB/s 2024-04-21T23:00:32.235 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T23:00:32.291 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_4 2024-04-21T23:00:32.345 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T23:00:32.345 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T23:00:32.345 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-21T23:00:32.345 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:00:32.345 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T23:00:32.345 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-21 23:00:06.028630091 +0000 2024-04-21T23:00:32.345 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-21 22:54:44.824834156 +0000 2024-04-21T23:00:32.345 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-21 22:54:44.824834156 +0000 2024-04-21T23:00:32.346 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-04-21 22:54:44.824834156 +0000 2024-04-21T23:00:32.346 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T23:00:32.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:32 smithi069 ceph-mon[31816]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:32.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:32 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:32.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:32 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:32.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:32 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:32.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:32 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:32.407 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:32 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:32.407 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:32 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:32.407 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:32 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:00:32.407 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:32 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T23:00:32.407 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:32 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:00:32.407 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:32 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:32.407 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:32 smithi069 ceph-mon[31816]: Deploying daemon mgr.z on smithi069 2024-04-21T23:00:32.408 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-21T23:00:32.408 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-21T23:00:32.409 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000195596 s, 2.6 MB/s 2024-04-21T23:00:32.410 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T23:00:32.466 INFO:tasks.cephadm:Deploying osd.0 on smithi044 with /dev/vg_nvme/lv_4... 2024-04-21T23:00:32.466 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T23:00:32.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:32 smithi044 ceph-mon[27934]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:32.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:32 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:32.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:32 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:32.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:32 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:32.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:32 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:32.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:32 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:32.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:32 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:32.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:32 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:00:32.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:32 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T23:00:32.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:32 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:00:32.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:32 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:32.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:32 smithi044 ceph-mon[27934]: Deploying daemon mgr.z on smithi069 2024-04-21T23:00:32.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:32 smithi055 ceph-mon[31117]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:32.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:32 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:32.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:32 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:32.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:32 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:32.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:32 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:32.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:32 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:32.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:32 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:32.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:32 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:00:32.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:32 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T23:00:32.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:32 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:00:32.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:32 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:32.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:32 smithi055 ceph-mon[31117]: Deploying daemon mgr.z on smithi069 2024-04-21T23:00:32.718 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:00:33.465 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:33 smithi069 systemd[1]: Starting Ceph mgr.z for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:00:33.749 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:33 smithi069 podman[33699]: 2024-04-21 23:00:33.387414756 +0000 UTC m=+0.017924160 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:00:33.749 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:33 smithi069 podman[33699]: 2024-04-21 23:00:33.505949607 +0000 UTC m=+0.136458991 container create 2fc9ae974ee6ba6994ac8963a4ea5d7c658d2a3030bc3a0c301ed88069060511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:00:33.749 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:33 smithi069 podman[33699]: 2024-04-21 23:00:33.69167158 +0000 UTC m=+0.322180964 container init 2fc9ae974ee6ba6994ac8963a4ea5d7c658d2a3030bc3a0c301ed88069060511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-21T23:00:33.749 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:33 smithi069 podman[33699]: 2024-04-21 23:00:33.694812856 +0000 UTC m=+0.325322236 container start 2fc9ae974ee6ba6994ac8963a4ea5d7c658d2a3030bc3a0c301ed88069060511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:00:33.749 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:33 smithi069 ceph-mgr[33718]: -- 172.21.15.69:0/3492884911 <== mon.1 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5650d08a85a0 con 0x5650d087c800 2024-04-21T23:00:33.749 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:33 smithi069 bash[33699]: 2fc9ae974ee6ba6994ac8963a4ea5d7c658d2a3030bc3a0c301ed88069060511 2024-04-21T23:00:34.030 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:33 smithi069 systemd[1]: Started Ceph mgr.z for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:00:34.030 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:33 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:33.850+0000 7f5adf960200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T23:00:34.030 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:34 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:34.030+0000 7f5adf960200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:00:34.434 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:34 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:34.131+0000 7f5adf960200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T23:00:34.434 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:34 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:34.433+0000 7f5adf960200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T23:00:34.632 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T23:00:34.632 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T23:00:34.632 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2024-04-21T23:00:34.633 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2024-04-21T23:00:34.633 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110973 s, 945 MB/s 2024-04-21T23:00:34.633 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2024-04-21T23:00:34.714 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:34 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:34.517+0000 7f5adf960200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T23:00:34.901 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:34 smithi044 ceph-mon[27934]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:34.902 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:34 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:34.902 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:34 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:34.902 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:34 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:34.902 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:34 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:00:34.902 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:34 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T23:00:34.902 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:34 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:00:34.902 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:34 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:34.902 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:34 smithi044 ceph-mon[27934]: Deploying daemon mgr.y on smithi055 2024-04-21T23:00:35.155 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi044:vg_nvme/lv_4 2024-04-21T23:00:35.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:34 smithi055 ceph-mon[31117]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:35.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:34 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:35.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:34 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:35.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:34 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:35.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:34 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:00:35.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:34 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T23:00:35.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:34 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:00:35.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:34 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:35.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:34 smithi055 ceph-mon[31117]: Deploying daemon mgr.y on smithi055 2024-04-21T23:00:35.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:34 smithi069 ceph-mon[31816]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:35.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:34 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:35.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:34 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:35.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:34 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:35.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:34 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T23:00:35.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:34 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T23:00:35.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:34 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T23:00:35.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:34 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:35.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:34 smithi069 ceph-mon[31816]: Deploying daemon mgr.y on smithi055 2024-04-21T23:00:35.401 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:00:35.964 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:35 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:35.696+0000 7f5adf960200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T23:00:35.965 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:35 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:35.866+0000 7f5adf960200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T23:00:35.965 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:35 smithi055 systemd[1]: Starting Ceph mgr.y for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:00:36.337 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:36 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:36.038+0000 7f5adf960200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T23:00:36.338 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:36 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:36.336+0000 7f5adf960200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T23:00:36.340 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:36 smithi055 podman[32527]: 2024-04-21 23:00:36.077784221 +0000 UTC m=+0.115060381 container create 43fd150ed11af7a594d896e713d9606aa6e1d862306326e27ae0f235ed146823 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-21T23:00:36.340 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:36 smithi055 podman[32527]: 2024-04-21 23:00:35.981243005 +0000 UTC m=+0.018519100 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:00:36.340 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:36 smithi055 podman[32527]: 2024-04-21 23:00:36.313235705 +0000 UTC m=+0.350511800 container init 43fd150ed11af7a594d896e713d9606aa6e1d862306326e27ae0f235ed146823 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:00:36.340 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:36 smithi055 podman[32527]: 2024-04-21 23:00:36.316045172 +0000 UTC m=+0.353321267 container start 43fd150ed11af7a594d896e713d9606aa6e1d862306326e27ae0f235ed146823 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-21T23:00:36.643 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:36 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:36.431+0000 7f5adf960200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T23:00:36.643 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:36 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:36.642+0000 7f5adf960200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T23:00:36.663 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:36 smithi055 ceph-mgr[32547]: -- 172.21.15.55:0/859051949 <== mon.2 v2:172.21.15.55:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557e4bdfc5a0 con 0x557e4bdd1000 2024-04-21T23:00:36.663 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:36 smithi055 bash[32527]: 43fd150ed11af7a594d896e713d9606aa6e1d862306326e27ae0f235ed146823 2024-04-21T23:00:36.664 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:36 smithi055 systemd[1]: Started Ceph mgr.y for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:00:36.664 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:36 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:36.471+0000 7f2d79afd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T23:00:36.664 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:36 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:36.663+0000 7f2d79afd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:00:36.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.770+0000 7fb70e3ed700 1 -- 172.21.15.44:0/4207996677 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb708102d30 msgr2=0x7fb7081095e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:36.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.770+0000 7fb70e3ed700 1 --2- 172.21.15.44:0/4207996677 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb708102d30 0x7fb7081095e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb70000b0d0 tx=0x7fb70000b3e0 comp rx=0 tx=0).stop 2024-04-21T23:00:36.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.770+0000 7fb70e3ed700 1 -- 172.21.15.44:0/4207996677 shutdown_connections 2024-04-21T23:00:36.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.770+0000 7fb70e3ed700 1 --2- 172.21.15.44:0/4207996677 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb708102d30 0x7fb7081095e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:36.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.770+0000 7fb70e3ed700 1 --2- 172.21.15.44:0/4207996677 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb708101190 0x7fb7081015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:36.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.770+0000 7fb70e3ed700 1 --2- 172.21.15.44:0/4207996677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb708102390 0x7fb7081027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:36.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.770+0000 7fb70e3ed700 1 -- 172.21.15.44:0/4207996677 >> 172.21.15.44:0/4207996677 conn(0x7fb7080fc730 msgr2=0x7fb7080feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:00:36.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.770+0000 7fb70e3ed700 1 -- 172.21.15.44:0/4207996677 shutdown_connections 2024-04-21T23:00:36.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.770+0000 7fb70e3ed700 1 -- 172.21.15.44:0/4207996677 wait complete. 2024-04-21T23:00:36.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.771+0000 7fb70e3ed700 1 Processor -- start 2024-04-21T23:00:36.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.771+0000 7fb70e3ed700 1 -- start start 2024-04-21T23:00:36.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.771+0000 7fb70e3ed700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb708101190 0x7fb708105120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:36.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.771+0000 7fb70e3ed700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb708102390 0x7fb708103770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:36.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.771+0000 7fb70e3ed700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb708102d30 0x7fb708103cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:36.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.771+0000 7fb70e3ed700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb70810cd60 con 0x7fb708101190 2024-04-21T23:00:36.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.771+0000 7fb70e3ed700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb70810cbe0 con 0x7fb708102390 2024-04-21T23:00:36.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.771+0000 7fb70e3ed700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb70810cee0 con 0x7fb708102d30 2024-04-21T23:00:36.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.771+0000 7fb707fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb708101190 0x7fb708105120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:36.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.771+0000 7fb707fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb708101190 0x7fb708105120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43166/0 (socket says 172.21.15.44:43166) 2024-04-21T23:00:36.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.771+0000 7fb707fff700 1 -- 172.21.15.44:0/1103396883 learned_addr learned my addr 172.21.15.44:0/1103396883 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:00:36.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.771+0000 7fb70c98a700 1 --2- 172.21.15.44:0/1103396883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb708102d30 0x7fb708103cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:36.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.771+0000 7fb7077fe700 1 --2- 172.21.15.44:0/1103396883 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb708102390 0x7fb708103770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:36.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.771+0000 7fb707fff700 1 -- 172.21.15.44:0/1103396883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb708102d30 msgr2=0x7fb708103cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:36.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.771+0000 7fb707fff700 1 --2- 172.21.15.44:0/1103396883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb708102d30 0x7fb708103cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:36.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.771+0000 7fb707fff700 1 -- 172.21.15.44:0/1103396883 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb708102390 msgr2=0x7fb708103770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:36.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.771+0000 7fb707fff700 1 --2- 172.21.15.44:0/1103396883 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb708102390 0x7fb708103770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:36.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.771+0000 7fb707fff700 1 -- 172.21.15.44:0/1103396883 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7081041f0 con 0x7fb708101190 2024-04-21T23:00:36.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.772+0000 7fb707fff700 1 --2- 172.21.15.44:0/1103396883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb708101190 0x7fb708105120 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb6f800d980 tx=0x7fb6f800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:00:36.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.772+0000 7fb7057fa700 1 -- 172.21.15.44:0/1103396883 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8016070 con 0x7fb708101190 2024-04-21T23:00:36.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.772+0000 7fb7057fa700 1 -- 172.21.15.44:0/1103396883 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6f800fd70 con 0x7fb708101190 2024-04-21T23:00:36.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.772+0000 7fb70e3ed700 1 -- 172.21.15.44:0/1103396883 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7081044e0 con 0x7fb708101190 2024-04-21T23:00:36.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.772+0000 7fb7057fa700 1 -- 172.21.15.44:0/1103396883 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f8019360 con 0x7fb708101190 2024-04-21T23:00:36.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.772+0000 7fb70e3ed700 1 -- 172.21.15.44:0/1103396883 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7081a6f90 con 0x7fb708101190 2024-04-21T23:00:36.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.773+0000 7fb70e3ed700 1 -- 172.21.15.44:0/1103396883 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb708068f20 con 0x7fb708101190 2024-04-21T23:00:36.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.773+0000 7fb7057fa700 1 -- 172.21.15.44:0/1103396883 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fb6f8019070 con 0x7fb708101190 2024-04-21T23:00:36.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.773+0000 7fb7057fa700 1 --2- 172.21.15.44:0/1103396883 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb6f0039330 0x7fb6f003b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:00:36.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.773+0000 7fb7057fa700 1 -- 172.21.15.44:0/1103396883 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb6f804bb80 con 0x7fb708101190 2024-04-21T23:00:36.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.773+0000 7fb7077fe700 1 --2- 172.21.15.44:0/1103396883 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb6f0039330 0x7fb6f003b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:00:36.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.774+0000 7fb7077fe700 1 --2- 172.21.15.44:0/1103396883 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb6f0039330 0x7fb6f003b7f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc005950 tx=0x7fb6fc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:00:36.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.778+0000 7fb7057fa700 1 -- 172.21.15.44:0/1103396883 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb6f800b2e0 con 0x7fb708101190 2024-04-21T23:00:36.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:36 smithi044 ceph-mon[27934]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:36.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:36 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:36.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:36 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:36.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:36 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:36.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:36 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:36.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:36 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:36.964 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:36 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:36.770+0000 7f2d79afd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T23:00:36.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:36 smithi055 ceph-mon[31117]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:36.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:36 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:36.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:36 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:36.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:36 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:36.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:36 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:36.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:36 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:36.965 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:36 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:36.844+0000 7f5adf960200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:00:36.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:36 smithi069 ceph-mon[31816]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:36.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:36 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:36.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:36 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:36.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:36 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:36.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:36 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:36.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:36 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:36.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:36.996+0000 7fb70e3ed700 1 -- 172.21.15.44:0/1103396883 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fb7080fe8c0 con 0x7fb6f0039330 2024-04-21T23:00:37.464 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:37 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:37.103+0000 7f2d79afd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T23:00:37.464 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:37 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:37.197+0000 7f2d79afd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T23:00:37.830 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:37 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:37.476+0000 7f5adf960200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T23:00:37.932 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:37 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:00:37.932 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:37 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:00:37.932 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:37 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:38.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:37 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:00:38.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:37 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:00:38.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:37 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:38.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:37 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:00:38.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:37 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:00:38.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:37 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:38.215 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:37 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:37.912+0000 7f5adf960200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T23:00:38.216 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:38 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:38.017+0000 7f5adf960200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T23:00:38.714 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:38 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:38.221+0000 7f5adf960200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T23:00:38.832 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:38 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:38.499+0000 7f2d79afd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T23:00:38.832 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:38 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:38.675+0000 7f2d79afd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T23:00:38.931 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:38 smithi044 ceph-mon[27934]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:38.931 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:38 smithi044 ceph-mon[27934]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:00:38.931 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:38 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:38.931 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:38 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:38.931 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:38 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:38.931 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:38 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:38.931 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:38 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:38.931 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:38 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:38.931 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:38 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:38.932 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:38 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:38.932 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:38 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:38.932 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:38 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:38.932 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:38 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:38.932 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:38 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:39.079 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:38 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:38.747+0000 7f5adf960200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T23:00:39.079 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:38 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:38.841+0000 7f5adf960200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T23:00:39.079 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:38 smithi069 ceph-mon[31816]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:39.079 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:38 smithi069 ceph-mon[31816]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:00:39.079 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:38 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:39.079 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:38 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:39.079 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:38 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:39.079 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:38 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:39.080 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:38 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:39.080 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:38 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:39.080 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:38 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:39.080 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:38 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:39.080 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:38 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:39.080 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:38 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:39.080 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:38 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:39.080 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:38 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:39.151 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:38 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:38.853+0000 7f2d79afd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T23:00:39.151 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:39 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:39.150+0000 7f2d79afd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T23:00:39.151 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:38 smithi055 ceph-mon[31117]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:39.151 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:38 smithi055 ceph-mon[31117]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:00:39.151 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:38 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:39.151 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:38 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:39.151 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:38 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:39.151 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:38 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:39.151 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:38 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:39.151 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:38 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:39.151 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:38 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:39.152 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:38 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:39.152 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:38 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:39.152 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:38 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:39.152 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:38 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:39.152 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:38 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:39.419 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:39 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:39.078+0000 7f5adf960200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T23:00:39.419 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:39 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:39.209+0000 7f5adf960200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T23:00:39.419 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:39 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:39.302+0000 7f5adf960200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T23:00:39.419 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:39 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:39.418+0000 7f5adf960200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T23:00:39.456 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:39 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:39.246+0000 7f2d79afd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T23:00:39.456 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:39 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:39.456+0000 7f2d79afd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T23:00:39.714 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:39 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:39.655+0000 7f2d79afd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:00:39.715 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:39 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:39.508+0000 7f5adf960200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T23:00:40.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:39 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2924875427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaaab74d-dfe1-4669-9839-36b2664ac353"}]: dispatch 2024-04-21T23:00:40.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:39 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2924875427' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aaaab74d-dfe1-4669-9839-36b2664ac353"}]': finished 2024-04-21T23:00:40.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:39 smithi044 ceph-mon[27934]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T23:00:40.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:00:40.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:39 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2924875427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaaab74d-dfe1-4669-9839-36b2664ac353"}]: dispatch 2024-04-21T23:00:40.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:39 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2924875427' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aaaab74d-dfe1-4669-9839-36b2664ac353"}]': finished 2024-04-21T23:00:40.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:39 smithi055 ceph-mon[31117]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T23:00:40.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:39 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:00:40.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:39 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2924875427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaaab74d-dfe1-4669-9839-36b2664ac353"}]: dispatch 2024-04-21T23:00:40.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:39 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2924875427' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aaaab74d-dfe1-4669-9839-36b2664ac353"}]': finished 2024-04-21T23:00:40.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:39 smithi069 ceph-mon[31816]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T23:00:40.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:39 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:00:40.714 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:40 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:40.286+0000 7f2d79afd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T23:00:40.715 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:40 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:40.324+0000 7f5adf960200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T23:00:40.715 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:00:40 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:00:40.535+0000 7f5adf960200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T23:00:40.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:40.843+0000 7fb7057fa700 1 -- 172.21.15.44:0/1103396883 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mgrmap(e 14) ==== 92060+0+0 (secure 0 0 0) 0x7fb6f801a3b0 con 0x7fb708101190 2024-04-21T23:00:41.029 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:40 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:40.721+0000 7f2d79afd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T23:00:41.029 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:40 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:40.821+0000 7f2d79afd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T23:00:41.029 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:41 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:41.029+0000 7f2d79afd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T23:00:41.029 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:40 smithi055 ceph-mon[31117]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:41.030 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:40 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3421843896' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:00:41.030 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:40 smithi055 ceph-mon[31117]: Standby manager daemon z started 2024-04-21T23:00:41.030 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:40 smithi055 ceph-mon[31117]: from='mgr.? 172.21.15.69:0/2790674316' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T23:00:41.030 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:40 smithi055 ceph-mon[31117]: from='mgr.? 172.21.15.69:0/2790674316' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:00:41.030 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:40 smithi055 ceph-mon[31117]: from='mgr.? 172.21.15.69:0/2790674316' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T23:00:41.030 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:40 smithi055 ceph-mon[31117]: from='mgr.? 172.21.15.69:0/2790674316' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:00:41.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:40 smithi044 ceph-mon[27934]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:41.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:40 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3421843896' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:00:41.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:40 smithi044 ceph-mon[27934]: Standby manager daemon z started 2024-04-21T23:00:41.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:40 smithi044 ceph-mon[27934]: from='mgr.? 172.21.15.69:0/2790674316' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T23:00:41.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:40 smithi044 ceph-mon[27934]: from='mgr.? 172.21.15.69:0/2790674316' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:00:41.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:40 smithi044 ceph-mon[27934]: from='mgr.? 172.21.15.69:0/2790674316' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T23:00:41.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:40 smithi044 ceph-mon[27934]: from='mgr.? 172.21.15.69:0/2790674316' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:00:41.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:40 smithi069 ceph-mon[31816]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:41.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:40 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3421843896' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:00:41.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:40 smithi069 ceph-mon[31816]: Standby manager daemon z started 2024-04-21T23:00:41.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:40 smithi069 ceph-mon[31816]: from='mgr.? 172.21.15.69:0/2790674316' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T23:00:41.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:40 smithi069 ceph-mon[31816]: from='mgr.? 172.21.15.69:0/2790674316' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:00:41.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:40 smithi069 ceph-mon[31816]: from='mgr.? 172.21.15.69:0/2790674316' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T23:00:41.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:40 smithi069 ceph-mon[31816]: from='mgr.? 172.21.15.69:0/2790674316' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:00:41.847 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:41 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:41.574+0000 7f2d79afd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T23:00:41.847 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:41 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:41.668+0000 7f2d79afd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T23:00:42.101 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:41 smithi044 ceph-mon[27934]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-21T23:00:42.101 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:41 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T23:00:42.154 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:41 smithi055 ceph-mon[31117]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-21T23:00:42.154 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:41 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T23:00:42.155 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:41 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:41.919+0000 7f2d79afd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T23:00:42.155 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:42 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:42.057+0000 7f2d79afd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T23:00:42.155 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:42 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:42.154+0000 7f2d79afd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T23:00:42.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:41 smithi069 ceph-mon[31816]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-21T23:00:42.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:41 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T23:00:42.464 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:42 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:42.273+0000 7f2d79afd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T23:00:42.464 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:42 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:42.367+0000 7f2d79afd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T23:00:43.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:42 smithi044 ceph-mon[27934]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:43.213 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:42 smithi055 ceph-mon[31117]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:43.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:42 smithi069 ceph-mon[31816]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:43.464 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:43 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:43.213+0000 7f2d79afd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T23:00:43.464 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:00:43 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:00:43.433+0000 7f2d79afd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T23:00:43.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:43.867+0000 7fb7057fa700 1 -- 172.21.15.44:0/1103396883 <== mon.0 v2:172.21.15.44:3300/0 8 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7fb6f8004830 con 0x7fb708101190 2024-04-21T23:00:44.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:43 smithi044 ceph-mon[27934]: Standby manager daemon y started 2024-04-21T23:00:44.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:43 smithi044 ceph-mon[27934]: from='mgr.? 172.21.15.55:0/2422236459' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T23:00:44.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:43 smithi044 ceph-mon[27934]: from='mgr.? 172.21.15.55:0/2422236459' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:00:44.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:43 smithi044 ceph-mon[27934]: from='mgr.? 172.21.15.55:0/2422236459' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T23:00:44.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:43 smithi044 ceph-mon[27934]: from='mgr.? 172.21.15.55:0/2422236459' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:00:44.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:43 smithi055 ceph-mon[31117]: Standby manager daemon y started 2024-04-21T23:00:44.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:43 smithi055 ceph-mon[31117]: from='mgr.? 172.21.15.55:0/2422236459' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T23:00:44.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:43 smithi055 ceph-mon[31117]: from='mgr.? 172.21.15.55:0/2422236459' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:00:44.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:43 smithi055 ceph-mon[31117]: from='mgr.? 172.21.15.55:0/2422236459' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T23:00:44.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:43 smithi055 ceph-mon[31117]: from='mgr.? 172.21.15.55:0/2422236459' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:00:44.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:43 smithi069 ceph-mon[31816]: Standby manager daemon y started 2024-04-21T23:00:44.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:43 smithi069 ceph-mon[31816]: from='mgr.? 172.21.15.55:0/2422236459' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T23:00:44.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:43 smithi069 ceph-mon[31816]: from='mgr.? 172.21.15.55:0/2422236459' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:00:44.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:43 smithi069 ceph-mon[31816]: from='mgr.? 172.21.15.55:0/2422236459' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T23:00:44.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:43 smithi069 ceph-mon[31816]: from='mgr.? 172.21.15.55:0/2422236459' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:00:45.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:44 smithi044 ceph-mon[27934]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:45.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:44 smithi044 ceph-mon[27934]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-21T23:00:45.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:44 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T23:00:45.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:44 smithi055 ceph-mon[31117]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:45.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:44 smithi055 ceph-mon[31117]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-21T23:00:45.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:44 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T23:00:45.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:44 smithi069 ceph-mon[31816]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:45.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:44 smithi069 ceph-mon[31816]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-21T23:00:45.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:44 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T23:00:47.129 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:46 smithi069 ceph-mon[31816]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:47.129 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:46 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T23:00:47.130 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:46 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:47.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:46 smithi044 ceph-mon[27934]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:47.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:46 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T23:00:47.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:46 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:47.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:46 smithi055 ceph-mon[31117]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:47.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:46 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T23:00:47.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:46 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:48.089 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:47 smithi044 ceph-mon[27934]: Deploying daemon osd.0 on smithi044 2024-04-21T23:00:48.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:47 smithi055 ceph-mon[31117]: Deploying daemon osd.0 on smithi044 2024-04-21T23:00:48.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:47 smithi069 ceph-mon[31816]: Deploying daemon osd.0 on smithi044 2024-04-21T23:00:49.053 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:48 smithi044 ceph-mon[27934]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:49.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:48 smithi069 ceph-mon[31816]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:49.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:48 smithi055 ceph-mon[31117]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:51.055 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:50 smithi044 ceph-mon[27934]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:51.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:50 smithi055 ceph-mon[31117]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:51.217 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:50 smithi069 ceph-mon[31816]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:53.172 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:52 smithi044 ceph-mon[27934]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:53.172 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:52 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:53.173 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:52 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:53.173 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:52 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:53.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:52 smithi069 ceph-mon[31816]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:53.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:52 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:53.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:52 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:53.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:52 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:53.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:52 smithi055 ceph-mon[31117]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:53.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:52 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:53.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:52 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:53.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:52 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:54.896 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:54 smithi044 ceph-mon[27934]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:54.896 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:54.896 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:54.896 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:54.896 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:54.896 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:55.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:54 smithi055 ceph-mon[31117]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:55.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:54 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:55.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:54 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:55.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:54 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:55.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:54 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:55.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:54 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:55.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:54 smithi069 ceph-mon[31816]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:55.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:54 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:55.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:54 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:55.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:54 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:55.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:54 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:55.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:54 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:55.602 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 0 on host 'smithi044' 2024-04-21T23:00:55.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:55.602+0000 7fb7057fa700 1 -- 172.21.15.44:0/1103396883 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb7080fe8c0 con 0x7fb6f0039330 2024-04-21T23:00:55.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:55.604+0000 7fb70e3ed700 1 -- 172.21.15.44:0/1103396883 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb6f0039330 msgr2=0x7fb6f003b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:55.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:55.604+0000 7fb70e3ed700 1 --2- 172.21.15.44:0/1103396883 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb6f0039330 0x7fb6f003b7f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc005950 tx=0x7fb6fc00b410 comp rx=0 tx=0).stop 2024-04-21T23:00:55.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:55.605+0000 7fb70e3ed700 1 -- 172.21.15.44:0/1103396883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb708101190 msgr2=0x7fb708105120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:00:55.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:55.605+0000 7fb70e3ed700 1 --2- 172.21.15.44:0/1103396883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb708101190 0x7fb708105120 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb6f800d980 tx=0x7fb6f800dc90 comp rx=0 tx=0).stop 2024-04-21T23:00:55.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:55.605+0000 7fb70e3ed700 1 -- 172.21.15.44:0/1103396883 shutdown_connections 2024-04-21T23:00:55.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:55.605+0000 7fb70e3ed700 1 --2- 172.21.15.44:0/1103396883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb708102d30 0x7fb708103cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:55.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:55.605+0000 7fb70e3ed700 1 --2- 172.21.15.44:0/1103396883 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb6f0039330 0x7fb6f003b7f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:55.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:55.605+0000 7fb70e3ed700 1 --2- 172.21.15.44:0/1103396883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb708101190 0x7fb708105120 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:55.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:55.605+0000 7fb70e3ed700 1 --2- 172.21.15.44:0/1103396883 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb708102390 0x7fb708103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:00:55.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:55.605+0000 7fb70e3ed700 1 -- 172.21.15.44:0/1103396883 >> 172.21.15.44:0/1103396883 conn(0x7fb7080fc730 msgr2=0x7fb7080fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:00:55.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:55.605+0000 7fb70e3ed700 1 -- 172.21.15.44:0/1103396883 shutdown_connections 2024-04-21T23:00:55.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:00:55.605+0000 7fb70e3ed700 1 -- 172.21.15.44:0/1103396883 wait complete. 2024-04-21T23:00:55.919 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:55 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:55.919 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:55 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:55.919 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:55 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:55.919 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:55 smithi044 ceph-mon[27934]: from='osd.0 [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T23:00:56.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:55 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:56.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:55 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:56.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:55 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:56.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:55 smithi055 ceph-mon[31117]: from='osd.0 [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T23:00:56.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:55 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:00:56.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:55 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:56.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:55 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:56.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:55 smithi069 ceph-mon[31816]: from='osd.0 [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T23:00:56.259 DEBUG:teuthology.orchestra.run.smithi044:osd.0> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.0.service 2024-04-21T23:00:56.261 INFO:tasks.cephadm:Deploying osd.1 on smithi055 with /dev/vg_nvme/lv_4... 2024-04-21T23:00:56.262 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T23:00:56.513 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:00:56.946 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:56 smithi055 ceph-mon[31117]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:56.946 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:56 smithi055 ceph-mon[31117]: from='osd.0 [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T23:00:56.946 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:56 smithi055 ceph-mon[31117]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T23:00:56.946 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:56 smithi055 ceph-mon[31117]: from='osd.0 [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-21T23:00:56.946 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:56 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:00:57.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:56 smithi044 ceph-mon[27934]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:57.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:56 smithi044 ceph-mon[27934]: from='osd.0 [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T23:00:57.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:56 smithi044 ceph-mon[27934]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T23:00:57.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:56 smithi044 ceph-mon[27934]: from='osd.0 [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-21T23:00:57.102 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:56 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:00:57.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:56 smithi069 ceph-mon[31816]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:57.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:56 smithi069 ceph-mon[31816]: from='osd.0 [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T23:00:57.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:56 smithi069 ceph-mon[31816]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T23:00:57.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:56 smithi069 ceph-mon[31816]: from='osd.0 [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-21T23:00:57.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:56 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:00:57.797 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:00:57 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-0[40172]: 2024-04-21T23:00:57.602+0000 7fe7f105a700 -1 osd.0 0 waiting for initial osdmap 2024-04-21T23:00:57.798 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:00:57 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-0[40172]: 2024-04-21T23:00:57.606+0000 7fe7ece4d700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:00:58.430 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T23:00:58.431 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T23:00:58.431 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10+0 records in 2024-04-21T23:00:58.431 INFO:teuthology.orchestra.run.smithi055.stderr:10+0 records out 2024-04-21T23:00:58.431 INFO:teuthology.orchestra.run.smithi055.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0156537 s, 670 MB/s 2024-04-21T23:00:58.431 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping successful for: 2024-04-21T23:00:58.764 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:58 smithi055 ceph-mon[31117]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:58.764 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:58 smithi055 ceph-mon[31117]: from='osd.0 [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-04-21T23:00:58.764 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:58 smithi055 ceph-mon[31117]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T23:00:58.764 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:58 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:00:58.765 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:58 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:00:58.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:58 smithi044 ceph-mon[27934]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:58.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:58 smithi044 ceph-mon[27934]: from='osd.0 [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-04-21T23:00:58.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:58 smithi044 ceph-mon[27934]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T23:00:58.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:58 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:00:58.765 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:58 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:00:58.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:58 smithi069 ceph-mon[31816]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T23:00:58.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:58 smithi069 ceph-mon[31816]: from='osd.0 [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-04-21T23:00:58.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:58 smithi069 ceph-mon[31816]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T23:00:58.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:58 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:00:58.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:58 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:00:59.058 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi055:vg_nvme/lv_4 2024-04-21T23:00:59.303 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:00:59.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:59 smithi055 ceph-mon[31117]: purged_snaps scrub starts 2024-04-21T23:00:59.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:59 smithi055 ceph-mon[31117]: purged_snaps scrub ok 2024-04-21T23:00:59.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:59 smithi055 ceph-mon[31117]: osd.0 [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096] boot 2024-04-21T23:00:59.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:59 smithi055 ceph-mon[31117]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T23:00:59.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:59 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:00:59.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:59 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:59.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:59 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:59.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:59 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:00:59.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:59 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:59.716 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:59 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:59.716 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:59 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:59.716 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:00:59 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:59.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:59 smithi044 ceph-mon[27934]: purged_snaps scrub starts 2024-04-21T23:00:59.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:59 smithi044 ceph-mon[27934]: purged_snaps scrub ok 2024-04-21T23:00:59.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:59 smithi044 ceph-mon[27934]: osd.0 [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096] boot 2024-04-21T23:00:59.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:59 smithi044 ceph-mon[27934]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T23:00:59.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:59 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:00:59.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:59 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:59.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:59 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:59.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:59 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:00:59.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:59 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:59.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:59 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:59.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:59 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:59.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:00:59 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:59.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:59 smithi069 ceph-mon[31816]: purged_snaps scrub starts 2024-04-21T23:00:59.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:59 smithi069 ceph-mon[31816]: purged_snaps scrub ok 2024-04-21T23:00:59.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:59 smithi069 ceph-mon[31816]: osd.0 [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096] boot 2024-04-21T23:00:59.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:59 smithi069 ceph-mon[31816]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T23:00:59.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:59 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:00:59.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:59 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:59.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:59 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:59.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:59 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:00:59.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:59 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:00:59.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:59 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:00:59.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:59 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:00:59.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:00:59 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:00.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.800+0000 7fea82fbb700 1 -- 172.21.15.55:0/962798995 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea7c102290 msgr2=0x7fea7c1026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:01:00.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.800+0000 7fea82fbb700 1 --2- 172.21.15.55:0/962798995 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea7c102290 0x7fea7c1026b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fea700099c0 tx=0x7fea70009cd0 comp rx=0 tx=0).stop 2024-04-21T23:01:00.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.800+0000 7fea82fbb700 1 -- 172.21.15.55:0/962798995 shutdown_connections 2024-04-21T23:01:00.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.800+0000 7fea82fbb700 1 --2- 172.21.15.55:0/962798995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea7c103e30 0x7fea7c10a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:00.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.800+0000 7fea82fbb700 1 --2- 172.21.15.55:0/962798995 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea7c102290 0x7fea7c1026b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:00.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.800+0000 7fea82fbb700 1 --2- 172.21.15.55:0/962798995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea7c103490 0x7fea7c1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:00.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.800+0000 7fea82fbb700 1 -- 172.21.15.55:0/962798995 >> 172.21.15.55:0/962798995 conn(0x7fea7c0fd830 msgr2=0x7fea7c0ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:01:00.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.800+0000 7fea82fbb700 1 -- 172.21.15.55:0/962798995 shutdown_connections 2024-04-21T23:01:00.801 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.800+0000 7fea82fbb700 1 -- 172.21.15.55:0/962798995 wait complete. 2024-04-21T23:01:00.802 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:00 smithi055 ceph-mon[31117]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:00.802 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:00 smithi055 ceph-mon[31117]: Detected new or changed devices on smithi044 2024-04-21T23:01:00.802 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:00 smithi055 ceph-mon[31117]: Adjusting osd_memory_target on smithi044 to 17093M 2024-04-21T23:01:00.802 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:00 smithi055 ceph-mon[31117]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T23:01:00.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.801+0000 7fea82fbb700 1 Processor -- start 2024-04-21T23:01:00.803 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.801+0000 7fea82fbb700 1 -- start start 2024-04-21T23:01:00.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.801+0000 7fea82fbb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea7c102290 0x7fea7c075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:01:00.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.801+0000 7fea82fbb700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea7c103490 0x7fea7c074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:01:00.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.801+0000 7fea82fbb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea7c103e30 0x7fea7c074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:01:00.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.801+0000 7fea82fbb700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fea7c10de50 con 0x7fea7c103490 2024-04-21T23:01:00.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.801+0000 7fea82fbb700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fea7c10dcd0 con 0x7fea7c102290 2024-04-21T23:01:00.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.801+0000 7fea82fbb700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fea7c10dfd0 con 0x7fea7c103e30 2024-04-21T23:01:00.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.801+0000 7fea80d57700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea7c102290 0x7fea7c075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:01:00.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.801+0000 7fea80d57700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea7c102290 0x7fea7c075fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:42118/0 (socket says 172.21.15.55:42118) 2024-04-21T23:01:00.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.801+0000 7fea80d57700 1 -- 172.21.15.55:0/54935393 learned_addr learned my addr 172.21.15.55:0/54935393 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-04-21T23:01:00.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.801+0000 7fea81558700 1 --2- 172.21.15.55:0/54935393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea7c103e30 0x7fea7c074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:01:00.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.801+0000 7fea7bfff700 1 --2- 172.21.15.55:0/54935393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea7c103490 0x7fea7c074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:01:00.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.802+0000 7fea80d57700 1 -- 172.21.15.55:0/54935393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea7c103e30 msgr2=0x7fea7c074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:01:00.804 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.802+0000 7fea80d57700 1 --2- 172.21.15.55:0/54935393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea7c103e30 0x7fea7c074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:00.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.802+0000 7fea80d57700 1 -- 172.21.15.55:0/54935393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea7c103490 msgr2=0x7fea7c074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:01:00.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.802+0000 7fea80d57700 1 --2- 172.21.15.55:0/54935393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea7c103490 0x7fea7c074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:00.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.802+0000 7fea80d57700 1 -- 172.21.15.55:0/54935393 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea7c075410 con 0x7fea7c102290 2024-04-21T23:01:00.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.802+0000 7fea80d57700 1 --2- 172.21.15.55:0/54935393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea7c102290 0x7fea7c075fd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fea7000c010 tx=0x7fea70009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:01:00.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.802+0000 7fea79ffb700 1 -- 172.21.15.55:0/54935393 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea70027070 con 0x7fea7c102290 2024-04-21T23:01:00.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.802+0000 7fea79ffb700 1 -- 172.21.15.55:0/54935393 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea70004150 con 0x7fea7c102290 2024-04-21T23:01:00.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.802+0000 7fea79ffb700 1 -- 172.21.15.55:0/54935393 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea70022d40 con 0x7fea7c102290 2024-04-21T23:01:00.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.802+0000 7fea82fbb700 1 -- 172.21.15.55:0/54935393 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea7c1a25f0 con 0x7fea7c102290 2024-04-21T23:01:00.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.802+0000 7fea82fbb700 1 -- 172.21.15.55:0/54935393 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea7c1a2a50 con 0x7fea7c102290 2024-04-21T23:01:00.805 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.803+0000 7fea82fbb700 1 -- 172.21.15.55:0/54935393 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea7c068f20 con 0x7fea7c102290 2024-04-21T23:01:00.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.804+0000 7fea79ffb700 1 -- 172.21.15.55:0/54935393 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7fea7002a070 con 0x7fea7c102290 2024-04-21T23:01:00.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.805+0000 7fea79ffb700 1 --2- 172.21.15.55:0/54935393 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fea6c0a2b90 0x7fea6c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:01:00.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.805+0000 7fea79ffb700 1 -- 172.21.15.55:0/54935393 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fea700ce430 con 0x7fea7c102290 2024-04-21T23:01:00.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.805+0000 7fea7bfff700 1 --2- 172.21.15.55:0/54935393 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fea6c0a2b90 0x7fea6c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:01:00.806 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.806+0000 7fea7bfff700 1 --2- 172.21.15.55:0/54935393 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fea6c0a2b90 0x7fea6c0a5050 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fea68005950 tx=0x7fea6800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:01:00.809 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:00.810+0000 7fea79ffb700 1 -- 172.21.15.55:0/54935393 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fea7005e6f0 con 0x7fea7c102290 2024-04-21T23:01:00.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:00 smithi044 ceph-mon[27934]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:00.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:00 smithi044 ceph-mon[27934]: Detected new or changed devices on smithi044 2024-04-21T23:01:00.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:00 smithi044 ceph-mon[27934]: Adjusting osd_memory_target on smithi044 to 17093M 2024-04-21T23:01:00.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:00 smithi044 ceph-mon[27934]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T23:01:00.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:00 smithi069 ceph-mon[31816]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:00.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:00 smithi069 ceph-mon[31816]: Detected new or changed devices on smithi044 2024-04-21T23:01:00.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:00 smithi069 ceph-mon[31816]: Adjusting osd_memory_target on smithi044 to 17093M 2024-04-21T23:01:00.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:00 smithi069 ceph-mon[31816]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T23:01:01.025 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:01.025+0000 7fea82fbb700 1 -- 172.21.15.55:0/54935393 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fea7c062670 con 0x7fea6c0a2b90 2024-04-21T23:01:01.930 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:01 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:01:01.930 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:01 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:01:01.930 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:01 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:01.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:01 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:01:01.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:01 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:01:01.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:01 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:01.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:01 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:01:01.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:01 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:01:01.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:01 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:02.887 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:02 smithi055 ceph-mon[31117]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:02.887 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:02 smithi055 ceph-mon[31117]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:01:02.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:02 smithi044 ceph-mon[27934]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:02.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:02 smithi044 ceph-mon[27934]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:01:02.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:02 smithi069 ceph-mon[31816]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:02.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:02 smithi069 ceph-mon[31816]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:01:04.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:04 smithi044 ceph-mon[27934]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:04.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:04 smithi044 ceph-mon[27934]: from='client.? 172.21.15.55:0/119892589' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7950725c-5dea-4a3b-a06d-4e50b621110b"}]: dispatch 2024-04-21T23:01:04.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:04 smithi044 ceph-mon[27934]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7950725c-5dea-4a3b-a06d-4e50b621110b"}]: dispatch 2024-04-21T23:01:04.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:04 smithi044 ceph-mon[27934]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7950725c-5dea-4a3b-a06d-4e50b621110b"}]': finished 2024-04-21T23:01:04.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:04 smithi044 ceph-mon[27934]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T23:01:04.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:04 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:01:04.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:04 smithi044 ceph-mon[27934]: from='client.? 172.21.15.55:0/1362143862' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:01:04.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:04 smithi055 ceph-mon[31117]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:04.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:04 smithi055 ceph-mon[31117]: from='client.? 172.21.15.55:0/119892589' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7950725c-5dea-4a3b-a06d-4e50b621110b"}]: dispatch 2024-04-21T23:01:04.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:04 smithi055 ceph-mon[31117]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7950725c-5dea-4a3b-a06d-4e50b621110b"}]: dispatch 2024-04-21T23:01:04.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:04 smithi055 ceph-mon[31117]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7950725c-5dea-4a3b-a06d-4e50b621110b"}]': finished 2024-04-21T23:01:04.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:04 smithi055 ceph-mon[31117]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T23:01:04.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:04 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:01:04.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:04 smithi055 ceph-mon[31117]: from='client.? 172.21.15.55:0/1362143862' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:01:04.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:04 smithi069 ceph-mon[31816]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:04.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:04 smithi069 ceph-mon[31816]: from='client.? 172.21.15.55:0/119892589' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7950725c-5dea-4a3b-a06d-4e50b621110b"}]: dispatch 2024-04-21T23:01:04.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:04 smithi069 ceph-mon[31816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7950725c-5dea-4a3b-a06d-4e50b621110b"}]: dispatch 2024-04-21T23:01:04.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:04 smithi069 ceph-mon[31816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7950725c-5dea-4a3b-a06d-4e50b621110b"}]': finished 2024-04-21T23:01:04.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:04 smithi069 ceph-mon[31816]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T23:01:04.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:04 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:01:04.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:04 smithi069 ceph-mon[31816]: from='client.? 172.21.15.55:0/1362143862' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:01:06.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:06 smithi044 ceph-mon[27934]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:06.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:06 smithi069 ceph-mon[31816]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:06.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:06 smithi055 ceph-mon[31117]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:08.818 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:08 smithi055 ceph-mon[31117]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:08.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:08 smithi044 ceph-mon[27934]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:08.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:08 smithi069 ceph-mon[31816]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:10.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:10 smithi044 ceph-mon[27934]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:10.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:10 smithi055 ceph-mon[31117]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:10.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:10 smithi069 ceph-mon[31816]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:11.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:11 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T23:01:11.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:11 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:11.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:11 smithi044 ceph-mon[27934]: Deploying daemon osd.1 on smithi055 2024-04-21T23:01:11.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:11 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T23:01:11.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:11 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:11.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:11 smithi069 ceph-mon[31816]: Deploying daemon osd.1 on smithi055 2024-04-21T23:01:11.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:11 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T23:01:11.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:11 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:11.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:11 smithi055 ceph-mon[31117]: Deploying daemon osd.1 on smithi055 2024-04-21T23:01:12.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:12 smithi044 ceph-mon[27934]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:12.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:12 smithi069 ceph-mon[31816]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:12.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:12 smithi055 ceph-mon[31117]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:14.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:14 smithi044 ceph-mon[27934]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:14.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:14 smithi069 ceph-mon[31816]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:14.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:14 smithi055 ceph-mon[31117]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:17.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:16 smithi044 ceph-mon[27934]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:17.780 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:16 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:17.780 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:16 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:17.780 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:16 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:01:17.781 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:16 smithi055 ceph-mon[31117]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:17.781 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:17.781 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:17.782 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:01:17.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:16 smithi069 ceph-mon[31816]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:17.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:16 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:17.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:16 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:17.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:16 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:01:19.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:18 smithi044 ceph-mon[27934]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:19.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:18 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:19.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:18 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:19.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:18 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:19.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:18 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:01:19.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:18 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:19.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:18 smithi055 ceph-mon[31117]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:19.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:18 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:19.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:18 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:19.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:18 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:19.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:18 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:01:19.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:18 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:19.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:18 smithi069 ceph-mon[31816]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:19.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:18 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:19.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:18 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:19.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:18 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:19.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:18 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:01:19.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:18 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:19.598 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:19.598+0000 7fea79ffb700 1 -- 172.21.15.55:0/54935393 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fea7c062670 con 0x7fea6c0a2b90 2024-04-21T23:01:19.598 INFO:teuthology.orchestra.run.smithi055.stdout:Created osd(s) 1 on host 'smithi055' 2024-04-21T23:01:19.600 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:19.600+0000 7fea82fbb700 1 -- 172.21.15.55:0/54935393 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fea6c0a2b90 msgr2=0x7fea6c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:01:19.601 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:19.600+0000 7fea82fbb700 1 --2- 172.21.15.55:0/54935393 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fea6c0a2b90 0x7fea6c0a5050 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fea68005950 tx=0x7fea6800b410 comp rx=0 tx=0).stop 2024-04-21T23:01:19.601 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:19.600+0000 7fea82fbb700 1 -- 172.21.15.55:0/54935393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea7c102290 msgr2=0x7fea7c075fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:01:19.601 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:19.600+0000 7fea82fbb700 1 --2- 172.21.15.55:0/54935393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea7c102290 0x7fea7c075fd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fea7000c010 tx=0x7fea70009510 comp rx=0 tx=0).stop 2024-04-21T23:01:19.601 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:19.600+0000 7fea82fbb700 1 -- 172.21.15.55:0/54935393 shutdown_connections 2024-04-21T23:01:19.601 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:19.600+0000 7fea82fbb700 1 --2- 172.21.15.55:0/54935393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea7c103e30 0x7fea7c074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:19.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:19.600+0000 7fea82fbb700 1 --2- 172.21.15.55:0/54935393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea7c102290 0x7fea7c075fd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:19.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:19.600+0000 7fea82fbb700 1 --2- 172.21.15.55:0/54935393 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fea6c0a2b90 0x7fea6c0a5050 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:19.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:19.600+0000 7fea82fbb700 1 --2- 172.21.15.55:0/54935393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea7c103490 0x7fea7c074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:19.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:19.600+0000 7fea82fbb700 1 -- 172.21.15.55:0/54935393 >> 172.21.15.55:0/54935393 conn(0x7fea7c0fd830 msgr2=0x7fea7c1068b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:01:19.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:19.601+0000 7fea82fbb700 1 -- 172.21.15.55:0/54935393 shutdown_connections 2024-04-21T23:01:19.602 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:01:19.601+0000 7fea82fbb700 1 -- 172.21.15.55:0/54935393 wait complete. 2024-04-21T23:01:20.281 DEBUG:teuthology.orchestra.run.smithi055:osd.1> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.1.service 2024-04-21T23:01:20.283 INFO:tasks.cephadm:Deploying osd.2 on smithi069 with /dev/vg_nvme/lv_4... 2024-04-21T23:01:20.283 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T23:01:20.537 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:01:20.658 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:20 smithi069 ceph-mon[31816]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:20.658 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:20 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:20.658 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:20 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:20.658 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:20 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:01:20.658 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:20 smithi069 ceph-mon[31816]: from='osd.1 [v2:172.21.15.55:6800/1988017680,v1:172.21.15.55:6801/1988017680]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T23:01:20.658 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:20 smithi069 ceph-mon[31816]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T23:01:20.849 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:20 smithi055 ceph-mon[31117]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:20.849 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:20 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:20.849 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:20 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:20.849 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:20 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:01:20.850 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:20 smithi055 ceph-mon[31117]: from='osd.1 [v2:172.21.15.55:6800/1988017680,v1:172.21.15.55:6801/1988017680]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T23:01:20.850 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:20 smithi055 ceph-mon[31117]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T23:01:20.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:20 smithi044 ceph-mon[27934]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:20.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:20 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:20.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:20 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:20.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:20 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:01:20.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:20 smithi044 ceph-mon[27934]: from='osd.1 [v2:172.21.15.55:6800/1988017680,v1:172.21.15.55:6801/1988017680]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T23:01:20.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:20 smithi044 ceph-mon[27934]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T23:01:21.661 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:21 smithi069 ceph-mon[31816]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T23:01:21.661 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:21 smithi069 ceph-mon[31816]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T23:01:21.661 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:21 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:01:21.662 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:21 smithi069 ceph-mon[31816]: from='osd.1 [v2:172.21.15.55:6800/1988017680,v1:172.21.15.55:6801/1988017680]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:01:21.662 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:21 smithi069 ceph-mon[31816]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:01:21.841 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:21 smithi055 ceph-mon[31117]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T23:01:21.841 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:21 smithi055 ceph-mon[31117]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T23:01:21.841 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:21 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:01:21.841 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:21 smithi055 ceph-mon[31117]: from='osd.1 [v2:172.21.15.55:6800/1988017680,v1:172.21.15.55:6801/1988017680]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:01:21.842 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:21 smithi055 ceph-mon[31117]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:01:21.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:21 smithi044 ceph-mon[27934]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T23:01:21.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:21 smithi044 ceph-mon[27934]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T23:01:21.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:21 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:01:21.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:21 smithi044 ceph-mon[27934]: from='osd.1 [v2:172.21.15.55:6800/1988017680,v1:172.21.15.55:6801/1988017680]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:01:21.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:21 smithi044 ceph-mon[27934]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:01:22.255 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T23:01:22.255 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T23:01:22.255 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2024-04-21T23:01:22.255 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2024-04-21T23:01:22.255 INFO:teuthology.orchestra.run.smithi069.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011114 s, 943 MB/s 2024-04-21T23:01:22.255 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2024-04-21T23:01:22.569 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:01:22 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-1[35754]: 2024-04-21T23:01:22.279+0000 7f0cbd0f0700 -1 osd.1 0 waiting for initial osdmap 2024-04-21T23:01:22.569 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:01:22 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-1[35754]: 2024-04-21T23:01:22.283+0000 7f0cb4edb700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:01:22.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:22 smithi069 ceph-mon[31816]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:22.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:22 smithi069 ceph-mon[31816]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-04-21T23:01:22.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:22 smithi069 ceph-mon[31816]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T23:01:22.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:22 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:01:22.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:22 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:01:22.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:22 smithi069 ceph-mon[31816]: from='osd.1 ' entity='osd.1' 2024-04-21T23:01:22.848 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi069:vg_nvme/lv_4 2024-04-21T23:01:22.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:22 smithi044 ceph-mon[27934]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:22.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:22 smithi044 ceph-mon[27934]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-04-21T23:01:22.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:22 smithi044 ceph-mon[27934]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T23:01:22.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:22 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:01:22.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:22 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:01:22.937 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:22 smithi044 ceph-mon[27934]: from='osd.1 ' entity='osd.1' 2024-04-21T23:01:22.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:22 smithi055 ceph-mon[31117]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:22.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:22 smithi055 ceph-mon[31117]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-04-21T23:01:22.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:22 smithi055 ceph-mon[31117]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T23:01:22.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:22 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:01:22.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:22 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:01:22.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:22 smithi055 ceph-mon[31117]: from='osd.1 ' entity='osd.1' 2024-04-21T23:01:23.087 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:01:23.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:23 smithi055 ceph-mon[31117]: purged_snaps scrub starts 2024-04-21T23:01:23.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:23 smithi055 ceph-mon[31117]: purged_snaps scrub ok 2024-04-21T23:01:23.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:01:23.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:23 smithi055 ceph-mon[31117]: osd.1 [v2:172.21.15.55:6800/1988017680,v1:172.21.15.55:6801/1988017680] boot 2024-04-21T23:01:23.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:23 smithi055 ceph-mon[31117]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T23:01:23.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:01:23.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:23.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:23.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:01:23.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:23.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:23.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:01:23.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:23.858 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:23 smithi069 ceph-mon[31816]: purged_snaps scrub starts 2024-04-21T23:01:23.858 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:23 smithi069 ceph-mon[31816]: purged_snaps scrub ok 2024-04-21T23:01:23.858 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:01:23.858 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:23 smithi069 ceph-mon[31816]: osd.1 [v2:172.21.15.55:6800/1988017680,v1:172.21.15.55:6801/1988017680] boot 2024-04-21T23:01:23.858 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:23 smithi069 ceph-mon[31816]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T23:01:23.859 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:01:23.859 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:23.861 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:23.861 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:01:23.861 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:23.861 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:23.861 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:01:23.861 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:23.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:23 smithi044 ceph-mon[27934]: purged_snaps scrub starts 2024-04-21T23:01:23.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:23 smithi044 ceph-mon[27934]: purged_snaps scrub ok 2024-04-21T23:01:23.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:01:23.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:23 smithi044 ceph-mon[27934]: osd.1 [v2:172.21.15.55:6800/1988017680,v1:172.21.15.55:6801/1988017680] boot 2024-04-21T23:01:23.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:23 smithi044 ceph-mon[27934]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T23:01:23.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:01:23.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:23.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:23.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:01:23.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:23.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:23.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:01:23.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:24.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.606+0000 7fa15d146700 1 -- 172.21.15.69:0/897723792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1581034b0 msgr2=0x7fa158103910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:01:24.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.606+0000 7fa15d146700 1 --2- 172.21.15.69:0/897723792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1581034b0 0x7fa158103910 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa140009990 tx=0x7fa140009ca0 comp rx=0 tx=0).stop 2024-04-21T23:01:24.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.606+0000 7fa15d146700 1 -- 172.21.15.69:0/897723792 shutdown_connections 2024-04-21T23:01:24.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.606+0000 7fa15d146700 1 --2- 172.21.15.69:0/897723792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa158103e50 0x7fa15810a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:24.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.606+0000 7fa15d146700 1 --2- 172.21.15.69:0/897723792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa1581022b0 0x7fa1581026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:24.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.606+0000 7fa15d146700 1 --2- 172.21.15.69:0/897723792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1581034b0 0x7fa158103910 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:24.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.606+0000 7fa15d146700 1 -- 172.21.15.69:0/897723792 >> 172.21.15.69:0/897723792 conn(0x7fa1580fd850 msgr2=0x7fa1580ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:01:24.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.606+0000 7fa15d146700 1 -- 172.21.15.69:0/897723792 shutdown_connections 2024-04-21T23:01:24.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.606+0000 7fa15d146700 1 -- 172.21.15.69:0/897723792 wait complete. 2024-04-21T23:01:24.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.607+0000 7fa15d146700 1 Processor -- start 2024-04-21T23:01:24.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.607+0000 7fa15d146700 1 -- start start 2024-04-21T23:01:24.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.607+0000 7fa15d146700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa1581022b0 0x7fa158198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:01:24.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.607+0000 7fa15d146700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1581034b0 0x7fa158198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:01:24.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.607+0000 7fa15d146700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa158103e50 0x7fa15819fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:01:24.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.607+0000 7fa15d146700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa15810dfe0 con 0x7fa158103e50 2024-04-21T23:01:24.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.607+0000 7fa15d146700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa15810de60 con 0x7fa1581022b0 2024-04-21T23:01:24.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.607+0000 7fa15d146700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa15810e160 con 0x7fa1581034b0 2024-04-21T23:01:24.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.607+0000 7fa14e59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1581034b0 0x7fa158198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:01:24.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.607+0000 7fa14e59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1581034b0 0x7fa158198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43924/0 (socket says 172.21.15.69:43924) 2024-04-21T23:01:24.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.607+0000 7fa14e59c700 1 -- 172.21.15.69:0/1112036788 learned_addr learned my addr 172.21.15.69:0/1112036788 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-21T23:01:24.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.607+0000 7fa156d9d700 1 --2- 172.21.15.69:0/1112036788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa1581022b0 0x7fa158198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:01:24.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.607+0000 7fa15759e700 1 --2- 172.21.15.69:0/1112036788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa158103e50 0x7fa15819fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:01:24.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.607+0000 7fa14e59c700 1 -- 172.21.15.69:0/1112036788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa1581022b0 msgr2=0x7fa158198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:01:24.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.607+0000 7fa14e59c700 1 --2- 172.21.15.69:0/1112036788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa1581022b0 0x7fa158198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:24.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.607+0000 7fa14e59c700 1 -- 172.21.15.69:0/1112036788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa158103e50 msgr2=0x7fa15819fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:01:24.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.607+0000 7fa14e59c700 1 --2- 172.21.15.69:0/1112036788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa158103e50 0x7fa15819fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:24.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.607+0000 7fa14e59c700 1 -- 172.21.15.69:0/1112036788 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1581a02c0 con 0x7fa1581034b0 2024-04-21T23:01:24.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.608+0000 7fa15759e700 1 --2- 172.21.15.69:0/1112036788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa158103e50 0x7fa15819fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:01:24.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.608+0000 7fa14e59c700 1 --2- 172.21.15.69:0/1112036788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1581034b0 0x7fa158198ba0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa140005910 tx=0x7fa1400047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:01:24.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.608+0000 7fa154d99700 1 -- 172.21.15.69:0/1112036788 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa140027070 con 0x7fa1581034b0 2024-04-21T23:01:24.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.608+0000 7fa154d99700 1 -- 172.21.15.69:0/1112036788 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa140022470 con 0x7fa1581034b0 2024-04-21T23:01:24.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.608+0000 7fa15d146700 1 -- 172.21.15.69:0/1112036788 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1581a0550 con 0x7fa1581034b0 2024-04-21T23:01:24.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.608+0000 7fa154d99700 1 -- 172.21.15.69:0/1112036788 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14000f6c0 con 0x7fa1581034b0 2024-04-21T23:01:24.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.608+0000 7fa15d146700 1 -- 172.21.15.69:0/1112036788 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1581a0a20 con 0x7fa1581034b0 2024-04-21T23:01:24.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.609+0000 7fa15d146700 1 -- 172.21.15.69:0/1112036788 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa158068f20 con 0x7fa1581034b0 2024-04-21T23:01:24.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.610+0000 7fa154d99700 1 -- 172.21.15.69:0/1112036788 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7fa14002a070 con 0x7fa1581034b0 2024-04-21T23:01:24.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.610+0000 7fa154d99700 1 --2- 172.21.15.69:0/1112036788 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa1440a2b90 0x7fa1440a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:01:24.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.611+0000 7fa154d99700 1 -- 172.21.15.69:0/1112036788 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7fa1400ce8c0 con 0x7fa1581034b0 2024-04-21T23:01:24.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.611+0000 7fa156d9d700 1 --2- 172.21.15.69:0/1112036788 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa1440a2b90 0x7fa1440a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:01:24.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.611+0000 7fa156d9d700 1 --2- 172.21.15.69:0/1112036788 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa1440a2b90 0x7fa1440a5050 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa158103310 tx=0x7fa14800d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:01:24.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.614+0000 7fa154d99700 1 -- 172.21.15.69:0/1112036788 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa140013070 con 0x7fa1581034b0 2024-04-21T23:01:24.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:24.818+0000 7fa15d146700 1 -- 172.21.15.69:0/1112036788 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fa158062670 con 0x7fa1440a2b90 2024-04-21T23:01:24.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:24 smithi069 ceph-mon[31816]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:24.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:24 smithi069 ceph-mon[31816]: Detected new or changed devices on smithi055 2024-04-21T23:01:24.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:24 smithi069 ceph-mon[31816]: Adjusting osd_memory_target on smithi055 to 17093M 2024-04-21T23:01:24.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:24 smithi044 ceph-mon[27934]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:24.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:24 smithi044 ceph-mon[27934]: Detected new or changed devices on smithi055 2024-04-21T23:01:24.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:24 smithi044 ceph-mon[27934]: Adjusting osd_memory_target on smithi055 to 17093M 2024-04-21T23:01:24.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:24 smithi055 ceph-mon[31117]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T23:01:24.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:24 smithi055 ceph-mon[31117]: Detected new or changed devices on smithi055 2024-04-21T23:01:24.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:24 smithi055 ceph-mon[31117]: Adjusting osd_memory_target on smithi055 to 17093M 2024-04-21T23:01:25.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:25 smithi069 ceph-mon[31816]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T23:01:25.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:25 smithi069 ceph-mon[31816]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:01:25.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:25 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:01:25.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:25 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:01:25.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:25 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:25.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:25 smithi044 ceph-mon[27934]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T23:01:25.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:25 smithi044 ceph-mon[27934]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:01:25.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:25 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:01:25.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:25 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:01:25.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:25 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:25.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:25 smithi055 ceph-mon[31117]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T23:01:25.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:25 smithi055 ceph-mon[31117]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:01:25.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:25 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:01:25.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:25 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:01:25.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:25 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:26.629 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:26 smithi069 ceph-mon[31816]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:26.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:26 smithi044 ceph-mon[27934]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:26.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:26 smithi055 ceph-mon[31117]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:27.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:27 smithi044 ceph-mon[27934]: from='client.? 172.21.15.69:0/3711727912' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e99888f4-9a3e-4856-89a8-1a35817893b6"}]: dispatch 2024-04-21T23:01:27.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:27 smithi044 ceph-mon[27934]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e99888f4-9a3e-4856-89a8-1a35817893b6"}]: dispatch 2024-04-21T23:01:27.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:27 smithi044 ceph-mon[27934]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e99888f4-9a3e-4856-89a8-1a35817893b6"}]': finished 2024-04-21T23:01:27.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:27 smithi044 ceph-mon[27934]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T23:01:27.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:01:27.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:27 smithi055 ceph-mon[31117]: from='client.? 172.21.15.69:0/3711727912' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e99888f4-9a3e-4856-89a8-1a35817893b6"}]: dispatch 2024-04-21T23:01:27.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:27 smithi055 ceph-mon[31117]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e99888f4-9a3e-4856-89a8-1a35817893b6"}]: dispatch 2024-04-21T23:01:27.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:27 smithi055 ceph-mon[31117]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e99888f4-9a3e-4856-89a8-1a35817893b6"}]': finished 2024-04-21T23:01:27.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:27 smithi055 ceph-mon[31117]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T23:01:27.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:01:27.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:27 smithi069 ceph-mon[31816]: from='client.? 172.21.15.69:0/3711727912' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e99888f4-9a3e-4856-89a8-1a35817893b6"}]: dispatch 2024-04-21T23:01:27.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:27 smithi069 ceph-mon[31816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e99888f4-9a3e-4856-89a8-1a35817893b6"}]: dispatch 2024-04-21T23:01:27.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:27 smithi069 ceph-mon[31816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e99888f4-9a3e-4856-89a8-1a35817893b6"}]': finished 2024-04-21T23:01:27.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:27 smithi069 ceph-mon[31816]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T23:01:27.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:01:28.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:28 smithi044 ceph-mon[27934]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:28.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:28 smithi044 ceph-mon[27934]: from='client.? 172.21.15.69:0/560657627' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:01:28.967 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:28 smithi055 ceph-mon[31117]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:28.967 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:28 smithi055 ceph-mon[31117]: from='client.? 172.21.15.69:0/560657627' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:01:28.968 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:28 smithi069 ceph-mon[31816]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:28.968 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:28 smithi069 ceph-mon[31816]: from='client.? 172.21.15.69:0/560657627' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:01:30.886 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:30 smithi069 ceph-mon[31816]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:30.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:30 smithi044 ceph-mon[27934]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:30.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:30 smithi055 ceph-mon[31117]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:32.947 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:32 smithi044 ceph-mon[27934]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:32.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:32 smithi069 ceph-mon[31816]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:32.969 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:32 smithi055 ceph-mon[31117]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:34.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:34 smithi044 ceph-mon[27934]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:34.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:34 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T23:01:34.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:34 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:34.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:34 smithi069 ceph-mon[31816]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:34.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:34 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T23:01:34.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:34 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:34.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:34 smithi055 ceph-mon[31117]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:34.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:34 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T23:01:34.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:34 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:35.644 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:35 smithi069 ceph-mon[31816]: Deploying daemon osd.2 on smithi069 2024-04-21T23:01:35.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:35 smithi044 ceph-mon[27934]: Deploying daemon osd.2 on smithi069 2024-04-21T23:01:35.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:35 smithi055 ceph-mon[31117]: Deploying daemon osd.2 on smithi069 2024-04-21T23:01:36.859 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:36 smithi069 ceph-mon[31816]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:36.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:36 smithi044 ceph-mon[27934]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:36.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:36 smithi055 ceph-mon[31117]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:38.882 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:38 smithi069 ceph-mon[31816]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:38.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:38 smithi044 ceph-mon[27934]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:38.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:38 smithi055 ceph-mon[31117]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:40.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:40 smithi069 ceph-mon[31816]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:40.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:40 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:40.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:40 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:40.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:40 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:01:41.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:40 smithi044 ceph-mon[27934]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:41.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:40 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:41.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:40 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:41.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:40 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:01:41.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:40 smithi055 ceph-mon[31117]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:41.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:40 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:41.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:40 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:41.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:40 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:01:42.861 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:42 smithi069 ceph-mon[31816]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:42.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:42.935+0000 7fa154d99700 1 -- 172.21.15.69:0/1112036788 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa158062670 con 0x7fa1440a2b90 2024-04-21T23:01:42.936 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 2 on host 'smithi069' 2024-04-21T23:01:42.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:42.937+0000 7fa15d146700 1 -- 172.21.15.69:0/1112036788 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa1440a2b90 msgr2=0x7fa1440a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:01:42.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:42.937+0000 7fa15d146700 1 --2- 172.21.15.69:0/1112036788 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa1440a2b90 0x7fa1440a5050 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa158103310 tx=0x7fa14800d040 comp rx=0 tx=0).stop 2024-04-21T23:01:42.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:42.937+0000 7fa15d146700 1 -- 172.21.15.69:0/1112036788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1581034b0 msgr2=0x7fa158198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:01:42.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:42.937+0000 7fa15d146700 1 --2- 172.21.15.69:0/1112036788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1581034b0 0x7fa158198ba0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa140005910 tx=0x7fa1400047b0 comp rx=0 tx=0).stop 2024-04-21T23:01:42.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:42.937+0000 7fa15d146700 1 -- 172.21.15.69:0/1112036788 shutdown_connections 2024-04-21T23:01:42.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:42.937+0000 7fa15d146700 1 --2- 172.21.15.69:0/1112036788 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa1440a2b90 0x7fa1440a5050 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:42.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:42.937+0000 7fa15d146700 1 --2- 172.21.15.69:0/1112036788 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa158103e50 0x7fa15819fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:42.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:42.937+0000 7fa15d146700 1 --2- 172.21.15.69:0/1112036788 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa1581022b0 0x7fa158198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:42.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:42.937+0000 7fa15d146700 1 --2- 172.21.15.69:0/1112036788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1581034b0 0x7fa158198ba0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:42.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:42.937+0000 7fa15d146700 1 -- 172.21.15.69:0/1112036788 >> 172.21.15.69:0/1112036788 conn(0x7fa1580fd850 msgr2=0x7fa158106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:01:42.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:42.937+0000 7fa15d146700 1 -- 172.21.15.69:0/1112036788 shutdown_connections 2024-04-21T23:01:42.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:01:42.937+0000 7fa15d146700 1 -- 172.21.15.69:0/1112036788 wait complete. 2024-04-21T23:01:43.007 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:42 smithi044 ceph-mon[27934]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:43.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:42 smithi055 ceph-mon[31117]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:43.509 DEBUG:teuthology.orchestra.run.smithi069:osd.2> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.2.service 2024-04-21T23:01:43.511 INFO:tasks.cephadm:Deploying osd.3 on smithi044 with /dev/vg_nvme/lv_3... 2024-04-21T23:01:43.512 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T23:01:43.714 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:01:43 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-2[37031]: 2024-04-21T23:01:43.616+0000 7ff41aca37c0 -1 osd.2 0 log_to_monitors true 2024-04-21T23:01:43.767 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:01:44.178 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:43 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:44.179 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:43 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:44.179 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:43 smithi044 ceph-mon[27934]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:44.179 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:43 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:44.179 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:43 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:44.179 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:43 smithi044 ceph-mon[27934]: from='osd.2 [v2:172.21.15.69:6800/3485429011,v1:172.21.15.69:6801/3485429011]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T23:01:44.179 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:43 smithi044 ceph-mon[27934]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T23:01:44.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:43 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:44.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:43 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:44.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:43 smithi069 ceph-mon[31816]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:44.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:43 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:44.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:43 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:44.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:43 smithi069 ceph-mon[31816]: from='osd.2 [v2:172.21.15.69:6800/3485429011,v1:172.21.15.69:6801/3485429011]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T23:01:44.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:43 smithi069 ceph-mon[31816]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T23:01:44.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:43 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:44.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:43 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:44.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:43 smithi055 ceph-mon[31117]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:44.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:43 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:44.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:43 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:44.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:43 smithi055 ceph-mon[31117]: from='osd.2 [v2:172.21.15.69:6800/3485429011,v1:172.21.15.69:6801/3485429011]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T23:01:44.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:43 smithi055 ceph-mon[31117]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T23:01:45.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:45 smithi044 ceph-mon[27934]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T23:01:45.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:45 smithi044 ceph-mon[27934]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T23:01:45.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:45 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:01:45.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:45 smithi044 ceph-mon[27934]: from='osd.2 [v2:172.21.15.69:6800/3485429011,v1:172.21.15.69:6801/3485429011]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:01:45.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:45 smithi044 ceph-mon[27934]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:01:45.870 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T23:01:45.870 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T23:01:45.870 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2024-04-21T23:01:45.870 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2024-04-21T23:01:45.870 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110332 s, 950 MB/s 2024-04-21T23:01:45.870 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2024-04-21T23:01:45.878 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:45 smithi069 ceph-mon[31816]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T23:01:45.878 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:45 smithi069 ceph-mon[31816]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T23:01:45.878 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:45 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:01:45.878 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:45 smithi069 ceph-mon[31816]: from='osd.2 [v2:172.21.15.69:6800/3485429011,v1:172.21.15.69:6801/3485429011]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:01:45.878 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:45 smithi069 ceph-mon[31816]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:01:45.878 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:01:45 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-2[37031]: 2024-04-21T23:01:45.615+0000 7ff410308700 -1 osd.2 0 waiting for initial osdmap 2024-04-21T23:01:45.878 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:01:45 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-2[37031]: 2024-04-21T23:01:45.620+0000 7ff40c0fb700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:01:45.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:45 smithi055 ceph-mon[31117]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T23:01:45.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:45 smithi055 ceph-mon[31117]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T23:01:45.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:45 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:01:45.966 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:45 smithi055 ceph-mon[31117]: from='osd.2 [v2:172.21.15.69:6800/3485429011,v1:172.21.15.69:6801/3485429011]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:01:45.966 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:45 smithi055 ceph-mon[31117]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:01:46.547 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi044:vg_nvme/lv_3 2024-04-21T23:01:46.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:46 smithi069 ceph-mon[31816]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:46.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:46 smithi069 ceph-mon[31816]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-21T23:01:46.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:46 smithi069 ceph-mon[31816]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T23:01:46.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:46 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:01:46.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:46 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:01:46.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:46 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:01:46.798 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:01:46.920 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:46 smithi044 ceph-mon[27934]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:46.920 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:46 smithi044 ceph-mon[27934]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-21T23:01:46.920 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:46 smithi044 ceph-mon[27934]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T23:01:46.920 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:46 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:01:46.920 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:46 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:01:46.920 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:46 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:01:46.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:46 smithi055 ceph-mon[31117]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:46.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:46 smithi055 ceph-mon[31117]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-21T23:01:46.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:46 smithi055 ceph-mon[31117]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T23:01:46.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:46 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:01:46.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:46 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:01:46.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:46 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:01:47.863 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:47 smithi044 ceph-mon[27934]: purged_snaps scrub starts 2024-04-21T23:01:47.863 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:47 smithi044 ceph-mon[27934]: purged_snaps scrub ok 2024-04-21T23:01:47.863 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:47 smithi044 ceph-mon[27934]: osd.2 [v2:172.21.15.69:6800/3485429011,v1:172.21.15.69:6801/3485429011] boot 2024-04-21T23:01:47.863 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:47 smithi044 ceph-mon[27934]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T23:01:47.863 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:47 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:01:47.863 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:47 smithi044 ceph-mon[27934]: Detected new or changed devices on smithi069 2024-04-21T23:01:47.864 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:47 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:47.864 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:47 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:47.864 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:47 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:01:47.864 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:47 smithi044 ceph-mon[27934]: Adjusting osd_memory_target on smithi069 to 17088M 2024-04-21T23:01:47.864 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:47 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:47.864 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:47 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:47.864 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:47 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:01:47.864 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:47 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:47.864 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:47 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:01:47.864 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:47 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:01:47.864 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:47 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:47.864 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:47 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:01:47.864 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:47 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:47.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:47 smithi055 ceph-mon[31117]: purged_snaps scrub starts 2024-04-21T23:01:47.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:47 smithi055 ceph-mon[31117]: purged_snaps scrub ok 2024-04-21T23:01:47.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:47 smithi055 ceph-mon[31117]: osd.2 [v2:172.21.15.69:6800/3485429011,v1:172.21.15.69:6801/3485429011] boot 2024-04-21T23:01:47.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:47 smithi055 ceph-mon[31117]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T23:01:47.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:47 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:01:47.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:47 smithi055 ceph-mon[31117]: Detected new or changed devices on smithi069 2024-04-21T23:01:47.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:47 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:47.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:47 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:47.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:47 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:01:47.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:47 smithi055 ceph-mon[31117]: Adjusting osd_memory_target on smithi069 to 17088M 2024-04-21T23:01:47.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:47 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:47.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:47 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:47.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:47 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:01:47.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:47 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:47.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:47 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:01:47.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:47 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:01:47.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:47 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:47.966 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:47 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:01:47.966 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:47 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:47.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:47 smithi069 ceph-mon[31816]: purged_snaps scrub starts 2024-04-21T23:01:47.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:47 smithi069 ceph-mon[31816]: purged_snaps scrub ok 2024-04-21T23:01:47.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:47 smithi069 ceph-mon[31816]: osd.2 [v2:172.21.15.69:6800/3485429011,v1:172.21.15.69:6801/3485429011] boot 2024-04-21T23:01:47.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:47 smithi069 ceph-mon[31816]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T23:01:47.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:47 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:01:47.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:47 smithi069 ceph-mon[31816]: Detected new or changed devices on smithi069 2024-04-21T23:01:47.967 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:47 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:47.967 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:47 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:47.967 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:47 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:01:47.967 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:47 smithi069 ceph-mon[31816]: Adjusting osd_memory_target on smithi069 to 17088M 2024-04-21T23:01:47.967 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:47 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:47.967 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:47 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:47.967 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:47 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:01:47.967 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:47 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:47.967 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:47 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:01:47.967 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:47 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:01:47.967 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:47 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:47.967 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:47 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:01:47.967 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:47 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:48.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.505+0000 7f58c86e1700 1 -- 172.21.15.44:0/2404019910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58c0100550 msgr2=0x7f58c0100970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:01:48.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.505+0000 7f58c86e1700 1 --2- 172.21.15.44:0/2404019910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58c0100550 0x7f58c0100970 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f58bc009990 tx=0x7f58bc009ca0 comp rx=0 tx=0).stop 2024-04-21T23:01:48.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.505+0000 7f58c86e1700 1 -- 172.21.15.44:0/2404019910 shutdown_connections 2024-04-21T23:01:48.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.505+0000 7f58c86e1700 1 --2- 172.21.15.44:0/2404019910 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58c0103ed0 0x7f58c010a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:48.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.505+0000 7f58c86e1700 1 --2- 172.21.15.44:0/2404019910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58c0100550 0x7f58c0100970 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:48.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.505+0000 7f58c86e1700 1 --2- 172.21.15.44:0/2404019910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58c0103530 0x7f58c0103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:48.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.505+0000 7f58c86e1700 1 -- 172.21.15.44:0/2404019910 >> 172.21.15.44:0/2404019910 conn(0x7f58c00fbe90 msgr2=0x7f58c00fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:01:48.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.505+0000 7f58c86e1700 1 -- 172.21.15.44:0/2404019910 shutdown_connections 2024-04-21T23:01:48.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.505+0000 7f58c86e1700 1 -- 172.21.15.44:0/2404019910 wait complete. 2024-04-21T23:01:48.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.506+0000 7f58c86e1700 1 Processor -- start 2024-04-21T23:01:48.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.506+0000 7f58c86e1700 1 -- start start 2024-04-21T23:01:48.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.506+0000 7f58c86e1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58c0100550 0x7f58c0198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:01:48.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.506+0000 7f58c86e1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58c0103530 0x7f58c0198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:01:48.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.506+0000 7f58c86e1700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58c0103ed0 0x7f58c019f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:01:48.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.506+0000 7f58c86e1700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f58c010de60 con 0x7f58c0103ed0 2024-04-21T23:01:48.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.506+0000 7f58c86e1700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f58c010dce0 con 0x7f58c0103530 2024-04-21T23:01:48.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.506+0000 7f58c86e1700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f58c010dfe0 con 0x7f58c0100550 2024-04-21T23:01:48.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.506+0000 7f58c6c7e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58c0103ed0 0x7f58c019f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:01:48.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.506+0000 7f58c6c7e700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58c0103ed0 0x7f58c019f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51358/0 (socket says 172.21.15.44:51358) 2024-04-21T23:01:48.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.506+0000 7f58c6c7e700 1 -- 172.21.15.44:0/745533818 learned_addr learned my addr 172.21.15.44:0/745533818 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:01:48.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.506+0000 7f58c5c7c700 1 --2- 172.21.15.44:0/745533818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58c0103530 0x7f58c0198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:01:48.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.506+0000 7f58c647d700 1 --2- 172.21.15.44:0/745533818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58c0100550 0x7f58c0198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:01:48.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.507+0000 7f58c6c7e700 1 -- 172.21.15.44:0/745533818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58c0100550 msgr2=0x7f58c0198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:01:48.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.507+0000 7f58c6c7e700 1 --2- 172.21.15.44:0/745533818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58c0100550 0x7f58c0198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:48.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.507+0000 7f58c6c7e700 1 -- 172.21.15.44:0/745533818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58c0103530 msgr2=0x7f58c0198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:01:48.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.507+0000 7f58c6c7e700 1 --2- 172.21.15.44:0/745533818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58c0103530 0x7f58c0198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:01:48.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.507+0000 7f58c6c7e700 1 -- 172.21.15.44:0/745533818 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c01a0080 con 0x7f58c0103ed0 2024-04-21T23:01:48.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.507+0000 7f58c6c7e700 1 --2- 172.21.15.44:0/745533818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58c0103ed0 0x7f58c019f9e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f58b8007f10 tx=0x7f58b800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:01:48.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.507+0000 7f58b77fe700 1 -- 172.21.15.44:0/745533818 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b8004bb0 con 0x7f58c0103ed0 2024-04-21T23:01:48.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.507+0000 7f58c86e1700 1 -- 172.21.15.44:0/745533818 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58c01a0370 con 0x7f58c0103ed0 2024-04-21T23:01:48.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.507+0000 7f58b77fe700 1 -- 172.21.15.44:0/745533818 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58b8004d50 con 0x7f58c0103ed0 2024-04-21T23:01:48.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.507+0000 7f58b77fe700 1 -- 172.21.15.44:0/745533818 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b80059f0 con 0x7f58c0103ed0 2024-04-21T23:01:48.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.507+0000 7f58c86e1700 1 -- 172.21.15.44:0/745533818 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58c01a08b0 con 0x7f58c0103ed0 2024-04-21T23:01:48.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.508+0000 7f58c86e1700 1 -- 172.21.15.44:0/745533818 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58c0068f20 con 0x7f58c0103ed0 2024-04-21T23:01:48.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.509+0000 7f58b77fe700 1 -- 172.21.15.44:0/745533818 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f58b801a020 con 0x7f58c0103ed0 2024-04-21T23:01:48.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.510+0000 7f58b77fe700 1 --2- 172.21.15.44:0/745533818 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f58ac0a2b90 0x7f58ac0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:01:48.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.510+0000 7f58c647d700 1 --2- 172.21.15.44:0/745533818 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f58ac0a2b90 0x7f58ac0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:01:48.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.510+0000 7f58b77fe700 1 -- 172.21.15.44:0/745533818 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7f58b80cd880 con 0x7f58c0103ed0 2024-04-21T23:01:48.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.510+0000 7f58c647d700 1 --2- 172.21.15.44:0/745533818 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f58ac0a2b90 0x7f58ac0a5050 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f58bc003890 tx=0x7f58bc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:01:48.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.514+0000 7f58b77fe700 1 -- 172.21.15.44:0/745533818 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f58b8010070 con 0x7f58c0103ed0 2024-04-21T23:01:48.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:48.747+0000 7f58c86e1700 1 -- 172.21.15.44:0/745533818 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f58c0062670 con 0x7f58ac0a2b90 2024-04-21T23:01:48.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:48 smithi044 ceph-mon[27934]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:48.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:48 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T23:01:48.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:48 smithi044 ceph-mon[27934]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T23:01:48.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:48 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:01:48.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:48 smithi069 ceph-mon[31816]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:48.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:48 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T23:01:48.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:48 smithi069 ceph-mon[31816]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T23:01:48.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:48 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:01:48.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:48 smithi055 ceph-mon[31117]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-21T23:01:48.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:48 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T23:01:48.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:48 smithi055 ceph-mon[31117]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T23:01:48.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:48 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:01:49.521 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:01:49 smithi044 sudo[44802]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T23:01:49.521 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:01:49 smithi044 sudo[44802]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:01:49.521 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:01:49 smithi044 sudo[44802]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:01:49.521 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:01:49 smithi044 sudo[44802]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:01:49.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:01:49.657+0000 7f58b77fe700 1 -- 172.21.15.44:0/745533818 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f58b8097c10 con 0x7f58c0103ed0 2024-04-21T23:01:49.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:49 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T23:01:49.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:49 smithi044 ceph-mon[27934]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T23:01:49.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:49 smithi044 ceph-mon[27934]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:01:49.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:49 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:01:49.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:49 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:01:49.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:49 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:49.936 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:01:49 smithi044 sudo[44878]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T23:01:49.936 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:01:49 smithi044 sudo[44878]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:01:49.936 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:01:49 smithi044 sudo[44878]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:01:49.936 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:01:49 smithi044 sudo[44878]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:01:49.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:49 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T23:01:49.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:49 smithi069 ceph-mon[31816]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T23:01:49.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:49 smithi069 ceph-mon[31816]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:01:49.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:49 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:01:49.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:49 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:01:49.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:49 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:49.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:49 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T23:01:49.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:49 smithi055 ceph-mon[31117]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T23:01:49.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:49 smithi055 ceph-mon[31117]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:01:49.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:49 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:01:49.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:49 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:01:49.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:49 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:01:50.465 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:01:50 smithi055 sudo[39894]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T23:01:50.466 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:01:50 smithi055 sudo[39894]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:01:50.466 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:01:50 smithi055 sudo[39894]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:01:50.466 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:01:50 smithi055 sudo[39894]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:01:50.925 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:50 smithi044 ceph-mon[27934]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:01:50.925 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:50 smithi044 ceph-mon[27934]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-21T23:01:50.925 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:50 smithi044 ceph-mon[27934]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T23:01:50.925 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:50 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:50.925 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:50 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:50.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:50 smithi069 ceph-mon[31816]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:01:50.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:50 smithi069 ceph-mon[31816]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-21T23:01:50.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:50 smithi069 ceph-mon[31816]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T23:01:50.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:50 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:50.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:50 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:50.965 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:01:50 smithi069 sudo[40664]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T23:01:50.965 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:01:50 smithi069 sudo[40664]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:01:50.965 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:01:50 smithi069 sudo[40664]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:01:50.965 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:01:50 smithi069 sudo[40664]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:01:50.966 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:01:50 smithi055 sudo[39897]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T23:01:50.966 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:01:50 smithi055 sudo[39897]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:01:50.966 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:01:50 smithi055 sudo[39897]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:01:50.966 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:01:50 smithi055 sudo[39897]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:01:50.966 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:50 smithi055 ceph-mon[31117]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:01:50.966 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:50 smithi055 ceph-mon[31117]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-21T23:01:50.966 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:50 smithi055 ceph-mon[31117]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T23:01:50.967 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:50 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:50.967 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:50 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:51.464 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:01:51 smithi069 sudo[40667]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T23:01:51.464 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:01:51 smithi069 sudo[40667]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:01:51.464 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:01:51 smithi069 sudo[40667]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:01:51.465 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:01:51 smithi069 sudo[40667]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:01:51.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:51 smithi044 sudo[44956]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T23:01:51.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:51 smithi044 sudo[44956]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:01:51.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:51 smithi044 sudo[44956]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:01:51.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:51 smithi044 sudo[44956]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:01:52.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:52 smithi044 sudo[44959]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T23:01:52.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:52 smithi044 sudo[44959]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:01:52.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:52 smithi044 sudo[44959]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:01:52.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:52 smithi044 sudo[44959]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:01:52.545 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:52 smithi069 sudo[40670]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T23:01:52.545 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:52 smithi069 sudo[40670]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:01:52.545 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:52 smithi069 sudo[40670]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:01:52.545 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:52 smithi069 sudo[40670]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:01:52.932 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:52 smithi044 ceph-mon[27934]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:01:52.932 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:52 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:52.932 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:52 smithi044 ceph-mon[27934]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:01:52.932 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:52 smithi044 ceph-mon[27934]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:01:52.932 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:52 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:01:52.932 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:52 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:01:52.932 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:52 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:01:52.932 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:52 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:01:52.932 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:52 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:01:52.933 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:52 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:01:52.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:52 smithi055 ceph-mon[31117]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:01:52.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:52 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:52.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:52 smithi055 ceph-mon[31117]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:01:52.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:52 smithi055 ceph-mon[31117]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:01:52.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:52 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:01:52.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:52 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:01:52.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:52 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:01:52.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:52 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:01:52.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:52 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:01:52.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:52 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:01:52.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:52 smithi069 sudo[40673]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T23:01:52.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:52 smithi069 sudo[40673]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:01:52.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:52 smithi069 sudo[40673]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:01:52.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:52 smithi069 sudo[40673]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:01:52.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:52 smithi069 ceph-mon[31816]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:01:52.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:52 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:01:52.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:52 smithi069 ceph-mon[31816]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:01:52.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:52 smithi069 ceph-mon[31816]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:01:52.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:52 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:01:52.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:52 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:01:52.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:52 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:01:52.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:52 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:01:52.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:52 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:01:52.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:52 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:01:53.457 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:53 smithi055 sudo[39900]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T23:01:53.457 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:53 smithi055 sudo[39900]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:01:53.457 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:53 smithi055 sudo[39900]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:01:53.458 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:53 smithi055 sudo[39900]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:01:53.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:53 smithi055 sudo[39903]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T23:01:53.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:53 smithi055 sudo[39903]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T23:01:53.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:53 smithi055 sudo[39903]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T23:01:53.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:53 smithi055 sudo[39903]: pam_unix(sudo:session): session closed for user root 2024-04-21T23:01:53.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:53 smithi055 ceph-mon[31117]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:01:53.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:53 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3181658768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "508c528e-8e90-408b-97e3-5eded769bf1b"}]: dispatch 2024-04-21T23:01:53.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:53 smithi055 ceph-mon[31117]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:01:53.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:53 smithi055 ceph-mon[31117]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:01:53.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:53 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:01:53.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:53 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:01:53.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:53 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3181658768' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "508c528e-8e90-408b-97e3-5eded769bf1b"}]': finished 2024-04-21T23:01:53.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:53 smithi055 ceph-mon[31117]: osdmap e22: 4 total, 3 up, 4 in 2024-04-21T23:01:53.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:53 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:01:53.716 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:53 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:01:53.716 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:53 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1837698741' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:01:53.716 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:53 smithi055 ceph-mon[31117]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:01:53.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:53 smithi044 ceph-mon[27934]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:01:53.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:53 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3181658768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "508c528e-8e90-408b-97e3-5eded769bf1b"}]: dispatch 2024-04-21T23:01:53.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:53 smithi044 ceph-mon[27934]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:01:53.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:53 smithi044 ceph-mon[27934]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:01:53.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:53 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:01:53.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:53 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:01:53.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:53 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3181658768' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "508c528e-8e90-408b-97e3-5eded769bf1b"}]': finished 2024-04-21T23:01:53.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:53 smithi044 ceph-mon[27934]: osdmap e22: 4 total, 3 up, 4 in 2024-04-21T23:01:53.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:53 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:01:53.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:53 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:01:53.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:53 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1837698741' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:01:53.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:53 smithi044 ceph-mon[27934]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:01:53.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:53 smithi069 ceph-mon[31816]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:01:53.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:53 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3181658768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "508c528e-8e90-408b-97e3-5eded769bf1b"}]: dispatch 2024-04-21T23:01:53.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:53 smithi069 ceph-mon[31816]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:01:53.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:53 smithi069 ceph-mon[31816]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T23:01:53.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:53 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:01:53.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:53 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:01:53.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:53 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3181658768' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "508c528e-8e90-408b-97e3-5eded769bf1b"}]': finished 2024-04-21T23:01:53.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:53 smithi069 ceph-mon[31816]: osdmap e22: 4 total, 3 up, 4 in 2024-04-21T23:01:53.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:53 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:01:53.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:53 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:01:53.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:53 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1837698741' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:01:53.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:53 smithi069 ceph-mon[31816]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T23:01:54.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:54 smithi044 ceph-mon[27934]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:01:54.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:54 smithi055 ceph-mon[31117]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:01:54.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:54 smithi069 ceph-mon[31816]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:01:56.838 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:56 smithi044 ceph-mon[27934]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:01:56.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:56 smithi069 ceph-mon[31816]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:01:56.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:56 smithi055 ceph-mon[31117]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:01:58.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:01:58 smithi044 ceph-mon[27934]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:01:58.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:01:58 smithi069 ceph-mon[31816]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:01:58.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:01:58 smithi055 ceph-mon[31117]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:00.682 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:00 smithi044 ceph-mon[27934]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:00.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:00 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T23:02:00.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:00 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:00.683 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:00 smithi044 ceph-mon[27934]: Deploying daemon osd.3 on smithi044 2024-04-21T23:02:00.875 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:00 smithi055 ceph-mon[31117]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:00.876 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:00 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T23:02:00.876 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:00 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:00.876 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:00 smithi055 ceph-mon[31117]: Deploying daemon osd.3 on smithi044 2024-04-21T23:02:00.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:00 smithi069 ceph-mon[31816]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:00.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:00 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T23:02:00.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:00 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:00.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:00 smithi069 ceph-mon[31816]: Deploying daemon osd.3 on smithi044 2024-04-21T23:02:02.849 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:02 smithi044 ceph-mon[27934]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:02.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:02 smithi055 ceph-mon[31117]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:02.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:02 smithi069 ceph-mon[31816]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:04.825 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:04 smithi044 ceph-mon[27934]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:04.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:04 smithi055 ceph-mon[31117]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:04.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:04 smithi069 ceph-mon[31816]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:06.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:06 smithi044 ceph-mon[27934]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:06.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:06 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:06.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:06 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:06.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:06 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:02:06.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:06 smithi055 ceph-mon[31117]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:06.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:06 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:06.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:06 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:06.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:06 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:02:06.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:06 smithi069 ceph-mon[31816]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:06.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:06 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:06.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:06 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:06.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:06 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:02:08.521 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:08 smithi044 ceph-mon[27934]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:08.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:08 smithi055 ceph-mon[31117]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:08.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:08 smithi069 ceph-mon[31816]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:08.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:02:08.859+0000 7f58b77fe700 1 -- 172.21.15.44:0/745533818 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f58c0062670 con 0x7f58ac0a2b90 2024-04-21T23:02:08.859 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 3 on host 'smithi044' 2024-04-21T23:02:08.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:02:08.861+0000 7f58c86e1700 1 -- 172.21.15.44:0/745533818 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f58ac0a2b90 msgr2=0x7f58ac0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:02:08.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:02:08.861+0000 7f58c86e1700 1 --2- 172.21.15.44:0/745533818 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f58ac0a2b90 0x7f58ac0a5050 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f58bc003890 tx=0x7f58bc005fd0 comp rx=0 tx=0).stop 2024-04-21T23:02:08.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:02:08.861+0000 7f58c86e1700 1 -- 172.21.15.44:0/745533818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58c0103ed0 msgr2=0x7f58c019f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:02:08.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:02:08.861+0000 7f58c86e1700 1 --2- 172.21.15.44:0/745533818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58c0103ed0 0x7f58c019f9e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f58b8007f10 tx=0x7f58b800d3b0 comp rx=0 tx=0).stop 2024-04-21T23:02:08.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:02:08.861+0000 7f58c86e1700 1 -- 172.21.15.44:0/745533818 shutdown_connections 2024-04-21T23:02:08.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:02:08.861+0000 7f58c86e1700 1 --2- 172.21.15.44:0/745533818 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f58ac0a2b90 0x7f58ac0a5050 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:08.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:02:08.861+0000 7f58c86e1700 1 --2- 172.21.15.44:0/745533818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f58c0103ed0 0x7f58c019f9e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:08.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:02:08.861+0000 7f58c86e1700 1 --2- 172.21.15.44:0/745533818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58c0100550 0x7f58c0198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:08.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:02:08.861+0000 7f58c86e1700 1 --2- 172.21.15.44:0/745533818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f58c0103530 0x7f58c0198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:08.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:02:08.861+0000 7f58c86e1700 1 -- 172.21.15.44:0/745533818 >> 172.21.15.44:0/745533818 conn(0x7f58c00fbe90 msgr2=0x7f58c0106850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:02:08.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:02:08.861+0000 7f58c86e1700 1 -- 172.21.15.44:0/745533818 shutdown_connections 2024-04-21T23:02:08.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:02:08.861+0000 7f58c86e1700 1 -- 172.21.15.44:0/745533818 wait complete. 2024-04-21T23:02:09.496 DEBUG:teuthology.orchestra.run.smithi044:osd.3> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.3.service 2024-04-21T23:02:09.499 INFO:tasks.cephadm:Deploying osd.4 on smithi055 with /dev/vg_nvme/lv_3... 2024-04-21T23:02:09.499 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T23:02:09.752 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:02:10.103 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:09 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:10.103 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:09 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:10.103 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:09 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:10.104 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:09 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:10.104 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:09 smithi055 ceph-mon[31117]: from='osd.3 [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T23:02:10.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:09 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:10.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:09 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:10.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:09 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:10.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:09 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:10.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:09 smithi044 ceph-mon[27934]: from='osd.3 [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T23:02:10.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:09 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:10.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:09 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:10.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:09 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:10.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:09 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:10.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:09 smithi069 ceph-mon[31816]: from='osd.3 [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T23:02:11.098 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:10 smithi055 ceph-mon[31117]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:11.099 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:10 smithi055 ceph-mon[31117]: from='osd.3 [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T23:02:11.099 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:10 smithi055 ceph-mon[31117]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T23:02:11.099 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:10 smithi055 ceph-mon[31117]: from='osd.3 [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-21T23:02:11.099 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:10 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:02:11.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:10 smithi044 ceph-mon[27934]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:11.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:10 smithi044 ceph-mon[27934]: from='osd.3 [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T23:02:11.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:10 smithi044 ceph-mon[27934]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T23:02:11.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:10 smithi044 ceph-mon[27934]: from='osd.3 [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-21T23:02:11.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:10 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:02:11.186 INFO:journalctl@ceph.osd.3.smithi044.stdout:Apr 21 23:02:10 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-3[46996]: 2024-04-21T23:02:10.951+0000 7f6004eca700 -1 osd.3 0 waiting for initial osdmap 2024-04-21T23:02:11.186 INFO:journalctl@ceph.osd.3.smithi044.stdout:Apr 21 23:02:10 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-3[46996]: 2024-04-21T23:02:10.958+0000 7f6000cbd700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:02:11.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:10 smithi069 ceph-mon[31816]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:11.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:10 smithi069 ceph-mon[31816]: from='osd.3 [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T23:02:11.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:10 smithi069 ceph-mon[31816]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T23:02:11.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:10 smithi069 ceph-mon[31816]: from='osd.3 [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-21T23:02:11.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:10 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:02:11.705 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T23:02:11.705 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T23:02:11.705 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10+0 records in 2024-04-21T23:02:11.705 INFO:teuthology.orchestra.run.smithi055.stderr:10+0 records out 2024-04-21T23:02:11.705 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114498 s, 916 MB/s 2024-04-21T23:02:11.705 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping successful for: 2024-04-21T23:02:12.030 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:11 smithi055 ceph-mon[31117]: from='osd.3 [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-04-21T23:02:12.030 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:11 smithi055 ceph-mon[31117]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T23:02:12.030 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:11 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:02:12.031 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:11 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:02:12.031 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:11 smithi055 ceph-mon[31117]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:12.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:11 smithi044 ceph-mon[27934]: from='osd.3 [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-04-21T23:02:12.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:11 smithi044 ceph-mon[27934]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T23:02:12.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:11 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:02:12.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:11 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:02:12.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:11 smithi044 ceph-mon[27934]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:12.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:11 smithi069 ceph-mon[31816]: from='osd.3 [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-04-21T23:02:12.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:11 smithi069 ceph-mon[31816]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T23:02:12.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:11 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:02:12.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:11 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:02:12.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:11 smithi069 ceph-mon[31816]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-21T23:02:12.322 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi055:vg_nvme/lv_3 2024-04-21T23:02:12.578 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:02:13.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:12 smithi069 ceph-mon[31816]: purged_snaps scrub starts 2024-04-21T23:02:13.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:12 smithi069 ceph-mon[31816]: purged_snaps scrub ok 2024-04-21T23:02:13.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:12 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:02:13.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:12 smithi069 ceph-mon[31816]: osd.3 [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227] boot 2024-04-21T23:02:13.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:12 smithi069 ceph-mon[31816]: osdmap e25: 4 total, 4 up, 4 in 2024-04-21T23:02:13.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:12 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:02:13.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:12 smithi069 ceph-mon[31816]: Detected new or changed devices on smithi044 2024-04-21T23:02:13.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:12 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:13.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:12 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:13.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:12 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:02:13.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:12 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:02:13.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:12 smithi069 ceph-mon[31816]: Adjusting osd_memory_target on smithi044 to 8546M 2024-04-21T23:02:13.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:12 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:13.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:12 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:13.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:12 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:02:13.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:12 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:13.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:12 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:02:13.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:12 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:13.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:12 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:02:13.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:12 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:13.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:12 smithi055 ceph-mon[31117]: purged_snaps scrub starts 2024-04-21T23:02:13.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:12 smithi055 ceph-mon[31117]: purged_snaps scrub ok 2024-04-21T23:02:13.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:12 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:02:13.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:12 smithi055 ceph-mon[31117]: osd.3 [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227] boot 2024-04-21T23:02:13.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:12 smithi055 ceph-mon[31117]: osdmap e25: 4 total, 4 up, 4 in 2024-04-21T23:02:13.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:12 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:02:13.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:12 smithi055 ceph-mon[31117]: Detected new or changed devices on smithi044 2024-04-21T23:02:13.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:12 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:13.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:12 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:13.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:12 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:02:13.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:12 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:02:13.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:12 smithi055 ceph-mon[31117]: Adjusting osd_memory_target on smithi044 to 8546M 2024-04-21T23:02:13.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:12 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:13.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:12 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:13.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:12 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:02:13.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:12 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:13.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:12 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:02:13.218 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:12 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:13.218 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:12 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:02:13.218 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:12 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:13.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:12 smithi044 ceph-mon[27934]: purged_snaps scrub starts 2024-04-21T23:02:13.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:12 smithi044 ceph-mon[27934]: purged_snaps scrub ok 2024-04-21T23:02:13.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:02:13.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:12 smithi044 ceph-mon[27934]: osd.3 [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227] boot 2024-04-21T23:02:13.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:12 smithi044 ceph-mon[27934]: osdmap e25: 4 total, 4 up, 4 in 2024-04-21T23:02:13.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:02:13.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:12 smithi044 ceph-mon[27934]: Detected new or changed devices on smithi044 2024-04-21T23:02:13.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:13.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:13.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:02:13.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:02:13.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:12 smithi044 ceph-mon[27934]: Adjusting osd_memory_target on smithi044 to 8546M 2024-04-21T23:02:13.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:13.437 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:13.437 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:02:13.437 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:13.437 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:02:13.437 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:13.437 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:02:13.437 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:14.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.148+0000 7f8159b1c700 1 -- 172.21.15.55:0/1481467787 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8154107f70 msgr2=0x7f815410a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:02:14.148 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.148+0000 7f8159b1c700 1 --2- 172.21.15.55:0/1481467787 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8154107f70 0x7f815410a3e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f814c00b0a0 tx=0x7f814c00b3b0 comp rx=0 tx=0).stop 2024-04-21T23:02:14.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.148+0000 7f8159b1c700 1 -- 172.21.15.55:0/1481467787 shutdown_connections 2024-04-21T23:02:14.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.148+0000 7f8159b1c700 1 --2- 172.21.15.55:0/1481467787 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8154107f70 0x7f815410a3e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:14.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.148+0000 7f8159b1c700 1 --2- 172.21.15.55:0/1481467787 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8154102d10 0x7f8154105100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:14.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.148+0000 7f8159b1c700 1 --2- 172.21.15.55:0/1481467787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8154105640 0x7f8154107a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:14.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.148+0000 7f8159b1c700 1 -- 172.21.15.55:0/1481467787 >> 172.21.15.55:0/1481467787 conn(0x7f81540fc720 msgr2=0x7f81540feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:02:14.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.148+0000 7f8159b1c700 1 -- 172.21.15.55:0/1481467787 shutdown_connections 2024-04-21T23:02:14.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.148+0000 7f8159b1c700 1 -- 172.21.15.55:0/1481467787 wait complete. 2024-04-21T23:02:14.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.149+0000 7f8159b1c700 1 Processor -- start 2024-04-21T23:02:14.149 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.149+0000 7f8159b1c700 1 -- start start 2024-04-21T23:02:14.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.149+0000 7f8159b1c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8154102d10 0x7f81541984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:02:14.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.149+0000 7f8159b1c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8154105640 0x7f8154198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:02:14.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.149+0000 7f8159b1c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8154107f70 0x7f815419fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:02:14.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.149+0000 7f8159b1c700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f815410df00 con 0x7f8154107f70 2024-04-21T23:02:14.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.149+0000 7f8159b1c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f815410dd80 con 0x7f8154105640 2024-04-21T23:02:14.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.149+0000 7f8159b1c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f815410e080 con 0x7f8154102d10 2024-04-21T23:02:14.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.149+0000 7f8152ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8154105640 0x7f8154198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:02:14.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.149+0000 7f8152ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8154105640 0x7f8154198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52986/0 (socket says 172.21.15.55:52986) 2024-04-21T23:02:14.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.149+0000 7f8152ffd700 1 -- 172.21.15.55:0/3147651146 learned_addr learned my addr 172.21.15.55:0/3147651146 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-04-21T23:02:14.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.149+0000 7f8153fff700 1 --2- 172.21.15.55:0/3147651146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8154107f70 0x7f815419fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:02:14.152 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.149+0000 7f81537fe700 1 --2- 172.21.15.55:0/3147651146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8154102d10 0x7f81541984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:02:14.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.150+0000 7f8152ffd700 1 -- 172.21.15.55:0/3147651146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8154102d10 msgr2=0x7f81541984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:02:14.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.150+0000 7f8152ffd700 1 --2- 172.21.15.55:0/3147651146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8154102d10 0x7f81541984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:14.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.150+0000 7f8152ffd700 1 -- 172.21.15.55:0/3147651146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8154107f70 msgr2=0x7f815419fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:02:14.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.150+0000 7f8152ffd700 1 --2- 172.21.15.55:0/3147651146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8154107f70 0x7f815419fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:14.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.150+0000 7f8152ffd700 1 -- 172.21.15.55:0/3147651146 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81541a0150 con 0x7f8154105640 2024-04-21T23:02:14.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.150+0000 7f8153fff700 1 --2- 172.21.15.55:0/3147651146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8154107f70 0x7f815419fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:02:14.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.150+0000 7f8152ffd700 1 --2- 172.21.15.55:0/3147651146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8154105640 0x7f8154198a30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f814800b530 tx=0x7f814800b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:02:14.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.150+0000 7f8150ff9700 1 -- 172.21.15.55:0/3147651146 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8148015020 con 0x7f8154105640 2024-04-21T23:02:14.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.150+0000 7f8159b1c700 1 -- 172.21.15.55:0/3147651146 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81541a0440 con 0x7f8154105640 2024-04-21T23:02:14.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.150+0000 7f8150ff9700 1 -- 172.21.15.55:0/3147651146 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8148011d10 con 0x7f8154105640 2024-04-21T23:02:14.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.150+0000 7f8150ff9700 1 -- 172.21.15.55:0/3147651146 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f814800f2e0 con 0x7f8154105640 2024-04-21T23:02:14.153 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.150+0000 7f8159b1c700 1 -- 172.21.15.55:0/3147651146 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81541a0980 con 0x7f8154105640 2024-04-21T23:02:14.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.151+0000 7f8159b1c700 1 -- 172.21.15.55:0/3147651146 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8154068f20 con 0x7f8154105640 2024-04-21T23:02:14.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.152+0000 7f8150ff9700 1 -- 172.21.15.55:0/3147651146 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8148018020 con 0x7f8154105640 2024-04-21T23:02:14.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.153+0000 7f8150ff9700 1 --2- 172.21.15.55:0/3147651146 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f813c0a2c20 0x7f813c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:02:14.154 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.153+0000 7f8150ff9700 1 -- 172.21.15.55:0/3147651146 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7f81480cd540 con 0x7f8154105640 2024-04-21T23:02:14.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.153+0000 7f81537fe700 1 --2- 172.21.15.55:0/3147651146 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f813c0a2c20 0x7f813c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:02:14.155 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.153+0000 7f81537fe700 1 --2- 172.21.15.55:0/3147651146 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f813c0a2c20 0x7f813c0a50e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8144009c60 tx=0x7f8144009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:02:14.157 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.158+0000 7f8150ff9700 1 -- 172.21.15.55:0/3147651146 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f814801f020 con 0x7f8154105640 2024-04-21T23:02:14.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:13 smithi044 ceph-mon[27934]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T23:02:14.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:13 smithi044 ceph-mon[27934]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:02:14.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:13 smithi055 ceph-mon[31117]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T23:02:14.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:13 smithi055 ceph-mon[31117]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:02:14.377 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:14.376+0000 7f8159b1c700 1 -- 172.21.15.55:0/3147651146 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f8154062670 con 0x7f813c0a2c20 2024-04-21T23:02:14.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:13 smithi069 ceph-mon[31816]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T23:02:14.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:13 smithi069 ceph-mon[31816]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:02:15.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:14 smithi044 ceph-mon[27934]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T23:02:15.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:14 smithi044 ceph-mon[27934]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:02:15.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:14 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:02:15.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:14 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:02:15.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:14 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:15.377 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:14 smithi055 ceph-mon[31117]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T23:02:15.378 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:14 smithi055 ceph-mon[31117]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:02:15.378 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:14 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:02:15.378 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:14 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:02:15.378 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:14 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:15.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:14 smithi069 ceph-mon[31816]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T23:02:15.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:14 smithi069 ceph-mon[31816]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:02:15.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:14 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:02:15.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:14 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:02:15.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:14 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:16.106 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:15 smithi055 ceph-mon[31117]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T23:02:16.106 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:15 smithi055 ceph-mon[31117]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:02:16.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:15 smithi044 ceph-mon[27934]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T23:02:16.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:15 smithi044 ceph-mon[27934]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:02:16.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:15 smithi069 ceph-mon[31816]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T23:02:16.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:15 smithi069 ceph-mon[31816]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:02:17.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:16 smithi044 ceph-mon[27934]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T23:02:17.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:16 smithi055 ceph-mon[31117]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T23:02:17.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:16 smithi069 ceph-mon[31816]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T23:02:18.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:17 smithi044 ceph-mon[27934]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:02:18.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:17 smithi044 ceph-mon[27934]: from='client.? 172.21.15.55:0/3246485371' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44a9925a-84ae-474a-8d5c-bec46baaa91c"}]: dispatch 2024-04-21T23:02:18.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:17 smithi044 ceph-mon[27934]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44a9925a-84ae-474a-8d5c-bec46baaa91c"}]: dispatch 2024-04-21T23:02:18.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:17 smithi044 ceph-mon[27934]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44a9925a-84ae-474a-8d5c-bec46baaa91c"}]': finished 2024-04-21T23:02:18.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:17 smithi044 ceph-mon[27934]: osdmap e30: 5 total, 4 up, 5 in 2024-04-21T23:02:18.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:17 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:02:18.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:17 smithi044 ceph-mon[27934]: from='client.? 172.21.15.55:0/72114732' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:02:18.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:17 smithi055 ceph-mon[31117]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:02:18.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:17 smithi055 ceph-mon[31117]: from='client.? 172.21.15.55:0/3246485371' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44a9925a-84ae-474a-8d5c-bec46baaa91c"}]: dispatch 2024-04-21T23:02:18.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:17 smithi055 ceph-mon[31117]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44a9925a-84ae-474a-8d5c-bec46baaa91c"}]: dispatch 2024-04-21T23:02:18.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:17 smithi055 ceph-mon[31117]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44a9925a-84ae-474a-8d5c-bec46baaa91c"}]': finished 2024-04-21T23:02:18.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:17 smithi055 ceph-mon[31117]: osdmap e30: 5 total, 4 up, 5 in 2024-04-21T23:02:18.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:17 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:02:18.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:17 smithi055 ceph-mon[31117]: from='client.? 172.21.15.55:0/72114732' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:02:18.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:17 smithi069 ceph-mon[31816]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:02:18.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:17 smithi069 ceph-mon[31816]: from='client.? 172.21.15.55:0/3246485371' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44a9925a-84ae-474a-8d5c-bec46baaa91c"}]: dispatch 2024-04-21T23:02:18.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:17 smithi069 ceph-mon[31816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44a9925a-84ae-474a-8d5c-bec46baaa91c"}]: dispatch 2024-04-21T23:02:18.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:17 smithi069 ceph-mon[31816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44a9925a-84ae-474a-8d5c-bec46baaa91c"}]': finished 2024-04-21T23:02:18.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:17 smithi069 ceph-mon[31816]: osdmap e30: 5 total, 4 up, 5 in 2024-04-21T23:02:18.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:17 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:02:18.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:17 smithi069 ceph-mon[31816]: from='client.? 172.21.15.55:0/72114732' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:02:20.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:20 smithi044 ceph-mon[27934]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-21T23:02:20.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:20 smithi055 ceph-mon[31117]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-21T23:02:20.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:20 smithi069 ceph-mon[31816]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-21T23:02:22.336 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:22 smithi055 ceph-mon[31117]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T23:02:22.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:22 smithi044 ceph-mon[27934]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T23:02:22.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:22 smithi069 ceph-mon[31816]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T23:02:24.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:24 smithi055 ceph-mon[31117]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T23:02:24.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:24 smithi044 ceph-mon[27934]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T23:02:24.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:24 smithi069 ceph-mon[31816]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T23:02:25.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:25 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T23:02:25.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:25 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:25.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:25 smithi044 ceph-mon[27934]: Deploying daemon osd.4 on smithi055 2024-04-21T23:02:25.337 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:25 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T23:02:25.337 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:25 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:25.337 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:25 smithi055 ceph-mon[31117]: Deploying daemon osd.4 on smithi055 2024-04-21T23:02:25.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:25 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T23:02:25.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:25 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:25.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:25 smithi069 ceph-mon[31816]: Deploying daemon osd.4 on smithi055 2024-04-21T23:02:26.340 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:26 smithi055 ceph-mon[31117]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-21T23:02:26.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:26 smithi044 ceph-mon[27934]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-21T23:02:26.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:26 smithi069 ceph-mon[31816]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-21T23:02:28.388 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:28 smithi055 ceph-mon[31117]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:02:28.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:28 smithi044 ceph-mon[27934]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:02:28.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:28 smithi069 ceph-mon[31816]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:02:30.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:30 smithi044 ceph-mon[27934]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-21T23:02:30.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:30 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:30.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:30 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:30.456 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:30 smithi055 ceph-mon[31117]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-21T23:02:30.456 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:30 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:30.456 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:30 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:30.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:30 smithi069 ceph-mon[31816]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-21T23:02:30.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:30 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:30.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:30 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:31.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:31 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:02:31.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:31 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:02:31.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:31 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:02:32.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:32 smithi044 ceph-mon[27934]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:02:32.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:32 smithi055 ceph-mon[31117]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:02:32.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:32 smithi069 ceph-mon[31816]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:02:33.844 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:33.844+0000 7f8150ff9700 1 -- 172.21.15.55:0/3147651146 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8154062670 con 0x7f813c0a2c20 2024-04-21T23:02:33.845 INFO:teuthology.orchestra.run.smithi055.stdout:Created osd(s) 4 on host 'smithi055' 2024-04-21T23:02:33.846 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:33.846+0000 7f8159b1c700 1 -- 172.21.15.55:0/3147651146 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f813c0a2c20 msgr2=0x7f813c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:02:33.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:33.846+0000 7f8159b1c700 1 --2- 172.21.15.55:0/3147651146 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f813c0a2c20 0x7f813c0a50e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8144009c60 tx=0x7f8144009260 comp rx=0 tx=0).stop 2024-04-21T23:02:33.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:33.846+0000 7f8159b1c700 1 -- 172.21.15.55:0/3147651146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8154105640 msgr2=0x7f8154198a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:02:33.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:33.846+0000 7f8159b1c700 1 --2- 172.21.15.55:0/3147651146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8154105640 0x7f8154198a30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f814800b530 tx=0x7f814800b840 comp rx=0 tx=0).stop 2024-04-21T23:02:33.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:33.846+0000 7f8159b1c700 1 -- 172.21.15.55:0/3147651146 shutdown_connections 2024-04-21T23:02:33.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:33.846+0000 7f8159b1c700 1 --2- 172.21.15.55:0/3147651146 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f813c0a2c20 0x7f813c0a50e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:33.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:33.846+0000 7f8159b1c700 1 --2- 172.21.15.55:0/3147651146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8154107f70 0x7f815419fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:33.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:33.846+0000 7f8159b1c700 1 --2- 172.21.15.55:0/3147651146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8154102d10 0x7f81541984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:33.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:33.846+0000 7f8159b1c700 1 --2- 172.21.15.55:0/3147651146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8154105640 0x7f8154198a30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:33.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:33.846+0000 7f8159b1c700 1 -- 172.21.15.55:0/3147651146 >> 172.21.15.55:0/3147651146 conn(0x7f81540fc720 msgr2=0x7f8154103670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:02:33.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:33.847+0000 7f8159b1c700 1 -- 172.21.15.55:0/3147651146 shutdown_connections 2024-04-21T23:02:33.847 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:02:33.847+0000 7f8159b1c700 1 -- 172.21.15.55:0/3147651146 wait complete. 2024-04-21T23:02:34.116 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:33 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:34.116 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:33 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:34.117 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:33 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:34.117 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:33 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:02:34.117 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:33 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:34.117 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:33 smithi055 ceph-mon[31117]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:02:34.117 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:33 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:34.117 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:33 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:34.117 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:33 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:02:34.117 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:33 smithi055 ceph-mon[31117]: from='osd.4 [v2:172.21.15.55:6808/988958724,v1:172.21.15.55:6809/988958724]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T23:02:34.117 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:33 smithi055 ceph-mon[31117]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T23:02:34.267 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:33 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:34.267 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:33 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:34.267 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:33 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:34.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:33 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:02:34.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:33 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:34.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:33 smithi069 ceph-mon[31816]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:02:34.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:33 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:34.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:33 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:34.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:33 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:02:34.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:33 smithi069 ceph-mon[31816]: from='osd.4 [v2:172.21.15.55:6808/988958724,v1:172.21.15.55:6809/988958724]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T23:02:34.268 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:33 smithi069 ceph-mon[31816]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T23:02:34.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:33 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:34.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:33 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:34.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:33 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:34.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:33 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:02:34.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:33 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:34.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:33 smithi044 ceph-mon[27934]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:02:34.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:33 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:34.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:33 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:34.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:33 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:02:34.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:33 smithi044 ceph-mon[27934]: from='osd.4 [v2:172.21.15.55:6808/988958724,v1:172.21.15.55:6809/988958724]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T23:02:34.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:33 smithi044 ceph-mon[27934]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T23:02:34.461 DEBUG:teuthology.orchestra.run.smithi055:osd.4> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.4.service 2024-04-21T23:02:34.463 INFO:tasks.cephadm:Deploying osd.5 on smithi069 with /dev/vg_nvme/lv_3... 2024-04-21T23:02:34.463 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T23:02:34.708 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:02:35.865 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:35 smithi055 ceph-mon[31117]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T23:02:35.866 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:35 smithi055 ceph-mon[31117]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T23:02:35.866 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:35 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:02:35.866 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:35 smithi055 ceph-mon[31117]: from='osd.4 [v2:172.21.15.55:6808/988958724,v1:172.21.15.55:6809/988958724]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:02:35.866 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:35 smithi055 ceph-mon[31117]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:02:36.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:35 smithi069 ceph-mon[31816]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T23:02:36.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:35 smithi069 ceph-mon[31816]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T23:02:36.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:35 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:02:36.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:35 smithi069 ceph-mon[31816]: from='osd.4 [v2:172.21.15.55:6808/988958724,v1:172.21.15.55:6809/988958724]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:02:36.095 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:35 smithi069 ceph-mon[31816]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:02:36.124 INFO:journalctl@ceph.osd.4.smithi055.stdout:Apr 21 23:02:35 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-4[42691]: 2024-04-21T23:02:35.864+0000 7fd34ba32700 -1 osd.4 0 waiting for initial osdmap 2024-04-21T23:02:36.125 INFO:journalctl@ceph.osd.4.smithi055.stdout:Apr 21 23:02:35 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-4[42691]: 2024-04-21T23:02:35.872+0000 7fd347825700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:02:36.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:35 smithi044 ceph-mon[27934]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T23:02:36.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:35 smithi044 ceph-mon[27934]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T23:02:36.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:35 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:02:36.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:35 smithi044 ceph-mon[27934]: from='osd.4 [v2:172.21.15.55:6808/988958724,v1:172.21.15.55:6809/988958724]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:02:36.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:35 smithi044 ceph-mon[27934]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:02:36.675 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T23:02:36.675 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T23:02:36.675 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2024-04-21T23:02:36.675 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2024-04-21T23:02:36.675 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0108975 s, 962 MB/s 2024-04-21T23:02:36.676 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2024-04-21T23:02:37.087 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:36 smithi055 ceph-mon[31117]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:02:37.087 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:36 smithi055 ceph-mon[31117]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-04-21T23:02:37.087 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:36 smithi055 ceph-mon[31117]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T23:02:37.087 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:36 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:02:37.087 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:36 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:02:37.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:36 smithi044 ceph-mon[27934]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:02:37.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:36 smithi044 ceph-mon[27934]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-04-21T23:02:37.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:36 smithi044 ceph-mon[27934]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T23:02:37.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:36 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:02:37.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:36 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:02:37.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:36 smithi069 ceph-mon[31816]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-21T23:02:37.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:36 smithi069 ceph-mon[31816]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-04-21T23:02:37.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:36 smithi069 ceph-mon[31816]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T23:02:37.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:36 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:02:37.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:36 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:02:37.257 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi069:vg_nvme/lv_3 2024-04-21T23:02:37.502 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:02:38.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:37 smithi044 ceph-mon[27934]: purged_snaps scrub starts 2024-04-21T23:02:38.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:37 smithi044 ceph-mon[27934]: purged_snaps scrub ok 2024-04-21T23:02:38.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:37 smithi044 ceph-mon[27934]: osd.4 [v2:172.21.15.55:6808/988958724,v1:172.21.15.55:6809/988958724] boot 2024-04-21T23:02:38.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:37 smithi044 ceph-mon[27934]: osdmap e33: 5 total, 5 up, 5 in 2024-04-21T23:02:38.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:37 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:02:38.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:37 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:38.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:37 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:38.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:37 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:02:38.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:37 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:02:38.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:37 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:38.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:37 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:38.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:37 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:02:38.187 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:37 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:38.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:37 smithi055 ceph-mon[31117]: purged_snaps scrub starts 2024-04-21T23:02:38.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:37 smithi055 ceph-mon[31117]: purged_snaps scrub ok 2024-04-21T23:02:38.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:37 smithi055 ceph-mon[31117]: osd.4 [v2:172.21.15.55:6808/988958724,v1:172.21.15.55:6809/988958724] boot 2024-04-21T23:02:38.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:37 smithi055 ceph-mon[31117]: osdmap e33: 5 total, 5 up, 5 in 2024-04-21T23:02:38.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:37 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:02:38.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:37 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:38.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:37 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:38.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:37 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:02:38.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:37 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:02:38.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:37 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:38.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:37 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:38.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:37 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:02:38.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:37 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:38.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:37 smithi069 ceph-mon[31816]: purged_snaps scrub starts 2024-04-21T23:02:38.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:37 smithi069 ceph-mon[31816]: purged_snaps scrub ok 2024-04-21T23:02:38.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:37 smithi069 ceph-mon[31816]: osd.4 [v2:172.21.15.55:6808/988958724,v1:172.21.15.55:6809/988958724] boot 2024-04-21T23:02:38.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:37 smithi069 ceph-mon[31816]: osdmap e33: 5 total, 5 up, 5 in 2024-04-21T23:02:38.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:37 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:02:38.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:37 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:38.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:37 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:38.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:37 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:02:38.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:37 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:02:38.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:37 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:38.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:37 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:38.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:37 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:02:38.217 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:37 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:38.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:38 smithi069 ceph-mon[31816]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:38.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:38 smithi069 ceph-mon[31816]: Detected new or changed devices on smithi055 2024-04-21T23:02:38.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:38 smithi069 ceph-mon[31816]: Adjusting osd_memory_target on smithi055 to 8546M 2024-04-21T23:02:38.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:38 smithi069 ceph-mon[31816]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T23:02:39.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.159+0000 7fb471e24700 1 -- 172.21.15.69:0/341285502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb46c101180 msgr2=0x7fb46c1015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:02:39.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.159+0000 7fb471e24700 1 --2- 172.21.15.69:0/341285502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb46c101180 0x7fb46c1015a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb45c00b0a0 tx=0x7fb45c00b3b0 comp rx=0 tx=0).stop 2024-04-21T23:02:39.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.159+0000 7fb471e24700 1 -- 172.21.15.69:0/341285502 shutdown_connections 2024-04-21T23:02:39.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.159+0000 7fb471e24700 1 --2- 172.21.15.69:0/341285502 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb46c102d20 0x7fb46c1095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:39.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.159+0000 7fb471e24700 1 --2- 172.21.15.69:0/341285502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb46c101180 0x7fb46c1015a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:39.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.159+0000 7fb471e24700 1 --2- 172.21.15.69:0/341285502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb46c102380 0x7fb46c1027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:39.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.159+0000 7fb471e24700 1 -- 172.21.15.69:0/341285502 >> 172.21.15.69:0/341285502 conn(0x7fb46c0fc720 msgr2=0x7fb46c0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:02:39.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.159+0000 7fb471e24700 1 -- 172.21.15.69:0/341285502 shutdown_connections 2024-04-21T23:02:39.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.159+0000 7fb471e24700 1 -- 172.21.15.69:0/341285502 wait complete. 2024-04-21T23:02:39.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.160+0000 7fb471e24700 1 Processor -- start 2024-04-21T23:02:39.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.160+0000 7fb471e24700 1 -- start start 2024-04-21T23:02:39.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.160+0000 7fb471e24700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb46c101180 0x7fb46c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:02:39.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.160+0000 7fb471e24700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb46c102380 0x7fb46c198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:02:39.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.160+0000 7fb471e24700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb46c102d20 0x7fb46c19fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:02:39.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.160+0000 7fb471e24700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb46c10ce20 con 0x7fb46c102d20 2024-04-21T23:02:39.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.160+0000 7fb471e24700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb46c10cca0 con 0x7fb46c101180 2024-04-21T23:02:39.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.160+0000 7fb471e24700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb46c10cfa0 con 0x7fb46c102380 2024-04-21T23:02:39.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.160+0000 7fb462ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb46c102380 0x7fb46c198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:02:39.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.160+0000 7fb462ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb46c102380 0x7fb46c198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50794/0 (socket says 172.21.15.69:50794) 2024-04-21T23:02:39.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.160+0000 7fb462ffd700 1 -- 172.21.15.69:0/355562583 learned_addr learned my addr 172.21.15.69:0/355562583 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-21T23:02:39.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.160+0000 7fb46bfff700 1 --2- 172.21.15.69:0/355562583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb46c102d20 0x7fb46c19fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:02:39.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.160+0000 7fb46b7fe700 1 --2- 172.21.15.69:0/355562583 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb46c101180 0x7fb46c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:02:39.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.160+0000 7fb462ffd700 1 -- 172.21.15.69:0/355562583 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb46c101180 msgr2=0x7fb46c198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:02:39.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.160+0000 7fb462ffd700 1 --2- 172.21.15.69:0/355562583 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb46c101180 0x7fb46c198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:39.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.160+0000 7fb462ffd700 1 -- 172.21.15.69:0/355562583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb46c102d20 msgr2=0x7fb46c19fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:02:39.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.160+0000 7fb462ffd700 1 --2- 172.21.15.69:0/355562583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb46c102d20 0x7fb46c19fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:39.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.160+0000 7fb462ffd700 1 -- 172.21.15.69:0/355562583 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb46c1a01c0 con 0x7fb46c102380 2024-04-21T23:02:39.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.160+0000 7fb462ffd700 1 --2- 172.21.15.69:0/355562583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb46c102380 0x7fb46c198aa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb45400d980 tx=0x7fb45400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:02:39.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.161+0000 7fb4697fa700 1 -- 172.21.15.69:0/355562583 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb454016070 con 0x7fb46c102380 2024-04-21T23:02:39.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.161+0000 7fb471e24700 1 -- 172.21.15.69:0/355562583 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb46c1a04b0 con 0x7fb46c102380 2024-04-21T23:02:39.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.161+0000 7fb4697fa700 1 -- 172.21.15.69:0/355562583 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb454009db0 con 0x7fb46c102380 2024-04-21T23:02:39.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.161+0000 7fb471e24700 1 -- 172.21.15.69:0/355562583 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb46c1a0ac0 con 0x7fb46c102380 2024-04-21T23:02:39.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.161+0000 7fb4697fa700 1 -- 172.21.15.69:0/355562583 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45400f3c0 con 0x7fb46c102380 2024-04-21T23:02:39.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.162+0000 7fb471e24700 1 -- 172.21.15.69:0/355562583 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb46c068f20 con 0x7fb46c102380 2024-04-21T23:02:39.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.163+0000 7fb4697fa700 1 -- 172.21.15.69:0/355562583 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb454019070 con 0x7fb46c102380 2024-04-21T23:02:39.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.163+0000 7fb4697fa700 1 --2- 172.21.15.69:0/355562583 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb4580a2c20 0x7fb4580a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:02:39.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.164+0000 7fb4697fa700 1 -- 172.21.15.69:0/355562583 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7fb4540cda90 con 0x7fb46c102380 2024-04-21T23:02:39.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.164+0000 7fb46b7fe700 1 --2- 172.21.15.69:0/355562583 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb4580a2c20 0x7fb4580a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:02:39.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.164+0000 7fb46b7fe700 1 --2- 172.21.15.69:0/355562583 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb4580a2c20 0x7fb4580a50e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb45c009040 tx=0x7fb45c006000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:02:39.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.168+0000 7fb4697fa700 1 -- 172.21.15.69:0/355562583 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb45409ba60 con 0x7fb46c102380 2024-04-21T23:02:39.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:38 smithi044 ceph-mon[27934]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:39.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:38 smithi044 ceph-mon[27934]: Detected new or changed devices on smithi055 2024-04-21T23:02:39.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:38 smithi044 ceph-mon[27934]: Adjusting osd_memory_target on smithi055 to 8546M 2024-04-21T23:02:39.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:38 smithi044 ceph-mon[27934]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T23:02:39.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:38 smithi055 ceph-mon[31117]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:39.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:38 smithi055 ceph-mon[31117]: Detected new or changed devices on smithi055 2024-04-21T23:02:39.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:38 smithi055 ceph-mon[31117]: Adjusting osd_memory_target on smithi055 to 8546M 2024-04-21T23:02:39.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:38 smithi055 ceph-mon[31117]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T23:02:39.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:39.375+0000 7fb471e24700 1 -- 172.21.15.69:0/355562583 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fb46c0fd460 con 0x7fb4580a2c20 2024-04-21T23:02:40.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:02:40.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:02:40.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:40.192 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:39 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:02:40.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:39 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:02:40.193 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:39 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:40.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:39 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:02:40.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:39 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:02:40.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:39 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:40.923 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:40 smithi069 ceph-mon[31816]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:40.923 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:40 smithi069 ceph-mon[31816]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:02:41.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:40 smithi044 ceph-mon[27934]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:41.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:40 smithi044 ceph-mon[27934]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:02:41.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:40 smithi055 ceph-mon[31117]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:41.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:40 smithi055 ceph-mon[31117]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:02:43.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:42 smithi044 ceph-mon[27934]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:43.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:42 smithi044 ceph-mon[27934]: from='client.? 172.21.15.69:0/3151840733' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f219c5c9-af53-4cf0-a77a-56595b2db93b"}]: dispatch 2024-04-21T23:02:43.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:42 smithi044 ceph-mon[27934]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f219c5c9-af53-4cf0-a77a-56595b2db93b"}]: dispatch 2024-04-21T23:02:43.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:42 smithi044 ceph-mon[27934]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f219c5c9-af53-4cf0-a77a-56595b2db93b"}]': finished 2024-04-21T23:02:43.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:42 smithi044 ceph-mon[27934]: osdmap e35: 6 total, 5 up, 6 in 2024-04-21T23:02:43.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:42 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:02:43.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:42 smithi044 ceph-mon[27934]: from='client.? 172.21.15.69:0/4274587949' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:02:43.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:42 smithi055 ceph-mon[31117]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:43.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:42 smithi055 ceph-mon[31117]: from='client.? 172.21.15.69:0/3151840733' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f219c5c9-af53-4cf0-a77a-56595b2db93b"}]: dispatch 2024-04-21T23:02:43.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:42 smithi055 ceph-mon[31117]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f219c5c9-af53-4cf0-a77a-56595b2db93b"}]: dispatch 2024-04-21T23:02:43.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:42 smithi055 ceph-mon[31117]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f219c5c9-af53-4cf0-a77a-56595b2db93b"}]': finished 2024-04-21T23:02:43.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:42 smithi055 ceph-mon[31117]: osdmap e35: 6 total, 5 up, 6 in 2024-04-21T23:02:43.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:42 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:02:43.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:42 smithi055 ceph-mon[31117]: from='client.? 172.21.15.69:0/4274587949' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:02:43.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:42 smithi069 ceph-mon[31816]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:43.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:42 smithi069 ceph-mon[31816]: from='client.? 172.21.15.69:0/3151840733' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f219c5c9-af53-4cf0-a77a-56595b2db93b"}]: dispatch 2024-04-21T23:02:43.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:42 smithi069 ceph-mon[31816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f219c5c9-af53-4cf0-a77a-56595b2db93b"}]: dispatch 2024-04-21T23:02:43.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:42 smithi069 ceph-mon[31816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f219c5c9-af53-4cf0-a77a-56595b2db93b"}]': finished 2024-04-21T23:02:43.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:42 smithi069 ceph-mon[31816]: osdmap e35: 6 total, 5 up, 6 in 2024-04-21T23:02:43.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:42 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:02:43.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:42 smithi069 ceph-mon[31816]: from='client.? 172.21.15.69:0/4274587949' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:02:45.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:44 smithi044 ceph-mon[27934]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:45.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:44 smithi055 ceph-mon[31117]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:45.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:44 smithi069 ceph-mon[31816]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:46.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:46 smithi055 ceph-mon[31117]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:47.184 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:46 smithi069 ceph-mon[31816]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:47.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:46 smithi044 ceph-mon[27934]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:49.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:48 smithi044 ceph-mon[27934]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:49.200 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:48 smithi069 ceph-mon[31816]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:49.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:48 smithi055 ceph-mon[31117]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:50.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:49 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T23:02:50.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:49 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:50.192 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:49 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T23:02:50.192 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:49 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:50.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:49 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T23:02:50.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:49 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:50.949 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:50 smithi069 ceph-mon[31816]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:50.949 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:50 smithi069 ceph-mon[31816]: Deploying daemon osd.5 on smithi069 2024-04-21T23:02:51.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:50 smithi044 ceph-mon[27934]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:51.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:50 smithi044 ceph-mon[27934]: Deploying daemon osd.5 on smithi069 2024-04-21T23:02:51.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:50 smithi055 ceph-mon[31117]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:51.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:50 smithi055 ceph-mon[31117]: Deploying daemon osd.5 on smithi069 2024-04-21T23:02:53.161 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:52 smithi069 ceph-mon[31816]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:53.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:52 smithi044 ceph-mon[27934]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:53.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:52 smithi055 ceph-mon[31117]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:54.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:54 smithi069 ceph-mon[31816]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:54.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:54 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:54.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:54 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:54.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:54 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:02:55.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:54 smithi044 ceph-mon[27934]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:55.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:55.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:55.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:02:55.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:54 smithi055 ceph-mon[31117]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:55.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:54 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:55.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:54 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:55.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:54 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:02:57.138 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:56 smithi069 ceph-mon[31816]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:57.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:56 smithi044 ceph-mon[27934]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:57.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:56 smithi055 ceph-mon[31117]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:58.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:58 smithi069 ceph-mon[31816]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:58.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:58 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:58.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:58 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:58.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:58 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:58.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:58 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:02:58.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:58 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:58.543 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 5 on host 'smithi069' 2024-04-21T23:02:58.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:58.544+0000 7fb4697fa700 1 -- 172.21.15.69:0/355562583 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb46c0fd460 con 0x7fb4580a2c20 2024-04-21T23:02:58.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:58.546+0000 7fb471e24700 1 -- 172.21.15.69:0/355562583 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb4580a2c20 msgr2=0x7fb4580a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:02:58.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:58.546+0000 7fb471e24700 1 --2- 172.21.15.69:0/355562583 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb4580a2c20 0x7fb4580a50e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb45c009040 tx=0x7fb45c006000 comp rx=0 tx=0).stop 2024-04-21T23:02:58.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:58.546+0000 7fb471e24700 1 -- 172.21.15.69:0/355562583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb46c102380 msgr2=0x7fb46c198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:02:58.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:58.546+0000 7fb471e24700 1 --2- 172.21.15.69:0/355562583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb46c102380 0x7fb46c198aa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb45400d980 tx=0x7fb45400dc90 comp rx=0 tx=0).stop 2024-04-21T23:02:58.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:58.546+0000 7fb471e24700 1 -- 172.21.15.69:0/355562583 shutdown_connections 2024-04-21T23:02:58.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:58.546+0000 7fb471e24700 1 --2- 172.21.15.69:0/355562583 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb4580a2c20 0x7fb4580a50e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:58.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:58.546+0000 7fb471e24700 1 --2- 172.21.15.69:0/355562583 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb46c102d20 0x7fb46c19fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:58.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:58.546+0000 7fb471e24700 1 --2- 172.21.15.69:0/355562583 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb46c101180 0x7fb46c198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:58.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:58.547+0000 7fb471e24700 1 --2- 172.21.15.69:0/355562583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb46c102380 0x7fb46c198aa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:02:58.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:58.547+0000 7fb471e24700 1 -- 172.21.15.69:0/355562583 >> 172.21.15.69:0/355562583 conn(0x7fb46c0fc720 msgr2=0x7fb46c0fcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:02:58.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:58.547+0000 7fb471e24700 1 -- 172.21.15.69:0/355562583 shutdown_connections 2024-04-21T23:02:58.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:02:58.547+0000 7fb471e24700 1 -- 172.21.15.69:0/355562583 wait complete. 2024-04-21T23:02:58.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:58 smithi044 ceph-mon[27934]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:58.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:58 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:58.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:58 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:58.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:58 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:58.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:58 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:02:58.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:58 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:58.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:58 smithi055 ceph-mon[31117]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:02:58.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:58 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:58.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:58 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:58.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:58 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:02:58.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:58 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:02:58.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:58 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:59.115 DEBUG:teuthology.orchestra.run.smithi069:osd.5> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.5.service 2024-04-21T23:02:59.118 INFO:tasks.cephadm:Deploying osd.6 on smithi044 with /dev/vg_nvme/lv_2... 2024-04-21T23:02:59.118 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T23:02:59.375 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:02:59.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:59 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:59.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:59 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:02:59.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:59 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:59.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:59 smithi069 ceph-mon[31816]: from='osd.5 [v2:172.21.15.69:6808/2597202863,v1:172.21.15.69:6809/2597202863]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T23:02:59.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:02:59 smithi069 ceph-mon[31816]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T23:02:59.775 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:59 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:59.775 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:59 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:02:59.775 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:59 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:59.775 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:59 smithi044 ceph-mon[27934]: from='osd.5 [v2:172.21.15.69:6808/2597202863,v1:172.21.15.69:6809/2597202863]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T23:02:59.775 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:02:59 smithi044 ceph-mon[27934]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T23:02:59.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:59 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:59.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:59 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:02:59.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:59 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:02:59.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:59 smithi055 ceph-mon[31117]: from='osd.5 [v2:172.21.15.69:6808/2597202863,v1:172.21.15.69:6809/2597202863]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T23:02:59.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:02:59 smithi055 ceph-mon[31117]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T23:03:00.524 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:00 smithi069 ceph-mon[31816]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:03:00.524 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:00 smithi069 ceph-mon[31816]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T23:03:00.524 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:00 smithi069 ceph-mon[31816]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T23:03:00.525 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:00 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:03:00.525 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:00 smithi069 ceph-mon[31816]: from='osd.5 [v2:172.21.15.69:6808/2597202863,v1:172.21.15.69:6809/2597202863]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:03:00.525 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:00 smithi069 ceph-mon[31816]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:03:00.525 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:03:00 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:03:00.518+0000 7f00f9c13700 -1 osd.5 0 waiting for initial osdmap 2024-04-21T23:03:00.525 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:03:00 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:03:00.524+0000 7f00f5a06700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:03:00.591 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:00 smithi044 ceph-mon[27934]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:03:00.591 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:00 smithi044 ceph-mon[27934]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T23:03:00.591 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:00 smithi044 ceph-mon[27934]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T23:03:00.591 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:00 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:03:00.591 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:00 smithi044 ceph-mon[27934]: from='osd.5 [v2:172.21.15.69:6808/2597202863,v1:172.21.15.69:6809/2597202863]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:03:00.592 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:00 smithi044 ceph-mon[27934]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:03:00.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:00 smithi055 ceph-mon[31117]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:03:00.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:00 smithi055 ceph-mon[31117]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T23:03:00.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:00 smithi055 ceph-mon[31117]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T23:03:00.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:00 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:03:00.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:00 smithi055 ceph-mon[31117]: from='osd.5 [v2:172.21.15.69:6808/2597202863,v1:172.21.15.69:6809/2597202863]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:03:00.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:00 smithi055 ceph-mon[31117]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:03:01.182 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T23:03:01.182 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T23:03:01.183 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2024-04-21T23:03:01.183 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2024-04-21T23:03:01.183 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128516 s, 816 MB/s 2024-04-21T23:03:01.183 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2024-04-21T23:03:01.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:01 smithi069 ceph-mon[31816]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-21T23:03:01.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:01 smithi069 ceph-mon[31816]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T23:03:01.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:01 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:03:01.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:01 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:03:01.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:01 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:03:01.747 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:01 smithi044 ceph-mon[27934]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-21T23:03:01.747 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:01 smithi044 ceph-mon[27934]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T23:03:01.747 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:01 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:03:01.747 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:01 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:03:01.747 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:01 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:03:01.773 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi044:vg_nvme/lv_2 2024-04-21T23:03:01.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:01 smithi055 ceph-mon[31117]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-21T23:03:01.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:01 smithi055 ceph-mon[31117]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T23:03:01.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:01 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:03:01.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:01 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:03:01.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:01 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:03:02.032 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:03:02.843 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:02 smithi044 ceph-mon[27934]: purged_snaps scrub starts 2024-04-21T23:03:02.843 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:02 smithi044 ceph-mon[27934]: purged_snaps scrub ok 2024-04-21T23:03:02.843 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:02 smithi044 ceph-mon[27934]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:03:02.843 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:02 smithi044 ceph-mon[27934]: osd.5 [v2:172.21.15.69:6808/2597202863,v1:172.21.15.69:6809/2597202863] boot 2024-04-21T23:03:02.843 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:02 smithi044 ceph-mon[27934]: osdmap e38: 6 total, 6 up, 6 in 2024-04-21T23:03:02.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:03:02.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:02.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:02.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:02.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:02.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:02.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:02.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:03:02.844 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:02.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:02 smithi055 ceph-mon[31117]: purged_snaps scrub starts 2024-04-21T23:03:02.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:02 smithi055 ceph-mon[31117]: purged_snaps scrub ok 2024-04-21T23:03:02.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:02 smithi055 ceph-mon[31117]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:03:02.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:02 smithi055 ceph-mon[31117]: osd.5 [v2:172.21.15.69:6808/2597202863,v1:172.21.15.69:6809/2597202863] boot 2024-04-21T23:03:02.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:02 smithi055 ceph-mon[31117]: osdmap e38: 6 total, 6 up, 6 in 2024-04-21T23:03:02.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:02 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:03:02.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:02 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:02.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:02 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:02.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:02 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:02.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:02 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:02.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:02 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:02.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:02 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:02.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:02 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:03:02.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:02 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:02.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:02 smithi069 ceph-mon[31816]: purged_snaps scrub starts 2024-04-21T23:03:02.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:02 smithi069 ceph-mon[31816]: purged_snaps scrub ok 2024-04-21T23:03:02.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:02 smithi069 ceph-mon[31816]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-21T23:03:02.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:02 smithi069 ceph-mon[31816]: osd.5 [v2:172.21.15.69:6808/2597202863,v1:172.21.15.69:6809/2597202863] boot 2024-04-21T23:03:02.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:02 smithi069 ceph-mon[31816]: osdmap e38: 6 total, 6 up, 6 in 2024-04-21T23:03:02.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:02 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:03:02.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:02 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:02.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:02 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:02.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:02 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:02.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:02 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:02.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:02 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:02.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:02 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:02.967 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:02 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:03:02.967 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:02 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:03.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.619+0000 7f042e875700 1 -- 172.21.15.44:0/2581301629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0428100550 msgr2=0x7f0428100970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:03:03.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.619+0000 7f042e875700 1 --2- 172.21.15.44:0/2581301629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0428100550 0x7f0428100970 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0418009990 tx=0x7f0418009ca0 comp rx=0 tx=0).stop 2024-04-21T23:03:03.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.619+0000 7f042e875700 1 -- 172.21.15.44:0/2581301629 shutdown_connections 2024-04-21T23:03:03.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.619+0000 7f042e875700 1 --2- 172.21.15.44:0/2581301629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0428103ed0 0x7f042810a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:03.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.619+0000 7f042e875700 1 --2- 172.21.15.44:0/2581301629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0428100550 0x7f0428100970 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:03.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.619+0000 7f042e875700 1 --2- 172.21.15.44:0/2581301629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0428103530 0x7f0428103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:03.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.619+0000 7f042e875700 1 -- 172.21.15.44:0/2581301629 >> 172.21.15.44:0/2581301629 conn(0x7f04280fbe90 msgr2=0x7f04280fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:03:03.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.619+0000 7f042e875700 1 -- 172.21.15.44:0/2581301629 shutdown_connections 2024-04-21T23:03:03.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.619+0000 7f042e875700 1 -- 172.21.15.44:0/2581301629 wait complete. 2024-04-21T23:03:03.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.620+0000 7f042e875700 1 Processor -- start 2024-04-21T23:03:03.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.620+0000 7f042e875700 1 -- start start 2024-04-21T23:03:03.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.620+0000 7f042e875700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0428100550 0x7f0428102e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:03:03.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.620+0000 7f042e875700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0428103530 0x7f04281014e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:03:03.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.620+0000 7f042e875700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0428103ed0 0x7f0428101a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:03:03.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.620+0000 7f042e875700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f042810deb0 con 0x7f0428103ed0 2024-04-21T23:03:03.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.620+0000 7f042e875700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f042810dd30 con 0x7f0428100550 2024-04-21T23:03:03.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.620+0000 7f042e875700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f042810e030 con 0x7f0428103530 2024-04-21T23:03:03.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.620+0000 7f042ce12700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0428103ed0 0x7f0428101a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:03:03.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.620+0000 7f042ce12700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0428103ed0 0x7f0428101a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54646/0 (socket says 172.21.15.44:54646) 2024-04-21T23:03:03.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.620+0000 7f042ce12700 1 -- 172.21.15.44:0/3761211497 learned_addr learned my addr 172.21.15.44:0/3761211497 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:03:03.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.620+0000 7f0427fff700 1 --2- 172.21.15.44:0/3761211497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0428100550 0x7f0428102e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:03:03.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.620+0000 7f04277fe700 1 --2- 172.21.15.44:0/3761211497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0428103530 0x7f04281014e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:03:03.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.620+0000 7f042ce12700 1 -- 172.21.15.44:0/3761211497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0428103530 msgr2=0x7f04281014e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:03:03.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.620+0000 7f042ce12700 1 --2- 172.21.15.44:0/3761211497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0428103530 0x7f04281014e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:03.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.620+0000 7f042ce12700 1 -- 172.21.15.44:0/3761211497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0428100550 msgr2=0x7f0428102e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:03:03.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.620+0000 7f042ce12700 1 --2- 172.21.15.44:0/3761211497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0428100550 0x7f0428102e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:03.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.620+0000 7f042ce12700 1 -- 172.21.15.44:0/3761211497 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04281022d0 con 0x7f0428103ed0 2024-04-21T23:03:03.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.621+0000 7f042ce12700 1 --2- 172.21.15.44:0/3761211497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0428103ed0 0x7f0428101a20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f042000ef30 tx=0x7f042000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:03:03.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.621+0000 7f04257fa700 1 -- 172.21.15.44:0/3761211497 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04200092e0 con 0x7f0428103ed0 2024-04-21T23:03:03.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.621+0000 7f042e875700 1 -- 172.21.15.44:0/3761211497 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04281a25f0 con 0x7f0428103ed0 2024-04-21T23:03:03.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.621+0000 7f04257fa700 1 -- 172.21.15.44:0/3761211497 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0420004bb0 con 0x7f0428103ed0 2024-04-21T23:03:03.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.621+0000 7f042e875700 1 -- 172.21.15.44:0/3761211497 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04281a2b00 con 0x7f0428103ed0 2024-04-21T23:03:03.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.621+0000 7f04257fa700 1 -- 172.21.15.44:0/3761211497 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0420004280 con 0x7f0428103ed0 2024-04-21T23:03:03.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.622+0000 7f042e875700 1 -- 172.21.15.44:0/3761211497 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f042810e670 con 0x7f0428103ed0 2024-04-21T23:03:03.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.623+0000 7f04257fa700 1 -- 172.21.15.44:0/3761211497 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f042001b030 con 0x7f0428103ed0 2024-04-21T23:03:03.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.624+0000 7f04257fa700 1 --2- 172.21.15.44:0/3761211497 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f04100a2c20 0x7f04100a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:03:03.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.624+0000 7f0427fff700 1 --2- 172.21.15.44:0/3761211497 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f04100a2c20 0x7f04100a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:03:03.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.624+0000 7f04257fa700 1 -- 172.21.15.44:0/3761211497 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7f04200ce590 con 0x7f0428103ed0 2024-04-21T23:03:03.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.624+0000 7f0427fff700 1 --2- 172.21.15.44:0/3761211497 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f04100a2c20 0x7f04100a50e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0418000c00 tx=0x7f0418005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:03:03.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.628+0000 7f04257fa700 1 -- 172.21.15.44:0/3761211497 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0420022020 con 0x7f0428103ed0 2024-04-21T23:03:03.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:03 smithi044 ceph-mon[27934]: Detected new or changed devices on smithi069 2024-04-21T23:03:03.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:03 smithi044 ceph-mon[27934]: Adjusting osd_memory_target on smithi069 to 8544M 2024-04-21T23:03:03.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:03 smithi044 ceph-mon[27934]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T23:03:03.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:03.844+0000 7f042e875700 1 -- 172.21.15.44:0/3761211497 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f0428062670 con 0x7f04100a2c20 2024-04-21T23:03:03.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:03 smithi055 ceph-mon[31117]: Detected new or changed devices on smithi069 2024-04-21T23:03:03.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:03 smithi055 ceph-mon[31117]: Adjusting osd_memory_target on smithi069 to 8544M 2024-04-21T23:03:03.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:03 smithi055 ceph-mon[31117]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T23:03:03.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:03 smithi069 ceph-mon[31816]: Detected new or changed devices on smithi069 2024-04-21T23:03:03.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:03 smithi069 ceph-mon[31816]: Adjusting osd_memory_target on smithi069 to 8544M 2024-04-21T23:03:03.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:03 smithi069 ceph-mon[31816]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T23:03:04.742 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:04 smithi044 ceph-mon[27934]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:03:04.742 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:04 smithi044 ceph-mon[27934]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T23:03:04.742 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:04 smithi044 ceph-mon[27934]: from='client.14382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:03:04.742 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:04 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:03:04.742 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:04 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:03:04.742 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:04 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:04.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:04 smithi069 ceph-mon[31816]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:03:04.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:04 smithi069 ceph-mon[31816]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T23:03:04.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:04 smithi069 ceph-mon[31816]: from='client.14382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:03:04.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:04 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:03:04.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:04 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:03:04.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:04 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:04.866 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:04 smithi055 ceph-mon[31117]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:03:04.867 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:04 smithi055 ceph-mon[31117]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T23:03:04.867 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:04 smithi055 ceph-mon[31117]: from='client.14382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:03:04.867 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:04 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:03:04.867 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:04 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:03:04.867 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:04 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:05.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:05 smithi044 ceph-mon[27934]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T23:03:05.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:05 smithi055 ceph-mon[31117]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T23:03:05.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:05 smithi069 ceph-mon[31816]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T23:03:06.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:06 smithi044 ceph-mon[27934]: pgmap v142: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-21T23:03:06.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:06 smithi044 ceph-mon[27934]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-21T23:03:06.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:06 smithi044 ceph-mon[27934]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T23:03:06.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:06 smithi069 ceph-mon[31816]: pgmap v142: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-21T23:03:06.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:06 smithi069 ceph-mon[31816]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-21T23:03:06.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:06 smithi069 ceph-mon[31816]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T23:03:06.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:06 smithi055 ceph-mon[31117]: pgmap v142: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-21T23:03:06.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:06 smithi055 ceph-mon[31117]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-21T23:03:06.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:06 smithi055 ceph-mon[31117]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T23:03:07.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:07 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3305293779' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c9b8cd5-e20b-49eb-af02-87709b05e2e5"}]: dispatch 2024-04-21T23:03:07.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:07 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3305293779' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c9b8cd5-e20b-49eb-af02-87709b05e2e5"}]': finished 2024-04-21T23:03:07.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:07 smithi044 ceph-mon[27934]: osdmap e43: 7 total, 6 up, 7 in 2024-04-21T23:03:07.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:07 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:03:07.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:07 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/341495473' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:03:07.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:07 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3305293779' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c9b8cd5-e20b-49eb-af02-87709b05e2e5"}]: dispatch 2024-04-21T23:03:07.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:07 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3305293779' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c9b8cd5-e20b-49eb-af02-87709b05e2e5"}]': finished 2024-04-21T23:03:07.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:07 smithi069 ceph-mon[31816]: osdmap e43: 7 total, 6 up, 7 in 2024-04-21T23:03:07.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:07 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:03:07.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:07 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/341495473' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:03:07.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:07 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3305293779' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c9b8cd5-e20b-49eb-af02-87709b05e2e5"}]: dispatch 2024-04-21T23:03:07.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:07 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3305293779' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c9b8cd5-e20b-49eb-af02-87709b05e2e5"}]': finished 2024-04-21T23:03:07.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:07 smithi055 ceph-mon[31117]: osdmap e43: 7 total, 6 up, 7 in 2024-04-21T23:03:07.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:07 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:03:07.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:07 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/341495473' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:03:08.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:08 smithi044 ceph-mon[27934]: pgmap v145: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-21T23:03:08.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:08 smithi069 ceph-mon[31816]: pgmap v145: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-21T23:03:08.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:08 smithi055 ceph-mon[31117]: pgmap v145: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-21T23:03:09.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:09 smithi044 ceph-mon[27934]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-21T23:03:09.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:09 smithi044 ceph-mon[27934]: Cluster is now healthy 2024-04-21T23:03:09.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:09 smithi055 ceph-mon[31117]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-21T23:03:09.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:09 smithi055 ceph-mon[31117]: Cluster is now healthy 2024-04-21T23:03:09.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:09 smithi069 ceph-mon[31816]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-21T23:03:09.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:09 smithi069 ceph-mon[31816]: Cluster is now healthy 2024-04-21T23:03:10.930 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:10 smithi044 ceph-mon[27934]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-21T23:03:10.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:10 smithi055 ceph-mon[31117]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-21T23:03:10.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:10 smithi069 ceph-mon[31816]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-21T23:03:12.831 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:12 smithi044 ceph-mon[27934]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-21T23:03:12.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:12 smithi069 ceph-mon[31816]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-21T23:03:12.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:12 smithi055 ceph-mon[31117]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-21T23:03:14.857 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:14 smithi044 ceph-mon[27934]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T23:03:14.857 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:14 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T23:03:14.857 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:14 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:14.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:14 smithi055 ceph-mon[31117]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T23:03:14.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:14 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T23:03:14.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:14 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:14.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:14 smithi069 ceph-mon[31816]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T23:03:14.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:14 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T23:03:14.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:14 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:15.888 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:15 smithi044 ceph-mon[27934]: Deploying daemon osd.6 on smithi044 2024-04-21T23:03:15.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:15 smithi055 ceph-mon[31117]: Deploying daemon osd.6 on smithi044 2024-04-21T23:03:15.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:15 smithi069 ceph-mon[31816]: Deploying daemon osd.6 on smithi044 2024-04-21T23:03:16.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:16 smithi044 ceph-mon[27934]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-21T23:03:16.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:16 smithi055 ceph-mon[31117]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-21T23:03:16.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:16 smithi069 ceph-mon[31816]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-21T23:03:18.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:18 smithi044 ceph-mon[27934]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-21T23:03:18.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:18 smithi055 ceph-mon[31117]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-21T23:03:18.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:18 smithi069 ceph-mon[31816]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-21T23:03:20.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:20 smithi044 ceph-mon[27934]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-21T23:03:20.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:20 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:20.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:20 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:20.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:20 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:03:20.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:20 smithi055 ceph-mon[31117]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-21T23:03:20.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:20 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:20.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:20 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:20.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:20 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:03:20.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:20 smithi069 ceph-mon[31816]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-21T23:03:20.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:20 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:20.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:20 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:20.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:20 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:03:22.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:22 smithi055 ceph-mon[31117]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:03:22.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:22 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:22.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:22 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:22.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:22 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:22.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:22 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:03:22.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:22 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:22.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:22 smithi069 ceph-mon[31816]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:03:22.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:22 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:22.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:22 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:22.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:22 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:22.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:22 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:03:22.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:22 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:23.099 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:22 smithi044 ceph-mon[27934]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:03:23.099 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:22 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:23.100 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:22 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:23.100 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:22 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:23.100 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:22 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:03:23.100 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:22 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:23.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:23.550+0000 7f04257fa700 1 -- 172.21.15.44:0/3761211497 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0428062670 con 0x7f04100a2c20 2024-04-21T23:03:23.551 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 6 on host 'smithi044' 2024-04-21T23:03:23.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:23.552+0000 7f042e875700 1 -- 172.21.15.44:0/3761211497 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f04100a2c20 msgr2=0x7f04100a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:03:23.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:23.552+0000 7f042e875700 1 --2- 172.21.15.44:0/3761211497 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f04100a2c20 0x7f04100a50e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0418000c00 tx=0x7f0418005fb0 comp rx=0 tx=0).stop 2024-04-21T23:03:23.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:23.552+0000 7f042e875700 1 -- 172.21.15.44:0/3761211497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0428103ed0 msgr2=0x7f0428101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:03:23.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:23.552+0000 7f042e875700 1 --2- 172.21.15.44:0/3761211497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0428103ed0 0x7f0428101a20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f042000ef30 tx=0x7f042000c3a0 comp rx=0 tx=0).stop 2024-04-21T23:03:23.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:23.553+0000 7f042e875700 1 -- 172.21.15.44:0/3761211497 shutdown_connections 2024-04-21T23:03:23.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:23.553+0000 7f042e875700 1 --2- 172.21.15.44:0/3761211497 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f04100a2c20 0x7f04100a50e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:23.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:23.553+0000 7f042e875700 1 --2- 172.21.15.44:0/3761211497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0428103ed0 0x7f0428101a20 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:23.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:23.553+0000 7f042e875700 1 --2- 172.21.15.44:0/3761211497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0428100550 0x7f0428102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:23.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:23.553+0000 7f042e875700 1 --2- 172.21.15.44:0/3761211497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0428103530 0x7f04281014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:23.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:23.553+0000 7f042e875700 1 -- 172.21.15.44:0/3761211497 >> 172.21.15.44:0/3761211497 conn(0x7f04280fbe90 msgr2=0x7f0428106830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:03:23.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:23.553+0000 7f042e875700 1 -- 172.21.15.44:0/3761211497 shutdown_connections 2024-04-21T23:03:23.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:03:23.553+0000 7f042e875700 1 -- 172.21.15.44:0/3761211497 wait complete. 2024-04-21T23:03:24.153 DEBUG:teuthology.orchestra.run.smithi044:osd.6> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.6.service 2024-04-21T23:03:24.155 INFO:tasks.cephadm:Deploying osd.7 on smithi055 with /dev/vg_nvme/lv_2... 2024-04-21T23:03:24.155 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T23:03:24.409 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:03:24.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:24 smithi044 ceph-mon[27934]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:03:24.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:24.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:03:24.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:24.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:24 smithi044 ceph-mon[27934]: from='osd.6 [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T23:03:24.850 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:24 smithi055 ceph-mon[31117]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:03:24.850 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:24.850 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:03:24.850 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:24.850 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:24 smithi055 ceph-mon[31117]: from='osd.6 [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T23:03:24.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:24 smithi069 ceph-mon[31816]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:03:24.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:24.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:03:24.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:24.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:24 smithi069 ceph-mon[31816]: from='osd.6 [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T23:03:25.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:25 smithi044 ceph-mon[27934]: from='osd.6 [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T23:03:25.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:25 smithi044 ceph-mon[27934]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T23:03:25.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:25 smithi044 ceph-mon[27934]: from='osd.6 [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-21T23:03:25.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:25 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:03:25.686 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:03:25 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:03:25.561+0000 7fee2bef5700 -1 osd.6 0 waiting for initial osdmap 2024-04-21T23:03:25.686 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:03:25 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:03:25.573+0000 7fee27ce8700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:03:25.890 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:25 smithi055 ceph-mon[31117]: from='osd.6 [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T23:03:25.891 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:25 smithi055 ceph-mon[31117]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T23:03:25.891 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:25 smithi055 ceph-mon[31117]: from='osd.6 [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-21T23:03:25.891 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:25 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:03:25.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:25 smithi069 ceph-mon[31816]: from='osd.6 [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T23:03:25.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:25 smithi069 ceph-mon[31816]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T23:03:25.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:25 smithi069 ceph-mon[31816]: from='osd.6 [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-21T23:03:25.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:25 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:03:26.496 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T23:03:26.497 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T23:03:26.497 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10+0 records in 2024-04-21T23:03:26.497 INFO:teuthology.orchestra.run.smithi055.stderr:10+0 records out 2024-04-21T23:03:26.497 INFO:teuthology.orchestra.run.smithi055.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0156452 s, 670 MB/s 2024-04-21T23:03:26.497 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping successful for: 2024-04-21T23:03:26.837 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:26 smithi055 ceph-mon[31117]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:03:26.838 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:26 smithi055 ceph-mon[31117]: from='osd.6 [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-04-21T23:03:26.838 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:26 smithi055 ceph-mon[31117]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T23:03:26.838 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:26 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:03:26.838 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:26 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:03:26.867 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:26 smithi044 ceph-mon[27934]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:03:26.867 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:26 smithi044 ceph-mon[27934]: from='osd.6 [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-04-21T23:03:26.867 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:26 smithi044 ceph-mon[27934]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T23:03:26.867 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:26 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:03:26.867 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:26 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:03:26.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:26 smithi069 ceph-mon[31816]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-21T23:03:26.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:26 smithi069 ceph-mon[31816]: from='osd.6 [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-04-21T23:03:26.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:26 smithi069 ceph-mon[31816]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T23:03:26.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:26 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:03:26.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:26 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:03:27.157 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi055:vg_nvme/lv_2 2024-04-21T23:03:27.408 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:03:27.776 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:27 smithi055 ceph-mon[31117]: purged_snaps scrub starts 2024-04-21T23:03:27.776 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:27 smithi055 ceph-mon[31117]: purged_snaps scrub ok 2024-04-21T23:03:27.776 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:27 smithi055 ceph-mon[31117]: osd.6 [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748] boot 2024-04-21T23:03:27.776 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:27 smithi055 ceph-mon[31117]: osdmap e46: 7 total, 7 up, 7 in 2024-04-21T23:03:27.776 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:03:27.776 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:27.776 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:27.777 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:27.777 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:27.777 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:27.777 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:27.777 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:27.777 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:03:27.777 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:27.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:27 smithi044 ceph-mon[27934]: purged_snaps scrub starts 2024-04-21T23:03:27.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:27 smithi044 ceph-mon[27934]: purged_snaps scrub ok 2024-04-21T23:03:27.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:27 smithi044 ceph-mon[27934]: osd.6 [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748] boot 2024-04-21T23:03:27.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:27 smithi044 ceph-mon[27934]: osdmap e46: 7 total, 7 up, 7 in 2024-04-21T23:03:27.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:03:27.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:27.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:27.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:27.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:27.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:27.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:27.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:27.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:03:27.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:27.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:27 smithi069 ceph-mon[31816]: purged_snaps scrub starts 2024-04-21T23:03:27.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:27 smithi069 ceph-mon[31816]: purged_snaps scrub ok 2024-04-21T23:03:27.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:27 smithi069 ceph-mon[31816]: osd.6 [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748] boot 2024-04-21T23:03:27.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:27 smithi069 ceph-mon[31816]: osdmap e46: 7 total, 7 up, 7 in 2024-04-21T23:03:27.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:03:27.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:27.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:27.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:27.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:27.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:27.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:27.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:27.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:03:27.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:28.792 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:28 smithi055 ceph-mon[31117]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:28.792 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:28 smithi055 ceph-mon[31117]: Detected new or changed devices on smithi044 2024-04-21T23:03:28.792 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:28 smithi055 ceph-mon[31117]: Adjusting osd_memory_target on smithi044 to 5697M 2024-04-21T23:03:28.792 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:28 smithi055 ceph-mon[31117]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T23:03:28.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:28 smithi044 ceph-mon[27934]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:28.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:28 smithi044 ceph-mon[27934]: Detected new or changed devices on smithi044 2024-04-21T23:03:28.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:28 smithi044 ceph-mon[27934]: Adjusting osd_memory_target on smithi044 to 5697M 2024-04-21T23:03:28.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:28 smithi044 ceph-mon[27934]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T23:03:28.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:28 smithi069 ceph-mon[31816]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:28.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:28 smithi069 ceph-mon[31816]: Detected new or changed devices on smithi044 2024-04-21T23:03:28.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:28 smithi069 ceph-mon[31816]: Adjusting osd_memory_target on smithi044 to 5697M 2024-04-21T23:03:28.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:28 smithi069 ceph-mon[31816]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T23:03:29.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.038+0000 7fc85853f700 1 -- 172.21.15.55:0/1881457017 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8501022b0 msgr2=0x7fc8501026d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:03:29.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.038+0000 7fc85853f700 1 --2- 172.21.15.55:0/1881457017 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8501022b0 0x7fc8501026d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc84c0099c0 tx=0x7fc84c009cd0 comp rx=0 tx=0).stop 2024-04-21T23:03:29.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.038+0000 7fc85853f700 1 -- 172.21.15.55:0/1881457017 shutdown_connections 2024-04-21T23:03:29.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.038+0000 7fc85853f700 1 --2- 172.21.15.55:0/1881457017 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc850103e50 0x7fc85010a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:29.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.038+0000 7fc85853f700 1 --2- 172.21.15.55:0/1881457017 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc8501022b0 0x7fc8501026d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:29.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.038+0000 7fc85853f700 1 --2- 172.21.15.55:0/1881457017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8501034b0 0x7fc850103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:29.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.038+0000 7fc85853f700 1 -- 172.21.15.55:0/1881457017 >> 172.21.15.55:0/1881457017 conn(0x7fc8500fd850 msgr2=0x7fc8500ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:03:29.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.038+0000 7fc85853f700 1 -- 172.21.15.55:0/1881457017 shutdown_connections 2024-04-21T23:03:29.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.038+0000 7fc85853f700 1 -- 172.21.15.55:0/1881457017 wait complete. 2024-04-21T23:03:29.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.038+0000 7fc85853f700 1 Processor -- start 2024-04-21T23:03:29.043 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.038+0000 7fc85853f700 1 -- start start 2024-04-21T23:03:29.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc85853f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8501022b0 0x7fc8501984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:03:29.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc85853f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8501034b0 0x7fc850198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:03:29.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc85853f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc850103e50 0x7fc85019fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:03:29.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc85853f700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc85010de70 con 0x7fc8501034b0 2024-04-21T23:03:29.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc85853f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc85010dcf0 con 0x7fc850103e50 2024-04-21T23:03:29.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc85853f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc85010dff0 con 0x7fc8501022b0 2024-04-21T23:03:29.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc856adc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc850103e50 0x7fc85019fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:03:29.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc856adc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc850103e50 0x7fc85019fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:44048/0 (socket says 172.21.15.55:44048) 2024-04-21T23:03:29.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc856adc700 1 -- 172.21.15.55:0/509284030 learned_addr learned my addr 172.21.15.55:0/509284030 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-04-21T23:03:29.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc855ada700 1 --2- 172.21.15.55:0/509284030 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8501034b0 0x7fc850198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:03:29.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc8562db700 1 --2- 172.21.15.55:0/509284030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8501022b0 0x7fc8501984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:03:29.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc856adc700 1 -- 172.21.15.55:0/509284030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8501022b0 msgr2=0x7fc8501984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:03:29.044 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc856adc700 1 --2- 172.21.15.55:0/509284030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8501022b0 0x7fc8501984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:29.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc856adc700 1 -- 172.21.15.55:0/509284030 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8501034b0 msgr2=0x7fc850198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:03:29.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc856adc700 1 --2- 172.21.15.55:0/509284030 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8501034b0 0x7fc850198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:29.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc856adc700 1 -- 172.21.15.55:0/509284030 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8501a0120 con 0x7fc850103e50 2024-04-21T23:03:29.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc8562db700 1 --2- 172.21.15.55:0/509284030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8501022b0 0x7fc8501984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:03:29.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc856adc700 1 --2- 172.21.15.55:0/509284030 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc850103e50 0x7fc85019fa80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc848007ee0 tx=0x7fc84800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:03:29.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc8477fe700 1 -- 172.21.15.55:0/509284030 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84800dbd0 con 0x7fc850103e50 2024-04-21T23:03:29.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc85853f700 1 -- 172.21.15.55:0/509284030 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8501a0410 con 0x7fc850103e50 2024-04-21T23:03:29.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc8477fe700 1 -- 172.21.15.55:0/509284030 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc84800dd70 con 0x7fc850103e50 2024-04-21T23:03:29.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc8477fe700 1 -- 172.21.15.55:0/509284030 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc848005a40 con 0x7fc850103e50 2024-04-21T23:03:29.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.039+0000 7fc85853f700 1 -- 172.21.15.55:0/509284030 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8501a0900 con 0x7fc850103e50 2024-04-21T23:03:29.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.040+0000 7fc85853f700 1 -- 172.21.15.55:0/509284030 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc850068f20 con 0x7fc850103e50 2024-04-21T23:03:29.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.041+0000 7fc8477fe700 1 -- 172.21.15.55:0/509284030 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc84801a020 con 0x7fc850103e50 2024-04-21T23:03:29.045 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.042+0000 7fc8477fe700 1 --2- 172.21.15.55:0/509284030 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc83c0a2c20 0x7fc83c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:03:29.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.042+0000 7fc8477fe700 1 -- 172.21.15.55:0/509284030 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7fc8480ce0a0 con 0x7fc850103e50 2024-04-21T23:03:29.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.042+0000 7fc8562db700 1 --2- 172.21.15.55:0/509284030 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc83c0a2c20 0x7fc83c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:03:29.046 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.043+0000 7fc8562db700 1 --2- 172.21.15.55:0/509284030 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc83c0a2c20 0x7fc83c0a50e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00b580 tx=0x7fc84c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:03:29.047 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.048+0000 7fc8477fe700 1 -- 172.21.15.55:0/509284030 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc848010070 con 0x7fc850103e50 2024-04-21T23:03:29.264 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:29.264+0000 7fc85853f700 1 -- 172.21.15.55:0/509284030 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fc850062730 con 0x7fc83c0a2c20 2024-04-21T23:03:29.901 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:29 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:03:29.901 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:29 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:03:29.901 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:29 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:29.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:29 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:03:29.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:29 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:03:29.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:29 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:29.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:29 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:03:29.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:29 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:03:29.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:29 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:30.875 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:30 smithi055 ceph-mon[31117]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:30.876 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:30 smithi055 ceph-mon[31117]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:03:30.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:30 smithi044 ceph-mon[27934]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:30.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:30 smithi044 ceph-mon[27934]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:03:30.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:30 smithi069 ceph-mon[31816]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:30.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:30 smithi069 ceph-mon[31816]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:03:32.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:32 smithi055 ceph-mon[31117]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:32.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:32 smithi055 ceph-mon[31117]: from='client.? 172.21.15.55:0/3304540040' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12c98dc9-81df-4a59-9e6f-e480f5162390"}]: dispatch 2024-04-21T23:03:32.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:32 smithi055 ceph-mon[31117]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12c98dc9-81df-4a59-9e6f-e480f5162390"}]: dispatch 2024-04-21T23:03:32.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:32 smithi055 ceph-mon[31117]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12c98dc9-81df-4a59-9e6f-e480f5162390"}]': finished 2024-04-21T23:03:32.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:32 smithi055 ceph-mon[31117]: osdmap e48: 8 total, 7 up, 8 in 2024-04-21T23:03:32.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:32 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:03:32.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:32 smithi044 ceph-mon[27934]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:32.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:32 smithi044 ceph-mon[27934]: from='client.? 172.21.15.55:0/3304540040' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12c98dc9-81df-4a59-9e6f-e480f5162390"}]: dispatch 2024-04-21T23:03:32.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:32 smithi044 ceph-mon[27934]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12c98dc9-81df-4a59-9e6f-e480f5162390"}]: dispatch 2024-04-21T23:03:32.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:32 smithi044 ceph-mon[27934]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12c98dc9-81df-4a59-9e6f-e480f5162390"}]': finished 2024-04-21T23:03:32.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:32 smithi044 ceph-mon[27934]: osdmap e48: 8 total, 7 up, 8 in 2024-04-21T23:03:32.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:32 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:03:32.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:32 smithi069 ceph-mon[31816]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:32.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:32 smithi069 ceph-mon[31816]: from='client.? 172.21.15.55:0/3304540040' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12c98dc9-81df-4a59-9e6f-e480f5162390"}]: dispatch 2024-04-21T23:03:32.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:32 smithi069 ceph-mon[31816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12c98dc9-81df-4a59-9e6f-e480f5162390"}]: dispatch 2024-04-21T23:03:32.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:32 smithi069 ceph-mon[31816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12c98dc9-81df-4a59-9e6f-e480f5162390"}]': finished 2024-04-21T23:03:32.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:32 smithi069 ceph-mon[31816]: osdmap e48: 8 total, 7 up, 8 in 2024-04-21T23:03:32.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:32 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:03:33.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:33 smithi044 ceph-mon[27934]: from='client.? 172.21.15.55:0/1848363364' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:03:33.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:33 smithi055 ceph-mon[31117]: from='client.? 172.21.15.55:0/1848363364' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:03:33.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:33 smithi069 ceph-mon[31816]: from='client.? 172.21.15.55:0/1848363364' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:03:34.865 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:34 smithi055 ceph-mon[31117]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:34.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:34 smithi069 ceph-mon[31816]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:34.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:34 smithi044 ceph-mon[27934]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:36.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:36 smithi055 ceph-mon[31117]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:36.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:36 smithi044 ceph-mon[27934]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:36.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:36 smithi069 ceph-mon[31816]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:38.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:38 smithi055 ceph-mon[31117]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:38.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:38 smithi044 ceph-mon[27934]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:38.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:38 smithi069 ceph-mon[31816]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:39.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:39 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T23:03:39.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:39 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:39.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T23:03:39.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:39.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:39 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T23:03:39.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:39 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:40.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:40 smithi044 ceph-mon[27934]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:40.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:40 smithi044 ceph-mon[27934]: Deploying daemon osd.7 on smithi055 2024-04-21T23:03:40.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:40 smithi069 ceph-mon[31816]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:40.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:40 smithi069 ceph-mon[31816]: Deploying daemon osd.7 on smithi055 2024-04-21T23:03:40.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:40 smithi055 ceph-mon[31117]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:40.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:40 smithi055 ceph-mon[31117]: Deploying daemon osd.7 on smithi055 2024-04-21T23:03:42.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:42 smithi044 ceph-mon[27934]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:42.936 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:42 smithi055 ceph-mon[31117]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:42.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:42 smithi069 ceph-mon[31816]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:44.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:44 smithi055 ceph-mon[31117]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:44.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:44 smithi044 ceph-mon[27934]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:44.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:44 smithi069 ceph-mon[31816]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:46.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:46 smithi055 ceph-mon[31117]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:46.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:46 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:46.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:46 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:46.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:46 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:03:46.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:46 smithi044 ceph-mon[27934]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:46.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:46 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:46.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:46 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:46.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:46 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:03:46.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:46 smithi069 ceph-mon[31816]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:46.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:46 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:46.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:46 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:46.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:46 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:03:48.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:48 smithi055 ceph-mon[31117]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:48.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:48 smithi044 ceph-mon[27934]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:48.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:48 smithi069 ceph-mon[31816]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:48.586 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:48.586+0000 7fc8477fe700 1 -- 172.21.15.55:0/509284030 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc850062730 con 0x7fc83c0a2c20 2024-04-21T23:03:48.587 INFO:teuthology.orchestra.run.smithi055.stdout:Created osd(s) 7 on host 'smithi055' 2024-04-21T23:03:48.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:48.589+0000 7fc85853f700 1 -- 172.21.15.55:0/509284030 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc83c0a2c20 msgr2=0x7fc83c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:03:48.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:48.589+0000 7fc85853f700 1 --2- 172.21.15.55:0/509284030 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc83c0a2c20 0x7fc83c0a50e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00b580 tx=0x7fc84c005fb0 comp rx=0 tx=0).stop 2024-04-21T23:03:48.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:48.589+0000 7fc85853f700 1 -- 172.21.15.55:0/509284030 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc850103e50 msgr2=0x7fc85019fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:03:48.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:48.589+0000 7fc85853f700 1 --2- 172.21.15.55:0/509284030 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc850103e50 0x7fc85019fa80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc848007ee0 tx=0x7fc84800d3b0 comp rx=0 tx=0).stop 2024-04-21T23:03:48.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:48.589+0000 7fc85853f700 1 -- 172.21.15.55:0/509284030 shutdown_connections 2024-04-21T23:03:48.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:48.589+0000 7fc85853f700 1 --2- 172.21.15.55:0/509284030 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc850103e50 0x7fc85019fa80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:48.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:48.589+0000 7fc85853f700 1 --2- 172.21.15.55:0/509284030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8501022b0 0x7fc8501984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:48.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:48.589+0000 7fc85853f700 1 --2- 172.21.15.55:0/509284030 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc83c0a2c20 0x7fc83c0a50e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:48.589 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:48.589+0000 7fc85853f700 1 --2- 172.21.15.55:0/509284030 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc8501034b0 0x7fc850198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:48.590 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:48.589+0000 7fc85853f700 1 -- 172.21.15.55:0/509284030 >> 172.21.15.55:0/509284030 conn(0x7fc8500fd850 msgr2=0x7fc8501067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:03:48.590 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:48.589+0000 7fc85853f700 1 -- 172.21.15.55:0/509284030 shutdown_connections 2024-04-21T23:03:48.590 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:03:48.589+0000 7fc85853f700 1 -- 172.21.15.55:0/509284030 wait complete. 2024-04-21T23:03:49.217 DEBUG:teuthology.orchestra.run.smithi055:osd.7> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.7.service 2024-04-21T23:03:49.219 INFO:tasks.cephadm:Deploying osd.8 on smithi069 with /dev/vg_nvme/lv_2... 2024-04-21T23:03:49.220 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T23:03:49.471 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:03:49.829 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:49 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:49.829 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:49 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:49.829 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:49 smithi069 ceph-mon[31816]: from='osd.7 [v2:172.21.15.55:6816/3062702978,v1:172.21.15.55:6817/3062702978]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T23:03:49.830 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:49 smithi069 ceph-mon[31816]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T23:03:49.830 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:49 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:49.830 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:49 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:49.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:49 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:49.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:49 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:49.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:49 smithi044 ceph-mon[27934]: from='osd.7 [v2:172.21.15.55:6816/3062702978,v1:172.21.15.55:6817/3062702978]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T23:03:49.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:49 smithi044 ceph-mon[27934]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T23:03:49.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:49 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:49.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:49 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:49.962 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:49 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:49.962 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:49 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:49.962 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:49 smithi055 ceph-mon[31117]: from='osd.7 [v2:172.21.15.55:6816/3062702978,v1:172.21.15.55:6817/3062702978]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T23:03:49.962 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:49 smithi055 ceph-mon[31117]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T23:03:49.962 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:49 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:49.962 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:49 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:50.853 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:50 smithi069 ceph-mon[31816]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:50.853 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:50 smithi069 ceph-mon[31816]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T23:03:50.853 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:50 smithi069 ceph-mon[31816]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T23:03:50.853 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:50 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:03:50.853 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:50 smithi069 ceph-mon[31816]: from='osd.7 [v2:172.21.15.55:6816/3062702978,v1:172.21.15.55:6817/3062702978]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:03:50.853 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:50 smithi069 ceph-mon[31816]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:03:50.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:50 smithi044 ceph-mon[27934]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:50.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:50 smithi044 ceph-mon[27934]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T23:03:50.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:50 smithi044 ceph-mon[27934]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T23:03:50.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:50 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:03:50.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:50 smithi044 ceph-mon[27934]: from='osd.7 [v2:172.21.15.55:6816/3062702978,v1:172.21.15.55:6817/3062702978]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:03:50.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:50 smithi044 ceph-mon[27934]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:03:50.940 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:50 smithi055 ceph-mon[31117]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:50.940 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:50 smithi055 ceph-mon[31117]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T23:03:50.940 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:50 smithi055 ceph-mon[31117]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T23:03:50.940 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:50 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:03:50.940 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:50 smithi055 ceph-mon[31117]: from='osd.7 [v2:172.21.15.55:6816/3062702978,v1:172.21.15.55:6817/3062702978]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:03:50.940 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:50 smithi055 ceph-mon[31117]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:03:50.940 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:03:50 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:03:50.596+0000 7f1ba2d9d700 -1 osd.7 0 waiting for initial osdmap 2024-04-21T23:03:50.941 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:03:50 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:03:50.605+0000 7f1b9eb90700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:03:51.434 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T23:03:51.434 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T23:03:51.434 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2024-04-21T23:03:51.434 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2024-04-21T23:03:51.434 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110571 s, 948 MB/s 2024-04-21T23:03:51.434 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2024-04-21T23:03:51.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:51 smithi069 ceph-mon[31816]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-04-21T23:03:51.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:51 smithi069 ceph-mon[31816]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T23:03:51.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:51 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:03:51.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:51 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:03:51.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:51 smithi069 ceph-mon[31816]: osd.7 [v2:172.21.15.55:6816/3062702978,v1:172.21.15.55:6817/3062702978] boot 2024-04-21T23:03:51.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:51 smithi069 ceph-mon[31816]: osdmap e51: 8 total, 8 up, 8 in 2024-04-21T23:03:51.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:51 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:03:51.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:51 smithi044 ceph-mon[27934]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-04-21T23:03:51.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:51 smithi044 ceph-mon[27934]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T23:03:51.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:51 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:03:51.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:51 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:03:51.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:51 smithi044 ceph-mon[27934]: osd.7 [v2:172.21.15.55:6816/3062702978,v1:172.21.15.55:6817/3062702978] boot 2024-04-21T23:03:51.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:51 smithi044 ceph-mon[27934]: osdmap e51: 8 total, 8 up, 8 in 2024-04-21T23:03:51.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:51 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:03:51.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:51 smithi055 ceph-mon[31117]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-04-21T23:03:51.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:51 smithi055 ceph-mon[31117]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T23:03:51.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:51 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:03:51.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:51 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:03:51.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:51 smithi055 ceph-mon[31117]: osd.7 [v2:172.21.15.55:6816/3062702978,v1:172.21.15.55:6817/3062702978] boot 2024-04-21T23:03:51.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:51 smithi055 ceph-mon[31117]: osdmap e51: 8 total, 8 up, 8 in 2024-04-21T23:03:51.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:51 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:03:52.117 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi069:vg_nvme/lv_2 2024-04-21T23:03:52.358 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:03:52.698 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:52 smithi055 ceph-mon[31117]: purged_snaps scrub starts 2024-04-21T23:03:52.698 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:52 smithi055 ceph-mon[31117]: purged_snaps scrub ok 2024-04-21T23:03:52.698 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:52 smithi055 ceph-mon[31117]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:52.698 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:52 smithi055 ceph-mon[31117]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T23:03:52.701 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:52 smithi069 ceph-mon[31816]: purged_snaps scrub starts 2024-04-21T23:03:52.701 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:52 smithi069 ceph-mon[31816]: purged_snaps scrub ok 2024-04-21T23:03:52.701 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:52 smithi069 ceph-mon[31816]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:52.701 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:52 smithi069 ceph-mon[31816]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T23:03:52.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:52 smithi044 ceph-mon[27934]: purged_snaps scrub starts 2024-04-21T23:03:52.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:52 smithi044 ceph-mon[27934]: purged_snaps scrub ok 2024-04-21T23:03:52.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:52 smithi044 ceph-mon[27934]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-04-21T23:03:52.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:52 smithi044 ceph-mon[27934]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T23:03:53.959 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:53 smithi069 ceph-mon[31816]: Detected new or changed devices on smithi055 2024-04-21T23:03:53.959 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:53 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:53.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:53 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:53.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:53 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:53.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:53 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:53.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:53 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:53.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:53 smithi069 ceph-mon[31816]: Adjusting osd_memory_target on smithi055 to 5697M 2024-04-21T23:03:53.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:53 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:53.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:53 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:53.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:53 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:03:53.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:53 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:53.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:53 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:03:53.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:53 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:53.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:53 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:03:53.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:53 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:53.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:53 smithi069 ceph-mon[31816]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T23:03:53.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.971+0000 7f6a49b45700 1 -- 172.21.15.69:0/2103897605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44103e30 msgr2=0x7f6a4410a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:03:53.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.971+0000 7f6a49b45700 1 --2- 172.21.15.69:0/2103897605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44103e30 0x7f6a4410a6e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400b0a0 tx=0x7f6a3400b3b0 comp rx=0 tx=0).stop 2024-04-21T23:03:53.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.971+0000 7f6a49b45700 1 -- 172.21.15.69:0/2103897605 shutdown_connections 2024-04-21T23:03:53.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.971+0000 7f6a49b45700 1 --2- 172.21.15.69:0/2103897605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44103e30 0x7f6a4410a6e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:53.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.971+0000 7f6a49b45700 1 --2- 172.21.15.69:0/2103897605 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a44102290 0x7f6a441026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:53.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.971+0000 7f6a49b45700 1 --2- 172.21.15.69:0/2103897605 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a44103490 0x7f6a441038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:53.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.971+0000 7f6a49b45700 1 -- 172.21.15.69:0/2103897605 >> 172.21.15.69:0/2103897605 conn(0x7f6a440fd830 msgr2=0x7f6a440ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:03:53.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.971+0000 7f6a49b45700 1 -- 172.21.15.69:0/2103897605 shutdown_connections 2024-04-21T23:03:53.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.971+0000 7f6a49b45700 1 -- 172.21.15.69:0/2103897605 wait complete. 2024-04-21T23:03:53.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.972+0000 7f6a49b45700 1 Processor -- start 2024-04-21T23:03:53.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.972+0000 7f6a49b45700 1 -- start start 2024-04-21T23:03:53.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.972+0000 7f6a49b45700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44102290 0x7f6a44075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:03:53.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.972+0000 7f6a49b45700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a44103490 0x7f6a44074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:03:53.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.972+0000 7f6a49b45700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a44103e30 0x7f6a44074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:03:53.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.972+0000 7f6a49b45700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4410df70 con 0x7f6a44103e30 2024-04-21T23:03:53.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.972+0000 7f6a49b45700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4410ddf0 con 0x7f6a44103490 2024-04-21T23:03:53.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.972+0000 7f6a49b45700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4410e0f0 con 0x7f6a44102290 2024-04-21T23:03:53.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.972+0000 7f6a437fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44102290 0x7f6a44075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:03:53.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.972+0000 7f6a437fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44102290 0x7f6a44075fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44214/0 (socket says 172.21.15.69:44214) 2024-04-21T23:03:53.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.972+0000 7f6a437fe700 1 -- 172.21.15.69:0/2534402459 learned_addr learned my addr 172.21.15.69:0/2534402459 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-21T23:03:53.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.972+0000 7f6a43fff700 1 --2- 172.21.15.69:0/2534402459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a44103e30 0x7f6a44074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:03:53.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.972+0000 7f6a42ffd700 1 --2- 172.21.15.69:0/2534402459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a44103490 0x7f6a44074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:03:53.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.972+0000 7f6a437fe700 1 -- 172.21.15.69:0/2534402459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a44103490 msgr2=0x7f6a44074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:03:53.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.972+0000 7f6a437fe700 1 --2- 172.21.15.69:0/2534402459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a44103490 0x7f6a44074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:53.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.973+0000 7f6a437fe700 1 -- 172.21.15.69:0/2534402459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a44103e30 msgr2=0x7f6a44074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:03:53.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.973+0000 7f6a437fe700 1 --2- 172.21.15.69:0/2534402459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a44103e30 0x7f6a44074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:03:53.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.973+0000 7f6a437fe700 1 -- 172.21.15.69:0/2534402459 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a44075410 con 0x7f6a44102290 2024-04-21T23:03:53.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.973+0000 7f6a437fe700 1 --2- 172.21.15.69:0/2534402459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44102290 0x7f6a44075fd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c00b500 tx=0x7f6a3c00b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:03:53.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.973+0000 7f6a40ff9700 1 -- 172.21.15.69:0/2534402459 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c015020 con 0x7f6a44102290 2024-04-21T23:03:53.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.973+0000 7f6a49b45700 1 -- 172.21.15.69:0/2534402459 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a441a2750 con 0x7f6a44102290 2024-04-21T23:03:53.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.973+0000 7f6a40ff9700 1 -- 172.21.15.69:0/2534402459 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a3c011d10 con 0x7f6a44102290 2024-04-21T23:03:53.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.973+0000 7f6a40ff9700 1 -- 172.21.15.69:0/2534402459 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c010e50 con 0x7f6a44102290 2024-04-21T23:03:53.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.973+0000 7f6a49b45700 1 -- 172.21.15.69:0/2534402459 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a441a2ce0 con 0x7f6a44102290 2024-04-21T23:03:53.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.974+0000 7f6a49b45700 1 -- 172.21.15.69:0/2534402459 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a44068f20 con 0x7f6a44102290 2024-04-21T23:03:53.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.975+0000 7f6a40ff9700 1 -- 172.21.15.69:0/2534402459 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6a3c018020 con 0x7f6a44102290 2024-04-21T23:03:53.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.976+0000 7f6a40ff9700 1 --2- 172.21.15.69:0/2534402459 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6a2c0a2dc0 0x7f6a2c0a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:03:53.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.976+0000 7f6a40ff9700 1 -- 172.21.15.69:0/2534402459 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7f6a3c0ce200 con 0x7f6a44102290 2024-04-21T23:03:53.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.976+0000 7f6a42ffd700 1 --2- 172.21.15.69:0/2534402459 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6a2c0a2dc0 0x7f6a2c0a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:03:53.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.976+0000 7f6a42ffd700 1 --2- 172.21.15.69:0/2534402459 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6a2c0a2dc0 0x7f6a2c0a5280 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6a38006fd0 tx=0x7f6a38008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:03:53.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:53.979+0000 7f6a40ff9700 1 -- 172.21.15.69:0/2534402459 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6a3c01f020 con 0x7f6a44102290 2024-04-21T23:03:54.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:53 smithi044 ceph-mon[27934]: Detected new or changed devices on smithi055 2024-04-21T23:03:54.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:53 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:54.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:53 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:54.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:53 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:54.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:53 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:54.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:53 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:54.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:53 smithi044 ceph-mon[27934]: Adjusting osd_memory_target on smithi055 to 5697M 2024-04-21T23:03:54.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:53 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:54.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:53 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:54.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:53 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:03:54.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:53 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:54.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:53 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:03:54.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:53 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:54.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:53 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:03:54.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:53 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:54.076 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:53 smithi044 ceph-mon[27934]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T23:03:54.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:03:54.181+0000 7f6a49b45700 1 -- 172.21.15.69:0/2534402459 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f6a44062670 con 0x7f6a2c0a2dc0 2024-04-21T23:03:54.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:53 smithi055 ceph-mon[31117]: Detected new or changed devices on smithi055 2024-04-21T23:03:54.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:53 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:54.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:53 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:54.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:53 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:54.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:53 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:54.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:53 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:03:54.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:53 smithi055 ceph-mon[31117]: Adjusting osd_memory_target on smithi055 to 5697M 2024-04-21T23:03:54.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:53 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:54.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:53 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:54.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:53 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:03:54.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:53 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:54.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:53 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:03:54.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:53 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:54.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:53 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:03:54.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:53 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:03:54.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:53 smithi055 ceph-mon[31117]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T23:03:55.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:54 smithi069 ceph-mon[31816]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:03:55.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:54 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:03:55.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:54 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:03:55.072 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:54 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:55.072 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:54 smithi069 ceph-mon[31816]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T23:03:55.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:54 smithi044 ceph-mon[27934]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:03:55.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:03:55.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:03:55.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:55.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:54 smithi044 ceph-mon[27934]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T23:03:55.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:54 smithi055 ceph-mon[31117]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:03:55.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:54 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:03:55.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:54 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:03:55.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:54 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:03:55.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:54 smithi055 ceph-mon[31117]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T23:03:56.026 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:55 smithi069 ceph-mon[31816]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:03:56.026 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:55 smithi069 ceph-mon[31816]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T23:03:56.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:55 smithi044 ceph-mon[27934]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:03:56.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:55 smithi044 ceph-mon[27934]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T23:03:56.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:55 smithi055 ceph-mon[31117]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:03:56.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:55 smithi055 ceph-mon[31117]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T23:03:57.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:56 smithi044 ceph-mon[27934]: pgmap v180: 1 pgs: 1 remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:03:57.210 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:56 smithi069 ceph-mon[31816]: pgmap v180: 1 pgs: 1 remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:03:57.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:56 smithi055 ceph-mon[31117]: pgmap v180: 1 pgs: 1 remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:03:58.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:57 smithi044 ceph-mon[27934]: from='client.? 172.21.15.69:0/1779123768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1633bc7-35b7-491c-be5c-4375af06fd67"}]: dispatch 2024-04-21T23:03:58.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:57 smithi044 ceph-mon[27934]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1633bc7-35b7-491c-be5c-4375af06fd67"}]: dispatch 2024-04-21T23:03:58.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:57 smithi044 ceph-mon[27934]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1633bc7-35b7-491c-be5c-4375af06fd67"}]': finished 2024-04-21T23:03:58.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:57 smithi044 ceph-mon[27934]: osdmap e56: 9 total, 8 up, 9 in 2024-04-21T23:03:58.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:57 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:03:58.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:57 smithi044 ceph-mon[27934]: from='client.? 172.21.15.69:0/3325580336' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:03:58.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:57 smithi055 ceph-mon[31117]: from='client.? 172.21.15.69:0/1779123768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1633bc7-35b7-491c-be5c-4375af06fd67"}]: dispatch 2024-04-21T23:03:58.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:57 smithi055 ceph-mon[31117]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1633bc7-35b7-491c-be5c-4375af06fd67"}]: dispatch 2024-04-21T23:03:58.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:57 smithi055 ceph-mon[31117]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1633bc7-35b7-491c-be5c-4375af06fd67"}]': finished 2024-04-21T23:03:58.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:57 smithi055 ceph-mon[31117]: osdmap e56: 9 total, 8 up, 9 in 2024-04-21T23:03:58.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:57 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:03:58.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:57 smithi055 ceph-mon[31117]: from='client.? 172.21.15.69:0/3325580336' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:03:58.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:57 smithi069 ceph-mon[31816]: from='client.? 172.21.15.69:0/1779123768' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1633bc7-35b7-491c-be5c-4375af06fd67"}]: dispatch 2024-04-21T23:03:58.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:57 smithi069 ceph-mon[31816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1633bc7-35b7-491c-be5c-4375af06fd67"}]: dispatch 2024-04-21T23:03:58.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:57 smithi069 ceph-mon[31816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1633bc7-35b7-491c-be5c-4375af06fd67"}]': finished 2024-04-21T23:03:58.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:57 smithi069 ceph-mon[31816]: osdmap e56: 9 total, 8 up, 9 in 2024-04-21T23:03:58.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:57 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:03:58.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:57 smithi069 ceph-mon[31816]: from='client.? 172.21.15.69:0/3325580336' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:03:59.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:03:58 smithi044 ceph-mon[27934]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:03:59.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:03:58 smithi055 ceph-mon[31117]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:03:59.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:03:58 smithi069 ceph-mon[31816]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:01.100 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:00 smithi069 ceph-mon[31816]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:01.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:00 smithi044 ceph-mon[27934]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:01.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:00 smithi055 ceph-mon[31117]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:03.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:02 smithi044 ceph-mon[27934]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:03.210 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:02 smithi069 ceph-mon[31816]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:03.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:02 smithi055 ceph-mon[31117]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:04.942 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:04 smithi069 ceph-mon[31816]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:04.942 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:04 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T23:04:04.942 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:04 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:05.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:04 smithi044 ceph-mon[27934]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:05.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:04 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T23:04:05.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:04 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:05.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:04 smithi055 ceph-mon[31117]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:05.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:04 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T23:04:05.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:04 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:05.978 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:05 smithi069 ceph-mon[31816]: Deploying daemon osd.8 on smithi069 2024-04-21T23:04:06.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:05 smithi044 ceph-mon[27934]: Deploying daemon osd.8 on smithi069 2024-04-21T23:04:06.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:05 smithi055 ceph-mon[31117]: Deploying daemon osd.8 on smithi069 2024-04-21T23:04:07.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:06 smithi044 ceph-mon[27934]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:07.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:06 smithi055 ceph-mon[31117]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:07.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:06 smithi069 ceph-mon[31816]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:09.132 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:08 smithi069 ceph-mon[31816]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:09.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:08 smithi044 ceph-mon[27934]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:09.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:08 smithi055 ceph-mon[31117]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:10.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:10 smithi069 ceph-mon[31816]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:10.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:10 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:10.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:10 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:10.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:10 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:04:10.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:10 smithi044 ceph-mon[27934]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:10.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:10 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:10.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:10 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:10.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:10 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:04:10.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:10 smithi055 ceph-mon[31117]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:10.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:10 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:10.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:10 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:10.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:10 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:04:12.654 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:12 smithi069 ceph-mon[31816]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:12.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:12 smithi044 ceph-mon[27934]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:12.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:12 smithi055 ceph-mon[31117]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:13.000 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 8 on host 'smithi069' 2024-04-21T23:04:13.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:04:13.001+0000 7f6a40ff9700 1 -- 172.21.15.69:0/2534402459 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6a44062670 con 0x7f6a2c0a2dc0 2024-04-21T23:04:13.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:04:13.003+0000 7f6a49b45700 1 -- 172.21.15.69:0/2534402459 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6a2c0a2dc0 msgr2=0x7f6a2c0a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:04:13.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:04:13.003+0000 7f6a49b45700 1 --2- 172.21.15.69:0/2534402459 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6a2c0a2dc0 0x7f6a2c0a5280 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6a38006fd0 tx=0x7f6a38008040 comp rx=0 tx=0).stop 2024-04-21T23:04:13.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:04:13.003+0000 7f6a49b45700 1 -- 172.21.15.69:0/2534402459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44102290 msgr2=0x7f6a44075fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:04:13.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:04:13.003+0000 7f6a49b45700 1 --2- 172.21.15.69:0/2534402459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44102290 0x7f6a44075fd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c00b500 tx=0x7f6a3c00b810 comp rx=0 tx=0).stop 2024-04-21T23:04:13.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:04:13.003+0000 7f6a49b45700 1 -- 172.21.15.69:0/2534402459 shutdown_connections 2024-04-21T23:04:13.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:04:13.003+0000 7f6a49b45700 1 --2- 172.21.15.69:0/2534402459 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6a2c0a2dc0 0x7f6a2c0a5280 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:04:13.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:04:13.003+0000 7f6a49b45700 1 --2- 172.21.15.69:0/2534402459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a44103e30 0x7f6a44074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:04:13.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:04:13.003+0000 7f6a49b45700 1 --2- 172.21.15.69:0/2534402459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44102290 0x7f6a44075fd0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:04:13.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:04:13.003+0000 7f6a49b45700 1 --2- 172.21.15.69:0/2534402459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a44103490 0x7f6a44074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:04:13.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:04:13.003+0000 7f6a49b45700 1 -- 172.21.15.69:0/2534402459 >> 172.21.15.69:0/2534402459 conn(0x7f6a440fd830 msgr2=0x7f6a44106810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:04:13.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:04:13.003+0000 7f6a49b45700 1 -- 172.21.15.69:0/2534402459 shutdown_connections 2024-04-21T23:04:13.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:04:13.003+0000 7f6a49b45700 1 -- 172.21.15.69:0/2534402459 wait complete. 2024-04-21T23:04:13.690 DEBUG:teuthology.orchestra.run.smithi069:osd.8> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.8.service 2024-04-21T23:04:13.693 INFO:tasks.cephadm:Deploying osd.9 on smithi044 with /dev/vg_nvme/lv_1... 2024-04-21T23:04:13.693 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T23:04:13.957 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:04:14.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:13 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:14.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:13 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:14.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:13 smithi044 ceph-mon[27934]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:14.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:13 smithi044 ceph-mon[27934]: from='osd.8 [v2:172.21.15.69:6816/635270546,v1:172.21.15.69:6817/635270546]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T23:04:14.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:13 smithi044 ceph-mon[27934]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T23:04:14.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:13 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:14.080 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:13 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:14.418 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:13 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:14.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:13 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:14.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:13 smithi069 ceph-mon[31816]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:14.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:13 smithi069 ceph-mon[31816]: from='osd.8 [v2:172.21.15.69:6816/635270546,v1:172.21.15.69:6817/635270546]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T23:04:14.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:13 smithi069 ceph-mon[31816]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T23:04:14.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:13 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:14.419 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:13 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:14.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:13 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:14.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:13 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:14.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:13 smithi055 ceph-mon[31117]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:14.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:13 smithi055 ceph-mon[31117]: from='osd.8 [v2:172.21.15.69:6816/635270546,v1:172.21.15.69:6817/635270546]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T23:04:14.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:13 smithi055 ceph-mon[31117]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T23:04:14.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:13 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:14.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:13 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:15.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:15 smithi044 ceph-mon[27934]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T23:04:15.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:15 smithi044 ceph-mon[27934]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T23:04:15.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:15 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:04:15.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:15 smithi044 ceph-mon[27934]: from='osd.8 [v2:172.21.15.69:6816/635270546,v1:172.21.15.69:6817/635270546]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:04:15.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:15 smithi044 ceph-mon[27934]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:04:15.430 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:15 smithi069 ceph-mon[31816]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T23:04:15.430 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:15 smithi069 ceph-mon[31816]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T23:04:15.430 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:15 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:04:15.430 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:15 smithi069 ceph-mon[31816]: from='osd.8 [v2:172.21.15.69:6816/635270546,v1:172.21.15.69:6817/635270546]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:04:15.430 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:15 smithi069 ceph-mon[31816]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:04:15.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:15 smithi055 ceph-mon[31117]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T23:04:15.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:15 smithi055 ceph-mon[31117]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T23:04:15.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:15 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:04:15.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:15 smithi055 ceph-mon[31117]: from='osd.8 [v2:172.21.15.69:6816/635270546,v1:172.21.15.69:6817/635270546]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:04:15.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:15 smithi055 ceph-mon[31117]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:04:16.011 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:04:15 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:04:15.767+0000 7f810a1eb700 -1 osd.8 0 waiting for initial osdmap 2024-04-21T23:04:16.012 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:04:15 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:04:15.780+0000 7f81017d5700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:04:16.165 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T23:04:16.166 INFO:teuthology.orchestra.run.smithi044.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T23:04:16.166 INFO:teuthology.orchestra.run.smithi044.stderr: stderr: 10+0 records in 2024-04-21T23:04:16.166 INFO:teuthology.orchestra.run.smithi044.stderr:10+0 records out 2024-04-21T23:04:16.166 INFO:teuthology.orchestra.run.smithi044.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110261 s, 951 MB/s 2024-04-21T23:04:16.166 INFO:teuthology.orchestra.run.smithi044.stderr:--> Zapping successful for: 2024-04-21T23:04:16.190 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:16 smithi044 ceph-mon[27934]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-21T23:04:16.190 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:16 smithi044 ceph-mon[27934]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T23:04:16.190 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:16 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:04:16.190 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:16 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:04:16.190 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:16 smithi044 ceph-mon[27934]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:16.190 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:16 smithi044 ceph-mon[27934]: from='osd.8 ' entity='osd.8' 2024-04-21T23:04:16.190 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:16 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:04:16.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:16 smithi055 ceph-mon[31117]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-21T23:04:16.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:16 smithi055 ceph-mon[31117]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T23:04:16.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:04:16.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:04:16.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:16 smithi055 ceph-mon[31117]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:16.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:16 smithi055 ceph-mon[31117]: from='osd.8 ' entity='osd.8' 2024-04-21T23:04:16.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:04:16.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:16 smithi069 ceph-mon[31816]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-21T23:04:16.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:16 smithi069 ceph-mon[31816]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T23:04:16.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:16 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:04:16.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:16 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:04:16.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:16 smithi069 ceph-mon[31816]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-04-21T23:04:16.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:16 smithi069 ceph-mon[31816]: from='osd.8 ' entity='osd.8' 2024-04-21T23:04:16.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:16 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:04:16.937 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi044:vg_nvme/lv_1 2024-04-21T23:04:17.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:17 smithi044 ceph-mon[27934]: purged_snaps scrub starts 2024-04-21T23:04:17.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:17 smithi044 ceph-mon[27934]: purged_snaps scrub ok 2024-04-21T23:04:17.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:17 smithi044 ceph-mon[27934]: osd.8 [v2:172.21.15.69:6816/635270546,v1:172.21.15.69:6817/635270546] boot 2024-04-21T23:04:17.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:17 smithi044 ceph-mon[27934]: osdmap e59: 9 total, 9 up, 9 in 2024-04-21T23:04:17.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:17 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:04:17.186 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:04:17.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:17 smithi069 ceph-mon[31816]: purged_snaps scrub starts 2024-04-21T23:04:17.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:17 smithi069 ceph-mon[31816]: purged_snaps scrub ok 2024-04-21T23:04:17.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:17 smithi069 ceph-mon[31816]: osd.8 [v2:172.21.15.69:6816/635270546,v1:172.21.15.69:6817/635270546] boot 2024-04-21T23:04:17.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:17 smithi069 ceph-mon[31816]: osdmap e59: 9 total, 9 up, 9 in 2024-04-21T23:04:17.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:17 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:04:17.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:17 smithi055 ceph-mon[31117]: purged_snaps scrub starts 2024-04-21T23:04:17.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:17 smithi055 ceph-mon[31117]: purged_snaps scrub ok 2024-04-21T23:04:17.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:17 smithi055 ceph-mon[31117]: osd.8 [v2:172.21.15.69:6816/635270546,v1:172.21.15.69:6817/635270546] boot 2024-04-21T23:04:17.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:17 smithi055 ceph-mon[31117]: osdmap e59: 9 total, 9 up, 9 in 2024-04-21T23:04:17.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:17 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:04:18.426 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:18 smithi044 ceph-mon[27934]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T23:04:18.426 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:18 smithi044 ceph-mon[27934]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:18.427 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:18 smithi044 ceph-mon[27934]: Detected new or changed devices on smithi069 2024-04-21T23:04:18.427 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:18 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:18.427 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:18 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:18.427 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:18 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:18.427 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:18 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:18.427 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:18 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:18.427 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:18 smithi044 ceph-mon[27934]: Adjusting osd_memory_target on smithi069 to 5696M 2024-04-21T23:04:18.427 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:18 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:18.427 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:18 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:18.427 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:18 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:04:18.427 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:18 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:18.428 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:18 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:04:18.428 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:18 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:18.428 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:18 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:04:18.428 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:18 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:18.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:18 smithi055 ceph-mon[31117]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T23:04:18.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:18 smithi055 ceph-mon[31117]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:18.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:18 smithi055 ceph-mon[31117]: Detected new or changed devices on smithi069 2024-04-21T23:04:18.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:18 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:18.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:18 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:18.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:18 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:18.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:18 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:18.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:18 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:18.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:18 smithi055 ceph-mon[31117]: Adjusting osd_memory_target on smithi069 to 5696M 2024-04-21T23:04:18.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:18 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:18.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:18 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:18.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:18 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:04:18.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:18 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:18.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:18 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:04:18.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:18 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:18.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:18 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:04:18.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:18 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:18.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:18 smithi069 ceph-mon[31816]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T23:04:18.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:18 smithi069 ceph-mon[31816]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:18.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:18 smithi069 ceph-mon[31816]: Detected new or changed devices on smithi069 2024-04-21T23:04:18.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:18 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:18.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:18 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:18.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:18 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:18.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:18 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:18.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:18 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:18.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:18 smithi069 ceph-mon[31816]: Adjusting osd_memory_target on smithi069 to 5696M 2024-04-21T23:04:18.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:18 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:18.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:18 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:18.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:18 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:04:18.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:18 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:18.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:18 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:04:18.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:18 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:18.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:18 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:04:18.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:18 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:18.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.976+0000 7f206f018700 1 -- 172.21.15.44:0/494262282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2068102290 msgr2=0x7f20681026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:04:18.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.976+0000 7f206f018700 1 --2- 172.21.15.44:0/494262282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2068102290 0x7f20681026b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f205c0099c0 tx=0x7f205c009cd0 comp rx=0 tx=0).stop 2024-04-21T23:04:18.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.976+0000 7f206f018700 1 -- 172.21.15.44:0/494262282 shutdown_connections 2024-04-21T23:04:18.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.976+0000 7f206f018700 1 --2- 172.21.15.44:0/494262282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2068103e30 0x7f206810a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:04:18.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.976+0000 7f206f018700 1 --2- 172.21.15.44:0/494262282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2068102290 0x7f20681026b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:04:18.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.976+0000 7f206f018700 1 --2- 172.21.15.44:0/494262282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2068103490 0x7f20681038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:04:18.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.976+0000 7f206f018700 1 -- 172.21.15.44:0/494262282 >> 172.21.15.44:0/494262282 conn(0x7f20680fd830 msgr2=0x7f20680ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:04:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.976+0000 7f206f018700 1 -- 172.21.15.44:0/494262282 shutdown_connections 2024-04-21T23:04:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.976+0000 7f206f018700 1 -- 172.21.15.44:0/494262282 wait complete. 2024-04-21T23:04:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.976+0000 7f206f018700 1 Processor -- start 2024-04-21T23:04:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.976+0000 7f206f018700 1 -- start start 2024-04-21T23:04:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.977+0000 7f206f018700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2068102290 0x7f20681984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:04:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.977+0000 7f206f018700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2068103490 0x7f20681989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:04:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.977+0000 7f206f018700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2068103e30 0x7f206819fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:04:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.977+0000 7f206f018700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f206810de70 con 0x7f2068103490 2024-04-21T23:04:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.977+0000 7f206f018700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f206810dcf0 con 0x7f2068103e30 2024-04-21T23:04:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.977+0000 7f206f018700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f206810dff0 con 0x7f2068102290 2024-04-21T23:04:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.977+0000 7f2067fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2068103490 0x7f20681989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:04:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.977+0000 7f2067fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2068103490 0x7f20681989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35224/0 (socket says 172.21.15.44:35224) 2024-04-21T23:04:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.977+0000 7f2067fff700 1 -- 172.21.15.44:0/2279415945 learned_addr learned my addr 172.21.15.44:0/2279415945 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:04:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.977+0000 7f206cdb4700 1 --2- 172.21.15.44:0/2279415945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2068102290 0x7f20681984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:04:18.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.977+0000 7f206d5b5700 1 --2- 172.21.15.44:0/2279415945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2068103e30 0x7f206819fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:04:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.977+0000 7f2067fff700 1 -- 172.21.15.44:0/2279415945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2068102290 msgr2=0x7f20681984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:04:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.977+0000 7f2067fff700 1 --2- 172.21.15.44:0/2279415945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2068102290 0x7f20681984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:04:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.977+0000 7f2067fff700 1 -- 172.21.15.44:0/2279415945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2068103e30 msgr2=0x7f206819fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:04:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.977+0000 7f2067fff700 1 --2- 172.21.15.44:0/2279415945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2068103e30 0x7f206819fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:04:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.977+0000 7f2067fff700 1 -- 172.21.15.44:0/2279415945 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20681a0110 con 0x7f2068103490 2024-04-21T23:04:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.977+0000 7f206d5b5700 1 --2- 172.21.15.44:0/2279415945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2068103e30 0x7f206819fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:04:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.977+0000 7f2067fff700 1 --2- 172.21.15.44:0/2279415945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2068103490 0x7f20681989f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f205400c9a0 tx=0x7f205400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:04:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.977+0000 7f2065ffb700 1 -- 172.21.15.44:0/2279415945 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2054007a60 con 0x7f2068103490 2024-04-21T23:04:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.978+0000 7f2065ffb700 1 -- 172.21.15.44:0/2279415945 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2054004d10 con 0x7f2068103490 2024-04-21T23:04:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.978+0000 7f206f018700 1 -- 172.21.15.44:0/2279415945 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20681a0400 con 0x7f2068103490 2024-04-21T23:04:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.978+0000 7f2065ffb700 1 -- 172.21.15.44:0/2279415945 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2054005690 con 0x7f2068103490 2024-04-21T23:04:18.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.978+0000 7f206f018700 1 -- 172.21.15.44:0/2279415945 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2068074630 con 0x7f2068103490 2024-04-21T23:04:18.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.979+0000 7f206f018700 1 -- 172.21.15.44:0/2279415945 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2068068f20 con 0x7f2068103490 2024-04-21T23:04:18.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.980+0000 7f2065ffb700 1 -- 172.21.15.44:0/2279415945 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2054007c00 con 0x7f2068103490 2024-04-21T23:04:18.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.981+0000 7f2065ffb700 1 --2- 172.21.15.44:0/2279415945 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f20580a2c20 0x7f20580a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:04:18.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.981+0000 7f206cdb4700 1 --2- 172.21.15.44:0/2279415945 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f20580a2c20 0x7f20580a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:04:18.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.981+0000 7f2065ffb700 1 -- 172.21.15.44:0/2279415945 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5858+0+0 (secure 0 0 0) 0x7f20540ce700 con 0x7f2068103490 2024-04-21T23:04:18.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.981+0000 7f206cdb4700 1 --2- 172.21.15.44:0/2279415945 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f20580a2c20 0x7f20580a50e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f205c003890 tx=0x7f205c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:04:18.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:18.985+0000 7f2065ffb700 1 -- 172.21.15.44:0/2279415945 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f205409c110 con 0x7f2068103490 2024-04-21T23:04:19.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:19.204+0000 7f206f018700 1 -- 172.21.15.44:0/2279415945 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2068062670 con 0x7f20580a2c20 2024-04-21T23:04:19.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:19 smithi044 ceph-mon[27934]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T23:04:19.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:19 smithi055 ceph-mon[31117]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T23:04:19.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:19 smithi069 ceph-mon[31816]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T23:04:20.097 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:20 smithi044 ceph-mon[27934]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:20.097 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:20 smithi044 ceph-mon[27934]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T23:04:20.097 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:20 smithi044 ceph-mon[27934]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:04:20.097 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:20 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:04:20.097 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:20 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:04:20.097 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:20 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:20.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:20 smithi069 ceph-mon[31816]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:20.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:20 smithi069 ceph-mon[31816]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T23:04:20.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:20 smithi069 ceph-mon[31816]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:04:20.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:20 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:04:20.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:20 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:04:20.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:20 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:20.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:20 smithi055 ceph-mon[31117]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:20.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:20 smithi055 ceph-mon[31117]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T23:04:20.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:20 smithi055 ceph-mon[31117]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi044:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:04:20.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:20 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:04:20.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:20 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:04:20.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:20 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:21.143 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:21 smithi044 ceph-mon[27934]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T23:04:21.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:21 smithi055 ceph-mon[31117]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T23:04:21.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:21 smithi069 ceph-mon[31816]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T23:04:22.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:22 smithi044 ceph-mon[27934]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:22.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:22 smithi055 ceph-mon[31117]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:22.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:22 smithi069 ceph-mon[31816]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:23.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:23 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1098690731' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc0ed998-2d10-4bc6-9a3a-d6afe9135386"}]: dispatch 2024-04-21T23:04:23.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:23 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1098690731' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc0ed998-2d10-4bc6-9a3a-d6afe9135386"}]': finished 2024-04-21T23:04:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:23 smithi044 ceph-mon[27934]: osdmap e64: 10 total, 9 up, 10 in 2024-04-21T23:04:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:04:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:23 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1338826734' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:04:23.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:23 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1098690731' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc0ed998-2d10-4bc6-9a3a-d6afe9135386"}]: dispatch 2024-04-21T23:04:23.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:23 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1098690731' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc0ed998-2d10-4bc6-9a3a-d6afe9135386"}]': finished 2024-04-21T23:04:23.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:23 smithi055 ceph-mon[31117]: osdmap e64: 10 total, 9 up, 10 in 2024-04-21T23:04:23.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:04:23.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:23 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1338826734' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:04:23.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:23 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1098690731' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc0ed998-2d10-4bc6-9a3a-d6afe9135386"}]: dispatch 2024-04-21T23:04:23.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:23 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1098690731' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc0ed998-2d10-4bc6-9a3a-d6afe9135386"}]': finished 2024-04-21T23:04:23.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:23 smithi069 ceph-mon[31816]: osdmap e64: 10 total, 9 up, 10 in 2024-04-21T23:04:23.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:04:23.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:23 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1338826734' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:04:24.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:24 smithi044 ceph-mon[27934]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-21T23:04:24.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:24 smithi055 ceph-mon[31117]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-21T23:04:24.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:24 smithi069 ceph-mon[31816]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-21T23:04:26.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:26 smithi044 ceph-mon[27934]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T23:04:26.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:26 smithi069 ceph-mon[31816]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T23:04:26.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:26 smithi055 ceph-mon[31117]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T23:04:28.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:28 smithi044 ceph-mon[27934]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T23:04:28.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:28 smithi069 ceph-mon[31816]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T23:04:28.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:28 smithi055 ceph-mon[31117]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T23:04:30.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:30 smithi044 ceph-mon[27934]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-21T23:04:30.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:30 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T23:04:30.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:30 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:30.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:30 smithi044 ceph-mon[27934]: Deploying daemon osd.9 on smithi044 2024-04-21T23:04:30.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:30 smithi055 ceph-mon[31117]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-21T23:04:30.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:30 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T23:04:30.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:30 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:30.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:30 smithi055 ceph-mon[31117]: Deploying daemon osd.9 on smithi044 2024-04-21T23:04:30.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:30 smithi069 ceph-mon[31816]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-21T23:04:30.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:30 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T23:04:30.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:30 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:30.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:30 smithi069 ceph-mon[31816]: Deploying daemon osd.9 on smithi044 2024-04-21T23:04:32.288 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:32 smithi044 ceph-mon[27934]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:04:32.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:32 smithi055 ceph-mon[31117]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:04:32.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:32 smithi069 ceph-mon[31816]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T23:04:34.430 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:34 smithi044 ceph-mon[27934]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:34.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:34 smithi055 ceph-mon[31117]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:34.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:34 smithi069 ceph-mon[31816]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:37.167 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:36 smithi055 ceph-mon[31117]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:37.167 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:36 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:37.167 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:36 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:37.167 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:36 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:04:37.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:36 smithi044 ceph-mon[27934]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:37.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:36 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:37.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:36 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:37.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:36 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:04:37.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:36 smithi069 ceph-mon[31816]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:37.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:36 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:37.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:36 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:37.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:36 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:04:38.897 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:38 smithi044 ceph-mon[27934]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:39.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:38 smithi055 ceph-mon[31117]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:39.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:38 smithi069 ceph-mon[31816]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:39.664 INFO:teuthology.orchestra.run.smithi044.stdout:Created osd(s) 9 on host 'smithi044' 2024-04-21T23:04:39.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:39.665+0000 7f2065ffb700 1 -- 172.21.15.44:0/2279415945 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2068062670 con 0x7f20580a2c20 2024-04-21T23:04:39.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:39.668+0000 7f206f018700 1 -- 172.21.15.44:0/2279415945 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f20580a2c20 msgr2=0x7f20580a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:04:39.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:39.668+0000 7f206f018700 1 --2- 172.21.15.44:0/2279415945 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f20580a2c20 0x7f20580a50e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f205c003890 tx=0x7f205c005fb0 comp rx=0 tx=0).stop 2024-04-21T23:04:39.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:39.668+0000 7f206f018700 1 -- 172.21.15.44:0/2279415945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2068103490 msgr2=0x7f20681989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:04:39.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:39.668+0000 7f206f018700 1 --2- 172.21.15.44:0/2279415945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2068103490 0x7f20681989f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f205400c9a0 tx=0x7f205400ccb0 comp rx=0 tx=0).stop 2024-04-21T23:04:39.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:39.668+0000 7f206f018700 1 -- 172.21.15.44:0/2279415945 shutdown_connections 2024-04-21T23:04:39.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:39.668+0000 7f206f018700 1 --2- 172.21.15.44:0/2279415945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2068103e30 0x7f206819fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:04:39.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:39.668+0000 7f206f018700 1 --2- 172.21.15.44:0/2279415945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2068102290 0x7f20681984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:04:39.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:39.668+0000 7f206f018700 1 --2- 172.21.15.44:0/2279415945 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f20580a2c20 0x7f20580a50e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:04:39.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:39.668+0000 7f206f018700 1 --2- 172.21.15.44:0/2279415945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2068103490 0x7f20681989f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:04:39.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:39.668+0000 7f206f018700 1 -- 172.21.15.44:0/2279415945 >> 172.21.15.44:0/2279415945 conn(0x7f20680fd830 msgr2=0x7f20681067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:04:39.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:39.668+0000 7f206f018700 1 -- 172.21.15.44:0/2279415945 shutdown_connections 2024-04-21T23:04:39.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:04:39.668+0000 7f206f018700 1 -- 172.21.15.44:0/2279415945 wait complete. 2024-04-21T23:04:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:39.766 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:39.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:04:39.767 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:39.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:39.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:39.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:39 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:04:39.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:39 smithi044 ceph-mon[27934]: from='osd.9 [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T23:04:40.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:39 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:40.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:39 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:40.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:39 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:40.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:39 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:04:40.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:39 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:40.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:39 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:40.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:39 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:40.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:39 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:04:40.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:39 smithi069 ceph-mon[31816]: from='osd.9 [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T23:04:40.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:39 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:40.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:39 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:40.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:39 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:40.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:39 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:04:40.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:39 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:40.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:39 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:40.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:39 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:40.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:39 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:04:40.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:39 smithi055 ceph-mon[31117]: from='osd.9 [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T23:04:40.359 DEBUG:teuthology.orchestra.run.smithi044:osd.9> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.9.service 2024-04-21T23:04:40.362 INFO:tasks.cephadm:Deploying osd.10 on smithi055 with /dev/vg_nvme/lv_1... 2024-04-21T23:04:40.362 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T23:04:40.622 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:04:40.928 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:40 smithi044 ceph-mon[27934]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:40.928 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:40 smithi044 ceph-mon[27934]: from='osd.9 [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T23:04:40.928 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:40 smithi044 ceph-mon[27934]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T23:04:40.928 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:40 smithi044 ceph-mon[27934]: from='osd.9 [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-21T23:04:40.928 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:40 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:04:40.999 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:40 smithi055 ceph-mon[31117]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:40.999 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:40 smithi055 ceph-mon[31117]: from='osd.9 [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T23:04:40.999 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:40 smithi055 ceph-mon[31117]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T23:04:41.000 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:40 smithi055 ceph-mon[31117]: from='osd.9 [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-21T23:04:41.000 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:40 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:04:41.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:40 smithi069 ceph-mon[31816]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:41.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:40 smithi069 ceph-mon[31816]: from='osd.9 [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T23:04:41.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:40 smithi069 ceph-mon[31816]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T23:04:41.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:40 smithi069 ceph-mon[31816]: from='osd.9 [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]: dispatch 2024-04-21T23:04:41.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:40 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:04:41.921 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:04:41 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:04:41.636+0000 7f88f0e0c700 -1 osd.9 0 waiting for initial osdmap 2024-04-21T23:04:41.921 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:04:41 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:04:41.647+0000 7f88ecbff700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:04:42.730 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T23:04:42.730 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T23:04:42.730 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10+0 records in 2024-04-21T23:04:42.730 INFO:teuthology.orchestra.run.smithi055.stderr:10+0 records out 2024-04-21T23:04:42.730 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0149826 s, 700 MB/s 2024-04-21T23:04:42.730 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping successful for: 2024-04-21T23:04:42.730 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:42 smithi055 ceph-mon[31117]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:42.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:42 smithi055 ceph-mon[31117]: from='osd.9 [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-04-21T23:04:42.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:42 smithi055 ceph-mon[31117]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T23:04:42.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:42 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:04:42.731 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:42 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:04:42.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:42 smithi044 ceph-mon[27934]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:42.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:42 smithi044 ceph-mon[27934]: from='osd.9 [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-04-21T23:04:42.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:42 smithi044 ceph-mon[27934]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T23:04:42.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:42 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:04:42.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:42 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:04:42.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:42 smithi069 ceph-mon[31816]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-04-21T23:04:42.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:42 smithi069 ceph-mon[31816]: from='osd.9 [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi044", "root=default"]}]': finished 2024-04-21T23:04:42.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:42 smithi069 ceph-mon[31816]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T23:04:42.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:42 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:04:42.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:42 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:04:43.549 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi055:vg_nvme/lv_1 2024-04-21T23:04:43.797 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:04:43.920 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:43 smithi055 ceph-mon[31117]: purged_snaps scrub starts 2024-04-21T23:04:43.920 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:43 smithi055 ceph-mon[31117]: purged_snaps scrub ok 2024-04-21T23:04:43.920 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:43 smithi055 ceph-mon[31117]: osd.9 [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908] boot 2024-04-21T23:04:43.920 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:43 smithi055 ceph-mon[31117]: osdmap e67: 10 total, 10 up, 10 in 2024-04-21T23:04:43.920 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:43 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:04:43.920 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:43 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:43.920 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:43 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:43.920 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:43 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:43.920 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:43 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:43.921 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:43 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:43.921 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:43 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:43.921 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:43 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:43.921 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:43 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:43.921 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:43 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:04:43.921 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:43 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:43.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:43 smithi044 ceph-mon[27934]: purged_snaps scrub starts 2024-04-21T23:04:43.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:43 smithi044 ceph-mon[27934]: purged_snaps scrub ok 2024-04-21T23:04:43.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:43 smithi044 ceph-mon[27934]: osd.9 [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908] boot 2024-04-21T23:04:43.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:43 smithi044 ceph-mon[27934]: osdmap e67: 10 total, 10 up, 10 in 2024-04-21T23:04:43.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:43 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:04:43.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:43 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:43.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:43 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:43.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:43 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:43.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:43 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:43.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:43 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:43.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:43 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:43.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:43 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:43.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:43 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:43.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:43 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:04:43.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:43 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:43.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:43 smithi069 ceph-mon[31816]: purged_snaps scrub starts 2024-04-21T23:04:43.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:43 smithi069 ceph-mon[31816]: purged_snaps scrub ok 2024-04-21T23:04:43.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:43 smithi069 ceph-mon[31816]: osd.9 [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908] boot 2024-04-21T23:04:43.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:43 smithi069 ceph-mon[31816]: osdmap e67: 10 total, 10 up, 10 in 2024-04-21T23:04:43.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:43 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:04:43.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:43 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:43.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:43 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:43.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:43 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:43.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:43 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:43.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:43 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:43.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:43 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:04:43.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:43 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:43.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:43 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:43.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:43 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:04:43.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:43 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:04:44.883 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:44 smithi055 ceph-mon[31117]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:44.883 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:44 smithi055 ceph-mon[31117]: Detected new or changed devices on smithi044 2024-04-21T23:04:44.883 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:44 smithi055 ceph-mon[31117]: Adjusting osd_memory_target on smithi044 to 4273M 2024-04-21T23:04:44.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:44 smithi044 ceph-mon[27934]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:44.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:44 smithi044 ceph-mon[27934]: Detected new or changed devices on smithi044 2024-04-21T23:04:44.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:44 smithi044 ceph-mon[27934]: Adjusting osd_memory_target on smithi044 to 4273M 2024-04-21T23:04:44.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:44 smithi069 ceph-mon[31816]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:44.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:44 smithi069 ceph-mon[31816]: Detected new or changed devices on smithi044 2024-04-21T23:04:44.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:44 smithi069 ceph-mon[31816]: Adjusting osd_memory_target on smithi044 to 4273M 2024-04-21T23:04:45.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.506+0000 7fa85b6f4700 1 -- 172.21.15.55:0/1621939497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa854103e50 msgr2=0x7fa85410a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:04:45.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.506+0000 7fa85b6f4700 1 --2- 172.21.15.55:0/1621939497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa854103e50 0x7fa85410a700 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00b0d0 tx=0x7fa84c00b3e0 comp rx=0 tx=0).stop 2024-04-21T23:04:45.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.506+0000 7fa85b6f4700 1 -- 172.21.15.55:0/1621939497 shutdown_connections 2024-04-21T23:04:45.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.506+0000 7fa85b6f4700 1 --2- 172.21.15.55:0/1621939497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa854103e50 0x7fa85410a700 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:04:45.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.506+0000 7fa85b6f4700 1 --2- 172.21.15.55:0/1621939497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8541022b0 0x7fa8541026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:04:45.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.506+0000 7fa85b6f4700 1 --2- 172.21.15.55:0/1621939497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa8541034b0 0x7fa854103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:04:45.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.506+0000 7fa85b6f4700 1 -- 172.21.15.55:0/1621939497 >> 172.21.15.55:0/1621939497 conn(0x7fa8540fd850 msgr2=0x7fa8540ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:04:45.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.507+0000 7fa85b6f4700 1 -- 172.21.15.55:0/1621939497 shutdown_connections 2024-04-21T23:04:45.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.507+0000 7fa85b6f4700 1 -- 172.21.15.55:0/1621939497 wait complete. 2024-04-21T23:04:45.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.507+0000 7fa85b6f4700 1 Processor -- start 2024-04-21T23:04:45.509 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.507+0000 7fa85b6f4700 1 -- start start 2024-04-21T23:04:45.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.507+0000 7fa85b6f4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8541022b0 0x7fa854198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:04:45.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.507+0000 7fa85b6f4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8541034b0 0x7fa854198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:04:45.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.507+0000 7fa85b6f4700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa854103e50 0x7fa85419f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:04:45.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.507+0000 7fa85b6f4700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa85410de90 con 0x7fa854103e50 2024-04-21T23:04:45.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.507+0000 7fa85b6f4700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa85410dd10 con 0x7fa8541034b0 2024-04-21T23:04:45.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.508+0000 7fa85b6f4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa85410e010 con 0x7fa8541022b0 2024-04-21T23:04:45.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.508+0000 7fa858c8f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8541034b0 0x7fa854198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:04:45.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.508+0000 7fa858c8f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8541034b0 0x7fa854198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:34810/0 (socket says 172.21.15.55:34810) 2024-04-21T23:04:45.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.508+0000 7fa858c8f700 1 -- 172.21.15.55:0/696390630 learned_addr learned my addr 172.21.15.55:0/696390630 (peer_addr_for_me v2:172.21.15.55:0/0) 2024-04-21T23:04:45.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.508+0000 7fa859490700 1 --2- 172.21.15.55:0/696390630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8541022b0 0x7fa854198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:04:45.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.508+0000 7fa859c91700 1 --2- 172.21.15.55:0/696390630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa854103e50 0x7fa85419f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:04:45.510 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.508+0000 7fa858c8f700 1 -- 172.21.15.55:0/696390630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8541022b0 msgr2=0x7fa854198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:04:45.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.508+0000 7fa858c8f700 1 --2- 172.21.15.55:0/696390630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8541022b0 0x7fa854198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:04:45.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.508+0000 7fa858c8f700 1 -- 172.21.15.55:0/696390630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa854103e50 msgr2=0x7fa85419f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:04:45.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.508+0000 7fa858c8f700 1 --2- 172.21.15.55:0/696390630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa854103e50 0x7fa85419f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:04:45.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.508+0000 7fa858c8f700 1 -- 172.21.15.55:0/696390630 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8541a0070 con 0x7fa8541034b0 2024-04-21T23:04:45.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.508+0000 7fa858c8f700 1 --2- 172.21.15.55:0/696390630 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8541034b0 0x7fa854198950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa85000c920 tx=0x7fa85000cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:04:45.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.508+0000 7fa84a7fc700 1 -- 172.21.15.55:0/696390630 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa850004a30 con 0x7fa8541034b0 2024-04-21T23:04:45.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.508+0000 7fa84a7fc700 1 -- 172.21.15.55:0/696390630 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa850005500 con 0x7fa8541034b0 2024-04-21T23:04:45.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.508+0000 7fa85b6f4700 1 -- 172.21.15.55:0/696390630 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8541a0360 con 0x7fa8541034b0 2024-04-21T23:04:45.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.508+0000 7fa84a7fc700 1 -- 172.21.15.55:0/696390630 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa850007dd0 con 0x7fa8541034b0 2024-04-21T23:04:45.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.508+0000 7fa85b6f4700 1 -- 172.21.15.55:0/696390630 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8541a08f0 con 0x7fa8541034b0 2024-04-21T23:04:45.511 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.509+0000 7fa85b6f4700 1 -- 172.21.15.55:0/696390630 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa854068f20 con 0x7fa8541034b0 2024-04-21T23:04:45.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.511+0000 7fa84a7fc700 1 -- 172.21.15.55:0/696390630 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa850005000 con 0x7fa8541034b0 2024-04-21T23:04:45.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.511+0000 7fa84a7fc700 1 --2- 172.21.15.55:0/696390630 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa8400a2ca0 0x7fa8400a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:04:45.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.511+0000 7fa84a7fc700 1 -- 172.21.15.55:0/696390630 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7fa8500ce6c0 con 0x7fa8541034b0 2024-04-21T23:04:45.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.511+0000 7fa859490700 1 --2- 172.21.15.55:0/696390630 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa8400a2ca0 0x7fa8400a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:04:45.512 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.512+0000 7fa859490700 1 --2- 172.21.15.55:0/696390630 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa8400a2ca0 0x7fa8400a5160 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa854103310 tx=0x7fa84400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:04:45.514 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.515+0000 7fa84a7fc700 1 -- 172.21.15.55:0/696390630 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa850014030 con 0x7fa8541034b0 2024-04-21T23:04:45.730 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:04:45.730+0000 7fa85b6f4700 1 -- 172.21.15.55:0/696390630 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fa854062670 con 0x7fa8400a2ca0 2024-04-21T23:04:45.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:45 smithi044 ceph-mon[27934]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T23:04:45.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:45 smithi055 ceph-mon[31117]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T23:04:45.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:45 smithi069 ceph-mon[31816]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T23:04:46.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:46 smithi044 ceph-mon[27934]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:46.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:46 smithi044 ceph-mon[27934]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:04:46.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:46 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:04:46.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:46 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:04:46.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:46 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:46.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:46 smithi055 ceph-mon[31117]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:46.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:46 smithi055 ceph-mon[31117]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:04:46.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:46 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:04:46.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:46 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:04:46.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:46 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:46.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:46 smithi069 ceph-mon[31816]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:46.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:46 smithi069 ceph-mon[31816]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:04:46.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:46 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:04:46.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:46 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:04:46.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:46 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:48.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:48 smithi044 ceph-mon[27934]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:48.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:48 smithi069 ceph-mon[31816]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:48.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:48 smithi055 ceph-mon[31117]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:49.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:49 smithi044 ceph-mon[27934]: from='client.? 172.21.15.55:0/2730256841' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8dddb89-8c25-4df2-8993-3f61e52c53c0"}]: dispatch 2024-04-21T23:04:49.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:49 smithi044 ceph-mon[27934]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8dddb89-8c25-4df2-8993-3f61e52c53c0"}]: dispatch 2024-04-21T23:04:49.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:49 smithi044 ceph-mon[27934]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8dddb89-8c25-4df2-8993-3f61e52c53c0"}]': finished 2024-04-21T23:04:49.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:49 smithi044 ceph-mon[27934]: osdmap e69: 11 total, 10 up, 11 in 2024-04-21T23:04:49.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:49 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:04:49.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:49 smithi044 ceph-mon[27934]: from='client.? 172.21.15.55:0/2797056455' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:04:49.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:49 smithi069 ceph-mon[31816]: from='client.? 172.21.15.55:0/2730256841' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8dddb89-8c25-4df2-8993-3f61e52c53c0"}]: dispatch 2024-04-21T23:04:49.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:49 smithi069 ceph-mon[31816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8dddb89-8c25-4df2-8993-3f61e52c53c0"}]: dispatch 2024-04-21T23:04:49.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:49 smithi069 ceph-mon[31816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8dddb89-8c25-4df2-8993-3f61e52c53c0"}]': finished 2024-04-21T23:04:49.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:49 smithi069 ceph-mon[31816]: osdmap e69: 11 total, 10 up, 11 in 2024-04-21T23:04:49.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:49 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:04:49.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:49 smithi069 ceph-mon[31816]: from='client.? 172.21.15.55:0/2797056455' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:04:49.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:49 smithi055 ceph-mon[31117]: from='client.? 172.21.15.55:0/2730256841' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8dddb89-8c25-4df2-8993-3f61e52c53c0"}]: dispatch 2024-04-21T23:04:49.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:49 smithi055 ceph-mon[31117]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8dddb89-8c25-4df2-8993-3f61e52c53c0"}]: dispatch 2024-04-21T23:04:49.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:49 smithi055 ceph-mon[31117]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8dddb89-8c25-4df2-8993-3f61e52c53c0"}]': finished 2024-04-21T23:04:49.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:49 smithi055 ceph-mon[31117]: osdmap e69: 11 total, 10 up, 11 in 2024-04-21T23:04:49.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:49 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:04:49.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:49 smithi055 ceph-mon[31117]: from='client.? 172.21.15.55:0/2797056455' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:04:51.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:50 smithi044 ceph-mon[27934]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:51.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:50 smithi055 ceph-mon[31117]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:51.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:50 smithi069 ceph-mon[31816]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:53.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:52 smithi044 ceph-mon[27934]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:53.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:52 smithi069 ceph-mon[31816]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:53.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:52 smithi055 ceph-mon[31117]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:55.022 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:54 smithi055 ceph-mon[31117]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:55.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:54 smithi044 ceph-mon[27934]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:55.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:54 smithi069 ceph-mon[31816]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:57.038 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:56 smithi055 ceph-mon[31117]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:57.039 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:56 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T23:04:57.039 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:56 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:57.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:56 smithi044 ceph-mon[27934]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:57.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:56 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T23:04:57.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:56 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:57.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:56 smithi069 ceph-mon[31816]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:57.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:56 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T23:04:57.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:56 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:04:58.042 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:57 smithi055 ceph-mon[31117]: Deploying daemon osd.10 on smithi055 2024-04-21T23:04:58.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:57 smithi044 ceph-mon[27934]: Deploying daemon osd.10 on smithi055 2024-04-21T23:04:58.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:57 smithi069 ceph-mon[31816]: Deploying daemon osd.10 on smithi055 2024-04-21T23:04:58.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:04:58 smithi055 ceph-mon[31117]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:59.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:04:58 smithi044 ceph-mon[27934]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:04:59.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:04:58 smithi069 ceph-mon[31816]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:05:00.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:00 smithi055 ceph-mon[31117]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:05:01.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:00 smithi044 ceph-mon[27934]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:05:01.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:00 smithi069 ceph-mon[31816]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:05:03.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:02 smithi044 ceph-mon[27934]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:05:03.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:03.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:03.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:05:03.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:02 smithi055 ceph-mon[31117]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:05:03.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:02 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:03.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:02 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:03.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:02 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:05:03.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:02 smithi069 ceph-mon[31816]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:05:03.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:02 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:03.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:02 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:03.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:02 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:05:05.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:04 smithi044 ceph-mon[27934]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:05:05.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:04 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:05.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:04 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:05.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:04 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:05.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:04 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:05:05.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:04 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:05.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:04 smithi069 ceph-mon[31816]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:05:05.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:04 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:05.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:04 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:05.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:04 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:05.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:04 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:05:05.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:04 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:05.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:04 smithi055 ceph-mon[31117]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:05:05.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:04 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:05.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:04 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:05.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:04 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:05.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:04 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:05:05.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:04 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:05.655 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:05:05.656+0000 7fa84a7fc700 1 -- 172.21.15.55:0/696390630 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fa854062670 con 0x7fa8400a2ca0 2024-04-21T23:05:05.656 INFO:teuthology.orchestra.run.smithi055.stdout:Created osd(s) 10 on host 'smithi055' 2024-04-21T23:05:05.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:05:05.658+0000 7fa85b6f4700 1 -- 172.21.15.55:0/696390630 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa8400a2ca0 msgr2=0x7fa8400a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:05.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:05:05.658+0000 7fa85b6f4700 1 --2- 172.21.15.55:0/696390630 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa8400a2ca0 0x7fa8400a5160 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa854103310 tx=0x7fa84400b410 comp rx=0 tx=0).stop 2024-04-21T23:05:05.658 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:05:05.658+0000 7fa85b6f4700 1 -- 172.21.15.55:0/696390630 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8541034b0 msgr2=0x7fa854198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:05.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:05:05.658+0000 7fa85b6f4700 1 --2- 172.21.15.55:0/696390630 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8541034b0 0x7fa854198950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa85000c920 tx=0x7fa85000cc30 comp rx=0 tx=0).stop 2024-04-21T23:05:05.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:05:05.658+0000 7fa85b6f4700 1 -- 172.21.15.55:0/696390630 shutdown_connections 2024-04-21T23:05:05.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:05:05.658+0000 7fa85b6f4700 1 --2- 172.21.15.55:0/696390630 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa8400a2ca0 0x7fa8400a5160 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:05.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:05:05.658+0000 7fa85b6f4700 1 --2- 172.21.15.55:0/696390630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa854103e50 0x7fa85419f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:05.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:05:05.658+0000 7fa85b6f4700 1 --2- 172.21.15.55:0/696390630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8541022b0 0x7fa854198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:05.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:05:05.658+0000 7fa85b6f4700 1 --2- 172.21.15.55:0/696390630 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8541034b0 0x7fa854198950 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:05.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:05:05.658+0000 7fa85b6f4700 1 -- 172.21.15.55:0/696390630 >> 172.21.15.55:0/696390630 conn(0x7fa8540fd850 msgr2=0x7fa854106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:05.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:05:05.658+0000 7fa85b6f4700 1 -- 172.21.15.55:0/696390630 shutdown_connections 2024-04-21T23:05:05.659 INFO:teuthology.orchestra.run.smithi055.stderr:2024-04-21T23:05:05.659+0000 7fa85b6f4700 1 -- 172.21.15.55:0/696390630 wait complete. 2024-04-21T23:05:06.320 DEBUG:teuthology.orchestra.run.smithi055:osd.10> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.10.service 2024-04-21T23:05:06.322 INFO:tasks.cephadm:Deploying osd.11 on smithi069 with /dev/vg_nvme/lv_1... 2024-04-21T23:05:06.322 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T23:05:06.554 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:05:06.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:06 smithi069 ceph-mon[31816]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:05:06.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:06 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:06.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:06 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:05:06.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:06 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:06.931 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:06 smithi069 ceph-mon[31816]: from='osd.10 [v2:172.21.15.55:6824/3066586400,v1:172.21.15.55:6825/3066586400]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T23:05:06.931 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:06 smithi069 ceph-mon[31816]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T23:05:06.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:06 smithi044 ceph-mon[27934]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:05:06.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:06 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:06.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:06 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:05:06.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:06 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:06.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:06 smithi044 ceph-mon[27934]: from='osd.10 [v2:172.21.15.55:6824/3066586400,v1:172.21.15.55:6825/3066586400]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T23:05:06.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:06 smithi044 ceph-mon[27934]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T23:05:06.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:06 smithi055 ceph-mon[31117]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:05:06.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:06 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:06.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:06 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:05:06.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:06 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:06.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:06 smithi055 ceph-mon[31117]: from='osd.10 [v2:172.21.15.55:6824/3066586400,v1:172.21.15.55:6825/3066586400]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T23:05:06.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:06 smithi055 ceph-mon[31117]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T23:05:07.915 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:07 smithi069 ceph-mon[31816]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T23:05:07.915 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:07 smithi069 ceph-mon[31816]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T23:05:07.915 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:07 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:05:07.915 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:07 smithi069 ceph-mon[31816]: from='osd.10 [v2:172.21.15.55:6824/3066586400,v1:172.21.15.55:6825/3066586400]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:05:07.916 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:07 smithi069 ceph-mon[31816]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:05:07.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:07 smithi044 ceph-mon[27934]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T23:05:07.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:07 smithi044 ceph-mon[27934]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T23:05:07.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:07 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:05:07.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:07 smithi044 ceph-mon[27934]: from='osd.10 [v2:172.21.15.55:6824/3066586400,v1:172.21.15.55:6825/3066586400]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:05:07.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:07 smithi044 ceph-mon[27934]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:05:07.950 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:07 smithi055 ceph-mon[31117]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T23:05:07.950 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:07 smithi055 ceph-mon[31117]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T23:05:07.950 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:07 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:05:07.950 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:07 smithi055 ceph-mon[31117]: from='osd.10 [v2:172.21.15.55:6824/3066586400,v1:172.21.15.55:6825/3066586400]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:05:07.950 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:07 smithi055 ceph-mon[31117]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2024-04-21T23:05:08.629 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:05:08 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:05:08.376+0000 7f277d73f700 -1 osd.10 0 waiting for initial osdmap 2024-04-21T23:05:08.629 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:05:08 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:05:08.394+0000 7f277552a700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:05:08.690 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T23:05:08.690 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T23:05:08.690 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2024-04-21T23:05:08.690 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2024-04-21T23:05:08.690 INFO:teuthology.orchestra.run.smithi069.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125145 s, 838 MB/s 2024-04-21T23:05:08.690 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2024-04-21T23:05:08.891 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:08 smithi055 ceph-mon[31117]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:05:08.891 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:08 smithi055 ceph-mon[31117]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-04-21T23:05:08.891 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:08 smithi055 ceph-mon[31117]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T23:05:08.891 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:08 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:05:08.891 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:08 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:05:08.891 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:08 smithi055 ceph-mon[31117]: osdmap e72: 11 total, 10 up, 11 in 2024-04-21T23:05:08.891 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:08 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:05:08.891 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:08 smithi055 ceph-mon[31117]: from='osd.10 ' entity='osd.10' 2024-04-21T23:05:08.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:08 smithi044 ceph-mon[27934]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:05:08.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:08 smithi044 ceph-mon[27934]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-04-21T23:05:08.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:08 smithi044 ceph-mon[27934]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T23:05:08.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:08 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:05:08.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:08 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:05:08.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:08 smithi044 ceph-mon[27934]: osdmap e72: 11 total, 10 up, 11 in 2024-04-21T23:05:08.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:08 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:05:08.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:08 smithi044 ceph-mon[27934]: from='osd.10 ' entity='osd.10' 2024-04-21T23:05:08.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:08 smithi069 ceph-mon[31816]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:05:08.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:08 smithi069 ceph-mon[31816]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2024-04-21T23:05:08.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:08 smithi069 ceph-mon[31816]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T23:05:08.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:08 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:05:08.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:08 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:05:08.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:08 smithi069 ceph-mon[31816]: osdmap e72: 11 total, 10 up, 11 in 2024-04-21T23:05:08.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:08 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:05:08.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:08 smithi069 ceph-mon[31816]: from='osd.10 ' entity='osd.10' 2024-04-21T23:05:09.313 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi069:vg_nvme/lv_1 2024-04-21T23:05:09.548 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:05:09.924 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:09 smithi055 ceph-mon[31117]: purged_snaps scrub starts 2024-04-21T23:05:09.924 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:09 smithi055 ceph-mon[31117]: purged_snaps scrub ok 2024-04-21T23:05:09.924 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:09 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:05:09.924 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:09 smithi055 ceph-mon[31117]: osd.10 [v2:172.21.15.55:6824/3066586400,v1:172.21.15.55:6825/3066586400] boot 2024-04-21T23:05:09.924 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:09 smithi055 ceph-mon[31117]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T23:05:09.924 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:09 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:05:09.924 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:09 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:09.924 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:09 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:09.924 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:09 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:09.924 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:09 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:09.924 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:09 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:09.925 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:09 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:09.925 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:09 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:09.925 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:09 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:09.925 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:09 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:05:09.925 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:09 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:09.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:09 smithi044 ceph-mon[27934]: purged_snaps scrub starts 2024-04-21T23:05:09.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:09 smithi044 ceph-mon[27934]: purged_snaps scrub ok 2024-04-21T23:05:09.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:09 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:05:09.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:09 smithi044 ceph-mon[27934]: osd.10 [v2:172.21.15.55:6824/3066586400,v1:172.21.15.55:6825/3066586400] boot 2024-04-21T23:05:09.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:09 smithi044 ceph-mon[27934]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T23:05:09.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:09 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:05:09.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:09 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:09.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:09 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:09.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:09 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:09.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:09 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:09.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:09 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:09.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:09 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:09.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:09 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:09.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:09 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:09.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:09 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:05:09.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:09 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:09.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:09 smithi069 ceph-mon[31816]: purged_snaps scrub starts 2024-04-21T23:05:09.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:09 smithi069 ceph-mon[31816]: purged_snaps scrub ok 2024-04-21T23:05:09.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:09 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:05:09.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:09 smithi069 ceph-mon[31816]: osd.10 [v2:172.21.15.55:6824/3066586400,v1:172.21.15.55:6825/3066586400] boot 2024-04-21T23:05:09.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:09 smithi069 ceph-mon[31816]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T23:05:09.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:09 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:05:09.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:09 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:09.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:09 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:09.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:09 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:09.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:09 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:09.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:09 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:09.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:09 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:09.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:09 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:09.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:09 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:09.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:09 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:05:09.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:09 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:10.897 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:10 smithi069 ceph-mon[31816]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:05:10.898 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:10 smithi069 ceph-mon[31816]: Detected new or changed devices on smithi055 2024-04-21T23:05:10.898 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:10 smithi069 ceph-mon[31816]: Adjusting osd_memory_target on smithi055 to 4273M 2024-04-21T23:05:10.898 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:10 smithi069 ceph-mon[31816]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T23:05:10.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:10 smithi044 ceph-mon[27934]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:05:10.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:10 smithi044 ceph-mon[27934]: Detected new or changed devices on smithi055 2024-04-21T23:05:10.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:10 smithi044 ceph-mon[27934]: Adjusting osd_memory_target on smithi055 to 4273M 2024-04-21T23:05:10.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:10 smithi044 ceph-mon[27934]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T23:05:10.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:10 smithi055 ceph-mon[31117]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-04-21T23:05:10.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:10 smithi055 ceph-mon[31117]: Detected new or changed devices on smithi055 2024-04-21T23:05:10.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:10 smithi055 ceph-mon[31117]: Adjusting osd_memory_target on smithi055 to 4273M 2024-04-21T23:05:10.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:10 smithi055 ceph-mon[31117]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T23:05:11.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.132+0000 7f6f932ca700 1 -- 172.21.15.69:0/1326060220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f8c103e20 msgr2=0x7f6f8c10a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:11.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.132+0000 7f6f932ca700 1 --2- 172.21.15.69:0/1326060220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f8c103e20 0x7f6f8c10a6d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6f8400b0a0 tx=0x7f6f8400b3b0 comp rx=0 tx=0).stop 2024-04-21T23:05:11.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.132+0000 7f6f932ca700 1 -- 172.21.15.69:0/1326060220 shutdown_connections 2024-04-21T23:05:11.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.132+0000 7f6f932ca700 1 --2- 172.21.15.69:0/1326060220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f8c103e20 0x7f6f8c10a6d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:11.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.132+0000 7f6f932ca700 1 --2- 172.21.15.69:0/1326060220 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f8c102280 0x7f6f8c1026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:11.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.132+0000 7f6f932ca700 1 --2- 172.21.15.69:0/1326060220 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f8c103480 0x7f6f8c1038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:11.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.132+0000 7f6f932ca700 1 -- 172.21.15.69:0/1326060220 >> 172.21.15.69:0/1326060220 conn(0x7f6f8c0fd820 msgr2=0x7f6f8c0ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:11.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.133+0000 7f6f932ca700 1 -- 172.21.15.69:0/1326060220 shutdown_connections 2024-04-21T23:05:11.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.133+0000 7f6f932ca700 1 -- 172.21.15.69:0/1326060220 wait complete. 2024-04-21T23:05:11.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.133+0000 7f6f932ca700 1 Processor -- start 2024-04-21T23:05:11.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.133+0000 7f6f932ca700 1 -- start start 2024-04-21T23:05:11.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.133+0000 7f6f932ca700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f8c102280 0x7f6f8c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:11.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.133+0000 7f6f932ca700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f8c103480 0x7f6f8c198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:11.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.133+0000 7f6f932ca700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f8c103e20 0x7f6f8c19f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:11.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.133+0000 7f6f932ca700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8c10de60 con 0x7f6f8c103e20 2024-04-21T23:05:11.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.133+0000 7f6f932ca700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8c10dce0 con 0x7f6f8c103480 2024-04-21T23:05:11.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.133+0000 7f6f932ca700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8c10dfe0 con 0x7f6f8c102280 2024-04-21T23:05:11.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.133+0000 7f6f91066700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f8c102280 0x7f6f8c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:11.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.133+0000 7f6f91066700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f8c102280 0x7f6f8c198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48736/0 (socket says 172.21.15.69:48736) 2024-04-21T23:05:11.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.133+0000 7f6f91066700 1 -- 172.21.15.69:0/2731397350 learned_addr learned my addr 172.21.15.69:0/2731397350 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-21T23:05:11.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.133+0000 7f6f90865700 1 --2- 172.21.15.69:0/2731397350 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f8c103480 0x7f6f8c198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:11.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.134+0000 7f6f91867700 1 --2- 172.21.15.69:0/2731397350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f8c103e20 0x7f6f8c19f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:11.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.134+0000 7f6f91066700 1 -- 172.21.15.69:0/2731397350 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f8c103480 msgr2=0x7f6f8c198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:11.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.134+0000 7f6f91066700 1 --2- 172.21.15.69:0/2731397350 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f8c103480 0x7f6f8c198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:11.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.134+0000 7f6f91066700 1 -- 172.21.15.69:0/2731397350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f8c103e20 msgr2=0x7f6f8c19f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:11.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.134+0000 7f6f91066700 1 --2- 172.21.15.69:0/2731397350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f8c103e20 0x7f6f8c19f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:11.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.134+0000 7f6f91066700 1 -- 172.21.15.69:0/2731397350 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f8c1a0070 con 0x7f6f8c102280 2024-04-21T23:05:11.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.134+0000 7f6f91066700 1 --2- 172.21.15.69:0/2731397350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f8c102280 0x7f6f8c198410 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6f8800c9a0 tx=0x7f6f8800ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:11.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.134+0000 7f6f827fc700 1 -- 172.21.15.69:0/2731397350 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f88004220 con 0x7f6f8c102280 2024-04-21T23:05:11.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.134+0000 7f6f932ca700 1 -- 172.21.15.69:0/2731397350 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f8c1a0360 con 0x7f6f8c102280 2024-04-21T23:05:11.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.134+0000 7f6f827fc700 1 -- 172.21.15.69:0/2731397350 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f88007d40 con 0x7f6f8c102280 2024-04-21T23:05:11.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.134+0000 7f6f827fc700 1 -- 172.21.15.69:0/2731397350 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f88003da0 con 0x7f6f8c102280 2024-04-21T23:05:11.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.134+0000 7f6f932ca700 1 -- 172.21.15.69:0/2731397350 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f8c1a08f0 con 0x7f6f8c102280 2024-04-21T23:05:11.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.135+0000 7f6f932ca700 1 -- 172.21.15.69:0/2731397350 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f8c068f20 con 0x7f6f8c102280 2024-04-21T23:05:11.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.136+0000 7f6f827fc700 1 -- 172.21.15.69:0/2731397350 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6f88007860 con 0x7f6f8c102280 2024-04-21T23:05:11.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.137+0000 7f6f827fc700 1 --2- 172.21.15.69:0/2731397350 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6f780a2c20 0x7f6f780a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:11.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.137+0000 7f6f827fc700 1 -- 172.21.15.69:0/2731397350 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6723+0+0 (secure 0 0 0) 0x7f6f880ce6a0 con 0x7f6f8c102280 2024-04-21T23:05:11.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.137+0000 7f6f90865700 1 --2- 172.21.15.69:0/2731397350 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6f780a2c20 0x7f6f780a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:11.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.137+0000 7f6f90865700 1 --2- 172.21.15.69:0/2731397350 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6f780a2c20 0x7f6f780a50e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c004630 tx=0x7f6f7c00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:11.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.141+0000 7f6f827fc700 1 -- 172.21.15.69:0/2731397350 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6f88014030 con 0x7f6f8c102280 2024-04-21T23:05:11.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:11.346+0000 7f6f932ca700 1 -- 172.21.15.69:0/2731397350 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f6f8c062670 con 0x7f6f780a2c20 2024-04-21T23:05:12.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:12 smithi069 ceph-mon[31816]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:12.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:12 smithi069 ceph-mon[31816]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T23:05:12.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:12 smithi069 ceph-mon[31816]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:05:12.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:12 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:05:12.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:12 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:05:12.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:12 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:12.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:12 smithi044 ceph-mon[27934]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:12.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:12 smithi044 ceph-mon[27934]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T23:05:12.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:12 smithi044 ceph-mon[27934]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:05:12.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:05:12.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:05:12.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:12 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:12.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:12 smithi055 ceph-mon[31117]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:12.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:12 smithi055 ceph-mon[31117]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T23:05:12.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:12 smithi055 ceph-mon[31117]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:05:12.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:12 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T23:05:12.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:12 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T23:05:12.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:12 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:13.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:13 smithi069 ceph-mon[31816]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T23:05:13.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:13 smithi044 ceph-mon[27934]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T23:05:13.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:13 smithi055 ceph-mon[31117]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T23:05:14.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:14 smithi044 ceph-mon[27934]: pgmap v241: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T23:05:14.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:14 smithi044 ceph-mon[27934]: osdmap e77: 11 total, 11 up, 11 in 2024-04-21T23:05:14.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:14 smithi055 ceph-mon[31117]: pgmap v241: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T23:05:14.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:14 smithi055 ceph-mon[31117]: osdmap e77: 11 total, 11 up, 11 in 2024-04-21T23:05:14.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:14 smithi069 ceph-mon[31816]: pgmap v241: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T23:05:14.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:14 smithi069 ceph-mon[31816]: osdmap e77: 11 total, 11 up, 11 in 2024-04-21T23:05:15.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:15 smithi044 ceph-mon[27934]: from='client.? 172.21.15.69:0/299518074' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1623764d-b8ca-4222-a1e4-2714f17d64e6"}]: dispatch 2024-04-21T23:05:15.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:15 smithi044 ceph-mon[27934]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1623764d-b8ca-4222-a1e4-2714f17d64e6"}]: dispatch 2024-04-21T23:05:15.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:15 smithi044 ceph-mon[27934]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1623764d-b8ca-4222-a1e4-2714f17d64e6"}]': finished 2024-04-21T23:05:15.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:15 smithi044 ceph-mon[27934]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T23:05:15.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:15 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:05:15.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:15 smithi044 ceph-mon[27934]: from='client.? 172.21.15.69:0/2745889118' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:05:15.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:15 smithi069 ceph-mon[31816]: from='client.? 172.21.15.69:0/299518074' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1623764d-b8ca-4222-a1e4-2714f17d64e6"}]: dispatch 2024-04-21T23:05:15.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:15 smithi069 ceph-mon[31816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1623764d-b8ca-4222-a1e4-2714f17d64e6"}]: dispatch 2024-04-21T23:05:15.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:15 smithi069 ceph-mon[31816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1623764d-b8ca-4222-a1e4-2714f17d64e6"}]': finished 2024-04-21T23:05:15.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:15 smithi069 ceph-mon[31816]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T23:05:15.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:15 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:05:15.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:15 smithi069 ceph-mon[31816]: from='client.? 172.21.15.69:0/2745889118' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:05:15.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:15 smithi055 ceph-mon[31117]: from='client.? 172.21.15.69:0/299518074' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1623764d-b8ca-4222-a1e4-2714f17d64e6"}]: dispatch 2024-04-21T23:05:15.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:15 smithi055 ceph-mon[31117]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1623764d-b8ca-4222-a1e4-2714f17d64e6"}]: dispatch 2024-04-21T23:05:15.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:15 smithi055 ceph-mon[31117]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1623764d-b8ca-4222-a1e4-2714f17d64e6"}]': finished 2024-04-21T23:05:15.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:15 smithi055 ceph-mon[31117]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T23:05:15.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:15 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:05:15.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:15 smithi055 ceph-mon[31117]: from='client.? 172.21.15.69:0/2745889118' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T23:05:16.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:16 smithi044 ceph-mon[27934]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T23:05:16.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:16 smithi069 ceph-mon[31816]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T23:05:16.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:16 smithi055 ceph-mon[31117]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T23:05:18.630 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:18 smithi069 ceph-mon[31816]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T23:05:18.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:18 smithi044 ceph-mon[27934]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T23:05:18.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:18 smithi055 ceph-mon[31117]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T23:05:20.544 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:20 smithi069 ceph-mon[31816]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:20.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:20 smithi044 ceph-mon[27934]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:20.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:20 smithi055 ceph-mon[31117]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:22.543 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:22 smithi069 ceph-mon[31816]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:22.543 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:22 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T23:05:22.543 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:22 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:22.543 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:22 smithi069 ceph-mon[31816]: Deploying daemon osd.11 on smithi069 2024-04-21T23:05:22.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:22 smithi044 ceph-mon[27934]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:22.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:22 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T23:05:22.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:22 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:22.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:22 smithi044 ceph-mon[27934]: Deploying daemon osd.11 on smithi069 2024-04-21T23:05:22.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:22 smithi055 ceph-mon[31117]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:22.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:22 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T23:05:22.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:22 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:22.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:22 smithi055 ceph-mon[31117]: Deploying daemon osd.11 on smithi069 2024-04-21T23:05:24.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:24 smithi069 ceph-mon[31816]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:24.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:24 smithi044 ceph-mon[27934]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:24.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:24 smithi055 ceph-mon[31117]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:26.455 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:26 smithi069 ceph-mon[31816]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:26.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:26 smithi044 ceph-mon[27934]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:26.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:26 smithi055 ceph-mon[31117]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:28.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:28 smithi044 ceph-mon[27934]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:28.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:28 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:28.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:28 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:28.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:28 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:05:28.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:28 smithi055 ceph-mon[31117]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:28.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:28 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:28.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:28 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:28.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:28 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:05:28.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:28 smithi069 ceph-mon[31816]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:28.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:28 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:28.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:28 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:28.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:28 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:05:30.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:30 smithi069 ceph-mon[31816]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:30.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:30 smithi044 ceph-mon[27934]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:30.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:30 smithi055 ceph-mon[31117]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:31.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:31.169+0000 7f6f827fc700 1 -- 172.21.15.69:0/2731397350 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f6f8c062670 con 0x7f6f780a2c20 2024-04-21T23:05:31.169 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 11 on host 'smithi069' 2024-04-21T23:05:31.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:31.171+0000 7f6f932ca700 1 -- 172.21.15.69:0/2731397350 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6f780a2c20 msgr2=0x7f6f780a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:31.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:31.171+0000 7f6f932ca700 1 --2- 172.21.15.69:0/2731397350 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6f780a2c20 0x7f6f780a50e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c004630 tx=0x7f6f7c00b3c0 comp rx=0 tx=0).stop 2024-04-21T23:05:31.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:31.171+0000 7f6f932ca700 1 -- 172.21.15.69:0/2731397350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f8c102280 msgr2=0x7f6f8c198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:31.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:31.171+0000 7f6f932ca700 1 --2- 172.21.15.69:0/2731397350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f8c102280 0x7f6f8c198410 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6f8800c9a0 tx=0x7f6f8800ccb0 comp rx=0 tx=0).stop 2024-04-21T23:05:31.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:31.171+0000 7f6f932ca700 1 -- 172.21.15.69:0/2731397350 shutdown_connections 2024-04-21T23:05:31.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:31.171+0000 7f6f932ca700 1 --2- 172.21.15.69:0/2731397350 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6f780a2c20 0x7f6f780a50e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:31.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:31.171+0000 7f6f932ca700 1 --2- 172.21.15.69:0/2731397350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f8c103e20 0x7f6f8c19f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:31.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:31.171+0000 7f6f932ca700 1 --2- 172.21.15.69:0/2731397350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f8c102280 0x7f6f8c198410 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:31.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:31.171+0000 7f6f932ca700 1 --2- 172.21.15.69:0/2731397350 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f8c103480 0x7f6f8c198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:31.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:31.171+0000 7f6f932ca700 1 -- 172.21.15.69:0/2731397350 >> 172.21.15.69:0/2731397350 conn(0x7f6f8c0fd820 msgr2=0x7f6f8c106780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:31.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:31.172+0000 7f6f932ca700 1 -- 172.21.15.69:0/2731397350 shutdown_connections 2024-04-21T23:05:31.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:31.172+0000 7f6f932ca700 1 -- 172.21.15.69:0/2731397350 wait complete. 2024-04-21T23:05:31.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:31 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:31.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:31 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:31.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:31 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:31.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:31 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:05:31.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:31 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:31.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:31 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:31.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:31 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:31.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:31 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:05:31.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:31 smithi069 ceph-mon[31816]: from='osd.11 [v2:172.21.15.69:6824/1593504610,v1:172.21.15.69:6825/1593504610]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T23:05:31.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:31 smithi069 ceph-mon[31816]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T23:05:31.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:31 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:31.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:31 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:31.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:31 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:31.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:31 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:05:31.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:31 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:31.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:31 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:31.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:31 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:31.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:31 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:05:31.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:31 smithi044 ceph-mon[27934]: from='osd.11 [v2:172.21.15.69:6824/1593504610,v1:172.21.15.69:6825/1593504610]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T23:05:31.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:31 smithi044 ceph-mon[27934]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T23:05:31.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:31 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:31.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:31 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:31.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:31 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:31.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:31 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:05:31.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:31 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:31.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:31 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:31.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:31 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:31.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:31 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:05:31.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:31 smithi055 ceph-mon[31117]: from='osd.11 [v2:172.21.15.69:6824/1593504610,v1:172.21.15.69:6825/1593504610]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T23:05:31.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:31 smithi055 ceph-mon[31117]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T23:05:31.858 DEBUG:teuthology.orchestra.run.smithi069:osd.11> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.11.service 2024-04-21T23:05:31.861 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-21T23:05:31.861 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-21T23:05:32.120 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:05:32.366 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:32 smithi069 ceph-mon[31816]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:32.366 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:32 smithi069 ceph-mon[31816]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T23:05:32.366 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:32 smithi069 ceph-mon[31816]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T23:05:32.366 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:32 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:05:32.366 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:32 smithi069 ceph-mon[31816]: from='osd.11 [v2:172.21.15.69:6824/1593504610,v1:172.21.15.69:6825/1593504610]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:05:32.366 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:32 smithi069 ceph-mon[31816]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:05:32.555 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:32 smithi044 ceph-mon[27934]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:32.555 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:32 smithi044 ceph-mon[27934]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T23:05:32.555 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:32 smithi044 ceph-mon[27934]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T23:05:32.555 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:32 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:05:32.556 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:32 smithi044 ceph-mon[27934]: from='osd.11 [v2:172.21.15.69:6824/1593504610,v1:172.21.15.69:6825/1593504610]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:05:32.556 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:32 smithi044 ceph-mon[27934]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:05:32.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:32 smithi055 ceph-mon[31117]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:32.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:32 smithi055 ceph-mon[31117]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T23:05:32.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:32 smithi055 ceph-mon[31117]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T23:05:32.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:32 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:05:32.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:32 smithi055 ceph-mon[31117]: from='osd.11 [v2:172.21.15.69:6824/1593504610,v1:172.21.15.69:6825/1593504610]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:05:32.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:32 smithi055 ceph-mon[31117]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-21T23:05:33.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.897+0000 7f2ca3508700 1 -- 172.21.15.44:0/1869453441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c9c1056c0 msgr2=0x7f2c9c107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:33.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.897+0000 7f2ca3508700 1 --2- 172.21.15.44:0/1869453441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c9c1056c0 0x7f2c9c107ab0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2c90009990 tx=0x7f2c90009ca0 comp rx=0 tx=0).stop 2024-04-21T23:05:33.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.897+0000 7f2ca3508700 1 -- 172.21.15.44:0/1869453441 shutdown_connections 2024-04-21T23:05:33.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.897+0000 7f2ca3508700 1 --2- 172.21.15.44:0/1869453441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c9c107ff0 0x7f2c9c10a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:33.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.897+0000 7f2ca3508700 1 --2- 172.21.15.44:0/1869453441 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2c9c102d90 0x7f2c9c105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:33.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.897+0000 7f2ca3508700 1 --2- 172.21.15.44:0/1869453441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c9c1056c0 0x7f2c9c107ab0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:33.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.897+0000 7f2ca3508700 1 -- 172.21.15.44:0/1869453441 >> 172.21.15.44:0/1869453441 conn(0x7f2c9c0fc720 msgr2=0x7f2c9c0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:33.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.898+0000 7f2ca3508700 1 -- 172.21.15.44:0/1869453441 shutdown_connections 2024-04-21T23:05:33.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.898+0000 7f2ca3508700 1 -- 172.21.15.44:0/1869453441 wait complete. 2024-04-21T23:05:33.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.898+0000 7f2ca3508700 1 Processor -- start 2024-04-21T23:05:33.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.898+0000 7f2ca3508700 1 -- start start 2024-04-21T23:05:33.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.898+0000 7f2ca3508700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c9c102d90 0x7f2c9c1026f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:33.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.898+0000 7f2ca3508700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2c9c1056c0 0x7f2c9c100d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:33.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.898+0000 7f2ca3508700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c9c107ff0 0x7f2c9c101280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:33.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.898+0000 7f2ca3508700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9c10de90 con 0x7f2c9c107ff0 2024-04-21T23:05:33.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.898+0000 7f2ca3508700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9c10dd10 con 0x7f2c9c1056c0 2024-04-21T23:05:33.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.898+0000 7f2ca3508700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9c10e010 con 0x7f2c9c102d90 2024-04-21T23:05:33.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.899+0000 7f2ca1aa5700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c9c107ff0 0x7f2c9c101280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:33.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.899+0000 7f2ca12a4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c9c102d90 0x7f2c9c1026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:33.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.899+0000 7f2ca1aa5700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c9c107ff0 0x7f2c9c101280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54546/0 (socket says 172.21.15.44:54546) 2024-04-21T23:05:33.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.899+0000 7f2ca1aa5700 1 -- 172.21.15.44:0/2274969711 learned_addr learned my addr 172.21.15.44:0/2274969711 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:05:33.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.899+0000 7f2ca0aa3700 1 --2- 172.21.15.44:0/2274969711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2c9c1056c0 0x7f2c9c100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:33.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.899+0000 7f2ca12a4700 1 -- 172.21.15.44:0/2274969711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2c9c1056c0 msgr2=0x7f2c9c100d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:33.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.899+0000 7f2ca12a4700 1 --2- 172.21.15.44:0/2274969711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2c9c1056c0 0x7f2c9c100d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:33.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.899+0000 7f2ca12a4700 1 -- 172.21.15.44:0/2274969711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c9c107ff0 msgr2=0x7f2c9c101280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:33.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.899+0000 7f2ca12a4700 1 --2- 172.21.15.44:0/2274969711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c9c107ff0 0x7f2c9c101280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:33.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.899+0000 7f2ca12a4700 1 -- 172.21.15.44:0/2274969711 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c9c101b30 con 0x7f2c9c102d90 2024-04-21T23:05:33.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.899+0000 7f2ca1aa5700 1 --2- 172.21.15.44:0/2274969711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c9c107ff0 0x7f2c9c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:05:33.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.899+0000 7f2ca12a4700 1 --2- 172.21.15.44:0/2274969711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c9c102d90 0x7f2c9c1026f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2c9800d750 tx=0x7f2c9800da60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:33.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.899+0000 7f2c8e7fc700 1 -- 172.21.15.44:0/2274969711 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98014030 con 0x7f2c9c102d90 2024-04-21T23:05:33.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.899+0000 7f2ca3508700 1 -- 172.21.15.44:0/2274969711 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c9c1a2630 con 0x7f2c9c102d90 2024-04-21T23:05:33.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.899+0000 7f2c8e7fc700 1 -- 172.21.15.44:0/2274969711 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c98004500 con 0x7f2c9c102d90 2024-04-21T23:05:33.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.899+0000 7f2c8e7fc700 1 -- 172.21.15.44:0/2274969711 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98005400 con 0x7f2c9c102d90 2024-04-21T23:05:33.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.899+0000 7f2ca3508700 1 -- 172.21.15.44:0/2274969711 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c9c1a2b70 con 0x7f2c9c102d90 2024-04-21T23:05:33.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.900+0000 7f2ca3508700 1 -- 172.21.15.44:0/2274969711 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c9c068f20 con 0x7f2c9c102d90 2024-04-21T23:05:33.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.902+0000 7f2c8e7fc700 1 -- 172.21.15.44:0/2274969711 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2c98004750 con 0x7f2c9c102d90 2024-04-21T23:05:33.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.902+0000 7f2c8e7fc700 1 --2- 172.21.15.44:0/2274969711 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2c880a2c20 0x7f2c880a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:33.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.902+0000 7f2ca0aa3700 1 --2- 172.21.15.44:0/2274969711 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2c880a2c20 0x7f2c880a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:33.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.902+0000 7f2c8e7fc700 1 -- 172.21.15.44:0/2274969711 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6847+0+0 (secure 0 0 0) 0x7f2c980ce8b0 con 0x7f2c9c102d90 2024-04-21T23:05:33.902 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.903+0000 7f2ca0aa3700 1 --2- 172.21.15.44:0/2274969711 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2c880a2c20 0x7f2c880a50e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2c9000b2e0 tx=0x7f2c90005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:33.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:33.907+0000 7f2c8e7fc700 1 -- 172.21.15.44:0/2274969711 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2c9809bd30 con 0x7f2c9c102d90 2024-04-21T23:05:34.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:34.116+0000 7f2ca3508700 1 -- 172.21.15.44:0/2274969711 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f2c9c02df60 con 0x7f2c9c102d90 2024-04-21T23:05:34.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:34.117+0000 7f2c8e7fc700 1 -- 172.21.15.44:0/2274969711 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f2c9809b630 con 0x7f2c9c102d90 2024-04-21T23:05:34.116 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:05:34.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:34.119+0000 7f2ca3508700 1 -- 172.21.15.44:0/2274969711 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2c880a2c20 msgr2=0x7f2c880a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:34.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:34.119+0000 7f2ca3508700 1 --2- 172.21.15.44:0/2274969711 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2c880a2c20 0x7f2c880a50e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2c9000b2e0 tx=0x7f2c90005fd0 comp rx=0 tx=0).stop 2024-04-21T23:05:34.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:34.120+0000 7f2ca3508700 1 -- 172.21.15.44:0/2274969711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c9c102d90 msgr2=0x7f2c9c1026f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:34.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:34.120+0000 7f2ca3508700 1 --2- 172.21.15.44:0/2274969711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c9c102d90 0x7f2c9c1026f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2c9800d750 tx=0x7f2c9800da60 comp rx=0 tx=0).stop 2024-04-21T23:05:34.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:34.120+0000 7f2ca3508700 1 -- 172.21.15.44:0/2274969711 shutdown_connections 2024-04-21T23:05:34.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:34.120+0000 7f2ca3508700 1 --2- 172.21.15.44:0/2274969711 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2c880a2c20 0x7f2c880a50e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:34.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:34.120+0000 7f2ca3508700 1 --2- 172.21.15.44:0/2274969711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2c9c107ff0 0x7f2c9c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:34.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:34.120+0000 7f2ca3508700 1 --2- 172.21.15.44:0/2274969711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c9c102d90 0x7f2c9c1026f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:34.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:34.120+0000 7f2ca3508700 1 --2- 172.21.15.44:0/2274969711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2c9c1056c0 0x7f2c9c100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:34.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:34.120+0000 7f2ca3508700 1 -- 172.21.15.44:0/2274969711 >> 172.21.15.44:0/2274969711 conn(0x7f2c9c0fc720 msgr2=0x7f2c9c1036f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:34.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:34.120+0000 7f2ca3508700 1 -- 172.21.15.44:0/2274969711 shutdown_connections 2024-04-21T23:05:34.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:34.120+0000 7f2ca3508700 1 -- 172.21.15.44:0/2274969711 wait complete. 2024-04-21T23:05:34.155 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:05:33 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:05:33.872+0000 7f2db06df700 -1 osd.11 0 waiting for initial osdmap 2024-04-21T23:05:34.155 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:05:33 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:05:33.885+0000 7f2da84ca700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T23:05:34.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:34 smithi044 ceph-mon[27934]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:34.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:34 smithi044 ceph-mon[27934]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-21T23:05:34.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:34 smithi044 ceph-mon[27934]: osdmap e80: 12 total, 11 up, 12 in 2024-04-21T23:05:34.418 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:34 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:05:34.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:34 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:05:34.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:34 smithi044 ceph-mon[27934]: from='osd.11 ' entity='osd.11' 2024-04-21T23:05:34.419 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:34 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2274969711' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T23:05:34.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:34 smithi055 ceph-mon[31117]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:34.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:34 smithi055 ceph-mon[31117]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-21T23:05:34.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:34 smithi055 ceph-mon[31117]: osdmap e80: 12 total, 11 up, 12 in 2024-04-21T23:05:34.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:34 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:05:34.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:34 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:05:34.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:34 smithi055 ceph-mon[31117]: from='osd.11 ' entity='osd.11' 2024-04-21T23:05:34.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:34 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2274969711' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T23:05:34.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:34 smithi069 ceph-mon[31816]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:34.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:34 smithi069 ceph-mon[31816]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-21T23:05:34.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:34 smithi069 ceph-mon[31816]: osdmap e80: 12 total, 11 up, 12 in 2024-04-21T23:05:34.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:34 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:05:34.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:34 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:05:34.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:34 smithi069 ceph-mon[31816]: from='osd.11 ' entity='osd.11' 2024-04-21T23:05:34.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:34 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2274969711' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T23:05:35.011 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":80,"num_osds":12,"num_up_osds":11,"osd_up_since":1713740709,"num_in_osds":12,"osd_in_since":1713740714,"num_remapped_pgs":0} 2024-04-21T23:05:35.400 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:35 smithi044 ceph-mon[27934]: purged_snaps scrub starts 2024-04-21T23:05:35.400 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:35 smithi044 ceph-mon[27934]: purged_snaps scrub ok 2024-04-21T23:05:35.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:35 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:05:35.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:35 smithi044 ceph-mon[27934]: osd.11 [v2:172.21.15.69:6824/1593504610,v1:172.21.15.69:6825/1593504610] boot 2024-04-21T23:05:35.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:35 smithi044 ceph-mon[27934]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T23:05:35.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:35 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:05:35.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:35 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:35.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:35 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:35.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:35 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:35.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:35 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:35.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:35 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:35.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:35 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:35.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:35 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:35.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:35 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:35.401 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:35 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:05:35.402 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:35 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:35.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:35 smithi055 ceph-mon[31117]: purged_snaps scrub starts 2024-04-21T23:05:35.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:35 smithi055 ceph-mon[31117]: purged_snaps scrub ok 2024-04-21T23:05:35.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:35 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:05:35.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:35 smithi055 ceph-mon[31117]: osd.11 [v2:172.21.15.69:6824/1593504610,v1:172.21.15.69:6825/1593504610] boot 2024-04-21T23:05:35.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:35 smithi055 ceph-mon[31117]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T23:05:35.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:35 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:05:35.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:35 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:35.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:35 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:35.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:35 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:35.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:35 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:35.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:35 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:35.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:35 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:35.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:35 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:35.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:35 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:35.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:35 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:05:35.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:35 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:35.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:35 smithi069 ceph-mon[31816]: purged_snaps scrub starts 2024-04-21T23:05:35.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:35 smithi069 ceph-mon[31816]: purged_snaps scrub ok 2024-04-21T23:05:35.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:35 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:05:35.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:35 smithi069 ceph-mon[31816]: osd.11 [v2:172.21.15.69:6824/1593504610,v1:172.21.15.69:6825/1593504610] boot 2024-04-21T23:05:35.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:35 smithi069 ceph-mon[31816]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T23:05:35.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:35 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:05:35.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:35 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:35.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:35 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:35.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:35 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:35.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:35 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:35.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:35 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:35.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:35 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:05:35.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:35 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:35.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:35 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:35.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:35 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:05:35.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:35 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:36.012 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-21T23:05:36.275 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:05:36.398 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:36 smithi044 ceph-mon[27934]: Detected new or changed devices on smithi069 2024-04-21T23:05:36.398 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:36 smithi044 ceph-mon[27934]: Adjusting osd_memory_target on smithi069 to 4272M 2024-04-21T23:05:36.398 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:36 smithi044 ceph-mon[27934]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:36.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:36 smithi069 ceph-mon[31816]: Detected new or changed devices on smithi069 2024-04-21T23:05:36.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:36 smithi069 ceph-mon[31816]: Adjusting osd_memory_target on smithi069 to 4272M 2024-04-21T23:05:36.433 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:36 smithi069 ceph-mon[31816]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:36.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:36 smithi055 ceph-mon[31117]: Detected new or changed devices on smithi069 2024-04-21T23:05:36.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:36 smithi055 ceph-mon[31117]: Adjusting osd_memory_target on smithi069 to 4272M 2024-04-21T23:05:36.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:36 smithi055 ceph-mon[31117]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-21T23:05:37.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:37 smithi044 ceph-mon[27934]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T23:05:37.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:37 smithi055 ceph-mon[31117]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T23:05:37.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:37 smithi069 ceph-mon[31816]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T23:05:38.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.128+0000 7f00796f1700 1 -- 172.21.15.44:0/3730741883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0074101190 msgr2=0x7f00741015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:38.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.128+0000 7f00796f1700 1 --2- 172.21.15.44:0/3730741883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0074101190 0x7f00741015b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f00640099c0 tx=0x7f0064009cd0 comp rx=0 tx=0).stop 2024-04-21T23:05:38.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.128+0000 7f00796f1700 1 -- 172.21.15.44:0/3730741883 shutdown_connections 2024-04-21T23:05:38.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.128+0000 7f00796f1700 1 --2- 172.21.15.44:0/3730741883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0074102d30 0x7f00741095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.128+0000 7f00796f1700 1 --2- 172.21.15.44:0/3730741883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0074101190 0x7f00741015b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.128+0000 7f00796f1700 1 --2- 172.21.15.44:0/3730741883 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0074102390 0x7f00741027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.128+0000 7f00796f1700 1 -- 172.21.15.44:0/3730741883 >> 172.21.15.44:0/3730741883 conn(0x7f00740fc730 msgr2=0x7f00740feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.128+0000 7f00796f1700 1 -- 172.21.15.44:0/3730741883 shutdown_connections 2024-04-21T23:05:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.128+0000 7f00796f1700 1 -- 172.21.15.44:0/3730741883 wait complete. 2024-04-21T23:05:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.129+0000 7f00796f1700 1 Processor -- start 2024-04-21T23:05:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.129+0000 7f00796f1700 1 -- start start 2024-04-21T23:05:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.129+0000 7f00796f1700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0074101190 0x7f0074105120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.129+0000 7f00796f1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0074102390 0x7f0074103770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.129+0000 7f00796f1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0074102d30 0x7f0074103cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.129+0000 7f00796f1700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f007410cd50 con 0x7f0074101190 2024-04-21T23:05:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.129+0000 7f00796f1700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f007410cbd0 con 0x7f0074102390 2024-04-21T23:05:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.129+0000 7f00796f1700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f007410ced0 con 0x7f0074102d30 2024-04-21T23:05:38.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.129+0000 7f0072ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0074101190 0x7f0074105120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:38.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.129+0000 7f0072ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0074101190 0x7f0074105120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54572/0 (socket says 172.21.15.44:54572) 2024-04-21T23:05:38.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.129+0000 7f00737fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0074102d30 0x7f0074103cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:38.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.129+0000 7f0072ffd700 1 -- 172.21.15.44:0/2189379575 learned_addr learned my addr 172.21.15.44:0/2189379575 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:05:38.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.129+0000 7f00727fc700 1 --2- 172.21.15.44:0/2189379575 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0074102390 0x7f0074103770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:38.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.130+0000 7f0072ffd700 1 -- 172.21.15.44:0/2189379575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0074102d30 msgr2=0x7f0074103cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:38.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.130+0000 7f0072ffd700 1 --2- 172.21.15.44:0/2189379575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0074102d30 0x7f0074103cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:38.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.130+0000 7f0072ffd700 1 -- 172.21.15.44:0/2189379575 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0074102390 msgr2=0x7f0074103770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:38.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.130+0000 7f0072ffd700 1 --2- 172.21.15.44:0/2189379575 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0074102390 0x7f0074103770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:38.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.130+0000 7f0072ffd700 1 -- 172.21.15.44:0/2189379575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0074104560 con 0x7f0074101190 2024-04-21T23:05:38.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.130+0000 7f00737fe700 1 --2- 172.21.15.44:0/2189379575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0074102d30 0x7f0074103cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:05:38.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.130+0000 7f0072ffd700 1 --2- 172.21.15.44:0/2189379575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0074101190 0x7f0074105120 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0064003890 tx=0x7f0064009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:38.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.130+0000 7f005bfff700 1 -- 172.21.15.44:0/2189379575 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0064027070 con 0x7f0074101190 2024-04-21T23:05:38.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.130+0000 7f00796f1700 1 -- 172.21.15.44:0/2189379575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00741a2670 con 0x7f0074101190 2024-04-21T23:05:38.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.130+0000 7f005bfff700 1 -- 172.21.15.44:0/2189379575 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0064003f40 con 0x7f0074101190 2024-04-21T23:05:38.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.130+0000 7f005bfff700 1 -- 172.21.15.44:0/2189379575 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0064022bc0 con 0x7f0074101190 2024-04-21T23:05:38.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.130+0000 7f00796f1700 1 -- 172.21.15.44:0/2189379575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00741a2b20 con 0x7f0074101190 2024-04-21T23:05:38.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.131+0000 7f00796f1700 1 -- 172.21.15.44:0/2189379575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0074068f20 con 0x7f0074101190 2024-04-21T23:05:38.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.132+0000 7f005bfff700 1 -- 172.21.15.44:0/2189379575 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f006402a070 con 0x7f0074101190 2024-04-21T23:05:38.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.133+0000 7f005bfff700 1 --2- 172.21.15.44:0/2189379575 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f005c0a2c20 0x7f005c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:38.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.133+0000 7f00727fc700 1 --2- 172.21.15.44:0/2189379575 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f005c0a2c20 0x7f005c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:38.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.133+0000 7f005bfff700 1 -- 172.21.15.44:0/2189379575 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f00640cfb80 con 0x7f0074101190 2024-04-21T23:05:38.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.134+0000 7f00727fc700 1 --2- 172.21.15.44:0/2189379575 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f005c0a2c20 0x7f005c0a50e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0068005950 tx=0x7f006800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:38.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.138+0000 7f005bfff700 1 -- 172.21.15.44:0/2189379575 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f006405e6f0 con 0x7f0074101190 2024-04-21T23:05:38.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.345+0000 7f00796f1700 1 -- 172.21.15.44:0/2189379575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f00741a3440 con 0x7f0074101190 2024-04-21T23:05:38.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.346+0000 7f005bfff700 1 -- 172.21.15.44:0/2189379575 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) ==== 74+0+133 (secure 0 0 0) 0x7f0064013070 con 0x7f0074101190 2024-04-21T23:05:38.345 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:05:38.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.348+0000 7f00796f1700 1 -- 172.21.15.44:0/2189379575 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f005c0a2c20 msgr2=0x7f005c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:38.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.348+0000 7f00796f1700 1 --2- 172.21.15.44:0/2189379575 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f005c0a2c20 0x7f005c0a50e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0068005950 tx=0x7f006800b410 comp rx=0 tx=0).stop 2024-04-21T23:05:38.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.348+0000 7f00796f1700 1 -- 172.21.15.44:0/2189379575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0074101190 msgr2=0x7f0074105120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:38.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.348+0000 7f00796f1700 1 --2- 172.21.15.44:0/2189379575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0074101190 0x7f0074105120 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0064003890 tx=0x7f0064009580 comp rx=0 tx=0).stop 2024-04-21T23:05:38.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.348+0000 7f00796f1700 1 -- 172.21.15.44:0/2189379575 shutdown_connections 2024-04-21T23:05:38.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.348+0000 7f00796f1700 1 --2- 172.21.15.44:0/2189379575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0074102d30 0x7f0074103cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:38.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.348+0000 7f00796f1700 1 --2- 172.21.15.44:0/2189379575 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f005c0a2c20 0x7f005c0a50e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:38.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.348+0000 7f00796f1700 1 --2- 172.21.15.44:0/2189379575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0074101190 0x7f0074105120 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:38.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.348+0000 7f00796f1700 1 --2- 172.21.15.44:0/2189379575 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0074102390 0x7f0074103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:38.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.348+0000 7f00796f1700 1 -- 172.21.15.44:0/2189379575 >> 172.21.15.44:0/2189379575 conn(0x7f00740fc730 msgr2=0x7f00740fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:38.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.348+0000 7f00796f1700 1 -- 172.21.15.44:0/2189379575 shutdown_connections 2024-04-21T23:05:38.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:38.348+0000 7f00796f1700 1 -- 172.21.15.44:0/2189379575 wait complete. 2024-04-21T23:05:38.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:38 smithi044 ceph-mon[27934]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:38.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:38 smithi055 ceph-mon[31117]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:38.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:38 smithi069 ceph-mon[31816]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:39.030 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1713740734,"num_in_osds":12,"osd_in_since":1713740714,"num_remapped_pgs":0} 2024-04-21T23:05:39.031 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-21T23:05:39.280 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:05:39.402 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:39 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2189379575' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T23:05:39.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:39 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2189379575' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T23:05:39.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:39 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2189379575' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T23:05:40.426 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:40 smithi044 ceph-mon[27934]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:40.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:40 smithi069 ceph-mon[31816]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:40.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:40 smithi055 ceph-mon[31117]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:40.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.980+0000 7f600c876700 1 -- 172.21.15.44:0/1116030497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f600806ac40 msgr2=0x7f600806b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:40.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.980+0000 7f600c876700 1 --2- 172.21.15.44:0/1116030497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f600806ac40 0x7f600806b020 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5ff80099c0 tx=0x7f5ff8009cd0 comp rx=0 tx=0).stop 2024-04-21T23:05:40.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.980+0000 7f600c876700 1 -- 172.21.15.44:0/1116030497 shutdown_connections 2024-04-21T23:05:40.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.980+0000 7f600c876700 1 --2- 172.21.15.44:0/1116030497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6008107590 0x7f600810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:40.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.980+0000 7f600c876700 1 --2- 172.21.15.44:0/1116030497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f600806ac40 0x7f600806b020 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:40.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.980+0000 7f600c876700 1 --2- 172.21.15.44:0/1116030497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f600806b5f0 0x7f6008106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:40.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.980+0000 7f600c876700 1 -- 172.21.15.44:0/1116030497 >> 172.21.15.44:0/1116030497 conn(0x7f60080781d0 msgr2=0x7f60080785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:40.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.980+0000 7f600c876700 1 -- 172.21.15.44:0/1116030497 shutdown_connections 2024-04-21T23:05:40.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.980+0000 7f600c876700 1 -- 172.21.15.44:0/1116030497 wait complete. 2024-04-21T23:05:40.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.981+0000 7f600c876700 1 Processor -- start 2024-04-21T23:05:40.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.981+0000 7f600c876700 1 -- start start 2024-04-21T23:05:40.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.981+0000 7f600c876700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f600806ac40 0x7f600819e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:40.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.981+0000 7f600c876700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f600806b5f0 0x7f600819ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:40.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.981+0000 7f600c876700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6008107590 0x7f6008198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:40.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.981+0000 7f600c876700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f60081125a0 con 0x7f600806ac40 2024-04-21T23:05:40.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.981+0000 7f600c876700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6008112420 con 0x7f6008107590 2024-04-21T23:05:40.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.981+0000 7f600c876700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6008112720 con 0x7f600806b5f0 2024-04-21T23:05:40.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.981+0000 7f600659c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f600806ac40 0x7f600819e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:40.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.982+0000 7f600659c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f600806ac40 0x7f600819e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36534/0 (socket says 172.21.15.44:36534) 2024-04-21T23:05:40.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.982+0000 7f600659c700 1 -- 172.21.15.44:0/3622146837 learned_addr learned my addr 172.21.15.44:0/3622146837 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:05:40.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.982+0000 7f6006d9d700 1 --2- 172.21.15.44:0/3622146837 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6008107590 0x7f6008198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:40.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.982+0000 7f6005d9b700 1 --2- 172.21.15.44:0/3622146837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f600806b5f0 0x7f600819ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:40.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.982+0000 7f600659c700 1 -- 172.21.15.44:0/3622146837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f600806b5f0 msgr2=0x7f600819ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:40.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.982+0000 7f600659c700 1 --2- 172.21.15.44:0/3622146837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f600806b5f0 0x7f600819ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:40.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.982+0000 7f600659c700 1 -- 172.21.15.44:0/3622146837 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6008107590 msgr2=0x7f6008198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:40.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.982+0000 7f600659c700 1 --2- 172.21.15.44:0/3622146837 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6008107590 0x7f6008198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:40.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.982+0000 7f600659c700 1 -- 172.21.15.44:0/3622146837 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6008199020 con 0x7f600806ac40 2024-04-21T23:05:40.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.982+0000 7f600659c700 1 --2- 172.21.15.44:0/3622146837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f600806ac40 0x7f600819e6c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8006010 tx=0x7f5ff8009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:40.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.982+0000 7f5ff77fe700 1 -- 172.21.15.44:0/3622146837 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff8026070 con 0x7f600806ac40 2024-04-21T23:05:40.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.982+0000 7f600c876700 1 -- 172.21.15.44:0/3622146837 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60081992b0 con 0x7f600806ac40 2024-04-21T23:05:40.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.982+0000 7f5ff77fe700 1 -- 172.21.15.44:0/3622146837 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ff8004cd0 con 0x7f600806ac40 2024-04-21T23:05:40.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.982+0000 7f5ff77fe700 1 -- 172.21.15.44:0/3622146837 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff8003ba0 con 0x7f600806ac40 2024-04-21T23:05:40.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.982+0000 7f600c876700 1 -- 172.21.15.44:0/3622146837 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60081010d0 con 0x7f600806ac40 2024-04-21T23:05:40.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.983+0000 7f600c876700 1 -- 172.21.15.44:0/3622146837 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6008103990 con 0x7f600806ac40 2024-04-21T23:05:40.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.984+0000 7f5ff77fe700 1 -- 172.21.15.44:0/3622146837 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5ff8029070 con 0x7f600806ac40 2024-04-21T23:05:40.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.985+0000 7f5ff77fe700 1 --2- 172.21.15.44:0/3622146837 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5ff00a2c20 0x7f5ff00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:40.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.985+0000 7f5ff77fe700 1 -- 172.21.15.44:0/3622146837 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f5ff80cf480 con 0x7f600806ac40 2024-04-21T23:05:40.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.985+0000 7f6005d9b700 1 --2- 172.21.15.44:0/3622146837 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5ff00a2c20 0x7f5ff00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:40.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.986+0000 7f6005d9b700 1 --2- 172.21.15.44:0/3622146837 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5ff00a2c20 0x7f5ff00a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6008107070 tx=0x7f5ffc0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:40.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:40.990+0000 7f5ff77fe700 1 -- 172.21.15.44:0/3622146837 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5ff805d6f0 con 0x7f600806ac40 2024-04-21T23:05:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:41.197+0000 7f600c876700 1 -- 172.21.15.44:0/3622146837 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6008199d70 con 0x7f600806ac40 2024-04-21T23:05:41.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:41.198+0000 7f5ff77fe700 1 -- 172.21.15.44:0/3622146837 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18360 (secure 0 0 0) 0x7f5ff8013070 con 0x7f600806ac40 2024-04-21T23:05:41.197 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:05:41.197 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":82,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:05:36.159839+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:05:41.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:41.200+0000 7f600c876700 1 -- 172.21.15.44:0/3622146837 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5ff00a2c20 msgr2=0x7f5ff00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:41.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:41.200+0000 7f600c876700 1 --2- 172.21.15.44:0/3622146837 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5ff00a2c20 0x7f5ff00a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6008107070 tx=0x7f5ffc0073d0 comp rx=0 tx=0).stop 2024-04-21T23:05:41.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:41.200+0000 7f600c876700 1 -- 172.21.15.44:0/3622146837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f600806ac40 msgr2=0x7f600819e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:41.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:41.200+0000 7f600c876700 1 --2- 172.21.15.44:0/3622146837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f600806ac40 0x7f600819e6c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8006010 tx=0x7f5ff8009f80 comp rx=0 tx=0).stop 2024-04-21T23:05:41.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:41.200+0000 7f600c876700 1 -- 172.21.15.44:0/3622146837 shutdown_connections 2024-04-21T23:05:41.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:41.200+0000 7f600c876700 1 --2- 172.21.15.44:0/3622146837 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6008107590 0x7f6008198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:41.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:41.200+0000 7f600c876700 1 --2- 172.21.15.44:0/3622146837 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5ff00a2c20 0x7f5ff00a50e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:41.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:41.200+0000 7f600c876700 1 --2- 172.21.15.44:0/3622146837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f600806ac40 0x7f600819e6c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:41.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:41.200+0000 7f600c876700 1 --2- 172.21.15.44:0/3622146837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f600806b5f0 0x7f600819ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:41.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:41.200+0000 7f600c876700 1 -- 172.21.15.44:0/3622146837 >> 172.21.15.44:0/3622146837 conn(0x7f60080781d0 msgr2=0x7f60080ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:41.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:41.200+0000 7f600c876700 1 -- 172.21.15.44:0/3622146837 shutdown_connections 2024-04-21T23:05:41.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:41.200+0000 7f600c876700 1 -- 172.21.15.44:0/3622146837 wait complete. 2024-04-21T23:05:41.944 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:01:47.507503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T23:05:41.944 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-21T23:05:42.194 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:05:42.440 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:42 smithi044 ceph-mon[27934]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:42.440 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:42 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3622146837' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:05:42.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:42 smithi069 ceph-mon[31816]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:42.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:42 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3622146837' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:05:42.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:42 smithi055 ceph-mon[31117]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:42.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:42 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3622146837' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:05:43.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.908+0000 7f88e899d700 1 -- 172.21.15.44:0/606229164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88e4103cb0 msgr2=0x7f88e410c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:43.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.908+0000 7f88e899d700 1 --2- 172.21.15.44:0/606229164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88e4103cb0 0x7f88e410c1a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f88d8009990 tx=0x7f88d8009ca0 comp rx=0 tx=0).stop 2024-04-21T23:05:43.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.908+0000 7f88e899d700 1 -- 172.21.15.44:0/606229164 shutdown_connections 2024-04-21T23:05:43.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.908+0000 7f88e899d700 1 --2- 172.21.15.44:0/606229164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88e410c7c0 0x7f88e410ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:43.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.908+0000 7f88e899d700 1 --2- 172.21.15.44:0/606229164 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88e4103390 0x7f88e4103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:43.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.908+0000 7f88e899d700 1 --2- 172.21.15.44:0/606229164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88e4103cb0 0x7f88e410c1a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:43.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.908+0000 7f88e899d700 1 -- 172.21.15.44:0/606229164 >> 172.21.15.44:0/606229164 conn(0x7f88e40fcbe0 msgr2=0x7f88e40ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:43.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.908+0000 7f88e899d700 1 -- 172.21.15.44:0/606229164 shutdown_connections 2024-04-21T23:05:43.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.908+0000 7f88e899d700 1 -- 172.21.15.44:0/606229164 wait complete. 2024-04-21T23:05:43.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.908+0000 7f88e899d700 1 Processor -- start 2024-04-21T23:05:43.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.909+0000 7f88e899d700 1 -- start start 2024-04-21T23:05:43.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.909+0000 7f88e899d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88e4103390 0x7f88e419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:43.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.909+0000 7f88e899d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88e4103cb0 0x7f88e419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:43.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.909+0000 7f88e899d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88e410c7c0 0x7f88e4198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:43.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.909+0000 7f88e899d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f88e4112560 con 0x7f88e4103cb0 2024-04-21T23:05:43.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.909+0000 7f88e899d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f88e41123e0 con 0x7f88e410c7c0 2024-04-21T23:05:43.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.909+0000 7f88e899d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f88e41126e0 con 0x7f88e4103390 2024-04-21T23:05:43.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.909+0000 7f88e2ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88e4103cb0 0x7f88e419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:43.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.909+0000 7f88e2ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88e4103cb0 0x7f88e419ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36552/0 (socket says 172.21.15.44:36552) 2024-04-21T23:05:43.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.909+0000 7f88e2ffd700 1 -- 172.21.15.44:0/2169428844 learned_addr learned my addr 172.21.15.44:0/2169428844 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:05:43.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.909+0000 7f88e37fe700 1 --2- 172.21.15.44:0/2169428844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88e4103390 0x7f88e419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:43.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.909+0000 7f88e3fff700 1 --2- 172.21.15.44:0/2169428844 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88e410c7c0 0x7f88e4198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:43.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.909+0000 7f88e2ffd700 1 -- 172.21.15.44:0/2169428844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88e4103390 msgr2=0x7f88e419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:43.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.909+0000 7f88e2ffd700 1 --2- 172.21.15.44:0/2169428844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88e4103390 0x7f88e419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:43.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.909+0000 7f88e2ffd700 1 -- 172.21.15.44:0/2169428844 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88e410c7c0 msgr2=0x7f88e4198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:43.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.909+0000 7f88e2ffd700 1 --2- 172.21.15.44:0/2169428844 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88e410c7c0 0x7f88e4198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:43.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.909+0000 7f88e2ffd700 1 -- 172.21.15.44:0/2169428844 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e4199000 con 0x7f88e4103cb0 2024-04-21T23:05:43.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.910+0000 7f88e2ffd700 1 --2- 172.21.15.44:0/2169428844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88e4103cb0 0x7f88e419ebe0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f88d8000c00 tx=0x7f88d80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:43.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.910+0000 7f88e0ff9700 1 -- 172.21.15.44:0/2169428844 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d8027070 con 0x7f88e4103cb0 2024-04-21T23:05:43.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.910+0000 7f88e0ff9700 1 -- 172.21.15.44:0/2169428844 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88d8004050 con 0x7f88e4103cb0 2024-04-21T23:05:43.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.910+0000 7f88e0ff9700 1 -- 172.21.15.44:0/2169428844 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d8022e10 con 0x7f88e4103cb0 2024-04-21T23:05:43.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.910+0000 7f88e899d700 1 -- 172.21.15.44:0/2169428844 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88e4199290 con 0x7f88e4103cb0 2024-04-21T23:05:43.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.910+0000 7f88e899d700 1 -- 172.21.15.44:0/2169428844 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88e41a5500 con 0x7f88e4103cb0 2024-04-21T23:05:43.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.911+0000 7f88e899d700 1 -- 172.21.15.44:0/2169428844 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88e406b080 con 0x7f88e4103cb0 2024-04-21T23:05:43.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.912+0000 7f88e0ff9700 1 -- 172.21.15.44:0/2169428844 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f88d802a070 con 0x7f88e4103cb0 2024-04-21T23:05:43.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.912+0000 7f88e0ff9700 1 --2- 172.21.15.44:0/2169428844 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f88cc0a2c20 0x7f88cc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:43.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.913+0000 7f88e37fe700 1 --2- 172.21.15.44:0/2169428844 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f88cc0a2c20 0x7f88cc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:43.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.913+0000 7f88e0ff9700 1 -- 172.21.15.44:0/2169428844 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f88d80cf8a0 con 0x7f88e4103cb0 2024-04-21T23:05:43.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.913+0000 7f88e37fe700 1 --2- 172.21.15.44:0/2169428844 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f88cc0a2c20 0x7f88cc0a50e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f88d4005950 tx=0x7f88d400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:43.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:43.917+0000 7f88e0ff9700 1 -- 172.21.15.44:0/2169428844 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f88d8013070 con 0x7f88e4103cb0 2024-04-21T23:05:44.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:44.121+0000 7f88e899d700 1 -- 172.21.15.44:0/2169428844 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f88e4199d50 con 0x7f88e4103cb0 2024-04-21T23:05:44.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:44.122+0000 7f88e0ff9700 1 -- 172.21.15.44:0/2169428844 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f88d809cd30 con 0x7f88e4103cb0 2024-04-21T23:05:44.121 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-21T23:05:44.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:44.124+0000 7f88e899d700 1 -- 172.21.15.44:0/2169428844 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f88cc0a2c20 msgr2=0x7f88cc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:44.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:44.124+0000 7f88e899d700 1 --2- 172.21.15.44:0/2169428844 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f88cc0a2c20 0x7f88cc0a50e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f88d4005950 tx=0x7f88d400a300 comp rx=0 tx=0).stop 2024-04-21T23:05:44.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:44.124+0000 7f88e899d700 1 -- 172.21.15.44:0/2169428844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88e4103cb0 msgr2=0x7f88e419ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:44.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:44.124+0000 7f88e899d700 1 --2- 172.21.15.44:0/2169428844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88e4103cb0 0x7f88e419ebe0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f88d8000c00 tx=0x7f88d80047b0 comp rx=0 tx=0).stop 2024-04-21T23:05:44.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:44.124+0000 7f88e899d700 1 -- 172.21.15.44:0/2169428844 shutdown_connections 2024-04-21T23:05:44.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:44.124+0000 7f88e899d700 1 --2- 172.21.15.44:0/2169428844 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88e410c7c0 0x7f88e4198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:44.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:44.124+0000 7f88e899d700 1 --2- 172.21.15.44:0/2169428844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88e4103390 0x7f88e419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:44.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:44.124+0000 7f88e899d700 1 --2- 172.21.15.44:0/2169428844 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f88cc0a2c20 0x7f88cc0a50e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:44.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:44.124+0000 7f88e899d700 1 --2- 172.21.15.44:0/2169428844 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88e4103cb0 0x7f88e419ebe0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:44.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:44.124+0000 7f88e899d700 1 -- 172.21.15.44:0/2169428844 >> 172.21.15.44:0/2169428844 conn(0x7f88e40fcbe0 msgr2=0x7f88e40fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:44.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:44.124+0000 7f88e899d700 1 -- 172.21.15.44:0/2169428844 shutdown_connections 2024-04-21T23:05:44.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:44.124+0000 7f88e899d700 1 -- 172.21.15.44:0/2169428844 wait complete. 2024-04-21T23:05:44.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:44 smithi044 ceph-mon[27934]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:44.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:44 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2169428844' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:05:44.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:44 smithi069 ceph-mon[31816]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:44.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:44 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2169428844' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:05:44.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:44 smithi055 ceph-mon[31117]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:44.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:44 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2169428844' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:05:44.897 INFO:tasks.cephadm:Adding mds.a on smithi044 2024-04-21T23:05:44.898 INFO:tasks.cephadm:Adding mds.d on smithi044 2024-04-21T23:05:44.898 INFO:tasks.cephadm:Adding mds.g on smithi044 2024-04-21T23:05:44.898 INFO:tasks.cephadm:Adding mds.j on smithi044 2024-04-21T23:05:44.898 INFO:tasks.cephadm:Adding mds.b on smithi055 2024-04-21T23:05:44.898 INFO:tasks.cephadm:Adding mds.e on smithi055 2024-04-21T23:05:44.898 INFO:tasks.cephadm:Adding mds.h on smithi055 2024-04-21T23:05:44.898 INFO:tasks.cephadm:Adding mds.k on smithi055 2024-04-21T23:05:44.898 INFO:tasks.cephadm:Adding mds.c on smithi069 2024-04-21T23:05:44.898 INFO:tasks.cephadm:Adding mds.f on smithi069 2024-04-21T23:05:44.898 INFO:tasks.cephadm:Adding mds.i on smithi069 2024-04-21T23:05:44.898 INFO:tasks.cephadm:Adding mds.l on smithi069 2024-04-21T23:05:44.898 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi055=b;smithi055=e;smithi055=h;smithi055=k;smithi069=c;smithi069=f;smithi069=i;smithi069=l' 2024-04-21T23:05:45.148 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:05:46.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:46 smithi055 ceph-mon[31117]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:46.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:46 smithi069 ceph-mon[31816]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:46.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:46 smithi044 ceph-mon[27934]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:46.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.908+0000 7fcca4847700 1 -- 172.21.15.69:0/1199388484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca0102d20 msgr2=0x7fcca0105110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:46.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.908+0000 7fcca4847700 1 --2- 172.21.15.69:0/1199388484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca0102d20 0x7fcca0105110 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcc90009990 tx=0x7fcc90009ca0 comp rx=0 tx=0).stop 2024-04-21T23:05:46.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.908+0000 7fcca4847700 1 -- 172.21.15.69:0/1199388484 shutdown_connections 2024-04-21T23:05:46.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.908+0000 7fcca4847700 1 --2- 172.21.15.69:0/1199388484 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcca0107f80 0x7fcca010a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:46.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.908+0000 7fcca4847700 1 --2- 172.21.15.69:0/1199388484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca0102d20 0x7fcca0105110 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:46.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.908+0000 7fcca4847700 1 --2- 172.21.15.69:0/1199388484 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcca0105650 0x7fcca0107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:46.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.908+0000 7fcca4847700 1 -- 172.21.15.69:0/1199388484 >> 172.21.15.69:0/1199388484 conn(0x7fcca00fc730 msgr2=0x7fcca00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:46.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.908+0000 7fcca4847700 1 -- 172.21.15.69:0/1199388484 shutdown_connections 2024-04-21T23:05:46.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.908+0000 7fcca4847700 1 -- 172.21.15.69:0/1199388484 wait complete. 2024-04-21T23:05:46.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.909+0000 7fcca4847700 1 Processor -- start 2024-04-21T23:05:46.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.909+0000 7fcca4847700 1 -- start start 2024-04-21T23:05:46.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.909+0000 7fcca4847700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcca0102d20 0x7fcca0198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:46.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.909+0000 7fcca4847700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca0105650 0x7fcca0198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:46.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.909+0000 7fcca4847700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcca0107f80 0x7fcca019fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:46.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.909+0000 7fcca4847700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcca010df00 con 0x7fcca0102d20 2024-04-21T23:05:46.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.909+0000 7fcca4847700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcca010dd80 con 0x7fcca0107f80 2024-04-21T23:05:46.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.909+0000 7fcca4847700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcca010e080 con 0x7fcca0105650 2024-04-21T23:05:46.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.909+0000 7fcc9dd9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca0105650 0x7fcca0198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:46.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.909+0000 7fcc9dd9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca0105650 0x7fcca0198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55490/0 (socket says 172.21.15.69:55490) 2024-04-21T23:05:46.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.909+0000 7fcc9dd9b700 1 -- 172.21.15.69:0/1883294525 learned_addr learned my addr 172.21.15.69:0/1883294525 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-21T23:05:46.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.909+0000 7fcc9ed9d700 1 --2- 172.21.15.69:0/1883294525 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcca0107f80 0x7fcca019fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:46.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.909+0000 7fcc9e59c700 1 --2- 172.21.15.69:0/1883294525 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcca0102d20 0x7fcca0198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:46.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.910+0000 7fcc9dd9b700 1 -- 172.21.15.69:0/1883294525 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcca0107f80 msgr2=0x7fcca019fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:46.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.910+0000 7fcc9dd9b700 1 --2- 172.21.15.69:0/1883294525 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcca0107f80 0x7fcca019fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:46.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.910+0000 7fcc9dd9b700 1 -- 172.21.15.69:0/1883294525 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcca0102d20 msgr2=0x7fcca0198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:46.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.910+0000 7fcc9dd9b700 1 --2- 172.21.15.69:0/1883294525 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcca0102d20 0x7fcca0198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:46.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.910+0000 7fcc9dd9b700 1 -- 172.21.15.69:0/1883294525 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca01a01d0 con 0x7fcca0105650 2024-04-21T23:05:46.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.910+0000 7fcc9dd9b700 1 --2- 172.21.15.69:0/1883294525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca0105650 0x7fcca0198ab0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcc9400b560 tx=0x7fcc9400b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:46.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.910+0000 7fcc8f7fe700 1 -- 172.21.15.69:0/1883294525 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc94015020 con 0x7fcca0105650 2024-04-21T23:05:46.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.910+0000 7fcca4847700 1 -- 172.21.15.69:0/1883294525 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcca01a04c0 con 0x7fcca0105650 2024-04-21T23:05:46.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.910+0000 7fcc8f7fe700 1 -- 172.21.15.69:0/1883294525 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc94011d10 con 0x7fcca0105650 2024-04-21T23:05:46.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.910+0000 7fcc8f7fe700 1 -- 172.21.15.69:0/1883294525 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9400f2e0 con 0x7fcca0105650 2024-04-21T23:05:46.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.910+0000 7fcca4847700 1 -- 172.21.15.69:0/1883294525 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcca01a0a80 con 0x7fcca0105650 2024-04-21T23:05:46.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.911+0000 7fcca4847700 1 -- 172.21.15.69:0/1883294525 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcca0068f20 con 0x7fcca0105650 2024-04-21T23:05:46.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.912+0000 7fcc8f7fe700 1 -- 172.21.15.69:0/1883294525 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fcc94018020 con 0x7fcca0105650 2024-04-21T23:05:46.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.913+0000 7fcc8f7fe700 1 --2- 172.21.15.69:0/1883294525 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcc880a2c20 0x7fcc880a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:46.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.913+0000 7fcc8f7fe700 1 -- 172.21.15.69:0/1883294525 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fcc940ce780 con 0x7fcca0105650 2024-04-21T23:05:46.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.913+0000 7fcc9e59c700 1 --2- 172.21.15.69:0/1883294525 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcc880a2c20 0x7fcc880a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:46.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.913+0000 7fcc9e59c700 1 --2- 172.21.15.69:0/1883294525 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcc880a2c20 0x7fcc880a50e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcc9000b5c0 tx=0x7fcc90003960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:46.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:46.916+0000 7fcc8f7fe700 1 -- 172.21.15.69:0/1883294525 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fcc9401f020 con 0x7fcca0105650 2024-04-21T23:05:47.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:47.120+0000 7fcca4847700 1 -- 172.21.15.69:0/1883294525 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi055=b;smithi055=e;smithi055=h;smithi055=k;smithi069=c;smithi069=f;smithi069=i;smithi069=l", "target": ["mon-mgr", ""]}) -- 0x7fcca0062870 con 0x7fcc880a2c20 2024-04-21T23:05:47.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:47.128+0000 7fcc8f7fe700 1 -- 172.21.15.69:0/1883294525 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fcca0062870 con 0x7fcc880a2c20 2024-04-21T23:05:47.128 INFO:teuthology.orchestra.run.smithi069.stdout:Scheduled mds.all update... 2024-04-21T23:05:47.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:47.130+0000 7fcca4847700 1 -- 172.21.15.69:0/1883294525 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcc880a2c20 msgr2=0x7fcc880a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:47.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:47.130+0000 7fcca4847700 1 --2- 172.21.15.69:0/1883294525 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcc880a2c20 0x7fcc880a50e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcc9000b5c0 tx=0x7fcc90003960 comp rx=0 tx=0).stop 2024-04-21T23:05:47.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:47.130+0000 7fcca4847700 1 -- 172.21.15.69:0/1883294525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca0105650 msgr2=0x7fcca0198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:47.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:47.130+0000 7fcca4847700 1 --2- 172.21.15.69:0/1883294525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca0105650 0x7fcca0198ab0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcc9400b560 tx=0x7fcc9400b870 comp rx=0 tx=0).stop 2024-04-21T23:05:47.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:47.130+0000 7fcca4847700 1 -- 172.21.15.69:0/1883294525 shutdown_connections 2024-04-21T23:05:47.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:47.130+0000 7fcca4847700 1 --2- 172.21.15.69:0/1883294525 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcca0107f80 0x7fcca019fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:47.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:47.130+0000 7fcca4847700 1 --2- 172.21.15.69:0/1883294525 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcc880a2c20 0x7fcc880a50e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:47.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:47.130+0000 7fcca4847700 1 --2- 172.21.15.69:0/1883294525 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcca0102d20 0x7fcca0198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:47.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:47.130+0000 7fcca4847700 1 --2- 172.21.15.69:0/1883294525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca0105650 0x7fcca0198ab0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:47.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:47.130+0000 7fcca4847700 1 -- 172.21.15.69:0/1883294525 >> 172.21.15.69:0/1883294525 conn(0x7fcca00fc730 msgr2=0x7fcca01037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:47.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:47.130+0000 7fcca4847700 1 -- 172.21.15.69:0/1883294525 shutdown_connections 2024-04-21T23:05:47.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-21T23:05:47.130+0000 7fcca4847700 1 -- 172.21.15.69:0/1883294525 wait complete. 2024-04-21T23:05:47.776 DEBUG:teuthology.orchestra.run.smithi044:mds.a> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.a.service 2024-04-21T23:05:47.778 DEBUG:teuthology.orchestra.run.smithi044:mds.d> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.d.service 2024-04-21T23:05:47.821 DEBUG:teuthology.orchestra.run.smithi044:mds.g> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.g.service 2024-04-21T23:05:47.864 DEBUG:teuthology.orchestra.run.smithi044:mds.j> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.j.service 2024-04-21T23:05:47.907 DEBUG:teuthology.orchestra.run.smithi055:mds.b> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.b.service 2024-04-21T23:05:47.910 DEBUG:teuthology.orchestra.run.smithi055:mds.e> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.e.service 2024-04-21T23:05:47.953 DEBUG:teuthology.orchestra.run.smithi055:mds.h> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.h.service 2024-04-21T23:05:47.996 DEBUG:teuthology.orchestra.run.smithi055:mds.k> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.k.service 2024-04-21T23:05:48.039 DEBUG:teuthology.orchestra.run.smithi069:mds.c> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.c.service 2024-04-21T23:05:48.041 DEBUG:teuthology.orchestra.run.smithi069:mds.f> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.f.service 2024-04-21T23:05:48.084 DEBUG:teuthology.orchestra.run.smithi069:mds.i> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.i.service 2024-04-21T23:05:48.127 DEBUG:teuthology.orchestra.run.smithi069:mds.l> sudo journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.l.service 2024-04-21T23:05:48.170 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-21T23:05:48.170 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:05:48.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:48 smithi055 ceph-mon[31117]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:48.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:48 smithi055 ceph-mon[31117]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi055=b;smithi055=e;smithi055=h;smithi055=k;smithi069=c;smithi069=f;smithi069=i;smithi069=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:05:48.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:48 smithi055 ceph-mon[31117]: Saving service mds.all spec with placement smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi055=b;smithi055=e;smithi055=h;smithi055=k;smithi069=c;smithi069=f;smithi069=i;smithi069=l;count:12 2024-04-21T23:05:48.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:48 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:48.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:48 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:05:48.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:48 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:48.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:48 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:05:48.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:48 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:48.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:48 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:05:48.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:48 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:05:48.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:48 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:48.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:48 smithi055 ceph-mon[31117]: Deploying daemon mds.l on smithi069 2024-04-21T23:05:48.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.287+0000 7f6f70040640 1 -- 172.21.15.44:0/1275375877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f68101890 msgr2=0x7f6f68101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:48.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.287+0000 7f6f70040640 1 --2- 172.21.15.44:0/1275375877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f68101890 0x7f6f68101d10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6f58009900 tx=0x7f6f5802bc30 comp rx=0 tx=0).stop 2024-04-21T23:05:48.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.287+0000 7f6f70040640 1 -- 172.21.15.44:0/1275375877 shutdown_connections 2024-04-21T23:05:48.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.287+0000 7f6f70040640 1 --2- 172.21.15.44:0/1275375877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f68102250 0x7f6f68108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.287+0000 7f6f70040640 1 --2- 172.21.15.44:0/1275375877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f68101890 0x7f6f68101d10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.287+0000 7f6f70040640 1 --2- 172.21.15.44:0/1275375877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f681006a0 0x7f6f68100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.287+0000 7f6f70040640 1 -- 172.21.15.44:0/1275375877 >> 172.21.15.44:0/1275375877 conn(0x7f6f680fbc50 msgr2=0x7f6f680fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:48.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.287+0000 7f6f70040640 1 -- 172.21.15.44:0/1275375877 shutdown_connections 2024-04-21T23:05:48.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.288+0000 7f6f70040640 1 -- 172.21.15.44:0/1275375877 wait complete. 2024-04-21T23:05:48.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.288+0000 7f6f70040640 1 Processor -- start 2024-04-21T23:05:48.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.288+0000 7f6f70040640 1 -- start start 2024-04-21T23:05:48.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.288+0000 7f6f70040640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f681006a0 0x7f6f68199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:48.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.288+0000 7f6f70040640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f68101890 0x7f6f6819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:48.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.288+0000 7f6f70040640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f68102250 0x7f6f681a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:48.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.288+0000 7f6f70040640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6810d440 con 0x7f6f68102250 2024-04-21T23:05:48.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.288+0000 7f6f70040640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6810d2c0 con 0x7f6f681006a0 2024-04-21T23:05:48.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.288+0000 7f6f70040640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6810d5c0 con 0x7f6f68101890 2024-04-21T23:05:48.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.288+0000 7f6f6e5dd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f68102250 0x7f6f681a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:48.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.288+0000 7f6f6dddc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f681006a0 0x7f6f68199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:48.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.288+0000 7f6f6e5dd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f68102250 0x7f6f681a1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36570/0 (socket says 172.21.15.44:36570) 2024-04-21T23:05:48.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.288+0000 7f6f6e5dd640 1 -- 172.21.15.44:0/1941979870 learned_addr learned my addr 172.21.15.44:0/1941979870 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:05:48.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.288+0000 7f6f6d5db640 1 --2- 172.21.15.44:0/1941979870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f68101890 0x7f6f6819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:48.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.289+0000 7f6f6e5dd640 1 -- 172.21.15.44:0/1941979870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f68101890 msgr2=0x7f6f6819a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:48.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.289+0000 7f6f6e5dd640 1 --2- 172.21.15.44:0/1941979870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f68101890 0x7f6f6819a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.289+0000 7f6f6e5dd640 1 -- 172.21.15.44:0/1941979870 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f681006a0 msgr2=0x7f6f68199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:48.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.289+0000 7f6f6e5dd640 1 --2- 172.21.15.44:0/1941979870 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f681006a0 0x7f6f68199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.289+0000 7f6f6e5dd640 1 -- 172.21.15.44:0/1941979870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f681a18f0 con 0x7f6f68102250 2024-04-21T23:05:48.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.289+0000 7f6f6e5dd640 1 --2- 172.21.15.44:0/1941979870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f68102250 0x7f6f681a1250 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6f6000eb10 tx=0x7f6f6000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:48.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.289+0000 7f6f5effd640 1 -- 172.21.15.44:0/1941979870 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f60004b20 con 0x7f6f68102250 2024-04-21T23:05:48.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.289+0000 7f6f5effd640 1 -- 172.21.15.44:0/1941979870 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f60004cc0 con 0x7f6f68102250 2024-04-21T23:05:48.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.289+0000 7f6f70040640 1 -- 172.21.15.44:0/1941979870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f681a1be0 con 0x7f6f68102250 2024-04-21T23:05:48.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.289+0000 7f6f5effd640 1 -- 172.21.15.44:0/1941979870 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f60010760 con 0x7f6f68102250 2024-04-21T23:05:48.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.289+0000 7f6f70040640 1 -- 172.21.15.44:0/1941979870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f681a2100 con 0x7f6f68102250 2024-04-21T23:05:48.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.290+0000 7f6f70040640 1 -- 172.21.15.44:0/1941979870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f681a2330 con 0x7f6f68102250 2024-04-21T23:05:48.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.291+0000 7f6f5effd640 1 -- 172.21.15.44:0/1941979870 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6f60005020 con 0x7f6f68102250 2024-04-21T23:05:48.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.292+0000 7f6f5effd640 1 --2- 172.21.15.44:0/1941979870 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6f540a2c10 0x7f6f540a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:48.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.292+0000 7f6f6dddc640 1 --2- 172.21.15.44:0/1941979870 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6f540a2c10 0x7f6f540a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:48.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.292+0000 7f6f5effd640 1 -- 172.21.15.44:0/1941979870 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f6f600cf170 con 0x7f6f68102250 2024-04-21T23:05:48.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.292+0000 7f6f6dddc640 1 --2- 172.21.15.44:0/1941979870 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6f540a2c10 0x7f6f540a50d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6f6819ab80 tx=0x7f6f6400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:48.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.296+0000 7f6f5effd640 1 -- 172.21.15.44:0/1941979870 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6f6009c4d0 con 0x7f6f68102250 2024-04-21T23:05:48.427 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:48 smithi069 ceph-mon[31816]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:48.427 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:48 smithi069 ceph-mon[31816]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi055=b;smithi055=e;smithi055=h;smithi055=k;smithi069=c;smithi069=f;smithi069=i;smithi069=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:05:48.427 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:48 smithi069 ceph-mon[31816]: Saving service mds.all spec with placement smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi055=b;smithi055=e;smithi055=h;smithi055=k;smithi069=c;smithi069=f;smithi069=i;smithi069=l;count:12 2024-04-21T23:05:48.427 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:48 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:48.427 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:48 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:05:48.427 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:48 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:48.427 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:48 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:05:48.427 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:48 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:48.427 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:48 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:05:48.427 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:48 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:05:48.427 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:48 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:48.428 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:48 smithi069 ceph-mon[31816]: Deploying daemon mds.l on smithi069 2024-04-21T23:05:48.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:48 smithi044 ceph-mon[27934]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:48.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:48 smithi044 ceph-mon[27934]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi055=b;smithi055=e;smithi055=h;smithi055=k;smithi069=c;smithi069=f;smithi069=i;smithi069=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:05:48.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:48 smithi044 ceph-mon[27934]: Saving service mds.all spec with placement smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi055=b;smithi055=e;smithi055=h;smithi055=k;smithi069=c;smithi069=f;smithi069=i;smithi069=l;count:12 2024-04-21T23:05:48.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:48 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:48.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:48 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:05:48.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:48 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:48.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:48 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:05:48.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:48 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:48.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:48 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:05:48.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:48 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:05:48.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:48 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:48.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:48 smithi044 ceph-mon[27934]: Deploying daemon mds.l on smithi069 2024-04-21T23:05:48.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.483+0000 7f6f70040640 1 -- 172.21.15.44:0/1941979870 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6f681a25a0 con 0x7f6f68102250 2024-04-21T23:05:48.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.484+0000 7f6f5effd640 1 -- 172.21.15.44:0/1941979870 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18360 (secure 0 0 0) 0x7f6f6009bdd0 con 0x7f6f68102250 2024-04-21T23:05:48.484 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:05:48.484 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":82,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:05:36.159839+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:05:48.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.486+0000 7f6f70040640 1 -- 172.21.15.44:0/1941979870 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6f540a2c10 msgr2=0x7f6f540a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:48.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.486+0000 7f6f70040640 1 --2- 172.21.15.44:0/1941979870 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6f540a2c10 0x7f6f540a50d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6f6819ab80 tx=0x7f6f6400b390 comp rx=0 tx=0).stop 2024-04-21T23:05:48.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.486+0000 7f6f70040640 1 -- 172.21.15.44:0/1941979870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f68102250 msgr2=0x7f6f681a1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:48.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.486+0000 7f6f70040640 1 --2- 172.21.15.44:0/1941979870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f68102250 0x7f6f681a1250 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6f6000eb10 tx=0x7f6f6000c2d0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.486+0000 7f6f70040640 1 -- 172.21.15.44:0/1941979870 shutdown_connections 2024-04-21T23:05:48.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.486+0000 7f6f70040640 1 --2- 172.21.15.44:0/1941979870 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6f540a2c10 0x7f6f540a50d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.486+0000 7f6f70040640 1 --2- 172.21.15.44:0/1941979870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f68101890 0x7f6f6819a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.486+0000 7f6f70040640 1 --2- 172.21.15.44:0/1941979870 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f68102250 0x7f6f681a1250 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.486+0000 7f6f70040640 1 --2- 172.21.15.44:0/1941979870 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f681006a0 0x7f6f68199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.486+0000 7f6f70040640 1 -- 172.21.15.44:0/1941979870 >> 172.21.15.44:0/1941979870 conn(0x7f6f680fbc50 msgr2=0x7f6f681094a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:48.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.486+0000 7f6f70040640 1 -- 172.21.15.44:0/1941979870 shutdown_connections 2024-04-21T23:05:48.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.486+0000 7f6f70040640 1 -- 172.21.15.44:0/1941979870 wait complete. 2024-04-21T23:05:48.497 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:01:47.507503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T23:05:48.497 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:05:48.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.602+0000 7f7295185640 1 -- 172.21.15.44:0/1176592828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7290103350 msgr2=0x7f7290109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:48.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.602+0000 7f7295185640 1 --2- 172.21.15.44:0/1176592828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7290103350 0x7f7290109be0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7280009ca0 tx=0x7f728002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.602+0000 7f7295185640 1 -- 172.21.15.44:0/1176592828 shutdown_connections 2024-04-21T23:05:48.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.602+0000 7f7295185640 1 --2- 172.21.15.44:0/1176592828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7290102990 0x7f7290102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.602+0000 7f7295185640 1 --2- 172.21.15.44:0/1176592828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7290103350 0x7f7290109be0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.602+0000 7f7295185640 1 --2- 172.21.15.44:0/1176592828 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72901017a0 0x7f7290101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.602+0000 7f7295185640 1 -- 172.21.15.44:0/1176592828 >> 172.21.15.44:0/1176592828 conn(0x7f72900fcd10 msgr2=0x7f72900ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:48.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.602+0000 7f7295185640 1 -- 172.21.15.44:0/1176592828 shutdown_connections 2024-04-21T23:05:48.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.602+0000 7f7295185640 1 -- 172.21.15.44:0/1176592828 wait complete. 2024-04-21T23:05:48.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.603+0000 7f7295185640 1 Processor -- start 2024-04-21T23:05:48.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.603+0000 7f7295185640 1 -- start start 2024-04-21T23:05:48.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.603+0000 7f7295185640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72901017a0 0x7f7290197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:48.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.603+0000 7f7295185640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7290102990 0x7f7290197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:48.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f7295185640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7290103350 0x7f729019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:48.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f7295185640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f729010d420 con 0x7f72901017a0 2024-04-21T23:05:48.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f7295185640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f729010d2a0 con 0x7f7290103350 2024-04-21T23:05:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f7295185640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f729010d5a0 con 0x7f7290102990 2024-04-21T23:05:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f728ed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72901017a0 0x7f7290197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f728ed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72901017a0 0x7f7290197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36586/0 (socket says 172.21.15.44:36586) 2024-04-21T23:05:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f728ed9d640 1 -- 172.21.15.44:0/2919876350 learned_addr learned my addr 172.21.15.44:0/2919876350 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:05:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f728e59c640 1 --2- 172.21.15.44:0/2919876350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7290102990 0x7f7290197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f728f59e640 1 --2- 172.21.15.44:0/2919876350 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7290103350 0x7f729019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f728ed9d640 1 -- 172.21.15.44:0/2919876350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7290102990 msgr2=0x7f7290197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f728ed9d640 1 --2- 172.21.15.44:0/2919876350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7290102990 0x7f7290197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f728ed9d640 1 -- 172.21.15.44:0/2919876350 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7290103350 msgr2=0x7f729019f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f728ed9d640 1 --2- 172.21.15.44:0/2919876350 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7290103350 0x7f729019f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f728ed9d640 1 -- 172.21.15.44:0/2919876350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f729019f690 con 0x7f72901017a0 2024-04-21T23:05:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f728f59e640 1 --2- 172.21.15.44:0/2919876350 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7290103350 0x7f729019f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:05:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f728ed9d640 1 --2- 172.21.15.44:0/2919876350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72901017a0 0x7f7290197a70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f728800c970 tx=0x7f728800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:48.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f7287fff640 1 -- 172.21.15.44:0/2919876350 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7288004220 con 0x7f72901017a0 2024-04-21T23:05:48.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f7295185640 1 -- 172.21.15.44:0/2919876350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f729019f980 con 0x7f72901017a0 2024-04-21T23:05:48.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f7287fff640 1 -- 172.21.15.44:0/2919876350 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72880043c0 con 0x7f72901017a0 2024-04-21T23:05:48.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f7287fff640 1 -- 172.21.15.44:0/2919876350 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728800f450 con 0x7f72901017a0 2024-04-21T23:05:48.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.604+0000 7f7295185640 1 -- 172.21.15.44:0/2919876350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f729019fed0 con 0x7f72901017a0 2024-04-21T23:05:48.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.605+0000 7f7295185640 1 -- 172.21.15.44:0/2919876350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7290108ac0 con 0x7f72901017a0 2024-04-21T23:05:48.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.606+0000 7f7287fff640 1 -- 172.21.15.44:0/2919876350 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f72880079b0 con 0x7f72901017a0 2024-04-21T23:05:48.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.607+0000 7f7287fff640 1 --2- 172.21.15.44:0/2919876350 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f727c0a2b90 0x7f727c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:48.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.607+0000 7f728e59c640 1 --2- 172.21.15.44:0/2919876350 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f727c0a2b90 0x7f727c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:48.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.607+0000 7f7287fff640 1 -- 172.21.15.44:0/2919876350 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f72880ce9e0 con 0x7f72901017a0 2024-04-21T23:05:48.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.607+0000 7f728e59c640 1 --2- 172.21.15.44:0/2919876350 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f727c0a2b90 0x7f727c0a5050 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7278005fd0 tx=0x7f7278005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:48.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.612+0000 7f7287fff640 1 -- 172.21.15.44:0/2919876350 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7288014030 con 0x7f72901017a0 2024-04-21T23:05:48.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.797+0000 7f7295185640 1 -- 172.21.15.44:0/2919876350 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7290062340 con 0x7f72901017a0 2024-04-21T23:05:48.797 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.797+0000 7f7287fff640 1 -- 172.21.15.44:0/2919876350 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f728809bde0 con 0x7f72901017a0 2024-04-21T23:05:48.797 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-21T23:05:48.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.799+0000 7f7295185640 1 -- 172.21.15.44:0/2919876350 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f727c0a2b90 msgr2=0x7f727c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:48.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.799+0000 7f7295185640 1 --2- 172.21.15.44:0/2919876350 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f727c0a2b90 0x7f727c0a5050 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7278005fd0 tx=0x7f7278005850 comp rx=0 tx=0).stop 2024-04-21T23:05:48.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.799+0000 7f7295185640 1 -- 172.21.15.44:0/2919876350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72901017a0 msgr2=0x7f7290197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:48.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.799+0000 7f7295185640 1 --2- 172.21.15.44:0/2919876350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72901017a0 0x7f7290197a70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f728800c970 tx=0x7f728800ce30 comp rx=0 tx=0).stop 2024-04-21T23:05:48.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.799+0000 7f7295185640 1 -- 172.21.15.44:0/2919876350 shutdown_connections 2024-04-21T23:05:48.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.799+0000 7f7295185640 1 --2- 172.21.15.44:0/2919876350 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f727c0a2b90 0x7f727c0a5050 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.799+0000 7f7295185640 1 --2- 172.21.15.44:0/2919876350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7290102990 0x7f7290197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.799+0000 7f7295185640 1 --2- 172.21.15.44:0/2919876350 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7290103350 0x7f729019f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.799+0000 7f7295185640 1 --2- 172.21.15.44:0/2919876350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72901017a0 0x7f7290197a70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.799+0000 7f7295185640 1 -- 172.21.15.44:0/2919876350 >> 172.21.15.44:0/2919876350 conn(0x7f72900fcd10 msgr2=0x7f7290105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:48.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.799+0000 7f7295185640 1 -- 172.21.15.44:0/2919876350 shutdown_connections 2024-04-21T23:05:48.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.799+0000 7f7295185640 1 -- 172.21.15.44:0/2919876350 wait complete. 2024-04-21T23:05:48.810 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:05:48.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.912+0000 7f7c7920e640 1 -- 172.21.15.44:0/3024177343 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c74101890 msgr2=0x7f7c74101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:48.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.912+0000 7f7c7920e640 1 --2- 172.21.15.44:0/3024177343 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c74101890 0x7f7c74101d10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7c68009900 tx=0x7f7c6802bc30 comp rx=0 tx=0).stop 2024-04-21T23:05:48.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.912+0000 7f7c7920e640 1 -- 172.21.15.44:0/3024177343 shutdown_connections 2024-04-21T23:05:48.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.912+0000 7f7c7920e640 1 --2- 172.21.15.44:0/3024177343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c74102250 0x7f7c74108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.912+0000 7f7c7920e640 1 --2- 172.21.15.44:0/3024177343 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c74101890 0x7f7c74101d10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.912+0000 7f7c7920e640 1 --2- 172.21.15.44:0/3024177343 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c741006a0 0x7f7c74100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.912+0000 7f7c7920e640 1 -- 172.21.15.44:0/3024177343 >> 172.21.15.44:0/3024177343 conn(0x7f7c740fbc10 msgr2=0x7f7c740fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:48.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.912+0000 7f7c7920e640 1 -- 172.21.15.44:0/3024177343 shutdown_connections 2024-04-21T23:05:48.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.912+0000 7f7c7920e640 1 -- 172.21.15.44:0/3024177343 wait complete. 2024-04-21T23:05:48.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.913+0000 7f7c7920e640 1 Processor -- start 2024-04-21T23:05:48.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.913+0000 7f7c7920e640 1 -- start start 2024-04-21T23:05:48.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.913+0000 7f7c7920e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c741006a0 0x7f7c7410b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:48.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.913+0000 7f7c7920e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c74101890 0x7f7c74109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:48.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.913+0000 7f7c7920e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c74102250 0x7f7c74109ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:48.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.913+0000 7f7c7920e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7410d440 con 0x7f7c741006a0 2024-04-21T23:05:48.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.913+0000 7f7c7920e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7410d2c0 con 0x7f7c74101890 2024-04-21T23:05:48.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.913+0000 7f7c7920e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7410d5c0 con 0x7f7c74102250 2024-04-21T23:05:48.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.913+0000 7f7c72d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c741006a0 0x7f7c7410b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:48.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.914+0000 7f7c7259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c74101890 0x7f7c74109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:48.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.914+0000 7f7c72d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c741006a0 0x7f7c7410b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36602/0 (socket says 172.21.15.44:36602) 2024-04-21T23:05:48.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.914+0000 7f7c72d9d640 1 -- 172.21.15.44:0/2359660582 learned_addr learned my addr 172.21.15.44:0/2359660582 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:05:48.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.914+0000 7f7c7359e640 1 --2- 172.21.15.44:0/2359660582 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c74102250 0x7f7c74109ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:48.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.914+0000 7f7c72d9d640 1 -- 172.21.15.44:0/2359660582 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c74102250 msgr2=0x7f7c74109ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:48.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.914+0000 7f7c72d9d640 1 --2- 172.21.15.44:0/2359660582 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c74102250 0x7f7c74109ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.914+0000 7f7c72d9d640 1 -- 172.21.15.44:0/2359660582 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c74101890 msgr2=0x7f7c74109660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:48.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.914+0000 7f7c72d9d640 1 --2- 172.21.15.44:0/2359660582 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c74101890 0x7f7c74109660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:48.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.914+0000 7f7c72d9d640 1 -- 172.21.15.44:0/2359660582 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c7410a430 con 0x7f7c741006a0 2024-04-21T23:05:48.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.914+0000 7f7c7359e640 1 --2- 172.21.15.44:0/2359660582 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c74102250 0x7f7c74109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:05:48.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.914+0000 7f7c72d9d640 1 --2- 172.21.15.44:0/2359660582 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c741006a0 0x7f7c7410b020 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7c6400c970 tx=0x7f7c6400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:48.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.914+0000 7f7c5bfff640 1 -- 172.21.15.44:0/2359660582 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c64004220 con 0x7f7c741006a0 2024-04-21T23:05:48.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.914+0000 7f7c5bfff640 1 -- 172.21.15.44:0/2359660582 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c640043c0 con 0x7f7c741006a0 2024-04-21T23:05:48.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.914+0000 7f7c7920e640 1 -- 172.21.15.44:0/2359660582 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c741a1ba0 con 0x7f7c741006a0 2024-04-21T23:05:48.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.914+0000 7f7c5bfff640 1 -- 172.21.15.44:0/2359660582 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6400f450 con 0x7f7c741006a0 2024-04-21T23:05:48.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.914+0000 7f7c7920e640 1 -- 172.21.15.44:0/2359660582 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c741a20c0 con 0x7f7c741006a0 2024-04-21T23:05:48.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.915+0000 7f7c7920e640 1 -- 172.21.15.44:0/2359660582 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c740685f0 con 0x7f7c741006a0 2024-04-21T23:05:48.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.916+0000 7f7c5bfff640 1 -- 172.21.15.44:0/2359660582 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7c640079b0 con 0x7f7c741006a0 2024-04-21T23:05:48.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.917+0000 7f7c5bfff640 1 --2- 172.21.15.44:0/2359660582 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f7c5c0a2b90 0x7f7c5c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:48.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.917+0000 7f7c7259c640 1 --2- 172.21.15.44:0/2359660582 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f7c5c0a2b90 0x7f7c5c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:48.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.917+0000 7f7c5bfff640 1 -- 172.21.15.44:0/2359660582 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f7c640ce9b0 con 0x7f7c741006a0 2024-04-21T23:05:48.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.917+0000 7f7c7259c640 1 --2- 172.21.15.44:0/2359660582 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f7c5c0a2b90 0x7f7c5c0a5050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7c6802c7b0 tx=0x7f7c68005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:48.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:48.921+0000 7f7c5bfff640 1 -- 172.21.15.44:0/2359660582 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7c64014030 con 0x7f7c741006a0 2024-04-21T23:05:49.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:49.146+0000 7f7c7920e640 1 -- 172.21.15.44:0/2359660582 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c741079c0 con 0x7f7c741006a0 2024-04-21T23:05:49.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:49.146+0000 7f7c5bfff640 1 -- 172.21.15.44:0/2359660582 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f7c6409bdb0 con 0x7f7c741006a0 2024-04-21T23:05:49.146 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:05:49.146 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-21T23:05:49.146 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 1 2024-04-21T23:05:49.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:49.148+0000 7f7c7920e640 1 -- 172.21.15.44:0/2359660582 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f7c5c0a2b90 msgr2=0x7f7c5c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:49.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:49.148+0000 7f7c7920e640 1 --2- 172.21.15.44:0/2359660582 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f7c5c0a2b90 0x7f7c5c0a5050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7c6802c7b0 tx=0x7f7c68005fd0 comp rx=0 tx=0).stop 2024-04-21T23:05:49.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:49.148+0000 7f7c7920e640 1 -- 172.21.15.44:0/2359660582 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c741006a0 msgr2=0x7f7c7410b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:49.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:49.148+0000 7f7c7920e640 1 --2- 172.21.15.44:0/2359660582 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c741006a0 0x7f7c7410b020 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7c6400c970 tx=0x7f7c6400ce30 comp rx=0 tx=0).stop 2024-04-21T23:05:49.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:49.148+0000 7f7c7920e640 1 -- 172.21.15.44:0/2359660582 shutdown_connections 2024-04-21T23:05:49.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:49.148+0000 7f7c7920e640 1 --2- 172.21.15.44:0/2359660582 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f7c5c0a2b90 0x7f7c5c0a5050 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:49.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:49.148+0000 7f7c7920e640 1 --2- 172.21.15.44:0/2359660582 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c74102250 0x7f7c74109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:49.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:49.148+0000 7f7c7920e640 1 --2- 172.21.15.44:0/2359660582 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c74101890 0x7f7c74109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:49.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:49.148+0000 7f7c7920e640 1 --2- 172.21.15.44:0/2359660582 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7c741006a0 0x7f7c7410b020 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:49.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:49.148+0000 7f7c7920e640 1 -- 172.21.15.44:0/2359660582 >> 172.21.15.44:0/2359660582 conn(0x7f7c740fbc10 msgr2=0x7f7c74104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:49.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:49.148+0000 7f7c7920e640 1 -- 172.21.15.44:0/2359660582 shutdown_connections 2024-04-21T23:05:49.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:49.148+0000 7f7c7920e640 1 -- 172.21.15.44:0/2359660582 wait complete. 2024-04-21T23:05:49.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:49 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1941979870' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:05:49.410 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:49 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2919876350' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:05:49.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:49 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1941979870' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:05:49.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:49 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2919876350' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:05:49.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:49 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1941979870' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:05:49.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:49 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2919876350' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:05:49.964 INFO:journalctl@ceph.mds.l.smithi069.stdout:Apr 21 23:05:49 smithi069 systemd[1]: Starting Ceph mds.l for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:05:50.386 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:50 smithi069 ceph-mon[31816]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:50.386 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:50 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2359660582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:05:50.386 INFO:journalctl@ceph.mds.l.smithi069.stdout:Apr 21 23:05:50 smithi069 podman[62248]: 2024-04-21 23:05:50.004533163 +0000 UTC m=+0.017865234 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:05:50.386 INFO:journalctl@ceph.mds.l.smithi069.stdout:Apr 21 23:05:50 smithi069 podman[62248]: 2024-04-21 23:05:50.14307698 +0000 UTC m=+0.156409030 container create 4512bfa6f430ef16036785807c87e7ce38128a41f2c8e90473903b94d33c8c5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-21T23:05:50.386 INFO:journalctl@ceph.mds.l.smithi069.stdout:Apr 21 23:05:50 smithi069 podman[62248]: 2024-04-21 23:05:50.387478737 +0000 UTC m=+0.400810795 container init 4512bfa6f430ef16036785807c87e7ce38128a41f2c8e90473903b94d33c8c5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-l, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-21T23:05:50.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:50 smithi044 ceph-mon[27934]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:50.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:50 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2359660582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:05:50.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:50 smithi055 ceph-mon[31117]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:50.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:50 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2359660582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:05:50.714 INFO:journalctl@ceph.mds.l.smithi069.stdout:Apr 21 23:05:50 smithi069 podman[62248]: 2024-04-21 23:05:50.390077319 +0000 UTC m=+0.403409370 container start 4512bfa6f430ef16036785807c87e7ce38128a41f2c8e90473903b94d33c8c5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-820a2f8) 2024-04-21T23:05:50.714 INFO:journalctl@ceph.mds.l.smithi069.stdout:Apr 21 23:05:50 smithi069 ceph-mds[62267]: -- 172.21.15.69:0/1879130703 <== mon.1 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5653656a1a40 con 0x565365675000 2024-04-21T23:05:50.714 INFO:journalctl@ceph.mds.l.smithi069.stdout:Apr 21 23:05:50 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-l[62263]: starting mds.l at 2024-04-21T23:05:50.714 INFO:journalctl@ceph.mds.l.smithi069.stdout:Apr 21 23:05:50 smithi069 bash[62248]: 4512bfa6f430ef16036785807c87e7ce38128a41f2c8e90473903b94d33c8c5c 2024-04-21T23:05:50.715 INFO:journalctl@ceph.mds.l.smithi069.stdout:Apr 21 23:05:50 smithi069 systemd[1]: Started Ceph mds.l for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:05:51.160 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:05:51.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.275+0000 7fbed6c8d640 1 -- 172.21.15.44:0/1720927652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbed0102250 msgr2=0x7fbed0108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:51.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.275+0000 7fbed6c8d640 1 --2- 172.21.15.44:0/1720927652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbed0102250 0x7fbed0108ae0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbec8009ca0 tx=0x7fbec802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:05:51.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.275+0000 7fbed6c8d640 1 -- 172.21.15.44:0/1720927652 shutdown_connections 2024-04-21T23:05:51.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.275+0000 7fbed6c8d640 1 --2- 172.21.15.44:0/1720927652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed0101890 0x7fbed0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:51.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.275+0000 7fbed6c8d640 1 --2- 172.21.15.44:0/1720927652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbed0102250 0x7fbed0108ae0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:51.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.275+0000 7fbed6c8d640 1 --2- 172.21.15.44:0/1720927652 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbed01006a0 0x7fbed0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:51.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.275+0000 7fbed6c8d640 1 -- 172.21.15.44:0/1720927652 >> 172.21.15.44:0/1720927652 conn(0x7fbed00fbc50 msgr2=0x7fbed00fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:51.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.275+0000 7fbed6c8d640 1 -- 172.21.15.44:0/1720927652 shutdown_connections 2024-04-21T23:05:51.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.275+0000 7fbed6c8d640 1 -- 172.21.15.44:0/1720927652 wait complete. 2024-04-21T23:05:51.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.275+0000 7fbed6c8d640 1 Processor -- start 2024-04-21T23:05:51.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.276+0000 7fbed6c8d640 1 -- start start 2024-04-21T23:05:51.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.276+0000 7fbed6c8d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed01006a0 0x7fbed006a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:51.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.276+0000 7fbed6c8d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbed0101890 0x7fbed019c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:51.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.276+0000 7fbed6c8d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbed0102250 0x7fbed019ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:51.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.276+0000 7fbed6c8d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbed010c300 con 0x7fbed0102250 2024-04-21T23:05:51.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.276+0000 7fbed6c8d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbed010c180 con 0x7fbed0101890 2024-04-21T23:05:51.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.276+0000 7fbed6c8d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbed010c480 con 0x7fbed01006a0 2024-04-21T23:05:51.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.276+0000 7fbed522a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbed0102250 0x7fbed019ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:51.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.276+0000 7fbed522a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbed0102250 0x7fbed019ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40824/0 (socket says 172.21.15.44:40824) 2024-04-21T23:05:51.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.276+0000 7fbed522a640 1 -- 172.21.15.44:0/971932575 learned_addr learned my addr 172.21.15.44:0/971932575 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:05:51.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.276+0000 7fbed4a29640 1 --2- 172.21.15.44:0/971932575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed01006a0 0x7fbed006a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:51.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.276+0000 7fbecffff640 1 --2- 172.21.15.44:0/971932575 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbed0101890 0x7fbed019c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:51.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.276+0000 7fbed522a640 1 -- 172.21.15.44:0/971932575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed01006a0 msgr2=0x7fbed006a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:51.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.276+0000 7fbed522a640 1 --2- 172.21.15.44:0/971932575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed01006a0 0x7fbed006a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:51.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.276+0000 7fbed522a640 1 -- 172.21.15.44:0/971932575 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbed0101890 msgr2=0x7fbed019c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:51.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.276+0000 7fbed522a640 1 --2- 172.21.15.44:0/971932575 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbed0101890 0x7fbed019c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:51.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.276+0000 7fbed522a640 1 -- 172.21.15.44:0/971932575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed019f690 con 0x7fbed0102250 2024-04-21T23:05:51.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.276+0000 7fbed522a640 1 --2- 172.21.15.44:0/971932575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbed0102250 0x7fbed019ed80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbec8037040 tx=0x7fbec8032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:51.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.276+0000 7fbecdffb640 1 -- 172.21.15.44:0/971932575 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8041020 con 0x7fbed0102250 2024-04-21T23:05:51.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.277+0000 7fbed6c8d640 1 -- 172.21.15.44:0/971932575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed019f8c0 con 0x7fbed0102250 2024-04-21T23:05:51.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.277+0000 7fbecdffb640 1 -- 172.21.15.44:0/971932575 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbec8034cd0 con 0x7fbed0102250 2024-04-21T23:05:51.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.277+0000 7fbecdffb640 1 -- 172.21.15.44:0/971932575 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8039040 con 0x7fbed0102250 2024-04-21T23:05:51.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.277+0000 7fbed6c8d640 1 -- 172.21.15.44:0/971932575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbed019fdb0 con 0x7fbed0102250 2024-04-21T23:05:51.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.278+0000 7fbed6c8d640 1 -- 172.21.15.44:0/971932575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbed01079c0 con 0x7fbed0102250 2024-04-21T23:05:51.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.279+0000 7fbecdffb640 1 -- 172.21.15.44:0/971932575 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbec8044020 con 0x7fbed0102250 2024-04-21T23:05:51.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.279+0000 7fbecdffb640 1 --2- 172.21.15.44:0/971932575 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbec00a2b90 0x7fbec00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:51.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.280+0000 7fbecdffb640 1 -- 172.21.15.44:0/971932575 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fbec80f0e90 con 0x7fbed0102250 2024-04-21T23:05:51.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.280+0000 7fbed4a29640 1 --2- 172.21.15.44:0/971932575 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbec00a2b90 0x7fbec00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:51.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.280+0000 7fbed4a29640 1 --2- 172.21.15.44:0/971932575 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbec00a2b90 0x7fbec00a5050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbed01016f0 tx=0x7fbec4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:51.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.283+0000 7fbecdffb640 1 -- 172.21.15.44:0/971932575 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fbec804b020 con 0x7fbed0102250 2024-04-21T23:05:51.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.506+0000 7fbed6c8d640 1 -- 172.21.15.44:0/971932575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbed002d330 con 0x7fbed0102250 2024-04-21T23:05:51.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.507+0000 7fbecdffb640 1 -- 172.21.15.44:0/971932575 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1387 (secure 0 0 0) 0x7fbec80a8090 con 0x7fbed0102250 2024-04-21T23:05:51.507 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:05:51.507 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T23:05:51.507 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 2 2024-04-21T23:05:51.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.509+0000 7fbed6c8d640 1 -- 172.21.15.44:0/971932575 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbec00a2b90 msgr2=0x7fbec00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:51.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.509+0000 7fbed6c8d640 1 --2- 172.21.15.44:0/971932575 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbec00a2b90 0x7fbec00a5050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbed01016f0 tx=0x7fbec4005f90 comp rx=0 tx=0).stop 2024-04-21T23:05:51.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.509+0000 7fbed6c8d640 1 -- 172.21.15.44:0/971932575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbed0102250 msgr2=0x7fbed019ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:51.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.509+0000 7fbed6c8d640 1 --2- 172.21.15.44:0/971932575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbed0102250 0x7fbed019ed80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbec8037040 tx=0x7fbec8032f40 comp rx=0 tx=0).stop 2024-04-21T23:05:51.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.509+0000 7fbed6c8d640 1 -- 172.21.15.44:0/971932575 shutdown_connections 2024-04-21T23:05:51.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.509+0000 7fbed6c8d640 1 --2- 172.21.15.44:0/971932575 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbec00a2b90 0x7fbec00a5050 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:51.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.509+0000 7fbed6c8d640 1 --2- 172.21.15.44:0/971932575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbed0102250 0x7fbed019ed80 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:51.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.509+0000 7fbed6c8d640 1 --2- 172.21.15.44:0/971932575 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbed0101890 0x7fbed019c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:51.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.509+0000 7fbed6c8d640 1 --2- 172.21.15.44:0/971932575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed01006a0 0x7fbed006a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:51.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.509+0000 7fbed6c8d640 1 -- 172.21.15.44:0/971932575 >> 172.21.15.44:0/971932575 conn(0x7fbed00fbc50 msgr2=0x7fbed0104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:51.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.509+0000 7fbed6c8d640 1 -- 172.21.15.44:0/971932575 shutdown_connections 2024-04-21T23:05:51.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:51.509+0000 7fbed6c8d640 1 -- 172.21.15.44:0/971932575 wait complete. 2024-04-21T23:05:51.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:51 smithi044 ceph-mon[27934]: mds.? [v2:172.21.15.69:6832/43117883,v1:172.21.15.69:6833/43117883] up:boot 2024-04-21T23:05:51.518 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:51 smithi044 ceph-mon[27934]: fsmap 1 up:standby 2024-04-21T23:05:51.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:51 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T23:05:51.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:51 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:51.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:51 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:51.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:51 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:51.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:51 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:05:51.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:51 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:05:51.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:51 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:51.519 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:51 smithi044 ceph-mon[27934]: Deploying daemon mds.k on smithi055 2024-04-21T23:05:51.525 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:51 smithi055 ceph-mon[31117]: mds.? [v2:172.21.15.69:6832/43117883,v1:172.21.15.69:6833/43117883] up:boot 2024-04-21T23:05:51.525 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:51 smithi055 ceph-mon[31117]: fsmap 1 up:standby 2024-04-21T23:05:51.525 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:51 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T23:05:51.525 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:51 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:51.525 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:51 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:51.525 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:51 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:51.525 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:51 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:05:51.526 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:51 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:05:51.526 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:51 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:51.526 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:51 smithi055 ceph-mon[31117]: Deploying daemon mds.k on smithi055 2024-04-21T23:05:51.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:51 smithi069 ceph-mon[31816]: mds.? [v2:172.21.15.69:6832/43117883,v1:172.21.15.69:6833/43117883] up:boot 2024-04-21T23:05:51.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:51 smithi069 ceph-mon[31816]: fsmap 1 up:standby 2024-04-21T23:05:51.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:51 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T23:05:51.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:51 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:51.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:51 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:51.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:51 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:51.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:51 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:05:51.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:51 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:05:51.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:51 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:51.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:51 smithi069 ceph-mon[31816]: Deploying daemon mds.k on smithi055 2024-04-21T23:05:52.589 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:52 smithi055 ceph-mon[31117]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:52.589 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:52 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/971932575' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:05:52.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:52 smithi044 ceph-mon[27934]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:52.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:52 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/971932575' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:05:52.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:52 smithi069 ceph-mon[31816]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:52.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:52 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/971932575' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:05:52.964 INFO:journalctl@ceph.mds.k.smithi055.stdout:Apr 21 23:05:52 smithi055 systemd[1]: Starting Ceph mds.k for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:05:53.294 INFO:journalctl@ceph.mds.k.smithi055.stdout:Apr 21 23:05:53 smithi055 podman[60595]: 2024-04-21 23:05:52.90376977 +0000 UTC m=+0.018457041 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:05:53.294 INFO:journalctl@ceph.mds.k.smithi055.stdout:Apr 21 23:05:53 smithi055 podman[60595]: 2024-04-21 23:05:53.025595959 +0000 UTC m=+0.140283221 container create 3800890d2e1efc0e0f4102cb6fb7e676caf462c103554086d1a24f80fe003c8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-k, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:05:53.294 INFO:journalctl@ceph.mds.k.smithi055.stdout:Apr 21 23:05:53 smithi055 podman[60595]: 2024-04-21 23:05:53.295137329 +0000 UTC m=+0.409824594 container init 3800890d2e1efc0e0f4102cb6fb7e676caf462c103554086d1a24f80fe003c8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-k, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:05:53.521 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:05:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.641+0000 7f3a0adec640 1 -- 172.21.15.44:0/1670466098 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a04102250 msgr2=0x7f3a04108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.641+0000 7f3a0adec640 1 --2- 172.21.15.44:0/1670466098 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a04102250 0x7f3a04108ae0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f39fc009ca0 tx=0x7f39fc02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:05:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.641+0000 7f3a0adec640 1 -- 172.21.15.44:0/1670466098 shutdown_connections 2024-04-21T23:05:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.641+0000 7f3a0adec640 1 --2- 172.21.15.44:0/1670466098 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a04102250 0x7f3a04108ae0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.641+0000 7f3a0adec640 1 --2- 172.21.15.44:0/1670466098 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a04101890 0x7f3a04101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.641+0000 7f3a0adec640 1 --2- 172.21.15.44:0/1670466098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a041006a0 0x7f3a04100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.641+0000 7f3a0adec640 1 -- 172.21.15.44:0/1670466098 >> 172.21.15.44:0/1670466098 conn(0x7f3a040fbc10 msgr2=0x7f3a040fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.641+0000 7f3a0adec640 1 -- 172.21.15.44:0/1670466098 shutdown_connections 2024-04-21T23:05:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.641+0000 7f3a0adec640 1 -- 172.21.15.44:0/1670466098 wait complete. 2024-04-21T23:05:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.642+0000 7f3a0adec640 1 Processor -- start 2024-04-21T23:05:53.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.642+0000 7f3a0adec640 1 -- start start 2024-04-21T23:05:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.642+0000 7f3a0adec640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a041006a0 0x7f3a04197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.642+0000 7f3a0adec640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a04101890 0x7f3a04197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.642+0000 7f3a0adec640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a04102250 0x7f3a0419ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.642+0000 7f3a0adec640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0410c320 con 0x7f3a04102250 2024-04-21T23:05:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.642+0000 7f3a0adec640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0410c1a0 con 0x7f3a04101890 2024-04-21T23:05:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.642+0000 7f3a0adec640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0410c4a0 con 0x7f3a041006a0 2024-04-21T23:05:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.642+0000 7f3a09389640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a04102250 0x7f3a0419ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.642+0000 7f3a09389640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a04102250 0x7f3a0419ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40850/0 (socket says 172.21.15.44:40850) 2024-04-21T23:05:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.642+0000 7f3a09389640 1 -- 172.21.15.44:0/3519976384 learned_addr learned my addr 172.21.15.44:0/3519976384 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:05:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.642+0000 7f3a03fff640 1 --2- 172.21.15.44:0/3519976384 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a04101890 0x7f3a04197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.642+0000 7f3a08b88640 1 --2- 172.21.15.44:0/3519976384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a041006a0 0x7f3a04197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.642+0000 7f3a09389640 1 -- 172.21.15.44:0/3519976384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a041006a0 msgr2=0x7f3a04197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.642+0000 7f3a09389640 1 --2- 172.21.15.44:0/3519976384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a041006a0 0x7f3a04197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.642+0000 7f3a09389640 1 -- 172.21.15.44:0/3519976384 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a04101890 msgr2=0x7f3a04197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:53.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.642+0000 7f3a09389640 1 --2- 172.21.15.44:0/3519976384 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a04101890 0x7f3a04197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:53.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.642+0000 7f3a09389640 1 -- 172.21.15.44:0/3519976384 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a0419f590 con 0x7f3a04102250 2024-04-21T23:05:53.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.642+0000 7f3a09389640 1 --2- 172.21.15.44:0/3519976384 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a04102250 0x7f3a0419ef20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f39fc009bf0 tx=0x7f39fc035ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:53.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.643+0000 7f3a01ffb640 1 -- 172.21.15.44:0/3519976384 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39fc039040 con 0x7f3a04102250 2024-04-21T23:05:53.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.643+0000 7f3a01ffb640 1 -- 172.21.15.44:0/3519976384 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39fc044070 con 0x7f3a04102250 2024-04-21T23:05:53.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.643+0000 7f3a0adec640 1 -- 172.21.15.44:0/3519976384 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a0419f820 con 0x7f3a04102250 2024-04-21T23:05:53.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.643+0000 7f3a01ffb640 1 -- 172.21.15.44:0/3519976384 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39fc0043c0 con 0x7f3a04102250 2024-04-21T23:05:53.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.643+0000 7f3a0adec640 1 -- 172.21.15.44:0/3519976384 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a0419fd10 con 0x7f3a04102250 2024-04-21T23:05:53.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.644+0000 7f3a0adec640 1 -- 172.21.15.44:0/3519976384 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a0419ff70 con 0x7f3a04102250 2024-04-21T23:05:53.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.645+0000 7f3a01ffb640 1 -- 172.21.15.44:0/3519976384 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f39fc003a70 con 0x7f3a04102250 2024-04-21T23:05:53.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.646+0000 7f3a01ffb640 1 --2- 172.21.15.44:0/3519976384 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f39f00a2b90 0x7f39f00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:53.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.646+0000 7f3a08b88640 1 --2- 172.21.15.44:0/3519976384 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f39f00a2b90 0x7f39f00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:53.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.646+0000 7f3a01ffb640 1 -- 172.21.15.44:0/3519976384 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f39fc0f18e0 con 0x7f3a04102250 2024-04-21T23:05:53.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.646+0000 7f3a08b88640 1 --2- 172.21.15.44:0/3519976384 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f39f00a2b90 0x7f39f00a5050 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f39ec0058c0 tx=0x7f39ec005c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:53.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.649+0000 7f3a01ffb640 1 -- 172.21.15.44:0/3519976384 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f39fc0bec40 con 0x7f3a04102250 2024-04-21T23:05:53.714 INFO:journalctl@ceph.mds.k.smithi055.stdout:Apr 21 23:05:53 smithi055 podman[60595]: 2024-04-21 23:05:53.298224405 +0000 UTC m=+0.412911674 container start 3800890d2e1efc0e0f4102cb6fb7e676caf462c103554086d1a24f80fe003c8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-21T23:05:53.714 INFO:journalctl@ceph.mds.k.smithi055.stdout:Apr 21 23:05:53 smithi055 ceph-mds[60614]: -- 172.21.15.55:0/641811925 <== mon.2 v2:172.21.15.55:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558f93afda40 con 0x558f93ad0c00 2024-04-21T23:05:53.715 INFO:journalctl@ceph.mds.k.smithi055.stdout:Apr 21 23:05:53 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-k[60610]: starting mds.k at 2024-04-21T23:05:53.715 INFO:journalctl@ceph.mds.k.smithi055.stdout:Apr 21 23:05:53 smithi055 bash[60595]: 3800890d2e1efc0e0f4102cb6fb7e676caf462c103554086d1a24f80fe003c8b 2024-04-21T23:05:53.715 INFO:journalctl@ceph.mds.k.smithi055.stdout:Apr 21 23:05:53 smithi055 systemd[1]: Started Ceph mds.k for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:05:53.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.873+0000 7f3a0adec640 1 -- 172.21.15.44:0/3519976384 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a041a0300 con 0x7f3a04102250 2024-04-21T23:05:53.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.874+0000 7f3a01ffb640 1 -- 172.21.15.44:0/3519976384 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2212 (secure 0 0 0) 0x7f39fc0be540 con 0x7f3a04102250 2024-04-21T23:05:53.874 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:05:53.874 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T23:05:53.874 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 3 2024-04-21T23:05:53.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.876+0000 7f3a0adec640 1 -- 172.21.15.44:0/3519976384 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f39f00a2b90 msgr2=0x7f39f00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:53.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.876+0000 7f3a0adec640 1 --2- 172.21.15.44:0/3519976384 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f39f00a2b90 0x7f39f00a5050 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f39ec0058c0 tx=0x7f39ec005c80 comp rx=0 tx=0).stop 2024-04-21T23:05:53.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.876+0000 7f3a0adec640 1 -- 172.21.15.44:0/3519976384 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a04102250 msgr2=0x7f3a0419ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:53.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.876+0000 7f3a0adec640 1 --2- 172.21.15.44:0/3519976384 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a04102250 0x7f3a0419ef20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f39fc009bf0 tx=0x7f39fc035ae0 comp rx=0 tx=0).stop 2024-04-21T23:05:53.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.876+0000 7f3a0adec640 1 -- 172.21.15.44:0/3519976384 shutdown_connections 2024-04-21T23:05:53.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.876+0000 7f3a0adec640 1 --2- 172.21.15.44:0/3519976384 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f39f00a2b90 0x7f39f00a5050 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:53.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.876+0000 7f3a0adec640 1 --2- 172.21.15.44:0/3519976384 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3a04102250 0x7f3a0419ef20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:53.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.876+0000 7f3a0adec640 1 --2- 172.21.15.44:0/3519976384 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3a04101890 0x7f3a04197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:53.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.876+0000 7f3a0adec640 1 --2- 172.21.15.44:0/3519976384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a041006a0 0x7f3a04197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:53.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.876+0000 7f3a0adec640 1 -- 172.21.15.44:0/3519976384 >> 172.21.15.44:0/3519976384 conn(0x7f3a040fbc10 msgr2=0x7f3a040fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:53.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.876+0000 7f3a0adec640 1 -- 172.21.15.44:0/3519976384 shutdown_connections 2024-04-21T23:05:53.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:53.876+0000 7f3a0adec640 1 -- 172.21.15.44:0/3519976384 wait complete. 2024-04-21T23:05:54.434 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:54 smithi044 ceph-mon[27934]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:54.434 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:54 smithi044 ceph-mon[27934]: mds.? [v2:172.21.15.55:6832/1288947368,v1:172.21.15.55:6833/1288947368] up:boot 2024-04-21T23:05:54.434 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:54 smithi044 ceph-mon[27934]: fsmap 2 up:standby 2024-04-21T23:05:54.434 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T23:05:54.434 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:54.434 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:54.434 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:54.434 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:05:54.434 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:05:54.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:54 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:54.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:54 smithi044 ceph-mon[27934]: Deploying daemon mds.j on smithi044 2024-04-21T23:05:54.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:54 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3519976384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:05:54.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:54 smithi055 ceph-mon[31117]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:54.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:54 smithi055 ceph-mon[31117]: mds.? [v2:172.21.15.55:6832/1288947368,v1:172.21.15.55:6833/1288947368] up:boot 2024-04-21T23:05:54.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:54 smithi055 ceph-mon[31117]: fsmap 2 up:standby 2024-04-21T23:05:54.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:54 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T23:05:54.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:54 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:54.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:54 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:54.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:54 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:54.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:54 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:05:54.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:54 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:05:54.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:54 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:54.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:54 smithi055 ceph-mon[31117]: Deploying daemon mds.j on smithi044 2024-04-21T23:05:54.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:54 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3519976384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:05:54.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:54 smithi069 ceph-mon[31816]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:54.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:54 smithi069 ceph-mon[31816]: mds.? [v2:172.21.15.55:6832/1288947368,v1:172.21.15.55:6833/1288947368] up:boot 2024-04-21T23:05:54.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:54 smithi069 ceph-mon[31816]: fsmap 2 up:standby 2024-04-21T23:05:54.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:54 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T23:05:54.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:54 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:54.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:54 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:54.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:54 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:54.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:54 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:05:54.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:54 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:05:54.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:54 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:54.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:54 smithi069 ceph-mon[31816]: Deploying daemon mds.j on smithi044 2024-04-21T23:05:54.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:54 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3519976384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:05:55.888 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:05:55.912 INFO:journalctl@ceph.mds.j.smithi044.stdout:Apr 21 23:05:55 smithi044 systemd[1]: Starting Ceph mds.j for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:05:55.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.997+0000 7fac56d9f640 1 -- 172.21.15.44:0/1451917160 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac50101780 msgr2=0x7fac50101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:55.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.997+0000 7fac56d9f640 1 --2- 172.21.15.44:0/1451917160 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac50101780 0x7fac50101c00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fac40009900 tx=0x7fac4002bc30 comp rx=0 tx=0).stop 2024-04-21T23:05:55.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.997+0000 7fac56d9f640 1 -- 172.21.15.44:0/1451917160 shutdown_connections 2024-04-21T23:05:55.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.997+0000 7fac56d9f640 1 --2- 172.21.15.44:0/1451917160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac50102140 0x7fac501089d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:55.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.997+0000 7fac56d9f640 1 --2- 172.21.15.44:0/1451917160 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac50101780 0x7fac50101c00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:55.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.997+0000 7fac56d9f640 1 --2- 172.21.15.44:0/1451917160 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac50100590 0x7fac501009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:55.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.997+0000 7fac56d9f640 1 -- 172.21.15.44:0/1451917160 >> 172.21.15.44:0/1451917160 conn(0x7fac500fbb00 msgr2=0x7fac500fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:55.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.997+0000 7fac56d9f640 1 -- 172.21.15.44:0/1451917160 shutdown_connections 2024-04-21T23:05:55.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.997+0000 7fac56d9f640 1 -- 172.21.15.44:0/1451917160 wait complete. 2024-04-21T23:05:56.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.997+0000 7fac56d9f640 1 Processor -- start 2024-04-21T23:05:56.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac56d9f640 1 -- start start 2024-04-21T23:05:56.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac56d9f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac50100590 0x7fac50197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:56.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac56d9f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac50101780 0x7fac50197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:56.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac56d9f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac50102140 0x7fac5019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:56.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac56d9f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fac5010c2a0 con 0x7fac50102140 2024-04-21T23:05:56.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac56d9f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fac5010c120 con 0x7fac50101780 2024-04-21T23:05:56.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac56d9f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fac5010c420 con 0x7fac50100590 2024-04-21T23:05:56.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac5659e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac50102140 0x7fac5019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:56.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac5659e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac50102140 0x7fac5019efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40878/0 (socket says 172.21.15.44:40878) 2024-04-21T23:05:56.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac5659e640 1 -- 172.21.15.44:0/4136578330 learned_addr learned my addr 172.21.15.44:0/4136578330 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:05:56.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac5559c640 1 --2- 172.21.15.44:0/4136578330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac50101780 0x7fac50197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:56.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac55d9d640 1 --2- 172.21.15.44:0/4136578330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac50100590 0x7fac50197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:56.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac5659e640 1 -- 172.21.15.44:0/4136578330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac50100590 msgr2=0x7fac50197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:56.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac5659e640 1 --2- 172.21.15.44:0/4136578330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac50100590 0x7fac50197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:56.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac5659e640 1 -- 172.21.15.44:0/4136578330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac50101780 msgr2=0x7fac50197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:56.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac5659e640 1 --2- 172.21.15.44:0/4136578330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac50101780 0x7fac50197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:56.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac5659e640 1 -- 172.21.15.44:0/4136578330 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5019f620 con 0x7fac50102140 2024-04-21T23:05:56.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac55d9d640 1 --2- 172.21.15.44:0/4136578330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac50100590 0x7fac50197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:05:56.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac5659e640 1 --2- 172.21.15.44:0/4136578330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac50102140 0x7fac5019efb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fac4800eaa0 tx=0x7fac4800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:56.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac46ffd640 1 -- 172.21.15.44:0/4136578330 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4800cdb0 con 0x7fac50102140 2024-04-21T23:05:56.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac46ffd640 1 -- 172.21.15.44:0/4136578330 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac48005020 con 0x7fac50102140 2024-04-21T23:05:56.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac56d9f640 1 -- 172.21.15.44:0/4136578330 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac5019f910 con 0x7fac50102140 2024-04-21T23:05:56.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.998+0000 7fac46ffd640 1 -- 172.21.15.44:0/4136578330 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac480116b0 con 0x7fac50102140 2024-04-21T23:05:56.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:55.999+0000 7fac56d9f640 1 -- 172.21.15.44:0/4136578330 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac5019fe80 con 0x7fac50102140 2024-04-21T23:05:56.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.000+0000 7fac56d9f640 1 -- 172.21.15.44:0/4136578330 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac5004f570 con 0x7fac50102140 2024-04-21T23:05:56.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.001+0000 7fac46ffd640 1 -- 172.21.15.44:0/4136578330 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fac48005560 con 0x7fac50102140 2024-04-21T23:05:56.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.001+0000 7fac46ffd640 1 --2- 172.21.15.44:0/4136578330 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fac3c0a2c00 0x7fac3c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:56.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.001+0000 7fac55d9d640 1 --2- 172.21.15.44:0/4136578330 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fac3c0a2c00 0x7fac3c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:56.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.001+0000 7fac46ffd640 1 -- 172.21.15.44:0/4136578330 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fac480ceec0 con 0x7fac50102140 2024-04-21T23:05:56.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.002+0000 7fac55d9d640 1 --2- 172.21.15.44:0/4136578330 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fac3c0a2c00 0x7fac3c0a50c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fac4c006fd0 tx=0x7fac4c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:56.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.006+0000 7fac46ffd640 1 -- 172.21.15.44:0/4136578330 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fac48016030 con 0x7fac50102140 2024-04-21T23:05:56.185 INFO:journalctl@ceph.mds.j.smithi044.stdout:Apr 21 23:05:55 smithi044 podman[65747]: 2024-04-21 23:05:55.845610955 +0000 UTC m=+0.018497052 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:05:56.185 INFO:journalctl@ceph.mds.j.smithi044.stdout:Apr 21 23:05:55 smithi044 podman[65747]: 2024-04-21 23:05:55.963196335 +0000 UTC m=+0.136082430 container create a058353c7bf20855a3fe8914bc84d667efda97a0592989028146f7861faef1ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, GIT_CLEAN=True) 2024-04-21T23:05:56.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.232+0000 7fac56d9f640 1 -- 172.21.15.44:0/4136578330 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac501a01b0 con 0x7fac50102140 2024-04-21T23:05:56.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.232+0000 7fac46ffd640 1 -- 172.21.15.44:0/4136578330 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2212 (secure 0 0 0) 0x7fac4809c220 con 0x7fac50102140 2024-04-21T23:05:56.232 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:05:56.232 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T23:05:56.233 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 3 2024-04-21T23:05:56.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.235+0000 7fac56d9f640 1 -- 172.21.15.44:0/4136578330 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fac3c0a2c00 msgr2=0x7fac3c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:56.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.235+0000 7fac56d9f640 1 --2- 172.21.15.44:0/4136578330 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fac3c0a2c00 0x7fac3c0a50c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fac4c006fd0 tx=0x7fac4c008040 comp rx=0 tx=0).stop 2024-04-21T23:05:56.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.235+0000 7fac56d9f640 1 -- 172.21.15.44:0/4136578330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac50102140 msgr2=0x7fac5019efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:56.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.235+0000 7fac56d9f640 1 --2- 172.21.15.44:0/4136578330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac50102140 0x7fac5019efb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fac4800eaa0 tx=0x7fac4800ef60 comp rx=0 tx=0).stop 2024-04-21T23:05:56.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.235+0000 7fac56d9f640 1 -- 172.21.15.44:0/4136578330 shutdown_connections 2024-04-21T23:05:56.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.235+0000 7fac56d9f640 1 --2- 172.21.15.44:0/4136578330 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fac3c0a2c00 0x7fac3c0a50c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:56.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.235+0000 7fac56d9f640 1 --2- 172.21.15.44:0/4136578330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fac50102140 0x7fac5019efb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:56.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.235+0000 7fac56d9f640 1 --2- 172.21.15.44:0/4136578330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fac50101780 0x7fac50197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:56.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.235+0000 7fac56d9f640 1 --2- 172.21.15.44:0/4136578330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac50100590 0x7fac50197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:56.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.235+0000 7fac56d9f640 1 -- 172.21.15.44:0/4136578330 >> 172.21.15.44:0/4136578330 conn(0x7fac500fbb00 msgr2=0x7fac500fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:56.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.235+0000 7fac56d9f640 1 -- 172.21.15.44:0/4136578330 shutdown_connections 2024-04-21T23:05:56.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:56.235+0000 7fac56d9f640 1 -- 172.21.15.44:0/4136578330 wait complete. 2024-04-21T23:05:56.685 INFO:journalctl@ceph.mds.j.smithi044.stdout:Apr 21 23:05:56 smithi044 podman[65747]: 2024-04-21 23:05:56.20382754 +0000 UTC m=+0.376713636 container init a058353c7bf20855a3fe8914bc84d667efda97a0592989028146f7861faef1ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-21T23:05:56.685 INFO:journalctl@ceph.mds.j.smithi044.stdout:Apr 21 23:05:56 smithi044 podman[65747]: 2024-04-21 23:05:56.206533862 +0000 UTC m=+0.379419958 container start a058353c7bf20855a3fe8914bc84d667efda97a0592989028146f7861faef1ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-820a2f8) 2024-04-21T23:05:56.685 INFO:journalctl@ceph.mds.j.smithi044.stdout:Apr 21 23:05:56 smithi044 ceph-mds[65805]: -- 172.21.15.44:0/2806256884 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5627c8f59a40 con 0x5627c8f2cc00 2024-04-21T23:05:56.686 INFO:journalctl@ceph.mds.j.smithi044.stdout:Apr 21 23:05:56 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-j[65801]: starting mds.j at 2024-04-21T23:05:56.686 INFO:journalctl@ceph.mds.j.smithi044.stdout:Apr 21 23:05:56 smithi044 bash[65747]: a058353c7bf20855a3fe8914bc84d667efda97a0592989028146f7861faef1ba 2024-04-21T23:05:56.686 INFO:journalctl@ceph.mds.j.smithi044.stdout:Apr 21 23:05:56 smithi044 systemd[1]: Started Ceph mds.j for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:05:56.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:56 smithi044 ceph-mon[27934]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:56.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:56 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/4136578330' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:05:56.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:56 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:56.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:56 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:56.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:56 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:56.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:56 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:05:56.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:56 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:05:56.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:56 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:56.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:56 smithi069 ceph-mon[31816]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:56.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:56 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/4136578330' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:05:56.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:56 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:56.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:56 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:56.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:56 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:56.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:56 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:05:56.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:56 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:05:56.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:56 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:56.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:56 smithi055 ceph-mon[31117]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:56.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:56 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/4136578330' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:05:56.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:56 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:56.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:56 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:56.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:56 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:56.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:56 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:05:56.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:56 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:05:56.716 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:56 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:57.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:57 smithi044 ceph-mon[27934]: Deploying daemon mds.d on smithi044 2024-04-21T23:05:57.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:57 smithi044 ceph-mon[27934]: mds.? [v2:172.21.15.44:6834/634418901,v1:172.21.15.44:6835/634418901] up:boot 2024-04-21T23:05:57.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:57 smithi044 ceph-mon[27934]: fsmap 3 up:standby 2024-04-21T23:05:57.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:57 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T23:05:57.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:57 smithi055 ceph-mon[31117]: Deploying daemon mds.d on smithi044 2024-04-21T23:05:57.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:57 smithi055 ceph-mon[31117]: mds.? [v2:172.21.15.44:6834/634418901,v1:172.21.15.44:6835/634418901] up:boot 2024-04-21T23:05:57.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:57 smithi055 ceph-mon[31117]: fsmap 3 up:standby 2024-04-21T23:05:57.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:57 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T23:05:57.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:57 smithi069 ceph-mon[31816]: Deploying daemon mds.d on smithi044 2024-04-21T23:05:57.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:57 smithi069 ceph-mon[31816]: mds.? [v2:172.21.15.44:6834/634418901,v1:172.21.15.44:6835/634418901] up:boot 2024-04-21T23:05:57.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:57 smithi069 ceph-mon[31816]: fsmap 3 up:standby 2024-04-21T23:05:57.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:57 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T23:05:58.247 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:05:58.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.367+0000 7f6ae88ed640 1 -- 172.21.15.44:0/245501751 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ae4101890 msgr2=0x7f6ae4101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:58.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.367+0000 7f6ae88ed640 1 --2- 172.21.15.44:0/245501751 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ae4101890 0x7f6ae4101d10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009900 tx=0x7f6ad802bc30 comp rx=0 tx=0).stop 2024-04-21T23:05:58.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.367+0000 7f6ae88ed640 1 -- 172.21.15.44:0/245501751 shutdown_connections 2024-04-21T23:05:58.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.367+0000 7f6ae88ed640 1 --2- 172.21.15.44:0/245501751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ae4102250 0x7f6ae4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:58.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.367+0000 7f6ae88ed640 1 --2- 172.21.15.44:0/245501751 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ae4101890 0x7f6ae4101d10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:58.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.367+0000 7f6ae88ed640 1 --2- 172.21.15.44:0/245501751 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6ae41006a0 0x7f6ae4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:58.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.367+0000 7f6ae88ed640 1 -- 172.21.15.44:0/245501751 >> 172.21.15.44:0/245501751 conn(0x7f6ae40fbc10 msgr2=0x7f6ae40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:58.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.368+0000 7f6ae88ed640 1 -- 172.21.15.44:0/245501751 shutdown_connections 2024-04-21T23:05:58.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.368+0000 7f6ae88ed640 1 -- 172.21.15.44:0/245501751 wait complete. 2024-04-21T23:05:58.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.368+0000 7f6ae88ed640 1 Processor -- start 2024-04-21T23:05:58.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.368+0000 7f6ae88ed640 1 -- start start 2024-04-21T23:05:58.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.369+0000 7f6ae88ed640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ae41006a0 0x7f6ae4197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:58.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.369+0000 7f6ae88ed640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6ae4101890 0x7f6ae4198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:58.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.369+0000 7f6ae88ed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ae4102250 0x7f6ae419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:58.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.369+0000 7f6ae88ed640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae410c300 con 0x7f6ae4102250 2024-04-21T23:05:58.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.369+0000 7f6ae88ed640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae410c180 con 0x7f6ae4101890 2024-04-21T23:05:58.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.369+0000 7f6ae88ed640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae410c480 con 0x7f6ae41006a0 2024-04-21T23:05:58.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.369+0000 7f6ae2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ae4102250 0x7f6ae419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:58.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.369+0000 7f6ae2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ae4102250 0x7f6ae419f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40916/0 (socket says 172.21.15.44:40916) 2024-04-21T23:05:58.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.369+0000 7f6ae2d9d640 1 -- 172.21.15.44:0/4049667137 learned_addr learned my addr 172.21.15.44:0/4049667137 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:05:58.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.369+0000 7f6ae1d9b640 1 --2- 172.21.15.44:0/4049667137 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6ae4101890 0x7f6ae4198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:58.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.369+0000 7f6ae259c640 1 --2- 172.21.15.44:0/4049667137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ae41006a0 0x7f6ae4197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:58.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.369+0000 7f6ae2d9d640 1 -- 172.21.15.44:0/4049667137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ae41006a0 msgr2=0x7f6ae4197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:58.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.369+0000 7f6ae2d9d640 1 --2- 172.21.15.44:0/4049667137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ae41006a0 0x7f6ae4197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:58.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.369+0000 7f6ae2d9d640 1 -- 172.21.15.44:0/4049667137 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6ae4101890 msgr2=0x7f6ae4198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:58.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.369+0000 7f6ae2d9d640 1 --2- 172.21.15.44:0/4049667137 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6ae4101890 0x7f6ae4198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:58.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.369+0000 7f6ae2d9d640 1 -- 172.21.15.44:0/4049667137 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae419f700 con 0x7f6ae4102250 2024-04-21T23:05:58.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.369+0000 7f6ae2d9d640 1 --2- 172.21.15.44:0/4049667137 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ae4102250 0x7f6ae419f090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6adc009fd0 tx=0x7f6adc00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:58.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.370+0000 7f6ad37fe640 1 -- 172.21.15.44:0/4049667137 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc017070 con 0x7f6ae4102250 2024-04-21T23:05:58.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.370+0000 7f6ae88ed640 1 -- 172.21.15.44:0/4049667137 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ae419f9f0 con 0x7f6ae4102250 2024-04-21T23:05:58.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.370+0000 7f6ad37fe640 1 -- 172.21.15.44:0/4049667137 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6adc003e50 con 0x7f6ae4102250 2024-04-21T23:05:58.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.370+0000 7f6ad37fe640 1 -- 172.21.15.44:0/4049667137 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc0106c0 con 0x7f6ae4102250 2024-04-21T23:05:58.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.370+0000 7f6ae88ed640 1 -- 172.21.15.44:0/4049667137 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ae419ff10 con 0x7f6ae4102250 2024-04-21T23:05:58.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.371+0000 7f6ae88ed640 1 -- 172.21.15.44:0/4049667137 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ae41a01a0 con 0x7f6ae4102250 2024-04-21T23:05:58.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.372+0000 7f6ad37fe640 1 -- 172.21.15.44:0/4049667137 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6adc003970 con 0x7f6ae4102250 2024-04-21T23:05:58.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.373+0000 7f6ad37fe640 1 --2- 172.21.15.44:0/4049667137 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6acc0a2b90 0x7f6acc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:05:58.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.373+0000 7f6ae259c640 1 --2- 172.21.15.44:0/4049667137 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6acc0a2b90 0x7f6acc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:05:58.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.373+0000 7f6ad37fe640 1 -- 172.21.15.44:0/4049667137 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f6adc0ce8b0 con 0x7f6ae4102250 2024-04-21T23:05:58.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.373+0000 7f6ae259c640 1 --2- 172.21.15.44:0/4049667137 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6acc0a2b90 0x7f6acc0a5050 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6ae41016f0 tx=0x7f6ad400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:05:58.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.377+0000 7f6ad37fe640 1 -- 172.21.15.44:0/4049667137 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6adc09b660 con 0x7f6ae4102250 2024-04-21T23:05:58.546 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:58 smithi044 ceph-mon[27934]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:58.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.616+0000 7f6ae88ed640 1 -- 172.21.15.44:0/4049667137 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ae41a0410 con 0x7f6ae4102250 2024-04-21T23:05:58.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.617+0000 7f6ad37fe640 1 -- 172.21.15.44:0/4049667137 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3034 (secure 0 0 0) 0x7f6adc09f960 con 0x7f6ae4102250 2024-04-21T23:05:58.617 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:05:58.617 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T23:05:58.617 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 4 2024-04-21T23:05:58.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.619+0000 7f6ae88ed640 1 -- 172.21.15.44:0/4049667137 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6acc0a2b90 msgr2=0x7f6acc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:58.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.619+0000 7f6ae88ed640 1 --2- 172.21.15.44:0/4049667137 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6acc0a2b90 0x7f6acc0a5050 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6ae41016f0 tx=0x7f6ad400b390 comp rx=0 tx=0).stop 2024-04-21T23:05:58.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.619+0000 7f6ae88ed640 1 -- 172.21.15.44:0/4049667137 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ae4102250 msgr2=0x7f6ae419f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:05:58.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.619+0000 7f6ae88ed640 1 --2- 172.21.15.44:0/4049667137 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ae4102250 0x7f6ae419f090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6adc009fd0 tx=0x7f6adc00b260 comp rx=0 tx=0).stop 2024-04-21T23:05:58.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.619+0000 7f6ae88ed640 1 -- 172.21.15.44:0/4049667137 shutdown_connections 2024-04-21T23:05:58.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.619+0000 7f6ae88ed640 1 --2- 172.21.15.44:0/4049667137 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6acc0a2b90 0x7f6acc0a5050 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:58.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.619+0000 7f6ae88ed640 1 --2- 172.21.15.44:0/4049667137 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ae4102250 0x7f6ae419f090 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:58.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.619+0000 7f6ae88ed640 1 --2- 172.21.15.44:0/4049667137 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6ae4101890 0x7f6ae4198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:58.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.619+0000 7f6ae88ed640 1 --2- 172.21.15.44:0/4049667137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ae41006a0 0x7f6ae4197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:05:58.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.619+0000 7f6ae88ed640 1 -- 172.21.15.44:0/4049667137 >> 172.21.15.44:0/4049667137 conn(0x7f6ae40fbc10 msgr2=0x7f6ae40fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:05:58.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.619+0000 7f6ae88ed640 1 -- 172.21.15.44:0/4049667137 shutdown_connections 2024-04-21T23:05:58.620 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:05:58.619+0000 7f6ae88ed640 1 -- 172.21.15.44:0/4049667137 wait complete. 2024-04-21T23:05:58.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:58 smithi055 ceph-mon[31117]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:58.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:58 smithi069 ceph-mon[31816]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:05:58.935 INFO:journalctl@ceph.mds.d.smithi044.stdout:Apr 21 23:05:58 smithi044 systemd[1]: Starting Ceph mds.d for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:05:59.213 INFO:journalctl@ceph.mds.d.smithi044.stdout:Apr 21 23:05:58 smithi044 podman[66138]: 2024-04-21 23:05:58.863190901 +0000 UTC m=+0.018384437 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:05:59.213 INFO:journalctl@ceph.mds.d.smithi044.stdout:Apr 21 23:05:58 smithi044 podman[66138]: 2024-04-21 23:05:58.978259029 +0000 UTC m=+0.133452579 container create 126f059275fe00e55915d9306d286b9b9862e9b6a0ccdf6aa0fdea108ad01431 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-d, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS) 2024-04-21T23:05:59.213 INFO:journalctl@ceph.mds.d.smithi044.stdout:Apr 21 23:05:59 smithi044 podman[66138]: 2024-04-21 23:05:59.214490512 +0000 UTC m=+0.369684032 container init 126f059275fe00e55915d9306d286b9b9862e9b6a0ccdf6aa0fdea108ad01431 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-21T23:05:59.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:59 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/4049667137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:05:59.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:59 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:59.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:59 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:59.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:59 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:59.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:59 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:05:59.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:59 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:05:59.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:05:59 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:59.686 INFO:journalctl@ceph.mds.d.smithi044.stdout:Apr 21 23:05:59 smithi044 podman[66138]: 2024-04-21 23:05:59.217273721 +0000 UTC m=+0.372467228 container start 126f059275fe00e55915d9306d286b9b9862e9b6a0ccdf6aa0fdea108ad01431 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:05:59.687 INFO:journalctl@ceph.mds.d.smithi044.stdout:Apr 21 23:05:59 smithi044 ceph-mds[66157]: -- 172.21.15.44:0/767717609 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d51cbeda40 con 0x55d51cbc0800 2024-04-21T23:05:59.687 INFO:journalctl@ceph.mds.d.smithi044.stdout:Apr 21 23:05:59 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-d[66153]: starting mds.d at 2024-04-21T23:05:59.687 INFO:journalctl@ceph.mds.d.smithi044.stdout:Apr 21 23:05:59 smithi044 bash[66138]: 126f059275fe00e55915d9306d286b9b9862e9b6a0ccdf6aa0fdea108ad01431 2024-04-21T23:05:59.687 INFO:journalctl@ceph.mds.d.smithi044.stdout:Apr 21 23:05:59 smithi044 systemd[1]: Started Ceph mds.d for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:05:59.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:59 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/4049667137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:05:59.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:59 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:59.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:59 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:59.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:59 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:59.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:59 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:05:59.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:59 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:05:59.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:05:59 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:05:59.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:59 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/4049667137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:05:59.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:59 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:59.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:59 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:59.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:59 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:05:59.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:59 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:05:59.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:59 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:05:59.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:05:59 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:00.633 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:06:00.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:00 smithi069 ceph-mon[31816]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:00.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:00 smithi069 ceph-mon[31816]: Deploying daemon mds.e on smithi055 2024-04-21T23:06:00.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:00 smithi069 ceph-mon[31816]: mds.? [v2:172.21.15.44:6836/3848365642,v1:172.21.15.44:6837/3848365642] up:boot 2024-04-21T23:06:00.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:00 smithi069 ceph-mon[31816]: fsmap 4 up:standby 2024-04-21T23:06:00.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:00 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T23:06:00.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:00 smithi055 ceph-mon[31117]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:00.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:00 smithi055 ceph-mon[31117]: Deploying daemon mds.e on smithi055 2024-04-21T23:06:00.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:00 smithi055 ceph-mon[31117]: mds.? [v2:172.21.15.44:6836/3848365642,v1:172.21.15.44:6837/3848365642] up:boot 2024-04-21T23:06:00.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:00 smithi055 ceph-mon[31117]: fsmap 4 up:standby 2024-04-21T23:06:00.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:00 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T23:06:00.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.751+0000 7f6f8b682640 1 -- 172.21.15.44:0/2502061235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f840741a0 msgr2=0x7f6f84074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:00.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.751+0000 7f6f8b682640 1 --2- 172.21.15.44:0/2502061235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f840741a0 0x7f6f84074620 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6f74009930 tx=0x7f6f7402bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.752+0000 7f6f8b682640 1 -- 172.21.15.44:0/2502061235 shutdown_connections 2024-04-21T23:06:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.752+0000 7f6f8b682640 1 --2- 172.21.15.44:0/2502061235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f8410dd70 0x7f6f84110160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.752+0000 7f6f8b682640 1 --2- 172.21.15.44:0/2502061235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f840741a0 0x7f6f84074620 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.752+0000 7f6f8b682640 1 --2- 172.21.15.44:0/2502061235 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f84075780 0x7f6f84073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.752+0000 7f6f8b682640 1 -- 172.21.15.44:0/2502061235 >> 172.21.15.44:0/2502061235 conn(0x7f6f840fcd30 msgr2=0x7f6f840ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.752+0000 7f6f8b682640 1 -- 172.21.15.44:0/2502061235 shutdown_connections 2024-04-21T23:06:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.752+0000 7f6f8b682640 1 -- 172.21.15.44:0/2502061235 wait complete. 2024-04-21T23:06:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.752+0000 7f6f8b682640 1 Processor -- start 2024-04-21T23:06:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.752+0000 7f6f8b682640 1 -- start start 2024-04-21T23:06:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.752+0000 7f6f8b682640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f840741a0 0x7f6f8419e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.752+0000 7f6f8b682640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f84075780 0x7f6f8419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.752+0000 7f6f8b682640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f8410dd70 0x7f6f841a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:00.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.752+0000 7f6f8b682640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6f84113a10 con 0x7f6f840741a0 2024-04-21T23:06:00.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.752+0000 7f6f8b682640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6f84113890 con 0x7f6f8410dd70 2024-04-21T23:06:00.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.752+0000 7f6f8b682640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6f84113b90 con 0x7f6f84075780 2024-04-21T23:06:00.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.753+0000 7f6f8941e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f840741a0 0x7f6f8419e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:00.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.753+0000 7f6f8941e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f840741a0 0x7f6f8419e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43248/0 (socket says 172.21.15.44:43248) 2024-04-21T23:06:00.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.753+0000 7f6f8941e640 1 -- 172.21.15.44:0/2341285068 learned_addr learned my addr 172.21.15.44:0/2341285068 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:00.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.753+0000 7f6f88c1d640 1 --2- 172.21.15.44:0/2341285068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f84075780 0x7f6f8419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:00.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.753+0000 7f6f8941e640 1 -- 172.21.15.44:0/2341285068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f84075780 msgr2=0x7f6f8419e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:00.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.753+0000 7f6f8941e640 1 --2- 172.21.15.44:0/2341285068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f84075780 0x7f6f8419e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:00.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.753+0000 7f6f89c1f640 1 --2- 172.21.15.44:0/2341285068 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f8410dd70 0x7f6f841a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:00.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.753+0000 7f6f8941e640 1 -- 172.21.15.44:0/2341285068 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f8410dd70 msgr2=0x7f6f841a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:00.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.753+0000 7f6f8941e640 1 --2- 172.21.15.44:0/2341285068 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f8410dd70 0x7f6f841a56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:00.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.753+0000 7f6f8941e640 1 -- 172.21.15.44:0/2341285068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f841a5d20 con 0x7f6f840741a0 2024-04-21T23:06:00.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.753+0000 7f6f8941e640 1 --2- 172.21.15.44:0/2341285068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f840741a0 0x7f6f8419e100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6f8000d950 tx=0x7f6f8000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:00.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.753+0000 7f6f7a7fc640 1 -- 172.21.15.44:0/2341285068 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f80017070 con 0x7f6f840741a0 2024-04-21T23:06:00.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.753+0000 7f6f8b682640 1 -- 172.21.15.44:0/2341285068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f841a5fb0 con 0x7f6f840741a0 2024-04-21T23:06:00.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.753+0000 7f6f7a7fc640 1 -- 172.21.15.44:0/2341285068 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f80009c30 con 0x7f6f840741a0 2024-04-21T23:06:00.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.753+0000 7f6f7a7fc640 1 -- 172.21.15.44:0/2341285068 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f8000f6b0 con 0x7f6f840741a0 2024-04-21T23:06:00.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.753+0000 7f6f8b682640 1 -- 172.21.15.44:0/2341285068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f841a6470 con 0x7f6f840741a0 2024-04-21T23:06:00.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.754+0000 7f6f8b682640 1 -- 172.21.15.44:0/2341285068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f84103f40 con 0x7f6f840741a0 2024-04-21T23:06:00.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.755+0000 7f6f7a7fc640 1 -- 172.21.15.44:0/2341285068 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6f800109d0 con 0x7f6f840741a0 2024-04-21T23:06:00.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.756+0000 7f6f7a7fc640 1 --2- 172.21.15.44:0/2341285068 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6f700a2b90 0x7f6f700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:00.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.756+0000 7f6f88c1d640 1 --2- 172.21.15.44:0/2341285068 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6f700a2b90 0x7f6f700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:00.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.756+0000 7f6f7a7fc640 1 -- 172.21.15.44:0/2341285068 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f6f800ce900 con 0x7f6f840741a0 2024-04-21T23:06:00.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.757+0000 7f6f88c1d640 1 --2- 172.21.15.44:0/2341285068 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6f700a2b90 0x7f6f700a5050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6f74009900 tx=0x7f6f74005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:00.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.760+0000 7f6f7a7fc640 1 -- 172.21.15.44:0/2341285068 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6f80009de0 con 0x7f6f840741a0 2024-04-21T23:06:00.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:00 smithi044 ceph-mon[27934]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:00.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:00 smithi044 ceph-mon[27934]: Deploying daemon mds.e on smithi055 2024-04-21T23:06:00.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:00 smithi044 ceph-mon[27934]: mds.? [v2:172.21.15.44:6836/3848365642,v1:172.21.15.44:6837/3848365642] up:boot 2024-04-21T23:06:00.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:00 smithi044 ceph-mon[27934]: fsmap 4 up:standby 2024-04-21T23:06:00.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:00 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T23:06:00.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.983+0000 7f6f8b682640 1 -- 172.21.15.44:0/2341285068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f8402d170 con 0x7f6f840741a0 2024-04-21T23:06:00.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.984+0000 7f6f7a7fc640 1 -- 172.21.15.44:0/2341285068 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3859 (secure 0 0 0) 0x7f6f8009bc60 con 0x7f6f840741a0 2024-04-21T23:06:00.984 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:00.984 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T23:06:00.984 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 5 2024-04-21T23:06:00.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.986+0000 7f6f8b682640 1 -- 172.21.15.44:0/2341285068 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6f700a2b90 msgr2=0x7f6f700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:00.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.986+0000 7f6f8b682640 1 --2- 172.21.15.44:0/2341285068 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6f700a2b90 0x7f6f700a5050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6f74009900 tx=0x7f6f74005fd0 comp rx=0 tx=0).stop 2024-04-21T23:06:00.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.986+0000 7f6f8b682640 1 -- 172.21.15.44:0/2341285068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f840741a0 msgr2=0x7f6f8419e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:00.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.986+0000 7f6f8b682640 1 --2- 172.21.15.44:0/2341285068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f840741a0 0x7f6f8419e100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6f8000d950 tx=0x7f6f8000de10 comp rx=0 tx=0).stop 2024-04-21T23:06:00.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.986+0000 7f6f8b682640 1 -- 172.21.15.44:0/2341285068 shutdown_connections 2024-04-21T23:06:00.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.986+0000 7f6f8b682640 1 --2- 172.21.15.44:0/2341285068 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6f700a2b90 0x7f6f700a5050 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:00.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.986+0000 7f6f8b682640 1 --2- 172.21.15.44:0/2341285068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f84075780 0x7f6f8419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:00.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.986+0000 7f6f8b682640 1 --2- 172.21.15.44:0/2341285068 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6f8410dd70 0x7f6f841a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:00.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.986+0000 7f6f8b682640 1 --2- 172.21.15.44:0/2341285068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6f840741a0 0x7f6f8419e100 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:00.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.986+0000 7f6f8b682640 1 -- 172.21.15.44:0/2341285068 >> 172.21.15.44:0/2341285068 conn(0x7f6f840fcd30 msgr2=0x7f6f84101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:00.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.986+0000 7f6f8b682640 1 -- 172.21.15.44:0/2341285068 shutdown_connections 2024-04-21T23:06:00.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:00.986+0000 7f6f8b682640 1 -- 172.21.15.44:0/2341285068 wait complete. 2024-04-21T23:06:01.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:01 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2341285068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:01.714 INFO:journalctl@ceph.mds.e.smithi055.stdout:Apr 21 23:06:01 smithi055 systemd[1]: Starting Ceph mds.e for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:06:01.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:01 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2341285068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:01.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:01 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2341285068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:02.203 INFO:journalctl@ceph.mds.e.smithi055.stdout:Apr 21 23:06:01 smithi055 podman[60915]: 2024-04-21 23:06:01.792766718 +0000 UTC m=+0.018433465 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:06:02.203 INFO:journalctl@ceph.mds.e.smithi055.stdout:Apr 21 23:06:01 smithi055 podman[60915]: 2024-04-21 23:06:01.901965398 +0000 UTC m=+0.127632179 container create b2cad9b74fe43640e56397bf03c733fa9e2b56af61c8ed5e8db38892686ace04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-21T23:06:02.204 INFO:journalctl@ceph.mds.e.smithi055.stdout:Apr 21 23:06:02 smithi055 podman[60915]: 2024-04-21 23:06:02.112749601 +0000 UTC m=+0.338416336 container init b2cad9b74fe43640e56397bf03c733fa9e2b56af61c8ed5e8db38892686ace04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS) 2024-04-21T23:06:02.204 INFO:journalctl@ceph.mds.e.smithi055.stdout:Apr 21 23:06:02 smithi055 podman[60915]: 2024-04-21 23:06:02.115441307 +0000 UTC m=+0.341108045 container start b2cad9b74fe43640e56397bf03c733fa9e2b56af61c8ed5e8db38892686ace04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-e, ceph=True, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-21T23:06:02.204 INFO:journalctl@ceph.mds.e.smithi055.stdout:Apr 21 23:06:02 smithi055 ceph-mds[60934]: -- 172.21.15.55:0/231784326 <== mon.2 v2:172.21.15.55:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b09ad85a40 con 0x55b09ad59000 2024-04-21T23:06:02.204 INFO:journalctl@ceph.mds.e.smithi055.stdout:Apr 21 23:06:02 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-e[60930]: starting mds.e at 2024-04-21T23:06:02.204 INFO:journalctl@ceph.mds.e.smithi055.stdout:Apr 21 23:06:02 smithi055 bash[60915]: b2cad9b74fe43640e56397bf03c733fa9e2b56af61c8ed5e8db38892686ace04 2024-04-21T23:06:02.464 INFO:journalctl@ceph.mds.e.smithi055.stdout:Apr 21 23:06:02 smithi055 systemd[1]: Started Ceph mds.e for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:06:02.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:02 smithi069 ceph-mon[31816]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:02.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:02 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:02.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:02 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:02.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:02 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:02.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:02 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:02.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:02 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:02.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:02 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:02.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:02 smithi044 ceph-mon[27934]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:02.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:02.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:02.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:02.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:02.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:02.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:02 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:02.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:02 smithi055 ceph-mon[31117]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:02.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:02 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:02.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:02 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:02.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:02 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:02.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:02 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:02.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:02 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:02.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:02 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:02.997 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:06:03.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.113+0000 7f072a5be640 1 -- 172.21.15.44:0/1497064082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0724102250 msgr2=0x7f0724108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:03.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.113+0000 7f072a5be640 1 --2- 172.21.15.44:0/1497064082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0724102250 0x7f0724108ae0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f071c009ca0 tx=0x7f071c02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:03.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.113+0000 7f072a5be640 1 -- 172.21.15.44:0/1497064082 shutdown_connections 2024-04-21T23:06:03.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.113+0000 7f072a5be640 1 --2- 172.21.15.44:0/1497064082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0724102250 0x7f0724108ae0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:03.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.113+0000 7f072a5be640 1 --2- 172.21.15.44:0/1497064082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0724101890 0x7f0724101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:03.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.113+0000 7f072a5be640 1 --2- 172.21.15.44:0/1497064082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07241006a0 0x7f0724100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:03.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.113+0000 7f072a5be640 1 -- 172.21.15.44:0/1497064082 >> 172.21.15.44:0/1497064082 conn(0x7f07240fbc50 msgr2=0x7f07240fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:03.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.113+0000 7f072a5be640 1 -- 172.21.15.44:0/1497064082 shutdown_connections 2024-04-21T23:06:03.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.113+0000 7f072a5be640 1 -- 172.21.15.44:0/1497064082 wait complete. 2024-04-21T23:06:03.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.113+0000 7f072a5be640 1 Processor -- start 2024-04-21T23:06:03.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.113+0000 7f072a5be640 1 -- start start 2024-04-21T23:06:03.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f072a5be640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07241006a0 0x7f0724197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:03.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f072a5be640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0724101890 0x7f0724197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:03.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f072a5be640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0724102250 0x7f072419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:03.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f072a5be640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f072410d450 con 0x7f0724102250 2024-04-21T23:06:03.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f072a5be640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f072410d2d0 con 0x7f0724101890 2024-04-21T23:06:03.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f072a5be640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f072410d5d0 con 0x7f07241006a0 2024-04-21T23:06:03.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f0728b5b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0724102250 0x7f072419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:03.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f0728b5b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0724102250 0x7f072419efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43272/0 (socket says 172.21.15.44:43272) 2024-04-21T23:06:03.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f0728b5b640 1 -- 172.21.15.44:0/3149831336 learned_addr learned my addr 172.21.15.44:0/3149831336 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:03.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f0723fff640 1 --2- 172.21.15.44:0/3149831336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07241006a0 0x7f0724197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:03.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f07237fe640 1 --2- 172.21.15.44:0/3149831336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0724101890 0x7f0724197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:03.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f0728b5b640 1 -- 172.21.15.44:0/3149831336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07241006a0 msgr2=0x7f0724197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:03.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f0728b5b640 1 --2- 172.21.15.44:0/3149831336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07241006a0 0x7f0724197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:03.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f0728b5b640 1 -- 172.21.15.44:0/3149831336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0724101890 msgr2=0x7f0724197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:03.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f0728b5b640 1 --2- 172.21.15.44:0/3149831336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0724101890 0x7f0724197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:03.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f0728b5b640 1 -- 172.21.15.44:0/3149831336 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f072419f640 con 0x7f0724102250 2024-04-21T23:06:03.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f07237fe640 1 --2- 172.21.15.44:0/3149831336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0724101890 0x7f0724197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:06:03.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f0728b5b640 1 --2- 172.21.15.44:0/3149831336 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0724102250 0x7f072419efd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f071c02c630 tx=0x7f071c032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:03.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f07217fa640 1 -- 172.21.15.44:0/3149831336 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c041020 con 0x7f0724102250 2024-04-21T23:06:03.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f072a5be640 1 -- 172.21.15.44:0/3149831336 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f072419f8d0 con 0x7f0724102250 2024-04-21T23:06:03.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f07217fa640 1 -- 172.21.15.44:0/3149831336 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f071c034cd0 con 0x7f0724102250 2024-04-21T23:06:03.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f07217fa640 1 -- 172.21.15.44:0/3149831336 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c039040 con 0x7f0724102250 2024-04-21T23:06:03.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.114+0000 7f072a5be640 1 -- 172.21.15.44:0/3149831336 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f072419fdc0 con 0x7f0724102250 2024-04-21T23:06:03.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.115+0000 7f072a5be640 1 -- 172.21.15.44:0/3149831336 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07241079c0 con 0x7f0724102250 2024-04-21T23:06:03.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.117+0000 7f07217fa640 1 -- 172.21.15.44:0/3149831336 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f071c044020 con 0x7f0724102250 2024-04-21T23:06:03.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.117+0000 7f07217fa640 1 --2- 172.21.15.44:0/3149831336 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f07100a2b90 0x7f07100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:03.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.117+0000 7f07217fa640 1 -- 172.21.15.44:0/3149831336 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f071c0f0ba0 con 0x7f0724102250 2024-04-21T23:06:03.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.117+0000 7f0723fff640 1 --2- 172.21.15.44:0/3149831336 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f07100a2b90 0x7f07100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:03.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.118+0000 7f0723fff640 1 --2- 172.21.15.44:0/3149831336 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f07100a2b90 0x7f07100a5050 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f07241016f0 tx=0x7f0714005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:03.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.122+0000 7f07217fa640 1 -- 172.21.15.44:0/3149831336 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f071c04b020 con 0x7f0724102250 2024-04-21T23:06:03.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.345+0000 7f072a5be640 1 -- 172.21.15.44:0/3149831336 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0724062340 con 0x7f0724102250 2024-04-21T23:06:03.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.346+0000 7f07217fa640 1 -- 172.21.15.44:0/3149831336 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4684 (secure 0 0 0) 0x7f071c0bdfa0 con 0x7f0724102250 2024-04-21T23:06:03.346 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:03.346 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-21T23:06:03.346 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 6 2024-04-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.348+0000 7f072a5be640 1 -- 172.21.15.44:0/3149831336 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f07100a2b90 msgr2=0x7f07100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.348+0000 7f072a5be640 1 --2- 172.21.15.44:0/3149831336 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f07100a2b90 0x7f07100a5050 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f07241016f0 tx=0x7f0714005f90 comp rx=0 tx=0).stop 2024-04-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.348+0000 7f072a5be640 1 -- 172.21.15.44:0/3149831336 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0724102250 msgr2=0x7f072419efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.348+0000 7f072a5be640 1 --2- 172.21.15.44:0/3149831336 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0724102250 0x7f072419efd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f071c02c630 tx=0x7f071c032ea0 comp rx=0 tx=0).stop 2024-04-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.348+0000 7f072a5be640 1 -- 172.21.15.44:0/3149831336 shutdown_connections 2024-04-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.348+0000 7f072a5be640 1 --2- 172.21.15.44:0/3149831336 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f07100a2b90 0x7f07100a5050 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.348+0000 7f072a5be640 1 --2- 172.21.15.44:0/3149831336 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0724102250 0x7f072419efd0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.348+0000 7f072a5be640 1 --2- 172.21.15.44:0/3149831336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0724101890 0x7f0724197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.348+0000 7f072a5be640 1 --2- 172.21.15.44:0/3149831336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07241006a0 0x7f0724197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.348+0000 7f072a5be640 1 -- 172.21.15.44:0/3149831336 >> 172.21.15.44:0/3149831336 conn(0x7f07240fbc50 msgr2=0x7f0724104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:03.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.348+0000 7f072a5be640 1 -- 172.21.15.44:0/3149831336 shutdown_connections 2024-04-21T23:06:03.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:03.348+0000 7f072a5be640 1 -- 172.21.15.44:0/3149831336 wait complete. 2024-04-21T23:06:03.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:03 smithi069 ceph-mon[31816]: Deploying daemon mds.f on smithi069 2024-04-21T23:06:03.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:03 smithi069 ceph-mon[31816]: mds.? [v2:172.21.15.55:6834/1607421809,v1:172.21.15.55:6835/1607421809] up:boot 2024-04-21T23:06:03.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:03 smithi069 ceph-mon[31816]: fsmap 5 up:standby 2024-04-21T23:06:03.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:03 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T23:06:03.626 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:03 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3149831336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:03.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:03 smithi044 ceph-mon[27934]: Deploying daemon mds.f on smithi069 2024-04-21T23:06:03.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:03 smithi044 ceph-mon[27934]: mds.? [v2:172.21.15.55:6834/1607421809,v1:172.21.15.55:6835/1607421809] up:boot 2024-04-21T23:06:03.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:03 smithi044 ceph-mon[27934]: fsmap 5 up:standby 2024-04-21T23:06:03.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:03 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T23:06:03.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:03 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3149831336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:03.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:03 smithi055 ceph-mon[31117]: Deploying daemon mds.f on smithi069 2024-04-21T23:06:03.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:03 smithi055 ceph-mon[31117]: mds.? [v2:172.21.15.55:6834/1607421809,v1:172.21.15.55:6835/1607421809] up:boot 2024-04-21T23:06:03.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:03 smithi055 ceph-mon[31117]: fsmap 5 up:standby 2024-04-21T23:06:03.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:03 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T23:06:03.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:03 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3149831336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:04.464 INFO:journalctl@ceph.mds.f.smithi069.stdout:Apr 21 23:06:04 smithi069 systemd[1]: Starting Ceph mds.f for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:06:04.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:04 smithi069 ceph-mon[31816]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:04.819 INFO:journalctl@ceph.mds.f.smithi069.stdout:Apr 21 23:06:04 smithi069 podman[62555]: 2024-04-21 23:06:04.483601499 +0000 UTC m=+0.018208943 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:06:04.819 INFO:journalctl@ceph.mds.f.smithi069.stdout:Apr 21 23:06:04 smithi069 podman[62555]: 2024-04-21 23:06:04.597729912 +0000 UTC m=+0.132337356 container create 81754f9122c7465848eb9e454aa7b4a255acba0530f845cbe697d087a32df721 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-f, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-21T23:06:04.819 INFO:journalctl@ceph.mds.f.smithi069.stdout:Apr 21 23:06:04 smithi069 podman[62555]: 2024-04-21 23:06:04.817060581 +0000 UTC m=+0.351668027 container init 81754f9122c7465848eb9e454aa7b4a255acba0530f845cbe697d087a32df721 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8) 2024-04-21T23:06:04.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:04 smithi044 ceph-mon[27934]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:04.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:04 smithi055 ceph-mon[31117]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:05.214 INFO:journalctl@ceph.mds.f.smithi069.stdout:Apr 21 23:06:04 smithi069 podman[62555]: 2024-04-21 23:06:04.819588409 +0000 UTC m=+0.354195853 container start 81754f9122c7465848eb9e454aa7b4a255acba0530f845cbe697d087a32df721 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-21T23:06:05.214 INFO:journalctl@ceph.mds.f.smithi069.stdout:Apr 21 23:06:04 smithi069 ceph-mds[62573]: -- 172.21.15.69:0/2693630084 <== mon.1 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55adee537a40 con 0x55adee50b000 2024-04-21T23:06:05.214 INFO:journalctl@ceph.mds.f.smithi069.stdout:Apr 21 23:06:04 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-f[62569]: starting mds.f at 2024-04-21T23:06:05.214 INFO:journalctl@ceph.mds.f.smithi069.stdout:Apr 21 23:06:04 smithi069 bash[62555]: 81754f9122c7465848eb9e454aa7b4a255acba0530f845cbe697d087a32df721 2024-04-21T23:06:05.215 INFO:journalctl@ceph.mds.f.smithi069.stdout:Apr 21 23:06:04 smithi069 systemd[1]: Started Ceph mds.f for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:06:05.360 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:06:05.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.476+0000 7f66f714f640 1 -- 172.21.15.44:0/3126835481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66f01017a0 msgr2=0x7f66f0101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:05.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.476+0000 7f66f714f640 1 --2- 172.21.15.44:0/3126835481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66f01017a0 0x7f66f0101bc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f66e8009ca0 tx=0x7f66e802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:05.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.476+0000 7f66f714f640 1 -- 172.21.15.44:0/3126835481 shutdown_connections 2024-04-21T23:06:05.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.476+0000 7f66f714f640 1 --2- 172.21.15.44:0/3126835481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f0102990 0x7f66f0102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:05.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.476+0000 7f66f714f640 1 --2- 172.21.15.44:0/3126835481 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66f0103350 0x7f66f0109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:05.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.476+0000 7f66f714f640 1 --2- 172.21.15.44:0/3126835481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66f01017a0 0x7f66f0101bc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:05.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.476+0000 7f66f714f640 1 -- 172.21.15.44:0/3126835481 >> 172.21.15.44:0/3126835481 conn(0x7f66f00fcd30 msgr2=0x7f66f00ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:05.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.476+0000 7f66f714f640 1 -- 172.21.15.44:0/3126835481 shutdown_connections 2024-04-21T23:06:05.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.476+0000 7f66f714f640 1 -- 172.21.15.44:0/3126835481 wait complete. 2024-04-21T23:06:05.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.477+0000 7f66f714f640 1 Processor -- start 2024-04-21T23:06:05.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.477+0000 7f66f714f640 1 -- start start 2024-04-21T23:06:05.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.477+0000 7f66f714f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f01017a0 0x7f66f0197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:05.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.477+0000 7f66f714f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66f0102990 0x7f66f0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:05.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.477+0000 7f66f714f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66f0103350 0x7f66f019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:05.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.477+0000 7f66f714f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f66f010d410 con 0x7f66f0103350 2024-04-21T23:06:05.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.477+0000 7f66f714f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f66f010d290 con 0x7f66f0102990 2024-04-21T23:06:05.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.477+0000 7f66f714f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f66f010d590 con 0x7f66f01017a0 2024-04-21T23:06:05.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.477+0000 7f66f56ec640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66f0103350 0x7f66f019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:05.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.477+0000 7f66f56ec640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66f0103350 0x7f66f019f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43292/0 (socket says 172.21.15.44:43292) 2024-04-21T23:06:05.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.477+0000 7f66f56ec640 1 -- 172.21.15.44:0/4131012066 learned_addr learned my addr 172.21.15.44:0/4131012066 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:05.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.477+0000 7f66f4eeb640 1 --2- 172.21.15.44:0/4131012066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f01017a0 0x7f66f0197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:05.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.477+0000 7f66effff640 1 --2- 172.21.15.44:0/4131012066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66f0102990 0x7f66f0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:05.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.477+0000 7f66f56ec640 1 -- 172.21.15.44:0/4131012066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f01017a0 msgr2=0x7f66f0197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:05.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.477+0000 7f66f56ec640 1 --2- 172.21.15.44:0/4131012066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f01017a0 0x7f66f0197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:05.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.477+0000 7f66f56ec640 1 -- 172.21.15.44:0/4131012066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66f0102990 msgr2=0x7f66f0197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:05.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.477+0000 7f66f56ec640 1 --2- 172.21.15.44:0/4131012066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66f0102990 0x7f66f0197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:05.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.477+0000 7f66f56ec640 1 -- 172.21.15.44:0/4131012066 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f019f6d0 con 0x7f66f0103350 2024-04-21T23:06:05.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.477+0000 7f66effff640 1 --2- 172.21.15.44:0/4131012066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66f0102990 0x7f66f0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:06:05.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.477+0000 7f66f56ec640 1 --2- 172.21.15.44:0/4131012066 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66f0103350 0x7f66f019f060 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f66e000d750 tx=0x7f66e000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:05.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.478+0000 7f66edffb640 1 -- 172.21.15.44:0/4131012066 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e0017070 con 0x7f66f0103350 2024-04-21T23:06:05.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.478+0000 7f66f714f640 1 -- 172.21.15.44:0/4131012066 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66f019f9c0 con 0x7f66f0103350 2024-04-21T23:06:05.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.478+0000 7f66edffb640 1 -- 172.21.15.44:0/4131012066 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66e0005500 con 0x7f66f0103350 2024-04-21T23:06:05.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.478+0000 7f66edffb640 1 -- 172.21.15.44:0/4131012066 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e0004d90 con 0x7f66f0103350 2024-04-21T23:06:05.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.478+0000 7f66f714f640 1 -- 172.21.15.44:0/4131012066 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66f019ff10 con 0x7f66f0103350 2024-04-21T23:06:05.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.479+0000 7f66f714f640 1 -- 172.21.15.44:0/4131012066 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66f0108b40 con 0x7f66f0103350 2024-04-21T23:06:05.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.480+0000 7f66edffb640 1 -- 172.21.15.44:0/4131012066 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f66e0004f30 con 0x7f66f0103350 2024-04-21T23:06:05.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.481+0000 7f66edffb640 1 --2- 172.21.15.44:0/4131012066 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f66d80a2b90 0x7f66d80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:05.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.481+0000 7f66f4eeb640 1 --2- 172.21.15.44:0/4131012066 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f66d80a2b90 0x7f66d80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:05.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.481+0000 7f66edffb640 1 -- 172.21.15.44:0/4131012066 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f66e00ce5c0 con 0x7f66f0103350 2024-04-21T23:06:05.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.481+0000 7f66f4eeb640 1 --2- 172.21.15.44:0/4131012066 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f66d80a2b90 0x7f66d80a5050 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f66e802c700 tx=0x7f66e8032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:05.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.485+0000 7f66edffb640 1 -- 172.21.15.44:0/4131012066 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f66e009b9c0 con 0x7f66f0103350 2024-04-21T23:06:05.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.708+0000 7f66f714f640 1 -- 172.21.15.44:0/4131012066 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66f0062340 con 0x7f66f0103350 2024-04-21T23:06:05.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.709+0000 7f66edffb640 1 -- 172.21.15.44:0/4131012066 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5509 (secure 0 0 0) 0x7f66e009b2c0 con 0x7f66f0103350 2024-04-21T23:06:05.709 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:05.709 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-21T23:06:05.709 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 7 2024-04-21T23:06:05.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.711+0000 7f66f714f640 1 -- 172.21.15.44:0/4131012066 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f66d80a2b90 msgr2=0x7f66d80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:05.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.711+0000 7f66f714f640 1 --2- 172.21.15.44:0/4131012066 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f66d80a2b90 0x7f66d80a5050 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f66e802c700 tx=0x7f66e8032590 comp rx=0 tx=0).stop 2024-04-21T23:06:05.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.711+0000 7f66f714f640 1 -- 172.21.15.44:0/4131012066 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66f0103350 msgr2=0x7f66f019f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:05.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.711+0000 7f66f714f640 1 --2- 172.21.15.44:0/4131012066 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66f0103350 0x7f66f019f060 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f66e000d750 tx=0x7f66e000dc10 comp rx=0 tx=0).stop 2024-04-21T23:06:05.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.711+0000 7f66f714f640 1 -- 172.21.15.44:0/4131012066 shutdown_connections 2024-04-21T23:06:05.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.711+0000 7f66f714f640 1 --2- 172.21.15.44:0/4131012066 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f66d80a2b90 0x7f66d80a5050 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:05.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.711+0000 7f66f714f640 1 --2- 172.21.15.44:0/4131012066 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f66f0103350 0x7f66f019f060 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:05.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.711+0000 7f66f714f640 1 --2- 172.21.15.44:0/4131012066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66f0102990 0x7f66f0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:05.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.711+0000 7f66f714f640 1 --2- 172.21.15.44:0/4131012066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f01017a0 0x7f66f0197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:05.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.711+0000 7f66f714f640 1 -- 172.21.15.44:0/4131012066 >> 172.21.15.44:0/4131012066 conn(0x7f66f00fcd30 msgr2=0x7f66f0105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:05.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.711+0000 7f66f714f640 1 -- 172.21.15.44:0/4131012066 shutdown_connections 2024-04-21T23:06:05.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:05.711+0000 7f66f714f640 1 -- 172.21.15.44:0/4131012066 wait complete. 2024-04-21T23:06:06.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:05 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:06.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:05 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:06.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:05 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:06.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:05 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:06.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:05 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:06.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:05 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:06.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:05 smithi044 ceph-mon[27934]: Deploying daemon mds.c on smithi069 2024-04-21T23:06:06.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:05 smithi044 ceph-mon[27934]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:06.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:05 smithi044 ceph-mon[27934]: mds.? [v2:172.21.15.69:6834/2637621987,v1:172.21.15.69:6835/2637621987] up:boot 2024-04-21T23:06:06.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:05 smithi044 ceph-mon[27934]: fsmap 6 up:standby 2024-04-21T23:06:06.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:05 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T23:06:06.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:05 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/4131012066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:06.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:05 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:06.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:05 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:06.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:05 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:06.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:05 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:06.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:05 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:06.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:05 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:06.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:05 smithi055 ceph-mon[31117]: Deploying daemon mds.c on smithi069 2024-04-21T23:06:06.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:05 smithi055 ceph-mon[31117]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:06.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:05 smithi055 ceph-mon[31117]: mds.? [v2:172.21.15.69:6834/2637621987,v1:172.21.15.69:6835/2637621987] up:boot 2024-04-21T23:06:06.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:05 smithi055 ceph-mon[31117]: fsmap 6 up:standby 2024-04-21T23:06:06.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:05 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T23:06:06.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:05 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/4131012066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:06.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:05 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:06.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:05 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:06.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:05 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:06.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:05 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:06.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:05 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:06.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:05 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:06.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:05 smithi069 ceph-mon[31816]: Deploying daemon mds.c on smithi069 2024-04-21T23:06:06.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:05 smithi069 ceph-mon[31816]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:06.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:05 smithi069 ceph-mon[31816]: mds.? [v2:172.21.15.69:6834/2637621987,v1:172.21.15.69:6835/2637621987] up:boot 2024-04-21T23:06:06.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:05 smithi069 ceph-mon[31816]: fsmap 6 up:standby 2024-04-21T23:06:06.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:05 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T23:06:06.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:05 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/4131012066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:07.714 INFO:journalctl@ceph.mds.c.smithi069.stdout:Apr 21 23:06:07 smithi069 systemd[1]: Started Ceph mds.c for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:06:07.723 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:06:07.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.838+0000 7f055d9f7640 1 -- 172.21.15.44:0/257638544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0558101890 msgr2=0x7f0558101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:07.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.838+0000 7f055d9f7640 1 --2- 172.21.15.44:0/257638544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0558101890 0x7f0558101d10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f054c009900 tx=0x7f054c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:07.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.838+0000 7f055d9f7640 1 -- 172.21.15.44:0/257638544 shutdown_connections 2024-04-21T23:06:07.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.838+0000 7f055d9f7640 1 --2- 172.21.15.44:0/257638544 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0558102250 0x7f0558108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:07.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.838+0000 7f055d9f7640 1 --2- 172.21.15.44:0/257638544 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0558101890 0x7f0558101d10 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:07.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.838+0000 7f055d9f7640 1 --2- 172.21.15.44:0/257638544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05581006a0 0x7f0558100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:07.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.838+0000 7f055d9f7640 1 -- 172.21.15.44:0/257638544 >> 172.21.15.44:0/257638544 conn(0x7f05580fbc50 msgr2=0x7f05580fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:07.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.838+0000 7f055d9f7640 1 -- 172.21.15.44:0/257638544 shutdown_connections 2024-04-21T23:06:07.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.838+0000 7f055d9f7640 1 -- 172.21.15.44:0/257638544 wait complete. 2024-04-21T23:06:07.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.838+0000 7f055d9f7640 1 Processor -- start 2024-04-21T23:06:07.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.839+0000 7f055d9f7640 1 -- start start 2024-04-21T23:06:07.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.839+0000 7f055d9f7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05581006a0 0x7f055806aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:07.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.839+0000 7f055d9f7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0558101890 0x7f055819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:07.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.839+0000 7f055d9f7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0558102250 0x7f055819ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:07.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.839+0000 7f055d9f7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f055810c320 con 0x7f0558102250 2024-04-21T23:06:07.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.839+0000 7f055d9f7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f055810c1a0 con 0x7f05581006a0 2024-04-21T23:06:07.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.839+0000 7f055d9f7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f055810c4a0 con 0x7f0558101890 2024-04-21T23:06:07.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.839+0000 7f05577fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0558102250 0x7f055819ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:07.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.839+0000 7f05577fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0558102250 0x7f055819ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43308/0 (socket says 172.21.15.44:43308) 2024-04-21T23:06:07.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.839+0000 7f05577fe640 1 -- 172.21.15.44:0/3779017156 learned_addr learned my addr 172.21.15.44:0/3779017156 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:07.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.839+0000 7f0556ffd640 1 --2- 172.21.15.44:0/3779017156 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05581006a0 0x7f055806aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:07.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.839+0000 7f05567fc640 1 --2- 172.21.15.44:0/3779017156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0558101890 0x7f055819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:07.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.839+0000 7f05577fe640 1 -- 172.21.15.44:0/3779017156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0558101890 msgr2=0x7f055819c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:07.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.839+0000 7f05577fe640 1 --2- 172.21.15.44:0/3779017156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0558101890 0x7f055819c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:07.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.839+0000 7f05577fe640 1 -- 172.21.15.44:0/3779017156 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05581006a0 msgr2=0x7f055806aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:07.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.839+0000 7f05577fe640 1 --2- 172.21.15.44:0/3779017156 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05581006a0 0x7f055806aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:07.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.839+0000 7f05577fe640 1 -- 172.21.15.44:0/3779017156 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f055819f6e0 con 0x7f0558102250 2024-04-21T23:06:07.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.839+0000 7f05577fe640 1 --2- 172.21.15.44:0/3779017156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0558102250 0x7f055819ee20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f055000eb10 tx=0x7f055000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:07.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.840+0000 7f055c9f5640 1 -- 172.21.15.44:0/3779017156 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0550004b20 con 0x7f0558102250 2024-04-21T23:06:07.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.840+0000 7f055d9f7640 1 -- 172.21.15.44:0/3779017156 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f055819f9d0 con 0x7f0558102250 2024-04-21T23:06:07.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.840+0000 7f055c9f5640 1 -- 172.21.15.44:0/3779017156 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0550004cc0 con 0x7f0558102250 2024-04-21T23:06:07.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.840+0000 7f055c9f5640 1 -- 172.21.15.44:0/3779017156 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0550010760 con 0x7f0558102250 2024-04-21T23:06:07.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.840+0000 7f055d9f7640 1 -- 172.21.15.44:0/3779017156 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f055819fef0 con 0x7f0558102250 2024-04-21T23:06:07.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.841+0000 7f055d9f7640 1 -- 172.21.15.44:0/3779017156 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05581079c0 con 0x7f0558102250 2024-04-21T23:06:07.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.842+0000 7f055c9f5640 1 -- 172.21.15.44:0/3779017156 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0550010900 con 0x7f0558102250 2024-04-21T23:06:07.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.843+0000 7f055c9f5640 1 --2- 172.21.15.44:0/3779017156 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f05400a2b90 0x7f05400a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:07.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.843+0000 7f0556ffd640 1 --2- 172.21.15.44:0/3779017156 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f05400a2b90 0x7f05400a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:07.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.843+0000 7f055c9f5640 1 -- 172.21.15.44:0/3779017156 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f05500cf3c0 con 0x7f0558102250 2024-04-21T23:06:07.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.843+0000 7f0556ffd640 1 --2- 172.21.15.44:0/3779017156 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f05400a2b90 0x7f05400a5050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f05581016f0 tx=0x7f0548005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:07.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:07.846+0000 7f055c9f5640 1 -- 172.21.15.44:0/3779017156 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f05500536f0 con 0x7f0558102250 2024-04-21T23:06:08.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:08.072+0000 7f055d9f7640 1 -- 172.21.15.44:0/3779017156 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f055802d170 con 0x7f0558102250 2024-04-21T23:06:08.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:08.073+0000 7f055c9f5640 1 -- 172.21.15.44:0/3779017156 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5509 (secure 0 0 0) 0x7f055009c720 con 0x7f0558102250 2024-04-21T23:06:08.073 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:08.073 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-21T23:06:08.073 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 7 2024-04-21T23:06:08.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:08.074+0000 7f055d9f7640 1 -- 172.21.15.44:0/3779017156 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f05400a2b90 msgr2=0x7f05400a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:08.074+0000 7f055d9f7640 1 --2- 172.21.15.44:0/3779017156 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f05400a2b90 0x7f05400a5050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f05581016f0 tx=0x7f0548005f90 comp rx=0 tx=0).stop 2024-04-21T23:06:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:08.075+0000 7f055d9f7640 1 -- 172.21.15.44:0/3779017156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0558102250 msgr2=0x7f055819ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:08.075+0000 7f055d9f7640 1 --2- 172.21.15.44:0/3779017156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0558102250 0x7f055819ee20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f055000eb10 tx=0x7f055000c2d0 comp rx=0 tx=0).stop 2024-04-21T23:06:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:08.075+0000 7f055d9f7640 1 -- 172.21.15.44:0/3779017156 shutdown_connections 2024-04-21T23:06:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:08.075+0000 7f055d9f7640 1 --2- 172.21.15.44:0/3779017156 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f05400a2b90 0x7f05400a5050 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:08.075+0000 7f055d9f7640 1 --2- 172.21.15.44:0/3779017156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0558101890 0x7f055819c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:08.075+0000 7f055d9f7640 1 --2- 172.21.15.44:0/3779017156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0558102250 0x7f055819ee20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:08.075+0000 7f055d9f7640 1 --2- 172.21.15.44:0/3779017156 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05581006a0 0x7f055806aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:08.075+0000 7f055d9f7640 1 -- 172.21.15.44:0/3779017156 >> 172.21.15.44:0/3779017156 conn(0x7f05580fbc50 msgr2=0x7f0558104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:08.075+0000 7f055d9f7640 1 -- 172.21.15.44:0/3779017156 shutdown_connections 2024-04-21T23:06:08.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:08.075+0000 7f055d9f7640 1 -- 172.21.15.44:0/3779017156 wait complete. 2024-04-21T23:06:09.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:08 smithi044 ceph-mon[27934]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:09.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:08 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:09.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:08 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:09.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:08 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:09.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:08 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:09.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:08 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:09.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:08 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:09.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:08 smithi044 ceph-mon[27934]: Deploying daemon mds.h on smithi055 2024-04-21T23:06:09.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:08 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3779017156' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:09.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:08 smithi044 ceph-mon[27934]: mds.? [v2:172.21.15.69:6836/3371833935,v1:172.21.15.69:6837/3371833935] up:boot 2024-04-21T23:06:09.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:08 smithi044 ceph-mon[27934]: fsmap 7 up:standby 2024-04-21T23:06:09.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:08 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T23:06:09.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:08 smithi055 ceph-mon[31117]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:09.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:08 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:09.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:08 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:09.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:08 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:09.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:08 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:09.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:08 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:09.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:08 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:09.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:08 smithi055 ceph-mon[31117]: Deploying daemon mds.h on smithi055 2024-04-21T23:06:09.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:08 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3779017156' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:09.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:08 smithi055 ceph-mon[31117]: mds.? [v2:172.21.15.69:6836/3371833935,v1:172.21.15.69:6837/3371833935] up:boot 2024-04-21T23:06:09.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:08 smithi055 ceph-mon[31117]: fsmap 7 up:standby 2024-04-21T23:06:09.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:08 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T23:06:09.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:08 smithi069 ceph-mon[31816]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:09.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:08 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:09.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:08 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:09.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:08 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:09.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:08 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:09.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:08 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:09.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:08 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:09.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:08 smithi069 ceph-mon[31816]: Deploying daemon mds.h on smithi055 2024-04-21T23:06:09.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:08 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3779017156' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:09.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:08 smithi069 ceph-mon[31816]: mds.? [v2:172.21.15.69:6836/3371833935,v1:172.21.15.69:6837/3371833935] up:boot 2024-04-21T23:06:09.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:08 smithi069 ceph-mon[31816]: fsmap 7 up:standby 2024-04-21T23:06:09.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:08 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T23:06:10.087 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:06:10.131 INFO:journalctl@ceph.mds.h.smithi055.stdout:Apr 21 23:06:09 smithi055 systemd[1]: Starting Ceph mds.h for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:06:10.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.202+0000 7f616acc2640 1 -- 172.21.15.44:0/722759715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6164103350 msgr2=0x7f6164109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:10.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.202+0000 7f616acc2640 1 --2- 172.21.15.44:0/722759715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6164103350 0x7f6164109be0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f615c009ca0 tx=0x7f615c02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:10.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.202+0000 7f616acc2640 1 -- 172.21.15.44:0/722759715 shutdown_connections 2024-04-21T23:06:10.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.202+0000 7f616acc2640 1 --2- 172.21.15.44:0/722759715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6164103350 0x7f6164109be0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:10.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.202+0000 7f616acc2640 1 --2- 172.21.15.44:0/722759715 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6164102990 0x7f6164102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:10.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.202+0000 7f616acc2640 1 --2- 172.21.15.44:0/722759715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f61641017a0 0x7f6164101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:10.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.202+0000 7f616acc2640 1 -- 172.21.15.44:0/722759715 >> 172.21.15.44:0/722759715 conn(0x7f61640fcd30 msgr2=0x7f61640ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:10.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.202+0000 7f616acc2640 1 -- 172.21.15.44:0/722759715 shutdown_connections 2024-04-21T23:06:10.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.202+0000 7f616acc2640 1 -- 172.21.15.44:0/722759715 wait complete. 2024-04-21T23:06:10.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.203+0000 7f616acc2640 1 Processor -- start 2024-04-21T23:06:10.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.203+0000 7f616acc2640 1 -- start start 2024-04-21T23:06:10.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.203+0000 7f616acc2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61641017a0 0x7f61640754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:10.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.203+0000 7f616acc2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6164102990 0x7f6164073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:10.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.203+0000 7f616acc2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6164103350 0x7f6164074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:10.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.203+0000 7f616acc2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f616410d400 con 0x7f6164102990 2024-04-21T23:06:10.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.203+0000 7f616acc2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f616410d280 con 0x7f61641017a0 2024-04-21T23:06:10.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.203+0000 7f616acc2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f616410d580 con 0x7f6164103350 2024-04-21T23:06:10.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.203+0000 7f6163fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6164102990 0x7f6164073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:10.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.203+0000 7f6163fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6164102990 0x7f6164073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43328/0 (socket says 172.21.15.44:43328) 2024-04-21T23:06:10.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.203+0000 7f6163fff640 1 -- 172.21.15.44:0/3316012792 learned_addr learned my addr 172.21.15.44:0/3316012792 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:10.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.203+0000 7f6168a5e640 1 --2- 172.21.15.44:0/3316012792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61641017a0 0x7f61640754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:10.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.203+0000 7f616925f640 1 --2- 172.21.15.44:0/3316012792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6164103350 0x7f6164074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:10.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.203+0000 7f6163fff640 1 -- 172.21.15.44:0/3316012792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6164103350 msgr2=0x7f6164074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:10.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.203+0000 7f6163fff640 1 --2- 172.21.15.44:0/3316012792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6164103350 0x7f6164074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:10.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.203+0000 7f6163fff640 1 -- 172.21.15.44:0/3316012792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61641017a0 msgr2=0x7f61640754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:10.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.203+0000 7f6163fff640 1 --2- 172.21.15.44:0/3316012792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61641017a0 0x7f61640754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:10.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.203+0000 7f6163fff640 1 -- 172.21.15.44:0/3316012792 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6164074900 con 0x7f6164102990 2024-04-21T23:06:10.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.203+0000 7f6168a5e640 1 --2- 172.21.15.44:0/3316012792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61641017a0 0x7f61640754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:06:10.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.203+0000 7f6163fff640 1 --2- 172.21.15.44:0/3316012792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6164102990 0x7f6164073b30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f615800c920 tx=0x7f615800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:10.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.204+0000 7f6161ffb640 1 -- 172.21.15.44:0/3316012792 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6158007c70 con 0x7f6164102990 2024-04-21T23:06:10.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.204+0000 7f616acc2640 1 -- 172.21.15.44:0/3316012792 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61641a1aa0 con 0x7f6164102990 2024-04-21T23:06:10.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.204+0000 7f6161ffb640 1 -- 172.21.15.44:0/3316012792 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6158007e10 con 0x7f6164102990 2024-04-21T23:06:10.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.204+0000 7f6161ffb640 1 -- 172.21.15.44:0/3316012792 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6158018890 con 0x7f6164102990 2024-04-21T23:06:10.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.204+0000 7f616acc2640 1 -- 172.21.15.44:0/3316012792 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61641a2040 con 0x7f6164102990 2024-04-21T23:06:10.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.205+0000 7f616acc2640 1 -- 172.21.15.44:0/3316012792 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6164108a60 con 0x7f6164102990 2024-04-21T23:06:10.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.206+0000 7f6161ffb640 1 -- 172.21.15.44:0/3316012792 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6158018b30 con 0x7f6164102990 2024-04-21T23:06:10.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.207+0000 7f6161ffb640 1 --2- 172.21.15.44:0/3316012792 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f614c0a2b90 0x7f614c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:10.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.207+0000 7f6168a5e640 1 --2- 172.21.15.44:0/3316012792 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f614c0a2b90 0x7f614c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:10.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.207+0000 7f6161ffb640 1 -- 172.21.15.44:0/3316012792 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f61580cedd0 con 0x7f6164102990 2024-04-21T23:06:10.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.207+0000 7f6168a5e640 1 --2- 172.21.15.44:0/3316012792 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f614c0a2b90 0x7f614c0a5050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f61641027f0 tx=0x7f615400a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:10.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.211+0000 7f6161ffb640 1 -- 172.21.15.44:0/3316012792 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6158014030 con 0x7f6164102990 2024-04-21T23:06:10.409 INFO:journalctl@ceph.mds.h.smithi055.stdout:Apr 21 23:06:10 smithi055 podman[61224]: 2024-04-21 23:06:10.035327758 +0000 UTC m=+0.018498712 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:06:10.409 INFO:journalctl@ceph.mds.h.smithi055.stdout:Apr 21 23:06:10 smithi055 podman[61224]: 2024-04-21 23:06:10.14882283 +0000 UTC m=+0.131993762 container create 4ecdcab93f9b8f822121d19431bee23217e89a0a1e5ecbf2e77c39b6ef36ebf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-21T23:06:10.409 INFO:journalctl@ceph.mds.h.smithi055.stdout:Apr 21 23:06:10 smithi055 podman[61224]: 2024-04-21 23:06:10.410290077 +0000 UTC m=+0.393461010 container init 4ecdcab93f9b8f822121d19431bee23217e89a0a1e5ecbf2e77c39b6ef36ebf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-h, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-21T23:06:10.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.439+0000 7f616acc2640 1 -- 172.21.15.44:0/3316012792 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f616402d170 con 0x7f6164102990 2024-04-21T23:06:10.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.440+0000 7f6161ffb640 1 -- 172.21.15.44:0/3316012792 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6334 (secure 0 0 0) 0x7f615800de20 con 0x7f6164102990 2024-04-21T23:06:10.439 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:10.440 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-21T23:06:10.440 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 8 2024-04-21T23:06:10.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.442+0000 7f616acc2640 1 -- 172.21.15.44:0/3316012792 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f614c0a2b90 msgr2=0x7f614c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:10.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.442+0000 7f616acc2640 1 --2- 172.21.15.44:0/3316012792 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f614c0a2b90 0x7f614c0a5050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f61641027f0 tx=0x7f615400a560 comp rx=0 tx=0).stop 2024-04-21T23:06:10.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.442+0000 7f616acc2640 1 -- 172.21.15.44:0/3316012792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6164102990 msgr2=0x7f6164073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:10.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.442+0000 7f616acc2640 1 --2- 172.21.15.44:0/3316012792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6164102990 0x7f6164073b30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f615800c920 tx=0x7f615800cde0 comp rx=0 tx=0).stop 2024-04-21T23:06:10.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.442+0000 7f616acc2640 1 -- 172.21.15.44:0/3316012792 shutdown_connections 2024-04-21T23:06:10.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.442+0000 7f616acc2640 1 --2- 172.21.15.44:0/3316012792 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f614c0a2b90 0x7f614c0a5050 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:10.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.442+0000 7f616acc2640 1 --2- 172.21.15.44:0/3316012792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6164103350 0x7f6164074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:10.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.442+0000 7f616acc2640 1 --2- 172.21.15.44:0/3316012792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6164102990 0x7f6164073b30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:10.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.442+0000 7f616acc2640 1 --2- 172.21.15.44:0/3316012792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61641017a0 0x7f61640754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:10.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.442+0000 7f616acc2640 1 -- 172.21.15.44:0/3316012792 >> 172.21.15.44:0/3316012792 conn(0x7f61640fcd30 msgr2=0x7f6164105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:10.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.442+0000 7f616acc2640 1 -- 172.21.15.44:0/3316012792 shutdown_connections 2024-04-21T23:06:10.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:10.442+0000 7f616acc2640 1 -- 172.21.15.44:0/3316012792 wait complete. 2024-04-21T23:06:10.714 INFO:journalctl@ceph.mds.h.smithi055.stdout:Apr 21 23:06:10 smithi055 podman[61224]: 2024-04-21 23:06:10.413152826 +0000 UTC m=+0.396323765 container start 4ecdcab93f9b8f822121d19431bee23217e89a0a1e5ecbf2e77c39b6ef36ebf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8) 2024-04-21T23:06:10.715 INFO:journalctl@ceph.mds.h.smithi055.stdout:Apr 21 23:06:10 smithi055 ceph-mds[61244]: -- 172.21.15.55:0/2779701351 <== mon.2 v2:172.21.15.55:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f4c604da40 con 0x55f4c6020800 2024-04-21T23:06:10.715 INFO:journalctl@ceph.mds.h.smithi055.stdout:Apr 21 23:06:10 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-h[61239]: starting mds.h at 2024-04-21T23:06:10.715 INFO:journalctl@ceph.mds.h.smithi055.stdout:Apr 21 23:06:10 smithi055 bash[61224]: 4ecdcab93f9b8f822121d19431bee23217e89a0a1e5ecbf2e77c39b6ef36ebf6 2024-04-21T23:06:10.715 INFO:journalctl@ceph.mds.h.smithi055.stdout:Apr 21 23:06:10 smithi055 systemd[1]: Started Ceph mds.h for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:06:11.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:10 smithi044 ceph-mon[27934]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:11.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:10 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3316012792' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:11.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:10 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:11.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:10 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:11.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:10 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:11.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:10 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:11.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:10 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:11.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:10 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:11.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:10 smithi055 ceph-mon[31117]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:11.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:10 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3316012792' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:11.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:10 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:11.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:10 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:11.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:10 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:11.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:10 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:11.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:10 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:11.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:10 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:11.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:10 smithi069 ceph-mon[31816]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:11.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:10 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3316012792' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:11.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:10 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:11.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:10 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:11.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:10 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:11.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:10 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:11.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:10 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:11.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:10 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:11.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:11 smithi055 ceph-mon[31117]: Deploying daemon mds.i on smithi069 2024-04-21T23:06:11.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:11 smithi055 ceph-mon[31117]: mds.? [v2:172.21.15.55:6836/1372428654,v1:172.21.15.55:6837/1372428654] up:boot 2024-04-21T23:06:11.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:11 smithi055 ceph-mon[31117]: fsmap 8 up:standby 2024-04-21T23:06:11.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:11 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T23:06:12.087 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:11 smithi044 ceph-mon[27934]: Deploying daemon mds.i on smithi069 2024-04-21T23:06:12.087 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:11 smithi044 ceph-mon[27934]: mds.? [v2:172.21.15.55:6836/1372428654,v1:172.21.15.55:6837/1372428654] up:boot 2024-04-21T23:06:12.088 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:11 smithi044 ceph-mon[27934]: fsmap 8 up:standby 2024-04-21T23:06:12.088 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:11 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T23:06:12.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:11 smithi069 ceph-mon[31816]: Deploying daemon mds.i on smithi069 2024-04-21T23:06:12.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:11 smithi069 ceph-mon[31816]: mds.? [v2:172.21.15.55:6836/1372428654,v1:172.21.15.55:6837/1372428654] up:boot 2024-04-21T23:06:12.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:11 smithi069 ceph-mon[31816]: fsmap 8 up:standby 2024-04-21T23:06:12.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:11 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T23:06:12.454 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:06:12.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.570+0000 7fbd08488640 1 -- 172.21.15.44:0/4200258730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd00101890 msgr2=0x7fbd00101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:12.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.570+0000 7fbd08488640 1 --2- 172.21.15.44:0/4200258730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd00101890 0x7fbd00101d10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbcf0009900 tx=0x7fbcf002bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:12.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.570+0000 7fbd08488640 1 -- 172.21.15.44:0/4200258730 shutdown_connections 2024-04-21T23:06:12.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.570+0000 7fbd08488640 1 --2- 172.21.15.44:0/4200258730 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd00102250 0x7fbd00108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:12.573 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.570+0000 7fbd08488640 1 --2- 172.21.15.44:0/4200258730 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd00101890 0x7fbd00101d10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.570+0000 7fbd08488640 1 --2- 172.21.15.44:0/4200258730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd001006a0 0x7fbd00100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.570+0000 7fbd08488640 1 -- 172.21.15.44:0/4200258730 >> 172.21.15.44:0/4200258730 conn(0x7fbd000fbc10 msgr2=0x7fbd000fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.570+0000 7fbd08488640 1 -- 172.21.15.44:0/4200258730 shutdown_connections 2024-04-21T23:06:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.571+0000 7fbd08488640 1 -- 172.21.15.44:0/4200258730 wait complete. 2024-04-21T23:06:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.571+0000 7fbd08488640 1 Processor -- start 2024-04-21T23:06:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.571+0000 7fbd08488640 1 -- start start 2024-04-21T23:06:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.571+0000 7fbd08488640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd001006a0 0x7fbd00104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.571+0000 7fbd08488640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd00101890 0x7fbd00102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.571+0000 7fbd08488640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd00102250 0x7fbd001031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.571+0000 7fbd08488640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0010c320 con 0x7fbd00102250 2024-04-21T23:06:12.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.571+0000 7fbd08488640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0010c1a0 con 0x7fbd001006a0 2024-04-21T23:06:12.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.571+0000 7fbd08488640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0010c4a0 con 0x7fbd00101890 2024-04-21T23:06:12.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.572+0000 7fbd05a23640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd00101890 0x7fbd00102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:12.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.572+0000 7fbd06a25640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd00102250 0x7fbd001031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:12.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.572+0000 7fbd06224640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd001006a0 0x7fbd00104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:12.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.572+0000 7fbd05a23640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd00101890 0x7fbd00102c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:43512/0 (socket says 172.21.15.44:43512) 2024-04-21T23:06:12.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.572+0000 7fbd06224640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd001006a0 0x7fbd00104630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:58754/0 (socket says 172.21.15.44:58754) 2024-04-21T23:06:12.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.572+0000 7fbd05a23640 1 -- 172.21.15.44:0/2065299000 learned_addr learned my addr 172.21.15.44:0/2065299000 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:12.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.572+0000 7fbd05a23640 1 -- 172.21.15.44:0/2065299000 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd001006a0 msgr2=0x7fbd00104630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:12.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.572+0000 7fbd05a23640 1 --2- 172.21.15.44:0/2065299000 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd001006a0 0x7fbd00104630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:12.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.572+0000 7fbd05a23640 1 -- 172.21.15.44:0/2065299000 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd00102250 msgr2=0x7fbd001031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:12.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.572+0000 7fbd05a23640 1 --2- 172.21.15.44:0/2065299000 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd00102250 0x7fbd001031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:12.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.572+0000 7fbd05a23640 1 -- 172.21.15.44:0/2065299000 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd00103a40 con 0x7fbd00101890 2024-04-21T23:06:12.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.572+0000 7fbd06a25640 1 --2- 172.21.15.44:0/2065299000 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd00102250 0x7fbd001031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:06:12.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.572+0000 7fbd06224640 1 --2- 172.21.15.44:0/2065299000 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd001006a0 0x7fbd00104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:06:12.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.572+0000 7fbd05a23640 1 --2- 172.21.15.44:0/2065299000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd00101890 0x7fbd00102c70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbcf0009230 tx=0x7fbcf002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:12.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.572+0000 7fbcf77fe640 1 -- 172.21.15.44:0/2065299000 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf0041070 con 0x7fbd00101890 2024-04-21T23:06:12.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.572+0000 7fbd08488640 1 -- 172.21.15.44:0/2065299000 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd001a1b70 con 0x7fbd00101890 2024-04-21T23:06:12.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.572+0000 7fbcf77fe640 1 -- 172.21.15.44:0/2065299000 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcf0004450 con 0x7fbd00101890 2024-04-21T23:06:12.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.572+0000 7fbcf77fe640 1 -- 172.21.15.44:0/2065299000 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf0003e20 con 0x7fbd00101890 2024-04-21T23:06:12.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.573+0000 7fbd08488640 1 -- 172.21.15.44:0/2065299000 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd001a20c0 con 0x7fbd00101890 2024-04-21T23:06:12.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.573+0000 7fbd08488640 1 -- 172.21.15.44:0/2065299000 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd001a2290 con 0x7fbd00101890 2024-04-21T23:06:12.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.575+0000 7fbcf77fe640 1 -- 172.21.15.44:0/2065299000 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbcf0044070 con 0x7fbd00101890 2024-04-21T23:06:12.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.576+0000 7fbcf77fe640 1 --2- 172.21.15.44:0/2065299000 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbcec0a2b90 0x7fbcec0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:12.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.576+0000 7fbcf77fe640 1 -- 172.21.15.44:0/2065299000 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fbcf00f0670 con 0x7fbd00101890 2024-04-21T23:06:12.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.576+0000 7fbd06224640 1 --2- 172.21.15.44:0/2065299000 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbcec0a2b90 0x7fbcec0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:12.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.577+0000 7fbd06224640 1 --2- 172.21.15.44:0/2065299000 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbcec0a2b90 0x7fbcec0a5050 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbd001016f0 tx=0x7fbcfc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:12.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.579+0000 7fbcf77fe640 1 -- 172.21.15.44:0/2065299000 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fbcf00bd420 con 0x7fbd00101890 2024-04-21T23:06:12.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.807+0000 7fbd08488640 1 -- 172.21.15.44:0/2065299000 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd001a2460 con 0x7fbd00101890 2024-04-21T23:06:12.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.807+0000 7fbcf77fe640 1 -- 172.21.15.44:0/2065299000 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7159 (secure 0 0 0) 0x7fbcf00c1720 con 0x7fbd00101890 2024-04-21T23:06:12.807 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:12.807 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-21T23:06:12.808 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 9 2024-04-21T23:06:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.809+0000 7fbd08488640 1 -- 172.21.15.44:0/2065299000 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbcec0a2b90 msgr2=0x7fbcec0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:12.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.809+0000 7fbd08488640 1 --2- 172.21.15.44:0/2065299000 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbcec0a2b90 0x7fbcec0a5050 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbd001016f0 tx=0x7fbcfc005f90 comp rx=0 tx=0).stop 2024-04-21T23:06:12.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.809+0000 7fbd08488640 1 -- 172.21.15.44:0/2065299000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd00101890 msgr2=0x7fbd00102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:12.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.809+0000 7fbd08488640 1 --2- 172.21.15.44:0/2065299000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd00101890 0x7fbd00102c70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbcf0009230 tx=0x7fbcf002c650 comp rx=0 tx=0).stop 2024-04-21T23:06:12.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.809+0000 7fbd08488640 1 -- 172.21.15.44:0/2065299000 shutdown_connections 2024-04-21T23:06:12.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.809+0000 7fbd08488640 1 --2- 172.21.15.44:0/2065299000 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbcec0a2b90 0x7fbcec0a5050 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:12.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.809+0000 7fbd08488640 1 --2- 172.21.15.44:0/2065299000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd00101890 0x7fbd00102c70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:12.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.809+0000 7fbd08488640 1 --2- 172.21.15.44:0/2065299000 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd00102250 0x7fbd001031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:12.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.809+0000 7fbd08488640 1 --2- 172.21.15.44:0/2065299000 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd001006a0 0x7fbd00104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:12.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.809+0000 7fbd08488640 1 -- 172.21.15.44:0/2065299000 >> 172.21.15.44:0/2065299000 conn(0x7fbd000fbc10 msgr2=0x7fbd000fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:12.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.809+0000 7fbd08488640 1 -- 172.21.15.44:0/2065299000 shutdown_connections 2024-04-21T23:06:12.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:12.809+0000 7fbd08488640 1 -- 172.21.15.44:0/2065299000 wait complete. 2024-04-21T23:06:12.960 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:12 smithi069 ceph-mon[31816]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:13.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:12 smithi044 ceph-mon[27934]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:13.211 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:06:13 smithi069 ceph-mds[63204]: -- 172.21.15.69:0/45832789 <== mon.1 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560a66ea9a40 con 0x560a66e7cc00 2024-04-21T23:06:13.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:12 smithi055 ceph-mon[31117]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:13.464 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:06:13 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: starting mds.i at 2024-04-21T23:06:13.464 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:06:13 smithi069 bash[63185]: fa6d4b0214a33607ad61e9bcbd42c69147945602695b3896aaf4739d44caef61 2024-04-21T23:06:13.464 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:06:13 smithi069 systemd[1]: Started Ceph mds.i for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:06:14.022 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:13 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2065299000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:14.022 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:13 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:14.022 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:13 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:14.022 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:13 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:14.022 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:13 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:14.022 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:13 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:14.022 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:13 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:14.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:13 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2065299000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:14.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:13 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:14.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:13 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:14.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:13 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:14.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:13 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:14.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:13 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:14.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:13 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:14.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:13 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2065299000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:14.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:13 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:14.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:13 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:14.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:13 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:14.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:13 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:14.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:13 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:14.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:13 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:14.822 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:06:14.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.941+0000 7fd3d3b8a640 1 -- 172.21.15.44:0/1121266051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3cc0ff450 msgr2=0x7fd3cc0ff870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:14.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.941+0000 7fd3d3b8a640 1 --2- 172.21.15.44:0/1121266051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3cc0ff450 0x7fd3cc0ff870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009900 tx=0x7fd3c802bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:14.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.941+0000 7fd3d3b8a640 1 -- 172.21.15.44:0/1121266051 shutdown_connections 2024-04-21T23:06:14.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.941+0000 7fd3d3b8a640 1 --2- 172.21.15.44:0/1121266051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3cc1074d0 0x7fd3cc109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:14.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.941+0000 7fd3d3b8a640 1 --2- 172.21.15.44:0/1121266051 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3cc0ffe40 0x7fd3cc106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:14.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.941+0000 7fd3d3b8a640 1 --2- 172.21.15.44:0/1121266051 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3cc0ff450 0x7fd3cc0ff870 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:14.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.941+0000 7fd3d3b8a640 1 -- 172.21.15.44:0/1121266051 >> 172.21.15.44:0/1121266051 conn(0x7fd3cc0faf80 msgr2=0x7fd3cc0fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:14.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.941+0000 7fd3d3b8a640 1 -- 172.21.15.44:0/1121266051 shutdown_connections 2024-04-21T23:06:14.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.941+0000 7fd3d3b8a640 1 -- 172.21.15.44:0/1121266051 wait complete. 2024-04-21T23:06:14.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.942+0000 7fd3d3b8a640 1 Processor -- start 2024-04-21T23:06:14.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.942+0000 7fd3d3b8a640 1 -- start start 2024-04-21T23:06:14.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.942+0000 7fd3d3b8a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3cc0ff450 0x7fd3cc1028e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:14.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.942+0000 7fd3d3b8a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3cc0ffe40 0x7fd3cc100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:14.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.942+0000 7fd3d3b8a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3cc1074d0 0x7fd3cc101460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:14.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.942+0000 7fd3d3b8a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc10d420 con 0x7fd3cc1074d0 2024-04-21T23:06:14.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.942+0000 7fd3d3b8a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc10d2a0 con 0x7fd3cc0ffe40 2024-04-21T23:06:14.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.942+0000 7fd3d3b8a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc10d5a0 con 0x7fd3cc0ff450 2024-04-21T23:06:14.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.942+0000 7fd3d2127640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3cc1074d0 0x7fd3cc101460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:14.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.942+0000 7fd3d2127640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3cc1074d0 0x7fd3cc101460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50748/0 (socket says 172.21.15.44:50748) 2024-04-21T23:06:14.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.942+0000 7fd3d2127640 1 -- 172.21.15.44:0/2574229496 learned_addr learned my addr 172.21.15.44:0/2574229496 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:14.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.942+0000 7fd3d1125640 1 --2- 172.21.15.44:0/2574229496 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3cc0ffe40 0x7fd3cc100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:14.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.942+0000 7fd3d1926640 1 --2- 172.21.15.44:0/2574229496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3cc0ff450 0x7fd3cc1028e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:14.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.942+0000 7fd3d2127640 1 -- 172.21.15.44:0/2574229496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3cc0ff450 msgr2=0x7fd3cc1028e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:14.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.942+0000 7fd3d2127640 1 --2- 172.21.15.44:0/2574229496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3cc0ff450 0x7fd3cc1028e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:14.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.942+0000 7fd3d2127640 1 -- 172.21.15.44:0/2574229496 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3cc0ffe40 msgr2=0x7fd3cc100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:14.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.942+0000 7fd3d2127640 1 --2- 172.21.15.44:0/2574229496 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3cc0ffe40 0x7fd3cc100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:14.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.942+0000 7fd3d2127640 1 -- 172.21.15.44:0/2574229496 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3cc101cf0 con 0x7fd3cc1074d0 2024-04-21T23:06:14.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.943+0000 7fd3d2127640 1 --2- 172.21.15.44:0/2574229496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3cc1074d0 0x7fd3cc101460 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd3c400eb40 tx=0x7fd3c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:14.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.943+0000 7fd3c2ffd640 1 -- 172.21.15.44:0/2574229496 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4004b20 con 0x7fd3cc1074d0 2024-04-21T23:06:14.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.943+0000 7fd3c2ffd640 1 -- 172.21.15.44:0/2574229496 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3c4004cc0 con 0x7fd3cc1074d0 2024-04-21T23:06:14.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.943+0000 7fd3d3b8a640 1 -- 172.21.15.44:0/2574229496 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3cc1a1b80 con 0x7fd3cc1074d0 2024-04-21T23:06:14.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.943+0000 7fd3c2ffd640 1 -- 172.21.15.44:0/2574229496 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4010850 con 0x7fd3cc1074d0 2024-04-21T23:06:14.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.943+0000 7fd3d3b8a640 1 -- 172.21.15.44:0/2574229496 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3cc1a2120 con 0x7fd3cc1074d0 2024-04-21T23:06:14.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.944+0000 7fd3d3b8a640 1 -- 172.21.15.44:0/2574229496 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3cc105da0 con 0x7fd3cc1074d0 2024-04-21T23:06:14.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.945+0000 7fd3c2ffd640 1 -- 172.21.15.44:0/2574229496 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd3c4005000 con 0x7fd3cc1074d0 2024-04-21T23:06:14.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.946+0000 7fd3c2ffd640 1 --2- 172.21.15.44:0/2574229496 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd3b80a2c10 0x7fd3b80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:14.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.946+0000 7fd3d1926640 1 --2- 172.21.15.44:0/2574229496 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd3b80a2c10 0x7fd3b80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:14.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.946+0000 7fd3c2ffd640 1 -- 172.21.15.44:0/2574229496 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd3c40cf320 con 0x7fd3cc1074d0 2024-04-21T23:06:14.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.946+0000 7fd3d1926640 1 --2- 172.21.15.44:0/2574229496 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd3b80a2c10 0x7fd3b80a50d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8006010 tx=0x7fd3c8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:14.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:14.949+0000 7fd3c2ffd640 1 -- 172.21.15.44:0/2574229496 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd3c409c720 con 0x7fd3cc1074d0 2024-04-21T23:06:15.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:14 smithi044 ceph-mon[27934]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:15.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:14 smithi044 ceph-mon[27934]: Deploying daemon mds.g on smithi044 2024-04-21T23:06:15.047 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:14 smithi044 ceph-mon[27934]: mds.? [v2:172.21.15.69:6838/3743898853,v1:172.21.15.69:6839/3743898853] up:boot 2024-04-21T23:06:15.048 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:14 smithi044 ceph-mon[27934]: fsmap 9 up:standby 2024-04-21T23:06:15.048 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:14 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:06:15.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:15.173+0000 7fd3d3b8a640 1 -- 172.21.15.44:0/2574229496 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3cc02d170 con 0x7fd3cc1074d0 2024-04-21T23:06:15.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:15.174+0000 7fd3c2ffd640 1 -- 172.21.15.44:0/2574229496 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7986 (secure 0 0 0) 0x7fd3c409c020 con 0x7fd3cc1074d0 2024-04-21T23:06:15.174 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:15.174 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24526,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/3743898853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3743898853},{"type":"v1","addr":"172.21.15.69:6839","nonce":3743898853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T23:06:15.174 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 10 2024-04-21T23:06:15.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:15.176+0000 7fd3d3b8a640 1 -- 172.21.15.44:0/2574229496 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd3b80a2c10 msgr2=0x7fd3b80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:15.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:15.176+0000 7fd3d3b8a640 1 --2- 172.21.15.44:0/2574229496 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd3b80a2c10 0x7fd3b80a50d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8006010 tx=0x7fd3c8000bc0 comp rx=0 tx=0).stop 2024-04-21T23:06:15.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:15.176+0000 7fd3d3b8a640 1 -- 172.21.15.44:0/2574229496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3cc1074d0 msgr2=0x7fd3cc101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:15.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:15.176+0000 7fd3d3b8a640 1 --2- 172.21.15.44:0/2574229496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3cc1074d0 0x7fd3cc101460 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd3c400eb40 tx=0x7fd3c400c2d0 comp rx=0 tx=0).stop 2024-04-21T23:06:15.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:15.176+0000 7fd3d3b8a640 1 -- 172.21.15.44:0/2574229496 shutdown_connections 2024-04-21T23:06:15.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:15.176+0000 7fd3d3b8a640 1 --2- 172.21.15.44:0/2574229496 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd3b80a2c10 0x7fd3b80a50d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:15.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:15.176+0000 7fd3d3b8a640 1 --2- 172.21.15.44:0/2574229496 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3cc1074d0 0x7fd3cc101460 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:15.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:15.176+0000 7fd3d3b8a640 1 --2- 172.21.15.44:0/2574229496 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3cc0ffe40 0x7fd3cc100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:15.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:15.176+0000 7fd3d3b8a640 1 --2- 172.21.15.44:0/2574229496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3cc0ff450 0x7fd3cc1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:15.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:15.176+0000 7fd3d3b8a640 1 -- 172.21.15.44:0/2574229496 >> 172.21.15.44:0/2574229496 conn(0x7fd3cc0faf80 msgr2=0x7fd3cc102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:15.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:15.176+0000 7fd3d3b8a640 1 -- 172.21.15.44:0/2574229496 shutdown_connections 2024-04-21T23:06:15.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:15.176+0000 7fd3d3b8a640 1 -- 172.21.15.44:0/2574229496 wait complete. 2024-04-21T23:06:15.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:14 smithi055 ceph-mon[31117]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:15.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:14 smithi055 ceph-mon[31117]: Deploying daemon mds.g on smithi044 2024-04-21T23:06:15.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:14 smithi055 ceph-mon[31117]: mds.? [v2:172.21.15.69:6838/3743898853,v1:172.21.15.69:6839/3743898853] up:boot 2024-04-21T23:06:15.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:14 smithi055 ceph-mon[31117]: fsmap 9 up:standby 2024-04-21T23:06:15.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:14 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:06:15.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:14 smithi069 ceph-mon[31816]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:15.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:14 smithi069 ceph-mon[31816]: Deploying daemon mds.g on smithi044 2024-04-21T23:06:15.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:14 smithi069 ceph-mon[31816]: mds.? [v2:172.21.15.69:6838/3743898853,v1:172.21.15.69:6839/3743898853] up:boot 2024-04-21T23:06:15.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:14 smithi069 ceph-mon[31816]: fsmap 9 up:standby 2024-04-21T23:06:15.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:14 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:06:15.685 INFO:journalctl@ceph.mds.g.smithi044.stdout:Apr 21 23:06:15 smithi044 systemd[1]: Starting Ceph mds.g for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:06:16.061 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:15 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2574229496' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:16.062 INFO:journalctl@ceph.mds.g.smithi044.stdout:Apr 21 23:06:15 smithi044 podman[66750]: 2024-04-21 23:06:15.707125767 +0000 UTC m=+0.018565815 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:06:16.062 INFO:journalctl@ceph.mds.g.smithi044.stdout:Apr 21 23:06:15 smithi044 podman[66750]: 2024-04-21 23:06:15.818641898 +0000 UTC m=+0.130081987 container create 0bc183c485bb907862c5a1f46ea168313a615e7967aa2a3f43dd4d3a6f751810 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:06:16.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:15 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2574229496' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:16.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:15 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2574229496' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:16.435 INFO:journalctl@ceph.mds.g.smithi044.stdout:Apr 21 23:06:16 smithi044 podman[66750]: 2024-04-21 23:06:16.063136949 +0000 UTC m=+0.374576999 container init 0bc183c485bb907862c5a1f46ea168313a615e7967aa2a3f43dd4d3a6f751810 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-g, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-21T23:06:16.435 INFO:journalctl@ceph.mds.g.smithi044.stdout:Apr 21 23:06:16 smithi044 podman[66750]: 2024-04-21 23:06:16.065969824 +0000 UTC m=+0.377409874 container start 0bc183c485bb907862c5a1f46ea168313a615e7967aa2a3f43dd4d3a6f751810 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux ) 2024-04-21T23:06:16.435 INFO:journalctl@ceph.mds.g.smithi044.stdout:Apr 21 23:06:16 smithi044 ceph-mds[66769]: -- 172.21.15.44:0/668838680 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56406e16da40 con 0x56406e140c00 2024-04-21T23:06:16.435 INFO:journalctl@ceph.mds.g.smithi044.stdout:Apr 21 23:06:16 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-g[66765]: starting mds.g at 2024-04-21T23:06:16.436 INFO:journalctl@ceph.mds.g.smithi044.stdout:Apr 21 23:06:16 smithi044 bash[66750]: 0bc183c485bb907862c5a1f46ea168313a615e7967aa2a3f43dd4d3a6f751810 2024-04-21T23:06:16.436 INFO:journalctl@ceph.mds.g.smithi044.stdout:Apr 21 23:06:16 smithi044 systemd[1]: Started Ceph mds.g for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:06:17.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:16 smithi044 ceph-mon[27934]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:17.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:16 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:17.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:16 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:17.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:16 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:17.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:16 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:17.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:16 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:17.037 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:16 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:17.188 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:06:17.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:16 smithi069 ceph-mon[31816]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:17.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:16 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:17.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:16 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:17.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:16 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:17.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:16 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:17.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:16 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:17.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:16 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:17.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:16 smithi055 ceph-mon[31117]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:17.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:17.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:17.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:17.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:17.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:17.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:16 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:17.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.295+0000 7f17cad77640 1 -- 172.21.15.44:0/553353162 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17c41006a0 msgr2=0x7f17c4100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:17.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.295+0000 7f17cad77640 1 --2- 172.21.15.44:0/553353162 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17c41006a0 0x7f17c4100ac0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f17b8009900 tx=0x7f17b802bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:17.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.295+0000 7f17cad77640 1 -- 172.21.15.44:0/553353162 shutdown_connections 2024-04-21T23:06:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.295+0000 7f17cad77640 1 --2- 172.21.15.44:0/553353162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17c4101890 0x7f17c4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.295+0000 7f17cad77640 1 --2- 172.21.15.44:0/553353162 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17c4102250 0x7f17c4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.295+0000 7f17cad77640 1 --2- 172.21.15.44:0/553353162 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17c41006a0 0x7f17c4100ac0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.295+0000 7f17cad77640 1 -- 172.21.15.44:0/553353162 >> 172.21.15.44:0/553353162 conn(0x7f17c40fbc50 msgr2=0x7f17c40fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.295+0000 7f17cad77640 1 -- 172.21.15.44:0/553353162 shutdown_connections 2024-04-21T23:06:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.295+0000 7f17cad77640 1 -- 172.21.15.44:0/553353162 wait complete. 2024-04-21T23:06:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.296+0000 7f17cad77640 1 Processor -- start 2024-04-21T23:06:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.296+0000 7f17cad77640 1 -- start start 2024-04-21T23:06:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.296+0000 7f17cad77640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17c41006a0 0x7f17c406a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.296+0000 7f17cad77640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17c4101890 0x7f17c419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.296+0000 7f17cad77640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17c4102250 0x7f17c419eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.296+0000 7f17cad77640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f17c410c3e0 con 0x7f17c41006a0 2024-04-21T23:06:17.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.296+0000 7f17cad77640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f17c410c260 con 0x7f17c4101890 2024-04-21T23:06:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.296+0000 7f17cad77640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f17c410c560 con 0x7f17c4102250 2024-04-21T23:06:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.296+0000 7f17c8b13640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17c41006a0 0x7f17c406a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.296+0000 7f17c8b13640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17c41006a0 0x7f17c406a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50792/0 (socket says 172.21.15.44:50792) 2024-04-21T23:06:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.296+0000 7f17c8b13640 1 -- 172.21.15.44:0/4237602029 learned_addr learned my addr 172.21.15.44:0/4237602029 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.296+0000 7f17c9314640 1 --2- 172.21.15.44:0/4237602029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17c4102250 0x7f17c419eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.296+0000 7f17c3fff640 1 --2- 172.21.15.44:0/4237602029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17c4101890 0x7f17c419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.296+0000 7f17c8b13640 1 -- 172.21.15.44:0/4237602029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17c4102250 msgr2=0x7f17c419eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.296+0000 7f17c8b13640 1 --2- 172.21.15.44:0/4237602029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17c4102250 0x7f17c419eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.296+0000 7f17c8b13640 1 -- 172.21.15.44:0/4237602029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17c4101890 msgr2=0x7f17c419c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.296+0000 7f17c8b13640 1 --2- 172.21.15.44:0/4237602029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17c4101890 0x7f17c419c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.296+0000 7f17c8b13640 1 -- 172.21.15.44:0/4237602029 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c419f7d0 con 0x7f17c41006a0 2024-04-21T23:06:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.296+0000 7f17c3fff640 1 --2- 172.21.15.44:0/4237602029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17c4101890 0x7f17c419c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:06:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.296+0000 7f17c8b13640 1 --2- 172.21.15.44:0/4237602029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17c41006a0 0x7f17c406a9c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f17b8000c00 tx=0x7f17b8005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.297+0000 7f17c1ffb640 1 -- 172.21.15.44:0/4237602029 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b8043020 con 0x7f17c41006a0 2024-04-21T23:06:17.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.297+0000 7f17c1ffb640 1 -- 172.21.15.44:0/4237602029 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17b80054e0 con 0x7f17c41006a0 2024-04-21T23:06:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.297+0000 7f17cad77640 1 -- 172.21.15.44:0/4237602029 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17c419fa00 con 0x7f17c41006a0 2024-04-21T23:06:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.297+0000 7f17c1ffb640 1 -- 172.21.15.44:0/4237602029 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b8003e60 con 0x7f17c41006a0 2024-04-21T23:06:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.297+0000 7f17cad77640 1 -- 172.21.15.44:0/4237602029 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17c419ffa0 con 0x7f17c41006a0 2024-04-21T23:06:17.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.298+0000 7f17cad77640 1 -- 172.21.15.44:0/4237602029 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17c4068a70 con 0x7f17c41006a0 2024-04-21T23:06:17.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.299+0000 7f17c1ffb640 1 -- 172.21.15.44:0/4237602029 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f17b8046020 con 0x7f17c41006a0 2024-04-21T23:06:17.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.299+0000 7f17c1ffb640 1 --2- 172.21.15.44:0/4237602029 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f17b40a2c10 0x7f17b40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:17.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.300+0000 7f17c3fff640 1 --2- 172.21.15.44:0/4237602029 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f17b40a2c10 0x7f17b40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:17.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.300+0000 7f17c1ffb640 1 -- 172.21.15.44:0/4237602029 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f17b80f0f80 con 0x7f17c41006a0 2024-04-21T23:06:17.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.300+0000 7f17c3fff640 1 --2- 172.21.15.44:0/4237602029 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f17b40a2c10 0x7f17b40a50d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f17c406a730 tx=0x7f17b0005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:17.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.304+0000 7f17c1ffb640 1 -- 172.21.15.44:0/4237602029 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f17c4068a70 con 0x7f17c41006a0 2024-04-21T23:06:17.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.530+0000 7f17cad77640 1 -- 172.21.15.44:0/4237602029 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17c4062370 con 0x7f17c41006a0 2024-04-21T23:06:17.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.530+0000 7f17c1ffb640 1 -- 172.21.15.44:0/4237602029 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8812 (secure 0 0 0) 0x7f17b804d020 con 0x7f17c41006a0 2024-04-21T23:06:17.531 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:17.531 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24526,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/3743898853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3743898853},{"type":"v1","addr":"172.21.15.69:6839","nonce":3743898853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T23:06:17.531 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 11 2024-04-21T23:06:17.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.533+0000 7f17cad77640 1 -- 172.21.15.44:0/4237602029 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f17b40a2c10 msgr2=0x7f17b40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:17.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.533+0000 7f17cad77640 1 --2- 172.21.15.44:0/4237602029 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f17b40a2c10 0x7f17b40a50d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f17c406a730 tx=0x7f17b0005e50 comp rx=0 tx=0).stop 2024-04-21T23:06:17.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.533+0000 7f17cad77640 1 -- 172.21.15.44:0/4237602029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17c41006a0 msgr2=0x7f17c406a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:17.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.533+0000 7f17cad77640 1 --2- 172.21.15.44:0/4237602029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17c41006a0 0x7f17c406a9c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f17b8000c00 tx=0x7f17b8005dd0 comp rx=0 tx=0).stop 2024-04-21T23:06:17.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.533+0000 7f17cad77640 1 -- 172.21.15.44:0/4237602029 shutdown_connections 2024-04-21T23:06:17.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.533+0000 7f17cad77640 1 --2- 172.21.15.44:0/4237602029 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f17b40a2c10 0x7f17b40a50d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:17.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.533+0000 7f17cad77640 1 --2- 172.21.15.44:0/4237602029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17c4102250 0x7f17c419eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:17.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.533+0000 7f17cad77640 1 --2- 172.21.15.44:0/4237602029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17c4101890 0x7f17c419c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:17.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.533+0000 7f17cad77640 1 --2- 172.21.15.44:0/4237602029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f17c41006a0 0x7f17c406a9c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:17.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.533+0000 7f17cad77640 1 -- 172.21.15.44:0/4237602029 >> 172.21.15.44:0/4237602029 conn(0x7f17c40fbc50 msgr2=0x7f17c4104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:17.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.533+0000 7f17cad77640 1 -- 172.21.15.44:0/4237602029 shutdown_connections 2024-04-21T23:06:17.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:17.533+0000 7f17cad77640 1 -- 172.21.15.44:0/4237602029 wait complete. 2024-04-21T23:06:17.904 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:17 smithi044 ceph-mon[27934]: Deploying daemon mds.a on smithi044 2024-04-21T23:06:17.904 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:17 smithi044 ceph-mon[27934]: mds.? [v2:172.21.15.44:6838/1544956047,v1:172.21.15.44:6839/1544956047] up:boot 2024-04-21T23:06:17.904 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:17 smithi044 ceph-mon[27934]: fsmap 10 up:standby 2024-04-21T23:06:17.904 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:17 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T23:06:17.904 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:17 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/4237602029' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:17.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:17 smithi069 ceph-mon[31816]: Deploying daemon mds.a on smithi044 2024-04-21T23:06:17.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:17 smithi069 ceph-mon[31816]: mds.? [v2:172.21.15.44:6838/1544956047,v1:172.21.15.44:6839/1544956047] up:boot 2024-04-21T23:06:17.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:17 smithi069 ceph-mon[31816]: fsmap 10 up:standby 2024-04-21T23:06:17.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:17 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T23:06:17.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:17 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/4237602029' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:18.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:17 smithi055 ceph-mon[31117]: Deploying daemon mds.a on smithi044 2024-04-21T23:06:18.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:17 smithi055 ceph-mon[31117]: mds.? [v2:172.21.15.44:6838/1544956047,v1:172.21.15.44:6839/1544956047] up:boot 2024-04-21T23:06:18.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:17 smithi055 ceph-mon[31117]: fsmap 10 up:standby 2024-04-21T23:06:18.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:17 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T23:06:18.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:17 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/4237602029' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:18.602 INFO:journalctl@ceph.mds.a.smithi044.stdout:Apr 21 23:06:18 smithi044 systemd[1]: Starting Ceph mds.a for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:06:18.857 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:18 smithi044 ceph-mon[27934]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:18.857 INFO:journalctl@ceph.mds.a.smithi044.stdout:Apr 21 23:06:18 smithi044 podman[67097]: 2024-04-21 23:06:18.507069859 +0000 UTC m=+0.018646317 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:06:18.857 INFO:journalctl@ceph.mds.a.smithi044.stdout:Apr 21 23:06:18 smithi044 podman[67097]: 2024-04-21 23:06:18.708427121 +0000 UTC m=+0.220003559 container create 8d3075c20f536433232d7aba0d67f3399ecade9e9dff0b0404fcd14bb89cd97d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416) 2024-04-21T23:06:19.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:18 smithi055 ceph-mon[31117]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:19.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:18 smithi069 ceph-mon[31816]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:19.435 INFO:journalctl@ceph.mds.a.smithi044.stdout:Apr 21 23:06:18 smithi044 podman[67097]: 2024-04-21 23:06:18.978014948 +0000 UTC m=+0.489591387 container init 8d3075c20f536433232d7aba0d67f3399ecade9e9dff0b0404fcd14bb89cd97d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-21T23:06:19.436 INFO:journalctl@ceph.mds.a.smithi044.stdout:Apr 21 23:06:18 smithi044 podman[67097]: 2024-04-21 23:06:18.980761491 +0000 UTC m=+0.492337937 container start 8d3075c20f536433232d7aba0d67f3399ecade9e9dff0b0404fcd14bb89cd97d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-a, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-21T23:06:19.436 INFO:journalctl@ceph.mds.a.smithi044.stdout:Apr 21 23:06:19 smithi044 ceph-mds[67116]: -- 172.21.15.44:0/568075365 <== mon.0 v2:172.21.15.44:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d68052ba40 con 0x55d6804fec00 2024-04-21T23:06:19.436 INFO:journalctl@ceph.mds.a.smithi044.stdout:Apr 21 23:06:19 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-a[67112]: starting mds.a at 2024-04-21T23:06:19.436 INFO:journalctl@ceph.mds.a.smithi044.stdout:Apr 21 23:06:19 smithi044 bash[67097]: 8d3075c20f536433232d7aba0d67f3399ecade9e9dff0b0404fcd14bb89cd97d 2024-04-21T23:06:19.436 INFO:journalctl@ceph.mds.a.smithi044.stdout:Apr 21 23:06:19 smithi044 systemd[1]: Started Ceph mds.a for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:06:19.544 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:06:19.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.655+0000 7fad910c9640 1 -- 172.21.15.44:0/1877077720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad8c103360 msgr2=0x7fad8c109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:19.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.655+0000 7fad910c9640 1 --2- 172.21.15.44:0/1877077720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad8c103360 0x7fad8c109bf0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fad84009ca0 tx=0x7fad8402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:19.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.655+0000 7fad910c9640 1 -- 172.21.15.44:0/1877077720 shutdown_connections 2024-04-21T23:06:19.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.655+0000 7fad910c9640 1 --2- 172.21.15.44:0/1877077720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad8c1029a0 0x7fad8c102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:19.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.655+0000 7fad910c9640 1 --2- 172.21.15.44:0/1877077720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad8c103360 0x7fad8c109bf0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:19.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.655+0000 7fad910c9640 1 --2- 172.21.15.44:0/1877077720 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fad8c1017b0 0x7fad8c101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:19.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.655+0000 7fad910c9640 1 -- 172.21.15.44:0/1877077720 >> 172.21.15.44:0/1877077720 conn(0x7fad8c0fcd20 msgr2=0x7fad8c0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:19.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.655+0000 7fad910c9640 1 -- 172.21.15.44:0/1877077720 shutdown_connections 2024-04-21T23:06:19.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.655+0000 7fad910c9640 1 -- 172.21.15.44:0/1877077720 wait complete. 2024-04-21T23:06:19.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.655+0000 7fad910c9640 1 Processor -- start 2024-04-21T23:06:19.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad910c9640 1 -- start start 2024-04-21T23:06:19.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad910c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad8c1017b0 0x7fad8c0754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:19.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad910c9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fad8c1029a0 0x7fad8c073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:19.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad910c9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad8c103360 0x7fad8c074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:19.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad910c9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fad8c10d430 con 0x7fad8c103360 2024-04-21T23:06:19.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad910c9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fad8c10d2b0 con 0x7fad8c1029a0 2024-04-21T23:06:19.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad910c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fad8c10d5b0 con 0x7fad8c1017b0 2024-04-21T23:06:19.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad8b59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad8c103360 0x7fad8c074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:19.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad8b59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad8c103360 0x7fad8c074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50824/0 (socket says 172.21.15.44:50824) 2024-04-21T23:06:19.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad8b59e640 1 -- 172.21.15.44:0/2397624932 learned_addr learned my addr 172.21.15.44:0/2397624932 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:19.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad8259c640 1 --2- 172.21.15.44:0/2397624932 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fad8c1029a0 0x7fad8c073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:19.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad8ad9d640 1 --2- 172.21.15.44:0/2397624932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad8c1017b0 0x7fad8c0754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:19.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad8b59e640 1 -- 172.21.15.44:0/2397624932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad8c1017b0 msgr2=0x7fad8c0754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:19.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad8b59e640 1 --2- 172.21.15.44:0/2397624932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad8c1017b0 0x7fad8c0754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:19.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad8b59e640 1 -- 172.21.15.44:0/2397624932 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fad8c1029a0 msgr2=0x7fad8c073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:19.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad8b59e640 1 --2- 172.21.15.44:0/2397624932 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fad8c1029a0 0x7fad8c073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:19.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad8b59e640 1 -- 172.21.15.44:0/2397624932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad8c074900 con 0x7fad8c103360 2024-04-21T23:06:19.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad8ad9d640 1 --2- 172.21.15.44:0/2397624932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad8c1017b0 0x7fad8c0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:06:19.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad8259c640 1 --2- 172.21.15.44:0/2397624932 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fad8c1029a0 0x7fad8c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:06:19.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad8b59e640 1 --2- 172.21.15.44:0/2397624932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad8c103360 0x7fad8c074070 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fad84037040 tx=0x7fad84032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:19.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.656+0000 7fad88d99640 1 -- 172.21.15.44:0/2397624932 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad84041020 con 0x7fad8c103360 2024-04-21T23:06:19.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.657+0000 7fad910c9640 1 -- 172.21.15.44:0/2397624932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad8c1a1ad0 con 0x7fad8c103360 2024-04-21T23:06:19.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.657+0000 7fad88d99640 1 -- 172.21.15.44:0/2397624932 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad84034d90 con 0x7fad8c103360 2024-04-21T23:06:19.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.657+0000 7fad88d99640 1 -- 172.21.15.44:0/2397624932 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad84039040 con 0x7fad8c103360 2024-04-21T23:06:19.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.657+0000 7fad910c9640 1 -- 172.21.15.44:0/2397624932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad8c1a2020 con 0x7fad8c103360 2024-04-21T23:06:19.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.658+0000 7fad910c9640 1 -- 172.21.15.44:0/2397624932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad8c108a70 con 0x7fad8c103360 2024-04-21T23:06:19.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.659+0000 7fad88d99640 1 -- 172.21.15.44:0/2397624932 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fad84044020 con 0x7fad8c103360 2024-04-21T23:06:19.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.660+0000 7fad88d99640 1 --2- 172.21.15.44:0/2397624932 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fad780a2b90 0x7fad780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:19.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.660+0000 7fad8ad9d640 1 --2- 172.21.15.44:0/2397624932 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fad780a2b90 0x7fad780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:19.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.660+0000 7fad88d99640 1 -- 172.21.15.44:0/2397624932 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fad840f0b30 con 0x7fad8c103360 2024-04-21T23:06:19.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.660+0000 7fad8ad9d640 1 --2- 172.21.15.44:0/2397624932 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fad780a2b90 0x7fad780a5050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fad8c102800 tx=0x7fad7c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:19.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.664+0000 7fad88d99640 1 -- 172.21.15.44:0/2397624932 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fad8404b020 con 0x7fad8c103360 2024-04-21T23:06:19.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.889+0000 7fad910c9640 1 -- 172.21.15.44:0/2397624932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad8c062340 con 0x7fad8c103360 2024-04-21T23:06:19.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.890+0000 7fad88d99640 1 -- 172.21.15.44:0/2397624932 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9638 (secure 0 0 0) 0x7fad840bdf30 con 0x7fad8c103360 2024-04-21T23:06:19.890 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:19.890 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24526,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/3743898853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3743898853},{"type":"v1","addr":"172.21.15.69:6839","nonce":3743898853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T23:06:19.891 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 12 2024-04-21T23:06:19.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.892+0000 7fad910c9640 1 -- 172.21.15.44:0/2397624932 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fad780a2b90 msgr2=0x7fad780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:19.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.892+0000 7fad910c9640 1 --2- 172.21.15.44:0/2397624932 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fad780a2b90 0x7fad780a5050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fad8c102800 tx=0x7fad7c005f90 comp rx=0 tx=0).stop 2024-04-21T23:06:19.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.892+0000 7fad910c9640 1 -- 172.21.15.44:0/2397624932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad8c103360 msgr2=0x7fad8c074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:19.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.892+0000 7fad910c9640 1 --2- 172.21.15.44:0/2397624932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad8c103360 0x7fad8c074070 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fad84037040 tx=0x7fad84032f20 comp rx=0 tx=0).stop 2024-04-21T23:06:19.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.892+0000 7fad910c9640 1 -- 172.21.15.44:0/2397624932 shutdown_connections 2024-04-21T23:06:19.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.892+0000 7fad910c9640 1 --2- 172.21.15.44:0/2397624932 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fad780a2b90 0x7fad780a5050 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:19.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.892+0000 7fad910c9640 1 --2- 172.21.15.44:0/2397624932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad8c103360 0x7fad8c074070 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:19.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.892+0000 7fad910c9640 1 --2- 172.21.15.44:0/2397624932 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fad8c1029a0 0x7fad8c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:19.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.892+0000 7fad910c9640 1 --2- 172.21.15.44:0/2397624932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad8c1017b0 0x7fad8c0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:19.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.892+0000 7fad910c9640 1 -- 172.21.15.44:0/2397624932 >> 172.21.15.44:0/2397624932 conn(0x7fad8c0fcd20 msgr2=0x7fad8c105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:19.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.892+0000 7fad910c9640 1 -- 172.21.15.44:0/2397624932 shutdown_connections 2024-04-21T23:06:19.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:19.892+0000 7fad910c9640 1 -- 172.21.15.44:0/2397624932 wait complete. 2024-04-21T23:06:20.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:20 smithi044 ceph-mon[27934]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:20.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:20 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:20.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:20 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:20.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:20 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:20.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:20 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:20.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:20 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:20.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:20 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:20.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:20 smithi044 ceph-mon[27934]: Deploying daemon mds.b on smithi055 2024-04-21T23:06:20.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:20 smithi044 ceph-mon[27934]: mds.? [v2:172.21.15.44:6840/1863599778,v1:172.21.15.44:6841/1863599778] up:boot 2024-04-21T23:06:20.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:20 smithi044 ceph-mon[27934]: fsmap 11 up:standby 2024-04-21T23:06:20.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:20 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T23:06:20.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:20 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2397624932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:20.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:20 smithi069 ceph-mon[31816]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:20.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:20 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:20.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:20 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:20.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:20 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:20.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:20 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:20.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:20 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:20.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:20 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:20.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:20 smithi069 ceph-mon[31816]: Deploying daemon mds.b on smithi055 2024-04-21T23:06:20.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:20 smithi069 ceph-mon[31816]: mds.? [v2:172.21.15.44:6840/1863599778,v1:172.21.15.44:6841/1863599778] up:boot 2024-04-21T23:06:20.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:20 smithi069 ceph-mon[31816]: fsmap 11 up:standby 2024-04-21T23:06:20.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:20 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T23:06:20.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:20 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2397624932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:20.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:20 smithi055 ceph-mon[31117]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:20.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:20 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:20.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:20 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:20.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:20 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:20.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:20 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T23:06:20.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:20 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T23:06:20.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:20 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:20.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:20 smithi055 ceph-mon[31117]: Deploying daemon mds.b on smithi055 2024-04-21T23:06:20.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:20 smithi055 ceph-mon[31117]: mds.? [v2:172.21.15.44:6840/1863599778,v1:172.21.15.44:6841/1863599778] up:boot 2024-04-21T23:06:20.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:20 smithi055 ceph-mon[31117]: fsmap 11 up:standby 2024-04-21T23:06:20.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:20 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T23:06:20.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:20 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2397624932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:21.845 INFO:journalctl@ceph.mds.b.smithi055.stdout:Apr 21 23:06:21 smithi055 podman[61545]: 2024-04-21 23:06:21.727002634 +0000 UTC m=+0.019138971 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:06:21.845 INFO:journalctl@ceph.mds.b.smithi055.stdout:Apr 21 23:06:21 smithi055 podman[61545]: 2024-04-21 23:06:21.846461096 +0000 UTC m=+0.138597434 container create c87ce9581cca077152dacb68c6ca328247ae81f1ef015fb0371aeb85cf16ac13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-21T23:06:21.904 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:06:22.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.019+0000 7fcecbe0e640 1 -- 172.21.15.44:0/84992403 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcec41006a0 msgr2=0x7fcec4100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:22.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.019+0000 7fcecbe0e640 1 --2- 172.21.15.44:0/84992403 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcec41006a0 0x7fcec4100ac0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcec0009900 tx=0x7fcec002bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:22.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.019+0000 7fcecbe0e640 1 -- 172.21.15.44:0/84992403 shutdown_connections 2024-04-21T23:06:22.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.019+0000 7fcecbe0e640 1 --2- 172.21.15.44:0/84992403 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec4102250 0x7fcec4108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.019+0000 7fcecbe0e640 1 --2- 172.21.15.44:0/84992403 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcec4101890 0x7fcec4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.019+0000 7fcecbe0e640 1 --2- 172.21.15.44:0/84992403 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcec41006a0 0x7fcec4100ac0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.019+0000 7fcecbe0e640 1 -- 172.21.15.44:0/84992403 >> 172.21.15.44:0/84992403 conn(0x7fcec40fbc50 msgr2=0x7fcec40fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:22.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.019+0000 7fcecbe0e640 1 -- 172.21.15.44:0/84992403 shutdown_connections 2024-04-21T23:06:22.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.019+0000 7fcecbe0e640 1 -- 172.21.15.44:0/84992403 wait complete. 2024-04-21T23:06:22.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.020+0000 7fcecbe0e640 1 Processor -- start 2024-04-21T23:06:22.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.020+0000 7fcecbe0e640 1 -- start start 2024-04-21T23:06:22.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.020+0000 7fcecbe0e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcec41006a0 0x7fcec406aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:22.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.020+0000 7fcecbe0e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec4101890 0x7fcec419c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:22.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.020+0000 7fcecbe0e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcec4102250 0x7fcec419ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:22.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.020+0000 7fcecbe0e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcec410c370 con 0x7fcec4102250 2024-04-21T23:06:22.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.020+0000 7fcecbe0e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcec410c1f0 con 0x7fcec41006a0 2024-04-21T23:06:22.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.020+0000 7fcecbe0e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcec410c4f0 con 0x7fcec4101890 2024-04-21T23:06:22.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.020+0000 7fceca3ab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcec4102250 0x7fcec419ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:22.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.020+0000 7fceca3ab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcec4102250 0x7fcec419ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36122/0 (socket says 172.21.15.44:36122) 2024-04-21T23:06:22.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.020+0000 7fceca3ab640 1 -- 172.21.15.44:0/3048695338 learned_addr learned my addr 172.21.15.44:0/3048695338 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.020+0000 7fcec93a9640 1 --2- 172.21.15.44:0/3048695338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec4101890 0x7fcec419c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.020+0000 7fceca3ab640 1 -- 172.21.15.44:0/3048695338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec4101890 msgr2=0x7fcec419c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.020+0000 7fceca3ab640 1 --2- 172.21.15.44:0/3048695338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec4101890 0x7fcec419c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.020+0000 7fceca3ab640 1 -- 172.21.15.44:0/3048695338 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcec41006a0 msgr2=0x7fcec406aa20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T23:06:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.020+0000 7fceca3ab640 1 --2- 172.21.15.44:0/3048695338 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcec41006a0 0x7fcec406aa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.020+0000 7fceca3ab640 1 -- 172.21.15.44:0/3048695338 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec419f730 con 0x7fcec4102250 2024-04-21T23:06:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.020+0000 7fcec93a9640 1 --2- 172.21.15.44:0/3048695338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec4101890 0x7fcec419c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:06:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.020+0000 7fceca3ab640 1 --2- 172.21.15.44:0/3048695338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcec4102250 0x7fcec419ee70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00b970 tx=0x7fcebc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.020+0000 7fcebaffd640 1 -- 172.21.15.44:0/3048695338 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc019070 con 0x7fcec4102250 2024-04-21T23:06:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.021+0000 7fcebaffd640 1 -- 172.21.15.44:0/3048695338 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcebc00cb40 con 0x7fcec4102250 2024-04-21T23:06:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.021+0000 7fcecbe0e640 1 -- 172.21.15.44:0/3048695338 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcec419fa20 con 0x7fcec4102250 2024-04-21T23:06:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.021+0000 7fcebaffd640 1 -- 172.21.15.44:0/3048695338 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc007650 con 0x7fcec4102250 2024-04-21T23:06:22.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.021+0000 7fcecbe0e640 1 -- 172.21.15.44:0/3048695338 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec419ff70 con 0x7fcec4102250 2024-04-21T23:06:22.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.022+0000 7fcecbe0e640 1 -- 172.21.15.44:0/3048695338 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcec4107a40 con 0x7fcec4102250 2024-04-21T23:06:22.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.023+0000 7fcebaffd640 1 -- 172.21.15.44:0/3048695338 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fcebc01c070 con 0x7fcec4102250 2024-04-21T23:06:22.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.023+0000 7fcebaffd640 1 --2- 172.21.15.44:0/3048695338 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fceb00a2c10 0x7fceb00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:22.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.024+0000 7fcec9baa640 1 --2- 172.21.15.44:0/3048695338 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fceb00a2c10 0x7fceb00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:22.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.024+0000 7fcebaffd640 1 -- 172.21.15.44:0/3048695338 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fcebc0ce6d0 con 0x7fcec4102250 2024-04-21T23:06:22.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.024+0000 7fcec9baa640 1 --2- 172.21.15.44:0/3048695338 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fceb00a2c10 0x7fceb00a50d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcec002c710 tx=0x7fcec0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:22.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.027+0000 7fcebaffd640 1 -- 172.21.15.44:0/3048695338 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fcebc09ba30 con 0x7fcec4102250 2024-04-21T23:06:22.116 INFO:journalctl@ceph.mds.b.smithi055.stdout:Apr 21 23:06:22 smithi055 podman[61545]: 2024-04-21 23:06:22.091035816 +0000 UTC m=+0.383172153 container init c87ce9581cca077152dacb68c6ca328247ae81f1ef015fb0371aeb85cf16ac13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-21T23:06:22.116 INFO:journalctl@ceph.mds.b.smithi055.stdout:Apr 21 23:06:22 smithi055 podman[61545]: 2024-04-21 23:06:22.094029499 +0000 UTC m=+0.386165838 container start c87ce9581cca077152dacb68c6ca328247ae81f1ef015fb0371aeb85cf16ac13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:06:22.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:22 smithi044 ceph-mon[27934]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:22.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.253+0000 7fcecbe0e640 1 -- 172.21.15.44:0/3048695338 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcec402d170 con 0x7fcec4102250 2024-04-21T23:06:22.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.254+0000 7fcebaffd640 1 -- 172.21.15.44:0/3048695338 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10464 (secure 0 0 0) 0x7fcebc09b330 con 0x7fcec4102250 2024-04-21T23:06:22.254 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:22.254 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24526,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/3743898853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3743898853},{"type":"v1","addr":"172.21.15.69:6839","nonce":3743898853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-21T23:06:22.254 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 13 2024-04-21T23:06:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.256+0000 7fcecbe0e640 1 -- 172.21.15.44:0/3048695338 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fceb00a2c10 msgr2=0x7fceb00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.256+0000 7fcecbe0e640 1 --2- 172.21.15.44:0/3048695338 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fceb00a2c10 0x7fceb00a50d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcec002c710 tx=0x7fcec0005fd0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.256+0000 7fcecbe0e640 1 -- 172.21.15.44:0/3048695338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcec4102250 msgr2=0x7fcec419ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.256+0000 7fcecbe0e640 1 --2- 172.21.15.44:0/3048695338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcec4102250 0x7fcec419ee70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00b970 tx=0x7fcebc00be30 comp rx=0 tx=0).stop 2024-04-21T23:06:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.256+0000 7fcecbe0e640 1 -- 172.21.15.44:0/3048695338 shutdown_connections 2024-04-21T23:06:22.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.256+0000 7fcecbe0e640 1 --2- 172.21.15.44:0/3048695338 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fceb00a2c10 0x7fceb00a50d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.256+0000 7fcecbe0e640 1 --2- 172.21.15.44:0/3048695338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec4101890 0x7fcec419c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.256+0000 7fcecbe0e640 1 --2- 172.21.15.44:0/3048695338 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcec4102250 0x7fcec419ee70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.256+0000 7fcecbe0e640 1 --2- 172.21.15.44:0/3048695338 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcec41006a0 0x7fcec406aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.256+0000 7fcecbe0e640 1 -- 172.21.15.44:0/3048695338 >> 172.21.15.44:0/3048695338 conn(0x7fcec40fbc50 msgr2=0x7fcec4104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.256+0000 7fcecbe0e640 1 -- 172.21.15.44:0/3048695338 shutdown_connections 2024-04-21T23:06:22.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.256+0000 7fcecbe0e640 1 -- 172.21.15.44:0/3048695338 wait complete. 2024-04-21T23:06:22.268 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:06:22.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.370+0000 7fbb76166640 1 -- 172.21.15.44:0/426043454 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb701006a0 msgr2=0x7fbb70100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:22.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.370+0000 7fbb76166640 1 --2- 172.21.15.44:0/426043454 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb701006a0 0x7fbb70100ac0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbb60009900 tx=0x7fbb6002bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:22.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.370+0000 7fbb76166640 1 -- 172.21.15.44:0/426043454 shutdown_connections 2024-04-21T23:06:22.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.370+0000 7fbb76166640 1 --2- 172.21.15.44:0/426043454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb70102250 0x7fbb70108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.370+0000 7fbb76166640 1 --2- 172.21.15.44:0/426043454 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb70101890 0x7fbb70101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.370+0000 7fbb76166640 1 --2- 172.21.15.44:0/426043454 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb701006a0 0x7fbb70100ac0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.370+0000 7fbb76166640 1 -- 172.21.15.44:0/426043454 >> 172.21.15.44:0/426043454 conn(0x7fbb700fbc50 msgr2=0x7fbb700fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:22.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.370+0000 7fbb76166640 1 -- 172.21.15.44:0/426043454 shutdown_connections 2024-04-21T23:06:22.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.370+0000 7fbb76166640 1 -- 172.21.15.44:0/426043454 wait complete. 2024-04-21T23:06:22.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.371+0000 7fbb76166640 1 Processor -- start 2024-04-21T23:06:22.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.371+0000 7fbb76166640 1 -- start start 2024-04-21T23:06:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.371+0000 7fbb76166640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb701006a0 0x7fbb70197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.371+0000 7fbb76166640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb70101890 0x7fbb701980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.371+0000 7fbb76166640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb70102250 0x7fbb7019f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.371+0000 7fbb76166640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7010d4c0 con 0x7fbb701006a0 2024-04-21T23:06:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.371+0000 7fbb76166640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7010d340 con 0x7fbb70102250 2024-04-21T23:06:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.371+0000 7fbb76166640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7010d640 con 0x7fbb70101890 2024-04-21T23:06:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.371+0000 7fbb6f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb701006a0 0x7fbb70197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.371+0000 7fbb6f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb701006a0 0x7fbb70197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36140/0 (socket says 172.21.15.44:36140) 2024-04-21T23:06:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.371+0000 7fbb6f7fe640 1 -- 172.21.15.44:0/1230829171 learned_addr learned my addr 172.21.15.44:0/1230829171 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.371+0000 7fbb6effd640 1 --2- 172.21.15.44:0/1230829171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb70101890 0x7fbb701980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:22.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.371+0000 7fbb6ffff640 1 --2- 172.21.15.44:0/1230829171 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb70102250 0x7fbb7019f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.371+0000 7fbb6f7fe640 1 -- 172.21.15.44:0/1230829171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb70101890 msgr2=0x7fbb701980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.371+0000 7fbb6f7fe640 1 --2- 172.21.15.44:0/1230829171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb70101890 0x7fbb701980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.371+0000 7fbb6f7fe640 1 -- 172.21.15.44:0/1230829171 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb70102250 msgr2=0x7fbb7019f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.371+0000 7fbb6f7fe640 1 --2- 172.21.15.44:0/1230829171 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb70102250 0x7fbb7019f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.371+0000 7fbb6f7fe640 1 -- 172.21.15.44:0/1230829171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb7019f7b0 con 0x7fbb701006a0 2024-04-21T23:06:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.372+0000 7fbb6f7fe640 1 --2- 172.21.15.44:0/1230829171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb701006a0 0x7fbb70197b90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbb60000c00 tx=0x7fbb60004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.372+0000 7fbb6cff9640 1 -- 172.21.15.44:0/1230829171 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb60043020 con 0x7fbb701006a0 2024-04-21T23:06:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.372+0000 7fbb76166640 1 -- 172.21.15.44:0/1230829171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb7019fa40 con 0x7fbb701006a0 2024-04-21T23:06:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.372+0000 7fbb6cff9640 1 -- 172.21.15.44:0/1230829171 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb60004440 con 0x7fbb701006a0 2024-04-21T23:06:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.372+0000 7fbb6cff9640 1 -- 172.21.15.44:0/1230829171 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6003e450 con 0x7fbb701006a0 2024-04-21T23:06:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.372+0000 7fbb76166640 1 -- 172.21.15.44:0/1230829171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb7019ff90 con 0x7fbb701006a0 2024-04-21T23:06:22.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.373+0000 7fbb76166640 1 -- 172.21.15.44:0/1230829171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb701078f0 con 0x7fbb701006a0 2024-04-21T23:06:22.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.374+0000 7fbb6cff9640 1 -- 172.21.15.44:0/1230829171 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbb60046020 con 0x7fbb701006a0 2024-04-21T23:06:22.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.375+0000 7fbb6cff9640 1 --2- 172.21.15.44:0/1230829171 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbb580a2b90 0x7fbb580a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:22.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.375+0000 7fbb6effd640 1 --2- 172.21.15.44:0/1230829171 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbb580a2b90 0x7fbb580a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:22.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.375+0000 7fbb6cff9640 1 -- 172.21.15.44:0/1230829171 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fbb600f12c0 con 0x7fbb701006a0 2024-04-21T23:06:22.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.376+0000 7fbb6effd640 1 --2- 172.21.15.44:0/1230829171 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbb580a2b90 0x7fbb580a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbb701990a0 tx=0x7fbb6400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:22.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.379+0000 7fbb6cff9640 1 -- 172.21.15.44:0/1230829171 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fbb6004d020 con 0x7fbb701006a0 2024-04-21T23:06:22.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:22 smithi055 ceph-mon[31117]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:22.464 INFO:journalctl@ceph.mds.b.smithi055.stdout:Apr 21 23:06:22 smithi055 ceph-mds[61564]: -- 172.21.15.55:0/683272229 <== mon.2 v2:172.21.15.55:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56041d845a40 con 0x56041d819000 2024-04-21T23:06:22.464 INFO:journalctl@ceph.mds.b.smithi055.stdout:Apr 21 23:06:22 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-b[61560]: starting mds.b at 2024-04-21T23:06:22.464 INFO:journalctl@ceph.mds.b.smithi055.stdout:Apr 21 23:06:22 smithi055 bash[61545]: c87ce9581cca077152dacb68c6ca328247ae81f1ef015fb0371aeb85cf16ac13 2024-04-21T23:06:22.464 INFO:journalctl@ceph.mds.b.smithi055.stdout:Apr 21 23:06:22 smithi055 systemd[1]: Started Ceph mds.b for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:06:22.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:22 smithi069 ceph-mon[31816]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:22.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.565+0000 7fbb76166640 1 -- 172.21.15.44:0/1230829171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbb7002d3e0 con 0x7fbb701006a0 2024-04-21T23:06:22.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.566+0000 7fbb6cff9640 1 -- 172.21.15.44:0/1230829171 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18360 (secure 0 0 0) 0x7fbb600be6c0 con 0x7fbb701006a0 2024-04-21T23:06:22.566 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:22.566 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":82,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:05:36.159839+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:06:22.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.568+0000 7fbb76166640 1 -- 172.21.15.44:0/1230829171 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbb580a2b90 msgr2=0x7fbb580a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:22.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.568+0000 7fbb76166640 1 --2- 172.21.15.44:0/1230829171 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbb580a2b90 0x7fbb580a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbb701990a0 tx=0x7fbb6400a2d0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.568+0000 7fbb76166640 1 -- 172.21.15.44:0/1230829171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb701006a0 msgr2=0x7fbb70197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:22.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.568+0000 7fbb76166640 1 --2- 172.21.15.44:0/1230829171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb701006a0 0x7fbb70197b90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbb60000c00 tx=0x7fbb60004040 comp rx=0 tx=0).stop 2024-04-21T23:06:22.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.568+0000 7fbb76166640 1 -- 172.21.15.44:0/1230829171 shutdown_connections 2024-04-21T23:06:22.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.568+0000 7fbb76166640 1 --2- 172.21.15.44:0/1230829171 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbb580a2b90 0x7fbb580a5050 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.568+0000 7fbb76166640 1 --2- 172.21.15.44:0/1230829171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb70101890 0x7fbb701980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.568+0000 7fbb76166640 1 --2- 172.21.15.44:0/1230829171 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb70102250 0x7fbb7019f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.568+0000 7fbb76166640 1 --2- 172.21.15.44:0/1230829171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb701006a0 0x7fbb70197b90 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.568+0000 7fbb76166640 1 -- 172.21.15.44:0/1230829171 >> 172.21.15.44:0/1230829171 conn(0x7fbb700fbc50 msgr2=0x7fbb70104ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:22.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.568+0000 7fbb76166640 1 -- 172.21.15.44:0/1230829171 shutdown_connections 2024-04-21T23:06:22.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.569+0000 7fbb76166640 1 -- 172.21.15.44:0/1230829171 wait complete. 2024-04-21T23:06:22.580 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:01:47.507503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T23:06:22.581 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:06:22.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.687+0000 7fdd29713640 1 -- 172.21.15.44:0/3168661784 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd24101870 msgr2=0x7fdd24101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:22.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.687+0000 7fdd29713640 1 --2- 172.21.15.44:0/3168661784 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd24101870 0x7fdd24101cf0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdd14009900 tx=0x7fdd1402bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:22.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.687+0000 7fdd29713640 1 -- 172.21.15.44:0/3168661784 shutdown_connections 2024-04-21T23:06:22.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.687+0000 7fdd29713640 1 --2- 172.21.15.44:0/3168661784 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd24102230 0x7fdd24108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.687+0000 7fdd29713640 1 --2- 172.21.15.44:0/3168661784 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd24101870 0x7fdd24101cf0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.687+0000 7fdd29713640 1 --2- 172.21.15.44:0/3168661784 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdd24100680 0x7fdd24100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.687+0000 7fdd29713640 1 -- 172.21.15.44:0/3168661784 >> 172.21.15.44:0/3168661784 conn(0x7fdd240fbc10 msgr2=0x7fdd240fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:22.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.687+0000 7fdd29713640 1 -- 172.21.15.44:0/3168661784 shutdown_connections 2024-04-21T23:06:22.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.687+0000 7fdd29713640 1 -- 172.21.15.44:0/3168661784 wait complete. 2024-04-21T23:06:22.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.687+0000 7fdd29713640 1 Processor -- start 2024-04-21T23:06:22.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.688+0000 7fdd29713640 1 -- start start 2024-04-21T23:06:22.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.688+0000 7fdd29713640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdd24100680 0x7fdd24199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:22.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.688+0000 7fdd29713640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd24101870 0x7fdd2419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:22.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.688+0000 7fdd29713640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd24102230 0x7fdd241a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:22.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.688+0000 7fdd29713640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2410c300 con 0x7fdd24102230 2024-04-21T23:06:22.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.688+0000 7fdd29713640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2410c180 con 0x7fdd24100680 2024-04-21T23:06:22.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.688+0000 7fdd29713640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2410c480 con 0x7fdd24101870 2024-04-21T23:06:22.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.688+0000 7fdd237fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd24102230 0x7fdd241a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:22.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.688+0000 7fdd22ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdd24100680 0x7fdd24199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:22.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.688+0000 7fdd237fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd24102230 0x7fdd241a1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36162/0 (socket says 172.21.15.44:36162) 2024-04-21T23:06:22.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.688+0000 7fdd237fe640 1 -- 172.21.15.44:0/438059808 learned_addr learned my addr 172.21.15.44:0/438059808 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:22.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.688+0000 7fdd227fc640 1 --2- 172.21.15.44:0/438059808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd24101870 0x7fdd2419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:22.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.688+0000 7fdd237fe640 1 -- 172.21.15.44:0/438059808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd24101870 msgr2=0x7fdd2419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:22.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.688+0000 7fdd237fe640 1 --2- 172.21.15.44:0/438059808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd24101870 0x7fdd2419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.688+0000 7fdd237fe640 1 -- 172.21.15.44:0/438059808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdd24100680 msgr2=0x7fdd24199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:22.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.688+0000 7fdd237fe640 1 --2- 172.21.15.44:0/438059808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdd24100680 0x7fdd24199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.688+0000 7fdd237fe640 1 -- 172.21.15.44:0/438059808 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd241a1900 con 0x7fdd24102230 2024-04-21T23:06:22.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.688+0000 7fdd227fc640 1 --2- 172.21.15.44:0/438059808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd24101870 0x7fdd2419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:06:22.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.688+0000 7fdd22ffd640 1 --2- 172.21.15.44:0/438059808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdd24100680 0x7fdd24199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:06:22.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.688+0000 7fdd237fe640 1 --2- 172.21.15.44:0/438059808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd24102230 0x7fdd241a1260 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00eb10 tx=0x7fdd1c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:22.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.689+0000 7fdd1bfff640 1 -- 172.21.15.44:0/438059808 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c004b20 con 0x7fdd24102230 2024-04-21T23:06:22.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.689+0000 7fdd1bfff640 1 -- 172.21.15.44:0/438059808 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd1c004cc0 con 0x7fdd24102230 2024-04-21T23:06:22.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.689+0000 7fdd29713640 1 -- 172.21.15.44:0/438059808 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd241a1bf0 con 0x7fdd24102230 2024-04-21T23:06:22.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.689+0000 7fdd1bfff640 1 -- 172.21.15.44:0/438059808 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c010760 con 0x7fdd24102230 2024-04-21T23:06:22.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.689+0000 7fdd29713640 1 -- 172.21.15.44:0/438059808 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd241a2110 con 0x7fdd24102230 2024-04-21T23:06:22.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.690+0000 7fdd29713640 1 -- 172.21.15.44:0/438059808 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd241a2340 con 0x7fdd24102230 2024-04-21T23:06:22.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.691+0000 7fdd1bfff640 1 -- 172.21.15.44:0/438059808 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fdd1c005020 con 0x7fdd24102230 2024-04-21T23:06:22.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.691+0000 7fdd1bfff640 1 --2- 172.21.15.44:0/438059808 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdd100a2b90 0x7fdd100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:22.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.691+0000 7fdd22ffd640 1 --2- 172.21.15.44:0/438059808 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdd100a2b90 0x7fdd100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:22.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.692+0000 7fdd1bfff640 1 -- 172.21.15.44:0/438059808 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fdd1c0cf240 con 0x7fdd24102230 2024-04-21T23:06:22.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.692+0000 7fdd22ffd640 1 --2- 172.21.15.44:0/438059808 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdd100a2b90 0x7fdd100a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdd24069750 tx=0x7fdd0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:22.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.696+0000 7fdd1bfff640 1 -- 172.21.15.44:0/438059808 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fdd1c09c640 con 0x7fdd24102230 2024-04-21T23:06:22.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.882+0000 7fdd29713640 1 -- 172.21.15.44:0/438059808 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdd241a25b0 con 0x7fdd24102230 2024-04-21T23:06:22.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.882+0000 7fdd1bfff640 1 -- 172.21.15.44:0/438059808 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fdd1c09bf40 con 0x7fdd24102230 2024-04-21T23:06:22.883 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-21T23:06:22.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.884+0000 7fdd29713640 1 -- 172.21.15.44:0/438059808 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdd100a2b90 msgr2=0x7fdd100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:22.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.884+0000 7fdd29713640 1 --2- 172.21.15.44:0/438059808 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdd100a2b90 0x7fdd100a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdd24069750 tx=0x7fdd0c005f90 comp rx=0 tx=0).stop 2024-04-21T23:06:22.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.884+0000 7fdd29713640 1 -- 172.21.15.44:0/438059808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd24102230 msgr2=0x7fdd241a1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:22.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.884+0000 7fdd29713640 1 --2- 172.21.15.44:0/438059808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd24102230 0x7fdd241a1260 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00eb10 tx=0x7fdd1c00c2d0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.884+0000 7fdd29713640 1 -- 172.21.15.44:0/438059808 shutdown_connections 2024-04-21T23:06:22.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.884+0000 7fdd29713640 1 --2- 172.21.15.44:0/438059808 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdd100a2b90 0x7fdd100a5050 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.884+0000 7fdd29713640 1 --2- 172.21.15.44:0/438059808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd24101870 0x7fdd2419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.884+0000 7fdd29713640 1 --2- 172.21.15.44:0/438059808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdd24102230 0x7fdd241a1260 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.884+0000 7fdd29713640 1 --2- 172.21.15.44:0/438059808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdd24100680 0x7fdd24199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:22.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.884+0000 7fdd29713640 1 -- 172.21.15.44:0/438059808 >> 172.21.15.44:0/438059808 conn(0x7fdd240fbc10 msgr2=0x7fdd240fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:22.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.884+0000 7fdd29713640 1 -- 172.21.15.44:0/438059808 shutdown_connections 2024-04-21T23:06:22.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:22.884+0000 7fdd29713640 1 -- 172.21.15.44:0/438059808 wait complete. 2024-04-21T23:06:22.895 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-21T23:06:23.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.010+0000 7fe8ad288640 1 -- 172.21.15.44:0/3894534659 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a81006a0 msgr2=0x7fe8a8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:23.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.010+0000 7fe8ad288640 1 --2- 172.21.15.44:0/3894534659 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a81006a0 0x7fe8a8100ac0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe898009900 tx=0x7fe89802bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:23.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.010+0000 7fe8ad288640 1 -- 172.21.15.44:0/3894534659 shutdown_connections 2024-04-21T23:06:23.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.010+0000 7fe8ad288640 1 --2- 172.21.15.44:0/3894534659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8a8102250 0x7fe8a8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:23.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.010+0000 7fe8ad288640 1 --2- 172.21.15.44:0/3894534659 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8a8101890 0x7fe8a8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:23.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.010+0000 7fe8ad288640 1 --2- 172.21.15.44:0/3894534659 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a81006a0 0x7fe8a8100ac0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:23.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.010+0000 7fe8ad288640 1 -- 172.21.15.44:0/3894534659 >> 172.21.15.44:0/3894534659 conn(0x7fe8a80fbc10 msgr2=0x7fe8a80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:23.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.010+0000 7fe8ad288640 1 -- 172.21.15.44:0/3894534659 shutdown_connections 2024-04-21T23:06:23.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.010+0000 7fe8ad288640 1 -- 172.21.15.44:0/3894534659 wait complete. 2024-04-21T23:06:23.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.011+0000 7fe8ad288640 1 Processor -- start 2024-04-21T23:06:23.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.011+0000 7fe8ad288640 1 -- start start 2024-04-21T23:06:23.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.011+0000 7fe8ad288640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8a81006a0 0x7fe8a80695b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:23.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.011+0000 7fe8ad288640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8a8101890 0x7fe8a819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:23.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.011+0000 7fe8ad288640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a8102250 0x7fe8a819ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:23.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.011+0000 7fe8ad288640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a810c320 con 0x7fe8a8102250 2024-04-21T23:06:23.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.011+0000 7fe8ad288640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a810c1a0 con 0x7fe8a8101890 2024-04-21T23:06:23.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.011+0000 7fe8ad288640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a810c4a0 con 0x7fe8a81006a0 2024-04-21T23:06:23.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.011+0000 7fe8a77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a8102250 0x7fe8a819ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:23.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.011+0000 7fe8a77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a8102250 0x7fe8a819ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36184/0 (socket says 172.21.15.44:36184) 2024-04-21T23:06:23.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.011+0000 7fe8a77fe640 1 -- 172.21.15.44:0/3176856378 learned_addr learned my addr 172.21.15.44:0/3176856378 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:23.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.011+0000 7fe8a67fc640 1 --2- 172.21.15.44:0/3176856378 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8a8101890 0x7fe8a819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:23.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.011+0000 7fe8a6ffd640 1 --2- 172.21.15.44:0/3176856378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8a81006a0 0x7fe8a80695b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:23.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.012+0000 7fe8a77fe640 1 -- 172.21.15.44:0/3176856378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8a81006a0 msgr2=0x7fe8a80695b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:23.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.012+0000 7fe8a77fe640 1 --2- 172.21.15.44:0/3176856378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8a81006a0 0x7fe8a80695b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:23.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.012+0000 7fe8a77fe640 1 -- 172.21.15.44:0/3176856378 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8a8101890 msgr2=0x7fe8a819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:23.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.012+0000 7fe8a77fe640 1 --2- 172.21.15.44:0/3176856378 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8a8101890 0x7fe8a819c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:23.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.012+0000 7fe8a77fe640 1 -- 172.21.15.44:0/3176856378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a819f750 con 0x7fe8a8102250 2024-04-21T23:06:23.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.012+0000 7fe8a67fc640 1 --2- 172.21.15.44:0/3176856378 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8a8101890 0x7fe8a819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:06:23.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.012+0000 7fe8a77fe640 1 --2- 172.21.15.44:0/3176856378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a8102250 0x7fe8a819ee40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe8a000eb10 tx=0x7fe8a000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:23.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.012+0000 7fe88ffff640 1 -- 172.21.15.44:0/3176856378 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a0004b20 con 0x7fe8a8102250 2024-04-21T23:06:23.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.012+0000 7fe8ad288640 1 -- 172.21.15.44:0/3176856378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8a819f9e0 con 0x7fe8a8102250 2024-04-21T23:06:23.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.012+0000 7fe88ffff640 1 -- 172.21.15.44:0/3176856378 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8a0004cc0 con 0x7fe8a8102250 2024-04-21T23:06:23.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.012+0000 7fe8ad288640 1 -- 172.21.15.44:0/3176856378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8a819ff80 con 0x7fe8a8102250 2024-04-21T23:06:23.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.012+0000 7fe88ffff640 1 -- 172.21.15.44:0/3176856378 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a0010760 con 0x7fe8a8102250 2024-04-21T23:06:23.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.013+0000 7fe8ad288640 1 -- 172.21.15.44:0/3176856378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8a81079c0 con 0x7fe8a8102250 2024-04-21T23:06:23.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.014+0000 7fe88ffff640 1 -- 172.21.15.44:0/3176856378 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe8a0005020 con 0x7fe8a8102250 2024-04-21T23:06:23.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.015+0000 7fe88ffff640 1 --2- 172.21.15.44:0/3176856378 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe8900a2c10 0x7fe8900a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:23.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.015+0000 7fe88ffff640 1 -- 172.21.15.44:0/3176856378 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe8a00cf220 con 0x7fe8a8102250 2024-04-21T23:06:23.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.015+0000 7fe8a6ffd640 1 --2- 172.21.15.44:0/3176856378 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe8900a2c10 0x7fe8900a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:23.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.015+0000 7fe8a6ffd640 1 --2- 172.21.15.44:0/3176856378 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe8900a2c10 0x7fe8900a50d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe898000c00 tx=0x7fe898005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:23.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.019+0000 7fe88ffff640 1 -- 172.21.15.44:0/3176856378 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe8a009c580 con 0x7fe8a8102250 2024-04-21T23:06:23.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.225+0000 7fe8ad288640 1 -- 172.21.15.44:0/3176856378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fe8a8062340 con 0x7fe8a8102250 2024-04-21T23:06:23.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.225+0000 7fe88ffff640 1 -- 172.21.15.44:0/3176856378 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7fe8a009be80 con 0x7fe8a8102250 2024-04-21T23:06:23.225 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:23.225 INFO:teuthology.orchestra.run.smithi044.stdout:[ 2024-04-21T23:06:23.225 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:06:23.225 INFO:teuthology.orchestra.run.smithi044.stdout: "poolnum": 1, 2024-04-21T23:06:23.225 INFO:teuthology.orchestra.run.smithi044.stdout: "poolname": ".mgr" 2024-04-21T23:06:23.225 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:06:23.225 INFO:teuthology.orchestra.run.smithi044.stdout:] 2024-04-21T23:06:23.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.227+0000 7fe8ad288640 1 -- 172.21.15.44:0/3176856378 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe8900a2c10 msgr2=0x7fe8900a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:23.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.227+0000 7fe8ad288640 1 --2- 172.21.15.44:0/3176856378 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe8900a2c10 0x7fe8900a50d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe898000c00 tx=0x7fe898005fd0 comp rx=0 tx=0).stop 2024-04-21T23:06:23.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.227+0000 7fe8ad288640 1 -- 172.21.15.44:0/3176856378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a8102250 msgr2=0x7fe8a819ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:23.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.227+0000 7fe8ad288640 1 --2- 172.21.15.44:0/3176856378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a8102250 0x7fe8a819ee40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe8a000eb10 tx=0x7fe8a000c2d0 comp rx=0 tx=0).stop 2024-04-21T23:06:23.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.227+0000 7fe8ad288640 1 -- 172.21.15.44:0/3176856378 shutdown_connections 2024-04-21T23:06:23.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.227+0000 7fe8ad288640 1 --2- 172.21.15.44:0/3176856378 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe8900a2c10 0x7fe8900a50d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:23.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.227+0000 7fe8ad288640 1 --2- 172.21.15.44:0/3176856378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe8a8102250 0x7fe8a819ee40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:23.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.227+0000 7fe8ad288640 1 --2- 172.21.15.44:0/3176856378 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe8a8101890 0x7fe8a819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:23.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.227+0000 7fe8ad288640 1 --2- 172.21.15.44:0/3176856378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8a81006a0 0x7fe8a80695b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:23.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.227+0000 7fe8ad288640 1 -- 172.21.15.44:0/3176856378 >> 172.21.15.44:0/3176856378 conn(0x7fe8a80fbc10 msgr2=0x7fe8a8104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:23.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.228+0000 7fe8ad288640 1 -- 172.21.15.44:0/3176856378 shutdown_connections 2024-04-21T23:06:23.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.228+0000 7fe8ad288640 1 -- 172.21.15.44:0/3176856378 wait complete. 2024-04-21T23:06:23.238 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-21T23:06:23.238 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-21T23:06:23.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.347+0000 7f1a6ff3a640 1 -- 172.21.15.44:0/3198389795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a68101870 msgr2=0x7f1a68101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:23.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.347+0000 7f1a6ff3a640 1 --2- 172.21.15.44:0/3198389795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a68101870 0x7f1a68101cf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1a64009900 tx=0x7f1a6402bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:23.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.347+0000 7f1a6ff3a640 1 -- 172.21.15.44:0/3198389795 shutdown_connections 2024-04-21T23:06:23.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.347+0000 7f1a6ff3a640 1 --2- 172.21.15.44:0/3198389795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a68102230 0x7f1a68108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:23.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.347+0000 7f1a6ff3a640 1 --2- 172.21.15.44:0/3198389795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a68101870 0x7f1a68101cf0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:23.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.347+0000 7f1a6ff3a640 1 --2- 172.21.15.44:0/3198389795 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a68100680 0x7f1a68100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:23.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.347+0000 7f1a6ff3a640 1 -- 172.21.15.44:0/3198389795 >> 172.21.15.44:0/3198389795 conn(0x7f1a680fbc10 msgr2=0x7f1a680fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:23.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.347+0000 7f1a6ff3a640 1 -- 172.21.15.44:0/3198389795 shutdown_connections 2024-04-21T23:06:23.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.347+0000 7f1a6ff3a640 1 -- 172.21.15.44:0/3198389795 wait complete. 2024-04-21T23:06:23.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.347+0000 7f1a6ff3a640 1 Processor -- start 2024-04-21T23:06:23.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.348+0000 7f1a6ff3a640 1 -- start start 2024-04-21T23:06:23.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.348+0000 7f1a6ff3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a68100680 0x7f1a68196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:23.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.348+0000 7f1a6ff3a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a68101870 0x7f1a68196fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:23.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.348+0000 7f1a6ff3a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a68102230 0x7f1a6819e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:23.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.348+0000 7f1a6ff3a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6810c3c0 con 0x7f1a68102230 2024-04-21T23:06:23.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.348+0000 7f1a6ff3a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6810c240 con 0x7f1a68101870 2024-04-21T23:06:23.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.348+0000 7f1a6ff3a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6810c540 con 0x7f1a68100680 2024-04-21T23:06:23.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.348+0000 7f1a6e4d7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a68102230 0x7f1a6819e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:23.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.348+0000 7f1a6dcd6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a68100680 0x7f1a68196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:23.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.348+0000 7f1a6dcd6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a68100680 0x7f1a68196a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:42906/0 (socket says 172.21.15.44:42906) 2024-04-21T23:06:23.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.348+0000 7f1a6dcd6640 1 -- 172.21.15.44:0/2769105674 learned_addr learned my addr 172.21.15.44:0/2769105674 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:23.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.348+0000 7f1a6e4d7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a68102230 0x7f1a6819e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36204/0 (socket says 172.21.15.44:36204) 2024-04-21T23:06:23.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.348+0000 7f1a6d4d5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a68101870 0x7f1a68196fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:23.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.348+0000 7f1a6e4d7640 1 -- 172.21.15.44:0/2769105674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a68100680 msgr2=0x7f1a68196a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:23.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.348+0000 7f1a6e4d7640 1 --2- 172.21.15.44:0/2769105674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a68100680 0x7f1a68196a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:23.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.348+0000 7f1a6e4d7640 1 -- 172.21.15.44:0/2769105674 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a68101870 msgr2=0x7f1a68196fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:23.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.348+0000 7f1a6e4d7640 1 --2- 172.21.15.44:0/2769105674 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a68101870 0x7f1a68196fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:23.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.348+0000 7f1a6e4d7640 1 -- 172.21.15.44:0/2769105674 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a6819e680 con 0x7f1a68102230 2024-04-21T23:06:23.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.348+0000 7f1a6dcd6640 1 --2- 172.21.15.44:0/2769105674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a68100680 0x7f1a68196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:06:23.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.348+0000 7f1a6e4d7640 1 --2- 172.21.15.44:0/2769105674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a68102230 0x7f1a6819e010 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1a6000da10 tx=0x7f1a6000ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:23.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.349+0000 7f1a5effd640 1 -- 172.21.15.44:0/2769105674 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6000bd70 con 0x7f1a68102230 2024-04-21T23:06:23.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.349+0000 7f1a5effd640 1 -- 172.21.15.44:0/2769105674 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a60003e50 con 0x7f1a68102230 2024-04-21T23:06:23.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.349+0000 7f1a6ff3a640 1 -- 172.21.15.44:0/2769105674 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a6819e970 con 0x7f1a68102230 2024-04-21T23:06:23.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.349+0000 7f1a5effd640 1 -- 172.21.15.44:0/2769105674 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a600106a0 con 0x7f1a68102230 2024-04-21T23:06:23.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.349+0000 7f1a6ff3a640 1 -- 172.21.15.44:0/2769105674 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a6819ee40 con 0x7f1a68102230 2024-04-21T23:06:23.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.350+0000 7f1a6ff3a640 1 -- 172.21.15.44:0/2769105674 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a681079f0 con 0x7f1a68102230 2024-04-21T23:06:23.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.351+0000 7f1a5effd640 1 -- 172.21.15.44:0/2769105674 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1a60003970 con 0x7f1a68102230 2024-04-21T23:06:23.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.352+0000 7f1a5effd640 1 --2- 172.21.15.44:0/2769105674 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f1a540a2b90 0x7f1a540a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:23.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.352+0000 7f1a6dcd6640 1 --2- 172.21.15.44:0/2769105674 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f1a540a2b90 0x7f1a540a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:23.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.352+0000 7f1a5effd640 1 -- 172.21.15.44:0/2769105674 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f1a600cefd0 con 0x7f1a68102230 2024-04-21T23:06:23.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.353+0000 7f1a6dcd6640 1 --2- 172.21.15.44:0/2769105674 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f1a540a2b90 0x7f1a540a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1a58009e00 tx=0x7f1a58009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.357+0000 7f1a5effd640 1 -- 172.21.15.44:0/2769105674 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1a6009c3d0 con 0x7f1a68102230 2024-04-21T23:06:23.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:23 smithi044 ceph-mon[27934]: mds.? [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] up:boot 2024-04-21T23:06:23.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:23 smithi044 ceph-mon[27934]: fsmap 12 up:standby 2024-04-21T23:06:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:06:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:23 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3048695338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:06:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:23 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1230829171' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:23 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/438059808' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:06:23.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:23.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:23 smithi069 ceph-mon[31816]: mds.? [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] up:boot 2024-04-21T23:06:23.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:23 smithi069 ceph-mon[31816]: fsmap 12 up:standby 2024-04-21T23:06:23.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:06:23.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:23.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:23.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:23.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:23 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3048695338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:23.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:23.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:23.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:06:23.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:23 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1230829171' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:23.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:23 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/438059808' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:06:23.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:23.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:23 smithi055 ceph-mon[31117]: mds.? [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] up:boot 2024-04-21T23:06:23.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:23 smithi055 ceph-mon[31117]: fsmap 12 up:standby 2024-04-21T23:06:23.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:06:23.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:23.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:23.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:23.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:23 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3048695338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:23.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:23.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:23.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:06:23.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:23 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1230829171' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:23.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:23 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/438059808' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:06:23.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:23.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:23.545+0000 7f1a6ff3a640 1 -- 172.21.15.44:0/2769105674 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f1a6802d370 con 0x7f1a68102230 2024-04-21T23:06:24.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.185+0000 7f1a5effd640 1 -- 172.21.15.44:0/2769105674 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7f1a6009bcd0 con 0x7f1a68102230 2024-04-21T23:06:24.185 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' created 2024-04-21T23:06:24.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.187+0000 7f1a6ff3a640 1 -- 172.21.15.44:0/2769105674 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f1a540a2b90 msgr2=0x7f1a540a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:24.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.187+0000 7f1a6ff3a640 1 --2- 172.21.15.44:0/2769105674 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f1a540a2b90 0x7f1a540a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1a58009e00 tx=0x7f1a58009190 comp rx=0 tx=0).stop 2024-04-21T23:06:24.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.187+0000 7f1a6ff3a640 1 -- 172.21.15.44:0/2769105674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a68102230 msgr2=0x7f1a6819e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:24.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.187+0000 7f1a6ff3a640 1 --2- 172.21.15.44:0/2769105674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a68102230 0x7f1a6819e010 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1a6000da10 tx=0x7f1a6000ded0 comp rx=0 tx=0).stop 2024-04-21T23:06:24.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.187+0000 7f1a6ff3a640 1 -- 172.21.15.44:0/2769105674 shutdown_connections 2024-04-21T23:06:24.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.187+0000 7f1a6ff3a640 1 --2- 172.21.15.44:0/2769105674 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f1a540a2b90 0x7f1a540a5050 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:24.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.187+0000 7f1a6ff3a640 1 --2- 172.21.15.44:0/2769105674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1a68102230 0x7f1a6819e010 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:24.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.187+0000 7f1a6ff3a640 1 --2- 172.21.15.44:0/2769105674 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1a68101870 0x7f1a68196fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:24.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.187+0000 7f1a6ff3a640 1 --2- 172.21.15.44:0/2769105674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a68100680 0x7f1a68196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:24.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.187+0000 7f1a6ff3a640 1 -- 172.21.15.44:0/2769105674 >> 172.21.15.44:0/2769105674 conn(0x7f1a680fbc10 msgr2=0x7f1a68104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:24.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.188+0000 7f1a6ff3a640 1 -- 172.21.15.44:0/2769105674 shutdown_connections 2024-04-21T23:06:24.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.188+0000 7f1a6ff3a640 1 -- 172.21.15.44:0/2769105674 wait complete. 2024-04-21T23:06:24.204 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-21T23:06:24.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.322+0000 7fbaf922e640 1 -- 172.21.15.44:0/2450180658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaf4101860 msgr2=0x7fbaf4101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:24.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.322+0000 7fbaf922e640 1 --2- 172.21.15.44:0/2450180658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaf4101860 0x7fbaf4101ce0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbae8009900 tx=0x7fbae802bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:24.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.322+0000 7fbaf922e640 1 -- 172.21.15.44:0/2450180658 shutdown_connections 2024-04-21T23:06:24.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.322+0000 7fbaf922e640 1 --2- 172.21.15.44:0/2450180658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbaf4102220 0x7fbaf4108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:24.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.322+0000 7fbaf922e640 1 --2- 172.21.15.44:0/2450180658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaf4101860 0x7fbaf4101ce0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:24.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.322+0000 7fbaf922e640 1 --2- 172.21.15.44:0/2450180658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaf4100670 0x7fbaf4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:24.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.322+0000 7fbaf922e640 1 -- 172.21.15.44:0/2450180658 >> 172.21.15.44:0/2450180658 conn(0x7fbaf40fbbc0 msgr2=0x7fbaf40fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:24.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.322+0000 7fbaf922e640 1 -- 172.21.15.44:0/2450180658 shutdown_connections 2024-04-21T23:06:24.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.322+0000 7fbaf922e640 1 -- 172.21.15.44:0/2450180658 wait complete. 2024-04-21T23:06:24.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.323+0000 7fbaf922e640 1 Processor -- start 2024-04-21T23:06:24.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.323+0000 7fbaf922e640 1 -- start start 2024-04-21T23:06:24.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.323+0000 7fbaf922e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaf4100670 0x7fbaf406a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:24.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.323+0000 7fbaf922e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaf4101860 0x7fbaf419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:24.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.323+0000 7fbaf922e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbaf4102220 0x7fbaf419ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:24.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.323+0000 7fbaf922e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf410c340 con 0x7fbaf4101860 2024-04-21T23:06:24.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.323+0000 7fbaf922e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf410c1c0 con 0x7fbaf4102220 2024-04-21T23:06:24.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.323+0000 7fbaf922e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf410c4c0 con 0x7fbaf4100670 2024-04-21T23:06:24.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.323+0000 7fbaf259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaf4101860 0x7fbaf419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:24.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.323+0000 7fbaf2d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaf4100670 0x7fbaf406a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:24.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.323+0000 7fbaf259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaf4101860 0x7fbaf419c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36212/0 (socket says 172.21.15.44:36212) 2024-04-21T23:06:24.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.323+0000 7fbaf259c640 1 -- 172.21.15.44:0/981109371 learned_addr learned my addr 172.21.15.44:0/981109371 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:24.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.323+0000 7fbaf359e640 1 --2- 172.21.15.44:0/981109371 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbaf4102220 0x7fbaf419ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:24.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.323+0000 7fbaf259c640 1 -- 172.21.15.44:0/981109371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaf4100670 msgr2=0x7fbaf406a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:24.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.323+0000 7fbaf259c640 1 --2- 172.21.15.44:0/981109371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaf4100670 0x7fbaf406a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:24.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.323+0000 7fbaf259c640 1 -- 172.21.15.44:0/981109371 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbaf4102220 msgr2=0x7fbaf419ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:24.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.323+0000 7fbaf259c640 1 --2- 172.21.15.44:0/981109371 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbaf4102220 0x7fbaf419ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:24.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.323+0000 7fbaf259c640 1 -- 172.21.15.44:0/981109371 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaf419f740 con 0x7fbaf4101860 2024-04-21T23:06:24.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.323+0000 7fbaf2d9d640 1 --2- 172.21.15.44:0/981109371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaf4100670 0x7fbaf406a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:06:24.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.323+0000 7fbaf359e640 1 --2- 172.21.15.44:0/981109371 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbaf4102220 0x7fbaf419ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:06:24.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.324+0000 7fbaf259c640 1 --2- 172.21.15.44:0/981109371 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaf4101860 0x7fbaf419c840 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbae8005ab0 tx=0x7fbae802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:24.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.324+0000 7fbadbfff640 1 -- 172.21.15.44:0/981109371 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae8043020 con 0x7fbaf4101860 2024-04-21T23:06:24.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.324+0000 7fbaf922e640 1 -- 172.21.15.44:0/981109371 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaf419f9d0 con 0x7fbaf4101860 2024-04-21T23:06:24.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.324+0000 7fbadbfff640 1 -- 172.21.15.44:0/981109371 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbae80054e0 con 0x7fbaf4101860 2024-04-21T23:06:24.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.324+0000 7fbadbfff640 1 -- 172.21.15.44:0/981109371 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae8003ce0 con 0x7fbaf4101860 2024-04-21T23:06:24.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.324+0000 7fbaf922e640 1 -- 172.21.15.44:0/981109371 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaf419ff20 con 0x7fbaf4101860 2024-04-21T23:06:24.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.325+0000 7fbaf922e640 1 -- 172.21.15.44:0/981109371 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaf41078c0 con 0x7fbaf4101860 2024-04-21T23:06:24.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.326+0000 7fbadbfff640 1 -- 172.21.15.44:0/981109371 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbae8046020 con 0x7fbaf4101860 2024-04-21T23:06:24.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.327+0000 7fbadbfff640 1 --2- 172.21.15.44:0/981109371 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbadc0a2b90 0x7fbadc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:24.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.327+0000 7fbaf2d9d640 1 --2- 172.21.15.44:0/981109371 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbadc0a2b90 0x7fbadc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:24.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.327+0000 7fbadbfff640 1 -- 172.21.15.44:0/981109371 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7497+0+0 (secure 0 0 0) 0x7fbae80f0e80 con 0x7fbaf4101860 2024-04-21T23:06:24.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.327+0000 7fbaf2d9d640 1 --2- 172.21.15.44:0/981109371 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbadc0a2b90 0x7fbadc0a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbaf41016c0 tx=0x7fbae4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:24.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.331+0000 7fbadbfff640 1 -- 172.21.15.44:0/981109371 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fbae804d020 con 0x7fbaf4101860 2024-04-21T23:06:24.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:24 smithi044 ceph-mon[27934]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:24.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:24 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3176856378' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T23:06:24.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:24 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2769105674' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T23:06:24.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:24 smithi055 ceph-mon[31117]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:24.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:24 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3176856378' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T23:06:24.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:24 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2769105674' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T23:06:24.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:24 smithi069 ceph-mon[31816]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:24.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:24 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3176856378' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T23:06:24.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:24 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2769105674' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T23:06:24.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:24.516+0000 7fbaf922e640 1 -- 172.21.15.44:0/981109371 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fbaf402d170 con 0x7fbaf4101860 2024-04-21T23:06:25.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.189+0000 7fbadbfff640 1 -- 172.21.15.44:0/981109371 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v84) ==== 171+0+0 (secure 0 0 0) 0x7fbae80be110 con 0x7fbaf4101860 2024-04-21T23:06:25.189 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' created 2024-04-21T23:06:25.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.192+0000 7fbaf922e640 1 -- 172.21.15.44:0/981109371 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbadc0a2b90 msgr2=0x7fbadc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:25.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.192+0000 7fbaf922e640 1 --2- 172.21.15.44:0/981109371 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbadc0a2b90 0x7fbadc0a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbaf41016c0 tx=0x7fbae4005f90 comp rx=0 tx=0).stop 2024-04-21T23:06:25.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.192+0000 7fbaf922e640 1 -- 172.21.15.44:0/981109371 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaf4101860 msgr2=0x7fbaf419c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:25.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.192+0000 7fbaf922e640 1 --2- 172.21.15.44:0/981109371 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaf4101860 0x7fbaf419c840 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbae8005ab0 tx=0x7fbae802c6c0 comp rx=0 tx=0).stop 2024-04-21T23:06:25.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.193+0000 7fbaf922e640 1 -- 172.21.15.44:0/981109371 shutdown_connections 2024-04-21T23:06:25.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.193+0000 7fbaf922e640 1 --2- 172.21.15.44:0/981109371 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbadc0a2b90 0x7fbadc0a5050 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:25.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.193+0000 7fbaf922e640 1 --2- 172.21.15.44:0/981109371 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbaf4102220 0x7fbaf419ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:25.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.193+0000 7fbaf922e640 1 --2- 172.21.15.44:0/981109371 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbaf4101860 0x7fbaf419c840 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:25.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.193+0000 7fbaf922e640 1 --2- 172.21.15.44:0/981109371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaf4100670 0x7fbaf406a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:25.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.193+0000 7fbaf922e640 1 -- 172.21.15.44:0/981109371 >> 172.21.15.44:0/981109371 conn(0x7fbaf40fbbc0 msgr2=0x7fbaf4104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:25.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.196+0000 7fbaf922e640 1 -- 172.21.15.44:0/981109371 shutdown_connections 2024-04-21T23:06:25.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.197+0000 7fbaf922e640 1 -- 172.21.15.44:0/981109371 wait complete. 2024-04-21T23:06:25.219 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-21T23:06:25.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.325+0000 7f5c658d4640 1 -- 172.21.15.44:0/766758952 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c60102230 msgr2=0x7f5c60108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:25.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.325+0000 7f5c658d4640 1 --2- 172.21.15.44:0/766758952 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c60102230 0x7f5c60108ac0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5c58009ca0 tx=0x7f5c5802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:25.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.325+0000 7f5c658d4640 1 -- 172.21.15.44:0/766758952 shutdown_connections 2024-04-21T23:06:25.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.325+0000 7f5c658d4640 1 --2- 172.21.15.44:0/766758952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c60101870 0x7f5c60101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:25.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.325+0000 7f5c658d4640 1 --2- 172.21.15.44:0/766758952 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c60102230 0x7f5c60108ac0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:25.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.325+0000 7f5c658d4640 1 --2- 172.21.15.44:0/766758952 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5c60100680 0x7f5c60100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:25.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.325+0000 7f5c658d4640 1 -- 172.21.15.44:0/766758952 >> 172.21.15.44:0/766758952 conn(0x7f5c600fbc10 msgr2=0x7f5c600fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:25.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.325+0000 7f5c658d4640 1 -- 172.21.15.44:0/766758952 shutdown_connections 2024-04-21T23:06:25.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.325+0000 7f5c658d4640 1 -- 172.21.15.44:0/766758952 wait complete. 2024-04-21T23:06:25.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.326+0000 7f5c658d4640 1 Processor -- start 2024-04-21T23:06:25.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.326+0000 7f5c658d4640 1 -- start start 2024-04-21T23:06:25.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.326+0000 7f5c658d4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c60100680 0x7f5c60104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:25.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.326+0000 7f5c658d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c60101870 0x7f5c60102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:25.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.326+0000 7f5c658d4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5c60102230 0x7f5c60103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:25.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.326+0000 7f5c658d4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6010c2e0 con 0x7f5c60100680 2024-04-21T23:06:25.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.326+0000 7f5c658d4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6010c160 con 0x7f5c60102230 2024-04-21T23:06:25.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.326+0000 7f5c658d4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6010c460 con 0x7f5c60101870 2024-04-21T23:06:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.326+0000 7f5c5effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c60100680 0x7f5c60104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.326+0000 7f5c5e7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c60101870 0x7f5c60102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.326+0000 7f5c5effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c60100680 0x7f5c60104610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36234/0 (socket says 172.21.15.44:36234) 2024-04-21T23:06:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.326+0000 7f5c5f7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5c60102230 0x7f5c60103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.326+0000 7f5c5effd640 1 -- 172.21.15.44:0/3088627137 learned_addr learned my addr 172.21.15.44:0/3088627137 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.327+0000 7f5c5effd640 1 -- 172.21.15.44:0/3088627137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c60101870 msgr2=0x7f5c60102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.327+0000 7f5c5effd640 1 --2- 172.21.15.44:0/3088627137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c60101870 0x7f5c60102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.327+0000 7f5c5effd640 1 -- 172.21.15.44:0/3088627137 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5c60102230 msgr2=0x7f5c60103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.327+0000 7f5c5effd640 1 --2- 172.21.15.44:0/3088627137 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5c60102230 0x7f5c60103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.327+0000 7f5c5effd640 1 -- 172.21.15.44:0/3088627137 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c60103a20 con 0x7f5c60100680 2024-04-21T23:06:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.327+0000 7f5c5f7fe640 1 --2- 172.21.15.44:0/3088627137 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5c60102230 0x7f5c60103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:06:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.327+0000 7f5c5effd640 1 --2- 172.21.15.44:0/3088627137 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c60100680 0x7f5c60104610 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5c5000d950 tx=0x7f5c5000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.327+0000 7f5c648d2640 1 -- 172.21.15.44:0/3088627137 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c50017020 con 0x7f5c60100680 2024-04-21T23:06:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.327+0000 7f5c648d2640 1 -- 172.21.15.44:0/3088627137 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c50009c30 con 0x7f5c60100680 2024-04-21T23:06:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.327+0000 7f5c658d4640 1 -- 172.21.15.44:0/3088627137 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c601a1b90 con 0x7f5c60100680 2024-04-21T23:06:25.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.327+0000 7f5c648d2640 1 -- 172.21.15.44:0/3088627137 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5000f720 con 0x7f5c60100680 2024-04-21T23:06:25.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.327+0000 7f5c658d4640 1 -- 172.21.15.44:0/3088627137 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c601a2130 con 0x7f5c60100680 2024-04-21T23:06:25.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.328+0000 7f5c658d4640 1 -- 172.21.15.44:0/3088627137 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c601a2300 con 0x7f5c60100680 2024-04-21T23:06:25.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.330+0000 7f5c648d2640 1 -- 172.21.15.44:0/3088627137 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5c5001a020 con 0x7f5c60100680 2024-04-21T23:06:25.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.331+0000 7f5c648d2640 1 --2- 172.21.15.44:0/3088627137 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5c480a2c10 0x7f5c480a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:25.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.331+0000 7f5c5e7fc640 1 --2- 172.21.15.44:0/3088627137 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5c480a2c10 0x7f5c480a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:25.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.331+0000 7f5c648d2640 1 -- 172.21.15.44:0/3088627137 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7879+0+0 (secure 0 0 0) 0x7f5c500cf190 con 0x7f5c60100680 2024-04-21T23:06:25.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.332+0000 7f5c5e7fc640 1 --2- 172.21.15.44:0/3088627137 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5c480a2c10 0x7f5c480a50d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5c60104380 tx=0x7f5c5400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:25.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.334+0000 7f5c648d2640 1 -- 172.21.15.44:0/3088627137 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5c50021020 con 0x7f5c60100680 2024-04-21T23:06:25.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:25 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2769105674' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T23:06:25.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:25 smithi055 ceph-mon[31117]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T23:06:25.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:25 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:25 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:25 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:25 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:25 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:25 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:25 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/981109371' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T23:06:25.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:25 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:25.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:25 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:06:25.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:25 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:25 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:06:25.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:25 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2769105674' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T23:06:25.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:25 smithi069 ceph-mon[31816]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T23:06:25.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:25 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:25 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:25 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:25 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:25 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:25 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:25 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/981109371' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T23:06:25.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:25 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:25.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:25 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:06:25.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:25 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:25 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:06:25.560 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:25.560+0000 7f5c658d4640 1 -- 172.21.15.44:0/3088627137 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f5c601a24d0 con 0x7f5c60100680 2024-04-21T23:06:25.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:25 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2769105674' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T23:06:25.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:25 smithi044 ceph-mon[27934]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T23:06:25.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:25 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:25 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:25 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:25 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:25 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:25 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:25 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/981109371' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T23:06:25.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:25 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:25.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:25 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:06:25.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:25 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:25.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:25 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:06:26.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.205+0000 7f5c648d2640 1 -- 172.21.15.44:0/3088627137 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f5c50009de0 con 0x7f5c60100680 2024-04-21T23:06:26.205 INFO:teuthology.orchestra.run.smithi044.stderr:new fs with metadata pool 2 and data pool 3 2024-04-21T23:06:26.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.208+0000 7f5c658d4640 1 -- 172.21.15.44:0/3088627137 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5c480a2c10 msgr2=0x7f5c480a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:26.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.208+0000 7f5c658d4640 1 --2- 172.21.15.44:0/3088627137 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5c480a2c10 0x7f5c480a50d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5c60104380 tx=0x7f5c5400b390 comp rx=0 tx=0).stop 2024-04-21T23:06:26.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.208+0000 7f5c658d4640 1 -- 172.21.15.44:0/3088627137 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c60100680 msgr2=0x7f5c60104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:26.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.208+0000 7f5c658d4640 1 --2- 172.21.15.44:0/3088627137 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c60100680 0x7f5c60104610 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5c5000d950 tx=0x7f5c5000de10 comp rx=0 tx=0).stop 2024-04-21T23:06:26.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.208+0000 7f5c658d4640 1 -- 172.21.15.44:0/3088627137 shutdown_connections 2024-04-21T23:06:26.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.208+0000 7f5c658d4640 1 --2- 172.21.15.44:0/3088627137 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5c480a2c10 0x7f5c480a50d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:26.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.208+0000 7f5c658d4640 1 --2- 172.21.15.44:0/3088627137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c60101870 0x7f5c60102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:26.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.208+0000 7f5c658d4640 1 --2- 172.21.15.44:0/3088627137 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5c60102230 0x7f5c60103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:26.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.208+0000 7f5c658d4640 1 --2- 172.21.15.44:0/3088627137 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5c60100680 0x7f5c60104610 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:26.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.208+0000 7f5c658d4640 1 -- 172.21.15.44:0/3088627137 >> 172.21.15.44:0/3088627137 conn(0x7f5c600fbc10 msgr2=0x7f5c600fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:26.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.208+0000 7f5c658d4640 1 -- 172.21.15.44:0/3088627137 shutdown_connections 2024-04-21T23:06:26.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.208+0000 7f5c658d4640 1 -- 172.21.15.44:0/3088627137 wait complete. 2024-04-21T23:06:26.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:26 smithi055 ceph-mon[31117]: pgmap v284: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:26.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:26 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/981109371' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T23:06:26.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:26 smithi055 ceph-mon[31117]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T23:06:26.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:26 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3088627137' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T23:06:26.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:26 smithi055 ceph-mon[31117]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T23:06:26.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:26 smithi055 ceph-mon[31117]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:06:26.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:26 smithi055 ceph-mon[31117]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T23:06:26.222 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-21T23:06:26.222 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-21T23:06:26.243 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:26 smithi044 ceph-mon[27934]: pgmap v284: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:26.243 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:26 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/981109371' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T23:06:26.243 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:26 smithi044 ceph-mon[27934]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T23:06:26.243 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:26 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3088627137' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T23:06:26.243 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:26 smithi044 ceph-mon[27934]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T23:06:26.244 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:26 smithi044 ceph-mon[27934]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:06:26.244 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:26 smithi044 ceph-mon[27934]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T23:06:26.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.329+0000 7fcd8219a640 1 -- 172.21.15.44:0/1386093296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd7c107440 msgr2=0x7fcd7c109890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:26.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.329+0000 7fcd8219a640 1 --2- 172.21.15.44:0/1386093296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd7c107440 0x7fcd7c109890 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcd74009ca0 tx=0x7fcd7402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:26.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.329+0000 7fcd8219a640 1 -- 172.21.15.44:0/1386093296 shutdown_connections 2024-04-21T23:06:26.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.329+0000 7fcd8219a640 1 --2- 172.21.15.44:0/1386093296 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd7c107440 0x7fcd7c109890 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:26.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.329+0000 7fcd8219a640 1 --2- 172.21.15.44:0/1386093296 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcd7c104b10 0x7fcd7c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:26.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.329+0000 7fcd8219a640 1 --2- 172.21.15.44:0/1386093296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd7c1021e0 0x7fcd7c1045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:26.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.329+0000 7fcd8219a640 1 -- 172.21.15.44:0/1386093296 >> 172.21.15.44:0/1386093296 conn(0x7fcd7c0fbbb0 msgr2=0x7fcd7c0fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:26.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.329+0000 7fcd8219a640 1 -- 172.21.15.44:0/1386093296 shutdown_connections 2024-04-21T23:06:26.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.329+0000 7fcd8219a640 1 -- 172.21.15.44:0/1386093296 wait complete. 2024-04-21T23:06:26.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.330+0000 7fcd8219a640 1 Processor -- start 2024-04-21T23:06:26.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.330+0000 7fcd8219a640 1 -- start start 2024-04-21T23:06:26.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.330+0000 7fcd8219a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd7c1021e0 0x7fcd7c197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:26.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.330+0000 7fcd8219a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcd7c104b10 0x7fcd7c197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:26.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.330+0000 7fcd8219a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd7c107440 0x7fcd7c19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:26.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.330+0000 7fcd8219a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7c10d3e0 con 0x7fcd7c107440 2024-04-21T23:06:26.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.330+0000 7fcd8219a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7c10d260 con 0x7fcd7c104b10 2024-04-21T23:06:26.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.330+0000 7fcd8219a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7c10d560 con 0x7fcd7c1021e0 2024-04-21T23:06:26.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.330+0000 7fcd7bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd7c107440 0x7fcd7c19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:26.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.330+0000 7fcd7bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd7c107440 0x7fcd7c19ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36252/0 (socket says 172.21.15.44:36252) 2024-04-21T23:06:26.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.330+0000 7fcd7bfff640 1 -- 172.21.15.44:0/4209931967 learned_addr learned my addr 172.21.15.44:0/4209931967 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:26.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.330+0000 7fcd7b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd7c1021e0 0x7fcd7c197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:26.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.330+0000 7fcd7affd640 1 --2- 172.21.15.44:0/4209931967 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcd7c104b10 0x7fcd7c197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:26.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.331+0000 7fcd7bfff640 1 -- 172.21.15.44:0/4209931967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd7c1021e0 msgr2=0x7fcd7c197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:26.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.331+0000 7fcd7bfff640 1 --2- 172.21.15.44:0/4209931967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd7c1021e0 0x7fcd7c197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:26.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.331+0000 7fcd7bfff640 1 -- 172.21.15.44:0/4209931967 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcd7c104b10 msgr2=0x7fcd7c197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:26.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.331+0000 7fcd7bfff640 1 --2- 172.21.15.44:0/4209931967 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcd7c104b10 0x7fcd7c197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:26.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.331+0000 7fcd7bfff640 1 -- 172.21.15.44:0/4209931967 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd7c19f580 con 0x7fcd7c107440 2024-04-21T23:06:26.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.331+0000 7fcd7bfff640 1 --2- 172.21.15.44:0/4209931967 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd7c107440 0x7fcd7c19ef10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcd7402c630 tx=0x7fcd74032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:26.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.331+0000 7fcd78ff9640 1 -- 172.21.15.44:0/4209931967 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd74041020 con 0x7fcd7c107440 2024-04-21T23:06:26.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.331+0000 7fcd78ff9640 1 -- 172.21.15.44:0/4209931967 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd74034cd0 con 0x7fcd7c107440 2024-04-21T23:06:26.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.331+0000 7fcd8219a640 1 -- 172.21.15.44:0/4209931967 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd7c19f810 con 0x7fcd7c107440 2024-04-21T23:06:26.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.331+0000 7fcd78ff9640 1 -- 172.21.15.44:0/4209931967 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd74039040 con 0x7fcd7c107440 2024-04-21T23:06:26.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.331+0000 7fcd8219a640 1 -- 172.21.15.44:0/4209931967 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd7c19fd50 con 0x7fcd7c107440 2024-04-21T23:06:26.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.332+0000 7fcd8219a640 1 -- 172.21.15.44:0/4209931967 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd7c0fd6a0 con 0x7fcd7c107440 2024-04-21T23:06:26.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.333+0000 7fcd78ff9640 1 -- 172.21.15.44:0/4209931967 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fcd74044020 con 0x7fcd7c107440 2024-04-21T23:06:26.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.334+0000 7fcd78ff9640 1 --2- 172.21.15.44:0/4209931967 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcd640a2c10 0x7fcd640a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:26.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.334+0000 7fcd7b7fe640 1 --2- 172.21.15.44:0/4209931967 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcd640a2c10 0x7fcd640a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:26.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.334+0000 7fcd78ff9640 1 -- 172.21.15.44:0/4209931967 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fcd740f0d50 con 0x7fcd7c107440 2024-04-21T23:06:26.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.334+0000 7fcd7b7fe640 1 --2- 172.21.15.44:0/4209931967 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcd640a2c10 0x7fcd640a50d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c009bd0 tx=0x7fcd6c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:26.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.338+0000 7fcd78ff9640 1 -- 172.21.15.44:0/4209931967 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fcd740f4030 con 0x7fcd7c107440 2024-04-21T23:06:26.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:26 smithi069 ceph-mon[31816]: pgmap v284: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:06:26.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:26 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/981109371' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T23:06:26.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:26 smithi069 ceph-mon[31816]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T23:06:26.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:26 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3088627137' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T23:06:26.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:26 smithi069 ceph-mon[31816]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T23:06:26.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:26 smithi069 ceph-mon[31816]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:06:26.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:26 smithi069 ceph-mon[31816]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T23:06:26.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.554+0000 7fcd8219a640 1 -- 172.21.15.44:0/4209931967 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fcd7c02d170 con 0x7fcd7c107440 2024-04-21T23:06:26.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.962+0000 7fcd78ff9640 1 -- 172.21.15.44:0/4209931967 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v86) ==== 157+0+0 (secure 0 0 0) 0x7fcd7404b020 con 0x7fcd7c107440 2024-04-21T23:06:26.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.964+0000 7fcd8219a640 1 -- 172.21.15.44:0/4209931967 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcd640a2c10 msgr2=0x7fcd640a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:26.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.964+0000 7fcd8219a640 1 --2- 172.21.15.44:0/4209931967 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcd640a2c10 0x7fcd640a50d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c009bd0 tx=0x7fcd6c005f90 comp rx=0 tx=0).stop 2024-04-21T23:06:26.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.964+0000 7fcd8219a640 1 -- 172.21.15.44:0/4209931967 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd7c107440 msgr2=0x7fcd7c19ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:26.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.964+0000 7fcd8219a640 1 --2- 172.21.15.44:0/4209931967 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd7c107440 0x7fcd7c19ef10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcd7402c630 tx=0x7fcd74032ea0 comp rx=0 tx=0).stop 2024-04-21T23:06:26.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.964+0000 7fcd8219a640 1 -- 172.21.15.44:0/4209931967 shutdown_connections 2024-04-21T23:06:26.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.964+0000 7fcd8219a640 1 --2- 172.21.15.44:0/4209931967 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcd640a2c10 0x7fcd640a50d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:26.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.964+0000 7fcd8219a640 1 --2- 172.21.15.44:0/4209931967 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcd7c107440 0x7fcd7c19ef10 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:26.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.964+0000 7fcd8219a640 1 --2- 172.21.15.44:0/4209931967 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcd7c104b10 0x7fcd7c197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:26.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.964+0000 7fcd8219a640 1 --2- 172.21.15.44:0/4209931967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd7c1021e0 0x7fcd7c197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:26.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.964+0000 7fcd8219a640 1 -- 172.21.15.44:0/4209931967 >> 172.21.15.44:0/4209931967 conn(0x7fcd7c0fbbb0 msgr2=0x7fcd7c102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:26.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.964+0000 7fcd8219a640 1 -- 172.21.15.44:0/4209931967 shutdown_connections 2024-04-21T23:06:26.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:26.964+0000 7fcd8219a640 1 -- 172.21.15.44:0/4209931967 wait complete. 2024-04-21T23:06:26.976 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-21T23:06:27.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.133+0000 7fc37e42e640 1 -- 172.21.15.44:0/4016408045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc378107490 msgr2=0x7fc3781098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:27.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.133+0000 7fc37e42e640 1 --2- 172.21.15.44:0/4016408045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc378107490 0x7fc3781098e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc370009ca0 tx=0x7fc37002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:27.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.133+0000 7fc37e42e640 1 -- 172.21.15.44:0/4016408045 shutdown_connections 2024-04-21T23:06:27.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.133+0000 7fc37e42e640 1 --2- 172.21.15.44:0/4016408045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc378104b60 0x7fc378106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:27.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.133+0000 7fc37e42e640 1 --2- 172.21.15.44:0/4016408045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc378107490 0x7fc3781098e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:27.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.133+0000 7fc37e42e640 1 --2- 172.21.15.44:0/4016408045 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc378102230 0x7fc378104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:27.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.133+0000 7fc37e42e640 1 -- 172.21.15.44:0/4016408045 >> 172.21.15.44:0/4016408045 conn(0x7fc3780fbc20 msgr2=0x7fc3780fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:27.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.133+0000 7fc37e42e640 1 -- 172.21.15.44:0/4016408045 shutdown_connections 2024-04-21T23:06:27.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.133+0000 7fc37e42e640 1 -- 172.21.15.44:0/4016408045 wait complete. 2024-04-21T23:06:27.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.134+0000 7fc37e42e640 1 Processor -- start 2024-04-21T23:06:27.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.134+0000 7fc37e42e640 1 -- start start 2024-04-21T23:06:27.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.134+0000 7fc37e42e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc378102230 0x7fc378197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:27.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.134+0000 7fc37e42e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc378104b60 0x7fc378197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:27.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.134+0000 7fc37e42e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc378107490 0x7fc37819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.134+0000 7fc37e42e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc37810d450 con 0x7fc378107490 2024-04-21T23:06:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.134+0000 7fc37e42e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc37810d2d0 con 0x7fc378104b60 2024-04-21T23:06:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.134+0000 7fc37e42e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc37810d5d0 con 0x7fc378102230 2024-04-21T23:06:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.134+0000 7fc37c9cb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc378107490 0x7fc37819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.134+0000 7fc37c9cb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc378107490 0x7fc37819efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36274/0 (socket says 172.21.15.44:36274) 2024-04-21T23:06:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.134+0000 7fc37c9cb640 1 -- 172.21.15.44:0/1234521091 learned_addr learned my addr 172.21.15.44:0/1234521091 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.134+0000 7fc3777fe640 1 --2- 172.21.15.44:0/1234521091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc378104b60 0x7fc378197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.134+0000 7fc377fff640 1 --2- 172.21.15.44:0/1234521091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc378102230 0x7fc378197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.134+0000 7fc37c9cb640 1 -- 172.21.15.44:0/1234521091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc378102230 msgr2=0x7fc378197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.134+0000 7fc37c9cb640 1 --2- 172.21.15.44:0/1234521091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc378102230 0x7fc378197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.134+0000 7fc37c9cb640 1 -- 172.21.15.44:0/1234521091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc378104b60 msgr2=0x7fc378197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.134+0000 7fc37c9cb640 1 --2- 172.21.15.44:0/1234521091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc378104b60 0x7fc378197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:27.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.135+0000 7fc37c9cb640 1 -- 172.21.15.44:0/1234521091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc37819f650 con 0x7fc378107490 2024-04-21T23:06:27.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.135+0000 7fc37c9cb640 1 --2- 172.21.15.44:0/1234521091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc378107490 0x7fc37819efe0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc370037040 tx=0x7fc370032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:27.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.135+0000 7fc3757fa640 1 -- 172.21.15.44:0/1234521091 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc370041020 con 0x7fc378107490 2024-04-21T23:06:27.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.135+0000 7fc37e42e640 1 -- 172.21.15.44:0/1234521091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc37819f8e0 con 0x7fc378107490 2024-04-21T23:06:27.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.135+0000 7fc3757fa640 1 -- 172.21.15.44:0/1234521091 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc370034cd0 con 0x7fc378107490 2024-04-21T23:06:27.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.135+0000 7fc3757fa640 1 -- 172.21.15.44:0/1234521091 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc370039040 con 0x7fc378107490 2024-04-21T23:06:27.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.135+0000 7fc37e42e640 1 -- 172.21.15.44:0/1234521091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc37819fe20 con 0x7fc378107490 2024-04-21T23:06:27.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.136+0000 7fc37e42e640 1 -- 172.21.15.44:0/1234521091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3780fd710 con 0x7fc378107490 2024-04-21T23:06:27.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.137+0000 7fc3757fa640 1 -- 172.21.15.44:0/1234521091 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc370044020 con 0x7fc378107490 2024-04-21T23:06:27.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.138+0000 7fc3757fa640 1 --2- 172.21.15.44:0/1234521091 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc3600a2b90 0x7fc3600a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:27.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.138+0000 7fc3757fa640 1 -- 172.21.15.44:0/1234521091 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8277+0+0 (secure 0 0 0) 0x7fc3700f1280 con 0x7fc378107490 2024-04-21T23:06:27.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.138+0000 7fc377fff640 1 --2- 172.21.15.44:0/1234521091 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc3600a2b90 0x7fc3600a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:27.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.139+0000 7fc377fff640 1 --2- 172.21.15.44:0/1234521091 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc3600a2b90 0x7fc3600a5050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc368009bd0 tx=0x7fc368005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:27.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.143+0000 7fc3757fa640 1 -- 172.21.15.44:0/1234521091 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc37004b020 con 0x7fc378107490 2024-04-21T23:06:27.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:27.328+0000 7fc37e42e640 1 -- 172.21.15.44:0/1234521091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc37802d170 con 0x7fc378107490 2024-04-21T23:06:27.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:27 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3088627137' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T23:06:27.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:27 smithi044 ceph-mon[27934]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T23:06:27.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:27 smithi044 ceph-mon[27934]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T23:06:27.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:27 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/4209931967' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T23:06:27.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:27.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:06:27.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:27 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/4209931967' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T23:06:27.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:27 smithi044 ceph-mon[27934]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T23:06:27.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:27 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3088627137' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T23:06:27.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:27 smithi055 ceph-mon[31117]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T23:06:27.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:27 smithi055 ceph-mon[31117]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T23:06:27.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:27 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/4209931967' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T23:06:27.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:27.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:06:27.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:27 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/4209931967' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T23:06:27.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:27 smithi055 ceph-mon[31117]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T23:06:27.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:27 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3088627137' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T23:06:27.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:27 smithi069 ceph-mon[31816]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T23:06:27.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:27 smithi069 ceph-mon[31816]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T23:06:27.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:27 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/4209931967' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T23:06:27.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:06:27.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:06:27.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:06:27.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:27 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/4209931967' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T23:06:27.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:27 smithi069 ceph-mon[31816]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T23:06:28.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:28 smithi044 ceph-mon[27934]: pgmap v288: 97 pgs: 36 active+clean, 3 creating+peering, 58 unknown; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 6 op/s 2024-04-21T23:06:28.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:28 smithi044 ceph-mon[27934]: mds.? [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] up:active 2024-04-21T23:06:28.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:28 smithi044 ceph-mon[27934]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:06:28.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:28 smithi044 ceph-mon[27934]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:06:28.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:28 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1234521091' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T23:06:28.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:28 smithi044 ceph-mon[27934]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T23:06:28.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:28 smithi044 ceph-mon[27934]: Cluster is now healthy 2024-04-21T23:06:28.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:28 smithi069 ceph-mon[31816]: pgmap v288: 97 pgs: 36 active+clean, 3 creating+peering, 58 unknown; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 6 op/s 2024-04-21T23:06:28.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:28 smithi069 ceph-mon[31816]: mds.? [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] up:active 2024-04-21T23:06:28.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:28 smithi069 ceph-mon[31816]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:06:28.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:28 smithi069 ceph-mon[31816]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:06:28.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:28 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1234521091' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T23:06:28.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:28 smithi069 ceph-mon[31816]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T23:06:28.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:28 smithi069 ceph-mon[31816]: Cluster is now healthy 2024-04-21T23:06:28.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:28 smithi055 ceph-mon[31117]: pgmap v288: 97 pgs: 36 active+clean, 3 creating+peering, 58 unknown; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 6 op/s 2024-04-21T23:06:28.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:28 smithi055 ceph-mon[31117]: mds.? [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] up:active 2024-04-21T23:06:28.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:28 smithi055 ceph-mon[31117]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:06:28.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:28 smithi055 ceph-mon[31117]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:06:28.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:28 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1234521091' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T23:06:28.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:28 smithi055 ceph-mon[31117]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T23:06:28.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:28 smithi055 ceph-mon[31117]: Cluster is now healthy 2024-04-21T23:06:29.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.240+0000 7fc3757fa640 1 -- 172.21.15.44:0/1234521091 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v88) ==== 228+0+0 (secure 0 0 0) 0x7fc3700be240 con 0x7fc378107490 2024-04-21T23:06:29.240 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data_ec' created 2024-04-21T23:06:29.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.244+0000 7fc37e42e640 1 -- 172.21.15.44:0/1234521091 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc3600a2b90 msgr2=0x7fc3600a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:29.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.244+0000 7fc37e42e640 1 --2- 172.21.15.44:0/1234521091 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc3600a2b90 0x7fc3600a5050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc368009bd0 tx=0x7fc368005f90 comp rx=0 tx=0).stop 2024-04-21T23:06:29.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.244+0000 7fc37e42e640 1 -- 172.21.15.44:0/1234521091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc378107490 msgr2=0x7fc37819efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:29.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.244+0000 7fc37e42e640 1 --2- 172.21.15.44:0/1234521091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc378107490 0x7fc37819efe0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc370037040 tx=0x7fc370032f40 comp rx=0 tx=0).stop 2024-04-21T23:06:29.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.244+0000 7fc37e42e640 1 -- 172.21.15.44:0/1234521091 shutdown_connections 2024-04-21T23:06:29.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.244+0000 7fc37e42e640 1 --2- 172.21.15.44:0/1234521091 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc3600a2b90 0x7fc3600a5050 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:29.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.244+0000 7fc37e42e640 1 --2- 172.21.15.44:0/1234521091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc378107490 0x7fc37819efe0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:29.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.244+0000 7fc37e42e640 1 --2- 172.21.15.44:0/1234521091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc378104b60 0x7fc378197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:29.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.244+0000 7fc37e42e640 1 --2- 172.21.15.44:0/1234521091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc378102230 0x7fc378197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:29.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.244+0000 7fc37e42e640 1 -- 172.21.15.44:0/1234521091 >> 172.21.15.44:0/1234521091 conn(0x7fc3780fbc20 msgr2=0x7fc378102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:29.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.244+0000 7fc37e42e640 1 -- 172.21.15.44:0/1234521091 shutdown_connections 2024-04-21T23:06:29.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.244+0000 7fc37e42e640 1 -- 172.21.15.44:0/1234521091 wait complete. 2024-04-21T23:06:29.263 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-21T23:06:29.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.375+0000 7f193d325640 1 -- 172.21.15.44:0/409294886 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19381022c0 msgr2=0x7f19381046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:29.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.375+0000 7f193d325640 1 --2- 172.21.15.44:0/409294886 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19381022c0 0x7f19381046b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1928009900 tx=0x7f192802bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:29.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.375+0000 7f193d325640 1 -- 172.21.15.44:0/409294886 shutdown_connections 2024-04-21T23:06:29.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.376+0000 7f193d325640 1 --2- 172.21.15.44:0/409294886 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1938107520 0x7f1938109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:29.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.376+0000 7f193d325640 1 --2- 172.21.15.44:0/409294886 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1938104bf0 0x7f1938106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:29.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.376+0000 7f193d325640 1 --2- 172.21.15.44:0/409294886 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19381022c0 0x7f19381046b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:29.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.376+0000 7f193d325640 1 -- 172.21.15.44:0/409294886 >> 172.21.15.44:0/409294886 conn(0x7f19380fbc10 msgr2=0x7f19380fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.376+0000 7f193d325640 1 -- 172.21.15.44:0/409294886 shutdown_connections 2024-04-21T23:06:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.376+0000 7f193d325640 1 -- 172.21.15.44:0/409294886 wait complete. 2024-04-21T23:06:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.376+0000 7f193d325640 1 Processor -- start 2024-04-21T23:06:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.376+0000 7f193d325640 1 -- start start 2024-04-21T23:06:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.376+0000 7f193d325640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19381022c0 0x7f1938101c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.376+0000 7f193d325640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1938104bf0 0x7f1938100270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.376+0000 7f193d325640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1938107520 0x7f19381007b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.376+0000 7f193d325640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f193810d480 con 0x7f19381022c0 2024-04-21T23:06:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.376+0000 7f193d325640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f193810d300 con 0x7f1938104bf0 2024-04-21T23:06:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.376+0000 7f193d325640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f193810d600 con 0x7f1938107520 2024-04-21T23:06:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.377+0000 7f1936ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19381022c0 0x7f1938101c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.377+0000 7f1936ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19381022c0 0x7f1938101c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36282/0 (socket says 172.21.15.44:36282) 2024-04-21T23:06:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.377+0000 7f1936ffd640 1 -- 172.21.15.44:0/3072582638 learned_addr learned my addr 172.21.15.44:0/3072582638 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.377+0000 7f19377fe640 1 --2- 172.21.15.44:0/3072582638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1938107520 0x7f19381007b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:29.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.377+0000 7f19367fc640 1 --2- 172.21.15.44:0/3072582638 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1938104bf0 0x7f1938100270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.377+0000 7f1936ffd640 1 -- 172.21.15.44:0/3072582638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1938107520 msgr2=0x7f19381007b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.377+0000 7f1936ffd640 1 --2- 172.21.15.44:0/3072582638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1938107520 0x7f19381007b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.377+0000 7f1936ffd640 1 -- 172.21.15.44:0/3072582638 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1938104bf0 msgr2=0x7f1938100270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.377+0000 7f1936ffd640 1 --2- 172.21.15.44:0/3072582638 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1938104bf0 0x7f1938100270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.377+0000 7f1936ffd640 1 -- 172.21.15.44:0/3072582638 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1938101040 con 0x7f19381022c0 2024-04-21T23:06:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.377+0000 7f19367fc640 1 --2- 172.21.15.44:0/3072582638 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1938104bf0 0x7f1938100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:06:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.377+0000 7f1936ffd640 1 --2- 172.21.15.44:0/3072582638 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19381022c0 0x7f1938101c30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1928000c00 tx=0x7f1928004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.377+0000 7f191ffff640 1 -- 172.21.15.44:0/3072582638 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1928043020 con 0x7f19381022c0 2024-04-21T23:06:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.377+0000 7f193d325640 1 -- 172.21.15.44:0/3072582638 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19381a1bc0 con 0x7f19381022c0 2024-04-21T23:06:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.377+0000 7f191ffff640 1 -- 172.21.15.44:0/3072582638 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1928004440 con 0x7f19381022c0 2024-04-21T23:06:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.377+0000 7f191ffff640 1 -- 172.21.15.44:0/3072582638 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192803e450 con 0x7f19381022c0 2024-04-21T23:06:29.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.377+0000 7f193d325640 1 -- 172.21.15.44:0/3072582638 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19381a2160 con 0x7f19381022c0 2024-04-21T23:06:29.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.378+0000 7f193d325640 1 -- 172.21.15.44:0/3072582638 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19380fd700 con 0x7f19381022c0 2024-04-21T23:06:29.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.380+0000 7f191ffff640 1 -- 172.21.15.44:0/3072582638 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1928046020 con 0x7f19381022c0 2024-04-21T23:06:29.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.380+0000 7f191ffff640 1 --2- 172.21.15.44:0/3072582638 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19200a2b90 0x7f19200a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:29.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.380+0000 7f191ffff640 1 -- 172.21.15.44:0/3072582638 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f19280f19d0 con 0x7f19381022c0 2024-04-21T23:06:29.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.380+0000 7f19367fc640 1 --2- 172.21.15.44:0/3072582638 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19200a2b90 0x7f19200a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:29.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.381+0000 7f19367fc640 1 --2- 172.21.15.44:0/3072582638 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19200a2b90 0x7f19200a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f19381019a0 tx=0x7f192c005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:29.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.384+0000 7f191ffff640 1 -- 172.21.15.44:0/3072582638 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f192804d020 con 0x7f19381022c0 2024-04-21T23:06:29.571 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:29.572+0000 7f193d325640 1 -- 172.21.15.44:0/3072582638 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f193802d170 con 0x7f19381022c0 2024-04-21T23:06:29.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:29 smithi044 ceph-mon[27934]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T23:06:29.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:29 smithi055 ceph-mon[31117]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T23:06:29.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:29 smithi069 ceph-mon[31816]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T23:06:30.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.242+0000 7f191ffff640 1 -- 172.21.15.44:0/3072582638 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v89) ==== 169+0+0 (secure 0 0 0) 0x7f19280be710 con 0x7f19381022c0 2024-04-21T23:06:30.242 INFO:teuthology.orchestra.run.smithi044.stderr:set pool 4 allow_ec_overwrites to true 2024-04-21T23:06:30.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.246+0000 7f193d325640 1 -- 172.21.15.44:0/3072582638 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19200a2b90 msgr2=0x7f19200a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:30.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.246+0000 7f193d325640 1 --2- 172.21.15.44:0/3072582638 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19200a2b90 0x7f19200a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f19381019a0 tx=0x7f192c005d90 comp rx=0 tx=0).stop 2024-04-21T23:06:30.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.246+0000 7f193d325640 1 -- 172.21.15.44:0/3072582638 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19381022c0 msgr2=0x7f1938101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:30.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.246+0000 7f193d325640 1 --2- 172.21.15.44:0/3072582638 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19381022c0 0x7f1938101c30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1928000c00 tx=0x7f1928004040 comp rx=0 tx=0).stop 2024-04-21T23:06:30.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.246+0000 7f193d325640 1 -- 172.21.15.44:0/3072582638 shutdown_connections 2024-04-21T23:06:30.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.246+0000 7f193d325640 1 --2- 172.21.15.44:0/3072582638 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19200a2b90 0x7f19200a5050 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:30.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.246+0000 7f193d325640 1 --2- 172.21.15.44:0/3072582638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1938107520 0x7f19381007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:30.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.246+0000 7f193d325640 1 --2- 172.21.15.44:0/3072582638 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1938104bf0 0x7f1938100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:30.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.246+0000 7f193d325640 1 --2- 172.21.15.44:0/3072582638 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f19381022c0 0x7f1938101c30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:30.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.246+0000 7f193d325640 1 -- 172.21.15.44:0/3072582638 >> 172.21.15.44:0/3072582638 conn(0x7f19380fbc10 msgr2=0x7f1938102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:30.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.246+0000 7f193d325640 1 -- 172.21.15.44:0/3072582638 shutdown_connections 2024-04-21T23:06:30.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.246+0000 7f193d325640 1 -- 172.21.15.44:0/3072582638 wait complete. 2024-04-21T23:06:30.269 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-21T23:06:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.380+0000 7f1934208640 1 -- 172.21.15.44:0/124987394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f192c103350 msgr2=0x7f192c109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.380+0000 7f1934208640 1 --2- 172.21.15.44:0/124987394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f192c103350 0x7f192c109be0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1924009ca0 tx=0x7f192402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.380+0000 7f1934208640 1 -- 172.21.15.44:0/124987394 shutdown_connections 2024-04-21T23:06:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.380+0000 7f1934208640 1 --2- 172.21.15.44:0/124987394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f192c103350 0x7f192c109be0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.380+0000 7f1934208640 1 --2- 172.21.15.44:0/124987394 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f192c102990 0x7f192c102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.380+0000 7f1934208640 1 --2- 172.21.15.44:0/124987394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f192c1017a0 0x7f192c101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.380+0000 7f1934208640 1 -- 172.21.15.44:0/124987394 >> 172.21.15.44:0/124987394 conn(0x7f192c0fcd10 msgr2=0x7f192c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.380+0000 7f1934208640 1 -- 172.21.15.44:0/124987394 shutdown_connections 2024-04-21T23:06:30.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.380+0000 7f1934208640 1 -- 172.21.15.44:0/124987394 wait complete. 2024-04-21T23:06:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.380+0000 7f1934208640 1 Processor -- start 2024-04-21T23:06:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.380+0000 7f1934208640 1 -- start start 2024-04-21T23:06:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.380+0000 7f1934208640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f192c1017a0 0x7f192c197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.380+0000 7f1934208640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f192c102990 0x7f192c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.380+0000 7f1934208640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f192c103350 0x7f192c19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.380+0000 7f1934208640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f192c10d400 con 0x7f192c102990 2024-04-21T23:06:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.380+0000 7f1934208640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f192c10d280 con 0x7f192c1017a0 2024-04-21T23:06:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.381+0000 7f1934208640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f192c10d580 con 0x7f192c103350 2024-04-21T23:06:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.381+0000 7f19317a3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f192c102990 0x7f192c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.381+0000 7f19317a3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f192c102990 0x7f192c197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36296/0 (socket says 172.21.15.44:36296) 2024-04-21T23:06:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.381+0000 7f19317a3640 1 -- 172.21.15.44:0/1197510634 learned_addr learned my addr 172.21.15.44:0/1197510634 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.381+0000 7f1931fa4640 1 --2- 172.21.15.44:0/1197510634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f192c1017a0 0x7f192c197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.381+0000 7f19327a5640 1 --2- 172.21.15.44:0/1197510634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f192c103350 0x7f192c19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:30.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.381+0000 7f19317a3640 1 -- 172.21.15.44:0/1197510634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f192c103350 msgr2=0x7f192c19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:30.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.381+0000 7f19317a3640 1 --2- 172.21.15.44:0/1197510634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f192c103350 0x7f192c19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:30.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.381+0000 7f19317a3640 1 -- 172.21.15.44:0/1197510634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f192c1017a0 msgr2=0x7f192c197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:30.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.381+0000 7f19317a3640 1 --2- 172.21.15.44:0/1197510634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f192c1017a0 0x7f192c197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:30.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.381+0000 7f19317a3640 1 -- 172.21.15.44:0/1197510634 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f192c19f660 con 0x7f192c102990 2024-04-21T23:06:30.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.381+0000 7f19317a3640 1 --2- 172.21.15.44:0/1197510634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f192c102990 0x7f192c197f80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f191c00d950 tx=0x7f191c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:30.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.381+0000 7f1922ffd640 1 -- 172.21.15.44:0/1197510634 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c017070 con 0x7f192c102990 2024-04-21T23:06:30.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.381+0000 7f1922ffd640 1 -- 172.21.15.44:0/1197510634 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f191c009c30 con 0x7f192c102990 2024-04-21T23:06:30.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.381+0000 7f1934208640 1 -- 172.21.15.44:0/1197510634 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f192c19f950 con 0x7f192c102990 2024-04-21T23:06:30.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.381+0000 7f1922ffd640 1 -- 172.21.15.44:0/1197510634 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c00f6b0 con 0x7f192c102990 2024-04-21T23:06:30.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.381+0000 7f1934208640 1 -- 172.21.15.44:0/1197510634 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f192c19fef0 con 0x7f192c102990 2024-04-21T23:06:30.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.382+0000 7f1934208640 1 -- 172.21.15.44:0/1197510634 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f192c068a40 con 0x7f192c102990 2024-04-21T23:06:30.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.384+0000 7f1922ffd640 1 -- 172.21.15.44:0/1197510634 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f191c0109d0 con 0x7f192c102990 2024-04-21T23:06:30.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.384+0000 7f1922ffd640 1 --2- 172.21.15.44:0/1197510634 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19180a2c10 0x7f19180a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:30.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.384+0000 7f1931fa4640 1 --2- 172.21.15.44:0/1197510634 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19180a2c10 0x7f19180a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:30.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.384+0000 7f1922ffd640 1 -- 172.21.15.44:0/1197510634 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8770+0+0 (secure 0 0 0) 0x7f191c0ced50 con 0x7f192c102990 2024-04-21T23:06:30.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.385+0000 7f1931fa4640 1 --2- 172.21.15.44:0/1197510634 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19180a2c10 0x7f19180a50d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1928005fd0 tx=0x7f192800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:30.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.388+0000 7f1922ffd640 1 -- 172.21.15.44:0/1197510634 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f191c009de0 con 0x7f192c102990 2024-04-21T23:06:30.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:30.617+0000 7f1934208640 1 -- 172.21.15.44:0/1197510634 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f192c108a60 con 0x7f192c102990 2024-04-21T23:06:30.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:30 smithi044 ceph-mon[27934]: pgmap v290: 97 pgs: 83 active+clean, 3 creating+peering, 11 unknown; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-04-21T23:06:30.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:30 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1234521091' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T23:06:30.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:30 smithi044 ceph-mon[27934]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T23:06:30.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:30 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3072582638' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T23:06:30.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:30 smithi055 ceph-mon[31117]: pgmap v290: 97 pgs: 83 active+clean, 3 creating+peering, 11 unknown; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-04-21T23:06:30.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:30 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1234521091' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T23:06:30.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:30 smithi055 ceph-mon[31117]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T23:06:30.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:30 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3072582638' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T23:06:30.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:30 smithi069 ceph-mon[31816]: pgmap v290: 97 pgs: 83 active+clean, 3 creating+peering, 11 unknown; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-04-21T23:06:30.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:30 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1234521091' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T23:06:30.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:30 smithi069 ceph-mon[31816]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T23:06:30.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:30 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3072582638' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T23:06:31.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.265+0000 7f1922ffd640 1 -- 172.21.15.44:0/1197510634 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f191c09bb10 con 0x7f192c102990 2024-04-21T23:06:31.265 INFO:teuthology.orchestra.run.smithi044.stderr:added data pool 4 to fsmap 2024-04-21T23:06:31.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.268+0000 7f1934208640 1 -- 172.21.15.44:0/1197510634 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19180a2c10 msgr2=0x7f19180a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:31.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.268+0000 7f1934208640 1 --2- 172.21.15.44:0/1197510634 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19180a2c10 0x7f19180a50d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1928005fd0 tx=0x7f192800b410 comp rx=0 tx=0).stop 2024-04-21T23:06:31.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.268+0000 7f1934208640 1 -- 172.21.15.44:0/1197510634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f192c102990 msgr2=0x7f192c197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:31.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.268+0000 7f1934208640 1 --2- 172.21.15.44:0/1197510634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f192c102990 0x7f192c197f80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f191c00d950 tx=0x7f191c00de10 comp rx=0 tx=0).stop 2024-04-21T23:06:31.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.268+0000 7f1934208640 1 -- 172.21.15.44:0/1197510634 shutdown_connections 2024-04-21T23:06:31.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.268+0000 7f1934208640 1 --2- 172.21.15.44:0/1197510634 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19180a2c10 0x7f19180a50d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.268+0000 7f1934208640 1 --2- 172.21.15.44:0/1197510634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f192c103350 0x7f192c19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.268+0000 7f1934208640 1 --2- 172.21.15.44:0/1197510634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f192c102990 0x7f192c197f80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.268+0000 7f1934208640 1 --2- 172.21.15.44:0/1197510634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f192c1017a0 0x7f192c197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.268+0000 7f1934208640 1 -- 172.21.15.44:0/1197510634 >> 172.21.15.44:0/1197510634 conn(0x7f192c0fcd10 msgr2=0x7f192c105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:31.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.269+0000 7f1934208640 1 -- 172.21.15.44:0/1197510634 shutdown_connections 2024-04-21T23:06:31.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.269+0000 7f1934208640 1 -- 172.21.15.44:0/1197510634 wait complete. 2024-04-21T23:06:31.282 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:06:31.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.391+0000 7fa2c2dc9640 1 -- 172.21.15.44:0/269996582 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc1017a0 msgr2=0x7fa2bc101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:31.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.391+0000 7fa2c2dc9640 1 --2- 172.21.15.44:0/269996582 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc1017a0 0x7fa2bc101bc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0009900 tx=0x7fa2b002bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:31.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.391+0000 7fa2c2dc9640 1 -- 172.21.15.44:0/269996582 shutdown_connections 2024-04-21T23:06:31.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.391+0000 7fa2c2dc9640 1 --2- 172.21.15.44:0/269996582 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2bc103350 0x7fa2bc109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.391+0000 7fa2c2dc9640 1 --2- 172.21.15.44:0/269996582 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2bc102990 0x7fa2bc102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.391+0000 7fa2c2dc9640 1 --2- 172.21.15.44:0/269996582 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc1017a0 0x7fa2bc101bc0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.391+0000 7fa2c2dc9640 1 -- 172.21.15.44:0/269996582 >> 172.21.15.44:0/269996582 conn(0x7fa2bc0fcd30 msgr2=0x7fa2bc0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:31.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.391+0000 7fa2c2dc9640 1 -- 172.21.15.44:0/269996582 shutdown_connections 2024-04-21T23:06:31.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.391+0000 7fa2c2dc9640 1 -- 172.21.15.44:0/269996582 wait complete. 2024-04-21T23:06:31.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.391+0000 7fa2c2dc9640 1 Processor -- start 2024-04-21T23:06:31.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.391+0000 7fa2c2dc9640 1 -- start start 2024-04-21T23:06:31.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2c2dc9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc1017a0 0x7fa2bc197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:31.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2c2dc9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2bc102990 0x7fa2bc197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:31.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2c2dc9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2bc103350 0x7fa2bc19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:31.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2c2dc9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc10d420 con 0x7fa2bc1017a0 2024-04-21T23:06:31.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2c2dc9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc10d2a0 con 0x7fa2bc103350 2024-04-21T23:06:31.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2c2dc9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa2bc10d5a0 con 0x7fa2bc102990 2024-04-21T23:06:31.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2c0b65640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc1017a0 0x7fa2bc197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:31.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2c0b65640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc1017a0 0x7fa2bc197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41402/0 (socket says 172.21.15.44:41402) 2024-04-21T23:06:31.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2c0b65640 1 -- 172.21.15.44:0/1669914396 learned_addr learned my addr 172.21.15.44:0/1669914396 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:31.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2bbfff640 1 --2- 172.21.15.44:0/1669914396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2bc102990 0x7fa2bc197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:31.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2c1366640 1 --2- 172.21.15.44:0/1669914396 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2bc103350 0x7fa2bc19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:31.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2c0b65640 1 -- 172.21.15.44:0/1669914396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2bc102990 msgr2=0x7fa2bc197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:31.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2c0b65640 1 --2- 172.21.15.44:0/1669914396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2bc102990 0x7fa2bc197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2c0b65640 1 -- 172.21.15.44:0/1669914396 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2bc103350 msgr2=0x7fa2bc19f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:31.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2c0b65640 1 --2- 172.21.15.44:0/1669914396 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2bc103350 0x7fa2bc19f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2c0b65640 1 -- 172.21.15.44:0/1669914396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2bc19f6c0 con 0x7fa2bc1017a0 2024-04-21T23:06:31.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2bbfff640 1 --2- 172.21.15.44:0/1669914396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2bc102990 0x7fa2bc197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:06:31.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2c0b65640 1 --2- 172.21.15.44:0/1669914396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc1017a0 0x7fa2bc197aa0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0000c00 tx=0x7fa2b0005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:31.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2b9ffb640 1 -- 172.21.15.44:0/1669914396 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0041070 con 0x7fa2bc1017a0 2024-04-21T23:06:31.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2b9ffb640 1 -- 172.21.15.44:0/1669914396 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2b00044d0 con 0x7fa2bc1017a0 2024-04-21T23:06:31.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2c2dc9640 1 -- 172.21.15.44:0/1669914396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2bc19f950 con 0x7fa2bc1017a0 2024-04-21T23:06:31.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.392+0000 7fa2b9ffb640 1 -- 172.21.15.44:0/1669914396 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0034440 con 0x7fa2bc1017a0 2024-04-21T23:06:31.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.393+0000 7fa2c2dc9640 1 -- 172.21.15.44:0/1669914396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2bc19fe90 con 0x7fa2bc1017a0 2024-04-21T23:06:31.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.394+0000 7fa2c2dc9640 1 -- 172.21.15.44:0/1669914396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2bc108ac0 con 0x7fa2bc1017a0 2024-04-21T23:06:31.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.395+0000 7fa2b9ffb640 1 -- 172.21.15.44:0/1669914396 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa2b0044070 con 0x7fa2bc1017a0 2024-04-21T23:06:31.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.395+0000 7fa2b9ffb640 1 --2- 172.21.15.44:0/1669914396 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa2ac0a2c10 0x7fa2ac0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:31.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.396+0000 7fa2bbfff640 1 --2- 172.21.15.44:0/1669914396 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa2ac0a2c10 0x7fa2ac0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:31.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.396+0000 7fa2b9ffb640 1 -- 172.21.15.44:0/1669914396 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fa2b00f0eb0 con 0x7fa2bc1017a0 2024-04-21T23:06:31.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.396+0000 7fa2bbfff640 1 --2- 172.21.15.44:0/1669914396 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa2ac0a2c10 0x7fa2ac0a50d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc198fb0 tx=0x7fa2a8005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:31.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.400+0000 7fa2b9ffb640 1 -- 172.21.15.44:0/1669914396 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa2b00bdb70 con 0x7fa2bc1017a0 2024-04-21T23:06:31.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.624+0000 7fa2c2dc9640 1 -- 172.21.15.44:0/1669914396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2bc02d170 con 0x7fa2bc1017a0 2024-04-21T23:06:31.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.625+0000 7fa2b9ffb640 1 -- 172.21.15.44:0/1669914396 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11782 (secure 0 0 0) 0x7fa2b00bd470 con 0x7fa2bc1017a0 2024-04-21T23:06:31.624 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:31.625 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24526,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/3743898853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3743898853},{"type":"v1","addr":"172.21.15.69:6839","nonce":3743898853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:06:30.619342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:06:31.625 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 17 2024-04-21T23:06:31.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.627+0000 7fa2c2dc9640 1 -- 172.21.15.44:0/1669914396 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa2ac0a2c10 msgr2=0x7fa2ac0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:31.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.627+0000 7fa2c2dc9640 1 --2- 172.21.15.44:0/1669914396 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa2ac0a2c10 0x7fa2ac0a50d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc198fb0 tx=0x7fa2a8005ec0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.627+0000 7fa2c2dc9640 1 -- 172.21.15.44:0/1669914396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc1017a0 msgr2=0x7fa2bc197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:31.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.627+0000 7fa2c2dc9640 1 --2- 172.21.15.44:0/1669914396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc1017a0 0x7fa2bc197aa0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0000c00 tx=0x7fa2b0005d50 comp rx=0 tx=0).stop 2024-04-21T23:06:31.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.627+0000 7fa2c2dc9640 1 -- 172.21.15.44:0/1669914396 shutdown_connections 2024-04-21T23:06:31.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.627+0000 7fa2c2dc9640 1 --2- 172.21.15.44:0/1669914396 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa2ac0a2c10 0x7fa2ac0a50d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.627+0000 7fa2c2dc9640 1 --2- 172.21.15.44:0/1669914396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2bc102990 0x7fa2bc197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.627+0000 7fa2c2dc9640 1 --2- 172.21.15.44:0/1669914396 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2bc103350 0x7fa2bc19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.627+0000 7fa2c2dc9640 1 --2- 172.21.15.44:0/1669914396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa2bc1017a0 0x7fa2bc197aa0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.627+0000 7fa2c2dc9640 1 -- 172.21.15.44:0/1669914396 >> 172.21.15.44:0/1669914396 conn(0x7fa2bc0fcd30 msgr2=0x7fa2bc105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:31.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.627+0000 7fa2c2dc9640 1 -- 172.21.15.44:0/1669914396 shutdown_connections 2024-04-21T23:06:31.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.627+0000 7fa2c2dc9640 1 -- 172.21.15.44:0/1669914396 wait complete. 2024-04-21T23:06:31.636 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:31 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3072582638' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T23:06:31.636 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:31 smithi044 ceph-mon[27934]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T23:06:31.636 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:31 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1197510634' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T23:06:31.638 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:06:31.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:31 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3072582638' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T23:06:31.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:31 smithi069 ceph-mon[31816]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T23:06:31.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:31 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1197510634' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T23:06:31.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:31 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3072582638' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T23:06:31.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:31 smithi055 ceph-mon[31117]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T23:06:31.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:31 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1197510634' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T23:06:31.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.739+0000 7f5e8cf97640 1 -- 172.21.15.44:0/2102141100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e88107540 msgr2=0x7f5e88109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:31.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.739+0000 7f5e8cf97640 1 --2- 172.21.15.44:0/2102141100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e88107540 0x7f5e88109930 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5e78009ca0 tx=0x7f5e7802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.739+0000 7f5e8cf97640 1 -- 172.21.15.44:0/2102141100 shutdown_connections 2024-04-21T23:06:31.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.739+0000 7f5e8cf97640 1 --2- 172.21.15.44:0/2102141100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e88104c10 0x7f5e88107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.739+0000 7f5e8cf97640 1 --2- 172.21.15.44:0/2102141100 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e88107540 0x7f5e88109930 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.739+0000 7f5e8cf97640 1 --2- 172.21.15.44:0/2102141100 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5e881022e0 0x7f5e881046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.739+0000 7f5e8cf97640 1 -- 172.21.15.44:0/2102141100 >> 172.21.15.44:0/2102141100 conn(0x7f5e880fbc70 msgr2=0x7f5e880fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:31.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.739+0000 7f5e8cf97640 1 -- 172.21.15.44:0/2102141100 shutdown_connections 2024-04-21T23:06:31.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.739+0000 7f5e8cf97640 1 -- 172.21.15.44:0/2102141100 wait complete. 2024-04-21T23:06:31.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.740+0000 7f5e8cf97640 1 Processor -- start 2024-04-21T23:06:31.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.740+0000 7f5e8cf97640 1 -- start start 2024-04-21T23:06:31.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.740+0000 7f5e8cf97640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5e881022e0 0x7f5e881979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:31.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.740+0000 7f5e8cf97640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e88104c10 0x7f5e88197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:31.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.740+0000 7f5e8cf97640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e88107540 0x7f5e8819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:31.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.740+0000 7f5e8cf97640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8810d430 con 0x7f5e88107540 2024-04-21T23:06:31.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.740+0000 7f5e8cf97640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8810d2b0 con 0x7f5e881022e0 2024-04-21T23:06:31.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.740+0000 7f5e8cf97640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8810d5b0 con 0x7f5e88104c10 2024-04-21T23:06:31.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.740+0000 7f5e86d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e88107540 0x7f5e8819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:31.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.740+0000 7f5e86d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e88107540 0x7f5e8819ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41412/0 (socket says 172.21.15.44:41412) 2024-04-21T23:06:31.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.740+0000 7f5e86d9d640 1 -- 172.21.15.44:0/1928667091 learned_addr learned my addr 172.21.15.44:0/1928667091 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:31.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.740+0000 7f5e8659c640 1 --2- 172.21.15.44:0/1928667091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5e881022e0 0x7f5e881979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:31.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.741+0000 7f5e85d9b640 1 --2- 172.21.15.44:0/1928667091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e88104c10 0x7f5e88197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:31.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.741+0000 7f5e86d9d640 1 -- 172.21.15.44:0/1928667091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e88104c10 msgr2=0x7f5e88197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:31.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.741+0000 7f5e86d9d640 1 --2- 172.21.15.44:0/1928667091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e88104c10 0x7f5e88197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.741+0000 7f5e86d9d640 1 -- 172.21.15.44:0/1928667091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5e881022e0 msgr2=0x7f5e881979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:31.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.741+0000 7f5e86d9d640 1 --2- 172.21.15.44:0/1928667091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5e881022e0 0x7f5e881979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.741+0000 7f5e86d9d640 1 -- 172.21.15.44:0/1928667091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e8819f600 con 0x7f5e88107540 2024-04-21T23:06:31.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.741+0000 7f5e86d9d640 1 --2- 172.21.15.44:0/1928667091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e88107540 0x7f5e8819ef90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5e78037040 tx=0x7f5e78032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:31.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.741+0000 7f5e7f7fe640 1 -- 172.21.15.44:0/1928667091 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e78041020 con 0x7f5e88107540 2024-04-21T23:06:31.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.741+0000 7f5e7f7fe640 1 -- 172.21.15.44:0/1928667091 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e78034cd0 con 0x7f5e88107540 2024-04-21T23:06:31.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.741+0000 7f5e8cf97640 1 -- 172.21.15.44:0/1928667091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e8819f890 con 0x7f5e88107540 2024-04-21T23:06:31.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.741+0000 7f5e7f7fe640 1 -- 172.21.15.44:0/1928667091 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e78039040 con 0x7f5e88107540 2024-04-21T23:06:31.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.741+0000 7f5e8cf97640 1 -- 172.21.15.44:0/1928667091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e8819fdd0 con 0x7f5e88107540 2024-04-21T23:06:31.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.742+0000 7f5e8cf97640 1 -- 172.21.15.44:0/1928667091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e880fd760 con 0x7f5e88107540 2024-04-21T23:06:31.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.743+0000 7f5e7f7fe640 1 -- 172.21.15.44:0/1928667091 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5e78044020 con 0x7f5e88107540 2024-04-21T23:06:31.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.744+0000 7f5e7f7fe640 1 --2- 172.21.15.44:0/1928667091 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5e740a2c10 0x7f5e740a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:31.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.744+0000 7f5e8659c640 1 --2- 172.21.15.44:0/1928667091 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5e740a2c10 0x7f5e740a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:31.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.744+0000 7f5e7f7fe640 1 -- 172.21.15.44:0/1928667091 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f5e780f15b0 con 0x7f5e88107540 2024-04-21T23:06:31.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.744+0000 7f5e8659c640 1 --2- 172.21.15.44:0/1928667091 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5e740a2c10 0x7f5e740a50d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5e8000abe0 tx=0x7f5e8000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:31.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.747+0000 7f5e7f7fe640 1 -- 172.21.15.44:0/1928667091 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5e7804b020 con 0x7f5e88107540 2024-04-21T23:06:31.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.931+0000 7f5e8cf97640 1 -- 172.21.15.44:0/1928667091 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5e8802d370 con 0x7f5e88107540 2024-04-21T23:06:31.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.933+0000 7f5e7f7fe640 1 -- 172.21.15.44:0/1928667091 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24032 (secure 0 0 0) 0x7f5e7804e350 con 0x7f5e88107540 2024-04-21T23:06:31.933 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:31.933 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":90,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:06:30.619155+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:06:31.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.935+0000 7f5e8cf97640 1 -- 172.21.15.44:0/1928667091 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5e740a2c10 msgr2=0x7f5e740a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:31.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.935+0000 7f5e8cf97640 1 --2- 172.21.15.44:0/1928667091 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5e740a2c10 0x7f5e740a50d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5e8000abe0 tx=0x7f5e8000a2d0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.935+0000 7f5e8cf97640 1 -- 172.21.15.44:0/1928667091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e88107540 msgr2=0x7f5e8819ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:31.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.935+0000 7f5e8cf97640 1 --2- 172.21.15.44:0/1928667091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e88107540 0x7f5e8819ef90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5e78037040 tx=0x7f5e78032f20 comp rx=0 tx=0).stop 2024-04-21T23:06:31.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.935+0000 7f5e8cf97640 1 -- 172.21.15.44:0/1928667091 shutdown_connections 2024-04-21T23:06:31.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.935+0000 7f5e8cf97640 1 --2- 172.21.15.44:0/1928667091 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5e740a2c10 0x7f5e740a50d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.935+0000 7f5e8cf97640 1 --2- 172.21.15.44:0/1928667091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e88104c10 0x7f5e88197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.935+0000 7f5e8cf97640 1 --2- 172.21.15.44:0/1928667091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5e88107540 0x7f5e8819ef90 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.935+0000 7f5e8cf97640 1 --2- 172.21.15.44:0/1928667091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5e881022e0 0x7f5e881979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:31.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.935+0000 7f5e8cf97640 1 -- 172.21.15.44:0/1928667091 >> 172.21.15.44:0/1928667091 conn(0x7f5e880fbc70 msgr2=0x7f5e88102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:31.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.935+0000 7f5e8cf97640 1 -- 172.21.15.44:0/1928667091 shutdown_connections 2024-04-21T23:06:31.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:31.935+0000 7f5e8cf97640 1 -- 172.21.15.44:0/1928667091 wait complete. 2024-04-21T23:06:31.947 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:06:32.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.049+0000 7fdc0e597640 1 -- 172.21.15.44:0/2076871675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc081017a0 msgr2=0x7fdc08101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:32.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.049+0000 7fdc0e597640 1 --2- 172.21.15.44:0/2076871675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc081017a0 0x7fdc08101bc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009900 tx=0x7fdbfc02bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:32.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.049+0000 7fdc0e597640 1 -- 172.21.15.44:0/2076871675 shutdown_connections 2024-04-21T23:06:32.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.049+0000 7fdc0e597640 1 --2- 172.21.15.44:0/2076871675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc08102990 0x7fdc08102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.049+0000 7fdc0e597640 1 --2- 172.21.15.44:0/2076871675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdc08103350 0x7fdc08109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.049+0000 7fdc0e597640 1 --2- 172.21.15.44:0/2076871675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc081017a0 0x7fdc08101bc0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.049+0000 7fdc0e597640 1 -- 172.21.15.44:0/2076871675 >> 172.21.15.44:0/2076871675 conn(0x7fdc080fcd30 msgr2=0x7fdc080ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:32.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.049+0000 7fdc0e597640 1 -- 172.21.15.44:0/2076871675 shutdown_connections 2024-04-21T23:06:32.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.049+0000 7fdc0e597640 1 -- 172.21.15.44:0/2076871675 wait complete. 2024-04-21T23:06:32.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.050+0000 7fdc0e597640 1 Processor -- start 2024-04-21T23:06:32.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.050+0000 7fdc0e597640 1 -- start start 2024-04-21T23:06:32.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.050+0000 7fdc0e597640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdc081017a0 0x7fdc08197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:32.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.050+0000 7fdc0e597640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc08102990 0x7fdc08197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:32.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.050+0000 7fdc0e597640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc08103350 0x7fdc0819f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:32.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.050+0000 7fdc0e597640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0810d470 con 0x7fdc08102990 2024-04-21T23:06:32.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.050+0000 7fdc0e597640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0810d2f0 con 0x7fdc081017a0 2024-04-21T23:06:32.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.050+0000 7fdc0e597640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0810d5f0 con 0x7fdc08103350 2024-04-21T23:06:32.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.050+0000 7fdc077fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc08102990 0x7fdc08197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:32.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.050+0000 7fdc077fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc08102990 0x7fdc08197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41438/0 (socket says 172.21.15.44:41438) 2024-04-21T23:06:32.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.050+0000 7fdc077fe640 1 -- 172.21.15.44:0/42872138 learned_addr learned my addr 172.21.15.44:0/42872138 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:32.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.050+0000 7fdc07fff640 1 --2- 172.21.15.44:0/42872138 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdc081017a0 0x7fdc08197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:32.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.050+0000 7fdc0cb34640 1 --2- 172.21.15.44:0/42872138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc08103350 0x7fdc0819f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:32.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.051+0000 7fdc077fe640 1 -- 172.21.15.44:0/42872138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc08103350 msgr2=0x7fdc0819f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:32.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.051+0000 7fdc077fe640 1 --2- 172.21.15.44:0/42872138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc08103350 0x7fdc0819f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.051+0000 7fdc077fe640 1 -- 172.21.15.44:0/42872138 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdc081017a0 msgr2=0x7fdc08197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:32.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.051+0000 7fdc077fe640 1 --2- 172.21.15.44:0/42872138 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdc081017a0 0x7fdc08197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.051+0000 7fdc077fe640 1 -- 172.21.15.44:0/42872138 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc0819f6a0 con 0x7fdc08102990 2024-04-21T23:06:32.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.051+0000 7fdc077fe640 1 --2- 172.21.15.44:0/42872138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc08102990 0x7fdc08197fc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdbf800c920 tx=0x7fdbf800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:32.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.051+0000 7fdc057fa640 1 -- 172.21.15.44:0/42872138 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8007c70 con 0x7fdc08102990 2024-04-21T23:06:32.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.051+0000 7fdc057fa640 1 -- 172.21.15.44:0/42872138 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf8007e10 con 0x7fdc08102990 2024-04-21T23:06:32.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.051+0000 7fdc057fa640 1 -- 172.21.15.44:0/42872138 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8018890 con 0x7fdc08102990 2024-04-21T23:06:32.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.051+0000 7fdc0e597640 1 -- 172.21.15.44:0/42872138 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc0819f990 con 0x7fdc08102990 2024-04-21T23:06:32.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.051+0000 7fdc0e597640 1 -- 172.21.15.44:0/42872138 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc0819ff30 con 0x7fdc08102990 2024-04-21T23:06:32.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.052+0000 7fdc0e597640 1 -- 172.21.15.44:0/42872138 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc08108a60 con 0x7fdc08102990 2024-04-21T23:06:32.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.053+0000 7fdc057fa640 1 -- 172.21.15.44:0/42872138 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fdbf8018a70 con 0x7fdc08102990 2024-04-21T23:06:32.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.054+0000 7fdc057fa640 1 --2- 172.21.15.44:0/42872138 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdbf00a2c10 0x7fdbf00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:32.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.054+0000 7fdc07fff640 1 --2- 172.21.15.44:0/42872138 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdbf00a2c10 0x7fdbf00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:32.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.054+0000 7fdc057fa640 1 -- 172.21.15.44:0/42872138 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fdbf80cf220 con 0x7fdc08102990 2024-04-21T23:06:32.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.055+0000 7fdc07fff640 1 --2- 172.21.15.44:0/42872138 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdbf00a2c10 0x7fdbf00a50d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc000c00 tx=0x7fdbfc004f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:32.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.059+0000 7fdc057fa640 1 -- 172.21.15.44:0/42872138 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fdbf8014030 con 0x7fdc08102990 2024-04-21T23:06:32.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.243+0000 7fdc0e597640 1 -- 172.21.15.44:0/42872138 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdc0802d380 con 0x7fdc08102990 2024-04-21T23:06:32.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.244+0000 7fdc057fa640 1 -- 172.21.15.44:0/42872138 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24032 (secure 0 0 0) 0x7fdbf809bfe0 con 0x7fdc08102990 2024-04-21T23:06:32.244 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:32.244 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":90,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:06:30.619155+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:06:32.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.246+0000 7fdc0e597640 1 -- 172.21.15.44:0/42872138 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdbf00a2c10 msgr2=0x7fdbf00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:32.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.246+0000 7fdc0e597640 1 --2- 172.21.15.44:0/42872138 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdbf00a2c10 0x7fdbf00a50d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc000c00 tx=0x7fdbfc004f10 comp rx=0 tx=0).stop 2024-04-21T23:06:32.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.246+0000 7fdc0e597640 1 -- 172.21.15.44:0/42872138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc08102990 msgr2=0x7fdc08197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:32.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.246+0000 7fdc0e597640 1 --2- 172.21.15.44:0/42872138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc08102990 0x7fdc08197fc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdbf800c920 tx=0x7fdbf800cde0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.246+0000 7fdc0e597640 1 -- 172.21.15.44:0/42872138 shutdown_connections 2024-04-21T23:06:32.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.246+0000 7fdc0e597640 1 --2- 172.21.15.44:0/42872138 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdbf00a2c10 0x7fdbf00a50d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.246+0000 7fdc0e597640 1 --2- 172.21.15.44:0/42872138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc08103350 0x7fdc0819f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.246+0000 7fdc0e597640 1 --2- 172.21.15.44:0/42872138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdc08102990 0x7fdc08197fc0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.246+0000 7fdc0e597640 1 --2- 172.21.15.44:0/42872138 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdc081017a0 0x7fdc08197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.246+0000 7fdc0e597640 1 -- 172.21.15.44:0/42872138 >> 172.21.15.44:0/42872138 conn(0x7fdc080fcd30 msgr2=0x7fdc08105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:32.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.246+0000 7fdc0e597640 1 -- 172.21.15.44:0/42872138 shutdown_connections 2024-04-21T23:06:32.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.246+0000 7fdc0e597640 1 -- 172.21.15.44:0/42872138 wait complete. 2024-04-21T23:06:32.272 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:06:32.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.373+0000 7ff6e877a640 1 -- 172.21.15.44:0/2477008679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e0102220 msgr2=0x7ff6e0104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:32.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.373+0000 7ff6e877a640 1 --2- 172.21.15.44:0/2477008679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e0102220 0x7ff6e0104610 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009900 tx=0x7ff6d002bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:32.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.374+0000 7ff6e877a640 1 -- 172.21.15.44:0/2477008679 shutdown_connections 2024-04-21T23:06:32.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.374+0000 7ff6e877a640 1 --2- 172.21.15.44:0/2477008679 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e0104b50 0x7ff6e0106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.374+0000 7ff6e877a640 1 --2- 172.21.15.44:0/2477008679 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e0107480 0x7ff6e01098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.374+0000 7ff6e877a640 1 --2- 172.21.15.44:0/2477008679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e0102220 0x7ff6e0104610 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.374+0000 7ff6e877a640 1 -- 172.21.15.44:0/2477008679 >> 172.21.15.44:0/2477008679 conn(0x7ff6e00fbc10 msgr2=0x7ff6e00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:32.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.374+0000 7ff6e877a640 1 -- 172.21.15.44:0/2477008679 shutdown_connections 2024-04-21T23:06:32.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.374+0000 7ff6e877a640 1 -- 172.21.15.44:0/2477008679 wait complete. 2024-04-21T23:06:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.374+0000 7ff6e877a640 1 Processor -- start 2024-04-21T23:06:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.374+0000 7ff6e877a640 1 -- start start 2024-04-21T23:06:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.374+0000 7ff6e877a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e0102220 0x7ff6e0101b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.374+0000 7ff6e877a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e0104b50 0x7ff6e01001d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.374+0000 7ff6e877a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e0107480 0x7ff6e0100710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.374+0000 7ff6e877a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e010d450 con 0x7ff6e0107480 2024-04-21T23:06:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.374+0000 7ff6e877a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e010d2d0 con 0x7ff6e0102220 2024-04-21T23:06:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.374+0000 7ff6e877a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e010d5d0 con 0x7ff6e0104b50 2024-04-21T23:06:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.375+0000 7ff6e6d17640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e0107480 0x7ff6e0100710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.375+0000 7ff6e6516640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e0102220 0x7ff6e0101b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.375+0000 7ff6e6d17640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e0107480 0x7ff6e0100710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41452/0 (socket says 172.21.15.44:41452) 2024-04-21T23:06:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.375+0000 7ff6e6d17640 1 -- 172.21.15.44:0/3357783561 learned_addr learned my addr 172.21.15.44:0/3357783561 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.375+0000 7ff6e5d15640 1 --2- 172.21.15.44:0/3357783561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e0104b50 0x7ff6e01001d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:32.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.375+0000 7ff6e6d17640 1 -- 172.21.15.44:0/3357783561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e0104b50 msgr2=0x7ff6e01001d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:32.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.375+0000 7ff6e6d17640 1 --2- 172.21.15.44:0/3357783561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e0104b50 0x7ff6e01001d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.375+0000 7ff6e6d17640 1 -- 172.21.15.44:0/3357783561 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e0102220 msgr2=0x7ff6e0101b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:32.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.375+0000 7ff6e6d17640 1 --2- 172.21.15.44:0/3357783561 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e0102220 0x7ff6e0101b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.375+0000 7ff6e6d17640 1 -- 172.21.15.44:0/3357783561 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6e0100fa0 con 0x7ff6e0107480 2024-04-21T23:06:32.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.375+0000 7ff6e6516640 1 --2- 172.21.15.44:0/3357783561 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e0102220 0x7ff6e0101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:06:32.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.375+0000 7ff6e5d15640 1 --2- 172.21.15.44:0/3357783561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e0104b50 0x7ff6e01001d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:06:32.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.375+0000 7ff6e6d17640 1 --2- 172.21.15.44:0/3357783561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e0107480 0x7ff6e0100710 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff6d800d750 tx=0x7ff6d800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:32.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.375+0000 7ff6d77fe640 1 -- 172.21.15.44:0/3357783561 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8017070 con 0x7ff6e0107480 2024-04-21T23:06:32.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.375+0000 7ff6e877a640 1 -- 172.21.15.44:0/3357783561 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6e01a1be0 con 0x7ff6e0107480 2024-04-21T23:06:32.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.375+0000 7ff6d77fe640 1 -- 172.21.15.44:0/3357783561 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6d8005500 con 0x7ff6e0107480 2024-04-21T23:06:32.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.375+0000 7ff6d77fe640 1 -- 172.21.15.44:0/3357783561 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8004d90 con 0x7ff6e0107480 2024-04-21T23:06:32.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.375+0000 7ff6e877a640 1 -- 172.21.15.44:0/3357783561 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6e01a20e0 con 0x7ff6e0107480 2024-04-21T23:06:32.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.376+0000 7ff6e877a640 1 -- 172.21.15.44:0/3357783561 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6e00fd700 con 0x7ff6e0107480 2024-04-21T23:06:32.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.377+0000 7ff6d77fe640 1 -- 172.21.15.44:0/3357783561 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff6d8004f30 con 0x7ff6e0107480 2024-04-21T23:06:32.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.378+0000 7ff6d77fe640 1 --2- 172.21.15.44:0/3357783561 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff6cc0a2b90 0x7ff6cc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:32.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.378+0000 7ff6e6516640 1 --2- 172.21.15.44:0/3357783561 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff6cc0a2b90 0x7ff6cc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:32.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.378+0000 7ff6d77fe640 1 -- 172.21.15.44:0/3357783561 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff6d80ced20 con 0x7ff6e0107480 2024-04-21T23:06:32.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.379+0000 7ff6e6516640 1 --2- 172.21.15.44:0/3357783561 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff6cc0a2b90 0x7ff6cc0a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff6d002c7b0 tx=0x7ff6d0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:32.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.383+0000 7ff6d77fe640 1 -- 172.21.15.44:0/3357783561 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff6d809b9e0 con 0x7ff6e0107480 2024-04-21T23:06:32.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:32 smithi055 ceph-mon[31117]: pgmap v293: 129 pgs: 3 creating+peering, 97 active+clean, 29 unknown; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s wr, 9 op/s 2024-04-21T23:06:32.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:32 smithi055 ceph-mon[31117]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:06:32.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:32 smithi055 ceph-mon[31117]: osdmap e90: 12 total, 12 up, 12 in 2024-04-21T23:06:32.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:32 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1197510634' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T23:06:32.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:32 smithi055 ceph-mon[31117]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:06:32.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:32 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1669914396' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:32.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:32 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1928667091' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:32.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:32 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/42872138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:32.465 INFO:journalctl@ceph.mds.b.smithi055.stdout:Apr 21 23:06:32 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-b[61560]: 2024-04-21T23:06:32.183+0000 7fda0d0e5700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-21T23:06:32.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.606+0000 7ff6e877a640 1 -- 172.21.15.44:0/3357783561 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6e00685f0 con 0x7ff6e0107480 2024-04-21T23:06:32.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.607+0000 7ff6d77fe640 1 -- 172.21.15.44:0/3357783561 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11782 (secure 0 0 0) 0x7ff6d809b2e0 con 0x7ff6e0107480 2024-04-21T23:06:32.607 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:32.607 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24526,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/3743898853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3743898853},{"type":"v1","addr":"172.21.15.69:6839","nonce":3743898853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:06:30.619342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:06:32.608 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 17 2024-04-21T23:06:32.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.609+0000 7ff6e877a640 1 -- 172.21.15.44:0/3357783561 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff6cc0a2b90 msgr2=0x7ff6cc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:32.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.609+0000 7ff6e877a640 1 --2- 172.21.15.44:0/3357783561 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff6cc0a2b90 0x7ff6cc0a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff6d002c7b0 tx=0x7ff6d0005fd0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.609+0000 7ff6e877a640 1 -- 172.21.15.44:0/3357783561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e0107480 msgr2=0x7ff6e0100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:32.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.609+0000 7ff6e877a640 1 --2- 172.21.15.44:0/3357783561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e0107480 0x7ff6e0100710 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff6d800d750 tx=0x7ff6d800dc10 comp rx=0 tx=0).stop 2024-04-21T23:06:32.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.609+0000 7ff6e877a640 1 -- 172.21.15.44:0/3357783561 shutdown_connections 2024-04-21T23:06:32.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.609+0000 7ff6e877a640 1 --2- 172.21.15.44:0/3357783561 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff6cc0a2b90 0x7ff6cc0a5050 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.609+0000 7ff6e877a640 1 --2- 172.21.15.44:0/3357783561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e0104b50 0x7ff6e01001d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.609+0000 7ff6e877a640 1 --2- 172.21.15.44:0/3357783561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e0107480 0x7ff6e0100710 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.609+0000 7ff6e877a640 1 --2- 172.21.15.44:0/3357783561 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e0102220 0x7ff6e0101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.609+0000 7ff6e877a640 1 -- 172.21.15.44:0/3357783561 >> 172.21.15.44:0/3357783561 conn(0x7ff6e00fbc10 msgr2=0x7ff6e0102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:32.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.610+0000 7ff6e877a640 1 -- 172.21.15.44:0/3357783561 shutdown_connections 2024-04-21T23:06:32.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.610+0000 7ff6e877a640 1 -- 172.21.15.44:0/3357783561 wait complete. 2024-04-21T23:06:32.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:32 smithi044 ceph-mon[27934]: pgmap v293: 129 pgs: 3 creating+peering, 97 active+clean, 29 unknown; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s wr, 9 op/s 2024-04-21T23:06:32.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:32 smithi044 ceph-mon[27934]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:06:32.619 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:32 smithi044 ceph-mon[27934]: osdmap e90: 12 total, 12 up, 12 in 2024-04-21T23:06:32.620 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:32 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1197510634' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T23:06:32.620 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:32 smithi044 ceph-mon[27934]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:06:32.620 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:32 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1669914396' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:32.620 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:32 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1928667091' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:32.620 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:32 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/42872138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:32.621 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:06:32.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:32 smithi069 ceph-mon[31816]: pgmap v293: 129 pgs: 3 creating+peering, 97 active+clean, 29 unknown; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s wr, 9 op/s 2024-04-21T23:06:32.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:32 smithi069 ceph-mon[31816]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T23:06:32.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:32 smithi069 ceph-mon[31816]: osdmap e90: 12 total, 12 up, 12 in 2024-04-21T23:06:32.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:32 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1197510634' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T23:06:32.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:32 smithi069 ceph-mon[31816]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:06:32.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:32 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1669914396' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:32.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:32 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1928667091' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:32.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:32 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/42872138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:32.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.723+0000 7f89f70ed640 1 -- 172.21.15.44:0/1975344547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89f0101890 msgr2=0x7f89f0101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:32.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.723+0000 7f89f70ed640 1 --2- 172.21.15.44:0/1975344547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89f0101890 0x7f89f0101d10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f89dc009900 tx=0x7f89dc02bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:32.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.723+0000 7f89f70ed640 1 -- 172.21.15.44:0/1975344547 shutdown_connections 2024-04-21T23:06:32.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.723+0000 7f89f70ed640 1 --2- 172.21.15.44:0/1975344547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89f0102250 0x7f89f0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.723+0000 7f89f70ed640 1 --2- 172.21.15.44:0/1975344547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89f0101890 0x7f89f0101d10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.723+0000 7f89f70ed640 1 --2- 172.21.15.44:0/1975344547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89f01006a0 0x7f89f0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.723+0000 7f89f70ed640 1 -- 172.21.15.44:0/1975344547 >> 172.21.15.44:0/1975344547 conn(0x7f89f00fbc50 msgr2=0x7f89f00fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:32.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.723+0000 7f89f70ed640 1 -- 172.21.15.44:0/1975344547 shutdown_connections 2024-04-21T23:06:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.723+0000 7f89f70ed640 1 -- 172.21.15.44:0/1975344547 wait complete. 2024-04-21T23:06:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.724+0000 7f89f70ed640 1 Processor -- start 2024-04-21T23:06:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.724+0000 7f89f70ed640 1 -- start start 2024-04-21T23:06:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.724+0000 7f89f70ed640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89f01006a0 0x7f89f006a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.724+0000 7f89f70ed640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89f0101890 0x7f89f019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.724+0000 7f89f70ed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89f0102250 0x7f89f019ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.724+0000 7f89f70ed640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f89f010c350 con 0x7f89f0102250 2024-04-21T23:06:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.724+0000 7f89f70ed640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f89f010c1d0 con 0x7f89f0101890 2024-04-21T23:06:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.724+0000 7f89f70ed640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f89f010c4d0 con 0x7f89f01006a0 2024-04-21T23:06:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.724+0000 7f89f568a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89f0102250 0x7f89f019ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.724+0000 7f89f568a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89f0102250 0x7f89f019ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41464/0 (socket says 172.21.15.44:41464) 2024-04-21T23:06:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.724+0000 7f89f568a640 1 -- 172.21.15.44:0/2847743148 learned_addr learned my addr 172.21.15.44:0/2847743148 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:32.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.724+0000 7f89f4e89640 1 --2- 172.21.15.44:0/2847743148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89f01006a0 0x7f89f006a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:32.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.724+0000 7f89effff640 1 --2- 172.21.15.44:0/2847743148 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89f0101890 0x7f89f019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:32.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.724+0000 7f89f568a640 1 -- 172.21.15.44:0/2847743148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89f01006a0 msgr2=0x7f89f006a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:32.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.724+0000 7f89f568a640 1 --2- 172.21.15.44:0/2847743148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89f01006a0 0x7f89f006a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.724+0000 7f89f568a640 1 -- 172.21.15.44:0/2847743148 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89f0101890 msgr2=0x7f89f019c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:32.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.724+0000 7f89f568a640 1 --2- 172.21.15.44:0/2847743148 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89f0101890 0x7f89f019c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.724+0000 7f89f568a640 1 -- 172.21.15.44:0/2847743148 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89f019f740 con 0x7f89f0102250 2024-04-21T23:06:32.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.724+0000 7f89effff640 1 --2- 172.21.15.44:0/2847743148 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89f0101890 0x7f89f019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:06:32.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.725+0000 7f89f568a640 1 --2- 172.21.15.44:0/2847743148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89f0102250 0x7f89f019ee30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f89e800cad0 tx=0x7f89e800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:32.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.725+0000 7f89edffb640 1 -- 172.21.15.44:0/2847743148 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e800ae20 con 0x7f89f0102250 2024-04-21T23:06:32.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.725+0000 7f89edffb640 1 -- 172.21.15.44:0/2847743148 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89e8005020 con 0x7f89f0102250 2024-04-21T23:06:32.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.725+0000 7f89f70ed640 1 -- 172.21.15.44:0/2847743148 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89f019f9d0 con 0x7f89f0102250 2024-04-21T23:06:32.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.725+0000 7f89edffb640 1 -- 172.21.15.44:0/2847743148 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e800f6f0 con 0x7f89f0102250 2024-04-21T23:06:32.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.725+0000 7f89f70ed640 1 -- 172.21.15.44:0/2847743148 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89f019fed0 con 0x7f89f0102250 2024-04-21T23:06:32.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.726+0000 7f89f70ed640 1 -- 172.21.15.44:0/2847743148 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89f0107a40 con 0x7f89f0102250 2024-04-21T23:06:32.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.727+0000 7f89edffb640 1 -- 172.21.15.44:0/2847743148 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f89e80051d0 con 0x7f89f0102250 2024-04-21T23:06:32.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.728+0000 7f89edffb640 1 --2- 172.21.15.44:0/2847743148 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f89e00a2c10 0x7f89e00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:32.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.728+0000 7f89f4e89640 1 --2- 172.21.15.44:0/2847743148 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f89e00a2c10 0x7f89e00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:32.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.728+0000 7f89edffb640 1 -- 172.21.15.44:0/2847743148 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f89e80cf380 con 0x7f89f0102250 2024-04-21T23:06:32.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.728+0000 7f89f4e89640 1 --2- 172.21.15.44:0/2847743148 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f89e00a2c10 0x7f89e00a50d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f89e4005d00 tx=0x7f89e4005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:32.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.732+0000 7f89edffb640 1 -- 172.21.15.44:0/2847743148 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f89e8014030 con 0x7f89f0102250 2024-04-21T23:06:32.919 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.920+0000 7f89f70ed640 1 -- 172.21.15.44:0/2847743148 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f89f00623e0 con 0x7f89f0102250 2024-04-21T23:06:32.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.921+0000 7f89edffb640 1 -- 172.21.15.44:0/2847743148 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24032 (secure 0 0 0) 0x7f89e809c040 con 0x7f89f0102250 2024-04-21T23:06:32.921 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:32.921 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":90,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:06:30.619155+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:06:32.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.923+0000 7f89f70ed640 1 -- 172.21.15.44:0/2847743148 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f89e00a2c10 msgr2=0x7f89e00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:32.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.923+0000 7f89f70ed640 1 --2- 172.21.15.44:0/2847743148 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f89e00a2c10 0x7f89e00a50d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f89e4005d00 tx=0x7f89e4005c50 comp rx=0 tx=0).stop 2024-04-21T23:06:32.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.923+0000 7f89f70ed640 1 -- 172.21.15.44:0/2847743148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89f0102250 msgr2=0x7f89f019ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:32.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.923+0000 7f89f70ed640 1 --2- 172.21.15.44:0/2847743148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89f0102250 0x7f89f019ee30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f89e800cad0 tx=0x7f89e800cf90 comp rx=0 tx=0).stop 2024-04-21T23:06:32.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.923+0000 7f89f70ed640 1 -- 172.21.15.44:0/2847743148 shutdown_connections 2024-04-21T23:06:32.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.924+0000 7f89f70ed640 1 --2- 172.21.15.44:0/2847743148 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f89e00a2c10 0x7f89e00a50d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.924+0000 7f89f70ed640 1 --2- 172.21.15.44:0/2847743148 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89f0102250 0x7f89f019ee30 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.924+0000 7f89f70ed640 1 --2- 172.21.15.44:0/2847743148 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89f0101890 0x7f89f019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.924+0000 7f89f70ed640 1 --2- 172.21.15.44:0/2847743148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89f01006a0 0x7f89f006a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:32.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.924+0000 7f89f70ed640 1 -- 172.21.15.44:0/2847743148 >> 172.21.15.44:0/2847743148 conn(0x7f89f00fbc50 msgr2=0x7f89f0104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:32.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.924+0000 7f89f70ed640 1 -- 172.21.15.44:0/2847743148 shutdown_connections 2024-04-21T23:06:32.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:32.924+0000 7f89f70ed640 1 -- 172.21.15.44:0/2847743148 wait complete. 2024-04-21T23:06:32.935 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-21T23:06:25.562805+0000', 'modified': '2024-04-21T23:06:30.619342+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24530}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24530': {'gid': 24530, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.55:6839/2066817543', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6838', 'nonce': 2066817543}, {'type': 'v1', 'addr': '172.21.15.55:6839', 'nonce': 2066817543}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24530, 'qdb_cluster': [24530]} 2024-04-21T23:06:32.935 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-21T23:06:32.935 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T23:06:33.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.039+0000 7fb36f39f640 1 -- 172.21.15.44:0/1285459643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb368103180 msgr2=0x7fb368109a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:33.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.039+0000 7fb36f39f640 1 --2- 172.21.15.44:0/1285459643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb368103180 0x7fb368109a10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb360009ca0 tx=0x7fb36002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.039+0000 7fb36f39f640 1 -- 172.21.15.44:0/1285459643 shutdown_connections 2024-04-21T23:06:33.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.039+0000 7fb36f39f640 1 --2- 172.21.15.44:0/1285459643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb368103180 0x7fb368109a10 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.039+0000 7fb36f39f640 1 --2- 172.21.15.44:0/1285459643 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3681027c0 0x7fb368102c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.039+0000 7fb36f39f640 1 --2- 172.21.15.44:0/1285459643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3681015d0 0x7fb3681019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.039+0000 7fb36f39f640 1 -- 172.21.15.44:0/1285459643 >> 172.21.15.44:0/1285459643 conn(0x7fb3680fcb40 msgr2=0x7fb3680fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:33.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.039+0000 7fb36f39f640 1 -- 172.21.15.44:0/1285459643 shutdown_connections 2024-04-21T23:06:33.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.039+0000 7fb36f39f640 1 -- 172.21.15.44:0/1285459643 wait complete. 2024-04-21T23:06:33.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb36f39f640 1 Processor -- start 2024-04-21T23:06:33.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb36f39f640 1 -- start start 2024-04-21T23:06:33.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb36f39f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3681015d0 0x7fb3681978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:33.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb36f39f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3681027c0 0x7fb368197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:33.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb36f39f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb368103180 0x7fb36819ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:33.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb36f39f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb36810d230 con 0x7fb368103180 2024-04-21T23:06:33.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb36f39f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb36810d0b0 con 0x7fb3681015d0 2024-04-21T23:06:33.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb36f39f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb36810d3b0 con 0x7fb3681027c0 2024-04-21T23:06:33.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb36eb9e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb368103180 0x7fb36819ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:33.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb36eb9e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb368103180 0x7fb36819ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41472/0 (socket says 172.21.15.44:41472) 2024-04-21T23:06:33.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb36eb9e640 1 -- 172.21.15.44:0/2066141406 learned_addr learned my addr 172.21.15.44:0/2066141406 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:33.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb36db9c640 1 --2- 172.21.15.44:0/2066141406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3681027c0 0x7fb368197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:33.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb36e39d640 1 --2- 172.21.15.44:0/2066141406 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3681015d0 0x7fb3681978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:33.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb36eb9e640 1 -- 172.21.15.44:0/2066141406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3681027c0 msgr2=0x7fb368197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:33.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb36eb9e640 1 --2- 172.21.15.44:0/2066141406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3681027c0 0x7fb368197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb36eb9e640 1 -- 172.21.15.44:0/2066141406 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3681015d0 msgr2=0x7fb3681978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:33.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb36eb9e640 1 --2- 172.21.15.44:0/2066141406 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3681015d0 0x7fb3681978a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb36eb9e640 1 -- 172.21.15.44:0/2066141406 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36819f4c0 con 0x7fb368103180 2024-04-21T23:06:33.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb36e39d640 1 --2- 172.21.15.44:0/2066141406 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3681015d0 0x7fb3681978a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:06:33.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb36eb9e640 1 --2- 172.21.15.44:0/2066141406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb368103180 0x7fb36819ee50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb360037040 tx=0x7fb360032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:33.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.040+0000 7fb35f7fe640 1 -- 172.21.15.44:0/2066141406 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb360041020 con 0x7fb368103180 2024-04-21T23:06:33.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.041+0000 7fb35f7fe640 1 -- 172.21.15.44:0/2066141406 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb360034d90 con 0x7fb368103180 2024-04-21T23:06:33.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.041+0000 7fb36f39f640 1 -- 172.21.15.44:0/2066141406 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb36819f750 con 0x7fb368103180 2024-04-21T23:06:33.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.041+0000 7fb35f7fe640 1 -- 172.21.15.44:0/2066141406 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb360039040 con 0x7fb368103180 2024-04-21T23:06:33.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.041+0000 7fb36f39f640 1 -- 172.21.15.44:0/2066141406 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb36819fc90 con 0x7fb368103180 2024-04-21T23:06:33.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.043+0000 7fb35f7fe640 1 -- 172.21.15.44:0/2066141406 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb360044020 con 0x7fb368103180 2024-04-21T23:06:33.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.043+0000 7fb3731eb740 1 Processor -- start 2024-04-21T23:06:33.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb3731eb740 1 -- start start 2024-04-21T23:06:33.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb3731eb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 0x561db01e1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:33.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb3731eb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 0x561db01e6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:33.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb3731eb740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e6dd0 0x561db01e9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:33.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb3731eb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561db0145df0 con 0x561db01e1c60 2024-04-21T23:06:33.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb3731eb740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x561db00ee640 con 0x561db01e6dd0 2024-04-21T23:06:33.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb3731eb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x561db0141db0 con 0x561db01e1270 2024-04-21T23:06:33.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb35f7fe640 1 --2- 172.21.15.44:0/2066141406 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb3540a2c10 0x7fb3540a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:33.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb36e39d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 0x561db01e6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:33.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb36e39d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 0x561db01e6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41476/0 (socket says 172.21.15.44:41476) 2024-04-21T23:06:33.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb36e39d640 1 -- 172.21.15.44:0/1142366811 learned_addr learned my addr 172.21.15.44:0/1142366811 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:33.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb36db9c640 1 --2- 172.21.15.44:0/1142366811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 0x561db01e1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:33.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb36eb9e640 1 --2- 172.21.15.44:0/2066141406 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb3540a2c10 0x7fb3540a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:33.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb36db9c640 1 --2- 172.21.15.44:0/1142366811 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e6dd0 0x561db01e9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:33.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb36e39d640 1 -- 172.21.15.44:0/1142366811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 msgr2=0x561db01e1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:06:33.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb35f7fe640 1 -- 172.21.15.44:0/2066141406 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fb3600f1260 con 0x7fb368103180 2024-04-21T23:06:33.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb36e39d640 1 --2- 172.21.15.44:0/1142366811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 0x561db01e1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb36e39d640 1 -- 172.21.15.44:0/1142366811 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e6dd0 msgr2=0x561db01e9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:06:33.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb36e39d640 1 --2- 172.21.15.44:0/1142366811 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e6dd0 0x561db01e9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb36e39d640 1 -- 172.21.15.44:0/1142366811 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dafe5a2d0 con 0x561db01e1c60 2024-04-21T23:06:33.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb36db9c640 1 --2- 172.21.15.44:0/1142366811 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e6dd0 0x561db01e9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:06:33.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.044+0000 7fb36e39d640 1 --2- 172.21.15.44:0/1142366811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 0x561db01e6760 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb36400bb70 tx=0x7fb36400d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a03ffd3f9119646c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:33.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.045+0000 7fb36eb9e640 1 --2- 172.21.15.44:0/2066141406 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb3540a2c10 0x7fb3540a50d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb3600ba1f0 tx=0x7fb3600908e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:33.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.045+0000 7fb35cbfe640 1 -- 172.21.15.44:0/1142366811 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb364005220 con 0x561db01e1c60 2024-04-21T23:06:33.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.045+0000 7fb35cbfe640 1 -- 172.21.15.44:0/1142366811 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb364004c80 con 0x561db01e1c60 2024-04-21T23:06:33.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.045+0000 7fb35cbfe640 1 -- 172.21.15.44:0/1142366811 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36400f810 con 0x561db01e1c60 2024-04-21T23:06:33.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.045+0000 7fb3731eb740 1 -- 172.21.15.44:0/1142366811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 msgr2=0x561db01e6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:33.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.045+0000 7fb3731eb740 1 --2- 172.21.15.44:0/1142366811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 0x561db01e6760 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb36400bb70 tx=0x7fb36400d710 comp rx=0 tx=0).stop 2024-04-21T23:06:33.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.045+0000 7fb3731eb740 1 -- 172.21.15.44:0/1142366811 shutdown_connections 2024-04-21T23:06:33.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.045+0000 7fb3731eb740 1 --2- 172.21.15.44:0/1142366811 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e6dd0 0x561db01e9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.045+0000 7fb3731eb740 1 --2- 172.21.15.44:0/1142366811 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 0x561db01e6760 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.045+0000 7fb3731eb740 1 --2- 172.21.15.44:0/1142366811 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 0x561db01e1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.045+0000 7fb3731eb740 1 -- 172.21.15.44:0/1142366811 >> 172.21.15.44:0/1142366811 conn(0x561db01dc7e0 msgr2=0x561db01dcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:33.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb3731eb740 1 -- 172.21.15.44:0/1142366811 shutdown_connections 2024-04-21T23:06:33.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb3731eb740 1 -- 172.21.15.44:0/1142366811 wait complete. 2024-04-21T23:06:33.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb3731eb740 1 -- start start 2024-04-21T23:06:33.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb3731eb740 1 Processor -- start 2024-04-21T23:06:33.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb3731eb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 0x561db01eb750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:33.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb3731eb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 0x561db01ebc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:33.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb3731eb740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e6dd0 0x561db01f2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:33.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb3731eb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561db01e9d80 con 0x561db01e1c60 2024-04-21T23:06:33.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb3731eb740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x561db0101be0 con 0x561db01e6dd0 2024-04-21T23:06:33.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb3731eb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x561db01e9f00 con 0x561db01e1270 2024-04-21T23:06:33.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb3731eb740 10 client.0 _ll_register_callbacks cb 0x561db01015d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:06:33.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb36e39d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 0x561db01ebc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:33.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb36e39d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 0x561db01ebc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41484/0 (socket says 172.21.15.44:41484) 2024-04-21T23:06:33.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb36e39d640 1 -- 172.21.15.44:0/3326620378 learned_addr learned my addr 172.21.15.44:0/3326620378 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:33.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb36db9c640 1 --2- 172.21.15.44:0/3326620378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 0x561db01eb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:33.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb36db9c640 1 --2- 172.21.15.44:0/3326620378 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e6dd0 0x561db01f2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:33.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb36e39d640 1 -- 172.21.15.44:0/3326620378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 msgr2=0x561db01eb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:06:33.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb36e39d640 1 --2- 172.21.15.44:0/3326620378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 0x561db01eb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb36e39d640 1 -- 172.21.15.44:0/3326620378 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e6dd0 msgr2=0x561db01f2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:06:33.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb36e39d640 1 --2- 172.21.15.44:0/3326620378 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e6dd0 0x561db01f2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb36e39d640 1 -- 172.21.15.44:0/3326620378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561daff246b0 con 0x561db01e1c60 2024-04-21T23:06:33.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.046+0000 7fb36db9c640 1 --2- 172.21.15.44:0/3326620378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 0x561db01eb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:06:33.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.047+0000 7fb36e39d640 1 --2- 172.21.15.44:0/3326620378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 0x561db01ebc90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb364000c00 tx=0x7fb364004eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a454ec04f5557e33 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:33.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.047+0000 7fb351ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:06:33.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.047+0000 7fb351ffb640 1 -- 172.21.15.44:0/3326620378 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb364025030 con 0x561db01e1c60 2024-04-21T23:06:33.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.047+0000 7fb3731eb740 1 -- 172.21.15.44:0/3326620378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb364004c80 con 0x561db01e1c60 2024-04-21T23:06:33.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.047+0000 7fb351ffb640 1 -- 172.21.15.44:0/3326620378 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fb36401c070 con 0x561db01e1c60 2024-04-21T23:06:33.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.047+0000 7fb351ffb640 1 -- 172.21.15.44:0/3326620378 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fb36402f050 con 0x561db01e1c60 2024-04-21T23:06:33.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.047+0000 7fb3731eb740 10 client.14772 fetch_fsmap learned FSMap version 17 2024-04-21T23:06:33.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.047+0000 7fb3731eb740 1 -- 172.21.15.44:0/3326620378 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x561db01f5d00 con 0x561db01e1c60 2024-04-21T23:06:33.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.047+0000 7fb351ffb640 1 -- 172.21.15.44:0/3326620378 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb364004c80 con 0x561db01e1c60 2024-04-21T23:06:33.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.047+0000 7fb351ffb640 1 -- 172.21.15.44:0/3326620378 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fb36400fca0 con 0x561db01e1c60 2024-04-21T23:06:33.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.047+0000 7fb3731eb740 10 client.14772 fetch_fsmap finished waiting for FSMap version 17 2024-04-21T23:06:33.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.047+0000 7fb3731eb740 10 client.14772 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T23:06:33.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.047+0000 7fb3731eb740 20 client.14772 populate_metadata read hostname 'smithi044' 2024-04-21T23:06:33.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.047+0000 7fb3731eb740 1 --2- 172.21.15.44:0/3326620378 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x561db01f7f40 0x561db01f93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:33.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.047+0000 7fb3731eb740 4 client.14772 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:06:33.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.047+0000 7fb3731eb740 1 -- 172.21.15.44:0/3326620378 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561db0153330 con 0x561db01f7f40 2024-04-21T23:06:33.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.048+0000 7fb36db9c640 1 --2- 172.21.15.44:0/3326620378 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x561db01f7f40 0x561db01f93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:33.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.048+0000 7fb351ffb640 10 client.14772 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:06:33.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.048+0000 7fb36db9c640 1 --2- 172.21.15.44:0/3326620378 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x561db01f7f40 0x561db01f93b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0f9f250cf42e543 server_cookie=33e1e106957217e5 in_seq=0 out_seq=0 2024-04-21T23:06:33.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.049+0000 7fb351ffb640 1 -- 172.21.15.44:0/3326620378 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561db0153330 con 0x561db01f7f40 2024-04-21T23:06:33.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.049+0000 7fb351ffb640 10 client.14772 handle_command_reply: tid=0 2024-04-21T23:06:33.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.049+0000 7fb3731eb740 1 client.14772 shutdown 2024-04-21T23:06:33.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.049+0000 7fb3731eb740 1 -- 172.21.15.44:0/3326620378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 msgr2=0x561db01ebc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:33.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.049+0000 7fb3731eb740 1 --2- 172.21.15.44:0/3326620378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 0x561db01ebc90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb364000c00 tx=0x7fb364004eb0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.049+0000 7fb3731eb740 1 -- 172.21.15.44:0/3326620378 shutdown_connections 2024-04-21T23:06:33.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.049+0000 7fb3731eb740 1 --2- 172.21.15.44:0/3326620378 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x561db01f7f40 0x561db01f93b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.049+0000 7fb3731eb740 1 --2- 172.21.15.44:0/3326620378 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e6dd0 0x561db01f2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.049+0000 7fb3731eb740 1 --2- 172.21.15.44:0/3326620378 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 0x561db01ebc90 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.049+0000 7fb351ffb640 0 client.14772 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:06:33.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.049+0000 7fb3731eb740 1 --2- 172.21.15.44:0/3326620378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 0x561db01eb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.049+0000 7fb3731eb740 1 -- 172.21.15.44:0/3326620378 >> 172.21.15.44:0/3326620378 conn(0x561db01dc7e0 msgr2=0x561db01db9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:33.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.049+0000 7fb3731eb740 1 -- 172.21.15.44:0/3326620378 shutdown_connections 2024-04-21T23:06:33.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.049+0000 7fb3731eb740 1 -- 172.21.15.44:0/3326620378 wait complete. 2024-04-21T23:06:33.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.050+0000 7fb3731eb740 20 client.14772 trim_cache size 0 max 16384 2024-04-21T23:06:33.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.068+0000 7fb3731eb740 1 Processor -- start 2024-04-21T23:06:33.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.068+0000 7fb3731eb740 1 -- start start 2024-04-21T23:06:33.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.068+0000 7fb3731eb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 0x561db01dcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:33.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.068+0000 7fb3731eb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 0x561db01f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:33.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.068+0000 7fb3731eb740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e6dd0 0x561db01eb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:33.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.068+0000 7fb3731eb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561db01f76f0 con 0x561db01e1c60 2024-04-21T23:06:33.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.068+0000 7fb3731eb740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x561db01f7dc0 con 0x561db01e6dd0 2024-04-21T23:06:33.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.069+0000 7fb3731eb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x561db01f8490 con 0x561db01e1270 2024-04-21T23:06:33.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.069+0000 7fb36e39d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 0x561db01f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:33.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.069+0000 7fb36e39d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 0x561db01f2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41486/0 (socket says 172.21.15.44:41486) 2024-04-21T23:06:33.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.069+0000 7fb36e39d640 1 -- 172.21.15.44:0/1534543625 learned_addr learned my addr 172.21.15.44:0/1534543625 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:33.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.069+0000 7fb36db9c640 1 --2- 172.21.15.44:0/1534543625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 0x561db01dcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:33.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.069+0000 7fb36e39d640 1 -- 172.21.15.44:0/1534543625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 msgr2=0x561db01dcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:06:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.069+0000 7fb36e39d640 1 --2- 172.21.15.44:0/1534543625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 0x561db01dcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.069+0000 7fb36db9c640 1 --2- 172.21.15.44:0/1534543625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e6dd0 0x561db01eb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.069+0000 7fb36e39d640 1 -- 172.21.15.44:0/1534543625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e6dd0 msgr2=0x561db01eb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:06:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.069+0000 7fb36e39d640 1 --2- 172.21.15.44:0/1534543625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e6dd0 0x561db01eb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.069+0000 7fb36e39d640 1 -- 172.21.15.44:0/1534543625 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561db01e1060 con 0x561db01e1c60 2024-04-21T23:06:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.069+0000 7fb36db9c640 1 --2- 172.21.15.44:0/1534543625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e6dd0 0x561db01eb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:06:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.069+0000 7fb36e39d640 1 --2- 172.21.15.44:0/1534543625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 0x561db01f2c90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb364021720 tx=0x7fb36400b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b897183cad2c0a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.069+0000 7fb353fff640 1 -- 172.21.15.44:0/1534543625 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb364038070 con 0x561db01e1c60 2024-04-21T23:06:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.069+0000 7fb353fff640 1 -- 172.21.15.44:0/1534543625 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb364005560 con 0x561db01e1c60 2024-04-21T23:06:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.069+0000 7fb353fff640 1 -- 172.21.15.44:0/1534543625 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb364025070 con 0x561db01e1c60 2024-04-21T23:06:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.070+0000 7fb3731eb740 1 -- 172.21.15.44:0/1534543625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 msgr2=0x561db01f2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.070+0000 7fb3731eb740 1 --2- 172.21.15.44:0/1534543625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 0x561db01f2c90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb364021720 tx=0x7fb36400b420 comp rx=0 tx=0).stop 2024-04-21T23:06:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.070+0000 7fb3731eb740 1 -- 172.21.15.44:0/1534543625 shutdown_connections 2024-04-21T23:06:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.070+0000 7fb3731eb740 1 --2- 172.21.15.44:0/1534543625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e6dd0 0x561db01eb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.070+0000 7fb3731eb740 1 --2- 172.21.15.44:0/1534543625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e1c60 0x561db01f2c90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.070+0000 7fb3731eb740 1 --2- 172.21.15.44:0/1534543625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 0x561db01dcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.070+0000 7fb3731eb740 1 -- 172.21.15.44:0/1534543625 >> 172.21.15.44:0/1534543625 conn(0x561db01dc7e0 msgr2=0x561db01e26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:33.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.070+0000 7fb3731eb740 1 -- 172.21.15.44:0/1534543625 shutdown_connections 2024-04-21T23:06:33.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.070+0000 7fb3731eb740 1 -- 172.21.15.44:0/1534543625 wait complete. 2024-04-21T23:06:33.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.070+0000 7fb3731eb740 1 -- start start 2024-04-21T23:06:33.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.070+0000 7fb3731eb740 1 Processor -- start 2024-04-21T23:06:33.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.070+0000 7fb3731eb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 0x561db01eeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:33.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.070+0000 7fb3731eb740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e1c60 0x561db01eeff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:33.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.070+0000 7fb3731eb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e6dd0 0x561db01f2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:33.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.070+0000 7fb3731eb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561db01f9230 con 0x561db01e6dd0 2024-04-21T23:06:33.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.070+0000 7fb3731eb740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x561db01f8b60 con 0x561db01e1c60 2024-04-21T23:06:33.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.070+0000 7fb3731eb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x561db01f9900 con 0x561db01e1270 2024-04-21T23:06:33.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.071+0000 7fb3731eb740 10 client.0 _ll_register_callbacks cb 0x7fb364005330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:06:33.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.071+0000 7fb36db9c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e6dd0 0x561db01f2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:33.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.071+0000 7fb36db9c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e6dd0 0x561db01f2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41496/0 (socket says 172.21.15.44:41496) 2024-04-21T23:06:33.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.071+0000 7fb36db9c640 1 -- 172.21.15.44:0/3986544442 learned_addr learned my addr 172.21.15.44:0/3986544442 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:33.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.071+0000 7fb36e39d640 1 --2- 172.21.15.44:0/3986544442 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e1c60 0x561db01eeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:33.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.071+0000 7fb36db9c640 1 --2- 172.21.15.44:0/3986544442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 0x561db01eeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:33.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.071+0000 7fb36db9c640 1 -- 172.21.15.44:0/3986544442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 msgr2=0x561db01eeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:06:33.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.071+0000 7fb36db9c640 1 --2- 172.21.15.44:0/3986544442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 0x561db01eeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.071+0000 7fb36db9c640 1 -- 172.21.15.44:0/3986544442 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e1c60 msgr2=0x561db01eeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:06:33.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.071+0000 7fb36db9c640 1 --2- 172.21.15.44:0/3986544442 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e1c60 0x561db01eeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.071+0000 7fb36db9c640 1 -- 172.21.15.44:0/3986544442 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561db01f2600 con 0x561db01e6dd0 2024-04-21T23:06:33.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.071+0000 7fb36db9c640 1 --2- 172.21.15.44:0/3986544442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e6dd0 0x561db01f2010 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb358000c00 tx=0x7fb364005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee9f83931a740865 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:33.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.071+0000 7fb351ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:06:33.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.071+0000 7fb351ffb640 1 -- 172.21.15.44:0/3986544442 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb358004030 con 0x561db01e6dd0 2024-04-21T23:06:33.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.071+0000 7fb351ffb640 1 -- 172.21.15.44:0/3986544442 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fb35801b040 con 0x561db01e6dd0 2024-04-21T23:06:33.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.071+0000 7fb3731eb740 1 -- 172.21.15.44:0/3986544442 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb364005560 con 0x561db01e6dd0 2024-04-21T23:06:33.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.071+0000 7fb351ffb640 1 -- 172.21.15.44:0/3986544442 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fb3580041d0 con 0x561db01e6dd0 2024-04-21T23:06:33.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.071+0000 7fb3731eb740 10 client.14778 fetch_fsmap learned FSMap version 17 2024-04-21T23:06:33.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.071+0000 7fb3731eb740 1 -- 172.21.15.44:0/3986544442 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x561db025e5b0 con 0x561db01e6dd0 2024-04-21T23:06:33.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.072+0000 7fb351ffb640 1 -- 172.21.15.44:0/3986544442 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb364005560 con 0x561db01e6dd0 2024-04-21T23:06:33.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.072+0000 7fb351ffb640 1 -- 172.21.15.44:0/3986544442 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fb3580035f0 con 0x561db01e6dd0 2024-04-21T23:06:33.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.072+0000 7fb3731eb740 10 client.14778 fetch_fsmap finished waiting for FSMap version 17 2024-04-21T23:06:33.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.072+0000 7fb3731eb740 10 client.14778 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T23:06:33.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.072+0000 7fb3731eb740 20 client.14778 populate_metadata read hostname 'smithi044' 2024-04-21T23:06:33.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.072+0000 7fb3731eb740 1 --2- 172.21.15.44:0/3986544442 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x561db02631d0 0x561db025f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:33.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.072+0000 7fb3731eb740 4 client.14778 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T23:06:33.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.072+0000 7fb3731eb740 1 -- 172.21.15.44:0/3986544442 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "status"}) -- 0x561db014d680 con 0x561db02631d0 2024-04-21T23:06:33.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.072+0000 7fb36e39d640 1 --2- 172.21.15.44:0/3986544442 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x561db02631d0 0x561db025f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:33.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.072+0000 7fb351ffb640 10 client.14778 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:06:33.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.072+0000 7fb36e39d640 1 --2- 172.21.15.44:0/3986544442 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x561db02631d0 0x561db025f610 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=765e7c4190e73118 server_cookie=9139251c38846e99 in_seq=0 out_seq=0 2024-04-21T23:06:33.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb351ffb640 1 -- 172.21.15.44:0/3986544442 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x561db014d680 con 0x561db02631d0 2024-04-21T23:06:33.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb351ffb640 10 client.14778 handle_command_reply: tid=0 2024-04-21T23:06:33.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb3731eb740 1 client.14778 shutdown 2024-04-21T23:06:33.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb3731eb740 1 -- 172.21.15.44:0/3986544442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e6dd0 msgr2=0x561db01f2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:33.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb3731eb740 1 --2- 172.21.15.44:0/3986544442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e6dd0 0x561db01f2010 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb358000c00 tx=0x7fb364005c70 comp rx=0 tx=0).stop 2024-04-21T23:06:33.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb3731eb740 1 -- 172.21.15.44:0/3986544442 shutdown_connections 2024-04-21T23:06:33.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb3731eb740 1 --2- 172.21.15.44:0/3986544442 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x561db02631d0 0x561db025f610 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb3731eb740 1 --2- 172.21.15.44:0/3986544442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561db01e6dd0 0x561db01f2010 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb3731eb740 1 --2- 172.21.15.44:0/3986544442 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561db01e1c60 0x561db01eeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb3731eb740 1 --2- 172.21.15.44:0/3986544442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561db01e1270 0x561db01eeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb351ffb640 0 client.14778 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:06:33.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb3731eb740 1 -- 172.21.15.44:0/3986544442 >> 172.21.15.44:0/3986544442 conn(0x561db01dc7e0 msgr2=0x561db01e25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:33.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb3731eb740 1 -- 172.21.15.44:0/3986544442 shutdown_connections 2024-04-21T23:06:33.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb3731eb740 1 -- 172.21.15.44:0/3986544442 wait complete. 2024-04-21T23:06:33.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb3731eb740 20 client.14778 trim_cache size 0 max 16384 2024-04-21T23:06:33.080 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:06:33.080 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "80909b94-0032-11ef-bc93-c7b262605968", 2024-04-21T23:06:33.080 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-04-21T23:06:33.081 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24530, 2024-04-21T23:06:33.081 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-04-21T23:06:33.081 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-04-21T23:06:33.081 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-04-21T23:06:33.081 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 6.8660948980000001, 2024-04-21T23:06:33.081 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 17, 2024-04-21T23:06:33.081 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 90, 2024-04-21T23:06:33.081 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 90, 2024-04-21T23:06:33.081 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 10.953513752999999 2024-04-21T23:06:33.081 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:06:33.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb36f39f640 1 -- 172.21.15.44:0/2066141406 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb3540a2c10 msgr2=0x7fb3540a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:33.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb36f39f640 1 --2- 172.21.15.44:0/2066141406 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb3540a2c10 0x7fb3540a50d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb3600ba1f0 tx=0x7fb3600908e0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb36f39f640 1 -- 172.21.15.44:0/2066141406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb368103180 msgr2=0x7fb36819ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:33.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb36f39f640 1 --2- 172.21.15.44:0/2066141406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb368103180 0x7fb36819ee50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb360037040 tx=0x7fb360032f20 comp rx=0 tx=0).stop 2024-04-21T23:06:33.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb36f39f640 1 -- 172.21.15.44:0/2066141406 shutdown_connections 2024-04-21T23:06:33.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb36f39f640 1 --2- 172.21.15.44:0/2066141406 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb3540a2c10 0x7fb3540a50d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb36f39f640 1 --2- 172.21.15.44:0/2066141406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3681027c0 0x7fb368197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb36f39f640 1 --2- 172.21.15.44:0/2066141406 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb368103180 0x7fb36819ee50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb36f39f640 1 --2- 172.21.15.44:0/2066141406 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3681015d0 0x7fb3681978a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:33.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb36f39f640 1 -- 172.21.15.44:0/2066141406 >> 172.21.15.44:0/2066141406 conn(0x7fb3680fcb40 msgr2=0x7fb368105ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:33.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.073+0000 7fb36f39f640 1 -- 172.21.15.44:0/2066141406 shutdown_connections 2024-04-21T23:06:33.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:33.074+0000 7fb36f39f640 1 -- 172.21.15.44:0/2066141406 wait complete. 2024-04-21T23:06:33.084 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-21T23:06:33.084 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:06:33.084 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-21T23:06:33.331 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:06:33.620 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:33 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3357783561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:33.620 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:33 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2847743148' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:33.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:33 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3357783561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:33.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:33 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2847743148' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:33.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:33 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3357783561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:33.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:33 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2847743148' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:34.670 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:34 smithi044 ceph-mon[27934]: pgmap v295: 129 pgs: 3 creating+peering, 123 active+clean, 3 unknown; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 637 B/s wr, 1 op/s 2024-04-21T23:06:34.671 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:34 smithi044 ceph-mon[27934]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T23:06:34.671 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:34 smithi044 ceph-mon[27934]: Cluster is now healthy 2024-04-21T23:06:34.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:34 smithi055 ceph-mon[31117]: pgmap v295: 129 pgs: 3 creating+peering, 123 active+clean, 3 unknown; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 637 B/s wr, 1 op/s 2024-04-21T23:06:34.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:34 smithi055 ceph-mon[31117]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T23:06:34.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:34 smithi055 ceph-mon[31117]: Cluster is now healthy 2024-04-21T23:06:34.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:34 smithi069 ceph-mon[31816]: pgmap v295: 129 pgs: 3 creating+peering, 123 active+clean, 3 unknown; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 637 B/s wr, 1 op/s 2024-04-21T23:06:34.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:34 smithi069 ceph-mon[31816]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T23:06:34.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:34 smithi069 ceph-mon[31816]: Cluster is now healthy 2024-04-21T23:06:35.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.180+0000 7f2411809700 1 -- 172.21.15.44:0/3140447971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240c102ec0 msgr2=0x7f240c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:35.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.180+0000 7f2411809700 1 --2- 172.21.15.44:0/3140447971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240c102ec0 0x7f240c106f00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f24000099c0 tx=0x7f2400009cd0 comp rx=0 tx=0).stop 2024-04-21T23:06:35.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.180+0000 7f2411809700 1 -- 172.21.15.44:0/3140447971 shutdown_connections 2024-04-21T23:06:35.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.180+0000 7f2411809700 1 --2- 172.21.15.44:0/3140447971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240c1075e0 0x7f240c10f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:35.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.180+0000 7f2411809700 1 --2- 172.21.15.44:0/3140447971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f240c102510 0x7f240c1028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:35.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.180+0000 7f2411809700 1 --2- 172.21.15.44:0/3140447971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240c102ec0 0x7f240c106f00 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:35.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.180+0000 7f2411809700 1 -- 172.21.15.44:0/3140447971 >> 172.21.15.44:0/3140447971 conn(0x7f240c0793e0 msgr2=0x7f240c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:35.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.180+0000 7f2411809700 1 -- 172.21.15.44:0/3140447971 shutdown_connections 2024-04-21T23:06:35.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.180+0000 7f2411809700 1 -- 172.21.15.44:0/3140447971 wait complete. 2024-04-21T23:06:35.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.181+0000 7f2411809700 1 Processor -- start 2024-04-21T23:06:35.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.181+0000 7f2411809700 1 -- start start 2024-04-21T23:06:35.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.181+0000 7f2411809700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240c102510 0x7f240c074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:35.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.181+0000 7f2411809700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f240c102ec0 0x7f240c074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:35.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.181+0000 7f2411809700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240c1075e0 0x7f240c074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:35.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.181+0000 7f2411809700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f240c1125a0 con 0x7f240c1075e0 2024-04-21T23:06:35.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.181+0000 7f2411809700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f240c112420 con 0x7f240c102ec0 2024-04-21T23:06:35.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.181+0000 7f2411809700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f240c112720 con 0x7f240c102510 2024-04-21T23:06:35.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.181+0000 7f240affd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240c102510 0x7f240c074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:35.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.181+0000 7f240b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240c1075e0 0x7f240c074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:35.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.181+0000 7f240affd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240c102510 0x7f240c074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:49240/0 (socket says 172.21.15.44:49240) 2024-04-21T23:06:35.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.181+0000 7f240affd700 1 -- 172.21.15.44:0/2975808585 learned_addr learned my addr 172.21.15.44:0/2975808585 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:35.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.181+0000 7f240a7fc700 1 --2- 172.21.15.44:0/2975808585 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f240c102ec0 0x7f240c074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:35.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.182+0000 7f240b7fe700 1 -- 172.21.15.44:0/2975808585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240c102510 msgr2=0x7f240c074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:35.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.182+0000 7f240b7fe700 1 --2- 172.21.15.44:0/2975808585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240c102510 0x7f240c074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:35.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.182+0000 7f240b7fe700 1 -- 172.21.15.44:0/2975808585 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f240c102ec0 msgr2=0x7f240c074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:35.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.182+0000 7f240b7fe700 1 --2- 172.21.15.44:0/2975808585 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f240c102ec0 0x7f240c074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:35.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.182+0000 7f240b7fe700 1 -- 172.21.15.44:0/2975808585 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f240c1a6f00 con 0x7f240c1075e0 2024-04-21T23:06:35.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.182+0000 7f240a7fc700 1 --2- 172.21.15.44:0/2975808585 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f240c102ec0 0x7f240c074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:06:35.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.182+0000 7f240b7fe700 1 --2- 172.21.15.44:0/2975808585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240c1075e0 0x7f240c074fb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f240400cc50 tx=0x7f240400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:35.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.182+0000 7f2410807700 1 -- 172.21.15.44:0/2975808585 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2404016020 con 0x7f240c1075e0 2024-04-21T23:06:35.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.182+0000 7f2410807700 1 -- 172.21.15.44:0/2975808585 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f240400ab40 con 0x7f240c1075e0 2024-04-21T23:06:35.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.182+0000 7f2411809700 1 -- 172.21.15.44:0/2975808585 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f240c1a71f0 con 0x7f240c1075e0 2024-04-21T23:06:35.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.182+0000 7f2410807700 1 -- 172.21.15.44:0/2975808585 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24040115e0 con 0x7f240c1075e0 2024-04-21T23:06:35.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.182+0000 7f2411809700 1 -- 172.21.15.44:0/2975808585 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f240c1a76b0 con 0x7f240c1075e0 2024-04-21T23:06:35.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.183+0000 7f2411809700 1 -- 172.21.15.44:0/2975808585 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f240c068f20 con 0x7f240c1075e0 2024-04-21T23:06:35.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.184+0000 7f2410807700 1 -- 172.21.15.44:0/2975808585 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2404019020 con 0x7f240c1075e0 2024-04-21T23:06:35.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.185+0000 7f2410807700 1 --2- 172.21.15.44:0/2975808585 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f23f40a2c20 0x7f23f40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:35.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.185+0000 7f240affd700 1 --2- 172.21.15.44:0/2975808585 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f23f40a2c20 0x7f23f40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:35.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.185+0000 7f2410807700 1 -- 172.21.15.44:0/2975808585 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f24040cf5b0 con 0x7f240c1075e0 2024-04-21T23:06:35.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.185+0000 7f240affd700 1 --2- 172.21.15.44:0/2975808585 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f23f40a2c20 0x7f23f40a50e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f23fc007c90 tx=0x7f23fc007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:35.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.189+0000 7f2410807700 1 -- 172.21.15.44:0/2975808585 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2404020020 con 0x7f240c1075e0 2024-04-21T23:06:35.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.441+0000 7f2411809700 1 -- 172.21.15.44:0/2975808585 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f240c1a7940 con 0x7f240c1075e0 2024-04-21T23:06:35.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.441+0000 7f2410807700 1 -- 172.21.15.44:0/2975808585 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f2404022300 con 0x7f240c1075e0 2024-04-21T23:06:35.440 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-21T23:06:35.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.443+0000 7f2411809700 1 -- 172.21.15.44:0/2975808585 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f23f40a2c20 msgr2=0x7f23f40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:35.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.443+0000 7f2411809700 1 --2- 172.21.15.44:0/2975808585 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f23f40a2c20 0x7f23f40a50e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f23fc007c90 tx=0x7f23fc007480 comp rx=0 tx=0).stop 2024-04-21T23:06:35.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.443+0000 7f2411809700 1 -- 172.21.15.44:0/2975808585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240c1075e0 msgr2=0x7f240c074fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:35.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.443+0000 7f2411809700 1 --2- 172.21.15.44:0/2975808585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240c1075e0 0x7f240c074fb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f240400cc50 tx=0x7f240400cf60 comp rx=0 tx=0).stop 2024-04-21T23:06:35.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.443+0000 7f2411809700 1 -- 172.21.15.44:0/2975808585 shutdown_connections 2024-04-21T23:06:35.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.443+0000 7f2411809700 1 --2- 172.21.15.44:0/2975808585 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f23f40a2c20 0x7f23f40a50e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:35.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.443+0000 7f2411809700 1 --2- 172.21.15.44:0/2975808585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f240c1075e0 0x7f240c074fb0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:35.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.443+0000 7f2411809700 1 --2- 172.21.15.44:0/2975808585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f240c102510 0x7f240c074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:35.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.443+0000 7f2411809700 1 --2- 172.21.15.44:0/2975808585 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f240c102ec0 0x7f240c074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:35.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.443+0000 7f2411809700 1 -- 172.21.15.44:0/2975808585 >> 172.21.15.44:0/2975808585 conn(0x7f240c0793e0 msgr2=0x7f240c100220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:35.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.443+0000 7f2411809700 1 -- 172.21.15.44:0/2975808585 shutdown_connections 2024-04-21T23:06:35.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:35.443+0000 7f2411809700 1 -- 172.21.15.44:0/2975808585 wait complete. 2024-04-21T23:06:36.068 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-21T23:06:36.068 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-21T23:06:36.068 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-21T23:06:36.068 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-21T23:06:36.069 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:06:36.069 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-21T23:06:36.069 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-21T23:06:36.069 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-21T23:06:36.069 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-21T23:06:36.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:06:36.069 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-21T23:06:36.086 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-21T23:06:36.087 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-21T23:06:36.087 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-21T23:06:36.087 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-21T23:06:36.087 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-21T23:06:36.087 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-21T23:06:36.087 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T23:06:36.087 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-21T23:06:36.087 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-21T23:06:36.087 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T23:06:36.087 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-21T23:06:36.087 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-21T23:06:36.087 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-21T23:06:36.087 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-21T23:06:36.088 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-21T23:06:36.088 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-21T23:06:36.088 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 42169sec preferred_lft 42169sec 2024-04-21T23:06:36.088 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-21T23:06:36.088 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-21T23:06:36.088 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T23:06:36.088 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-21T23:06:36.088 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-21T23:06:36.089 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-21T23:06:36.089 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:06:36.089 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-21T23:06:36.089 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add name ceph-brx type bridge 2024-04-21T23:06:36.089 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip addr flush dev ceph-brx 2024-04-21T23:06:36.089 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set ceph-brx up 2024-04-21T23:06:36.089 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-21T23:06:36.089 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-21T23:06:36.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:06:36.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:06:36.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:06:36.284 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-21T23:06:36.290 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:36 smithi044 ceph-mon[27934]: pgmap v296: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-04-21T23:06:36.291 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:36 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2975808585' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T23:06:36.309 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-21T23:06:36.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:06:36.310 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-21T23:06:36.400 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-21T23:06:36.400 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-21T23:06:36.400 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-21T23:06:36.400 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-21T23:06:36.400 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-21T23:06:36.400 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:06:36.400 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-21T23:06:36.401 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-21T23:06:36.401 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-21T23:06:36.401 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-21T23:06:36.401 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-21T23:06:36.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:06:36.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:06:36.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:06:36.640 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-21T23:06:36.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:06:36.656 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list-id 2024-04-21T23:06:36.712 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:06:36.712 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-21T23:06:36.712 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T23:06:36.712 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-21T23:06:36.712 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-21T23:06:36.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:36 smithi055 ceph-mon[31117]: pgmap v296: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-04-21T23:06:36.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:36 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2975808585' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T23:06:36.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:36 smithi069 ceph-mon[31816]: pgmap v296: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-04-21T23:06:36.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:36 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2975808585' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T23:06:36.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:06:36.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:06:36.832 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-21T23:06:36.832 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:06:36.832 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-21T23:06:36.832 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-21T23:06:36.832 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-21T23:06:36.832 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-21T23:06:36.832 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-21T23:06:36.832 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-21T23:06:36.832 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-21T23:06:36.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:06:36.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:06:36.977 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:06:36.977 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-21T23:06:36.977 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-04-21T23:06:36.977 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set dev brx.0 master ceph-brx 2024-04-21T23:06:36.978 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-21T23:06:37.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:06:37.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:06:37.055 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-21T23:06:37.055 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T23:06:37.055 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:06:37.070 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-21T23:06:37.070 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T23:06:37.070 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:06:37.125 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-21T23:06:37.191 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-21T23:06:37.247 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-21T23:06:37.248 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-21T23:06:37.248 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-21T23:06:37.248 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-21T23:06:37.248 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-21T23:06:37.248 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-21T23:06:37.248 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-21T23:06:37.248 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-21T23:06:37.248 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-21T23:06:37.248 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-21T23:06:37.248 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-21T23:06:37.248 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-21T23:06:37.248 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T23:06:37.248 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-21T23:06:37.249 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-21T23:06:37.249 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-21T23:06:37.249 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-21T23:06:37.249 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysusers.service 2024-04-21T23:06:37.249 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-21T23:06:37.249 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-21T23:06:37.249 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-21T23:06:37.249 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-21T23:06:37.249 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-21T23:06:37.249 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-21T23:06:37.249 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-21T23:06:37.249 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-21T23:06:37.249 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay 2024-04-21T23:06:37.249 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/650ed9f13e3907146542019a1289620b2bfcc1ac323caee7c6abd6a66f82700a/merged 2024-04-21T23:06:37.250 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/64637c1396637e86290a2ec6f1dcfdc3e43b4c15747d275b7e0cbba0264821f5/merged 2024-04-21T23:06:37.250 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/0 2024-04-21T23:06:37.250 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T23:06:37.250 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/08fa679ca5c5e4c9222236bdd85cb0314e308387719ca11e26ff44ebc1203556/merged 2024-04-21T23:06:37.250 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/0dbb1c212601680377af7b48a3112ef2b2b96c69aeee02aa3f8e1955a539309a/merged 2024-04-21T23:06:37.250 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/cf0c9a3a63744a3ec72ed4450bb9f435d1921ab38c69ba45cebab1529372a381/merged 2024-04-21T23:06:37.250 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/a31ef6488e01b7cc866c21c3561ba05e701335ece5dab48f01c81d7ab155d43d/merged 2024-04-21T23:06:37.250 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/c07aeb9b00a213ff04368c23cbb7e0503e25378c747f32db9c63377659c8dc10/merged 2024-04-21T23:06:37.250 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/8b08b7c8b9037b7ec5b7b5a868fb2d7ae87c2997f599107299e5a8307a1dfa9f/merged 2024-04-21T23:06:37.250 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/89d4b1c0ca7195499cd99996d2191ec186680ded39c972d97c2f639bf7775348/merged 2024-04-21T23:06:37.250 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/051f7a66ac58b3e158ac9fda7cac97663389b73061eece6cf8c3b4920b4a159e/merged 2024-04-21T23:06:37.250 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-21T23:06:37.250 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T23:06:37.250 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T23:06:37.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:06:37.251 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-21T23:06:37.303 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-21T23:06:37.303 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-21T23:06:37.345 DEBUG:teuthology.orchestra.run.smithi044:> sudo modprobe fuse 2024-04-21T23:06:37.369 DEBUG:teuthology.orchestra.run.smithi044:> cat /proc/self/mounts | awk '{print $2}' 2024-04-21T23:06:37.426 INFO:teuthology.orchestra.run.smithi044.stdout:/proc 2024-04-21T23:06:37.426 INFO:teuthology.orchestra.run.smithi044.stdout:/sys 2024-04-21T23:06:37.426 INFO:teuthology.orchestra.run.smithi044.stdout:/dev 2024-04-21T23:06:37.426 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/security 2024-04-21T23:06:37.426 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/shm 2024-04-21T23:06:37.426 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/pts 2024-04-21T23:06:37.426 INFO:teuthology.orchestra.run.smithi044.stdout:/run 2024-04-21T23:06:37.426 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/cgroup 2024-04-21T23:06:37.426 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/pstore 2024-04-21T23:06:37.426 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/bpf 2024-04-21T23:06:37.426 INFO:teuthology.orchestra.run.smithi044.stdout:/ 2024-04-21T23:06:37.427 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/selinux 2024-04-21T23:06:37.427 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T23:06:37.427 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/hugepages 2024-04-21T23:06:37.427 INFO:teuthology.orchestra.run.smithi044.stdout:/dev/mqueue 2024-04-21T23:06:37.427 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/debug 2024-04-21T23:06:37.427 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/tracing 2024-04-21T23:06:37.427 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysusers.service 2024-04-21T23:06:37.427 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/kernel/config 2024-04-21T23:06:37.427 INFO:teuthology.orchestra.run.smithi044.stdout:/sys/fs/fuse/connections 2024-04-21T23:06:37.427 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-sysctl.service 2024-04-21T23:06:37.427 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-21T23:06:37.427 INFO:teuthology.orchestra.run.smithi044.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-21T23:06:37.427 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/nfs/rpc_pipefs 2024-04-21T23:06:37.427 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/1000 2024-04-21T23:06:37.427 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/ceph 2024-04-21T23:06:37.427 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay 2024-04-21T23:06:37.428 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/650ed9f13e3907146542019a1289620b2bfcc1ac323caee7c6abd6a66f82700a/merged 2024-04-21T23:06:37.428 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/64637c1396637e86290a2ec6f1dcfdc3e43b4c15747d275b7e0cbba0264821f5/merged 2024-04-21T23:06:37.428 INFO:teuthology.orchestra.run.smithi044.stdout:/run/user/0 2024-04-21T23:06:37.428 INFO:teuthology.orchestra.run.smithi044.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T23:06:37.428 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/08fa679ca5c5e4c9222236bdd85cb0314e308387719ca11e26ff44ebc1203556/merged 2024-04-21T23:06:37.428 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/0dbb1c212601680377af7b48a3112ef2b2b96c69aeee02aa3f8e1955a539309a/merged 2024-04-21T23:06:37.428 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/cf0c9a3a63744a3ec72ed4450bb9f435d1921ab38c69ba45cebab1529372a381/merged 2024-04-21T23:06:37.428 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/a31ef6488e01b7cc866c21c3561ba05e701335ece5dab48f01c81d7ab155d43d/merged 2024-04-21T23:06:37.428 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/c07aeb9b00a213ff04368c23cbb7e0503e25378c747f32db9c63377659c8dc10/merged 2024-04-21T23:06:37.428 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/8b08b7c8b9037b7ec5b7b5a868fb2d7ae87c2997f599107299e5a8307a1dfa9f/merged 2024-04-21T23:06:37.428 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/89d4b1c0ca7195499cd99996d2191ec186680ded39c972d97c2f639bf7775348/merged 2024-04-21T23:06:37.428 INFO:teuthology.orchestra.run.smithi044.stdout:/var/lib/containers/storage/overlay/051f7a66ac58b3e158ac9fda7cac97663389b73061eece6cf8c3b4920b4a159e/merged 2024-04-21T23:06:37.428 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns 2024-04-21T23:06:37.428 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T23:06:37.429 INFO:teuthology.orchestra.run.smithi044.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T23:06:37.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:06:37.429 DEBUG:teuthology.orchestra.run.smithi044:> ls /sys/fs/fuse/connections 2024-04-21T23:06:37.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi044.stderr:2024-04-21T23:06:37.442+0000 7fc5d883c480 -1 init, newargv = 0x5582ec872270 newargc=15 2024-04-21T23:06:37.441 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi044.stderr:ceph-fuse[69227]: starting ceph client 2024-04-21T23:06:37.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi044.stderr:ceph-fuse[69227]: starting fuse 2024-04-21T23:06:37.456 INFO:teuthology.orchestra.run.smithi044.stdout:115 2024-04-21T23:06:37.457 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [115] 2024-04-21T23:06:37.457 DEBUG:teuthology.orchestra.run.smithi044:> sudo stdin-killer -- python3 -c ' 2024-04-21T23:06:37.457 DEBUG:teuthology.orchestra.run.smithi044:> import glob 2024-04-21T23:06:37.457 DEBUG:teuthology.orchestra.run.smithi044:> import re 2024-04-21T23:06:37.457 DEBUG:teuthology.orchestra.run.smithi044:> import os 2024-04-21T23:06:37.457 DEBUG:teuthology.orchestra.run.smithi044:> import subprocess 2024-04-21T23:06:37.457 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-21T23:06:37.457 DEBUG:teuthology.orchestra.run.smithi044:> def _find_admin_socket(client_name): 2024-04-21T23:06:37.457 DEBUG:teuthology.orchestra.run.smithi044:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-21T23:06:37.457 DEBUG:teuthology.orchestra.run.smithi044:> files = glob.glob(asok_path) 2024-04-21T23:06:37.457 DEBUG:teuthology.orchestra.run.smithi044:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-21T23:06:37.457 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-21T23:06:37.458 DEBUG:teuthology.orchestra.run.smithi044:> # Given a non-glob path, it better be there 2024-04-21T23:06:37.458 DEBUG:teuthology.orchestra.run.smithi044:> if "*" not in asok_path: 2024-04-21T23:06:37.458 DEBUG:teuthology.orchestra.run.smithi044:> assert(len(files) == 1) 2024-04-21T23:06:37.458 DEBUG:teuthology.orchestra.run.smithi044:> return files[0] 2024-04-21T23:06:37.458 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-21T23:06:37.458 DEBUG:teuthology.orchestra.run.smithi044:> for f in files: 2024-04-21T23:06:37.458 DEBUG:teuthology.orchestra.run.smithi044:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-21T23:06:37.458 DEBUG:teuthology.orchestra.run.smithi044:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-21T23:06:37.458 DEBUG:teuthology.orchestra.run.smithi044:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-21T23:06:37.458 DEBUG:teuthology.orchestra.run.smithi044:> contents = proc_f.read() 2024-04-21T23:06:37.458 DEBUG:teuthology.orchestra.run.smithi044:> if mountpoint in contents: 2024-04-21T23:06:37.458 DEBUG:teuthology.orchestra.run.smithi044:> return f 2024-04-21T23:06:37.458 DEBUG:teuthology.orchestra.run.smithi044:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-21T23:06:37.458 DEBUG:teuthology.orchestra.run.smithi044:> 2024-04-21T23:06:37.458 DEBUG:teuthology.orchestra.run.smithi044:> print(_find_admin_socket("client.admin")) 2024-04-21T23:06:37.458 DEBUG:teuthology.orchestra.run.smithi044:> ' 2024-04-21T23:06:37.576 INFO:teuthology.orchestra.run.smithi044.stdout:/var/run/ceph/ceph-client.admin.69227.asok 2024-04-21T23:06:37.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:06:37.586 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.69227.asok 2024-04-21T23:06:37.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:06:37.587 DEBUG:teuthology.orchestra.run.smithi044:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.69227.asok status 2024-04-21T23:06:37.723 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:06:37.723 INFO:teuthology.orchestra.run.smithi044.stdout: "metadata": { 2024-04-21T23:06:37.723 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_sha1": "820a2f8ced66152ed3d765c31444b8076ea28c6b", 2024-04-21T23:06:37.723 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_version": "ceph version 19.0.0-2405-g820a2f8c (820a2f8ced66152ed3d765c31444b8076ea28c6b) squid (dev)", 2024-04-21T23:06:37.723 INFO:teuthology.orchestra.run.smithi044.stdout: "entity_id": "admin", 2024-04-21T23:06:37.723 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": "smithi044", 2024-04-21T23:06:37.723 INFO:teuthology.orchestra.run.smithi044.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-21T23:06:37.723 INFO:teuthology.orchestra.run.smithi044.stdout: "pid": "69227", 2024-04-21T23:06:37.723 INFO:teuthology.orchestra.run.smithi044.stdout: "root": "/" 2024-04-21T23:06:37.723 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:06:37.723 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_count": 0, 2024-04-21T23:06:37.723 INFO:teuthology.orchestra.run.smithi044.stdout: "dentry_pinned_count": 0, 2024-04-21T23:06:37.723 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 14790, 2024-04-21T23:06:37.724 INFO:teuthology.orchestra.run.smithi044.stdout: "inst": { 2024-04-21T23:06:37.724 INFO:teuthology.orchestra.run.smithi044.stdout: "name": { 2024-04-21T23:06:37.724 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "client", 2024-04-21T23:06:37.724 INFO:teuthology.orchestra.run.smithi044.stdout: "num": 14790 2024-04-21T23:06:37.724 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:06:37.724 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-21T23:06:37.724 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-21T23:06:37.724 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-21T23:06:37.724 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 1168313435 2024-04-21T23:06:37.724 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:06:37.724 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:06:37.724 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": { 2024-04-21T23:06:37.724 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-21T23:06:37.724 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "192.168.0.1:0", 2024-04-21T23:06:37.725 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 1168313435 2024-04-21T23:06:37.725 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:06:37.725 INFO:teuthology.orchestra.run.smithi044.stdout: "inst_str": "client.14790 192.168.0.1:0/1168313435", 2024-04-21T23:06:37.725 INFO:teuthology.orchestra.run.smithi044.stdout: "addr_str": "192.168.0.1:0/1168313435", 2024-04-21T23:06:37.725 INFO:teuthology.orchestra.run.smithi044.stdout: "inode_count": 1, 2024-04-21T23:06:37.725 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_epoch": 17, 2024-04-21T23:06:37.725 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch": 90, 2024-04-21T23:06:37.725 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_epoch_barrier": 0, 2024-04-21T23:06:37.725 INFO:teuthology.orchestra.run.smithi044.stdout: "blocklisted": false, 2024-04-21T23:06:37.725 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs" 2024-04-21T23:06:37.725 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:06:37.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:06:37.731 DEBUG:teuthology.orchestra.run.smithi044:> stat --file-system '--printf=%T 2024-04-21T23:06:37.731 DEBUG:teuthology.orchestra.run.smithi044:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:06:37.747 INFO:teuthology.orchestra.run.smithi044.stdout:fuseblk 2024-04-21T23:06:37.747 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:06:37.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:06:37.747 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:06:37.813 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-21T23:06:37.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:37 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T23:06:37.926 INFO:teuthology.orchestra.run.smithi044.stdout:# file: . 2024-04-21T23:06:37.926 INFO:teuthology.orchestra.run.smithi044.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-21T23:06:37.926 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:37.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:06:37.932 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T23:06:37.932 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T23:06:37.949 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:06:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.093+0000 7f7de7102640 1 -- 172.21.15.44:0/3411565857 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7de0101890 msgr2=0x7f7de0101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.093+0000 7f7de7102640 1 --2- 172.21.15.44:0/3411565857 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7de0101890 0x7f7de0101d10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009900 tx=0x7f7dcc02bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.093+0000 7f7de7102640 1 -- 172.21.15.44:0/3411565857 shutdown_connections 2024-04-21T23:06:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.093+0000 7f7de7102640 1 --2- 172.21.15.44:0/3411565857 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7de0102250 0x7f7de0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.093+0000 7f7de7102640 1 --2- 172.21.15.44:0/3411565857 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7de0101890 0x7f7de0101d10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.093+0000 7f7de7102640 1 --2- 172.21.15.44:0/3411565857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7de01006a0 0x7f7de0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.093+0000 7f7de7102640 1 -- 172.21.15.44:0/3411565857 >> 172.21.15.44:0/3411565857 conn(0x7f7de00fbc50 msgr2=0x7f7de00fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.093+0000 7f7de7102640 1 -- 172.21.15.44:0/3411565857 shutdown_connections 2024-04-21T23:06:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.093+0000 7f7de7102640 1 -- 172.21.15.44:0/3411565857 wait complete. 2024-04-21T23:06:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.094+0000 7f7de7102640 1 Processor -- start 2024-04-21T23:06:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.094+0000 7f7de7102640 1 -- start start 2024-04-21T23:06:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.094+0000 7f7de7102640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7de01006a0 0x7f7de010b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.094+0000 7f7de7102640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7de0101890 0x7f7de0109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.094+0000 7f7de7102640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7de0102250 0x7f7de0109ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.094+0000 7f7de7102640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7de010d4a0 con 0x7f7de0102250 2024-04-21T23:06:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.094+0000 7f7de7102640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7de010d320 con 0x7f7de01006a0 2024-04-21T23:06:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.094+0000 7f7de7102640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7de010d620 con 0x7f7de0101890 2024-04-21T23:06:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.094+0000 7f7de569f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7de0102250 0x7f7de0109ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.094+0000 7f7de569f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7de0102250 0x7f7de0109ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41532/0 (socket says 172.21.15.44:41532) 2024-04-21T23:06:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.094+0000 7f7de569f640 1 -- 172.21.15.44:0/646915892 learned_addr learned my addr 172.21.15.44:0/646915892 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.094+0000 7f7ddffff640 1 --2- 172.21.15.44:0/646915892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7de0101890 0x7f7de0109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.094+0000 7f7de4e9e640 1 --2- 172.21.15.44:0/646915892 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7de01006a0 0x7f7de010b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.094+0000 7f7de569f640 1 -- 172.21.15.44:0/646915892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7de0101890 msgr2=0x7f7de0109660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.094+0000 7f7de569f640 1 --2- 172.21.15.44:0/646915892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7de0101890 0x7f7de0109660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.094+0000 7f7de569f640 1 -- 172.21.15.44:0/646915892 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7de01006a0 msgr2=0x7f7de010b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.094+0000 7f7de569f640 1 --2- 172.21.15.44:0/646915892 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7de01006a0 0x7f7de010b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.094+0000 7f7de569f640 1 -- 172.21.15.44:0/646915892 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7de010a430 con 0x7f7de0102250 2024-04-21T23:06:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.095+0000 7f7de569f640 1 --2- 172.21.15.44:0/646915892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7de0102250 0x7f7de0109ba0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7dd800eb10 tx=0x7f7dd800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.095+0000 7f7dddffb640 1 -- 172.21.15.44:0/646915892 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd8004b20 con 0x7f7de0102250 2024-04-21T23:06:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.095+0000 7f7de7102640 1 -- 172.21.15.44:0/646915892 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7de01a1ba0 con 0x7f7de0102250 2024-04-21T23:06:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.095+0000 7f7dddffb640 1 -- 172.21.15.44:0/646915892 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dd8004cc0 con 0x7f7de0102250 2024-04-21T23:06:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.095+0000 7f7dddffb640 1 -- 172.21.15.44:0/646915892 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd8010760 con 0x7f7de0102250 2024-04-21T23:06:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.095+0000 7f7de7102640 1 -- 172.21.15.44:0/646915892 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7de01a20c0 con 0x7f7de0102250 2024-04-21T23:06:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.096+0000 7f7de7102640 1 -- 172.21.15.44:0/646915892 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7de0068a40 con 0x7f7de0102250 2024-04-21T23:06:38.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.097+0000 7f7dddffb640 1 -- 172.21.15.44:0/646915892 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7dd8005020 con 0x7f7de0102250 2024-04-21T23:06:38.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.098+0000 7f7dddffb640 1 --2- 172.21.15.44:0/646915892 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f7dd00a2b90 0x7f7dd00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:38.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.098+0000 7f7de4e9e640 1 --2- 172.21.15.44:0/646915892 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f7dd00a2b90 0x7f7dd00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:38.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.098+0000 7f7dddffb640 1 -- 172.21.15.44:0/646915892 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f7dd80cf980 con 0x7f7de0102250 2024-04-21T23:06:38.098 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.098+0000 7f7de4e9e640 1 --2- 172.21.15.44:0/646915892 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f7dd00a2b90 0x7f7dd00a5050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4005fd0 tx=0x7f7dd4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:38.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.103+0000 7f7dddffb640 1 -- 172.21.15.44:0/646915892 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7dd809c640 con 0x7f7de0102250 2024-04-21T23:06:38.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.290+0000 7f7de7102640 1 -- 172.21.15.44:0/646915892 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7de0062340 con 0x7f7de0102250 2024-04-21T23:06:38.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.292+0000 7f7dddffb640 1 -- 172.21.15.44:0/646915892 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24032 (secure 0 0 0) 0x7f7dd809bf40 con 0x7f7de0102250 2024-04-21T23:06:38.291 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:38.291 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":90,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:06:30.619155+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:06:38.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.294+0000 7f7de7102640 1 -- 172.21.15.44:0/646915892 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f7dd00a2b90 msgr2=0x7f7dd00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:38.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.294+0000 7f7de7102640 1 --2- 172.21.15.44:0/646915892 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f7dd00a2b90 0x7f7dd00a5050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4005fd0 tx=0x7f7dd4005850 comp rx=0 tx=0).stop 2024-04-21T23:06:38.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.294+0000 7f7de7102640 1 -- 172.21.15.44:0/646915892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7de0102250 msgr2=0x7f7de0109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:38.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.294+0000 7f7de7102640 1 --2- 172.21.15.44:0/646915892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7de0102250 0x7f7de0109ba0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7dd800eb10 tx=0x7f7dd800c2d0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.294+0000 7f7de7102640 1 -- 172.21.15.44:0/646915892 shutdown_connections 2024-04-21T23:06:38.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.294+0000 7f7de7102640 1 --2- 172.21.15.44:0/646915892 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f7dd00a2b90 0x7f7dd00a5050 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.294+0000 7f7de7102640 1 --2- 172.21.15.44:0/646915892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7de0101890 0x7f7de0109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.294+0000 7f7de7102640 1 --2- 172.21.15.44:0/646915892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7de0102250 0x7f7de0109ba0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.294+0000 7f7de7102640 1 --2- 172.21.15.44:0/646915892 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7de01006a0 0x7f7de010b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.294+0000 7f7de7102640 1 -- 172.21.15.44:0/646915892 >> 172.21.15.44:0/646915892 conn(0x7f7de00fbc50 msgr2=0x7f7de0104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:38.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.294+0000 7f7de7102640 1 -- 172.21.15.44:0/646915892 shutdown_connections 2024-04-21T23:06:38.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.294+0000 7f7de7102640 1 -- 172.21.15.44:0/646915892 wait complete. 2024-04-21T23:06:38.304 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:01:47.507503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:06:23.993760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:06:24.954661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:06:28.677618+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:06:38.304 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:06:38.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.411+0000 7fcfac1c9640 1 -- 172.21.15.44:0/3131114580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa4102240 msgr2=0x7fcfa4108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:38.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.411+0000 7fcfac1c9640 1 --2- 172.21.15.44:0/3131114580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa4102240 0x7fcfa4108ad0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c009ca0 tx=0x7fcf9c02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.411+0000 7fcfac1c9640 1 -- 172.21.15.44:0/3131114580 shutdown_connections 2024-04-21T23:06:38.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.411+0000 7fcfac1c9640 1 --2- 172.21.15.44:0/3131114580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa4101880 0x7fcfa4101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.411+0000 7fcfac1c9640 1 --2- 172.21.15.44:0/3131114580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa4102240 0x7fcfa4108ad0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.411+0000 7fcfac1c9640 1 --2- 172.21.15.44:0/3131114580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfa4100690 0x7fcfa4100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.411+0000 7fcfac1c9640 1 -- 172.21.15.44:0/3131114580 >> 172.21.15.44:0/3131114580 conn(0x7fcfa40fbc20 msgr2=0x7fcfa40fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:38.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.411+0000 7fcfac1c9640 1 -- 172.21.15.44:0/3131114580 shutdown_connections 2024-04-21T23:06:38.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.411+0000 7fcfac1c9640 1 -- 172.21.15.44:0/3131114580 wait complete. 2024-04-21T23:06:38.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.412+0000 7fcfac1c9640 1 Processor -- start 2024-04-21T23:06:38.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.412+0000 7fcfac1c9640 1 -- start start 2024-04-21T23:06:38.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.412+0000 7fcfac1c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa4100690 0x7fcfa4196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:38.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.412+0000 7fcfac1c9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfa4101880 0x7fcfa4196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:38.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.412+0000 7fcfac1c9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa4102240 0x7fcfa419df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:38.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.412+0000 7fcfac1c9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa410c310 con 0x7fcfa4102240 2024-04-21T23:06:38.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.412+0000 7fcfac1c9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa410c190 con 0x7fcfa4101880 2024-04-21T23:06:38.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.412+0000 7fcfac1c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa410c490 con 0x7fcfa4100690 2024-04-21T23:06:38.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.412+0000 7fcfaa766640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa4102240 0x7fcfa419df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:38.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.412+0000 7fcfaa766640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa4102240 0x7fcfa419df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41550/0 (socket says 172.21.15.44:41550) 2024-04-21T23:06:38.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.412+0000 7fcfaa766640 1 -- 172.21.15.44:0/1586577246 learned_addr learned my addr 172.21.15.44:0/1586577246 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:38.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.412+0000 7fcfa9f65640 1 --2- 172.21.15.44:0/1586577246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa4100690 0x7fcfa4196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:38.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.412+0000 7fcfa9764640 1 --2- 172.21.15.44:0/1586577246 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfa4101880 0x7fcfa4196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:38.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.412+0000 7fcfaa766640 1 -- 172.21.15.44:0/1586577246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa4100690 msgr2=0x7fcfa4196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:38.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.412+0000 7fcfaa766640 1 --2- 172.21.15.44:0/1586577246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa4100690 0x7fcfa4196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.412+0000 7fcfaa766640 1 -- 172.21.15.44:0/1586577246 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfa4101880 msgr2=0x7fcfa4196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:38.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.412+0000 7fcfaa766640 1 --2- 172.21.15.44:0/1586577246 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfa4101880 0x7fcfa4196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.412+0000 7fcfaa766640 1 -- 172.21.15.44:0/1586577246 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa419e580 con 0x7fcfa4102240 2024-04-21T23:06:38.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.412+0000 7fcfaa766640 1 --2- 172.21.15.44:0/1586577246 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa4102240 0x7fcfa419df10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c02c700 tx=0x7fcf9c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:38.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.413+0000 7fcf9affd640 1 -- 172.21.15.44:0/1586577246 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c039040 con 0x7fcfa4102240 2024-04-21T23:06:38.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.413+0000 7fcf9affd640 1 -- 172.21.15.44:0/1586577246 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf9c044070 con 0x7fcfa4102240 2024-04-21T23:06:38.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.413+0000 7fcfac1c9640 1 -- 172.21.15.44:0/1586577246 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa419e810 con 0x7fcfa4102240 2024-04-21T23:06:38.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.413+0000 7fcf9affd640 1 -- 172.21.15.44:0/1586577246 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c0042a0 con 0x7fcfa4102240 2024-04-21T23:06:38.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.413+0000 7fcfac1c9640 1 -- 172.21.15.44:0/1586577246 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa419ed50 con 0x7fcfa4102240 2024-04-21T23:06:38.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.414+0000 7fcfac1c9640 1 -- 172.21.15.44:0/1586577246 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfa4107950 con 0x7fcfa4102240 2024-04-21T23:06:38.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.415+0000 7fcf9affd640 1 -- 172.21.15.44:0/1586577246 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fcf9c003970 con 0x7fcfa4102240 2024-04-21T23:06:38.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.415+0000 7fcf9affd640 1 --2- 172.21.15.44:0/1586577246 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcf900a2c10 0x7fcf900a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:38.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.416+0000 7fcfa9f65640 1 --2- 172.21.15.44:0/1586577246 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcf900a2c10 0x7fcf900a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:38.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.416+0000 7fcf9affd640 1 -- 172.21.15.44:0/1586577246 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fcf9c0f1de0 con 0x7fcfa4102240 2024-04-21T23:06:38.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.416+0000 7fcfa9f65640 1 --2- 172.21.15.44:0/1586577246 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcf900a2c10 0x7fcf900a50d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcfa41016e0 tx=0x7fcfa000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:38.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.419+0000 7fcf9affd640 1 -- 172.21.15.44:0/1586577246 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fcf9c0beb30 con 0x7fcfa4102240 2024-04-21T23:06:38.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.603+0000 7fcfac1c9640 1 -- 172.21.15.44:0/1586577246 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcfa402d300 con 0x7fcfa4102240 2024-04-21T23:06:38.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.604+0000 7fcf9affd640 1 -- 172.21.15.44:0/1586577246 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7fcf9c0be430 con 0x7fcfa4102240 2024-04-21T23:06:38.603 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-21T23:06:38.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.606+0000 7fcfac1c9640 1 -- 172.21.15.44:0/1586577246 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcf900a2c10 msgr2=0x7fcf900a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:38.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.606+0000 7fcfac1c9640 1 --2- 172.21.15.44:0/1586577246 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcf900a2c10 0x7fcf900a50d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcfa41016e0 tx=0x7fcfa000b390 comp rx=0 tx=0).stop 2024-04-21T23:06:38.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.606+0000 7fcfac1c9640 1 -- 172.21.15.44:0/1586577246 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa4102240 msgr2=0x7fcfa419df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:38.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.606+0000 7fcfac1c9640 1 --2- 172.21.15.44:0/1586577246 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa4102240 0x7fcfa419df10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c02c700 tx=0x7fcf9c032f20 comp rx=0 tx=0).stop 2024-04-21T23:06:38.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.606+0000 7fcfac1c9640 1 -- 172.21.15.44:0/1586577246 shutdown_connections 2024-04-21T23:06:38.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.606+0000 7fcfac1c9640 1 --2- 172.21.15.44:0/1586577246 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcf900a2c10 0x7fcf900a50d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.606+0000 7fcfac1c9640 1 --2- 172.21.15.44:0/1586577246 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfa4102240 0x7fcfa419df10 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.606+0000 7fcfac1c9640 1 --2- 172.21.15.44:0/1586577246 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfa4101880 0x7fcfa4196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.606+0000 7fcfac1c9640 1 --2- 172.21.15.44:0/1586577246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa4100690 0x7fcfa4196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.606+0000 7fcfac1c9640 1 -- 172.21.15.44:0/1586577246 >> 172.21.15.44:0/1586577246 conn(0x7fcfa40fbc20 msgr2=0x7fcfa4104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:38.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.606+0000 7fcfac1c9640 1 -- 172.21.15.44:0/1586577246 shutdown_connections 2024-04-21T23:06:38.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.606+0000 7fcfac1c9640 1 -- 172.21.15.44:0/1586577246 wait complete. 2024-04-21T23:06:38.614 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:38 smithi044 ceph-mon[27934]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 393 B/s wr, 0 op/s 2024-04-21T23:06:38.615 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:06:38.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:38 smithi055 ceph-mon[31117]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 393 B/s wr, 0 op/s 2024-04-21T23:06:38.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:38 smithi069 ceph-mon[31816]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 393 B/s wr, 0 op/s 2024-04-21T23:06:38.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.722+0000 7f4c42427640 1 -- 172.21.15.44:0/3250100049 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c3c102230 msgr2=0x7f4c3c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:38.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.722+0000 7f4c42427640 1 --2- 172.21.15.44:0/3250100049 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c3c102230 0x7f4c3c108ac0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4c34009ca0 tx=0x7f4c3402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.722+0000 7f4c42427640 1 -- 172.21.15.44:0/3250100049 shutdown_connections 2024-04-21T23:06:38.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.722+0000 7f4c42427640 1 --2- 172.21.15.44:0/3250100049 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c3c101870 0x7f4c3c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.722+0000 7f4c42427640 1 --2- 172.21.15.44:0/3250100049 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c3c102230 0x7f4c3c108ac0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.722+0000 7f4c42427640 1 --2- 172.21.15.44:0/3250100049 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c3c100680 0x7f4c3c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.722+0000 7f4c42427640 1 -- 172.21.15.44:0/3250100049 >> 172.21.15.44:0/3250100049 conn(0x7f4c3c0fbc10 msgr2=0x7f4c3c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:38.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.722+0000 7f4c42427640 1 -- 172.21.15.44:0/3250100049 shutdown_connections 2024-04-21T23:06:38.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.722+0000 7f4c42427640 1 -- 172.21.15.44:0/3250100049 wait complete. 2024-04-21T23:06:38.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.722+0000 7f4c42427640 1 Processor -- start 2024-04-21T23:06:38.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.722+0000 7f4c42427640 1 -- start start 2024-04-21T23:06:38.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.723+0000 7f4c42427640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c3c100680 0x7f4c3c197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:38.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.723+0000 7f4c42427640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c3c101870 0x7f4c3c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:38.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.723+0000 7f4c42427640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c3c102230 0x7f4c3c19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:38.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.723+0000 7f4c42427640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3c10c2e0 con 0x7f4c3c101870 2024-04-21T23:06:38.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.723+0000 7f4c42427640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3c10c160 con 0x7f4c3c100680 2024-04-21T23:06:38.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.723+0000 7f4c42427640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3c10c460 con 0x7f4c3c102230 2024-04-21T23:06:38.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.723+0000 7f4c3b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c3c101870 0x7f4c3c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:38.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.723+0000 7f4c3b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c3c101870 0x7f4c3c198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41574/0 (socket says 172.21.15.44:41574) 2024-04-21T23:06:38.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.723+0000 7f4c3b7fe640 1 -- 172.21.15.44:0/917785678 learned_addr learned my addr 172.21.15.44:0/917785678 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:38.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.723+0000 7f4c3bfff640 1 --2- 172.21.15.44:0/917785678 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c3c100680 0x7f4c3c197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:38.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.723+0000 7f4c409c4640 1 --2- 172.21.15.44:0/917785678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c3c102230 0x7f4c3c19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:38.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.723+0000 7f4c3b7fe640 1 -- 172.21.15.44:0/917785678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c3c102230 msgr2=0x7f4c3c19f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:38.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.723+0000 7f4c3b7fe640 1 --2- 172.21.15.44:0/917785678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c3c102230 0x7f4c3c19f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.723+0000 7f4c3b7fe640 1 -- 172.21.15.44:0/917785678 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c3c100680 msgr2=0x7f4c3c197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:38.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.723+0000 7f4c3b7fe640 1 --2- 172.21.15.44:0/917785678 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c3c100680 0x7f4c3c197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.723+0000 7f4c3b7fe640 1 -- 172.21.15.44:0/917785678 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c3c19f720 con 0x7f4c3c101870 2024-04-21T23:06:38.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.723+0000 7f4c3bfff640 1 --2- 172.21.15.44:0/917785678 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c3c100680 0x7f4c3c197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:06:38.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.723+0000 7f4c3b7fe640 1 --2- 172.21.15.44:0/917785678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c3c101870 0x7f4c3c198040 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4c3000d950 tx=0x7f4c3000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:38.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.724+0000 7f4c397fa640 1 -- 172.21.15.44:0/917785678 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c30017030 con 0x7f4c3c101870 2024-04-21T23:06:38.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.724+0000 7f4c397fa640 1 -- 172.21.15.44:0/917785678 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c30004020 con 0x7f4c3c101870 2024-04-21T23:06:38.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.724+0000 7f4c42427640 1 -- 172.21.15.44:0/917785678 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c3c19fa10 con 0x7f4c3c101870 2024-04-21T23:06:38.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.724+0000 7f4c397fa640 1 -- 172.21.15.44:0/917785678 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c300047a0 con 0x7f4c3c101870 2024-04-21T23:06:38.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.724+0000 7f4c42427640 1 -- 172.21.15.44:0/917785678 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c3c19ffb0 con 0x7f4c3c101870 2024-04-21T23:06:38.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.725+0000 7f4c42427640 1 -- 172.21.15.44:0/917785678 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c3c1a0240 con 0x7f4c3c101870 2024-04-21T23:06:38.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.726+0000 7f4c397fa640 1 -- 172.21.15.44:0/917785678 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4c300041d0 con 0x7f4c3c101870 2024-04-21T23:06:38.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.726+0000 7f4c397fa640 1 --2- 172.21.15.44:0/917785678 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f4c240a2c10 0x7f4c240a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:38.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.727+0000 7f4c3bfff640 1 --2- 172.21.15.44:0/917785678 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f4c240a2c10 0x7f4c240a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:38.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.727+0000 7f4c397fa640 1 -- 172.21.15.44:0/917785678 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f4c300cebf0 con 0x7f4c3c101870 2024-04-21T23:06:38.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.727+0000 7f4c3bfff640 1 --2- 172.21.15.44:0/917785678 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f4c240a2c10 0x7f4c240a50d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c1016d0 tx=0x7f4c2c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:38.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.731+0000 7f4c397fa640 1 -- 172.21.15.44:0/917785678 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4c3009b910 con 0x7f4c3c101870 2024-04-21T23:06:38.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.916+0000 7f4c42427640 1 -- 172.21.15.44:0/917785678 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4c3c1a0410 con 0x7f4c3c101870 2024-04-21T23:06:38.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.917+0000 7f4c397fa640 1 -- 172.21.15.44:0/917785678 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f4c3009b210 con 0x7f4c3c101870 2024-04-21T23:06:38.916 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:06:38.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.919+0000 7f4c42427640 1 -- 172.21.15.44:0/917785678 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f4c240a2c10 msgr2=0x7f4c240a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:38.917 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.919+0000 7f4c42427640 1 --2- 172.21.15.44:0/917785678 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f4c240a2c10 0x7f4c240a50d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c1016d0 tx=0x7f4c2c005f90 comp rx=0 tx=0).stop 2024-04-21T23:06:38.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.919+0000 7f4c42427640 1 -- 172.21.15.44:0/917785678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c3c101870 msgr2=0x7f4c3c198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:38.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.919+0000 7f4c42427640 1 --2- 172.21.15.44:0/917785678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c3c101870 0x7f4c3c198040 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4c3000d950 tx=0x7f4c3000de10 comp rx=0 tx=0).stop 2024-04-21T23:06:38.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.919+0000 7f4c42427640 1 -- 172.21.15.44:0/917785678 shutdown_connections 2024-04-21T23:06:38.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.919+0000 7f4c42427640 1 --2- 172.21.15.44:0/917785678 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f4c240a2c10 0x7f4c240a50d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.919+0000 7f4c42427640 1 --2- 172.21.15.44:0/917785678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c3c102230 0x7f4c3c19f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.919+0000 7f4c42427640 1 --2- 172.21.15.44:0/917785678 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c3c101870 0x7f4c3c198040 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.919+0000 7f4c42427640 1 --2- 172.21.15.44:0/917785678 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c3c100680 0x7f4c3c197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:38.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.919+0000 7f4c42427640 1 -- 172.21.15.44:0/917785678 >> 172.21.15.44:0/917785678 conn(0x7f4c3c0fbc10 msgr2=0x7f4c3c0fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:38.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.919+0000 7f4c42427640 1 -- 172.21.15.44:0/917785678 shutdown_connections 2024-04-21T23:06:38.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:38.919+0000 7f4c42427640 1 -- 172.21.15.44:0/917785678 wait complete. 2024-04-21T23:06:38.928 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:06:39.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.035+0000 7f26ad4c2640 1 -- 172.21.15.44:0/3322317032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26a8102240 msgr2=0x7f26a8108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:39.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.035+0000 7f26ad4c2640 1 --2- 172.21.15.44:0/3322317032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26a8102240 0x7f26a8108ad0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f26a0009ca0 tx=0x7f26a002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.035+0000 7f26ad4c2640 1 -- 172.21.15.44:0/3322317032 shutdown_connections 2024-04-21T23:06:39.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.035+0000 7f26ad4c2640 1 --2- 172.21.15.44:0/3322317032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26a8101880 0x7f26a8101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.035+0000 7f26ad4c2640 1 --2- 172.21.15.44:0/3322317032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26a8102240 0x7f26a8108ad0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.035+0000 7f26ad4c2640 1 --2- 172.21.15.44:0/3322317032 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26a8100690 0x7f26a8100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.035+0000 7f26ad4c2640 1 -- 172.21.15.44:0/3322317032 >> 172.21.15.44:0/3322317032 conn(0x7f26a80fbc20 msgr2=0x7f26a80fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.035+0000 7f26ad4c2640 1 -- 172.21.15.44:0/3322317032 shutdown_connections 2024-04-21T23:06:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.035+0000 7f26ad4c2640 1 -- 172.21.15.44:0/3322317032 wait complete. 2024-04-21T23:06:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.036+0000 7f26ad4c2640 1 Processor -- start 2024-04-21T23:06:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.036+0000 7f26ad4c2640 1 -- start start 2024-04-21T23:06:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.036+0000 7f26ad4c2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26a8100690 0x7f26a8197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.036+0000 7f26ad4c2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26a8101880 0x7f26a8198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.036+0000 7f26ad4c2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26a8102240 0x7f26a819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.036+0000 7f26ad4c2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f26a810d430 con 0x7f26a8100690 2024-04-21T23:06:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.036+0000 7f26ad4c2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f26a810d2b0 con 0x7f26a8102240 2024-04-21T23:06:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.036+0000 7f26ad4c2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f26a810d5b0 con 0x7f26a8101880 2024-04-21T23:06:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.036+0000 7f26a6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26a8100690 0x7f26a8197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.036+0000 7f26a6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26a8100690 0x7f26a8197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41586/0 (socket says 172.21.15.44:41586) 2024-04-21T23:06:39.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.036+0000 7f26a6ffd640 1 -- 172.21.15.44:0/711123428 learned_addr learned my addr 172.21.15.44:0/711123428 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.036+0000 7f26a77fe640 1 --2- 172.21.15.44:0/711123428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26a8102240 0x7f26a819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.036+0000 7f26a67fc640 1 --2- 172.21.15.44:0/711123428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26a8101880 0x7f26a8198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.036+0000 7f26a6ffd640 1 -- 172.21.15.44:0/711123428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26a8101880 msgr2=0x7f26a8198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.036+0000 7f26a6ffd640 1 --2- 172.21.15.44:0/711123428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26a8101880 0x7f26a8198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.036+0000 7f26a6ffd640 1 -- 172.21.15.44:0/711123428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26a8102240 msgr2=0x7f26a819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.036+0000 7f26a6ffd640 1 --2- 172.21.15.44:0/711123428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26a8102240 0x7f26a819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.036+0000 7f26a6ffd640 1 -- 172.21.15.44:0/711123428 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a819f6e0 con 0x7f26a8100690 2024-04-21T23:06:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.036+0000 7f26a77fe640 1 --2- 172.21.15.44:0/711123428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26a8102240 0x7f26a819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:06:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.037+0000 7f26a6ffd640 1 --2- 172.21.15.44:0/711123428 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26a8100690 0x7f26a8197ac0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f269800e9c0 tx=0x7f269800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.037+0000 7f268ffff640 1 -- 172.21.15.44:0/711123428 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f269800cd80 con 0x7f26a8100690 2024-04-21T23:06:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.037+0000 7f268ffff640 1 -- 172.21.15.44:0/711123428 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26980035f0 con 0x7f26a8100690 2024-04-21T23:06:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.037+0000 7f268ffff640 1 -- 172.21.15.44:0/711123428 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2698010690 con 0x7f26a8100690 2024-04-21T23:06:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.037+0000 7f26ad4c2640 1 -- 172.21.15.44:0/711123428 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26a819f9d0 con 0x7f26a8100690 2024-04-21T23:06:39.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.037+0000 7f26ad4c2640 1 -- 172.21.15.44:0/711123428 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26a819fed0 con 0x7f26a8100690 2024-04-21T23:06:39.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.038+0000 7f26ad4c2640 1 -- 172.21.15.44:0/711123428 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26a8107950 con 0x7f26a8100690 2024-04-21T23:06:39.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.039+0000 7f268ffff640 1 -- 172.21.15.44:0/711123428 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f26980037a0 con 0x7f26a8100690 2024-04-21T23:06:39.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.040+0000 7f268ffff640 1 --2- 172.21.15.44:0/711123428 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f26900a2b90 0x7f26900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:39.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.040+0000 7f26a67fc640 1 --2- 172.21.15.44:0/711123428 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f26900a2b90 0x7f26900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:39.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.040+0000 7f268ffff640 1 -- 172.21.15.44:0/711123428 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f26980cf770 con 0x7f26a8100690 2024-04-21T23:06:39.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.040+0000 7f26a67fc640 1 --2- 172.21.15.44:0/711123428 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f26900a2b90 0x7f26900a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f26a8198fd0 tx=0x7f269c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:39.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.044+0000 7f268ffff640 1 -- 172.21.15.44:0/711123428 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f269809c430 con 0x7f26a8100690 2024-04-21T23:06:39.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.228+0000 7f26ad4c2640 1 -- 172.21.15.44:0/711123428 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f26a802d3d0 con 0x7f26a8100690 2024-04-21T23:06:39.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.228+0000 7f268ffff640 1 -- 172.21.15.44:0/711123428 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f269809bd30 con 0x7f26a8100690 2024-04-21T23:06:39.227 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-21T23:06:39.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.230+0000 7f26ad4c2640 1 -- 172.21.15.44:0/711123428 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f26900a2b90 msgr2=0x7f26900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:39.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.230+0000 7f26ad4c2640 1 --2- 172.21.15.44:0/711123428 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f26900a2b90 0x7f26900a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f26a8198fd0 tx=0x7f269c005f90 comp rx=0 tx=0).stop 2024-04-21T23:06:39.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.230+0000 7f26ad4c2640 1 -- 172.21.15.44:0/711123428 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26a8100690 msgr2=0x7f26a8197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:39.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.230+0000 7f26ad4c2640 1 --2- 172.21.15.44:0/711123428 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26a8100690 0x7f26a8197ac0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f269800e9c0 tx=0x7f269800ee80 comp rx=0 tx=0).stop 2024-04-21T23:06:39.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.230+0000 7f26ad4c2640 1 -- 172.21.15.44:0/711123428 shutdown_connections 2024-04-21T23:06:39.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.230+0000 7f26ad4c2640 1 --2- 172.21.15.44:0/711123428 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f26900a2b90 0x7f26900a5050 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.230+0000 7f26ad4c2640 1 --2- 172.21.15.44:0/711123428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26a8101880 0x7f26a8198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.230+0000 7f26ad4c2640 1 --2- 172.21.15.44:0/711123428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26a8102240 0x7f26a819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.230+0000 7f26ad4c2640 1 --2- 172.21.15.44:0/711123428 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26a8100690 0x7f26a8197ac0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.230+0000 7f26ad4c2640 1 -- 172.21.15.44:0/711123428 >> 172.21.15.44:0/711123428 conn(0x7f26a80fbc20 msgr2=0x7f26a8104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:39.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.230+0000 7f26ad4c2640 1 -- 172.21.15.44:0/711123428 shutdown_connections 2024-04-21T23:06:39.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.230+0000 7f26ad4c2640 1 -- 172.21.15.44:0/711123428 wait complete. 2024-04-21T23:06:39.240 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:06:39.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.345+0000 7fcddb1bf640 1 -- 172.21.15.44:0/575221350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcdd4102230 msgr2=0x7fcdd4108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:39.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.345+0000 7fcddb1bf640 1 --2- 172.21.15.44:0/575221350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcdd4102230 0x7fcdd4108ac0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4009900 tx=0x7fcdc402bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:39.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.345+0000 7fcddb1bf640 1 -- 172.21.15.44:0/575221350 shutdown_connections 2024-04-21T23:06:39.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.345+0000 7fcddb1bf640 1 --2- 172.21.15.44:0/575221350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd4101870 0x7fcdd4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.345+0000 7fcddb1bf640 1 --2- 172.21.15.44:0/575221350 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcdd4102230 0x7fcdd4108ac0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.345+0000 7fcddb1bf640 1 --2- 172.21.15.44:0/575221350 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcdd4100680 0x7fcdd4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.345+0000 7fcddb1bf640 1 -- 172.21.15.44:0/575221350 >> 172.21.15.44:0/575221350 conn(0x7fcdd40fbc10 msgr2=0x7fcdd40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.346+0000 7fcddb1bf640 1 -- 172.21.15.44:0/575221350 shutdown_connections 2024-04-21T23:06:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.346+0000 7fcddb1bf640 1 -- 172.21.15.44:0/575221350 wait complete. 2024-04-21T23:06:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.346+0000 7fcddb1bf640 1 Processor -- start 2024-04-21T23:06:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.346+0000 7fcddb1bf640 1 -- start start 2024-04-21T23:06:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.346+0000 7fcddb1bf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd4100680 0x7fcdd4104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.346+0000 7fcddb1bf640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcdd4101870 0x7fcdd4102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.346+0000 7fcddb1bf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcdd4102230 0x7fcdd4103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.346+0000 7fcddb1bf640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd410c300 con 0x7fcdd4102230 2024-04-21T23:06:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.346+0000 7fcddb1bf640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd410c180 con 0x7fcdd4101870 2024-04-21T23:06:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.346+0000 7fcddb1bf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd410c480 con 0x7fcdd4100680 2024-04-21T23:06:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.346+0000 7fcdd975c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcdd4102230 0x7fcdd4103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.346+0000 7fcdd975c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcdd4102230 0x7fcdd4103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41608/0 (socket says 172.21.15.44:41608) 2024-04-21T23:06:39.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.346+0000 7fcdd975c640 1 -- 172.21.15.44:0/174498436 learned_addr learned my addr 172.21.15.44:0/174498436 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.346+0000 7fcdd3fff640 1 --2- 172.21.15.44:0/174498436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcdd4101870 0x7fcdd4102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.346+0000 7fcdd8f5b640 1 --2- 172.21.15.44:0/174498436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd4100680 0x7fcdd4104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.347+0000 7fcdd975c640 1 -- 172.21.15.44:0/174498436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd4100680 msgr2=0x7fcdd4104610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.347+0000 7fcdd975c640 1 --2- 172.21.15.44:0/174498436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd4100680 0x7fcdd4104610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.347+0000 7fcdd975c640 1 -- 172.21.15.44:0/174498436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcdd4101870 msgr2=0x7fcdd4102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.347+0000 7fcdd975c640 1 --2- 172.21.15.44:0/174498436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcdd4101870 0x7fcdd4102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.347+0000 7fcdd975c640 1 -- 172.21.15.44:0/174498436 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd4103a20 con 0x7fcdd4102230 2024-04-21T23:06:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.347+0000 7fcdd3fff640 1 --2- 172.21.15.44:0/174498436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcdd4101870 0x7fcdd4102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:06:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.347+0000 7fcdd975c640 1 --2- 172.21.15.44:0/174498436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcdd4102230 0x7fcdd4103190 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4009230 tx=0x7fcdc4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.347+0000 7fcdd1ffb640 1 -- 172.21.15.44:0/174498436 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4041070 con 0x7fcdd4102230 2024-04-21T23:06:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.347+0000 7fcddb1bf640 1 -- 172.21.15.44:0/174498436 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdd41a1bf0 con 0x7fcdd4102230 2024-04-21T23:06:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.347+0000 7fcdd1ffb640 1 -- 172.21.15.44:0/174498436 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdc40054e0 con 0x7fcdd4102230 2024-04-21T23:06:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.347+0000 7fcdd1ffb640 1 -- 172.21.15.44:0/174498436 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4034440 con 0x7fcdd4102230 2024-04-21T23:06:39.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.347+0000 7fcddb1bf640 1 -- 172.21.15.44:0/174498436 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdd41a2190 con 0x7fcdd4102230 2024-04-21T23:06:39.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.348+0000 7fcddb1bf640 1 -- 172.21.15.44:0/174498436 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdd41a23c0 con 0x7fcdd4102230 2024-04-21T23:06:39.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.350+0000 7fcdd1ffb640 1 -- 172.21.15.44:0/174498436 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fcdc4044070 con 0x7fcdd4102230 2024-04-21T23:06:39.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.350+0000 7fcdd1ffb640 1 --2- 172.21.15.44:0/174498436 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcdc00a2b90 0x7fcdc00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:39.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.351+0000 7fcdd8f5b640 1 --2- 172.21.15.44:0/174498436 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcdc00a2b90 0x7fcdc00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:39.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.351+0000 7fcdd1ffb640 1 -- 172.21.15.44:0/174498436 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fcdc40f1250 con 0x7fcdd4102230 2024-04-21T23:06:39.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.351+0000 7fcdd8f5b640 1 --2- 172.21.15.44:0/174498436 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcdc00a2b90 0x7fcdc00a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcdd41016d0 tx=0x7fcdcc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:39.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.354+0000 7fcdd1ffb640 1 -- 172.21.15.44:0/174498436 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fcdc40bd920 con 0x7fcdd4102230 2024-04-21T23:06:39.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.539+0000 7fcddb1bf640 1 -- 172.21.15.44:0/174498436 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fcdd41a2590 con 0x7fcdd4102230 2024-04-21T23:06:39.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.540+0000 7fcdd1ffb640 1 -- 172.21.15.44:0/174498436 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7fcdc40c1c20 con 0x7fcdd4102230 2024-04-21T23:06:39.539 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:06:39.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.542+0000 7fcddb1bf640 1 -- 172.21.15.44:0/174498436 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcdc00a2b90 msgr2=0x7fcdc00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:39.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.542+0000 7fcddb1bf640 1 --2- 172.21.15.44:0/174498436 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcdc00a2b90 0x7fcdc00a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcdd41016d0 tx=0x7fcdcc009500 comp rx=0 tx=0).stop 2024-04-21T23:06:39.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.542+0000 7fcddb1bf640 1 -- 172.21.15.44:0/174498436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcdd4102230 msgr2=0x7fcdd4103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:39.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.542+0000 7fcddb1bf640 1 --2- 172.21.15.44:0/174498436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcdd4102230 0x7fcdd4103190 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4009230 tx=0x7fcdc4005d50 comp rx=0 tx=0).stop 2024-04-21T23:06:39.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.542+0000 7fcddb1bf640 1 -- 172.21.15.44:0/174498436 shutdown_connections 2024-04-21T23:06:39.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.542+0000 7fcddb1bf640 1 --2- 172.21.15.44:0/174498436 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcdc00a2b90 0x7fcdc00a5050 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.542+0000 7fcddb1bf640 1 --2- 172.21.15.44:0/174498436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcdd4102230 0x7fcdd4103190 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.542+0000 7fcddb1bf640 1 --2- 172.21.15.44:0/174498436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcdd4101870 0x7fcdd4102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.542+0000 7fcddb1bf640 1 --2- 172.21.15.44:0/174498436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd4100680 0x7fcdd4104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.542+0000 7fcddb1bf640 1 -- 172.21.15.44:0/174498436 >> 172.21.15.44:0/174498436 conn(0x7fcdd40fbc10 msgr2=0x7fcdd40fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:39.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.542+0000 7fcddb1bf640 1 -- 172.21.15.44:0/174498436 shutdown_connections 2024-04-21T23:06:39.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.542+0000 7fcddb1bf640 1 -- 172.21.15.44:0/174498436 wait complete. 2024-04-21T23:06:39.551 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:39 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/646915892' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:39.551 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:39 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1586577246' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:06:39.551 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:39 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/917785678' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:06:39.551 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:39 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/711123428' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:06:39.552 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-21T23:06:39.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.657+0000 7f92a5bf7640 1 -- 172.21.15.44:0/3652073588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92a0104b30 msgr2=0x7f92a0106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:39.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.657+0000 7f92a5bf7640 1 --2- 172.21.15.44:0/3652073588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92a0104b30 0x7f92a0106f20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9294009900 tx=0x7f929402bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.657+0000 7f92a5bf7640 1 -- 172.21.15.44:0/3652073588 shutdown_connections 2024-04-21T23:06:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.657+0000 7f92a5bf7640 1 --2- 172.21.15.44:0/3652073588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92a0107460 0x7f92a01098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.657+0000 7f92a5bf7640 1 --2- 172.21.15.44:0/3652073588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92a0104b30 0x7f92a0106f20 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.657+0000 7f92a5bf7640 1 --2- 172.21.15.44:0/3652073588 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92a0102200 0x7f92a01045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.657+0000 7f92a5bf7640 1 -- 172.21.15.44:0/3652073588 >> 172.21.15.44:0/3652073588 conn(0x7f92a00fbc10 msgr2=0x7f92a00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.657+0000 7f92a5bf7640 1 -- 172.21.15.44:0/3652073588 shutdown_connections 2024-04-21T23:06:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.657+0000 7f92a5bf7640 1 -- 172.21.15.44:0/3652073588 wait complete. 2024-04-21T23:06:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.657+0000 7f92a5bf7640 1 Processor -- start 2024-04-21T23:06:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.657+0000 7f92a5bf7640 1 -- start start 2024-04-21T23:06:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f92a5bf7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92a0102200 0x7f92a0101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f92a5bf7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92a0104b30 0x7f92a01001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f92a5bf7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92a0107460 0x7f92a01006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f92a5bf7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f92a010d400 con 0x7f92a0102200 2024-04-21T23:06:39.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f92a5bf7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f92a010d280 con 0x7f92a0104b30 2024-04-21T23:06:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f92a5bf7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f92a010d580 con 0x7f92a0107460 2024-04-21T23:06:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f929f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92a0102200 0x7f92a0101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f929f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92a0102200 0x7f92a0101b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41620/0 (socket says 172.21.15.44:41620) 2024-04-21T23:06:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f929f7fe640 1 -- 172.21.15.44:0/3964184722 learned_addr learned my addr 172.21.15.44:0/3964184722 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f929effd640 1 --2- 172.21.15.44:0/3964184722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92a0104b30 0x7f92a01001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f929ffff640 1 --2- 172.21.15.44:0/3964184722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92a0107460 0x7f92a01006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f929f7fe640 1 -- 172.21.15.44:0/3964184722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92a0107460 msgr2=0x7f92a01006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f929f7fe640 1 --2- 172.21.15.44:0/3964184722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92a0107460 0x7f92a01006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f929f7fe640 1 -- 172.21.15.44:0/3964184722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92a0104b30 msgr2=0x7f92a01001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f929f7fe640 1 --2- 172.21.15.44:0/3964184722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92a0104b30 0x7f92a01001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f929f7fe640 1 -- 172.21.15.44:0/3964184722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a0100f80 con 0x7f92a0102200 2024-04-21T23:06:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f929f7fe640 1 --2- 172.21.15.44:0/3964184722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92a0102200 0x7f92a0101b70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9290009d50 tx=0x7f929000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f929cff9640 1 -- 172.21.15.44:0/3964184722 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290003b70 con 0x7f92a0102200 2024-04-21T23:06:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f92a5bf7640 1 -- 172.21.15.44:0/3964184722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92a01a1bf0 con 0x7f92a0102200 2024-04-21T23:06:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f929cff9640 1 -- 172.21.15.44:0/3964184722 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9290003d10 con 0x7f92a0102200 2024-04-21T23:06:39.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f929cff9640 1 -- 172.21.15.44:0/3964184722 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290010a20 con 0x7f92a0102200 2024-04-21T23:06:39.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.658+0000 7f92a5bf7640 1 -- 172.21.15.44:0/3964184722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92a01a2160 con 0x7f92a0102200 2024-04-21T23:06:39.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.659+0000 7f92a5bf7640 1 -- 172.21.15.44:0/3964184722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92a00fd700 con 0x7f92a0102200 2024-04-21T23:06:39.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.661+0000 7f929cff9640 1 -- 172.21.15.44:0/3964184722 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9290010bc0 con 0x7f92a0102200 2024-04-21T23:06:39.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.661+0000 7f929cff9640 1 --2- 172.21.15.44:0/3964184722 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f92880a2b90 0x7f92880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:39.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.661+0000 7f929effd640 1 --2- 172.21.15.44:0/3964184722 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f92880a2b90 0x7f92880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:39.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.661+0000 7f929cff9640 1 -- 172.21.15.44:0/3964184722 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f92900cfac0 con 0x7f92a0102200 2024-04-21T23:06:39.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.662+0000 7f929effd640 1 --2- 172.21.15.44:0/3964184722 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f92880a2b90 0x7f92880a5050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9294000c00 tx=0x7f9294005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:39.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.664+0000 7f929cff9640 1 -- 172.21.15.44:0/3964184722 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f929009c870 con 0x7f92a0102200 2024-04-21T23:06:39.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:39 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/646915892' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:39.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:39 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1586577246' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:06:39.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:39 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/917785678' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:06:39.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:39 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/711123428' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:06:39.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:39 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/646915892' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:39.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:39 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1586577246' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:06:39.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:39 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/917785678' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:06:39.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:39 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/711123428' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:06:39.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.862+0000 7f92a5bf7640 1 -- 172.21.15.44:0/3964184722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f92a002d170 con 0x7f92a0102200 2024-04-21T23:06:39.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.862+0000 7f929cff9640 1 -- 172.21.15.44:0/3964184722 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+590 (secure 0 0 0) 0x7f92a002d170 con 0x7f92a0102200 2024-04-21T23:06:39.861 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2729363939 2024-04-22T22:59:22.969731+0000 2024-04-21T23:06:39.862 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3462209685 2024-04-22T22:59:22.969731+0000 2024-04-21T23:06:39.862 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/4238737803 2024-04-22T22:59:22.969731+0000 2024-04-21T23:06:39.862 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/4238737803 2024-04-22T22:59:22.969731+0000 2024-04-21T23:06:39.862 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/2900331072 2024-04-22T22:58:56.511296+0000 2024-04-21T23:06:39.862 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/908290973 2024-04-22T22:58:56.511296+0000 2024-04-21T23:06:39.862 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2994566338 2024-04-22T22:58:56.511296+0000 2024-04-21T23:06:39.862 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/2900331072 2024-04-22T22:58:56.511296+0000 2024-04-21T23:06:39.862 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/693229699 2024-04-22T22:58:56.511296+0000 2024-04-21T23:06:39.862 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1931984406 2024-04-22T22:59:22.969731+0000 2024-04-21T23:06:39.862 INFO:teuthology.orchestra.run.smithi044.stderr:listed 10 entries 2024-04-21T23:06:39.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.865+0000 7f92a5bf7640 1 -- 172.21.15.44:0/3964184722 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f92880a2b90 msgr2=0x7f92880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:39.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.865+0000 7f92a5bf7640 1 --2- 172.21.15.44:0/3964184722 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f92880a2b90 0x7f92880a5050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9294000c00 tx=0x7f9294005fd0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.865+0000 7f92a5bf7640 1 -- 172.21.15.44:0/3964184722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92a0102200 msgr2=0x7f92a0101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:39.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.865+0000 7f92a5bf7640 1 --2- 172.21.15.44:0/3964184722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92a0102200 0x7f92a0101b70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9290009d50 tx=0x7f929000c6a0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.865+0000 7f92a5bf7640 1 -- 172.21.15.44:0/3964184722 shutdown_connections 2024-04-21T23:06:39.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.865+0000 7f92a5bf7640 1 --2- 172.21.15.44:0/3964184722 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f92880a2b90 0x7f92880a5050 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.865+0000 7f92a5bf7640 1 --2- 172.21.15.44:0/3964184722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92a0107460 0x7f92a01006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.865+0000 7f92a5bf7640 1 --2- 172.21.15.44:0/3964184722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92a0104b30 0x7f92a01001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.865+0000 7f92a5bf7640 1 --2- 172.21.15.44:0/3964184722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92a0102200 0x7f92a0101b70 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:39.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.865+0000 7f92a5bf7640 1 -- 172.21.15.44:0/3964184722 >> 172.21.15.44:0/3964184722 conn(0x7f92a00fbc10 msgr2=0x7f92a0102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:39.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.865+0000 7f92a5bf7640 1 -- 172.21.15.44:0/3964184722 shutdown_connections 2024-04-21T23:06:39.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:39.865+0000 7f92a5bf7640 1 -- 172.21.15.44:0/3964184722 wait complete. 2024-04-21T23:06:39.875 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T23:06:39.875 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T23:06:39.891 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:06:40.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.035+0000 7f170718c640 1 -- 172.21.15.44:0/2388807977 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1700101890 msgr2=0x7f1700101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:40.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.035+0000 7f170718c640 1 --2- 172.21.15.44:0/2388807977 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1700101890 0x7f1700101d10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009900 tx=0x7f16ec02bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:40.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.035+0000 7f170718c640 1 -- 172.21.15.44:0/2388807977 shutdown_connections 2024-04-21T23:06:40.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.035+0000 7f170718c640 1 --2- 172.21.15.44:0/2388807977 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1700102250 0x7f1700108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.035+0000 7f170718c640 1 --2- 172.21.15.44:0/2388807977 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1700101890 0x7f1700101d10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.035+0000 7f170718c640 1 --2- 172.21.15.44:0/2388807977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17001006a0 0x7f1700100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.035+0000 7f170718c640 1 -- 172.21.15.44:0/2388807977 >> 172.21.15.44:0/2388807977 conn(0x7f17000fbc10 msgr2=0x7f17000fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:40.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.035+0000 7f170718c640 1 -- 172.21.15.44:0/2388807977 shutdown_connections 2024-04-21T23:06:40.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.035+0000 7f170718c640 1 -- 172.21.15.44:0/2388807977 wait complete. 2024-04-21T23:06:40.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.035+0000 7f170718c640 1 Processor -- start 2024-04-21T23:06:40.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.035+0000 7f170718c640 1 -- start start 2024-04-21T23:06:40.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.035+0000 7f170718c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17001006a0 0x7f1700197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:40.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.035+0000 7f170718c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1700101890 0x7f1700197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:40.037 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.035+0000 7f170718c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1700102250 0x7f170019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:40.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.035+0000 7f170718c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f170010d440 con 0x7f1700102250 2024-04-21T23:06:40.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.035+0000 7f170718c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f170010d2c0 con 0x7f1700101890 2024-04-21T23:06:40.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.035+0000 7f170718c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f170010d5c0 con 0x7f17001006a0 2024-04-21T23:06:40.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.036+0000 7f1705729640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1700102250 0x7f170019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:40.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.036+0000 7f1705729640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1700102250 0x7f170019f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41640/0 (socket says 172.21.15.44:41640) 2024-04-21T23:06:40.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.036+0000 7f1705729640 1 -- 172.21.15.44:0/2182146107 learned_addr learned my addr 172.21.15.44:0/2182146107 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:40.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.036+0000 7f16fffff640 1 --2- 172.21.15.44:0/2182146107 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1700101890 0x7f1700197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:40.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.036+0000 7f1704f28640 1 --2- 172.21.15.44:0/2182146107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17001006a0 0x7f1700197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:40.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.036+0000 7f1705729640 1 -- 172.21.15.44:0/2182146107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17001006a0 msgr2=0x7f1700197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:40.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.036+0000 7f1705729640 1 --2- 172.21.15.44:0/2182146107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17001006a0 0x7f1700197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.036+0000 7f1705729640 1 -- 172.21.15.44:0/2182146107 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1700101890 msgr2=0x7f1700197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:40.038 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.036+0000 7f1705729640 1 --2- 172.21.15.44:0/2182146107 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1700101890 0x7f1700197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.036+0000 7f1705729640 1 -- 172.21.15.44:0/2182146107 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170019f6b0 con 0x7f1700102250 2024-04-21T23:06:40.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.036+0000 7f16fffff640 1 --2- 172.21.15.44:0/2182146107 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1700101890 0x7f1700197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:06:40.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.036+0000 7f1704f28640 1 --2- 172.21.15.44:0/2182146107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17001006a0 0x7f1700197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:06:40.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.036+0000 7f1705729640 1 --2- 172.21.15.44:0/2182146107 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1700102250 0x7f170019f040 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f16f800cad0 tx=0x7f16f800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:40.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.036+0000 7f16fdffb640 1 -- 172.21.15.44:0/2182146107 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f800ae20 con 0x7f1700102250 2024-04-21T23:06:40.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.036+0000 7f170718c640 1 -- 172.21.15.44:0/2182146107 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f170019f9a0 con 0x7f1700102250 2024-04-21T23:06:40.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.036+0000 7f16fdffb640 1 -- 172.21.15.44:0/2182146107 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16f8005020 con 0x7f1700102250 2024-04-21T23:06:40.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.036+0000 7f16fdffb640 1 -- 172.21.15.44:0/2182146107 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f800f6f0 con 0x7f1700102250 2024-04-21T23:06:40.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.036+0000 7f170718c640 1 -- 172.21.15.44:0/2182146107 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f170019fea0 con 0x7f1700102250 2024-04-21T23:06:40.039 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.037+0000 7f170718c640 1 -- 172.21.15.44:0/2182146107 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17001079c0 con 0x7f1700102250 2024-04-21T23:06:40.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.038+0000 7f16fdffb640 1 -- 172.21.15.44:0/2182146107 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f16f80051d0 con 0x7f1700102250 2024-04-21T23:06:40.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.039+0000 7f16fdffb640 1 --2- 172.21.15.44:0/2182146107 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f16f00a2b90 0x7f16f00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:40.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.039+0000 7f16fdffb640 1 -- 172.21.15.44:0/2182146107 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f16f80cf3f0 con 0x7f1700102250 2024-04-21T23:06:40.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.039+0000 7f1704f28640 1 --2- 172.21.15.44:0/2182146107 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f16f00a2b90 0x7f16f00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:40.040 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.040+0000 7f1704f28640 1 --2- 172.21.15.44:0/2182146107 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f16f00a2b90 0x7f16f00a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f17001016f0 tx=0x7f16f4005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:40.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.043+0000 7f16fdffb640 1 -- 172.21.15.44:0/2182146107 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f16f8014030 con 0x7f1700102250 2024-04-21T23:06:40.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.229+0000 7f170718c640 1 -- 172.21.15.44:0/2182146107 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1700062340 con 0x7f1700102250 2024-04-21T23:06:40.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.230+0000 7f16fdffb640 1 -- 172.21.15.44:0/2182146107 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24032 (secure 0 0 0) 0x7f16f809c110 con 0x7f1700102250 2024-04-21T23:06:40.229 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:40.229 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":90,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:06:30.619155+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:06:40.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.232+0000 7f170718c640 1 -- 172.21.15.44:0/2182146107 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f16f00a2b90 msgr2=0x7f16f00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:40.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.232+0000 7f170718c640 1 --2- 172.21.15.44:0/2182146107 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f16f00a2b90 0x7f16f00a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f17001016f0 tx=0x7f16f4005e70 comp rx=0 tx=0).stop 2024-04-21T23:06:40.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.232+0000 7f170718c640 1 -- 172.21.15.44:0/2182146107 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1700102250 msgr2=0x7f170019f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:40.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.232+0000 7f170718c640 1 --2- 172.21.15.44:0/2182146107 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1700102250 0x7f170019f040 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f16f800cad0 tx=0x7f16f800cf90 comp rx=0 tx=0).stop 2024-04-21T23:06:40.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.232+0000 7f170718c640 1 -- 172.21.15.44:0/2182146107 shutdown_connections 2024-04-21T23:06:40.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.232+0000 7f170718c640 1 --2- 172.21.15.44:0/2182146107 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f16f00a2b90 0x7f16f00a5050 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.232+0000 7f170718c640 1 --2- 172.21.15.44:0/2182146107 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1700102250 0x7f170019f040 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.232+0000 7f170718c640 1 --2- 172.21.15.44:0/2182146107 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1700101890 0x7f1700197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.232+0000 7f170718c640 1 --2- 172.21.15.44:0/2182146107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17001006a0 0x7f1700197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.232+0000 7f170718c640 1 -- 172.21.15.44:0/2182146107 >> 172.21.15.44:0/2182146107 conn(0x7f17000fbc10 msgr2=0x7f1700104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:40.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.232+0000 7f170718c640 1 -- 172.21.15.44:0/2182146107 shutdown_connections 2024-04-21T23:06:40.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.233+0000 7f170718c640 1 -- 172.21.15.44:0/2182146107 wait complete. 2024-04-21T23:06:40.243 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:01:47.507503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:06:23.993760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:06:24.954661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:06:28.677618+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:06:40.243 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:06:40.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.349+0000 7f0665ab4640 1 -- 172.21.15.44:0/1507323229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06601033f0 msgr2=0x7f0660109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:40.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.349+0000 7f0665ab4640 1 --2- 172.21.15.44:0/1507323229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06601033f0 0x7f0660109c80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0658009cd0 tx=0x7f065802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.349+0000 7f0665ab4640 1 -- 172.21.15.44:0/1507323229 shutdown_connections 2024-04-21T23:06:40.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.349+0000 7f0665ab4640 1 --2- 172.21.15.44:0/1507323229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0660102a30 0x7f0660102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.349+0000 7f0665ab4640 1 --2- 172.21.15.44:0/1507323229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06601033f0 0x7f0660109c80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.349+0000 7f0665ab4640 1 --2- 172.21.15.44:0/1507323229 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06600ffa60 0x7f06600ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.349+0000 7f0665ab4640 1 -- 172.21.15.44:0/1507323229 >> 172.21.15.44:0/1507323229 conn(0x7f06600fb370 msgr2=0x7f06600fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:40.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.349+0000 7f0665ab4640 1 -- 172.21.15.44:0/1507323229 shutdown_connections 2024-04-21T23:06:40.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.349+0000 7f0665ab4640 1 -- 172.21.15.44:0/1507323229 wait complete. 2024-04-21T23:06:40.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.350+0000 7f0665ab4640 1 Processor -- start 2024-04-21T23:06:40.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.350+0000 7f0665ab4640 1 -- start start 2024-04-21T23:06:40.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.350+0000 7f0665ab4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06600ffa60 0x7f0660197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:40.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.350+0000 7f0665ab4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0660102a30 0x7f0660197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:40.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.350+0000 7f0665ab4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06601033f0 0x7f066019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:40.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.350+0000 7f0665ab4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f066010d440 con 0x7f0660102a30 2024-04-21T23:06:40.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.350+0000 7f0665ab4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f066010d2c0 con 0x7f06600ffa60 2024-04-21T23:06:40.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.350+0000 7f0665ab4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f066010d5c0 con 0x7f06601033f0 2024-04-21T23:06:40.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.350+0000 7f065effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0660102a30 0x7f0660197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:40.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.350+0000 7f065effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0660102a30 0x7f0660197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41646/0 (socket says 172.21.15.44:41646) 2024-04-21T23:06:40.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.350+0000 7f065effd640 1 -- 172.21.15.44:0/3268745725 learned_addr learned my addr 172.21.15.44:0/3268745725 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:40.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.350+0000 7f065f7fe640 1 --2- 172.21.15.44:0/3268745725 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06600ffa60 0x7f0660197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:40.351 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.350+0000 7f065effd640 1 -- 172.21.15.44:0/3268745725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06601033f0 msgr2=0x7f066019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:40.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.350+0000 7f065ffff640 1 --2- 172.21.15.44:0/3268745725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06601033f0 0x7f066019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:40.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.350+0000 7f065effd640 1 --2- 172.21.15.44:0/3268745725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06601033f0 0x7f066019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.350+0000 7f065effd640 1 -- 172.21.15.44:0/3268745725 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06600ffa60 msgr2=0x7f0660197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:40.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.350+0000 7f065effd640 1 --2- 172.21.15.44:0/3268745725 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06600ffa60 0x7f0660197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.350+0000 7f065effd640 1 -- 172.21.15.44:0/3268745725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f066019f6d0 con 0x7f0660102a30 2024-04-21T23:06:40.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.350+0000 7f065f7fe640 1 --2- 172.21.15.44:0/3268745725 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06600ffa60 0x7f0660197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:06:40.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.350+0000 7f065ffff640 1 --2- 172.21.15.44:0/3268745725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06601033f0 0x7f066019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:06:40.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.351+0000 7f065effd640 1 --2- 172.21.15.44:0/3268745725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0660102a30 0x7f0660197ff0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f065400d770 tx=0x7f065400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:40.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.351+0000 7f065cff9640 1 -- 172.21.15.44:0/3268745725 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0654017070 con 0x7f0660102a30 2024-04-21T23:06:40.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.351+0000 7f065cff9640 1 -- 172.21.15.44:0/3268745725 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0654005500 con 0x7f0660102a30 2024-04-21T23:06:40.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.351+0000 7f0665ab4640 1 -- 172.21.15.44:0/3268745725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f066019f9c0 con 0x7f0660102a30 2024-04-21T23:06:40.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.351+0000 7f065cff9640 1 -- 172.21.15.44:0/3268745725 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0654004e00 con 0x7f0660102a30 2024-04-21T23:06:40.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.351+0000 7f0665ab4640 1 -- 172.21.15.44:0/3268745725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f066019ff10 con 0x7f0660102a30 2024-04-21T23:06:40.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.352+0000 7f0665ab4640 1 -- 172.21.15.44:0/3268745725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0660108b60 con 0x7f0660102a30 2024-04-21T23:06:40.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.353+0000 7f065cff9640 1 -- 172.21.15.44:0/3268745725 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0654004fa0 con 0x7f0660102a30 2024-04-21T23:06:40.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.354+0000 7f065cff9640 1 --2- 172.21.15.44:0/3268745725 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f06480a2c10 0x7f06480a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:40.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.354+0000 7f065f7fe640 1 --2- 172.21.15.44:0/3268745725 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f06480a2c10 0x7f06480a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:40.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.354+0000 7f065cff9640 1 -- 172.21.15.44:0/3268745725 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f06540cece0 con 0x7f0660102a30 2024-04-21T23:06:40.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.354+0000 7f065f7fe640 1 --2- 172.21.15.44:0/3268745725 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f06480a2c10 0x7f06480a50d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0660069c10 tx=0x7f065000a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:40.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.358+0000 7f065cff9640 1 -- 172.21.15.44:0/3268745725 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f065409ba00 con 0x7f0660102a30 2024-04-21T23:06:40.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.542+0000 7f0665ab4640 1 -- 172.21.15.44:0/3268745725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0660062370 con 0x7f0660102a30 2024-04-21T23:06:40.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.542+0000 7f065cff9640 1 -- 172.21.15.44:0/3268745725 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f065409b300 con 0x7f0660102a30 2024-04-21T23:06:40.541 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-21T23:06:40.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.544+0000 7f0665ab4640 1 -- 172.21.15.44:0/3268745725 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f06480a2c10 msgr2=0x7f06480a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:40.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.544+0000 7f0665ab4640 1 --2- 172.21.15.44:0/3268745725 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f06480a2c10 0x7f06480a50d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0660069c10 tx=0x7f065000a560 comp rx=0 tx=0).stop 2024-04-21T23:06:40.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.544+0000 7f0665ab4640 1 -- 172.21.15.44:0/3268745725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0660102a30 msgr2=0x7f0660197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:40.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.544+0000 7f0665ab4640 1 --2- 172.21.15.44:0/3268745725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0660102a30 0x7f0660197ff0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f065400d770 tx=0x7f065400dc30 comp rx=0 tx=0).stop 2024-04-21T23:06:40.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.544+0000 7f0665ab4640 1 -- 172.21.15.44:0/3268745725 shutdown_connections 2024-04-21T23:06:40.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.544+0000 7f0665ab4640 1 --2- 172.21.15.44:0/3268745725 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f06480a2c10 0x7f06480a50d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.544+0000 7f0665ab4640 1 --2- 172.21.15.44:0/3268745725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06601033f0 0x7f066019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.544+0000 7f0665ab4640 1 --2- 172.21.15.44:0/3268745725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0660102a30 0x7f0660197ff0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.544+0000 7f0665ab4640 1 --2- 172.21.15.44:0/3268745725 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06600ffa60 0x7f0660197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.544+0000 7f0665ab4640 1 -- 172.21.15.44:0/3268745725 >> 172.21.15.44:0/3268745725 conn(0x7f06600fb370 msgr2=0x7f0660105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:40.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.545+0000 7f0665ab4640 1 -- 172.21.15.44:0/3268745725 shutdown_connections 2024-04-21T23:06:40.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.545+0000 7f0665ab4640 1 -- 172.21.15.44:0/3268745725 wait complete. 2024-04-21T23:06:40.553 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:40 smithi044 ceph-mon[27934]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-21T23:06:40.553 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:40 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/174498436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:06:40.553 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:40 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3964184722' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:06:40.553 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:40 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2182146107' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:40.554 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:06:40.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.661+0000 7f88c150c640 1 -- 172.21.15.44:0/453740045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88bc100690 msgr2=0x7f88bc100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:40.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.661+0000 7f88c150c640 1 --2- 172.21.15.44:0/453740045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88bc100690 0x7f88bc100ab0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f88ac009900 tx=0x7f88ac02bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:40.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.661+0000 7f88c150c640 1 -- 172.21.15.44:0/453740045 shutdown_connections 2024-04-21T23:06:40.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.661+0000 7f88c150c640 1 --2- 172.21.15.44:0/453740045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88bc102240 0x7f88bc108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.661+0000 7f88c150c640 1 --2- 172.21.15.44:0/453740045 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88bc101880 0x7f88bc101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.661+0000 7f88c150c640 1 --2- 172.21.15.44:0/453740045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88bc100690 0x7f88bc100ab0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.661+0000 7f88c150c640 1 -- 172.21.15.44:0/453740045 >> 172.21.15.44:0/453740045 conn(0x7f88bc0fbc20 msgr2=0x7f88bc0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:40.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.661+0000 7f88c150c640 1 -- 172.21.15.44:0/453740045 shutdown_connections 2024-04-21T23:06:40.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.661+0000 7f88c150c640 1 -- 172.21.15.44:0/453740045 wait complete. 2024-04-21T23:06:40.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.662+0000 7f88c150c640 1 Processor -- start 2024-04-21T23:06:40.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.662+0000 7f88c150c640 1 -- start start 2024-04-21T23:06:40.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.662+0000 7f88c150c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88bc100690 0x7f88bc06a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:40.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.662+0000 7f88c150c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88bc101880 0x7f88bc19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:40.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.662+0000 7f88c150c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88bc102240 0x7f88bc19ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:40.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.662+0000 7f88c150c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc10c330 con 0x7f88bc101880 2024-04-21T23:06:40.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.662+0000 7f88c150c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc10c1b0 con 0x7f88bc100690 2024-04-21T23:06:40.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.662+0000 7f88c150c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f88bc10c4b0 con 0x7f88bc102240 2024-04-21T23:06:40.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.662+0000 7f88b3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88bc101880 0x7f88bc19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:40.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.662+0000 7f88b3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88bc101880 0x7f88bc19c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38556/0 (socket says 172.21.15.44:38556) 2024-04-21T23:06:40.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.662+0000 7f88b3fff640 1 -- 172.21.15.44:0/203078901 learned_addr learned my addr 172.21.15.44:0/203078901 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:40.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.662+0000 7f88bb7fe640 1 --2- 172.21.15.44:0/203078901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88bc102240 0x7f88bc19ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:40.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.662+0000 7f88baffd640 1 --2- 172.21.15.44:0/203078901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88bc100690 0x7f88bc06a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:40.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.662+0000 7f88b3fff640 1 -- 172.21.15.44:0/203078901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88bc102240 msgr2=0x7f88bc19ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:40.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.662+0000 7f88b3fff640 1 --2- 172.21.15.44:0/203078901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88bc102240 0x7f88bc19ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.662+0000 7f88b3fff640 1 -- 172.21.15.44:0/203078901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88bc100690 msgr2=0x7f88bc06a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:40.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.662+0000 7f88b3fff640 1 --2- 172.21.15.44:0/203078901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88bc100690 0x7f88bc06a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.662+0000 7f88b3fff640 1 -- 172.21.15.44:0/203078901 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88bc19f6f0 con 0x7f88bc101880 2024-04-21T23:06:40.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.662+0000 7f88b3fff640 1 --2- 172.21.15.44:0/203078901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88bc101880 0x7f88bc19c7c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f88a400d950 tx=0x7f88a400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:40.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.663+0000 7f88b8ff9640 1 -- 172.21.15.44:0/203078901 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a4017070 con 0x7f88bc101880 2024-04-21T23:06:40.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.663+0000 7f88b8ff9640 1 -- 172.21.15.44:0/203078901 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88a4009c30 con 0x7f88bc101880 2024-04-21T23:06:40.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.663+0000 7f88c150c640 1 -- 172.21.15.44:0/203078901 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88bc19f980 con 0x7f88bc101880 2024-04-21T23:06:40.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.663+0000 7f88b8ff9640 1 -- 172.21.15.44:0/203078901 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a400f6b0 con 0x7f88bc101880 2024-04-21T23:06:40.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.663+0000 7f88c150c640 1 -- 172.21.15.44:0/203078901 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88bc19fed0 con 0x7f88bc101880 2024-04-21T23:06:40.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.664+0000 7f88c150c640 1 -- 172.21.15.44:0/203078901 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88bc1079b0 con 0x7f88bc101880 2024-04-21T23:06:40.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.665+0000 7f88b8ff9640 1 -- 172.21.15.44:0/203078901 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f88a40109d0 con 0x7f88bc101880 2024-04-21T23:06:40.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.665+0000 7f88b8ff9640 1 --2- 172.21.15.44:0/203078901 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f88a80a2b90 0x7f88a80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:40.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.666+0000 7f88b8ff9640 1 -- 172.21.15.44:0/203078901 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f88a40cee70 con 0x7f88bc101880 2024-04-21T23:06:40.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.666+0000 7f88baffd640 1 --2- 172.21.15.44:0/203078901 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f88a80a2b90 0x7f88a80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:40.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.666+0000 7f88baffd640 1 --2- 172.21.15.44:0/203078901 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f88a80a2b90 0x7f88a80a5050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f88ac02c170 tx=0x7f88ac02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:40.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.669+0000 7f88b8ff9640 1 -- 172.21.15.44:0/203078901 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f88a4009de0 con 0x7f88bc101880 2024-04-21T23:06:40.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:40 smithi069 ceph-mon[31816]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-21T23:06:40.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:40 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/174498436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:06:40.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:40 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3964184722' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:06:40.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:40 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2182146107' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:40.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:40 smithi055 ceph-mon[31117]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-21T23:06:40.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:40 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/174498436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:06:40.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:40 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3964184722' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:06:40.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:40 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2182146107' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:40.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.852+0000 7f88c150c640 1 -- 172.21.15.44:0/203078901 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f88bc02d3c0 con 0x7f88bc101880 2024-04-21T23:06:40.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.852+0000 7f88b8ff9640 1 -- 172.21.15.44:0/203078901 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f88a409b4e0 con 0x7f88bc101880 2024-04-21T23:06:40.851 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:06:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.854+0000 7f88c150c640 1 -- 172.21.15.44:0/203078901 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f88a80a2b90 msgr2=0x7f88a80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.854+0000 7f88c150c640 1 --2- 172.21.15.44:0/203078901 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f88a80a2b90 0x7f88a80a5050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f88ac02c170 tx=0x7f88ac02df90 comp rx=0 tx=0).stop 2024-04-21T23:06:40.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.854+0000 7f88c150c640 1 -- 172.21.15.44:0/203078901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88bc101880 msgr2=0x7f88bc19c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:40.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.854+0000 7f88c150c640 1 --2- 172.21.15.44:0/203078901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88bc101880 0x7f88bc19c7c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f88a400d950 tx=0x7f88a400de10 comp rx=0 tx=0).stop 2024-04-21T23:06:40.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.854+0000 7f88c150c640 1 -- 172.21.15.44:0/203078901 shutdown_connections 2024-04-21T23:06:40.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.854+0000 7f88c150c640 1 --2- 172.21.15.44:0/203078901 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f88a80a2b90 0x7f88a80a5050 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.854+0000 7f88c150c640 1 --2- 172.21.15.44:0/203078901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88bc102240 0x7f88bc19ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.854+0000 7f88c150c640 1 --2- 172.21.15.44:0/203078901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f88bc101880 0x7f88bc19c7c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.854+0000 7f88c150c640 1 --2- 172.21.15.44:0/203078901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f88bc100690 0x7f88bc06a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.854+0000 7f88c150c640 1 -- 172.21.15.44:0/203078901 >> 172.21.15.44:0/203078901 conn(0x7f88bc0fbc20 msgr2=0x7f88bc104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:40.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.855+0000 7f88c150c640 1 -- 172.21.15.44:0/203078901 shutdown_connections 2024-04-21T23:06:40.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.855+0000 7f88c150c640 1 -- 172.21.15.44:0/203078901 wait complete. 2024-04-21T23:06:40.864 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:06:40.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.970+0000 7f0750e88640 1 -- 172.21.15.44:0/3099129411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f074c102240 msgr2=0x7f074c108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:40.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.970+0000 7f0750e88640 1 --2- 172.21.15.44:0/3099129411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f074c102240 0x7f074c108ad0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0744009ca0 tx=0x7f074402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.970+0000 7f0750e88640 1 -- 172.21.15.44:0/3099129411 shutdown_connections 2024-04-21T23:06:40.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.970+0000 7f0750e88640 1 --2- 172.21.15.44:0/3099129411 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f074c102240 0x7f074c108ad0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.970+0000 7f0750e88640 1 --2- 172.21.15.44:0/3099129411 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f074c101880 0x7f074c101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.970+0000 7f0750e88640 1 --2- 172.21.15.44:0/3099129411 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f074c100690 0x7f074c100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.970+0000 7f0750e88640 1 -- 172.21.15.44:0/3099129411 >> 172.21.15.44:0/3099129411 conn(0x7f074c0fbc20 msgr2=0x7f074c0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:40.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.970+0000 7f0750e88640 1 -- 172.21.15.44:0/3099129411 shutdown_connections 2024-04-21T23:06:40.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.970+0000 7f0750e88640 1 -- 172.21.15.44:0/3099129411 wait complete. 2024-04-21T23:06:40.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.971+0000 7f0750e88640 1 Processor -- start 2024-04-21T23:06:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.971+0000 7f0750e88640 1 -- start start 2024-04-21T23:06:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.971+0000 7f0750e88640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f074c100690 0x7f074c197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.971+0000 7f0750e88640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f074c101880 0x7f074c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.971+0000 7f0750e88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f074c102240 0x7f074c19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.971+0000 7f0750e88640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f074c10c2f0 con 0x7f074c101880 2024-04-21T23:06:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.971+0000 7f0750e88640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f074c10c170 con 0x7f074c100690 2024-04-21T23:06:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.971+0000 7f0750e88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f074c10c470 con 0x7f074c102240 2024-04-21T23:06:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.971+0000 7f0741d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f074c101880 0x7f074c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.971+0000 7f0741d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f074c101880 0x7f074c197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38568/0 (socket says 172.21.15.44:38568) 2024-04-21T23:06:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.971+0000 7f0741d9b640 1 -- 172.21.15.44:0/3937457006 learned_addr learned my addr 172.21.15.44:0/3937457006 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.971+0000 7f074ad9d640 1 --2- 172.21.15.44:0/3937457006 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f074c102240 0x7f074c19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.971+0000 7f074a59c640 1 --2- 172.21.15.44:0/3937457006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f074c100690 0x7f074c197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.971+0000 7f0741d9b640 1 -- 172.21.15.44:0/3937457006 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f074c102240 msgr2=0x7f074c19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.971+0000 7f0741d9b640 1 --2- 172.21.15.44:0/3937457006 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f074c102240 0x7f074c19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.971+0000 7f0741d9b640 1 -- 172.21.15.44:0/3937457006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f074c100690 msgr2=0x7f074c197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.971+0000 7f0741d9b640 1 --2- 172.21.15.44:0/3937457006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f074c100690 0x7f074c197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.971+0000 7f0741d9b640 1 -- 172.21.15.44:0/3937457006 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f074c19f660 con 0x7f074c101880 2024-04-21T23:06:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.971+0000 7f074a59c640 1 --2- 172.21.15.44:0/3937457006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f074c100690 0x7f074c197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:06:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.972+0000 7f0741d9b640 1 --2- 172.21.15.44:0/3937457006 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f074c101880 0x7f074c197f80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f073400d950 tx=0x7f073400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.972+0000 7f0743fff640 1 -- 172.21.15.44:0/3937457006 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0734017070 con 0x7f074c101880 2024-04-21T23:06:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.972+0000 7f0750e88640 1 -- 172.21.15.44:0/3937457006 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f074c19f950 con 0x7f074c101880 2024-04-21T23:06:40.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.972+0000 7f0743fff640 1 -- 172.21.15.44:0/3937457006 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0734009c30 con 0x7f074c101880 2024-04-21T23:06:40.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.972+0000 7f0743fff640 1 -- 172.21.15.44:0/3937457006 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073400f6b0 con 0x7f074c101880 2024-04-21T23:06:40.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.972+0000 7f0750e88640 1 -- 172.21.15.44:0/3937457006 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f074c19fea0 con 0x7f074c101880 2024-04-21T23:06:40.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.973+0000 7f0750e88640 1 -- 172.21.15.44:0/3937457006 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f074c1a00d0 con 0x7f074c101880 2024-04-21T23:06:40.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.974+0000 7f0743fff640 1 -- 172.21.15.44:0/3937457006 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f07340109d0 con 0x7f074c101880 2024-04-21T23:06:40.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.975+0000 7f0743fff640 1 --2- 172.21.15.44:0/3937457006 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f07380a2b90 0x7f07380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:40.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.975+0000 7f074a59c640 1 --2- 172.21.15.44:0/3937457006 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f07380a2b90 0x7f07380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:40.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.975+0000 7f0743fff640 1 -- 172.21.15.44:0/3937457006 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f07340ceeb0 con 0x7f074c101880 2024-04-21T23:06:40.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.975+0000 7f074a59c640 1 --2- 172.21.15.44:0/3937457006 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f07380a2b90 0x7f07380a5050 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f074c1016e0 tx=0x7f073c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:40.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:40.978+0000 7f0743fff640 1 -- 172.21.15.44:0/3937457006 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f07340fd6f0 con 0x7f074c101880 2024-04-21T23:06:41.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.163+0000 7f0750e88640 1 -- 172.21.15.44:0/3937457006 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f074c1a03c0 con 0x7f074c101880 2024-04-21T23:06:41.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.163+0000 7f0743fff640 1 -- 172.21.15.44:0/3937457006 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f0734009de0 con 0x7f074c101880 2024-04-21T23:06:41.162 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-21T23:06:41.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.165+0000 7f0750e88640 1 -- 172.21.15.44:0/3937457006 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f07380a2b90 msgr2=0x7f07380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:41.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.165+0000 7f0750e88640 1 --2- 172.21.15.44:0/3937457006 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f07380a2b90 0x7f07380a5050 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f074c1016e0 tx=0x7f073c005f90 comp rx=0 tx=0).stop 2024-04-21T23:06:41.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.165+0000 7f0750e88640 1 -- 172.21.15.44:0/3937457006 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f074c101880 msgr2=0x7f074c197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:41.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.165+0000 7f0750e88640 1 --2- 172.21.15.44:0/3937457006 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f074c101880 0x7f074c197f80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f073400d950 tx=0x7f073400de10 comp rx=0 tx=0).stop 2024-04-21T23:06:41.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.165+0000 7f0750e88640 1 -- 172.21.15.44:0/3937457006 shutdown_connections 2024-04-21T23:06:41.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.165+0000 7f0750e88640 1 --2- 172.21.15.44:0/3937457006 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f07380a2b90 0x7f07380a5050 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.165+0000 7f0750e88640 1 --2- 172.21.15.44:0/3937457006 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f074c102240 0x7f074c19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.165+0000 7f0750e88640 1 --2- 172.21.15.44:0/3937457006 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f074c101880 0x7f074c197f80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.165+0000 7f0750e88640 1 --2- 172.21.15.44:0/3937457006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f074c100690 0x7f074c197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.165+0000 7f0750e88640 1 -- 172.21.15.44:0/3937457006 >> 172.21.15.44:0/3937457006 conn(0x7f074c0fbc20 msgr2=0x7f074c069e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:41.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.165+0000 7f0750e88640 1 -- 172.21.15.44:0/3937457006 shutdown_connections 2024-04-21T23:06:41.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.165+0000 7f0750e88640 1 -- 172.21.15.44:0/3937457006 wait complete. 2024-04-21T23:06:41.175 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:06:41.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.286+0000 7fd27cbe5640 1 -- 172.21.15.44:0/610700067 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd278102990 msgr2=0x7fd278102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:41.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.286+0000 7fd27cbe5640 1 --2- 172.21.15.44:0/610700067 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd278102990 0x7fd278102e10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd26c009900 tx=0x7fd26c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:41.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.286+0000 7fd27cbe5640 1 -- 172.21.15.44:0/610700067 shutdown_connections 2024-04-21T23:06:41.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.286+0000 7fd27cbe5640 1 --2- 172.21.15.44:0/610700067 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd278103350 0x7fd278109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.286+0000 7fd27cbe5640 1 --2- 172.21.15.44:0/610700067 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd278102990 0x7fd278102e10 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.286+0000 7fd27cbe5640 1 --2- 172.21.15.44:0/610700067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2781017a0 0x7fd278101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.286+0000 7fd27cbe5640 1 -- 172.21.15.44:0/610700067 >> 172.21.15.44:0/610700067 conn(0x7fd2780fcd10 msgr2=0x7fd2780ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:41.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.286+0000 7fd27cbe5640 1 -- 172.21.15.44:0/610700067 shutdown_connections 2024-04-21T23:06:41.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.286+0000 7fd27cbe5640 1 -- 172.21.15.44:0/610700067 wait complete. 2024-04-21T23:06:41.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.287+0000 7fd27cbe5640 1 Processor -- start 2024-04-21T23:06:41.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.287+0000 7fd27cbe5640 1 -- start start 2024-04-21T23:06:41.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.287+0000 7fd27cbe5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2781017a0 0x7fd278197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:41.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.287+0000 7fd27cbe5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd278102990 0x7fd278198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:41.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.287+0000 7fd27cbe5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd278103350 0x7fd27819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:41.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.287+0000 7fd27cbe5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd27810d420 con 0x7fd278102990 2024-04-21T23:06:41.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.287+0000 7fd27cbe5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd27810d2a0 con 0x7fd2781017a0 2024-04-21T23:06:41.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.287+0000 7fd27cbe5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd27810d5a0 con 0x7fd278103350 2024-04-21T23:06:41.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.287+0000 7fd275d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd278102990 0x7fd278198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:41.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.287+0000 7fd275d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd278102990 0x7fd278198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38586/0 (socket says 172.21.15.44:38586) 2024-04-21T23:06:41.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.287+0000 7fd275d9b640 1 -- 172.21.15.44:0/2961063706 learned_addr learned my addr 172.21.15.44:0/2961063706 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:41.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.287+0000 7fd276d9d640 1 --2- 172.21.15.44:0/2961063706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd278103350 0x7fd27819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:41.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.287+0000 7fd27659c640 1 --2- 172.21.15.44:0/2961063706 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2781017a0 0x7fd278197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:41.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.287+0000 7fd275d9b640 1 -- 172.21.15.44:0/2961063706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd278103350 msgr2=0x7fd27819f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:41.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.287+0000 7fd275d9b640 1 --2- 172.21.15.44:0/2961063706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd278103350 0x7fd27819f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.287+0000 7fd275d9b640 1 -- 172.21.15.44:0/2961063706 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2781017a0 msgr2=0x7fd278197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:41.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.287+0000 7fd275d9b640 1 --2- 172.21.15.44:0/2961063706 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2781017a0 0x7fd278197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.287+0000 7fd275d9b640 1 -- 172.21.15.44:0/2961063706 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd27819f720 con 0x7fd278102990 2024-04-21T23:06:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.288+0000 7fd275d9b640 1 --2- 172.21.15.44:0/2961063706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd278102990 0x7fd278198090 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd26c009230 tx=0x7fd26c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.288+0000 7fd2677fe640 1 -- 172.21.15.44:0/2961063706 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c041070 con 0x7fd278102990 2024-04-21T23:06:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.288+0000 7fd2677fe640 1 -- 172.21.15.44:0/2961063706 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd26c0044d0 con 0x7fd278102990 2024-04-21T23:06:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.288+0000 7fd27cbe5640 1 -- 172.21.15.44:0/2961063706 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd27819f9b0 con 0x7fd278102990 2024-04-21T23:06:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.288+0000 7fd2677fe640 1 -- 172.21.15.44:0/2961063706 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c034440 con 0x7fd278102990 2024-04-21T23:06:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.288+0000 7fd27cbe5640 1 -- 172.21.15.44:0/2961063706 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd27819fea0 con 0x7fd278102990 2024-04-21T23:06:41.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.289+0000 7fd27cbe5640 1 -- 172.21.15.44:0/2961063706 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd278108b10 con 0x7fd278102990 2024-04-21T23:06:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.290+0000 7fd2677fe640 1 -- 172.21.15.44:0/2961063706 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd26c044070 con 0x7fd278102990 2024-04-21T23:06:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.291+0000 7fd2677fe640 1 --2- 172.21.15.44:0/2961063706 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd2600a2b90 0x7fd2600a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.291+0000 7fd27659c640 1 --2- 172.21.15.44:0/2961063706 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd2600a2b90 0x7fd2600a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.291+0000 7fd2677fe640 1 -- 172.21.15.44:0/2961063706 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fd26c0f0fa0 con 0x7fd278102990 2024-04-21T23:06:41.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.291+0000 7fd27659c640 1 --2- 172.21.15.44:0/2961063706 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd2600a2b90 0x7fd2600a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd2781027f0 tx=0x7fd268005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:41.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.295+0000 7fd2677fe640 1 -- 172.21.15.44:0/2961063706 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd26c0bdcc0 con 0x7fd278102990 2024-04-21T23:06:41.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:41 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3268745725' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:06:41.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:41 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/203078901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:06:41.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:41 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3937457006' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:06:41.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.482+0000 7fd27cbe5640 1 -- 172.21.15.44:0/2961063706 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd278062370 con 0x7fd278102990 2024-04-21T23:06:41.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.482+0000 7fd2677fe640 1 -- 172.21.15.44:0/2961063706 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7fd26c0bd5c0 con 0x7fd278102990 2024-04-21T23:06:41.481 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:06:41.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.485+0000 7fd27cbe5640 1 -- 172.21.15.44:0/2961063706 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd2600a2b90 msgr2=0x7fd2600a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:41.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.485+0000 7fd27cbe5640 1 --2- 172.21.15.44:0/2961063706 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd2600a2b90 0x7fd2600a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd2781027f0 tx=0x7fd268005f90 comp rx=0 tx=0).stop 2024-04-21T23:06:41.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.485+0000 7fd27cbe5640 1 -- 172.21.15.44:0/2961063706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd278102990 msgr2=0x7fd278198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:41.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.485+0000 7fd27cbe5640 1 --2- 172.21.15.44:0/2961063706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd278102990 0x7fd278198090 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd26c009230 tx=0x7fd26c005d50 comp rx=0 tx=0).stop 2024-04-21T23:06:41.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.485+0000 7fd27cbe5640 1 -- 172.21.15.44:0/2961063706 shutdown_connections 2024-04-21T23:06:41.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.485+0000 7fd27cbe5640 1 --2- 172.21.15.44:0/2961063706 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd2600a2b90 0x7fd2600a5050 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.485+0000 7fd27cbe5640 1 --2- 172.21.15.44:0/2961063706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd278103350 0x7fd27819f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.485+0000 7fd27cbe5640 1 --2- 172.21.15.44:0/2961063706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd278102990 0x7fd278198090 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.485+0000 7fd27cbe5640 1 --2- 172.21.15.44:0/2961063706 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2781017a0 0x7fd278197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.485+0000 7fd27cbe5640 1 -- 172.21.15.44:0/2961063706 >> 172.21.15.44:0/2961063706 conn(0x7fd2780fcd10 msgr2=0x7fd278105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:41.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.485+0000 7fd27cbe5640 1 -- 172.21.15.44:0/2961063706 shutdown_connections 2024-04-21T23:06:41.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.485+0000 7fd27cbe5640 1 -- 172.21.15.44:0/2961063706 wait complete. 2024-04-21T23:06:41.495 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-21T23:06:41.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.600+0000 7fa99b65f640 1 -- 172.21.15.44:0/96958127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa994075780 msgr2=0x7fa994073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:41.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.600+0000 7fa99b65f640 1 --2- 172.21.15.44:0/96958127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa994075780 0x7fa994073bd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa98c009ca0 tx=0x7fa98c02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.600+0000 7fa99b65f640 1 -- 172.21.15.44:0/96958127 shutdown_connections 2024-04-21T23:06:41.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.600+0000 7fa99b65f640 1 --2- 172.21.15.44:0/96958127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9940741a0 0x7fa994074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.600+0000 7fa99b65f640 1 --2- 172.21.15.44:0/96958127 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa994109660 0x7fa99410baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.600+0000 7fa99b65f640 1 --2- 172.21.15.44:0/96958127 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa994075780 0x7fa994073bd0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.600+0000 7fa99b65f640 1 -- 172.21.15.44:0/96958127 >> 172.21.15.44:0/96958127 conn(0x7fa9940fcd10 msgr2=0x7fa9940ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:41.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.600+0000 7fa99b65f640 1 -- 172.21.15.44:0/96958127 shutdown_connections 2024-04-21T23:06:41.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.600+0000 7fa99b65f640 1 -- 172.21.15.44:0/96958127 wait complete. 2024-04-21T23:06:41.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.601+0000 7fa99b65f640 1 Processor -- start 2024-04-21T23:06:41.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.601+0000 7fa99b65f640 1 -- start start 2024-04-21T23:06:41.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.601+0000 7fa99b65f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9940741a0 0x7fa994199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:41.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.601+0000 7fa99b65f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa994075780 0x7fa99419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:41.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.601+0000 7fa99b65f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa994109660 0x7fa9941a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:41.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.601+0000 7fa99b65f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa99410f600 con 0x7fa994075780 2024-04-21T23:06:41.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.601+0000 7fa99b65f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa99410f480 con 0x7fa9940741a0 2024-04-21T23:06:41.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.601+0000 7fa99b65f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa99410f780 con 0x7fa994109660 2024-04-21T23:06:41.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.601+0000 7fa998bfa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa994075780 0x7fa99419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:41.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.601+0000 7fa998bfa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa994075780 0x7fa99419a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38602/0 (socket says 172.21.15.44:38602) 2024-04-21T23:06:41.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.601+0000 7fa9993fb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9940741a0 0x7fa994199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:41.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.601+0000 7fa998bfa640 1 -- 172.21.15.44:0/2079607626 learned_addr learned my addr 172.21.15.44:0/2079607626 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:41.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.601+0000 7fa999bfc640 1 --2- 172.21.15.44:0/2079607626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa994109660 0x7fa9941a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:41.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.601+0000 7fa998bfa640 1 -- 172.21.15.44:0/2079607626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa994109660 msgr2=0x7fa9941a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:41.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.601+0000 7fa998bfa640 1 --2- 172.21.15.44:0/2079607626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa994109660 0x7fa9941a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.601+0000 7fa998bfa640 1 -- 172.21.15.44:0/2079607626 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9940741a0 msgr2=0x7fa994199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:41.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.601+0000 7fa998bfa640 1 --2- 172.21.15.44:0/2079607626 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9940741a0 0x7fa994199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.601+0000 7fa998bfa640 1 -- 172.21.15.44:0/2079607626 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9941a1880 con 0x7fa994075780 2024-04-21T23:06:41.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.602+0000 7fa998bfa640 1 --2- 172.21.15.44:0/2079607626 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa994075780 0x7fa99419a1b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa98400d770 tx=0x7fa98400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:41.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.602+0000 7fa98a7fc640 1 -- 172.21.15.44:0/2079607626 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa984017070 con 0x7fa994075780 2024-04-21T23:06:41.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.602+0000 7fa98a7fc640 1 -- 172.21.15.44:0/2079607626 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa984005500 con 0x7fa994075780 2024-04-21T23:06:41.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.602+0000 7fa99b65f640 1 -- 172.21.15.44:0/2079607626 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9941a1b70 con 0x7fa994075780 2024-04-21T23:06:41.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.602+0000 7fa98a7fc640 1 -- 172.21.15.44:0/2079607626 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa984004e00 con 0x7fa994075780 2024-04-21T23:06:41.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.602+0000 7fa99b65f640 1 -- 172.21.15.44:0/2079607626 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9941a2070 con 0x7fa994075780 2024-04-21T23:06:41.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.603+0000 7fa99b65f640 1 -- 172.21.15.44:0/2079607626 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa994108390 con 0x7fa994075780 2024-04-21T23:06:41.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.604+0000 7fa98a7fc640 1 -- 172.21.15.44:0/2079607626 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa984004fa0 con 0x7fa994075780 2024-04-21T23:06:41.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.604+0000 7fa98a7fc640 1 --2- 172.21.15.44:0/2079607626 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa9800a2b90 0x7fa9800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:41.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.605+0000 7fa9993fb640 1 --2- 172.21.15.44:0/2079607626 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa9800a2b90 0x7fa9800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:41.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.605+0000 7fa98a7fc640 1 -- 172.21.15.44:0/2079607626 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fa9840ced40 con 0x7fa994075780 2024-04-21T23:06:41.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.605+0000 7fa9993fb640 1 --2- 172.21.15.44:0/2079607626 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa9800a2b90 0x7fa9800a5050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa98c02c700 tx=0x7fa98c032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:41.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.609+0000 7fa98a7fc640 1 -- 172.21.15.44:0/2079607626 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa98409b3b0 con 0x7fa994075780 2024-04-21T23:06:41.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:41 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3268745725' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:06:41.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:41 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/203078901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:06:41.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:41 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3937457006' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:06:41.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:41 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3268745725' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:06:41.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:41 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/203078901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:06:41.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:41 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3937457006' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:06:41.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.806+0000 7fa99b65f640 1 -- 172.21.15.44:0/2079607626 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa994062340 con 0x7fa994075780 2024-04-21T23:06:41.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.807+0000 7fa98a7fc640 1 -- 172.21.15.44:0/2079607626 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+590 (secure 0 0 0) 0x7fa98409f6b0 con 0x7fa994075780 2024-04-21T23:06:41.806 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2729363939 2024-04-22T22:59:22.969731+0000 2024-04-21T23:06:41.806 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3462209685 2024-04-22T22:59:22.969731+0000 2024-04-21T23:06:41.806 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/4238737803 2024-04-22T22:59:22.969731+0000 2024-04-21T23:06:41.806 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/4238737803 2024-04-22T22:59:22.969731+0000 2024-04-21T23:06:41.806 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/2900331072 2024-04-22T22:58:56.511296+0000 2024-04-21T23:06:41.806 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/908290973 2024-04-22T22:58:56.511296+0000 2024-04-21T23:06:41.806 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2994566338 2024-04-22T22:58:56.511296+0000 2024-04-21T23:06:41.806 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/2900331072 2024-04-22T22:58:56.511296+0000 2024-04-21T23:06:41.806 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/693229699 2024-04-22T22:58:56.511296+0000 2024-04-21T23:06:41.807 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1931984406 2024-04-22T22:59:22.969731+0000 2024-04-21T23:06:41.807 INFO:teuthology.orchestra.run.smithi044.stderr:listed 10 entries 2024-04-21T23:06:41.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.809+0000 7fa99b65f640 1 -- 172.21.15.44:0/2079607626 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa9800a2b90 msgr2=0x7fa9800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:41.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.809+0000 7fa99b65f640 1 --2- 172.21.15.44:0/2079607626 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa9800a2b90 0x7fa9800a5050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa98c02c700 tx=0x7fa98c032590 comp rx=0 tx=0).stop 2024-04-21T23:06:41.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.809+0000 7fa99b65f640 1 -- 172.21.15.44:0/2079607626 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa994075780 msgr2=0x7fa99419a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:41.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.809+0000 7fa99b65f640 1 --2- 172.21.15.44:0/2079607626 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa994075780 0x7fa99419a1b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa98400d770 tx=0x7fa98400dc30 comp rx=0 tx=0).stop 2024-04-21T23:06:41.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.809+0000 7fa99b65f640 1 -- 172.21.15.44:0/2079607626 shutdown_connections 2024-04-21T23:06:41.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.809+0000 7fa99b65f640 1 --2- 172.21.15.44:0/2079607626 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa9800a2b90 0x7fa9800a5050 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.809+0000 7fa99b65f640 1 --2- 172.21.15.44:0/2079607626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa994109660 0x7fa9941a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.809+0000 7fa99b65f640 1 --2- 172.21.15.44:0/2079607626 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa994075780 0x7fa99419a1b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.809+0000 7fa99b65f640 1 --2- 172.21.15.44:0/2079607626 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9940741a0 0x7fa994199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:41.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.809+0000 7fa99b65f640 1 -- 172.21.15.44:0/2079607626 >> 172.21.15.44:0/2079607626 conn(0x7fa9940fcd10 msgr2=0x7fa994105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:41.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.809+0000 7fa99b65f640 1 -- 172.21.15.44:0/2079607626 shutdown_connections 2024-04-21T23:06:41.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:41.809+0000 7fa99b65f640 1 -- 172.21.15.44:0/2079607626 wait complete. 2024-04-21T23:06:41.819 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi044.front.sepia.ceph.com... 2024-04-21T23:06:41.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:06:41.819 DEBUG:teuthology.orchestra.run.smithi044:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-21T23:06:41.841 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi044.stderr:ceph-fuse[69227]: fuse finished with error 0 and tester_r 0 2024-04-21T23:06:41.843 INFO:teuthology.orchestra.run:waiting for 300 2024-04-21T23:06:42.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:42 smithi044 ceph-mon[27934]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-04-21T23:06:42.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:42 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2961063706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:06:42.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:42 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2079607626' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:06:42.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:42 smithi055 ceph-mon[31117]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-04-21T23:06:42.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:42 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2961063706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:06:42.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:42 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2079607626' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:06:42.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:42 smithi069 ceph-mon[31816]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-04-21T23:06:42.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:42 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2961063706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:06:42.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:42 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2079607626' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:06:44.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:44 smithi044 ceph-mon[27934]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 260 B/s wr, 0 op/s 2024-04-21T23:06:44.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:44 smithi055 ceph-mon[31117]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 260 B/s wr, 0 op/s 2024-04-21T23:06:44.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:44 smithi069 ceph-mon[31816]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 260 B/s wr, 0 op/s 2024-04-21T23:06:46.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:46 smithi044 ceph-mon[27934]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:06:46.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:46 smithi055 ceph-mon[31117]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:06:46.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:46 smithi069 ceph-mon[31816]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:06:47.945 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-21T23:06:47.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:06:47.945 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-21T23:06:47.981 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:06:48.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.138+0000 7f02ec3e5640 1 -- 172.21.15.44:0/3593860084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02e41029f0 msgr2=0x7f02e4102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:48.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.138+0000 7f02ec3e5640 1 --2- 172.21.15.44:0/3593860084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02e41029f0 0x7f02e4102e70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f02d8009900 tx=0x7f02d802bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:48.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.139+0000 7f02ec3e5640 1 -- 172.21.15.44:0/3593860084 shutdown_connections 2024-04-21T23:06:48.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.139+0000 7f02ec3e5640 1 --2- 172.21.15.44:0/3593860084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02e41033b0 0x7f02e4109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.139+0000 7f02ec3e5640 1 --2- 172.21.15.44:0/3593860084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02e41029f0 0x7f02e4102e70 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.139+0000 7f02ec3e5640 1 --2- 172.21.15.44:0/3593860084 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02e4101800 0x7f02e4101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.139+0000 7f02ec3e5640 1 -- 172.21.15.44:0/3593860084 >> 172.21.15.44:0/3593860084 conn(0x7f02e40fcd70 msgr2=0x7f02e40ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:48.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.139+0000 7f02ec3e5640 1 -- 172.21.15.44:0/3593860084 shutdown_connections 2024-04-21T23:06:48.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.139+0000 7f02ec3e5640 1 -- 172.21.15.44:0/3593860084 wait complete. 2024-04-21T23:06:48.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.139+0000 7f02ec3e5640 1 Processor -- start 2024-04-21T23:06:48.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.139+0000 7f02ec3e5640 1 -- start start 2024-04-21T23:06:48.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.139+0000 7f02ec3e5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02e4101800 0x7f02e4197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:48.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.139+0000 7f02ec3e5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02e41029f0 0x7f02e4197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:48.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.139+0000 7f02ec3e5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02e41033b0 0x7f02e419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:48.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.139+0000 7f02ec3e5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f02e410d3f0 con 0x7f02e41033b0 2024-04-21T23:06:48.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.139+0000 7f02ec3e5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f02e410d270 con 0x7f02e41029f0 2024-04-21T23:06:48.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.139+0000 7f02ec3e5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f02e410d570 con 0x7f02e4101800 2024-04-21T23:06:48.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.140+0000 7f02ea982640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02e41033b0 0x7f02e419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:48.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.140+0000 7f02ea982640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02e41033b0 0x7f02e419f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38632/0 (socket says 172.21.15.44:38632) 2024-04-21T23:06:48.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.140+0000 7f02ea982640 1 -- 172.21.15.44:0/159494949 learned_addr learned my addr 172.21.15.44:0/159494949 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:48.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.140+0000 7f02e9980640 1 --2- 172.21.15.44:0/159494949 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02e41029f0 0x7f02e4197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:48.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.140+0000 7f02ea181640 1 --2- 172.21.15.44:0/159494949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02e4101800 0x7f02e4197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:48.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.140+0000 7f02ea982640 1 -- 172.21.15.44:0/159494949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02e4101800 msgr2=0x7f02e4197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:48.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.140+0000 7f02ea982640 1 --2- 172.21.15.44:0/159494949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02e4101800 0x7f02e4197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.140+0000 7f02ea982640 1 -- 172.21.15.44:0/159494949 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02e41029f0 msgr2=0x7f02e4197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:48.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.140+0000 7f02ea982640 1 --2- 172.21.15.44:0/159494949 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02e41029f0 0x7f02e4197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.140+0000 7f02ea982640 1 -- 172.21.15.44:0/159494949 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e419f690 con 0x7f02e41033b0 2024-04-21T23:06:48.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.140+0000 7f02ea181640 1 --2- 172.21.15.44:0/159494949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02e4101800 0x7f02e4197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:06:48.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.140+0000 7f02ea982640 1 --2- 172.21.15.44:0/159494949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02e41033b0 0x7f02e419f020 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f02dc00eaa0 tx=0x7f02dc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:48.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.140+0000 7f02d77fe640 1 -- 172.21.15.44:0/159494949 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc00cdb0 con 0x7f02e41033b0 2024-04-21T23:06:48.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.140+0000 7f02d77fe640 1 -- 172.21.15.44:0/159494949 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02dc005020 con 0x7f02e41033b0 2024-04-21T23:06:48.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.140+0000 7f02ec3e5640 1 -- 172.21.15.44:0/159494949 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02e419f980 con 0x7f02e41033b0 2024-04-21T23:06:48.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.140+0000 7f02d77fe640 1 -- 172.21.15.44:0/159494949 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc0116b0 con 0x7f02e41033b0 2024-04-21T23:06:48.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.140+0000 7f02ec3e5640 1 -- 172.21.15.44:0/159494949 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02e419fea0 con 0x7f02e41033b0 2024-04-21T23:06:48.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.141+0000 7f02ec3e5640 1 -- 172.21.15.44:0/159494949 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02e4108b20 con 0x7f02e41033b0 2024-04-21T23:06:48.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.142+0000 7f02d77fe640 1 -- 172.21.15.44:0/159494949 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f02dc005560 con 0x7f02e41033b0 2024-04-21T23:06:48.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.143+0000 7f02d77fe640 1 --2- 172.21.15.44:0/159494949 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f02d00a2c10 0x7f02d00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:48.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.143+0000 7f02ea181640 1 --2- 172.21.15.44:0/159494949 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f02d00a2c10 0x7f02d00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:48.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.143+0000 7f02d77fe640 1 -- 172.21.15.44:0/159494949 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f02dc0cf4a0 con 0x7f02e41033b0 2024-04-21T23:06:48.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.144+0000 7f02ea181640 1 --2- 172.21.15.44:0/159494949 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f02d00a2c10 0x7f02d00a50d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f02e4102850 tx=0x7f02e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:48.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.148+0000 7f02d77fe640 1 -- 172.21.15.44:0/159494949 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f02dc016030 con 0x7f02e41033b0 2024-04-21T23:06:48.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.337+0000 7f02ec3e5640 1 -- 172.21.15.44:0/159494949 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f02e402d1c0 con 0x7f02e41033b0 2024-04-21T23:06:48.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.339+0000 7f02d77fe640 1 -- 172.21.15.44:0/159494949 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24032 (secure 0 0 0) 0x7f02dc09c1c0 con 0x7f02e41033b0 2024-04-21T23:06:48.339 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:48.339 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":90,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:06:30.619155+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:06:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.341+0000 7f02ec3e5640 1 -- 172.21.15.44:0/159494949 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f02d00a2c10 msgr2=0x7f02d00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.341+0000 7f02ec3e5640 1 --2- 172.21.15.44:0/159494949 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f02d00a2c10 0x7f02d00a50d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f02e4102850 tx=0x7f02e0005f90 comp rx=0 tx=0).stop 2024-04-21T23:06:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.341+0000 7f02ec3e5640 1 -- 172.21.15.44:0/159494949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02e41033b0 msgr2=0x7f02e419f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.341+0000 7f02ec3e5640 1 --2- 172.21.15.44:0/159494949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02e41033b0 0x7f02e419f020 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f02dc00eaa0 tx=0x7f02dc00ef60 comp rx=0 tx=0).stop 2024-04-21T23:06:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.341+0000 7f02ec3e5640 1 -- 172.21.15.44:0/159494949 shutdown_connections 2024-04-21T23:06:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.341+0000 7f02ec3e5640 1 --2- 172.21.15.44:0/159494949 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f02d00a2c10 0x7f02d00a50d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.341+0000 7f02ec3e5640 1 --2- 172.21.15.44:0/159494949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02e41033b0 0x7f02e419f020 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.341+0000 7f02ec3e5640 1 --2- 172.21.15.44:0/159494949 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02e41029f0 0x7f02e4197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.341+0000 7f02ec3e5640 1 --2- 172.21.15.44:0/159494949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02e4101800 0x7f02e4197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.341+0000 7f02ec3e5640 1 -- 172.21.15.44:0/159494949 >> 172.21.15.44:0/159494949 conn(0x7f02e40fcd70 msgr2=0x7f02e4105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:48.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.341+0000 7f02ec3e5640 1 -- 172.21.15.44:0/159494949 shutdown_connections 2024-04-21T23:06:48.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.341+0000 7f02ec3e5640 1 -- 172.21.15.44:0/159494949 wait complete. 2024-04-21T23:06:48.352 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:06:48.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.454+0000 7f81032a7640 1 -- 172.21.15.44:0/1572233038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80fc0ff570 msgr2=0x7f80fc0ff990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:48.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.454+0000 7f81032a7640 1 --2- 172.21.15.44:0/1572233038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80fc0ff570 0x7f80fc0ff990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f80f8009900 tx=0x7f80f802bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:48.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.454+0000 7f81032a7640 1 -- 172.21.15.44:0/1572233038 shutdown_connections 2024-04-21T23:06:48.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.454+0000 7f81032a7640 1 --2- 172.21.15.44:0/1572233038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80fc0ffed0 0x7f80fc106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.454+0000 7f81032a7640 1 --2- 172.21.15.44:0/1572233038 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80fc107460 0x7f80fc109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.454+0000 7f81032a7640 1 --2- 172.21.15.44:0/1572233038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80fc0ff570 0x7f80fc0ff990 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.454+0000 7f81032a7640 1 -- 172.21.15.44:0/1572233038 >> 172.21.15.44:0/1572233038 conn(0x7f80fc0faf80 msgr2=0x7f80fc0fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:48.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.454+0000 7f81032a7640 1 -- 172.21.15.44:0/1572233038 shutdown_connections 2024-04-21T23:06:48.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.454+0000 7f81032a7640 1 -- 172.21.15.44:0/1572233038 wait complete. 2024-04-21T23:06:48.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.455+0000 7f81032a7640 1 Processor -- start 2024-04-21T23:06:48.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.455+0000 7f81032a7640 1 -- start start 2024-04-21T23:06:48.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.455+0000 7f81032a7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80fc0ff570 0x7f80fc1027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:48.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.455+0000 7f81032a7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80fc0ffed0 0x7f80fc100e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:48.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.455+0000 7f81032a7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80fc107460 0x7f80fc1013a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:48.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.455+0000 7f81032a7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc10d3e0 con 0x7f80fc0ff570 2024-04-21T23:06:48.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.455+0000 7f81032a7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc10d260 con 0x7f80fc107460 2024-04-21T23:06:48.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.455+0000 7f81032a7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc10d560 con 0x7f80fc0ffed0 2024-04-21T23:06:48.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.455+0000 7f8101043640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80fc0ff570 0x7f80fc1027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:48.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.455+0000 7f8101043640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80fc0ff570 0x7f80fc1027d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38640/0 (socket says 172.21.15.44:38640) 2024-04-21T23:06:48.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.455+0000 7f8101043640 1 -- 172.21.15.44:0/2645471172 learned_addr learned my addr 172.21.15.44:0/2645471172 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:48.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.455+0000 7f8100842640 1 --2- 172.21.15.44:0/2645471172 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80fc0ffed0 0x7f80fc100e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:48.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.455+0000 7f8101844640 1 --2- 172.21.15.44:0/2645471172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80fc107460 0x7f80fc1013a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:48.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.455+0000 7f8101043640 1 -- 172.21.15.44:0/2645471172 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80fc0ffed0 msgr2=0x7f80fc100e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:48.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.455+0000 7f8101043640 1 --2- 172.21.15.44:0/2645471172 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80fc0ffed0 0x7f80fc100e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.455+0000 7f8101043640 1 -- 172.21.15.44:0/2645471172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80fc107460 msgr2=0x7f80fc1013a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:48.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.455+0000 7f8101043640 1 --2- 172.21.15.44:0/2645471172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80fc107460 0x7f80fc1013a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.455+0000 7f8101043640 1 -- 172.21.15.44:0/2645471172 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80fc101c60 con 0x7f80fc0ff570 2024-04-21T23:06:48.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.456+0000 7f8101043640 1 --2- 172.21.15.44:0/2645471172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80fc0ff570 0x7f80fc1027d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f80f8009230 tx=0x7f80f8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:48.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.456+0000 7f80f27fc640 1 -- 172.21.15.44:0/2645471172 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f8041070 con 0x7f80fc0ff570 2024-04-21T23:06:48.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.456+0000 7f80f27fc640 1 -- 172.21.15.44:0/2645471172 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80f80044d0 con 0x7f80fc0ff570 2024-04-21T23:06:48.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.456+0000 7f81032a7640 1 -- 172.21.15.44:0/2645471172 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80fc1a1af0 con 0x7f80fc0ff570 2024-04-21T23:06:48.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.456+0000 7f80f27fc640 1 -- 172.21.15.44:0/2645471172 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f8034440 con 0x7f80fc0ff570 2024-04-21T23:06:48.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.456+0000 7f81032a7640 1 -- 172.21.15.44:0/2645471172 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80fc1a2000 con 0x7f80fc0ff570 2024-04-21T23:06:48.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.457+0000 7f81032a7640 1 -- 172.21.15.44:0/2645471172 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80fc105ce0 con 0x7f80fc0ff570 2024-04-21T23:06:48.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.458+0000 7f80f27fc640 1 -- 172.21.15.44:0/2645471172 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f80f8044070 con 0x7f80fc0ff570 2024-04-21T23:06:48.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.458+0000 7f80f27fc640 1 --2- 172.21.15.44:0/2645471172 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f80e80a2c10 0x7f80e80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:48.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.459+0000 7f8100842640 1 --2- 172.21.15.44:0/2645471172 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f80e80a2c10 0x7f80e80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:48.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.459+0000 7f80f27fc640 1 -- 172.21.15.44:0/2645471172 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f80f80f0f40 con 0x7f80fc0ff570 2024-04-21T23:06:48.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.459+0000 7f8100842640 1 --2- 172.21.15.44:0/2645471172 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f80e80a2c10 0x7f80e80a50d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f80ec0058c0 tx=0x7f80ec005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:48.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.463+0000 7f80f27fc640 1 -- 172.21.15.44:0/2645471172 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f80f80bdc60 con 0x7f80fc0ff570 2024-04-21T23:06:48.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.650+0000 7f81032a7640 1 -- 172.21.15.44:0/2645471172 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f80fc062370 con 0x7f80fc0ff570 2024-04-21T23:06:48.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.652+0000 7f80f27fc640 1 -- 172.21.15.44:0/2645471172 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24032 (secure 0 0 0) 0x7f80f80bd560 con 0x7f80fc0ff570 2024-04-21T23:06:48.652 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:48.652 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":90,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:06:30.619155+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:06:48.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.654+0000 7f81032a7640 1 -- 172.21.15.44:0/2645471172 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f80e80a2c10 msgr2=0x7f80e80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:48.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.654+0000 7f81032a7640 1 --2- 172.21.15.44:0/2645471172 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f80e80a2c10 0x7f80e80a50d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f80ec0058c0 tx=0x7f80ec005850 comp rx=0 tx=0).stop 2024-04-21T23:06:48.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.654+0000 7f81032a7640 1 -- 172.21.15.44:0/2645471172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80fc0ff570 msgr2=0x7f80fc1027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:48.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.654+0000 7f81032a7640 1 --2- 172.21.15.44:0/2645471172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80fc0ff570 0x7f80fc1027d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f80f8009230 tx=0x7f80f8005d50 comp rx=0 tx=0).stop 2024-04-21T23:06:48.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.654+0000 7f81032a7640 1 -- 172.21.15.44:0/2645471172 shutdown_connections 2024-04-21T23:06:48.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.654+0000 7f81032a7640 1 --2- 172.21.15.44:0/2645471172 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f80e80a2c10 0x7f80e80a50d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.654+0000 7f81032a7640 1 --2- 172.21.15.44:0/2645471172 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80fc0ffed0 0x7f80fc100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.654+0000 7f81032a7640 1 --2- 172.21.15.44:0/2645471172 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80fc107460 0x7f80fc1013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.654+0000 7f81032a7640 1 --2- 172.21.15.44:0/2645471172 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80fc0ff570 0x7f80fc1027d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.654+0000 7f81032a7640 1 -- 172.21.15.44:0/2645471172 >> 172.21.15.44:0/2645471172 conn(0x7f80fc0faf80 msgr2=0x7f80fc102eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:48.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.654+0000 7f81032a7640 1 -- 172.21.15.44:0/2645471172 shutdown_connections 2024-04-21T23:06:48.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.654+0000 7f81032a7640 1 -- 172.21.15.44:0/2645471172 wait complete. 2024-04-21T23:06:48.663 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:48 smithi044 ceph-mon[27934]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:06:48.665 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-21T23:06:48.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:48 smithi055 ceph-mon[31117]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:06:48.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:48 smithi069 ceph-mon[31816]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:06:48.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.770+0000 7fe30b97a640 1 -- 172.21.15.44:0/698209206 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe304101870 msgr2=0x7fe304101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:48.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.770+0000 7fe30b97a640 1 --2- 172.21.15.44:0/698209206 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe304101870 0x7fe304101cf0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8009900 tx=0x7fe2f802bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.770+0000 7fe30b97a640 1 -- 172.21.15.44:0/698209206 shutdown_connections 2024-04-21T23:06:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.770+0000 7fe30b97a640 1 --2- 172.21.15.44:0/698209206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe304102230 0x7fe304108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.770+0000 7fe30b97a640 1 --2- 172.21.15.44:0/698209206 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe304101870 0x7fe304101cf0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.770+0000 7fe30b97a640 1 --2- 172.21.15.44:0/698209206 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe304100680 0x7fe304100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.770+0000 7fe30b97a640 1 -- 172.21.15.44:0/698209206 >> 172.21.15.44:0/698209206 conn(0x7fe3040fbc10 msgr2=0x7fe3040fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.771+0000 7fe30b97a640 1 -- 172.21.15.44:0/698209206 shutdown_connections 2024-04-21T23:06:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.771+0000 7fe30b97a640 1 -- 172.21.15.44:0/698209206 wait complete. 2024-04-21T23:06:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.771+0000 7fe30b97a640 1 Processor -- start 2024-04-21T23:06:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.771+0000 7fe30b97a640 1 -- start start 2024-04-21T23:06:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.771+0000 7fe30b97a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe304100680 0x7fe3041045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.771+0000 7fe30b97a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe304101870 0x7fe304102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.771+0000 7fe30b97a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe304102230 0x7fe304103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.771+0000 7fe30b97a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe30410c300 con 0x7fe304102230 2024-04-21T23:06:48.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.771+0000 7fe30b97a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe30410c180 con 0x7fe304101870 2024-04-21T23:06:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.771+0000 7fe30b97a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe30410c480 con 0x7fe304100680 2024-04-21T23:06:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.771+0000 7fe309f17640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe304102230 0x7fe304103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.771+0000 7fe309f17640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe304102230 0x7fe304103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38662/0 (socket says 172.21.15.44:38662) 2024-04-21T23:06:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.771+0000 7fe309f17640 1 -- 172.21.15.44:0/3227915117 learned_addr learned my addr 172.21.15.44:0/3227915117 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.771+0000 7fe308f15640 1 --2- 172.21.15.44:0/3227915117 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe304101870 0x7fe304102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.772+0000 7fe309716640 1 --2- 172.21.15.44:0/3227915117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe304100680 0x7fe3041045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.772+0000 7fe309f17640 1 -- 172.21.15.44:0/3227915117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe304100680 msgr2=0x7fe3041045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.772+0000 7fe309f17640 1 --2- 172.21.15.44:0/3227915117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe304100680 0x7fe3041045c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.772+0000 7fe309f17640 1 -- 172.21.15.44:0/3227915117 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe304101870 msgr2=0x7fe304102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.772+0000 7fe309f17640 1 --2- 172.21.15.44:0/3227915117 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe304101870 0x7fe304102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.772+0000 7fe309f17640 1 -- 172.21.15.44:0/3227915117 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe304103a50 con 0x7fe304102230 2024-04-21T23:06:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.772+0000 7fe309f17640 1 --2- 172.21.15.44:0/3227915117 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe304102230 0x7fe304103190 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc00d750 tx=0x7fe2fc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.772+0000 7fe3027fc640 1 -- 172.21.15.44:0/3227915117 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc017070 con 0x7fe304102230 2024-04-21T23:06:48.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.772+0000 7fe30b97a640 1 -- 172.21.15.44:0/3227915117 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3041a1c40 con 0x7fe304102230 2024-04-21T23:06:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.772+0000 7fe3027fc640 1 -- 172.21.15.44:0/3227915117 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2fc005500 con 0x7fe304102230 2024-04-21T23:06:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.772+0000 7fe3027fc640 1 -- 172.21.15.44:0/3227915117 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc004d90 con 0x7fe304102230 2024-04-21T23:06:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.772+0000 7fe30b97a640 1 -- 172.21.15.44:0/3227915117 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3041a21b0 con 0x7fe304102230 2024-04-21T23:06:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.773+0000 7fe30b97a640 1 -- 172.21.15.44:0/3227915117 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3041a2380 con 0x7fe304102230 2024-04-21T23:06:48.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.775+0000 7fe3027fc640 1 -- 172.21.15.44:0/3227915117 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe2fc004f30 con 0x7fe304102230 2024-04-21T23:06:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.775+0000 7fe3027fc640 1 --2- 172.21.15.44:0/3227915117 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe2ec0a2b90 0x7fe2ec0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.776+0000 7fe309716640 1 --2- 172.21.15.44:0/3227915117 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe2ec0a2b90 0x7fe2ec0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.776+0000 7fe3027fc640 1 -- 172.21.15.44:0/3227915117 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fe2fc0cefe0 con 0x7fe304102230 2024-04-21T23:06:48.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.776+0000 7fe309716640 1 --2- 172.21.15.44:0/3227915117 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe2ec0a2b90 0x7fe2ec0a5050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe3041016d0 tx=0x7fe2f4009870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:48.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:48.779+0000 7fe3027fc640 1 -- 172.21.15.44:0/3227915117 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe2fc09bd00 con 0x7fe304102230 2024-04-21T23:06:49.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.001+0000 7fe30b97a640 1 -- 172.21.15.44:0/3227915117 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fe3041a26d0 con 0x7fe304102230 2024-04-21T23:06:49.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.316+0000 7fe3027fc640 1 -- 172.21.15.44:0/3227915117 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7fe2fc09b600 con 0x7fe304102230 2024-04-21T23:06:49.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.319+0000 7fe30b97a640 1 -- 172.21.15.44:0/3227915117 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe2ec0a2b90 msgr2=0x7fe2ec0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:49.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.319+0000 7fe30b97a640 1 --2- 172.21.15.44:0/3227915117 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe2ec0a2b90 0x7fe2ec0a5050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe3041016d0 tx=0x7fe2f4009870 comp rx=0 tx=0).stop 2024-04-21T23:06:49.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.319+0000 7fe30b97a640 1 -- 172.21.15.44:0/3227915117 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe304102230 msgr2=0x7fe304103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:49.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.319+0000 7fe30b97a640 1 --2- 172.21.15.44:0/3227915117 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe304102230 0x7fe304103190 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc00d750 tx=0x7fe2fc00dc10 comp rx=0 tx=0).stop 2024-04-21T23:06:49.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.319+0000 7fe30b97a640 1 -- 172.21.15.44:0/3227915117 shutdown_connections 2024-04-21T23:06:49.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.319+0000 7fe30b97a640 1 --2- 172.21.15.44:0/3227915117 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe2ec0a2b90 0x7fe2ec0a5050 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:49.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.319+0000 7fe30b97a640 1 --2- 172.21.15.44:0/3227915117 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe304102230 0x7fe304103190 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:49.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.319+0000 7fe30b97a640 1 --2- 172.21.15.44:0/3227915117 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe304101870 0x7fe304102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:49.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.319+0000 7fe30b97a640 1 --2- 172.21.15.44:0/3227915117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe304100680 0x7fe3041045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:49.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.319+0000 7fe30b97a640 1 -- 172.21.15.44:0/3227915117 >> 172.21.15.44:0/3227915117 conn(0x7fe3040fbc10 msgr2=0x7fe3040fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:49.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.319+0000 7fe30b97a640 1 -- 172.21.15.44:0/3227915117 shutdown_connections 2024-04-21T23:06:49.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.319+0000 7fe30b97a640 1 -- 172.21.15.44:0/3227915117 wait complete. 2024-04-21T23:06:49.331 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 1, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-04-21T23:06:49.331 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-21T23:06:49.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.437+0000 7f5f8313a640 1 -- 172.21.15.44:0/294842047 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f7c103350 msgr2=0x7f5f7c109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:49.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.437+0000 7f5f8313a640 1 --2- 172.21.15.44:0/294842047 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f7c103350 0x7f5f7c109be0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5f74009ca0 tx=0x7f5f7402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:49.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.437+0000 7f5f8313a640 1 -- 172.21.15.44:0/294842047 shutdown_connections 2024-04-21T23:06:49.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.437+0000 7f5f8313a640 1 --2- 172.21.15.44:0/294842047 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f7c102990 0x7f5f7c102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:49.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.437+0000 7f5f8313a640 1 --2- 172.21.15.44:0/294842047 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f7c103350 0x7f5f7c109be0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:49.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.437+0000 7f5f8313a640 1 --2- 172.21.15.44:0/294842047 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f7c1017a0 0x7f5f7c101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:49.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.437+0000 7f5f8313a640 1 -- 172.21.15.44:0/294842047 >> 172.21.15.44:0/294842047 conn(0x7f5f7c0fcd10 msgr2=0x7f5f7c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:49.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.437+0000 7f5f8313a640 1 -- 172.21.15.44:0/294842047 shutdown_connections 2024-04-21T23:06:49.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.437+0000 7f5f8313a640 1 -- 172.21.15.44:0/294842047 wait complete. 2024-04-21T23:06:49.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.437+0000 7f5f8313a640 1 Processor -- start 2024-04-21T23:06:49.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.437+0000 7f5f8313a640 1 -- start start 2024-04-21T23:06:49.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f8313a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f7c1017a0 0x7f5f7c0754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:49.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f8313a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f7c102990 0x7f5f7c073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:49.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f8313a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f7c103350 0x7f5f7c074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:49.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f8313a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c10d420 con 0x7f5f7c1017a0 2024-04-21T23:06:49.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f8313a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c10d2a0 con 0x7f5f7c102990 2024-04-21T23:06:49.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f8313a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c10d5a0 con 0x7f5f7c103350 2024-04-21T23:06:49.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f80ed6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f7c1017a0 0x7f5f7c0754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:49.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f80ed6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f7c1017a0 0x7f5f7c0754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38676/0 (socket says 172.21.15.44:38676) 2024-04-21T23:06:49.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f80ed6640 1 -- 172.21.15.44:0/3025519889 learned_addr learned my addr 172.21.15.44:0/3025519889 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:49.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f7bfff640 1 --2- 172.21.15.44:0/3025519889 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f7c102990 0x7f5f7c073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:49.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f816d7640 1 --2- 172.21.15.44:0/3025519889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f7c103350 0x7f5f7c074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:49.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f80ed6640 1 -- 172.21.15.44:0/3025519889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f7c103350 msgr2=0x7f5f7c074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:49.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f80ed6640 1 --2- 172.21.15.44:0/3025519889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f7c103350 0x7f5f7c074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:49.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f80ed6640 1 -- 172.21.15.44:0/3025519889 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f7c102990 msgr2=0x7f5f7c073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:49.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f80ed6640 1 --2- 172.21.15.44:0/3025519889 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f7c102990 0x7f5f7c073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:49.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f80ed6640 1 -- 172.21.15.44:0/3025519889 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f7c074900 con 0x7f5f7c1017a0 2024-04-21T23:06:49.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f7bfff640 1 --2- 172.21.15.44:0/3025519889 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f7c102990 0x7f5f7c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:06:49.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f816d7640 1 --2- 172.21.15.44:0/3025519889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f7c103350 0x7f5f7c074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:06:49.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f80ed6640 1 --2- 172.21.15.44:0/3025519889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f7c1017a0 0x7f5f7c0754f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5f7000d950 tx=0x7f5f7000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:49.441 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f79ffb640 1 -- 172.21.15.44:0/3025519889 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f70017020 con 0x7f5f7c1017a0 2024-04-21T23:06:49.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f8313a640 1 -- 172.21.15.44:0/3025519889 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f7c1a1b90 con 0x7f5f7c1017a0 2024-04-21T23:06:49.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f79ffb640 1 -- 172.21.15.44:0/3025519889 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f70009c30 con 0x7f5f7c1017a0 2024-04-21T23:06:49.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.438+0000 7f5f79ffb640 1 -- 172.21.15.44:0/3025519889 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7000f6b0 con 0x7f5f7c1017a0 2024-04-21T23:06:49.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.439+0000 7f5f8313a640 1 -- 172.21.15.44:0/3025519889 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f7c1a2130 con 0x7f5f7c1017a0 2024-04-21T23:06:49.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.439+0000 7f5f8313a640 1 -- 172.21.15.44:0/3025519889 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f7c108ac0 con 0x7f5f7c1017a0 2024-04-21T23:06:49.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.441+0000 7f5f79ffb640 1 -- 172.21.15.44:0/3025519889 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5f7001a020 con 0x7f5f7c1017a0 2024-04-21T23:06:49.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.441+0000 7f5f79ffb640 1 --2- 172.21.15.44:0/3025519889 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5f6c0a2b90 0x7f5f6c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:49.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.441+0000 7f5f79ffb640 1 -- 172.21.15.44:0/3025519889 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f5f700cf770 con 0x7f5f7c1017a0 2024-04-21T23:06:49.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.441+0000 7f5f7bfff640 1 --2- 172.21.15.44:0/3025519889 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5f6c0a2b90 0x7f5f6c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:49.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.442+0000 7f5f7bfff640 1 --2- 172.21.15.44:0/3025519889 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5f6c0a2b90 0x7f5f6c0a5050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5f68005d70 tx=0x7f5f68005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:49.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.445+0000 7f5f79ffb640 1 -- 172.21.15.44:0/3025519889 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5f70021020 con 0x7f5f7c1017a0 2024-04-21T23:06:49.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:49.670+0000 7f5f8313a640 1 -- 172.21.15.44:0/3025519889 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f5f7c02d170 con 0x7f5f7c1017a0 2024-04-21T23:06:49.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:49 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/159494949' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:49.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:49 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2645471172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:49.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:49 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3227915117' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-21T23:06:49.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:49 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/159494949' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:49.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:49 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2645471172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:49.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:49 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3227915117' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-21T23:06:49.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:49 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/159494949' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:49.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:49 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2645471172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:49.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:49 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3227915117' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-21T23:06:50.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.324+0000 7f5f79ffb640 1 -- 172.21.15.44:0/3025519889 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7f5f7009c490 con 0x7f5f7c1017a0 2024-04-21T23:06:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.327+0000 7f5f8313a640 1 -- 172.21.15.44:0/3025519889 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5f6c0a2b90 msgr2=0x7f5f6c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.327+0000 7f5f8313a640 1 --2- 172.21.15.44:0/3025519889 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5f6c0a2b90 0x7f5f6c0a5050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5f68005d70 tx=0x7f5f68005d00 comp rx=0 tx=0).stop 2024-04-21T23:06:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.327+0000 7f5f8313a640 1 -- 172.21.15.44:0/3025519889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f7c1017a0 msgr2=0x7f5f7c0754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.327+0000 7f5f8313a640 1 --2- 172.21.15.44:0/3025519889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f7c1017a0 0x7f5f7c0754f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5f7000d950 tx=0x7f5f7000de10 comp rx=0 tx=0).stop 2024-04-21T23:06:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.327+0000 7f5f8313a640 1 -- 172.21.15.44:0/3025519889 shutdown_connections 2024-04-21T23:06:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.327+0000 7f5f8313a640 1 --2- 172.21.15.44:0/3025519889 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5f6c0a2b90 0x7f5f6c0a5050 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.327+0000 7f5f8313a640 1 --2- 172.21.15.44:0/3025519889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f7c103350 0x7f5f7c074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.327+0000 7f5f8313a640 1 --2- 172.21.15.44:0/3025519889 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f7c102990 0x7f5f7c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.327+0000 7f5f8313a640 1 --2- 172.21.15.44:0/3025519889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f7c1017a0 0x7f5f7c0754f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.327+0000 7f5f8313a640 1 -- 172.21.15.44:0/3025519889 >> 172.21.15.44:0/3025519889 conn(0x7f5f7c0fcd10 msgr2=0x7f5f7c105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:50.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.327+0000 7f5f8313a640 1 -- 172.21.15.44:0/3025519889 shutdown_connections 2024-04-21T23:06:50.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.327+0000 7f5f8313a640 1 -- 172.21.15.44:0/3025519889 wait complete. 2024-04-21T23:06:50.340 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-21T23:06:50.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.450+0000 7f8ebb0c6640 1 -- 172.21.15.44:0/3010229889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8eb4103340 msgr2=0x7f8eb4109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:50.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.450+0000 7f8ebb0c6640 1 --2- 172.21.15.44:0/3010229889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8eb4103340 0x7f8eb4109bd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8eac009ca0 tx=0x7f8eac02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:50.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.450+0000 7f8ebb0c6640 1 -- 172.21.15.44:0/3010229889 shutdown_connections 2024-04-21T23:06:50.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.450+0000 7f8ebb0c6640 1 --2- 172.21.15.44:0/3010229889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8eb4102980 0x7f8eb4102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:50.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.450+0000 7f8ebb0c6640 1 --2- 172.21.15.44:0/3010229889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8eb4103340 0x7f8eb4109bd0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:50.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.450+0000 7f8ebb0c6640 1 --2- 172.21.15.44:0/3010229889 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8eb4101790 0x7f8eb4101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:50.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.450+0000 7f8ebb0c6640 1 -- 172.21.15.44:0/3010229889 >> 172.21.15.44:0/3010229889 conn(0x7f8eb40fcd00 msgr2=0x7f8eb40ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:50.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.451+0000 7f8ebb0c6640 1 -- 172.21.15.44:0/3010229889 shutdown_connections 2024-04-21T23:06:50.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.451+0000 7f8ebb0c6640 1 -- 172.21.15.44:0/3010229889 wait complete. 2024-04-21T23:06:50.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.451+0000 7f8ebb0c6640 1 Processor -- start 2024-04-21T23:06:50.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.451+0000 7f8ebb0c6640 1 -- start start 2024-04-21T23:06:50.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.451+0000 7f8ebb0c6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8eb4101790 0x7f8eb4197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:50.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.451+0000 7f8ebb0c6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8eb4102980 0x7f8eb4197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:50.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.451+0000 7f8ebb0c6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8eb4103340 0x7f8eb419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:50.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.451+0000 7f8ebb0c6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb410d410 con 0x7f8eb4101790 2024-04-21T23:06:50.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.451+0000 7f8ebb0c6640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb410d290 con 0x7f8eb4102980 2024-04-21T23:06:50.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.451+0000 7f8ebb0c6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb410d590 con 0x7f8eb4103340 2024-04-21T23:06:50.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.451+0000 7f8eb8e62640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8eb4101790 0x7f8eb4197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:50.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.451+0000 7f8eb8e62640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8eb4101790 0x7f8eb4197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38688/0 (socket says 172.21.15.44:38688) 2024-04-21T23:06:50.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.451+0000 7f8eb8e62640 1 -- 172.21.15.44:0/1246591919 learned_addr learned my addr 172.21.15.44:0/1246591919 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:50.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.451+0000 7f8eb3fff640 1 --2- 172.21.15.44:0/1246591919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8eb4102980 0x7f8eb4197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:50.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.452+0000 7f8eb9663640 1 --2- 172.21.15.44:0/1246591919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8eb4103340 0x7f8eb419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:50.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.452+0000 7f8eb8e62640 1 -- 172.21.15.44:0/1246591919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8eb4103340 msgr2=0x7f8eb419f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:50.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.452+0000 7f8eb8e62640 1 --2- 172.21.15.44:0/1246591919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8eb4103340 0x7f8eb419f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:50.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.452+0000 7f8eb8e62640 1 -- 172.21.15.44:0/1246591919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8eb4102980 msgr2=0x7f8eb4197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:50.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.452+0000 7f8eb8e62640 1 --2- 172.21.15.44:0/1246591919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8eb4102980 0x7f8eb4197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:50.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.452+0000 7f8eb8e62640 1 -- 172.21.15.44:0/1246591919 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eb419f6a0 con 0x7f8eb4101790 2024-04-21T23:06:50.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.452+0000 7f8eb9663640 1 --2- 172.21.15.44:0/1246591919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8eb4103340 0x7f8eb419f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:06:50.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.452+0000 7f8eb8e62640 1 --2- 172.21.15.44:0/1246591919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8eb4101790 0x7f8eb4197a80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400d770 tx=0x7f8ea400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:50.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.452+0000 7f8eb1ffb640 1 -- 172.21.15.44:0/1246591919 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4017070 con 0x7f8eb4101790 2024-04-21T23:06:50.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.452+0000 7f8eb1ffb640 1 -- 172.21.15.44:0/1246591919 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ea4009c30 con 0x7f8eb4101790 2024-04-21T23:06:50.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.452+0000 7f8ebb0c6640 1 -- 172.21.15.44:0/1246591919 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8eb419f990 con 0x7f8eb4101790 2024-04-21T23:06:50.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.452+0000 7f8eb1ffb640 1 -- 172.21.15.44:0/1246591919 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4003dd0 con 0x7f8eb4101790 2024-04-21T23:06:50.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.452+0000 7f8ebb0c6640 1 -- 172.21.15.44:0/1246591919 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8eb419fee0 con 0x7f8eb4101790 2024-04-21T23:06:50.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.453+0000 7f8ebb0c6640 1 -- 172.21.15.44:0/1246591919 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8eb4108ab0 con 0x7f8eb4101790 2024-04-21T23:06:50.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.454+0000 7f8eb1ffb640 1 -- 172.21.15.44:0/1246591919 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8ea4003fb0 con 0x7f8eb4101790 2024-04-21T23:06:50.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.455+0000 7f8eb1ffb640 1 --2- 172.21.15.44:0/1246591919 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f8e9c0a2c10 0x7f8e9c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:50.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.455+0000 7f8eb3fff640 1 --2- 172.21.15.44:0/1246591919 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f8e9c0a2c10 0x7f8e9c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:50.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.455+0000 7f8eb1ffb640 1 -- 172.21.15.44:0/1246591919 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f8ea40cec90 con 0x7f8eb4101790 2024-04-21T23:06:50.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.455+0000 7f8eb3fff640 1 --2- 172.21.15.44:0/1246591919 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f8e9c0a2c10 0x7f8e9c0a50d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4198f90 tx=0x7f8ea800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:50.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.459+0000 7f8eb1ffb640 1 -- 172.21.15.44:0/1246591919 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f8ea409b950 con 0x7f8eb4101790 2024-04-21T23:06:50.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:50.684+0000 7f8ebb0c6640 1 -- 172.21.15.44:0/1246591919 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f8eb402d170 con 0x7f8eb4101790 2024-04-21T23:06:50.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:50 smithi044 ceph-mon[27934]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:06:50.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:50 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3227915117' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-21T23:06:50.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:50 smithi044 ceph-mon[27934]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:06:50.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:50 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3025519889' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-21T23:06:50.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:50 smithi069 ceph-mon[31816]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:06:50.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:50 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3227915117' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-21T23:06:50.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:50 smithi069 ceph-mon[31816]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:06:50.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:50 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3025519889' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-21T23:06:50.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:50 smithi055 ceph-mon[31117]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:06:50.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:50 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3227915117' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-21T23:06:50.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:50 smithi055 ceph-mon[31117]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:06:50.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:50 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3025519889' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-21T23:06:51.338 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.338+0000 7f8eb1ffb640 1 -- 172.21.15.44:0/1246591919 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) ==== 115+0+0 (secure 0 0 0) 0x7f8ea4009de0 con 0x7f8eb4101790 2024-04-21T23:06:51.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.341+0000 7f8ebb0c6640 1 -- 172.21.15.44:0/1246591919 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f8e9c0a2c10 msgr2=0x7f8e9c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:51.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.341+0000 7f8ebb0c6640 1 --2- 172.21.15.44:0/1246591919 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f8e9c0a2c10 0x7f8e9c0a50d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4198f90 tx=0x7f8ea800b390 comp rx=0 tx=0).stop 2024-04-21T23:06:51.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.341+0000 7f8ebb0c6640 1 -- 172.21.15.44:0/1246591919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8eb4101790 msgr2=0x7f8eb4197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:51.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.341+0000 7f8ebb0c6640 1 --2- 172.21.15.44:0/1246591919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8eb4101790 0x7f8eb4197a80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400d770 tx=0x7f8ea400dc30 comp rx=0 tx=0).stop 2024-04-21T23:06:51.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.341+0000 7f8ebb0c6640 1 -- 172.21.15.44:0/1246591919 shutdown_connections 2024-04-21T23:06:51.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.341+0000 7f8ebb0c6640 1 --2- 172.21.15.44:0/1246591919 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f8e9c0a2c10 0x7f8e9c0a50d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:51.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.341+0000 7f8ebb0c6640 1 --2- 172.21.15.44:0/1246591919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8eb4103340 0x7f8eb419f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:51.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.341+0000 7f8ebb0c6640 1 --2- 172.21.15.44:0/1246591919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8eb4102980 0x7f8eb4197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:51.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.341+0000 7f8ebb0c6640 1 --2- 172.21.15.44:0/1246591919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8eb4101790 0x7f8eb4197a80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:51.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.341+0000 7f8ebb0c6640 1 -- 172.21.15.44:0/1246591919 >> 172.21.15.44:0/1246591919 conn(0x7f8eb40fcd00 msgr2=0x7f8eb4105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:51.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.341+0000 7f8ebb0c6640 1 -- 172.21.15.44:0/1246591919 shutdown_connections 2024-04-21T23:06:51.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.341+0000 7f8ebb0c6640 1 -- 172.21.15.44:0/1246591919 wait complete. 2024-04-21T23:06:51.355 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-04-21T23:06:51.355 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-21T23:06:51.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.467+0000 7f6a780b5640 1 -- 172.21.15.44:0/204899863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a70100680 msgr2=0x7f6a70100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:51.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.467+0000 7f6a780b5640 1 --2- 172.21.15.44:0/204899863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a70100680 0x7f6a70100aa0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009900 tx=0x7f6a6002bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:51.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.467+0000 7f6a780b5640 1 -- 172.21.15.44:0/204899863 shutdown_connections 2024-04-21T23:06:51.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.467+0000 7f6a780b5640 1 --2- 172.21.15.44:0/204899863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a70102230 0x7f6a70108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:51.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.467+0000 7f6a780b5640 1 --2- 172.21.15.44:0/204899863 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a70101870 0x7f6a70101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:51.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.467+0000 7f6a780b5640 1 --2- 172.21.15.44:0/204899863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a70100680 0x7f6a70100aa0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:51.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.467+0000 7f6a780b5640 1 -- 172.21.15.44:0/204899863 >> 172.21.15.44:0/204899863 conn(0x7f6a700fbc10 msgr2=0x7f6a700fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:51.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.467+0000 7f6a780b5640 1 -- 172.21.15.44:0/204899863 shutdown_connections 2024-04-21T23:06:51.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.467+0000 7f6a780b5640 1 -- 172.21.15.44:0/204899863 wait complete. 2024-04-21T23:06:51.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.467+0000 7f6a780b5640 1 Processor -- start 2024-04-21T23:06:51.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.467+0000 7f6a780b5640 1 -- start start 2024-04-21T23:06:51.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.467+0000 7f6a780b5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a70100680 0x7f6a700695b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:51.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a780b5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a70101870 0x7f6a7019c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:51.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a780b5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a70102230 0x7f6a7019edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:51.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a780b5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6a7010c2d0 con 0x7f6a70102230 2024-04-21T23:06:51.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a780b5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6a7010c150 con 0x7f6a70100680 2024-04-21T23:06:51.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a780b5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6a7010c450 con 0x7f6a70101870 2024-04-21T23:06:51.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a76652640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a70102230 0x7f6a7019edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:51.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a76652640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a70102230 0x7f6a7019edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50662/0 (socket says 172.21.15.44:50662) 2024-04-21T23:06:51.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a76652640 1 -- 172.21.15.44:0/953010181 learned_addr learned my addr 172.21.15.44:0/953010181 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:51.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a75e51640 1 --2- 172.21.15.44:0/953010181 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a70100680 0x7f6a700695b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:51.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a75650640 1 --2- 172.21.15.44:0/953010181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a70101870 0x7f6a7019c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:51.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a76652640 1 -- 172.21.15.44:0/953010181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a70101870 msgr2=0x7f6a7019c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:51.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a76652640 1 --2- 172.21.15.44:0/953010181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a70101870 0x7f6a7019c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:51.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a76652640 1 -- 172.21.15.44:0/953010181 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a70100680 msgr2=0x7f6a700695b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:51.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a76652640 1 --2- 172.21.15.44:0/953010181 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a70100680 0x7f6a700695b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:51.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a76652640 1 -- 172.21.15.44:0/953010181 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a7019f6c0 con 0x7f6a70102230 2024-04-21T23:06:51.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a76652640 1 --2- 172.21.15.44:0/953010181 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a70102230 0x7f6a7019edb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6a6800cce0 tx=0x7f6a6800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:51.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a66ffd640 1 -- 172.21.15.44:0/953010181 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a6800f450 con 0x7f6a70102230 2024-04-21T23:06:51.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a66ffd640 1 -- 172.21.15.44:0/953010181 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a6800fad0 con 0x7f6a70102230 2024-04-21T23:06:51.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a780b5640 1 -- 172.21.15.44:0/953010181 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a7019f950 con 0x7f6a70102230 2024-04-21T23:06:51.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a66ffd640 1 -- 172.21.15.44:0/953010181 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a68018c10 con 0x7f6a70102230 2024-04-21T23:06:51.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.468+0000 7f6a780b5640 1 -- 172.21.15.44:0/953010181 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a7019fea0 con 0x7f6a70102230 2024-04-21T23:06:51.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.469+0000 7f6a780b5640 1 -- 172.21.15.44:0/953010181 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a701079c0 con 0x7f6a70102230 2024-04-21T23:06:51.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.471+0000 7f6a66ffd640 1 -- 172.21.15.44:0/953010181 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6a6800f5f0 con 0x7f6a70102230 2024-04-21T23:06:51.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.471+0000 7f6a66ffd640 1 --2- 172.21.15.44:0/953010181 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6a5c0a2c10 0x7f6a5c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:51.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.471+0000 7f6a75e51640 1 --2- 172.21.15.44:0/953010181 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6a5c0a2c10 0x7f6a5c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:51.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.471+0000 7f6a66ffd640 1 -- 172.21.15.44:0/953010181 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f6a680cf820 con 0x7f6a70102230 2024-04-21T23:06:51.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.472+0000 7f6a75e51640 1 --2- 172.21.15.44:0/953010181 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6a5c0a2c10 0x7f6a5c0a50d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6a60000c00 tx=0x7f6a60005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:51.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.476+0000 7f6a66ffd640 1 -- 172.21.15.44:0/953010181 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6a68014030 con 0x7f6a70102230 2024-04-21T23:06:51.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:51 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3025519889' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-21T23:06:51.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:51 smithi044 ceph-mon[27934]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:06:51.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:51 smithi044 ceph-mon[27934]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:06:51.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:51 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1246591919' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-21T23:06:51.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.687+0000 7f6a780b5640 1 -- 172.21.15.44:0/953010181 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f6a70061b80 con 0x7f6a5c0a2c10 2024-04-21T23:06:51.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.708+0000 7f6a66ffd640 1 -- 172.21.15.44:0/953010181 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6a70061b80 con 0x7f6a5c0a2c10 2024-04-21T23:06:51.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.710+0000 7f6a780b5640 1 -- 172.21.15.44:0/953010181 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6a5c0a2c10 msgr2=0x7f6a5c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:51.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.710+0000 7f6a780b5640 1 --2- 172.21.15.44:0/953010181 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6a5c0a2c10 0x7f6a5c0a50d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6a60000c00 tx=0x7f6a60005fd0 comp rx=0 tx=0).stop 2024-04-21T23:06:51.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.710+0000 7f6a780b5640 1 -- 172.21.15.44:0/953010181 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a70102230 msgr2=0x7f6a7019edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:51.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.710+0000 7f6a780b5640 1 --2- 172.21.15.44:0/953010181 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a70102230 0x7f6a7019edb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6a6800cce0 tx=0x7f6a6800a620 comp rx=0 tx=0).stop 2024-04-21T23:06:51.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.710+0000 7f6a780b5640 1 -- 172.21.15.44:0/953010181 shutdown_connections 2024-04-21T23:06:51.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.710+0000 7f6a780b5640 1 --2- 172.21.15.44:0/953010181 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6a5c0a2c10 0x7f6a5c0a50d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:51.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.710+0000 7f6a780b5640 1 --2- 172.21.15.44:0/953010181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a70101870 0x7f6a7019c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:51.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.710+0000 7f6a780b5640 1 --2- 172.21.15.44:0/953010181 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6a70102230 0x7f6a7019edb0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:51.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.710+0000 7f6a780b5640 1 --2- 172.21.15.44:0/953010181 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6a70100680 0x7f6a700695b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:51.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.710+0000 7f6a780b5640 1 -- 172.21.15.44:0/953010181 >> 172.21.15.44:0/953010181 conn(0x7f6a700fbc10 msgr2=0x7f6a70104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:51.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.710+0000 7f6a780b5640 1 -- 172.21.15.44:0/953010181 shutdown_connections 2024-04-21T23:06:51.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.710+0000 7f6a780b5640 1 -- 172.21.15.44:0/953010181 wait complete. 2024-04-21T23:06:51.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:51 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3025519889' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-21T23:06:51.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:51 smithi055 ceph-mon[31117]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:06:51.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:51 smithi055 ceph-mon[31117]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:06:51.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:51 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1246591919' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-21T23:06:51.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:51 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3025519889' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-21T23:06:51.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:51 smithi069 ceph-mon[31816]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:06:51.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:51 smithi069 ceph-mon[31816]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:06:51.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:51 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1246591919' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-21T23:06:51.721 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-21T23:06:51.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.828+0000 7f6e5ffa7640 1 -- 172.21.15.44:0/1518559629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e58100960 msgr2=0x7f6e58100dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:51.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.828+0000 7f6e5ffa7640 1 --2- 172.21.15.44:0/1518559629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e58100960 0x7f6e58100dc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c009900 tx=0x7f6e4c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:51.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.828+0000 7f6e5ffa7640 1 -- 172.21.15.44:0/1518559629 shutdown_connections 2024-04-21T23:06:51.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.828+0000 7f6e5ffa7640 1 --2- 172.21.15.44:0/1518559629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e58101300 0x7f6e58101780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:51.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.828+0000 7f6e5ffa7640 1 --2- 172.21.15.44:0/1518559629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e58100960 0x7f6e58100dc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:51.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.828+0000 7f6e5ffa7640 1 --2- 172.21.15.44:0/1518559629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e58100000 0x7f6e58100420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:51.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.828+0000 7f6e5ffa7640 1 -- 172.21.15.44:0/1518559629 >> 172.21.15.44:0/1518559629 conn(0x7f6e580fbc20 msgr2=0x7f6e580fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:51.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.828+0000 7f6e5ffa7640 1 -- 172.21.15.44:0/1518559629 shutdown_connections 2024-04-21T23:06:51.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.828+0000 7f6e5ffa7640 1 -- 172.21.15.44:0/1518559629 wait complete. 2024-04-21T23:06:51.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.829+0000 7f6e5ffa7640 1 Processor -- start 2024-04-21T23:06:51.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.829+0000 7f6e5ffa7640 1 -- start start 2024-04-21T23:06:51.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.829+0000 7f6e5ffa7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e58100000 0x7f6e58199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:51.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.829+0000 7f6e5ffa7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e58100960 0x7f6e5819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:51.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.829+0000 7f6e5ffa7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e58101300 0x7f6e581a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:51.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.829+0000 7f6e5ffa7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5810f5d0 con 0x7f6e58100960 2024-04-21T23:06:51.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.829+0000 7f6e5ffa7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5810f450 con 0x7f6e58101300 2024-04-21T23:06:51.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.829+0000 7f6e5ffa7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5810f750 con 0x7f6e58100000 2024-04-21T23:06:51.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.829+0000 7f6e5d542640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e58100960 0x7f6e5819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:51.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.829+0000 7f6e5d542640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e58100960 0x7f6e5819a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50694/0 (socket says 172.21.15.44:50694) 2024-04-21T23:06:51.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.829+0000 7f6e5d542640 1 -- 172.21.15.44:0/4219565963 learned_addr learned my addr 172.21.15.44:0/4219565963 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:51.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.829+0000 7f6e5dd43640 1 --2- 172.21.15.44:0/4219565963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e58100000 0x7f6e58199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:51.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.829+0000 7f6e5e544640 1 --2- 172.21.15.44:0/4219565963 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e58101300 0x7f6e581a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:51.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.829+0000 7f6e5d542640 1 -- 172.21.15.44:0/4219565963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e58100000 msgr2=0x7f6e58199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:51.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.829+0000 7f6e5d542640 1 --2- 172.21.15.44:0/4219565963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e58100000 0x7f6e58199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:51.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.829+0000 7f6e5d542640 1 -- 172.21.15.44:0/4219565963 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e58101300 msgr2=0x7f6e581a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:51.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.829+0000 7f6e5d542640 1 --2- 172.21.15.44:0/4219565963 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e58101300 0x7f6e581a1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:51.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.829+0000 7f6e5d542640 1 -- 172.21.15.44:0/4219565963 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e581a18c0 con 0x7f6e58100960 2024-04-21T23:06:51.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.829+0000 7f6e5d542640 1 --2- 172.21.15.44:0/4219565963 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e58100960 0x7f6e5819a1c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c009230 tx=0x7f6e4c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:51.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.829+0000 7f6e56ffd640 1 -- 172.21.15.44:0/4219565963 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c041070 con 0x7f6e58100960 2024-04-21T23:06:51.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.830+0000 7f6e56ffd640 1 -- 172.21.15.44:0/4219565963 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e4c0054e0 con 0x7f6e58100960 2024-04-21T23:06:51.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.830+0000 7f6e5ffa7640 1 -- 172.21.15.44:0/4219565963 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e581a1b50 con 0x7f6e58100960 2024-04-21T23:06:51.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.830+0000 7f6e56ffd640 1 -- 172.21.15.44:0/4219565963 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c034440 con 0x7f6e58100960 2024-04-21T23:06:51.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.830+0000 7f6e5ffa7640 1 -- 172.21.15.44:0/4219565963 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e581a2090 con 0x7f6e58100960 2024-04-21T23:06:51.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.830+0000 7f6e5ffa7640 1 -- 172.21.15.44:0/4219565963 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e58068a70 con 0x7f6e58100960 2024-04-21T23:06:51.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.832+0000 7f6e56ffd640 1 -- 172.21.15.44:0/4219565963 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6e4c044070 con 0x7f6e58100960 2024-04-21T23:06:51.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.832+0000 7f6e56ffd640 1 --2- 172.21.15.44:0/4219565963 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6e400a2c90 0x7f6e400a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:51.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.833+0000 7f6e5dd43640 1 --2- 172.21.15.44:0/4219565963 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6e400a2c90 0x7f6e400a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:51.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.833+0000 7f6e56ffd640 1 -- 172.21.15.44:0/4219565963 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f6e4c0f1400 con 0x7f6e58100960 2024-04-21T23:06:51.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.833+0000 7f6e5dd43640 1 --2- 172.21.15.44:0/4219565963 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6e400a2c90 0x7f6e400a5150 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6e48005c40 tx=0x7f6e4800a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:51.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:51.836+0000 7f6e56ffd640 1 -- 172.21.15.44:0/4219565963 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6e58068a70 con 0x7f6e58100960 2024-04-21T23:06:52.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.045+0000 7f6e5ffa7640 1 -- 172.21.15.44:0/4219565963 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f6e58061b80 con 0x7f6e400a2c90 2024-04-21T23:06:52.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.071+0000 7f6e56ffd640 1 -- 172.21.15.44:0/4219565963 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6e58061b80 con 0x7f6e400a2c90 2024-04-21T23:06:52.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.073+0000 7f6e5ffa7640 1 -- 172.21.15.44:0/4219565963 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6e400a2c90 msgr2=0x7f6e400a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:52.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.074+0000 7f6e5ffa7640 1 --2- 172.21.15.44:0/4219565963 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6e400a2c90 0x7f6e400a5150 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6e48005c40 tx=0x7f6e4800a3b0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.074+0000 7f6e5ffa7640 1 -- 172.21.15.44:0/4219565963 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e58100960 msgr2=0x7f6e5819a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.074+0000 7f6e5ffa7640 1 --2- 172.21.15.44:0/4219565963 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e58100960 0x7f6e5819a1c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c009230 tx=0x7f6e4c005dd0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.074+0000 7f6e5ffa7640 1 -- 172.21.15.44:0/4219565963 shutdown_connections 2024-04-21T23:06:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.074+0000 7f6e5ffa7640 1 --2- 172.21.15.44:0/4219565963 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6e400a2c90 0x7f6e400a5150 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.074+0000 7f6e5ffa7640 1 --2- 172.21.15.44:0/4219565963 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e58101300 0x7f6e581a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.074+0000 7f6e5ffa7640 1 --2- 172.21.15.44:0/4219565963 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e58100960 0x7f6e5819a1c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.074+0000 7f6e5ffa7640 1 --2- 172.21.15.44:0/4219565963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e58100000 0x7f6e58199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.074+0000 7f6e5ffa7640 1 -- 172.21.15.44:0/4219565963 >> 172.21.15.44:0/4219565963 conn(0x7f6e580fbc20 msgr2=0x7f6e580fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.074+0000 7f6e5ffa7640 1 -- 172.21.15.44:0/4219565963 shutdown_connections 2024-04-21T23:06:52.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.074+0000 7f6e5ffa7640 1 -- 172.21.15.44:0/4219565963 wait complete. 2024-04-21T23:06:52.083 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:06:51 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:06:51.692+0000 7f38cb426700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:06:52.083 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:06:51 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:06:51.692+0000 7f38cb426700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:06:52.083 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:06:51 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:06:51.692+0000 7f38cb426700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:06:52.083 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:06:51 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:06:51.692+0000 7f38cb426700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:06:52.083 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:06:51 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:06:51.692+0000 7f38cb426700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:06:52.084 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-21T23:06:52.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.191+0000 7f2ef3908640 1 -- 172.21.15.44:0/4054118066 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eec101880 msgr2=0x7f2eec101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:52.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.191+0000 7f2ef3908640 1 --2- 172.21.15.44:0/4054118066 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eec101880 0x7f2eec101d00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0009900 tx=0x7f2ee002bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:52.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.191+0000 7f2ef3908640 1 -- 172.21.15.44:0/4054118066 shutdown_connections 2024-04-21T23:06:52.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.191+0000 7f2ef3908640 1 --2- 172.21.15.44:0/4054118066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eec102240 0x7f2eec108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.191+0000 7f2ef3908640 1 --2- 172.21.15.44:0/4054118066 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eec101880 0x7f2eec101d00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.191+0000 7f2ef3908640 1 --2- 172.21.15.44:0/4054118066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eec100690 0x7f2eec100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.191+0000 7f2ef3908640 1 -- 172.21.15.44:0/4054118066 >> 172.21.15.44:0/4054118066 conn(0x7f2eec0fbc20 msgr2=0x7f2eec0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:52.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.191+0000 7f2ef3908640 1 -- 172.21.15.44:0/4054118066 shutdown_connections 2024-04-21T23:06:52.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.191+0000 7f2ef3908640 1 -- 172.21.15.44:0/4054118066 wait complete. 2024-04-21T23:06:52.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.192+0000 7f2ef3908640 1 Processor -- start 2024-04-21T23:06:52.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.192+0000 7f2ef3908640 1 -- start start 2024-04-21T23:06:52.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.192+0000 7f2ef3908640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eec100690 0x7f2eec197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:52.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.192+0000 7f2ef3908640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eec101880 0x7f2eec198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:52.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.192+0000 7f2ef3908640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eec102240 0x7f2eec19f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:52.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.192+0000 7f2ef3908640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec10c370 con 0x7f2eec102240 2024-04-21T23:06:52.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.192+0000 7f2ef3908640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec10c1f0 con 0x7f2eec101880 2024-04-21T23:06:52.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.192+0000 7f2ef3908640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec10c4f0 con 0x7f2eec100690 2024-04-21T23:06:52.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.192+0000 7f2ef1ea5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eec102240 0x7f2eec19f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:52.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.192+0000 7f2ef1ea5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eec102240 0x7f2eec19f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50710/0 (socket says 172.21.15.44:50710) 2024-04-21T23:06:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.192+0000 7f2ef1ea5640 1 -- 172.21.15.44:0/830968439 learned_addr learned my addr 172.21.15.44:0/830968439 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.192+0000 7f2ef16a4640 1 --2- 172.21.15.44:0/830968439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eec100690 0x7f2eec197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.192+0000 7f2ef0ea3640 1 --2- 172.21.15.44:0/830968439 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eec101880 0x7f2eec198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.193+0000 7f2ef1ea5640 1 -- 172.21.15.44:0/830968439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eec100690 msgr2=0x7f2eec197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.193+0000 7f2ef1ea5640 1 --2- 172.21.15.44:0/830968439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eec100690 0x7f2eec197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.193+0000 7f2ef1ea5640 1 -- 172.21.15.44:0/830968439 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eec101880 msgr2=0x7f2eec198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.193+0000 7f2ef1ea5640 1 --2- 172.21.15.44:0/830968439 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eec101880 0x7f2eec198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.193+0000 7f2ef1ea5640 1 -- 172.21.15.44:0/830968439 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eec19f760 con 0x7f2eec102240 2024-04-21T23:06:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.193+0000 7f2ef0ea3640 1 --2- 172.21.15.44:0/830968439 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eec101880 0x7f2eec198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:06:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.193+0000 7f2ef1ea5640 1 --2- 172.21.15.44:0/830968439 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eec102240 0x7f2eec19f0f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4009fd0 tx=0x7f2ee400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.193+0000 7f2eea7fc640 1 -- 172.21.15.44:0/830968439 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4011830 con 0x7f2eec102240 2024-04-21T23:06:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.193+0000 7f2ef3908640 1 -- 172.21.15.44:0/830968439 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eec19fa50 con 0x7f2eec102240 2024-04-21T23:06:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.193+0000 7f2eea7fc640 1 -- 172.21.15.44:0/830968439 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ee4016070 con 0x7f2eec102240 2024-04-21T23:06:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.193+0000 7f2eea7fc640 1 -- 172.21.15.44:0/830968439 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee400f2e0 con 0x7f2eec102240 2024-04-21T23:06:52.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.193+0000 7f2ef3908640 1 -- 172.21.15.44:0/830968439 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eec19ff20 con 0x7f2eec102240 2024-04-21T23:06:52.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.194+0000 7f2ef3908640 1 -- 172.21.15.44:0/830968439 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eec1a01b0 con 0x7f2eec102240 2024-04-21T23:06:52.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.195+0000 7f2eea7fc640 1 -- 172.21.15.44:0/830968439 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2ee40119d0 con 0x7f2eec102240 2024-04-21T23:06:52.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.196+0000 7f2eea7fc640 1 --2- 172.21.15.44:0/830968439 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2ed40a2b90 0x7f2ed40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:52.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.196+0000 7f2ef16a4640 1 --2- 172.21.15.44:0/830968439 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2ed40a2b90 0x7f2ed40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:52.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.196+0000 7f2eea7fc640 1 -- 172.21.15.44:0/830968439 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f2ee40ce950 con 0x7f2eec102240 2024-04-21T23:06:52.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.196+0000 7f2ef16a4640 1 --2- 172.21.15.44:0/830968439 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2ed40a2b90 0x7f2ed40a5050 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2eec1016e0 tx=0x7f2edc00a590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:52.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.200+0000 7f2eea7fc640 1 -- 172.21.15.44:0/830968439 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2ee409b610 con 0x7f2eec102240 2024-04-21T23:06:52.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.415+0000 7f2ef3908640 1 -- 172.21.15.44:0/830968439 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f2eec1a0380 con 0x7f2ed40a2b90 2024-04-21T23:06:52.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.420+0000 7f2eea7fc640 1 -- 172.21.15.44:0/830968439 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f2eec1a0380 con 0x7f2ed40a2b90 2024-04-21T23:06:52.419 INFO:teuthology.orchestra.run.smithi044.stdout:/volumes/qa/sv_0/a18e91c0-28b3-4949-9346-ed4059df7ea1 2024-04-21T23:06:52.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.422+0000 7f2ef3908640 1 -- 172.21.15.44:0/830968439 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2ed40a2b90 msgr2=0x7f2ed40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:52.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.422+0000 7f2ef3908640 1 --2- 172.21.15.44:0/830968439 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2ed40a2b90 0x7f2ed40a5050 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2eec1016e0 tx=0x7f2edc00a590 comp rx=0 tx=0).stop 2024-04-21T23:06:52.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.422+0000 7f2ef3908640 1 -- 172.21.15.44:0/830968439 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eec102240 msgr2=0x7f2eec19f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:52.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.422+0000 7f2ef3908640 1 --2- 172.21.15.44:0/830968439 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eec102240 0x7f2eec19f0f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4009fd0 tx=0x7f2ee400b8e0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.422+0000 7f2ef3908640 1 -- 172.21.15.44:0/830968439 shutdown_connections 2024-04-21T23:06:52.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.422+0000 7f2ef3908640 1 --2- 172.21.15.44:0/830968439 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2ed40a2b90 0x7f2ed40a5050 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.422+0000 7f2ef3908640 1 --2- 172.21.15.44:0/830968439 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2eec102240 0x7f2eec19f0f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.422+0000 7f2ef3908640 1 --2- 172.21.15.44:0/830968439 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2eec101880 0x7f2eec198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.422+0000 7f2ef3908640 1 --2- 172.21.15.44:0/830968439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eec100690 0x7f2eec197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.422+0000 7f2ef3908640 1 -- 172.21.15.44:0/830968439 >> 172.21.15.44:0/830968439 conn(0x7f2eec0fbc20 msgr2=0x7f2eec0fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:52.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.422+0000 7f2ef3908640 1 -- 172.21.15.44:0/830968439 shutdown_connections 2024-04-21T23:06:52.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.422+0000 7f2ef3908640 1 -- 172.21.15.44:0/830968439 wait complete. 2024-04-21T23:06:52.433 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-21T23:06:52.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.539+0000 7f6e48977640 1 -- 172.21.15.44:0/2310744883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e44100690 msgr2=0x7f6e44100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:52.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.539+0000 7f6e48977640 1 --2- 172.21.15.44:0/2310744883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e44100690 0x7f6e44100ab0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c009ca0 tx=0x7f6e3c02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.539+0000 7f6e48977640 1 -- 172.21.15.44:0/2310744883 shutdown_connections 2024-04-21T23:06:52.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.539+0000 7f6e48977640 1 --2- 172.21.15.44:0/2310744883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e44102240 0x7f6e44108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.539+0000 7f6e48977640 1 --2- 172.21.15.44:0/2310744883 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e44101880 0x7f6e44101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.539+0000 7f6e48977640 1 --2- 172.21.15.44:0/2310744883 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e44100690 0x7f6e44100ab0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.539+0000 7f6e48977640 1 -- 172.21.15.44:0/2310744883 >> 172.21.15.44:0/2310744883 conn(0x7f6e440fbc20 msgr2=0x7f6e440fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:52.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.539+0000 7f6e48977640 1 -- 172.21.15.44:0/2310744883 shutdown_connections 2024-04-21T23:06:52.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.539+0000 7f6e48977640 1 -- 172.21.15.44:0/2310744883 wait complete. 2024-04-21T23:06:52.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.540+0000 7f6e48977640 1 Processor -- start 2024-04-21T23:06:52.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.540+0000 7f6e48977640 1 -- start start 2024-04-21T23:06:52.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.540+0000 7f6e48977640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e44100690 0x7f6e44196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:52.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.540+0000 7f6e48977640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e44101880 0x7f6e44196eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:52.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.540+0000 7f6e48977640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e44102240 0x7f6e4419df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:52.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.540+0000 7f6e48977640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4410c2e0 con 0x7f6e44102240 2024-04-21T23:06:52.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.540+0000 7f6e48977640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4410c160 con 0x7f6e44100690 2024-04-21T23:06:52.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.540+0000 7f6e48977640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4410c460 con 0x7f6e44101880 2024-04-21T23:06:52.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.540+0000 7f6e42d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e44102240 0x7f6e4419df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:52.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.540+0000 7f6e42d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e44102240 0x7f6e4419df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50714/0 (socket says 172.21.15.44:50714) 2024-04-21T23:06:52.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.540+0000 7f6e42d9d640 1 -- 172.21.15.44:0/1160719432 learned_addr learned my addr 172.21.15.44:0/1160719432 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:52.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.541+0000 7f6e4259c640 1 --2- 172.21.15.44:0/1160719432 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e44100690 0x7f6e44196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:52.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.541+0000 7f6e41d9b640 1 --2- 172.21.15.44:0/1160719432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e44101880 0x7f6e44196eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:52.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.541+0000 7f6e42d9d640 1 -- 172.21.15.44:0/1160719432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e44101880 msgr2=0x7f6e44196eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:52.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.541+0000 7f6e42d9d640 1 --2- 172.21.15.44:0/1160719432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e44101880 0x7f6e44196eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.541+0000 7f6e42d9d640 1 -- 172.21.15.44:0/1160719432 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e44100690 msgr2=0x7f6e44196970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:52.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.541+0000 7f6e42d9d640 1 --2- 172.21.15.44:0/1160719432 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e44100690 0x7f6e44196970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.541+0000 7f6e42d9d640 1 -- 172.21.15.44:0/1160719432 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e4419e590 con 0x7f6e44102240 2024-04-21T23:06:52.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.541+0000 7f6e41d9b640 1 --2- 172.21.15.44:0/1160719432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e44101880 0x7f6e44196eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:06:52.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.541+0000 7f6e42d9d640 1 --2- 172.21.15.44:0/1160719432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e44102240 0x7f6e4419df20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6e3400d950 tx=0x7f6e3400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:52.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.541+0000 7f6e337fe640 1 -- 172.21.15.44:0/1160719432 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e34017030 con 0x7f6e44102240 2024-04-21T23:06:52.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.541+0000 7f6e337fe640 1 -- 172.21.15.44:0/1160719432 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e34004020 con 0x7f6e44102240 2024-04-21T23:06:52.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.541+0000 7f6e48977640 1 -- 172.21.15.44:0/1160719432 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e4419e880 con 0x7f6e44102240 2024-04-21T23:06:52.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.541+0000 7f6e337fe640 1 -- 172.21.15.44:0/1160719432 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e34003c00 con 0x7f6e44102240 2024-04-21T23:06:52.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.541+0000 7f6e48977640 1 -- 172.21.15.44:0/1160719432 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e4419ed80 con 0x7f6e44102240 2024-04-21T23:06:52.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.542+0000 7f6e48977640 1 -- 172.21.15.44:0/1160719432 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e44107a50 con 0x7f6e44102240 2024-04-21T23:06:52.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.543+0000 7f6e337fe640 1 -- 172.21.15.44:0/1160719432 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6e3401a070 con 0x7f6e44102240 2024-04-21T23:06:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.544+0000 7f6e337fe640 1 --2- 172.21.15.44:0/1160719432 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6e2c0a2b90 0x7f6e2c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.544+0000 7f6e4259c640 1 --2- 172.21.15.44:0/1160719432 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6e2c0a2b90 0x7f6e2c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.544+0000 7f6e337fe640 1 -- 172.21.15.44:0/1160719432 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f6e340cebd0 con 0x7f6e44102240 2024-04-21T23:06:52.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.544+0000 7f6e4259c640 1 --2- 172.21.15.44:0/1160719432 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6e2c0a2b90 0x7f6e2c0a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c02c630 tx=0x7f6e3c009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:52.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.548+0000 7f6e337fe640 1 -- 172.21.15.44:0/1160719432 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6e34021070 con 0x7f6e44102240 2024-04-21T23:06:52.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:52 smithi044 ceph-mon[27934]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:06:52.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:52 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1246591919' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-21T23:06:52.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:52 smithi044 ceph-mon[27934]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:06:52.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:52 smithi044 ceph-mon[27934]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:06:52.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:52 smithi055 ceph-mon[31117]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:06:52.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:52 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1246591919' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-21T23:06:52.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:52 smithi055 ceph-mon[31117]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:06:52.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:52 smithi055 ceph-mon[31117]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:06:52.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:52 smithi069 ceph-mon[31816]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T23:06:52.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:52 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1246591919' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-21T23:06:52.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:52 smithi069 ceph-mon[31816]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:06:52.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:52 smithi069 ceph-mon[31816]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:06:52.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.760+0000 7f6e48977640 1 -- 172.21.15.44:0/1160719432 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f6e44061b80 con 0x7f6e2c0a2b90 2024-04-21T23:06:52.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.784+0000 7f6e337fe640 1 -- 172.21.15.44:0/1160719432 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6e44061b80 con 0x7f6e2c0a2b90 2024-04-21T23:06:52.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.786+0000 7f6e48977640 1 -- 172.21.15.44:0/1160719432 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6e2c0a2b90 msgr2=0x7f6e2c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:52.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.786+0000 7f6e48977640 1 --2- 172.21.15.44:0/1160719432 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6e2c0a2b90 0x7f6e2c0a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c02c630 tx=0x7f6e3c009360 comp rx=0 tx=0).stop 2024-04-21T23:06:52.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.786+0000 7f6e48977640 1 -- 172.21.15.44:0/1160719432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e44102240 msgr2=0x7f6e4419df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:52.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.786+0000 7f6e48977640 1 --2- 172.21.15.44:0/1160719432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e44102240 0x7f6e4419df20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6e3400d950 tx=0x7f6e3400de10 comp rx=0 tx=0).stop 2024-04-21T23:06:52.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.786+0000 7f6e48977640 1 -- 172.21.15.44:0/1160719432 shutdown_connections 2024-04-21T23:06:52.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.786+0000 7f6e48977640 1 --2- 172.21.15.44:0/1160719432 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f6e2c0a2b90 0x7f6e2c0a5050 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.786 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.786+0000 7f6e48977640 1 --2- 172.21.15.44:0/1160719432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e44101880 0x7f6e44196eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.786+0000 7f6e48977640 1 --2- 172.21.15.44:0/1160719432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e44102240 0x7f6e4419df20 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.786+0000 7f6e48977640 1 --2- 172.21.15.44:0/1160719432 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e44100690 0x7f6e44196970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.786+0000 7f6e48977640 1 -- 172.21.15.44:0/1160719432 >> 172.21.15.44:0/1160719432 conn(0x7f6e440fbc20 msgr2=0x7f6e44104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:52.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.786+0000 7f6e48977640 1 -- 172.21.15.44:0/1160719432 shutdown_connections 2024-04-21T23:06:52.787 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.786+0000 7f6e48977640 1 -- 172.21.15.44:0/1160719432 wait complete. 2024-04-21T23:06:52.797 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-21T23:06:52.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.903+0000 7fda0c00e640 1 -- 172.21.15.44:0/93243802 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda04100680 msgr2=0x7fda04100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:52.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.903+0000 7fda0c00e640 1 --2- 172.21.15.44:0/93243802 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda04100680 0x7fda04100aa0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fda00009900 tx=0x7fda0002bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:52.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.903+0000 7fda0c00e640 1 -- 172.21.15.44:0/93243802 shutdown_connections 2024-04-21T23:06:52.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.903+0000 7fda0c00e640 1 --2- 172.21.15.44:0/93243802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda04101870 0x7fda04101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.903+0000 7fda0c00e640 1 --2- 172.21.15.44:0/93243802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda04102230 0x7fda04108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.903+0000 7fda0c00e640 1 --2- 172.21.15.44:0/93243802 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda04100680 0x7fda04100aa0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.903+0000 7fda0c00e640 1 -- 172.21.15.44:0/93243802 >> 172.21.15.44:0/93243802 conn(0x7fda040fbc10 msgr2=0x7fda040fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:52.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.903+0000 7fda0c00e640 1 -- 172.21.15.44:0/93243802 shutdown_connections 2024-04-21T23:06:52.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.903+0000 7fda0c00e640 1 -- 172.21.15.44:0/93243802 wait complete. 2024-04-21T23:06:52.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.904+0000 7fda0c00e640 1 Processor -- start 2024-04-21T23:06:52.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.904+0000 7fda0c00e640 1 -- start start 2024-04-21T23:06:52.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.904+0000 7fda0c00e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda04100680 0x7fda04104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:52.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.904+0000 7fda0c00e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda04101870 0x7fda04102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:52.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.904+0000 7fda0c00e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda04102230 0x7fda04103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:52.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.904+0000 7fda0c00e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fda0410c350 con 0x7fda04102230 2024-04-21T23:06:52.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.904+0000 7fda0c00e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fda0410c1d0 con 0x7fda04100680 2024-04-21T23:06:52.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.904+0000 7fda0c00e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fda0410c4d0 con 0x7fda04101870 2024-04-21T23:06:52.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.904+0000 7fda0a5ab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda04102230 0x7fda04103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:52.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.904+0000 7fda0a5ab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda04102230 0x7fda04103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50728/0 (socket says 172.21.15.44:50728) 2024-04-21T23:06:52.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.904+0000 7fda0a5ab640 1 -- 172.21.15.44:0/590285871 learned_addr learned my addr 172.21.15.44:0/590285871 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:52.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.904+0000 7fda09daa640 1 --2- 172.21.15.44:0/590285871 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda04100680 0x7fda04104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:52.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.904+0000 7fda095a9640 1 --2- 172.21.15.44:0/590285871 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda04101870 0x7fda04102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:52.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.905+0000 7fda0a5ab640 1 -- 172.21.15.44:0/590285871 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda04101870 msgr2=0x7fda04102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:52.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.905+0000 7fda0a5ab640 1 --2- 172.21.15.44:0/590285871 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda04101870 0x7fda04102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.905+0000 7fda0a5ab640 1 -- 172.21.15.44:0/590285871 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda04100680 msgr2=0x7fda04104610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:52.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.905+0000 7fda0a5ab640 1 --2- 172.21.15.44:0/590285871 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda04100680 0x7fda04104610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:52.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.905+0000 7fda0a5ab640 1 -- 172.21.15.44:0/590285871 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda04103a20 con 0x7fda04102230 2024-04-21T23:06:52.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.905+0000 7fda095a9640 1 --2- 172.21.15.44:0/590285871 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda04101870 0x7fda04102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:06:52.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.905+0000 7fda0a5ab640 1 --2- 172.21.15.44:0/590285871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda04102230 0x7fda04103190 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc00cce0 tx=0x7fd9fc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:52.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.905+0000 7fd9faffd640 1 -- 172.21.15.44:0/590285871 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc00f450 con 0x7fda04102230 2024-04-21T23:06:52.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.905+0000 7fda0c00e640 1 -- 172.21.15.44:0/590285871 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda041a1c10 con 0x7fda04102230 2024-04-21T23:06:52.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.905+0000 7fd9faffd640 1 -- 172.21.15.44:0/590285871 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9fc00fad0 con 0x7fda04102230 2024-04-21T23:06:52.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.905+0000 7fd9faffd640 1 -- 172.21.15.44:0/590285871 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc018c10 con 0x7fda04102230 2024-04-21T23:06:52.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.905+0000 7fda0c00e640 1 -- 172.21.15.44:0/590285871 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda041a2160 con 0x7fda04102230 2024-04-21T23:06:52.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.906+0000 7fda0c00e640 1 -- 172.21.15.44:0/590285871 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda041a2390 con 0x7fda04102230 2024-04-21T23:06:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.907+0000 7fd9faffd640 1 -- 172.21.15.44:0/590285871 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd9fc00f5f0 con 0x7fda04102230 2024-04-21T23:06:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.908+0000 7fd9faffd640 1 --2- 172.21.15.44:0/590285871 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd9f00a2b90 0x7fd9f00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.908+0000 7fda09daa640 1 --2- 172.21.15.44:0/590285871 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd9f00a2b90 0x7fd9f00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.908+0000 7fd9faffd640 1 -- 172.21.15.44:0/590285871 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fd9fc0cf9c0 con 0x7fda04102230 2024-04-21T23:06:52.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.908+0000 7fda09daa640 1 --2- 172.21.15.44:0/590285871 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd9f00a2b90 0x7fd9f00a5050 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fda00000c00 tx=0x7fda00004f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:52.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:52.912+0000 7fd9faffd640 1 -- 172.21.15.44:0/590285871 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd9fc0d1e20 con 0x7fda04102230 2024-04-21T23:06:53.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.127+0000 7fda0c00e640 1 -- 172.21.15.44:0/590285871 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fda040fde00 con 0x7fd9f00a2b90 2024-04-21T23:06:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.134+0000 7fd9faffd640 1 -- 172.21.15.44:0/590285871 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fda040fde00 con 0x7fd9f00a2b90 2024-04-21T23:06:53.133 INFO:teuthology.orchestra.run.smithi044.stdout:/volumes/qa/sv_1/52aa91e0-dc47-4a01-8a26-3fb0130744af 2024-04-21T23:06:53.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.136+0000 7fda0c00e640 1 -- 172.21.15.44:0/590285871 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd9f00a2b90 msgr2=0x7fd9f00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:53.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.136+0000 7fda0c00e640 1 --2- 172.21.15.44:0/590285871 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd9f00a2b90 0x7fd9f00a5050 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fda00000c00 tx=0x7fda00004f10 comp rx=0 tx=0).stop 2024-04-21T23:06:53.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.136+0000 7fda0c00e640 1 -- 172.21.15.44:0/590285871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda04102230 msgr2=0x7fda04103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:53.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.136+0000 7fda0c00e640 1 --2- 172.21.15.44:0/590285871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda04102230 0x7fda04103190 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc00cce0 tx=0x7fd9fc00a620 comp rx=0 tx=0).stop 2024-04-21T23:06:53.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.136+0000 7fda0c00e640 1 -- 172.21.15.44:0/590285871 shutdown_connections 2024-04-21T23:06:53.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.136+0000 7fda0c00e640 1 --2- 172.21.15.44:0/590285871 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd9f00a2b90 0x7fd9f00a5050 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.136+0000 7fda0c00e640 1 --2- 172.21.15.44:0/590285871 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda04101870 0x7fda04102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.136+0000 7fda0c00e640 1 --2- 172.21.15.44:0/590285871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fda04102230 0x7fda04103190 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.136+0000 7fda0c00e640 1 --2- 172.21.15.44:0/590285871 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fda04100680 0x7fda04104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.136+0000 7fda0c00e640 1 -- 172.21.15.44:0/590285871 >> 172.21.15.44:0/590285871 conn(0x7fda040fbc10 msgr2=0x7fda040fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:53.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.136+0000 7fda0c00e640 1 -- 172.21.15.44:0/590285871 shutdown_connections 2024-04-21T23:06:53.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.136+0000 7fda0c00e640 1 -- 172.21.15.44:0/590285871 wait complete. 2024-04-21T23:06:53.147 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:06:53.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.250+0000 7f3912d9a640 1 -- 172.21.15.44:0/2932387431 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f390c103350 msgr2=0x7f390c109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:53.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.250+0000 7f3912d9a640 1 --2- 172.21.15.44:0/2932387431 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f390c103350 0x7f390c109be0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3904009ca0 tx=0x7f390402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.250+0000 7f3912d9a640 1 -- 172.21.15.44:0/2932387431 shutdown_connections 2024-04-21T23:06:53.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.250+0000 7f3912d9a640 1 --2- 172.21.15.44:0/2932387431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f390c102990 0x7f390c102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.250+0000 7f3912d9a640 1 --2- 172.21.15.44:0/2932387431 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f390c103350 0x7f390c109be0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.250+0000 7f3912d9a640 1 --2- 172.21.15.44:0/2932387431 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f390c1017a0 0x7f390c101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.250+0000 7f3912d9a640 1 -- 172.21.15.44:0/2932387431 >> 172.21.15.44:0/2932387431 conn(0x7f390c0fcd30 msgr2=0x7f390c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:53.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.250+0000 7f3912d9a640 1 -- 172.21.15.44:0/2932387431 shutdown_connections 2024-04-21T23:06:53.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.250+0000 7f3912d9a640 1 -- 172.21.15.44:0/2932387431 wait complete. 2024-04-21T23:06:53.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.250+0000 7f3912d9a640 1 Processor -- start 2024-04-21T23:06:53.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.250+0000 7f3912d9a640 1 -- start start 2024-04-21T23:06:53.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.250+0000 7f3912d9a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f390c1017a0 0x7f390c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:53.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.250+0000 7f3912d9a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f390c102990 0x7f390c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:53.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.250+0000 7f3912d9a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f390c103350 0x7f390c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:53.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.250+0000 7f3912d9a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f390c10d420 con 0x7f390c103350 2024-04-21T23:06:53.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.250+0000 7f3912d9a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f390c10d2a0 con 0x7f390c1017a0 2024-04-21T23:06:53.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.250+0000 7f3912d9a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f390c10d5a0 con 0x7f390c102990 2024-04-21T23:06:53.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.251+0000 7f3911337640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f390c103350 0x7f390c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:53.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.251+0000 7f3911337640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f390c103350 0x7f390c19efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50758/0 (socket says 172.21.15.44:50758) 2024-04-21T23:06:53.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.251+0000 7f3911337640 1 -- 172.21.15.44:0/2866041990 learned_addr learned my addr 172.21.15.44:0/2866041990 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:53.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.251+0000 7f3910b36640 1 --2- 172.21.15.44:0/2866041990 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f390c1017a0 0x7f390c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:53.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.251+0000 7f390bfff640 1 --2- 172.21.15.44:0/2866041990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f390c102990 0x7f390c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:53.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.251+0000 7f3911337640 1 -- 172.21.15.44:0/2866041990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f390c102990 msgr2=0x7f390c197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:53.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.251+0000 7f3911337640 1 --2- 172.21.15.44:0/2866041990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f390c102990 0x7f390c197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.251+0000 7f3911337640 1 -- 172.21.15.44:0/2866041990 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f390c1017a0 msgr2=0x7f390c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:53.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.251+0000 7f3911337640 1 --2- 172.21.15.44:0/2866041990 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f390c1017a0 0x7f390c197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.251+0000 7f3911337640 1 -- 172.21.15.44:0/2866041990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f390c19f650 con 0x7f390c103350 2024-04-21T23:06:53.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.251+0000 7f3910b36640 1 --2- 172.21.15.44:0/2866041990 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f390c1017a0 0x7f390c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:06:53.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.251+0000 7f390bfff640 1 --2- 172.21.15.44:0/2866041990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f390c102990 0x7f390c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:06:53.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.251+0000 7f3911337640 1 --2- 172.21.15.44:0/2866041990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f390c103350 0x7f390c19efe0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3904005e30 tx=0x7f3904032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:53.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.251+0000 7f3909ffb640 1 -- 172.21.15.44:0/2866041990 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3904041020 con 0x7f390c103350 2024-04-21T23:06:53.254 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.251+0000 7f3912d9a640 1 -- 172.21.15.44:0/2866041990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f390c19f8e0 con 0x7f390c103350 2024-04-21T23:06:53.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.251+0000 7f3909ffb640 1 -- 172.21.15.44:0/2866041990 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3904034cd0 con 0x7f390c103350 2024-04-21T23:06:53.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.251+0000 7f3909ffb640 1 -- 172.21.15.44:0/2866041990 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3904039040 con 0x7f390c103350 2024-04-21T23:06:53.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.251+0000 7f3912d9a640 1 -- 172.21.15.44:0/2866041990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f390c19fe20 con 0x7f390c103350 2024-04-21T23:06:53.255 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.252+0000 7f3912d9a640 1 -- 172.21.15.44:0/2866041990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f390c108a60 con 0x7f390c103350 2024-04-21T23:06:53.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.254+0000 7f3909ffb640 1 -- 172.21.15.44:0/2866041990 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3904044020 con 0x7f390c103350 2024-04-21T23:06:53.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.254+0000 7f3909ffb640 1 --2- 172.21.15.44:0/2866041990 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f38fc0a2b90 0x7f38fc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:53.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.254+0000 7f3910b36640 1 --2- 172.21.15.44:0/2866041990 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f38fc0a2b90 0x7f38fc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:53.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.254+0000 7f3909ffb640 1 -- 172.21.15.44:0/2866041990 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f39040f1370 con 0x7f390c103350 2024-04-21T23:06:53.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.255+0000 7f3910b36640 1 --2- 172.21.15.44:0/2866041990 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f38fc0a2b90 0x7f38fc0a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3900005d00 tx=0x7f3900005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:53.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.259+0000 7f3909ffb640 1 -- 172.21.15.44:0/2866041990 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f390404b020 con 0x7f390c103350 2024-04-21T23:06:53.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.355+0000 7f3909ffb640 1 -- 172.21.15.44:0/2866041990 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f39040ba000 con 0x7f390c103350 2024-04-21T23:06:53.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:53 smithi044 ceph-mon[27934]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:06:53.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:53 smithi044 ceph-mon[27934]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:06:53.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:53 smithi044 ceph-mon[27934]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:06:53.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.485+0000 7f3912d9a640 1 -- 172.21.15.44:0/2866041990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f390c02d170 con 0x7f390c103350 2024-04-21T23:06:53.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.486+0000 7f3909ffb640 1 -- 172.21.15.44:0/2866041990 <== mon.0 v2:172.21.15.44:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11791 (secure 0 0 0) 0x7f39040be090 con 0x7f390c103350 2024-04-21T23:06:53.485 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:53.486 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:06:50.685663+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3743898853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3743898853},{"type":"v1","addr":"172.21.15.69:6839","nonce":3743898853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:06:53.486 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2024-04-21T23:06:53.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.488+0000 7f3912d9a640 1 -- 172.21.15.44:0/2866041990 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f38fc0a2b90 msgr2=0x7f38fc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:53.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.488+0000 7f3912d9a640 1 --2- 172.21.15.44:0/2866041990 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f38fc0a2b90 0x7f38fc0a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3900005d00 tx=0x7f3900005c90 comp rx=0 tx=0).stop 2024-04-21T23:06:53.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.488+0000 7f3912d9a640 1 -- 172.21.15.44:0/2866041990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f390c103350 msgr2=0x7f390c19efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:53.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.488+0000 7f3912d9a640 1 --2- 172.21.15.44:0/2866041990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f390c103350 0x7f390c19efe0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3904005e30 tx=0x7f3904032ec0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.488+0000 7f3912d9a640 1 -- 172.21.15.44:0/2866041990 shutdown_connections 2024-04-21T23:06:53.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.488+0000 7f3912d9a640 1 --2- 172.21.15.44:0/2866041990 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f38fc0a2b90 0x7f38fc0a5050 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.488+0000 7f3912d9a640 1 --2- 172.21.15.44:0/2866041990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f390c102990 0x7f390c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.488+0000 7f3912d9a640 1 --2- 172.21.15.44:0/2866041990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f390c103350 0x7f390c19efe0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.488+0000 7f3912d9a640 1 --2- 172.21.15.44:0/2866041990 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f390c1017a0 0x7f390c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.488+0000 7f3912d9a640 1 -- 172.21.15.44:0/2866041990 >> 172.21.15.44:0/2866041990 conn(0x7f390c0fcd30 msgr2=0x7f390c105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:53.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.488+0000 7f3912d9a640 1 -- 172.21.15.44:0/2866041990 shutdown_connections 2024-04-21T23:06:53.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.488+0000 7f3912d9a640 1 -- 172.21.15.44:0/2866041990 wait complete. 2024-04-21T23:06:53.499 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:06:53.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.601+0000 7f13297d7640 1 -- 172.21.15.44:0/2326418987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1324102990 msgr2=0x7f1324102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:53.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.601+0000 7f13297d7640 1 --2- 172.21.15.44:0/2326418987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1324102990 0x7f1324102e10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1318009900 tx=0x7f131802bc30 comp rx=0 tx=0).stop 2024-04-21T23:06:53.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.601+0000 7f13297d7640 1 -- 172.21.15.44:0/2326418987 shutdown_connections 2024-04-21T23:06:53.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.601+0000 7f13297d7640 1 --2- 172.21.15.44:0/2326418987 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1324103350 0x7f1324109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.601+0000 7f13297d7640 1 --2- 172.21.15.44:0/2326418987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1324102990 0x7f1324102e10 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.601+0000 7f13297d7640 1 --2- 172.21.15.44:0/2326418987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13241017a0 0x7f1324101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.601+0000 7f13297d7640 1 -- 172.21.15.44:0/2326418987 >> 172.21.15.44:0/2326418987 conn(0x7f13240fcd30 msgr2=0x7f13240ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:53.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.601+0000 7f13297d7640 1 -- 172.21.15.44:0/2326418987 shutdown_connections 2024-04-21T23:06:53.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.601+0000 7f13297d7640 1 -- 172.21.15.44:0/2326418987 wait complete. 2024-04-21T23:06:53.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.601+0000 7f13297d7640 1 Processor -- start 2024-04-21T23:06:53.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.602+0000 7f13297d7640 1 -- start start 2024-04-21T23:06:53.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.602+0000 7f13297d7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13241017a0 0x7f1324197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:53.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.602+0000 7f13297d7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1324102990 0x7f1324198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:53.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.602+0000 7f13297d7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1324103350 0x7f132419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:53.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.602+0000 7f13297d7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f132410d420 con 0x7f1324102990 2024-04-21T23:06:53.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.602+0000 7f13297d7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f132410d2a0 con 0x7f13241017a0 2024-04-21T23:06:53.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.602+0000 7f13297d7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f132410d5a0 con 0x7f1324103350 2024-04-21T23:06:53.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.602+0000 7f13227fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1324102990 0x7f1324198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:53.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.602+0000 7f13227fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1324102990 0x7f1324198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50780/0 (socket says 172.21.15.44:50780) 2024-04-21T23:06:53.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.602+0000 7f13227fc640 1 -- 172.21.15.44:0/1978088580 learned_addr learned my addr 172.21.15.44:0/1978088580 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:53.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.602+0000 7f1322ffd640 1 --2- 172.21.15.44:0/1978088580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13241017a0 0x7f1324197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:53.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.602+0000 7f13237fe640 1 --2- 172.21.15.44:0/1978088580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1324103350 0x7f132419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:53.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.602+0000 7f13227fc640 1 -- 172.21.15.44:0/1978088580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1324103350 msgr2=0x7f132419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:53.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.602+0000 7f13227fc640 1 --2- 172.21.15.44:0/1978088580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1324103350 0x7f132419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.602+0000 7f13227fc640 1 -- 172.21.15.44:0/1978088580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13241017a0 msgr2=0x7f1324197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:53.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.602+0000 7f13227fc640 1 --2- 172.21.15.44:0/1978088580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13241017a0 0x7f1324197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.602+0000 7f13227fc640 1 -- 172.21.15.44:0/1978088580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132419f6f0 con 0x7f1324102990 2024-04-21T23:06:53.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.602+0000 7f13227fc640 1 --2- 172.21.15.44:0/1978088580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1324102990 0x7f1324198010 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1318009230 tx=0x7f1318005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:53.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.602+0000 7f130bfff640 1 -- 172.21.15.44:0/1978088580 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1318041070 con 0x7f1324102990 2024-04-21T23:06:53.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.603+0000 7f130bfff640 1 -- 172.21.15.44:0/1978088580 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13180054e0 con 0x7f1324102990 2024-04-21T23:06:53.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.603+0000 7f13297d7640 1 -- 172.21.15.44:0/1978088580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f132419f980 con 0x7f1324102990 2024-04-21T23:06:53.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.603+0000 7f130bfff640 1 -- 172.21.15.44:0/1978088580 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1318034440 con 0x7f1324102990 2024-04-21T23:06:53.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.603+0000 7f13297d7640 1 -- 172.21.15.44:0/1978088580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f132419fe70 con 0x7f1324102990 2024-04-21T23:06:53.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.603+0000 7f13297d7640 1 -- 172.21.15.44:0/1978088580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1324108ac0 con 0x7f1324102990 2024-04-21T23:06:53.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.605+0000 7f130bfff640 1 -- 172.21.15.44:0/1978088580 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1318044070 con 0x7f1324102990 2024-04-21T23:06:53.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.605+0000 7f130bfff640 1 --2- 172.21.15.44:0/1978088580 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f130c0a2c20 0x7f130c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:53.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.605+0000 7f1322ffd640 1 --2- 172.21.15.44:0/1978088580 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f130c0a2c20 0x7f130c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:53.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.605+0000 7f130bfff640 1 -- 172.21.15.44:0/1978088580 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f13180f1380 con 0x7f1324102990 2024-04-21T23:06:53.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.606+0000 7f1322ffd640 1 --2- 172.21.15.44:0/1978088580 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f130c0a2c20 0x7f130c0a50e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1314005fd0 tx=0x7f131400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:53.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.609+0000 7f130bfff640 1 -- 172.21.15.44:0/1978088580 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f13180bda00 con 0x7f1324102990 2024-04-21T23:06:53.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:53 smithi055 ceph-mon[31117]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:06:53.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:53 smithi055 ceph-mon[31117]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:06:53.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:53 smithi055 ceph-mon[31117]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:06:53.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:53 smithi069 ceph-mon[31816]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:06:53.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:53 smithi069 ceph-mon[31816]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:06:53.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:53 smithi069 ceph-mon[31816]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:06:53.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.795+0000 7f13297d7640 1 -- 172.21.15.44:0/1978088580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f132402d3c0 con 0x7f1324102990 2024-04-21T23:06:53.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.796+0000 7f130bfff640 1 -- 172.21.15.44:0/1978088580 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24032 (secure 0 0 0) 0x7f13180c1d00 con 0x7f1324102990 2024-04-21T23:06:53.796 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:53.797 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":90,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:06:30.619155+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:06:53.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.798+0000 7f13297d7640 1 -- 172.21.15.44:0/1978088580 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f130c0a2c20 msgr2=0x7f130c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:53.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.798+0000 7f13297d7640 1 --2- 172.21.15.44:0/1978088580 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f130c0a2c20 0x7f130c0a50e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1314005fd0 tx=0x7f131400b390 comp rx=0 tx=0).stop 2024-04-21T23:06:53.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.798+0000 7f13297d7640 1 -- 172.21.15.44:0/1978088580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1324102990 msgr2=0x7f1324198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:53.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.798+0000 7f13297d7640 1 --2- 172.21.15.44:0/1978088580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1324102990 0x7f1324198010 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1318009230 tx=0x7f1318005dd0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.798+0000 7f13297d7640 1 -- 172.21.15.44:0/1978088580 shutdown_connections 2024-04-21T23:06:53.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.798+0000 7f13297d7640 1 --2- 172.21.15.44:0/1978088580 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f130c0a2c20 0x7f130c0a50e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.798+0000 7f13297d7640 1 --2- 172.21.15.44:0/1978088580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1324103350 0x7f132419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.798+0000 7f13297d7640 1 --2- 172.21.15.44:0/1978088580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1324102990 0x7f1324198010 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.798+0000 7f13297d7640 1 --2- 172.21.15.44:0/1978088580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f13241017a0 0x7f1324197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.798+0000 7f13297d7640 1 -- 172.21.15.44:0/1978088580 >> 172.21.15.44:0/1978088580 conn(0x7f13240fcd30 msgr2=0x7f1324105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:53.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.799+0000 7f13297d7640 1 -- 172.21.15.44:0/1978088580 shutdown_connections 2024-04-21T23:06:53.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.799+0000 7f13297d7640 1 -- 172.21.15.44:0/1978088580 wait complete. 2024-04-21T23:06:53.810 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-21T23:06:53.810 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T23:06:53.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.919+0000 7f2fdb302640 1 -- 172.21.15.44:0/4114529386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fd4102250 msgr2=0x7f2fd4108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:53.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.919+0000 7f2fdb302640 1 --2- 172.21.15.44:0/4114529386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fd4102250 0x7f2fd4108ae0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4009ca0 tx=0x7f2fc402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.919+0000 7f2fdb302640 1 -- 172.21.15.44:0/4114529386 shutdown_connections 2024-04-21T23:06:53.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.919+0000 7f2fdb302640 1 --2- 172.21.15.44:0/4114529386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fd4102250 0x7f2fd4108ae0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.919+0000 7f2fdb302640 1 --2- 172.21.15.44:0/4114529386 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2fd4101890 0x7f2fd4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.919+0000 7f2fdb302640 1 --2- 172.21.15.44:0/4114529386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fd41006a0 0x7f2fd4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.919+0000 7f2fdb302640 1 -- 172.21.15.44:0/4114529386 >> 172.21.15.44:0/4114529386 conn(0x7f2fd40fbc10 msgr2=0x7f2fd40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:53.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.919+0000 7f2fdb302640 1 -- 172.21.15.44:0/4114529386 shutdown_connections 2024-04-21T23:06:53.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.919+0000 7f2fdb302640 1 -- 172.21.15.44:0/4114529386 wait complete. 2024-04-21T23:06:53.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fdb302640 1 Processor -- start 2024-04-21T23:06:53.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fdb302640 1 -- start start 2024-04-21T23:06:53.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fdb302640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fd41006a0 0x7f2fd410b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:53.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fdb302640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fd4101890 0x7f2fd4109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:53.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fdb302640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2fd4102250 0x7f2fd4109ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fdb302640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd410d440 con 0x7f2fd41006a0 2024-04-21T23:06:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fdb302640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd410d2c0 con 0x7f2fd4102250 2024-04-21T23:06:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fdb302640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd410d5c0 con 0x7f2fd4101890 2024-04-21T23:06:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fd909e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fd41006a0 0x7f2fd410b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fd909e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fd41006a0 0x7f2fd410b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50794/0 (socket says 172.21.15.44:50794) 2024-04-21T23:06:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fd909e640 1 -- 172.21.15.44:0/643766382 learned_addr learned my addr 172.21.15.44:0/643766382 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fd989f640 1 --2- 172.21.15.44:0/643766382 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2fd4102250 0x7f2fd4109ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fd889d640 1 --2- 172.21.15.44:0/643766382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fd4101890 0x7f2fd4109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fd909e640 1 -- 172.21.15.44:0/643766382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fd4101890 msgr2=0x7f2fd4109660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fd909e640 1 --2- 172.21.15.44:0/643766382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fd4101890 0x7f2fd4109660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fd909e640 1 -- 172.21.15.44:0/643766382 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2fd4102250 msgr2=0x7f2fd4109ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fd909e640 1 --2- 172.21.15.44:0/643766382 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2fd4102250 0x7f2fd4109ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fd909e640 1 -- 172.21.15.44:0/643766382 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd410a430 con 0x7f2fd41006a0 2024-04-21T23:06:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fd889d640 1 --2- 172.21.15.44:0/643766382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fd4101890 0x7f2fd4109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:06:53.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fd989f640 1 --2- 172.21.15.44:0/643766382 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2fd4102250 0x7f2fd4109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:06:53.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.920+0000 7f2fd909e640 1 --2- 172.21.15.44:0/643766382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fd41006a0 0x7f2fd410b020 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc00e9c0 tx=0x7f2fcc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:53.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.921+0000 7f2fd27fc640 1 -- 172.21.15.44:0/643766382 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc00cd80 con 0x7f2fd41006a0 2024-04-21T23:06:53.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.921+0000 7f2fd27fc640 1 -- 172.21.15.44:0/643766382 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fcc0035f0 con 0x7f2fd41006a0 2024-04-21T23:06:53.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.921+0000 7f2fdb302640 1 -- 172.21.15.44:0/643766382 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fd41a1b20 con 0x7f2fd41006a0 2024-04-21T23:06:53.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.921+0000 7f2fd27fc640 1 -- 172.21.15.44:0/643766382 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc010690 con 0x7f2fd41006a0 2024-04-21T23:06:53.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.921+0000 7f2fdb302640 1 -- 172.21.15.44:0/643766382 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fd41a2020 con 0x7f2fd41006a0 2024-04-21T23:06:53.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.922+0000 7f2fdb302640 1 -- 172.21.15.44:0/643766382 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fd41079c0 con 0x7f2fd41006a0 2024-04-21T23:06:53.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.923+0000 7f2fd27fc640 1 -- 172.21.15.44:0/643766382 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f2fcc0037a0 con 0x7f2fd41006a0 2024-04-21T23:06:53.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.923+0000 7f2fd27fc640 1 --2- 172.21.15.44:0/643766382 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2fbc0a2c20 0x7f2fbc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:53.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.924+0000 7f2fd27fc640 1 -- 172.21.15.44:0/643766382 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f2fcc0cf9b0 con 0x7f2fd41006a0 2024-04-21T23:06:53.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.924+0000 7f2fd889d640 1 --2- 172.21.15.44:0/643766382 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2fbc0a2c20 0x7f2fbc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:53.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.924+0000 7f2fd889d640 1 --2- 172.21.15.44:0/643766382 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2fbc0a2c20 0x7f2fbc0a50e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2fd410ad90 tx=0x7f2fc8005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:53.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:53.928+0000 7f2fd27fc640 1 -- 172.21.15.44:0/643766382 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2fcc09c510 con 0x7f2fd41006a0 2024-04-21T23:06:54.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.110+0000 7f2fdb302640 1 -- 172.21.15.44:0/643766382 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2fd4061b80 con 0x7f2fbc0a2c20 2024-04-21T23:06:54.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.120+0000 7f2fd27fc640 1 -- 172.21.15.44:0/643766382 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340216 (secure 0 0 0) 0x7f2fd4061b80 con 0x7f2fbc0a2c20 2024-04-21T23:06:54.124 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:54.126 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-21T23:06:54.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.124+0000 7f2fdb302640 1 -- 172.21.15.44:0/643766382 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2fbc0a2c20 msgr2=0x7f2fbc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:54.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.124+0000 7f2fdb302640 1 --2- 172.21.15.44:0/643766382 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2fbc0a2c20 0x7f2fbc0a50e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2fd410ad90 tx=0x7f2fc8005d30 comp rx=0 tx=0).stop 2024-04-21T23:06:54.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.124+0000 7f2fdb302640 1 -- 172.21.15.44:0/643766382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fd41006a0 msgr2=0x7f2fd410b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:54.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.124+0000 7f2fdb302640 1 --2- 172.21.15.44:0/643766382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fd41006a0 0x7f2fd410b020 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc00e9c0 tx=0x7f2fcc00ee80 comp rx=0 tx=0).stop 2024-04-21T23:06:54.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.124+0000 7f2fdb302640 1 -- 172.21.15.44:0/643766382 shutdown_connections 2024-04-21T23:06:54.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.124+0000 7f2fdb302640 1 --2- 172.21.15.44:0/643766382 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2fbc0a2c20 0x7f2fbc0a50e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:54.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.124+0000 7f2fdb302640 1 --2- 172.21.15.44:0/643766382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fd4101890 0x7f2fd4109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:54.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.124+0000 7f2fdb302640 1 --2- 172.21.15.44:0/643766382 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2fd4102250 0x7f2fd4109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:54.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.124+0000 7f2fdb302640 1 --2- 172.21.15.44:0/643766382 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2fd41006a0 0x7f2fd410b020 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:54.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.124+0000 7f2fdb302640 1 -- 172.21.15.44:0/643766382 >> 172.21.15.44:0/643766382 conn(0x7f2fd40fbc10 msgr2=0x7f2fd4104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:54.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.124+0000 7f2fdb302640 1 -- 172.21.15.44:0/643766382 shutdown_connections 2024-04-21T23:06:54.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.124+0000 7f2fdb302640 1 -- 172.21.15.44:0/643766382 wait complete. 2024-04-21T23:06:54.134 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-04-21T23:06:53.057326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":142,"num_read_kb":125,"num_write":232,"num_write_kb":2958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181,"ondisk_log_size":181,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":362732,"kb_used_data":13528,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124579092,"statfs":{"total":1151940427776,"available":1151568990208,"internally_reserved":0,"allocated":13852672,"data_stored":7619490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003565"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416229+0000","last_change":"2024-04-21T23:06:25.980091+0000","last_active":"2024-04-21T23:06:31.416229+0000","last_peered":"2024-04-21T23:06:31.416229+0000","last_clean":"2024-04-21T23:06:31.416229+0000","last_became_active":"2024-04-21T23:06:25.979763+0000","last_became_peered":"2024-04-21T23:06:25.979763+0000","last_unstale":"2024-04-21T23:06:31.416229+0000","last_undegraded":"2024-04-21T23:06:31.416229+0000","last_fullsized":"2024-04-21T23:06:31.416229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:12:23.771931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414545+0000","last_change":"2024-04-21T23:06:25.992461+0000","last_active":"2024-04-21T23:06:31.414545+0000","last_peered":"2024-04-21T23:06:31.414545+0000","last_clean":"2024-04-21T23:06:31.414545+0000","last_became_active":"2024-04-21T23:06:25.992018+0000","last_became_peered":"2024-04-21T23:06:25.992018+0000","last_unstale":"2024-04-21T23:06:31.414545+0000","last_undegraded":"2024-04-21T23:06:31.414545+0000","last_fullsized":"2024-04-21T23:06:31.414545+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:46.651683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414373+0000","last_change":"2024-04-21T23:06:25.974812+0000","last_active":"2024-04-21T23:06:31.414373+0000","last_peered":"2024-04-21T23:06:31.414373+0000","last_clean":"2024-04-21T23:06:31.414373+0000","last_became_active":"2024-04-21T23:06:25.974420+0000","last_became_peered":"2024-04-21T23:06:25.974420+0000","last_unstale":"2024-04-21T23:06:31.414373+0000","last_undegraded":"2024-04-21T23:06:31.414373+0000","last_fullsized":"2024-04-21T23:06:31.414373+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:00.734218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416020+0000","last_change":"2024-04-21T23:06:25.988806+0000","last_active":"2024-04-21T23:06:31.416020+0000","last_peered":"2024-04-21T23:06:31.416020+0000","last_clean":"2024-04-21T23:06:31.416020+0000","last_became_active":"2024-04-21T23:06:25.988344+0000","last_became_peered":"2024-04-21T23:06:25.988344+0000","last_unstale":"2024-04-21T23:06:31.416020+0000","last_undegraded":"2024-04-21T23:06:31.416020+0000","last_fullsized":"2024-04-21T23:06:31.416020+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:15:40.079273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414684+0000","last_change":"2024-04-21T23:06:25.979125+0000","last_active":"2024-04-21T23:06:31.414684+0000","last_peered":"2024-04-21T23:06:31.414684+0000","last_clean":"2024-04-21T23:06:31.414684+0000","last_became_active":"2024-04-21T23:06:25.978778+0000","last_became_peered":"2024-04-21T23:06:25.978778+0000","last_unstale":"2024-04-21T23:06:31.414684+0000","last_undegraded":"2024-04-21T23:06:31.414684+0000","last_fullsized":"2024-04-21T23:06:31.414684+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:44.655404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416464+0000","last_change":"2024-04-21T23:06:25.983983+0000","last_active":"2024-04-21T23:06:31.416464+0000","last_peered":"2024-04-21T23:06:31.416464+0000","last_clean":"2024-04-21T23:06:31.416464+0000","last_became_active":"2024-04-21T23:06:25.983463+0000","last_became_peered":"2024-04-21T23:06:25.983463+0000","last_unstale":"2024-04-21T23:06:31.416464+0000","last_undegraded":"2024-04-21T23:06:31.416464+0000","last_fullsized":"2024-04-21T23:06:31.416464+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:32:12.784752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414557+0000","last_change":"2024-04-21T23:06:25.987479+0000","last_active":"2024-04-21T23:06:31.414557+0000","last_peered":"2024-04-21T23:06:31.414557+0000","last_clean":"2024-04-21T23:06:31.414557+0000","last_became_active":"2024-04-21T23:06:25.987133+0000","last_became_peered":"2024-04-21T23:06:25.987133+0000","last_unstale":"2024-04-21T23:06:31.414557+0000","last_undegraded":"2024-04-21T23:06:31.414557+0000","last_fullsized":"2024-04-21T23:06:31.414557+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:08.033184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414851+0000","last_change":"2024-04-21T23:06:25.987714+0000","last_active":"2024-04-21T23:06:31.414851+0000","last_peered":"2024-04-21T23:06:31.414851+0000","last_clean":"2024-04-21T23:06:31.414851+0000","last_became_active":"2024-04-21T23:06:25.987457+0000","last_became_peered":"2024-04-21T23:06:25.987457+0000","last_unstale":"2024-04-21T23:06:31.414851+0000","last_undegraded":"2024-04-21T23:06:31.414851+0000","last_fullsized":"2024-04-21T23:06:31.414851+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:06:06.184525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.264178+0000","last_change":"2024-04-21T23:06:25.975957+0000","last_active":"2024-04-21T23:06:31.264178+0000","last_peered":"2024-04-21T23:06:31.264178+0000","last_clean":"2024-04-21T23:06:31.264178+0000","last_became_active":"2024-04-21T23:06:25.974710+0000","last_became_peered":"2024-04-21T23:06:25.974710+0000","last_unstale":"2024-04-21T23:06:31.264178+0000","last_undegraded":"2024-04-21T23:06:31.264178+0000","last_fullsized":"2024-04-21T23:06:31.264178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:31:40.062273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416666+0000","last_change":"2024-04-21T23:06:25.995085+0000","last_active":"2024-04-21T23:06:31.416666+0000","last_peered":"2024-04-21T23:06:31.416666+0000","last_clean":"2024-04-21T23:06:31.416666+0000","last_became_active":"2024-04-21T23:06:25.993552+0000","last_became_peered":"2024-04-21T23:06:25.993552+0000","last_unstale":"2024-04-21T23:06:31.416666+0000","last_undegraded":"2024-04-21T23:06:31.416666+0000","last_fullsized":"2024-04-21T23:06:31.416666+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:11.405113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415519+0000","last_change":"2024-04-21T23:06:25.993191+0000","last_active":"2024-04-21T23:06:31.415519+0000","last_peered":"2024-04-21T23:06:31.415519+0000","last_clean":"2024-04-21T23:06:31.415519+0000","last_became_active":"2024-04-21T23:06:25.989399+0000","last_became_peered":"2024-04-21T23:06:25.989399+0000","last_unstale":"2024-04-21T23:06:31.415519+0000","last_undegraded":"2024-04-21T23:06:31.415519+0000","last_fullsized":"2024-04-21T23:06:31.415519+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:30:20.035599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415962+0000","last_change":"2024-04-21T23:06:25.994158+0000","last_active":"2024-04-21T23:06:31.415962+0000","last_peered":"2024-04-21T23:06:31.415962+0000","last_clean":"2024-04-21T23:06:31.415962+0000","last_became_active":"2024-04-21T23:06:25.993780+0000","last_became_peered":"2024-04-21T23:06:25.993780+0000","last_unstale":"2024-04-21T23:06:31.415962+0000","last_undegraded":"2024-04-21T23:06:31.415962+0000","last_fullsized":"2024-04-21T23:06:31.415962+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:52:17.181898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.258935+0000","last_change":"2024-04-21T23:06:25.989274+0000","last_active":"2024-04-21T23:06:31.258935+0000","last_peered":"2024-04-21T23:06:31.258935+0000","last_clean":"2024-04-21T23:06:31.258935+0000","last_became_active":"2024-04-21T23:06:25.988931+0000","last_became_peered":"2024-04-21T23:06:25.988931+0000","last_unstale":"2024-04-21T23:06:31.258935+0000","last_undegraded":"2024-04-21T23:06:31.258935+0000","last_fullsized":"2024-04-21T23:06:31.258935+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:11:04.576136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.413914+0000","last_change":"2024-04-21T23:06:25.982976+0000","last_active":"2024-04-21T23:06:31.413914+0000","last_peered":"2024-04-21T23:06:31.413914+0000","last_clean":"2024-04-21T23:06:31.413914+0000","last_became_active":"2024-04-21T23:06:25.982625+0000","last_became_peered":"2024-04-21T23:06:25.982625+0000","last_unstale":"2024-04-21T23:06:31.413914+0000","last_undegraded":"2024-04-21T23:06:31.413914+0000","last_fullsized":"2024-04-21T23:06:31.413914+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:51:02.193044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.413821+0000","last_change":"2024-04-21T23:06:25.994805+0000","last_active":"2024-04-21T23:06:31.413821+0000","last_peered":"2024-04-21T23:06:31.413821+0000","last_clean":"2024-04-21T23:06:31.413821+0000","last_became_active":"2024-04-21T23:06:25.994482+0000","last_became_peered":"2024-04-21T23:06:25.994482+0000","last_unstale":"2024-04-21T23:06:31.413821+0000","last_undegraded":"2024-04-21T23:06:31.413821+0000","last_fullsized":"2024-04-21T23:06:31.413821+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:24:21.792358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.263419+0000","last_change":"2024-04-21T23:06:25.990840+0000","last_active":"2024-04-21T23:06:31.263419+0000","last_peered":"2024-04-21T23:06:31.263419+0000","last_clean":"2024-04-21T23:06:31.263419+0000","last_became_active":"2024-04-21T23:06:25.990517+0000","last_became_peered":"2024-04-21T23:06:25.990517+0000","last_unstale":"2024-04-21T23:06:31.263419+0000","last_undegraded":"2024-04-21T23:06:31.263419+0000","last_fullsized":"2024-04-21T23:06:31.263419+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:48:09.265280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416081+0000","last_change":"2024-04-21T23:06:25.976810+0000","last_active":"2024-04-21T23:06:31.416081+0000","last_peered":"2024-04-21T23:06:31.416081+0000","last_clean":"2024-04-21T23:06:31.416081+0000","last_became_active":"2024-04-21T23:06:25.976489+0000","last_became_peered":"2024-04-21T23:06:25.976489+0000","last_unstale":"2024-04-21T23:06:31.416081+0000","last_undegraded":"2024-04-21T23:06:31.416081+0000","last_fullsized":"2024-04-21T23:06:31.416081+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:07:03.794115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415712+0000","last_change":"2024-04-21T23:06:25.988603+0000","last_active":"2024-04-21T23:06:31.415712+0000","last_peered":"2024-04-21T23:06:31.415712+0000","last_clean":"2024-04-21T23:06:31.415712+0000","last_became_active":"2024-04-21T23:06:25.988254+0000","last_became_peered":"2024-04-21T23:06:25.988254+0000","last_unstale":"2024-04-21T23:06:31.415712+0000","last_undegraded":"2024-04-21T23:06:31.415712+0000","last_fullsized":"2024-04-21T23:06:31.415712+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:25:46.099853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415311+0000","last_change":"2024-04-21T23:06:25.980831+0000","last_active":"2024-04-21T23:06:31.415311+0000","last_peered":"2024-04-21T23:06:31.415311+0000","last_clean":"2024-04-21T23:06:31.415311+0000","last_became_active":"2024-04-21T23:06:25.977051+0000","last_became_peered":"2024-04-21T23:06:25.977051+0000","last_unstale":"2024-04-21T23:06:31.415311+0000","last_undegraded":"2024-04-21T23:06:31.415311+0000","last_fullsized":"2024-04-21T23:06:31.415311+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:55:03.684734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416058+0000","last_change":"2024-04-21T23:06:30.256875+0000","last_active":"2024-04-21T23:06:31.416058+0000","last_peered":"2024-04-21T23:06:31.416058+0000","last_clean":"2024-04-21T23:06:31.416058+0000","last_became_active":"2024-04-21T23:06:30.256459+0000","last_became_peered":"2024-04-21T23:06:30.256459+0000","last_unstale":"2024-04-21T23:06:31.416058+0000","last_undegraded":"2024-04-21T23:06:31.416058+0000","last_fullsized":"2024-04-21T23:06:31.416058+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:45:32.376270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415309+0000","last_change":"2024-04-21T23:06:25.209605+0000","last_active":"2024-04-21T23:06:31.415309+0000","last_peered":"2024-04-21T23:06:31.415309+0000","last_clean":"2024-04-21T23:06:31.415309+0000","last_became_active":"2024-04-21T23:06:25.209277+0000","last_became_peered":"2024-04-21T23:06:25.209277+0000","last_unstale":"2024-04-21T23:06:31.415309+0000","last_undegraded":"2024-04-21T23:06:31.415309+0000","last_fullsized":"2024-04-21T23:06:31.415309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:15:49.742940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415528+0000","last_change":"2024-04-21T23:06:25.981774+0000","last_active":"2024-04-21T23:06:31.415528+0000","last_peered":"2024-04-21T23:06:31.415528+0000","last_clean":"2024-04-21T23:06:31.415528+0000","last_became_active":"2024-04-21T23:06:25.981432+0000","last_became_peered":"2024-04-21T23:06:25.981432+0000","last_unstale":"2024-04-21T23:06:31.415528+0000","last_undegraded":"2024-04-21T23:06:31.415528+0000","last_fullsized":"2024-04-21T23:06:31.415528+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:28:28.712149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414558+0000","last_change":"2024-04-21T23:06:30.271508+0000","last_active":"2024-04-21T23:06:31.414558+0000","last_peered":"2024-04-21T23:06:31.414558+0000","last_clean":"2024-04-21T23:06:31.414558+0000","last_became_active":"2024-04-21T23:06:30.270461+0000","last_became_peered":"2024-04-21T23:06:30.270461+0000","last_unstale":"2024-04-21T23:06:31.414558+0000","last_undegraded":"2024-04-21T23:06:31.414558+0000","last_fullsized":"2024-04-21T23:06:31.414558+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:43:07.473365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416426+0000","last_change":"2024-04-21T23:06:25.218773+0000","last_active":"2024-04-21T23:06:31.416426+0000","last_peered":"2024-04-21T23:06:31.416426+0000","last_clean":"2024-04-21T23:06:31.416426+0000","last_became_active":"2024-04-21T23:06:25.218458+0000","last_became_peered":"2024-04-21T23:06:25.218458+0000","last_unstale":"2024-04-21T23:06:31.416426+0000","last_undegraded":"2024-04-21T23:06:31.416426+0000","last_fullsized":"2024-04-21T23:06:31.416426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:25:16.372689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415140+0000","last_change":"2024-04-21T23:06:25.977669+0000","last_active":"2024-04-21T23:06:31.415140+0000","last_peered":"2024-04-21T23:06:31.415140+0000","last_clean":"2024-04-21T23:06:31.415140+0000","last_became_active":"2024-04-21T23:06:25.977143+0000","last_became_peered":"2024-04-21T23:06:25.977143+0000","last_unstale":"2024-04-21T23:06:31.415140+0000","last_undegraded":"2024-04-21T23:06:31.415140+0000","last_fullsized":"2024-04-21T23:06:31.415140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:39.289208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414991+0000","last_change":"2024-04-21T23:06:30.271934+0000","last_active":"2024-04-21T23:06:31.414991+0000","last_peered":"2024-04-21T23:06:31.414991+0000","last_clean":"2024-04-21T23:06:31.414991+0000","last_became_active":"2024-04-21T23:06:30.270567+0000","last_became_peered":"2024-04-21T23:06:30.270567+0000","last_unstale":"2024-04-21T23:06:31.414991+0000","last_undegraded":"2024-04-21T23:06:31.414991+0000","last_fullsized":"2024-04-21T23:06:31.414991+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:43:34.908748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415901+0000","last_change":"2024-04-21T23:06:25.214590+0000","last_active":"2024-04-21T23:06:31.415901+0000","last_peered":"2024-04-21T23:06:31.415901+0000","last_clean":"2024-04-21T23:06:31.415901+0000","last_became_active":"2024-04-21T23:06:25.214087+0000","last_became_peered":"2024-04-21T23:06:25.214087+0000","last_unstale":"2024-04-21T23:06:31.415901+0000","last_undegraded":"2024-04-21T23:06:31.415901+0000","last_fullsized":"2024-04-21T23:06:31.415901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:49:51.294478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.263882+0000","last_change":"2024-04-21T23:06:25.988197+0000","last_active":"2024-04-21T23:06:31.263882+0000","last_peered":"2024-04-21T23:06:31.263882+0000","last_clean":"2024-04-21T23:06:31.263882+0000","last_became_active":"2024-04-21T23:06:25.987872+0000","last_became_peered":"2024-04-21T23:06:25.987872+0000","last_unstale":"2024-04-21T23:06:31.263882+0000","last_undegraded":"2024-04-21T23:06:31.263882+0000","last_fullsized":"2024-04-21T23:06:31.263882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:25.389020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414666+0000","last_change":"2024-04-21T23:06:30.265262+0000","last_active":"2024-04-21T23:06:31.414666+0000","last_peered":"2024-04-21T23:06:31.414666+0000","last_clean":"2024-04-21T23:06:31.414666+0000","last_became_active":"2024-04-21T23:06:30.264781+0000","last_became_peered":"2024-04-21T23:06:30.264781+0000","last_unstale":"2024-04-21T23:06:31.414666+0000","last_undegraded":"2024-04-21T23:06:31.414666+0000","last_fullsized":"2024-04-21T23:06:31.414666+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:03:30.756792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414876+0000","last_change":"2024-04-21T23:06:25.205070+0000","last_active":"2024-04-21T23:06:31.414876+0000","last_peered":"2024-04-21T23:06:31.414876+0000","last_clean":"2024-04-21T23:06:31.414876+0000","last_became_active":"2024-04-21T23:06:25.204737+0000","last_became_peered":"2024-04-21T23:06:25.204737+0000","last_unstale":"2024-04-21T23:06:31.414876+0000","last_undegraded":"2024-04-21T23:06:31.414876+0000","last_fullsized":"2024-04-21T23:06:31.414876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:37.014534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.260026+0000","last_change":"2024-04-21T23:06:25.986584+0000","last_active":"2024-04-21T23:06:31.260026+0000","last_peered":"2024-04-21T23:06:31.260026+0000","last_clean":"2024-04-21T23:06:31.260026+0000","last_became_active":"2024-04-21T23:06:25.986268+0000","last_became_peered":"2024-04-21T23:06:25.986268+0000","last_unstale":"2024-04-21T23:06:31.260026+0000","last_undegraded":"2024-04-21T23:06:31.260026+0000","last_fullsized":"2024-04-21T23:06:31.260026+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:19:43.864279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415555+0000","last_change":"2024-04-21T23:06:30.261900+0000","last_active":"2024-04-21T23:06:31.415555+0000","last_peered":"2024-04-21T23:06:31.415555+0000","last_clean":"2024-04-21T23:06:31.415555+0000","last_became_active":"2024-04-21T23:06:30.261444+0000","last_became_peered":"2024-04-21T23:06:30.261444+0000","last_unstale":"2024-04-21T23:06:31.415555+0000","last_undegraded":"2024-04-21T23:06:31.415555+0000","last_fullsized":"2024-04-21T23:06:31.415555+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:10:40.810555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259706+0000","last_change":"2024-04-21T23:06:25.210702+0000","last_active":"2024-04-21T23:06:31.259706+0000","last_peered":"2024-04-21T23:06:31.259706+0000","last_clean":"2024-04-21T23:06:31.259706+0000","last_became_active":"2024-04-21T23:06:25.210383+0000","last_became_peered":"2024-04-21T23:06:25.210383+0000","last_unstale":"2024-04-21T23:06:31.259706+0000","last_undegraded":"2024-04-21T23:06:31.259706+0000","last_fullsized":"2024-04-21T23:06:31.259706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:07:45.733370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415353+0000","last_change":"2024-04-21T23:06:25.975999+0000","last_active":"2024-04-21T23:06:31.415353+0000","last_peered":"2024-04-21T23:06:31.415353+0000","last_clean":"2024-04-21T23:06:31.415353+0000","last_became_active":"2024-04-21T23:06:25.975677+0000","last_became_peered":"2024-04-21T23:06:25.975677+0000","last_unstale":"2024-04-21T23:06:31.415353+0000","last_undegraded":"2024-04-21T23:06:31.415353+0000","last_fullsized":"2024-04-21T23:06:31.415353+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:52:03.177248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415728+0000","last_change":"2024-04-21T23:06:30.261234+0000","last_active":"2024-04-21T23:06:31.415728+0000","last_peered":"2024-04-21T23:06:31.415728+0000","last_clean":"2024-04-21T23:06:31.415728+0000","last_became_active":"2024-04-21T23:06:30.260770+0000","last_became_peered":"2024-04-21T23:06:30.260770+0000","last_unstale":"2024-04-21T23:06:31.415728+0000","last_undegraded":"2024-04-21T23:06:31.415728+0000","last_fullsized":"2024-04-21T23:06:31.415728+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:51:22.460253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416378+0000","last_change":"2024-04-21T23:06:25.214893+0000","last_active":"2024-04-21T23:06:31.416378+0000","last_peered":"2024-04-21T23:06:31.416378+0000","last_clean":"2024-04-21T23:06:31.416378+0000","last_became_active":"2024-04-21T23:06:25.212359+0000","last_became_peered":"2024-04-21T23:06:25.212359+0000","last_unstale":"2024-04-21T23:06:31.416378+0000","last_undegraded":"2024-04-21T23:06:31.416378+0000","last_fullsized":"2024-04-21T23:06:31.416378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:18:03.278875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259801+0000","last_change":"2024-04-21T23:06:25.986849+0000","last_active":"2024-04-21T23:06:31.259801+0000","last_peered":"2024-04-21T23:06:31.259801+0000","last_clean":"2024-04-21T23:06:31.259801+0000","last_became_active":"2024-04-21T23:06:25.986334+0000","last_became_peered":"2024-04-21T23:06:25.986334+0000","last_unstale":"2024-04-21T23:06:31.259801+0000","last_undegraded":"2024-04-21T23:06:31.259801+0000","last_fullsized":"2024-04-21T23:06:31.259801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:57:42.195629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416004+0000","last_change":"2024-04-21T23:06:30.270812+0000","last_active":"2024-04-21T23:06:31.416004+0000","last_peered":"2024-04-21T23:06:31.416004+0000","last_clean":"2024-04-21T23:06:31.416004+0000","last_became_active":"2024-04-21T23:06:30.270370+0000","last_became_peered":"2024-04-21T23:06:30.270370+0000","last_unstale":"2024-04-21T23:06:31.416004+0000","last_undegraded":"2024-04-21T23:06:31.416004+0000","last_fullsized":"2024-04-21T23:06:31.416004+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:53.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.258606+0000","last_change":"2024-04-21T23:06:25.201603+0000","last_active":"2024-04-21T23:06:31.258606+0000","last_peered":"2024-04-21T23:06:31.258606+0000","last_clean":"2024-04-21T23:06:31.258606+0000","last_became_active":"2024-04-21T23:06:25.201265+0000","last_became_peered":"2024-04-21T23:06:25.201265+0000","last_unstale":"2024-04-21T23:06:31.258606+0000","last_undegraded":"2024-04-21T23:06:31.258606+0000","last_fullsized":"2024-04-21T23:06:31.258606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:23:19.134215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415130+0000","last_change":"2024-04-21T23:06:25.982420+0000","last_active":"2024-04-21T23:06:31.415130+0000","last_peered":"2024-04-21T23:06:31.415130+0000","last_clean":"2024-04-21T23:06:31.415130+0000","last_became_active":"2024-04-21T23:06:25.981867+0000","last_became_peered":"2024-04-21T23:06:25.981867+0000","last_unstale":"2024-04-21T23:06:31.415130+0000","last_undegraded":"2024-04-21T23:06:31.415130+0000","last_fullsized":"2024-04-21T23:06:31.415130+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:10:29.433243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414491+0000","last_change":"2024-04-21T23:06:30.273907+0000","last_active":"2024-04-21T23:06:31.414491+0000","last_peered":"2024-04-21T23:06:31.414491+0000","last_clean":"2024-04-21T23:06:31.414491+0000","last_became_active":"2024-04-21T23:06:30.273472+0000","last_became_peered":"2024-04-21T23:06:30.273472+0000","last_unstale":"2024-04-21T23:06:31.414491+0000","last_undegraded":"2024-04-21T23:06:31.414491+0000","last_fullsized":"2024-04-21T23:06:31.414491+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:58:50.914940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259369+0000","last_change":"2024-04-21T23:06:25.211117+0000","last_active":"2024-04-21T23:06:31.259369+0000","last_peered":"2024-04-21T23:06:31.259369+0000","last_clean":"2024-04-21T23:06:31.259369+0000","last_became_active":"2024-04-21T23:06:25.210626+0000","last_became_peered":"2024-04-21T23:06:25.210626+0000","last_unstale":"2024-04-21T23:06:31.259369+0000","last_undegraded":"2024-04-21T23:06:31.259369+0000","last_fullsized":"2024-04-21T23:06:31.259369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:56:39.917072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414670+0000","last_change":"2024-04-21T23:06:25.980537+0000","last_active":"2024-04-21T23:06:31.414670+0000","last_peered":"2024-04-21T23:06:31.414670+0000","last_clean":"2024-04-21T23:06:31.414670+0000","last_became_active":"2024-04-21T23:06:25.980188+0000","last_became_peered":"2024-04-21T23:06:25.980188+0000","last_unstale":"2024-04-21T23:06:31.414670+0000","last_undegraded":"2024-04-21T23:06:31.414670+0000","last_fullsized":"2024-04-21T23:06:31.414670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:58:02.828632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415055+0000","last_change":"2024-04-21T23:06:30.263565+0000","last_active":"2024-04-21T23:06:31.415055+0000","last_peered":"2024-04-21T23:06:31.415055+0000","last_clean":"2024-04-21T23:06:31.415055+0000","last_became_active":"2024-04-21T23:06:30.263092+0000","last_became_peered":"2024-04-21T23:06:30.263092+0000","last_unstale":"2024-04-21T23:06:31.415055+0000","last_undegraded":"2024-04-21T23:06:31.415055+0000","last_fullsized":"2024-04-21T23:06:31.415055+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:58:25.709551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416668+0000","last_change":"2024-04-21T23:06:25.212567+0000","last_active":"2024-04-21T23:06:31.416668+0000","last_peered":"2024-04-21T23:06:31.416668+0000","last_clean":"2024-04-21T23:06:31.416668+0000","last_became_active":"2024-04-21T23:06:25.210593+0000","last_became_peered":"2024-04-21T23:06:25.210593+0000","last_unstale":"2024-04-21T23:06:31.416668+0000","last_undegraded":"2024-04-21T23:06:31.416668+0000","last_fullsized":"2024-04-21T23:06:31.416668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:53:33.552742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.264226+0000","last_change":"2024-04-21T23:06:25.989525+0000","last_active":"2024-04-21T23:06:31.264226+0000","last_peered":"2024-04-21T23:06:31.264226+0000","last_clean":"2024-04-21T23:06:31.264226+0000","last_became_active":"2024-04-21T23:06:25.989130+0000","last_became_peered":"2024-04-21T23:06:25.989130+0000","last_unstale":"2024-04-21T23:06:31.264226+0000","last_undegraded":"2024-04-21T23:06:31.264226+0000","last_fullsized":"2024-04-21T23:06:31.264226+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:19:31.000530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415581+0000","last_change":"2024-04-21T23:06:30.269309+0000","last_active":"2024-04-21T23:06:31.415581+0000","last_peered":"2024-04-21T23:06:31.415581+0000","last_clean":"2024-04-21T23:06:31.415581+0000","last_became_active":"2024-04-21T23:06:30.268903+0000","last_became_peered":"2024-04-21T23:06:30.268903+0000","last_unstale":"2024-04-21T23:06:31.415581+0000","last_undegraded":"2024-04-21T23:06:31.415581+0000","last_fullsized":"2024-04-21T23:06:31.415581+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:52:16.043870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414857+0000","last_change":"2024-04-21T23:06:25.213938+0000","last_active":"2024-04-21T23:06:31.414857+0000","last_peered":"2024-04-21T23:06:31.414857+0000","last_clean":"2024-04-21T23:06:31.414857+0000","last_became_active":"2024-04-21T23:06:25.213242+0000","last_became_peered":"2024-04-21T23:06:25.213242+0000","last_unstale":"2024-04-21T23:06:31.414857+0000","last_undegraded":"2024-04-21T23:06:31.414857+0000","last_fullsized":"2024-04-21T23:06:31.414857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:27.679122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415390+0000","last_change":"2024-04-21T23:06:25.994061+0000","last_active":"2024-04-21T23:06:31.415390+0000","last_peered":"2024-04-21T23:06:31.415390+0000","last_clean":"2024-04-21T23:06:31.415390+0000","last_became_active":"2024-04-21T23:06:25.993714+0000","last_became_peered":"2024-04-21T23:06:25.993714+0000","last_unstale":"2024-04-21T23:06:31.415390+0000","last_undegraded":"2024-04-21T23:06:31.415390+0000","last_fullsized":"2024-04-21T23:06:31.415390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:54:45.571815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415389+0000","last_change":"2024-04-21T23:06:30.272206+0000","last_active":"2024-04-21T23:06:31.415389+0000","last_peered":"2024-04-21T23:06:31.415389+0000","last_clean":"2024-04-21T23:06:31.415389+0000","last_became_active":"2024-04-21T23:06:30.271389+0000","last_became_peered":"2024-04-21T23:06:30.271389+0000","last_unstale":"2024-04-21T23:06:31.415389+0000","last_undegraded":"2024-04-21T23:06:31.415389+0000","last_fullsized":"2024-04-21T23:06:31.415389+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:54:47.996166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259925+0000","last_change":"2024-04-21T23:06:25.217655+0000","last_active":"2024-04-21T23:06:31.259925+0000","last_peered":"2024-04-21T23:06:31.259925+0000","last_clean":"2024-04-21T23:06:31.259925+0000","last_became_active":"2024-04-21T23:06:25.217337+0000","last_became_peered":"2024-04-21T23:06:25.217337+0000","last_unstale":"2024-04-21T23:06:31.259925+0000","last_undegraded":"2024-04-21T23:06:31.259925+0000","last_fullsized":"2024-04-21T23:06:31.259925+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:29:28.690841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.264198+0000","last_change":"2024-04-21T23:06:25.995757+0000","last_active":"2024-04-21T23:06:31.264198+0000","last_peered":"2024-04-21T23:06:31.264198+0000","last_clean":"2024-04-21T23:06:31.264198+0000","last_became_active":"2024-04-21T23:06:25.995443+0000","last_became_peered":"2024-04-21T23:06:25.995443+0000","last_unstale":"2024-04-21T23:06:31.264198+0000","last_undegraded":"2024-04-21T23:06:31.264198+0000","last_fullsized":"2024-04-21T23:06:31.264198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:40.359633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416030+0000","last_change":"2024-04-21T23:06:30.266441+0000","last_active":"2024-04-21T23:06:31.416030+0000","last_peered":"2024-04-21T23:06:31.416030+0000","last_clean":"2024-04-21T23:06:31.416030+0000","last_became_active":"2024-04-21T23:06:30.264065+0000","last_became_peered":"2024-04-21T23:06:30.264065+0000","last_unstale":"2024-04-21T23:06:31.416030+0000","last_undegraded":"2024-04-21T23:06:31.416030+0000","last_fullsized":"2024-04-21T23:06:31.416030+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:53:22.548096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'10","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:41.846239+0000","last_change":"2024-04-21T23:06:25.213919+0000","last_active":"2024-04-21T23:06:41.846239+0000","last_peered":"2024-04-21T23:06:41.846239+0000","last_clean":"2024-04-21T23:06:41.846239+0000","last_became_active":"2024-04-21T23:06:25.212244+0000","last_became_peered":"2024-04-21T23:06:25.212244+0000","last_unstale":"2024-04-21T23:06:41.846239+0000","last_undegraded":"2024-04-21T23:06:41.846239+0000","last_fullsized":"2024-04-21T23:06:41.846239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:17:25.287718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415644+0000","last_change":"2024-04-21T23:06:25.993666+0000","last_active":"2024-04-21T23:06:31.415644+0000","last_peered":"2024-04-21T23:06:31.415644+0000","last_clean":"2024-04-21T23:06:31.415644+0000","last_became_active":"2024-04-21T23:06:25.993330+0000","last_became_peered":"2024-04-21T23:06:25.993330+0000","last_unstale":"2024-04-21T23:06:31.415644+0000","last_undegraded":"2024-04-21T23:06:31.415644+0000","last_fullsized":"2024-04-21T23:06:31.415644+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:11:00.109782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.260359+0000","last_change":"2024-04-21T23:06:30.265807+0000","last_active":"2024-04-21T23:06:31.260359+0000","last_peered":"2024-04-21T23:06:31.260359+0000","last_clean":"2024-04-21T23:06:31.260359+0000","last_became_active":"2024-04-21T23:06:30.265116+0000","last_became_peered":"2024-04-21T23:06:30.265116+0000","last_unstale":"2024-04-21T23:06:31.260359+0000","last_undegraded":"2024-04-21T23:06:31.260359+0000","last_fullsized":"2024-04-21T23:06:31.260359+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:14:04.178281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415461+0000","last_change":"2024-04-21T23:06:25.208319+0000","last_active":"2024-04-21T23:06:31.415461+0000","last_peered":"2024-04-21T23:06:31.415461+0000","last_clean":"2024-04-21T23:06:31.415461+0000","last_became_active":"2024-04-21T23:06:25.208025+0000","last_became_peered":"2024-04-21T23:06:25.208025+0000","last_unstale":"2024-04-21T23:06:31.415461+0000","last_undegraded":"2024-04-21T23:06:31.415461+0000","last_fullsized":"2024-04-21T23:06:31.415461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:14.723170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.258707+0000","last_change":"2024-04-21T23:06:25.993604+0000","last_active":"2024-04-21T23:06:31.258707+0000","last_peered":"2024-04-21T23:06:31.258707+0000","last_clean":"2024-04-21T23:06:31.258707+0000","last_became_active":"2024-04-21T23:06:25.993217+0000","last_became_peered":"2024-04-21T23:06:25.993217+0000","last_unstale":"2024-04-21T23:06:31.258707+0000","last_undegraded":"2024-04-21T23:06:31.258707+0000","last_fullsized":"2024-04-21T23:06:31.258707+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:23:00.435688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414741+0000","last_change":"2024-04-21T23:06:30.270991+0000","last_active":"2024-04-21T23:06:31.414741+0000","last_peered":"2024-04-21T23:06:31.414741+0000","last_clean":"2024-04-21T23:06:31.414741+0000","last_became_active":"2024-04-21T23:06:30.270511+0000","last_became_peered":"2024-04-21T23:06:30.270511+0000","last_unstale":"2024-04-21T23:06:31.414741+0000","last_undegraded":"2024-04-21T23:06:31.414741+0000","last_fullsized":"2024-04-21T23:06:31.414741+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:04.265855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259788+0000","last_change":"2024-04-21T23:06:25.211787+0000","last_active":"2024-04-21T23:06:31.259788+0000","last_peered":"2024-04-21T23:06:31.259788+0000","last_clean":"2024-04-21T23:06:31.259788+0000","last_became_active":"2024-04-21T23:06:25.211473+0000","last_became_peered":"2024-04-21T23:06:25.211473+0000","last_unstale":"2024-04-21T23:06:31.259788+0000","last_undegraded":"2024-04-21T23:06:31.259788+0000","last_fullsized":"2024-04-21T23:06:31.259788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:58:51.364974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414912+0000","last_change":"2024-04-21T23:06:25.990681+0000","last_active":"2024-04-21T23:06:31.414912+0000","last_peered":"2024-04-21T23:06:31.414912+0000","last_clean":"2024-04-21T23:06:31.414912+0000","last_became_active":"2024-04-21T23:06:25.990094+0000","last_became_peered":"2024-04-21T23:06:25.990094+0000","last_unstale":"2024-04-21T23:06:31.414912+0000","last_undegraded":"2024-04-21T23:06:31.414912+0000","last_fullsized":"2024-04-21T23:06:31.414912+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:15:26.907912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414912+0000","last_change":"2024-04-21T23:06:30.271352+0000","last_active":"2024-04-21T23:06:31.414912+0000","last_peered":"2024-04-21T23:06:31.414912+0000","last_clean":"2024-04-21T23:06:31.414912+0000","last_became_active":"2024-04-21T23:06:30.270601+0000","last_became_peered":"2024-04-21T23:06:30.270601+0000","last_unstale":"2024-04-21T23:06:31.414912+0000","last_undegraded":"2024-04-21T23:06:31.414912+0000","last_fullsized":"2024-04-21T23:06:31.414912+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:37.266987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414696+0000","last_change":"2024-04-21T23:06:25.213288+0000","last_active":"2024-04-21T23:06:31.414696+0000","last_peered":"2024-04-21T23:06:31.414696+0000","last_clean":"2024-04-21T23:06:31.414696+0000","last_became_active":"2024-04-21T23:06:25.212936+0000","last_became_peered":"2024-04-21T23:06:25.212936+0000","last_unstale":"2024-04-21T23:06:31.414696+0000","last_undegraded":"2024-04-21T23:06:31.414696+0000","last_fullsized":"2024-04-21T23:06:31.414696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:35.303093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414833+0000","last_change":"2024-04-21T23:06:25.998485+0000","last_active":"2024-04-21T23:06:31.414833+0000","last_peered":"2024-04-21T23:06:31.414833+0000","last_clean":"2024-04-21T23:06:31.414833+0000","last_became_active":"2024-04-21T23:06:25.998184+0000","last_became_peered":"2024-04-21T23:06:25.998184+0000","last_unstale":"2024-04-21T23:06:31.414833+0000","last_undegraded":"2024-04-21T23:06:31.414833+0000","last_fullsized":"2024-04-21T23:06:31.414833+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:35:26.613636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259325+0000","last_change":"2024-04-21T23:06:30.270089+0000","last_active":"2024-04-21T23:06:31.259325+0000","last_peered":"2024-04-21T23:06:31.259325+0000","last_clean":"2024-04-21T23:06:31.259325+0000","last_became_active":"2024-04-21T23:06:30.269695+0000","last_became_peered":"2024-04-21T23:06:30.269695+0000","last_unstale":"2024-04-21T23:06:31.259325+0000","last_undegraded":"2024-04-21T23:06:31.259325+0000","last_fullsized":"2024-04-21T23:06:31.259325+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:42.718580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414466+0000","last_change":"2024-04-21T23:06:25.218247+0000","last_active":"2024-04-21T23:06:31.414466+0000","last_peered":"2024-04-21T23:06:31.414466+0000","last_clean":"2024-04-21T23:06:31.414466+0000","last_became_active":"2024-04-21T23:06:25.217924+0000","last_became_peered":"2024-04-21T23:06:25.217924+0000","last_unstale":"2024-04-21T23:06:31.414466+0000","last_undegraded":"2024-04-21T23:06:31.414466+0000","last_fullsized":"2024-04-21T23:06:31.414466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:27:12.155597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259384+0000","last_change":"2024-04-21T23:06:25.994109+0000","last_active":"2024-04-21T23:06:31.259384+0000","last_peered":"2024-04-21T23:06:31.259384+0000","last_clean":"2024-04-21T23:06:31.259384+0000","last_became_active":"2024-04-21T23:06:25.993785+0000","last_became_peered":"2024-04-21T23:06:25.993785+0000","last_unstale":"2024-04-21T23:06:31.259384+0000","last_undegraded":"2024-04-21T23:06:31.259384+0000","last_fullsized":"2024-04-21T23:06:31.259384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:12.629364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259595+0000","last_change":"2024-04-21T23:06:30.265444+0000","last_active":"2024-04-21T23:06:31.259595+0000","last_peered":"2024-04-21T23:06:31.259595+0000","last_clean":"2024-04-21T23:06:31.259595+0000","last_became_active":"2024-04-21T23:06:30.264971+0000","last_became_peered":"2024-04-21T23:06:30.264971+0000","last_unstale":"2024-04-21T23:06:31.259595+0000","last_undegraded":"2024-04-21T23:06:31.259595+0000","last_fullsized":"2024-04-21T23:06:31.259595+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:03.596631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416086+0000","last_change":"2024-04-21T23:06:25.205988+0000","last_active":"2024-04-21T23:06:31.416086+0000","last_peered":"2024-04-21T23:06:31.416086+0000","last_clean":"2024-04-21T23:06:31.416086+0000","last_became_active":"2024-04-21T23:06:25.205642+0000","last_became_peered":"2024-04-21T23:06:25.205642+0000","last_unstale":"2024-04-21T23:06:31.416086+0000","last_undegraded":"2024-04-21T23:06:31.416086+0000","last_fullsized":"2024-04-21T23:06:31.416086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:02.225147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415864+0000","last_change":"2024-04-21T23:06:25.990193+0000","last_active":"2024-04-21T23:06:31.415864+0000","last_peered":"2024-04-21T23:06:31.415864+0000","last_clean":"2024-04-21T23:06:31.415864+0000","last_became_active":"2024-04-21T23:06:25.989842+0000","last_became_peered":"2024-04-21T23:06:25.989842+0000","last_unstale":"2024-04-21T23:06:31.415864+0000","last_undegraded":"2024-04-21T23:06:31.415864+0000","last_fullsized":"2024-04-21T23:06:31.415864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:54:53.421257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415295+0000","last_change":"2024-04-21T23:06:30.405155+0000","last_active":"2024-04-21T23:06:31.415295+0000","last_peered":"2024-04-21T23:06:31.415295+0000","last_clean":"2024-04-21T23:06:31.415295+0000","last_became_active":"2024-04-21T23:06:30.404657+0000","last_became_peered":"2024-04-21T23:06:30.404657+0000","last_unstale":"2024-04-21T23:06:31.415295+0000","last_undegraded":"2024-04-21T23:06:31.415295+0000","last_fullsized":"2024-04-21T23:06:31.415295+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:38:22.286830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414890+0000","last_change":"2024-04-21T23:06:25.218208+0000","last_active":"2024-04-21T23:06:31.414890+0000","last_peered":"2024-04-21T23:06:31.414890+0000","last_clean":"2024-04-21T23:06:31.414890+0000","last_became_active":"2024-04-21T23:06:25.217889+0000","last_became_peered":"2024-04-21T23:06:25.217889+0000","last_unstale":"2024-04-21T23:06:31.414890+0000","last_undegraded":"2024-04-21T23:06:31.414890+0000","last_fullsized":"2024-04-21T23:06:31.414890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:00:08.775682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416345+0000","last_change":"2024-04-21T23:06:25.987615+0000","last_active":"2024-04-21T23:06:31.416345+0000","last_peered":"2024-04-21T23:06:31.416345+0000","last_clean":"2024-04-21T23:06:31.416345+0000","last_became_active":"2024-04-21T23:06:25.987305+0000","last_became_peered":"2024-04-21T23:06:25.987305+0000","last_unstale":"2024-04-21T23:06:31.416345+0000","last_undegraded":"2024-04-21T23:06:31.416345+0000","last_fullsized":"2024-04-21T23:06:31.416345+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:23:49.386998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259112+0000","last_change":"2024-04-21T23:06:30.268157+0000","last_active":"2024-04-21T23:06:31.259112+0000","last_peered":"2024-04-21T23:06:31.259112+0000","last_clean":"2024-04-21T23:06:31.259112+0000","last_became_active":"2024-04-21T23:06:30.267732+0000","last_became_peered":"2024-04-21T23:06:30.267732+0000","last_unstale":"2024-04-21T23:06:31.259112+0000","last_undegraded":"2024-04-21T23:06:31.259112+0000","last_fullsized":"2024-04-21T23:06:31.259112+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:29:24.250271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415607+0000","last_change":"2024-04-21T23:06:25.215401+0000","last_active":"2024-04-21T23:06:31.415607+0000","last_peered":"2024-04-21T23:06:31.415607+0000","last_clean":"2024-04-21T23:06:31.415607+0000","last_became_active":"2024-04-21T23:06:25.215067+0000","last_became_peered":"2024-04-21T23:06:25.215067+0000","last_unstale":"2024-04-21T23:06:31.415607+0000","last_undegraded":"2024-04-21T23:06:31.415607+0000","last_fullsized":"2024-04-21T23:06:31.415607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:16:58.977797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416030+0000","last_change":"2024-04-21T23:06:25.996421+0000","last_active":"2024-04-21T23:06:31.416030+0000","last_peered":"2024-04-21T23:06:31.416030+0000","last_clean":"2024-04-21T23:06:31.416030+0000","last_became_active":"2024-04-21T23:06:25.996102+0000","last_became_peered":"2024-04-21T23:06:25.996102+0000","last_unstale":"2024-04-21T23:06:31.416030+0000","last_undegraded":"2024-04-21T23:06:31.416030+0000","last_fullsized":"2024-04-21T23:06:31.416030+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:06:42.630182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259254+0000","last_change":"2024-04-21T23:06:30.404509+0000","last_active":"2024-04-21T23:06:31.259254+0000","last_peered":"2024-04-21T23:06:31.259254+0000","last_clean":"2024-04-21T23:06:31.259254+0000","last_became_active":"2024-04-21T23:06:30.403987+0000","last_became_peered":"2024-04-21T23:06:30.403987+0000","last_unstale":"2024-04-21T23:06:31.259254+0000","last_undegraded":"2024-04-21T23:06:31.259254+0000","last_fullsized":"2024-04-21T23:06:31.259254+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:12:28.412117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259536+0000","last_change":"2024-04-21T23:06:25.994076+0000","last_active":"2024-04-21T23:06:31.259536+0000","last_peered":"2024-04-21T23:06:31.259536+0000","last_clean":"2024-04-21T23:06:31.259536+0000","last_became_active":"2024-04-21T23:06:25.993747+0000","last_became_peered":"2024-04-21T23:06:25.993747+0000","last_unstale":"2024-04-21T23:06:31.259536+0000","last_undegraded":"2024-04-21T23:06:31.259536+0000","last_fullsized":"2024-04-21T23:06:31.259536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:17.771165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.258484+0000","last_change":"2024-04-21T23:06:25.209756+0000","last_active":"2024-04-21T23:06:31.258484+0000","last_peered":"2024-04-21T23:06:31.258484+0000","last_clean":"2024-04-21T23:06:31.258484+0000","last_became_active":"2024-04-21T23:06:25.209445+0000","last_became_peered":"2024-04-21T23:06:25.209445+0000","last_unstale":"2024-04-21T23:06:31.258484+0000","last_undegraded":"2024-04-21T23:06:31.258484+0000","last_fullsized":"2024-04-21T23:06:31.258484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:58:49.453047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415301+0000","last_change":"2024-04-21T23:06:30.261565+0000","last_active":"2024-04-21T23:06:31.415301+0000","last_peered":"2024-04-21T23:06:31.415301+0000","last_clean":"2024-04-21T23:06:31.415301+0000","last_became_active":"2024-04-21T23:06:30.261095+0000","last_became_peered":"2024-04-21T23:06:30.261095+0000","last_unstale":"2024-04-21T23:06:31.415301+0000","last_undegraded":"2024-04-21T23:06:31.415301+0000","last_fullsized":"2024-04-21T23:06:31.415301+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:37:23.174152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416085+0000","last_change":"2024-04-21T23:06:25.981402+0000","last_active":"2024-04-21T23:06:31.416085+0000","last_peered":"2024-04-21T23:06:31.416085+0000","last_clean":"2024-04-21T23:06:31.416085+0000","last_became_active":"2024-04-21T23:06:25.981062+0000","last_became_peered":"2024-04-21T23:06:25.981062+0000","last_unstale":"2024-04-21T23:06:31.416085+0000","last_undegraded":"2024-04-21T23:06:31.416085+0000","last_fullsized":"2024-04-21T23:06:31.416085+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:32:32.556123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415144+0000","last_change":"2024-04-21T23:06:25.208968+0000","last_active":"2024-04-21T23:06:31.415144+0000","last_peered":"2024-04-21T23:06:31.415144+0000","last_clean":"2024-04-21T23:06:31.415144+0000","last_became_active":"2024-04-21T23:06:25.208606+0000","last_became_peered":"2024-04-21T23:06:25.208606+0000","last_unstale":"2024-04-21T23:06:31.415144+0000","last_undegraded":"2024-04-21T23:06:31.415144+0000","last_fullsized":"2024-04-21T23:06:31.415144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:31:46.147119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":358,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414934+0000","last_change":"2024-04-21T23:05:13.077846+0000","last_active":"2024-04-21T23:06:31.414934+0000","last_peered":"2024-04-21T23:06:31.414934+0000","last_clean":"2024-04-21T23:06:31.414934+0000","last_became_active":"2024-04-21T23:05:13.077436+0000","last_became_peered":"2024-04-21T23:05:13.077436+0000","last_unstale":"2024-04-21T23:06:31.414934+0000","last_undegraded":"2024-04-21T23:06:31.414934+0000","last_fullsized":"2024-04-21T23:06:31.414934+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:01:47.619212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:01:47.619212+0000","last_clean_scrub_stamp":"2024-04-21T23:01:47.619212+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:12:09.826839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.260038+0000","last_change":"2024-04-21T23:06:30.405121+0000","last_active":"2024-04-21T23:06:31.260038+0000","last_peered":"2024-04-21T23:06:31.260038+0000","last_clean":"2024-04-21T23:06:31.260038+0000","last_became_active":"2024-04-21T23:06:30.404612+0000","last_became_peered":"2024-04-21T23:06:30.404612+0000","last_unstale":"2024-04-21T23:06:31.260038+0000","last_undegraded":"2024-04-21T23:06:31.260038+0000","last_fullsized":"2024-04-21T23:06:31.260038+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:59:01.737337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.258456+0000","last_change":"2024-04-21T23:06:25.997879+0000","last_active":"2024-04-21T23:06:31.258456+0000","last_peered":"2024-04-21T23:06:31.258456+0000","last_clean":"2024-04-21T23:06:31.258456+0000","last_became_active":"2024-04-21T23:06:25.997507+0000","last_became_peered":"2024-04-21T23:06:25.997507+0000","last_unstale":"2024-04-21T23:06:31.258456+0000","last_undegraded":"2024-04-21T23:06:31.258456+0000","last_fullsized":"2024-04-21T23:06:31.258456+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:13:16.044456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414717+0000","last_change":"2024-04-21T23:06:25.209714+0000","last_active":"2024-04-21T23:06:31.414717+0000","last_peered":"2024-04-21T23:06:31.414717+0000","last_clean":"2024-04-21T23:06:31.414717+0000","last_became_active":"2024-04-21T23:06:25.209373+0000","last_became_peered":"2024-04-21T23:06:25.209373+0000","last_unstale":"2024-04-21T23:06:31.414717+0000","last_undegraded":"2024-04-21T23:06:31.414717+0000","last_fullsized":"2024-04-21T23:06:31.414717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:21.654143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415616+0000","last_change":"2024-04-21T23:06:30.264873+0000","last_active":"2024-04-21T23:06:31.415616+0000","last_peered":"2024-04-21T23:06:31.415616+0000","last_clean":"2024-04-21T23:06:31.415616+0000","last_became_active":"2024-04-21T23:06:30.264318+0000","last_became_peered":"2024-04-21T23:06:30.264318+0000","last_unstale":"2024-04-21T23:06:31.415616+0000","last_undegraded":"2024-04-21T23:06:31.415616+0000","last_fullsized":"2024-04-21T23:06:31.415616+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:01:22.558782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415784+0000","last_change":"2024-04-21T23:06:25.218767+0000","last_active":"2024-04-21T23:06:31.415784+0000","last_peered":"2024-04-21T23:06:31.415784+0000","last_clean":"2024-04-21T23:06:31.415784+0000","last_became_active":"2024-04-21T23:06:25.218278+0000","last_became_peered":"2024-04-21T23:06:25.218278+0000","last_unstale":"2024-04-21T23:06:31.415784+0000","last_undegraded":"2024-04-21T23:06:31.415784+0000","last_fullsized":"2024-04-21T23:06:31.415784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:03:05.063515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259189+0000","last_change":"2024-04-21T23:06:25.968551+0000","last_active":"2024-04-21T23:06:31.259189+0000","last_peered":"2024-04-21T23:06:31.259189+0000","last_clean":"2024-04-21T23:06:31.259189+0000","last_became_active":"2024-04-21T23:06:25.968167+0000","last_became_peered":"2024-04-21T23:06:25.968167+0000","last_unstale":"2024-04-21T23:06:31.259189+0000","last_undegraded":"2024-04-21T23:06:31.259189+0000","last_fullsized":"2024-04-21T23:06:31.259189+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:08.920677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415074+0000","last_change":"2024-04-21T23:06:30.404811+0000","last_active":"2024-04-21T23:06:31.415074+0000","last_peered":"2024-04-21T23:06:31.415074+0000","last_clean":"2024-04-21T23:06:31.415074+0000","last_became_active":"2024-04-21T23:06:30.404375+0000","last_became_peered":"2024-04-21T23:06:30.404375+0000","last_unstale":"2024-04-21T23:06:31.415074+0000","last_undegraded":"2024-04-21T23:06:31.415074+0000","last_fullsized":"2024-04-21T23:06:31.415074+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:10:15.688917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415668+0000","last_change":"2024-04-21T23:06:25.204371+0000","last_active":"2024-04-21T23:06:31.415668+0000","last_peered":"2024-04-21T23:06:31.415668+0000","last_clean":"2024-04-21T23:06:31.415668+0000","last_became_active":"2024-04-21T23:06:25.204025+0000","last_became_peered":"2024-04-21T23:06:25.204025+0000","last_unstale":"2024-04-21T23:06:31.415668+0000","last_undegraded":"2024-04-21T23:06:31.415668+0000","last_fullsized":"2024-04-21T23:06:31.415668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:34:31.071013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416396+0000","last_change":"2024-04-21T23:06:25.990800+0000","last_active":"2024-04-21T23:06:31.416396+0000","last_peered":"2024-04-21T23:06:31.416396+0000","last_clean":"2024-04-21T23:06:31.416396+0000","last_became_active":"2024-04-21T23:06:25.990477+0000","last_became_peered":"2024-04-21T23:06:25.990477+0000","last_unstale":"2024-04-21T23:06:31.416396+0000","last_undegraded":"2024-04-21T23:06:31.416396+0000","last_fullsized":"2024-04-21T23:06:31.416396+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:55:33.282303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415545+0000","last_change":"2024-04-21T23:06:30.260090+0000","last_active":"2024-04-21T23:06:31.415545+0000","last_peered":"2024-04-21T23:06:31.415545+0000","last_clean":"2024-04-21T23:06:31.415545+0000","last_became_active":"2024-04-21T23:06:30.259609+0000","last_became_peered":"2024-04-21T23:06:30.259609+0000","last_unstale":"2024-04-21T23:06:31.415545+0000","last_undegraded":"2024-04-21T23:06:31.415545+0000","last_fullsized":"2024-04-21T23:06:31.415545+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:11:17.933683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415346+0000","last_change":"2024-04-21T23:06:25.215446+0000","last_active":"2024-04-21T23:06:31.415346+0000","last_peered":"2024-04-21T23:06:31.415346+0000","last_clean":"2024-04-21T23:06:31.415346+0000","last_became_active":"2024-04-21T23:06:25.214773+0000","last_became_peered":"2024-04-21T23:06:25.214773+0000","last_unstale":"2024-04-21T23:06:31.415346+0000","last_undegraded":"2024-04-21T23:06:31.415346+0000","last_fullsized":"2024-04-21T23:06:31.415346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:25:01.090699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414575+0000","last_change":"2024-04-21T23:06:25.986640+0000","last_active":"2024-04-21T23:06:31.414575+0000","last_peered":"2024-04-21T23:06:31.414575+0000","last_clean":"2024-04-21T23:06:31.414575+0000","last_became_active":"2024-04-21T23:06:25.986285+0000","last_became_peered":"2024-04-21T23:06:25.986285+0000","last_unstale":"2024-04-21T23:06:31.414575+0000","last_undegraded":"2024-04-21T23:06:31.414575+0000","last_fullsized":"2024-04-21T23:06:31.414575+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:01:05.498199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416147+0000","last_change":"2024-04-21T23:06:30.405165+0000","last_active":"2024-04-21T23:06:31.416147+0000","last_peered":"2024-04-21T23:06:31.416147+0000","last_clean":"2024-04-21T23:06:31.416147+0000","last_became_active":"2024-04-21T23:06:30.404622+0000","last_became_peered":"2024-04-21T23:06:30.404622+0000","last_unstale":"2024-04-21T23:06:31.416147+0000","last_undegraded":"2024-04-21T23:06:31.416147+0000","last_fullsized":"2024-04-21T23:06:31.416147+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:34:16.995792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.263538+0000","last_change":"2024-04-21T23:06:25.217298+0000","last_active":"2024-04-21T23:06:31.263538+0000","last_peered":"2024-04-21T23:06:31.263538+0000","last_clean":"2024-04-21T23:06:31.263538+0000","last_became_active":"2024-04-21T23:06:25.216965+0000","last_became_peered":"2024-04-21T23:06:25.216965+0000","last_unstale":"2024-04-21T23:06:31.263538+0000","last_undegraded":"2024-04-21T23:06:31.263538+0000","last_fullsized":"2024-04-21T23:06:31.263538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:56:38.543697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414523+0000","last_change":"2024-04-21T23:06:25.981400+0000","last_active":"2024-04-21T23:06:31.414523+0000","last_peered":"2024-04-21T23:06:31.414523+0000","last_clean":"2024-04-21T23:06:31.414523+0000","last_became_active":"2024-04-21T23:06:25.981118+0000","last_became_peered":"2024-04-21T23:06:25.981118+0000","last_unstale":"2024-04-21T23:06:31.414523+0000","last_undegraded":"2024-04-21T23:06:31.414523+0000","last_fullsized":"2024-04-21T23:06:31.414523+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:20:59.631062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415183+0000","last_change":"2024-04-21T23:06:30.270351+0000","last_active":"2024-04-21T23:06:31.415183+0000","last_peered":"2024-04-21T23:06:31.415183+0000","last_clean":"2024-04-21T23:06:31.415183+0000","last_became_active":"2024-04-21T23:06:30.269659+0000","last_became_peered":"2024-04-21T23:06:30.269659+0000","last_unstale":"2024-04-21T23:06:31.415183+0000","last_undegraded":"2024-04-21T23:06:31.415183+0000","last_fullsized":"2024-04-21T23:06:31.415183+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:19:14.132191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415265+0000","last_change":"2024-04-21T23:06:25.212787+0000","last_active":"2024-04-21T23:06:31.415265+0000","last_peered":"2024-04-21T23:06:31.415265+0000","last_clean":"2024-04-21T23:06:31.415265+0000","last_became_active":"2024-04-21T23:06:25.212423+0000","last_became_peered":"2024-04-21T23:06:25.212423+0000","last_unstale":"2024-04-21T23:06:31.415265+0000","last_undegraded":"2024-04-21T23:06:31.415265+0000","last_fullsized":"2024-04-21T23:06:31.415265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:45.875664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414465+0000","last_change":"2024-04-21T23:06:25.978205+0000","last_active":"2024-04-21T23:06:31.414465+0000","last_peered":"2024-04-21T23:06:31.414465+0000","last_clean":"2024-04-21T23:06:31.414465+0000","last_became_active":"2024-04-21T23:06:25.977876+0000","last_became_peered":"2024-04-21T23:06:25.977876+0000","last_unstale":"2024-04-21T23:06:31.414465+0000","last_undegraded":"2024-04-21T23:06:31.414465+0000","last_fullsized":"2024-04-21T23:06:31.414465+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:43:29.280255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415829+0000","last_change":"2024-04-21T23:06:30.271928+0000","last_active":"2024-04-21T23:06:31.415829+0000","last_peered":"2024-04-21T23:06:31.415829+0000","last_clean":"2024-04-21T23:06:31.415829+0000","last_became_active":"2024-04-21T23:06:30.271321+0000","last_became_peered":"2024-04-21T23:06:30.271321+0000","last_unstale":"2024-04-21T23:06:31.415829+0000","last_undegraded":"2024-04-21T23:06:31.415829+0000","last_fullsized":"2024-04-21T23:06:31.415829+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:22:03.343620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415140+0000","last_change":"2024-04-21T23:06:25.204175+0000","last_active":"2024-04-21T23:06:31.415140+0000","last_peered":"2024-04-21T23:06:31.415140+0000","last_clean":"2024-04-21T23:06:31.415140+0000","last_became_active":"2024-04-21T23:06:25.203826+0000","last_became_peered":"2024-04-21T23:06:25.203826+0000","last_unstale":"2024-04-21T23:06:31.415140+0000","last_undegraded":"2024-04-21T23:06:31.415140+0000","last_fullsized":"2024-04-21T23:06:31.415140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:28:53.402395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415671+0000","last_change":"2024-04-21T23:06:25.990559+0000","last_active":"2024-04-21T23:06:31.415671+0000","last_peered":"2024-04-21T23:06:31.415671+0000","last_clean":"2024-04-21T23:06:31.415671+0000","last_became_active":"2024-04-21T23:06:25.990239+0000","last_became_peered":"2024-04-21T23:06:25.990239+0000","last_unstale":"2024-04-21T23:06:31.415671+0000","last_undegraded":"2024-04-21T23:06:31.415671+0000","last_fullsized":"2024-04-21T23:06:31.415671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:49:26.891190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416117+0000","last_change":"2024-04-21T23:06:30.406584+0000","last_active":"2024-04-21T23:06:31.416117+0000","last_peered":"2024-04-21T23:06:31.416117+0000","last_clean":"2024-04-21T23:06:31.416117+0000","last_became_active":"2024-04-21T23:06:30.404576+0000","last_became_peered":"2024-04-21T23:06:30.404576+0000","last_unstale":"2024-04-21T23:06:31.416117+0000","last_undegraded":"2024-04-21T23:06:31.416117+0000","last_fullsized":"2024-04-21T23:06:31.416117+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:56.047199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414864+0000","last_change":"2024-04-21T23:06:25.214502+0000","last_active":"2024-04-21T23:06:31.414864+0000","last_peered":"2024-04-21T23:06:31.414864+0000","last_clean":"2024-04-21T23:06:31.414864+0000","last_became_active":"2024-04-21T23:06:25.214035+0000","last_became_peered":"2024-04-21T23:06:25.214035+0000","last_unstale":"2024-04-21T23:06:31.414864+0000","last_undegraded":"2024-04-21T23:06:31.414864+0000","last_fullsized":"2024-04-21T23:06:31.414864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:51:51.760402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415715+0000","last_change":"2024-04-21T23:06:25.988925+0000","last_active":"2024-04-21T23:06:31.415715+0000","last_peered":"2024-04-21T23:06:31.415715+0000","last_clean":"2024-04-21T23:06:31.415715+0000","last_became_active":"2024-04-21T23:06:25.988480+0000","last_became_peered":"2024-04-21T23:06:25.988480+0000","last_unstale":"2024-04-21T23:06:31.415715+0000","last_undegraded":"2024-04-21T23:06:31.415715+0000","last_fullsized":"2024-04-21T23:06:31.415715+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:32:02.867858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.263752+0000","last_change":"2024-04-21T23:06:30.260841+0000","last_active":"2024-04-21T23:06:31.263752+0000","last_peered":"2024-04-21T23:06:31.263752+0000","last_clean":"2024-04-21T23:06:31.263752+0000","last_became_active":"2024-04-21T23:06:30.260376+0000","last_became_peered":"2024-04-21T23:06:30.260376+0000","last_unstale":"2024-04-21T23:06:31.263752+0000","last_undegraded":"2024-04-21T23:06:31.263752+0000","last_fullsized":"2024-04-21T23:06:31.263752+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:24:57.429169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415872+0000","last_change":"2024-04-21T23:06:25.217920+0000","last_active":"2024-04-21T23:06:31.415872+0000","last_peered":"2024-04-21T23:06:31.415872+0000","last_clean":"2024-04-21T23:06:31.415872+0000","last_became_active":"2024-04-21T23:06:25.217379+0000","last_became_peered":"2024-04-21T23:06:25.217379+0000","last_unstale":"2024-04-21T23:06:31.415872+0000","last_undegraded":"2024-04-21T23:06:31.415872+0000","last_fullsized":"2024-04-21T23:06:31.415872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:09.570463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415629+0000","last_change":"2024-04-21T23:06:25.993533+0000","last_active":"2024-04-21T23:06:31.415629+0000","last_peered":"2024-04-21T23:06:31.415629+0000","last_clean":"2024-04-21T23:06:31.415629+0000","last_became_active":"2024-04-21T23:06:25.989739+0000","last_became_peered":"2024-04-21T23:06:25.989739+0000","last_unstale":"2024-04-21T23:06:31.415629+0000","last_undegraded":"2024-04-21T23:06:31.415629+0000","last_fullsized":"2024-04-21T23:06:31.415629+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:24:05.594026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414078+0000","last_change":"2024-04-21T23:06:30.276121+0000","last_active":"2024-04-21T23:06:31.414078+0000","last_peered":"2024-04-21T23:06:31.414078+0000","last_clean":"2024-04-21T23:06:31.414078+0000","last_became_active":"2024-04-21T23:06:30.275650+0000","last_became_peered":"2024-04-21T23:06:30.275650+0000","last_unstale":"2024-04-21T23:06:31.414078+0000","last_undegraded":"2024-04-21T23:06:31.414078+0000","last_fullsized":"2024-04-21T23:06:31.414078+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:00:19.559482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.260186+0000","last_change":"2024-04-21T23:06:25.207079+0000","last_active":"2024-04-21T23:06:31.260186+0000","last_peered":"2024-04-21T23:06:31.260186+0000","last_clean":"2024-04-21T23:06:31.260186+0000","last_became_active":"2024-04-21T23:06:25.206724+0000","last_became_peered":"2024-04-21T23:06:25.206724+0000","last_unstale":"2024-04-21T23:06:31.260186+0000","last_undegraded":"2024-04-21T23:06:31.260186+0000","last_fullsized":"2024-04-21T23:06:31.260186+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:27.221282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416154+0000","last_change":"2024-04-21T23:06:25.988144+0000","last_active":"2024-04-21T23:06:31.416154+0000","last_peered":"2024-04-21T23:06:31.416154+0000","last_clean":"2024-04-21T23:06:31.416154+0000","last_became_active":"2024-04-21T23:06:25.987813+0000","last_became_peered":"2024-04-21T23:06:25.987813+0000","last_unstale":"2024-04-21T23:06:31.416154+0000","last_undegraded":"2024-04-21T23:06:31.416154+0000","last_fullsized":"2024-04-21T23:06:31.416154+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:23.733514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259282+0000","last_change":"2024-04-21T23:06:30.261548+0000","last_active":"2024-04-21T23:06:31.259282+0000","last_peered":"2024-04-21T23:06:31.259282+0000","last_clean":"2024-04-21T23:06:31.259282+0000","last_became_active":"2024-04-21T23:06:30.261152+0000","last_became_peered":"2024-04-21T23:06:30.261152+0000","last_unstale":"2024-04-21T23:06:31.259282+0000","last_undegraded":"2024-04-21T23:06:31.259282+0000","last_fullsized":"2024-04-21T23:06:31.259282+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:00:16.644253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259147+0000","last_change":"2024-04-21T23:06:25.208749+0000","last_active":"2024-04-21T23:06:31.259147+0000","last_peered":"2024-04-21T23:06:31.259147+0000","last_clean":"2024-04-21T23:06:31.259147+0000","last_became_active":"2024-04-21T23:06:25.201511+0000","last_became_peered":"2024-04-21T23:06:25.201511+0000","last_unstale":"2024-04-21T23:06:31.259147+0000","last_undegraded":"2024-04-21T23:06:31.259147+0000","last_fullsized":"2024-04-21T23:06:31.259147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:41.928835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415249+0000","last_change":"2024-04-21T23:06:25.996041+0000","last_active":"2024-04-21T23:06:31.415249+0000","last_peered":"2024-04-21T23:06:31.415249+0000","last_clean":"2024-04-21T23:06:31.415249+0000","last_became_active":"2024-04-21T23:06:25.994482+0000","last_became_peered":"2024-04-21T23:06:25.994482+0000","last_unstale":"2024-04-21T23:06:31.415249+0000","last_undegraded":"2024-04-21T23:06:31.415249+0000","last_fullsized":"2024-04-21T23:06:31.415249+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:15:46.792618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414710+0000","last_change":"2024-04-21T23:06:25.974061+0000","last_active":"2024-04-21T23:06:31.414710+0000","last_peered":"2024-04-21T23:06:31.414710+0000","last_clean":"2024-04-21T23:06:31.414710+0000","last_became_active":"2024-04-21T23:06:25.973708+0000","last_became_peered":"2024-04-21T23:06:25.973708+0000","last_unstale":"2024-04-21T23:06:31.414710+0000","last_undegraded":"2024-04-21T23:06:31.414710+0000","last_fullsized":"2024-04-21T23:06:31.414710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:58.702947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415980+0000","last_change":"2024-04-21T23:06:25.988412+0000","last_active":"2024-04-21T23:06:31.415980+0000","last_peered":"2024-04-21T23:06:31.415980+0000","last_clean":"2024-04-21T23:06:31.415980+0000","last_became_active":"2024-04-21T23:06:25.988100+0000","last_became_peered":"2024-04-21T23:06:25.988100+0000","last_unstale":"2024-04-21T23:06:31.415980+0000","last_undegraded":"2024-04-21T23:06:31.415980+0000","last_fullsized":"2024-04-21T23:06:31.415980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:16:42.626123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415058+0000","last_change":"2024-04-21T23:06:25.980144+0000","last_active":"2024-04-21T23:06:31.415058+0000","last_peered":"2024-04-21T23:06:31.415058+0000","last_clean":"2024-04-21T23:06:31.415058+0000","last_became_active":"2024-04-21T23:06:25.979760+0000","last_became_peered":"2024-04-21T23:06:25.979760+0000","last_unstale":"2024-04-21T23:06:31.415058+0000","last_undegraded":"2024-04-21T23:06:31.415058+0000","last_fullsized":"2024-04-21T23:06:31.415058+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:14:32.496184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414155+0000","last_change":"2024-04-21T23:06:25.994732+0000","last_active":"2024-04-21T23:06:31.414155+0000","last_peered":"2024-04-21T23:06:31.414155+0000","last_clean":"2024-04-21T23:06:31.414155+0000","last_became_active":"2024-04-21T23:06:25.994408+0000","last_became_peered":"2024-04-21T23:06:25.994408+0000","last_unstale":"2024-04-21T23:06:31.414155+0000","last_undegraded":"2024-04-21T23:06:31.414155+0000","last_fullsized":"2024-04-21T23:06:31.414155+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:28:59.778049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259238+0000","last_change":"2024-04-21T23:06:25.988439+0000","last_active":"2024-04-21T23:06:31.259238+0000","last_peered":"2024-04-21T23:06:31.259238+0000","last_clean":"2024-04-21T23:06:31.259238+0000","last_became_active":"2024-04-21T23:06:25.988068+0000","last_became_peered":"2024-04-21T23:06:25.988068+0000","last_unstale":"2024-04-21T23:06:31.259238+0000","last_undegraded":"2024-04-21T23:06:31.259238+0000","last_fullsized":"2024-04-21T23:06:31.259238+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:45:26.345083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415112+0000","last_change":"2024-04-21T23:06:25.973850+0000","last_active":"2024-04-21T23:06:31.415112+0000","last_peered":"2024-04-21T23:06:31.415112+0000","last_clean":"2024-04-21T23:06:31.415112+0000","last_became_active":"2024-04-21T23:06:25.973522+0000","last_became_peered":"2024-04-21T23:06:25.973522+0000","last_unstale":"2024-04-21T23:06:31.415112+0000","last_undegraded":"2024-04-21T23:06:31.415112+0000","last_fullsized":"2024-04-21T23:06:31.415112+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:43:58.417551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415377+0000","last_change":"2024-04-21T23:06:25.978114+0000","last_active":"2024-04-21T23:06:31.415377+0000","last_peered":"2024-04-21T23:06:31.415377+0000","last_clean":"2024-04-21T23:06:31.415377+0000","last_became_active":"2024-04-21T23:06:25.977115+0000","last_became_peered":"2024-04-21T23:06:25.977115+0000","last_unstale":"2024-04-21T23:06:31.415377+0000","last_undegraded":"2024-04-21T23:06:31.415377+0000","last_fullsized":"2024-04-21T23:06:31.415377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:26:46.756490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416493+0000","last_change":"2024-04-21T23:06:25.994451+0000","last_active":"2024-04-21T23:06:31.416493+0000","last_peered":"2024-04-21T23:06:31.416493+0000","last_clean":"2024-04-21T23:06:31.416493+0000","last_became_active":"2024-04-21T23:06:25.994085+0000","last_became_peered":"2024-04-21T23:06:25.994085+0000","last_unstale":"2024-04-21T23:06:31.416493+0000","last_undegraded":"2024-04-21T23:06:31.416493+0000","last_fullsized":"2024-04-21T23:06:31.416493+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:53.652014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259410+0000","last_change":"2024-04-21T23:06:25.982536+0000","last_active":"2024-04-21T23:06:31.259410+0000","last_peered":"2024-04-21T23:06:31.259410+0000","last_clean":"2024-04-21T23:06:31.259410+0000","last_became_active":"2024-04-21T23:06:25.981318+0000","last_became_peered":"2024-04-21T23:06:25.981318+0000","last_unstale":"2024-04-21T23:06:31.259410+0000","last_undegraded":"2024-04-21T23:06:31.259410+0000","last_fullsized":"2024-04-21T23:06:31.259410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:35.784679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415749+0000","last_change":"2024-04-21T23:06:25.981915+0000","last_active":"2024-04-21T23:06:31.415749+0000","last_peered":"2024-04-21T23:06:31.415749+0000","last_clean":"2024-04-21T23:06:31.415749+0000","last_became_active":"2024-04-21T23:06:25.981547+0000","last_became_peered":"2024-04-21T23:06:25.981547+0000","last_unstale":"2024-04-21T23:06:31.415749+0000","last_undegraded":"2024-04-21T23:06:31.415749+0000","last_fullsized":"2024-04-21T23:06:31.415749+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:03:11.154582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.263827+0000","last_change":"2024-04-21T23:06:25.991687+0000","last_active":"2024-04-21T23:06:31.263827+0000","last_peered":"2024-04-21T23:06:31.263827+0000","last_clean":"2024-04-21T23:06:31.263827+0000","last_became_active":"2024-04-21T23:06:25.991248+0000","last_became_peered":"2024-04-21T23:06:25.991248+0000","last_unstale":"2024-04-21T23:06:31.263827+0000","last_undegraded":"2024-04-21T23:06:31.263827+0000","last_fullsized":"2024-04-21T23:06:31.263827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:05:04.205660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.258784+0000","last_change":"2024-04-21T23:06:25.993658+0000","last_active":"2024-04-21T23:06:31.258784+0000","last_peered":"2024-04-21T23:06:31.258784+0000","last_clean":"2024-04-21T23:06:31.258784+0000","last_became_active":"2024-04-21T23:06:25.993298+0000","last_became_peered":"2024-04-21T23:06:25.993298+0000","last_unstale":"2024-04-21T23:06:31.258784+0000","last_undegraded":"2024-04-21T23:06:31.258784+0000","last_fullsized":"2024-04-21T23:06:31.258784+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:18:30.216961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415341+0000","last_change":"2024-04-21T23:06:25.993849+0000","last_active":"2024-04-21T23:06:31.415341+0000","last_peered":"2024-04-21T23:06:31.415341+0000","last_clean":"2024-04-21T23:06:31.415341+0000","last_became_active":"2024-04-21T23:06:25.993501+0000","last_became_peered":"2024-04-21T23:06:25.993501+0000","last_unstale":"2024-04-21T23:06:31.415341+0000","last_undegraded":"2024-04-21T23:06:31.415341+0000","last_fullsized":"2024-04-21T23:06:31.415341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:40:10.461775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":51,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":37850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350994,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":995328,"data_stored":483870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612631,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32752,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712400,"statfs":{"total":95995035648,"available":95961497600,"internally_reserved":0,"allocated":1613824,"data_stored":1078274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":484452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070498,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1003520,"data_stored":484474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332133,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28584,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716568,"statfs":{"total":95995035648,"available":95965765632,"internally_reserved":0,"allocated":1593344,"data_stored":1074820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":483870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":487712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":483960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":511676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":483974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920821,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":999424,"data_stored":483892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28584,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716568,"statfs":{"total":95995035648,"available":95965765632,"internally_reserved":0,"allocated":1605632,"data_stored":1078516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":27806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T23:06:54.141 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T23:06:54.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.278+0000 7ff9c10f4640 1 -- 172.21.15.44:0/331834247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9bc102260 msgr2=0x7ff9bc108af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:54.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.278+0000 7ff9c10f4640 1 --2- 172.21.15.44:0/331834247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9bc102260 0x7ff9bc108af0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff9b4009ca0 tx=0x7ff9b402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:06:54.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.278+0000 7ff9c10f4640 1 -- 172.21.15.44:0/331834247 shutdown_connections 2024-04-21T23:06:54.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.278+0000 7ff9c10f4640 1 --2- 172.21.15.44:0/331834247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9bc1018a0 0x7ff9bc101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:54.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.278+0000 7ff9c10f4640 1 --2- 172.21.15.44:0/331834247 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9bc102260 0x7ff9bc108af0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:54.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.278+0000 7ff9c10f4640 1 --2- 172.21.15.44:0/331834247 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9bc1006b0 0x7ff9bc100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:54.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.278+0000 7ff9c10f4640 1 -- 172.21.15.44:0/331834247 >> 172.21.15.44:0/331834247 conn(0x7ff9bc0fbc20 msgr2=0x7ff9bc0fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:54.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.278+0000 7ff9c10f4640 1 -- 172.21.15.44:0/331834247 shutdown_connections 2024-04-21T23:06:54.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.278+0000 7ff9c10f4640 1 -- 172.21.15.44:0/331834247 wait complete. 2024-04-21T23:06:54.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9c10f4640 1 Processor -- start 2024-04-21T23:06:54.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9c10f4640 1 -- start start 2024-04-21T23:06:54.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9c10f4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9bc1006b0 0x7ff9bc10b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:54.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9c10f4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9bc1018a0 0x7ff9bc109670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:54.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9c10f4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9bc102260 0x7ff9bc109bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:54.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9c10f4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc10d450 con 0x7ff9bc102260 2024-04-21T23:06:54.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9c10f4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc10d2d0 con 0x7ff9bc1006b0 2024-04-21T23:06:54.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9c10f4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc10d5d0 con 0x7ff9bc1018a0 2024-04-21T23:06:54.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9bb59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9bc102260 0x7ff9bc109bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:54.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9bb59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9bc102260 0x7ff9bc109bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50806/0 (socket says 172.21.15.44:50806) 2024-04-21T23:06:54.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9bb59e640 1 -- 172.21.15.44:0/685763135 learned_addr learned my addr 172.21.15.44:0/685763135 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:54.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9bad9d640 1 --2- 172.21.15.44:0/685763135 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9bc1006b0 0x7ff9bc10b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:54.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9b3fff640 1 --2- 172.21.15.44:0/685763135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9bc1018a0 0x7ff9bc109670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:54.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9bb59e640 1 -- 172.21.15.44:0/685763135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9bc1018a0 msgr2=0x7ff9bc109670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:54.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9bb59e640 1 --2- 172.21.15.44:0/685763135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9bc1018a0 0x7ff9bc109670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:54.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9bb59e640 1 -- 172.21.15.44:0/685763135 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9bc1006b0 msgr2=0x7ff9bc10b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:54.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9bb59e640 1 --2- 172.21.15.44:0/685763135 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9bc1006b0 0x7ff9bc10b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:54.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9bb59e640 1 -- 172.21.15.44:0/685763135 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9bc10a440 con 0x7ff9bc102260 2024-04-21T23:06:54.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9b3fff640 1 --2- 172.21.15.44:0/685763135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9bc1018a0 0x7ff9bc109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:06:54.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9bad9d640 1 --2- 172.21.15.44:0/685763135 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9bc1006b0 0x7ff9bc10b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:06:54.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.279+0000 7ff9bb59e640 1 --2- 172.21.15.44:0/685763135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9bc102260 0x7ff9bc109bb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff9b402c700 tx=0x7ff9b4032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:54.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.280+0000 7ff9b8d99640 1 -- 172.21.15.44:0/685763135 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4039040 con 0x7ff9bc102260 2024-04-21T23:06:54.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.280+0000 7ff9c10f4640 1 -- 172.21.15.44:0/685763135 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9bc1a1a90 con 0x7ff9bc102260 2024-04-21T23:06:54.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.280+0000 7ff9b8d99640 1 -- 172.21.15.44:0/685763135 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9b4044070 con 0x7ff9bc102260 2024-04-21T23:06:54.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.280+0000 7ff9b8d99640 1 -- 172.21.15.44:0/685763135 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b40042a0 con 0x7ff9bc102260 2024-04-21T23:06:54.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.280+0000 7ff9c10f4640 1 -- 172.21.15.44:0/685763135 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9bc1a1fe0 con 0x7ff9bc102260 2024-04-21T23:06:54.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.281+0000 7ff9c10f4640 1 -- 172.21.15.44:0/685763135 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9bc067d60 con 0x7ff9bc102260 2024-04-21T23:06:54.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.282+0000 7ff9b8d99640 1 -- 172.21.15.44:0/685763135 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff9b4003970 con 0x7ff9bc102260 2024-04-21T23:06:54.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.282+0000 7ff9b8d99640 1 --2- 172.21.15.44:0/685763135 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff9a80a2c20 0x7ff9a80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:54.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.282+0000 7ff9bad9d640 1 --2- 172.21.15.44:0/685763135 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff9a80a2c20 0x7ff9a80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:54.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.282+0000 7ff9b8d99640 1 -- 172.21.15.44:0/685763135 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff9b40f1900 con 0x7ff9bc102260 2024-04-21T23:06:54.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.283+0000 7ff9bad9d640 1 --2- 172.21.15.44:0/685763135 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff9a80a2c20 0x7ff9a80a50e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac0058c0 tx=0x7ff9ac005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:54.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.287+0000 7ff9b8d99640 1 -- 172.21.15.44:0/685763135 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff9b40333a0 con 0x7ff9bc102260 2024-04-21T23:06:54.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:54 smithi044 ceph-mon[27934]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2024-04-21T23:06:54.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:54 smithi044 ceph-mon[27934]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:06:54.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:54 smithi044 ceph-mon[27934]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-21T23:06:54.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:54 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2866041990' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:54.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:54 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1978088580' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:54.468 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.468+0000 7ff9c10f4640 1 -- 172.21.15.44:0/685763135 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff9bc061b80 con 0x7ff9a80a2c20 2024-04-21T23:06:54.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.478+0000 7ff9b8d99640 1 -- 172.21.15.44:0/685763135 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340216 (secure 0 0 0) 0x7ff9bc061b80 con 0x7ff9a80a2c20 2024-04-21T23:06:54.481 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:06:54.483 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-21T23:06:54.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.482+0000 7ff9c10f4640 1 -- 172.21.15.44:0/685763135 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff9a80a2c20 msgr2=0x7ff9a80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:54.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.482+0000 7ff9c10f4640 1 --2- 172.21.15.44:0/685763135 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff9a80a2c20 0x7ff9a80a50e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac0058c0 tx=0x7ff9ac005850 comp rx=0 tx=0).stop 2024-04-21T23:06:54.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.482+0000 7ff9c10f4640 1 -- 172.21.15.44:0/685763135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9bc102260 msgr2=0x7ff9bc109bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:54.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.482+0000 7ff9c10f4640 1 --2- 172.21.15.44:0/685763135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9bc102260 0x7ff9bc109bb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff9b402c700 tx=0x7ff9b4032f20 comp rx=0 tx=0).stop 2024-04-21T23:06:54.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.482+0000 7ff9c10f4640 1 -- 172.21.15.44:0/685763135 shutdown_connections 2024-04-21T23:06:54.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.482+0000 7ff9c10f4640 1 --2- 172.21.15.44:0/685763135 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff9a80a2c20 0x7ff9a80a50e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:54.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.482+0000 7ff9c10f4640 1 --2- 172.21.15.44:0/685763135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9bc1018a0 0x7ff9bc109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:54.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.482+0000 7ff9c10f4640 1 --2- 172.21.15.44:0/685763135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff9bc102260 0x7ff9bc109bb0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:54.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.482+0000 7ff9c10f4640 1 --2- 172.21.15.44:0/685763135 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff9bc1006b0 0x7ff9bc10b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:54.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.482+0000 7ff9c10f4640 1 -- 172.21.15.44:0/685763135 >> 172.21.15.44:0/685763135 conn(0x7ff9bc0fbc20 msgr2=0x7ff9bc104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:54.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.482+0000 7ff9c10f4640 1 -- 172.21.15.44:0/685763135 shutdown_connections 2024-04-21T23:06:54.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:54.482+0000 7ff9c10f4640 1 -- 172.21.15.44:0/685763135 wait complete. 2024-04-21T23:06:54.494 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-04-21T23:06:53.057326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":142,"num_read_kb":125,"num_write":232,"num_write_kb":2958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181,"ondisk_log_size":181,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":362732,"kb_used_data":13528,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124579092,"statfs":{"total":1151940427776,"available":1151568990208,"internally_reserved":0,"allocated":13852672,"data_stored":7619490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003565"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416229+0000","last_change":"2024-04-21T23:06:25.980091+0000","last_active":"2024-04-21T23:06:31.416229+0000","last_peered":"2024-04-21T23:06:31.416229+0000","last_clean":"2024-04-21T23:06:31.416229+0000","last_became_active":"2024-04-21T23:06:25.979763+0000","last_became_peered":"2024-04-21T23:06:25.979763+0000","last_unstale":"2024-04-21T23:06:31.416229+0000","last_undegraded":"2024-04-21T23:06:31.416229+0000","last_fullsized":"2024-04-21T23:06:31.416229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:12:23.771931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414545+0000","last_change":"2024-04-21T23:06:25.992461+0000","last_active":"2024-04-21T23:06:31.414545+0000","last_peered":"2024-04-21T23:06:31.414545+0000","last_clean":"2024-04-21T23:06:31.414545+0000","last_became_active":"2024-04-21T23:06:25.992018+0000","last_became_peered":"2024-04-21T23:06:25.992018+0000","last_unstale":"2024-04-21T23:06:31.414545+0000","last_undegraded":"2024-04-21T23:06:31.414545+0000","last_fullsized":"2024-04-21T23:06:31.414545+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:46.651683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414373+0000","last_change":"2024-04-21T23:06:25.974812+0000","last_active":"2024-04-21T23:06:31.414373+0000","last_peered":"2024-04-21T23:06:31.414373+0000","last_clean":"2024-04-21T23:06:31.414373+0000","last_became_active":"2024-04-21T23:06:25.974420+0000","last_became_peered":"2024-04-21T23:06:25.974420+0000","last_unstale":"2024-04-21T23:06:31.414373+0000","last_undegraded":"2024-04-21T23:06:31.414373+0000","last_fullsized":"2024-04-21T23:06:31.414373+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:00.734218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416020+0000","last_change":"2024-04-21T23:06:25.988806+0000","last_active":"2024-04-21T23:06:31.416020+0000","last_peered":"2024-04-21T23:06:31.416020+0000","last_clean":"2024-04-21T23:06:31.416020+0000","last_became_active":"2024-04-21T23:06:25.988344+0000","last_became_peered":"2024-04-21T23:06:25.988344+0000","last_unstale":"2024-04-21T23:06:31.416020+0000","last_undegraded":"2024-04-21T23:06:31.416020+0000","last_fullsized":"2024-04-21T23:06:31.416020+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:15:40.079273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414684+0000","last_change":"2024-04-21T23:06:25.979125+0000","last_active":"2024-04-21T23:06:31.414684+0000","last_peered":"2024-04-21T23:06:31.414684+0000","last_clean":"2024-04-21T23:06:31.414684+0000","last_became_active":"2024-04-21T23:06:25.978778+0000","last_became_peered":"2024-04-21T23:06:25.978778+0000","last_unstale":"2024-04-21T23:06:31.414684+0000","last_undegraded":"2024-04-21T23:06:31.414684+0000","last_fullsized":"2024-04-21T23:06:31.414684+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:44.655404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416464+0000","last_change":"2024-04-21T23:06:25.983983+0000","last_active":"2024-04-21T23:06:31.416464+0000","last_peered":"2024-04-21T23:06:31.416464+0000","last_clean":"2024-04-21T23:06:31.416464+0000","last_became_active":"2024-04-21T23:06:25.983463+0000","last_became_peered":"2024-04-21T23:06:25.983463+0000","last_unstale":"2024-04-21T23:06:31.416464+0000","last_undegraded":"2024-04-21T23:06:31.416464+0000","last_fullsized":"2024-04-21T23:06:31.416464+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:32:12.784752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414557+0000","last_change":"2024-04-21T23:06:25.987479+0000","last_active":"2024-04-21T23:06:31.414557+0000","last_peered":"2024-04-21T23:06:31.414557+0000","last_clean":"2024-04-21T23:06:31.414557+0000","last_became_active":"2024-04-21T23:06:25.987133+0000","last_became_peered":"2024-04-21T23:06:25.987133+0000","last_unstale":"2024-04-21T23:06:31.414557+0000","last_undegraded":"2024-04-21T23:06:31.414557+0000","last_fullsized":"2024-04-21T23:06:31.414557+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:08.033184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414851+0000","last_change":"2024-04-21T23:06:25.987714+0000","last_active":"2024-04-21T23:06:31.414851+0000","last_peered":"2024-04-21T23:06:31.414851+0000","last_clean":"2024-04-21T23:06:31.414851+0000","last_became_active":"2024-04-21T23:06:25.987457+0000","last_became_peered":"2024-04-21T23:06:25.987457+0000","last_unstale":"2024-04-21T23:06:31.414851+0000","last_undegraded":"2024-04-21T23:06:31.414851+0000","last_fullsized":"2024-04-21T23:06:31.414851+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:06:06.184525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.264178+0000","last_change":"2024-04-21T23:06:25.975957+0000","last_active":"2024-04-21T23:06:31.264178+0000","last_peered":"2024-04-21T23:06:31.264178+0000","last_clean":"2024-04-21T23:06:31.264178+0000","last_became_active":"2024-04-21T23:06:25.974710+0000","last_became_peered":"2024-04-21T23:06:25.974710+0000","last_unstale":"2024-04-21T23:06:31.264178+0000","last_undegraded":"2024-04-21T23:06:31.264178+0000","last_fullsized":"2024-04-21T23:06:31.264178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:31:40.062273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416666+0000","last_change":"2024-04-21T23:06:25.995085+0000","last_active":"2024-04-21T23:06:31.416666+0000","last_peered":"2024-04-21T23:06:31.416666+0000","last_clean":"2024-04-21T23:06:31.416666+0000","last_became_active":"2024-04-21T23:06:25.993552+0000","last_became_peered":"2024-04-21T23:06:25.993552+0000","last_unstale":"2024-04-21T23:06:31.416666+0000","last_undegraded":"2024-04-21T23:06:31.416666+0000","last_fullsized":"2024-04-21T23:06:31.416666+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:11.405113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415519+0000","last_change":"2024-04-21T23:06:25.993191+0000","last_active":"2024-04-21T23:06:31.415519+0000","last_peered":"2024-04-21T23:06:31.415519+0000","last_clean":"2024-04-21T23:06:31.415519+0000","last_became_active":"2024-04-21T23:06:25.989399+0000","last_became_peered":"2024-04-21T23:06:25.989399+0000","last_unstale":"2024-04-21T23:06:31.415519+0000","last_undegraded":"2024-04-21T23:06:31.415519+0000","last_fullsized":"2024-04-21T23:06:31.415519+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:30:20.035599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415962+0000","last_change":"2024-04-21T23:06:25.994158+0000","last_active":"2024-04-21T23:06:31.415962+0000","last_peered":"2024-04-21T23:06:31.415962+0000","last_clean":"2024-04-21T23:06:31.415962+0000","last_became_active":"2024-04-21T23:06:25.993780+0000","last_became_peered":"2024-04-21T23:06:25.993780+0000","last_unstale":"2024-04-21T23:06:31.415962+0000","last_undegraded":"2024-04-21T23:06:31.415962+0000","last_fullsized":"2024-04-21T23:06:31.415962+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:52:17.181898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.258935+0000","last_change":"2024-04-21T23:06:25.989274+0000","last_active":"2024-04-21T23:06:31.258935+0000","last_peered":"2024-04-21T23:06:31.258935+0000","last_clean":"2024-04-21T23:06:31.258935+0000","last_became_active":"2024-04-21T23:06:25.988931+0000","last_became_peered":"2024-04-21T23:06:25.988931+0000","last_unstale":"2024-04-21T23:06:31.258935+0000","last_undegraded":"2024-04-21T23:06:31.258935+0000","last_fullsized":"2024-04-21T23:06:31.258935+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:11:04.576136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.413914+0000","last_change":"2024-04-21T23:06:25.982976+0000","last_active":"2024-04-21T23:06:31.413914+0000","last_peered":"2024-04-21T23:06:31.413914+0000","last_clean":"2024-04-21T23:06:31.413914+0000","last_became_active":"2024-04-21T23:06:25.982625+0000","last_became_peered":"2024-04-21T23:06:25.982625+0000","last_unstale":"2024-04-21T23:06:31.413914+0000","last_undegraded":"2024-04-21T23:06:31.413914+0000","last_fullsized":"2024-04-21T23:06:31.413914+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:51:02.193044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.413821+0000","last_change":"2024-04-21T23:06:25.994805+0000","last_active":"2024-04-21T23:06:31.413821+0000","last_peered":"2024-04-21T23:06:31.413821+0000","last_clean":"2024-04-21T23:06:31.413821+0000","last_became_active":"2024-04-21T23:06:25.994482+0000","last_became_peered":"2024-04-21T23:06:25.994482+0000","last_unstale":"2024-04-21T23:06:31.413821+0000","last_undegraded":"2024-04-21T23:06:31.413821+0000","last_fullsized":"2024-04-21T23:06:31.413821+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:24:21.792358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.263419+0000","last_change":"2024-04-21T23:06:25.990840+0000","last_active":"2024-04-21T23:06:31.263419+0000","last_peered":"2024-04-21T23:06:31.263419+0000","last_clean":"2024-04-21T23:06:31.263419+0000","last_became_active":"2024-04-21T23:06:25.990517+0000","last_became_peered":"2024-04-21T23:06:25.990517+0000","last_unstale":"2024-04-21T23:06:31.263419+0000","last_undegraded":"2024-04-21T23:06:31.263419+0000","last_fullsized":"2024-04-21T23:06:31.263419+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:48:09.265280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416081+0000","last_change":"2024-04-21T23:06:25.976810+0000","last_active":"2024-04-21T23:06:31.416081+0000","last_peered":"2024-04-21T23:06:31.416081+0000","last_clean":"2024-04-21T23:06:31.416081+0000","last_became_active":"2024-04-21T23:06:25.976489+0000","last_became_peered":"2024-04-21T23:06:25.976489+0000","last_unstale":"2024-04-21T23:06:31.416081+0000","last_undegraded":"2024-04-21T23:06:31.416081+0000","last_fullsized":"2024-04-21T23:06:31.416081+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:07:03.794115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415712+0000","last_change":"2024-04-21T23:06:25.988603+0000","last_active":"2024-04-21T23:06:31.415712+0000","last_peered":"2024-04-21T23:06:31.415712+0000","last_clean":"2024-04-21T23:06:31.415712+0000","last_became_active":"2024-04-21T23:06:25.988254+0000","last_became_peered":"2024-04-21T23:06:25.988254+0000","last_unstale":"2024-04-21T23:06:31.415712+0000","last_undegraded":"2024-04-21T23:06:31.415712+0000","last_fullsized":"2024-04-21T23:06:31.415712+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:25:46.099853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415311+0000","last_change":"2024-04-21T23:06:25.980831+0000","last_active":"2024-04-21T23:06:31.415311+0000","last_peered":"2024-04-21T23:06:31.415311+0000","last_clean":"2024-04-21T23:06:31.415311+0000","last_became_active":"2024-04-21T23:06:25.977051+0000","last_became_peered":"2024-04-21T23:06:25.977051+0000","last_unstale":"2024-04-21T23:06:31.415311+0000","last_undegraded":"2024-04-21T23:06:31.415311+0000","last_fullsized":"2024-04-21T23:06:31.415311+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:55:03.684734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416058+0000","last_change":"2024-04-21T23:06:30.256875+0000","last_active":"2024-04-21T23:06:31.416058+0000","last_peered":"2024-04-21T23:06:31.416058+0000","last_clean":"2024-04-21T23:06:31.416058+0000","last_became_active":"2024-04-21T23:06:30.256459+0000","last_became_peered":"2024-04-21T23:06:30.256459+0000","last_unstale":"2024-04-21T23:06:31.416058+0000","last_undegraded":"2024-04-21T23:06:31.416058+0000","last_fullsized":"2024-04-21T23:06:31.416058+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:45:32.376270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415309+0000","last_change":"2024-04-21T23:06:25.209605+0000","last_active":"2024-04-21T23:06:31.415309+0000","last_peered":"2024-04-21T23:06:31.415309+0000","last_clean":"2024-04-21T23:06:31.415309+0000","last_became_active":"2024-04-21T23:06:25.209277+0000","last_became_peered":"2024-04-21T23:06:25.209277+0000","last_unstale":"2024-04-21T23:06:31.415309+0000","last_undegraded":"2024-04-21T23:06:31.415309+0000","last_fullsized":"2024-04-21T23:06:31.415309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:15:49.742940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415528+0000","last_change":"2024-04-21T23:06:25.981774+0000","last_active":"2024-04-21T23:06:31.415528+0000","last_peered":"2024-04-21T23:06:31.415528+0000","last_clean":"2024-04-21T23:06:31.415528+0000","last_became_active":"2024-04-21T23:06:25.981432+0000","last_became_peered":"2024-04-21T23:06:25.981432+0000","last_unstale":"2024-04-21T23:06:31.415528+0000","last_undegraded":"2024-04-21T23:06:31.415528+0000","last_fullsized":"2024-04-21T23:06:31.415528+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:28:28.712149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414558+0000","last_change":"2024-04-21T23:06:30.271508+0000","last_active":"2024-04-21T23:06:31.414558+0000","last_peered":"2024-04-21T23:06:31.414558+0000","last_clean":"2024-04-21T23:06:31.414558+0000","last_became_active":"2024-04-21T23:06:30.270461+0000","last_became_peered":"2024-04-21T23:06:30.270461+0000","last_unstale":"2024-04-21T23:06:31.414558+0000","last_undegraded":"2024-04-21T23:06:31.414558+0000","last_fullsized":"2024-04-21T23:06:31.414558+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:43:07.473365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416426+0000","last_change":"2024-04-21T23:06:25.218773+0000","last_active":"2024-04-21T23:06:31.416426+0000","last_peered":"2024-04-21T23:06:31.416426+0000","last_clean":"2024-04-21T23:06:31.416426+0000","last_became_active":"2024-04-21T23:06:25.218458+0000","last_became_peered":"2024-04-21T23:06:25.218458+0000","last_unstale":"2024-04-21T23:06:31.416426+0000","last_undegraded":"2024-04-21T23:06:31.416426+0000","last_fullsized":"2024-04-21T23:06:31.416426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:25:16.372689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415140+0000","last_change":"2024-04-21T23:06:25.977669+0000","last_active":"2024-04-21T23:06:31.415140+0000","last_peered":"2024-04-21T23:06:31.415140+0000","last_clean":"2024-04-21T23:06:31.415140+0000","last_became_active":"2024-04-21T23:06:25.977143+0000","last_became_peered":"2024-04-21T23:06:25.977143+0000","last_unstale":"2024-04-21T23:06:31.415140+0000","last_undegraded":"2024-04-21T23:06:31.415140+0000","last_fullsized":"2024-04-21T23:06:31.415140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:39.289208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414991+0000","last_change":"2024-04-21T23:06:30.271934+0000","last_active":"2024-04-21T23:06:31.414991+0000","last_peered":"2024-04-21T23:06:31.414991+0000","last_clean":"2024-04-21T23:06:31.414991+0000","last_became_active":"2024-04-21T23:06:30.270567+0000","last_became_peered":"2024-04-21T23:06:30.270567+0000","last_unstale":"2024-04-21T23:06:31.414991+0000","last_undegraded":"2024-04-21T23:06:31.414991+0000","last_fullsized":"2024-04-21T23:06:31.414991+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:43:34.908748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415901+0000","last_change":"2024-04-21T23:06:25.214590+0000","last_active":"2024-04-21T23:06:31.415901+0000","last_peered":"2024-04-21T23:06:31.415901+0000","last_clean":"2024-04-21T23:06:31.415901+0000","last_became_active":"2024-04-21T23:06:25.214087+0000","last_became_peered":"2024-04-21T23:06:25.214087+0000","last_unstale":"2024-04-21T23:06:31.415901+0000","last_undegraded":"2024-04-21T23:06:31.415901+0000","last_fullsized":"2024-04-21T23:06:31.415901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:49:51.294478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.263882+0000","last_change":"2024-04-21T23:06:25.988197+0000","last_active":"2024-04-21T23:06:31.263882+0000","last_peered":"2024-04-21T23:06:31.263882+0000","last_clean":"2024-04-21T23:06:31.263882+0000","last_became_active":"2024-04-21T23:06:25.987872+0000","last_became_peered":"2024-04-21T23:06:25.987872+0000","last_unstale":"2024-04-21T23:06:31.263882+0000","last_undegraded":"2024-04-21T23:06:31.263882+0000","last_fullsized":"2024-04-21T23:06:31.263882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:25.389020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414666+0000","last_change":"2024-04-21T23:06:30.265262+0000","last_active":"2024-04-21T23:06:31.414666+0000","last_peered":"2024-04-21T23:06:31.414666+0000","last_clean":"2024-04-21T23:06:31.414666+0000","last_became_active":"2024-04-21T23:06:30.264781+0000","last_became_peered":"2024-04-21T23:06:30.264781+0000","last_unstale":"2024-04-21T23:06:31.414666+0000","last_undegraded":"2024-04-21T23:06:31.414666+0000","last_fullsized":"2024-04-21T23:06:31.414666+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:03:30.756792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414876+0000","last_change":"2024-04-21T23:06:25.205070+0000","last_active":"2024-04-21T23:06:31.414876+0000","last_peered":"2024-04-21T23:06:31.414876+0000","last_clean":"2024-04-21T23:06:31.414876+0000","last_became_active":"2024-04-21T23:06:25.204737+0000","last_became_peered":"2024-04-21T23:06:25.204737+0000","last_unstale":"2024-04-21T23:06:31.414876+0000","last_undegraded":"2024-04-21T23:06:31.414876+0000","last_fullsized":"2024-04-21T23:06:31.414876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:37.014534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.260026+0000","last_change":"2024-04-21T23:06:25.986584+0000","last_active":"2024-04-21T23:06:31.260026+0000","last_peered":"2024-04-21T23:06:31.260026+0000","last_clean":"2024-04-21T23:06:31.260026+0000","last_became_active":"2024-04-21T23:06:25.986268+0000","last_became_peered":"2024-04-21T23:06:25.986268+0000","last_unstale":"2024-04-21T23:06:31.260026+0000","last_undegraded":"2024-04-21T23:06:31.260026+0000","last_fullsized":"2024-04-21T23:06:31.260026+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:19:43.864279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415555+0000","last_change":"2024-04-21T23:06:30.261900+0000","last_active":"2024-04-21T23:06:31.415555+0000","last_peered":"2024-04-21T23:06:31.415555+0000","last_clean":"2024-04-21T23:06:31.415555+0000","last_became_active":"2024-04-21T23:06:30.261444+0000","last_became_peered":"2024-04-21T23:06:30.261444+0000","last_unstale":"2024-04-21T23:06:31.415555+0000","last_undegraded":"2024-04-21T23:06:31.415555+0000","last_fullsized":"2024-04-21T23:06:31.415555+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:10:40.810555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259706+0000","last_change":"2024-04-21T23:06:25.210702+0000","last_active":"2024-04-21T23:06:31.259706+0000","last_peered":"2024-04-21T23:06:31.259706+0000","last_clean":"2024-04-21T23:06:31.259706+0000","last_became_active":"2024-04-21T23:06:25.210383+0000","last_became_peered":"2024-04-21T23:06:25.210383+0000","last_unstale":"2024-04-21T23:06:31.259706+0000","last_undegraded":"2024-04-21T23:06:31.259706+0000","last_fullsized":"2024-04-21T23:06:31.259706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:07:45.733370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415353+0000","last_change":"2024-04-21T23:06:25.975999+0000","last_active":"2024-04-21T23:06:31.415353+0000","last_peered":"2024-04-21T23:06:31.415353+0000","last_clean":"2024-04-21T23:06:31.415353+0000","last_became_active":"2024-04-21T23:06:25.975677+0000","last_became_peered":"2024-04-21T23:06:25.975677+0000","last_unstale":"2024-04-21T23:06:31.415353+0000","last_undegraded":"2024-04-21T23:06:31.415353+0000","last_fullsized":"2024-04-21T23:06:31.415353+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:52:03.177248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415728+0000","last_change":"2024-04-21T23:06:30.261234+0000","last_active":"2024-04-21T23:06:31.415728+0000","last_peered":"2024-04-21T23:06:31.415728+0000","last_clean":"2024-04-21T23:06:31.415728+0000","last_became_active":"2024-04-21T23:06:30.260770+0000","last_became_peered":"2024-04-21T23:06:30.260770+0000","last_unstale":"2024-04-21T23:06:31.415728+0000","last_undegraded":"2024-04-21T23:06:31.415728+0000","last_fullsized":"2024-04-21T23:06:31.415728+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:51:22.460253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416378+0000","last_change":"2024-04-21T23:06:25.214893+0000","last_active":"2024-04-21T23:06:31.416378+0000","last_peered":"2024-04-21T23:06:31.416378+0000","last_clean":"2024-04-21T23:06:31.416378+0000","last_became_active":"2024-04-21T23:06:25.212359+0000","last_became_peered":"2024-04-21T23:06:25.212359+0000","last_unstale":"2024-04-21T23:06:31.416378+0000","last_undegraded":"2024-04-21T23:06:31.416378+0000","last_fullsized":"2024-04-21T23:06:31.416378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:18:03.278875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259801+0000","last_change":"2024-04-21T23:06:25.986849+0000","last_active":"2024-04-21T23:06:31.259801+0000","last_peered":"2024-04-21T23:06:31.259801+0000","last_clean":"2024-04-21T23:06:31.259801+0000","last_became_active":"2024-04-21T23:06:25.986334+0000","last_became_peered":"2024-04-21T23:06:25.986334+0000","last_unstale":"2024-04-21T23:06:31.259801+0000","last_undegraded":"2024-04-21T23:06:31.259801+0000","last_fullsized":"2024-04-21T23:06:31.259801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:57:42.195629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416004+0000","last_change":"2024-04-21T23:06:30.270812+0000","last_active":"2024-04-21T23:06:31.416004+0000","last_peered":"2024-04-21T23:06:31.416004+0000","last_clean":"2024-04-21T23:06:31.416004+0000","last_became_active":"2024-04-21T23:06:30.270370+0000","last_became_peered":"2024-04-21T23:06:30.270370+0000","last_unstale":"2024-04-21T23:06:31.416004+0000","last_undegraded":"2024-04-21T23:06:31.416004+0000","last_fullsized":"2024-04-21T23:06:31.416004+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:53.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.258606+0000","last_change":"2024-04-21T23:06:25.201603+0000","last_active":"2024-04-21T23:06:31.258606+0000","last_peered":"2024-04-21T23:06:31.258606+0000","last_clean":"2024-04-21T23:06:31.258606+0000","last_became_active":"2024-04-21T23:06:25.201265+0000","last_became_peered":"2024-04-21T23:06:25.201265+0000","last_unstale":"2024-04-21T23:06:31.258606+0000","last_undegraded":"2024-04-21T23:06:31.258606+0000","last_fullsized":"2024-04-21T23:06:31.258606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:23:19.134215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415130+0000","last_change":"2024-04-21T23:06:25.982420+0000","last_active":"2024-04-21T23:06:31.415130+0000","last_peered":"2024-04-21T23:06:31.415130+0000","last_clean":"2024-04-21T23:06:31.415130+0000","last_became_active":"2024-04-21T23:06:25.981867+0000","last_became_peered":"2024-04-21T23:06:25.981867+0000","last_unstale":"2024-04-21T23:06:31.415130+0000","last_undegraded":"2024-04-21T23:06:31.415130+0000","last_fullsized":"2024-04-21T23:06:31.415130+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:10:29.433243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414491+0000","last_change":"2024-04-21T23:06:30.273907+0000","last_active":"2024-04-21T23:06:31.414491+0000","last_peered":"2024-04-21T23:06:31.414491+0000","last_clean":"2024-04-21T23:06:31.414491+0000","last_became_active":"2024-04-21T23:06:30.273472+0000","last_became_peered":"2024-04-21T23:06:30.273472+0000","last_unstale":"2024-04-21T23:06:31.414491+0000","last_undegraded":"2024-04-21T23:06:31.414491+0000","last_fullsized":"2024-04-21T23:06:31.414491+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:58:50.914940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259369+0000","last_change":"2024-04-21T23:06:25.211117+0000","last_active":"2024-04-21T23:06:31.259369+0000","last_peered":"2024-04-21T23:06:31.259369+0000","last_clean":"2024-04-21T23:06:31.259369+0000","last_became_active":"2024-04-21T23:06:25.210626+0000","last_became_peered":"2024-04-21T23:06:25.210626+0000","last_unstale":"2024-04-21T23:06:31.259369+0000","last_undegraded":"2024-04-21T23:06:31.259369+0000","last_fullsized":"2024-04-21T23:06:31.259369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:56:39.917072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414670+0000","last_change":"2024-04-21T23:06:25.980537+0000","last_active":"2024-04-21T23:06:31.414670+0000","last_peered":"2024-04-21T23:06:31.414670+0000","last_clean":"2024-04-21T23:06:31.414670+0000","last_became_active":"2024-04-21T23:06:25.980188+0000","last_became_peered":"2024-04-21T23:06:25.980188+0000","last_unstale":"2024-04-21T23:06:31.414670+0000","last_undegraded":"2024-04-21T23:06:31.414670+0000","last_fullsized":"2024-04-21T23:06:31.414670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:58:02.828632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415055+0000","last_change":"2024-04-21T23:06:30.263565+0000","last_active":"2024-04-21T23:06:31.415055+0000","last_peered":"2024-04-21T23:06:31.415055+0000","last_clean":"2024-04-21T23:06:31.415055+0000","last_became_active":"2024-04-21T23:06:30.263092+0000","last_became_peered":"2024-04-21T23:06:30.263092+0000","last_unstale":"2024-04-21T23:06:31.415055+0000","last_undegraded":"2024-04-21T23:06:31.415055+0000","last_fullsized":"2024-04-21T23:06:31.415055+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:58:25.709551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416668+0000","last_change":"2024-04-21T23:06:25.212567+0000","last_active":"2024-04-21T23:06:31.416668+0000","last_peered":"2024-04-21T23:06:31.416668+0000","last_clean":"2024-04-21T23:06:31.416668+0000","last_became_active":"2024-04-21T23:06:25.210593+0000","last_became_peered":"2024-04-21T23:06:25.210593+0000","last_unstale":"2024-04-21T23:06:31.416668+0000","last_undegraded":"2024-04-21T23:06:31.416668+0000","last_fullsized":"2024-04-21T23:06:31.416668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:53:33.552742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.264226+0000","last_change":"2024-04-21T23:06:25.989525+0000","last_active":"2024-04-21T23:06:31.264226+0000","last_peered":"2024-04-21T23:06:31.264226+0000","last_clean":"2024-04-21T23:06:31.264226+0000","last_became_active":"2024-04-21T23:06:25.989130+0000","last_became_peered":"2024-04-21T23:06:25.989130+0000","last_unstale":"2024-04-21T23:06:31.264226+0000","last_undegraded":"2024-04-21T23:06:31.264226+0000","last_fullsized":"2024-04-21T23:06:31.264226+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:19:31.000530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415581+0000","last_change":"2024-04-21T23:06:30.269309+0000","last_active":"2024-04-21T23:06:31.415581+0000","last_peered":"2024-04-21T23:06:31.415581+0000","last_clean":"2024-04-21T23:06:31.415581+0000","last_became_active":"2024-04-21T23:06:30.268903+0000","last_became_peered":"2024-04-21T23:06:30.268903+0000","last_unstale":"2024-04-21T23:06:31.415581+0000","last_undegraded":"2024-04-21T23:06:31.415581+0000","last_fullsized":"2024-04-21T23:06:31.415581+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:52:16.043870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414857+0000","last_change":"2024-04-21T23:06:25.213938+0000","last_active":"2024-04-21T23:06:31.414857+0000","last_peered":"2024-04-21T23:06:31.414857+0000","last_clean":"2024-04-21T23:06:31.414857+0000","last_became_active":"2024-04-21T23:06:25.213242+0000","last_became_peered":"2024-04-21T23:06:25.213242+0000","last_unstale":"2024-04-21T23:06:31.414857+0000","last_undegraded":"2024-04-21T23:06:31.414857+0000","last_fullsized":"2024-04-21T23:06:31.414857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:27.679122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415390+0000","last_change":"2024-04-21T23:06:25.994061+0000","last_active":"2024-04-21T23:06:31.415390+0000","last_peered":"2024-04-21T23:06:31.415390+0000","last_clean":"2024-04-21T23:06:31.415390+0000","last_became_active":"2024-04-21T23:06:25.993714+0000","last_became_peered":"2024-04-21T23:06:25.993714+0000","last_unstale":"2024-04-21T23:06:31.415390+0000","last_undegraded":"2024-04-21T23:06:31.415390+0000","last_fullsized":"2024-04-21T23:06:31.415390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:54:45.571815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415389+0000","last_change":"2024-04-21T23:06:30.272206+0000","last_active":"2024-04-21T23:06:31.415389+0000","last_peered":"2024-04-21T23:06:31.415389+0000","last_clean":"2024-04-21T23:06:31.415389+0000","last_became_active":"2024-04-21T23:06:30.271389+0000","last_became_peered":"2024-04-21T23:06:30.271389+0000","last_unstale":"2024-04-21T23:06:31.415389+0000","last_undegraded":"2024-04-21T23:06:31.415389+0000","last_fullsized":"2024-04-21T23:06:31.415389+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:54:47.996166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259925+0000","last_change":"2024-04-21T23:06:25.217655+0000","last_active":"2024-04-21T23:06:31.259925+0000","last_peered":"2024-04-21T23:06:31.259925+0000","last_clean":"2024-04-21T23:06:31.259925+0000","last_became_active":"2024-04-21T23:06:25.217337+0000","last_became_peered":"2024-04-21T23:06:25.217337+0000","last_unstale":"2024-04-21T23:06:31.259925+0000","last_undegraded":"2024-04-21T23:06:31.259925+0000","last_fullsized":"2024-04-21T23:06:31.259925+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:29:28.690841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.264198+0000","last_change":"2024-04-21T23:06:25.995757+0000","last_active":"2024-04-21T23:06:31.264198+0000","last_peered":"2024-04-21T23:06:31.264198+0000","last_clean":"2024-04-21T23:06:31.264198+0000","last_became_active":"2024-04-21T23:06:25.995443+0000","last_became_peered":"2024-04-21T23:06:25.995443+0000","last_unstale":"2024-04-21T23:06:31.264198+0000","last_undegraded":"2024-04-21T23:06:31.264198+0000","last_fullsized":"2024-04-21T23:06:31.264198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:40.359633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416030+0000","last_change":"2024-04-21T23:06:30.266441+0000","last_active":"2024-04-21T23:06:31.416030+0000","last_peered":"2024-04-21T23:06:31.416030+0000","last_clean":"2024-04-21T23:06:31.416030+0000","last_became_active":"2024-04-21T23:06:30.264065+0000","last_became_peered":"2024-04-21T23:06:30.264065+0000","last_unstale":"2024-04-21T23:06:31.416030+0000","last_undegraded":"2024-04-21T23:06:31.416030+0000","last_fullsized":"2024-04-21T23:06:31.416030+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:53:22.548096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'10","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:41.846239+0000","last_change":"2024-04-21T23:06:25.213919+0000","last_active":"2024-04-21T23:06:41.846239+0000","last_peered":"2024-04-21T23:06:41.846239+0000","last_clean":"2024-04-21T23:06:41.846239+0000","last_became_active":"2024-04-21T23:06:25.212244+0000","last_became_peered":"2024-04-21T23:06:25.212244+0000","last_unstale":"2024-04-21T23:06:41.846239+0000","last_undegraded":"2024-04-21T23:06:41.846239+0000","last_fullsized":"2024-04-21T23:06:41.846239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:17:25.287718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415644+0000","last_change":"2024-04-21T23:06:25.993666+0000","last_active":"2024-04-21T23:06:31.415644+0000","last_peered":"2024-04-21T23:06:31.415644+0000","last_clean":"2024-04-21T23:06:31.415644+0000","last_became_active":"2024-04-21T23:06:25.993330+0000","last_became_peered":"2024-04-21T23:06:25.993330+0000","last_unstale":"2024-04-21T23:06:31.415644+0000","last_undegraded":"2024-04-21T23:06:31.415644+0000","last_fullsized":"2024-04-21T23:06:31.415644+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:11:00.109782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.260359+0000","last_change":"2024-04-21T23:06:30.265807+0000","last_active":"2024-04-21T23:06:31.260359+0000","last_peered":"2024-04-21T23:06:31.260359+0000","last_clean":"2024-04-21T23:06:31.260359+0000","last_became_active":"2024-04-21T23:06:30.265116+0000","last_became_peered":"2024-04-21T23:06:30.265116+0000","last_unstale":"2024-04-21T23:06:31.260359+0000","last_undegraded":"2024-04-21T23:06:31.260359+0000","last_fullsized":"2024-04-21T23:06:31.260359+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:14:04.178281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415461+0000","last_change":"2024-04-21T23:06:25.208319+0000","last_active":"2024-04-21T23:06:31.415461+0000","last_peered":"2024-04-21T23:06:31.415461+0000","last_clean":"2024-04-21T23:06:31.415461+0000","last_became_active":"2024-04-21T23:06:25.208025+0000","last_became_peered":"2024-04-21T23:06:25.208025+0000","last_unstale":"2024-04-21T23:06:31.415461+0000","last_undegraded":"2024-04-21T23:06:31.415461+0000","last_fullsized":"2024-04-21T23:06:31.415461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:14.723170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.258707+0000","last_change":"2024-04-21T23:06:25.993604+0000","last_active":"2024-04-21T23:06:31.258707+0000","last_peered":"2024-04-21T23:06:31.258707+0000","last_clean":"2024-04-21T23:06:31.258707+0000","last_became_active":"2024-04-21T23:06:25.993217+0000","last_became_peered":"2024-04-21T23:06:25.993217+0000","last_unstale":"2024-04-21T23:06:31.258707+0000","last_undegraded":"2024-04-21T23:06:31.258707+0000","last_fullsized":"2024-04-21T23:06:31.258707+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:23:00.435688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414741+0000","last_change":"2024-04-21T23:06:30.270991+0000","last_active":"2024-04-21T23:06:31.414741+0000","last_peered":"2024-04-21T23:06:31.414741+0000","last_clean":"2024-04-21T23:06:31.414741+0000","last_became_active":"2024-04-21T23:06:30.270511+0000","last_became_peered":"2024-04-21T23:06:30.270511+0000","last_unstale":"2024-04-21T23:06:31.414741+0000","last_undegraded":"2024-04-21T23:06:31.414741+0000","last_fullsized":"2024-04-21T23:06:31.414741+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:04.265855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259788+0000","last_change":"2024-04-21T23:06:25.211787+0000","last_active":"2024-04-21T23:06:31.259788+0000","last_peered":"2024-04-21T23:06:31.259788+0000","last_clean":"2024-04-21T23:06:31.259788+0000","last_became_active":"2024-04-21T23:06:25.211473+0000","last_became_peered":"2024-04-21T23:06:25.211473+0000","last_unstale":"2024-04-21T23:06:31.259788+0000","last_undegraded":"2024-04-21T23:06:31.259788+0000","last_fullsized":"2024-04-21T23:06:31.259788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:58:51.364974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414912+0000","last_change":"2024-04-21T23:06:25.990681+0000","last_active":"2024-04-21T23:06:31.414912+0000","last_peered":"2024-04-21T23:06:31.414912+0000","last_clean":"2024-04-21T23:06:31.414912+0000","last_became_active":"2024-04-21T23:06:25.990094+0000","last_became_peered":"2024-04-21T23:06:25.990094+0000","last_unstale":"2024-04-21T23:06:31.414912+0000","last_undegraded":"2024-04-21T23:06:31.414912+0000","last_fullsized":"2024-04-21T23:06:31.414912+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:15:26.907912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414912+0000","last_change":"2024-04-21T23:06:30.271352+0000","last_active":"2024-04-21T23:06:31.414912+0000","last_peered":"2024-04-21T23:06:31.414912+0000","last_clean":"2024-04-21T23:06:31.414912+0000","last_became_active":"2024-04-21T23:06:30.270601+0000","last_became_peered":"2024-04-21T23:06:30.270601+0000","last_unstale":"2024-04-21T23:06:31.414912+0000","last_undegraded":"2024-04-21T23:06:31.414912+0000","last_fullsized":"2024-04-21T23:06:31.414912+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:37.266987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414696+0000","last_change":"2024-04-21T23:06:25.213288+0000","last_active":"2024-04-21T23:06:31.414696+0000","last_peered":"2024-04-21T23:06:31.414696+0000","last_clean":"2024-04-21T23:06:31.414696+0000","last_became_active":"2024-04-21T23:06:25.212936+0000","last_became_peered":"2024-04-21T23:06:25.212936+0000","last_unstale":"2024-04-21T23:06:31.414696+0000","last_undegraded":"2024-04-21T23:06:31.414696+0000","last_fullsized":"2024-04-21T23:06:31.414696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:35.303093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414833+0000","last_change":"2024-04-21T23:06:25.998485+0000","last_active":"2024-04-21T23:06:31.414833+0000","last_peered":"2024-04-21T23:06:31.414833+0000","last_clean":"2024-04-21T23:06:31.414833+0000","last_became_active":"2024-04-21T23:06:25.998184+0000","last_became_peered":"2024-04-21T23:06:25.998184+0000","last_unstale":"2024-04-21T23:06:31.414833+0000","last_undegraded":"2024-04-21T23:06:31.414833+0000","last_fullsized":"2024-04-21T23:06:31.414833+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:35:26.613636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259325+0000","last_change":"2024-04-21T23:06:30.270089+0000","last_active":"2024-04-21T23:06:31.259325+0000","last_peered":"2024-04-21T23:06:31.259325+0000","last_clean":"2024-04-21T23:06:31.259325+0000","last_became_active":"2024-04-21T23:06:30.269695+0000","last_became_peered":"2024-04-21T23:06:30.269695+0000","last_unstale":"2024-04-21T23:06:31.259325+0000","last_undegraded":"2024-04-21T23:06:31.259325+0000","last_fullsized":"2024-04-21T23:06:31.259325+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:42.718580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414466+0000","last_change":"2024-04-21T23:06:25.218247+0000","last_active":"2024-04-21T23:06:31.414466+0000","last_peered":"2024-04-21T23:06:31.414466+0000","last_clean":"2024-04-21T23:06:31.414466+0000","last_became_active":"2024-04-21T23:06:25.217924+0000","last_became_peered":"2024-04-21T23:06:25.217924+0000","last_unstale":"2024-04-21T23:06:31.414466+0000","last_undegraded":"2024-04-21T23:06:31.414466+0000","last_fullsized":"2024-04-21T23:06:31.414466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:27:12.155597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259384+0000","last_change":"2024-04-21T23:06:25.994109+0000","last_active":"2024-04-21T23:06:31.259384+0000","last_peered":"2024-04-21T23:06:31.259384+0000","last_clean":"2024-04-21T23:06:31.259384+0000","last_became_active":"2024-04-21T23:06:25.993785+0000","last_became_peered":"2024-04-21T23:06:25.993785+0000","last_unstale":"2024-04-21T23:06:31.259384+0000","last_undegraded":"2024-04-21T23:06:31.259384+0000","last_fullsized":"2024-04-21T23:06:31.259384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:12.629364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259595+0000","last_change":"2024-04-21T23:06:30.265444+0000","last_active":"2024-04-21T23:06:31.259595+0000","last_peered":"2024-04-21T23:06:31.259595+0000","last_clean":"2024-04-21T23:06:31.259595+0000","last_became_active":"2024-04-21T23:06:30.264971+0000","last_became_peered":"2024-04-21T23:06:30.264971+0000","last_unstale":"2024-04-21T23:06:31.259595+0000","last_undegraded":"2024-04-21T23:06:31.259595+0000","last_fullsized":"2024-04-21T23:06:31.259595+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:03.596631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416086+0000","last_change":"2024-04-21T23:06:25.205988+0000","last_active":"2024-04-21T23:06:31.416086+0000","last_peered":"2024-04-21T23:06:31.416086+0000","last_clean":"2024-04-21T23:06:31.416086+0000","last_became_active":"2024-04-21T23:06:25.205642+0000","last_became_peered":"2024-04-21T23:06:25.205642+0000","last_unstale":"2024-04-21T23:06:31.416086+0000","last_undegraded":"2024-04-21T23:06:31.416086+0000","last_fullsized":"2024-04-21T23:06:31.416086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:23:02.225147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415864+0000","last_change":"2024-04-21T23:06:25.990193+0000","last_active":"2024-04-21T23:06:31.415864+0000","last_peered":"2024-04-21T23:06:31.415864+0000","last_clean":"2024-04-21T23:06:31.415864+0000","last_became_active":"2024-04-21T23:06:25.989842+0000","last_became_peered":"2024-04-21T23:06:25.989842+0000","last_unstale":"2024-04-21T23:06:31.415864+0000","last_undegraded":"2024-04-21T23:06:31.415864+0000","last_fullsized":"2024-04-21T23:06:31.415864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:54:53.421257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415295+0000","last_change":"2024-04-21T23:06:30.405155+0000","last_active":"2024-04-21T23:06:31.415295+0000","last_peered":"2024-04-21T23:06:31.415295+0000","last_clean":"2024-04-21T23:06:31.415295+0000","last_became_active":"2024-04-21T23:06:30.404657+0000","last_became_peered":"2024-04-21T23:06:30.404657+0000","last_unstale":"2024-04-21T23:06:31.415295+0000","last_undegraded":"2024-04-21T23:06:31.415295+0000","last_fullsized":"2024-04-21T23:06:31.415295+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:38:22.286830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414890+0000","last_change":"2024-04-21T23:06:25.218208+0000","last_active":"2024-04-21T23:06:31.414890+0000","last_peered":"2024-04-21T23:06:31.414890+0000","last_clean":"2024-04-21T23:06:31.414890+0000","last_became_active":"2024-04-21T23:06:25.217889+0000","last_became_peered":"2024-04-21T23:06:25.217889+0000","last_unstale":"2024-04-21T23:06:31.414890+0000","last_undegraded":"2024-04-21T23:06:31.414890+0000","last_fullsized":"2024-04-21T23:06:31.414890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:00:08.775682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416345+0000","last_change":"2024-04-21T23:06:25.987615+0000","last_active":"2024-04-21T23:06:31.416345+0000","last_peered":"2024-04-21T23:06:31.416345+0000","last_clean":"2024-04-21T23:06:31.416345+0000","last_became_active":"2024-04-21T23:06:25.987305+0000","last_became_peered":"2024-04-21T23:06:25.987305+0000","last_unstale":"2024-04-21T23:06:31.416345+0000","last_undegraded":"2024-04-21T23:06:31.416345+0000","last_fullsized":"2024-04-21T23:06:31.416345+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:23:49.386998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259112+0000","last_change":"2024-04-21T23:06:30.268157+0000","last_active":"2024-04-21T23:06:31.259112+0000","last_peered":"2024-04-21T23:06:31.259112+0000","last_clean":"2024-04-21T23:06:31.259112+0000","last_became_active":"2024-04-21T23:06:30.267732+0000","last_became_peered":"2024-04-21T23:06:30.267732+0000","last_unstale":"2024-04-21T23:06:31.259112+0000","last_undegraded":"2024-04-21T23:06:31.259112+0000","last_fullsized":"2024-04-21T23:06:31.259112+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:29:24.250271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415607+0000","last_change":"2024-04-21T23:06:25.215401+0000","last_active":"2024-04-21T23:06:31.415607+0000","last_peered":"2024-04-21T23:06:31.415607+0000","last_clean":"2024-04-21T23:06:31.415607+0000","last_became_active":"2024-04-21T23:06:25.215067+0000","last_became_peered":"2024-04-21T23:06:25.215067+0000","last_unstale":"2024-04-21T23:06:31.415607+0000","last_undegraded":"2024-04-21T23:06:31.415607+0000","last_fullsized":"2024-04-21T23:06:31.415607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:16:58.977797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416030+0000","last_change":"2024-04-21T23:06:25.996421+0000","last_active":"2024-04-21T23:06:31.416030+0000","last_peered":"2024-04-21T23:06:31.416030+0000","last_clean":"2024-04-21T23:06:31.416030+0000","last_became_active":"2024-04-21T23:06:25.996102+0000","last_became_peered":"2024-04-21T23:06:25.996102+0000","last_unstale":"2024-04-21T23:06:31.416030+0000","last_undegraded":"2024-04-21T23:06:31.416030+0000","last_fullsized":"2024-04-21T23:06:31.416030+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:06:42.630182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259254+0000","last_change":"2024-04-21T23:06:30.404509+0000","last_active":"2024-04-21T23:06:31.259254+0000","last_peered":"2024-04-21T23:06:31.259254+0000","last_clean":"2024-04-21T23:06:31.259254+0000","last_became_active":"2024-04-21T23:06:30.403987+0000","last_became_peered":"2024-04-21T23:06:30.403987+0000","last_unstale":"2024-04-21T23:06:31.259254+0000","last_undegraded":"2024-04-21T23:06:31.259254+0000","last_fullsized":"2024-04-21T23:06:31.259254+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:12:28.412117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259536+0000","last_change":"2024-04-21T23:06:25.994076+0000","last_active":"2024-04-21T23:06:31.259536+0000","last_peered":"2024-04-21T23:06:31.259536+0000","last_clean":"2024-04-21T23:06:31.259536+0000","last_became_active":"2024-04-21T23:06:25.993747+0000","last_became_peered":"2024-04-21T23:06:25.993747+0000","last_unstale":"2024-04-21T23:06:31.259536+0000","last_undegraded":"2024-04-21T23:06:31.259536+0000","last_fullsized":"2024-04-21T23:06:31.259536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:17.771165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.258484+0000","last_change":"2024-04-21T23:06:25.209756+0000","last_active":"2024-04-21T23:06:31.258484+0000","last_peered":"2024-04-21T23:06:31.258484+0000","last_clean":"2024-04-21T23:06:31.258484+0000","last_became_active":"2024-04-21T23:06:25.209445+0000","last_became_peered":"2024-04-21T23:06:25.209445+0000","last_unstale":"2024-04-21T23:06:31.258484+0000","last_undegraded":"2024-04-21T23:06:31.258484+0000","last_fullsized":"2024-04-21T23:06:31.258484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:58:49.453047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415301+0000","last_change":"2024-04-21T23:06:30.261565+0000","last_active":"2024-04-21T23:06:31.415301+0000","last_peered":"2024-04-21T23:06:31.415301+0000","last_clean":"2024-04-21T23:06:31.415301+0000","last_became_active":"2024-04-21T23:06:30.261095+0000","last_became_peered":"2024-04-21T23:06:30.261095+0000","last_unstale":"2024-04-21T23:06:31.415301+0000","last_undegraded":"2024-04-21T23:06:31.415301+0000","last_fullsized":"2024-04-21T23:06:31.415301+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:37:23.174152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416085+0000","last_change":"2024-04-21T23:06:25.981402+0000","last_active":"2024-04-21T23:06:31.416085+0000","last_peered":"2024-04-21T23:06:31.416085+0000","last_clean":"2024-04-21T23:06:31.416085+0000","last_became_active":"2024-04-21T23:06:25.981062+0000","last_became_peered":"2024-04-21T23:06:25.981062+0000","last_unstale":"2024-04-21T23:06:31.416085+0000","last_undegraded":"2024-04-21T23:06:31.416085+0000","last_fullsized":"2024-04-21T23:06:31.416085+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:32:32.556123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415144+0000","last_change":"2024-04-21T23:06:25.208968+0000","last_active":"2024-04-21T23:06:31.415144+0000","last_peered":"2024-04-21T23:06:31.415144+0000","last_clean":"2024-04-21T23:06:31.415144+0000","last_became_active":"2024-04-21T23:06:25.208606+0000","last_became_peered":"2024-04-21T23:06:25.208606+0000","last_unstale":"2024-04-21T23:06:31.415144+0000","last_undegraded":"2024-04-21T23:06:31.415144+0000","last_fullsized":"2024-04-21T23:06:31.415144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:31:46.147119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":358,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414934+0000","last_change":"2024-04-21T23:05:13.077846+0000","last_active":"2024-04-21T23:06:31.414934+0000","last_peered":"2024-04-21T23:06:31.414934+0000","last_clean":"2024-04-21T23:06:31.414934+0000","last_became_active":"2024-04-21T23:05:13.077436+0000","last_became_peered":"2024-04-21T23:05:13.077436+0000","last_unstale":"2024-04-21T23:06:31.414934+0000","last_undegraded":"2024-04-21T23:06:31.414934+0000","last_fullsized":"2024-04-21T23:06:31.414934+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:01:47.619212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:01:47.619212+0000","last_clean_scrub_stamp":"2024-04-21T23:01:47.619212+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:12:09.826839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.260038+0000","last_change":"2024-04-21T23:06:30.405121+0000","last_active":"2024-04-21T23:06:31.260038+0000","last_peered":"2024-04-21T23:06:31.260038+0000","last_clean":"2024-04-21T23:06:31.260038+0000","last_became_active":"2024-04-21T23:06:30.404612+0000","last_became_peered":"2024-04-21T23:06:30.404612+0000","last_unstale":"2024-04-21T23:06:31.260038+0000","last_undegraded":"2024-04-21T23:06:31.260038+0000","last_fullsized":"2024-04-21T23:06:31.260038+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:59:01.737337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.258456+0000","last_change":"2024-04-21T23:06:25.997879+0000","last_active":"2024-04-21T23:06:31.258456+0000","last_peered":"2024-04-21T23:06:31.258456+0000","last_clean":"2024-04-21T23:06:31.258456+0000","last_became_active":"2024-04-21T23:06:25.997507+0000","last_became_peered":"2024-04-21T23:06:25.997507+0000","last_unstale":"2024-04-21T23:06:31.258456+0000","last_undegraded":"2024-04-21T23:06:31.258456+0000","last_fullsized":"2024-04-21T23:06:31.258456+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:13:16.044456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414717+0000","last_change":"2024-04-21T23:06:25.209714+0000","last_active":"2024-04-21T23:06:31.414717+0000","last_peered":"2024-04-21T23:06:31.414717+0000","last_clean":"2024-04-21T23:06:31.414717+0000","last_became_active":"2024-04-21T23:06:25.209373+0000","last_became_peered":"2024-04-21T23:06:25.209373+0000","last_unstale":"2024-04-21T23:06:31.414717+0000","last_undegraded":"2024-04-21T23:06:31.414717+0000","last_fullsized":"2024-04-21T23:06:31.414717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:21.654143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415616+0000","last_change":"2024-04-21T23:06:30.264873+0000","last_active":"2024-04-21T23:06:31.415616+0000","last_peered":"2024-04-21T23:06:31.415616+0000","last_clean":"2024-04-21T23:06:31.415616+0000","last_became_active":"2024-04-21T23:06:30.264318+0000","last_became_peered":"2024-04-21T23:06:30.264318+0000","last_unstale":"2024-04-21T23:06:31.415616+0000","last_undegraded":"2024-04-21T23:06:31.415616+0000","last_fullsized":"2024-04-21T23:06:31.415616+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:01:22.558782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415784+0000","last_change":"2024-04-21T23:06:25.218767+0000","last_active":"2024-04-21T23:06:31.415784+0000","last_peered":"2024-04-21T23:06:31.415784+0000","last_clean":"2024-04-21T23:06:31.415784+0000","last_became_active":"2024-04-21T23:06:25.218278+0000","last_became_peered":"2024-04-21T23:06:25.218278+0000","last_unstale":"2024-04-21T23:06:31.415784+0000","last_undegraded":"2024-04-21T23:06:31.415784+0000","last_fullsized":"2024-04-21T23:06:31.415784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:03:05.063515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259189+0000","last_change":"2024-04-21T23:06:25.968551+0000","last_active":"2024-04-21T23:06:31.259189+0000","last_peered":"2024-04-21T23:06:31.259189+0000","last_clean":"2024-04-21T23:06:31.259189+0000","last_became_active":"2024-04-21T23:06:25.968167+0000","last_became_peered":"2024-04-21T23:06:25.968167+0000","last_unstale":"2024-04-21T23:06:31.259189+0000","last_undegraded":"2024-04-21T23:06:31.259189+0000","last_fullsized":"2024-04-21T23:06:31.259189+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:08.920677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415074+0000","last_change":"2024-04-21T23:06:30.404811+0000","last_active":"2024-04-21T23:06:31.415074+0000","last_peered":"2024-04-21T23:06:31.415074+0000","last_clean":"2024-04-21T23:06:31.415074+0000","last_became_active":"2024-04-21T23:06:30.404375+0000","last_became_peered":"2024-04-21T23:06:30.404375+0000","last_unstale":"2024-04-21T23:06:31.415074+0000","last_undegraded":"2024-04-21T23:06:31.415074+0000","last_fullsized":"2024-04-21T23:06:31.415074+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:10:15.688917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415668+0000","last_change":"2024-04-21T23:06:25.204371+0000","last_active":"2024-04-21T23:06:31.415668+0000","last_peered":"2024-04-21T23:06:31.415668+0000","last_clean":"2024-04-21T23:06:31.415668+0000","last_became_active":"2024-04-21T23:06:25.204025+0000","last_became_peered":"2024-04-21T23:06:25.204025+0000","last_unstale":"2024-04-21T23:06:31.415668+0000","last_undegraded":"2024-04-21T23:06:31.415668+0000","last_fullsized":"2024-04-21T23:06:31.415668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:34:31.071013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416396+0000","last_change":"2024-04-21T23:06:25.990800+0000","last_active":"2024-04-21T23:06:31.416396+0000","last_peered":"2024-04-21T23:06:31.416396+0000","last_clean":"2024-04-21T23:06:31.416396+0000","last_became_active":"2024-04-21T23:06:25.990477+0000","last_became_peered":"2024-04-21T23:06:25.990477+0000","last_unstale":"2024-04-21T23:06:31.416396+0000","last_undegraded":"2024-04-21T23:06:31.416396+0000","last_fullsized":"2024-04-21T23:06:31.416396+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:55:33.282303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415545+0000","last_change":"2024-04-21T23:06:30.260090+0000","last_active":"2024-04-21T23:06:31.415545+0000","last_peered":"2024-04-21T23:06:31.415545+0000","last_clean":"2024-04-21T23:06:31.415545+0000","last_became_active":"2024-04-21T23:06:30.259609+0000","last_became_peered":"2024-04-21T23:06:30.259609+0000","last_unstale":"2024-04-21T23:06:31.415545+0000","last_undegraded":"2024-04-21T23:06:31.415545+0000","last_fullsized":"2024-04-21T23:06:31.415545+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:11:17.933683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415346+0000","last_change":"2024-04-21T23:06:25.215446+0000","last_active":"2024-04-21T23:06:31.415346+0000","last_peered":"2024-04-21T23:06:31.415346+0000","last_clean":"2024-04-21T23:06:31.415346+0000","last_became_active":"2024-04-21T23:06:25.214773+0000","last_became_peered":"2024-04-21T23:06:25.214773+0000","last_unstale":"2024-04-21T23:06:31.415346+0000","last_undegraded":"2024-04-21T23:06:31.415346+0000","last_fullsized":"2024-04-21T23:06:31.415346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:25:01.090699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414575+0000","last_change":"2024-04-21T23:06:25.986640+0000","last_active":"2024-04-21T23:06:31.414575+0000","last_peered":"2024-04-21T23:06:31.414575+0000","last_clean":"2024-04-21T23:06:31.414575+0000","last_became_active":"2024-04-21T23:06:25.986285+0000","last_became_peered":"2024-04-21T23:06:25.986285+0000","last_unstale":"2024-04-21T23:06:31.414575+0000","last_undegraded":"2024-04-21T23:06:31.414575+0000","last_fullsized":"2024-04-21T23:06:31.414575+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:01:05.498199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416147+0000","last_change":"2024-04-21T23:06:30.405165+0000","last_active":"2024-04-21T23:06:31.416147+0000","last_peered":"2024-04-21T23:06:31.416147+0000","last_clean":"2024-04-21T23:06:31.416147+0000","last_became_active":"2024-04-21T23:06:30.404622+0000","last_became_peered":"2024-04-21T23:06:30.404622+0000","last_unstale":"2024-04-21T23:06:31.416147+0000","last_undegraded":"2024-04-21T23:06:31.416147+0000","last_fullsized":"2024-04-21T23:06:31.416147+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:34:16.995792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.263538+0000","last_change":"2024-04-21T23:06:25.217298+0000","last_active":"2024-04-21T23:06:31.263538+0000","last_peered":"2024-04-21T23:06:31.263538+0000","last_clean":"2024-04-21T23:06:31.263538+0000","last_became_active":"2024-04-21T23:06:25.216965+0000","last_became_peered":"2024-04-21T23:06:25.216965+0000","last_unstale":"2024-04-21T23:06:31.263538+0000","last_undegraded":"2024-04-21T23:06:31.263538+0000","last_fullsized":"2024-04-21T23:06:31.263538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:56:38.543697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414523+0000","last_change":"2024-04-21T23:06:25.981400+0000","last_active":"2024-04-21T23:06:31.414523+0000","last_peered":"2024-04-21T23:06:31.414523+0000","last_clean":"2024-04-21T23:06:31.414523+0000","last_became_active":"2024-04-21T23:06:25.981118+0000","last_became_peered":"2024-04-21T23:06:25.981118+0000","last_unstale":"2024-04-21T23:06:31.414523+0000","last_undegraded":"2024-04-21T23:06:31.414523+0000","last_fullsized":"2024-04-21T23:06:31.414523+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:20:59.631062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415183+0000","last_change":"2024-04-21T23:06:30.270351+0000","last_active":"2024-04-21T23:06:31.415183+0000","last_peered":"2024-04-21T23:06:31.415183+0000","last_clean":"2024-04-21T23:06:31.415183+0000","last_became_active":"2024-04-21T23:06:30.269659+0000","last_became_peered":"2024-04-21T23:06:30.269659+0000","last_unstale":"2024-04-21T23:06:31.415183+0000","last_undegraded":"2024-04-21T23:06:31.415183+0000","last_fullsized":"2024-04-21T23:06:31.415183+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:19:14.132191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415265+0000","last_change":"2024-04-21T23:06:25.212787+0000","last_active":"2024-04-21T23:06:31.415265+0000","last_peered":"2024-04-21T23:06:31.415265+0000","last_clean":"2024-04-21T23:06:31.415265+0000","last_became_active":"2024-04-21T23:06:25.212423+0000","last_became_peered":"2024-04-21T23:06:25.212423+0000","last_unstale":"2024-04-21T23:06:31.415265+0000","last_undegraded":"2024-04-21T23:06:31.415265+0000","last_fullsized":"2024-04-21T23:06:31.415265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:45.875664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414465+0000","last_change":"2024-04-21T23:06:25.978205+0000","last_active":"2024-04-21T23:06:31.414465+0000","last_peered":"2024-04-21T23:06:31.414465+0000","last_clean":"2024-04-21T23:06:31.414465+0000","last_became_active":"2024-04-21T23:06:25.977876+0000","last_became_peered":"2024-04-21T23:06:25.977876+0000","last_unstale":"2024-04-21T23:06:31.414465+0000","last_undegraded":"2024-04-21T23:06:31.414465+0000","last_fullsized":"2024-04-21T23:06:31.414465+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:43:29.280255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415829+0000","last_change":"2024-04-21T23:06:30.271928+0000","last_active":"2024-04-21T23:06:31.415829+0000","last_peered":"2024-04-21T23:06:31.415829+0000","last_clean":"2024-04-21T23:06:31.415829+0000","last_became_active":"2024-04-21T23:06:30.271321+0000","last_became_peered":"2024-04-21T23:06:30.271321+0000","last_unstale":"2024-04-21T23:06:31.415829+0000","last_undegraded":"2024-04-21T23:06:31.415829+0000","last_fullsized":"2024-04-21T23:06:31.415829+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:22:03.343620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415140+0000","last_change":"2024-04-21T23:06:25.204175+0000","last_active":"2024-04-21T23:06:31.415140+0000","last_peered":"2024-04-21T23:06:31.415140+0000","last_clean":"2024-04-21T23:06:31.415140+0000","last_became_active":"2024-04-21T23:06:25.203826+0000","last_became_peered":"2024-04-21T23:06:25.203826+0000","last_unstale":"2024-04-21T23:06:31.415140+0000","last_undegraded":"2024-04-21T23:06:31.415140+0000","last_fullsized":"2024-04-21T23:06:31.415140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:28:53.402395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415671+0000","last_change":"2024-04-21T23:06:25.990559+0000","last_active":"2024-04-21T23:06:31.415671+0000","last_peered":"2024-04-21T23:06:31.415671+0000","last_clean":"2024-04-21T23:06:31.415671+0000","last_became_active":"2024-04-21T23:06:25.990239+0000","last_became_peered":"2024-04-21T23:06:25.990239+0000","last_unstale":"2024-04-21T23:06:31.415671+0000","last_undegraded":"2024-04-21T23:06:31.415671+0000","last_fullsized":"2024-04-21T23:06:31.415671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:49:26.891190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416117+0000","last_change":"2024-04-21T23:06:30.406584+0000","last_active":"2024-04-21T23:06:31.416117+0000","last_peered":"2024-04-21T23:06:31.416117+0000","last_clean":"2024-04-21T23:06:31.416117+0000","last_became_active":"2024-04-21T23:06:30.404576+0000","last_became_peered":"2024-04-21T23:06:30.404576+0000","last_unstale":"2024-04-21T23:06:31.416117+0000","last_undegraded":"2024-04-21T23:06:31.416117+0000","last_fullsized":"2024-04-21T23:06:31.416117+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:56.047199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414864+0000","last_change":"2024-04-21T23:06:25.214502+0000","last_active":"2024-04-21T23:06:31.414864+0000","last_peered":"2024-04-21T23:06:31.414864+0000","last_clean":"2024-04-21T23:06:31.414864+0000","last_became_active":"2024-04-21T23:06:25.214035+0000","last_became_peered":"2024-04-21T23:06:25.214035+0000","last_unstale":"2024-04-21T23:06:31.414864+0000","last_undegraded":"2024-04-21T23:06:31.414864+0000","last_fullsized":"2024-04-21T23:06:31.414864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:51:51.760402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415715+0000","last_change":"2024-04-21T23:06:25.988925+0000","last_active":"2024-04-21T23:06:31.415715+0000","last_peered":"2024-04-21T23:06:31.415715+0000","last_clean":"2024-04-21T23:06:31.415715+0000","last_became_active":"2024-04-21T23:06:25.988480+0000","last_became_peered":"2024-04-21T23:06:25.988480+0000","last_unstale":"2024-04-21T23:06:31.415715+0000","last_undegraded":"2024-04-21T23:06:31.415715+0000","last_fullsized":"2024-04-21T23:06:31.415715+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:32:02.867858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.263752+0000","last_change":"2024-04-21T23:06:30.260841+0000","last_active":"2024-04-21T23:06:31.263752+0000","last_peered":"2024-04-21T23:06:31.263752+0000","last_clean":"2024-04-21T23:06:31.263752+0000","last_became_active":"2024-04-21T23:06:30.260376+0000","last_became_peered":"2024-04-21T23:06:30.260376+0000","last_unstale":"2024-04-21T23:06:31.263752+0000","last_undegraded":"2024-04-21T23:06:31.263752+0000","last_fullsized":"2024-04-21T23:06:31.263752+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:24:57.429169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415872+0000","last_change":"2024-04-21T23:06:25.217920+0000","last_active":"2024-04-21T23:06:31.415872+0000","last_peered":"2024-04-21T23:06:31.415872+0000","last_clean":"2024-04-21T23:06:31.415872+0000","last_became_active":"2024-04-21T23:06:25.217379+0000","last_became_peered":"2024-04-21T23:06:25.217379+0000","last_unstale":"2024-04-21T23:06:31.415872+0000","last_undegraded":"2024-04-21T23:06:31.415872+0000","last_fullsized":"2024-04-21T23:06:31.415872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:09.570463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415629+0000","last_change":"2024-04-21T23:06:25.993533+0000","last_active":"2024-04-21T23:06:31.415629+0000","last_peered":"2024-04-21T23:06:31.415629+0000","last_clean":"2024-04-21T23:06:31.415629+0000","last_became_active":"2024-04-21T23:06:25.989739+0000","last_became_peered":"2024-04-21T23:06:25.989739+0000","last_unstale":"2024-04-21T23:06:31.415629+0000","last_undegraded":"2024-04-21T23:06:31.415629+0000","last_fullsized":"2024-04-21T23:06:31.415629+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:24:05.594026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414078+0000","last_change":"2024-04-21T23:06:30.276121+0000","last_active":"2024-04-21T23:06:31.414078+0000","last_peered":"2024-04-21T23:06:31.414078+0000","last_clean":"2024-04-21T23:06:31.414078+0000","last_became_active":"2024-04-21T23:06:30.275650+0000","last_became_peered":"2024-04-21T23:06:30.275650+0000","last_unstale":"2024-04-21T23:06:31.414078+0000","last_undegraded":"2024-04-21T23:06:31.414078+0000","last_fullsized":"2024-04-21T23:06:31.414078+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:00:19.559482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.260186+0000","last_change":"2024-04-21T23:06:25.207079+0000","last_active":"2024-04-21T23:06:31.260186+0000","last_peered":"2024-04-21T23:06:31.260186+0000","last_clean":"2024-04-21T23:06:31.260186+0000","last_became_active":"2024-04-21T23:06:25.206724+0000","last_became_peered":"2024-04-21T23:06:25.206724+0000","last_unstale":"2024-04-21T23:06:31.260186+0000","last_undegraded":"2024-04-21T23:06:31.260186+0000","last_fullsized":"2024-04-21T23:06:31.260186+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:27.221282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416154+0000","last_change":"2024-04-21T23:06:25.988144+0000","last_active":"2024-04-21T23:06:31.416154+0000","last_peered":"2024-04-21T23:06:31.416154+0000","last_clean":"2024-04-21T23:06:31.416154+0000","last_became_active":"2024-04-21T23:06:25.987813+0000","last_became_peered":"2024-04-21T23:06:25.987813+0000","last_unstale":"2024-04-21T23:06:31.416154+0000","last_undegraded":"2024-04-21T23:06:31.416154+0000","last_fullsized":"2024-04-21T23:06:31.416154+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:23.733514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259282+0000","last_change":"2024-04-21T23:06:30.261548+0000","last_active":"2024-04-21T23:06:31.259282+0000","last_peered":"2024-04-21T23:06:31.259282+0000","last_clean":"2024-04-21T23:06:31.259282+0000","last_became_active":"2024-04-21T23:06:30.261152+0000","last_became_peered":"2024-04-21T23:06:30.261152+0000","last_unstale":"2024-04-21T23:06:31.259282+0000","last_undegraded":"2024-04-21T23:06:31.259282+0000","last_fullsized":"2024-04-21T23:06:31.259282+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:00:16.644253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259147+0000","last_change":"2024-04-21T23:06:25.208749+0000","last_active":"2024-04-21T23:06:31.259147+0000","last_peered":"2024-04-21T23:06:31.259147+0000","last_clean":"2024-04-21T23:06:31.259147+0000","last_became_active":"2024-04-21T23:06:25.201511+0000","last_became_peered":"2024-04-21T23:06:25.201511+0000","last_unstale":"2024-04-21T23:06:31.259147+0000","last_undegraded":"2024-04-21T23:06:31.259147+0000","last_fullsized":"2024-04-21T23:06:31.259147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:41.928835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415249+0000","last_change":"2024-04-21T23:06:25.996041+0000","last_active":"2024-04-21T23:06:31.415249+0000","last_peered":"2024-04-21T23:06:31.415249+0000","last_clean":"2024-04-21T23:06:31.415249+0000","last_became_active":"2024-04-21T23:06:25.994482+0000","last_became_peered":"2024-04-21T23:06:25.994482+0000","last_unstale":"2024-04-21T23:06:31.415249+0000","last_undegraded":"2024-04-21T23:06:31.415249+0000","last_fullsized":"2024-04-21T23:06:31.415249+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:15:46.792618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414710+0000","last_change":"2024-04-21T23:06:25.974061+0000","last_active":"2024-04-21T23:06:31.414710+0000","last_peered":"2024-04-21T23:06:31.414710+0000","last_clean":"2024-04-21T23:06:31.414710+0000","last_became_active":"2024-04-21T23:06:25.973708+0000","last_became_peered":"2024-04-21T23:06:25.973708+0000","last_unstale":"2024-04-21T23:06:31.414710+0000","last_undegraded":"2024-04-21T23:06:31.414710+0000","last_fullsized":"2024-04-21T23:06:31.414710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:58.702947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415980+0000","last_change":"2024-04-21T23:06:25.988412+0000","last_active":"2024-04-21T23:06:31.415980+0000","last_peered":"2024-04-21T23:06:31.415980+0000","last_clean":"2024-04-21T23:06:31.415980+0000","last_became_active":"2024-04-21T23:06:25.988100+0000","last_became_peered":"2024-04-21T23:06:25.988100+0000","last_unstale":"2024-04-21T23:06:31.415980+0000","last_undegraded":"2024-04-21T23:06:31.415980+0000","last_fullsized":"2024-04-21T23:06:31.415980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:16:42.626123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415058+0000","last_change":"2024-04-21T23:06:25.980144+0000","last_active":"2024-04-21T23:06:31.415058+0000","last_peered":"2024-04-21T23:06:31.415058+0000","last_clean":"2024-04-21T23:06:31.415058+0000","last_became_active":"2024-04-21T23:06:25.979760+0000","last_became_peered":"2024-04-21T23:06:25.979760+0000","last_unstale":"2024-04-21T23:06:31.415058+0000","last_undegraded":"2024-04-21T23:06:31.415058+0000","last_fullsized":"2024-04-21T23:06:31.415058+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:14:32.496184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.414155+0000","last_change":"2024-04-21T23:06:25.994732+0000","last_active":"2024-04-21T23:06:31.414155+0000","last_peered":"2024-04-21T23:06:31.414155+0000","last_clean":"2024-04-21T23:06:31.414155+0000","last_became_active":"2024-04-21T23:06:25.994408+0000","last_became_peered":"2024-04-21T23:06:25.994408+0000","last_unstale":"2024-04-21T23:06:31.414155+0000","last_undegraded":"2024-04-21T23:06:31.414155+0000","last_fullsized":"2024-04-21T23:06:31.414155+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:28:59.778049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259238+0000","last_change":"2024-04-21T23:06:25.988439+0000","last_active":"2024-04-21T23:06:31.259238+0000","last_peered":"2024-04-21T23:06:31.259238+0000","last_clean":"2024-04-21T23:06:31.259238+0000","last_became_active":"2024-04-21T23:06:25.988068+0000","last_became_peered":"2024-04-21T23:06:25.988068+0000","last_unstale":"2024-04-21T23:06:31.259238+0000","last_undegraded":"2024-04-21T23:06:31.259238+0000","last_fullsized":"2024-04-21T23:06:31.259238+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:45:26.345083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415112+0000","last_change":"2024-04-21T23:06:25.973850+0000","last_active":"2024-04-21T23:06:31.415112+0000","last_peered":"2024-04-21T23:06:31.415112+0000","last_clean":"2024-04-21T23:06:31.415112+0000","last_became_active":"2024-04-21T23:06:25.973522+0000","last_became_peered":"2024-04-21T23:06:25.973522+0000","last_unstale":"2024-04-21T23:06:31.415112+0000","last_undegraded":"2024-04-21T23:06:31.415112+0000","last_fullsized":"2024-04-21T23:06:31.415112+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:43:58.417551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415377+0000","last_change":"2024-04-21T23:06:25.978114+0000","last_active":"2024-04-21T23:06:31.415377+0000","last_peered":"2024-04-21T23:06:31.415377+0000","last_clean":"2024-04-21T23:06:31.415377+0000","last_became_active":"2024-04-21T23:06:25.977115+0000","last_became_peered":"2024-04-21T23:06:25.977115+0000","last_unstale":"2024-04-21T23:06:31.415377+0000","last_undegraded":"2024-04-21T23:06:31.415377+0000","last_fullsized":"2024-04-21T23:06:31.415377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:26:46.756490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.416493+0000","last_change":"2024-04-21T23:06:25.994451+0000","last_active":"2024-04-21T23:06:31.416493+0000","last_peered":"2024-04-21T23:06:31.416493+0000","last_clean":"2024-04-21T23:06:31.416493+0000","last_became_active":"2024-04-21T23:06:25.994085+0000","last_became_peered":"2024-04-21T23:06:25.994085+0000","last_unstale":"2024-04-21T23:06:31.416493+0000","last_undegraded":"2024-04-21T23:06:31.416493+0000","last_fullsized":"2024-04-21T23:06:31.416493+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:53.652014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.259410+0000","last_change":"2024-04-21T23:06:25.982536+0000","last_active":"2024-04-21T23:06:31.259410+0000","last_peered":"2024-04-21T23:06:31.259410+0000","last_clean":"2024-04-21T23:06:31.259410+0000","last_became_active":"2024-04-21T23:06:25.981318+0000","last_became_peered":"2024-04-21T23:06:25.981318+0000","last_unstale":"2024-04-21T23:06:31.259410+0000","last_undegraded":"2024-04-21T23:06:31.259410+0000","last_fullsized":"2024-04-21T23:06:31.259410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:35.784679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415749+0000","last_change":"2024-04-21T23:06:25.981915+0000","last_active":"2024-04-21T23:06:31.415749+0000","last_peered":"2024-04-21T23:06:31.415749+0000","last_clean":"2024-04-21T23:06:31.415749+0000","last_became_active":"2024-04-21T23:06:25.981547+0000","last_became_peered":"2024-04-21T23:06:25.981547+0000","last_unstale":"2024-04-21T23:06:31.415749+0000","last_undegraded":"2024-04-21T23:06:31.415749+0000","last_fullsized":"2024-04-21T23:06:31.415749+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:03:11.154582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.263827+0000","last_change":"2024-04-21T23:06:25.991687+0000","last_active":"2024-04-21T23:06:31.263827+0000","last_peered":"2024-04-21T23:06:31.263827+0000","last_clean":"2024-04-21T23:06:31.263827+0000","last_became_active":"2024-04-21T23:06:25.991248+0000","last_became_peered":"2024-04-21T23:06:25.991248+0000","last_unstale":"2024-04-21T23:06:31.263827+0000","last_undegraded":"2024-04-21T23:06:31.263827+0000","last_fullsized":"2024-04-21T23:06:31.263827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:05:04.205660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.258784+0000","last_change":"2024-04-21T23:06:25.993658+0000","last_active":"2024-04-21T23:06:31.258784+0000","last_peered":"2024-04-21T23:06:31.258784+0000","last_clean":"2024-04-21T23:06:31.258784+0000","last_became_active":"2024-04-21T23:06:25.993298+0000","last_became_peered":"2024-04-21T23:06:25.993298+0000","last_unstale":"2024-04-21T23:06:31.258784+0000","last_undegraded":"2024-04-21T23:06:31.258784+0000","last_fullsized":"2024-04-21T23:06:31.258784+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:18:30.216961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-21T23:06:31.415341+0000","last_change":"2024-04-21T23:06:25.993849+0000","last_active":"2024-04-21T23:06:31.415341+0000","last_peered":"2024-04-21T23:06:31.415341+0000","last_clean":"2024-04-21T23:06:31.415341+0000","last_became_active":"2024-04-21T23:06:25.993501+0000","last_became_peered":"2024-04-21T23:06:25.993501+0000","last_unstale":"2024-04-21T23:06:31.415341+0000","last_undegraded":"2024-04-21T23:06:31.415341+0000","last_fullsized":"2024-04-21T23:06:31.415341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:40:10.461775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":51,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":37850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350994,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":995328,"data_stored":483870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612631,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32752,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712400,"statfs":{"total":95995035648,"available":95961497600,"internally_reserved":0,"allocated":1613824,"data_stored":1078274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":484452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070498,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1003520,"data_stored":484474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332133,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28584,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716568,"statfs":{"total":95995035648,"available":95965765632,"internally_reserved":0,"allocated":1593344,"data_stored":1074820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":483870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":487712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":483960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":511676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":483974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920821,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":999424,"data_stored":483892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28584,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716568,"statfs":{"total":95995035648,"available":95965765632,"internally_reserved":0,"allocated":1605632,"data_stored":1078516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":27806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T23:06:54.499 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-21T23:06:54.499 INFO:tasks.cephadm:Setting up client nodes... 2024-04-21T23:06:54.500 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-21T23:06:54.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:54 smithi055 ceph-mon[31117]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2024-04-21T23:06:54.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:54 smithi055 ceph-mon[31117]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:06:54.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:54 smithi055 ceph-mon[31117]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-21T23:06:54.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:54 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2866041990' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:54.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:54 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1978088580' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:54.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:54 smithi069 ceph-mon[31816]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 255 B/s wr, 0 op/s 2024-04-21T23:06:54.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:54 smithi069 ceph-mon[31816]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:06:54.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:54 smithi069 ceph-mon[31816]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-21T23:06:54.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:54 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2866041990' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:06:54.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:54 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1978088580' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:06:54.784 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:06:55.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:55 smithi044 ceph-mon[27934]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:06:55.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:55 smithi044 ceph-mon[27934]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:06:55.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:55 smithi069 ceph-mon[31816]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:06:55.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:55 smithi069 ceph-mon[31816]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:06:55.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:55 smithi055 ceph-mon[31117]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:06:55.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:55 smithi055 ceph-mon[31117]: from='client.14952 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:06:56.554 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:56 smithi044 ceph-mon[27934]: pgmap v306: 129 pgs: 129 active+clean; 617 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.7 KiB/s wr, 2 op/s 2024-04-21T23:06:56.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:56 smithi055 ceph-mon[31117]: pgmap v306: 129 pgs: 129 active+clean; 617 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.7 KiB/s wr, 2 op/s 2024-04-21T23:06:56.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:56 smithi069 ceph-mon[31816]: pgmap v306: 129 pgs: 129 active+clean; 617 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.7 KiB/s wr, 2 op/s 2024-04-21T23:06:56.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.811+0000 7fbf24253700 1 -- 172.21.15.44:0/2439316685 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf1c102da0 msgr2=0x7fbf1c105190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:56.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.811+0000 7fbf24253700 1 --2- 172.21.15.44:0/2439316685 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf1c102da0 0x7fbf1c105190 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbf18009990 tx=0x7fbf18009ca0 comp rx=0 tx=0).stop 2024-04-21T23:06:56.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.811+0000 7fbf24253700 1 -- 172.21.15.44:0/2439316685 shutdown_connections 2024-04-21T23:06:56.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.811+0000 7fbf24253700 1 --2- 172.21.15.44:0/2439316685 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbf1c108000 0x7fbf1c10a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:56.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.811+0000 7fbf24253700 1 --2- 172.21.15.44:0/2439316685 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf1c102da0 0x7fbf1c105190 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:56.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.811+0000 7fbf24253700 1 --2- 172.21.15.44:0/2439316685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf1c1056d0 0x7fbf1c107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:56.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.811+0000 7fbf24253700 1 -- 172.21.15.44:0/2439316685 >> 172.21.15.44:0/2439316685 conn(0x7fbf1c0fc750 msgr2=0x7fbf1c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:56.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.811+0000 7fbf24253700 1 -- 172.21.15.44:0/2439316685 shutdown_connections 2024-04-21T23:06:56.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.811+0000 7fbf24253700 1 -- 172.21.15.44:0/2439316685 wait complete. 2024-04-21T23:06:56.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.812+0000 7fbf24253700 1 Processor -- start 2024-04-21T23:06:56.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.812+0000 7fbf24253700 1 -- start start 2024-04-21T23:06:56.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.812+0000 7fbf24253700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf1c102da0 0x7fbf1c195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:56.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.812+0000 7fbf24253700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf1c1056d0 0x7fbf1c196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:56.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.812+0000 7fbf24253700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbf1c108000 0x7fbf1c1a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:56.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.812+0000 7fbf24253700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c10de90 con 0x7fbf1c102da0 2024-04-21T23:06:56.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.812+0000 7fbf24253700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c10dd10 con 0x7fbf1c108000 2024-04-21T23:06:56.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.812+0000 7fbf24253700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c10e010 con 0x7fbf1c1056d0 2024-04-21T23:06:56.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.812+0000 7fbf21fef700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf1c102da0 0x7fbf1c195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:56.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.812+0000 7fbf21fef700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf1c102da0 0x7fbf1c195fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50832/0 (socket says 172.21.15.44:50832) 2024-04-21T23:06:56.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.812+0000 7fbf21fef700 1 -- 172.21.15.44:0/320172975 learned_addr learned my addr 172.21.15.44:0/320172975 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:06:56.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.812+0000 7fbf217ee700 1 --2- 172.21.15.44:0/320172975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf1c1056d0 0x7fbf1c196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:56.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.812+0000 7fbf227f0700 1 --2- 172.21.15.44:0/320172975 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbf1c108000 0x7fbf1c1a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:56.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.813+0000 7fbf21fef700 1 -- 172.21.15.44:0/320172975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf1c1056d0 msgr2=0x7fbf1c196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:56.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.813+0000 7fbf21fef700 1 --2- 172.21.15.44:0/320172975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf1c1056d0 0x7fbf1c196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:56.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.813+0000 7fbf21fef700 1 -- 172.21.15.44:0/320172975 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbf1c108000 msgr2=0x7fbf1c1a61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:56.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.813+0000 7fbf21fef700 1 --2- 172.21.15.44:0/320172975 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbf1c108000 0x7fbf1c1a61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:56.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.813+0000 7fbf21fef700 1 -- 172.21.15.44:0/320172975 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf1c1a6700 con 0x7fbf1c102da0 2024-04-21T23:06:56.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.813+0000 7fbf227f0700 1 --2- 172.21.15.44:0/320172975 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbf1c108000 0x7fbf1c1a61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:06:56.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.813+0000 7fbf21fef700 1 --2- 172.21.15.44:0/320172975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf1c102da0 0x7fbf1c195fd0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbf18005910 tx=0x7fbf18009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:56.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.813+0000 7fbf12ffd700 1 -- 172.21.15.44:0/320172975 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf18027070 con 0x7fbf1c102da0 2024-04-21T23:06:56.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.813+0000 7fbf12ffd700 1 -- 172.21.15.44:0/320172975 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf18004150 con 0x7fbf1c102da0 2024-04-21T23:06:56.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.813+0000 7fbf24253700 1 -- 172.21.15.44:0/320172975 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf1c1a68d0 con 0x7fbf1c102da0 2024-04-21T23:06:56.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.813+0000 7fbf12ffd700 1 -- 172.21.15.44:0/320172975 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1800f460 con 0x7fbf1c102da0 2024-04-21T23:06:56.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.813+0000 7fbf24253700 1 -- 172.21.15.44:0/320172975 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf1c1a6d80 con 0x7fbf1c102da0 2024-04-21T23:06:56.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.814+0000 7fbf24253700 1 -- 172.21.15.44:0/320172975 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf1c0507a0 con 0x7fbf1c102da0 2024-04-21T23:06:56.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.816+0000 7fbf12ffd700 1 -- 172.21.15.44:0/320172975 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fbf1802a070 con 0x7fbf1c102da0 2024-04-21T23:06:56.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.817+0000 7fbf12ffd700 1 --2- 172.21.15.44:0/320172975 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbf080a2de0 0x7fbf080a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:06:56.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.817+0000 7fbf217ee700 1 --2- 172.21.15.44:0/320172975 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbf080a2de0 0x7fbf080a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:06:56.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.817+0000 7fbf12ffd700 1 -- 172.21.15.44:0/320172975 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fbf180d0210 con 0x7fbf1c102da0 2024-04-21T23:06:56.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.817+0000 7fbf217ee700 1 --2- 172.21.15.44:0/320172975 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbf080a2de0 0x7fbf080a52a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c005950 tx=0x7fbf0c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:06:56.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:56.821+0000 7fbf12ffd700 1 -- 172.21.15.44:0/320172975 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fbf1805e6f0 con 0x7fbf1c102da0 2024-04-21T23:06:57.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:57.113+0000 7fbf24253700 1 -- 172.21.15.44:0/320172975 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fbf1c02dc20 con 0x7fbf1c102da0 2024-04-21T23:06:57.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:57.116+0000 7fbf12ffd700 1 -- 172.21.15.44:0/320172975 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fbf1c02dc20 con 0x7fbf1c102da0 2024-04-21T23:06:57.115 INFO:teuthology.orchestra.run.smithi044.stdout:[client.0] 2024-04-21T23:06:57.115 INFO:teuthology.orchestra.run.smithi044.stdout: key = AQARnCVmYn7eBhAAY66isFCMlpNunCt8We+lbw== 2024-04-21T23:06:57.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:57.118+0000 7fbf24253700 1 -- 172.21.15.44:0/320172975 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbf080a2de0 msgr2=0x7fbf080a52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:57.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:57.118+0000 7fbf24253700 1 --2- 172.21.15.44:0/320172975 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbf080a2de0 0x7fbf080a52a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c005950 tx=0x7fbf0c00a300 comp rx=0 tx=0).stop 2024-04-21T23:06:57.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:57.118+0000 7fbf24253700 1 -- 172.21.15.44:0/320172975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf1c102da0 msgr2=0x7fbf1c195fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:06:57.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:57.118+0000 7fbf24253700 1 --2- 172.21.15.44:0/320172975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf1c102da0 0x7fbf1c195fd0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbf18005910 tx=0x7fbf18009580 comp rx=0 tx=0).stop 2024-04-21T23:06:57.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:57.118+0000 7fbf24253700 1 -- 172.21.15.44:0/320172975 shutdown_connections 2024-04-21T23:06:57.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:57.118+0000 7fbf24253700 1 --2- 172.21.15.44:0/320172975 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbf1c108000 0x7fbf1c1a61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:57.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:57.118+0000 7fbf24253700 1 --2- 172.21.15.44:0/320172975 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fbf080a2de0 0x7fbf080a52a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:57.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:57.118+0000 7fbf24253700 1 --2- 172.21.15.44:0/320172975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbf1c102da0 0x7fbf1c195fd0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:57.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:57.118+0000 7fbf24253700 1 --2- 172.21.15.44:0/320172975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf1c1056d0 0x7fbf1c196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:06:57.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:57.118+0000 7fbf24253700 1 -- 172.21.15.44:0/320172975 >> 172.21.15.44:0/320172975 conn(0x7fbf1c0fc750 msgr2=0x7fbf1c1024d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:06:57.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:57.118+0000 7fbf24253700 1 -- 172.21.15.44:0/320172975 shutdown_connections 2024-04-21T23:06:57.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:06:57.118+0000 7fbf24253700 1 -- 172.21.15.44:0/320172975 wait complete. 2024-04-21T23:06:57.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:57 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/320172975' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-21T23:06:57.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:57 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/320172975' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-21T23:06:57.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:57 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/320172975' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-21T23:06:57.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:57 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/320172975' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-21T23:06:57.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:57 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/320172975' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-21T23:06:57.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:57 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/320172975' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-21T23:06:57.945 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T23:06:57.945 DEBUG:teuthology.orchestra.run.smithi044:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-21T23:06:57.946 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-21T23:06:57.984 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-21T23:06:57.984 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-21T23:06:57.984 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-21T23:06:58.274 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:06:58.680 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:06:58 smithi044 ceph-mon[27934]: pgmap v307: 129 pgs: 129 active+clean; 617 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.7 KiB/s wr, 2 op/s 2024-04-21T23:06:58.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:06:58 smithi069 ceph-mon[31816]: pgmap v307: 129 pgs: 129 active+clean; 617 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.7 KiB/s wr, 2 op/s 2024-04-21T23:06:58.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:06:58 smithi055 ceph-mon[31117]: pgmap v307: 129 pgs: 129 active+clean; 617 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.7 KiB/s wr, 2 op/s 2024-04-21T23:07:00.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.221+0000 7fcee0be5700 1 -- 172.21.15.44:0/4159265836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcedc102e90 msgr2=0x7fcedc106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:00.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.221+0000 7fcee0be5700 1 --2- 172.21.15.44:0/4159265836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcedc102e90 0x7fcedc106e90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fced0009990 tx=0x7fced0009ca0 comp rx=0 tx=0).stop 2024-04-21T23:07:00.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.222+0000 7fcee0be5700 1 -- 172.21.15.44:0/4159265836 shutdown_connections 2024-04-21T23:07:00.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.222+0000 7fcee0be5700 1 --2- 172.21.15.44:0/4159265836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcedc107570 0x7fcedc10f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:00.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.222+0000 7fcee0be5700 1 --2- 172.21.15.44:0/4159265836 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcedc1024e0 0x7fcedc1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:00.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.222+0000 7fcee0be5700 1 --2- 172.21.15.44:0/4159265836 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcedc102e90 0x7fcedc106e90 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:00.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.222+0000 7fcee0be5700 1 -- 172.21.15.44:0/4159265836 >> 172.21.15.44:0/4159265836 conn(0x7fcedc079390 msgr2=0x7fcedc0797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:00.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.222+0000 7fcee0be5700 1 -- 172.21.15.44:0/4159265836 shutdown_connections 2024-04-21T23:07:00.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.222+0000 7fcee0be5700 1 -- 172.21.15.44:0/4159265836 wait complete. 2024-04-21T23:07:00.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.222+0000 7fcee0be5700 1 Processor -- start 2024-04-21T23:07:00.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.223+0000 7fcee0be5700 1 -- start start 2024-04-21T23:07:00.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.223+0000 7fcee0be5700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcedc1024e0 0x7fcedc19e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:00.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.223+0000 7fcee0be5700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcedc102e90 0x7fcedc19ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:00.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.223+0000 7fcee0be5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcedc107570 0x7fcedc1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:00.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.223+0000 7fcee0be5700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc112610 con 0x7fcedc1024e0 2024-04-21T23:07:00.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.223+0000 7fcee0be5700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc112490 con 0x7fcedc107570 2024-04-21T23:07:00.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.223+0000 7fcee0be5700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcedc112790 con 0x7fcedc102e90 2024-04-21T23:07:00.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.223+0000 7fceda59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcedc1024e0 0x7fcedc19e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:00.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.223+0000 7fceda59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcedc1024e0 0x7fcedc19e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50850/0 (socket says 172.21.15.44:50850) 2024-04-21T23:07:00.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.223+0000 7fceda59c700 1 -- 172.21.15.44:0/3058249509 learned_addr learned my addr 172.21.15.44:0/3058249509 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:00.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.223+0000 7fced9d9b700 1 --2- 172.21.15.44:0/3058249509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcedc102e90 0x7fcedc19ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:00.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.223+0000 7fcedad9d700 1 --2- 172.21.15.44:0/3058249509 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcedc107570 0x7fcedc1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:00.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.223+0000 7fceda59c700 1 -- 172.21.15.44:0/3058249509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcedc102e90 msgr2=0x7fcedc19ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:00.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.223+0000 7fceda59c700 1 --2- 172.21.15.44:0/3058249509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcedc102e90 0x7fcedc19ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:00.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.223+0000 7fceda59c700 1 -- 172.21.15.44:0/3058249509 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcedc107570 msgr2=0x7fcedc1989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:00.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.223+0000 7fceda59c700 1 --2- 172.21.15.44:0/3058249509 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcedc107570 0x7fcedc1989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:00.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.223+0000 7fceda59c700 1 -- 172.21.15.44:0/3058249509 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcedc199110 con 0x7fcedc1024e0 2024-04-21T23:07:00.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.223+0000 7fced9d9b700 1 --2- 172.21.15.44:0/3058249509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcedc102e90 0x7fcedc19ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:07:00.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.223+0000 7fcedad9d700 1 --2- 172.21.15.44:0/3058249509 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcedc107570 0x7fcedc1989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:07:00.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.223+0000 7fceda59c700 1 --2- 172.21.15.44:0/3058249509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcedc1024e0 0x7fcedc19e7f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fcecc00b530 tx=0x7fcecc00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:00.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.224+0000 7fcecb7fe700 1 -- 172.21.15.44:0/3058249509 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcecc015020 con 0x7fcedc1024e0 2024-04-21T23:07:00.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.224+0000 7fcee0be5700 1 -- 172.21.15.44:0/3058249509 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcedc199400 con 0x7fcedc1024e0 2024-04-21T23:07:00.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.224+0000 7fcecb7fe700 1 -- 172.21.15.44:0/3058249509 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcecc011d10 con 0x7fcedc1024e0 2024-04-21T23:07:00.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.224+0000 7fcecb7fe700 1 -- 172.21.15.44:0/3058249509 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcecc010e50 con 0x7fcedc1024e0 2024-04-21T23:07:00.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.224+0000 7fcee0be5700 1 -- 172.21.15.44:0/3058249509 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcedc1a56c0 con 0x7fcedc1024e0 2024-04-21T23:07:00.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.225+0000 7fcee0be5700 1 -- 172.21.15.44:0/3058249509 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcedc068f20 con 0x7fcedc1024e0 2024-04-21T23:07:00.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.226+0000 7fcecb7fe700 1 -- 172.21.15.44:0/3058249509 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fcecc018020 con 0x7fcedc1024e0 2024-04-21T23:07:00.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.226+0000 7fcecb7fe700 1 --2- 172.21.15.44:0/3058249509 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcec40a2e50 0x7fcec40a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:00.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.227+0000 7fced9d9b700 1 --2- 172.21.15.44:0/3058249509 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcec40a2e50 0x7fcec40a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:00.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.227+0000 7fcecb7fe700 1 -- 172.21.15.44:0/3058249509 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fcecc0ceb80 con 0x7fcedc1024e0 2024-04-21T23:07:00.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.227+0000 7fced9d9b700 1 --2- 172.21.15.44:0/3058249509 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcec40a2e50 0x7fcec40a5310 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fced000b5c0 tx=0x7fced0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:00.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.231+0000 7fcecb7fe700 1 -- 172.21.15.44:0/3058249509 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fcecc01f020 con 0x7fcedc1024e0 2024-04-21T23:07:00.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.513+0000 7fcee0be5700 1 -- 172.21.15.44:0/3058249509 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fcedc199ee0 con 0x7fcedc1024e0 2024-04-21T23:07:00.513 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:00 smithi044 ceph-mon[27934]: pgmap v308: 129 pgs: 129 active+clean; 617 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.8 KiB/s wr, 3 op/s 2024-04-21T23:07:00.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.518+0000 7fcecb7fe700 1 -- 172.21.15.44:0/3058249509 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263199 (secure 0 0 0) 0x7fcecc09b7f0 con 0x7fcedc1024e0 2024-04-21T23:07:00.523 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:07:00.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.521+0000 7fcee0be5700 1 -- 172.21.15.44:0/3058249509 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcec40a2e50 msgr2=0x7fcec40a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:00.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.521+0000 7fcee0be5700 1 --2- 172.21.15.44:0/3058249509 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcec40a2e50 0x7fcec40a5310 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fced000b5c0 tx=0x7fced0005fb0 comp rx=0 tx=0).stop 2024-04-21T23:07:00.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.521+0000 7fcee0be5700 1 -- 172.21.15.44:0/3058249509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcedc1024e0 msgr2=0x7fcedc19e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:00.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.521+0000 7fcee0be5700 1 --2- 172.21.15.44:0/3058249509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcedc1024e0 0x7fcedc19e7f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fcecc00b530 tx=0x7fcecc00b8f0 comp rx=0 tx=0).stop 2024-04-21T23:07:00.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.521+0000 7fcee0be5700 1 -- 172.21.15.44:0/3058249509 shutdown_connections 2024-04-21T23:07:00.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.521+0000 7fcee0be5700 1 --2- 172.21.15.44:0/3058249509 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcedc107570 0x7fcedc1989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:00.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.521+0000 7fcee0be5700 1 --2- 172.21.15.44:0/3058249509 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcec40a2e50 0x7fcec40a5310 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:00.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.521+0000 7fcee0be5700 1 --2- 172.21.15.44:0/3058249509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcedc1024e0 0x7fcedc19e7f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:00.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.521+0000 7fcee0be5700 1 --2- 172.21.15.44:0/3058249509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcedc102e90 0x7fcedc19ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:00.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.521+0000 7fcee0be5700 1 -- 172.21.15.44:0/3058249509 >> 172.21.15.44:0/3058249509 conn(0x7fcedc079390 msgr2=0x7fcedc100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:00.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.521+0000 7fcee0be5700 1 -- 172.21.15.44:0/3058249509 shutdown_connections 2024-04-21T23:07:00.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:00.521+0000 7fcee0be5700 1 -- 172.21.15.44:0/3058249509 wait complete. 2024-04-21T23:07:00.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:00 smithi055 ceph-mon[31117]: pgmap v308: 129 pgs: 129 active+clean; 617 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.8 KiB/s wr, 3 op/s 2024-04-21T23:07:00.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:00 smithi069 ceph-mon[31816]: pgmap v308: 129 pgs: 129 active+clean; 617 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.8 KiB/s wr, 3 op/s 2024-04-21T23:07:01.487 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6800","nonce":1417504446},{"type":"v1","addr":"172.21.15.44:6801","nonce":1417504446}]},"active_addr":"172.21.15.44:6801/1417504446","active_change":"2024-04-21T22:59:22.969841+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24119,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.44:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":792524018}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":101322314}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":3134653118}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":2303525779}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.44:0","nonce":1190207074}]}]} 2024-04-21T23:07:01.492 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-21T23:07:01.492 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-21T23:07:01.493 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-21T23:07:01.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:01 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3058249509' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-21T23:07:01.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:01 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3058249509' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-21T23:07:01.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:01 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3058249509' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-21T23:07:01.749 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:02.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:02 smithi044 ceph-mon[27934]: pgmap v309: 129 pgs: 129 active+clean; 629 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.2 KiB/s wr, 4 op/s 2024-04-21T23:07:02.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:02 smithi055 ceph-mon[31117]: pgmap v309: 129 pgs: 129 active+clean; 629 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.2 KiB/s wr, 4 op/s 2024-04-21T23:07:02.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:02 smithi069 ceph-mon[31816]: pgmap v309: 129 pgs: 129 active+clean; 629 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.2 KiB/s wr, 4 op/s 2024-04-21T23:07:03.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.644+0000 7f1e1dcf3700 1 -- 172.21.15.44:0/1398917819 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e1806b5f0 msgr2=0x7f1e18106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:03.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.644+0000 7f1e1dcf3700 1 --2- 172.21.15.44:0/1398917819 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e1806b5f0 0x7f1e18106eb0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c0099c0 tx=0x7f1e0c009cd0 comp rx=0 tx=0).stop 2024-04-21T23:07:03.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.644+0000 7f1e1dcf3700 1 -- 172.21.15.44:0/1398917819 shutdown_connections 2024-04-21T23:07:03.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.644+0000 7f1e1dcf3700 1 --2- 172.21.15.44:0/1398917819 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e18107590 0x7f1e1810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:03.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.644+0000 7f1e1dcf3700 1 --2- 172.21.15.44:0/1398917819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e1806ac40 0x7f1e1806b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:03.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.644+0000 7f1e1dcf3700 1 --2- 172.21.15.44:0/1398917819 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e1806b5f0 0x7f1e18106eb0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:03.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.644+0000 7f1e1dcf3700 1 -- 172.21.15.44:0/1398917819 >> 172.21.15.44:0/1398917819 conn(0x7f1e180781d0 msgr2=0x7f1e180785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:03.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.644+0000 7f1e1dcf3700 1 -- 172.21.15.44:0/1398917819 shutdown_connections 2024-04-21T23:07:03.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.644+0000 7f1e1dcf3700 1 -- 172.21.15.44:0/1398917819 wait complete. 2024-04-21T23:07:03.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.645+0000 7f1e1dcf3700 1 Processor -- start 2024-04-21T23:07:03.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.645+0000 7f1e1dcf3700 1 -- start start 2024-04-21T23:07:03.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.645+0000 7f1e1dcf3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e1806ac40 0x7f1e1819e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:03.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.645+0000 7f1e1dcf3700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e1806b5f0 0x7f1e1819ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:03.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.645+0000 7f1e1dcf3700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e18107590 0x7f1e18198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:03.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.645+0000 7f1e1dcf3700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1e18112550 con 0x7f1e18107590 2024-04-21T23:07:03.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.645+0000 7f1e1dcf3700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1e181123d0 con 0x7f1e1806b5f0 2024-04-21T23:07:03.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.645+0000 7f1e1dcf3700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1e181126d0 con 0x7f1e1806ac40 2024-04-21T23:07:03.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.645+0000 7f1e17fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e18107590 0x7f1e18198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:03.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.646+0000 7f1e17fff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e18107590 0x7f1e18198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42478/0 (socket says 172.21.15.44:42478) 2024-04-21T23:07:03.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.646+0000 7f1e17fff700 1 -- 172.21.15.44:0/3948486987 learned_addr learned my addr 172.21.15.44:0/3948486987 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:03.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.646+0000 7f1e177fe700 1 --2- 172.21.15.44:0/3948486987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e1806ac40 0x7f1e1819e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:03.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.646+0000 7f1e16ffd700 1 --2- 172.21.15.44:0/3948486987 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e1806b5f0 0x7f1e1819ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:03.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.646+0000 7f1e17fff700 1 -- 172.21.15.44:0/3948486987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e1806ac40 msgr2=0x7f1e1819e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:03.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.646+0000 7f1e17fff700 1 --2- 172.21.15.44:0/3948486987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e1806ac40 0x7f1e1819e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:03.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.646+0000 7f1e17fff700 1 -- 172.21.15.44:0/3948486987 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e1806b5f0 msgr2=0x7f1e1819ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:03.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.646+0000 7f1e17fff700 1 --2- 172.21.15.44:0/3948486987 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e1806b5f0 0x7f1e1819ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:03.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.646+0000 7f1e17fff700 1 -- 172.21.15.44:0/3948486987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e18199050 con 0x7f1e18107590 2024-04-21T23:07:03.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.646+0000 7f1e17fff700 1 --2- 172.21.15.44:0/3948486987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e18107590 0x7f1e18198870 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1e1000ef30 tx=0x7f1e1000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:03.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.646+0000 7f1e14ff9700 1 -- 172.21.15.44:0/3948486987 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e100092e0 con 0x7f1e18107590 2024-04-21T23:07:03.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.646+0000 7f1e14ff9700 1 -- 172.21.15.44:0/3948486987 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e10004bb0 con 0x7f1e18107590 2024-04-21T23:07:03.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.646+0000 7f1e14ff9700 1 -- 172.21.15.44:0/3948486987 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e10004280 con 0x7f1e18107590 2024-04-21T23:07:03.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.646+0000 7f1e1dcf3700 1 -- 172.21.15.44:0/3948486987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e181992e0 con 0x7f1e18107590 2024-04-21T23:07:03.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.646+0000 7f1e1dcf3700 1 -- 172.21.15.44:0/3948486987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e181a5500 con 0x7f1e18107590 2024-04-21T23:07:03.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.647+0000 7f1e1dcf3700 1 -- 172.21.15.44:0/3948486987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e18103990 con 0x7f1e18107590 2024-04-21T23:07:03.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.648+0000 7f1e14ff9700 1 -- 172.21.15.44:0/3948486987 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1e1001b030 con 0x7f1e18107590 2024-04-21T23:07:03.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.649+0000 7f1e14ff9700 1 --2- 172.21.15.44:0/3948486987 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f1e000a2cb0 0x7f1e000a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:03.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.649+0000 7f1e177fe700 1 --2- 172.21.15.44:0/3948486987 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f1e000a2cb0 0x7f1e000a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:03.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.649+0000 7f1e14ff9700 1 -- 172.21.15.44:0/3948486987 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f1e100ceed0 con 0x7f1e18107590 2024-04-21T23:07:03.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.649+0000 7f1e177fe700 1 --2- 172.21.15.44:0/3948486987 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f1e000a2cb0 0x7f1e000a5170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1e08005950 tx=0x7f1e0800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:03.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.654+0000 7f1e14ff9700 1 -- 172.21.15.44:0/3948486987 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1e10022020 con 0x7f1e18107590 2024-04-21T23:07:03.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.861+0000 7f1e1dcf3700 1 -- 172.21.15.44:0/3948486987 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1e1802db30 con 0x7f1e18107590 2024-04-21T23:07:03.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.863+0000 7f1e14ff9700 1 -- 172.21.15.44:0/3948486987 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24032 (secure 0 0 0) 0x7f1e10010040 con 0x7f1e18107590 2024-04-21T23:07:03.863 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:07:03.863 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":90,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:06:30.619155+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:07:03.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.865+0000 7f1e1dcf3700 1 -- 172.21.15.44:0/3948486987 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f1e000a2cb0 msgr2=0x7f1e000a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:03.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.865+0000 7f1e1dcf3700 1 --2- 172.21.15.44:0/3948486987 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f1e000a2cb0 0x7f1e000a5170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1e08005950 tx=0x7f1e0800b410 comp rx=0 tx=0).stop 2024-04-21T23:07:03.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.865+0000 7f1e1dcf3700 1 -- 172.21.15.44:0/3948486987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e18107590 msgr2=0x7f1e18198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:03.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.865+0000 7f1e1dcf3700 1 --2- 172.21.15.44:0/3948486987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e18107590 0x7f1e18198870 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1e1000ef30 tx=0x7f1e1000c3a0 comp rx=0 tx=0).stop 2024-04-21T23:07:03.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.865+0000 7f1e1dcf3700 1 -- 172.21.15.44:0/3948486987 shutdown_connections 2024-04-21T23:07:03.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.865+0000 7f1e1dcf3700 1 --2- 172.21.15.44:0/3948486987 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f1e000a2cb0 0x7f1e000a5170 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:03.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.865+0000 7f1e1dcf3700 1 --2- 172.21.15.44:0/3948486987 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1e18107590 0x7f1e18198870 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:03.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.865+0000 7f1e1dcf3700 1 --2- 172.21.15.44:0/3948486987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e1806ac40 0x7f1e1819e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:03.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.865+0000 7f1e1dcf3700 1 --2- 172.21.15.44:0/3948486987 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1e1806b5f0 0x7f1e1819ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:03.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.865+0000 7f1e1dcf3700 1 -- 172.21.15.44:0/3948486987 >> 172.21.15.44:0/3948486987 conn(0x7f1e180781d0 msgr2=0x7f1e180ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:03.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.865+0000 7f1e1dcf3700 1 -- 172.21.15.44:0/3948486987 shutdown_connections 2024-04-21T23:07:03.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:03.866+0000 7f1e1dcf3700 1 -- 172.21.15.44:0/3948486987 wait complete. 2024-04-21T23:07:04.614 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:04 smithi044 ceph-mon[27934]: pgmap v310: 129 pgs: 129 active+clean; 629 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.2 KiB/s wr, 5 op/s 2024-04-21T23:07:04.614 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:04 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3948486987' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:07:04.647 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-21T23:07:04.648 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-21T23:07:04.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:04 smithi069 ceph-mon[31816]: pgmap v310: 129 pgs: 129 active+clean; 629 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.2 KiB/s wr, 5 op/s 2024-04-21T23:07:04.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:04 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3948486987' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:07:04.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:04 smithi055 ceph-mon[31117]: pgmap v310: 129 pgs: 129 active+clean; 629 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.2 KiB/s wr, 5 op/s 2024-04-21T23:07:04.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:04 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3948486987' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:07:04.900 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:06.651 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:06 smithi044 ceph-mon[27934]: pgmap v311: 129 pgs: 129 active+clean; 638 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2024-04-21T23:07:06.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.659+0000 7f158cd23700 1 -- 172.21.15.44:0/770581108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15881013b0 msgr2=0x7f1588101790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:06.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.659+0000 7f158cd23700 1 --2- 172.21.15.44:0/770581108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15881013b0 0x7f1588101790 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f15780099c0 tx=0x7f1578009cd0 comp rx=0 tx=0).stop 2024-04-21T23:07:06.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.660+0000 7f158cd23700 1 -- 172.21.15.44:0/770581108 shutdown_connections 2024-04-21T23:07:06.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.660+0000 7f158cd23700 1 --2- 172.21.15.44:0/770581108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1588106490 0x7f158810e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:06.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.660+0000 7f158cd23700 1 --2- 172.21.15.44:0/770581108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15881013b0 0x7f1588101790 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:06.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.660+0000 7f158cd23700 1 --2- 172.21.15.44:0/770581108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1588101d60 0x7f1588105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:06.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.660+0000 7f158cd23700 1 -- 172.21.15.44:0/770581108 >> 172.21.15.44:0/770581108 conn(0x7f15880781d0 msgr2=0x7f15880785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:06.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.660+0000 7f158cd23700 1 -- 172.21.15.44:0/770581108 shutdown_connections 2024-04-21T23:07:06.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.660+0000 7f158cd23700 1 -- 172.21.15.44:0/770581108 wait complete. 2024-04-21T23:07:06.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.660+0000 7f158cd23700 1 Processor -- start 2024-04-21T23:07:06.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.660+0000 7f158cd23700 1 -- start start 2024-04-21T23:07:06.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.661+0000 7f158cd23700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15881013b0 0x7f1588199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:06.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.661+0000 7f158cd23700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1588101d60 0x7f1588199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:06.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.661+0000 7f158cd23700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1588106490 0x7f15881a4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:06.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.661+0000 7f158cd23700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f15881114f0 con 0x7f1588101d60 2024-04-21T23:07:06.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.661+0000 7f158cd23700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1588111370 con 0x7f1588106490 2024-04-21T23:07:06.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.661+0000 7f158cd23700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1588111670 con 0x7f15881013b0 2024-04-21T23:07:06.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.661+0000 7f1585d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1588101d60 0x7f1588199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:06.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.661+0000 7f1585d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1588101d60 0x7f1588199e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42500/0 (socket says 172.21.15.44:42500) 2024-04-21T23:07:06.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.661+0000 7f1585d9b700 1 -- 172.21.15.44:0/1834195342 learned_addr learned my addr 172.21.15.44:0/1834195342 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:06.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.661+0000 7f158659c700 1 --2- 172.21.15.44:0/1834195342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15881013b0 0x7f1588199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:06.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.661+0000 7f1586d9d700 1 --2- 172.21.15.44:0/1834195342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1588106490 0x7f15881a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:06.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.661+0000 7f1585d9b700 1 -- 172.21.15.44:0/1834195342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15881013b0 msgr2=0x7f1588199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:06.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.661+0000 7f1585d9b700 1 --2- 172.21.15.44:0/1834195342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15881013b0 0x7f1588199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:06.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.661+0000 7f1585d9b700 1 -- 172.21.15.44:0/1834195342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1588106490 msgr2=0x7f15881a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:06.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.661+0000 7f1585d9b700 1 --2- 172.21.15.44:0/1834195342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1588106490 0x7f15881a4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:06.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.661+0000 7f1585d9b700 1 -- 172.21.15.44:0/1834195342 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15881a4ed0 con 0x7f1588101d60 2024-04-21T23:07:06.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.661+0000 7f1585d9b700 1 --2- 172.21.15.44:0/1834195342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1588101d60 0x7f1588199e50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f157c00c9e0 tx=0x7f157c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:06.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.662+0000 7f15777fe700 1 -- 172.21.15.44:0/1834195342 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157c007aa0 con 0x7f1588101d60 2024-04-21T23:07:06.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.662+0000 7f158cd23700 1 -- 172.21.15.44:0/1834195342 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15881a50a0 con 0x7f1588101d60 2024-04-21T23:07:06.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.662+0000 7f15777fe700 1 -- 172.21.15.44:0/1834195342 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f157c007c40 con 0x7f1588101d60 2024-04-21T23:07:06.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.662+0000 7f15777fe700 1 -- 172.21.15.44:0/1834195342 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157c018640 con 0x7f1588101d60 2024-04-21T23:07:06.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.662+0000 7f158cd23700 1 -- 172.21.15.44:0/1834195342 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15881a55b0 con 0x7f1588101d60 2024-04-21T23:07:06.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.663+0000 7f158cd23700 1 -- 172.21.15.44:0/1834195342 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1588068f20 con 0x7f1588101d60 2024-04-21T23:07:06.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.664+0000 7f15777fe700 1 -- 172.21.15.44:0/1834195342 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f157c00f450 con 0x7f1588101d60 2024-04-21T23:07:06.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.665+0000 7f15777fe700 1 --2- 172.21.15.44:0/1834195342 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f15700a2cb0 0x7f15700a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:06.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.665+0000 7f158659c700 1 --2- 172.21.15.44:0/1834195342 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f15700a2cb0 0x7f15700a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:06.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.665+0000 7f15777fe700 1 -- 172.21.15.44:0/1834195342 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f157c0cf6c0 con 0x7f1588101d60 2024-04-21T23:07:06.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.665+0000 7f158659c700 1 --2- 172.21.15.44:0/1834195342 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f15700a2cb0 0x7f15700a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1578000c00 tx=0x7f1578005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:06.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.669+0000 7f15777fe700 1 -- 172.21.15.44:0/1834195342 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f157c014030 con 0x7f1588101d60 2024-04-21T23:07:06.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:06 smithi055 ceph-mon[31117]: pgmap v311: 129 pgs: 129 active+clean; 638 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2024-04-21T23:07:06.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:06 smithi069 ceph-mon[31816]: pgmap v311: 129 pgs: 129 active+clean; 638 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2024-04-21T23:07:06.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.877+0000 7f158cd23700 1 -- 172.21.15.44:0/1834195342 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f158806af50 con 0x7f1588101d60 2024-04-21T23:07:06.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.878+0000 7f15777fe700 1 -- 172.21.15.44:0/1834195342 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24032 (secure 0 0 0) 0x7f157c09c2b0 con 0x7f1588101d60 2024-04-21T23:07:06.878 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:07:06.878 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":90,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:06:30.619155+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:07:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.880+0000 7f158cd23700 1 -- 172.21.15.44:0/1834195342 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f15700a2cb0 msgr2=0x7f15700a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.880+0000 7f158cd23700 1 --2- 172.21.15.44:0/1834195342 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f15700a2cb0 0x7f15700a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1578000c00 tx=0x7f1578005fb0 comp rx=0 tx=0).stop 2024-04-21T23:07:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.880+0000 7f158cd23700 1 -- 172.21.15.44:0/1834195342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1588101d60 msgr2=0x7f1588199e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.880+0000 7f158cd23700 1 --2- 172.21.15.44:0/1834195342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1588101d60 0x7f1588199e50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f157c00c9e0 tx=0x7f157c00ccf0 comp rx=0 tx=0).stop 2024-04-21T23:07:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.880+0000 7f158cd23700 1 -- 172.21.15.44:0/1834195342 shutdown_connections 2024-04-21T23:07:06.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.880+0000 7f158cd23700 1 --2- 172.21.15.44:0/1834195342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1588106490 0x7f15881a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:06.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.880+0000 7f158cd23700 1 --2- 172.21.15.44:0/1834195342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15881013b0 0x7f1588199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:06.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.880+0000 7f158cd23700 1 --2- 172.21.15.44:0/1834195342 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f15700a2cb0 0x7f15700a5170 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:06.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.880+0000 7f158cd23700 1 --2- 172.21.15.44:0/1834195342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1588101d60 0x7f1588199e50 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:06.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.880+0000 7f158cd23700 1 -- 172.21.15.44:0/1834195342 >> 172.21.15.44:0/1834195342 conn(0x7f15880781d0 msgr2=0x7f15880ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:06.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.880+0000 7f158cd23700 1 -- 172.21.15.44:0/1834195342 shutdown_connections 2024-04-21T23:07:06.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:06.881+0000 7f158cd23700 1 -- 172.21.15.44:0/1834195342 wait complete. 2024-04-21T23:07:07.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:07 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1834195342' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:07:07.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:07 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1834195342' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:07:07.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:07 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1834195342' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:07:07.721 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-21T23:07:07.722 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-21T23:07:07.722 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-21T23:07:07.723 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-21T23:07:07.723 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-21T23:07:07.723 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-21T23:07:07.724 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-21T23:07:07.724 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-21T23:07:07.725 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-21T23:07:07.725 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-21T23:07:07.725 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-21T23:07:07.726 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-21T23:07:08.418 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:08.520 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:08.523 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:08.659 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:08 smithi044 ceph-mon[27934]: pgmap v312: 129 pgs: 129 active+clean; 638 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-04-21T23:07:08.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:08 smithi069 ceph-mon[31816]: pgmap v312: 129 pgs: 129 active+clean; 638 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-04-21T23:07:08.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:08 smithi055 ceph-mon[31117]: pgmap v312: 129 pgs: 129 active+clean; 638 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-04-21T23:07:09.250 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:09.593 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:09.593 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:09.594 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:09.594 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:09.594 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:09.595 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:10.443 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:10.448 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:10.645 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:10 smithi044 ceph-mon[27934]: pgmap v313: 129 pgs: 129 active+clean; 638 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.3 KiB/s wr, 3 op/s 2024-04-21T23:07:10.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:10 smithi055 ceph-mon[31117]: pgmap v313: 129 pgs: 129 active+clean; 638 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.3 KiB/s wr, 3 op/s 2024-04-21T23:07:10.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:10 smithi069 ceph-mon[31816]: pgmap v313: 129 pgs: 129 active+clean; 638 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.3 KiB/s wr, 3 op/s 2024-04-21T23:07:12.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:12 smithi044 ceph-mon[27934]: pgmap v314: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.3 KiB/s wr, 3 op/s 2024-04-21T23:07:12.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:12 smithi069 ceph-mon[31816]: pgmap v314: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.3 KiB/s wr, 3 op/s 2024-04-21T23:07:12.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:12 smithi055 ceph-mon[31117]: pgmap v314: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.3 KiB/s wr, 3 op/s 2024-04-21T23:07:14.556 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:14 smithi044 ceph-mon[27934]: pgmap v315: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 938 B/s wr, 2 op/s 2024-04-21T23:07:14.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:14 smithi055 ceph-mon[31117]: pgmap v315: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 938 B/s wr, 2 op/s 2024-04-21T23:07:14.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:14 smithi069 ceph-mon[31816]: pgmap v315: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 938 B/s wr, 2 op/s 2024-04-21T23:07:16.643 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:16 smithi044 ceph-mon[27934]: pgmap v316: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 938 B/s wr, 2 op/s 2024-04-21T23:07:16.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:16 smithi055 ceph-mon[31117]: pgmap v316: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 938 B/s wr, 2 op/s 2024-04-21T23:07:16.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:16 smithi069 ceph-mon[31816]: pgmap v316: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 938 B/s wr, 2 op/s 2024-04-21T23:07:17.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.491+0000 7ff8005ef700 1 -- 172.21.15.44:0/1102274526 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7f8102ec0 msgr2=0x7ff7f8106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:17.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.491+0000 7ff8005ef700 1 --2- 172.21.15.44:0/1102274526 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7f8102ec0 0x7ff7f8106eb0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8009990 tx=0x7ff7e8009ca0 comp rx=0 tx=0).stop 2024-04-21T23:07:17.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.491+0000 7ff8005ef700 1 -- 172.21.15.44:0/1102274526 shutdown_connections 2024-04-21T23:07:17.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.491+0000 7ff8005ef700 1 --2- 172.21.15.44:0/1102274526 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7f8107590 0x7ff7f810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:17.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.491+0000 7ff8005ef700 1 --2- 172.21.15.44:0/1102274526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7f8102510 0x7ff7f81028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:17.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.491+0000 7ff8005ef700 1 --2- 172.21.15.44:0/1102274526 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7f8102ec0 0x7ff7f8106eb0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:17.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.491+0000 7ff8005ef700 1 -- 172.21.15.44:0/1102274526 >> 172.21.15.44:0/1102274526 conn(0x7ff7f80793a0 msgr2=0x7ff7f80797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:17.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.491+0000 7ff8005ef700 1 -- 172.21.15.44:0/1102274526 shutdown_connections 2024-04-21T23:07:17.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.491+0000 7ff8005ef700 1 -- 172.21.15.44:0/1102274526 wait complete. 2024-04-21T23:07:17.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.491+0000 7ff8005ef700 1 Processor -- start 2024-04-21T23:07:17.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.492+0000 7ff8005ef700 1 -- start start 2024-04-21T23:07:17.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.492+0000 7ff8005ef700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7f8102510 0x7ff7f819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:17.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.492+0000 7ff8005ef700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7f8102ec0 0x7ff7f819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:17.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.492+0000 7ff8005ef700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7f8107590 0x7ff7f8198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:17.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.492+0000 7ff8005ef700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f8112550 con 0x7ff7f8107590 2024-04-21T23:07:17.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.492+0000 7ff8005ef700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f81123d0 con 0x7ff7f8102510 2024-04-21T23:07:17.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.492+0000 7ff8005ef700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f81126d0 con 0x7ff7f8102ec0 2024-04-21T23:07:17.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.492+0000 7ff7feb8c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7f8107590 0x7ff7f8198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:17.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.492+0000 7ff7feb8c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7f8107590 0x7ff7f8198810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34444/0 (socket says 172.21.15.44:34444) 2024-04-21T23:07:17.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.492+0000 7ff7feb8c700 1 -- 172.21.15.44:0/1630274926 learned_addr learned my addr 172.21.15.44:0/1630274926 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:17.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.492+0000 7ff7fdb8a700 1 --2- 172.21.15.44:0/1630274926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7f8102ec0 0x7ff7f819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:17.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.492+0000 7ff7fe38b700 1 --2- 172.21.15.44:0/1630274926 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7f8102510 0x7ff7f819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:17.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.492+0000 7ff7feb8c700 1 -- 172.21.15.44:0/1630274926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7f8102ec0 msgr2=0x7ff7f819eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:17.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.492+0000 7ff7feb8c700 1 --2- 172.21.15.44:0/1630274926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7f8102ec0 0x7ff7f819eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:17.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.492+0000 7ff7feb8c700 1 -- 172.21.15.44:0/1630274926 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7f8102510 msgr2=0x7ff7f819e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:17.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.492+0000 7ff7feb8c700 1 --2- 172.21.15.44:0/1630274926 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7f8102510 0x7ff7f819e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:17.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.492+0000 7ff7feb8c700 1 -- 172.21.15.44:0/1630274926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f8198ff0 con 0x7ff7f8107590 2024-04-21T23:07:17.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.492+0000 7ff7fe38b700 1 --2- 172.21.15.44:0/1630274926 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7f8102510 0x7ff7f819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:07:17.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.492+0000 7ff7feb8c700 1 --2- 172.21.15.44:0/1630274926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7f8107590 0x7ff7f8198810 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0007ee0 tx=0x7ff7f000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:17.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.493+0000 7ff7ef7fe700 1 -- 172.21.15.44:0/1630274926 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f000dbd0 con 0x7ff7f8107590 2024-04-21T23:07:17.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.493+0000 7ff8005ef700 1 -- 172.21.15.44:0/1630274926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7f81992e0 con 0x7ff7f8107590 2024-04-21T23:07:17.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.493+0000 7ff7ef7fe700 1 -- 172.21.15.44:0/1630274926 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7f000dd70 con 0x7ff7f8107590 2024-04-21T23:07:17.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.493+0000 7ff7ef7fe700 1 -- 172.21.15.44:0/1630274926 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0005a60 con 0x7ff7f8107590 2024-04-21T23:07:17.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.493+0000 7ff8005ef700 1 -- 172.21.15.44:0/1630274926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7f81a54a0 con 0x7ff7f8107590 2024-04-21T23:07:17.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.494+0000 7ff8005ef700 1 -- 172.21.15.44:0/1630274926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff7f80629c0 con 0x7ff7f8107590 2024-04-21T23:07:17.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.495+0000 7ff7ef7fe700 1 -- 172.21.15.44:0/1630274926 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff7f001a020 con 0x7ff7f8107590 2024-04-21T23:07:17.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.495+0000 7ff7ef7fe700 1 --2- 172.21.15.44:0/1630274926 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff7e40a2cb0 0x7ff7e40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:17.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.496+0000 7ff7fe38b700 1 --2- 172.21.15.44:0/1630274926 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff7e40a2cb0 0x7ff7e40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:17.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.496+0000 7ff7ef7fe700 1 -- 172.21.15.44:0/1630274926 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff7f00cf1e0 con 0x7ff7f8107590 2024-04-21T23:07:17.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.496+0000 7ff7ef7fe700 1 --2- 172.21.15.44:0/1630274926 >> [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096] conn(0x7ff7e40ad970 0x7ff7e40afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:17.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.496+0000 7ff7ef7fe700 1 -- 172.21.15.44:0/1630274926 --> [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff7e40b0480 con 0x7ff7e40ad970 2024-04-21T23:07:17.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.496+0000 7ff7ef7fe700 1 -- 172.21.15.44:0/1630274926 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7ff7f00cf590 con 0x7ff7f8107590 2024-04-21T23:07:17.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.496+0000 7ff7fdb8a700 1 --2- 172.21.15.44:0/1630274926 >> [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096] conn(0x7ff7e40ad970 0x7ff7e40afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:17.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.496+0000 7ff7fe38b700 1 --2- 172.21.15.44:0/1630274926 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff7e40a2cb0 0x7ff7e40a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009be0 tx=0x7ff7f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:17.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.496+0000 7ff7fdb8a700 1 --2- 172.21.15.44:0/1630274926 >> [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096] conn(0x7ff7e40ad970 0x7ff7e40afdb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:17.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.497+0000 7ff7ef7fe700 1 -- 172.21.15.44:0/1630274926 <== osd.0 v2:172.21.15.44:6802/4094542096 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7e40b0480 con 0x7ff7e40ad970 2024-04-21T23:07:17.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.513+0000 7ff8005ef700 1 -- 172.21.15.44:0/1630274926 --> [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff7f8062dc0 con 0x7ff7e40ad970 2024-04-21T23:07:17.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.514+0000 7ff7ef7fe700 1 -- 172.21.15.44:0/1630274926 <== osd.0 v2:172.21.15.44:6802/4094542096 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff7f8062dc0 con 0x7ff7e40ad970 2024-04-21T23:07:17.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.514+0000 7ff8005ef700 1 -- 172.21.15.44:0/1630274926 >> [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096] conn(0x7ff7e40ad970 msgr2=0x7ff7e40afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:17.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.514+0000 7ff8005ef700 1 --2- 172.21.15.44:0/1630274926 >> [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096] conn(0x7ff7e40ad970 0x7ff7e40afdb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:17.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.514+0000 7ff8005ef700 1 -- 172.21.15.44:0/1630274926 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff7e40a2cb0 msgr2=0x7ff7e40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:17.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.514+0000 7ff8005ef700 1 --2- 172.21.15.44:0/1630274926 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff7e40a2cb0 0x7ff7e40a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009be0 tx=0x7ff7f4009260 comp rx=0 tx=0).stop 2024-04-21T23:07:17.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.514+0000 7ff8005ef700 1 -- 172.21.15.44:0/1630274926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7f8107590 msgr2=0x7ff7f8198810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:17.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.514+0000 7ff8005ef700 1 --2- 172.21.15.44:0/1630274926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7f8107590 0x7ff7f8198810 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0007ee0 tx=0x7ff7f000d3b0 comp rx=0 tx=0).stop 2024-04-21T23:07:17.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.514+0000 7ff7feb8c700 1 -- 172.21.15.44:0/1630274926 reap_dead start 2024-04-21T23:07:17.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.514+0000 7ff8005ef700 1 -- 172.21.15.44:0/1630274926 shutdown_connections 2024-04-21T23:07:17.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.514+0000 7ff8005ef700 1 -- 172.21.15.44:0/1630274926 >> 172.21.15.44:0/1630274926 conn(0x7ff7f80793a0 msgr2=0x7ff7f81003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:17.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.514+0000 7ff8005ef700 1 -- 172.21.15.44:0/1630274926 shutdown_connections 2024-04-21T23:07:17.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:17.514+0000 7ff8005ef700 1 -- 172.21.15.44:0/1630274926 wait complete. 2024-04-21T23:07:18.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.516+0000 7f8a7c8eb700 1 -- 172.21.15.44:0/759672138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a78101ae0 msgr2=0x7f8a7810dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.516+0000 7f8a7c8eb700 1 --2- 172.21.15.44:0/759672138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a78101ae0 0x7f8a7810dfd0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8a7000b0a0 tx=0x7f8a7000b3b0 comp rx=0 tx=0).stop 2024-04-21T23:07:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.516+0000 7f8a7c8eb700 1 -- 172.21.15.44:0/759672138 shutdown_connections 2024-04-21T23:07:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.516+0000 7f8a7c8eb700 1 --2- 172.21.15.44:0/759672138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a78101ae0 0x7f8a7810dfd0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.516+0000 7f8a7c8eb700 1 --2- 172.21.15.44:0/759672138 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a78107190 0x7f8a78107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.516+0000 7f8a7c8eb700 1 --2- 172.21.15.44:0/759672138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a78101120 0x7f8a781015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.516+0000 7f8a7c8eb700 1 -- 172.21.15.44:0/759672138 >> 172.21.15.44:0/759672138 conn(0x7f8a780780a0 msgr2=0x7f8a780784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.516+0000 7f8a7c8eb700 1 -- 172.21.15.44:0/759672138 shutdown_connections 2024-04-21T23:07:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.516+0000 7f8a7c8eb700 1 -- 172.21.15.44:0/759672138 wait complete. 2024-04-21T23:07:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.517+0000 7f8a7c8eb700 1 Processor -- start 2024-04-21T23:07:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.517+0000 7f8a7c8eb700 1 -- start start 2024-04-21T23:07:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.517+0000 7f8a7c8eb700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a78101120 0x7f8a7819e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:18.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.517+0000 7f8a7c8eb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a78101ae0 0x7f8a7819ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:18.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.517+0000 7f8a7c8eb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a78107190 0x7f8a781987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:18.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.517+0000 7f8a7c8eb700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8a78112520 con 0x7f8a78101120 2024-04-21T23:07:18.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.517+0000 7f8a7c8eb700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8a781123a0 con 0x7f8a78107190 2024-04-21T23:07:18.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.517+0000 7f8a7c8eb700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a781126a0 con 0x7f8a78101ae0 2024-04-21T23:07:18.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.517+0000 7f8a7659c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a78101120 0x7f8a7819e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:18.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.517+0000 7f8a7659c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a78101120 0x7f8a7819e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34466/0 (socket says 172.21.15.44:34466) 2024-04-21T23:07:18.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.517+0000 7f8a7659c700 1 -- 172.21.15.44:0/1660457031 learned_addr learned my addr 172.21.15.44:0/1660457031 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:18.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.517+0000 7f8a75d9b700 1 --2- 172.21.15.44:0/1660457031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a78101ae0 0x7f8a7819ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:18.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.518+0000 7f8a76d9d700 1 --2- 172.21.15.44:0/1660457031 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a78107190 0x7f8a781987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:18.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.518+0000 7f8a7659c700 1 -- 172.21.15.44:0/1660457031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a78101ae0 msgr2=0x7f8a7819ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:18.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.518+0000 7f8a7659c700 1 --2- 172.21.15.44:0/1660457031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a78101ae0 0x7f8a7819ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:18.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.518+0000 7f8a7659c700 1 -- 172.21.15.44:0/1660457031 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a78107190 msgr2=0x7f8a781987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:18.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.518+0000 7f8a7659c700 1 --2- 172.21.15.44:0/1660457031 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a78107190 0x7f8a781987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:18.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.518+0000 7f8a7659c700 1 -- 172.21.15.44:0/1660457031 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a78199080 con 0x7f8a78101120 2024-04-21T23:07:18.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.518+0000 7f8a7659c700 1 --2- 172.21.15.44:0/1660457031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a78101120 0x7f8a7819e680 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800c920 tx=0x7f8a6800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:18.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.518+0000 7f8a677fe700 1 -- 172.21.15.44:0/1660457031 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68007b00 con 0x7f8a78101120 2024-04-21T23:07:18.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.518+0000 7f8a677fe700 1 -- 172.21.15.44:0/1660457031 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a68004d10 con 0x7f8a78101120 2024-04-21T23:07:18.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.518+0000 7f8a7c8eb700 1 -- 172.21.15.44:0/1660457031 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a78199370 con 0x7f8a78101120 2024-04-21T23:07:18.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.518+0000 7f8a677fe700 1 -- 172.21.15.44:0/1660457031 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68005680 con 0x7f8a78101120 2024-04-21T23:07:18.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.518+0000 7f8a7c8eb700 1 -- 172.21.15.44:0/1660457031 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a781a5600 con 0x7f8a78101120 2024-04-21T23:07:18.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.519+0000 7f8a7c8eb700 1 -- 172.21.15.44:0/1660457031 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8a78062980 con 0x7f8a78101120 2024-04-21T23:07:18.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.520+0000 7f8a677fe700 1 -- 172.21.15.44:0/1660457031 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f8a68004750 con 0x7f8a78101120 2024-04-21T23:07:18.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.521+0000 7f8a677fe700 1 --2- 172.21.15.44:0/1660457031 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f8a600a2cb0 0x7f8a600a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:18.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.521+0000 7f8a75d9b700 1 --2- 172.21.15.44:0/1660457031 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f8a600a2cb0 0x7f8a600a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:18.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.521+0000 7f8a677fe700 1 -- 172.21.15.44:0/1660457031 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f8a680cf330 con 0x7f8a78101120 2024-04-21T23:07:18.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.521+0000 7f8a677fe700 1 --2- 172.21.15.44:0/1660457031 >> [v2:172.21.15.55:6816/3062702978,v1:172.21.15.55:6817/3062702978] conn(0x7f8a600ad970 0x7f8a600afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:18.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.521+0000 7f8a677fe700 1 -- 172.21.15.44:0/1660457031 --> [v2:172.21.15.55:6816/3062702978,v1:172.21.15.55:6817/3062702978] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8a68022e50 con 0x7f8a600ad970 2024-04-21T23:07:18.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.521+0000 7f8a677fe700 1 -- 172.21.15.44:0/1660457031 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f8a680cf780 con 0x7f8a78101120 2024-04-21T23:07:18.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.521+0000 7f8a75d9b700 1 --2- 172.21.15.44:0/1660457031 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f8a600a2cb0 0x7f8a600a5170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8a78199f10 tx=0x7f8a6c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:18.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.521+0000 7f8a76d9d700 1 --2- 172.21.15.44:0/1660457031 >> [v2:172.21.15.55:6816/3062702978,v1:172.21.15.55:6817/3062702978] conn(0x7f8a600ad970 0x7f8a600afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:18.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.522+0000 7f8a76d9d700 1 --2- 172.21.15.44:0/1660457031 >> [v2:172.21.15.55:6816/3062702978,v1:172.21.15.55:6817/3062702978] conn(0x7f8a600ad970 0x7f8a600afdb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:18.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.523+0000 7f8a677fe700 1 -- 172.21.15.44:0/1660457031 <== osd.7 v2:172.21.15.55:6816/3062702978 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8a68022e50 con 0x7f8a600ad970 2024-04-21T23:07:18.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.539+0000 7f8a7c8eb700 1 -- 172.21.15.44:0/1660457031 --> [v2:172.21.15.55:6816/3062702978,v1:172.21.15.55:6817/3062702978] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8a78062e00 con 0x7f8a600ad970 2024-04-21T23:07:18.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.540+0000 7f8a677fe700 1 -- 172.21.15.44:0/1660457031 <== osd.7 v2:172.21.15.55:6816/3062702978 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8a78062e00 con 0x7f8a600ad970 2024-04-21T23:07:18.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.540+0000 7f8a7c8eb700 1 -- 172.21.15.44:0/1660457031 >> [v2:172.21.15.55:6816/3062702978,v1:172.21.15.55:6817/3062702978] conn(0x7f8a600ad970 msgr2=0x7f8a600afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:18.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.540+0000 7f8a7c8eb700 1 --2- 172.21.15.44:0/1660457031 >> [v2:172.21.15.55:6816/3062702978,v1:172.21.15.55:6817/3062702978] conn(0x7f8a600ad970 0x7f8a600afdb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:18.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.540+0000 7f8a7c8eb700 1 -- 172.21.15.44:0/1660457031 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f8a600a2cb0 msgr2=0x7f8a600a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:18.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.540+0000 7f8a7c8eb700 1 --2- 172.21.15.44:0/1660457031 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f8a600a2cb0 0x7f8a600a5170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8a78199f10 tx=0x7f8a6c00a300 comp rx=0 tx=0).stop 2024-04-21T23:07:18.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.540+0000 7f8a7c8eb700 1 -- 172.21.15.44:0/1660457031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a78101120 msgr2=0x7f8a7819e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:18.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.540+0000 7f8a7c8eb700 1 --2- 172.21.15.44:0/1660457031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a78101120 0x7f8a7819e680 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800c920 tx=0x7f8a6800cc30 comp rx=0 tx=0).stop 2024-04-21T23:07:18.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.540+0000 7f8a76d9d700 1 -- 172.21.15.44:0/1660457031 reap_dead start 2024-04-21T23:07:18.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.540+0000 7f8a7c8eb700 1 -- 172.21.15.44:0/1660457031 shutdown_connections 2024-04-21T23:07:18.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.540+0000 7f8a7c8eb700 1 -- 172.21.15.44:0/1660457031 >> 172.21.15.44:0/1660457031 conn(0x7f8a780780a0 msgr2=0x7f8a780ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:18.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.540+0000 7f8a7c8eb700 1 -- 172.21.15.44:0/1660457031 shutdown_connections 2024-04-21T23:07:18.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:18.540+0000 7f8a7c8eb700 1 -- 172.21.15.44:0/1660457031 wait complete. 2024-04-21T23:07:18.629 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:18 smithi044 ceph-mon[27934]: pgmap v317: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-04-21T23:07:18.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:18 smithi069 ceph-mon[31816]: pgmap v317: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-04-21T23:07:18.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:18 smithi055 ceph-mon[31117]: pgmap v317: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-04-21T23:07:19.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.047+0000 7fd8967db700 1 -- 172.21.15.44:0/1698760850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8901075a0 msgr2=0x7fd89010f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:19.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.047+0000 7fd8967db700 1 --2- 172.21.15.44:0/1698760850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8901075a0 0x7fd89010f170 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd88800b0a0 tx=0x7fd88800b3b0 comp rx=0 tx=0).stop 2024-04-21T23:07:19.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.047+0000 7fd8967db700 1 -- 172.21.15.44:0/1698760850 shutdown_connections 2024-04-21T23:07:19.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.047+0000 7fd8967db700 1 --2- 172.21.15.44:0/1698760850 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8901075a0 0x7fd89010f170 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:19.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.047+0000 7fd8967db700 1 --2- 172.21.15.44:0/1698760850 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8901024e0 0x7fd8901028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:19.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.047+0000 7fd8967db700 1 --2- 172.21.15.44:0/1698760850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd890102e90 0x7fd890106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:19.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.047+0000 7fd8967db700 1 -- 172.21.15.44:0/1698760850 >> 172.21.15.44:0/1698760850 conn(0x7fd8900793e0 msgr2=0x7fd8900797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:19.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.047+0000 7fd8967db700 1 -- 172.21.15.44:0/1698760850 shutdown_connections 2024-04-21T23:07:19.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.048+0000 7fd8967db700 1 -- 172.21.15.44:0/1698760850 wait complete. 2024-04-21T23:07:19.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.048+0000 7fd8967db700 1 Processor -- start 2024-04-21T23:07:19.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.048+0000 7fd8967db700 1 -- start start 2024-04-21T23:07:19.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.048+0000 7fd8967db700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8901024e0 0x7fd89019e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:19.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.048+0000 7fd8967db700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd890102e90 0x7fd89019ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:19.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.048+0000 7fd8967db700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8901075a0 0x7fd890198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:19.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.048+0000 7fd8967db700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd8901125d0 con 0x7fd8901024e0 2024-04-21T23:07:19.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.048+0000 7fd8967db700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd890112450 con 0x7fd8901075a0 2024-04-21T23:07:19.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.048+0000 7fd8967db700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd890112750 con 0x7fd890102e90 2024-04-21T23:07:19.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.049+0000 7fd88ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8901024e0 0x7fd89019e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:19.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.049+0000 7fd88ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8901024e0 0x7fd89019e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34490/0 (socket says 172.21.15.44:34490) 2024-04-21T23:07:19.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.049+0000 7fd88ffff700 1 -- 172.21.15.44:0/278620006 learned_addr learned my addr 172.21.15.44:0/278620006 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:19.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.049+0000 7fd88f7fe700 1 --2- 172.21.15.44:0/278620006 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd890102e90 0x7fd89019ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:19.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.049+0000 7fd894d78700 1 --2- 172.21.15.44:0/278620006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8901075a0 0x7fd890198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:19.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.049+0000 7fd88ffff700 1 -- 172.21.15.44:0/278620006 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd890102e90 msgr2=0x7fd89019ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:19.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.049+0000 7fd88ffff700 1 --2- 172.21.15.44:0/278620006 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd890102e90 0x7fd89019ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:19.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.049+0000 7fd88ffff700 1 -- 172.21.15.44:0/278620006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8901075a0 msgr2=0x7fd890198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:19.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.049+0000 7fd88ffff700 1 --2- 172.21.15.44:0/278620006 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8901075a0 0x7fd890198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:19.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.049+0000 7fd88ffff700 1 -- 172.21.15.44:0/278620006 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd890199050 con 0x7fd8901024e0 2024-04-21T23:07:19.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.049+0000 7fd88ffff700 1 --2- 172.21.15.44:0/278620006 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8901024e0 0x7fd89019e6a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd88400d950 tx=0x7fd88400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:19.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.049+0000 7fd88d7fa700 1 -- 172.21.15.44:0/278620006 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd884016070 con 0x7fd8901024e0 2024-04-21T23:07:19.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.049+0000 7fd88d7fa700 1 -- 172.21.15.44:0/278620006 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd884009e20 con 0x7fd8901024e0 2024-04-21T23:07:19.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.049+0000 7fd88d7fa700 1 -- 172.21.15.44:0/278620006 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88400f620 con 0x7fd8901024e0 2024-04-21T23:07:19.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.049+0000 7fd8967db700 1 -- 172.21.15.44:0/278620006 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd890199340 con 0x7fd8901024e0 2024-04-21T23:07:19.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.050+0000 7fd8967db700 1 -- 172.21.15.44:0/278620006 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8901a5620 con 0x7fd8901024e0 2024-04-21T23:07:19.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.050+0000 7fd8967db700 1 -- 172.21.15.44:0/278620006 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd890100c10 con 0x7fd8901024e0 2024-04-21T23:07:19.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.052+0000 7fd88d7fa700 1 -- 172.21.15.44:0/278620006 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd884019070 con 0x7fd8901024e0 2024-04-21T23:07:19.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.052+0000 7fd88d7fa700 1 --2- 172.21.15.44:0/278620006 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd8780a2d30 0x7fd8780a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:19.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.052+0000 7fd88f7fe700 1 --2- 172.21.15.44:0/278620006 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd8780a2d30 0x7fd8780a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:19.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.052+0000 7fd88d7fa700 1 -- 172.21.15.44:0/278620006 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fd8840cf440 con 0x7fd8901024e0 2024-04-21T23:07:19.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.053+0000 7fd88d7fa700 1 --2- 172.21.15.44:0/278620006 >> [v2:172.21.15.55:6824/3066586400,v1:172.21.15.55:6825/3066586400] conn(0x7fd8780ad9f0 0x7fd8780afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:19.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.053+0000 7fd88d7fa700 1 -- 172.21.15.44:0/278620006 --> [v2:172.21.15.55:6824/3066586400,v1:172.21.15.55:6825/3066586400] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd884010e50 con 0x7fd8780ad9f0 2024-04-21T23:07:19.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.053+0000 7fd88d7fa700 1 -- 172.21.15.44:0/278620006 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fd8840cf890 con 0x7fd8901024e0 2024-04-21T23:07:19.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.053+0000 7fd88f7fe700 1 --2- 172.21.15.44:0/278620006 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd8780a2d30 0x7fd8780a51f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd890199f30 tx=0x7fd88000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:19.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.053+0000 7fd894d78700 1 --2- 172.21.15.44:0/278620006 >> [v2:172.21.15.55:6824/3066586400,v1:172.21.15.55:6825/3066586400] conn(0x7fd8780ad9f0 0x7fd8780afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:19.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.053+0000 7fd894d78700 1 --2- 172.21.15.44:0/278620006 >> [v2:172.21.15.55:6824/3066586400,v1:172.21.15.55:6825/3066586400] conn(0x7fd8780ad9f0 0x7fd8780afe30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:19.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.054+0000 7fd88d7fa700 1 -- 172.21.15.44:0/278620006 <== osd.10 v2:172.21.15.55:6824/3066586400 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd884010e50 con 0x7fd8780ad9f0 2024-04-21T23:07:19.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.071+0000 7fd8967db700 1 -- 172.21.15.44:0/278620006 --> [v2:172.21.15.55:6824/3066586400,v1:172.21.15.55:6825/3066586400] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd8901039a0 con 0x7fd8780ad9f0 2024-04-21T23:07:19.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.071+0000 7fd88d7fa700 1 -- 172.21.15.44:0/278620006 <== osd.10 v2:172.21.15.55:6824/3066586400 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd8901039a0 con 0x7fd8780ad9f0 2024-04-21T23:07:19.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.072+0000 7fd8967db700 1 -- 172.21.15.44:0/278620006 >> [v2:172.21.15.55:6824/3066586400,v1:172.21.15.55:6825/3066586400] conn(0x7fd8780ad9f0 msgr2=0x7fd8780afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:19.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.072+0000 7fd8967db700 1 --2- 172.21.15.44:0/278620006 >> [v2:172.21.15.55:6824/3066586400,v1:172.21.15.55:6825/3066586400] conn(0x7fd8780ad9f0 0x7fd8780afe30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:19.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.072+0000 7fd8967db700 1 -- 172.21.15.44:0/278620006 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd8780a2d30 msgr2=0x7fd8780a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:19.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.072+0000 7fd8967db700 1 --2- 172.21.15.44:0/278620006 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd8780a2d30 0x7fd8780a51f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd890199f30 tx=0x7fd88000a380 comp rx=0 tx=0).stop 2024-04-21T23:07:19.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.072+0000 7fd8967db700 1 -- 172.21.15.44:0/278620006 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8901024e0 msgr2=0x7fd89019e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:19.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.072+0000 7fd8967db700 1 --2- 172.21.15.44:0/278620006 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8901024e0 0x7fd89019e6a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd88400d950 tx=0x7fd88400dc60 comp rx=0 tx=0).stop 2024-04-21T23:07:19.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.072+0000 7fd894d78700 1 -- 172.21.15.44:0/278620006 reap_dead start 2024-04-21T23:07:19.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.072+0000 7fd8967db700 1 -- 172.21.15.44:0/278620006 shutdown_connections 2024-04-21T23:07:19.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.072+0000 7fd8967db700 1 -- 172.21.15.44:0/278620006 >> 172.21.15.44:0/278620006 conn(0x7fd8900793e0 msgr2=0x7fd8901001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:19.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.072+0000 7fd8967db700 1 -- 172.21.15.44:0/278620006 shutdown_connections 2024-04-21T23:07:19.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:19.072+0000 7fd8967db700 1 -- 172.21.15.44:0/278620006 wait complete. 2024-04-21T23:07:20.313 INFO:teuthology.orchestra.run.smithi044.stdout:313532612637 2024-04-21T23:07:20.313 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-21T23:07:20.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:20 smithi044 ceph-mon[27934]: pgmap v318: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-04-21T23:07:20.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:20 smithi055 ceph-mon[31117]: pgmap v318: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-04-21T23:07:20.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:20 smithi069 ceph-mon[31816]: pgmap v318: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-04-21T23:07:22.672 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:22 smithi044 ceph-mon[27934]: pgmap v319: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-04-21T23:07:22.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:22 smithi055 ceph-mon[31117]: pgmap v319: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-04-21T23:07:22.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:22 smithi069 ceph-mon[31816]: pgmap v319: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-04-21T23:07:23.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.349+0000 7fc2944af700 1 -- 172.21.15.44:0/470170450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc28c107670 msgr2=0x7fc28c10f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:23.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.349+0000 7fc2944af700 1 --2- 172.21.15.44:0/470170450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc28c107670 0x7fc28c10f1d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc28400b0d0 tx=0x7fc28400b3e0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.350+0000 7fc2944af700 1 -- 172.21.15.44:0/470170450 shutdown_connections 2024-04-21T23:07:23.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.350+0000 7fc2944af700 1 --2- 172.21.15.44:0/470170450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc28c107670 0x7fc28c10f1d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.350+0000 7fc2944af700 1 --2- 172.21.15.44:0/470170450 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc28c100e50 0x7fc28c101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.350+0000 7fc2944af700 1 --2- 172.21.15.44:0/470170450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc28c101800 0x7fc28c106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.350+0000 7fc2944af700 1 -- 172.21.15.44:0/470170450 >> 172.21.15.44:0/470170450 conn(0x7fc28c076450 msgr2=0x7fc28c076860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:23.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.350+0000 7fc2944af700 1 -- 172.21.15.44:0/470170450 shutdown_connections 2024-04-21T23:07:23.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.350+0000 7fc2944af700 1 -- 172.21.15.44:0/470170450 wait complete. 2024-04-21T23:07:23.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.351+0000 7fc2944af700 1 Processor -- start 2024-04-21T23:07:23.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.351+0000 7fc2944af700 1 -- start start 2024-04-21T23:07:23.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.351+0000 7fc2944af700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc28c100e50 0x7fc28c19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:23.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.351+0000 7fc2944af700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc28c101800 0x7fc28c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:23.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.351+0000 7fc2944af700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc28c107670 0x7fc28c198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:23.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.351+0000 7fc2944af700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc28c112570 con 0x7fc28c101800 2024-04-21T23:07:23.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.351+0000 7fc2944af700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc28c1123f0 con 0x7fc28c100e50 2024-04-21T23:07:23.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.351+0000 7fc2944af700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc28c1126f0 con 0x7fc28c107670 2024-04-21T23:07:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.351+0000 7fc291a4a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc28c101800 0x7fc28c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.351+0000 7fc291a4a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc28c101800 0x7fc28c19eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53538/0 (socket says 172.21.15.44:53538) 2024-04-21T23:07:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.351+0000 7fc291a4a700 1 -- 172.21.15.44:0/1820256794 learned_addr learned my addr 172.21.15.44:0/1820256794 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.351+0000 7fc29224b700 1 --2- 172.21.15.44:0/1820256794 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc28c100e50 0x7fc28c19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:23.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.351+0000 7fc292a4c700 1 --2- 172.21.15.44:0/1820256794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc28c107670 0x7fc28c198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:23.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.351+0000 7fc291a4a700 1 -- 172.21.15.44:0/1820256794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc28c107670 msgr2=0x7fc28c198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:23.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.351+0000 7fc291a4a700 1 --2- 172.21.15.44:0/1820256794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc28c107670 0x7fc28c198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.351+0000 7fc291a4a700 1 -- 172.21.15.44:0/1820256794 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc28c100e50 msgr2=0x7fc28c19e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:23.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.351+0000 7fc291a4a700 1 --2- 172.21.15.44:0/1820256794 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc28c100e50 0x7fc28c19e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.351+0000 7fc291a4a700 1 -- 172.21.15.44:0/1820256794 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc28c199050 con 0x7fc28c101800 2024-04-21T23:07:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.351+0000 7fc29224b700 1 --2- 172.21.15.44:0/1820256794 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc28c100e50 0x7fc28c19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:07:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.352+0000 7fc291a4a700 1 --2- 172.21.15.44:0/1820256794 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc28c101800 0x7fc28c19eb90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc27c00d950 tx=0x7fc27c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.352+0000 7fc2837fe700 1 -- 172.21.15.44:0/1820256794 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27c016070 con 0x7fc28c101800 2024-04-21T23:07:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.352+0000 7fc2944af700 1 -- 172.21.15.44:0/1820256794 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc28c199340 con 0x7fc28c101800 2024-04-21T23:07:23.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.352+0000 7fc2837fe700 1 -- 172.21.15.44:0/1820256794 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc27c00fd70 con 0x7fc28c101800 2024-04-21T23:07:23.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.352+0000 7fc2837fe700 1 -- 172.21.15.44:0/1820256794 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27c00e610 con 0x7fc28c101800 2024-04-21T23:07:23.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.352+0000 7fc2944af700 1 -- 172.21.15.44:0/1820256794 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc28c1a55d0 con 0x7fc28c101800 2024-04-21T23:07:23.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.353+0000 7fc2944af700 1 -- 172.21.15.44:0/1820256794 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc28c062980 con 0x7fc28c101800 2024-04-21T23:07:23.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.354+0000 7fc2837fe700 1 -- 172.21.15.44:0/1820256794 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc27c019070 con 0x7fc28c101800 2024-04-21T23:07:23.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.355+0000 7fc2837fe700 1 --2- 172.21.15.44:0/1820256794 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc2780a2cb0 0x7fc2780a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:23.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.355+0000 7fc2837fe700 1 -- 172.21.15.44:0/1820256794 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc27c0cf3c0 con 0x7fc28c101800 2024-04-21T23:07:23.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.355+0000 7fc29224b700 1 --2- 172.21.15.44:0/1820256794 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc2780a2cb0 0x7fc2780a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:23.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.355+0000 7fc2837fe700 1 --2- 172.21.15.44:0/1820256794 >> [v2:172.21.15.69:6808/2597202863,v1:172.21.15.69:6809/2597202863] conn(0x7fc2780ad970 0x7fc2780afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:23.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.355+0000 7fc2837fe700 1 -- 172.21.15.44:0/1820256794 --> [v2:172.21.15.69:6808/2597202863,v1:172.21.15.69:6809/2597202863] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc2780b0480 con 0x7fc2780ad970 2024-04-21T23:07:23.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.355+0000 7fc2837fe700 1 -- 172.21.15.44:0/1820256794 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fc27c0cf810 con 0x7fc28c101800 2024-04-21T23:07:23.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.356+0000 7fc292a4c700 1 --2- 172.21.15.44:0/1820256794 >> [v2:172.21.15.69:6808/2597202863,v1:172.21.15.69:6809/2597202863] conn(0x7fc2780ad970 0x7fc2780afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:23.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.356+0000 7fc29224b700 1 --2- 172.21.15.44:0/1820256794 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc2780a2cb0 0x7fc2780a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc288005950 tx=0x7fc28800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:23.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.356+0000 7fc292a4c700 1 --2- 172.21.15.44:0/1820256794 >> [v2:172.21.15.69:6808/2597202863,v1:172.21.15.69:6809/2597202863] conn(0x7fc2780ad970 0x7fc2780afdb0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:23.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.357+0000 7fc2837fe700 1 -- 172.21.15.44:0/1820256794 <== osd.5 v2:172.21.15.69:6808/2597202863 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc2780b0480 con 0x7fc2780ad970 2024-04-21T23:07:23.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.374+0000 7fc2944af700 1 -- 172.21.15.44:0/1820256794 --> [v2:172.21.15.69:6808/2597202863,v1:172.21.15.69:6809/2597202863] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc28c062e00 con 0x7fc2780ad970 2024-04-21T23:07:23.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.374+0000 7fc2837fe700 1 -- 172.21.15.44:0/1820256794 <== osd.5 v2:172.21.15.69:6808/2597202863 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc28c062e00 con 0x7fc2780ad970 2024-04-21T23:07:23.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.375+0000 7fc2944af700 1 -- 172.21.15.44:0/1820256794 >> [v2:172.21.15.69:6808/2597202863,v1:172.21.15.69:6809/2597202863] conn(0x7fc2780ad970 msgr2=0x7fc2780afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:23.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.375+0000 7fc2944af700 1 --2- 172.21.15.44:0/1820256794 >> [v2:172.21.15.69:6808/2597202863,v1:172.21.15.69:6809/2597202863] conn(0x7fc2780ad970 0x7fc2780afdb0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.375+0000 7fc2944af700 1 -- 172.21.15.44:0/1820256794 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc2780a2cb0 msgr2=0x7fc2780a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:23.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.375+0000 7fc2944af700 1 --2- 172.21.15.44:0/1820256794 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc2780a2cb0 0x7fc2780a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc288005950 tx=0x7fc28800b410 comp rx=0 tx=0).stop 2024-04-21T23:07:23.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.375+0000 7fc2944af700 1 -- 172.21.15.44:0/1820256794 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc28c101800 msgr2=0x7fc28c19eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:23.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.375+0000 7fc2944af700 1 --2- 172.21.15.44:0/1820256794 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc28c101800 0x7fc28c19eb90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc27c00d950 tx=0x7fc27c00dc60 comp rx=0 tx=0).stop 2024-04-21T23:07:23.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.375+0000 7fc292a4c700 1 -- 172.21.15.44:0/1820256794 reap_dead start 2024-04-21T23:07:23.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.375+0000 7fc2944af700 1 -- 172.21.15.44:0/1820256794 shutdown_connections 2024-04-21T23:07:23.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.375+0000 7fc2944af700 1 -- 172.21.15.44:0/1820256794 >> 172.21.15.44:0/1820256794 conn(0x7fc28c076450 msgr2=0x7fc28c0ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:23.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.375+0000 7fc2944af700 1 -- 172.21.15.44:0/1820256794 shutdown_connections 2024-04-21T23:07:23.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:23.375+0000 7fc2944af700 1 -- 172.21.15.44:0/1820256794 wait complete. 2024-04-21T23:07:23.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.711+0000 7f58896f3700 1 -- 172.21.15.44:0/3563078785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5884100c60 msgr2=0x7f58841010e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:23.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.711+0000 7f58896f3700 1 --2- 172.21.15.44:0/3563078785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5884100c60 0x7f58841010e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5878009990 tx=0x7f5878009ca0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.711+0000 7f58896f3700 1 -- 172.21.15.44:0/3563078785 shutdown_connections 2024-04-21T23:07:23.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.711+0000 7f58896f3700 1 --2- 172.21.15.44:0/3563078785 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5884101620 0x7f588410f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.711+0000 7f58896f3700 1 --2- 172.21.15.44:0/3563078785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5884108410 0x7f58841087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.711+0000 7f58896f3700 1 --2- 172.21.15.44:0/3563078785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5884100c60 0x7f58841010e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.711+0000 7f58896f3700 1 -- 172.21.15.44:0/3563078785 >> 172.21.15.44:0/3563078785 conn(0x7f58840763a0 msgr2=0x7f58840767b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:23.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.711+0000 7f58896f3700 1 -- 172.21.15.44:0/3563078785 shutdown_connections 2024-04-21T23:07:23.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.711+0000 7f58896f3700 1 -- 172.21.15.44:0/3563078785 wait complete. 2024-04-21T23:07:23.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.712+0000 7f58896f3700 1 Processor -- start 2024-04-21T23:07:23.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.712+0000 7f58896f3700 1 -- start start 2024-04-21T23:07:23.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.712+0000 7f58896f3700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5884100c60 0x7f588419e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:23.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.712+0000 7f58896f3700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5884101620 0x7f588419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:23.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.712+0000 7f58896f3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5884108410 0x7f58841987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:23.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.712+0000 7f58896f3700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5884112530 con 0x7f5884100c60 2024-04-21T23:07:23.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.712+0000 7f58896f3700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f58841123b0 con 0x7f5884101620 2024-04-21T23:07:23.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.712+0000 7f58896f3700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f58841126b0 con 0x7f5884108410 2024-04-21T23:07:23.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.712+0000 7f5882ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5884100c60 0x7f588419e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:23.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.712+0000 7f5882ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5884100c60 0x7f588419e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53498/0 (socket says 172.21.15.44:53498) 2024-04-21T23:07:23.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.712+0000 7f5882ffd700 1 -- 172.21.15.44:0/2794670690 learned_addr learned my addr 172.21.15.44:0/2794670690 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:23.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.712+0000 7f58837fe700 1 --2- 172.21.15.44:0/2794670690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5884108410 0x7f58841987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:23.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.712+0000 7f58827fc700 1 --2- 172.21.15.44:0/2794670690 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5884101620 0x7f588419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:23.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.713+0000 7f5882ffd700 1 -- 172.21.15.44:0/2794670690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5884108410 msgr2=0x7f58841987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:23.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.713+0000 7f5882ffd700 1 --2- 172.21.15.44:0/2794670690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5884108410 0x7f58841987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.713+0000 7f5882ffd700 1 -- 172.21.15.44:0/2794670690 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5884101620 msgr2=0x7f588419eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:23.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.713+0000 7f5882ffd700 1 --2- 172.21.15.44:0/2794670690 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5884101620 0x7f588419eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.713+0000 7f5882ffd700 1 -- 172.21.15.44:0/2794670690 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5884198f90 con 0x7f5884100c60 2024-04-21T23:07:23.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.713+0000 7f58837fe700 1 --2- 172.21.15.44:0/2794670690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5884108410 0x7f58841987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:07:23.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.713+0000 7f5882ffd700 1 --2- 172.21.15.44:0/2794670690 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5884100c60 0x7f588419e630 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f587400c970 tx=0x7f587400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:23.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.713+0000 7f586bfff700 1 -- 172.21.15.44:0/2794670690 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5874007aa0 con 0x7f5884100c60 2024-04-21T23:07:23.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.713+0000 7f58896f3700 1 -- 172.21.15.44:0/2794670690 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5884199280 con 0x7f5884100c60 2024-04-21T23:07:23.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.713+0000 7f586bfff700 1 -- 172.21.15.44:0/2794670690 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5874004d10 con 0x7f5884100c60 2024-04-21T23:07:23.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.713+0000 7f586bfff700 1 -- 172.21.15.44:0/2794670690 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5874005690 con 0x7f5884100c60 2024-04-21T23:07:23.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.713+0000 7f58896f3700 1 -- 172.21.15.44:0/2794670690 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58841a5490 con 0x7f5884100c60 2024-04-21T23:07:23.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.714+0000 7f58896f3700 1 -- 172.21.15.44:0/2794670690 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f58840629c0 con 0x7f5884100c60 2024-04-21T23:07:23.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.715+0000 7f586bfff700 1 -- 172.21.15.44:0/2794670690 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5874004750 con 0x7f5884100c60 2024-04-21T23:07:23.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.716+0000 7f586bfff700 1 --2- 172.21.15.44:0/2794670690 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f586c0a2cb0 0x7f586c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:23.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.716+0000 7f58827fc700 1 --2- 172.21.15.44:0/2794670690 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f586c0a2cb0 0x7f586c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:23.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.716+0000 7f586bfff700 1 -- 172.21.15.44:0/2794670690 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f58740cf280 con 0x7f5884100c60 2024-04-21T23:07:23.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.716+0000 7f586bfff700 1 --2- 172.21.15.44:0/2794670690 >> [v2:172.21.15.55:6800/1988017680,v1:172.21.15.55:6801/1988017680] conn(0x7f586c0ad970 0x7f586c0afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:23.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.716+0000 7f586bfff700 1 -- 172.21.15.44:0/2794670690 --> [v2:172.21.15.55:6800/1988017680,v1:172.21.15.55:6801/1988017680] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f586c0b0480 con 0x7f586c0ad970 2024-04-21T23:07:23.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.716+0000 7f586bfff700 1 -- 172.21.15.44:0/2794670690 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f58740d6050 con 0x7f5884100c60 2024-04-21T23:07:23.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.716+0000 7f58827fc700 1 --2- 172.21.15.44:0/2794670690 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f586c0a2cb0 0x7f586c0a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5878009750 tx=0x7f587800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:23.548 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.716+0000 7f58837fe700 1 --2- 172.21.15.44:0/2794670690 >> [v2:172.21.15.55:6800/1988017680,v1:172.21.15.55:6801/1988017680] conn(0x7f586c0ad970 0x7f586c0afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:23.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.717+0000 7f58837fe700 1 --2- 172.21.15.44:0/2794670690 >> [v2:172.21.15.55:6800/1988017680,v1:172.21.15.55:6801/1988017680] conn(0x7f586c0ad970 0x7f586c0afdb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:23.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.718+0000 7f586bfff700 1 -- 172.21.15.44:0/2794670690 <== osd.1 v2:172.21.15.55:6800/1988017680 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f586c0b0480 con 0x7f586c0ad970 2024-04-21T23:07:23.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.736+0000 7f58896f3700 1 -- 172.21.15.44:0/2794670690 --> [v2:172.21.15.55:6800/1988017680,v1:172.21.15.55:6801/1988017680] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5884062e70 con 0x7f586c0ad970 2024-04-21T23:07:23.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.737+0000 7f586bfff700 1 -- 172.21.15.44:0/2794670690 <== osd.1 v2:172.21.15.55:6800/1988017680 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5884062e70 con 0x7f586c0ad970 2024-04-21T23:07:23.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.737+0000 7f58896f3700 1 -- 172.21.15.44:0/2794670690 >> [v2:172.21.15.55:6800/1988017680,v1:172.21.15.55:6801/1988017680] conn(0x7f586c0ad970 msgr2=0x7f586c0afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:23.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.737+0000 7f58896f3700 1 --2- 172.21.15.44:0/2794670690 >> [v2:172.21.15.55:6800/1988017680,v1:172.21.15.55:6801/1988017680] conn(0x7f586c0ad970 0x7f586c0afdb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.737+0000 7f58896f3700 1 -- 172.21.15.44:0/2794670690 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f586c0a2cb0 msgr2=0x7f586c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:23.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.737+0000 7f58896f3700 1 --2- 172.21.15.44:0/2794670690 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f586c0a2cb0 0x7f586c0a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5878009750 tx=0x7f587800b540 comp rx=0 tx=0).stop 2024-04-21T23:07:23.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.737+0000 7f58896f3700 1 -- 172.21.15.44:0/2794670690 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5884100c60 msgr2=0x7f588419e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:23.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.737+0000 7f58896f3700 1 --2- 172.21.15.44:0/2794670690 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5884100c60 0x7f588419e630 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f587400c970 tx=0x7f587400cd30 comp rx=0 tx=0).stop 2024-04-21T23:07:23.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.737+0000 7f58837fe700 1 -- 172.21.15.44:0/2794670690 reap_dead start 2024-04-21T23:07:23.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.737+0000 7f58896f3700 1 -- 172.21.15.44:0/2794670690 shutdown_connections 2024-04-21T23:07:23.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.737+0000 7f58896f3700 1 -- 172.21.15.44:0/2794670690 >> 172.21.15.44:0/2794670690 conn(0x7f58840763a0 msgr2=0x7f58840ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:23.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.738+0000 7f58896f3700 1 -- 172.21.15.44:0/2794670690 shutdown_connections 2024-04-21T23:07:23.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:21.738+0000 7f58896f3700 1 -- 172.21.15.44:0/2794670690 wait complete. 2024-04-21T23:07:23.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-21T23:07:23.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-04-21T23:07:23.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-04-21T23:07:23.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-04-21T23:07:23.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-21T23:07:23.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-21T23:07:23.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-04-21T23:07:23.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11, 5, 8]}]: dispatch 2024-04-21T23:07:23.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-04-21T23:07:23.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-21T23:07:23.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-04-21T23:07:23.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-04-21T23:07:23.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-04-21T23:07:23.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-21T23:07:23.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-21T23:07:23.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-04-21T23:07:23.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11, 5, 8]}]: dispatch 2024-04-21T23:07:23.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-04-21T23:07:23.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.626+0000 7f8b5dad4700 1 -- 172.21.15.44:0/2797422700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b581013c0 msgr2=0x7f8b581017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:23.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.626+0000 7f8b5dad4700 1 --2- 172.21.15.44:0/2797422700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b581013c0 0x7f8b581017a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8b480099c0 tx=0x7f8b48009cd0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.626+0000 7f8b5dad4700 1 -- 172.21.15.44:0/2797422700 shutdown_connections 2024-04-21T23:07:23.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.626+0000 7f8b5dad4700 1 --2- 172.21.15.44:0/2797422700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b58106450 0x7f8b5810e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.626+0000 7f8b5dad4700 1 --2- 172.21.15.44:0/2797422700 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b581013c0 0x7f8b581017a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.626+0000 7f8b5dad4700 1 --2- 172.21.15.44:0/2797422700 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b58101d70 0x7f8b58105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.626+0000 7f8b5dad4700 1 -- 172.21.15.44:0/2797422700 >> 172.21.15.44:0/2797422700 conn(0x7f8b580780e0 msgr2=0x7f8b580784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:23.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.626+0000 7f8b5dad4700 1 -- 172.21.15.44:0/2797422700 shutdown_connections 2024-04-21T23:07:23.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.626+0000 7f8b5dad4700 1 -- 172.21.15.44:0/2797422700 wait complete. 2024-04-21T23:07:23.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.627+0000 7f8b5dad4700 1 Processor -- start 2024-04-21T23:07:23.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.627+0000 7f8b5dad4700 1 -- start start 2024-04-21T23:07:23.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.627+0000 7f8b5dad4700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b581013c0 0x7f8b58199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:23.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.627+0000 7f8b5dad4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b58101d70 0x7f8b58199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:23.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.627+0000 7f8b5dad4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b58106450 0x7f8b581a4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:23.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.627+0000 7f8b5dad4700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8b581114b0 con 0x7f8b581013c0 2024-04-21T23:07:23.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.627+0000 7f8b5dad4700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8b58111330 con 0x7f8b58106450 2024-04-21T23:07:23.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.627+0000 7f8b5dad4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8b58111630 con 0x7f8b58101d70 2024-04-21T23:07:23.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.628+0000 7f8b577fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b581013c0 0x7f8b58199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:23.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.628+0000 7f8b577fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b581013c0 0x7f8b58199910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53518/0 (socket says 172.21.15.44:53518) 2024-04-21T23:07:23.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.628+0000 7f8b577fe700 1 -- 172.21.15.44:0/2084616646 learned_addr learned my addr 172.21.15.44:0/2084616646 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:23.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.628+0000 7f8b57fff700 1 --2- 172.21.15.44:0/2084616646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b58106450 0x7f8b581a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:23.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.628+0000 7f8b56ffd700 1 --2- 172.21.15.44:0/2084616646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b58101d70 0x7f8b58199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:23.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.628+0000 7f8b577fe700 1 -- 172.21.15.44:0/2084616646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b58101d70 msgr2=0x7f8b58199e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:23.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.628+0000 7f8b577fe700 1 --2- 172.21.15.44:0/2084616646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b58101d70 0x7f8b58199e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.628+0000 7f8b577fe700 1 -- 172.21.15.44:0/2084616646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b58106450 msgr2=0x7f8b581a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:23.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.628+0000 7f8b577fe700 1 --2- 172.21.15.44:0/2084616646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8b58106450 0x7f8b581a4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.628+0000 7f8b577fe700 1 -- 172.21.15.44:0/2084616646 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b581a4ed0 con 0x7f8b581013c0 2024-04-21T23:07:23.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.628+0000 7f8b56ffd700 1 --2- 172.21.15.44:0/2084616646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b58101d70 0x7f8b58199e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:07:23.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.628+0000 7f8b577fe700 1 --2- 172.21.15.44:0/2084616646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b581013c0 0x7f8b58199910 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8b48006010 tx=0x7f8b48009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:23.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.628+0000 7f8b54ff9700 1 -- 172.21.15.44:0/2084616646 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b48027070 con 0x7f8b581013c0 2024-04-21T23:07:23.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.628+0000 7f8b5dad4700 1 -- 172.21.15.44:0/2084616646 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b581a50a0 con 0x7f8b581013c0 2024-04-21T23:07:23.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.628+0000 7f8b54ff9700 1 -- 172.21.15.44:0/2084616646 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b48003780 con 0x7f8b581013c0 2024-04-21T23:07:23.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.628+0000 7f8b54ff9700 1 -- 172.21.15.44:0/2084616646 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b48022da0 con 0x7f8b581013c0 2024-04-21T23:07:23.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.628+0000 7f8b5dad4700 1 -- 172.21.15.44:0/2084616646 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b581a5550 con 0x7f8b581013c0 2024-04-21T23:07:23.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.629+0000 7f8b5dad4700 1 -- 172.21.15.44:0/2084616646 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8b58062940 con 0x7f8b581013c0 2024-04-21T23:07:23.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.631+0000 7f8b54ff9700 1 -- 172.21.15.44:0/2084616646 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f8b4802a070 con 0x7f8b581013c0 2024-04-21T23:07:23.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.632+0000 7f8b54ff9700 1 --2- 172.21.15.44:0/2084616646 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f8b400a2cb0 0x7f8b400a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:23.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.632+0000 7f8b56ffd700 1 --2- 172.21.15.44:0/2084616646 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f8b400a2cb0 0x7f8b400a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:23.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.632+0000 7f8b54ff9700 1 -- 172.21.15.44:0/2084616646 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f8b480d0610 con 0x7f8b581013c0 2024-04-21T23:07:23.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.632+0000 7f8b54ff9700 1 --2- 172.21.15.44:0/2084616646 >> [v2:172.21.15.69:6824/1593504610,v1:172.21.15.69:6825/1593504610] conn(0x7f8b400ad970 0x7f8b400afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:23.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.632+0000 7f8b54ff9700 1 -- 172.21.15.44:0/2084616646 --> [v2:172.21.15.69:6824/1593504610,v1:172.21.15.69:6825/1593504610] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8b480043d0 con 0x7f8b400ad970 2024-04-21T23:07:23.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.632+0000 7f8b54ff9700 1 -- 172.21.15.44:0/2084616646 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f8b480d0a50 con 0x7f8b581013c0 2024-04-21T23:07:23.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.632+0000 7f8b56ffd700 1 --2- 172.21.15.44:0/2084616646 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f8b400a2cb0 0x7f8b400a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8b58107960 tx=0x7f8b4c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:23.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.632+0000 7f8b57fff700 1 --2- 172.21.15.44:0/2084616646 >> [v2:172.21.15.69:6824/1593504610,v1:172.21.15.69:6825/1593504610] conn(0x7f8b400ad970 0x7f8b400afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:23.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.633+0000 7f8b57fff700 1 --2- 172.21.15.44:0/2084616646 >> [v2:172.21.15.69:6824/1593504610,v1:172.21.15.69:6825/1593504610] conn(0x7f8b400ad970 0x7f8b400afdb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:23.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.634+0000 7f8b54ff9700 1 -- 172.21.15.44:0/2084616646 <== osd.11 v2:172.21.15.69:6824/1593504610 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8b480043d0 con 0x7f8b400ad970 2024-04-21T23:07:23.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.660+0000 7f8b5dad4700 1 -- 172.21.15.44:0/2084616646 --> [v2:172.21.15.69:6824/1593504610,v1:172.21.15.69:6825/1593504610] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8b58062dc0 con 0x7f8b400ad970 2024-04-21T23:07:23.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.661+0000 7f8b54ff9700 1 -- 172.21.15.44:0/2084616646 <== osd.11 v2:172.21.15.69:6824/1593504610 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8b58062dc0 con 0x7f8b400ad970 2024-04-21T23:07:23.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.661+0000 7f8b5dad4700 1 -- 172.21.15.44:0/2084616646 >> [v2:172.21.15.69:6824/1593504610,v1:172.21.15.69:6825/1593504610] conn(0x7f8b400ad970 msgr2=0x7f8b400afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:23.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.661+0000 7f8b5dad4700 1 --2- 172.21.15.44:0/2084616646 >> [v2:172.21.15.69:6824/1593504610,v1:172.21.15.69:6825/1593504610] conn(0x7f8b400ad970 0x7f8b400afdb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:23.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.661+0000 7f8b5dad4700 1 -- 172.21.15.44:0/2084616646 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f8b400a2cb0 msgr2=0x7f8b400a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:23.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.661+0000 7f8b5dad4700 1 --2- 172.21.15.44:0/2084616646 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f8b400a2cb0 0x7f8b400a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8b58107960 tx=0x7f8b4c008040 comp rx=0 tx=0).stop 2024-04-21T23:07:23.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.662+0000 7f8b5dad4700 1 -- 172.21.15.44:0/2084616646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b581013c0 msgr2=0x7f8b58199910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:23.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.662+0000 7f8b5dad4700 1 --2- 172.21.15.44:0/2084616646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8b581013c0 0x7f8b58199910 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8b48006010 tx=0x7f8b48009f80 comp rx=0 tx=0).stop 2024-04-21T23:07:23.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.662+0000 7f8b57fff700 1 -- 172.21.15.44:0/2084616646 reap_dead start 2024-04-21T23:07:23.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.662+0000 7f8b5dad4700 1 -- 172.21.15.44:0/2084616646 shutdown_connections 2024-04-21T23:07:23.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.662+0000 7f8b5dad4700 1 -- 172.21.15.44:0/2084616646 >> 172.21.15.44:0/2084616646 conn(0x7f8b580780e0 msgr2=0x7f8b58100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:23.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.662+0000 7f8b5dad4700 1 -- 172.21.15.44:0/2084616646 shutdown_connections 2024-04-21T23:07:23.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:22.662+0000 7f8b5dad4700 1 -- 172.21.15.44:0/2084616646 wait complete. 2024-04-21T23:07:23.861 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-21T23:07:23.861 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-04-21T23:07:23.861 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-04-21T23:07:23.861 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-04-21T23:07:23.861 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-04-21T23:07:23.861 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-21T23:07:23.861 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-04-21T23:07:23.861 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11, 5, 8]}]: dispatch 2024-04-21T23:07:23.861 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-04-21T23:07:24.654 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:24 smithi044 ceph-mon[27934]: pgmap v320: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:07:24.654 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-21T23:07:24.654 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-04-21T23:07:24.654 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-04-21T23:07:24.654 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-04-21T23:07:24.655 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-21T23:07:24.655 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-21T23:07:24.655 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-04-21T23:07:24.655 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11, 5, 8]}]': finished 2024-04-21T23:07:24.655 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-04-21T23:07:24.655 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:24 smithi044 ceph-mon[27934]: osdmap e91: 12 total, 12 up, 12 in 2024-04-21T23:07:24.656 INFO:teuthology.orchestra.run.smithi044.stdout:55834574923 2024-04-21T23:07:24.656 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-21T23:07:24.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:24 smithi055 ceph-mon[31117]: pgmap v320: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:07:24.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-21T23:07:24.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-04-21T23:07:24.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-04-21T23:07:24.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-04-21T23:07:24.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-21T23:07:24.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-21T23:07:24.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-04-21T23:07:24.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11, 5, 8]}]': finished 2024-04-21T23:07:24.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-04-21T23:07:24.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:24 smithi055 ceph-mon[31117]: osdmap e91: 12 total, 12 up, 12 in 2024-04-21T23:07:24.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:24 smithi069 ceph-mon[31816]: pgmap v320: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:07:24.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-21T23:07:24.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-04-21T23:07:24.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-04-21T23:07:24.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-04-21T23:07:24.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-04-21T23:07:24.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-21T23:07:24.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-04-21T23:07:24.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11, 5, 8]}]': finished 2024-04-21T23:07:24.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-04-21T23:07:24.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:24 smithi069 ceph-mon[31816]: osdmap e91: 12 total, 12 up, 12 in 2024-04-21T23:07:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.849+0000 7f19835b4700 1 -- 172.21.15.44:0/776275342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f197c1071a0 msgr2=0x7f197c107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.849+0000 7f19835b4700 1 --2- 172.21.15.44:0/776275342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f197c1071a0 0x7f197c107580 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1978009990 tx=0x7f1978009ca0 comp rx=0 tx=0).stop 2024-04-21T23:07:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.849+0000 7f19835b4700 1 -- 172.21.15.44:0/776275342 shutdown_connections 2024-04-21T23:07:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.849+0000 7f19835b4700 1 --2- 172.21.15.44:0/776275342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f197c101af0 0x7f197c10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.849+0000 7f19835b4700 1 --2- 172.21.15.44:0/776275342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f197c1071a0 0x7f197c107580 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.849+0000 7f19835b4700 1 --2- 172.21.15.44:0/776275342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f197c101130 0x7f197c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.849+0000 7f19835b4700 1 -- 172.21.15.44:0/776275342 >> 172.21.15.44:0/776275342 conn(0x7f197c0780e0 msgr2=0x7f197c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.849+0000 7f19835b4700 1 -- 172.21.15.44:0/776275342 shutdown_connections 2024-04-21T23:07:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.850+0000 7f19835b4700 1 -- 172.21.15.44:0/776275342 wait complete. 2024-04-21T23:07:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.850+0000 7f19835b4700 1 Processor -- start 2024-04-21T23:07:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.850+0000 7f19835b4700 1 -- start start 2024-04-21T23:07:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.850+0000 7f19835b4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f197c101130 0x7f197c19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.850+0000 7f19835b4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f197c101af0 0x7f197c19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.850+0000 7f19835b4700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f197c1071a0 0x7f197c198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.851+0000 7f19835b4700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f197c111410 con 0x7f197c1071a0 2024-04-21T23:07:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.851+0000 7f19835b4700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f197c111290 con 0x7f197c101af0 2024-04-21T23:07:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.851+0000 7f19835b4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f197c111590 con 0x7f197c101130 2024-04-21T23:07:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.851+0000 7f1981b51700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f197c1071a0 0x7f197c198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.851+0000 7f1981b51700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f197c1071a0 0x7f197c198810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53560/0 (socket says 172.21.15.44:53560) 2024-04-21T23:07:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.851+0000 7f1981b51700 1 -- 172.21.15.44:0/2417944179 learned_addr learned my addr 172.21.15.44:0/2417944179 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.851+0000 7f1980b4f700 1 --2- 172.21.15.44:0/2417944179 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f197c101af0 0x7f197c19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.851+0000 7f1981350700 1 --2- 172.21.15.44:0/2417944179 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f197c101130 0x7f197c19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.851+0000 7f1981b51700 1 -- 172.21.15.44:0/2417944179 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f197c101130 msgr2=0x7f197c19e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.851+0000 7f1981b51700 1 --2- 172.21.15.44:0/2417944179 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f197c101130 0x7f197c19e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.851+0000 7f1981b51700 1 -- 172.21.15.44:0/2417944179 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f197c101af0 msgr2=0x7f197c19eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.851+0000 7f1981b51700 1 --2- 172.21.15.44:0/2417944179 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f197c101af0 0x7f197c19eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.851+0000 7f1981b51700 1 -- 172.21.15.44:0/2417944179 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f197c198ff0 con 0x7f197c1071a0 2024-04-21T23:07:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.851+0000 7f1981b51700 1 --2- 172.21.15.44:0/2417944179 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f197c1071a0 0x7f197c198810 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1974007ee0 tx=0x7f197400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.851+0000 7f19727fc700 1 -- 172.21.15.44:0/2417944179 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1974004bb0 con 0x7f197c1071a0 2024-04-21T23:07:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.852+0000 7f19835b4700 1 -- 172.21.15.44:0/2417944179 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f197c1992e0 con 0x7f197c1071a0 2024-04-21T23:07:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.852+0000 7f19835b4700 1 -- 172.21.15.44:0/2417944179 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f197c1a54a0 con 0x7f197c1071a0 2024-04-21T23:07:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.852+0000 7f19727fc700 1 -- 172.21.15.44:0/2417944179 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1974004d50 con 0x7f197c1071a0 2024-04-21T23:07:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.852+0000 7f19727fc700 1 -- 172.21.15.44:0/2417944179 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1974005960 con 0x7f197c1071a0 2024-04-21T23:07:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.853+0000 7f19835b4700 1 -- 172.21.15.44:0/2417944179 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1960001060 con 0x7f197c1071a0 2024-04-21T23:07:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.854+0000 7f19727fc700 1 -- 172.21.15.44:0/2417944179 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f197401a020 con 0x7f197c1071a0 2024-04-21T23:07:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.855+0000 7f19727fc700 1 --2- 172.21.15.44:0/2417944179 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19680a2d30 0x7f19680a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.855+0000 7f19727fc700 1 -- 172.21.15.44:0/2417944179 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f19740cf310 con 0x7f197c1071a0 2024-04-21T23:07:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.855+0000 7f1981350700 1 --2- 172.21.15.44:0/2417944179 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19680a2d30 0x7f19680a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.855+0000 7f19727fc700 1 --2- 172.21.15.44:0/2417944179 >> [v2:172.21.15.55:6808/988958724,v1:172.21.15.55:6809/988958724] conn(0x7f19680ade10 0x7f19680b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.855+0000 7f19727fc700 1 -- 172.21.15.44:0/2417944179 --> [v2:172.21.15.55:6808/988958724,v1:172.21.15.55:6809/988958724] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f19680b0980 con 0x7f19680ade10 2024-04-21T23:07:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.855+0000 7f19727fc700 1 -- 172.21.15.44:0/2417944179 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f19740cf7f0 con 0x7f197c1071a0 2024-04-21T23:07:24.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.856+0000 7f1980b4f700 1 --2- 172.21.15.44:0/2417944179 >> [v2:172.21.15.55:6808/988958724,v1:172.21.15.55:6809/988958724] conn(0x7f19680ade10 0x7f19680b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:24.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.856+0000 7f1981350700 1 --2- 172.21.15.44:0/2417944179 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19680a2d30 0x7f19680a51f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f197800b5c0 tx=0x7f1978005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:24.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.856+0000 7f1980b4f700 1 --2- 172.21.15.44:0/2417944179 >> [v2:172.21.15.55:6808/988958724,v1:172.21.15.55:6809/988958724] conn(0x7f19680ade10 0x7f19680b02b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:24.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.859+0000 7f19727fc700 1 -- 172.21.15.44:0/2417944179 <== osd.4 v2:172.21.15.55:6808/988958724 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f19680b0980 con 0x7f19680ade10 2024-04-21T23:07:24.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.878+0000 7f19835b4700 1 -- 172.21.15.44:0/2417944179 --> [v2:172.21.15.55:6808/988958724,v1:172.21.15.55:6809/988958724] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1960002e90 con 0x7f19680ade10 2024-04-21T23:07:24.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.879+0000 7f19727fc700 1 -- 172.21.15.44:0/2417944179 <== osd.4 v2:172.21.15.55:6808/988958724 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1960002e90 con 0x7f19680ade10 2024-04-21T23:07:24.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.879+0000 7f19835b4700 1 -- 172.21.15.44:0/2417944179 >> [v2:172.21.15.55:6808/988958724,v1:172.21.15.55:6809/988958724] conn(0x7f19680ade10 msgr2=0x7f19680b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:24.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.879+0000 7f19835b4700 1 --2- 172.21.15.44:0/2417944179 >> [v2:172.21.15.55:6808/988958724,v1:172.21.15.55:6809/988958724] conn(0x7f19680ade10 0x7f19680b02b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:24.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.879+0000 7f19835b4700 1 -- 172.21.15.44:0/2417944179 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19680a2d30 msgr2=0x7f19680a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:24.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.879+0000 7f19835b4700 1 --2- 172.21.15.44:0/2417944179 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f19680a2d30 0x7f19680a51f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f197800b5c0 tx=0x7f1978005960 comp rx=0 tx=0).stop 2024-04-21T23:07:24.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.879+0000 7f19835b4700 1 -- 172.21.15.44:0/2417944179 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f197c1071a0 msgr2=0x7f197c198810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:24.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.879+0000 7f19835b4700 1 --2- 172.21.15.44:0/2417944179 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f197c1071a0 0x7f197c198810 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1974007ee0 tx=0x7f197400d3b0 comp rx=0 tx=0).stop 2024-04-21T23:07:24.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.879+0000 7f1981b51700 1 -- 172.21.15.44:0/2417944179 reap_dead start 2024-04-21T23:07:24.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.879+0000 7f19835b4700 1 -- 172.21.15.44:0/2417944179 shutdown_connections 2024-04-21T23:07:24.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.879+0000 7f19835b4700 1 -- 172.21.15.44:0/2417944179 >> 172.21.15.44:0/2417944179 conn(0x7f197c0780e0 msgr2=0x7f197c100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:24.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.880+0000 7f19835b4700 1 -- 172.21.15.44:0/2417944179 shutdown_connections 2024-04-21T23:07:24.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:24.880+0000 7f19835b4700 1 -- 172.21.15.44:0/2417944179 wait complete. 2024-04-21T23:07:25.303 INFO:teuthology.orchestra.run.smithi044.stdout:34359738446 2024-04-21T23:07:25.303 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-21T23:07:25.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:25 smithi044 ceph-mon[27934]: osdmap e92: 12 total, 12 up, 12 in 2024-04-21T23:07:25.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:25 smithi055 ceph-mon[31117]: osdmap e92: 12 total, 12 up, 12 in 2024-04-21T23:07:25.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:25 smithi069 ceph-mon[31816]: osdmap e92: 12 total, 12 up, 12 in 2024-04-21T23:07:25.847 INFO:teuthology.orchestra.run.smithi044.stdout:219043332139 2024-04-21T23:07:25.847 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-21T23:07:26.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:26 smithi044 ceph-mon[27934]: pgmap v323: 129 pgs: 1 peering, 128 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:07:26.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:26 smithi069 ceph-mon[31816]: pgmap v323: 129 pgs: 1 peering, 128 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:07:26.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:26 smithi055 ceph-mon[31117]: pgmap v323: 129 pgs: 1 peering, 128 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:07:26.715 INFO:teuthology.orchestra.run.smithi044.stdout:347892351001 2024-04-21T23:07:26.716 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-21T23:07:27.737 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:27 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:07:27.901 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:27 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:07:27.966 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:27 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:07:28.520 INFO:teuthology.orchestra.run.smithi044.stdout:163208757304 2024-04-21T23:07:28.520 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-21T23:07:28.718 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:28 smithi044 ceph-mon[27934]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s rd, 2 op/s; 2 B/s, 0 objects/s recovering 2024-04-21T23:07:28.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.716+0000 7f78d297b700 1 -- 172.21.15.44:0/4193233328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78cc103140 msgr2=0x7f78cc103520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:28.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.716+0000 7f78d297b700 1 --2- 172.21.15.44:0/4193233328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78cc103140 0x7f78cc103520 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f78bc009990 tx=0x7f78bc009ca0 comp rx=0 tx=0).stop 2024-04-21T23:07:28.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.716+0000 7f78d297b700 1 -- 172.21.15.44:0/4193233328 shutdown_connections 2024-04-21T23:07:28.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.716+0000 7f78d297b700 1 --2- 172.21.15.44:0/4193233328 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78cc1012c0 0x7f78cc110fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:28.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.716+0000 7f78d297b700 1 --2- 172.21.15.44:0/4193233328 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78cc103140 0x7f78cc103520 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:28.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.716+0000 7f78d297b700 1 --2- 172.21.15.44:0/4193233328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78cc100820 0x7f78cc100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:28.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.716+0000 7f78d297b700 1 -- 172.21.15.44:0/4193233328 >> 172.21.15.44:0/4193233328 conn(0x7f78cc0777a0 msgr2=0x7f78cc0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:28.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.717+0000 7f78d297b700 1 -- 172.21.15.44:0/4193233328 shutdown_connections 2024-04-21T23:07:28.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.717+0000 7f78d297b700 1 -- 172.21.15.44:0/4193233328 wait complete. 2024-04-21T23:07:28.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.717+0000 7f78d297b700 1 Processor -- start 2024-04-21T23:07:28.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.717+0000 7f78d297b700 1 -- start start 2024-04-21T23:07:28.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.718+0000 7f78d297b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78cc100820 0x7f78cc1a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:28.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.718+0000 7f78d297b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78cc1012c0 0x7f78cc1a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:28.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.718+0000 7f78d297b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78cc103140 0x7f78cc19aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:28.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.718+0000 7f78d297b700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f78cc114730 con 0x7f78cc1012c0 2024-04-21T23:07:28.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.718+0000 7f78d297b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f78cc1145b0 con 0x7f78cc103140 2024-04-21T23:07:28.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.718+0000 7f78d297b700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f78cc1148b0 con 0x7f78cc100820 2024-04-21T23:07:28.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.718+0000 7f78cb7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78cc1012c0 0x7f78cc1a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:28.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.718+0000 7f78cb7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78cc1012c0 0x7f78cc1a0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53566/0 (socket says 172.21.15.44:53566) 2024-04-21T23:07:28.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.718+0000 7f78cb7fe700 1 -- 172.21.15.44:0/614906663 learned_addr learned my addr 172.21.15.44:0/614906663 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:28.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.718+0000 7f78d0f18700 1 --2- 172.21.15.44:0/614906663 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78cc103140 0x7f78cc19aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:28.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.718+0000 7f78cbfff700 1 --2- 172.21.15.44:0/614906663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78cc100820 0x7f78cc1a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:28.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.719+0000 7f78d0f18700 1 -- 172.21.15.44:0/614906663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78cc100820 msgr2=0x7f78cc1a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:28.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.719+0000 7f78d0f18700 1 --2- 172.21.15.44:0/614906663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78cc100820 0x7f78cc1a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:28.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.719+0000 7f78d0f18700 1 -- 172.21.15.44:0/614906663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78cc1012c0 msgr2=0x7f78cc1a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:28.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.719+0000 7f78d0f18700 1 --2- 172.21.15.44:0/614906663 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f78cc1012c0 0x7f78cc1a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:28.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.719+0000 7f78d0f18700 1 -- 172.21.15.44:0/614906663 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78cc19b170 con 0x7f78cc103140 2024-04-21T23:07:28.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.719+0000 7f78d0f18700 1 --2- 172.21.15.44:0/614906663 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78cc103140 0x7f78cc19aa10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f78c4007f10 tx=0x7f78c400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:28.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.720+0000 7f78c97fa700 1 -- 172.21.15.44:0/614906663 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c4004db0 con 0x7f78cc103140 2024-04-21T23:07:28.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.720+0000 7f78c97fa700 1 -- 172.21.15.44:0/614906663 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78c4009e20 con 0x7f78cc103140 2024-04-21T23:07:28.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.720+0000 7f78d297b700 1 -- 172.21.15.44:0/614906663 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78cc19b460 con 0x7f78cc103140 2024-04-21T23:07:28.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.720+0000 7f78c97fa700 1 -- 172.21.15.44:0/614906663 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c4015400 con 0x7f78cc103140 2024-04-21T23:07:28.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.720+0000 7f78d297b700 1 -- 172.21.15.44:0/614906663 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78cc1a76d0 con 0x7f78cc103140 2024-04-21T23:07:28.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.720+0000 7f78d297b700 1 -- 172.21.15.44:0/614906663 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f78cc062980 con 0x7f78cc103140 2024-04-21T23:07:28.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.722+0000 7f78c97fa700 1 -- 172.21.15.44:0/614906663 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f78c400db20 con 0x7f78cc103140 2024-04-21T23:07:28.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.722+0000 7f78c97fa700 1 --2- 172.21.15.44:0/614906663 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f78b40a2cb0 0x7f78b40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:28.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.723+0000 7f78c97fa700 1 -- 172.21.15.44:0/614906663 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f78c40cf9f0 con 0x7f78cc103140 2024-04-21T23:07:28.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.723+0000 7f78cbfff700 1 --2- 172.21.15.44:0/614906663 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f78b40a2cb0 0x7f78b40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:28.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.723+0000 7f78c97fa700 1 --2- 172.21.15.44:0/614906663 >> [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748] conn(0x7f78b40add90 0x7f78b40b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:28.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.723+0000 7f78c97fa700 1 -- 172.21.15.44:0/614906663 --> [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f78c4020e50 con 0x7f78b40add90 2024-04-21T23:07:28.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.723+0000 7f78c97fa700 1 -- 172.21.15.44:0/614906663 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f78c40cfda0 con 0x7f78cc103140 2024-04-21T23:07:28.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.723+0000 7f78cb7fe700 1 --2- 172.21.15.44:0/614906663 >> [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748] conn(0x7f78b40add90 0x7f78b40b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:28.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.723+0000 7f78cbfff700 1 --2- 172.21.15.44:0/614906663 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f78b40a2cb0 0x7f78b40a5170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f78cc0ff860 tx=0x7f78bc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:28.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.723+0000 7f78cb7fe700 1 --2- 172.21.15.44:0/614906663 >> [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748] conn(0x7f78b40add90 0x7f78b40b0230 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:28.730 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.725+0000 7f78c97fa700 1 -- 172.21.15.44:0/614906663 <== osd.6 v2:172.21.15.44:6818/1221340748 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f78c4020e50 con 0x7f78b40add90 2024-04-21T23:07:28.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.742+0000 7f78d297b700 1 -- 172.21.15.44:0/614906663 --> [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f78cc062e00 con 0x7f78b40add90 2024-04-21T23:07:28.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.743+0000 7f78c97fa700 1 -- 172.21.15.44:0/614906663 <== osd.6 v2:172.21.15.44:6818/1221340748 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f78cc062e00 con 0x7f78b40add90 2024-04-21T23:07:28.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.743+0000 7f78d297b700 1 -- 172.21.15.44:0/614906663 >> [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748] conn(0x7f78b40add90 msgr2=0x7f78b40b0230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:28.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.743+0000 7f78d297b700 1 --2- 172.21.15.44:0/614906663 >> [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748] conn(0x7f78b40add90 0x7f78b40b0230 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:28.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.744+0000 7f78d297b700 1 -- 172.21.15.44:0/614906663 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f78b40a2cb0 msgr2=0x7f78b40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:28.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.744+0000 7f78d297b700 1 --2- 172.21.15.44:0/614906663 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f78b40a2cb0 0x7f78b40a5170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f78cc0ff860 tx=0x7f78bc005960 comp rx=0 tx=0).stop 2024-04-21T23:07:28.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.744+0000 7f78d297b700 1 -- 172.21.15.44:0/614906663 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78cc103140 msgr2=0x7f78cc19aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:28.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.744+0000 7f78d297b700 1 --2- 172.21.15.44:0/614906663 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f78cc103140 0x7f78cc19aa10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f78c4007f10 tx=0x7f78c400d3b0 comp rx=0 tx=0).stop 2024-04-21T23:07:28.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.744+0000 7f78d0f18700 1 -- 172.21.15.44:0/614906663 reap_dead start 2024-04-21T23:07:28.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.744+0000 7f78d297b700 1 -- 172.21.15.44:0/614906663 shutdown_connections 2024-04-21T23:07:28.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.744+0000 7f78d297b700 1 -- 172.21.15.44:0/614906663 >> 172.21.15.44:0/614906663 conn(0x7f78cc0777a0 msgr2=0x7f78cc0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:28.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.744+0000 7f78d297b700 1 -- 172.21.15.44:0/614906663 shutdown_connections 2024-04-21T23:07:28.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:28.744+0000 7f78d297b700 1 -- 172.21.15.44:0/614906663 wait complete. 2024-04-21T23:07:28.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:28 smithi055 ceph-mon[31117]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s rd, 2 op/s; 2 B/s, 0 objects/s recovering 2024-04-21T23:07:28.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:28 smithi069 ceph-mon[31816]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s rd, 2 op/s; 2 B/s, 0 objects/s recovering 2024-04-21T23:07:29.284 INFO:teuthology.orchestra.run.smithi044.stdout:141733920828 2024-04-21T23:07:29.284 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-21T23:07:29.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:29 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:07:29.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:29 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:07:29.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:29 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:07:29.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:29 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:07:29.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:29 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:07:29.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:29 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:07:29.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:29 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:07:29.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:29 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:07:30.104 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:29 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:07:30.105 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:29 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:07:30.105 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:29 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:07:30.105 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:29 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:07:30.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.496+0000 7f9c3a64f700 1 -- 172.21.15.44:0/3144624152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c34101d60 msgr2=0x7f9c34105db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:30.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.497+0000 7f9c3a64f700 1 --2- 172.21.15.44:0/3144624152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c34101d60 0x7f9c34105db0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9c280099c0 tx=0x7f9c28009cd0 comp rx=0 tx=0).stop 2024-04-21T23:07:30.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.497+0000 7f9c3a64f700 1 -- 172.21.15.44:0/3144624152 shutdown_connections 2024-04-21T23:07:30.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.497+0000 7f9c3a64f700 1 --2- 172.21.15.44:0/3144624152 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9c34106490 0x7f9c3410e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:30.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.497+0000 7f9c3a64f700 1 --2- 172.21.15.44:0/3144624152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c341013b0 0x7f9c34101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:30.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.497+0000 7f9c3a64f700 1 --2- 172.21.15.44:0/3144624152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c34101d60 0x7f9c34105db0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:30.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.497+0000 7f9c3a64f700 1 -- 172.21.15.44:0/3144624152 >> 172.21.15.44:0/3144624152 conn(0x7f9c340781d0 msgr2=0x7f9c340785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:30.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.497+0000 7f9c3a64f700 1 -- 172.21.15.44:0/3144624152 shutdown_connections 2024-04-21T23:07:30.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.497+0000 7f9c3a64f700 1 -- 172.21.15.44:0/3144624152 wait complete. 2024-04-21T23:07:30.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.497+0000 7f9c3a64f700 1 Processor -- start 2024-04-21T23:07:30.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.497+0000 7f9c3a64f700 1 -- start start 2024-04-21T23:07:30.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.498+0000 7f9c3a64f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c341013b0 0x7f9c3419e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:30.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.498+0000 7f9c3a64f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c34101d60 0x7f9c3419ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:30.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.498+0000 7f9c3a64f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9c34106490 0x7f9c34198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:30.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.498+0000 7f9c3a64f700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9c34111470 con 0x7f9c34101d60 2024-04-21T23:07:30.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.498+0000 7f9c3a64f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9c341112f0 con 0x7f9c34106490 2024-04-21T23:07:30.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.498+0000 7f9c3a64f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9c341115f0 con 0x7f9c341013b0 2024-04-21T23:07:30.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.498+0000 7f9c337fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c34101d60 0x7f9c3419ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:30.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.498+0000 7f9c337fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c34101d60 0x7f9c3419ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53584/0 (socket says 172.21.15.44:53584) 2024-04-21T23:07:30.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.498+0000 7f9c337fe700 1 -- 172.21.15.44:0/3289562380 learned_addr learned my addr 172.21.15.44:0/3289562380 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:30.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.498+0000 7f9c33fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c341013b0 0x7f9c3419e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:30.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.498+0000 7f9c38bec700 1 --2- 172.21.15.44:0/3289562380 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9c34106490 0x7f9c34198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:30.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.498+0000 7f9c337fe700 1 -- 172.21.15.44:0/3289562380 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c341013b0 msgr2=0x7f9c3419e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:30.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.498+0000 7f9c337fe700 1 --2- 172.21.15.44:0/3289562380 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c341013b0 0x7f9c3419e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:30.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.498+0000 7f9c337fe700 1 -- 172.21.15.44:0/3289562380 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9c34106490 msgr2=0x7f9c34198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:30.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.498+0000 7f9c337fe700 1 --2- 172.21.15.44:0/3289562380 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9c34106490 0x7f9c34198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:30.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.498+0000 7f9c337fe700 1 -- 172.21.15.44:0/3289562380 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c34199060 con 0x7f9c34101d60 2024-04-21T23:07:30.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.498+0000 7f9c337fe700 1 --2- 172.21.15.44:0/3289562380 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c34101d60 0x7f9c3419ec80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9c28006010 tx=0x7f9c2800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:30.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.499+0000 7f9c317fa700 1 -- 172.21.15.44:0/3289562380 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c28027070 con 0x7f9c34101d60 2024-04-21T23:07:30.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.499+0000 7f9c317fa700 1 -- 172.21.15.44:0/3289562380 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c2800fe40 con 0x7f9c34101d60 2024-04-21T23:07:30.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.499+0000 7f9c3a64f700 1 -- 172.21.15.44:0/3289562380 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c341992f0 con 0x7f9c34101d60 2024-04-21T23:07:30.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.499+0000 7f9c317fa700 1 -- 172.21.15.44:0/3289562380 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c28017650 con 0x7f9c34101d60 2024-04-21T23:07:30.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.499+0000 7f9c3a64f700 1 -- 172.21.15.44:0/3289562380 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c341a5610 con 0x7f9c34101d60 2024-04-21T23:07:30.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.500+0000 7f9c3a64f700 1 -- 172.21.15.44:0/3289562380 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9c34062940 con 0x7f9c34101d60 2024-04-21T23:07:30.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.501+0000 7f9c317fa700 1 -- 172.21.15.44:0/3289562380 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9c2802a070 con 0x7f9c34101d60 2024-04-21T23:07:30.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.501+0000 7f9c317fa700 1 --2- 172.21.15.44:0/3289562380 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f9c1c0a2cb0 0x7f9c1c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:30.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.502+0000 7f9c33fff700 1 --2- 172.21.15.44:0/3289562380 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f9c1c0a2cb0 0x7f9c1c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:30.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.502+0000 7f9c317fa700 1 -- 172.21.15.44:0/3289562380 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f9c280d05f0 con 0x7f9c34101d60 2024-04-21T23:07:30.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.502+0000 7f9c317fa700 1 --2- 172.21.15.44:0/3289562380 >> [v2:172.21.15.69:6816/635270546,v1:172.21.15.69:6817/635270546] conn(0x7f9c1c0add90 0x7f9c1c0b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:30.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.502+0000 7f9c317fa700 1 -- 172.21.15.44:0/3289562380 --> [v2:172.21.15.69:6816/635270546,v1:172.21.15.69:6817/635270546] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9c28001d30 con 0x7f9c1c0add90 2024-04-21T23:07:30.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.502+0000 7f9c317fa700 1 -- 172.21.15.44:0/3289562380 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f9c280d0a40 con 0x7f9c34101d60 2024-04-21T23:07:30.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.502+0000 7f9c33fff700 1 --2- 172.21.15.44:0/3289562380 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f9c1c0a2cb0 0x7f9c1c0a5170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9c34199a40 tx=0x7f9c24009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:30.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.502+0000 7f9c38bec700 1 --2- 172.21.15.44:0/3289562380 >> [v2:172.21.15.69:6816/635270546,v1:172.21.15.69:6817/635270546] conn(0x7f9c1c0add90 0x7f9c1c0b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:30.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.502+0000 7f9c38bec700 1 --2- 172.21.15.44:0/3289562380 >> [v2:172.21.15.69:6816/635270546,v1:172.21.15.69:6817/635270546] conn(0x7f9c1c0add90 0x7f9c1c0b0230 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:30.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.503+0000 7f9c317fa700 1 -- 172.21.15.44:0/3289562380 <== osd.8 v2:172.21.15.69:6816/635270546 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9c28001d30 con 0x7f9c1c0add90 2024-04-21T23:07:30.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.520+0000 7f9c3a64f700 1 -- 172.21.15.44:0/3289562380 --> [v2:172.21.15.69:6816/635270546,v1:172.21.15.69:6817/635270546] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9c34062d40 con 0x7f9c1c0add90 2024-04-21T23:07:30.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.520+0000 7f9c317fa700 1 -- 172.21.15.44:0/3289562380 <== osd.8 v2:172.21.15.69:6816/635270546 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9c34062d40 con 0x7f9c1c0add90 2024-04-21T23:07:30.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.520+0000 7f9c3a64f700 1 -- 172.21.15.44:0/3289562380 >> [v2:172.21.15.69:6816/635270546,v1:172.21.15.69:6817/635270546] conn(0x7f9c1c0add90 msgr2=0x7f9c1c0b0230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:30.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.520+0000 7f9c3a64f700 1 --2- 172.21.15.44:0/3289562380 >> [v2:172.21.15.69:6816/635270546,v1:172.21.15.69:6817/635270546] conn(0x7f9c1c0add90 0x7f9c1c0b0230 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:30.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.520+0000 7f9c3a64f700 1 -- 172.21.15.44:0/3289562380 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f9c1c0a2cb0 msgr2=0x7f9c1c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:30.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.520+0000 7f9c3a64f700 1 --2- 172.21.15.44:0/3289562380 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f9c1c0a2cb0 0x7f9c1c0a5170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9c34199a40 tx=0x7f9c24009260 comp rx=0 tx=0).stop 2024-04-21T23:07:30.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.520+0000 7f9c3a64f700 1 -- 172.21.15.44:0/3289562380 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c34101d60 msgr2=0x7f9c3419ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:30.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.520+0000 7f9c3a64f700 1 --2- 172.21.15.44:0/3289562380 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9c34101d60 0x7f9c3419ec80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9c28006010 tx=0x7f9c2800fa00 comp rx=0 tx=0).stop 2024-04-21T23:07:30.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.520+0000 7f9c38bec700 1 -- 172.21.15.44:0/3289562380 reap_dead start 2024-04-21T23:07:30.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.521+0000 7f9c3a64f700 1 -- 172.21.15.44:0/3289562380 shutdown_connections 2024-04-21T23:07:30.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.521+0000 7f9c3a64f700 1 -- 172.21.15.44:0/3289562380 >> 172.21.15.44:0/3289562380 conn(0x7f9c340781d0 msgr2=0x7f9c3406acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:30.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.521+0000 7f9c3a64f700 1 -- 172.21.15.44:0/3289562380 shutdown_connections 2024-04-21T23:07:30.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:30.521+0000 7f9c3a64f700 1 -- 172.21.15.44:0/3289562380 wait complete. 2024-04-21T23:07:31.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:30 smithi044 ceph-mon[27934]: pgmap v325: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 0 op/s; 2 B/s, 0 objects/s recovering 2024-04-21T23:07:31.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:30 smithi069 ceph-mon[31816]: pgmap v325: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 0 op/s; 2 B/s, 0 objects/s recovering 2024-04-21T23:07:31.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:30 smithi055 ceph-mon[31117]: pgmap v325: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 0 op/s; 2 B/s, 0 objects/s recovering 2024-04-21T23:07:31.306 INFO:teuthology.orchestra.run.smithi044.stdout:253403070506 2024-04-21T23:07:31.306 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-21T23:07:32.438 INFO:teuthology.orchestra.run.smithi044.stdout:197568495667 2024-04-21T23:07:32.439 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-21T23:07:32.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.640+0000 7fdb2cfa0700 1 -- 172.21.15.44:0/390921639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb2810c750 msgr2=0x7fdb2810ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:32.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.640+0000 7fdb2cfa0700 1 --2- 172.21.15.44:0/390921639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb2810c750 0x7fdb2810ec10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000b0a0 tx=0x7fdb2000b3b0 comp rx=0 tx=0).stop 2024-04-21T23:07:32.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.640+0000 7fdb2cfa0700 1 -- 172.21.15.44:0/390921639 shutdown_connections 2024-04-21T23:07:32.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.640+0000 7fdb2cfa0700 1 --2- 172.21.15.44:0/390921639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb2810c750 0x7fdb2810ec10 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:32.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.640+0000 7fdb2cfa0700 1 --2- 172.21.15.44:0/390921639 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb28103320 0x7fdb28103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:32.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.640+0000 7fdb2cfa0700 1 --2- 172.21.15.44:0/390921639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb28103c40 0x7fdb2810c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:32.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.640+0000 7fdb2cfa0700 1 -- 172.21.15.44:0/390921639 >> 172.21.15.44:0/390921639 conn(0x7fdb280fcb50 msgr2=0x7fdb280fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:32.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.640+0000 7fdb2cfa0700 1 -- 172.21.15.44:0/390921639 shutdown_connections 2024-04-21T23:07:32.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.641+0000 7fdb2cfa0700 1 -- 172.21.15.44:0/390921639 wait complete. 2024-04-21T23:07:32.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.641+0000 7fdb2cfa0700 1 Processor -- start 2024-04-21T23:07:32.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.641+0000 7fdb2cfa0700 1 -- start start 2024-04-21T23:07:32.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.642+0000 7fdb2cfa0700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb28103320 0x7fdb2819e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:32.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.642+0000 7fdb2cfa0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb28103c40 0x7fdb2819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:32.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.642+0000 7fdb2cfa0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb2810c750 0x7fdb28198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:32.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.642+0000 7fdb2cfa0700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdb281124f0 con 0x7fdb28103320 2024-04-21T23:07:32.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.642+0000 7fdb2cfa0700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdb28112370 con 0x7fdb2810c750 2024-04-21T23:07:32.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.642+0000 7fdb2cfa0700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdb28112670 con 0x7fdb28103c40 2024-04-21T23:07:32.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.642+0000 7fdb277fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb28103320 0x7fdb2819e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:32.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.642+0000 7fdb277fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb28103320 0x7fdb2819e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42670/0 (socket says 172.21.15.44:42670) 2024-04-21T23:07:32.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.642+0000 7fdb277fe700 1 -- 172.21.15.44:0/261778418 learned_addr learned my addr 172.21.15.44:0/261778418 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:32.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.642+0000 7fdb26ffd700 1 --2- 172.21.15.44:0/261778418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb28103c40 0x7fdb2819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:32.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.642+0000 7fdb27fff700 1 --2- 172.21.15.44:0/261778418 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb2810c750 0x7fdb28198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:32.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.642+0000 7fdb277fe700 1 -- 172.21.15.44:0/261778418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb28103c40 msgr2=0x7fdb2819eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:32.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.642+0000 7fdb277fe700 1 --2- 172.21.15.44:0/261778418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb28103c40 0x7fdb2819eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:32.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.642+0000 7fdb277fe700 1 -- 172.21.15.44:0/261778418 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb2810c750 msgr2=0x7fdb28198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:32.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.642+0000 7fdb277fe700 1 --2- 172.21.15.44:0/261778418 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb2810c750 0x7fdb28198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:32.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.642+0000 7fdb277fe700 1 -- 172.21.15.44:0/261778418 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb28198f30 con 0x7fdb28103320 2024-04-21T23:07:32.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.642+0000 7fdb277fe700 1 --2- 172.21.15.44:0/261778418 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb28103320 0x7fdb2819e630 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c00e9a0 tx=0x7fdb1c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:32.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.643+0000 7fdb24ff9700 1 -- 172.21.15.44:0/261778418 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c00cbd0 con 0x7fdb28103320 2024-04-21T23:07:32.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.643+0000 7fdb2cfa0700 1 -- 172.21.15.44:0/261778418 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb28199220 con 0x7fdb28103320 2024-04-21T23:07:32.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.643+0000 7fdb24ff9700 1 -- 172.21.15.44:0/261778418 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb1c00cd70 con 0x7fdb28103320 2024-04-21T23:07:32.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.643+0000 7fdb24ff9700 1 -- 172.21.15.44:0/261778418 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c018750 con 0x7fdb28103320 2024-04-21T23:07:32.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.643+0000 7fdb2cfa0700 1 -- 172.21.15.44:0/261778418 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb281a5450 con 0x7fdb28103320 2024-04-21T23:07:32.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.644+0000 7fdb2cfa0700 1 -- 172.21.15.44:0/261778418 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdb28062900 con 0x7fdb28103320 2024-04-21T23:07:32.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.645+0000 7fdb24ff9700 1 -- 172.21.15.44:0/261778418 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fdb1c010430 con 0x7fdb28103320 2024-04-21T23:07:32.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.645+0000 7fdb24ff9700 1 --2- 172.21.15.44:0/261778418 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdb100a2cb0 0x7fdb100a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:32.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.646+0000 7fdb24ff9700 1 -- 172.21.15.44:0/261778418 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fdb1c0cff70 con 0x7fdb28103320 2024-04-21T23:07:32.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.646+0000 7fdb26ffd700 1 --2- 172.21.15.44:0/261778418 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdb100a2cb0 0x7fdb100a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:32.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.646+0000 7fdb24ff9700 1 --2- 172.21.15.44:0/261778418 >> [v2:172.21.15.69:6800/3485429011,v1:172.21.15.69:6801/3485429011] conn(0x7fdb100add90 0x7fdb100b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:32.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.646+0000 7fdb24ff9700 1 -- 172.21.15.44:0/261778418 --> [v2:172.21.15.69:6800/3485429011,v1:172.21.15.69:6801/3485429011] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb100b0900 con 0x7fdb100add90 2024-04-21T23:07:32.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.646+0000 7fdb24ff9700 1 -- 172.21.15.44:0/261778418 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fdb1c0d03c0 con 0x7fdb28103320 2024-04-21T23:07:32.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.646+0000 7fdb26ffd700 1 --2- 172.21.15.44:0/261778418 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdb100a2cb0 0x7fdb100a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdb2810ec80 tx=0x7fdb18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:32.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.646+0000 7fdb27fff700 1 --2- 172.21.15.44:0/261778418 >> [v2:172.21.15.69:6800/3485429011,v1:172.21.15.69:6801/3485429011] conn(0x7fdb100add90 0x7fdb100b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:32.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.647+0000 7fdb27fff700 1 --2- 172.21.15.44:0/261778418 >> [v2:172.21.15.69:6800/3485429011,v1:172.21.15.69:6801/3485429011] conn(0x7fdb100add90 0x7fdb100b0230 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:32.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.648+0000 7fdb24ff9700 1 -- 172.21.15.44:0/261778418 <== osd.2 v2:172.21.15.69:6800/3485429011 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdb100b0900 con 0x7fdb100add90 2024-04-21T23:07:32.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.677+0000 7fdb2cfa0700 1 -- 172.21.15.44:0/261778418 --> [v2:172.21.15.69:6800/3485429011,v1:172.21.15.69:6801/3485429011] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdb28062db0 con 0x7fdb100add90 2024-04-21T23:07:32.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.678+0000 7fdb24ff9700 1 -- 172.21.15.44:0/261778418 <== osd.2 v2:172.21.15.69:6800/3485429011 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdb28062db0 con 0x7fdb100add90 2024-04-21T23:07:32.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.678+0000 7fdb2cfa0700 1 -- 172.21.15.44:0/261778418 >> [v2:172.21.15.69:6800/3485429011,v1:172.21.15.69:6801/3485429011] conn(0x7fdb100add90 msgr2=0x7fdb100b0230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:32.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.678+0000 7fdb2cfa0700 1 --2- 172.21.15.44:0/261778418 >> [v2:172.21.15.69:6800/3485429011,v1:172.21.15.69:6801/3485429011] conn(0x7fdb100add90 0x7fdb100b0230 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:32.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.679+0000 7fdb2cfa0700 1 -- 172.21.15.44:0/261778418 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdb100a2cb0 msgr2=0x7fdb100a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:32.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.679+0000 7fdb2cfa0700 1 --2- 172.21.15.44:0/261778418 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdb100a2cb0 0x7fdb100a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdb2810ec80 tx=0x7fdb18009260 comp rx=0 tx=0).stop 2024-04-21T23:07:32.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.679+0000 7fdb2cfa0700 1 -- 172.21.15.44:0/261778418 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb28103320 msgr2=0x7fdb2819e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:32.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.679+0000 7fdb2cfa0700 1 --2- 172.21.15.44:0/261778418 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb28103320 0x7fdb2819e630 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c00e9a0 tx=0x7fdb1c00ecb0 comp rx=0 tx=0).stop 2024-04-21T23:07:32.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.679+0000 7fdb27fff700 1 -- 172.21.15.44:0/261778418 reap_dead start 2024-04-21T23:07:32.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.679+0000 7fdb2cfa0700 1 -- 172.21.15.44:0/261778418 shutdown_connections 2024-04-21T23:07:32.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.679+0000 7fdb2cfa0700 1 -- 172.21.15.44:0/261778418 >> 172.21.15.44:0/261778418 conn(0x7fdb280fcb50 msgr2=0x7fdb280fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:32.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.679+0000 7fdb2cfa0700 1 -- 172.21.15.44:0/261778418 shutdown_connections 2024-04-21T23:07:32.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.679+0000 7fdb2cfa0700 1 -- 172.21.15.44:0/261778418 wait complete. 2024-04-21T23:07:32.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.887+0000 7f2789092700 1 -- 172.21.15.44:0/2337926944 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2784102e90 msgr2=0x7f2784106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:32.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.887+0000 7f2789092700 1 --2- 172.21.15.44:0/2337926944 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2784102e90 0x7f2784106e90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2778009990 tx=0x7f2778009ca0 comp rx=0 tx=0).stop 2024-04-21T23:07:32.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.887+0000 7f2789092700 1 -- 172.21.15.44:0/2337926944 shutdown_connections 2024-04-21T23:07:32.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.887+0000 7f2789092700 1 --2- 172.21.15.44:0/2337926944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2784107570 0x7f278410f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:32.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.887+0000 7f2789092700 1 --2- 172.21.15.44:0/2337926944 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27841024e0 0x7f27841028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:32.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.887+0000 7f2789092700 1 --2- 172.21.15.44:0/2337926944 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2784102e90 0x7f2784106e90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:32.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.887+0000 7f2789092700 1 -- 172.21.15.44:0/2337926944 >> 172.21.15.44:0/2337926944 conn(0x7f2784079390 msgr2=0x7f27840797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:32.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.887+0000 7f2789092700 1 -- 172.21.15.44:0/2337926944 shutdown_connections 2024-04-21T23:07:32.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.887+0000 7f2789092700 1 -- 172.21.15.44:0/2337926944 wait complete. 2024-04-21T23:07:32.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.888+0000 7f2789092700 1 Processor -- start 2024-04-21T23:07:32.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.888+0000 7f2789092700 1 -- start start 2024-04-21T23:07:32.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.888+0000 7f2789092700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27841024e0 0x7f278419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:32.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.888+0000 7f2789092700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2784102e90 0x7f278419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:32.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.888+0000 7f2789092700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2784107570 0x7f2784198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:32.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.888+0000 7f2789092700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2784112560 con 0x7f27841024e0 2024-04-21T23:07:32.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.888+0000 7f2789092700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f27841123e0 con 0x7f2784102e90 2024-04-21T23:07:32.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.888+0000 7f2789092700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f27841126e0 con 0x7f2784107570 2024-04-21T23:07:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.889+0000 7f2782d9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27841024e0 0x7f278419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.889+0000 7f2782d9d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27841024e0 0x7f278419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42686/0 (socket says 172.21.15.44:42686) 2024-04-21T23:07:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.889+0000 7f2782d9d700 1 -- 172.21.15.44:0/345360003 learned_addr learned my addr 172.21.15.44:0/345360003 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.889+0000 7f278259c700 1 --2- 172.21.15.44:0/345360003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2784102e90 0x7f278419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.889+0000 7f278359e700 1 --2- 172.21.15.44:0/345360003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2784107570 0x7f2784198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.889+0000 7f2782d9d700 1 -- 172.21.15.44:0/345360003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2784107570 msgr2=0x7f2784198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.889+0000 7f2782d9d700 1 --2- 172.21.15.44:0/345360003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2784107570 0x7f2784198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.889+0000 7f2782d9d700 1 -- 172.21.15.44:0/345360003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2784102e90 msgr2=0x7f278419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.889+0000 7f2782d9d700 1 --2- 172.21.15.44:0/345360003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2784102e90 0x7f278419ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.889+0000 7f2782d9d700 1 -- 172.21.15.44:0/345360003 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2784199000 con 0x7f27841024e0 2024-04-21T23:07:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.889+0000 7f278359e700 1 --2- 172.21.15.44:0/345360003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2784107570 0x7f2784198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:07:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.889+0000 7f278259c700 1 --2- 172.21.15.44:0/345360003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2784102e90 0x7f278419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:07:32.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.889+0000 7f2782d9d700 1 --2- 172.21.15.44:0/345360003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27841024e0 0x7f278419e6a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f277400e9a0 tx=0x7f277400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.889+0000 7f276bfff700 1 -- 172.21.15.44:0/345360003 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277400cbd0 con 0x7f27841024e0 2024-04-21T23:07:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.889+0000 7f2789092700 1 -- 172.21.15.44:0/345360003 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27841992f0 con 0x7f27841024e0 2024-04-21T23:07:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.889+0000 7f276bfff700 1 -- 172.21.15.44:0/345360003 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f277400cd70 con 0x7f27841024e0 2024-04-21T23:07:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.889+0000 7f276bfff700 1 -- 172.21.15.44:0/345360003 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27740187c0 con 0x7f27841024e0 2024-04-21T23:07:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.889+0000 7f2789092700 1 -- 172.21.15.44:0/345360003 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27841a5500 con 0x7f27841024e0 2024-04-21T23:07:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.890+0000 7f2789092700 1 -- 172.21.15.44:0/345360003 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f27840629c0 con 0x7f27841024e0 2024-04-21T23:07:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.892+0000 7f276bfff700 1 -- 172.21.15.44:0/345360003 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f2774010430 con 0x7f27841024e0 2024-04-21T23:07:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.892+0000 7f276bfff700 1 --2- 172.21.15.44:0/345360003 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f276c0a2cb0 0x7f276c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.892+0000 7f278259c700 1 --2- 172.21.15.44:0/345360003 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f276c0a2cb0 0x7f276c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.892+0000 7f276bfff700 1 -- 172.21.15.44:0/345360003 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f27740d0020 con 0x7f27841024e0 2024-04-21T23:07:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.892+0000 7f276bfff700 1 --2- 172.21.15.44:0/345360003 >> [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908] conn(0x7f276c0add90 0x7f276c0b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.892+0000 7f276bfff700 1 -- 172.21.15.44:0/345360003 --> [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2774023e50 con 0x7f276c0add90 2024-04-21T23:07:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.892+0000 7f276bfff700 1 -- 172.21.15.44:0/345360003 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f27740d0470 con 0x7f27841024e0 2024-04-21T23:07:32.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.893+0000 7f278359e700 1 --2- 172.21.15.44:0/345360003 >> [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908] conn(0x7f276c0add90 0x7f276c0b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:32.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.893+0000 7f278259c700 1 --2- 172.21.15.44:0/345360003 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f276c0a2cb0 0x7f276c0a5170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2778006010 tx=0x7f277801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:32.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.893+0000 7f278359e700 1 --2- 172.21.15.44:0/345360003 >> [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908] conn(0x7f276c0add90 0x7f276c0b0230 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:32.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.895+0000 7f276bfff700 1 -- 172.21.15.44:0/345360003 <== osd.9 v2:172.21.15.44:6826/1043222908 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2774023e50 con 0x7f276c0add90 2024-04-21T23:07:32.904 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:32.904 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:32.908 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:32.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.928+0000 7f2789092700 1 -- 172.21.15.44:0/345360003 --> [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2784062dc0 con 0x7f276c0add90 2024-04-21T23:07:32.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.929+0000 7f276bfff700 1 -- 172.21.15.44:0/345360003 <== osd.9 v2:172.21.15.44:6826/1043222908 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2784062dc0 con 0x7f276c0add90 2024-04-21T23:07:32.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.929+0000 7f2789092700 1 -- 172.21.15.44:0/345360003 >> [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908] conn(0x7f276c0add90 msgr2=0x7f276c0b0230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:32.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.929+0000 7f2789092700 1 --2- 172.21.15.44:0/345360003 >> [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908] conn(0x7f276c0add90 0x7f276c0b0230 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:32.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.929+0000 7f2789092700 1 -- 172.21.15.44:0/345360003 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f276c0a2cb0 msgr2=0x7f276c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:32.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.929+0000 7f2789092700 1 --2- 172.21.15.44:0/345360003 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f276c0a2cb0 0x7f276c0a5170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2778006010 tx=0x7f277801a040 comp rx=0 tx=0).stop 2024-04-21T23:07:32.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.929+0000 7f2789092700 1 -- 172.21.15.44:0/345360003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27841024e0 msgr2=0x7f278419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:32.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.930+0000 7f2789092700 1 --2- 172.21.15.44:0/345360003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27841024e0 0x7f278419e6a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f277400e9a0 tx=0x7f277400ed60 comp rx=0 tx=0).stop 2024-04-21T23:07:32.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.930+0000 7f278359e700 1 -- 172.21.15.44:0/345360003 reap_dead start 2024-04-21T23:07:32.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.930+0000 7f2789092700 1 -- 172.21.15.44:0/345360003 shutdown_connections 2024-04-21T23:07:32.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.930+0000 7f2789092700 1 -- 172.21.15.44:0/345360003 >> 172.21.15.44:0/345360003 conn(0x7f2784079390 msgr2=0x7f2784100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:32.930 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.930+0000 7f2789092700 1 -- 172.21.15.44:0/345360003 shutdown_connections 2024-04-21T23:07:32.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:32.930+0000 7f2789092700 1 -- 172.21.15.44:0/345360003 wait complete. 2024-04-21T23:07:33.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:32 smithi044 ceph-mon[27934]: pgmap v326: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-04-21T23:07:33.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:32 smithi055 ceph-mon[31117]: pgmap v326: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-04-21T23:07:33.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:32 smithi069 ceph-mon[31816]: pgmap v326: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-04-21T23:07:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.654+0000 7fd5ce5ad700 1 -- 172.21.15.44:0/922199773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5c8107570 msgr2=0x7fd5c810f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.654+0000 7fd5ce5ad700 1 --2- 172.21.15.44:0/922199773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5c8107570 0x7fd5c810f140 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd5c000b0a0 tx=0x7fd5c000b3b0 comp rx=0 tx=0).stop 2024-04-21T23:07:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.654+0000 7fd5ce5ad700 1 -- 172.21.15.44:0/922199773 shutdown_connections 2024-04-21T23:07:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.654+0000 7fd5ce5ad700 1 --2- 172.21.15.44:0/922199773 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5c8107570 0x7fd5c810f140 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.654+0000 7fd5ce5ad700 1 --2- 172.21.15.44:0/922199773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5c81024e0 0x7fd5c81028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:33.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.654+0000 7fd5ce5ad700 1 --2- 172.21.15.44:0/922199773 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5c8102e90 0x7fd5c8106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.654+0000 7fd5ce5ad700 1 -- 172.21.15.44:0/922199773 >> 172.21.15.44:0/922199773 conn(0x7fd5c8079390 msgr2=0x7fd5c80797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.654+0000 7fd5ce5ad700 1 -- 172.21.15.44:0/922199773 shutdown_connections 2024-04-21T23:07:33.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.654+0000 7fd5ce5ad700 1 -- 172.21.15.44:0/922199773 wait complete. 2024-04-21T23:07:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.655+0000 7fd5ce5ad700 1 Processor -- start 2024-04-21T23:07:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.655+0000 7fd5ce5ad700 1 -- start start 2024-04-21T23:07:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.656+0000 7fd5ce5ad700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5c81024e0 0x7fd5c8074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.656+0000 7fd5ce5ad700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5c8102e90 0x7fd5c8074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:33.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.656+0000 7fd5ce5ad700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5c8107570 0x7fd5c8074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.656+0000 7fd5ce5ad700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c8112600 con 0x7fd5c8102e90 2024-04-21T23:07:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.656+0000 7fd5ce5ad700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c8112480 con 0x7fd5c8107570 2024-04-21T23:07:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.656+0000 7fd5ce5ad700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c8112780 con 0x7fd5c81024e0 2024-04-21T23:07:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.656+0000 7fd5c77fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5c8102e90 0x7fd5c8074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.656+0000 7fd5c77fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5c8102e90 0x7fd5c8074a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42696/0 (socket says 172.21.15.44:42696) 2024-04-21T23:07:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.656+0000 7fd5c77fe700 1 -- 172.21.15.44:0/583625221 learned_addr learned my addr 172.21.15.44:0/583625221 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.656+0000 7fd5ccb4a700 1 --2- 172.21.15.44:0/583625221 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5c8107570 0x7fd5c8074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.656+0000 7fd5c7fff700 1 --2- 172.21.15.44:0/583625221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5c81024e0 0x7fd5c8074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.656+0000 7fd5c77fe700 1 -- 172.21.15.44:0/583625221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5c81024e0 msgr2=0x7fd5c8074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.656+0000 7fd5c77fe700 1 --2- 172.21.15.44:0/583625221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5c81024e0 0x7fd5c8074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.656+0000 7fd5c77fe700 1 -- 172.21.15.44:0/583625221 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5c8107570 msgr2=0x7fd5c8074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.656+0000 7fd5c77fe700 1 --2- 172.21.15.44:0/583625221 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd5c8107570 0x7fd5c8074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.656+0000 7fd5c77fe700 1 -- 172.21.15.44:0/583625221 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5c81a7000 con 0x7fd5c8102e90 2024-04-21T23:07:33.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.656+0000 7fd5c77fe700 1 --2- 172.21.15.44:0/583625221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5c8102e90 0x7fd5c8074a70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc00d980 tx=0x7fd5bc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.656+0000 7fd5c57fa700 1 -- 172.21.15.44:0/583625221 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5bc017070 con 0x7fd5c8102e90 2024-04-21T23:07:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.657+0000 7fd5ce5ad700 1 -- 172.21.15.44:0/583625221 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5c81a72f0 con 0x7fd5c8102e90 2024-04-21T23:07:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.657+0000 7fd5c57fa700 1 -- 172.21.15.44:0/583625221 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5bc00fd70 con 0x7fd5c8102e90 2024-04-21T23:07:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.657+0000 7fd5c57fa700 1 -- 172.21.15.44:0/583625221 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5bc00e3b0 con 0x7fd5c8102e90 2024-04-21T23:07:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.657+0000 7fd5ce5ad700 1 -- 172.21.15.44:0/583625221 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5c81a7830 con 0x7fd5c8102e90 2024-04-21T23:07:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.657+0000 7fd5ce5ad700 1 -- 172.21.15.44:0/583625221 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd5c8062940 con 0x7fd5c8102e90 2024-04-21T23:07:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.660+0000 7fd5c57fa700 1 -- 172.21.15.44:0/583625221 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd5bc00f890 con 0x7fd5c8102e90 2024-04-21T23:07:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.660+0000 7fd5c57fa700 1 --2- 172.21.15.44:0/583625221 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd5b00a2cb0 0x7fd5b00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.661+0000 7fd5c57fa700 1 -- 172.21.15.44:0/583625221 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fd5bc0cf520 con 0x7fd5c8102e90 2024-04-21T23:07:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.661+0000 7fd5c7fff700 1 --2- 172.21.15.44:0/583625221 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd5b00a2cb0 0x7fd5b00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.661+0000 7fd5c57fa700 1 --2- 172.21.15.44:0/583625221 >> [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227] conn(0x7fd5b00add90 0x7fd5b00b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.661+0000 7fd5c57fa700 1 -- 172.21.15.44:0/583625221 --> [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd5bc026e50 con 0x7fd5b00add90 2024-04-21T23:07:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.661+0000 7fd5c57fa700 1 -- 172.21.15.44:0/583625221 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7fd5bc0cf8d0 con 0x7fd5c8102e90 2024-04-21T23:07:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.661+0000 7fd5ccb4a700 1 --2- 172.21.15.44:0/583625221 >> [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227] conn(0x7fd5b00add90 0x7fd5b00b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.661+0000 7fd5c7fff700 1 --2- 172.21.15.44:0/583625221 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd5b00a2cb0 0x7fd5b00a5170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8009a10 tx=0x7fd5b8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.661+0000 7fd5ccb4a700 1 --2- 172.21.15.44:0/583625221 >> [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227] conn(0x7fd5b00add90 0x7fd5b00b0230 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:33.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.662+0000 7fd5c57fa700 1 -- 172.21.15.44:0/583625221 <== osd.3 v2:172.21.15.44:6810/746259227 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd5bc026e50 con 0x7fd5b00add90 2024-04-21T23:07:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.679+0000 7fd5ce5ad700 1 -- 172.21.15.44:0/583625221 --> [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd5c8062d40 con 0x7fd5b00add90 2024-04-21T23:07:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.679+0000 7fd5c57fa700 1 -- 172.21.15.44:0/583625221 <== osd.3 v2:172.21.15.44:6810/746259227 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd5c8062d40 con 0x7fd5b00add90 2024-04-21T23:07:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.680+0000 7fd5ce5ad700 1 -- 172.21.15.44:0/583625221 >> [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227] conn(0x7fd5b00add90 msgr2=0x7fd5b00b0230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.680+0000 7fd5ce5ad700 1 --2- 172.21.15.44:0/583625221 >> [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227] conn(0x7fd5b00add90 0x7fd5b00b0230 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.680+0000 7fd5ce5ad700 1 -- 172.21.15.44:0/583625221 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd5b00a2cb0 msgr2=0x7fd5b00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.680+0000 7fd5ce5ad700 1 --2- 172.21.15.44:0/583625221 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd5b00a2cb0 0x7fd5b00a5170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8009a10 tx=0x7fd5b8009190 comp rx=0 tx=0).stop 2024-04-21T23:07:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.680+0000 7fd5ce5ad700 1 -- 172.21.15.44:0/583625221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5c8102e90 msgr2=0x7fd5c8074a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.680+0000 7fd5ce5ad700 1 --2- 172.21.15.44:0/583625221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd5c8102e90 0x7fd5c8074a70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc00d980 tx=0x7fd5bc00dc90 comp rx=0 tx=0).stop 2024-04-21T23:07:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.680+0000 7fd5ccb4a700 1 -- 172.21.15.44:0/583625221 reap_dead start 2024-04-21T23:07:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.680+0000 7fd5ce5ad700 1 -- 172.21.15.44:0/583625221 shutdown_connections 2024-04-21T23:07:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.680+0000 7fd5ce5ad700 1 -- 172.21.15.44:0/583625221 >> 172.21.15.44:0/583625221 conn(0x7fd5c8079390 msgr2=0x7fd5c8100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.680+0000 7fd5ce5ad700 1 -- 172.21.15.44:0/583625221 shutdown_connections 2024-04-21T23:07:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:33.680+0000 7fd5ce5ad700 1 -- 172.21.15.44:0/583625221 wait complete. 2024-04-21T23:07:34.084 INFO:teuthology.orchestra.run.smithi044.stdout:77309411401 2024-04-21T23:07:34.084 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-21T23:07:35.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:34 smithi044 ceph-mon[27934]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 638 B/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-04-21T23:07:35.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:34 smithi055 ceph-mon[31117]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 638 B/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-04-21T23:07:35.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:34 smithi069 ceph-mon[31816]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 638 B/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-04-21T23:07:35.430 INFO:teuthology.orchestra.run.smithi044.stdout:287762808869 2024-04-21T23:07:35.431 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-21T23:07:36.181 INFO:teuthology.orchestra.run.smithi044.stdout:107374182468 2024-04-21T23:07:36.181 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-21T23:07:36.594 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:36.595 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:36.602 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:37.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:36 smithi044 ceph-mon[27934]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s; 2 B/s, 0 objects/s recovering 2024-04-21T23:07:37.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:36 smithi055 ceph-mon[31117]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s; 2 B/s, 0 objects/s recovering 2024-04-21T23:07:37.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:36 smithi069 ceph-mon[31816]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s; 2 B/s, 0 objects/s recovering 2024-04-21T23:07:37.284 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:37.284 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:37.286 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:37.302 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:39.128 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:38 smithi044 ceph-mon[27934]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1 op/s; 1 B/s, 0 objects/s recovering 2024-04-21T23:07:39.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:38 smithi055 ceph-mon[31117]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1 op/s; 1 B/s, 0 objects/s recovering 2024-04-21T23:07:39.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:38 smithi069 ceph-mon[31816]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 1 op/s; 1 B/s, 0 objects/s recovering 2024-04-21T23:07:41.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:40 smithi044 ceph-mon[27934]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s 2024-04-21T23:07:41.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:40 smithi069 ceph-mon[31816]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s 2024-04-21T23:07:41.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:40 smithi055 ceph-mon[31117]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s 2024-04-21T23:07:42.070 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:43.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:42 smithi044 ceph-mon[27934]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:07:43.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:42 smithi055 ceph-mon[31117]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:07:43.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:42 smithi069 ceph-mon[31816]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:07:43.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.768+0000 7fe03956f700 1 -- 172.21.15.44:0/1418098355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe034107610 msgr2=0x7fe03410f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:43.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.768+0000 7fe03956f700 1 --2- 172.21.15.44:0/1418098355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe034107610 0x7fe03410f190 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe02c00b0d0 tx=0x7fe02c00b3e0 comp rx=0 tx=0).stop 2024-04-21T23:07:43.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.768+0000 7fe03956f700 1 -- 172.21.15.44:0/1418098355 shutdown_connections 2024-04-21T23:07:43.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.768+0000 7fe03956f700 1 --2- 172.21.15.44:0/1418098355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe034107610 0x7fe03410f190 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:43.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.768+0000 7fe03956f700 1 --2- 172.21.15.44:0/1418098355 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe03406ac00 0x7fe03406afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:43.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.768+0000 7fe03956f700 1 --2- 172.21.15.44:0/1418098355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe03406b5b0 0x7fe034106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:43.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.768+0000 7fe03956f700 1 -- 172.21.15.44:0/1418098355 >> 172.21.15.44:0/1418098355 conn(0x7fe034077180 msgr2=0x7fe034077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:43.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.768+0000 7fe03956f700 1 -- 172.21.15.44:0/1418098355 shutdown_connections 2024-04-21T23:07:43.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.768+0000 7fe03956f700 1 -- 172.21.15.44:0/1418098355 wait complete. 2024-04-21T23:07:43.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.769+0000 7fe03956f700 1 Processor -- start 2024-04-21T23:07:43.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.769+0000 7fe03956f700 1 -- start start 2024-04-21T23:07:43.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.769+0000 7fe03956f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe03406ac00 0x7fe034101290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:43.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.769+0000 7fe03956f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe03406b5b0 0x7fe0341017d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:43.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.769+0000 7fe03956f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe034107610 0x7fe034101d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:43.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.769+0000 7fe03956f700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe034112600 con 0x7fe03406b5b0 2024-04-21T23:07:43.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.769+0000 7fe03956f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe034112480 con 0x7fe034107610 2024-04-21T23:07:43.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.769+0000 7fe03956f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe034112780 con 0x7fe03406ac00 2024-04-21T23:07:43.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.769+0000 7fe0327fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe03406b5b0 0x7fe0341017d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:43.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.769+0000 7fe0327fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe03406b5b0 0x7fe0341017d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57178/0 (socket says 172.21.15.44:57178) 2024-04-21T23:07:43.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.769+0000 7fe0327fc700 1 -- 172.21.15.44:0/3298656843 learned_addr learned my addr 172.21.15.44:0/3298656843 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:43.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.769+0000 7fe0337fe700 1 --2- 172.21.15.44:0/3298656843 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe034107610 0x7fe034101d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:43.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.769+0000 7fe032ffd700 1 --2- 172.21.15.44:0/3298656843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe03406ac00 0x7fe034101290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:43.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.770+0000 7fe0327fc700 1 -- 172.21.15.44:0/3298656843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe03406ac00 msgr2=0x7fe034101290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:43.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.770+0000 7fe0327fc700 1 --2- 172.21.15.44:0/3298656843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe03406ac00 0x7fe034101290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:43.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.770+0000 7fe0327fc700 1 -- 172.21.15.44:0/3298656843 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe034107610 msgr2=0x7fe034101d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:43.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.770+0000 7fe0327fc700 1 --2- 172.21.15.44:0/3298656843 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe034107610 0x7fe034101d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:43.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.770+0000 7fe0327fc700 1 -- 172.21.15.44:0/3298656843 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0341a6f20 con 0x7fe03406b5b0 2024-04-21T23:07:43.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.770+0000 7fe032ffd700 1 --2- 172.21.15.44:0/3298656843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe03406ac00 0x7fe034101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:07:43.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.770+0000 7fe0327fc700 1 --2- 172.21.15.44:0/3298656843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe03406b5b0 0x7fe0341017d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe01c00d950 tx=0x7fe01c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:43.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.770+0000 7fe02bfff700 1 -- 172.21.15.44:0/3298656843 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe01c016070 con 0x7fe03406b5b0 2024-04-21T23:07:43.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.770+0000 7fe03956f700 1 -- 172.21.15.44:0/3298656843 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0341a7210 con 0x7fe03406b5b0 2024-04-21T23:07:43.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.770+0000 7fe02bfff700 1 -- 172.21.15.44:0/3298656843 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe01c004d10 con 0x7fe03406b5b0 2024-04-21T23:07:43.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.770+0000 7fe02bfff700 1 -- 172.21.15.44:0/3298656843 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe01c005610 con 0x7fe03406b5b0 2024-04-21T23:07:43.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.770+0000 7fe03956f700 1 -- 172.21.15.44:0/3298656843 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0341a77a0 con 0x7fe03406b5b0 2024-04-21T23:07:43.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.772+0000 7fe03956f700 1 -- 172.21.15.44:0/3298656843 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe034068f20 con 0x7fe03406b5b0 2024-04-21T23:07:43.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.772+0000 7fe02bfff700 1 -- 172.21.15.44:0/3298656843 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fe01c004750 con 0x7fe03406b5b0 2024-04-21T23:07:43.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.773+0000 7fe02bfff700 1 --2- 172.21.15.44:0/3298656843 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe0200a2cb0 0x7fe0200a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:43.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.773+0000 7fe032ffd700 1 --2- 172.21.15.44:0/3298656843 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe0200a2cb0 0x7fe0200a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:43.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.774+0000 7fe02bfff700 1 -- 172.21.15.44:0/3298656843 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fe01c0cf290 con 0x7fe03406b5b0 2024-04-21T23:07:43.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.774+0000 7fe032ffd700 1 --2- 172.21.15.44:0/3298656843 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe0200a2cb0 0x7fe0200a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe024009cb0 tx=0x7fe024009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:43.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.778+0000 7fe02bfff700 1 -- 172.21.15.44:0/3298656843 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe01c01f020 con 0x7fe03406b5b0 2024-04-21T23:07:43.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.987+0000 7fe03956f700 1 -- 172.21.15.44:0/3298656843 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fe03402db30 con 0x7fe03406b5b0 2024-04-21T23:07:43.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.987+0000 7fe02bfff700 1 -- 172.21.15.44:0/3298656843 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fe01c09bd50 con 0x7fe03406b5b0 2024-04-21T23:07:43.987 INFO:teuthology.orchestra.run.smithi044.stdout:347892351005 2024-04-21T23:07:43.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.989+0000 7fe03956f700 1 -- 172.21.15.44:0/3298656843 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe0200a2cb0 msgr2=0x7fe0200a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:43.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.989+0000 7fe03956f700 1 --2- 172.21.15.44:0/3298656843 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe0200a2cb0 0x7fe0200a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe024009cb0 tx=0x7fe024009260 comp rx=0 tx=0).stop 2024-04-21T23:07:43.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.989+0000 7fe03956f700 1 -- 172.21.15.44:0/3298656843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe03406b5b0 msgr2=0x7fe0341017d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:43.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.989+0000 7fe03956f700 1 --2- 172.21.15.44:0/3298656843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe03406b5b0 0x7fe0341017d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe01c00d950 tx=0x7fe01c00dc60 comp rx=0 tx=0).stop 2024-04-21T23:07:43.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.990+0000 7fe03956f700 1 -- 172.21.15.44:0/3298656843 shutdown_connections 2024-04-21T23:07:43.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.990+0000 7fe03956f700 1 --2- 172.21.15.44:0/3298656843 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe034107610 0x7fe034101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:43.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.990+0000 7fe03956f700 1 --2- 172.21.15.44:0/3298656843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe03406ac00 0x7fe034101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:43.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.990+0000 7fe03956f700 1 --2- 172.21.15.44:0/3298656843 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fe0200a2cb0 0x7fe0200a5170 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:43.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.990+0000 7fe03956f700 1 --2- 172.21.15.44:0/3298656843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe03406b5b0 0x7fe0341017d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:43.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.990+0000 7fe03956f700 1 -- 172.21.15.44:0/3298656843 >> 172.21.15.44:0/3298656843 conn(0x7fe034077180 msgr2=0x7fe0340ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:43.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.990+0000 7fe03956f700 1 -- 172.21.15.44:0/3298656843 shutdown_connections 2024-04-21T23:07:43.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:43.990+0000 7fe03956f700 1 -- 172.21.15.44:0/3298656843 wait complete. 2024-04-21T23:07:45.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:44 smithi044 ceph-mon[27934]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:07:45.186 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:44 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3298656843' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-21T23:07:45.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:44 smithi055 ceph-mon[31117]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:07:45.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:44 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3298656843' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-21T23:07:45.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:44 smithi069 ceph-mon[31816]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:07:45.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:44 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3298656843' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-21T23:07:45.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.734+0000 7fc5a0f3c700 1 -- 172.21.15.44:0/832922835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc59c102e90 msgr2=0x7fc59c106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:45.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.734+0000 7fc5a0f3c700 1 --2- 172.21.15.44:0/832922835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc59c102e90 0x7fc59c106e90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc590009990 tx=0x7fc590009ca0 comp rx=0 tx=0).stop 2024-04-21T23:07:45.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.734+0000 7fc5a0f3c700 1 -- 172.21.15.44:0/832922835 shutdown_connections 2024-04-21T23:07:45.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.734+0000 7fc5a0f3c700 1 --2- 172.21.15.44:0/832922835 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc59c107570 0x7fc59c10f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:45.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.734+0000 7fc5a0f3c700 1 --2- 172.21.15.44:0/832922835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc59c1024e0 0x7fc59c1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:45.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.734+0000 7fc5a0f3c700 1 --2- 172.21.15.44:0/832922835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc59c102e90 0x7fc59c106e90 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:45.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.734+0000 7fc5a0f3c700 1 -- 172.21.15.44:0/832922835 >> 172.21.15.44:0/832922835 conn(0x7fc59c079390 msgr2=0x7fc59c0797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:45.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.734+0000 7fc5a0f3c700 1 -- 172.21.15.44:0/832922835 shutdown_connections 2024-04-21T23:07:45.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.735+0000 7fc5a0f3c700 1 -- 172.21.15.44:0/832922835 wait complete. 2024-04-21T23:07:45.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.735+0000 7fc5a0f3c700 1 Processor -- start 2024-04-21T23:07:45.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.735+0000 7fc5a0f3c700 1 -- start start 2024-04-21T23:07:45.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.735+0000 7fc5a0f3c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc59c1024e0 0x7fc59c074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:45.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.735+0000 7fc5a0f3c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc59c102e90 0x7fc59c074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:45.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.735+0000 7fc5a0f3c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc59c107570 0x7fc59c074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:45.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.735+0000 7fc5a0f3c700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc59c112510 con 0x7fc59c1024e0 2024-04-21T23:07:45.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.735+0000 7fc5a0f3c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc59c112390 con 0x7fc59c102e90 2024-04-21T23:07:45.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.735+0000 7fc5a0f3c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc59c112690 con 0x7fc59c107570 2024-04-21T23:07:45.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.736+0000 7fc59a59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc59c1024e0 0x7fc59c074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:45.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.736+0000 7fc59a59c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc59c1024e0 0x7fc59c074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57182/0 (socket says 172.21.15.44:57182) 2024-04-21T23:07:45.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.736+0000 7fc59a59c700 1 -- 172.21.15.44:0/1128075283 learned_addr learned my addr 172.21.15.44:0/1128075283 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:45.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.736+0000 7fc599d9b700 1 --2- 172.21.15.44:0/1128075283 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc59c102e90 0x7fc59c074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:45.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.736+0000 7fc59ad9d700 1 --2- 172.21.15.44:0/1128075283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc59c107570 0x7fc59c074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:45.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.736+0000 7fc59a59c700 1 -- 172.21.15.44:0/1128075283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc59c107570 msgr2=0x7fc59c074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:45.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.736+0000 7fc59a59c700 1 --2- 172.21.15.44:0/1128075283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc59c107570 0x7fc59c074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:45.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.736+0000 7fc59a59c700 1 -- 172.21.15.44:0/1128075283 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc59c102e90 msgr2=0x7fc59c074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:45.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.736+0000 7fc59a59c700 1 --2- 172.21.15.44:0/1128075283 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc59c102e90 0x7fc59c074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:45.737 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.736+0000 7fc59a59c700 1 -- 172.21.15.44:0/1128075283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc59c1a6e90 con 0x7fc59c1024e0 2024-04-21T23:07:45.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.736+0000 7fc599d9b700 1 --2- 172.21.15.44:0/1128075283 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc59c102e90 0x7fc59c074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:07:45.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.736+0000 7fc59a59c700 1 --2- 172.21.15.44:0/1128075283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc59c1024e0 0x7fc59c074530 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc58c00c9b0 tx=0x7fc58c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:45.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.736+0000 7fc58b7fe700 1 -- 172.21.15.44:0/1128075283 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58c007b00 con 0x7fc59c1024e0 2024-04-21T23:07:45.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.736+0000 7fc5a0f3c700 1 -- 172.21.15.44:0/1128075283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc59c1a7180 con 0x7fc59c1024e0 2024-04-21T23:07:45.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.736+0000 7fc58b7fe700 1 -- 172.21.15.44:0/1128075283 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc58c00ce50 con 0x7fc59c1024e0 2024-04-21T23:07:45.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.736+0000 7fc58b7fe700 1 -- 172.21.15.44:0/1128075283 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58c018660 con 0x7fc59c1024e0 2024-04-21T23:07:45.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.737+0000 7fc5a0f3c700 1 -- 172.21.15.44:0/1128075283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc59c1a76e0 con 0x7fc59c1024e0 2024-04-21T23:07:45.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.738+0000 7fc5a0f3c700 1 -- 172.21.15.44:0/1128075283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc59c068f20 con 0x7fc59c1024e0 2024-04-21T23:07:45.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.739+0000 7fc58b7fe700 1 -- 172.21.15.44:0/1128075283 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc58c00f450 con 0x7fc59c1024e0 2024-04-21T23:07:45.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.739+0000 7fc58b7fe700 1 --2- 172.21.15.44:0/1128075283 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc5840a2cb0 0x7fc5840a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:45.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.739+0000 7fc599d9b700 1 --2- 172.21.15.44:0/1128075283 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc5840a2cb0 0x7fc5840a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:45.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.739+0000 7fc58b7fe700 1 -- 172.21.15.44:0/1128075283 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fc58c0cf440 con 0x7fc59c1024e0 2024-04-21T23:07:45.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.740+0000 7fc599d9b700 1 --2- 172.21.15.44:0/1128075283 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc5840a2cb0 0x7fc5840a5170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc59000b2e0 tx=0x7fc590005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:45.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.744+0000 7fc58b7fe700 1 -- 172.21.15.44:0/1128075283 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc58c014030 con 0x7fc59c1024e0 2024-04-21T23:07:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.954+0000 7fc5a0f3c700 1 -- 172.21.15.44:0/1128075283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fc59c02db30 con 0x7fc59c1024e0 2024-04-21T23:07:45.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.954+0000 7fc58b7fe700 1 -- 172.21.15.44:0/1128075283 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc58c09bf00 con 0x7fc59c1024e0 2024-04-21T23:07:45.954 INFO:teuthology.orchestra.run.smithi044.stdout:219043332144 2024-04-21T23:07:45.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.956+0000 7fc5a0f3c700 1 -- 172.21.15.44:0/1128075283 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc5840a2cb0 msgr2=0x7fc5840a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:45.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.956+0000 7fc5a0f3c700 1 --2- 172.21.15.44:0/1128075283 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc5840a2cb0 0x7fc5840a5170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc59000b2e0 tx=0x7fc590005960 comp rx=0 tx=0).stop 2024-04-21T23:07:45.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.956+0000 7fc5a0f3c700 1 -- 172.21.15.44:0/1128075283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc59c1024e0 msgr2=0x7fc59c074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:45.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.956+0000 7fc5a0f3c700 1 --2- 172.21.15.44:0/1128075283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc59c1024e0 0x7fc59c074530 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc58c00c9b0 tx=0x7fc58c00ccc0 comp rx=0 tx=0).stop 2024-04-21T23:07:45.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.956+0000 7fc5a0f3c700 1 -- 172.21.15.44:0/1128075283 shutdown_connections 2024-04-21T23:07:45.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.956+0000 7fc5a0f3c700 1 --2- 172.21.15.44:0/1128075283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc59c107570 0x7fc59c074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:45.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.956+0000 7fc5a0f3c700 1 --2- 172.21.15.44:0/1128075283 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc5840a2cb0 0x7fc5840a5170 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:45.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.956+0000 7fc5a0f3c700 1 --2- 172.21.15.44:0/1128075283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc59c1024e0 0x7fc59c074530 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:45.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.956+0000 7fc5a0f3c700 1 --2- 172.21.15.44:0/1128075283 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc59c102e90 0x7fc59c074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:45.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.956+0000 7fc5a0f3c700 1 -- 172.21.15.44:0/1128075283 >> 172.21.15.44:0/1128075283 conn(0x7fc59c079390 msgr2=0x7fc59c100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:45.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.957+0000 7fc5a0f3c700 1 -- 172.21.15.44:0/1128075283 shutdown_connections 2024-04-21T23:07:45.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:45.957+0000 7fc5a0f3c700 1 -- 172.21.15.44:0/1128075283 wait complete. 2024-04-21T23:07:46.395 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892351001 got 347892351005 for osd.11 2024-04-21T23:07:46.395 DEBUG:teuthology.parallel:result is None 2024-04-21T23:07:47.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:46 smithi044 ceph-mon[27934]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:07:47.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:46 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1128075283' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-21T23:07:47.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:46 smithi055 ceph-mon[31117]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:07:47.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:46 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1128075283' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-21T23:07:47.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:46 smithi069 ceph-mon[31816]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:07:47.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:46 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1128075283' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-21T23:07:47.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.929+0000 7f98c9c3c700 1 -- 172.21.15.44:0/799003007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c406b2a0 msgr2=0x7f98c410f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:47.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.929+0000 7f98c9c3c700 1 --2- 172.21.15.44:0/799003007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c406b2a0 0x7f98c410f110 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00b0a0 tx=0x7f98bc00b3b0 comp rx=0 tx=0).stop 2024-04-21T23:07:47.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.929+0000 7f98c9c3c700 1 -- 172.21.15.44:0/799003007 shutdown_connections 2024-04-21T23:07:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.929+0000 7f98c9c3c700 1 --2- 172.21.15.44:0/799003007 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c406b2a0 0x7f98c410f110 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.929+0000 7f98c9c3c700 1 --2- 172.21.15.44:0/799003007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98c41082d0 0x7f98c41086b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.929+0000 7f98c9c3c700 1 --2- 172.21.15.44:0/799003007 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98c406a900 0x7f98c406ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.929+0000 7f98c9c3c700 1 -- 172.21.15.44:0/799003007 >> 172.21.15.44:0/799003007 conn(0x7f98c40780e0 msgr2=0x7f98c40784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.929+0000 7f98c9c3c700 1 -- 172.21.15.44:0/799003007 shutdown_connections 2024-04-21T23:07:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.929+0000 7f98c9c3c700 1 -- 172.21.15.44:0/799003007 wait complete. 2024-04-21T23:07:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.930+0000 7f98c9c3c700 1 Processor -- start 2024-04-21T23:07:47.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.930+0000 7f98c9c3c700 1 -- start start 2024-04-21T23:07:47.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.930+0000 7f98c9c3c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c406a900 0x7f98c419e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:47.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.930+0000 7f98c9c3c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98c406b2a0 0x7f98c419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:47.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.930+0000 7f98c9c3c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98c41082d0 0x7f98c4198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:47.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.930+0000 7f98c9c3c700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f98c4112520 con 0x7f98c406a900 2024-04-21T23:07:47.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.930+0000 7f98c9c3c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41123a0 con 0x7f98c406b2a0 2024-04-21T23:07:47.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.930+0000 7f98c9c3c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41126a0 con 0x7f98c41082d0 2024-04-21T23:07:47.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.930+0000 7f98c37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c406a900 0x7f98c419e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:47.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.930+0000 7f98c37fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c406a900 0x7f98c419e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57206/0 (socket says 172.21.15.44:57206) 2024-04-21T23:07:47.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.930+0000 7f98c37fe700 1 -- 172.21.15.44:0/11603835 learned_addr learned my addr 172.21.15.44:0/11603835 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:47.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.930+0000 7f98c2ffd700 1 --2- 172.21.15.44:0/11603835 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98c406b2a0 0x7f98c419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:47.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.930+0000 7f98c3fff700 1 --2- 172.21.15.44:0/11603835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98c41082d0 0x7f98c4198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:47.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.931+0000 7f98c37fe700 1 -- 172.21.15.44:0/11603835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98c41082d0 msgr2=0x7f98c4198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:47.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.931+0000 7f98c37fe700 1 --2- 172.21.15.44:0/11603835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98c41082d0 0x7f98c4198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:47.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.931+0000 7f98c37fe700 1 -- 172.21.15.44:0/11603835 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98c406b2a0 msgr2=0x7f98c419eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:47.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.931+0000 7f98c37fe700 1 --2- 172.21.15.44:0/11603835 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98c406b2a0 0x7f98c419eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:47.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.931+0000 7f98c37fe700 1 -- 172.21.15.44:0/11603835 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c4198fe0 con 0x7f98c406a900 2024-04-21T23:07:47.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.931+0000 7f98c37fe700 1 --2- 172.21.15.44:0/11603835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c406a900 0x7f98c419e630 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f98b400c920 tx=0x7f98b400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:47.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.931+0000 7f98c0ff9700 1 -- 172.21.15.44:0/11603835 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b4007b00 con 0x7f98c406a900 2024-04-21T23:07:47.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.931+0000 7f98c9c3c700 1 -- 172.21.15.44:0/11603835 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98c41992d0 con 0x7f98c406a900 2024-04-21T23:07:47.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.931+0000 7f98c0ff9700 1 -- 172.21.15.44:0/11603835 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98b4004d10 con 0x7f98c406a900 2024-04-21T23:07:47.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.931+0000 7f98c0ff9700 1 -- 172.21.15.44:0/11603835 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b4005630 con 0x7f98c406a900 2024-04-21T23:07:47.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.931+0000 7f98c9c3c700 1 -- 172.21.15.44:0/11603835 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98c41a55b0 con 0x7f98c406a900 2024-04-21T23:07:47.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.932+0000 7f98c9c3c700 1 -- 172.21.15.44:0/11603835 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98c406a1b0 con 0x7f98c406a900 2024-04-21T23:07:47.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.933+0000 7f98c0ff9700 1 -- 172.21.15.44:0/11603835 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f98b4004750 con 0x7f98c406a900 2024-04-21T23:07:47.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.934+0000 7f98c0ff9700 1 --2- 172.21.15.44:0/11603835 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f98ac0a2cb0 0x7f98ac0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:47.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.934+0000 7f98c2ffd700 1 --2- 172.21.15.44:0/11603835 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f98ac0a2cb0 0x7f98ac0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:47.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.934+0000 7f98c0ff9700 1 -- 172.21.15.44:0/11603835 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f98b40cf000 con 0x7f98c406a900 2024-04-21T23:07:47.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.935+0000 7f98c2ffd700 1 --2- 172.21.15.44:0/11603835 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f98ac0a2cb0 0x7f98ac0a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f98c4199ec0 tx=0x7f98b800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:47.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:47.939+0000 7f98c0ff9700 1 -- 172.21.15.44:0/11603835 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f98b4014030 con 0x7f98c406a900 2024-04-21T23:07:48.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.153+0000 7f98c9c3c700 1 -- 172.21.15.44:0/11603835 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f98c4103980 con 0x7f98c406a900 2024-04-21T23:07:48.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.153+0000 7f98c0ff9700 1 -- 172.21.15.44:0/11603835 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f98b409bac0 con 0x7f98c406a900 2024-04-21T23:07:48.152 INFO:teuthology.orchestra.run.smithi044.stdout:197568495671 2024-04-21T23:07:48.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.155+0000 7f98c9c3c700 1 -- 172.21.15.44:0/11603835 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f98ac0a2cb0 msgr2=0x7f98ac0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:48.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.155+0000 7f98c9c3c700 1 --2- 172.21.15.44:0/11603835 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f98ac0a2cb0 0x7f98ac0a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f98c4199ec0 tx=0x7f98b800a380 comp rx=0 tx=0).stop 2024-04-21T23:07:48.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.155+0000 7f98c9c3c700 1 -- 172.21.15.44:0/11603835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c406a900 msgr2=0x7f98c419e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:48.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.155+0000 7f98c9c3c700 1 --2- 172.21.15.44:0/11603835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c406a900 0x7f98c419e630 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f98b400c920 tx=0x7f98b400cc30 comp rx=0 tx=0).stop 2024-04-21T23:07:48.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.155+0000 7f98c9c3c700 1 -- 172.21.15.44:0/11603835 shutdown_connections 2024-04-21T23:07:48.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.155+0000 7f98c9c3c700 1 --2- 172.21.15.44:0/11603835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98c41082d0 0x7f98c4198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:48.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.155+0000 7f98c9c3c700 1 --2- 172.21.15.44:0/11603835 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f98ac0a2cb0 0x7f98ac0a5170 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:48.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.155+0000 7f98c9c3c700 1 --2- 172.21.15.44:0/11603835 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f98c406a900 0x7f98c419e630 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:48.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.155+0000 7f98c9c3c700 1 --2- 172.21.15.44:0/11603835 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f98c406b2a0 0x7f98c419eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:48.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.155+0000 7f98c9c3c700 1 -- 172.21.15.44:0/11603835 >> 172.21.15.44:0/11603835 conn(0x7f98c40780e0 msgr2=0x7f98c4100040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:48.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.156+0000 7f98c9c3c700 1 -- 172.21.15.44:0/11603835 shutdown_connections 2024-04-21T23:07:48.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.156+0000 7f98c9c3c700 1 -- 172.21.15.44:0/11603835 wait complete. 2024-04-21T23:07:48.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.551+0000 7f440228a700 1 -- 172.21.15.44:0/2898550070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43fc107650 msgr2=0x7f43fc10f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:48.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.551+0000 7f440228a700 1 --2- 172.21.15.44:0/2898550070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43fc107650 0x7f43fc10f1b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f43f400b0d0 tx=0x7f43f400b3e0 comp rx=0 tx=0).stop 2024-04-21T23:07:48.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.551+0000 7f440228a700 1 -- 172.21.15.44:0/2898550070 shutdown_connections 2024-04-21T23:07:48.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.551+0000 7f440228a700 1 --2- 172.21.15.44:0/2898550070 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43fc107650 0x7f43fc10f1b0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:48.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.551+0000 7f440228a700 1 --2- 172.21.15.44:0/2898550070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43fc100e30 0x7f43fc101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:48.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.551+0000 7f440228a700 1 --2- 172.21.15.44:0/2898550070 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43fc1017e0 0x7f43fc106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:48.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.551+0000 7f440228a700 1 -- 172.21.15.44:0/2898550070 >> 172.21.15.44:0/2898550070 conn(0x7f43fc076450 msgr2=0x7f43fc076860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:48.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.551+0000 7f440228a700 1 -- 172.21.15.44:0/2898550070 shutdown_connections 2024-04-21T23:07:48.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.551+0000 7f440228a700 1 -- 172.21.15.44:0/2898550070 wait complete. 2024-04-21T23:07:48.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.552+0000 7f440228a700 1 Processor -- start 2024-04-21T23:07:48.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.552+0000 7f440228a700 1 -- start start 2024-04-21T23:07:48.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.552+0000 7f440228a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43fc100e30 0x7f43fc19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:48.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.552+0000 7f440228a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43fc1017e0 0x7f43fc19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:48.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.552+0000 7f440228a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43fc107650 0x7f43fc198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:48.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.552+0000 7f440228a700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f43fc112550 con 0x7f43fc1017e0 2024-04-21T23:07:48.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.552+0000 7f440228a700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f43fc1123d0 con 0x7f43fc100e30 2024-04-21T23:07:48.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.552+0000 7f440228a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f43fc1126d0 con 0x7f43fc107650 2024-04-21T23:07:48.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.552+0000 7f43fb7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43fc1017e0 0x7f43fc19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:48.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.552+0000 7f43fb7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43fc1017e0 0x7f43fc19eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57224/0 (socket says 172.21.15.44:57224) 2024-04-21T23:07:48.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.553+0000 7f43fb7fe700 1 -- 172.21.15.44:0/443898365 learned_addr learned my addr 172.21.15.44:0/443898365 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.553+0000 7f43fbfff700 1 --2- 172.21.15.44:0/443898365 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43fc100e30 0x7f43fc19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.553+0000 7f4400827700 1 --2- 172.21.15.44:0/443898365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43fc107650 0x7f43fc198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.553+0000 7f43fb7fe700 1 -- 172.21.15.44:0/443898365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43fc107650 msgr2=0x7f43fc198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.553+0000 7f43fb7fe700 1 --2- 172.21.15.44:0/443898365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43fc107650 0x7f43fc198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.553+0000 7f43fb7fe700 1 -- 172.21.15.44:0/443898365 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43fc100e30 msgr2=0x7f43fc19e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.553+0000 7f43fb7fe700 1 --2- 172.21.15.44:0/443898365 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43fc100e30 0x7f43fc19e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.553+0000 7f43fb7fe700 1 -- 172.21.15.44:0/443898365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43fc199010 con 0x7f43fc1017e0 2024-04-21T23:07:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.553+0000 7f4400827700 1 --2- 172.21.15.44:0/443898365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43fc107650 0x7f43fc198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:07:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.553+0000 7f43fb7fe700 1 --2- 172.21.15.44:0/443898365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43fc1017e0 0x7f43fc19eba0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f43f000d950 tx=0x7f43f000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.553+0000 7f43f97fa700 1 -- 172.21.15.44:0/443898365 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f0016070 con 0x7f43fc1017e0 2024-04-21T23:07:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.553+0000 7f440228a700 1 -- 172.21.15.44:0/443898365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43fc199300 con 0x7f43fc1017e0 2024-04-21T23:07:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.553+0000 7f43f97fa700 1 -- 172.21.15.44:0/443898365 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43f0009e20 con 0x7f43fc1017e0 2024-04-21T23:07:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.553+0000 7f43f97fa700 1 -- 172.21.15.44:0/443898365 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f000f620 con 0x7f43fc1017e0 2024-04-21T23:07:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.553+0000 7f440228a700 1 -- 172.21.15.44:0/443898365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43fc1a55e0 con 0x7f43fc1017e0 2024-04-21T23:07:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.554+0000 7f440228a700 1 -- 172.21.15.44:0/443898365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43fc06a480 con 0x7f43fc1017e0 2024-04-21T23:07:48.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.556+0000 7f43f97fa700 1 -- 172.21.15.44:0/443898365 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f43f0019070 con 0x7f43fc1017e0 2024-04-21T23:07:48.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.556+0000 7f43f97fa700 1 --2- 172.21.15.44:0/443898365 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f43e40a2cb0 0x7f43e40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:48.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.556+0000 7f43fbfff700 1 --2- 172.21.15.44:0/443898365 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f43e40a2cb0 0x7f43e40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:48.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.556+0000 7f43f97fa700 1 -- 172.21.15.44:0/443898365 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f43f00cf0d0 con 0x7f43fc1017e0 2024-04-21T23:07:48.557 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.557+0000 7f43fbfff700 1 --2- 172.21.15.44:0/443898365 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f43e40a2cb0 0x7f43e40a5170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f43ec005950 tx=0x7f43ec00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:48.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.560+0000 7f43f97fa700 1 -- 172.21.15.44:0/443898365 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f43f009bb90 con 0x7f43fc1017e0 2024-04-21T23:07:48.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.768+0000 7f440228a700 1 -- 172.21.15.44:0/443898365 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f43fc02db30 con 0x7f43fc1017e0 2024-04-21T23:07:48.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.769+0000 7f43f97fa700 1 -- 172.21.15.44:0/443898365 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f43f009b490 con 0x7f43fc1017e0 2024-04-21T23:07:48.769 INFO:teuthology.orchestra.run.smithi044.stdout:313532612643 2024-04-21T23:07:48.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.771+0000 7f440228a700 1 -- 172.21.15.44:0/443898365 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f43e40a2cb0 msgr2=0x7f43e40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:48.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.771+0000 7f440228a700 1 --2- 172.21.15.44:0/443898365 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f43e40a2cb0 0x7f43e40a5170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f43ec005950 tx=0x7f43ec00b410 comp rx=0 tx=0).stop 2024-04-21T23:07:48.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.771+0000 7f440228a700 1 -- 172.21.15.44:0/443898365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43fc1017e0 msgr2=0x7f43fc19eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:48.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.771+0000 7f440228a700 1 --2- 172.21.15.44:0/443898365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43fc1017e0 0x7f43fc19eba0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f43f000d950 tx=0x7f43f000dc60 comp rx=0 tx=0).stop 2024-04-21T23:07:48.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.771+0000 7f440228a700 1 -- 172.21.15.44:0/443898365 shutdown_connections 2024-04-21T23:07:48.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.771+0000 7f440228a700 1 --2- 172.21.15.44:0/443898365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43fc107650 0x7f43fc198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:48.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.771+0000 7f440228a700 1 --2- 172.21.15.44:0/443898365 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f43fc100e30 0x7f43fc19e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:48.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.771+0000 7f440228a700 1 --2- 172.21.15.44:0/443898365 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f43e40a2cb0 0x7f43e40a5170 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:48.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.771+0000 7f440228a700 1 --2- 172.21.15.44:0/443898365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43fc1017e0 0x7f43fc19eba0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:48.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.771+0000 7f440228a700 1 -- 172.21.15.44:0/443898365 >> 172.21.15.44:0/443898365 conn(0x7f43fc076450 msgr2=0x7f43fc0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:48.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.771+0000 7f440228a700 1 -- 172.21.15.44:0/443898365 shutdown_connections 2024-04-21T23:07:48.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:48.771+0000 7f440228a700 1 -- 172.21.15.44:0/443898365 wait complete. 2024-04-21T23:07:49.108 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:48 smithi044 ceph-mon[27934]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:07:49.108 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:48 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/11603835' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-21T23:07:49.109 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:48 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/443898365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-21T23:07:49.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:48 smithi055 ceph-mon[31117]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:07:49.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:48 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/11603835' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-21T23:07:49.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:48 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/443898365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-21T23:07:49.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:48 smithi069 ceph-mon[31816]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:07:49.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:48 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/11603835' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-21T23:07:49.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:48 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/443898365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-21T23:07:49.442 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612637 got 313532612643 for osd.10 2024-04-21T23:07:49.442 DEBUG:teuthology.parallel:result is None 2024-04-21T23:07:50.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.585+0000 7fba4099d700 1 -- 172.21.15.44:0/3586192711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba3c109230 msgr2=0x7fba3c109610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.585+0000 7fba4099d700 1 --2- 172.21.15.44:0/3586192711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba3c109230 0x7fba3c109610 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fba2c009990 tx=0x7fba2c009ca0 comp rx=0 tx=0).stop 2024-04-21T23:07:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.585+0000 7fba4099d700 1 -- 172.21.15.44:0/3586192711 shutdown_connections 2024-04-21T23:07:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.585+0000 7fba4099d700 1 --2- 172.21.15.44:0/3586192711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba3c115380 0x7fba3c117770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.585+0000 7fba4099d700 1 --2- 172.21.15.44:0/3586192711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba3c109230 0x7fba3c109610 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.585+0000 7fba4099d700 1 --2- 172.21.15.44:0/3586192711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fba3c109b50 0x7fba3c114e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.585+0000 7fba4099d700 1 -- 172.21.15.44:0/3586192711 >> 172.21.15.44:0/3586192711 conn(0x7fba3c06d680 msgr2=0x7fba3c06da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.585+0000 7fba4099d700 1 -- 172.21.15.44:0/3586192711 shutdown_connections 2024-04-21T23:07:50.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.586+0000 7fba4099d700 1 -- 172.21.15.44:0/3586192711 wait complete. 2024-04-21T23:07:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.586+0000 7fba4099d700 1 Processor -- start 2024-04-21T23:07:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.587+0000 7fba4099d700 1 -- start start 2024-04-21T23:07:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.587+0000 7fba4099d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba3c109230 0x7fba3c1a3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.587+0000 7fba4099d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba3c109b50 0x7fba3c1a3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.587+0000 7fba4099d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fba3c115380 0x7fba3c1a7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.587+0000 7fba4099d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c11ac80 con 0x7fba3c109230 2024-04-21T23:07:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.587+0000 7fba4099d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c11ab00 con 0x7fba3c115380 2024-04-21T23:07:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.587+0000 7fba4099d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c11ae00 con 0x7fba3c109b50 2024-04-21T23:07:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.587+0000 7fba3b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba3c109230 0x7fba3c1a3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.587+0000 7fba3b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba3c109230 0x7fba3c1a3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57238/0 (socket says 172.21.15.44:57238) 2024-04-21T23:07:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.587+0000 7fba3b7fe700 1 -- 172.21.15.44:0/2306227609 learned_addr learned my addr 172.21.15.44:0/2306227609 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.587+0000 7fba3affd700 1 --2- 172.21.15.44:0/2306227609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba3c109b50 0x7fba3c1a3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.587+0000 7fba3bfff700 1 --2- 172.21.15.44:0/2306227609 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fba3c115380 0x7fba3c1a7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.587+0000 7fba3b7fe700 1 -- 172.21.15.44:0/2306227609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba3c109b50 msgr2=0x7fba3c1a3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.587+0000 7fba3b7fe700 1 --2- 172.21.15.44:0/2306227609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba3c109b50 0x7fba3c1a3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.587+0000 7fba3b7fe700 1 -- 172.21.15.44:0/2306227609 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fba3c115380 msgr2=0x7fba3c1a7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.587+0000 7fba3b7fe700 1 --2- 172.21.15.44:0/2306227609 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fba3c115380 0x7fba3c1a7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.587+0000 7fba3b7fe700 1 -- 172.21.15.44:0/2306227609 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba3c1a8350 con 0x7fba3c109230 2024-04-21T23:07:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.588+0000 7fba3b7fe700 1 --2- 172.21.15.44:0/2306227609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba3c109230 0x7fba3c1a3340 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fba2c000c00 tx=0x7fba2c003890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.588+0000 7fba38ff9700 1 -- 172.21.15.44:0/2306227609 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c027070 con 0x7fba3c109230 2024-04-21T23:07:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.588+0000 7fba4099d700 1 -- 172.21.15.44:0/2306227609 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba3c1a85e0 con 0x7fba3c109230 2024-04-21T23:07:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.588+0000 7fba38ff9700 1 -- 172.21.15.44:0/2306227609 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba2c003e40 con 0x7fba3c109230 2024-04-21T23:07:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.588+0000 7fba38ff9700 1 -- 172.21.15.44:0/2306227609 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c022c90 con 0x7fba3c109230 2024-04-21T23:07:50.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.588+0000 7fba4099d700 1 -- 172.21.15.44:0/2306227609 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba3c1afec0 con 0x7fba3c109230 2024-04-21T23:07:50.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.589+0000 7fba4099d700 1 -- 172.21.15.44:0/2306227609 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba3c068f20 con 0x7fba3c109230 2024-04-21T23:07:50.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.590+0000 7fba38ff9700 1 -- 172.21.15.44:0/2306227609 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fba2c02a070 con 0x7fba3c109230 2024-04-21T23:07:50.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.591+0000 7fba38ff9700 1 --2- 172.21.15.44:0/2306227609 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fba240a2cb0 0x7fba240a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:50.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.591+0000 7fba38ff9700 1 -- 172.21.15.44:0/2306227609 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fba2c0d0a20 con 0x7fba3c109230 2024-04-21T23:07:50.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.591+0000 7fba3affd700 1 --2- 172.21.15.44:0/2306227609 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fba240a2cb0 0x7fba240a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:50.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.591+0000 7fba3affd700 1 --2- 172.21.15.44:0/2306227609 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fba240a2cb0 0x7fba240a5170 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fba3c1173c0 tx=0x7fba3000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:50.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.594+0000 7fba38ff9700 1 -- 172.21.15.44:0/2306227609 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fba2c031020 con 0x7fba3c109230 2024-04-21T23:07:50.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.804+0000 7fba4099d700 1 -- 172.21.15.44:0/2306227609 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fba3c1b0090 con 0x7fba3c109230 2024-04-21T23:07:50.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.805+0000 7fba38ff9700 1 -- 172.21.15.44:0/2306227609 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fba2c013070 con 0x7fba3c109230 2024-04-21T23:07:50.805 INFO:teuthology.orchestra.run.smithi044.stdout:163208757309 2024-04-21T23:07:50.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.807+0000 7fba4099d700 1 -- 172.21.15.44:0/2306227609 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fba240a2cb0 msgr2=0x7fba240a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:50.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.807+0000 7fba4099d700 1 --2- 172.21.15.44:0/2306227609 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fba240a2cb0 0x7fba240a5170 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fba3c1173c0 tx=0x7fba3000a380 comp rx=0 tx=0).stop 2024-04-21T23:07:50.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.807+0000 7fba4099d700 1 -- 172.21.15.44:0/2306227609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba3c109230 msgr2=0x7fba3c1a3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:50.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.807+0000 7fba4099d700 1 --2- 172.21.15.44:0/2306227609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba3c109230 0x7fba3c1a3340 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fba2c000c00 tx=0x7fba2c003890 comp rx=0 tx=0).stop 2024-04-21T23:07:50.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.807+0000 7fba4099d700 1 -- 172.21.15.44:0/2306227609 shutdown_connections 2024-04-21T23:07:50.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.807+0000 7fba4099d700 1 --2- 172.21.15.44:0/2306227609 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fba3c115380 0x7fba3c1a7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:50.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.807+0000 7fba4099d700 1 --2- 172.21.15.44:0/2306227609 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fba240a2cb0 0x7fba240a5170 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:50.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.807+0000 7fba4099d700 1 --2- 172.21.15.44:0/2306227609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fba3c109230 0x7fba3c1a3340 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:50.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.807+0000 7fba4099d700 1 --2- 172.21.15.44:0/2306227609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba3c109b50 0x7fba3c1a3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:50.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.807+0000 7fba4099d700 1 -- 172.21.15.44:0/2306227609 >> 172.21.15.44:0/2306227609 conn(0x7fba3c06d680 msgr2=0x7fba3c114370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:50.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.807+0000 7fba4099d700 1 -- 172.21.15.44:0/2306227609 shutdown_connections 2024-04-21T23:07:50.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:50.807+0000 7fba4099d700 1 -- 172.21.15.44:0/2306227609 wait complete. 2024-04-21T23:07:51.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:50 smithi055 ceph-mon[31117]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:07:51.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:50 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2306227609' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-21T23:07:51.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:50 smithi069 ceph-mon[31816]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:07:51.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:50 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2306227609' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-21T23:07:51.246 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:50 smithi044 ceph-mon[27934]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:07:51.247 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:50 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2306227609' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-21T23:07:51.998 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757304 got 163208757309 for osd.5 2024-04-21T23:07:52.003 DEBUG:teuthology.parallel:result is None 2024-04-21T23:07:52.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.518+0000 7f8561434700 1 -- 172.21.15.44:0/1335431631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f855c1071a0 msgr2=0x7f855c107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:52.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.518+0000 7f8561434700 1 --2- 172.21.15.44:0/1335431631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f855c1071a0 0x7f855c107580 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f854c009990 tx=0x7f854c009ca0 comp rx=0 tx=0).stop 2024-04-21T23:07:52.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.518+0000 7f8561434700 1 -- 172.21.15.44:0/1335431631 shutdown_connections 2024-04-21T23:07:52.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.518+0000 7f8561434700 1 --2- 172.21.15.44:0/1335431631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f855c101af0 0x7f855c10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:52.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.518+0000 7f8561434700 1 --2- 172.21.15.44:0/1335431631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f855c1071a0 0x7f855c107580 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:52.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.518+0000 7f8561434700 1 --2- 172.21.15.44:0/1335431631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f855c101130 0x7f855c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:52.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.518+0000 7f8561434700 1 -- 172.21.15.44:0/1335431631 >> 172.21.15.44:0/1335431631 conn(0x7f855c0780e0 msgr2=0x7f855c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:52.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.518+0000 7f8561434700 1 -- 172.21.15.44:0/1335431631 shutdown_connections 2024-04-21T23:07:52.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.518+0000 7f8561434700 1 -- 172.21.15.44:0/1335431631 wait complete. 2024-04-21T23:07:52.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.518+0000 7f8561434700 1 Processor -- start 2024-04-21T23:07:52.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.519+0000 7f8561434700 1 -- start start 2024-04-21T23:07:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.519+0000 7f8561434700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f855c101130 0x7f855c1975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.519+0000 7f8561434700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f855c101af0 0x7f855c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.519+0000 7f8561434700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f855c1071a0 0x7f855c198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.519+0000 7f8561434700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f855c111410 con 0x7f855c101af0 2024-04-21T23:07:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.519+0000 7f8561434700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f855c111290 con 0x7f855c101130 2024-04-21T23:07:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.519+0000 7f8561434700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f855c111590 con 0x7f855c1071a0 2024-04-21T23:07:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.519+0000 7f855a7fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f855c101af0 0x7f855c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.519+0000 7f855a7fc700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f855c101af0 0x7f855c197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51242/0 (socket says 172.21.15.44:51242) 2024-04-21T23:07:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.519+0000 7f855a7fc700 1 -- 172.21.15.44:0/1072743637 learned_addr learned my addr 172.21.15.44:0/1072743637 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.519+0000 7f855affd700 1 --2- 172.21.15.44:0/1072743637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f855c101130 0x7f855c1975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.519+0000 7f855b7fe700 1 --2- 172.21.15.44:0/1072743637 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f855c1071a0 0x7f855c198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.519+0000 7f855a7fc700 1 -- 172.21.15.44:0/1072743637 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f855c1071a0 msgr2=0x7f855c198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:52.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.519+0000 7f855a7fc700 1 --2- 172.21.15.44:0/1072743637 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f855c1071a0 0x7f855c198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:52.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.519+0000 7f855a7fc700 1 -- 172.21.15.44:0/1072743637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f855c101130 msgr2=0x7f855c1975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:52.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.520+0000 7f855a7fc700 1 --2- 172.21.15.44:0/1072743637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f855c101130 0x7f855c1975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:52.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.520+0000 7f855a7fc700 1 -- 172.21.15.44:0/1072743637 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855c1a6ee0 con 0x7f855c101af0 2024-04-21T23:07:52.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.520+0000 7f855affd700 1 --2- 172.21.15.44:0/1072743637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f855c101130 0x7f855c1975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:07:52.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.520+0000 7f855a7fc700 1 --2- 172.21.15.44:0/1072743637 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f855c101af0 0x7f855c197b20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f855000d950 tx=0x7f855000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:52.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.520+0000 7f8543fff700 1 -- 172.21.15.44:0/1072743637 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8550016070 con 0x7f855c101af0 2024-04-21T23:07:52.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.520+0000 7f8543fff700 1 -- 172.21.15.44:0/1072743637 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8550009e20 con 0x7f855c101af0 2024-04-21T23:07:52.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.520+0000 7f8561434700 1 -- 172.21.15.44:0/1072743637 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f855c1a71d0 con 0x7f855c101af0 2024-04-21T23:07:52.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.520+0000 7f8543fff700 1 -- 172.21.15.44:0/1072743637 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f855000f620 con 0x7f855c101af0 2024-04-21T23:07:52.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.520+0000 7f8561434700 1 -- 172.21.15.44:0/1072743637 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f855c1a7710 con 0x7f855c101af0 2024-04-21T23:07:52.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.521+0000 7f8561434700 1 -- 172.21.15.44:0/1072743637 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f855c068f20 con 0x7f855c101af0 2024-04-21T23:07:52.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.522+0000 7f8543fff700 1 -- 172.21.15.44:0/1072743637 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f8550019070 con 0x7f855c101af0 2024-04-21T23:07:52.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.523+0000 7f8543fff700 1 --2- 172.21.15.44:0/1072743637 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f85440a2cb0 0x7f85440a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:52.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.523+0000 7f855affd700 1 --2- 172.21.15.44:0/1072743637 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f85440a2cb0 0x7f85440a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:52.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.523+0000 7f8543fff700 1 -- 172.21.15.44:0/1072743637 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f85500cf140 con 0x7f855c101af0 2024-04-21T23:07:52.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.524+0000 7f855affd700 1 --2- 172.21.15.44:0/1072743637 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f85440a2cb0 0x7f85440a5170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f854c006010 tx=0x7f854c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:52.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.528+0000 7f8543fff700 1 -- 172.21.15.44:0/1072743637 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f855009bba0 con 0x7f855c101af0 2024-04-21T23:07:52.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.736+0000 7f8561434700 1 -- 172.21.15.44:0/1072743637 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f855c02db30 con 0x7f855c101af0 2024-04-21T23:07:52.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.736+0000 7f8543fff700 1 -- 172.21.15.44:0/1072743637 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f855009b4a0 con 0x7f855c101af0 2024-04-21T23:07:52.736 INFO:teuthology.orchestra.run.smithi044.stdout:253403070510 2024-04-21T23:07:52.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.738+0000 7f8561434700 1 -- 172.21.15.44:0/1072743637 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f85440a2cb0 msgr2=0x7f85440a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:52.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.738+0000 7f8561434700 1 --2- 172.21.15.44:0/1072743637 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f85440a2cb0 0x7f85440a5170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f854c006010 tx=0x7f854c00b540 comp rx=0 tx=0).stop 2024-04-21T23:07:52.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.738+0000 7f8561434700 1 -- 172.21.15.44:0/1072743637 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f855c101af0 msgr2=0x7f855c197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:52.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.738+0000 7f8561434700 1 --2- 172.21.15.44:0/1072743637 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f855c101af0 0x7f855c197b20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f855000d950 tx=0x7f855000dc60 comp rx=0 tx=0).stop 2024-04-21T23:07:52.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.738+0000 7f8561434700 1 -- 172.21.15.44:0/1072743637 shutdown_connections 2024-04-21T23:07:52.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.738+0000 7f8561434700 1 --2- 172.21.15.44:0/1072743637 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f855c1071a0 0x7f855c198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:52.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.738+0000 7f8561434700 1 --2- 172.21.15.44:0/1072743637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f855c101130 0x7f855c1975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:52.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.738+0000 7f8561434700 1 --2- 172.21.15.44:0/1072743637 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f85440a2cb0 0x7f85440a5170 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:52.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.738+0000 7f8561434700 1 --2- 172.21.15.44:0/1072743637 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f855c101af0 0x7f855c197b20 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:52.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.738+0000 7f8561434700 1 -- 172.21.15.44:0/1072743637 >> 172.21.15.44:0/1072743637 conn(0x7f855c0780e0 msgr2=0x7f855c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:52.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.738+0000 7f8561434700 1 -- 172.21.15.44:0/1072743637 shutdown_connections 2024-04-21T23:07:52.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:52.738+0000 7f8561434700 1 -- 172.21.15.44:0/1072743637 wait complete. 2024-04-21T23:07:53.135 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:52 smithi044 ceph-mon[27934]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:07:53.135 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:52 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1072743637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-21T23:07:53.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:52 smithi055 ceph-mon[31117]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:07:53.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:52 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1072743637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-21T23:07:53.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:52 smithi069 ceph-mon[31816]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:07:53.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:52 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1072743637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-21T23:07:53.729 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332139 got 219043332144 for osd.7 2024-04-21T23:07:53.730 DEBUG:teuthology.parallel:result is None 2024-04-21T23:07:54.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.340+0000 7fb74ac92700 1 -- 172.21.15.44:0/1312786620 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7441013c0 msgr2=0x7fb7441017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:54.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.340+0000 7fb74ac92700 1 --2- 172.21.15.44:0/1312786620 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7441013c0 0x7fb7441017a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb7340099c0 tx=0x7fb734009cd0 comp rx=0 tx=0).stop 2024-04-21T23:07:54.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.340+0000 7fb74ac92700 1 -- 172.21.15.44:0/1312786620 shutdown_connections 2024-04-21T23:07:54.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.340+0000 7fb74ac92700 1 --2- 172.21.15.44:0/1312786620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb744106450 0x7fb74410e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:54.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.340+0000 7fb74ac92700 1 --2- 172.21.15.44:0/1312786620 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb7441013c0 0x7fb7441017a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:54.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.340+0000 7fb74ac92700 1 --2- 172.21.15.44:0/1312786620 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb744101d70 0x7fb744105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:54.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.340+0000 7fb74ac92700 1 -- 172.21.15.44:0/1312786620 >> 172.21.15.44:0/1312786620 conn(0x7fb7440780e0 msgr2=0x7fb7440784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:54.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.341+0000 7fb74ac92700 1 -- 172.21.15.44:0/1312786620 shutdown_connections 2024-04-21T23:07:54.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.341+0000 7fb74ac92700 1 -- 172.21.15.44:0/1312786620 wait complete. 2024-04-21T23:07:54.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.341+0000 7fb74ac92700 1 Processor -- start 2024-04-21T23:07:54.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.341+0000 7fb74ac92700 1 -- start start 2024-04-21T23:07:54.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.341+0000 7fb74ac92700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7441013c0 0x7fb74419e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:54.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.341+0000 7fb74ac92700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb744101d70 0x7fb74419ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:54.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.341+0000 7fb74ac92700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb744106450 0x7fb744198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:54.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.341+0000 7fb74ac92700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb744111480 con 0x7fb744106450 2024-04-21T23:07:54.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.341+0000 7fb74ac92700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb744111300 con 0x7fb744101d70 2024-04-21T23:07:54.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.341+0000 7fb74ac92700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb744111600 con 0x7fb7441013c0 2024-04-21T23:07:54.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.341+0000 7fb74922f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb744106450 0x7fb744198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:54.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.341+0000 7fb74922f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb744106450 0x7fb744198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51258/0 (socket says 172.21.15.44:51258) 2024-04-21T23:07:54.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.342+0000 7fb74922f700 1 -- 172.21.15.44:0/430360106 learned_addr learned my addr 172.21.15.44:0/430360106 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:54.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.342+0000 7fb748a2e700 1 --2- 172.21.15.44:0/430360106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7441013c0 0x7fb74419e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:54.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.342+0000 7fb743fff700 1 --2- 172.21.15.44:0/430360106 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb744101d70 0x7fb74419ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:54.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.342+0000 7fb74922f700 1 -- 172.21.15.44:0/430360106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7441013c0 msgr2=0x7fb74419e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:54.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.342+0000 7fb74922f700 1 --2- 172.21.15.44:0/430360106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7441013c0 0x7fb74419e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:54.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.342+0000 7fb74922f700 1 -- 172.21.15.44:0/430360106 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb744101d70 msgr2=0x7fb74419ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:54.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.342+0000 7fb74922f700 1 --2- 172.21.15.44:0/430360106 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb744101d70 0x7fb74419ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:54.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.342+0000 7fb74922f700 1 -- 172.21.15.44:0/430360106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb744199010 con 0x7fb744106450 2024-04-21T23:07:54.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.342+0000 7fb743fff700 1 --2- 172.21.15.44:0/430360106 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb744101d70 0x7fb74419ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:07:54.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.342+0000 7fb74922f700 1 --2- 172.21.15.44:0/430360106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb744106450 0x7fb744198830 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb73c007e90 tx=0x7fb73c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:54.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.342+0000 7fb741ffb700 1 -- 172.21.15.44:0/430360106 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c004bb0 con 0x7fb744106450 2024-04-21T23:07:54.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.342+0000 7fb74ac92700 1 -- 172.21.15.44:0/430360106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb744199300 con 0x7fb744106450 2024-04-21T23:07:54.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.342+0000 7fb741ffb700 1 -- 172.21.15.44:0/430360106 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb73c004d50 con 0x7fb744106450 2024-04-21T23:07:54.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.342+0000 7fb741ffb700 1 -- 172.21.15.44:0/430360106 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c011d20 con 0x7fb744106450 2024-04-21T23:07:54.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.342+0000 7fb74ac92700 1 -- 172.21.15.44:0/430360106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb744102d40 con 0x7fb744106450 2024-04-21T23:07:54.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.344+0000 7fb74ac92700 1 -- 172.21.15.44:0/430360106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb744102850 con 0x7fb744106450 2024-04-21T23:07:54.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.345+0000 7fb741ffb700 1 -- 172.21.15.44:0/430360106 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb73c00c980 con 0x7fb744106450 2024-04-21T23:07:54.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.345+0000 7fb741ffb700 1 --2- 172.21.15.44:0/430360106 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb7300a2cb0 0x7fb7300a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:54.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.345+0000 7fb741ffb700 1 -- 172.21.15.44:0/430360106 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fb73c0d2d70 con 0x7fb744106450 2024-04-21T23:07:54.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.345+0000 7fb748a2e700 1 --2- 172.21.15.44:0/430360106 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb7300a2cb0 0x7fb7300a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:54.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.346+0000 7fb748a2e700 1 --2- 172.21.15.44:0/430360106 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb7300a2cb0 0x7fb7300a5170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb734000c00 tx=0x7fb734005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:54.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.350+0000 7fb741ffb700 1 -- 172.21.15.44:0/430360106 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb73c09f940 con 0x7fb744106450 2024-04-21T23:07:54.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.563+0000 7fb74ac92700 1 -- 172.21.15.44:0/430360106 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fb74402db30 con 0x7fb744106450 2024-04-21T23:07:54.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.563+0000 7fb741ffb700 1 -- 172.21.15.44:0/430360106 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb73c09f240 con 0x7fb744106450 2024-04-21T23:07:54.563 INFO:teuthology.orchestra.run.smithi044.stdout:34359738453 2024-04-21T23:07:54.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.565+0000 7fb74ac92700 1 -- 172.21.15.44:0/430360106 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb7300a2cb0 msgr2=0x7fb7300a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:54.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.565+0000 7fb74ac92700 1 --2- 172.21.15.44:0/430360106 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb7300a2cb0 0x7fb7300a5170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb734000c00 tx=0x7fb734005fb0 comp rx=0 tx=0).stop 2024-04-21T23:07:54.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.565+0000 7fb74ac92700 1 -- 172.21.15.44:0/430360106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb744106450 msgr2=0x7fb744198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:54.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.565+0000 7fb74ac92700 1 --2- 172.21.15.44:0/430360106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb744106450 0x7fb744198830 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb73c007e90 tx=0x7fb73c00c330 comp rx=0 tx=0).stop 2024-04-21T23:07:54.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.565+0000 7fb74ac92700 1 -- 172.21.15.44:0/430360106 shutdown_connections 2024-04-21T23:07:54.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.565+0000 7fb74ac92700 1 --2- 172.21.15.44:0/430360106 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb7300a2cb0 0x7fb7300a5170 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:54.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.565+0000 7fb74ac92700 1 --2- 172.21.15.44:0/430360106 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb744106450 0x7fb744198830 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:54.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.565+0000 7fb74ac92700 1 --2- 172.21.15.44:0/430360106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7441013c0 0x7fb74419e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:54.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.565+0000 7fb74ac92700 1 --2- 172.21.15.44:0/430360106 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb744101d70 0x7fb74419ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:54.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.565+0000 7fb74ac92700 1 -- 172.21.15.44:0/430360106 >> 172.21.15.44:0/430360106 conn(0x7fb7440780e0 msgr2=0x7fb74406aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:54.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.565+0000 7fb74ac92700 1 -- 172.21.15.44:0/430360106 shutdown_connections 2024-04-21T23:07:54.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:54.565+0000 7fb74ac92700 1 -- 172.21.15.44:0/430360106 wait complete. 2024-04-21T23:07:54.937 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:07:54.971 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495667 got 197568495671 for osd.6 2024-04-21T23:07:54.971 DEBUG:teuthology.parallel:result is None 2024-04-21T23:07:55.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:54 smithi044 ceph-mon[27934]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:07:55.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:54 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/430360106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-21T23:07:55.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:54 smithi055 ceph-mon[31117]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:07:55.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:54 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/430360106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-21T23:07:55.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:54 smithi069 ceph-mon[31816]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:07:55.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:54 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/430360106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-21T23:07:55.624 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070506 got 253403070510 for osd.8 2024-04-21T23:07:55.624 DEBUG:teuthology.parallel:result is None 2024-04-21T23:07:57.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.891+0000 7ff20324d700 1 -- 172.21.15.44:0/517819762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1fc102120 msgr2=0x7ff1fc105d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:57.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.891+0000 7ff20324d700 1 --2- 172.21.15.44:0/517819762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1fc102120 0x7ff1fc105d90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec00b0d0 tx=0x7ff1ec00b3e0 comp rx=0 tx=0).stop 2024-04-21T23:07:57.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.891+0000 7ff20324d700 1 -- 172.21.15.44:0/517819762 shutdown_connections 2024-04-21T23:07:57.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.891+0000 7ff20324d700 1 --2- 172.21.15.44:0/517819762 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1fc1064c0 0x7ff1fc10e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:57.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.891+0000 7ff20324d700 1 --2- 172.21.15.44:0/517819762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1fc101770 0x7ff1fc101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:57.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.891+0000 7ff20324d700 1 --2- 172.21.15.44:0/517819762 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1fc102120 0x7ff1fc105d90 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:57.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.891+0000 7ff20324d700 1 -- 172.21.15.44:0/517819762 >> 172.21.15.44:0/517819762 conn(0x7ff1fc0780e0 msgr2=0x7ff1fc0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:57.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.891+0000 7ff20324d700 1 -- 172.21.15.44:0/517819762 shutdown_connections 2024-04-21T23:07:57.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.891+0000 7ff20324d700 1 -- 172.21.15.44:0/517819762 wait complete. 2024-04-21T23:07:57.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.892+0000 7ff20324d700 1 Processor -- start 2024-04-21T23:07:57.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.892+0000 7ff20324d700 1 -- start start 2024-04-21T23:07:57.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.893+0000 7ff20324d700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1fc101770 0x7ff1fc19d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:57.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.893+0000 7ff20324d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1fc102120 0x7ff1fc19db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:57.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.893+0000 7ff20324d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1fc1064c0 0x7ff1fc197790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:57.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.893+0000 7ff20324d700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff1fc111450 con 0x7ff1fc101770 2024-04-21T23:07:57.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.893+0000 7ff20324d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff1fc1112d0 con 0x7ff1fc1064c0 2024-04-21T23:07:57.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.893+0000 7ff20324d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff1fc1115d0 con 0x7ff1fc102120 2024-04-21T23:07:57.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.893+0000 7ff200fe9700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1fc101770 0x7ff1fc19d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:57.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.893+0000 7ff200fe9700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1fc101770 0x7ff1fc19d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51280/0 (socket says 172.21.15.44:51280) 2024-04-21T23:07:57.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.893+0000 7ff200fe9700 1 -- 172.21.15.44:0/1130630627 learned_addr learned my addr 172.21.15.44:0/1130630627 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:57.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.893+0000 7ff2017ea700 1 --2- 172.21.15.44:0/1130630627 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1fc1064c0 0x7ff1fc197790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:57.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.893+0000 7ff200fe9700 1 -- 172.21.15.44:0/1130630627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1fc102120 msgr2=0x7ff1fc19db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:57.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.893+0000 7ff200fe9700 1 --2- 172.21.15.44:0/1130630627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1fc102120 0x7ff1fc19db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:57.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.893+0000 7ff200fe9700 1 -- 172.21.15.44:0/1130630627 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1fc1064c0 msgr2=0x7ff1fc197790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:57.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.893+0000 7ff200fe9700 1 --2- 172.21.15.44:0/1130630627 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1fc1064c0 0x7ff1fc197790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:57.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.893+0000 7ff200fe9700 1 -- 172.21.15.44:0/1130630627 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1fc197f70 con 0x7ff1fc101770 2024-04-21T23:07:57.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.893+0000 7ff200fe9700 1 --2- 172.21.15.44:0/1130630627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1fc101770 0x7ff1fc19d610 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff1e400d980 tx=0x7ff1e400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:57.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.893+0000 7ff1f9ffb700 1 -- 172.21.15.44:0/1130630627 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4017070 con 0x7ff1fc101770 2024-04-21T23:07:57.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.893+0000 7ff1f9ffb700 1 -- 172.21.15.44:0/1130630627 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1e4004d10 con 0x7ff1fc101770 2024-04-21T23:07:57.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.893+0000 7ff20324d700 1 -- 172.21.15.44:0/1130630627 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1fc198260 con 0x7ff1fc101770 2024-04-21T23:07:57.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.893+0000 7ff1f9ffb700 1 -- 172.21.15.44:0/1130630627 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e40055c0 con 0x7ff1fc101770 2024-04-21T23:07:57.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.894+0000 7ff20324d700 1 -- 172.21.15.44:0/1130630627 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1fc1a4470 con 0x7ff1fc101770 2024-04-21T23:07:57.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.895+0000 7ff20324d700 1 -- 172.21.15.44:0/1130630627 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1fc06a1b0 con 0x7ff1fc101770 2024-04-21T23:07:57.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.896+0000 7ff1f9ffb700 1 -- 172.21.15.44:0/1130630627 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff1e4004750 con 0x7ff1fc101770 2024-04-21T23:07:57.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.896+0000 7ff1f9ffb700 1 --2- 172.21.15.44:0/1130630627 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff1e80a2cb0 0x7ff1e80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:57.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.897+0000 7ff1fbfff700 1 --2- 172.21.15.44:0/1130630627 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff1e80a2cb0 0x7ff1e80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:57.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.897+0000 7ff1f9ffb700 1 -- 172.21.15.44:0/1130630627 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7ff1e40cf4c0 con 0x7ff1fc101770 2024-04-21T23:07:57.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.897+0000 7ff1fbfff700 1 --2- 172.21.15.44:0/1130630627 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff1e80a2cb0 0x7ff1e80a5170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc1010f0 tx=0x7ff1ec000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:57.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:56.901+0000 7ff1f9ffb700 1 -- 172.21.15.44:0/1130630627 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff1e4020020 con 0x7ff1fc101770 2024-04-21T23:07:57.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.111+0000 7ff20324d700 1 -- 172.21.15.44:0/1130630627 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7ff1fc102870 con 0x7ff1fc101770 2024-04-21T23:07:57.112 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:56 smithi044 ceph-mon[27934]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:07:57.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.112+0000 7ff1f9ffb700 1 -- 172.21.15.44:0/1130630627 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff1e409bf80 con 0x7ff1fc101770 2024-04-21T23:07:57.112 INFO:teuthology.orchestra.run.smithi044.stdout:77309411405 2024-04-21T23:07:57.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.114+0000 7ff20324d700 1 -- 172.21.15.44:0/1130630627 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff1e80a2cb0 msgr2=0x7ff1e80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:57.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.114+0000 7ff20324d700 1 --2- 172.21.15.44:0/1130630627 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff1e80a2cb0 0x7ff1e80a5170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc1010f0 tx=0x7ff1ec000f40 comp rx=0 tx=0).stop 2024-04-21T23:07:57.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.114+0000 7ff20324d700 1 -- 172.21.15.44:0/1130630627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1fc101770 msgr2=0x7ff1fc19d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:57.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.114+0000 7ff20324d700 1 --2- 172.21.15.44:0/1130630627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1fc101770 0x7ff1fc19d610 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff1e400d980 tx=0x7ff1e400dd40 comp rx=0 tx=0).stop 2024-04-21T23:07:57.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.114+0000 7ff20324d700 1 -- 172.21.15.44:0/1130630627 shutdown_connections 2024-04-21T23:07:57.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.114+0000 7ff20324d700 1 --2- 172.21.15.44:0/1130630627 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff1fc1064c0 0x7ff1fc197790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:57.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.114+0000 7ff20324d700 1 --2- 172.21.15.44:0/1130630627 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff1e80a2cb0 0x7ff1e80a5170 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:57.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.114+0000 7ff20324d700 1 --2- 172.21.15.44:0/1130630627 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff1fc101770 0x7ff1fc19d610 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:57.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.114+0000 7ff20324d700 1 --2- 172.21.15.44:0/1130630627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1fc102120 0x7ff1fc19db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:57.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.114+0000 7ff20324d700 1 -- 172.21.15.44:0/1130630627 >> 172.21.15.44:0/1130630627 conn(0x7ff1fc0780e0 msgr2=0x7ff1fc0ff710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:57.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.115+0000 7ff20324d700 1 -- 172.21.15.44:0/1130630627 shutdown_connections 2024-04-21T23:07:57.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.115+0000 7ff20324d700 1 -- 172.21.15.44:0/1130630627 wait complete. 2024-04-21T23:07:57.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:56 smithi055 ceph-mon[31117]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:07:57.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:56 smithi069 ceph-mon[31816]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:07:58.159 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738446 got 34359738453 for osd.0 2024-04-21T23:07:58.159 DEBUG:teuthology.parallel:result is None 2024-04-21T23:07:58.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:57 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1130630627' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-21T23:07:58.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:57 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/455573004' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-21T23:07:58.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:57 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1130630627' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-21T23:07:58.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:57 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/455573004' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-21T23:07:58.284 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:57 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1130630627' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-21T23:07:58.284 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:57 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/455573004' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-21T23:07:59.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.078+0000 7f57205f1700 1 -- 172.21.15.44:0/2978022581 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5718101160 msgr2=0x7f57181015c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:59.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.078+0000 7f57205f1700 1 --2- 172.21.15.44:0/2978022581 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5718101160 0x7f57181015c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5708009990 tx=0x7f5708009ca0 comp rx=0 tx=0).stop 2024-04-21T23:07:59.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.079+0000 7f57205f1700 1 -- 172.21.15.44:0/2978022581 shutdown_connections 2024-04-21T23:07:59.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.079+0000 7f57205f1700 1 --2- 172.21.15.44:0/2978022581 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5718101b00 0x7f571810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:59.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.079+0000 7f57205f1700 1 --2- 172.21.15.44:0/2978022581 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57181071a0 0x7f5718107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:59.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.079+0000 7f57205f1700 1 --2- 172.21.15.44:0/2978022581 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5718101160 0x7f57181015c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:59.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.079+0000 7f57205f1700 1 -- 172.21.15.44:0/2978022581 >> 172.21.15.44:0/2978022581 conn(0x7f57180780d0 msgr2=0x7f57180784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:59.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.079+0000 7f57205f1700 1 -- 172.21.15.44:0/2978022581 shutdown_connections 2024-04-21T23:07:59.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.079+0000 7f57205f1700 1 -- 172.21.15.44:0/2978022581 wait complete. 2024-04-21T23:07:59.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.079+0000 7f57205f1700 1 Processor -- start 2024-04-21T23:07:59.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.079+0000 7f57205f1700 1 -- start start 2024-04-21T23:07:59.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.080+0000 7f57205f1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5718101160 0x7f571819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:59.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.080+0000 7f57205f1700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5718101b00 0x7f571819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:59.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.080+0000 7f57205f1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57181071a0 0x7f57181987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:59.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.080+0000 7f57205f1700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f57181114a0 con 0x7f5718101b00 2024-04-21T23:07:59.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.080+0000 7f57205f1700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5718111320 con 0x7f57181071a0 2024-04-21T23:07:59.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.080+0000 7f57205f1700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5718111620 con 0x7f5718101160 2024-04-21T23:07:59.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.080+0000 7f571db8c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5718101b00 0x7f571819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:59.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.080+0000 7f571db8c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5718101b00 0x7f571819eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51310/0 (socket says 172.21.15.44:51310) 2024-04-21T23:07:59.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.080+0000 7f571db8c700 1 -- 172.21.15.44:0/616236559 learned_addr learned my addr 172.21.15.44:0/616236559 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:59.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.080+0000 7f571eb8e700 1 --2- 172.21.15.44:0/616236559 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57181071a0 0x7f57181987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:59.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.080+0000 7f571e38d700 1 --2- 172.21.15.44:0/616236559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5718101160 0x7f571819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:59.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.080+0000 7f571db8c700 1 -- 172.21.15.44:0/616236559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5718101160 msgr2=0x7f571819e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:59.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.080+0000 7f571db8c700 1 --2- 172.21.15.44:0/616236559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5718101160 0x7f571819e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:59.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.080+0000 7f571db8c700 1 -- 172.21.15.44:0/616236559 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57181071a0 msgr2=0x7f57181987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:59.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.080+0000 7f571db8c700 1 --2- 172.21.15.44:0/616236559 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57181071a0 0x7f57181987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:59.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.080+0000 7f571db8c700 1 -- 172.21.15.44:0/616236559 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5718198fc0 con 0x7f5718101b00 2024-04-21T23:07:59.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.080+0000 7f571db8c700 1 --2- 172.21.15.44:0/616236559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5718101b00 0x7f571819eba0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f5708000c00 tx=0x7f5708003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:59.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.080+0000 7f570f7fe700 1 -- 172.21.15.44:0/616236559 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5708027070 con 0x7f5718101b00 2024-04-21T23:07:59.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.080+0000 7f570f7fe700 1 -- 172.21.15.44:0/616236559 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5708004050 con 0x7f5718101b00 2024-04-21T23:07:59.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.081+0000 7f57205f1700 1 -- 172.21.15.44:0/616236559 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5718199250 con 0x7f5718101b00 2024-04-21T23:07:59.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.081+0000 7f570f7fe700 1 -- 172.21.15.44:0/616236559 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570800f460 con 0x7f5718101b00 2024-04-21T23:07:59.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.081+0000 7f57205f1700 1 -- 172.21.15.44:0/616236559 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57181a54c0 con 0x7f5718101b00 2024-04-21T23:07:59.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.081+0000 7f57205f1700 1 -- 172.21.15.44:0/616236559 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5718102850 con 0x7f5718101b00 2024-04-21T23:07:59.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.083+0000 7f570f7fe700 1 -- 172.21.15.44:0/616236559 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f570802a070 con 0x7f5718101b00 2024-04-21T23:07:59.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.083+0000 7f570f7fe700 1 --2- 172.21.15.44:0/616236559 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f57040a2d30 0x7f57040a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:59.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.083+0000 7f571e38d700 1 --2- 172.21.15.44:0/616236559 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f57040a2d30 0x7f57040a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:59.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.083+0000 7f570f7fe700 1 -- 172.21.15.44:0/616236559 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f57080d0950 con 0x7f5718101b00 2024-04-21T23:07:59.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.084+0000 7f571e38d700 1 --2- 172.21.15.44:0/616236559 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f57040a2d30 0x7f57040a51f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f571400ac50 tx=0x7f571400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:59.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.087+0000 7f570f7fe700 1 -- 172.21.15.44:0/616236559 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5708031070 con 0x7f5718101b00 2024-04-21T23:07:59.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:58 smithi044 ceph-mon[27934]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:07:59.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.296+0000 7f57205f1700 1 -- 172.21.15.44:0/616236559 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f57181a5690 con 0x7f5718101b00 2024-04-21T23:07:59.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.296+0000 7f570f7fe700 1 -- 172.21.15.44:0/616236559 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5708013070 con 0x7f5718101b00 2024-04-21T23:07:59.297 INFO:teuthology.orchestra.run.smithi044.stdout:55834574930 2024-04-21T23:07:59.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.298+0000 7f57205f1700 1 -- 172.21.15.44:0/616236559 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f57040a2d30 msgr2=0x7f57040a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:59.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.298+0000 7f57205f1700 1 --2- 172.21.15.44:0/616236559 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f57040a2d30 0x7f57040a51f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f571400ac50 tx=0x7f571400a380 comp rx=0 tx=0).stop 2024-04-21T23:07:59.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.298+0000 7f57205f1700 1 -- 172.21.15.44:0/616236559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5718101b00 msgr2=0x7f571819eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:59.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.298+0000 7f57205f1700 1 --2- 172.21.15.44:0/616236559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5718101b00 0x7f571819eba0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f5708000c00 tx=0x7f5708003680 comp rx=0 tx=0).stop 2024-04-21T23:07:59.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.298+0000 7f57205f1700 1 -- 172.21.15.44:0/616236559 shutdown_connections 2024-04-21T23:07:59.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.298+0000 7f57205f1700 1 --2- 172.21.15.44:0/616236559 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f57181071a0 0x7f57181987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:59.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.298+0000 7f57205f1700 1 --2- 172.21.15.44:0/616236559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5718101160 0x7f571819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:59.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.298+0000 7f57205f1700 1 --2- 172.21.15.44:0/616236559 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f57040a2d30 0x7f57040a51f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:59.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.298+0000 7f57205f1700 1 --2- 172.21.15.44:0/616236559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5718101b00 0x7f571819eba0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:59.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.298+0000 7f57205f1700 1 -- 172.21.15.44:0/616236559 >> 172.21.15.44:0/616236559 conn(0x7f57180780d0 msgr2=0x7f57181004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:59.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.299+0000 7f57205f1700 1 -- 172.21.15.44:0/616236559 shutdown_connections 2024-04-21T23:07:59.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.299+0000 7f57205f1700 1 -- 172.21.15.44:0/616236559 wait complete. 2024-04-21T23:07:59.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:58 smithi055 ceph-mon[31117]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:07:59.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:58 smithi069 ceph-mon[31816]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:07:59.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.754+0000 7f51f38d0700 1 -- 172.21.15.44:0/2512338961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51ec1082b0 msgr2=0x7f51ec108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:59.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.754+0000 7f51f38d0700 1 --2- 172.21.15.44:0/2512338961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51ec1082b0 0x7f51ec108690 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009990 tx=0x7f51dc009ca0 comp rx=0 tx=0).stop 2024-04-21T23:07:59.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.754+0000 7f51f38d0700 1 -- 172.21.15.44:0/2512338961 shutdown_connections 2024-04-21T23:07:59.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.754+0000 7f51f38d0700 1 --2- 172.21.15.44:0/2512338961 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51ec06a8e0 0x7f51ec10f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:59.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.754+0000 7f51f38d0700 1 --2- 172.21.15.44:0/2512338961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51ec1082b0 0x7f51ec108690 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:59.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.754+0000 7f51f38d0700 1 --2- 172.21.15.44:0/2512338961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51ec069f40 0x7f51ec06a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:59.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.754+0000 7f51f38d0700 1 -- 172.21.15.44:0/2512338961 >> 172.21.15.44:0/2512338961 conn(0x7f51ec0780e0 msgr2=0x7f51ec0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:59.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.754+0000 7f51f38d0700 1 -- 172.21.15.44:0/2512338961 shutdown_connections 2024-04-21T23:07:59.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.754+0000 7f51f38d0700 1 -- 172.21.15.44:0/2512338961 wait complete. 2024-04-21T23:07:59.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.755+0000 7f51f38d0700 1 Processor -- start 2024-04-21T23:07:59.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.755+0000 7f51f38d0700 1 -- start start 2024-04-21T23:07:59.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.756+0000 7f51f38d0700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51ec069f40 0x7f51ec19e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:59.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.756+0000 7f51f38d0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51ec06a8e0 0x7f51ec19ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:59.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.756+0000 7f51f38d0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51ec1082b0 0x7f51ec1988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:59.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.756+0000 7f51f38d0700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec1124f0 con 0x7f51ec069f40 2024-04-21T23:07:59.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.756+0000 7f51f38d0700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec112370 con 0x7f51ec06a8e0 2024-04-21T23:07:59.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.756+0000 7f51f38d0700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec112670 con 0x7f51ec1082b0 2024-04-21T23:07:59.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.756+0000 7f51f166c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51ec069f40 0x7f51ec19e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:59.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.756+0000 7f51f166c700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51ec069f40 0x7f51ec19e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51316/0 (socket says 172.21.15.44:51316) 2024-04-21T23:07:59.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.756+0000 7f51f166c700 1 -- 172.21.15.44:0/3244988468 learned_addr learned my addr 172.21.15.44:0/3244988468 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:07:59.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.756+0000 7f51f0e6b700 1 --2- 172.21.15.44:0/3244988468 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51ec06a8e0 0x7f51ec19ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:59.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.756+0000 7f51f1e6d700 1 --2- 172.21.15.44:0/3244988468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51ec1082b0 0x7f51ec1988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:59.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.756+0000 7f51f166c700 1 -- 172.21.15.44:0/3244988468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51ec1082b0 msgr2=0x7f51ec1988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:59.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.756+0000 7f51f166c700 1 --2- 172.21.15.44:0/3244988468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51ec1082b0 0x7f51ec1988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:59.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.756+0000 7f51f166c700 1 -- 172.21.15.44:0/3244988468 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51ec06a8e0 msgr2=0x7f51ec19ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:59.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.756+0000 7f51f166c700 1 --2- 172.21.15.44:0/3244988468 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51ec06a8e0 0x7f51ec19ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:59.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.756+0000 7f51f166c700 1 -- 172.21.15.44:0/3244988468 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ec199030 con 0x7f51ec069f40 2024-04-21T23:07:59.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.756+0000 7f51f166c700 1 --2- 172.21.15.44:0/3244988468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51ec069f40 0x7f51ec19e710 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0038b0 tx=0x7f51dc005590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:59.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.757+0000 7f51e27fc700 1 -- 172.21.15.44:0/3244988468 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51dc027070 con 0x7f51ec069f40 2024-04-21T23:07:59.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.757+0000 7f51e27fc700 1 -- 172.21.15.44:0/3244988468 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51dc0040d0 con 0x7f51ec069f40 2024-04-21T23:07:59.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.757+0000 7f51f38d0700 1 -- 172.21.15.44:0/3244988468 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51ec1992c0 con 0x7f51ec069f40 2024-04-21T23:07:59.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.757+0000 7f51e27fc700 1 -- 172.21.15.44:0/3244988468 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51dc022dd0 con 0x7f51ec069f40 2024-04-21T23:07:59.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.757+0000 7f51f38d0700 1 -- 172.21.15.44:0/3244988468 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51ec1a55e0 con 0x7f51ec069f40 2024-04-21T23:07:59.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.758+0000 7f51f38d0700 1 -- 172.21.15.44:0/3244988468 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51ec068f20 con 0x7f51ec069f40 2024-04-21T23:07:59.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.760+0000 7f51e27fc700 1 -- 172.21.15.44:0/3244988468 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f51dc02a070 con 0x7f51ec069f40 2024-04-21T23:07:59.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.761+0000 7f51e27fc700 1 --2- 172.21.15.44:0/3244988468 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f51d80a2d30 0x7f51d80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:07:59.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.761+0000 7f51e27fc700 1 -- 172.21.15.44:0/3244988468 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f51dc0d07e0 con 0x7f51ec069f40 2024-04-21T23:07:59.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.761+0000 7f51f0e6b700 1 --2- 172.21.15.44:0/3244988468 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f51d80a2d30 0x7f51d80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:07:59.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.762+0000 7f51f0e6b700 1 --2- 172.21.15.44:0/3244988468 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f51d80a2d30 0x7f51d80a51f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f51e8005fd0 tx=0x7f51e8005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:07:59.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.764+0000 7f51e27fc700 1 -- 172.21.15.44:0/3244988468 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f51dc031020 con 0x7f51ec069f40 2024-04-21T23:07:59.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.977+0000 7f51f38d0700 1 -- 172.21.15.44:0/3244988468 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f51ec199e00 con 0x7f51ec069f40 2024-04-21T23:07:59.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.977+0000 7f51e27fc700 1 -- 172.21.15.44:0/3244988468 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f51dc013070 con 0x7f51ec069f40 2024-04-21T23:07:59.977 INFO:teuthology.orchestra.run.smithi044.stdout:141733920835 2024-04-21T23:07:59.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.979+0000 7f51f38d0700 1 -- 172.21.15.44:0/3244988468 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f51d80a2d30 msgr2=0x7f51d80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:59.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.979+0000 7f51f38d0700 1 --2- 172.21.15.44:0/3244988468 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f51d80a2d30 0x7f51d80a51f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f51e8005fd0 tx=0x7f51e8005f60 comp rx=0 tx=0).stop 2024-04-21T23:07:59.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.979+0000 7f51f38d0700 1 -- 172.21.15.44:0/3244988468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51ec069f40 msgr2=0x7f51ec19e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:07:59.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.979+0000 7f51f38d0700 1 --2- 172.21.15.44:0/3244988468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51ec069f40 0x7f51ec19e710 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0038b0 tx=0x7f51dc005590 comp rx=0 tx=0).stop 2024-04-21T23:07:59.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.979+0000 7f51f38d0700 1 -- 172.21.15.44:0/3244988468 shutdown_connections 2024-04-21T23:07:59.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.979+0000 7f51f38d0700 1 --2- 172.21.15.44:0/3244988468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51ec1082b0 0x7f51ec1988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:59.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.979+0000 7f51f38d0700 1 --2- 172.21.15.44:0/3244988468 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f51d80a2d30 0x7f51d80a51f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:59.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.979+0000 7f51f38d0700 1 --2- 172.21.15.44:0/3244988468 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f51ec069f40 0x7f51ec19e710 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:59.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.979+0000 7f51f38d0700 1 --2- 172.21.15.44:0/3244988468 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f51ec06a8e0 0x7f51ec19ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:07:59.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.979+0000 7f51f38d0700 1 -- 172.21.15.44:0/3244988468 >> 172.21.15.44:0/3244988468 conn(0x7f51ec0780e0 msgr2=0x7f51ec0fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:07:59.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.980+0000 7f51f38d0700 1 -- 172.21.15.44:0/3244988468 shutdown_connections 2024-04-21T23:07:59.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:59.980+0000 7f51f38d0700 1 -- 172.21.15.44:0/3244988468 wait complete. 2024-04-21T23:08:00.294 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:59 smithi044 ceph-mon[27934]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:00.294 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:07:59 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/616236559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-21T23:08:00.354 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574923 got 55834574930 for osd.1 2024-04-21T23:08:00.354 DEBUG:teuthology.parallel:result is None 2024-04-21T23:08:00.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:59 smithi055 ceph-mon[31117]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:00.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:07:59 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/616236559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-21T23:08:00.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:59 smithi069 ceph-mon[31816]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:00.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:07:59 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/616236559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-21T23:08:00.772 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411401 got 77309411405 for osd.2 2024-04-21T23:08:00.773 DEBUG:teuthology.parallel:result is None 2024-04-21T23:08:00.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.987+0000 7f27e80a9700 1 -- 172.21.15.44:0/1960899341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27e0100820 msgr2=0x7f27e0100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:00.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.987+0000 7f27e80a9700 1 --2- 172.21.15.44:0/1960899341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27e0100820 0x7f27e0100c80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009990 tx=0x7f27dc009ca0 comp rx=0 tx=0).stop 2024-04-21T23:08:00.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.987+0000 7f27e80a9700 1 -- 172.21.15.44:0/1960899341 shutdown_connections 2024-04-21T23:08:00.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.988+0000 7f27e80a9700 1 --2- 172.21.15.44:0/1960899341 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27e01011c0 0x7f27e010f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:00.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.988+0000 7f27e80a9700 1 --2- 172.21.15.44:0/1960899341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27e01082c0 0x7f27e01086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:00.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.988+0000 7f27e80a9700 1 --2- 172.21.15.44:0/1960899341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27e0100820 0x7f27e0100c80 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:00.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.988+0000 7f27e80a9700 1 -- 172.21.15.44:0/1960899341 >> 172.21.15.44:0/1960899341 conn(0x7f27e00777a0 msgr2=0x7f27e00feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:00.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.988+0000 7f27e80a9700 1 -- 172.21.15.44:0/1960899341 shutdown_connections 2024-04-21T23:08:00.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.988+0000 7f27e80a9700 1 -- 172.21.15.44:0/1960899341 wait complete. 2024-04-21T23:08:00.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.988+0000 7f27e80a9700 1 Processor -- start 2024-04-21T23:08:00.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.988+0000 7f27e80a9700 1 -- start start 2024-04-21T23:08:00.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.988+0000 7f27e80a9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27e0100820 0x7f27e019e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:00.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.988+0000 7f27e80a9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27e01011c0 0x7f27e019eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:00.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.988+0000 7f27e80a9700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27e01082c0 0x7f27e01987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:00.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.989+0000 7f27e80a9700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f27e0112520 con 0x7f27e01082c0 2024-04-21T23:08:00.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.989+0000 7f27e80a9700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f27e01123a0 con 0x7f27e0100820 2024-04-21T23:08:00.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.989+0000 7f27e80a9700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f27e01126a0 con 0x7f27e01011c0 2024-04-21T23:08:00.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.989+0000 7f27e6646700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27e01082c0 0x7f27e01987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:00.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.989+0000 7f27e6646700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27e01082c0 0x7f27e01987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45968/0 (socket says 172.21.15.44:45968) 2024-04-21T23:08:00.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.989+0000 7f27e6646700 1 -- 172.21.15.44:0/1797707286 learned_addr learned my addr 172.21.15.44:0/1797707286 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:00.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.989+0000 7f27e5e45700 1 --2- 172.21.15.44:0/1797707286 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27e0100820 0x7f27e019e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:00.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.989+0000 7f27e5644700 1 --2- 172.21.15.44:0/1797707286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27e01011c0 0x7f27e019eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:00.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.989+0000 7f27e6646700 1 -- 172.21.15.44:0/1797707286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27e01011c0 msgr2=0x7f27e019eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:00.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.989+0000 7f27e6646700 1 --2- 172.21.15.44:0/1797707286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27e01011c0 0x7f27e019eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:00.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.989+0000 7f27e6646700 1 -- 172.21.15.44:0/1797707286 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27e0100820 msgr2=0x7f27e019e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:00.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.989+0000 7f27e6646700 1 --2- 172.21.15.44:0/1797707286 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27e0100820 0x7f27e019e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:00.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.989+0000 7f27e6646700 1 -- 172.21.15.44:0/1797707286 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e0198f80 con 0x7f27e01082c0 2024-04-21T23:08:00.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.989+0000 7f27e6646700 1 --2- 172.21.15.44:0/1797707286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27e01082c0 0x7f27e01987a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f27d8007ee0 tx=0x7f27d800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:00.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.989+0000 7f27d6ffd700 1 -- 172.21.15.44:0/1797707286 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d800dbd0 con 0x7f27e01082c0 2024-04-21T23:08:00.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.989+0000 7f27e80a9700 1 -- 172.21.15.44:0/1797707286 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27e0199270 con 0x7f27e01082c0 2024-04-21T23:08:00.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.989+0000 7f27d6ffd700 1 -- 172.21.15.44:0/1797707286 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27d800dd70 con 0x7f27e01082c0 2024-04-21T23:08:00.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.989+0000 7f27d6ffd700 1 -- 172.21.15.44:0/1797707286 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d80133f0 con 0x7f27e01082c0 2024-04-21T23:08:00.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.989+0000 7f27e80a9700 1 -- 172.21.15.44:0/1797707286 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27e01a5480 con 0x7f27e01082c0 2024-04-21T23:08:00.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.990+0000 7f27e80a9700 1 -- 172.21.15.44:0/1797707286 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27e0103b40 con 0x7f27e01082c0 2024-04-21T23:08:00.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.991+0000 7f27d6ffd700 1 -- 172.21.15.44:0/1797707286 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f27d801a020 con 0x7f27e01082c0 2024-04-21T23:08:00.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.992+0000 7f27d6ffd700 1 --2- 172.21.15.44:0/1797707286 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f27cc0a2d30 0x7f27cc0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:00.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.992+0000 7f27e5e45700 1 --2- 172.21.15.44:0/1797707286 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f27cc0a2d30 0x7f27cc0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:00.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.992+0000 7f27d6ffd700 1 -- 172.21.15.44:0/1797707286 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f27d80ced20 con 0x7f27e01082c0 2024-04-21T23:08:00.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.993+0000 7f27e5e45700 1 --2- 172.21.15.44:0/1797707286 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f27cc0a2d30 0x7f27cc0a51f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f27d0005950 tx=0x7f27d00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:00.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:00.997+0000 7f27d6ffd700 1 -- 172.21.15.44:0/1797707286 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f27d8010070 con 0x7f27e01082c0 2024-04-21T23:08:01.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:00 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3244988468' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-21T23:08:01.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:01.223+0000 7f27e80a9700 1 -- 172.21.15.44:0/1797707286 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f27e002db30 con 0x7f27e01082c0 2024-04-21T23:08:01.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:01.223+0000 7f27d6ffd700 1 -- 172.21.15.44:0/1797707286 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f27d809b780 con 0x7f27e01082c0 2024-04-21T23:08:01.223 INFO:teuthology.orchestra.run.smithi044.stdout:107374182473 2024-04-21T23:08:01.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:01.226+0000 7f27e80a9700 1 -- 172.21.15.44:0/1797707286 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f27cc0a2d30 msgr2=0x7f27cc0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:01.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:01.226+0000 7f27e80a9700 1 --2- 172.21.15.44:0/1797707286 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f27cc0a2d30 0x7f27cc0a51f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f27d0005950 tx=0x7f27d00058e0 comp rx=0 tx=0).stop 2024-04-21T23:08:01.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:01.226+0000 7f27e80a9700 1 -- 172.21.15.44:0/1797707286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27e01082c0 msgr2=0x7f27e01987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:01.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:01.226+0000 7f27e80a9700 1 --2- 172.21.15.44:0/1797707286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27e01082c0 0x7f27e01987a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f27d8007ee0 tx=0x7f27d800d3b0 comp rx=0 tx=0).stop 2024-04-21T23:08:01.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:01.226+0000 7f27e80a9700 1 -- 172.21.15.44:0/1797707286 shutdown_connections 2024-04-21T23:08:01.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:01.226+0000 7f27e80a9700 1 --2- 172.21.15.44:0/1797707286 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f27cc0a2d30 0x7f27cc0a51f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:01.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:01.226+0000 7f27e80a9700 1 --2- 172.21.15.44:0/1797707286 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f27e01082c0 0x7f27e01987a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:01.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:01.226+0000 7f27e80a9700 1 --2- 172.21.15.44:0/1797707286 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f27e0100820 0x7f27e019e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:01.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:01.226+0000 7f27e80a9700 1 --2- 172.21.15.44:0/1797707286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27e01011c0 0x7f27e019eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:01.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:01.226+0000 7f27e80a9700 1 -- 172.21.15.44:0/1797707286 >> 172.21.15.44:0/1797707286 conn(0x7f27e00777a0 msgr2=0x7f27e00fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:01.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:01.226+0000 7f27e80a9700 1 -- 172.21.15.44:0/1797707286 shutdown_connections 2024-04-21T23:08:01.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:01.226+0000 7f27e80a9700 1 -- 172.21.15.44:0/1797707286 wait complete. 2024-04-21T23:08:01.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:00 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3244988468' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-21T23:08:01.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:00 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3244988468' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-21T23:08:02.041 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182468 got 107374182473 for osd.3 2024-04-21T23:08:02.042 DEBUG:teuthology.parallel:result is None 2024-04-21T23:08:02.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.703+0000 7ff34ac51700 1 -- 172.21.15.44:0/1016619458 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3441013b0 msgr2=0x7ff344101790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:02.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.703+0000 7ff34ac51700 1 --2- 172.21.15.44:0/1016619458 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3441013b0 0x7ff344101790 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff3380099c0 tx=0x7ff338009cd0 comp rx=0 tx=0).stop 2024-04-21T23:08:02.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.703+0000 7ff34ac51700 1 -- 172.21.15.44:0/1016619458 shutdown_connections 2024-04-21T23:08:02.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.703+0000 7ff34ac51700 1 --2- 172.21.15.44:0/1016619458 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff344106490 0x7ff34410e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:02.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.703+0000 7ff34ac51700 1 --2- 172.21.15.44:0/1016619458 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff3441013b0 0x7ff344101790 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:02.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.703+0000 7ff34ac51700 1 --2- 172.21.15.44:0/1016619458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff344101d60 0x7ff344105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:02.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.703+0000 7ff34ac51700 1 -- 172.21.15.44:0/1016619458 >> 172.21.15.44:0/1016619458 conn(0x7ff3440781d0 msgr2=0x7ff3440785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:02.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.703+0000 7ff34ac51700 1 -- 172.21.15.44:0/1016619458 shutdown_connections 2024-04-21T23:08:02.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.703+0000 7ff34ac51700 1 -- 172.21.15.44:0/1016619458 wait complete. 2024-04-21T23:08:02.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.704+0000 7ff34ac51700 1 Processor -- start 2024-04-21T23:08:02.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.704+0000 7ff34ac51700 1 -- start start 2024-04-21T23:08:02.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.704+0000 7ff34ac51700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3441013b0 0x7ff344199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:02.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.704+0000 7ff34ac51700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff344101d60 0x7ff344199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:02.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.704+0000 7ff34ac51700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff344106490 0x7ff3441a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:02.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.704+0000 7ff34ac51700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff344111490 con 0x7ff344106490 2024-04-21T23:08:02.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.704+0000 7ff34ac51700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff344111310 con 0x7ff344101d60 2024-04-21T23:08:02.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.704+0000 7ff34ac51700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff344111610 con 0x7ff3441013b0 2024-04-21T23:08:02.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.704+0000 7ff3491ee700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff344106490 0x7ff3441a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:02.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.704+0000 7ff3491ee700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff344106490 0x7ff3441a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51302/0 (socket says 172.21.15.44:51302) 2024-04-21T23:08:02.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.704+0000 7ff3491ee700 1 -- 172.21.15.44:0/455573004 learned_addr learned my addr 172.21.15.44:0/455573004 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:02.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.704+0000 7ff3489ed700 1 --2- 172.21.15.44:0/455573004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3441013b0 0x7ff344199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:02.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.704+0000 7ff343fff700 1 --2- 172.21.15.44:0/455573004 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff344101d60 0x7ff344199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:02.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.705+0000 7ff3491ee700 1 -- 172.21.15.44:0/455573004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3441013b0 msgr2=0x7ff344199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:02.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.705+0000 7ff3491ee700 1 --2- 172.21.15.44:0/455573004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3441013b0 0x7ff344199830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:02.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.705+0000 7ff3491ee700 1 -- 172.21.15.44:0/455573004 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff344101d60 msgr2=0x7ff344199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:02.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.705+0000 7ff3491ee700 1 --2- 172.21.15.44:0/455573004 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff344101d60 0x7ff344199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:02.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.705+0000 7ff3491ee700 1 -- 172.21.15.44:0/455573004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3441a4df0 con 0x7ff344106490 2024-04-21T23:08:02.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.705+0000 7ff343fff700 1 --2- 172.21.15.44:0/455573004 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff344101d60 0x7ff344199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:08:02.125 INFO:teuthology.orchestra.run.smithi044.stdout:287762808873 2024-04-21T23:08:02.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.705+0000 7ff3491ee700 1 --2- 172.21.15.44:0/455573004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff344106490 0x7ff3441a48b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00ada0 tx=0x7ff33c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:02.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.705+0000 7ff341ffb700 1 -- 172.21.15.44:0/455573004 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33c017020 con 0x7ff344106490 2024-04-21T23:08:02.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.705+0000 7ff341ffb700 1 -- 172.21.15.44:0/455573004 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff33c00a0d0 con 0x7ff344106490 2024-04-21T23:08:02.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.705+0000 7ff341ffb700 1 -- 172.21.15.44:0/455573004 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33c005480 con 0x7ff344106490 2024-04-21T23:08:02.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.705+0000 7ff34ac51700 1 -- 172.21.15.44:0/455573004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3441a5020 con 0x7ff344106490 2024-04-21T23:08:02.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.705+0000 7ff34ac51700 1 -- 172.21.15.44:0/455573004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3441a5530 con 0x7ff344106490 2024-04-21T23:08:02.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.706+0000 7ff34ac51700 1 -- 172.21.15.44:0/455573004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff344068f20 con 0x7ff344106490 2024-04-21T23:08:02.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.707+0000 7ff341ffb700 1 -- 172.21.15.44:0/455573004 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff33c005620 con 0x7ff344106490 2024-04-21T23:08:02.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.708+0000 7ff341ffb700 1 --2- 172.21.15.44:0/455573004 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff3340a2cb0 0x7ff3340a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:02.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.708+0000 7ff3489ed700 1 --2- 172.21.15.44:0/455573004 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff3340a2cb0 0x7ff3340a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:02.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.708+0000 7ff341ffb700 1 -- 172.21.15.44:0/455573004 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7ff33c0cec20 con 0x7ff344106490 2024-04-21T23:08:02.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.708+0000 7ff3489ed700 1 --2- 172.21.15.44:0/455573004 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff3340a2cb0 0x7ff3340a5170 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff338000c00 tx=0x7ff338005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:02.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.712+0000 7ff341ffb700 1 -- 172.21.15.44:0/455573004 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff33c021020 con 0x7ff344106490 2024-04-21T23:08:02.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.923+0000 7ff34ac51700 1 -- 172.21.15.44:0/455573004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7ff34402dcf0 con 0x7ff344106490 2024-04-21T23:08:02.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.923+0000 7ff341ffb700 1 -- 172.21.15.44:0/455573004 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff33c09b710 con 0x7ff344106490 2024-04-21T23:08:02.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.925+0000 7ff34ac51700 1 -- 172.21.15.44:0/455573004 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff3340a2cb0 msgr2=0x7ff3340a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:02.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.925+0000 7ff34ac51700 1 --2- 172.21.15.44:0/455573004 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff3340a2cb0 0x7ff3340a5170 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff338000c00 tx=0x7ff338005fb0 comp rx=0 tx=0).stop 2024-04-21T23:08:02.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.925+0000 7ff34ac51700 1 -- 172.21.15.44:0/455573004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff344106490 msgr2=0x7ff3441a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:02.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.925+0000 7ff34ac51700 1 --2- 172.21.15.44:0/455573004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff344106490 0x7ff3441a48b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00ada0 tx=0x7ff33c00c330 comp rx=0 tx=0).stop 2024-04-21T23:08:02.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.925+0000 7ff34ac51700 1 -- 172.21.15.44:0/455573004 shutdown_connections 2024-04-21T23:08:02.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.925+0000 7ff34ac51700 1 --2- 172.21.15.44:0/455573004 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7ff3340a2cb0 0x7ff3340a5170 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:02.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.925+0000 7ff34ac51700 1 --2- 172.21.15.44:0/455573004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff344106490 0x7ff3441a48b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:02.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.925+0000 7ff34ac51700 1 --2- 172.21.15.44:0/455573004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3441013b0 0x7ff344199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:02.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.925+0000 7ff34ac51700 1 --2- 172.21.15.44:0/455573004 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff344101d60 0x7ff344199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:02.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.925+0000 7ff34ac51700 1 -- 172.21.15.44:0/455573004 >> 172.21.15.44:0/455573004 conn(0x7ff3440781d0 msgr2=0x7ff3441002e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:02.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.925+0000 7ff34ac51700 1 -- 172.21.15.44:0/455573004 shutdown_connections 2024-04-21T23:08:02.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:07:57.925+0000 7ff34ac51700 1 -- 172.21.15.44:0/455573004 wait complete. 2024-04-21T23:08:02.427 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:01 smithi044 ceph-mon[27934]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:08:02.427 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:01 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1797707286' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-21T23:08:02.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:01 smithi055 ceph-mon[31117]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:08:02.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:01 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1797707286' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-21T23:08:02.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:01 smithi069 ceph-mon[31816]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:08:02.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:01 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1797707286' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-21T23:08:02.905 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808869 got 287762808873 for osd.9 2024-04-21T23:08:02.906 DEBUG:teuthology.parallel:result is None 2024-04-21T23:08:03.346 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920828 got 141733920835 for osd.4 2024-04-21T23:08:03.347 DEBUG:teuthology.parallel:result is None 2024-04-21T23:08:03.347 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-21T23:08:03.347 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-21T23:08:03.636 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:08:04.917 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:04 smithi044 ceph-mon[27934]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:04.917 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:04 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:08:04.917 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:04 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:08:04.917 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:04 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:08:04.917 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:04 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:08:04.917 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:04 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:08:04.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:04 smithi069 ceph-mon[31816]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:04.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:04 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:08:04.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:04 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:08:04.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:04 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:08:04.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:04 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:08:04.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:04 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:08:04.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:04 smithi055 ceph-mon[31117]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:04.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:04 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:08:04.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:04 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:08:04.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:04 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:08:04.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:04 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:08:04.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:04 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' 2024-04-21T23:08:05.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.502+0000 7f00619e0700 1 -- 172.21.15.44:0/4292562133 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f005c100c60 msgr2=0x7f005c1010e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:05.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.502+0000 7f00619e0700 1 --2- 172.21.15.44:0/4292562133 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f005c100c60 0x7f005c1010e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0050009990 tx=0x7f0050009ca0 comp rx=0 tx=0).stop 2024-04-21T23:08:05.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.502+0000 7f00619e0700 1 -- 172.21.15.44:0/4292562133 shutdown_connections 2024-04-21T23:08:05.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.502+0000 7f00619e0700 1 --2- 172.21.15.44:0/4292562133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f005c101620 0x7f005c10f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:05.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.502+0000 7f00619e0700 1 --2- 172.21.15.44:0/4292562133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f005c108410 0x7f005c1087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:05.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.502+0000 7f00619e0700 1 --2- 172.21.15.44:0/4292562133 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f005c100c60 0x7f005c1010e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:05.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.502+0000 7f00619e0700 1 -- 172.21.15.44:0/4292562133 >> 172.21.15.44:0/4292562133 conn(0x7f005c0763a0 msgr2=0x7f005c0767b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:05.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.502+0000 7f00619e0700 1 -- 172.21.15.44:0/4292562133 shutdown_connections 2024-04-21T23:08:05.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.502+0000 7f00619e0700 1 -- 172.21.15.44:0/4292562133 wait complete. 2024-04-21T23:08:05.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.503+0000 7f00619e0700 1 Processor -- start 2024-04-21T23:08:05.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.503+0000 7f00619e0700 1 -- start start 2024-04-21T23:08:05.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.503+0000 7f00619e0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f005c100c60 0x7f005c1a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:05.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.503+0000 7f00619e0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f005c101620 0x7f005c1a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:05.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.503+0000 7f00619e0700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f005c108410 0x7f005c19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:05.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.503+0000 7f00619e0700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f005c112530 con 0x7f005c108410 2024-04-21T23:08:05.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.503+0000 7f00619e0700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f005c1123b0 con 0x7f005c101620 2024-04-21T23:08:05.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.503+0000 7f00619e0700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f005c1126b0 con 0x7f005c100c60 2024-04-21T23:08:05.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.503+0000 7f005a7fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f005c101620 0x7f005c1a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:05.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.503+0000 7f005b7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f005c108410 0x7f005c19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:05.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.503+0000 7f005a7fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f005c101620 0x7f005c1a0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:47154/0 (socket says 172.21.15.44:47154) 2024-04-21T23:08:05.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.503+0000 7f005a7fc700 1 -- 172.21.15.44:0/3227284591 learned_addr learned my addr 172.21.15.44:0/3227284591 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:05.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.503+0000 7f005affd700 1 --2- 172.21.15.44:0/3227284591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f005c100c60 0x7f005c1a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:05.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.504+0000 7f005a7fc700 1 -- 172.21.15.44:0/3227284591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f005c100c60 msgr2=0x7f005c1a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:05.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.504+0000 7f005a7fc700 1 --2- 172.21.15.44:0/3227284591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f005c100c60 0x7f005c1a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:05.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.504+0000 7f005a7fc700 1 -- 172.21.15.44:0/3227284591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f005c108410 msgr2=0x7f005c19aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:05.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.504+0000 7f005a7fc700 1 --2- 172.21.15.44:0/3227284591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f005c108410 0x7f005c19aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:05.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.504+0000 7f005a7fc700 1 -- 172.21.15.44:0/3227284591 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005c19b230 con 0x7f005c101620 2024-04-21T23:08:05.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.504+0000 7f005b7fe700 1 --2- 172.21.15.44:0/3227284591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f005c108410 0x7f005c19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:08:05.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.504+0000 7f005a7fc700 1 --2- 172.21.15.44:0/3227284591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f005c101620 0x7f005c1a0e50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0050000c00 tx=0x7f00500047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:05.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.504+0000 7f00609de700 1 -- 172.21.15.44:0/3227284591 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0050027070 con 0x7f005c101620 2024-04-21T23:08:05.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.504+0000 7f00619e0700 1 -- 172.21.15.44:0/3227284591 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f005c19b4c0 con 0x7f005c101620 2024-04-21T23:08:05.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.504+0000 7f00609de700 1 -- 172.21.15.44:0/3227284591 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0050003e40 con 0x7f005c101620 2024-04-21T23:08:05.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.504+0000 7f00609de700 1 -- 172.21.15.44:0/3227284591 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0050022d10 con 0x7f005c101620 2024-04-21T23:08:05.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.504+0000 7f00619e0700 1 -- 172.21.15.44:0/3227284591 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f005c1a77e0 con 0x7f005c101620 2024-04-21T23:08:05.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.505+0000 7f00619e0700 1 -- 172.21.15.44:0/3227284591 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f005c068f20 con 0x7f005c101620 2024-04-21T23:08:05.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.506+0000 7f00609de700 1 -- 172.21.15.44:0/3227284591 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f005002a070 con 0x7f005c101620 2024-04-21T23:08:05.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.507+0000 7f00609de700 1 --2- 172.21.15.44:0/3227284591 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f00440a2cb0 0x7f00440a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:05.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.507+0000 7f005affd700 1 --2- 172.21.15.44:0/3227284591 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f00440a2cb0 0x7f00440a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:05.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.507+0000 7f00609de700 1 -- 172.21.15.44:0/3227284591 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f00500d09e0 con 0x7f005c101620 2024-04-21T23:08:05.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.508+0000 7f005affd700 1 --2- 172.21.15.44:0/3227284591 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f00440a2cb0 0x7f00440a5170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f004c005950 tx=0x7f004c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:05.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.511+0000 7f00609de700 1 -- 172.21.15.44:0/3227284591 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0050031070 con 0x7f005c101620 2024-04-21T23:08:05.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.712+0000 7f00619e0700 1 -- 172.21.15.44:0/3227284591 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f005c0ffd30 con 0x7f00440a2cb0 2024-04-21T23:08:05.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.722+0000 7f00609de700 1 -- 172.21.15.44:0/3227284591 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340295 (secure 0 0 0) 0x7f005c0ffd30 con 0x7f00440a2cb0 2024-04-21T23:08:05.726 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:05.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.725+0000 7f00619e0700 1 -- 172.21.15.44:0/3227284591 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f00440a2cb0 msgr2=0x7f00440a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:05.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.725+0000 7f00619e0700 1 --2- 172.21.15.44:0/3227284591 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f00440a2cb0 0x7f00440a5170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f004c005950 tx=0x7f004c00b410 comp rx=0 tx=0).stop 2024-04-21T23:08:05.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.725+0000 7f00619e0700 1 -- 172.21.15.44:0/3227284591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f005c101620 msgr2=0x7f005c1a0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:05.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.725+0000 7f00619e0700 1 --2- 172.21.15.44:0/3227284591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f005c101620 0x7f005c1a0e50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0050000c00 tx=0x7f00500047b0 comp rx=0 tx=0).stop 2024-04-21T23:08:05.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.725+0000 7f00619e0700 1 -- 172.21.15.44:0/3227284591 shutdown_connections 2024-04-21T23:08:05.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.725+0000 7f00619e0700 1 --2- 172.21.15.44:0/3227284591 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f00440a2cb0 0x7f00440a5170 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:05.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.725+0000 7f00619e0700 1 --2- 172.21.15.44:0/3227284591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f005c108410 0x7f005c19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:05.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.725+0000 7f00619e0700 1 --2- 172.21.15.44:0/3227284591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f005c100c60 0x7f005c1a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:05.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.725+0000 7f00619e0700 1 --2- 172.21.15.44:0/3227284591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f005c101620 0x7f005c1a0e50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:05.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.725+0000 7f00619e0700 1 -- 172.21.15.44:0/3227284591 >> 172.21.15.44:0/3227284591 conn(0x7f005c0763a0 msgr2=0x7f005c0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:05.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.726+0000 7f00619e0700 1 -- 172.21.15.44:0/3227284591 shutdown_connections 2024-04-21T23:08:05.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:05.726+0000 7f00619e0700 1 -- 172.21.15.44:0/3227284591 wait complete. 2024-04-21T23:08:05.729 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-21T23:08:06.497 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-04-21T23:08:05.078277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654242,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":263,"num_write_kb":3031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":363328,"kb_used_data":14096,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124578496,"statfs":{"total":1151940427776,"available":1151568379904,"internally_reserved":0,"allocated":14434304,"data_stored":8027636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003298"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.449965+0000","last_change":"2024-04-21T23:06:25.980091+0000","last_active":"2024-04-21T23:07:24.449965+0000","last_peered":"2024-04-21T23:07:24.449965+0000","last_clean":"2024-04-21T23:07:24.449965+0000","last_became_active":"2024-04-21T23:06:25.979763+0000","last_became_peered":"2024-04-21T23:06:25.979763+0000","last_unstale":"2024-04-21T23:07:24.449965+0000","last_undegraded":"2024-04-21T23:07:24.449965+0000","last_fullsized":"2024-04-21T23:07:24.449965+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:12:23.771931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451425+0000","last_change":"2024-04-21T23:06:25.992461+0000","last_active":"2024-04-21T23:07:24.451425+0000","last_peered":"2024-04-21T23:07:24.451425+0000","last_clean":"2024-04-21T23:07:24.451425+0000","last_became_active":"2024-04-21T23:06:25.992018+0000","last_became_peered":"2024-04-21T23:06:25.992018+0000","last_unstale":"2024-04-21T23:07:24.451425+0000","last_undegraded":"2024-04-21T23:07:24.451425+0000","last_fullsized":"2024-04-21T23:07:24.451425+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:46.651683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451416+0000","last_change":"2024-04-21T23:06:25.974812+0000","last_active":"2024-04-21T23:07:24.451416+0000","last_peered":"2024-04-21T23:07:24.451416+0000","last_clean":"2024-04-21T23:07:24.451416+0000","last_became_active":"2024-04-21T23:06:25.974420+0000","last_became_peered":"2024-04-21T23:06:25.974420+0000","last_unstale":"2024-04-21T23:07:24.451416+0000","last_undegraded":"2024-04-21T23:07:24.451416+0000","last_fullsized":"2024-04-21T23:07:24.451416+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:00.734218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451907+0000","last_change":"2024-04-21T23:06:25.988806+0000","last_active":"2024-04-21T23:07:24.451907+0000","last_peered":"2024-04-21T23:07:24.451907+0000","last_clean":"2024-04-21T23:07:24.451907+0000","last_became_active":"2024-04-21T23:06:25.988344+0000","last_became_peered":"2024-04-21T23:06:25.988344+0000","last_unstale":"2024-04-21T23:07:24.451907+0000","last_undegraded":"2024-04-21T23:07:24.451907+0000","last_fullsized":"2024-04-21T23:07:24.451907+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:15:40.079273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.446257+0000","last_change":"2024-04-21T23:06:25.979125+0000","last_active":"2024-04-21T23:07:24.446257+0000","last_peered":"2024-04-21T23:07:24.446257+0000","last_clean":"2024-04-21T23:07:24.446257+0000","last_became_active":"2024-04-21T23:06:25.978778+0000","last_became_peered":"2024-04-21T23:06:25.978778+0000","last_unstale":"2024-04-21T23:07:24.446257+0000","last_undegraded":"2024-04-21T23:07:24.446257+0000","last_fullsized":"2024-04-21T23:07:24.446257+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:44.655404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.448111+0000","last_change":"2024-04-21T23:06:25.983983+0000","last_active":"2024-04-21T23:07:24.448111+0000","last_peered":"2024-04-21T23:07:24.448111+0000","last_clean":"2024-04-21T23:07:24.448111+0000","last_became_active":"2024-04-21T23:06:25.983463+0000","last_became_peered":"2024-04-21T23:06:25.983463+0000","last_unstale":"2024-04-21T23:07:24.448111+0000","last_undegraded":"2024-04-21T23:07:24.448111+0000","last_fullsized":"2024-04-21T23:07:24.448111+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:32:12.784752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450288+0000","last_change":"2024-04-21T23:06:25.987479+0000","last_active":"2024-04-21T23:07:24.450288+0000","last_peered":"2024-04-21T23:07:24.450288+0000","last_clean":"2024-04-21T23:07:24.450288+0000","last_became_active":"2024-04-21T23:06:25.987133+0000","last_became_peered":"2024-04-21T23:06:25.987133+0000","last_unstale":"2024-04-21T23:07:24.450288+0000","last_undegraded":"2024-04-21T23:07:24.450288+0000","last_fullsized":"2024-04-21T23:07:24.450288+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:08.033184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.814870+0000","last_change":"2024-04-21T23:06:25.987714+0000","last_active":"2024-04-21T23:07:24.814870+0000","last_peered":"2024-04-21T23:07:24.814870+0000","last_clean":"2024-04-21T23:07:24.814870+0000","last_became_active":"2024-04-21T23:06:25.987457+0000","last_became_peered":"2024-04-21T23:06:25.987457+0000","last_unstale":"2024-04-21T23:07:24.814870+0000","last_undegraded":"2024-04-21T23:07:24.814870+0000","last_fullsized":"2024-04-21T23:07:24.814870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:06:06.184525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.816484+0000","last_change":"2024-04-21T23:06:25.975957+0000","last_active":"2024-04-21T23:07:24.816484+0000","last_peered":"2024-04-21T23:07:24.816484+0000","last_clean":"2024-04-21T23:07:24.816484+0000","last_became_active":"2024-04-21T23:06:25.974710+0000","last_became_peered":"2024-04-21T23:06:25.974710+0000","last_unstale":"2024-04-21T23:07:24.816484+0000","last_undegraded":"2024-04-21T23:07:24.816484+0000","last_fullsized":"2024-04-21T23:07:24.816484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:31:40.062273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.448435+0000","last_change":"2024-04-21T23:06:25.995085+0000","last_active":"2024-04-21T23:07:24.448435+0000","last_peered":"2024-04-21T23:07:24.448435+0000","last_clean":"2024-04-21T23:07:24.448435+0000","last_became_active":"2024-04-21T23:06:25.993552+0000","last_became_peered":"2024-04-21T23:06:25.993552+0000","last_unstale":"2024-04-21T23:07:24.448435+0000","last_undegraded":"2024-04-21T23:07:24.448435+0000","last_fullsized":"2024-04-21T23:07:24.448435+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:11.405113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452004+0000","last_change":"2024-04-21T23:06:25.993191+0000","last_active":"2024-04-21T23:07:24.452004+0000","last_peered":"2024-04-21T23:07:24.452004+0000","last_clean":"2024-04-21T23:07:24.452004+0000","last_became_active":"2024-04-21T23:06:25.989399+0000","last_became_peered":"2024-04-21T23:06:25.989399+0000","last_unstale":"2024-04-21T23:07:24.452004+0000","last_undegraded":"2024-04-21T23:07:24.452004+0000","last_fullsized":"2024-04-21T23:07:24.452004+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:30:20.035599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451797+0000","last_change":"2024-04-21T23:06:25.994158+0000","last_active":"2024-04-21T23:07:24.451797+0000","last_peered":"2024-04-21T23:07:24.451797+0000","last_clean":"2024-04-21T23:07:24.451797+0000","last_became_active":"2024-04-21T23:06:25.993780+0000","last_became_peered":"2024-04-21T23:06:25.993780+0000","last_unstale":"2024-04-21T23:07:24.451797+0000","last_undegraded":"2024-04-21T23:07:24.451797+0000","last_fullsized":"2024-04-21T23:07:24.451797+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:52:17.181898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451391+0000","last_change":"2024-04-21T23:06:25.989274+0000","last_active":"2024-04-21T23:07:24.451391+0000","last_peered":"2024-04-21T23:07:24.451391+0000","last_clean":"2024-04-21T23:07:24.451391+0000","last_became_active":"2024-04-21T23:06:25.988931+0000","last_became_peered":"2024-04-21T23:06:25.988931+0000","last_unstale":"2024-04-21T23:07:24.451391+0000","last_undegraded":"2024-04-21T23:07:24.451391+0000","last_fullsized":"2024-04-21T23:07:24.451391+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:11:04.576136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451115+0000","last_change":"2024-04-21T23:06:25.982976+0000","last_active":"2024-04-21T23:07:24.451115+0000","last_peered":"2024-04-21T23:07:24.451115+0000","last_clean":"2024-04-21T23:07:24.451115+0000","last_became_active":"2024-04-21T23:06:25.982625+0000","last_became_peered":"2024-04-21T23:06:25.982625+0000","last_unstale":"2024-04-21T23:07:24.451115+0000","last_undegraded":"2024-04-21T23:07:24.451115+0000","last_fullsized":"2024-04-21T23:07:24.451115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:51:02.193044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450985+0000","last_change":"2024-04-21T23:06:25.994805+0000","last_active":"2024-04-21T23:07:24.450985+0000","last_peered":"2024-04-21T23:07:24.450985+0000","last_clean":"2024-04-21T23:07:24.450985+0000","last_became_active":"2024-04-21T23:06:25.994482+0000","last_became_peered":"2024-04-21T23:06:25.994482+0000","last_unstale":"2024-04-21T23:07:24.450985+0000","last_undegraded":"2024-04-21T23:07:24.450985+0000","last_fullsized":"2024-04-21T23:07:24.450985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:24:21.792358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815354+0000","last_change":"2024-04-21T23:06:25.990840+0000","last_active":"2024-04-21T23:07:24.815354+0000","last_peered":"2024-04-21T23:07:24.815354+0000","last_clean":"2024-04-21T23:07:24.815354+0000","last_became_active":"2024-04-21T23:06:25.990517+0000","last_became_peered":"2024-04-21T23:06:25.990517+0000","last_unstale":"2024-04-21T23:07:24.815354+0000","last_undegraded":"2024-04-21T23:07:24.815354+0000","last_fullsized":"2024-04-21T23:07:24.815354+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:48:09.265280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.449709+0000","last_change":"2024-04-21T23:06:25.976810+0000","last_active":"2024-04-21T23:07:24.449709+0000","last_peered":"2024-04-21T23:07:24.449709+0000","last_clean":"2024-04-21T23:07:24.449709+0000","last_became_active":"2024-04-21T23:06:25.976489+0000","last_became_peered":"2024-04-21T23:06:25.976489+0000","last_unstale":"2024-04-21T23:07:24.449709+0000","last_undegraded":"2024-04-21T23:07:24.449709+0000","last_fullsized":"2024-04-21T23:07:24.449709+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:07:03.794115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447953+0000","last_change":"2024-04-21T23:06:25.988603+0000","last_active":"2024-04-21T23:07:24.447953+0000","last_peered":"2024-04-21T23:07:24.447953+0000","last_clean":"2024-04-21T23:07:24.447953+0000","last_became_active":"2024-04-21T23:06:25.988254+0000","last_became_peered":"2024-04-21T23:06:25.988254+0000","last_unstale":"2024-04-21T23:07:24.447953+0000","last_undegraded":"2024-04-21T23:07:24.447953+0000","last_fullsized":"2024-04-21T23:07:24.447953+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:25:46.099853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452093+0000","last_change":"2024-04-21T23:06:25.980831+0000","last_active":"2024-04-21T23:07:24.452093+0000","last_peered":"2024-04-21T23:07:24.452093+0000","last_clean":"2024-04-21T23:07:24.452093+0000","last_became_active":"2024-04-21T23:06:25.977051+0000","last_became_peered":"2024-04-21T23:06:25.977051+0000","last_unstale":"2024-04-21T23:07:24.452093+0000","last_undegraded":"2024-04-21T23:07:24.452093+0000","last_fullsized":"2024-04-21T23:07:24.452093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:55:03.684734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450854+0000","last_change":"2024-04-21T23:06:30.256875+0000","last_active":"2024-04-21T23:07:24.450854+0000","last_peered":"2024-04-21T23:07:24.450854+0000","last_clean":"2024-04-21T23:07:24.450854+0000","last_became_active":"2024-04-21T23:06:30.256459+0000","last_became_peered":"2024-04-21T23:06:30.256459+0000","last_unstale":"2024-04-21T23:07:24.450854+0000","last_undegraded":"2024-04-21T23:07:24.450854+0000","last_fullsized":"2024-04-21T23:07:24.450854+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:45:32.376270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447817+0000","last_change":"2024-04-21T23:06:25.209605+0000","last_active":"2024-04-21T23:07:24.447817+0000","last_peered":"2024-04-21T23:07:24.447817+0000","last_clean":"2024-04-21T23:07:24.447817+0000","last_became_active":"2024-04-21T23:06:25.209277+0000","last_became_peered":"2024-04-21T23:06:25.209277+0000","last_unstale":"2024-04-21T23:07:24.447817+0000","last_undegraded":"2024-04-21T23:07:24.447817+0000","last_fullsized":"2024-04-21T23:07:24.447817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:15:49.742940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815277+0000","last_change":"2024-04-21T23:06:25.981774+0000","last_active":"2024-04-21T23:07:24.815277+0000","last_peered":"2024-04-21T23:07:24.815277+0000","last_clean":"2024-04-21T23:07:24.815277+0000","last_became_active":"2024-04-21T23:06:25.981432+0000","last_became_peered":"2024-04-21T23:06:25.981432+0000","last_unstale":"2024-04-21T23:07:24.815277+0000","last_undegraded":"2024-04-21T23:07:24.815277+0000","last_fullsized":"2024-04-21T23:07:24.815277+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:28:28.712149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451351+0000","last_change":"2024-04-21T23:06:30.271508+0000","last_active":"2024-04-21T23:07:24.451351+0000","last_peered":"2024-04-21T23:07:24.451351+0000","last_clean":"2024-04-21T23:07:24.451351+0000","last_became_active":"2024-04-21T23:06:30.270461+0000","last_became_peered":"2024-04-21T23:06:30.270461+0000","last_unstale":"2024-04-21T23:07:24.451351+0000","last_undegraded":"2024-04-21T23:07:24.451351+0000","last_fullsized":"2024-04-21T23:07:24.451351+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:43:07.473365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815649+0000","last_change":"2024-04-21T23:06:25.218773+0000","last_active":"2024-04-21T23:07:24.815649+0000","last_peered":"2024-04-21T23:07:24.815649+0000","last_clean":"2024-04-21T23:07:24.815649+0000","last_became_active":"2024-04-21T23:06:25.218458+0000","last_became_peered":"2024-04-21T23:06:25.218458+0000","last_unstale":"2024-04-21T23:07:24.815649+0000","last_undegraded":"2024-04-21T23:07:24.815649+0000","last_fullsized":"2024-04-21T23:07:24.815649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:25:16.372689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451480+0000","last_change":"2024-04-21T23:06:25.977669+0000","last_active":"2024-04-21T23:07:24.451480+0000","last_peered":"2024-04-21T23:07:24.451480+0000","last_clean":"2024-04-21T23:07:24.451480+0000","last_became_active":"2024-04-21T23:06:25.977143+0000","last_became_peered":"2024-04-21T23:06:25.977143+0000","last_unstale":"2024-04-21T23:07:24.451480+0000","last_undegraded":"2024-04-21T23:07:24.451480+0000","last_fullsized":"2024-04-21T23:07:24.451480+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:39.289208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451360+0000","last_change":"2024-04-21T23:06:30.271934+0000","last_active":"2024-04-21T23:07:24.451360+0000","last_peered":"2024-04-21T23:07:24.451360+0000","last_clean":"2024-04-21T23:07:24.451360+0000","last_became_active":"2024-04-21T23:06:30.270567+0000","last_became_peered":"2024-04-21T23:06:30.270567+0000","last_unstale":"2024-04-21T23:07:24.451360+0000","last_undegraded":"2024-04-21T23:07:24.451360+0000","last_fullsized":"2024-04-21T23:07:24.451360+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:43:34.908748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.462595+0000","last_change":"2024-04-21T23:07:24.462595+0000","last_active":"2024-04-21T23:07:24.462595+0000","last_peered":"2024-04-21T23:07:24.462595+0000","last_clean":"2024-04-21T23:07:24.462595+0000","last_became_active":"2024-04-21T23:07:24.454522+0000","last_became_peered":"2024-04-21T23:07:24.454522+0000","last_unstale":"2024-04-21T23:07:24.462595+0000","last_undegraded":"2024-04-21T23:07:24.462595+0000","last_fullsized":"2024-04-21T23:07:24.462595+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:07.457105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.816028+0000","last_change":"2024-04-21T23:06:25.988197+0000","last_active":"2024-04-21T23:07:24.816028+0000","last_peered":"2024-04-21T23:07:24.816028+0000","last_clean":"2024-04-21T23:07:24.816028+0000","last_became_active":"2024-04-21T23:06:25.987872+0000","last_became_peered":"2024-04-21T23:06:25.987872+0000","last_unstale":"2024-04-21T23:07:24.816028+0000","last_undegraded":"2024-04-21T23:07:24.816028+0000","last_fullsized":"2024-04-21T23:07:24.816028+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:25.389020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450816+0000","last_change":"2024-04-21T23:06:30.265262+0000","last_active":"2024-04-21T23:07:24.450816+0000","last_peered":"2024-04-21T23:07:24.450816+0000","last_clean":"2024-04-21T23:07:24.450816+0000","last_became_active":"2024-04-21T23:06:30.264781+0000","last_became_peered":"2024-04-21T23:06:30.264781+0000","last_unstale":"2024-04-21T23:07:24.450816+0000","last_undegraded":"2024-04-21T23:07:24.450816+0000","last_fullsized":"2024-04-21T23:07:24.450816+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:03:30.756792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.446480+0000","last_change":"2024-04-21T23:06:25.205070+0000","last_active":"2024-04-21T23:07:24.446480+0000","last_peered":"2024-04-21T23:07:24.446480+0000","last_clean":"2024-04-21T23:07:24.446480+0000","last_became_active":"2024-04-21T23:06:25.204737+0000","last_became_peered":"2024-04-21T23:06:25.204737+0000","last_unstale":"2024-04-21T23:07:24.446480+0000","last_undegraded":"2024-04-21T23:07:24.446480+0000","last_fullsized":"2024-04-21T23:07:24.446480+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:37.014534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.453979+0000","last_change":"2024-04-21T23:06:25.986584+0000","last_active":"2024-04-21T23:07:24.453979+0000","last_peered":"2024-04-21T23:07:24.453979+0000","last_clean":"2024-04-21T23:07:24.453979+0000","last_became_active":"2024-04-21T23:06:25.986268+0000","last_became_peered":"2024-04-21T23:06:25.986268+0000","last_unstale":"2024-04-21T23:07:24.453979+0000","last_undegraded":"2024-04-21T23:07:24.453979+0000","last_fullsized":"2024-04-21T23:07:24.453979+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:19:43.864279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451912+0000","last_change":"2024-04-21T23:06:30.261900+0000","last_active":"2024-04-21T23:07:24.451912+0000","last_peered":"2024-04-21T23:07:24.451912+0000","last_clean":"2024-04-21T23:07:24.451912+0000","last_became_active":"2024-04-21T23:06:30.261444+0000","last_became_peered":"2024-04-21T23:06:30.261444+0000","last_unstale":"2024-04-21T23:07:24.451912+0000","last_undegraded":"2024-04-21T23:07:24.451912+0000","last_fullsized":"2024-04-21T23:07:24.451912+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:10:40.810555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.817616+0000","last_change":"2024-04-21T23:07:24.817616+0000","last_active":"2024-04-21T23:07:24.817616+0000","last_peered":"2024-04-21T23:07:24.817616+0000","last_clean":"2024-04-21T23:07:24.817616+0000","last_became_active":"2024-04-21T23:07:24.816888+0000","last_became_peered":"2024-04-21T23:07:24.816888+0000","last_unstale":"2024-04-21T23:07:24.817616+0000","last_undegraded":"2024-04-21T23:07:24.817616+0000","last_fullsized":"2024-04-21T23:07:24.817616+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:01:03.069832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.446272+0000","last_change":"2024-04-21T23:06:25.975999+0000","last_active":"2024-04-21T23:07:24.446272+0000","last_peered":"2024-04-21T23:07:24.446272+0000","last_clean":"2024-04-21T23:07:24.446272+0000","last_became_active":"2024-04-21T23:06:25.975677+0000","last_became_peered":"2024-04-21T23:06:25.975677+0000","last_unstale":"2024-04-21T23:07:24.446272+0000","last_undegraded":"2024-04-21T23:07:24.446272+0000","last_fullsized":"2024-04-21T23:07:24.446272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:52:03.177248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.456926+0000","last_change":"2024-04-21T23:07:24.456926+0000","last_active":"2024-04-21T23:07:24.456926+0000","last_peered":"2024-04-21T23:07:24.456926+0000","last_clean":"2024-04-21T23:07:24.456926+0000","last_became_active":"2024-04-21T23:07:24.456481+0000","last_became_peered":"2024-04-21T23:07:24.456481+0000","last_unstale":"2024-04-21T23:07:24.456926+0000","last_undegraded":"2024-04-21T23:07:24.456926+0000","last_fullsized":"2024-04-21T23:07:24.456926+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:19:26.553012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447850+0000","last_change":"2024-04-21T23:06:25.214893+0000","last_active":"2024-04-21T23:07:24.447850+0000","last_peered":"2024-04-21T23:07:24.447850+0000","last_clean":"2024-04-21T23:07:24.447850+0000","last_became_active":"2024-04-21T23:06:25.212359+0000","last_became_peered":"2024-04-21T23:06:25.212359+0000","last_unstale":"2024-04-21T23:07:24.447850+0000","last_undegraded":"2024-04-21T23:07:24.447850+0000","last_fullsized":"2024-04-21T23:07:24.447850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:18:03.278875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.453767+0000","last_change":"2024-04-21T23:06:25.986849+0000","last_active":"2024-04-21T23:07:24.453767+0000","last_peered":"2024-04-21T23:07:24.453767+0000","last_clean":"2024-04-21T23:07:24.453767+0000","last_became_active":"2024-04-21T23:06:25.986334+0000","last_became_peered":"2024-04-21T23:06:25.986334+0000","last_unstale":"2024-04-21T23:07:24.453767+0000","last_undegraded":"2024-04-21T23:07:24.453767+0000","last_fullsized":"2024-04-21T23:07:24.453767+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:57:42.195629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.449586+0000","last_change":"2024-04-21T23:06:30.270812+0000","last_active":"2024-04-21T23:07:24.449586+0000","last_peered":"2024-04-21T23:07:24.449586+0000","last_clean":"2024-04-21T23:07:24.449586+0000","last_became_active":"2024-04-21T23:06:30.270370+0000","last_became_peered":"2024-04-21T23:06:30.270370+0000","last_unstale":"2024-04-21T23:07:24.449586+0000","last_undegraded":"2024-04-21T23:07:24.449586+0000","last_fullsized":"2024-04-21T23:07:24.449586+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:53.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450585+0000","last_change":"2024-04-21T23:06:25.201603+0000","last_active":"2024-04-21T23:07:24.450585+0000","last_peered":"2024-04-21T23:07:24.450585+0000","last_clean":"2024-04-21T23:07:24.450585+0000","last_became_active":"2024-04-21T23:06:25.201265+0000","last_became_peered":"2024-04-21T23:06:25.201265+0000","last_unstale":"2024-04-21T23:07:24.450585+0000","last_undegraded":"2024-04-21T23:07:24.450585+0000","last_fullsized":"2024-04-21T23:07:24.450585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:23:19.134215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.814984+0000","last_change":"2024-04-21T23:06:25.982420+0000","last_active":"2024-04-21T23:07:24.814984+0000","last_peered":"2024-04-21T23:07:24.814984+0000","last_clean":"2024-04-21T23:07:24.814984+0000","last_became_active":"2024-04-21T23:06:25.981867+0000","last_became_peered":"2024-04-21T23:06:25.981867+0000","last_unstale":"2024-04-21T23:07:24.814984+0000","last_undegraded":"2024-04-21T23:07:24.814984+0000","last_fullsized":"2024-04-21T23:07:24.814984+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:10:29.433243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451298+0000","last_change":"2024-04-21T23:06:30.273907+0000","last_active":"2024-04-21T23:07:24.451298+0000","last_peered":"2024-04-21T23:07:24.451298+0000","last_clean":"2024-04-21T23:07:24.451298+0000","last_became_active":"2024-04-21T23:06:30.273472+0000","last_became_peered":"2024-04-21T23:06:30.273472+0000","last_unstale":"2024-04-21T23:07:24.451298+0000","last_undegraded":"2024-04-21T23:07:24.451298+0000","last_fullsized":"2024-04-21T23:07:24.451298+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:58:50.914940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452141+0000","last_change":"2024-04-21T23:06:25.211117+0000","last_active":"2024-04-21T23:07:24.452141+0000","last_peered":"2024-04-21T23:07:24.452141+0000","last_clean":"2024-04-21T23:07:24.452141+0000","last_became_active":"2024-04-21T23:06:25.210626+0000","last_became_peered":"2024-04-21T23:06:25.210626+0000","last_unstale":"2024-04-21T23:07:24.452141+0000","last_undegraded":"2024-04-21T23:07:24.452141+0000","last_fullsized":"2024-04-21T23:07:24.452141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:56:39.917072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447267+0000","last_change":"2024-04-21T23:06:25.980537+0000","last_active":"2024-04-21T23:07:24.447267+0000","last_peered":"2024-04-21T23:07:24.447267+0000","last_clean":"2024-04-21T23:07:24.447267+0000","last_became_active":"2024-04-21T23:06:25.980188+0000","last_became_peered":"2024-04-21T23:06:25.980188+0000","last_unstale":"2024-04-21T23:07:24.447267+0000","last_undegraded":"2024-04-21T23:07:24.447267+0000","last_fullsized":"2024-04-21T23:07:24.447267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:58:02.828632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815003+0000","last_change":"2024-04-21T23:06:30.263565+0000","last_active":"2024-04-21T23:07:24.815003+0000","last_peered":"2024-04-21T23:07:24.815003+0000","last_clean":"2024-04-21T23:07:24.815003+0000","last_became_active":"2024-04-21T23:06:30.263092+0000","last_became_peered":"2024-04-21T23:06:30.263092+0000","last_unstale":"2024-04-21T23:07:24.815003+0000","last_undegraded":"2024-04-21T23:07:24.815003+0000","last_fullsized":"2024-04-21T23:07:24.815003+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:58:25.709551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.448432+0000","last_change":"2024-04-21T23:06:25.212567+0000","last_active":"2024-04-21T23:07:24.448432+0000","last_peered":"2024-04-21T23:07:24.448432+0000","last_clean":"2024-04-21T23:07:24.448432+0000","last_became_active":"2024-04-21T23:06:25.210593+0000","last_became_peered":"2024-04-21T23:06:25.210593+0000","last_unstale":"2024-04-21T23:07:24.448432+0000","last_undegraded":"2024-04-21T23:07:24.448432+0000","last_fullsized":"2024-04-21T23:07:24.448432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:53:33.552742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.816324+0000","last_change":"2024-04-21T23:06:25.989525+0000","last_active":"2024-04-21T23:07:24.816324+0000","last_peered":"2024-04-21T23:07:24.816324+0000","last_clean":"2024-04-21T23:07:24.816324+0000","last_became_active":"2024-04-21T23:06:25.989130+0000","last_became_peered":"2024-04-21T23:06:25.989130+0000","last_unstale":"2024-04-21T23:07:24.816324+0000","last_undegraded":"2024-04-21T23:07:24.816324+0000","last_fullsized":"2024-04-21T23:07:24.816324+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:19:31.000530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447165+0000","last_change":"2024-04-21T23:06:30.269309+0000","last_active":"2024-04-21T23:07:24.447165+0000","last_peered":"2024-04-21T23:07:24.447165+0000","last_clean":"2024-04-21T23:07:24.447165+0000","last_became_active":"2024-04-21T23:06:30.268903+0000","last_became_peered":"2024-04-21T23:06:30.268903+0000","last_unstale":"2024-04-21T23:07:24.447165+0000","last_undegraded":"2024-04-21T23:07:24.447165+0000","last_fullsized":"2024-04-21T23:07:24.447165+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:52:16.043870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451651+0000","last_change":"2024-04-21T23:06:25.213938+0000","last_active":"2024-04-21T23:07:24.451651+0000","last_peered":"2024-04-21T23:07:24.451651+0000","last_clean":"2024-04-21T23:07:24.451651+0000","last_became_active":"2024-04-21T23:06:25.213242+0000","last_became_peered":"2024-04-21T23:06:25.213242+0000","last_unstale":"2024-04-21T23:07:24.451651+0000","last_undegraded":"2024-04-21T23:07:24.451651+0000","last_fullsized":"2024-04-21T23:07:24.451651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:27.679122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.454702+0000","last_change":"2024-04-21T23:07:24.454702+0000","last_active":"2024-04-21T23:07:24.454702+0000","last_peered":"2024-04-21T23:07:24.454702+0000","last_clean":"2024-04-21T23:07:24.454702+0000","last_became_active":"2024-04-21T23:07:24.454211+0000","last_became_peered":"2024-04-21T23:07:24.454211+0000","last_unstale":"2024-04-21T23:07:24.454702+0000","last_undegraded":"2024-04-21T23:07:24.454702+0000","last_fullsized":"2024-04-21T23:07:24.454702+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:29:30.745026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451525+0000","last_change":"2024-04-21T23:06:30.272206+0000","last_active":"2024-04-21T23:07:24.451525+0000","last_peered":"2024-04-21T23:07:24.451525+0000","last_clean":"2024-04-21T23:07:24.451525+0000","last_became_active":"2024-04-21T23:06:30.271389+0000","last_became_peered":"2024-04-21T23:06:30.271389+0000","last_unstale":"2024-04-21T23:07:24.451525+0000","last_undegraded":"2024-04-21T23:07:24.451525+0000","last_fullsized":"2024-04-21T23:07:24.451525+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:54:47.996166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.454670+0000","last_change":"2024-04-21T23:06:25.217655+0000","last_active":"2024-04-21T23:07:24.454670+0000","last_peered":"2024-04-21T23:07:24.454670+0000","last_clean":"2024-04-21T23:07:24.454670+0000","last_became_active":"2024-04-21T23:06:25.217337+0000","last_became_peered":"2024-04-21T23:06:25.217337+0000","last_unstale":"2024-04-21T23:07:24.454670+0000","last_undegraded":"2024-04-21T23:07:24.454670+0000","last_fullsized":"2024-04-21T23:07:24.454670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:29:28.690841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.816161+0000","last_change":"2024-04-21T23:06:25.995757+0000","last_active":"2024-04-21T23:07:24.816161+0000","last_peered":"2024-04-21T23:07:24.816161+0000","last_clean":"2024-04-21T23:07:24.816161+0000","last_became_active":"2024-04-21T23:06:25.995443+0000","last_became_peered":"2024-04-21T23:06:25.995443+0000","last_unstale":"2024-04-21T23:07:24.816161+0000","last_undegraded":"2024-04-21T23:07:24.816161+0000","last_fullsized":"2024-04-21T23:07:24.816161+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:40.359633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447349+0000","last_change":"2024-04-21T23:06:30.266441+0000","last_active":"2024-04-21T23:07:24.447349+0000","last_peered":"2024-04-21T23:07:24.447349+0000","last_clean":"2024-04-21T23:07:24.447349+0000","last_became_active":"2024-04-21T23:06:30.264065+0000","last_became_peered":"2024-04-21T23:06:30.264065+0000","last_unstale":"2024-04-21T23:07:24.447349+0000","last_undegraded":"2024-04-21T23:07:24.447349+0000","last_fullsized":"2024-04-21T23:07:24.447349+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:53:22.548096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'31","reported_seq":213,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:08:03.878831+0000","last_change":"2024-04-21T23:06:25.213919+0000","last_active":"2024-04-21T23:08:03.878831+0000","last_peered":"2024-04-21T23:08:03.878831+0000","last_clean":"2024-04-21T23:08:03.878831+0000","last_became_active":"2024-04-21T23:06:25.212244+0000","last_became_peered":"2024-04-21T23:06:25.212244+0000","last_unstale":"2024-04-21T23:08:03.878831+0000","last_undegraded":"2024-04-21T23:08:03.878831+0000","last_fullsized":"2024-04-21T23:08:03.878831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:17:25.287718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":155,"num_read_kb":140,"num_write":31,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.446736+0000","last_change":"2024-04-21T23:06:25.993666+0000","last_active":"2024-04-21T23:07:24.446736+0000","last_peered":"2024-04-21T23:07:24.446736+0000","last_clean":"2024-04-21T23:07:24.446736+0000","last_became_active":"2024-04-21T23:06:25.993330+0000","last_became_peered":"2024-04-21T23:06:25.993330+0000","last_unstale":"2024-04-21T23:07:24.446736+0000","last_undegraded":"2024-04-21T23:07:24.446736+0000","last_fullsized":"2024-04-21T23:07:24.446736+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:11:00.109782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.454813+0000","last_change":"2024-04-21T23:06:30.265807+0000","last_active":"2024-04-21T23:07:24.454813+0000","last_peered":"2024-04-21T23:07:24.454813+0000","last_clean":"2024-04-21T23:07:24.454813+0000","last_became_active":"2024-04-21T23:06:30.265116+0000","last_became_peered":"2024-04-21T23:06:30.265116+0000","last_unstale":"2024-04-21T23:07:24.454813+0000","last_undegraded":"2024-04-21T23:07:24.454813+0000","last_fullsized":"2024-04-21T23:07:24.454813+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:14:04.178281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451692+0000","last_change":"2024-04-21T23:06:25.208319+0000","last_active":"2024-04-21T23:07:24.451692+0000","last_peered":"2024-04-21T23:07:24.451692+0000","last_clean":"2024-04-21T23:07:24.451692+0000","last_became_active":"2024-04-21T23:06:25.208025+0000","last_became_peered":"2024-04-21T23:06:25.208025+0000","last_unstale":"2024-04-21T23:07:24.451692+0000","last_undegraded":"2024-04-21T23:07:24.451692+0000","last_fullsized":"2024-04-21T23:07:24.451692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:14.723170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450822+0000","last_change":"2024-04-21T23:06:25.993604+0000","last_active":"2024-04-21T23:07:24.450822+0000","last_peered":"2024-04-21T23:07:24.450822+0000","last_clean":"2024-04-21T23:07:24.450822+0000","last_became_active":"2024-04-21T23:06:25.993217+0000","last_became_peered":"2024-04-21T23:06:25.993217+0000","last_unstale":"2024-04-21T23:07:24.450822+0000","last_undegraded":"2024-04-21T23:07:24.450822+0000","last_fullsized":"2024-04-21T23:07:24.450822+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:23:00.435688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.448793+0000","last_change":"2024-04-21T23:06:30.270991+0000","last_active":"2024-04-21T23:07:24.448793+0000","last_peered":"2024-04-21T23:07:24.448793+0000","last_clean":"2024-04-21T23:07:24.448793+0000","last_became_active":"2024-04-21T23:06:30.270511+0000","last_became_peered":"2024-04-21T23:06:30.270511+0000","last_unstale":"2024-04-21T23:07:24.448793+0000","last_undegraded":"2024-04-21T23:07:24.448793+0000","last_fullsized":"2024-04-21T23:07:24.448793+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:04.265855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.453270+0000","last_change":"2024-04-21T23:06:25.211787+0000","last_active":"2024-04-21T23:07:24.453270+0000","last_peered":"2024-04-21T23:07:24.453270+0000","last_clean":"2024-04-21T23:07:24.453270+0000","last_became_active":"2024-04-21T23:06:25.211473+0000","last_became_peered":"2024-04-21T23:06:25.211473+0000","last_unstale":"2024-04-21T23:07:24.453270+0000","last_undegraded":"2024-04-21T23:07:24.453270+0000","last_fullsized":"2024-04-21T23:07:24.453270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:58:51.364974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450870+0000","last_change":"2024-04-21T23:06:25.990681+0000","last_active":"2024-04-21T23:07:24.450870+0000","last_peered":"2024-04-21T23:07:24.450870+0000","last_clean":"2024-04-21T23:07:24.450870+0000","last_became_active":"2024-04-21T23:06:25.990094+0000","last_became_peered":"2024-04-21T23:06:25.990094+0000","last_unstale":"2024-04-21T23:07:24.450870+0000","last_undegraded":"2024-04-21T23:07:24.450870+0000","last_fullsized":"2024-04-21T23:07:24.450870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:15:26.907912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447564+0000","last_change":"2024-04-21T23:06:30.271352+0000","last_active":"2024-04-21T23:07:24.447564+0000","last_peered":"2024-04-21T23:07:24.447564+0000","last_clean":"2024-04-21T23:07:24.447564+0000","last_became_active":"2024-04-21T23:06:30.270601+0000","last_became_peered":"2024-04-21T23:06:30.270601+0000","last_unstale":"2024-04-21T23:07:24.447564+0000","last_undegraded":"2024-04-21T23:07:24.447564+0000","last_fullsized":"2024-04-21T23:07:24.447564+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:37.266987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451040+0000","last_change":"2024-04-21T23:06:25.213288+0000","last_active":"2024-04-21T23:07:24.451040+0000","last_peered":"2024-04-21T23:07:24.451040+0000","last_clean":"2024-04-21T23:07:24.451040+0000","last_became_active":"2024-04-21T23:06:25.212936+0000","last_became_peered":"2024-04-21T23:06:25.212936+0000","last_unstale":"2024-04-21T23:07:24.451040+0000","last_undegraded":"2024-04-21T23:07:24.451040+0000","last_fullsized":"2024-04-21T23:07:24.451040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:35.303093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450871+0000","last_change":"2024-04-21T23:06:25.998485+0000","last_active":"2024-04-21T23:07:24.450871+0000","last_peered":"2024-04-21T23:07:24.450871+0000","last_clean":"2024-04-21T23:07:24.450871+0000","last_became_active":"2024-04-21T23:06:25.998184+0000","last_became_peered":"2024-04-21T23:06:25.998184+0000","last_unstale":"2024-04-21T23:07:24.450871+0000","last_undegraded":"2024-04-21T23:07:24.450871+0000","last_fullsized":"2024-04-21T23:07:24.450871+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:35:26.613636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451711+0000","last_change":"2024-04-21T23:06:30.270089+0000","last_active":"2024-04-21T23:07:24.451711+0000","last_peered":"2024-04-21T23:07:24.451711+0000","last_clean":"2024-04-21T23:07:24.451711+0000","last_became_active":"2024-04-21T23:06:30.269695+0000","last_became_peered":"2024-04-21T23:06:30.269695+0000","last_unstale":"2024-04-21T23:07:24.451711+0000","last_undegraded":"2024-04-21T23:07:24.451711+0000","last_fullsized":"2024-04-21T23:07:24.451711+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:42.718580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.446128+0000","last_change":"2024-04-21T23:06:25.218247+0000","last_active":"2024-04-21T23:07:24.446128+0000","last_peered":"2024-04-21T23:07:24.446128+0000","last_clean":"2024-04-21T23:07:24.446128+0000","last_became_active":"2024-04-21T23:06:25.217924+0000","last_became_peered":"2024-04-21T23:06:25.217924+0000","last_unstale":"2024-04-21T23:07:24.446128+0000","last_undegraded":"2024-04-21T23:07:24.446128+0000","last_fullsized":"2024-04-21T23:07:24.446128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:27:12.155597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452772+0000","last_change":"2024-04-21T23:06:25.994109+0000","last_active":"2024-04-21T23:07:24.452772+0000","last_peered":"2024-04-21T23:07:24.452772+0000","last_clean":"2024-04-21T23:07:24.452772+0000","last_became_active":"2024-04-21T23:06:25.993785+0000","last_became_peered":"2024-04-21T23:06:25.993785+0000","last_unstale":"2024-04-21T23:07:24.452772+0000","last_undegraded":"2024-04-21T23:07:24.452772+0000","last_fullsized":"2024-04-21T23:07:24.452772+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:12.629364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452662+0000","last_change":"2024-04-21T23:06:30.265444+0000","last_active":"2024-04-21T23:07:24.452662+0000","last_peered":"2024-04-21T23:07:24.452662+0000","last_clean":"2024-04-21T23:07:24.452662+0000","last_became_active":"2024-04-21T23:06:30.264971+0000","last_became_peered":"2024-04-21T23:06:30.264971+0000","last_unstale":"2024-04-21T23:07:24.452662+0000","last_undegraded":"2024-04-21T23:07:24.452662+0000","last_fullsized":"2024-04-21T23:07:24.452662+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:03.596631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.465783+0000","last_change":"2024-04-21T23:07:24.465377+0000","last_active":"2024-04-21T23:07:24.465783+0000","last_peered":"2024-04-21T23:07:24.465783+0000","last_clean":"2024-04-21T23:07:24.465783+0000","last_became_active":"2024-04-21T23:07:24.454409+0000","last_became_peered":"2024-04-21T23:07:24.454409+0000","last_unstale":"2024-04-21T23:07:24.465783+0000","last_undegraded":"2024-04-21T23:07:24.465783+0000","last_fullsized":"2024-04-21T23:07:24.465783+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:06:22.836124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.448958+0000","last_change":"2024-04-21T23:06:25.990193+0000","last_active":"2024-04-21T23:07:24.448958+0000","last_peered":"2024-04-21T23:07:24.448958+0000","last_clean":"2024-04-21T23:07:24.448958+0000","last_became_active":"2024-04-21T23:06:25.989842+0000","last_became_peered":"2024-04-21T23:06:25.989842+0000","last_unstale":"2024-04-21T23:07:24.448958+0000","last_undegraded":"2024-04-21T23:07:24.448958+0000","last_fullsized":"2024-04-21T23:07:24.448958+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:54:53.421257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.817262+0000","last_change":"2024-04-21T23:07:24.817262+0000","last_active":"2024-04-21T23:07:24.817262+0000","last_peered":"2024-04-21T23:07:24.817262+0000","last_clean":"2024-04-21T23:07:24.817262+0000","last_became_active":"2024-04-21T23:07:24.816668+0000","last_became_peered":"2024-04-21T23:07:24.816668+0000","last_unstale":"2024-04-21T23:07:24.817262+0000","last_undegraded":"2024-04-21T23:07:24.817262+0000","last_fullsized":"2024-04-21T23:07:24.817262+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:25.302151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447428+0000","last_change":"2024-04-21T23:06:25.218208+0000","last_active":"2024-04-21T23:07:24.447428+0000","last_peered":"2024-04-21T23:07:24.447428+0000","last_clean":"2024-04-21T23:07:24.447428+0000","last_became_active":"2024-04-21T23:06:25.217889+0000","last_became_peered":"2024-04-21T23:06:25.217889+0000","last_unstale":"2024-04-21T23:07:24.447428+0000","last_undegraded":"2024-04-21T23:07:24.447428+0000","last_fullsized":"2024-04-21T23:07:24.447428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:00:08.775682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447991+0000","last_change":"2024-04-21T23:06:25.987615+0000","last_active":"2024-04-21T23:07:24.447991+0000","last_peered":"2024-04-21T23:07:24.447991+0000","last_clean":"2024-04-21T23:07:24.447991+0000","last_became_active":"2024-04-21T23:06:25.987305+0000","last_became_peered":"2024-04-21T23:06:25.987305+0000","last_unstale":"2024-04-21T23:07:24.447991+0000","last_undegraded":"2024-04-21T23:07:24.447991+0000","last_fullsized":"2024-04-21T23:07:24.447991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:23:49.386998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450947+0000","last_change":"2024-04-21T23:06:30.268157+0000","last_active":"2024-04-21T23:07:24.450947+0000","last_peered":"2024-04-21T23:07:24.450947+0000","last_clean":"2024-04-21T23:07:24.450947+0000","last_became_active":"2024-04-21T23:06:30.267732+0000","last_became_peered":"2024-04-21T23:06:30.267732+0000","last_unstale":"2024-04-21T23:07:24.450947+0000","last_undegraded":"2024-04-21T23:07:24.450947+0000","last_fullsized":"2024-04-21T23:07:24.450947+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:29:24.250271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450585+0000","last_change":"2024-04-21T23:06:25.215401+0000","last_active":"2024-04-21T23:07:24.450585+0000","last_peered":"2024-04-21T23:07:24.450585+0000","last_clean":"2024-04-21T23:07:24.450585+0000","last_became_active":"2024-04-21T23:06:25.215067+0000","last_became_peered":"2024-04-21T23:06:25.215067+0000","last_unstale":"2024-04-21T23:07:24.450585+0000","last_undegraded":"2024-04-21T23:07:24.450585+0000","last_fullsized":"2024-04-21T23:07:24.450585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:16:58.977797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447748+0000","last_change":"2024-04-21T23:06:25.996421+0000","last_active":"2024-04-21T23:07:24.447748+0000","last_peered":"2024-04-21T23:07:24.447748+0000","last_clean":"2024-04-21T23:07:24.447748+0000","last_became_active":"2024-04-21T23:06:25.996102+0000","last_became_peered":"2024-04-21T23:06:25.996102+0000","last_unstale":"2024-04-21T23:07:24.447748+0000","last_undegraded":"2024-04-21T23:07:24.447748+0000","last_fullsized":"2024-04-21T23:07:24.447748+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:06:42.630182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451425+0000","last_change":"2024-04-21T23:06:30.404509+0000","last_active":"2024-04-21T23:07:24.451425+0000","last_peered":"2024-04-21T23:07:24.451425+0000","last_clean":"2024-04-21T23:07:24.451425+0000","last_became_active":"2024-04-21T23:06:30.403987+0000","last_became_peered":"2024-04-21T23:06:30.403987+0000","last_unstale":"2024-04-21T23:07:24.451425+0000","last_undegraded":"2024-04-21T23:07:24.451425+0000","last_fullsized":"2024-04-21T23:07:24.451425+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:12:28.412117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452337+0000","last_change":"2024-04-21T23:06:25.994076+0000","last_active":"2024-04-21T23:07:24.452337+0000","last_peered":"2024-04-21T23:07:24.452337+0000","last_clean":"2024-04-21T23:07:24.452337+0000","last_became_active":"2024-04-21T23:06:25.993747+0000","last_became_peered":"2024-04-21T23:06:25.993747+0000","last_unstale":"2024-04-21T23:07:24.452337+0000","last_undegraded":"2024-04-21T23:07:24.452337+0000","last_fullsized":"2024-04-21T23:07:24.452337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:17.771165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450447+0000","last_change":"2024-04-21T23:06:25.209756+0000","last_active":"2024-04-21T23:07:24.450447+0000","last_peered":"2024-04-21T23:07:24.450447+0000","last_clean":"2024-04-21T23:07:24.450447+0000","last_became_active":"2024-04-21T23:06:25.209445+0000","last_became_peered":"2024-04-21T23:06:25.209445+0000","last_unstale":"2024-04-21T23:07:24.450447+0000","last_undegraded":"2024-04-21T23:07:24.450447+0000","last_fullsized":"2024-04-21T23:07:24.450447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:58:49.453047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451866+0000","last_change":"2024-04-21T23:06:30.261565+0000","last_active":"2024-04-21T23:07:24.451866+0000","last_peered":"2024-04-21T23:07:24.451866+0000","last_clean":"2024-04-21T23:07:24.451866+0000","last_became_active":"2024-04-21T23:06:30.261095+0000","last_became_peered":"2024-04-21T23:06:30.261095+0000","last_unstale":"2024-04-21T23:07:24.451866+0000","last_undegraded":"2024-04-21T23:07:24.451866+0000","last_fullsized":"2024-04-21T23:07:24.451866+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:37:23.174152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815182+0000","last_change":"2024-04-21T23:06:25.981402+0000","last_active":"2024-04-21T23:07:24.815182+0000","last_peered":"2024-04-21T23:07:24.815182+0000","last_clean":"2024-04-21T23:07:24.815182+0000","last_became_active":"2024-04-21T23:06:25.981062+0000","last_became_peered":"2024-04-21T23:06:25.981062+0000","last_unstale":"2024-04-21T23:07:24.815182+0000","last_undegraded":"2024-04-21T23:07:24.815182+0000","last_fullsized":"2024-04-21T23:07:24.815182+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:32:32.556123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451897+0000","last_change":"2024-04-21T23:06:25.208968+0000","last_active":"2024-04-21T23:07:24.451897+0000","last_peered":"2024-04-21T23:07:24.451897+0000","last_clean":"2024-04-21T23:07:24.451897+0000","last_became_active":"2024-04-21T23:06:25.208606+0000","last_became_peered":"2024-04-21T23:06:25.208606+0000","last_unstale":"2024-04-21T23:07:24.451897+0000","last_undegraded":"2024-04-21T23:07:24.451897+0000","last_fullsized":"2024-04-21T23:07:24.451897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:31:46.147119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":363,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450577+0000","last_change":"2024-04-21T23:05:13.077846+0000","last_active":"2024-04-21T23:07:24.450577+0000","last_peered":"2024-04-21T23:07:24.450577+0000","last_clean":"2024-04-21T23:07:24.450577+0000","last_became_active":"2024-04-21T23:05:13.077436+0000","last_became_peered":"2024-04-21T23:05:13.077436+0000","last_unstale":"2024-04-21T23:07:24.450577+0000","last_undegraded":"2024-04-21T23:07:24.450577+0000","last_fullsized":"2024-04-21T23:07:24.450577+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:01:47.619212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:01:47.619212+0000","last_clean_scrub_stamp":"2024-04-21T23:01:47.619212+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:12:09.826839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.454201+0000","last_change":"2024-04-21T23:06:30.405121+0000","last_active":"2024-04-21T23:07:24.454201+0000","last_peered":"2024-04-21T23:07:24.454201+0000","last_clean":"2024-04-21T23:07:24.454201+0000","last_became_active":"2024-04-21T23:06:30.404612+0000","last_became_peered":"2024-04-21T23:06:30.404612+0000","last_unstale":"2024-04-21T23:07:24.454201+0000","last_undegraded":"2024-04-21T23:07:24.454201+0000","last_fullsized":"2024-04-21T23:07:24.454201+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:59:01.737337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450475+0000","last_change":"2024-04-21T23:06:25.997879+0000","last_active":"2024-04-21T23:07:24.450475+0000","last_peered":"2024-04-21T23:07:24.450475+0000","last_clean":"2024-04-21T23:07:24.450475+0000","last_became_active":"2024-04-21T23:06:25.997507+0000","last_became_peered":"2024-04-21T23:06:25.997507+0000","last_unstale":"2024-04-21T23:07:24.450475+0000","last_undegraded":"2024-04-21T23:07:24.450475+0000","last_fullsized":"2024-04-21T23:07:24.450475+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:13:16.044456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450760+0000","last_change":"2024-04-21T23:06:25.209714+0000","last_active":"2024-04-21T23:07:24.450760+0000","last_peered":"2024-04-21T23:07:24.450760+0000","last_clean":"2024-04-21T23:07:24.450760+0000","last_became_active":"2024-04-21T23:06:25.209373+0000","last_became_peered":"2024-04-21T23:06:25.209373+0000","last_unstale":"2024-04-21T23:07:24.450760+0000","last_undegraded":"2024-04-21T23:07:24.450760+0000","last_fullsized":"2024-04-21T23:07:24.450760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:21.654143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450079+0000","last_change":"2024-04-21T23:06:30.264873+0000","last_active":"2024-04-21T23:07:24.450079+0000","last_peered":"2024-04-21T23:07:24.450079+0000","last_clean":"2024-04-21T23:07:24.450079+0000","last_became_active":"2024-04-21T23:06:30.264318+0000","last_became_peered":"2024-04-21T23:06:30.264318+0000","last_unstale":"2024-04-21T23:07:24.450079+0000","last_undegraded":"2024-04-21T23:07:24.450079+0000","last_fullsized":"2024-04-21T23:07:24.450079+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:01:22.558782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.446874+0000","last_change":"2024-04-21T23:06:25.218767+0000","last_active":"2024-04-21T23:07:24.446874+0000","last_peered":"2024-04-21T23:07:24.446874+0000","last_clean":"2024-04-21T23:07:24.446874+0000","last_became_active":"2024-04-21T23:06:25.218278+0000","last_became_peered":"2024-04-21T23:06:25.218278+0000","last_unstale":"2024-04-21T23:07:24.446874+0000","last_undegraded":"2024-04-21T23:07:24.446874+0000","last_fullsized":"2024-04-21T23:07:24.446874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:03:05.063515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451136+0000","last_change":"2024-04-21T23:06:25.968551+0000","last_active":"2024-04-21T23:07:24.451136+0000","last_peered":"2024-04-21T23:07:24.451136+0000","last_clean":"2024-04-21T23:07:24.451136+0000","last_became_active":"2024-04-21T23:06:25.968167+0000","last_became_peered":"2024-04-21T23:06:25.968167+0000","last_unstale":"2024-04-21T23:07:24.451136+0000","last_undegraded":"2024-04-21T23:07:24.451136+0000","last_fullsized":"2024-04-21T23:07:24.451136+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:08.920677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.814872+0000","last_change":"2024-04-21T23:06:30.404811+0000","last_active":"2024-04-21T23:07:24.814872+0000","last_peered":"2024-04-21T23:07:24.814872+0000","last_clean":"2024-04-21T23:07:24.814872+0000","last_became_active":"2024-04-21T23:06:30.404375+0000","last_became_peered":"2024-04-21T23:06:30.404375+0000","last_unstale":"2024-04-21T23:07:24.814872+0000","last_undegraded":"2024-04-21T23:07:24.814872+0000","last_fullsized":"2024-04-21T23:07:24.814872+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:10:15.688917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'6","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447940+0000","last_change":"2024-04-21T23:06:25.204371+0000","last_active":"2024-04-21T23:07:24.447940+0000","last_peered":"2024-04-21T23:07:24.447940+0000","last_clean":"2024-04-21T23:07:24.447940+0000","last_became_active":"2024-04-21T23:06:25.204025+0000","last_became_peered":"2024-04-21T23:06:25.204025+0000","last_unstale":"2024-04-21T23:07:24.447940+0000","last_undegraded":"2024-04-21T23:07:24.447940+0000","last_fullsized":"2024-04-21T23:07:24.447940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:34:31.071013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815355+0000","last_change":"2024-04-21T23:06:25.990800+0000","last_active":"2024-04-21T23:07:24.815355+0000","last_peered":"2024-04-21T23:07:24.815355+0000","last_clean":"2024-04-21T23:07:24.815355+0000","last_became_active":"2024-04-21T23:06:25.990477+0000","last_became_peered":"2024-04-21T23:06:25.990477+0000","last_unstale":"2024-04-21T23:07:24.815355+0000","last_undegraded":"2024-04-21T23:07:24.815355+0000","last_fullsized":"2024-04-21T23:07:24.815355+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:55:33.282303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450510+0000","last_change":"2024-04-21T23:06:30.260090+0000","last_active":"2024-04-21T23:07:24.450510+0000","last_peered":"2024-04-21T23:07:24.450510+0000","last_clean":"2024-04-21T23:07:24.450510+0000","last_became_active":"2024-04-21T23:06:30.259609+0000","last_became_peered":"2024-04-21T23:06:30.259609+0000","last_unstale":"2024-04-21T23:07:24.450510+0000","last_undegraded":"2024-04-21T23:07:24.450510+0000","last_fullsized":"2024-04-21T23:07:24.450510+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:11:17.933683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452284+0000","last_change":"2024-04-21T23:06:25.215446+0000","last_active":"2024-04-21T23:07:24.452284+0000","last_peered":"2024-04-21T23:07:24.452284+0000","last_clean":"2024-04-21T23:07:24.452284+0000","last_became_active":"2024-04-21T23:06:25.214773+0000","last_became_peered":"2024-04-21T23:06:25.214773+0000","last_unstale":"2024-04-21T23:07:24.452284+0000","last_undegraded":"2024-04-21T23:07:24.452284+0000","last_fullsized":"2024-04-21T23:07:24.452284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:25:01.090699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451547+0000","last_change":"2024-04-21T23:06:25.986640+0000","last_active":"2024-04-21T23:07:24.451547+0000","last_peered":"2024-04-21T23:07:24.451547+0000","last_clean":"2024-04-21T23:07:24.451547+0000","last_became_active":"2024-04-21T23:06:25.986285+0000","last_became_peered":"2024-04-21T23:06:25.986285+0000","last_unstale":"2024-04-21T23:07:24.451547+0000","last_undegraded":"2024-04-21T23:07:24.451547+0000","last_fullsized":"2024-04-21T23:07:24.451547+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:01:05.498199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.458291+0000","last_change":"2024-04-21T23:07:24.458291+0000","last_active":"2024-04-21T23:07:24.458291+0000","last_peered":"2024-04-21T23:07:24.458291+0000","last_clean":"2024-04-21T23:07:24.458291+0000","last_became_active":"2024-04-21T23:07:24.457677+0000","last_became_peered":"2024-04-21T23:07:24.457677+0000","last_unstale":"2024-04-21T23:07:24.458291+0000","last_undegraded":"2024-04-21T23:07:24.458291+0000","last_fullsized":"2024-04-21T23:07:24.458291+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:20:30.708901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,11],"acting":[4,8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815415+0000","last_change":"2024-04-21T23:06:25.217298+0000","last_active":"2024-04-21T23:07:24.815415+0000","last_peered":"2024-04-21T23:07:24.815415+0000","last_clean":"2024-04-21T23:07:24.815415+0000","last_became_active":"2024-04-21T23:06:25.216965+0000","last_became_peered":"2024-04-21T23:06:25.216965+0000","last_unstale":"2024-04-21T23:07:24.815415+0000","last_undegraded":"2024-04-21T23:07:24.815415+0000","last_fullsized":"2024-04-21T23:07:24.815415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:56:38.543697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.446128+0000","last_change":"2024-04-21T23:06:25.981400+0000","last_active":"2024-04-21T23:07:24.446128+0000","last_peered":"2024-04-21T23:07:24.446128+0000","last_clean":"2024-04-21T23:07:24.446128+0000","last_became_active":"2024-04-21T23:06:25.981118+0000","last_became_peered":"2024-04-21T23:06:25.981118+0000","last_unstale":"2024-04-21T23:07:24.446128+0000","last_undegraded":"2024-04-21T23:07:24.446128+0000","last_fullsized":"2024-04-21T23:07:24.446128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:20:59.631062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447715+0000","last_change":"2024-04-21T23:06:30.270351+0000","last_active":"2024-04-21T23:07:24.447715+0000","last_peered":"2024-04-21T23:07:24.447715+0000","last_clean":"2024-04-21T23:07:24.447715+0000","last_became_active":"2024-04-21T23:06:30.269659+0000","last_became_peered":"2024-04-21T23:06:30.269659+0000","last_unstale":"2024-04-21T23:07:24.447715+0000","last_undegraded":"2024-04-21T23:07:24.447715+0000","last_fullsized":"2024-04-21T23:07:24.447715+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:19:14.132191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.814984+0000","last_change":"2024-04-21T23:06:25.212787+0000","last_active":"2024-04-21T23:07:24.814984+0000","last_peered":"2024-04-21T23:07:24.814984+0000","last_clean":"2024-04-21T23:07:24.814984+0000","last_became_active":"2024-04-21T23:06:25.212423+0000","last_became_peered":"2024-04-21T23:06:25.212423+0000","last_unstale":"2024-04-21T23:07:24.814984+0000","last_undegraded":"2024-04-21T23:07:24.814984+0000","last_fullsized":"2024-04-21T23:07:24.814984+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:45.875664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.448604+0000","last_change":"2024-04-21T23:06:25.978205+0000","last_active":"2024-04-21T23:07:24.448604+0000","last_peered":"2024-04-21T23:07:24.448604+0000","last_clean":"2024-04-21T23:07:24.448604+0000","last_became_active":"2024-04-21T23:06:25.977876+0000","last_became_peered":"2024-04-21T23:06:25.977876+0000","last_unstale":"2024-04-21T23:07:24.448604+0000","last_undegraded":"2024-04-21T23:07:24.448604+0000","last_fullsized":"2024-04-21T23:07:24.448604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:43:29.280255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451607+0000","last_change":"2024-04-21T23:06:30.271928+0000","last_active":"2024-04-21T23:07:24.451607+0000","last_peered":"2024-04-21T23:07:24.451607+0000","last_clean":"2024-04-21T23:07:24.451607+0000","last_became_active":"2024-04-21T23:06:30.271321+0000","last_became_peered":"2024-04-21T23:06:30.271321+0000","last_unstale":"2024-04-21T23:07:24.451607+0000","last_undegraded":"2024-04-21T23:07:24.451607+0000","last_fullsized":"2024-04-21T23:07:24.451607+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:22:03.343620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450814+0000","last_change":"2024-04-21T23:06:25.204175+0000","last_active":"2024-04-21T23:07:24.450814+0000","last_peered":"2024-04-21T23:07:24.450814+0000","last_clean":"2024-04-21T23:07:24.450814+0000","last_became_active":"2024-04-21T23:06:25.203826+0000","last_became_peered":"2024-04-21T23:06:25.203826+0000","last_unstale":"2024-04-21T23:07:24.450814+0000","last_undegraded":"2024-04-21T23:07:24.450814+0000","last_fullsized":"2024-04-21T23:07:24.450814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:28:53.402395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815164+0000","last_change":"2024-04-21T23:06:25.990559+0000","last_active":"2024-04-21T23:07:24.815164+0000","last_peered":"2024-04-21T23:07:24.815164+0000","last_clean":"2024-04-21T23:07:24.815164+0000","last_became_active":"2024-04-21T23:06:25.990239+0000","last_became_peered":"2024-04-21T23:06:25.990239+0000","last_unstale":"2024-04-21T23:07:24.815164+0000","last_undegraded":"2024-04-21T23:07:24.815164+0000","last_fullsized":"2024-04-21T23:07:24.815164+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:49:26.891190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452205+0000","last_change":"2024-04-21T23:06:30.406584+0000","last_active":"2024-04-21T23:07:24.452205+0000","last_peered":"2024-04-21T23:07:24.452205+0000","last_clean":"2024-04-21T23:07:24.452205+0000","last_became_active":"2024-04-21T23:06:30.404576+0000","last_became_peered":"2024-04-21T23:06:30.404576+0000","last_unstale":"2024-04-21T23:07:24.452205+0000","last_undegraded":"2024-04-21T23:07:24.452205+0000","last_fullsized":"2024-04-21T23:07:24.452205+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:56.047199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.460921+0000","last_change":"2024-04-21T23:07:24.460375+0000","last_active":"2024-04-21T23:07:24.460921+0000","last_peered":"2024-04-21T23:07:24.460921+0000","last_clean":"2024-04-21T23:07:24.460921+0000","last_became_active":"2024-04-21T23:07:24.453873+0000","last_became_peered":"2024-04-21T23:07:24.453873+0000","last_unstale":"2024-04-21T23:07:24.460921+0000","last_undegraded":"2024-04-21T23:07:24.460921+0000","last_fullsized":"2024-04-21T23:07:24.460921+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:27.709414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450647+0000","last_change":"2024-04-21T23:06:25.988925+0000","last_active":"2024-04-21T23:07:24.450647+0000","last_peered":"2024-04-21T23:07:24.450647+0000","last_clean":"2024-04-21T23:07:24.450647+0000","last_became_active":"2024-04-21T23:06:25.988480+0000","last_became_peered":"2024-04-21T23:06:25.988480+0000","last_unstale":"2024-04-21T23:07:24.450647+0000","last_undegraded":"2024-04-21T23:07:24.450647+0000","last_fullsized":"2024-04-21T23:07:24.450647+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:32:02.867858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815558+0000","last_change":"2024-04-21T23:06:30.260841+0000","last_active":"2024-04-21T23:07:24.815558+0000","last_peered":"2024-04-21T23:07:24.815558+0000","last_clean":"2024-04-21T23:07:24.815558+0000","last_became_active":"2024-04-21T23:06:30.260376+0000","last_became_peered":"2024-04-21T23:06:30.260376+0000","last_unstale":"2024-04-21T23:07:24.815558+0000","last_undegraded":"2024-04-21T23:07:24.815558+0000","last_fullsized":"2024-04-21T23:07:24.815558+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:24:57.429169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452096+0000","last_change":"2024-04-21T23:06:25.217920+0000","last_active":"2024-04-21T23:07:24.452096+0000","last_peered":"2024-04-21T23:07:24.452096+0000","last_clean":"2024-04-21T23:07:24.452096+0000","last_became_active":"2024-04-21T23:06:25.217379+0000","last_became_peered":"2024-04-21T23:06:25.217379+0000","last_unstale":"2024-04-21T23:07:24.452096+0000","last_undegraded":"2024-04-21T23:07:24.452096+0000","last_fullsized":"2024-04-21T23:07:24.452096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:09.570463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452106+0000","last_change":"2024-04-21T23:06:25.993533+0000","last_active":"2024-04-21T23:07:24.452106+0000","last_peered":"2024-04-21T23:07:24.452106+0000","last_clean":"2024-04-21T23:07:24.452106+0000","last_became_active":"2024-04-21T23:06:25.989739+0000","last_became_peered":"2024-04-21T23:06:25.989739+0000","last_unstale":"2024-04-21T23:07:24.452106+0000","last_undegraded":"2024-04-21T23:07:24.452106+0000","last_fullsized":"2024-04-21T23:07:24.452106+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:24:05.594026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451828+0000","last_change":"2024-04-21T23:06:30.276121+0000","last_active":"2024-04-21T23:07:24.451828+0000","last_peered":"2024-04-21T23:07:24.451828+0000","last_clean":"2024-04-21T23:07:24.451828+0000","last_became_active":"2024-04-21T23:06:30.275650+0000","last_became_peered":"2024-04-21T23:06:30.275650+0000","last_unstale":"2024-04-21T23:07:24.451828+0000","last_undegraded":"2024-04-21T23:07:24.451828+0000","last_fullsized":"2024-04-21T23:07:24.451828+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:00:19.559482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.454084+0000","last_change":"2024-04-21T23:06:25.207079+0000","last_active":"2024-04-21T23:07:24.454084+0000","last_peered":"2024-04-21T23:07:24.454084+0000","last_clean":"2024-04-21T23:07:24.454084+0000","last_became_active":"2024-04-21T23:06:25.206724+0000","last_became_peered":"2024-04-21T23:06:25.206724+0000","last_unstale":"2024-04-21T23:07:24.454084+0000","last_undegraded":"2024-04-21T23:07:24.454084+0000","last_fullsized":"2024-04-21T23:07:24.454084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:27.221282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450876+0000","last_change":"2024-04-21T23:06:25.988144+0000","last_active":"2024-04-21T23:07:24.450876+0000","last_peered":"2024-04-21T23:07:24.450876+0000","last_clean":"2024-04-21T23:07:24.450876+0000","last_became_active":"2024-04-21T23:06:25.987813+0000","last_became_peered":"2024-04-21T23:06:25.987813+0000","last_unstale":"2024-04-21T23:07:24.450876+0000","last_undegraded":"2024-04-21T23:07:24.450876+0000","last_fullsized":"2024-04-21T23:07:24.450876+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:23.733514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451829+0000","last_change":"2024-04-21T23:06:30.261548+0000","last_active":"2024-04-21T23:07:24.451829+0000","last_peered":"2024-04-21T23:07:24.451829+0000","last_clean":"2024-04-21T23:07:24.451829+0000","last_became_active":"2024-04-21T23:06:30.261152+0000","last_became_peered":"2024-04-21T23:06:30.261152+0000","last_unstale":"2024-04-21T23:07:24.451829+0000","last_undegraded":"2024-04-21T23:07:24.451829+0000","last_fullsized":"2024-04-21T23:07:24.451829+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:00:16.644253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451184+0000","last_change":"2024-04-21T23:06:25.208749+0000","last_active":"2024-04-21T23:07:24.451184+0000","last_peered":"2024-04-21T23:07:24.451184+0000","last_clean":"2024-04-21T23:07:24.451184+0000","last_became_active":"2024-04-21T23:06:25.201511+0000","last_became_peered":"2024-04-21T23:06:25.201511+0000","last_unstale":"2024-04-21T23:07:24.451184+0000","last_undegraded":"2024-04-21T23:07:24.451184+0000","last_fullsized":"2024-04-21T23:07:24.451184+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:41.928835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451852+0000","last_change":"2024-04-21T23:06:25.996041+0000","last_active":"2024-04-21T23:07:24.451852+0000","last_peered":"2024-04-21T23:07:24.451852+0000","last_clean":"2024-04-21T23:07:24.451852+0000","last_became_active":"2024-04-21T23:06:25.994482+0000","last_became_peered":"2024-04-21T23:06:25.994482+0000","last_unstale":"2024-04-21T23:07:24.451852+0000","last_undegraded":"2024-04-21T23:07:24.451852+0000","last_fullsized":"2024-04-21T23:07:24.451852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:15:46.792618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.448677+0000","last_change":"2024-04-21T23:06:25.974061+0000","last_active":"2024-04-21T23:07:24.448677+0000","last_peered":"2024-04-21T23:07:24.448677+0000","last_clean":"2024-04-21T23:07:24.448677+0000","last_became_active":"2024-04-21T23:06:25.973708+0000","last_became_peered":"2024-04-21T23:06:25.973708+0000","last_unstale":"2024-04-21T23:07:24.448677+0000","last_undegraded":"2024-04-21T23:07:24.448677+0000","last_fullsized":"2024-04-21T23:07:24.448677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:58.702947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450989+0000","last_change":"2024-04-21T23:06:25.988412+0000","last_active":"2024-04-21T23:07:24.450989+0000","last_peered":"2024-04-21T23:07:24.450989+0000","last_clean":"2024-04-21T23:07:24.450989+0000","last_became_active":"2024-04-21T23:06:25.988100+0000","last_became_peered":"2024-04-21T23:06:25.988100+0000","last_unstale":"2024-04-21T23:07:24.450989+0000","last_undegraded":"2024-04-21T23:07:24.450989+0000","last_fullsized":"2024-04-21T23:07:24.450989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:16:42.626123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450695+0000","last_change":"2024-04-21T23:06:25.980144+0000","last_active":"2024-04-21T23:07:24.450695+0000","last_peered":"2024-04-21T23:07:24.450695+0000","last_clean":"2024-04-21T23:07:24.450695+0000","last_became_active":"2024-04-21T23:06:25.979760+0000","last_became_peered":"2024-04-21T23:06:25.979760+0000","last_unstale":"2024-04-21T23:07:24.450695+0000","last_undegraded":"2024-04-21T23:07:24.450695+0000","last_fullsized":"2024-04-21T23:07:24.450695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:14:32.496184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452099+0000","last_change":"2024-04-21T23:06:25.994732+0000","last_active":"2024-04-21T23:07:24.452099+0000","last_peered":"2024-04-21T23:07:24.452099+0000","last_clean":"2024-04-21T23:07:24.452099+0000","last_became_active":"2024-04-21T23:06:25.994408+0000","last_became_peered":"2024-04-21T23:06:25.994408+0000","last_unstale":"2024-04-21T23:07:24.452099+0000","last_undegraded":"2024-04-21T23:07:24.452099+0000","last_fullsized":"2024-04-21T23:07:24.452099+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:28:59.778049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.455416+0000","last_change":"2024-04-21T23:07:24.455416+0000","last_active":"2024-04-21T23:07:24.455416+0000","last_peered":"2024-04-21T23:07:24.455416+0000","last_clean":"2024-04-21T23:07:24.455416+0000","last_became_active":"2024-04-21T23:07:24.454954+0000","last_became_peered":"2024-04-21T23:07:24.454954+0000","last_unstale":"2024-04-21T23:07:24.455416+0000","last_undegraded":"2024-04-21T23:07:24.455416+0000","last_fullsized":"2024-04-21T23:07:24.455416+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:29:09.706719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.449201+0000","last_change":"2024-04-21T23:06:25.973850+0000","last_active":"2024-04-21T23:07:24.449201+0000","last_peered":"2024-04-21T23:07:24.449201+0000","last_clean":"2024-04-21T23:07:24.449201+0000","last_became_active":"2024-04-21T23:06:25.973522+0000","last_became_peered":"2024-04-21T23:06:25.973522+0000","last_unstale":"2024-04-21T23:07:24.449201+0000","last_undegraded":"2024-04-21T23:07:24.449201+0000","last_fullsized":"2024-04-21T23:07:24.449201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:43:58.417551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451068+0000","last_change":"2024-04-21T23:06:25.978114+0000","last_active":"2024-04-21T23:07:24.451068+0000","last_peered":"2024-04-21T23:07:24.451068+0000","last_clean":"2024-04-21T23:07:24.451068+0000","last_became_active":"2024-04-21T23:06:25.977115+0000","last_became_peered":"2024-04-21T23:06:25.977115+0000","last_unstale":"2024-04-21T23:07:24.451068+0000","last_undegraded":"2024-04-21T23:07:24.451068+0000","last_fullsized":"2024-04-21T23:07:24.451068+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:26:46.756490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815527+0000","last_change":"2024-04-21T23:06:25.994451+0000","last_active":"2024-04-21T23:07:24.815527+0000","last_peered":"2024-04-21T23:07:24.815527+0000","last_clean":"2024-04-21T23:07:24.815527+0000","last_became_active":"2024-04-21T23:06:25.994085+0000","last_became_peered":"2024-04-21T23:06:25.994085+0000","last_unstale":"2024-04-21T23:07:24.815527+0000","last_undegraded":"2024-04-21T23:07:24.815527+0000","last_fullsized":"2024-04-21T23:07:24.815527+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:53.652014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452244+0000","last_change":"2024-04-21T23:06:25.982536+0000","last_active":"2024-04-21T23:07:24.452244+0000","last_peered":"2024-04-21T23:07:24.452244+0000","last_clean":"2024-04-21T23:07:24.452244+0000","last_became_active":"2024-04-21T23:06:25.981318+0000","last_became_peered":"2024-04-21T23:06:25.981318+0000","last_unstale":"2024-04-21T23:07:24.452244+0000","last_undegraded":"2024-04-21T23:07:24.452244+0000","last_fullsized":"2024-04-21T23:07:24.452244+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:35.784679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815105+0000","last_change":"2024-04-21T23:06:25.981915+0000","last_active":"2024-04-21T23:07:24.815105+0000","last_peered":"2024-04-21T23:07:24.815105+0000","last_clean":"2024-04-21T23:07:24.815105+0000","last_became_active":"2024-04-21T23:06:25.981547+0000","last_became_peered":"2024-04-21T23:06:25.981547+0000","last_unstale":"2024-04-21T23:07:24.815105+0000","last_undegraded":"2024-04-21T23:07:24.815105+0000","last_fullsized":"2024-04-21T23:07:24.815105+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:03:11.154582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815749+0000","last_change":"2024-04-21T23:06:25.991687+0000","last_active":"2024-04-21T23:07:24.815749+0000","last_peered":"2024-04-21T23:07:24.815749+0000","last_clean":"2024-04-21T23:07:24.815749+0000","last_became_active":"2024-04-21T23:06:25.991248+0000","last_became_peered":"2024-04-21T23:06:25.991248+0000","last_unstale":"2024-04-21T23:07:24.815749+0000","last_undegraded":"2024-04-21T23:07:24.815749+0000","last_fullsized":"2024-04-21T23:07:24.815749+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:05:04.205660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451181+0000","last_change":"2024-04-21T23:06:25.993658+0000","last_active":"2024-04-21T23:07:24.451181+0000","last_peered":"2024-04-21T23:07:24.451181+0000","last_clean":"2024-04-21T23:07:24.451181+0000","last_became_active":"2024-04-21T23:06:25.993298+0000","last_became_peered":"2024-04-21T23:06:25.993298+0000","last_unstale":"2024-04-21T23:07:24.451181+0000","last_undegraded":"2024-04-21T23:07:24.451181+0000","last_fullsized":"2024-04-21T23:07:24.451181+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:18:30.216961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.448778+0000","last_change":"2024-04-21T23:06:25.993849+0000","last_active":"2024-04-21T23:07:24.448778+0000","last_peered":"2024-04-21T23:07:24.448778+0000","last_clean":"2024-04-21T23:07:24.448778+0000","last_became_active":"2024-04-21T23:06:25.993501+0000","last_became_peered":"2024-04-21T23:06:25.993501+0000","last_unstale":"2024-04-21T23:07:24.448778+0000","last_undegraded":"2024-04-21T23:07:24.448778+0000","last_fullsized":"2024-04-21T23:07:24.448778+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:40:10.461775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63640,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":158,"num_read_kb":143,"num_write":80,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":190920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351009,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1036288,"data_stored":506855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612647,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32852,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712300,"statfs":{"total":95995035648,"available":95961395200,"internally_reserved":0,"allocated":1708032,"data_stored":1156141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808875,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":511511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070513,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1036288,"data_stored":507415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332148,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28620,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716532,"statfs":{"total":95995035648,"available":95965728768,"internally_reserved":0,"allocated":1634304,"data_stored":1101879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495674,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":502737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1101824,"data_stored":565579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":502827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":565613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":502841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920836,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":506855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757312,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28620,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716532,"statfs":{"total":95995035648,"available":95965728768,"internally_reserved":0,"allocated":1638400,"data_stored":1097383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":63036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T23:08:06.506 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-21T23:08:06.762 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:08:06.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:06 smithi044 ceph-mon[27934]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:08:06.884 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:06 smithi044 ceph-mon[27934]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:08:06.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:06 smithi055 ceph-mon[31117]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:08:06.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:06 smithi055 ceph-mon[31117]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:08:06.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:06 smithi069 ceph-mon[31816]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:08:06.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:06 smithi069 ceph-mon[31816]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:08:08.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.497+0000 7fc665c8b700 1 -- 172.21.15.44:0/781338394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6601075e0 msgr2=0x7fc66010f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:08.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.497+0000 7fc665c8b700 1 --2- 172.21.15.44:0/781338394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6601075e0 0x7fc66010f1b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc65800b0d0 tx=0x7fc65800b3e0 comp rx=0 tx=0).stop 2024-04-21T23:08:08.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.497+0000 7fc665c8b700 1 -- 172.21.15.44:0/781338394 shutdown_connections 2024-04-21T23:08:08.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.497+0000 7fc665c8b700 1 --2- 172.21.15.44:0/781338394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6601075e0 0x7fc66010f1b0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:08.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.497+0000 7fc665c8b700 1 --2- 172.21.15.44:0/781338394 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc660102510 0x7fc6601028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:08.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.497+0000 7fc665c8b700 1 --2- 172.21.15.44:0/781338394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc660102ec0 0x7fc660106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:08.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.497+0000 7fc665c8b700 1 -- 172.21.15.44:0/781338394 >> 172.21.15.44:0/781338394 conn(0x7fc6600793e0 msgr2=0x7fc6600797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:08.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.497+0000 7fc665c8b700 1 -- 172.21.15.44:0/781338394 shutdown_connections 2024-04-21T23:08:08.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.497+0000 7fc665c8b700 1 -- 172.21.15.44:0/781338394 wait complete. 2024-04-21T23:08:08.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.498+0000 7fc665c8b700 1 Processor -- start 2024-04-21T23:08:08.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.498+0000 7fc665c8b700 1 -- start start 2024-04-21T23:08:08.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.498+0000 7fc665c8b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc660102510 0x7fc66019e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:08.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.498+0000 7fc665c8b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc660102ec0 0x7fc66019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:08.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.498+0000 7fc665c8b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6601075e0 0x7fc660198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:08.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.498+0000 7fc665c8b700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc6601125d0 con 0x7fc6601075e0 2024-04-21T23:08:08.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.498+0000 7fc665c8b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc660112450 con 0x7fc660102ec0 2024-04-21T23:08:08.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.498+0000 7fc665c8b700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc660112750 con 0x7fc660102510 2024-04-21T23:08:08.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.498+0000 7fc65ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6601075e0 0x7fc660198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:08.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.498+0000 7fc65ffff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6601075e0 0x7fc660198810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45990/0 (socket says 172.21.15.44:45990) 2024-04-21T23:08:08.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.498+0000 7fc65ffff700 1 -- 172.21.15.44:0/1266814650 learned_addr learned my addr 172.21.15.44:0/1266814650 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:08.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.498+0000 7fc65effd700 1 --2- 172.21.15.44:0/1266814650 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc660102ec0 0x7fc66019ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:08.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.498+0000 7fc65f7fe700 1 --2- 172.21.15.44:0/1266814650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc660102510 0x7fc66019e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:08.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.498+0000 7fc65ffff700 1 -- 172.21.15.44:0/1266814650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc660102510 msgr2=0x7fc66019e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:08.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.499+0000 7fc65ffff700 1 --2- 172.21.15.44:0/1266814650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc660102510 0x7fc66019e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:08.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.499+0000 7fc65ffff700 1 -- 172.21.15.44:0/1266814650 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc660102ec0 msgr2=0x7fc66019ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:08.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.499+0000 7fc65ffff700 1 --2- 172.21.15.44:0/1266814650 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc660102ec0 0x7fc66019ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:08.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.499+0000 7fc65ffff700 1 -- 172.21.15.44:0/1266814650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc660198ff0 con 0x7fc6601075e0 2024-04-21T23:08:08.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.499+0000 7fc65ffff700 1 --2- 172.21.15.44:0/1266814650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6601075e0 0x7fc660198810 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc658006070 tx=0x7fc658004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:08.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.499+0000 7fc65cff9700 1 -- 172.21.15.44:0/1266814650 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc658021070 con 0x7fc6601075e0 2024-04-21T23:08:08.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.499+0000 7fc665c8b700 1 -- 172.21.15.44:0/1266814650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc660199280 con 0x7fc6601075e0 2024-04-21T23:08:08.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.499+0000 7fc65cff9700 1 -- 172.21.15.44:0/1266814650 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc658007b80 con 0x7fc6601075e0 2024-04-21T23:08:08.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.499+0000 7fc65cff9700 1 -- 172.21.15.44:0/1266814650 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc658005330 con 0x7fc6601075e0 2024-04-21T23:08:08.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.499+0000 7fc665c8b700 1 -- 172.21.15.44:0/1266814650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6601a54f0 con 0x7fc6601075e0 2024-04-21T23:08:08.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.500+0000 7fc665c8b700 1 -- 172.21.15.44:0/1266814650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6601039e0 con 0x7fc6601075e0 2024-04-21T23:08:08.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.501+0000 7fc65cff9700 1 -- 172.21.15.44:0/1266814650 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc658024070 con 0x7fc6601075e0 2024-04-21T23:08:08.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.502+0000 7fc65cff9700 1 --2- 172.21.15.44:0/1266814650 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc6480a2cb0 0x7fc6480a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:08.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.502+0000 7fc65cff9700 1 -- 172.21.15.44:0/1266814650 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fc65800f020 con 0x7fc6601075e0 2024-04-21T23:08:08.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.502+0000 7fc65f7fe700 1 --2- 172.21.15.44:0/1266814650 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc6480a2cb0 0x7fc6480a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:08.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.502+0000 7fc65f7fe700 1 --2- 172.21.15.44:0/1266814650 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc6480a2cb0 0x7fc6480a5170 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc654005950 tx=0x7fc6540058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:08.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.506+0000 7fc65cff9700 1 -- 172.21.15.44:0/1266814650 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc65802b020 con 0x7fc6601075e0 2024-04-21T23:08:08.505 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:08 smithi044 ceph-mon[27934]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:08.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.706+0000 7fc665c8b700 1 -- 172.21.15.44:0/1266814650 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc660100930 con 0x7fc6480a2cb0 2024-04-21T23:08:08.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.716+0000 7fc65cff9700 1 -- 172.21.15.44:0/1266814650 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340295 (secure 0 0 0) 0x7fc660100930 con 0x7fc6480a2cb0 2024-04-21T23:08:08.720 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:08.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.720+0000 7fc665c8b700 1 -- 172.21.15.44:0/1266814650 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc6480a2cb0 msgr2=0x7fc6480a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:08.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.720+0000 7fc665c8b700 1 --2- 172.21.15.44:0/1266814650 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc6480a2cb0 0x7fc6480a5170 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc654005950 tx=0x7fc6540058e0 comp rx=0 tx=0).stop 2024-04-21T23:08:08.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.720+0000 7fc665c8b700 1 -- 172.21.15.44:0/1266814650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6601075e0 msgr2=0x7fc660198810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:08.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.720+0000 7fc665c8b700 1 --2- 172.21.15.44:0/1266814650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6601075e0 0x7fc660198810 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc658006070 tx=0x7fc658004d00 comp rx=0 tx=0).stop 2024-04-21T23:08:08.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.720+0000 7fc665c8b700 1 -- 172.21.15.44:0/1266814650 shutdown_connections 2024-04-21T23:08:08.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.720+0000 7fc665c8b700 1 --2- 172.21.15.44:0/1266814650 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc6480a2cb0 0x7fc6480a5170 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:08.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.720+0000 7fc665c8b700 1 --2- 172.21.15.44:0/1266814650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc6601075e0 0x7fc660198810 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:08.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.720+0000 7fc665c8b700 1 --2- 172.21.15.44:0/1266814650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc660102510 0x7fc66019e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:08.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.720+0000 7fc665c8b700 1 --2- 172.21.15.44:0/1266814650 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc660102ec0 0x7fc66019ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:08.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.720+0000 7fc665c8b700 1 -- 172.21.15.44:0/1266814650 >> 172.21.15.44:0/1266814650 conn(0x7fc6600793e0 msgr2=0x7fc660100220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:08.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.720+0000 7fc665c8b700 1 -- 172.21.15.44:0/1266814650 shutdown_connections 2024-04-21T23:08:08.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:08.720+0000 7fc665c8b700 1 -- 172.21.15.44:0/1266814650 wait complete. 2024-04-21T23:08:08.723 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-21T23:08:08.766 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:08 smithi069 ceph-mon[31816]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:08.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:08 smithi055 ceph-mon[31117]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:09.441 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-04-21T23:08:07.078801+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654242,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":263,"num_write_kb":3031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":363328,"kb_used_data":14096,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124578496,"statfs":{"total":1151940427776,"available":1151568379904,"internally_reserved":0,"allocated":14434304,"data_stored":8027636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003183"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.449965+0000","last_change":"2024-04-21T23:06:25.980091+0000","last_active":"2024-04-21T23:07:24.449965+0000","last_peered":"2024-04-21T23:07:24.449965+0000","last_clean":"2024-04-21T23:07:24.449965+0000","last_became_active":"2024-04-21T23:06:25.979763+0000","last_became_peered":"2024-04-21T23:06:25.979763+0000","last_unstale":"2024-04-21T23:07:24.449965+0000","last_undegraded":"2024-04-21T23:07:24.449965+0000","last_fullsized":"2024-04-21T23:07:24.449965+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:12:23.771931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451425+0000","last_change":"2024-04-21T23:06:25.992461+0000","last_active":"2024-04-21T23:07:24.451425+0000","last_peered":"2024-04-21T23:07:24.451425+0000","last_clean":"2024-04-21T23:07:24.451425+0000","last_became_active":"2024-04-21T23:06:25.992018+0000","last_became_peered":"2024-04-21T23:06:25.992018+0000","last_unstale":"2024-04-21T23:07:24.451425+0000","last_undegraded":"2024-04-21T23:07:24.451425+0000","last_fullsized":"2024-04-21T23:07:24.451425+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:46.651683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451416+0000","last_change":"2024-04-21T23:06:25.974812+0000","last_active":"2024-04-21T23:07:24.451416+0000","last_peered":"2024-04-21T23:07:24.451416+0000","last_clean":"2024-04-21T23:07:24.451416+0000","last_became_active":"2024-04-21T23:06:25.974420+0000","last_became_peered":"2024-04-21T23:06:25.974420+0000","last_unstale":"2024-04-21T23:07:24.451416+0000","last_undegraded":"2024-04-21T23:07:24.451416+0000","last_fullsized":"2024-04-21T23:07:24.451416+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:00.734218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451907+0000","last_change":"2024-04-21T23:06:25.988806+0000","last_active":"2024-04-21T23:07:24.451907+0000","last_peered":"2024-04-21T23:07:24.451907+0000","last_clean":"2024-04-21T23:07:24.451907+0000","last_became_active":"2024-04-21T23:06:25.988344+0000","last_became_peered":"2024-04-21T23:06:25.988344+0000","last_unstale":"2024-04-21T23:07:24.451907+0000","last_undegraded":"2024-04-21T23:07:24.451907+0000","last_fullsized":"2024-04-21T23:07:24.451907+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:15:40.079273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.446257+0000","last_change":"2024-04-21T23:06:25.979125+0000","last_active":"2024-04-21T23:07:24.446257+0000","last_peered":"2024-04-21T23:07:24.446257+0000","last_clean":"2024-04-21T23:07:24.446257+0000","last_became_active":"2024-04-21T23:06:25.978778+0000","last_became_peered":"2024-04-21T23:06:25.978778+0000","last_unstale":"2024-04-21T23:07:24.446257+0000","last_undegraded":"2024-04-21T23:07:24.446257+0000","last_fullsized":"2024-04-21T23:07:24.446257+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:44.655404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.448111+0000","last_change":"2024-04-21T23:06:25.983983+0000","last_active":"2024-04-21T23:07:24.448111+0000","last_peered":"2024-04-21T23:07:24.448111+0000","last_clean":"2024-04-21T23:07:24.448111+0000","last_became_active":"2024-04-21T23:06:25.983463+0000","last_became_peered":"2024-04-21T23:06:25.983463+0000","last_unstale":"2024-04-21T23:07:24.448111+0000","last_undegraded":"2024-04-21T23:07:24.448111+0000","last_fullsized":"2024-04-21T23:07:24.448111+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:32:12.784752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450288+0000","last_change":"2024-04-21T23:06:25.987479+0000","last_active":"2024-04-21T23:07:24.450288+0000","last_peered":"2024-04-21T23:07:24.450288+0000","last_clean":"2024-04-21T23:07:24.450288+0000","last_became_active":"2024-04-21T23:06:25.987133+0000","last_became_peered":"2024-04-21T23:06:25.987133+0000","last_unstale":"2024-04-21T23:07:24.450288+0000","last_undegraded":"2024-04-21T23:07:24.450288+0000","last_fullsized":"2024-04-21T23:07:24.450288+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:30:08.033184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.814870+0000","last_change":"2024-04-21T23:06:25.987714+0000","last_active":"2024-04-21T23:07:24.814870+0000","last_peered":"2024-04-21T23:07:24.814870+0000","last_clean":"2024-04-21T23:07:24.814870+0000","last_became_active":"2024-04-21T23:06:25.987457+0000","last_became_peered":"2024-04-21T23:06:25.987457+0000","last_unstale":"2024-04-21T23:07:24.814870+0000","last_undegraded":"2024-04-21T23:07:24.814870+0000","last_fullsized":"2024-04-21T23:07:24.814870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:06:06.184525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.816484+0000","last_change":"2024-04-21T23:06:25.975957+0000","last_active":"2024-04-21T23:07:24.816484+0000","last_peered":"2024-04-21T23:07:24.816484+0000","last_clean":"2024-04-21T23:07:24.816484+0000","last_became_active":"2024-04-21T23:06:25.974710+0000","last_became_peered":"2024-04-21T23:06:25.974710+0000","last_unstale":"2024-04-21T23:07:24.816484+0000","last_undegraded":"2024-04-21T23:07:24.816484+0000","last_fullsized":"2024-04-21T23:07:24.816484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:31:40.062273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.448435+0000","last_change":"2024-04-21T23:06:25.995085+0000","last_active":"2024-04-21T23:07:24.448435+0000","last_peered":"2024-04-21T23:07:24.448435+0000","last_clean":"2024-04-21T23:07:24.448435+0000","last_became_active":"2024-04-21T23:06:25.993552+0000","last_became_peered":"2024-04-21T23:06:25.993552+0000","last_unstale":"2024-04-21T23:07:24.448435+0000","last_undegraded":"2024-04-21T23:07:24.448435+0000","last_fullsized":"2024-04-21T23:07:24.448435+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:11.405113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452004+0000","last_change":"2024-04-21T23:06:25.993191+0000","last_active":"2024-04-21T23:07:24.452004+0000","last_peered":"2024-04-21T23:07:24.452004+0000","last_clean":"2024-04-21T23:07:24.452004+0000","last_became_active":"2024-04-21T23:06:25.989399+0000","last_became_peered":"2024-04-21T23:06:25.989399+0000","last_unstale":"2024-04-21T23:07:24.452004+0000","last_undegraded":"2024-04-21T23:07:24.452004+0000","last_fullsized":"2024-04-21T23:07:24.452004+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:30:20.035599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451797+0000","last_change":"2024-04-21T23:06:25.994158+0000","last_active":"2024-04-21T23:07:24.451797+0000","last_peered":"2024-04-21T23:07:24.451797+0000","last_clean":"2024-04-21T23:07:24.451797+0000","last_became_active":"2024-04-21T23:06:25.993780+0000","last_became_peered":"2024-04-21T23:06:25.993780+0000","last_unstale":"2024-04-21T23:07:24.451797+0000","last_undegraded":"2024-04-21T23:07:24.451797+0000","last_fullsized":"2024-04-21T23:07:24.451797+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:52:17.181898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451391+0000","last_change":"2024-04-21T23:06:25.989274+0000","last_active":"2024-04-21T23:07:24.451391+0000","last_peered":"2024-04-21T23:07:24.451391+0000","last_clean":"2024-04-21T23:07:24.451391+0000","last_became_active":"2024-04-21T23:06:25.988931+0000","last_became_peered":"2024-04-21T23:06:25.988931+0000","last_unstale":"2024-04-21T23:07:24.451391+0000","last_undegraded":"2024-04-21T23:07:24.451391+0000","last_fullsized":"2024-04-21T23:07:24.451391+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:11:04.576136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451115+0000","last_change":"2024-04-21T23:06:25.982976+0000","last_active":"2024-04-21T23:07:24.451115+0000","last_peered":"2024-04-21T23:07:24.451115+0000","last_clean":"2024-04-21T23:07:24.451115+0000","last_became_active":"2024-04-21T23:06:25.982625+0000","last_became_peered":"2024-04-21T23:06:25.982625+0000","last_unstale":"2024-04-21T23:07:24.451115+0000","last_undegraded":"2024-04-21T23:07:24.451115+0000","last_fullsized":"2024-04-21T23:07:24.451115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:51:02.193044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450985+0000","last_change":"2024-04-21T23:06:25.994805+0000","last_active":"2024-04-21T23:07:24.450985+0000","last_peered":"2024-04-21T23:07:24.450985+0000","last_clean":"2024-04-21T23:07:24.450985+0000","last_became_active":"2024-04-21T23:06:25.994482+0000","last_became_peered":"2024-04-21T23:06:25.994482+0000","last_unstale":"2024-04-21T23:07:24.450985+0000","last_undegraded":"2024-04-21T23:07:24.450985+0000","last_fullsized":"2024-04-21T23:07:24.450985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:24:21.792358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815354+0000","last_change":"2024-04-21T23:06:25.990840+0000","last_active":"2024-04-21T23:07:24.815354+0000","last_peered":"2024-04-21T23:07:24.815354+0000","last_clean":"2024-04-21T23:07:24.815354+0000","last_became_active":"2024-04-21T23:06:25.990517+0000","last_became_peered":"2024-04-21T23:06:25.990517+0000","last_unstale":"2024-04-21T23:07:24.815354+0000","last_undegraded":"2024-04-21T23:07:24.815354+0000","last_fullsized":"2024-04-21T23:07:24.815354+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:48:09.265280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.449709+0000","last_change":"2024-04-21T23:06:25.976810+0000","last_active":"2024-04-21T23:07:24.449709+0000","last_peered":"2024-04-21T23:07:24.449709+0000","last_clean":"2024-04-21T23:07:24.449709+0000","last_became_active":"2024-04-21T23:06:25.976489+0000","last_became_peered":"2024-04-21T23:06:25.976489+0000","last_unstale":"2024-04-21T23:07:24.449709+0000","last_undegraded":"2024-04-21T23:07:24.449709+0000","last_fullsized":"2024-04-21T23:07:24.449709+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:07:03.794115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447953+0000","last_change":"2024-04-21T23:06:25.988603+0000","last_active":"2024-04-21T23:07:24.447953+0000","last_peered":"2024-04-21T23:07:24.447953+0000","last_clean":"2024-04-21T23:07:24.447953+0000","last_became_active":"2024-04-21T23:06:25.988254+0000","last_became_peered":"2024-04-21T23:06:25.988254+0000","last_unstale":"2024-04-21T23:07:24.447953+0000","last_undegraded":"2024-04-21T23:07:24.447953+0000","last_fullsized":"2024-04-21T23:07:24.447953+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:25:46.099853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452093+0000","last_change":"2024-04-21T23:06:25.980831+0000","last_active":"2024-04-21T23:07:24.452093+0000","last_peered":"2024-04-21T23:07:24.452093+0000","last_clean":"2024-04-21T23:07:24.452093+0000","last_became_active":"2024-04-21T23:06:25.977051+0000","last_became_peered":"2024-04-21T23:06:25.977051+0000","last_unstale":"2024-04-21T23:07:24.452093+0000","last_undegraded":"2024-04-21T23:07:24.452093+0000","last_fullsized":"2024-04-21T23:07:24.452093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:55:03.684734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450854+0000","last_change":"2024-04-21T23:06:30.256875+0000","last_active":"2024-04-21T23:07:24.450854+0000","last_peered":"2024-04-21T23:07:24.450854+0000","last_clean":"2024-04-21T23:07:24.450854+0000","last_became_active":"2024-04-21T23:06:30.256459+0000","last_became_peered":"2024-04-21T23:06:30.256459+0000","last_unstale":"2024-04-21T23:07:24.450854+0000","last_undegraded":"2024-04-21T23:07:24.450854+0000","last_fullsized":"2024-04-21T23:07:24.450854+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:45:32.376270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447817+0000","last_change":"2024-04-21T23:06:25.209605+0000","last_active":"2024-04-21T23:07:24.447817+0000","last_peered":"2024-04-21T23:07:24.447817+0000","last_clean":"2024-04-21T23:07:24.447817+0000","last_became_active":"2024-04-21T23:06:25.209277+0000","last_became_peered":"2024-04-21T23:06:25.209277+0000","last_unstale":"2024-04-21T23:07:24.447817+0000","last_undegraded":"2024-04-21T23:07:24.447817+0000","last_fullsized":"2024-04-21T23:07:24.447817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:15:49.742940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815277+0000","last_change":"2024-04-21T23:06:25.981774+0000","last_active":"2024-04-21T23:07:24.815277+0000","last_peered":"2024-04-21T23:07:24.815277+0000","last_clean":"2024-04-21T23:07:24.815277+0000","last_became_active":"2024-04-21T23:06:25.981432+0000","last_became_peered":"2024-04-21T23:06:25.981432+0000","last_unstale":"2024-04-21T23:07:24.815277+0000","last_undegraded":"2024-04-21T23:07:24.815277+0000","last_fullsized":"2024-04-21T23:07:24.815277+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:28:28.712149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451351+0000","last_change":"2024-04-21T23:06:30.271508+0000","last_active":"2024-04-21T23:07:24.451351+0000","last_peered":"2024-04-21T23:07:24.451351+0000","last_clean":"2024-04-21T23:07:24.451351+0000","last_became_active":"2024-04-21T23:06:30.270461+0000","last_became_peered":"2024-04-21T23:06:30.270461+0000","last_unstale":"2024-04-21T23:07:24.451351+0000","last_undegraded":"2024-04-21T23:07:24.451351+0000","last_fullsized":"2024-04-21T23:07:24.451351+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:43:07.473365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815649+0000","last_change":"2024-04-21T23:06:25.218773+0000","last_active":"2024-04-21T23:07:24.815649+0000","last_peered":"2024-04-21T23:07:24.815649+0000","last_clean":"2024-04-21T23:07:24.815649+0000","last_became_active":"2024-04-21T23:06:25.218458+0000","last_became_peered":"2024-04-21T23:06:25.218458+0000","last_unstale":"2024-04-21T23:07:24.815649+0000","last_undegraded":"2024-04-21T23:07:24.815649+0000","last_fullsized":"2024-04-21T23:07:24.815649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:25:16.372689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451480+0000","last_change":"2024-04-21T23:06:25.977669+0000","last_active":"2024-04-21T23:07:24.451480+0000","last_peered":"2024-04-21T23:07:24.451480+0000","last_clean":"2024-04-21T23:07:24.451480+0000","last_became_active":"2024-04-21T23:06:25.977143+0000","last_became_peered":"2024-04-21T23:06:25.977143+0000","last_unstale":"2024-04-21T23:07:24.451480+0000","last_undegraded":"2024-04-21T23:07:24.451480+0000","last_fullsized":"2024-04-21T23:07:24.451480+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:39.289208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451360+0000","last_change":"2024-04-21T23:06:30.271934+0000","last_active":"2024-04-21T23:07:24.451360+0000","last_peered":"2024-04-21T23:07:24.451360+0000","last_clean":"2024-04-21T23:07:24.451360+0000","last_became_active":"2024-04-21T23:06:30.270567+0000","last_became_peered":"2024-04-21T23:06:30.270567+0000","last_unstale":"2024-04-21T23:07:24.451360+0000","last_undegraded":"2024-04-21T23:07:24.451360+0000","last_fullsized":"2024-04-21T23:07:24.451360+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:43:34.908748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.462595+0000","last_change":"2024-04-21T23:07:24.462595+0000","last_active":"2024-04-21T23:07:24.462595+0000","last_peered":"2024-04-21T23:07:24.462595+0000","last_clean":"2024-04-21T23:07:24.462595+0000","last_became_active":"2024-04-21T23:07:24.454522+0000","last_became_peered":"2024-04-21T23:07:24.454522+0000","last_unstale":"2024-04-21T23:07:24.462595+0000","last_undegraded":"2024-04-21T23:07:24.462595+0000","last_fullsized":"2024-04-21T23:07:24.462595+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:07.457105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.816028+0000","last_change":"2024-04-21T23:06:25.988197+0000","last_active":"2024-04-21T23:07:24.816028+0000","last_peered":"2024-04-21T23:07:24.816028+0000","last_clean":"2024-04-21T23:07:24.816028+0000","last_became_active":"2024-04-21T23:06:25.987872+0000","last_became_peered":"2024-04-21T23:06:25.987872+0000","last_unstale":"2024-04-21T23:07:24.816028+0000","last_undegraded":"2024-04-21T23:07:24.816028+0000","last_fullsized":"2024-04-21T23:07:24.816028+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:25.389020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450816+0000","last_change":"2024-04-21T23:06:30.265262+0000","last_active":"2024-04-21T23:07:24.450816+0000","last_peered":"2024-04-21T23:07:24.450816+0000","last_clean":"2024-04-21T23:07:24.450816+0000","last_became_active":"2024-04-21T23:06:30.264781+0000","last_became_peered":"2024-04-21T23:06:30.264781+0000","last_unstale":"2024-04-21T23:07:24.450816+0000","last_undegraded":"2024-04-21T23:07:24.450816+0000","last_fullsized":"2024-04-21T23:07:24.450816+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:03:30.756792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.446480+0000","last_change":"2024-04-21T23:06:25.205070+0000","last_active":"2024-04-21T23:07:24.446480+0000","last_peered":"2024-04-21T23:07:24.446480+0000","last_clean":"2024-04-21T23:07:24.446480+0000","last_became_active":"2024-04-21T23:06:25.204737+0000","last_became_peered":"2024-04-21T23:06:25.204737+0000","last_unstale":"2024-04-21T23:07:24.446480+0000","last_undegraded":"2024-04-21T23:07:24.446480+0000","last_fullsized":"2024-04-21T23:07:24.446480+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:37.014534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.453979+0000","last_change":"2024-04-21T23:06:25.986584+0000","last_active":"2024-04-21T23:07:24.453979+0000","last_peered":"2024-04-21T23:07:24.453979+0000","last_clean":"2024-04-21T23:07:24.453979+0000","last_became_active":"2024-04-21T23:06:25.986268+0000","last_became_peered":"2024-04-21T23:06:25.986268+0000","last_unstale":"2024-04-21T23:07:24.453979+0000","last_undegraded":"2024-04-21T23:07:24.453979+0000","last_fullsized":"2024-04-21T23:07:24.453979+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:19:43.864279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451912+0000","last_change":"2024-04-21T23:06:30.261900+0000","last_active":"2024-04-21T23:07:24.451912+0000","last_peered":"2024-04-21T23:07:24.451912+0000","last_clean":"2024-04-21T23:07:24.451912+0000","last_became_active":"2024-04-21T23:06:30.261444+0000","last_became_peered":"2024-04-21T23:06:30.261444+0000","last_unstale":"2024-04-21T23:07:24.451912+0000","last_undegraded":"2024-04-21T23:07:24.451912+0000","last_fullsized":"2024-04-21T23:07:24.451912+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:10:40.810555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.817616+0000","last_change":"2024-04-21T23:07:24.817616+0000","last_active":"2024-04-21T23:07:24.817616+0000","last_peered":"2024-04-21T23:07:24.817616+0000","last_clean":"2024-04-21T23:07:24.817616+0000","last_became_active":"2024-04-21T23:07:24.816888+0000","last_became_peered":"2024-04-21T23:07:24.816888+0000","last_unstale":"2024-04-21T23:07:24.817616+0000","last_undegraded":"2024-04-21T23:07:24.817616+0000","last_fullsized":"2024-04-21T23:07:24.817616+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:01:03.069832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.446272+0000","last_change":"2024-04-21T23:06:25.975999+0000","last_active":"2024-04-21T23:07:24.446272+0000","last_peered":"2024-04-21T23:07:24.446272+0000","last_clean":"2024-04-21T23:07:24.446272+0000","last_became_active":"2024-04-21T23:06:25.975677+0000","last_became_peered":"2024-04-21T23:06:25.975677+0000","last_unstale":"2024-04-21T23:07:24.446272+0000","last_undegraded":"2024-04-21T23:07:24.446272+0000","last_fullsized":"2024-04-21T23:07:24.446272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:52:03.177248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.456926+0000","last_change":"2024-04-21T23:07:24.456926+0000","last_active":"2024-04-21T23:07:24.456926+0000","last_peered":"2024-04-21T23:07:24.456926+0000","last_clean":"2024-04-21T23:07:24.456926+0000","last_became_active":"2024-04-21T23:07:24.456481+0000","last_became_peered":"2024-04-21T23:07:24.456481+0000","last_unstale":"2024-04-21T23:07:24.456926+0000","last_undegraded":"2024-04-21T23:07:24.456926+0000","last_fullsized":"2024-04-21T23:07:24.456926+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:19:26.553012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447850+0000","last_change":"2024-04-21T23:06:25.214893+0000","last_active":"2024-04-21T23:07:24.447850+0000","last_peered":"2024-04-21T23:07:24.447850+0000","last_clean":"2024-04-21T23:07:24.447850+0000","last_became_active":"2024-04-21T23:06:25.212359+0000","last_became_peered":"2024-04-21T23:06:25.212359+0000","last_unstale":"2024-04-21T23:07:24.447850+0000","last_undegraded":"2024-04-21T23:07:24.447850+0000","last_fullsized":"2024-04-21T23:07:24.447850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:18:03.278875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.453767+0000","last_change":"2024-04-21T23:06:25.986849+0000","last_active":"2024-04-21T23:07:24.453767+0000","last_peered":"2024-04-21T23:07:24.453767+0000","last_clean":"2024-04-21T23:07:24.453767+0000","last_became_active":"2024-04-21T23:06:25.986334+0000","last_became_peered":"2024-04-21T23:06:25.986334+0000","last_unstale":"2024-04-21T23:07:24.453767+0000","last_undegraded":"2024-04-21T23:07:24.453767+0000","last_fullsized":"2024-04-21T23:07:24.453767+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:57:42.195629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.449586+0000","last_change":"2024-04-21T23:06:30.270812+0000","last_active":"2024-04-21T23:07:24.449586+0000","last_peered":"2024-04-21T23:07:24.449586+0000","last_clean":"2024-04-21T23:07:24.449586+0000","last_became_active":"2024-04-21T23:06:30.270370+0000","last_became_peered":"2024-04-21T23:06:30.270370+0000","last_unstale":"2024-04-21T23:07:24.449586+0000","last_undegraded":"2024-04-21T23:07:24.449586+0000","last_fullsized":"2024-04-21T23:07:24.449586+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:53.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450585+0000","last_change":"2024-04-21T23:06:25.201603+0000","last_active":"2024-04-21T23:07:24.450585+0000","last_peered":"2024-04-21T23:07:24.450585+0000","last_clean":"2024-04-21T23:07:24.450585+0000","last_became_active":"2024-04-21T23:06:25.201265+0000","last_became_peered":"2024-04-21T23:06:25.201265+0000","last_unstale":"2024-04-21T23:07:24.450585+0000","last_undegraded":"2024-04-21T23:07:24.450585+0000","last_fullsized":"2024-04-21T23:07:24.450585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:23:19.134215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.814984+0000","last_change":"2024-04-21T23:06:25.982420+0000","last_active":"2024-04-21T23:07:24.814984+0000","last_peered":"2024-04-21T23:07:24.814984+0000","last_clean":"2024-04-21T23:07:24.814984+0000","last_became_active":"2024-04-21T23:06:25.981867+0000","last_became_peered":"2024-04-21T23:06:25.981867+0000","last_unstale":"2024-04-21T23:07:24.814984+0000","last_undegraded":"2024-04-21T23:07:24.814984+0000","last_fullsized":"2024-04-21T23:07:24.814984+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:10:29.433243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451298+0000","last_change":"2024-04-21T23:06:30.273907+0000","last_active":"2024-04-21T23:07:24.451298+0000","last_peered":"2024-04-21T23:07:24.451298+0000","last_clean":"2024-04-21T23:07:24.451298+0000","last_became_active":"2024-04-21T23:06:30.273472+0000","last_became_peered":"2024-04-21T23:06:30.273472+0000","last_unstale":"2024-04-21T23:07:24.451298+0000","last_undegraded":"2024-04-21T23:07:24.451298+0000","last_fullsized":"2024-04-21T23:07:24.451298+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:58:50.914940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452141+0000","last_change":"2024-04-21T23:06:25.211117+0000","last_active":"2024-04-21T23:07:24.452141+0000","last_peered":"2024-04-21T23:07:24.452141+0000","last_clean":"2024-04-21T23:07:24.452141+0000","last_became_active":"2024-04-21T23:06:25.210626+0000","last_became_peered":"2024-04-21T23:06:25.210626+0000","last_unstale":"2024-04-21T23:07:24.452141+0000","last_undegraded":"2024-04-21T23:07:24.452141+0000","last_fullsized":"2024-04-21T23:07:24.452141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:56:39.917072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447267+0000","last_change":"2024-04-21T23:06:25.980537+0000","last_active":"2024-04-21T23:07:24.447267+0000","last_peered":"2024-04-21T23:07:24.447267+0000","last_clean":"2024-04-21T23:07:24.447267+0000","last_became_active":"2024-04-21T23:06:25.980188+0000","last_became_peered":"2024-04-21T23:06:25.980188+0000","last_unstale":"2024-04-21T23:07:24.447267+0000","last_undegraded":"2024-04-21T23:07:24.447267+0000","last_fullsized":"2024-04-21T23:07:24.447267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:58:02.828632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815003+0000","last_change":"2024-04-21T23:06:30.263565+0000","last_active":"2024-04-21T23:07:24.815003+0000","last_peered":"2024-04-21T23:07:24.815003+0000","last_clean":"2024-04-21T23:07:24.815003+0000","last_became_active":"2024-04-21T23:06:30.263092+0000","last_became_peered":"2024-04-21T23:06:30.263092+0000","last_unstale":"2024-04-21T23:07:24.815003+0000","last_undegraded":"2024-04-21T23:07:24.815003+0000","last_fullsized":"2024-04-21T23:07:24.815003+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:58:25.709551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.448432+0000","last_change":"2024-04-21T23:06:25.212567+0000","last_active":"2024-04-21T23:07:24.448432+0000","last_peered":"2024-04-21T23:07:24.448432+0000","last_clean":"2024-04-21T23:07:24.448432+0000","last_became_active":"2024-04-21T23:06:25.210593+0000","last_became_peered":"2024-04-21T23:06:25.210593+0000","last_unstale":"2024-04-21T23:07:24.448432+0000","last_undegraded":"2024-04-21T23:07:24.448432+0000","last_fullsized":"2024-04-21T23:07:24.448432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:53:33.552742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.816324+0000","last_change":"2024-04-21T23:06:25.989525+0000","last_active":"2024-04-21T23:07:24.816324+0000","last_peered":"2024-04-21T23:07:24.816324+0000","last_clean":"2024-04-21T23:07:24.816324+0000","last_became_active":"2024-04-21T23:06:25.989130+0000","last_became_peered":"2024-04-21T23:06:25.989130+0000","last_unstale":"2024-04-21T23:07:24.816324+0000","last_undegraded":"2024-04-21T23:07:24.816324+0000","last_fullsized":"2024-04-21T23:07:24.816324+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:19:31.000530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447165+0000","last_change":"2024-04-21T23:06:30.269309+0000","last_active":"2024-04-21T23:07:24.447165+0000","last_peered":"2024-04-21T23:07:24.447165+0000","last_clean":"2024-04-21T23:07:24.447165+0000","last_became_active":"2024-04-21T23:06:30.268903+0000","last_became_peered":"2024-04-21T23:06:30.268903+0000","last_unstale":"2024-04-21T23:07:24.447165+0000","last_undegraded":"2024-04-21T23:07:24.447165+0000","last_fullsized":"2024-04-21T23:07:24.447165+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:52:16.043870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451651+0000","last_change":"2024-04-21T23:06:25.213938+0000","last_active":"2024-04-21T23:07:24.451651+0000","last_peered":"2024-04-21T23:07:24.451651+0000","last_clean":"2024-04-21T23:07:24.451651+0000","last_became_active":"2024-04-21T23:06:25.213242+0000","last_became_peered":"2024-04-21T23:06:25.213242+0000","last_unstale":"2024-04-21T23:07:24.451651+0000","last_undegraded":"2024-04-21T23:07:24.451651+0000","last_fullsized":"2024-04-21T23:07:24.451651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:27.679122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.454702+0000","last_change":"2024-04-21T23:07:24.454702+0000","last_active":"2024-04-21T23:07:24.454702+0000","last_peered":"2024-04-21T23:07:24.454702+0000","last_clean":"2024-04-21T23:07:24.454702+0000","last_became_active":"2024-04-21T23:07:24.454211+0000","last_became_peered":"2024-04-21T23:07:24.454211+0000","last_unstale":"2024-04-21T23:07:24.454702+0000","last_undegraded":"2024-04-21T23:07:24.454702+0000","last_fullsized":"2024-04-21T23:07:24.454702+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:29:30.745026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451525+0000","last_change":"2024-04-21T23:06:30.272206+0000","last_active":"2024-04-21T23:07:24.451525+0000","last_peered":"2024-04-21T23:07:24.451525+0000","last_clean":"2024-04-21T23:07:24.451525+0000","last_became_active":"2024-04-21T23:06:30.271389+0000","last_became_peered":"2024-04-21T23:06:30.271389+0000","last_unstale":"2024-04-21T23:07:24.451525+0000","last_undegraded":"2024-04-21T23:07:24.451525+0000","last_fullsized":"2024-04-21T23:07:24.451525+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:54:47.996166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.454670+0000","last_change":"2024-04-21T23:06:25.217655+0000","last_active":"2024-04-21T23:07:24.454670+0000","last_peered":"2024-04-21T23:07:24.454670+0000","last_clean":"2024-04-21T23:07:24.454670+0000","last_became_active":"2024-04-21T23:06:25.217337+0000","last_became_peered":"2024-04-21T23:06:25.217337+0000","last_unstale":"2024-04-21T23:07:24.454670+0000","last_undegraded":"2024-04-21T23:07:24.454670+0000","last_fullsized":"2024-04-21T23:07:24.454670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:29:28.690841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.816161+0000","last_change":"2024-04-21T23:06:25.995757+0000","last_active":"2024-04-21T23:07:24.816161+0000","last_peered":"2024-04-21T23:07:24.816161+0000","last_clean":"2024-04-21T23:07:24.816161+0000","last_became_active":"2024-04-21T23:06:25.995443+0000","last_became_peered":"2024-04-21T23:06:25.995443+0000","last_unstale":"2024-04-21T23:07:24.816161+0000","last_undegraded":"2024-04-21T23:07:24.816161+0000","last_fullsized":"2024-04-21T23:07:24.816161+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:40.359633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447349+0000","last_change":"2024-04-21T23:06:30.266441+0000","last_active":"2024-04-21T23:07:24.447349+0000","last_peered":"2024-04-21T23:07:24.447349+0000","last_clean":"2024-04-21T23:07:24.447349+0000","last_became_active":"2024-04-21T23:06:30.264065+0000","last_became_peered":"2024-04-21T23:06:30.264065+0000","last_unstale":"2024-04-21T23:07:24.447349+0000","last_undegraded":"2024-04-21T23:07:24.447349+0000","last_fullsized":"2024-04-21T23:07:24.447349+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:53:22.548096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'31","reported_seq":213,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:08:03.878831+0000","last_change":"2024-04-21T23:06:25.213919+0000","last_active":"2024-04-21T23:08:03.878831+0000","last_peered":"2024-04-21T23:08:03.878831+0000","last_clean":"2024-04-21T23:08:03.878831+0000","last_became_active":"2024-04-21T23:06:25.212244+0000","last_became_peered":"2024-04-21T23:06:25.212244+0000","last_unstale":"2024-04-21T23:08:03.878831+0000","last_undegraded":"2024-04-21T23:08:03.878831+0000","last_fullsized":"2024-04-21T23:08:03.878831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:17:25.287718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":155,"num_read_kb":140,"num_write":31,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.446736+0000","last_change":"2024-04-21T23:06:25.993666+0000","last_active":"2024-04-21T23:07:24.446736+0000","last_peered":"2024-04-21T23:07:24.446736+0000","last_clean":"2024-04-21T23:07:24.446736+0000","last_became_active":"2024-04-21T23:06:25.993330+0000","last_became_peered":"2024-04-21T23:06:25.993330+0000","last_unstale":"2024-04-21T23:07:24.446736+0000","last_undegraded":"2024-04-21T23:07:24.446736+0000","last_fullsized":"2024-04-21T23:07:24.446736+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:11:00.109782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.454813+0000","last_change":"2024-04-21T23:06:30.265807+0000","last_active":"2024-04-21T23:07:24.454813+0000","last_peered":"2024-04-21T23:07:24.454813+0000","last_clean":"2024-04-21T23:07:24.454813+0000","last_became_active":"2024-04-21T23:06:30.265116+0000","last_became_peered":"2024-04-21T23:06:30.265116+0000","last_unstale":"2024-04-21T23:07:24.454813+0000","last_undegraded":"2024-04-21T23:07:24.454813+0000","last_fullsized":"2024-04-21T23:07:24.454813+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:14:04.178281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451692+0000","last_change":"2024-04-21T23:06:25.208319+0000","last_active":"2024-04-21T23:07:24.451692+0000","last_peered":"2024-04-21T23:07:24.451692+0000","last_clean":"2024-04-21T23:07:24.451692+0000","last_became_active":"2024-04-21T23:06:25.208025+0000","last_became_peered":"2024-04-21T23:06:25.208025+0000","last_unstale":"2024-04-21T23:07:24.451692+0000","last_undegraded":"2024-04-21T23:07:24.451692+0000","last_fullsized":"2024-04-21T23:07:24.451692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:14.723170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450822+0000","last_change":"2024-04-21T23:06:25.993604+0000","last_active":"2024-04-21T23:07:24.450822+0000","last_peered":"2024-04-21T23:07:24.450822+0000","last_clean":"2024-04-21T23:07:24.450822+0000","last_became_active":"2024-04-21T23:06:25.993217+0000","last_became_peered":"2024-04-21T23:06:25.993217+0000","last_unstale":"2024-04-21T23:07:24.450822+0000","last_undegraded":"2024-04-21T23:07:24.450822+0000","last_fullsized":"2024-04-21T23:07:24.450822+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:23:00.435688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.448793+0000","last_change":"2024-04-21T23:06:30.270991+0000","last_active":"2024-04-21T23:07:24.448793+0000","last_peered":"2024-04-21T23:07:24.448793+0000","last_clean":"2024-04-21T23:07:24.448793+0000","last_became_active":"2024-04-21T23:06:30.270511+0000","last_became_peered":"2024-04-21T23:06:30.270511+0000","last_unstale":"2024-04-21T23:07:24.448793+0000","last_undegraded":"2024-04-21T23:07:24.448793+0000","last_fullsized":"2024-04-21T23:07:24.448793+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:04.265855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.453270+0000","last_change":"2024-04-21T23:06:25.211787+0000","last_active":"2024-04-21T23:07:24.453270+0000","last_peered":"2024-04-21T23:07:24.453270+0000","last_clean":"2024-04-21T23:07:24.453270+0000","last_became_active":"2024-04-21T23:06:25.211473+0000","last_became_peered":"2024-04-21T23:06:25.211473+0000","last_unstale":"2024-04-21T23:07:24.453270+0000","last_undegraded":"2024-04-21T23:07:24.453270+0000","last_fullsized":"2024-04-21T23:07:24.453270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:58:51.364974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450870+0000","last_change":"2024-04-21T23:06:25.990681+0000","last_active":"2024-04-21T23:07:24.450870+0000","last_peered":"2024-04-21T23:07:24.450870+0000","last_clean":"2024-04-21T23:07:24.450870+0000","last_became_active":"2024-04-21T23:06:25.990094+0000","last_became_peered":"2024-04-21T23:06:25.990094+0000","last_unstale":"2024-04-21T23:07:24.450870+0000","last_undegraded":"2024-04-21T23:07:24.450870+0000","last_fullsized":"2024-04-21T23:07:24.450870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:15:26.907912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447564+0000","last_change":"2024-04-21T23:06:30.271352+0000","last_active":"2024-04-21T23:07:24.447564+0000","last_peered":"2024-04-21T23:07:24.447564+0000","last_clean":"2024-04-21T23:07:24.447564+0000","last_became_active":"2024-04-21T23:06:30.270601+0000","last_became_peered":"2024-04-21T23:06:30.270601+0000","last_unstale":"2024-04-21T23:07:24.447564+0000","last_undegraded":"2024-04-21T23:07:24.447564+0000","last_fullsized":"2024-04-21T23:07:24.447564+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:37.266987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451040+0000","last_change":"2024-04-21T23:06:25.213288+0000","last_active":"2024-04-21T23:07:24.451040+0000","last_peered":"2024-04-21T23:07:24.451040+0000","last_clean":"2024-04-21T23:07:24.451040+0000","last_became_active":"2024-04-21T23:06:25.212936+0000","last_became_peered":"2024-04-21T23:06:25.212936+0000","last_unstale":"2024-04-21T23:07:24.451040+0000","last_undegraded":"2024-04-21T23:07:24.451040+0000","last_fullsized":"2024-04-21T23:07:24.451040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:35.303093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450871+0000","last_change":"2024-04-21T23:06:25.998485+0000","last_active":"2024-04-21T23:07:24.450871+0000","last_peered":"2024-04-21T23:07:24.450871+0000","last_clean":"2024-04-21T23:07:24.450871+0000","last_became_active":"2024-04-21T23:06:25.998184+0000","last_became_peered":"2024-04-21T23:06:25.998184+0000","last_unstale":"2024-04-21T23:07:24.450871+0000","last_undegraded":"2024-04-21T23:07:24.450871+0000","last_fullsized":"2024-04-21T23:07:24.450871+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:35:26.613636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451711+0000","last_change":"2024-04-21T23:06:30.270089+0000","last_active":"2024-04-21T23:07:24.451711+0000","last_peered":"2024-04-21T23:07:24.451711+0000","last_clean":"2024-04-21T23:07:24.451711+0000","last_became_active":"2024-04-21T23:06:30.269695+0000","last_became_peered":"2024-04-21T23:06:30.269695+0000","last_unstale":"2024-04-21T23:07:24.451711+0000","last_undegraded":"2024-04-21T23:07:24.451711+0000","last_fullsized":"2024-04-21T23:07:24.451711+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:42.718580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.446128+0000","last_change":"2024-04-21T23:06:25.218247+0000","last_active":"2024-04-21T23:07:24.446128+0000","last_peered":"2024-04-21T23:07:24.446128+0000","last_clean":"2024-04-21T23:07:24.446128+0000","last_became_active":"2024-04-21T23:06:25.217924+0000","last_became_peered":"2024-04-21T23:06:25.217924+0000","last_unstale":"2024-04-21T23:07:24.446128+0000","last_undegraded":"2024-04-21T23:07:24.446128+0000","last_fullsized":"2024-04-21T23:07:24.446128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:27:12.155597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452772+0000","last_change":"2024-04-21T23:06:25.994109+0000","last_active":"2024-04-21T23:07:24.452772+0000","last_peered":"2024-04-21T23:07:24.452772+0000","last_clean":"2024-04-21T23:07:24.452772+0000","last_became_active":"2024-04-21T23:06:25.993785+0000","last_became_peered":"2024-04-21T23:06:25.993785+0000","last_unstale":"2024-04-21T23:07:24.452772+0000","last_undegraded":"2024-04-21T23:07:24.452772+0000","last_fullsized":"2024-04-21T23:07:24.452772+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:12.629364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452662+0000","last_change":"2024-04-21T23:06:30.265444+0000","last_active":"2024-04-21T23:07:24.452662+0000","last_peered":"2024-04-21T23:07:24.452662+0000","last_clean":"2024-04-21T23:07:24.452662+0000","last_became_active":"2024-04-21T23:06:30.264971+0000","last_became_peered":"2024-04-21T23:06:30.264971+0000","last_unstale":"2024-04-21T23:07:24.452662+0000","last_undegraded":"2024-04-21T23:07:24.452662+0000","last_fullsized":"2024-04-21T23:07:24.452662+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:03.596631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.465783+0000","last_change":"2024-04-21T23:07:24.465377+0000","last_active":"2024-04-21T23:07:24.465783+0000","last_peered":"2024-04-21T23:07:24.465783+0000","last_clean":"2024-04-21T23:07:24.465783+0000","last_became_active":"2024-04-21T23:07:24.454409+0000","last_became_peered":"2024-04-21T23:07:24.454409+0000","last_unstale":"2024-04-21T23:07:24.465783+0000","last_undegraded":"2024-04-21T23:07:24.465783+0000","last_fullsized":"2024-04-21T23:07:24.465783+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:06:22.836124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.448958+0000","last_change":"2024-04-21T23:06:25.990193+0000","last_active":"2024-04-21T23:07:24.448958+0000","last_peered":"2024-04-21T23:07:24.448958+0000","last_clean":"2024-04-21T23:07:24.448958+0000","last_became_active":"2024-04-21T23:06:25.989842+0000","last_became_peered":"2024-04-21T23:06:25.989842+0000","last_unstale":"2024-04-21T23:07:24.448958+0000","last_undegraded":"2024-04-21T23:07:24.448958+0000","last_fullsized":"2024-04-21T23:07:24.448958+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:54:53.421257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.817262+0000","last_change":"2024-04-21T23:07:24.817262+0000","last_active":"2024-04-21T23:07:24.817262+0000","last_peered":"2024-04-21T23:07:24.817262+0000","last_clean":"2024-04-21T23:07:24.817262+0000","last_became_active":"2024-04-21T23:07:24.816668+0000","last_became_peered":"2024-04-21T23:07:24.816668+0000","last_unstale":"2024-04-21T23:07:24.817262+0000","last_undegraded":"2024-04-21T23:07:24.817262+0000","last_fullsized":"2024-04-21T23:07:24.817262+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:25.302151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447428+0000","last_change":"2024-04-21T23:06:25.218208+0000","last_active":"2024-04-21T23:07:24.447428+0000","last_peered":"2024-04-21T23:07:24.447428+0000","last_clean":"2024-04-21T23:07:24.447428+0000","last_became_active":"2024-04-21T23:06:25.217889+0000","last_became_peered":"2024-04-21T23:06:25.217889+0000","last_unstale":"2024-04-21T23:07:24.447428+0000","last_undegraded":"2024-04-21T23:07:24.447428+0000","last_fullsized":"2024-04-21T23:07:24.447428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:00:08.775682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447991+0000","last_change":"2024-04-21T23:06:25.987615+0000","last_active":"2024-04-21T23:07:24.447991+0000","last_peered":"2024-04-21T23:07:24.447991+0000","last_clean":"2024-04-21T23:07:24.447991+0000","last_became_active":"2024-04-21T23:06:25.987305+0000","last_became_peered":"2024-04-21T23:06:25.987305+0000","last_unstale":"2024-04-21T23:07:24.447991+0000","last_undegraded":"2024-04-21T23:07:24.447991+0000","last_fullsized":"2024-04-21T23:07:24.447991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:23:49.386998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450947+0000","last_change":"2024-04-21T23:06:30.268157+0000","last_active":"2024-04-21T23:07:24.450947+0000","last_peered":"2024-04-21T23:07:24.450947+0000","last_clean":"2024-04-21T23:07:24.450947+0000","last_became_active":"2024-04-21T23:06:30.267732+0000","last_became_peered":"2024-04-21T23:06:30.267732+0000","last_unstale":"2024-04-21T23:07:24.450947+0000","last_undegraded":"2024-04-21T23:07:24.450947+0000","last_fullsized":"2024-04-21T23:07:24.450947+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:29:24.250271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450585+0000","last_change":"2024-04-21T23:06:25.215401+0000","last_active":"2024-04-21T23:07:24.450585+0000","last_peered":"2024-04-21T23:07:24.450585+0000","last_clean":"2024-04-21T23:07:24.450585+0000","last_became_active":"2024-04-21T23:06:25.215067+0000","last_became_peered":"2024-04-21T23:06:25.215067+0000","last_unstale":"2024-04-21T23:07:24.450585+0000","last_undegraded":"2024-04-21T23:07:24.450585+0000","last_fullsized":"2024-04-21T23:07:24.450585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:16:58.977797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447748+0000","last_change":"2024-04-21T23:06:25.996421+0000","last_active":"2024-04-21T23:07:24.447748+0000","last_peered":"2024-04-21T23:07:24.447748+0000","last_clean":"2024-04-21T23:07:24.447748+0000","last_became_active":"2024-04-21T23:06:25.996102+0000","last_became_peered":"2024-04-21T23:06:25.996102+0000","last_unstale":"2024-04-21T23:07:24.447748+0000","last_undegraded":"2024-04-21T23:07:24.447748+0000","last_fullsized":"2024-04-21T23:07:24.447748+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:06:42.630182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451425+0000","last_change":"2024-04-21T23:06:30.404509+0000","last_active":"2024-04-21T23:07:24.451425+0000","last_peered":"2024-04-21T23:07:24.451425+0000","last_clean":"2024-04-21T23:07:24.451425+0000","last_became_active":"2024-04-21T23:06:30.403987+0000","last_became_peered":"2024-04-21T23:06:30.403987+0000","last_unstale":"2024-04-21T23:07:24.451425+0000","last_undegraded":"2024-04-21T23:07:24.451425+0000","last_fullsized":"2024-04-21T23:07:24.451425+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:12:28.412117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452337+0000","last_change":"2024-04-21T23:06:25.994076+0000","last_active":"2024-04-21T23:07:24.452337+0000","last_peered":"2024-04-21T23:07:24.452337+0000","last_clean":"2024-04-21T23:07:24.452337+0000","last_became_active":"2024-04-21T23:06:25.993747+0000","last_became_peered":"2024-04-21T23:06:25.993747+0000","last_unstale":"2024-04-21T23:07:24.452337+0000","last_undegraded":"2024-04-21T23:07:24.452337+0000","last_fullsized":"2024-04-21T23:07:24.452337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:17.771165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450447+0000","last_change":"2024-04-21T23:06:25.209756+0000","last_active":"2024-04-21T23:07:24.450447+0000","last_peered":"2024-04-21T23:07:24.450447+0000","last_clean":"2024-04-21T23:07:24.450447+0000","last_became_active":"2024-04-21T23:06:25.209445+0000","last_became_peered":"2024-04-21T23:06:25.209445+0000","last_unstale":"2024-04-21T23:07:24.450447+0000","last_undegraded":"2024-04-21T23:07:24.450447+0000","last_fullsized":"2024-04-21T23:07:24.450447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:58:49.453047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451866+0000","last_change":"2024-04-21T23:06:30.261565+0000","last_active":"2024-04-21T23:07:24.451866+0000","last_peered":"2024-04-21T23:07:24.451866+0000","last_clean":"2024-04-21T23:07:24.451866+0000","last_became_active":"2024-04-21T23:06:30.261095+0000","last_became_peered":"2024-04-21T23:06:30.261095+0000","last_unstale":"2024-04-21T23:07:24.451866+0000","last_undegraded":"2024-04-21T23:07:24.451866+0000","last_fullsized":"2024-04-21T23:07:24.451866+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:37:23.174152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815182+0000","last_change":"2024-04-21T23:06:25.981402+0000","last_active":"2024-04-21T23:07:24.815182+0000","last_peered":"2024-04-21T23:07:24.815182+0000","last_clean":"2024-04-21T23:07:24.815182+0000","last_became_active":"2024-04-21T23:06:25.981062+0000","last_became_peered":"2024-04-21T23:06:25.981062+0000","last_unstale":"2024-04-21T23:07:24.815182+0000","last_undegraded":"2024-04-21T23:07:24.815182+0000","last_fullsized":"2024-04-21T23:07:24.815182+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:32:32.556123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451897+0000","last_change":"2024-04-21T23:06:25.208968+0000","last_active":"2024-04-21T23:07:24.451897+0000","last_peered":"2024-04-21T23:07:24.451897+0000","last_clean":"2024-04-21T23:07:24.451897+0000","last_became_active":"2024-04-21T23:06:25.208606+0000","last_became_peered":"2024-04-21T23:06:25.208606+0000","last_unstale":"2024-04-21T23:07:24.451897+0000","last_undegraded":"2024-04-21T23:07:24.451897+0000","last_fullsized":"2024-04-21T23:07:24.451897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:31:46.147119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":363,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450577+0000","last_change":"2024-04-21T23:05:13.077846+0000","last_active":"2024-04-21T23:07:24.450577+0000","last_peered":"2024-04-21T23:07:24.450577+0000","last_clean":"2024-04-21T23:07:24.450577+0000","last_became_active":"2024-04-21T23:05:13.077436+0000","last_became_peered":"2024-04-21T23:05:13.077436+0000","last_unstale":"2024-04-21T23:07:24.450577+0000","last_undegraded":"2024-04-21T23:07:24.450577+0000","last_fullsized":"2024-04-21T23:07:24.450577+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:01:47.619212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:01:47.619212+0000","last_clean_scrub_stamp":"2024-04-21T23:01:47.619212+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:12:09.826839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.454201+0000","last_change":"2024-04-21T23:06:30.405121+0000","last_active":"2024-04-21T23:07:24.454201+0000","last_peered":"2024-04-21T23:07:24.454201+0000","last_clean":"2024-04-21T23:07:24.454201+0000","last_became_active":"2024-04-21T23:06:30.404612+0000","last_became_peered":"2024-04-21T23:06:30.404612+0000","last_unstale":"2024-04-21T23:07:24.454201+0000","last_undegraded":"2024-04-21T23:07:24.454201+0000","last_fullsized":"2024-04-21T23:07:24.454201+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:59:01.737337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450475+0000","last_change":"2024-04-21T23:06:25.997879+0000","last_active":"2024-04-21T23:07:24.450475+0000","last_peered":"2024-04-21T23:07:24.450475+0000","last_clean":"2024-04-21T23:07:24.450475+0000","last_became_active":"2024-04-21T23:06:25.997507+0000","last_became_peered":"2024-04-21T23:06:25.997507+0000","last_unstale":"2024-04-21T23:07:24.450475+0000","last_undegraded":"2024-04-21T23:07:24.450475+0000","last_fullsized":"2024-04-21T23:07:24.450475+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:13:16.044456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450760+0000","last_change":"2024-04-21T23:06:25.209714+0000","last_active":"2024-04-21T23:07:24.450760+0000","last_peered":"2024-04-21T23:07:24.450760+0000","last_clean":"2024-04-21T23:07:24.450760+0000","last_became_active":"2024-04-21T23:06:25.209373+0000","last_became_peered":"2024-04-21T23:06:25.209373+0000","last_unstale":"2024-04-21T23:07:24.450760+0000","last_undegraded":"2024-04-21T23:07:24.450760+0000","last_fullsized":"2024-04-21T23:07:24.450760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:21.654143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450079+0000","last_change":"2024-04-21T23:06:30.264873+0000","last_active":"2024-04-21T23:07:24.450079+0000","last_peered":"2024-04-21T23:07:24.450079+0000","last_clean":"2024-04-21T23:07:24.450079+0000","last_became_active":"2024-04-21T23:06:30.264318+0000","last_became_peered":"2024-04-21T23:06:30.264318+0000","last_unstale":"2024-04-21T23:07:24.450079+0000","last_undegraded":"2024-04-21T23:07:24.450079+0000","last_fullsized":"2024-04-21T23:07:24.450079+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:01:22.558782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.446874+0000","last_change":"2024-04-21T23:06:25.218767+0000","last_active":"2024-04-21T23:07:24.446874+0000","last_peered":"2024-04-21T23:07:24.446874+0000","last_clean":"2024-04-21T23:07:24.446874+0000","last_became_active":"2024-04-21T23:06:25.218278+0000","last_became_peered":"2024-04-21T23:06:25.218278+0000","last_unstale":"2024-04-21T23:07:24.446874+0000","last_undegraded":"2024-04-21T23:07:24.446874+0000","last_fullsized":"2024-04-21T23:07:24.446874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:03:05.063515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451136+0000","last_change":"2024-04-21T23:06:25.968551+0000","last_active":"2024-04-21T23:07:24.451136+0000","last_peered":"2024-04-21T23:07:24.451136+0000","last_clean":"2024-04-21T23:07:24.451136+0000","last_became_active":"2024-04-21T23:06:25.968167+0000","last_became_peered":"2024-04-21T23:06:25.968167+0000","last_unstale":"2024-04-21T23:07:24.451136+0000","last_undegraded":"2024-04-21T23:07:24.451136+0000","last_fullsized":"2024-04-21T23:07:24.451136+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:08.920677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.814872+0000","last_change":"2024-04-21T23:06:30.404811+0000","last_active":"2024-04-21T23:07:24.814872+0000","last_peered":"2024-04-21T23:07:24.814872+0000","last_clean":"2024-04-21T23:07:24.814872+0000","last_became_active":"2024-04-21T23:06:30.404375+0000","last_became_peered":"2024-04-21T23:06:30.404375+0000","last_unstale":"2024-04-21T23:07:24.814872+0000","last_undegraded":"2024-04-21T23:07:24.814872+0000","last_fullsized":"2024-04-21T23:07:24.814872+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:10:15.688917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'6","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447940+0000","last_change":"2024-04-21T23:06:25.204371+0000","last_active":"2024-04-21T23:07:24.447940+0000","last_peered":"2024-04-21T23:07:24.447940+0000","last_clean":"2024-04-21T23:07:24.447940+0000","last_became_active":"2024-04-21T23:06:25.204025+0000","last_became_peered":"2024-04-21T23:06:25.204025+0000","last_unstale":"2024-04-21T23:07:24.447940+0000","last_undegraded":"2024-04-21T23:07:24.447940+0000","last_fullsized":"2024-04-21T23:07:24.447940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:34:31.071013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815355+0000","last_change":"2024-04-21T23:06:25.990800+0000","last_active":"2024-04-21T23:07:24.815355+0000","last_peered":"2024-04-21T23:07:24.815355+0000","last_clean":"2024-04-21T23:07:24.815355+0000","last_became_active":"2024-04-21T23:06:25.990477+0000","last_became_peered":"2024-04-21T23:06:25.990477+0000","last_unstale":"2024-04-21T23:07:24.815355+0000","last_undegraded":"2024-04-21T23:07:24.815355+0000","last_fullsized":"2024-04-21T23:07:24.815355+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:55:33.282303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450510+0000","last_change":"2024-04-21T23:06:30.260090+0000","last_active":"2024-04-21T23:07:24.450510+0000","last_peered":"2024-04-21T23:07:24.450510+0000","last_clean":"2024-04-21T23:07:24.450510+0000","last_became_active":"2024-04-21T23:06:30.259609+0000","last_became_peered":"2024-04-21T23:06:30.259609+0000","last_unstale":"2024-04-21T23:07:24.450510+0000","last_undegraded":"2024-04-21T23:07:24.450510+0000","last_fullsized":"2024-04-21T23:07:24.450510+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:11:17.933683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452284+0000","last_change":"2024-04-21T23:06:25.215446+0000","last_active":"2024-04-21T23:07:24.452284+0000","last_peered":"2024-04-21T23:07:24.452284+0000","last_clean":"2024-04-21T23:07:24.452284+0000","last_became_active":"2024-04-21T23:06:25.214773+0000","last_became_peered":"2024-04-21T23:06:25.214773+0000","last_unstale":"2024-04-21T23:07:24.452284+0000","last_undegraded":"2024-04-21T23:07:24.452284+0000","last_fullsized":"2024-04-21T23:07:24.452284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:25:01.090699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451547+0000","last_change":"2024-04-21T23:06:25.986640+0000","last_active":"2024-04-21T23:07:24.451547+0000","last_peered":"2024-04-21T23:07:24.451547+0000","last_clean":"2024-04-21T23:07:24.451547+0000","last_became_active":"2024-04-21T23:06:25.986285+0000","last_became_peered":"2024-04-21T23:06:25.986285+0000","last_unstale":"2024-04-21T23:07:24.451547+0000","last_undegraded":"2024-04-21T23:07:24.451547+0000","last_fullsized":"2024-04-21T23:07:24.451547+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:01:05.498199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.458291+0000","last_change":"2024-04-21T23:07:24.458291+0000","last_active":"2024-04-21T23:07:24.458291+0000","last_peered":"2024-04-21T23:07:24.458291+0000","last_clean":"2024-04-21T23:07:24.458291+0000","last_became_active":"2024-04-21T23:07:24.457677+0000","last_became_peered":"2024-04-21T23:07:24.457677+0000","last_unstale":"2024-04-21T23:07:24.458291+0000","last_undegraded":"2024-04-21T23:07:24.458291+0000","last_fullsized":"2024-04-21T23:07:24.458291+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:20:30.708901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,11],"acting":[4,8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815415+0000","last_change":"2024-04-21T23:06:25.217298+0000","last_active":"2024-04-21T23:07:24.815415+0000","last_peered":"2024-04-21T23:07:24.815415+0000","last_clean":"2024-04-21T23:07:24.815415+0000","last_became_active":"2024-04-21T23:06:25.216965+0000","last_became_peered":"2024-04-21T23:06:25.216965+0000","last_unstale":"2024-04-21T23:07:24.815415+0000","last_undegraded":"2024-04-21T23:07:24.815415+0000","last_fullsized":"2024-04-21T23:07:24.815415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:56:38.543697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.446128+0000","last_change":"2024-04-21T23:06:25.981400+0000","last_active":"2024-04-21T23:07:24.446128+0000","last_peered":"2024-04-21T23:07:24.446128+0000","last_clean":"2024-04-21T23:07:24.446128+0000","last_became_active":"2024-04-21T23:06:25.981118+0000","last_became_peered":"2024-04-21T23:06:25.981118+0000","last_unstale":"2024-04-21T23:07:24.446128+0000","last_undegraded":"2024-04-21T23:07:24.446128+0000","last_fullsized":"2024-04-21T23:07:24.446128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:20:59.631062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.447715+0000","last_change":"2024-04-21T23:06:30.270351+0000","last_active":"2024-04-21T23:07:24.447715+0000","last_peered":"2024-04-21T23:07:24.447715+0000","last_clean":"2024-04-21T23:07:24.447715+0000","last_became_active":"2024-04-21T23:06:30.269659+0000","last_became_peered":"2024-04-21T23:06:30.269659+0000","last_unstale":"2024-04-21T23:07:24.447715+0000","last_undegraded":"2024-04-21T23:07:24.447715+0000","last_fullsized":"2024-04-21T23:07:24.447715+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:19:14.132191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.814984+0000","last_change":"2024-04-21T23:06:25.212787+0000","last_active":"2024-04-21T23:07:24.814984+0000","last_peered":"2024-04-21T23:07:24.814984+0000","last_clean":"2024-04-21T23:07:24.814984+0000","last_became_active":"2024-04-21T23:06:25.212423+0000","last_became_peered":"2024-04-21T23:06:25.212423+0000","last_unstale":"2024-04-21T23:07:24.814984+0000","last_undegraded":"2024-04-21T23:07:24.814984+0000","last_fullsized":"2024-04-21T23:07:24.814984+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:45.875664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.448604+0000","last_change":"2024-04-21T23:06:25.978205+0000","last_active":"2024-04-21T23:07:24.448604+0000","last_peered":"2024-04-21T23:07:24.448604+0000","last_clean":"2024-04-21T23:07:24.448604+0000","last_became_active":"2024-04-21T23:06:25.977876+0000","last_became_peered":"2024-04-21T23:06:25.977876+0000","last_unstale":"2024-04-21T23:07:24.448604+0000","last_undegraded":"2024-04-21T23:07:24.448604+0000","last_fullsized":"2024-04-21T23:07:24.448604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:43:29.280255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451607+0000","last_change":"2024-04-21T23:06:30.271928+0000","last_active":"2024-04-21T23:07:24.451607+0000","last_peered":"2024-04-21T23:07:24.451607+0000","last_clean":"2024-04-21T23:07:24.451607+0000","last_became_active":"2024-04-21T23:06:30.271321+0000","last_became_peered":"2024-04-21T23:06:30.271321+0000","last_unstale":"2024-04-21T23:07:24.451607+0000","last_undegraded":"2024-04-21T23:07:24.451607+0000","last_fullsized":"2024-04-21T23:07:24.451607+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:22:03.343620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450814+0000","last_change":"2024-04-21T23:06:25.204175+0000","last_active":"2024-04-21T23:07:24.450814+0000","last_peered":"2024-04-21T23:07:24.450814+0000","last_clean":"2024-04-21T23:07:24.450814+0000","last_became_active":"2024-04-21T23:06:25.203826+0000","last_became_peered":"2024-04-21T23:06:25.203826+0000","last_unstale":"2024-04-21T23:07:24.450814+0000","last_undegraded":"2024-04-21T23:07:24.450814+0000","last_fullsized":"2024-04-21T23:07:24.450814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:28:53.402395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815164+0000","last_change":"2024-04-21T23:06:25.990559+0000","last_active":"2024-04-21T23:07:24.815164+0000","last_peered":"2024-04-21T23:07:24.815164+0000","last_clean":"2024-04-21T23:07:24.815164+0000","last_became_active":"2024-04-21T23:06:25.990239+0000","last_became_peered":"2024-04-21T23:06:25.990239+0000","last_unstale":"2024-04-21T23:07:24.815164+0000","last_undegraded":"2024-04-21T23:07:24.815164+0000","last_fullsized":"2024-04-21T23:07:24.815164+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:49:26.891190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452205+0000","last_change":"2024-04-21T23:06:30.406584+0000","last_active":"2024-04-21T23:07:24.452205+0000","last_peered":"2024-04-21T23:07:24.452205+0000","last_clean":"2024-04-21T23:07:24.452205+0000","last_became_active":"2024-04-21T23:06:30.404576+0000","last_became_peered":"2024-04-21T23:06:30.404576+0000","last_unstale":"2024-04-21T23:07:24.452205+0000","last_undegraded":"2024-04-21T23:07:24.452205+0000","last_fullsized":"2024-04-21T23:07:24.452205+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:56.047199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.460921+0000","last_change":"2024-04-21T23:07:24.460375+0000","last_active":"2024-04-21T23:07:24.460921+0000","last_peered":"2024-04-21T23:07:24.460921+0000","last_clean":"2024-04-21T23:07:24.460921+0000","last_became_active":"2024-04-21T23:07:24.453873+0000","last_became_peered":"2024-04-21T23:07:24.453873+0000","last_unstale":"2024-04-21T23:07:24.460921+0000","last_undegraded":"2024-04-21T23:07:24.460921+0000","last_fullsized":"2024-04-21T23:07:24.460921+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:27.709414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450647+0000","last_change":"2024-04-21T23:06:25.988925+0000","last_active":"2024-04-21T23:07:24.450647+0000","last_peered":"2024-04-21T23:07:24.450647+0000","last_clean":"2024-04-21T23:07:24.450647+0000","last_became_active":"2024-04-21T23:06:25.988480+0000","last_became_peered":"2024-04-21T23:06:25.988480+0000","last_unstale":"2024-04-21T23:07:24.450647+0000","last_undegraded":"2024-04-21T23:07:24.450647+0000","last_fullsized":"2024-04-21T23:07:24.450647+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:32:02.867858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815558+0000","last_change":"2024-04-21T23:06:30.260841+0000","last_active":"2024-04-21T23:07:24.815558+0000","last_peered":"2024-04-21T23:07:24.815558+0000","last_clean":"2024-04-21T23:07:24.815558+0000","last_became_active":"2024-04-21T23:06:30.260376+0000","last_became_peered":"2024-04-21T23:06:30.260376+0000","last_unstale":"2024-04-21T23:07:24.815558+0000","last_undegraded":"2024-04-21T23:07:24.815558+0000","last_fullsized":"2024-04-21T23:07:24.815558+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:24:57.429169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452096+0000","last_change":"2024-04-21T23:06:25.217920+0000","last_active":"2024-04-21T23:07:24.452096+0000","last_peered":"2024-04-21T23:07:24.452096+0000","last_clean":"2024-04-21T23:07:24.452096+0000","last_became_active":"2024-04-21T23:06:25.217379+0000","last_became_peered":"2024-04-21T23:06:25.217379+0000","last_unstale":"2024-04-21T23:07:24.452096+0000","last_undegraded":"2024-04-21T23:07:24.452096+0000","last_fullsized":"2024-04-21T23:07:24.452096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:09.570463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452106+0000","last_change":"2024-04-21T23:06:25.993533+0000","last_active":"2024-04-21T23:07:24.452106+0000","last_peered":"2024-04-21T23:07:24.452106+0000","last_clean":"2024-04-21T23:07:24.452106+0000","last_became_active":"2024-04-21T23:06:25.989739+0000","last_became_peered":"2024-04-21T23:06:25.989739+0000","last_unstale":"2024-04-21T23:07:24.452106+0000","last_undegraded":"2024-04-21T23:07:24.452106+0000","last_fullsized":"2024-04-21T23:07:24.452106+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:24:05.594026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451828+0000","last_change":"2024-04-21T23:06:30.276121+0000","last_active":"2024-04-21T23:07:24.451828+0000","last_peered":"2024-04-21T23:07:24.451828+0000","last_clean":"2024-04-21T23:07:24.451828+0000","last_became_active":"2024-04-21T23:06:30.275650+0000","last_became_peered":"2024-04-21T23:06:30.275650+0000","last_unstale":"2024-04-21T23:07:24.451828+0000","last_undegraded":"2024-04-21T23:07:24.451828+0000","last_fullsized":"2024-04-21T23:07:24.451828+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:00:19.559482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.454084+0000","last_change":"2024-04-21T23:06:25.207079+0000","last_active":"2024-04-21T23:07:24.454084+0000","last_peered":"2024-04-21T23:07:24.454084+0000","last_clean":"2024-04-21T23:07:24.454084+0000","last_became_active":"2024-04-21T23:06:25.206724+0000","last_became_peered":"2024-04-21T23:06:25.206724+0000","last_unstale":"2024-04-21T23:07:24.454084+0000","last_undegraded":"2024-04-21T23:07:24.454084+0000","last_fullsized":"2024-04-21T23:07:24.454084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:27.221282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450876+0000","last_change":"2024-04-21T23:06:25.988144+0000","last_active":"2024-04-21T23:07:24.450876+0000","last_peered":"2024-04-21T23:07:24.450876+0000","last_clean":"2024-04-21T23:07:24.450876+0000","last_became_active":"2024-04-21T23:06:25.987813+0000","last_became_peered":"2024-04-21T23:06:25.987813+0000","last_unstale":"2024-04-21T23:07:24.450876+0000","last_undegraded":"2024-04-21T23:07:24.450876+0000","last_fullsized":"2024-04-21T23:07:24.450876+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:23.733514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451829+0000","last_change":"2024-04-21T23:06:30.261548+0000","last_active":"2024-04-21T23:07:24.451829+0000","last_peered":"2024-04-21T23:07:24.451829+0000","last_clean":"2024-04-21T23:07:24.451829+0000","last_became_active":"2024-04-21T23:06:30.261152+0000","last_became_peered":"2024-04-21T23:06:30.261152+0000","last_unstale":"2024-04-21T23:07:24.451829+0000","last_undegraded":"2024-04-21T23:07:24.451829+0000","last_fullsized":"2024-04-21T23:07:24.451829+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:00:16.644253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451184+0000","last_change":"2024-04-21T23:06:25.208749+0000","last_active":"2024-04-21T23:07:24.451184+0000","last_peered":"2024-04-21T23:07:24.451184+0000","last_clean":"2024-04-21T23:07:24.451184+0000","last_became_active":"2024-04-21T23:06:25.201511+0000","last_became_peered":"2024-04-21T23:06:25.201511+0000","last_unstale":"2024-04-21T23:07:24.451184+0000","last_undegraded":"2024-04-21T23:07:24.451184+0000","last_fullsized":"2024-04-21T23:07:24.451184+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:41.928835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451852+0000","last_change":"2024-04-21T23:06:25.996041+0000","last_active":"2024-04-21T23:07:24.451852+0000","last_peered":"2024-04-21T23:07:24.451852+0000","last_clean":"2024-04-21T23:07:24.451852+0000","last_became_active":"2024-04-21T23:06:25.994482+0000","last_became_peered":"2024-04-21T23:06:25.994482+0000","last_unstale":"2024-04-21T23:07:24.451852+0000","last_undegraded":"2024-04-21T23:07:24.451852+0000","last_fullsized":"2024-04-21T23:07:24.451852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:15:46.792618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.448677+0000","last_change":"2024-04-21T23:06:25.974061+0000","last_active":"2024-04-21T23:07:24.448677+0000","last_peered":"2024-04-21T23:07:24.448677+0000","last_clean":"2024-04-21T23:07:24.448677+0000","last_became_active":"2024-04-21T23:06:25.973708+0000","last_became_peered":"2024-04-21T23:06:25.973708+0000","last_unstale":"2024-04-21T23:07:24.448677+0000","last_undegraded":"2024-04-21T23:07:24.448677+0000","last_fullsized":"2024-04-21T23:07:24.448677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:58.702947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450989+0000","last_change":"2024-04-21T23:06:25.988412+0000","last_active":"2024-04-21T23:07:24.450989+0000","last_peered":"2024-04-21T23:07:24.450989+0000","last_clean":"2024-04-21T23:07:24.450989+0000","last_became_active":"2024-04-21T23:06:25.988100+0000","last_became_peered":"2024-04-21T23:06:25.988100+0000","last_unstale":"2024-04-21T23:07:24.450989+0000","last_undegraded":"2024-04-21T23:07:24.450989+0000","last_fullsized":"2024-04-21T23:07:24.450989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:16:42.626123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.450695+0000","last_change":"2024-04-21T23:06:25.980144+0000","last_active":"2024-04-21T23:07:24.450695+0000","last_peered":"2024-04-21T23:07:24.450695+0000","last_clean":"2024-04-21T23:07:24.450695+0000","last_became_active":"2024-04-21T23:06:25.979760+0000","last_became_peered":"2024-04-21T23:06:25.979760+0000","last_unstale":"2024-04-21T23:07:24.450695+0000","last_undegraded":"2024-04-21T23:07:24.450695+0000","last_fullsized":"2024-04-21T23:07:24.450695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:14:32.496184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452099+0000","last_change":"2024-04-21T23:06:25.994732+0000","last_active":"2024-04-21T23:07:24.452099+0000","last_peered":"2024-04-21T23:07:24.452099+0000","last_clean":"2024-04-21T23:07:24.452099+0000","last_became_active":"2024-04-21T23:06:25.994408+0000","last_became_peered":"2024-04-21T23:06:25.994408+0000","last_unstale":"2024-04-21T23:07:24.452099+0000","last_undegraded":"2024-04-21T23:07:24.452099+0000","last_fullsized":"2024-04-21T23:07:24.452099+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:28:59.778049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.455416+0000","last_change":"2024-04-21T23:07:24.455416+0000","last_active":"2024-04-21T23:07:24.455416+0000","last_peered":"2024-04-21T23:07:24.455416+0000","last_clean":"2024-04-21T23:07:24.455416+0000","last_became_active":"2024-04-21T23:07:24.454954+0000","last_became_peered":"2024-04-21T23:07:24.454954+0000","last_unstale":"2024-04-21T23:07:24.455416+0000","last_undegraded":"2024-04-21T23:07:24.455416+0000","last_fullsized":"2024-04-21T23:07:24.455416+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:29:09.706719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.449201+0000","last_change":"2024-04-21T23:06:25.973850+0000","last_active":"2024-04-21T23:07:24.449201+0000","last_peered":"2024-04-21T23:07:24.449201+0000","last_clean":"2024-04-21T23:07:24.449201+0000","last_became_active":"2024-04-21T23:06:25.973522+0000","last_became_peered":"2024-04-21T23:06:25.973522+0000","last_unstale":"2024-04-21T23:07:24.449201+0000","last_undegraded":"2024-04-21T23:07:24.449201+0000","last_fullsized":"2024-04-21T23:07:24.449201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:43:58.417551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451068+0000","last_change":"2024-04-21T23:06:25.978114+0000","last_active":"2024-04-21T23:07:24.451068+0000","last_peered":"2024-04-21T23:07:24.451068+0000","last_clean":"2024-04-21T23:07:24.451068+0000","last_became_active":"2024-04-21T23:06:25.977115+0000","last_became_peered":"2024-04-21T23:06:25.977115+0000","last_unstale":"2024-04-21T23:07:24.451068+0000","last_undegraded":"2024-04-21T23:07:24.451068+0000","last_fullsized":"2024-04-21T23:07:24.451068+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:26:46.756490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815527+0000","last_change":"2024-04-21T23:06:25.994451+0000","last_active":"2024-04-21T23:07:24.815527+0000","last_peered":"2024-04-21T23:07:24.815527+0000","last_clean":"2024-04-21T23:07:24.815527+0000","last_became_active":"2024-04-21T23:06:25.994085+0000","last_became_peered":"2024-04-21T23:06:25.994085+0000","last_unstale":"2024-04-21T23:07:24.815527+0000","last_undegraded":"2024-04-21T23:07:24.815527+0000","last_fullsized":"2024-04-21T23:07:24.815527+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:53.652014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.452244+0000","last_change":"2024-04-21T23:06:25.982536+0000","last_active":"2024-04-21T23:07:24.452244+0000","last_peered":"2024-04-21T23:07:24.452244+0000","last_clean":"2024-04-21T23:07:24.452244+0000","last_became_active":"2024-04-21T23:06:25.981318+0000","last_became_peered":"2024-04-21T23:06:25.981318+0000","last_unstale":"2024-04-21T23:07:24.452244+0000","last_undegraded":"2024-04-21T23:07:24.452244+0000","last_fullsized":"2024-04-21T23:07:24.452244+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:35.784679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815105+0000","last_change":"2024-04-21T23:06:25.981915+0000","last_active":"2024-04-21T23:07:24.815105+0000","last_peered":"2024-04-21T23:07:24.815105+0000","last_clean":"2024-04-21T23:07:24.815105+0000","last_became_active":"2024-04-21T23:06:25.981547+0000","last_became_peered":"2024-04-21T23:06:25.981547+0000","last_unstale":"2024-04-21T23:07:24.815105+0000","last_undegraded":"2024-04-21T23:07:24.815105+0000","last_fullsized":"2024-04-21T23:07:24.815105+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:03:11.154582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.815749+0000","last_change":"2024-04-21T23:06:25.991687+0000","last_active":"2024-04-21T23:07:24.815749+0000","last_peered":"2024-04-21T23:07:24.815749+0000","last_clean":"2024-04-21T23:07:24.815749+0000","last_became_active":"2024-04-21T23:06:25.991248+0000","last_became_peered":"2024-04-21T23:06:25.991248+0000","last_unstale":"2024-04-21T23:07:24.815749+0000","last_undegraded":"2024-04-21T23:07:24.815749+0000","last_fullsized":"2024-04-21T23:07:24.815749+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:05:04.205660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.451181+0000","last_change":"2024-04-21T23:06:25.993658+0000","last_active":"2024-04-21T23:07:24.451181+0000","last_peered":"2024-04-21T23:07:24.451181+0000","last_clean":"2024-04-21T23:07:24.451181+0000","last_became_active":"2024-04-21T23:06:25.993298+0000","last_became_peered":"2024-04-21T23:06:25.993298+0000","last_unstale":"2024-04-21T23:07:24.451181+0000","last_undegraded":"2024-04-21T23:07:24.451181+0000","last_fullsized":"2024-04-21T23:07:24.451181+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:18:30.216961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-21T23:07:24.448778+0000","last_change":"2024-04-21T23:06:25.993849+0000","last_active":"2024-04-21T23:07:24.448778+0000","last_peered":"2024-04-21T23:07:24.448778+0000","last_clean":"2024-04-21T23:07:24.448778+0000","last_became_active":"2024-04-21T23:06:25.993501+0000","last_became_peered":"2024-04-21T23:06:25.993501+0000","last_unstale":"2024-04-21T23:07:24.448778+0000","last_undegraded":"2024-04-21T23:07:24.448778+0000","last_fullsized":"2024-04-21T23:07:24.448778+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:40:10.461775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63640,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":158,"num_read_kb":143,"num_write":80,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":190920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351010,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1036288,"data_stored":506855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612647,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32852,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712300,"statfs":{"total":95995035648,"available":95961395200,"internally_reserved":0,"allocated":1708032,"data_stored":1156141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808875,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":511511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070514,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1036288,"data_stored":507415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332149,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28620,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716532,"statfs":{"total":95995035648,"available":95965728768,"internally_reserved":0,"allocated":1634304,"data_stored":1101879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495675,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":502737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1101824,"data_stored":565579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":502827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":565613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":502841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920837,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":506855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757312,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28620,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716532,"statfs":{"total":95995035648,"available":95965728768,"internally_reserved":0,"allocated":1638400,"data_stored":1097383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":63036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T23:08:09.446 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-21T23:08:09.446 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-21T23:08:09.446 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-21T23:08:09.446 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-21T23:08:09.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:09 smithi044 ceph-mon[27934]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:08:09.696 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:08:09.865 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:09 smithi055 ceph-mon[31117]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:08:09.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:09 smithi069 ceph-mon[31816]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:08:10.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:10 smithi044 ceph-mon[27934]: pgmap v345: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:10.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:10 smithi069 ceph-mon[31816]: pgmap v345: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:10.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:10 smithi055 ceph-mon[31117]: pgmap v345: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:11.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.505+0000 7f5bf481f700 1 -- 172.21.15.44:0/4139496032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bf006b5f0 msgr2=0x7f5bf0106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:11.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.505+0000 7f5bf481f700 1 --2- 172.21.15.44:0/4139496032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bf006b5f0 0x7f5bf0106eb0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80099c0 tx=0x7f5bd8009cd0 comp rx=0 tx=0).stop 2024-04-21T23:08:11.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.505+0000 7f5bf481f700 1 -- 172.21.15.44:0/4139496032 shutdown_connections 2024-04-21T23:08:11.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.505+0000 7f5bf481f700 1 --2- 172.21.15.44:0/4139496032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bf0107590 0x7f5bf010f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:11.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.505+0000 7f5bf481f700 1 --2- 172.21.15.44:0/4139496032 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5bf006ac40 0x7f5bf006b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:11.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.505+0000 7f5bf481f700 1 --2- 172.21.15.44:0/4139496032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bf006b5f0 0x7f5bf0106eb0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:11.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.505+0000 7f5bf481f700 1 -- 172.21.15.44:0/4139496032 >> 172.21.15.44:0/4139496032 conn(0x7f5bf00781d0 msgr2=0x7f5bf00785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:11.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.505+0000 7f5bf481f700 1 -- 172.21.15.44:0/4139496032 shutdown_connections 2024-04-21T23:08:11.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.505+0000 7f5bf481f700 1 -- 172.21.15.44:0/4139496032 wait complete. 2024-04-21T23:08:11.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.506+0000 7f5bf481f700 1 Processor -- start 2024-04-21T23:08:11.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.506+0000 7f5bf481f700 1 -- start start 2024-04-21T23:08:11.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.506+0000 7f5bf481f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5bf006ac40 0x7f5bf01011a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:11.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.506+0000 7f5bf481f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bf006b5f0 0x7f5bf01016e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:11.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.506+0000 7f5bf481f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bf0107590 0x7f5bf0101c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:11.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.506+0000 7f5bf481f700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf0112550 con 0x7f5bf006b5f0 2024-04-21T23:08:11.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.506+0000 7f5bf481f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf01123d0 con 0x7f5bf006ac40 2024-04-21T23:08:11.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.506+0000 7f5bf481f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf01126d0 con 0x7f5bf0107590 2024-04-21T23:08:11.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.506+0000 7f5be5d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bf006b5f0 0x7f5bf01016e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:11.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.507+0000 7f5be5d9b700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bf006b5f0 0x7f5bf01016e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40338/0 (socket says 172.21.15.44:40338) 2024-04-21T23:08:11.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.507+0000 7f5be5d9b700 1 -- 172.21.15.44:0/2115568903 learned_addr learned my addr 172.21.15.44:0/2115568903 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:11.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.507+0000 7f5bee59c700 1 --2- 172.21.15.44:0/2115568903 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5bf006ac40 0x7f5bf01011a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:11.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.507+0000 7f5beed9d700 1 --2- 172.21.15.44:0/2115568903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bf0107590 0x7f5bf0101c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:11.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.507+0000 7f5be5d9b700 1 -- 172.21.15.44:0/2115568903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bf0107590 msgr2=0x7f5bf0101c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:11.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.507+0000 7f5be5d9b700 1 --2- 172.21.15.44:0/2115568903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bf0107590 0x7f5bf0101c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:11.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.507+0000 7f5be5d9b700 1 -- 172.21.15.44:0/2115568903 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5bf006ac40 msgr2=0x7f5bf01011a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:11.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.507+0000 7f5be5d9b700 1 --2- 172.21.15.44:0/2115568903 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5bf006ac40 0x7f5bf01011a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:11.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.507+0000 7f5be5d9b700 1 -- 172.21.15.44:0/2115568903 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bf01a6f10 con 0x7f5bf006b5f0 2024-04-21T23:08:11.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.507+0000 7f5be5d9b700 1 --2- 172.21.15.44:0/2115568903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bf006b5f0 0x7f5bf01016e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8009fd0 tx=0x7f5bd8005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:11.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.507+0000 7f5be7fff700 1 -- 172.21.15.44:0/2115568903 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8026070 con 0x7f5bf006b5f0 2024-04-21T23:08:11.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.507+0000 7f5be7fff700 1 -- 172.21.15.44:0/2115568903 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bd8004cd0 con 0x7f5bf006b5f0 2024-04-21T23:08:11.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.507+0000 7f5bf481f700 1 -- 172.21.15.44:0/2115568903 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bf01a71a0 con 0x7f5bf006b5f0 2024-04-21T23:08:11.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.507+0000 7f5be7fff700 1 -- 172.21.15.44:0/2115568903 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8003dd0 con 0x7f5bf006b5f0 2024-04-21T23:08:11.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.507+0000 7f5bf481f700 1 -- 172.21.15.44:0/2115568903 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bf01a75f0 con 0x7f5bf006b5f0 2024-04-21T23:08:11.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.508+0000 7f5bf481f700 1 -- 172.21.15.44:0/2115568903 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bf0068f20 con 0x7f5bf006b5f0 2024-04-21T23:08:11.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.510+0000 7f5be7fff700 1 -- 172.21.15.44:0/2115568903 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5bd8029070 con 0x7f5bf006b5f0 2024-04-21T23:08:11.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.510+0000 7f5be7fff700 1 --2- 172.21.15.44:0/2115568903 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5bdc0a2cb0 0x7f5bdc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:11.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.510+0000 7f5be7fff700 1 -- 172.21.15.44:0/2115568903 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f5bd80d0300 con 0x7f5bf006b5f0 2024-04-21T23:08:11.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.510+0000 7f5bee59c700 1 --2- 172.21.15.44:0/2115568903 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5bdc0a2cb0 0x7f5bdc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:11.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.511+0000 7f5bee59c700 1 --2- 172.21.15.44:0/2115568903 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5bdc0a2cb0 0x7f5bdc0a5170 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5be000ac50 tx=0x7f5be000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:11.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.514+0000 7f5be7fff700 1 -- 172.21.15.44:0/2115568903 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5bd8030020 con 0x7f5bf006b5f0 2024-04-21T23:08:11.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.767+0000 7f5bf481f700 1 -- 172.21.15.44:0/2115568903 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5bf002dcf0 con 0x7f5bf006b5f0 2024-04-21T23:08:11.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.768+0000 7f5be7fff700 1 -- 172.21.15.44:0/2115568903 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f5bd8013070 con 0x7f5bf006b5f0 2024-04-21T23:08:11.768 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:11.768 INFO:teuthology.orchestra.run.smithi044.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-21T23:08:11.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.770+0000 7f5bf481f700 1 -- 172.21.15.44:0/2115568903 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5bdc0a2cb0 msgr2=0x7f5bdc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:11.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.770+0000 7f5bf481f700 1 --2- 172.21.15.44:0/2115568903 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5bdc0a2cb0 0x7f5bdc0a5170 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5be000ac50 tx=0x7f5be000a380 comp rx=0 tx=0).stop 2024-04-21T23:08:11.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.770+0000 7f5bf481f700 1 -- 172.21.15.44:0/2115568903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bf006b5f0 msgr2=0x7f5bf01016e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:11.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.770+0000 7f5bf481f700 1 --2- 172.21.15.44:0/2115568903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bf006b5f0 0x7f5bf01016e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8009fd0 tx=0x7f5bd8005f10 comp rx=0 tx=0).stop 2024-04-21T23:08:11.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.770+0000 7f5bf481f700 1 -- 172.21.15.44:0/2115568903 shutdown_connections 2024-04-21T23:08:11.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.770+0000 7f5bf481f700 1 --2- 172.21.15.44:0/2115568903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bf0107590 0x7f5bf0101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:11.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.770+0000 7f5bf481f700 1 --2- 172.21.15.44:0/2115568903 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5bf006ac40 0x7f5bf01011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:11.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.770+0000 7f5bf481f700 1 --2- 172.21.15.44:0/2115568903 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5bdc0a2cb0 0x7f5bdc0a5170 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:11.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.770+0000 7f5bf481f700 1 --2- 172.21.15.44:0/2115568903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5bf006b5f0 0x7f5bf01016e0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:11.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.770+0000 7f5bf481f700 1 -- 172.21.15.44:0/2115568903 >> 172.21.15.44:0/2115568903 conn(0x7f5bf00781d0 msgr2=0x7f5bf00ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:11.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.770+0000 7f5bf481f700 1 -- 172.21.15.44:0/2115568903 shutdown_connections 2024-04-21T23:08:11.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:11.770+0000 7f5bf481f700 1 -- 172.21.15.44:0/2115568903 wait complete. 2024-04-21T23:08:12.516 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-21T23:08:12.516 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:08:12.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.626+0000 7f5d09f1e640 1 -- 172.21.15.44:0/4203532673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d041006a0 msgr2=0x7f5d04100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:12.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.626+0000 7f5d09f1e640 1 --2- 172.21.15.44:0/4203532673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d041006a0 0x7f5d04100ac0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4009900 tx=0x7f5cf402bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:12.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.626+0000 7f5d09f1e640 1 -- 172.21.15.44:0/4203532673 shutdown_connections 2024-04-21T23:08:12.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.626+0000 7f5d09f1e640 1 --2- 172.21.15.44:0/4203532673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d04101890 0x7f5d04101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:12.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.626+0000 7f5d09f1e640 1 --2- 172.21.15.44:0/4203532673 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d04102250 0x7f5d04108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:12.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.626+0000 7f5d09f1e640 1 --2- 172.21.15.44:0/4203532673 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d041006a0 0x7f5d04100ac0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:12.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.626+0000 7f5d09f1e640 1 -- 172.21.15.44:0/4203532673 >> 172.21.15.44:0/4203532673 conn(0x7f5d040fbc50 msgr2=0x7f5d040fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:12.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.626+0000 7f5d09f1e640 1 -- 172.21.15.44:0/4203532673 shutdown_connections 2024-04-21T23:08:12.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.626+0000 7f5d09f1e640 1 -- 172.21.15.44:0/4203532673 wait complete. 2024-04-21T23:08:12.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.627+0000 7f5d09f1e640 1 Processor -- start 2024-04-21T23:08:12.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.627+0000 7f5d09f1e640 1 -- start start 2024-04-21T23:08:12.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.627+0000 7f5d09f1e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d041006a0 0x7f5d0410b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:12.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.627+0000 7f5d09f1e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d04101890 0x7f5d04109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:12.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.627+0000 7f5d09f1e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d04102250 0x7f5d04109ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:12.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.627+0000 7f5d09f1e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0410d4a0 con 0x7f5d04102250 2024-04-21T23:08:12.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.627+0000 7f5d09f1e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0410d320 con 0x7f5d041006a0 2024-04-21T23:08:12.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.627+0000 7f5d09f1e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0410d620 con 0x7f5d04101890 2024-04-21T23:08:12.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.627+0000 7f5d03fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d04102250 0x7f5d04109ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:12.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.627+0000 7f5d03fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d04102250 0x7f5d04109ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40358/0 (socket says 172.21.15.44:40358) 2024-04-21T23:08:12.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.627+0000 7f5d03fff640 1 -- 172.21.15.44:0/3114446130 learned_addr learned my addr 172.21.15.44:0/3114446130 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:12.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.627+0000 7f5d02ffd640 1 --2- 172.21.15.44:0/3114446130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d04101890 0x7f5d04109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:12.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.627+0000 7f5d037fe640 1 --2- 172.21.15.44:0/3114446130 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d041006a0 0x7f5d0410b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:12.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.628+0000 7f5d03fff640 1 -- 172.21.15.44:0/3114446130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d04101890 msgr2=0x7f5d04109660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:12.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.628+0000 7f5d03fff640 1 --2- 172.21.15.44:0/3114446130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d04101890 0x7f5d04109660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:12.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.628+0000 7f5d03fff640 1 -- 172.21.15.44:0/3114446130 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d041006a0 msgr2=0x7f5d0410b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:12.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.628+0000 7f5d03fff640 1 --2- 172.21.15.44:0/3114446130 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d041006a0 0x7f5d0410b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:12.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.628+0000 7f5d03fff640 1 -- 172.21.15.44:0/3114446130 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d0410a430 con 0x7f5d04102250 2024-04-21T23:08:12.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.628+0000 7f5d03fff640 1 --2- 172.21.15.44:0/3114446130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d04102250 0x7f5d04109ba0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00eaa0 tx=0x7f5cfc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:12.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.628+0000 7f5d00ff9640 1 -- 172.21.15.44:0/3114446130 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc00cdb0 con 0x7f5d04102250 2024-04-21T23:08:12.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.628+0000 7f5d00ff9640 1 -- 172.21.15.44:0/3114446130 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cfc005020 con 0x7f5d04102250 2024-04-21T23:08:12.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.628+0000 7f5d09f1e640 1 -- 172.21.15.44:0/3114446130 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d041a1c70 con 0x7f5d04102250 2024-04-21T23:08:12.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.628+0000 7f5d00ff9640 1 -- 172.21.15.44:0/3114446130 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc0116b0 con 0x7f5d04102250 2024-04-21T23:08:12.630 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.628+0000 7f5d09f1e640 1 -- 172.21.15.44:0/3114446130 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d041a2170 con 0x7f5d04102250 2024-04-21T23:08:12.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.629+0000 7f5d09f1e640 1 -- 172.21.15.44:0/3114446130 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d041079c0 con 0x7f5d04102250 2024-04-21T23:08:12.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.630+0000 7f5d00ff9640 1 -- 172.21.15.44:0/3114446130 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5cfc005560 con 0x7f5d04102250 2024-04-21T23:08:12.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.631+0000 7f5d00ff9640 1 --2- 172.21.15.44:0/3114446130 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5cec0a2c20 0x7f5cec0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:12.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.631+0000 7f5d037fe640 1 --2- 172.21.15.44:0/3114446130 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5cec0a2c20 0x7f5cec0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:12.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.631+0000 7f5d00ff9640 1 -- 172.21.15.44:0/3114446130 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f5cfc0cf750 con 0x7f5d04102250 2024-04-21T23:08:12.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.631+0000 7f5d037fe640 1 --2- 172.21.15.44:0/3114446130 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5cec0a2c20 0x7f5cec0a50e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5cf402c170 tx=0x7f5cf402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:12.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.635+0000 7f5d00ff9640 1 -- 172.21.15.44:0/3114446130 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5cfc016030 con 0x7f5d04102250 2024-04-21T23:08:12.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.823+0000 7f5d09f1e640 1 -- 172.21.15.44:0/3114446130 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d0402d3c0 con 0x7f5d04102250 2024-04-21T23:08:12.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.825+0000 7f5d00ff9640 1 -- 172.21.15.44:0/3114446130 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24530 (secure 0 0 0) 0x7f5cfc09bbc0 con 0x7f5d04102250 2024-04-21T23:08:12.825 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:12.825 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":92,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:07:24.439878+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":9,"to":11},{"from":5,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:08:12.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.827+0000 7f5d09f1e640 1 -- 172.21.15.44:0/3114446130 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5cec0a2c20 msgr2=0x7f5cec0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:12.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.827+0000 7f5d09f1e640 1 --2- 172.21.15.44:0/3114446130 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5cec0a2c20 0x7f5cec0a50e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5cf402c170 tx=0x7f5cf402df90 comp rx=0 tx=0).stop 2024-04-21T23:08:12.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.827+0000 7f5d09f1e640 1 -- 172.21.15.44:0/3114446130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d04102250 msgr2=0x7f5d04109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:12.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.827+0000 7f5d09f1e640 1 --2- 172.21.15.44:0/3114446130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d04102250 0x7f5d04109ba0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00eaa0 tx=0x7f5cfc00ef60 comp rx=0 tx=0).stop 2024-04-21T23:08:12.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.827+0000 7f5d09f1e640 1 -- 172.21.15.44:0/3114446130 shutdown_connections 2024-04-21T23:08:12.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.827+0000 7f5d09f1e640 1 --2- 172.21.15.44:0/3114446130 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5cec0a2c20 0x7f5cec0a50e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:12.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.827+0000 7f5d09f1e640 1 --2- 172.21.15.44:0/3114446130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d04101890 0x7f5d04109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:12.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.827+0000 7f5d09f1e640 1 --2- 172.21.15.44:0/3114446130 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d04102250 0x7f5d04109ba0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:12.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.827+0000 7f5d09f1e640 1 --2- 172.21.15.44:0/3114446130 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d041006a0 0x7f5d0410b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:12.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.827+0000 7f5d09f1e640 1 -- 172.21.15.44:0/3114446130 >> 172.21.15.44:0/3114446130 conn(0x7f5d040fbc50 msgr2=0x7f5d04104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:12.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.827+0000 7f5d09f1e640 1 -- 172.21.15.44:0/3114446130 shutdown_connections 2024-04-21T23:08:12.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.827+0000 7f5d09f1e640 1 -- 172.21.15.44:0/3114446130 wait complete. 2024-04-21T23:08:12.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:12 smithi044 ceph-mon[27934]: pgmap v346: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:08:12.837 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:12 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2115568903' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-21T23:08:12.838 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:01:47.507503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:06:23.993760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:06:24.954661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:06:28.677618+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:08:12.839 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:08:12.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.944+0000 7f942de4e640 1 -- 172.21.15.44:0/446702207 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9428100680 msgr2=0x7f9428100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:12.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.944+0000 7f942de4e640 1 --2- 172.21.15.44:0/446702207 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9428100680 0x7f9428100aa0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9418009900 tx=0x7f941802bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:12.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.944+0000 7f942de4e640 1 -- 172.21.15.44:0/446702207 shutdown_connections 2024-04-21T23:08:12.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.944+0000 7f942de4e640 1 --2- 172.21.15.44:0/446702207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9428101870 0x7f9428101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:12.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.944+0000 7f942de4e640 1 --2- 172.21.15.44:0/446702207 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9428102230 0x7f9428108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:12.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.944+0000 7f942de4e640 1 --2- 172.21.15.44:0/446702207 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9428100680 0x7f9428100aa0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:12.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.944+0000 7f942de4e640 1 -- 172.21.15.44:0/446702207 >> 172.21.15.44:0/446702207 conn(0x7f94280fbbd0 msgr2=0x7f94280fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:12.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.944+0000 7f942de4e640 1 -- 172.21.15.44:0/446702207 shutdown_connections 2024-04-21T23:08:12.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.944+0000 7f942de4e640 1 -- 172.21.15.44:0/446702207 wait complete. 2024-04-21T23:08:12.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.945+0000 7f942de4e640 1 Processor -- start 2024-04-21T23:08:12.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.945+0000 7f942de4e640 1 -- start start 2024-04-21T23:08:12.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.945+0000 7f942de4e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9428100680 0x7f9428197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:12.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.945+0000 7f942de4e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9428101870 0x7f9428198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:12.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.945+0000 7f942de4e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9428102230 0x7f942819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.945+0000 7f942de4e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f942810c300 con 0x7f9428100680 2024-04-21T23:08:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.945+0000 7f942de4e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f942810c180 con 0x7f9428101870 2024-04-21T23:08:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.945+0000 7f942de4e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f942810c480 con 0x7f9428102230 2024-04-21T23:08:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.945+0000 7f94277fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9428100680 0x7f9428197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.945+0000 7f94277fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9428100680 0x7f9428197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40370/0 (socket says 172.21.15.44:40370) 2024-04-21T23:08:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.945+0000 7f94277fe640 1 -- 172.21.15.44:0/2625509735 learned_addr learned my addr 172.21.15.44:0/2625509735 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.945+0000 7f9426ffd640 1 --2- 172.21.15.44:0/2625509735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9428101870 0x7f9428198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.945+0000 7f9427fff640 1 --2- 172.21.15.44:0/2625509735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9428102230 0x7f942819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.946+0000 7f94277fe640 1 -- 172.21.15.44:0/2625509735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9428102230 msgr2=0x7f942819f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.946+0000 7f94277fe640 1 --2- 172.21.15.44:0/2625509735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9428102230 0x7f942819f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.946+0000 7f94277fe640 1 -- 172.21.15.44:0/2625509735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9428101870 msgr2=0x7f9428198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.946+0000 7f94277fe640 1 --2- 172.21.15.44:0/2625509735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9428101870 0x7f9428198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:12.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.946+0000 7f94277fe640 1 -- 172.21.15.44:0/2625509735 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942819f720 con 0x7f9428100680 2024-04-21T23:08:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.946+0000 7f94277fe640 1 --2- 172.21.15.44:0/2625509735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9428100680 0x7f9428197b00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9418000c00 tx=0x7f9418005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.946+0000 7f9424ff9640 1 -- 172.21.15.44:0/2625509735 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418041070 con 0x7f9428100680 2024-04-21T23:08:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.946+0000 7f9424ff9640 1 -- 172.21.15.44:0/2625509735 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94180044d0 con 0x7f9428100680 2024-04-21T23:08:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.946+0000 7f942de4e640 1 -- 172.21.15.44:0/2625509735 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f942819f9b0 con 0x7f9428100680 2024-04-21T23:08:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.946+0000 7f9424ff9640 1 -- 172.21.15.44:0/2625509735 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418034440 con 0x7f9428100680 2024-04-21T23:08:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.946+0000 7f942de4e640 1 -- 172.21.15.44:0/2625509735 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f942819fea0 con 0x7f9428100680 2024-04-21T23:08:12.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.947+0000 7f942de4e640 1 -- 172.21.15.44:0/2625509735 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94281a0070 con 0x7f9428100680 2024-04-21T23:08:12.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.948+0000 7f9424ff9640 1 -- 172.21.15.44:0/2625509735 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9418044070 con 0x7f9428100680 2024-04-21T23:08:12.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.949+0000 7f9424ff9640 1 --2- 172.21.15.44:0/2625509735 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f94100a2c20 0x7f94100a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:12.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.949+0000 7f9424ff9640 1 -- 172.21.15.44:0/2625509735 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f94180f1180 con 0x7f9428100680 2024-04-21T23:08:12.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.949+0000 7f9426ffd640 1 --2- 172.21.15.44:0/2625509735 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f94100a2c20 0x7f94100a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:12.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.949+0000 7f9426ffd640 1 --2- 172.21.15.44:0/2625509735 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f94100a2c20 0x7f94100a50e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9428199010 tx=0x7f941c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:12.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:12.952+0000 7f9424ff9640 1 -- 172.21.15.44:0/2625509735 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f94180bdd50 con 0x7f9428100680 2024-04-21T23:08:12.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:12 smithi055 ceph-mon[31117]: pgmap v346: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:08:12.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:12 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2115568903' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-21T23:08:12.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:12 smithi069 ceph-mon[31816]: pgmap v346: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:08:12.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:12 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2115568903' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-21T23:08:13.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.136+0000 7f942de4e640 1 -- 172.21.15.44:0/2625509735 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f94281a0400 con 0x7f9428100680 2024-04-21T23:08:13.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.137+0000 7f9424ff9640 1 -- 172.21.15.44:0/2625509735 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f94180bd650 con 0x7f9428100680 2024-04-21T23:08:13.136 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-21T23:08:13.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.139+0000 7f942de4e640 1 -- 172.21.15.44:0/2625509735 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f94100a2c20 msgr2=0x7f94100a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:13.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.139+0000 7f942de4e640 1 --2- 172.21.15.44:0/2625509735 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f94100a2c20 0x7f94100a50e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9428199010 tx=0x7f941c005f90 comp rx=0 tx=0).stop 2024-04-21T23:08:13.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.139+0000 7f942de4e640 1 -- 172.21.15.44:0/2625509735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9428100680 msgr2=0x7f9428197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:13.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.139+0000 7f942de4e640 1 --2- 172.21.15.44:0/2625509735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9428100680 0x7f9428197b00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9418000c00 tx=0x7f9418005d50 comp rx=0 tx=0).stop 2024-04-21T23:08:13.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.139+0000 7f942de4e640 1 -- 172.21.15.44:0/2625509735 shutdown_connections 2024-04-21T23:08:13.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.139+0000 7f942de4e640 1 --2- 172.21.15.44:0/2625509735 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f94100a2c20 0x7f94100a50e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.139+0000 7f942de4e640 1 --2- 172.21.15.44:0/2625509735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9428102230 0x7f942819f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.139+0000 7f942de4e640 1 --2- 172.21.15.44:0/2625509735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9428101870 0x7f9428198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.139+0000 7f942de4e640 1 --2- 172.21.15.44:0/2625509735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9428100680 0x7f9428197b00 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.139+0000 7f942de4e640 1 -- 172.21.15.44:0/2625509735 >> 172.21.15.44:0/2625509735 conn(0x7f94280fbbd0 msgr2=0x7f94280fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:13.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.139+0000 7f942de4e640 1 -- 172.21.15.44:0/2625509735 shutdown_connections 2024-04-21T23:08:13.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.139+0000 7f942de4e640 1 -- 172.21.15.44:0/2625509735 wait complete. 2024-04-21T23:08:13.150 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:08:13.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.257+0000 7f1b92b5b640 1 -- 172.21.15.44:0/1181953243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b8c102230 msgr2=0x7f1b8c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:13.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.257+0000 7f1b92b5b640 1 --2- 172.21.15.44:0/1181953243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b8c102230 0x7f1b8c108ac0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1b84009ca0 tx=0x7f1b8402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.257+0000 7f1b92b5b640 1 -- 172.21.15.44:0/1181953243 shutdown_connections 2024-04-21T23:08:13.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.257+0000 7f1b92b5b640 1 --2- 172.21.15.44:0/1181953243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b8c101870 0x7f1b8c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.257+0000 7f1b92b5b640 1 --2- 172.21.15.44:0/1181953243 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b8c102230 0x7f1b8c108ac0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.257+0000 7f1b92b5b640 1 --2- 172.21.15.44:0/1181953243 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b8c100680 0x7f1b8c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.257+0000 7f1b92b5b640 1 -- 172.21.15.44:0/1181953243 >> 172.21.15.44:0/1181953243 conn(0x7f1b8c0fbc10 msgr2=0x7f1b8c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:13.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.257+0000 7f1b92b5b640 1 -- 172.21.15.44:0/1181953243 shutdown_connections 2024-04-21T23:08:13.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.257+0000 7f1b92b5b640 1 -- 172.21.15.44:0/1181953243 wait complete. 2024-04-21T23:08:13.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.258+0000 7f1b92b5b640 1 Processor -- start 2024-04-21T23:08:13.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.258+0000 7f1b92b5b640 1 -- start start 2024-04-21T23:08:13.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.258+0000 7f1b92b5b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b8c100680 0x7f1b8c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:13.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.258+0000 7f1b92b5b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b8c101870 0x7f1b8c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:13.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.258+0000 7f1b92b5b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b8c102230 0x7f1b8c19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:13.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.258+0000 7f1b92b5b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8c10c300 con 0x7f1b8c101870 2024-04-21T23:08:13.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.258+0000 7f1b92b5b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8c10c180 con 0x7f1b8c100680 2024-04-21T23:08:13.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.258+0000 7f1b92b5b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8c10c480 con 0x7f1b8c102230 2024-04-21T23:08:13.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.258+0000 7f1b8bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b8c101870 0x7f1b8c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:13.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.258+0000 7f1b8bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b8c101870 0x7f1b8c197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40394/0 (socket says 172.21.15.44:40394) 2024-04-21T23:08:13.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.258+0000 7f1b8bfff640 1 -- 172.21.15.44:0/993957379 learned_addr learned my addr 172.21.15.44:0/993957379 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:13.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.258+0000 7f1b910f8640 1 --2- 172.21.15.44:0/993957379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b8c102230 0x7f1b8c19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:13.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.259+0000 7f1b908f7640 1 --2- 172.21.15.44:0/993957379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b8c100680 0x7f1b8c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:13.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.259+0000 7f1b8bfff640 1 -- 172.21.15.44:0/993957379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b8c102230 msgr2=0x7f1b8c19f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:13.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.259+0000 7f1b8bfff640 1 --2- 172.21.15.44:0/993957379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b8c102230 0x7f1b8c19f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.259+0000 7f1b8bfff640 1 -- 172.21.15.44:0/993957379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b8c100680 msgr2=0x7f1b8c197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:13.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.259+0000 7f1b8bfff640 1 --2- 172.21.15.44:0/993957379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b8c100680 0x7f1b8c197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.259+0000 7f1b8bfff640 1 -- 172.21.15.44:0/993957379 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b8c19f6a0 con 0x7f1b8c101870 2024-04-21T23:08:13.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.259+0000 7f1b8bfff640 1 --2- 172.21.15.44:0/993957379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b8c101870 0x7f1b8c197fc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800d770 tx=0x7f1b7800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:13.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.259+0000 7f1b89ffb640 1 -- 172.21.15.44:0/993957379 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b78017070 con 0x7f1b8c101870 2024-04-21T23:08:13.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.259+0000 7f1b92b5b640 1 -- 172.21.15.44:0/993957379 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b8c19f990 con 0x7f1b8c101870 2024-04-21T23:08:13.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.259+0000 7f1b89ffb640 1 -- 172.21.15.44:0/993957379 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b78005500 con 0x7f1b8c101870 2024-04-21T23:08:13.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.259+0000 7f1b89ffb640 1 -- 172.21.15.44:0/993957379 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b78004e00 con 0x7f1b8c101870 2024-04-21T23:08:13.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.259+0000 7f1b92b5b640 1 -- 172.21.15.44:0/993957379 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b8c19fe90 con 0x7f1b8c101870 2024-04-21T23:08:13.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.260+0000 7f1b92b5b640 1 -- 172.21.15.44:0/993957379 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b8c1a00c0 con 0x7f1b8c101870 2024-04-21T23:08:13.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.261+0000 7f1b89ffb640 1 -- 172.21.15.44:0/993957379 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1b78004fa0 con 0x7f1b8c101870 2024-04-21T23:08:13.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.262+0000 7f1b89ffb640 1 --2- 172.21.15.44:0/993957379 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f1b7c0a2c20 0x7f1b7c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:13.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.262+0000 7f1b89ffb640 1 -- 172.21.15.44:0/993957379 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f1b780cef90 con 0x7f1b8c101870 2024-04-21T23:08:13.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.262+0000 7f1b908f7640 1 --2- 172.21.15.44:0/993957379 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f1b7c0a2c20 0x7f1b7c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:13.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.262+0000 7f1b908f7640 1 --2- 172.21.15.44:0/993957379 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f1b7c0a2c20 0x7f1b7c0a50e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c1016d0 tx=0x7f1b80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:13.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.271+0000 7f1b89ffb640 1 -- 172.21.15.44:0/993957379 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1b7809ba50 con 0x7f1b8c101870 2024-04-21T23:08:13.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.456+0000 7f1b92b5b640 1 -- 172.21.15.44:0/993957379 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1b8c1a0290 con 0x7f1b8c101870 2024-04-21T23:08:13.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.457+0000 7f1b89ffb640 1 -- 172.21.15.44:0/993957379 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f1b7809b350 con 0x7f1b8c101870 2024-04-21T23:08:13.457 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:08:13.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.459+0000 7f1b92b5b640 1 -- 172.21.15.44:0/993957379 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f1b7c0a2c20 msgr2=0x7f1b7c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:13.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.459+0000 7f1b92b5b640 1 --2- 172.21.15.44:0/993957379 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f1b7c0a2c20 0x7f1b7c0a50e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c1016d0 tx=0x7f1b80005f90 comp rx=0 tx=0).stop 2024-04-21T23:08:13.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.459+0000 7f1b92b5b640 1 -- 172.21.15.44:0/993957379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b8c101870 msgr2=0x7f1b8c197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:13.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.459+0000 7f1b92b5b640 1 --2- 172.21.15.44:0/993957379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b8c101870 0x7f1b8c197fc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800d770 tx=0x7f1b7800dc30 comp rx=0 tx=0).stop 2024-04-21T23:08:13.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.459+0000 7f1b92b5b640 1 -- 172.21.15.44:0/993957379 shutdown_connections 2024-04-21T23:08:13.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.459+0000 7f1b92b5b640 1 --2- 172.21.15.44:0/993957379 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f1b7c0a2c20 0x7f1b7c0a50e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.459+0000 7f1b92b5b640 1 --2- 172.21.15.44:0/993957379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b8c102230 0x7f1b8c19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.459+0000 7f1b92b5b640 1 --2- 172.21.15.44:0/993957379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b8c101870 0x7f1b8c197fc0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.459+0000 7f1b92b5b640 1 --2- 172.21.15.44:0/993957379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b8c100680 0x7f1b8c197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.459+0000 7f1b92b5b640 1 -- 172.21.15.44:0/993957379 >> 172.21.15.44:0/993957379 conn(0x7f1b8c0fbc10 msgr2=0x7f1b8c068a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:13.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.459+0000 7f1b92b5b640 1 -- 172.21.15.44:0/993957379 shutdown_connections 2024-04-21T23:08:13.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.459+0000 7f1b92b5b640 1 -- 172.21.15.44:0/993957379 wait complete. 2024-04-21T23:08:13.470 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:08:13.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.578+0000 7fc776ee2640 1 -- 172.21.15.44:0/2408632195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7701009b0 msgr2=0x7fc770100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.578+0000 7fc776ee2640 1 --2- 172.21.15.44:0/2408632195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7701009b0 0x7fc770100e10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc75c009900 tx=0x7fc75c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.578+0000 7fc776ee2640 1 -- 172.21.15.44:0/2408632195 shutdown_connections 2024-04-21T23:08:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.578+0000 7fc776ee2640 1 --2- 172.21.15.44:0/2408632195 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc770101350 0x7fc7701017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.578+0000 7fc776ee2640 1 --2- 172.21.15.44:0/2408632195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7701009b0 0x7fc770100e10 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.578+0000 7fc776ee2640 1 --2- 172.21.15.44:0/2408632195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc770100050 0x7fc770100470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.578+0000 7fc776ee2640 1 -- 172.21.15.44:0/2408632195 >> 172.21.15.44:0/2408632195 conn(0x7fc7700fbc10 msgr2=0x7fc7700fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.578+0000 7fc776ee2640 1 -- 172.21.15.44:0/2408632195 shutdown_connections 2024-04-21T23:08:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.578+0000 7fc776ee2640 1 -- 172.21.15.44:0/2408632195 wait complete. 2024-04-21T23:08:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.578+0000 7fc776ee2640 1 Processor -- start 2024-04-21T23:08:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.578+0000 7fc776ee2640 1 -- start start 2024-04-21T23:08:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.579+0000 7fc776ee2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc770100050 0x7fc770199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:13.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.579+0000 7fc776ee2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7701009b0 0x7fc77019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.579+0000 7fc776ee2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc770101350 0x7fc7701a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.579+0000 7fc776ee2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc77010f5e0 con 0x7fc7701009b0 2024-04-21T23:08:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.579+0000 7fc776ee2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc77010f460 con 0x7fc770100050 2024-04-21T23:08:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.579+0000 7fc776ee2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc77010f760 con 0x7fc770101350 2024-04-21T23:08:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.579+0000 7fc76ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7701009b0 0x7fc77019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.579+0000 7fc76ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7701009b0 0x7fc77019a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40400/0 (socket says 172.21.15.44:40400) 2024-04-21T23:08:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.579+0000 7fc76ffff640 1 -- 172.21.15.44:0/3093764932 learned_addr learned my addr 172.21.15.44:0/3093764932 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.579+0000 7fc774c7e640 1 --2- 172.21.15.44:0/3093764932 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc770100050 0x7fc770199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.579+0000 7fc77547f640 1 --2- 172.21.15.44:0/3093764932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc770101350 0x7fc7701a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.579+0000 7fc76ffff640 1 -- 172.21.15.44:0/3093764932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc770101350 msgr2=0x7fc7701a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.579+0000 7fc76ffff640 1 --2- 172.21.15.44:0/3093764932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc770101350 0x7fc7701a1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.579+0000 7fc76ffff640 1 -- 172.21.15.44:0/3093764932 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc770100050 msgr2=0x7fc770199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.579+0000 7fc76ffff640 1 --2- 172.21.15.44:0/3093764932 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc770100050 0x7fc770199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.579+0000 7fc76ffff640 1 -- 172.21.15.44:0/3093764932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7701a18c0 con 0x7fc7701009b0 2024-04-21T23:08:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.579+0000 7fc774c7e640 1 --2- 172.21.15.44:0/3093764932 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc770100050 0x7fc770199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:08:13.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.579+0000 7fc76ffff640 1 --2- 172.21.15.44:0/3093764932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7701009b0 0x7fc77019a1c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc75c009230 tx=0x7fc75c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:13.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.579+0000 7fc76dffb640 1 -- 172.21.15.44:0/3093764932 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75c041070 con 0x7fc7701009b0 2024-04-21T23:08:13.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.579+0000 7fc776ee2640 1 -- 172.21.15.44:0/3093764932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7701a1b50 con 0x7fc7701009b0 2024-04-21T23:08:13.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.580+0000 7fc776ee2640 1 -- 172.21.15.44:0/3093764932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7701a2090 con 0x7fc7701009b0 2024-04-21T23:08:13.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.580+0000 7fc76dffb640 1 -- 172.21.15.44:0/3093764932 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc75c0044d0 con 0x7fc7701009b0 2024-04-21T23:08:13.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.580+0000 7fc76dffb640 1 -- 172.21.15.44:0/3093764932 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75c04b060 con 0x7fc7701009b0 2024-04-21T23:08:13.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.581+0000 7fc776ee2640 1 -- 172.21.15.44:0/3093764932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7540052c0 con 0x7fc7701009b0 2024-04-21T23:08:13.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.583+0000 7fc76dffb640 1 -- 172.21.15.44:0/3093764932 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc75c044070 con 0x7fc7701009b0 2024-04-21T23:08:13.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.584+0000 7fc76dffb640 1 --2- 172.21.15.44:0/3093764932 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc7600a2c20 0x7fc7600a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:13.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.584+0000 7fc774c7e640 1 --2- 172.21.15.44:0/3093764932 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc7600a2c20 0x7fc7600a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:13.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.584+0000 7fc76dffb640 1 -- 172.21.15.44:0/3093764932 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fc75c0f0fe0 con 0x7fc7701009b0 2024-04-21T23:08:13.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.584+0000 7fc774c7e640 1 --2- 172.21.15.44:0/3093764932 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc7600a2c20 0x7fc7600a50e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc77019ab50 tx=0x7fc764005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:13.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.589+0000 7fc76dffb640 1 -- 172.21.15.44:0/3093764932 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc75c0bdbb0 con 0x7fc7701009b0 2024-04-21T23:08:13.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.780+0000 7fc776ee2640 1 -- 172.21.15.44:0/3093764932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc754005880 con 0x7fc7701009b0 2024-04-21T23:08:13.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.781+0000 7fc76dffb640 1 -- 172.21.15.44:0/3093764932 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7fc75c0bd4b0 con 0x7fc7701009b0 2024-04-21T23:08:13.781 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-21T23:08:13.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.783+0000 7fc776ee2640 1 -- 172.21.15.44:0/3093764932 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc7600a2c20 msgr2=0x7fc7600a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:13.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.783+0000 7fc776ee2640 1 --2- 172.21.15.44:0/3093764932 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc7600a2c20 0x7fc7600a50e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc77019ab50 tx=0x7fc764005f90 comp rx=0 tx=0).stop 2024-04-21T23:08:13.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.783+0000 7fc776ee2640 1 -- 172.21.15.44:0/3093764932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7701009b0 msgr2=0x7fc77019a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:13.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.783+0000 7fc776ee2640 1 --2- 172.21.15.44:0/3093764932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7701009b0 0x7fc77019a1c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc75c009230 tx=0x7fc75c005d50 comp rx=0 tx=0).stop 2024-04-21T23:08:13.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.783+0000 7fc776ee2640 1 -- 172.21.15.44:0/3093764932 shutdown_connections 2024-04-21T23:08:13.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.783+0000 7fc776ee2640 1 --2- 172.21.15.44:0/3093764932 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc7600a2c20 0x7fc7600a50e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.783+0000 7fc776ee2640 1 --2- 172.21.15.44:0/3093764932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc770101350 0x7fc7701a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.783+0000 7fc776ee2640 1 --2- 172.21.15.44:0/3093764932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc7701009b0 0x7fc77019a1c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.783+0000 7fc776ee2640 1 --2- 172.21.15.44:0/3093764932 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc770100050 0x7fc770199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.783+0000 7fc776ee2640 1 -- 172.21.15.44:0/3093764932 >> 172.21.15.44:0/3093764932 conn(0x7fc7700fbc10 msgr2=0x7fc7700fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:13.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.783+0000 7fc776ee2640 1 -- 172.21.15.44:0/3093764932 shutdown_connections 2024-04-21T23:08:13.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.783+0000 7fc776ee2640 1 -- 172.21.15.44:0/3093764932 wait complete. 2024-04-21T23:08:13.793 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:13 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3114446130' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:13.793 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:13 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2625509735' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:13.793 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:13 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/993957379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:13.794 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:08:13.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.901+0000 7f2d7091e640 1 -- 172.21.15.44:0/812696464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d6c101870 msgr2=0x7f2d6c101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:13.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.901+0000 7f2d7091e640 1 --2- 172.21.15.44:0/812696464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d6c101870 0x7f2d6c101cf0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c009900 tx=0x7f2d5c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:13.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.901+0000 7f2d7091e640 1 -- 172.21.15.44:0/812696464 shutdown_connections 2024-04-21T23:08:13.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.901+0000 7f2d7091e640 1 --2- 172.21.15.44:0/812696464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d6c102230 0x7f2d6c108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.901+0000 7f2d7091e640 1 --2- 172.21.15.44:0/812696464 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d6c101870 0x7f2d6c101cf0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.901+0000 7f2d7091e640 1 --2- 172.21.15.44:0/812696464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d6c100680 0x7f2d6c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.901+0000 7f2d7091e640 1 -- 172.21.15.44:0/812696464 >> 172.21.15.44:0/812696464 conn(0x7f2d6c0fbc10 msgr2=0x7f2d6c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:13.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.901+0000 7f2d7091e640 1 -- 172.21.15.44:0/812696464 shutdown_connections 2024-04-21T23:08:13.901 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.901+0000 7f2d7091e640 1 -- 172.21.15.44:0/812696464 wait complete. 2024-04-21T23:08:13.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.902+0000 7f2d7091e640 1 Processor -- start 2024-04-21T23:08:13.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.902+0000 7f2d7091e640 1 -- start start 2024-04-21T23:08:13.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.902+0000 7f2d7091e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d6c100680 0x7f2d6c199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:13.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.902+0000 7f2d7091e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d6c101870 0x7f2d6c19a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:13.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.902+0000 7f2d7091e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d6c102230 0x7f2d6c1a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:13.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.902+0000 7f2d7091e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6c10c2e0 con 0x7f2d6c102230 2024-04-21T23:08:13.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.902+0000 7f2d7091e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6c10c160 con 0x7f2d6c100680 2024-04-21T23:08:13.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.902+0000 7f2d7091e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6c10c460 con 0x7f2d6c101870 2024-04-21T23:08:13.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.902+0000 7f2d6ad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d6c102230 0x7f2d6c1a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:13.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.902+0000 7f2d6ad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d6c102230 0x7f2d6c1a1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40416/0 (socket says 172.21.15.44:40416) 2024-04-21T23:08:13.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.902+0000 7f2d6ad9d640 1 -- 172.21.15.44:0/1400687758 learned_addr learned my addr 172.21.15.44:0/1400687758 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:13.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.902+0000 7f2d69d9b640 1 --2- 172.21.15.44:0/1400687758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d6c101870 0x7f2d6c19a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:13.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.902+0000 7f2d6a59c640 1 --2- 172.21.15.44:0/1400687758 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d6c100680 0x7f2d6c199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:13.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.902+0000 7f2d6ad9d640 1 -- 172.21.15.44:0/1400687758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d6c101870 msgr2=0x7f2d6c19a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:13.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.902+0000 7f2d6ad9d640 1 --2- 172.21.15.44:0/1400687758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d6c101870 0x7f2d6c19a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.902+0000 7f2d6ad9d640 1 -- 172.21.15.44:0/1400687758 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d6c100680 msgr2=0x7f2d6c199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:13.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.902+0000 7f2d6ad9d640 1 --2- 172.21.15.44:0/1400687758 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d6c100680 0x7f2d6c199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:13.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.902+0000 7f2d6ad9d640 1 -- 172.21.15.44:0/1400687758 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d6c1a18c0 con 0x7f2d6c102230 2024-04-21T23:08:13.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.902+0000 7f2d69d9b640 1 --2- 172.21.15.44:0/1400687758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d6c101870 0x7f2d6c19a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:08:13.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.903+0000 7f2d6ad9d640 1 --2- 172.21.15.44:0/1400687758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d6c102230 0x7f2d6c1a1220 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2d6400cce0 tx=0x7f2d6400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:13.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.903+0000 7f2d5b7fe640 1 -- 172.21.15.44:0/1400687758 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6400f450 con 0x7f2d6c102230 2024-04-21T23:08:13.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.903+0000 7f2d7091e640 1 -- 172.21.15.44:0/1400687758 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d6c1a1bb0 con 0x7f2d6c102230 2024-04-21T23:08:13.905 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.903+0000 7f2d5b7fe640 1 -- 172.21.15.44:0/1400687758 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d6400fad0 con 0x7f2d6c102230 2024-04-21T23:08:13.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.903+0000 7f2d5b7fe640 1 -- 172.21.15.44:0/1400687758 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d64018c10 con 0x7f2d6c102230 2024-04-21T23:08:13.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.903+0000 7f2d7091e640 1 -- 172.21.15.44:0/1400687758 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d6c1a2080 con 0x7f2d6c102230 2024-04-21T23:08:13.906 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.904+0000 7f2d7091e640 1 -- 172.21.15.44:0/1400687758 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d6c1a22b0 con 0x7f2d6c102230 2024-04-21T23:08:13.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.905+0000 7f2d5b7fe640 1 -- 172.21.15.44:0/1400687758 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f2d6400f5f0 con 0x7f2d6c102230 2024-04-21T23:08:13.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.906+0000 7f2d5b7fe640 1 --2- 172.21.15.44:0/1400687758 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2d540a2c20 0x7f2d540a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:13.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.906+0000 7f2d5b7fe640 1 -- 172.21.15.44:0/1400687758 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f2d640cfcf0 con 0x7f2d6c102230 2024-04-21T23:08:13.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.906+0000 7f2d6a59c640 1 --2- 172.21.15.44:0/1400687758 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2d540a2c20 0x7f2d540a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:13.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.906+0000 7f2d6a59c640 1 --2- 172.21.15.44:0/1400687758 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2d540a2c20 0x7f2d540a50e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2d60005fd0 tx=0x7f2d60005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:13.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:13.913+0000 7f2d5b7fe640 1 -- 172.21.15.44:0/1400687758 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2d64014030 con 0x7f2d6c102230 2024-04-21T23:08:13.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:13 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3114446130' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:13.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:13 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2625509735' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:13.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:13 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/993957379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:13.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:13 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3114446130' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:13.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:13 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2625509735' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:13.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:13 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/993957379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:14.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.102+0000 7f2d7091e640 1 -- 172.21.15.44:0/1400687758 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f2d6c1a2480 con 0x7f2d6c102230 2024-04-21T23:08:14.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.102+0000 7f2d5b7fe640 1 -- 172.21.15.44:0/1400687758 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7f2d6409c840 con 0x7f2d6c102230 2024-04-21T23:08:14.102 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:08:14.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.104+0000 7f2d7091e640 1 -- 172.21.15.44:0/1400687758 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2d540a2c20 msgr2=0x7f2d540a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:14.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.104+0000 7f2d7091e640 1 --2- 172.21.15.44:0/1400687758 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2d540a2c20 0x7f2d540a50e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2d60005fd0 tx=0x7f2d60005850 comp rx=0 tx=0).stop 2024-04-21T23:08:14.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.104+0000 7f2d7091e640 1 -- 172.21.15.44:0/1400687758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d6c102230 msgr2=0x7f2d6c1a1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:14.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.104+0000 7f2d7091e640 1 --2- 172.21.15.44:0/1400687758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d6c102230 0x7f2d6c1a1220 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2d6400cce0 tx=0x7f2d6400a620 comp rx=0 tx=0).stop 2024-04-21T23:08:14.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.104+0000 7f2d7091e640 1 -- 172.21.15.44:0/1400687758 shutdown_connections 2024-04-21T23:08:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.104+0000 7f2d7091e640 1 --2- 172.21.15.44:0/1400687758 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f2d540a2c20 0x7f2d540a50e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.104+0000 7f2d7091e640 1 --2- 172.21.15.44:0/1400687758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d6c101870 0x7f2d6c19a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.104+0000 7f2d7091e640 1 --2- 172.21.15.44:0/1400687758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2d6c102230 0x7f2d6c1a1220 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.104+0000 7f2d7091e640 1 --2- 172.21.15.44:0/1400687758 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2d6c100680 0x7f2d6c199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.104+0000 7f2d7091e640 1 -- 172.21.15.44:0/1400687758 >> 172.21.15.44:0/1400687758 conn(0x7f2d6c0fbc10 msgr2=0x7f2d6c0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.104+0000 7f2d7091e640 1 -- 172.21.15.44:0/1400687758 shutdown_connections 2024-04-21T23:08:14.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.104+0000 7f2d7091e640 1 -- 172.21.15.44:0/1400687758 wait complete. 2024-04-21T23:08:14.115 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:08:14.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.219+0000 7f11eb69d640 1 -- 172.21.15.44:0/2458829341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11e4107520 msgr2=0x7f11e4109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:14.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.219+0000 7f11eb69d640 1 --2- 172.21.15.44:0/2458829341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11e4107520 0x7f11e4109910 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f11dc009ca0 tx=0x7f11dc02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.219+0000 7f11eb69d640 1 -- 172.21.15.44:0/2458829341 shutdown_connections 2024-04-21T23:08:14.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.219+0000 7f11eb69d640 1 --2- 172.21.15.44:0/2458829341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11e4104bf0 0x7f11e4106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.219+0000 7f11eb69d640 1 --2- 172.21.15.44:0/2458829341 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11e4107520 0x7f11e4109910 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.219+0000 7f11eb69d640 1 --2- 172.21.15.44:0/2458829341 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11e41022c0 0x7f11e41046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.219+0000 7f11eb69d640 1 -- 172.21.15.44:0/2458829341 >> 172.21.15.44:0/2458829341 conn(0x7f11e40fbc10 msgr2=0x7f11e40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:14.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.219+0000 7f11eb69d640 1 -- 172.21.15.44:0/2458829341 shutdown_connections 2024-04-21T23:08:14.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.219+0000 7f11eb69d640 1 -- 172.21.15.44:0/2458829341 wait complete. 2024-04-21T23:08:14.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.219+0000 7f11eb69d640 1 Processor -- start 2024-04-21T23:08:14.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.219+0000 7f11eb69d640 1 -- start start 2024-04-21T23:08:14.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11eb69d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11e41022c0 0x7f11e4197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:14.221 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11eb69d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11e4104bf0 0x7f11e4197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:14.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11eb69d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11e4107520 0x7f11e419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:14.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11eb69d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f11e410d440 con 0x7f11e4107520 2024-04-21T23:08:14.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11eb69d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f11e410d2c0 con 0x7f11e41022c0 2024-04-21T23:08:14.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11eb69d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f11e410d5c0 con 0x7f11e4104bf0 2024-04-21T23:08:14.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11e9c3a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11e4107520 0x7f11e419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:14.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11e9c3a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11e4107520 0x7f11e419f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40438/0 (socket says 172.21.15.44:40438) 2024-04-21T23:08:14.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11e9c3a640 1 -- 172.21.15.44:0/299936831 learned_addr learned my addr 172.21.15.44:0/299936831 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:14.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11e9439640 1 --2- 172.21.15.44:0/299936831 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11e41022c0 0x7f11e4197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:14.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11e8c38640 1 --2- 172.21.15.44:0/299936831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11e4104bf0 0x7f11e4197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:14.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11e9c3a640 1 -- 172.21.15.44:0/299936831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11e4104bf0 msgr2=0x7f11e4197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:14.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11e9c3a640 1 --2- 172.21.15.44:0/299936831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11e4104bf0 0x7f11e4197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11e9c3a640 1 -- 172.21.15.44:0/299936831 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11e41022c0 msgr2=0x7f11e4197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:14.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11e9c3a640 1 --2- 172.21.15.44:0/299936831 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11e41022c0 0x7f11e4197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11e9c3a640 1 -- 172.21.15.44:0/299936831 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e419f670 con 0x7f11e4107520 2024-04-21T23:08:14.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11e8c38640 1 --2- 172.21.15.44:0/299936831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11e4104bf0 0x7f11e4197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:08:14.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11e9c3a640 1 --2- 172.21.15.44:0/299936831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11e4107520 0x7f11e419f000 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f11dc037040 tx=0x7f11dc032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:14.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11da7fc640 1 -- 172.21.15.44:0/299936831 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc041020 con 0x7f11e4107520 2024-04-21T23:08:14.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11eb69d640 1 -- 172.21.15.44:0/299936831 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11e419f900 con 0x7f11e4107520 2024-04-21T23:08:14.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11da7fc640 1 -- 172.21.15.44:0/299936831 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11dc034d90 con 0x7f11e4107520 2024-04-21T23:08:14.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11da7fc640 1 -- 172.21.15.44:0/299936831 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc039040 con 0x7f11e4107520 2024-04-21T23:08:14.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.220+0000 7f11eb69d640 1 -- 172.21.15.44:0/299936831 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11e419fdf0 con 0x7f11e4107520 2024-04-21T23:08:14.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.221+0000 7f11eb69d640 1 -- 172.21.15.44:0/299936831 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11e40fd700 con 0x7f11e4107520 2024-04-21T23:08:14.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.223+0000 7f11da7fc640 1 -- 172.21.15.44:0/299936831 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f11dc044020 con 0x7f11e4107520 2024-04-21T23:08:14.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.223+0000 7f11da7fc640 1 --2- 172.21.15.44:0/299936831 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f11d00a2c20 0x7f11d00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:14.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.223+0000 7f11e9439640 1 --2- 172.21.15.44:0/299936831 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f11d00a2c20 0x7f11d00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:14.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.223+0000 7f11da7fc640 1 -- 172.21.15.44:0/299936831 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f11dc0f1310 con 0x7f11e4107520 2024-04-21T23:08:14.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.224+0000 7f11e9439640 1 --2- 172.21.15.44:0/299936831 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f11d00a2c20 0x7f11d00a50e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009bd0 tx=0x7f11e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:14.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.231+0000 7f11da7fc640 1 -- 172.21.15.44:0/299936831 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f11dc04b020 con 0x7f11e4107520 2024-04-21T23:08:14.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.460+0000 7f11eb69d640 1 -- 172.21.15.44:0/299936831 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11e4062340 con 0x7f11e4107520 2024-04-21T23:08:14.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.461+0000 7f11da7fc640 1 -- 172.21.15.44:0/299936831 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11791 (secure 0 0 0) 0x7f11dc0bdf80 con 0x7f11e4107520 2024-04-21T23:08:14.460 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:14.461 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:06:50.685663+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3743898853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3743898853},{"type":"v1","addr":"172.21.15.69:6839","nonce":3743898853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:08:14.461 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2024-04-21T23:08:14.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.463+0000 7f11eb69d640 1 -- 172.21.15.44:0/299936831 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f11d00a2c20 msgr2=0x7f11d00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:14.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.463+0000 7f11eb69d640 1 --2- 172.21.15.44:0/299936831 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f11d00a2c20 0x7f11d00a50e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009bd0 tx=0x7f11e0005f90 comp rx=0 tx=0).stop 2024-04-21T23:08:14.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.463+0000 7f11eb69d640 1 -- 172.21.15.44:0/299936831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11e4107520 msgr2=0x7f11e419f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:14.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.463+0000 7f11eb69d640 1 --2- 172.21.15.44:0/299936831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11e4107520 0x7f11e419f000 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f11dc037040 tx=0x7f11dc032f20 comp rx=0 tx=0).stop 2024-04-21T23:08:14.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.463+0000 7f11eb69d640 1 -- 172.21.15.44:0/299936831 shutdown_connections 2024-04-21T23:08:14.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.463+0000 7f11eb69d640 1 --2- 172.21.15.44:0/299936831 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f11d00a2c20 0x7f11d00a50e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.463+0000 7f11eb69d640 1 --2- 172.21.15.44:0/299936831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11e4104bf0 0x7f11e4197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.463+0000 7f11eb69d640 1 --2- 172.21.15.44:0/299936831 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f11e4107520 0x7f11e419f000 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.463+0000 7f11eb69d640 1 --2- 172.21.15.44:0/299936831 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11e41022c0 0x7f11e4197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.463+0000 7f11eb69d640 1 -- 172.21.15.44:0/299936831 >> 172.21.15.44:0/299936831 conn(0x7f11e40fbc10 msgr2=0x7f11e4102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:14.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.463+0000 7f11eb69d640 1 -- 172.21.15.44:0/299936831 shutdown_connections 2024-04-21T23:08:14.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.463+0000 7f11eb69d640 1 -- 172.21.15.44:0/299936831 wait complete. 2024-04-21T23:08:14.475 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:08:14.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.578+0000 7f70991af640 1 -- 172.21.15.44:0/352714607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7094102250 msgr2=0x7f7094108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:14.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.578+0000 7f70991af640 1 --2- 172.21.15.44:0/352714607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7094102250 0x7f7094108ae0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f708c009ca0 tx=0x7f708c02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.578+0000 7f70991af640 1 -- 172.21.15.44:0/352714607 shutdown_connections 2024-04-21T23:08:14.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.578+0000 7f70991af640 1 --2- 172.21.15.44:0/352714607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7094102250 0x7f7094108ae0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.578+0000 7f70991af640 1 --2- 172.21.15.44:0/352714607 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7094101890 0x7f7094101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.578+0000 7f70991af640 1 --2- 172.21.15.44:0/352714607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70941006a0 0x7f7094100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.578+0000 7f70991af640 1 -- 172.21.15.44:0/352714607 >> 172.21.15.44:0/352714607 conn(0x7f70940fbc50 msgr2=0x7f70940fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:14.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.578+0000 7f70991af640 1 -- 172.21.15.44:0/352714607 shutdown_connections 2024-04-21T23:08:14.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.578+0000 7f70991af640 1 -- 172.21.15.44:0/352714607 wait complete. 2024-04-21T23:08:14.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.578+0000 7f70991af640 1 Processor -- start 2024-04-21T23:08:14.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.579+0000 7f70991af640 1 -- start start 2024-04-21T23:08:14.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.579+0000 7f70991af640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70941006a0 0x7f7094197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:14.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.579+0000 7f70991af640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7094101890 0x7f7094197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:14.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.579+0000 7f70991af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7094102250 0x7f709419ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:14.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.579+0000 7f70991af640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f709410c320 con 0x7f7094102250 2024-04-21T23:08:14.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.579+0000 7f70991af640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f709410c1a0 con 0x7f70941006a0 2024-04-21T23:08:14.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.579+0000 7f70991af640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f709410c4a0 con 0x7f7094101890 2024-04-21T23:08:14.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.579+0000 7f709359e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7094102250 0x7f709419ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:14.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.579+0000 7f709359e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7094102250 0x7f709419ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40460/0 (socket says 172.21.15.44:40460) 2024-04-21T23:08:14.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.579+0000 7f709359e640 1 -- 172.21.15.44:0/997896187 learned_addr learned my addr 172.21.15.44:0/997896187 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:14.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.579+0000 7f709259c640 1 --2- 172.21.15.44:0/997896187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7094101890 0x7f7094197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:14.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.579+0000 7f7092d9d640 1 --2- 172.21.15.44:0/997896187 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70941006a0 0x7f7094197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:14.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.579+0000 7f709359e640 1 -- 172.21.15.44:0/997896187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7094101890 msgr2=0x7f7094197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:14.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.579+0000 7f709359e640 1 --2- 172.21.15.44:0/997896187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7094101890 0x7f7094197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.579+0000 7f709359e640 1 -- 172.21.15.44:0/997896187 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70941006a0 msgr2=0x7f7094197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:14.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.579+0000 7f709359e640 1 --2- 172.21.15.44:0/997896187 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70941006a0 0x7f7094197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.579+0000 7f709359e640 1 -- 172.21.15.44:0/997896187 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f709419f590 con 0x7f7094102250 2024-04-21T23:08:14.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.579+0000 7f7092d9d640 1 --2- 172.21.15.44:0/997896187 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70941006a0 0x7f7094197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:08:14.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.580+0000 7f709359e640 1 --2- 172.21.15.44:0/997896187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7094102250 0x7f709419ef20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f708c037040 tx=0x7f708c032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:14.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.580+0000 7f708bfff640 1 -- 172.21.15.44:0/997896187 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c039040 con 0x7f7094102250 2024-04-21T23:08:14.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.580+0000 7f708bfff640 1 -- 172.21.15.44:0/997896187 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f708c044070 con 0x7f7094102250 2024-04-21T23:08:14.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.580+0000 7f70991af640 1 -- 172.21.15.44:0/997896187 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f709419f820 con 0x7f7094102250 2024-04-21T23:08:14.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.580+0000 7f708bfff640 1 -- 172.21.15.44:0/997896187 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c004260 con 0x7f7094102250 2024-04-21T23:08:14.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.580+0000 7f70991af640 1 -- 172.21.15.44:0/997896187 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f709419fd10 con 0x7f7094102250 2024-04-21T23:08:14.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.581+0000 7f70991af640 1 -- 172.21.15.44:0/997896187 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f709419ff70 con 0x7f7094102250 2024-04-21T23:08:14.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.582+0000 7f708bfff640 1 -- 172.21.15.44:0/997896187 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f708c003970 con 0x7f7094102250 2024-04-21T23:08:14.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.582+0000 7f708bfff640 1 --2- 172.21.15.44:0/997896187 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f70800a2c20 0x7f70800a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:14.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.583+0000 7f7092d9d640 1 --2- 172.21.15.44:0/997896187 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f70800a2c20 0x7f70800a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:14.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.583+0000 7f708bfff640 1 -- 172.21.15.44:0/997896187 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f708c0f21b0 con 0x7f7094102250 2024-04-21T23:08:14.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.583+0000 7f7092d9d640 1 --2- 172.21.15.44:0/997896187 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f70800a2c20 0x7f70800a50e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f707c0058c0 tx=0x7f707c005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:14.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.586+0000 7f708bfff640 1 -- 172.21.15.44:0/997896187 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f708c0bed80 con 0x7f7094102250 2024-04-21T23:08:14.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:14 smithi044 ceph-mon[27934]: pgmap v347: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:14.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:14 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3093764932' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:14.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:14 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1400687758' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:14.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:14 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/299936831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:14.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.771+0000 7f70991af640 1 -- 172.21.15.44:0/997896187 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f70941a0300 con 0x7f7094102250 2024-04-21T23:08:14.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.773+0000 7f708bfff640 1 -- 172.21.15.44:0/997896187 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24530 (secure 0 0 0) 0x7f70941a0300 con 0x7f7094102250 2024-04-21T23:08:14.773 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:14.773 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":92,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:07:24.439878+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":9,"to":11},{"from":5,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:08:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.775+0000 7f70991af640 1 -- 172.21.15.44:0/997896187 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f70800a2c20 msgr2=0x7f70800a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.775+0000 7f70991af640 1 --2- 172.21.15.44:0/997896187 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f70800a2c20 0x7f70800a50e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f707c0058c0 tx=0x7f707c005d50 comp rx=0 tx=0).stop 2024-04-21T23:08:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.775+0000 7f70991af640 1 -- 172.21.15.44:0/997896187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7094102250 msgr2=0x7f709419ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.775+0000 7f70991af640 1 --2- 172.21.15.44:0/997896187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7094102250 0x7f709419ef20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f708c037040 tx=0x7f708c032f90 comp rx=0 tx=0).stop 2024-04-21T23:08:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.775+0000 7f70991af640 1 -- 172.21.15.44:0/997896187 shutdown_connections 2024-04-21T23:08:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.775+0000 7f70991af640 1 --2- 172.21.15.44:0/997896187 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f70800a2c20 0x7f70800a50e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.775+0000 7f70991af640 1 --2- 172.21.15.44:0/997896187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7094101890 0x7f7094197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.775+0000 7f70991af640 1 --2- 172.21.15.44:0/997896187 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7094102250 0x7f709419ef20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.775+0000 7f70991af640 1 --2- 172.21.15.44:0/997896187 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70941006a0 0x7f7094197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.775+0000 7f70991af640 1 -- 172.21.15.44:0/997896187 >> 172.21.15.44:0/997896187 conn(0x7f70940fbc50 msgr2=0x7f70940fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:14.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.775+0000 7f70991af640 1 -- 172.21.15.44:0/997896187 shutdown_connections 2024-04-21T23:08:14.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.775+0000 7f70991af640 1 -- 172.21.15.44:0/997896187 wait complete. 2024-04-21T23:08:14.786 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:01:47.507503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:06:23.993760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:06:24.954661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:06:28.677618+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:08:14.787 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:08:14.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.892+0000 7f4020d37640 1 -- 172.21.15.44:0/2993721760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f401c103490 msgr2=0x7f401c109d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:14.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.892+0000 7f4020d37640 1 --2- 172.21.15.44:0/2993721760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f401c103490 0x7f401c109d20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4014009ca0 tx=0x7f401402c140 comp rx=0 tx=0).stop 2024-04-21T23:08:14.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.892+0000 7f4020d37640 1 -- 172.21.15.44:0/2993721760 shutdown_connections 2024-04-21T23:08:14.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.892+0000 7f4020d37640 1 --2- 172.21.15.44:0/2993721760 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f401c103490 0x7f401c109d20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.892+0000 7f4020d37640 1 --2- 172.21.15.44:0/2993721760 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f401c102ad0 0x7f401c102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.892+0000 7f4020d37640 1 --2- 172.21.15.44:0/2993721760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f401c0ffa60 0x7f401c0ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.892+0000 7f4020d37640 1 -- 172.21.15.44:0/2993721760 >> 172.21.15.44:0/2993721760 conn(0x7f401c0fb370 msgr2=0x7f401c0fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:14.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.892+0000 7f4020d37640 1 -- 172.21.15.44:0/2993721760 shutdown_connections 2024-04-21T23:08:14.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.892+0000 7f4020d37640 1 -- 172.21.15.44:0/2993721760 wait complete. 2024-04-21T23:08:14.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.892+0000 7f4020d37640 1 Processor -- start 2024-04-21T23:08:14.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.893+0000 7f4020d37640 1 -- start start 2024-04-21T23:08:14.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.893+0000 7f4020d37640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f401c0ffa60 0x7f401c197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:14.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.893+0000 7f4020d37640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f401c102ad0 0x7f401c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:14.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.893+0000 7f4020d37640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f401c103490 0x7f401c19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:14.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.893+0000 7f4020d37640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f401c10d430 con 0x7f401c0ffa60 2024-04-21T23:08:14.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.893+0000 7f4020d37640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f401c10d2b0 con 0x7f401c102ad0 2024-04-21T23:08:14.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.893+0000 7f4020d37640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f401c10d5b0 con 0x7f401c103490 2024-04-21T23:08:14.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.893+0000 7f401a59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f401c0ffa60 0x7f401c197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:14.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.893+0000 7f401a59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f401c0ffa60 0x7f401c197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40478/0 (socket says 172.21.15.44:40478) 2024-04-21T23:08:14.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.893+0000 7f401a59c640 1 -- 172.21.15.44:0/4160527909 learned_addr learned my addr 172.21.15.44:0/4160527909 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:14.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.893+0000 7f401a59c640 1 -- 172.21.15.44:0/4160527909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f401c103490 msgr2=0x7f401c19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:14.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.893+0000 7f401a59c640 1 --2- 172.21.15.44:0/4160527909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f401c103490 0x7f401c19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.893+0000 7f401a59c640 1 -- 172.21.15.44:0/4160527909 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f401c102ad0 msgr2=0x7f401c197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:14.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.893+0000 7f401a59c640 1 --2- 172.21.15.44:0/4160527909 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f401c102ad0 0x7f401c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:14.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.893+0000 7f401a59c640 1 -- 172.21.15.44:0/4160527909 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f401c19f660 con 0x7f401c0ffa60 2024-04-21T23:08:14.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.893+0000 7f401a59c640 1 --2- 172.21.15.44:0/4160527909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f401c0ffa60 0x7f401c197a40 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f400c00d950 tx=0x7f400c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:14.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.893+0000 7f400b7fe640 1 -- 172.21.15.44:0/4160527909 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400c017070 con 0x7f401c0ffa60 2024-04-21T23:08:14.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.893+0000 7f400b7fe640 1 -- 172.21.15.44:0/4160527909 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f400c009c30 con 0x7f401c0ffa60 2024-04-21T23:08:14.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.893+0000 7f4020d37640 1 -- 172.21.15.44:0/4160527909 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f401c19f950 con 0x7f401c0ffa60 2024-04-21T23:08:14.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.893+0000 7f400b7fe640 1 -- 172.21.15.44:0/4160527909 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400c00f600 con 0x7f401c0ffa60 2024-04-21T23:08:14.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.894+0000 7f4020d37640 1 -- 172.21.15.44:0/4160527909 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f401c19fe50 con 0x7f401c0ffa60 2024-04-21T23:08:14.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.894+0000 7f4020d37640 1 -- 172.21.15.44:0/4160527909 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f401c108ba0 con 0x7f401c0ffa60 2024-04-21T23:08:14.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.896+0000 7f400b7fe640 1 -- 172.21.15.44:0/4160527909 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f400c01a070 con 0x7f401c0ffa60 2024-04-21T23:08:14.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.896+0000 7f400b7fe640 1 --2- 172.21.15.44:0/4160527909 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f40040a2c20 0x7f40040a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:14.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.896+0000 7f4019d9b640 1 --2- 172.21.15.44:0/4160527909 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f40040a2c20 0x7f40040a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:14.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.896+0000 7f400b7fe640 1 -- 172.21.15.44:0/4160527909 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f400c0cf030 con 0x7f401c0ffa60 2024-04-21T23:08:14.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.897+0000 7f4019d9b640 1 --2- 172.21.15.44:0/4160527909 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f40040a2c20 0x7f40040a50e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4010005fd0 tx=0x7f4010005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:14.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:14.900+0000 7f400b7fe640 1 -- 172.21.15.44:0/4160527909 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f400c009de0 con 0x7f401c0ffa60 2024-04-21T23:08:14.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:14 smithi055 ceph-mon[31117]: pgmap v347: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:14.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:14 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3093764932' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:14.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:14 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1400687758' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:14.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:14 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/299936831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:14.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:14 smithi069 ceph-mon[31816]: pgmap v347: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:14.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:14 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3093764932' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:14.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:14 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1400687758' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:14.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:14 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/299936831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:15.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.084+0000 7f4020d37640 1 -- 172.21.15.44:0/4160527909 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f401c02d3b0 con 0x7f401c0ffa60 2024-04-21T23:08:15.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.084+0000 7f400b7fe640 1 -- 172.21.15.44:0/4160527909 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f400c09baf0 con 0x7f401c0ffa60 2024-04-21T23:08:15.084 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-21T23:08:15.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.086+0000 7f4020d37640 1 -- 172.21.15.44:0/4160527909 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f40040a2c20 msgr2=0x7f40040a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:15.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.086+0000 7f4020d37640 1 --2- 172.21.15.44:0/4160527909 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f40040a2c20 0x7f40040a50e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4010005fd0 tx=0x7f4010005850 comp rx=0 tx=0).stop 2024-04-21T23:08:15.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.086+0000 7f4020d37640 1 -- 172.21.15.44:0/4160527909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f401c0ffa60 msgr2=0x7f401c197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:15.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.086+0000 7f4020d37640 1 --2- 172.21.15.44:0/4160527909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f401c0ffa60 0x7f401c197a40 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f400c00d950 tx=0x7f400c00de10 comp rx=0 tx=0).stop 2024-04-21T23:08:15.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.086+0000 7f4020d37640 1 -- 172.21.15.44:0/4160527909 shutdown_connections 2024-04-21T23:08:15.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.086+0000 7f4020d37640 1 --2- 172.21.15.44:0/4160527909 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f40040a2c20 0x7f40040a50e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.086+0000 7f4020d37640 1 --2- 172.21.15.44:0/4160527909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f401c103490 0x7f401c19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.086+0000 7f4020d37640 1 --2- 172.21.15.44:0/4160527909 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f401c102ad0 0x7f401c197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.086+0000 7f4020d37640 1 --2- 172.21.15.44:0/4160527909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f401c0ffa60 0x7f401c197a40 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.087+0000 7f4020d37640 1 -- 172.21.15.44:0/4160527909 >> 172.21.15.44:0/4160527909 conn(0x7f401c0fb370 msgr2=0x7f401c105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:15.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.087+0000 7f4020d37640 1 -- 172.21.15.44:0/4160527909 shutdown_connections 2024-04-21T23:08:15.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.087+0000 7f4020d37640 1 -- 172.21.15.44:0/4160527909 wait complete. 2024-04-21T23:08:15.097 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:08:15.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.203+0000 7f5915042640 1 -- 172.21.15.44:0/187270157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5910103350 msgr2=0x7f5910109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:15.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.203+0000 7f5915042640 1 --2- 172.21.15.44:0/187270157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5910103350 0x7f5910109be0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5908009ca0 tx=0x7f590802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.203+0000 7f5915042640 1 -- 172.21.15.44:0/187270157 shutdown_connections 2024-04-21T23:08:15.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.203+0000 7f5915042640 1 --2- 172.21.15.44:0/187270157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5910102990 0x7f5910102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.203+0000 7f5915042640 1 --2- 172.21.15.44:0/187270157 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5910103350 0x7f5910109be0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.203+0000 7f5915042640 1 --2- 172.21.15.44:0/187270157 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f59101017a0 0x7f5910101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.203+0000 7f5915042640 1 -- 172.21.15.44:0/187270157 >> 172.21.15.44:0/187270157 conn(0x7f59100fcd10 msgr2=0x7f59100ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:15.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.203+0000 7f5915042640 1 -- 172.21.15.44:0/187270157 shutdown_connections 2024-04-21T23:08:15.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.203+0000 7f5915042640 1 -- 172.21.15.44:0/187270157 wait complete. 2024-04-21T23:08:15.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.204+0000 7f5915042640 1 Processor -- start 2024-04-21T23:08:15.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.204+0000 7f5915042640 1 -- start start 2024-04-21T23:08:15.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.204+0000 7f5915042640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59101017a0 0x7f5910197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:15.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.204+0000 7f5915042640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5910102990 0x7f5910197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:15.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.204+0000 7f5915042640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5910103350 0x7f591019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:15.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.204+0000 7f5915042640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f591010d420 con 0x7f5910102990 2024-04-21T23:08:15.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.204+0000 7f5915042640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f591010d2a0 con 0x7f5910103350 2024-04-21T23:08:15.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.204+0000 7f5915042640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f591010d5a0 con 0x7f59101017a0 2024-04-21T23:08:15.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.204+0000 7f590e59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5910102990 0x7f5910197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:15.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.204+0000 7f590e59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5910102990 0x7f5910197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40506/0 (socket says 172.21.15.44:40506) 2024-04-21T23:08:15.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.204+0000 7f590e59c640 1 -- 172.21.15.44:0/2160026926 learned_addr learned my addr 172.21.15.44:0/2160026926 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:15.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.204+0000 7f590ed9d640 1 --2- 172.21.15.44:0/2160026926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59101017a0 0x7f5910197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:15.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.204+0000 7f590f59e640 1 --2- 172.21.15.44:0/2160026926 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5910103350 0x7f591019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:15.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.204+0000 7f590e59c640 1 -- 172.21.15.44:0/2160026926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59101017a0 msgr2=0x7f5910197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:15.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.204+0000 7f590e59c640 1 --2- 172.21.15.44:0/2160026926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59101017a0 0x7f5910197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.204+0000 7f590e59c640 1 -- 172.21.15.44:0/2160026926 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5910103350 msgr2=0x7f591019f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:15.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.204+0000 7f590e59c640 1 --2- 172.21.15.44:0/2160026926 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5910103350 0x7f591019f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.205+0000 7f590e59c640 1 -- 172.21.15.44:0/2160026926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f591019f690 con 0x7f5910102990 2024-04-21T23:08:15.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.205+0000 7f590e59c640 1 --2- 172.21.15.44:0/2160026926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5910102990 0x7f5910197fb0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f590400d770 tx=0x7f590400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:15.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.205+0000 7f58f7fff640 1 -- 172.21.15.44:0/2160026926 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5904017070 con 0x7f5910102990 2024-04-21T23:08:15.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.205+0000 7f5915042640 1 -- 172.21.15.44:0/2160026926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f591019f980 con 0x7f5910102990 2024-04-21T23:08:15.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.205+0000 7f58f7fff640 1 -- 172.21.15.44:0/2160026926 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5904005500 con 0x7f5910102990 2024-04-21T23:08:15.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.205+0000 7f58f7fff640 1 -- 172.21.15.44:0/2160026926 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5904004e00 con 0x7f5910102990 2024-04-21T23:08:15.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.205+0000 7f5915042640 1 -- 172.21.15.44:0/2160026926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f591019fe80 con 0x7f5910102990 2024-04-21T23:08:15.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.206+0000 7f5915042640 1 -- 172.21.15.44:0/2160026926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5910108ac0 con 0x7f5910102990 2024-04-21T23:08:15.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.207+0000 7f58f7fff640 1 -- 172.21.15.44:0/2160026926 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5904004fa0 con 0x7f5910102990 2024-04-21T23:08:15.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.208+0000 7f58f7fff640 1 --2- 172.21.15.44:0/2160026926 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f58f80a2c20 0x7f58f80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:15.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.208+0000 7f590ed9d640 1 --2- 172.21.15.44:0/2160026926 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f58f80a2c20 0x7f58f80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:15.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.208+0000 7f58f7fff640 1 -- 172.21.15.44:0/2160026926 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f59040cef40 con 0x7f5910102990 2024-04-21T23:08:15.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.208+0000 7f590ed9d640 1 --2- 172.21.15.44:0/2160026926 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f58f80a2c20 0x7f58f80a50e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f59101027f0 tx=0x7f590000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:15.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.212+0000 7f58f7fff640 1 -- 172.21.15.44:0/2160026926 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f590409b350 con 0x7f5910102990 2024-04-21T23:08:15.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.398+0000 7f5915042640 1 -- 172.21.15.44:0/2160026926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f591002d390 con 0x7f5910102990 2024-04-21T23:08:15.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.399+0000 7f58f7fff640 1 -- 172.21.15.44:0/2160026926 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f590409f650 con 0x7f5910102990 2024-04-21T23:08:15.399 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:08:15.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.401+0000 7f5915042640 1 -- 172.21.15.44:0/2160026926 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f58f80a2c20 msgr2=0x7f58f80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:15.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.401+0000 7f5915042640 1 --2- 172.21.15.44:0/2160026926 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f58f80a2c20 0x7f58f80a50e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f59101027f0 tx=0x7f590000a2d0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.401+0000 7f5915042640 1 -- 172.21.15.44:0/2160026926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5910102990 msgr2=0x7f5910197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:15.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.401+0000 7f5915042640 1 --2- 172.21.15.44:0/2160026926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5910102990 0x7f5910197fb0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f590400d770 tx=0x7f590400dc30 comp rx=0 tx=0).stop 2024-04-21T23:08:15.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.401+0000 7f5915042640 1 -- 172.21.15.44:0/2160026926 shutdown_connections 2024-04-21T23:08:15.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.401+0000 7f5915042640 1 --2- 172.21.15.44:0/2160026926 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f58f80a2c20 0x7f58f80a50e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.401+0000 7f5915042640 1 --2- 172.21.15.44:0/2160026926 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5910103350 0x7f591019f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.401+0000 7f5915042640 1 --2- 172.21.15.44:0/2160026926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5910102990 0x7f5910197fb0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.401+0000 7f5915042640 1 --2- 172.21.15.44:0/2160026926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59101017a0 0x7f5910197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.401+0000 7f5915042640 1 -- 172.21.15.44:0/2160026926 >> 172.21.15.44:0/2160026926 conn(0x7f59100fcd10 msgr2=0x7f5910105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:15.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.401+0000 7f5915042640 1 -- 172.21.15.44:0/2160026926 shutdown_connections 2024-04-21T23:08:15.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.401+0000 7f5915042640 1 -- 172.21.15.44:0/2160026926 wait complete. 2024-04-21T23:08:15.412 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:08:15.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.517+0000 7fa3f9b9a640 1 -- 172.21.15.44:0/465872691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3f4100690 msgr2=0x7fa3f4100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:15.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.517+0000 7fa3f9b9a640 1 --2- 172.21.15.44:0/465872691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3f4100690 0x7fa3f4100ab0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4009ca0 tx=0x7fa3e402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.517+0000 7fa3f9b9a640 1 -- 172.21.15.44:0/465872691 shutdown_connections 2024-04-21T23:08:15.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.517+0000 7fa3f9b9a640 1 --2- 172.21.15.44:0/465872691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3f4101880 0x7fa3f4101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.517+0000 7fa3f9b9a640 1 --2- 172.21.15.44:0/465872691 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3f4102240 0x7fa3f4108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.517+0000 7fa3f9b9a640 1 --2- 172.21.15.44:0/465872691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3f4100690 0x7fa3f4100ab0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.517+0000 7fa3f9b9a640 1 -- 172.21.15.44:0/465872691 >> 172.21.15.44:0/465872691 conn(0x7fa3f40fbc20 msgr2=0x7fa3f40fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:15.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.517+0000 7fa3f9b9a640 1 -- 172.21.15.44:0/465872691 shutdown_connections 2024-04-21T23:08:15.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.517+0000 7fa3f9b9a640 1 -- 172.21.15.44:0/465872691 wait complete. 2024-04-21T23:08:15.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.518+0000 7fa3f9b9a640 1 Processor -- start 2024-04-21T23:08:15.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.518+0000 7fa3f9b9a640 1 -- start start 2024-04-21T23:08:15.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.518+0000 7fa3f9b9a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3f4100690 0x7fa3f406a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:15.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.518+0000 7fa3f9b9a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3f4101880 0x7fa3f419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:15.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.518+0000 7fa3f9b9a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3f4102240 0x7fa3f419ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:15.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.518+0000 7fa3f9b9a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f410c2e0 con 0x7fa3f4101880 2024-04-21T23:08:15.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.518+0000 7fa3f9b9a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f410c160 con 0x7fa3f4100690 2024-04-21T23:08:15.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.518+0000 7fa3f9b9a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f410c460 con 0x7fa3f4102240 2024-04-21T23:08:15.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.518+0000 7fa3f2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3f4101880 0x7fa3f419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:15.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.518+0000 7fa3f2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3f4101880 0x7fa3f419c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40528/0 (socket says 172.21.15.44:40528) 2024-04-21T23:08:15.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.518+0000 7fa3f2ffd640 1 -- 172.21.15.44:0/1112624581 learned_addr learned my addr 172.21.15.44:0/1112624581 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:15.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.519+0000 7fa3f37fe640 1 --2- 172.21.15.44:0/1112624581 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3f4100690 0x7fa3f406a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:15.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.519+0000 7fa3f3fff640 1 --2- 172.21.15.44:0/1112624581 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3f4102240 0x7fa3f419ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:15.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.519+0000 7fa3f2ffd640 1 -- 172.21.15.44:0/1112624581 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3f4102240 msgr2=0x7fa3f419ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:15.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.519+0000 7fa3f2ffd640 1 --2- 172.21.15.44:0/1112624581 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3f4102240 0x7fa3f419ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.519+0000 7fa3f2ffd640 1 -- 172.21.15.44:0/1112624581 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3f4100690 msgr2=0x7fa3f406a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:15.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.519+0000 7fa3f2ffd640 1 --2- 172.21.15.44:0/1112624581 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3f4100690 0x7fa3f406a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.519+0000 7fa3f2ffd640 1 -- 172.21.15.44:0/1112624581 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3f419f6c0 con 0x7fa3f4101880 2024-04-21T23:08:15.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.519+0000 7fa3f2ffd640 1 --2- 172.21.15.44:0/1112624581 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3f4101880 0x7fa3f419c830 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa3e800d950 tx=0x7fa3e800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:15.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.519+0000 7fa3f0ff9640 1 -- 172.21.15.44:0/1112624581 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e8017030 con 0x7fa3f4101880 2024-04-21T23:08:15.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.519+0000 7fa3f0ff9640 1 -- 172.21.15.44:0/1112624581 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3e8004020 con 0x7fa3f4101880 2024-04-21T23:08:15.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.519+0000 7fa3f9b9a640 1 -- 172.21.15.44:0/1112624581 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3f419f9b0 con 0x7fa3f4101880 2024-04-21T23:08:15.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.519+0000 7fa3f0ff9640 1 -- 172.21.15.44:0/1112624581 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e80047a0 con 0x7fa3f4101880 2024-04-21T23:08:15.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.519+0000 7fa3f9b9a640 1 -- 172.21.15.44:0/1112624581 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3f419ff00 con 0x7fa3f4101880 2024-04-21T23:08:15.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.520+0000 7fa3f9b9a640 1 -- 172.21.15.44:0/1112624581 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3f4107a80 con 0x7fa3f4101880 2024-04-21T23:08:15.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.521+0000 7fa3f0ff9640 1 -- 172.21.15.44:0/1112624581 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa3e80041d0 con 0x7fa3f4101880 2024-04-21T23:08:15.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.522+0000 7fa3f0ff9640 1 --2- 172.21.15.44:0/1112624581 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa3dc0a2c20 0x7fa3dc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:15.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.522+0000 7fa3f37fe640 1 --2- 172.21.15.44:0/1112624581 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa3dc0a2c20 0x7fa3dc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:15.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.522+0000 7fa3f0ff9640 1 -- 172.21.15.44:0/1112624581 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fa3e80cf030 con 0x7fa3f4101880 2024-04-21T23:08:15.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.522+0000 7fa3f37fe640 1 --2- 172.21.15.44:0/1112624581 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa3dc0a2c20 0x7fa3dc0a50e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa3e402c700 tx=0x7fa3e4009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:15.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.526+0000 7fa3f0ff9640 1 -- 172.21.15.44:0/1112624581 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa3e809baf0 con 0x7fa3f4101880 2024-04-21T23:08:15.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:15 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/997896187' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:15.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:15 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/4160527909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:15.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:15 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2160026926' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:15.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.711+0000 7fa3f9b9a640 1 -- 172.21.15.44:0/1112624581 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa3f402d3c0 con 0x7fa3f4101880 2024-04-21T23:08:15.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.711+0000 7fa3f0ff9640 1 -- 172.21.15.44:0/1112624581 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7fa3e809b3f0 con 0x7fa3f4101880 2024-04-21T23:08:15.711 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-21T23:08:15.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.713+0000 7fa3f9b9a640 1 -- 172.21.15.44:0/1112624581 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa3dc0a2c20 msgr2=0x7fa3dc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:15.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.713+0000 7fa3f9b9a640 1 --2- 172.21.15.44:0/1112624581 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa3dc0a2c20 0x7fa3dc0a50e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa3e402c700 tx=0x7fa3e4009360 comp rx=0 tx=0).stop 2024-04-21T23:08:15.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.713+0000 7fa3f9b9a640 1 -- 172.21.15.44:0/1112624581 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3f4101880 msgr2=0x7fa3f419c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:15.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.713+0000 7fa3f9b9a640 1 --2- 172.21.15.44:0/1112624581 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3f4101880 0x7fa3f419c830 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa3e800d950 tx=0x7fa3e800de10 comp rx=0 tx=0).stop 2024-04-21T23:08:15.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.713+0000 7fa3f9b9a640 1 -- 172.21.15.44:0/1112624581 shutdown_connections 2024-04-21T23:08:15.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.713+0000 7fa3f9b9a640 1 --2- 172.21.15.44:0/1112624581 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa3dc0a2c20 0x7fa3dc0a50e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.713+0000 7fa3f9b9a640 1 --2- 172.21.15.44:0/1112624581 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3f4102240 0x7fa3f419ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.713+0000 7fa3f9b9a640 1 --2- 172.21.15.44:0/1112624581 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3f4101880 0x7fa3f419c830 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.713+0000 7fa3f9b9a640 1 --2- 172.21.15.44:0/1112624581 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3f4100690 0x7fa3f406a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.713+0000 7fa3f9b9a640 1 -- 172.21.15.44:0/1112624581 >> 172.21.15.44:0/1112624581 conn(0x7fa3f40fbc20 msgr2=0x7fa3f4104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:15.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.713+0000 7fa3f9b9a640 1 -- 172.21.15.44:0/1112624581 shutdown_connections 2024-04-21T23:08:15.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.713+0000 7fa3f9b9a640 1 -- 172.21.15.44:0/1112624581 wait complete. 2024-04-21T23:08:15.724 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:08:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.829+0000 7f96be866640 1 -- 172.21.15.44:0/2948645606 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96b8101880 msgr2=0x7f96b8101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.829+0000 7f96be866640 1 --2- 172.21.15.44:0/2948645606 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96b8101880 0x7f96b8101d00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009900 tx=0x7f96ac02bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.829+0000 7f96be866640 1 -- 172.21.15.44:0/2948645606 shutdown_connections 2024-04-21T23:08:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.829+0000 7f96be866640 1 --2- 172.21.15.44:0/2948645606 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96b8102240 0x7f96b8108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.829+0000 7f96be866640 1 --2- 172.21.15.44:0/2948645606 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96b8101880 0x7f96b8101d00 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.829+0000 7f96be866640 1 --2- 172.21.15.44:0/2948645606 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96b8100690 0x7f96b8100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.829+0000 7f96be866640 1 -- 172.21.15.44:0/2948645606 >> 172.21.15.44:0/2948645606 conn(0x7f96b80fbc20 msgr2=0x7f96b80fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:15.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.829+0000 7f96be866640 1 -- 172.21.15.44:0/2948645606 shutdown_connections 2024-04-21T23:08:15.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.829+0000 7f96be866640 1 -- 172.21.15.44:0/2948645606 wait complete. 2024-04-21T23:08:15.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.829+0000 7f96be866640 1 Processor -- start 2024-04-21T23:08:15.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.829+0000 7f96be866640 1 -- start start 2024-04-21T23:08:15.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.830+0000 7f96be866640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96b8100690 0x7f96b8197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:15.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.830+0000 7f96be866640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96b8101880 0x7f96b8197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:15.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.830+0000 7f96be866640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96b8102240 0x7f96b819f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:15.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.830+0000 7f96be866640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f96b810c2f0 con 0x7f96b8102240 2024-04-21T23:08:15.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.830+0000 7f96be866640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f96b810c170 con 0x7f96b8101880 2024-04-21T23:08:15.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.830+0000 7f96be866640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f96b810c470 con 0x7f96b8100690 2024-04-21T23:08:15.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.830+0000 7f96bce03640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96b8102240 0x7f96b819f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:15.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.830+0000 7f96bce03640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96b8102240 0x7f96b819f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40544/0 (socket says 172.21.15.44:40544) 2024-04-21T23:08:15.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.830+0000 7f96bce03640 1 -- 172.21.15.44:0/1774936398 learned_addr learned my addr 172.21.15.44:0/1774936398 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:15.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.830+0000 7f96b77fe640 1 --2- 172.21.15.44:0/1774936398 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96b8101880 0x7f96b8197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:15.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.830+0000 7f96b7fff640 1 --2- 172.21.15.44:0/1774936398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96b8100690 0x7f96b8197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:15.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.830+0000 7f96bce03640 1 -- 172.21.15.44:0/1774936398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96b8100690 msgr2=0x7f96b8197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:15.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.830+0000 7f96bce03640 1 --2- 172.21.15.44:0/1774936398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96b8100690 0x7f96b8197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.830+0000 7f96bce03640 1 -- 172.21.15.44:0/1774936398 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96b8101880 msgr2=0x7f96b8197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:15.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.830+0000 7f96bce03640 1 --2- 172.21.15.44:0/1774936398 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96b8101880 0x7f96b8197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:15.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.830+0000 7f96bce03640 1 -- 172.21.15.44:0/1774936398 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b819f6a0 con 0x7f96b8102240 2024-04-21T23:08:15.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.830+0000 7f96b77fe640 1 --2- 172.21.15.44:0/1774936398 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96b8101880 0x7f96b8197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:08:15.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.830+0000 7f96bce03640 1 --2- 172.21.15.44:0/1774936398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96b8102240 0x7f96b819f030 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f96b000cce0 tx=0x7f96b000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:15.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.831+0000 7f96b57fa640 1 -- 172.21.15.44:0/1774936398 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b000f450 con 0x7f96b8102240 2024-04-21T23:08:15.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.831+0000 7f96b57fa640 1 -- 172.21.15.44:0/1774936398 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96b000fad0 con 0x7f96b8102240 2024-04-21T23:08:15.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.831+0000 7f96be866640 1 -- 172.21.15.44:0/1774936398 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b819f990 con 0x7f96b8102240 2024-04-21T23:08:15.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.831+0000 7f96b57fa640 1 -- 172.21.15.44:0/1774936398 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b0018c10 con 0x7f96b8102240 2024-04-21T23:08:15.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.831+0000 7f96be866640 1 -- 172.21.15.44:0/1774936398 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b819fe90 con 0x7f96b8102240 2024-04-21T23:08:15.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.832+0000 7f96be866640 1 -- 172.21.15.44:0/1774936398 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96b81a0120 con 0x7f96b8102240 2024-04-21T23:08:15.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.833+0000 7f96b57fa640 1 -- 172.21.15.44:0/1774936398 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f96b000f5f0 con 0x7f96b8102240 2024-04-21T23:08:15.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.833+0000 7f96b57fa640 1 --2- 172.21.15.44:0/1774936398 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f96a00a2c20 0x7f96a00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:15.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.834+0000 7f96b7fff640 1 --2- 172.21.15.44:0/1774936398 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f96a00a2c20 0x7f96a00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:15.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.834+0000 7f96b57fa640 1 -- 172.21.15.44:0/1774936398 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f96b00cfc60 con 0x7f96b8102240 2024-04-21T23:08:15.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.834+0000 7f96b7fff640 1 --2- 172.21.15.44:0/1774936398 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f96a00a2c20 0x7f96a00a50e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f96a8005fd0 tx=0x7f96a8005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:15.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:15.838+0000 7f96b57fa640 1 -- 172.21.15.44:0/1774936398 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f96b0014030 con 0x7f96b8102240 2024-04-21T23:08:15.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:15 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/997896187' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:15.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:15 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/4160527909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:15.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:15 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2160026926' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:15.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:15 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/997896187' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:15.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:15 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/4160527909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:15.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:15 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2160026926' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:16.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.020+0000 7f96be866640 1 -- 172.21.15.44:0/1774936398 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f96b81a0390 con 0x7f96b8102240 2024-04-21T23:08:16.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.021+0000 7f96b57fa640 1 -- 172.21.15.44:0/1774936398 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7f96b009c720 con 0x7f96b8102240 2024-04-21T23:08:16.021 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:08:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.023+0000 7f96be866640 1 -- 172.21.15.44:0/1774936398 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f96a00a2c20 msgr2=0x7f96a00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.023+0000 7f96be866640 1 --2- 172.21.15.44:0/1774936398 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f96a00a2c20 0x7f96a00a50e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f96a8005fd0 tx=0x7f96a8005c70 comp rx=0 tx=0).stop 2024-04-21T23:08:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.023+0000 7f96be866640 1 -- 172.21.15.44:0/1774936398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96b8102240 msgr2=0x7f96b819f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.023+0000 7f96be866640 1 --2- 172.21.15.44:0/1774936398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96b8102240 0x7f96b819f030 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f96b000cce0 tx=0x7f96b000a620 comp rx=0 tx=0).stop 2024-04-21T23:08:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.023+0000 7f96be866640 1 -- 172.21.15.44:0/1774936398 shutdown_connections 2024-04-21T23:08:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.023+0000 7f96be866640 1 --2- 172.21.15.44:0/1774936398 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f96a00a2c20 0x7f96a00a50e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.023+0000 7f96be866640 1 --2- 172.21.15.44:0/1774936398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96b8102240 0x7f96b819f030 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.023+0000 7f96be866640 1 --2- 172.21.15.44:0/1774936398 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96b8101880 0x7f96b8197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.023+0000 7f96be866640 1 --2- 172.21.15.44:0/1774936398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96b8100690 0x7f96b8197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.023+0000 7f96be866640 1 -- 172.21.15.44:0/1774936398 >> 172.21.15.44:0/1774936398 conn(0x7f96b80fbc20 msgr2=0x7f96b80fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:16.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.023+0000 7f96be866640 1 -- 172.21.15.44:0/1774936398 shutdown_connections 2024-04-21T23:08:16.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.023+0000 7f96be866640 1 -- 172.21.15.44:0/1774936398 wait complete. 2024-04-21T23:08:16.034 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:08:16.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.136+0000 7f75d36df640 1 -- 172.21.15.44:0/695366392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75cc102250 msgr2=0x7f75cc108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:16.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.136+0000 7f75d36df640 1 --2- 172.21.15.44:0/695366392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75cc102250 0x7f75cc108ae0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f75c4009ca0 tx=0x7f75c402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.136+0000 7f75d36df640 1 -- 172.21.15.44:0/695366392 shutdown_connections 2024-04-21T23:08:16.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.136+0000 7f75d36df640 1 --2- 172.21.15.44:0/695366392 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75cc102250 0x7f75cc108ae0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.136+0000 7f75d36df640 1 --2- 172.21.15.44:0/695366392 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75cc101890 0x7f75cc101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.136+0000 7f75d36df640 1 --2- 172.21.15.44:0/695366392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75cc1006a0 0x7f75cc100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.136+0000 7f75d36df640 1 -- 172.21.15.44:0/695366392 >> 172.21.15.44:0/695366392 conn(0x7f75cc0fbc50 msgr2=0x7f75cc0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.136+0000 7f75d36df640 1 -- 172.21.15.44:0/695366392 shutdown_connections 2024-04-21T23:08:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.136+0000 7f75d36df640 1 -- 172.21.15.44:0/695366392 wait complete. 2024-04-21T23:08:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.137+0000 7f75d36df640 1 Processor -- start 2024-04-21T23:08:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.137+0000 7f75d36df640 1 -- start start 2024-04-21T23:08:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.137+0000 7f75d36df640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75cc1006a0 0x7f75cc10b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.137+0000 7f75d36df640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75cc101890 0x7f75cc109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.137+0000 7f75d36df640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75cc102250 0x7f75cc109ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.137+0000 7f75d36df640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f75cc10d440 con 0x7f75cc101890 2024-04-21T23:08:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.137+0000 7f75d36df640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f75cc10d2c0 con 0x7f75cc1006a0 2024-04-21T23:08:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.137+0000 7f75d36df640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f75cc10d5c0 con 0x7f75cc102250 2024-04-21T23:08:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.137+0000 7f75d0c7a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75cc101890 0x7f75cc109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.137+0000 7f75d0c7a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75cc101890 0x7f75cc109660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40570/0 (socket says 172.21.15.44:40570) 2024-04-21T23:08:16.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.137+0000 7f75d0c7a640 1 -- 172.21.15.44:0/1965274930 learned_addr learned my addr 172.21.15.44:0/1965274930 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:16.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.138+0000 7f75d147b640 1 --2- 172.21.15.44:0/1965274930 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75cc1006a0 0x7f75cc10b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:16.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.138+0000 7f75d1c7c640 1 --2- 172.21.15.44:0/1965274930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75cc102250 0x7f75cc109ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:16.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.138+0000 7f75d0c7a640 1 -- 172.21.15.44:0/1965274930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75cc102250 msgr2=0x7f75cc109ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:16.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.138+0000 7f75d0c7a640 1 --2- 172.21.15.44:0/1965274930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75cc102250 0x7f75cc109ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.138+0000 7f75d0c7a640 1 -- 172.21.15.44:0/1965274930 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75cc1006a0 msgr2=0x7f75cc10b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:16.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.138+0000 7f75d0c7a640 1 --2- 172.21.15.44:0/1965274930 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75cc1006a0 0x7f75cc10b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.138+0000 7f75d0c7a640 1 -- 172.21.15.44:0/1965274930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75cc10a430 con 0x7f75cc101890 2024-04-21T23:08:16.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.138+0000 7f75d0c7a640 1 --2- 172.21.15.44:0/1965274930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75cc101890 0x7f75cc109660 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f75bc00c890 tx=0x7f75bc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:16.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.138+0000 7f75c27fc640 1 -- 172.21.15.44:0/1965274930 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75bc004220 con 0x7f75cc101890 2024-04-21T23:08:16.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.138+0000 7f75c27fc640 1 -- 172.21.15.44:0/1965274930 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75bc0043c0 con 0x7f75cc101890 2024-04-21T23:08:16.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.138+0000 7f75d36df640 1 -- 172.21.15.44:0/1965274930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75cc1a1ba0 con 0x7f75cc101890 2024-04-21T23:08:16.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.138+0000 7f75c27fc640 1 -- 172.21.15.44:0/1965274930 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75bc0049e0 con 0x7f75cc101890 2024-04-21T23:08:16.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.138+0000 7f75d36df640 1 -- 172.21.15.44:0/1965274930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75cc1a20f0 con 0x7f75cc101890 2024-04-21T23:08:16.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.139+0000 7f75d36df640 1 -- 172.21.15.44:0/1965274930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75cc107960 con 0x7f75cc101890 2024-04-21T23:08:16.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.140+0000 7f75c27fc640 1 -- 172.21.15.44:0/1965274930 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f75bc0079b0 con 0x7f75cc101890 2024-04-21T23:08:16.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.141+0000 7f75c27fc640 1 --2- 172.21.15.44:0/1965274930 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f75b80a2c20 0x7f75b80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:16.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.141+0000 7f75c27fc640 1 -- 172.21.15.44:0/1965274930 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f75bc0cf710 con 0x7f75cc101890 2024-04-21T23:08:16.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.141+0000 7f75d147b640 1 --2- 172.21.15.44:0/1965274930 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f75b80a2c20 0x7f75b80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:16.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.141+0000 7f75d147b640 1 --2- 172.21.15.44:0/1965274930 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f75b80a2c20 0x7f75b80a50e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f75c8005fd0 tx=0x7f75c8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:16.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.145+0000 7f75c27fc640 1 -- 172.21.15.44:0/1965274930 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f75bc014030 con 0x7f75cc101890 2024-04-21T23:08:16.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.368+0000 7f75d36df640 1 -- 172.21.15.44:0/1965274930 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75cc02d170 con 0x7f75cc101890 2024-04-21T23:08:16.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.369+0000 7f75c27fc640 1 -- 172.21.15.44:0/1965274930 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11791 (secure 0 0 0) 0x7f75bc09c170 con 0x7f75cc101890 2024-04-21T23:08:16.369 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:16.369 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:06:50.685663+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3743898853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3743898853},{"type":"v1","addr":"172.21.15.69:6839","nonce":3743898853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:08:16.369 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2024-04-21T23:08:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.371+0000 7f75d36df640 1 -- 172.21.15.44:0/1965274930 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f75b80a2c20 msgr2=0x7f75b80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.371+0000 7f75d36df640 1 --2- 172.21.15.44:0/1965274930 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f75b80a2c20 0x7f75b80a50e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f75c8005fd0 tx=0x7f75c8005850 comp rx=0 tx=0).stop 2024-04-21T23:08:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.371+0000 7f75d36df640 1 -- 172.21.15.44:0/1965274930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75cc101890 msgr2=0x7f75cc109660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.371+0000 7f75d36df640 1 --2- 172.21.15.44:0/1965274930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75cc101890 0x7f75cc109660 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f75bc00c890 tx=0x7f75bc00cd50 comp rx=0 tx=0).stop 2024-04-21T23:08:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.371+0000 7f75d36df640 1 -- 172.21.15.44:0/1965274930 shutdown_connections 2024-04-21T23:08:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.371+0000 7f75d36df640 1 --2- 172.21.15.44:0/1965274930 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f75b80a2c20 0x7f75b80a50e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.371+0000 7f75d36df640 1 --2- 172.21.15.44:0/1965274930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75cc102250 0x7f75cc109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.371+0000 7f75d36df640 1 --2- 172.21.15.44:0/1965274930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75cc101890 0x7f75cc109660 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.371+0000 7f75d36df640 1 --2- 172.21.15.44:0/1965274930 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75cc1006a0 0x7f75cc10b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.371+0000 7f75d36df640 1 -- 172.21.15.44:0/1965274930 >> 172.21.15.44:0/1965274930 conn(0x7f75cc0fbc50 msgr2=0x7f75cc104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.371+0000 7f75d36df640 1 -- 172.21.15.44:0/1965274930 shutdown_connections 2024-04-21T23:08:16.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.371+0000 7f75d36df640 1 -- 172.21.15.44:0/1965274930 wait complete. 2024-04-21T23:08:16.382 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:08:16.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.484+0000 7f4c0fb77640 1 -- 172.21.15.44:0/2955950958 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c08107480 msgr2=0x7f4c081098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:16.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.484+0000 7f4c0fb77640 1 --2- 172.21.15.44:0/2955950958 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c08107480 0x7f4c081098d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009900 tx=0x7f4bf802bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:16.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.484+0000 7f4c0fb77640 1 -- 172.21.15.44:0/2955950958 shutdown_connections 2024-04-21T23:08:16.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.484+0000 7f4c0fb77640 1 --2- 172.21.15.44:0/2955950958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c08104b50 0x7f4c08106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.484+0000 7f4c0fb77640 1 --2- 172.21.15.44:0/2955950958 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c08107480 0x7f4c081098d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.484+0000 7f4c0fb77640 1 --2- 172.21.15.44:0/2955950958 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c08102220 0x7f4c08104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.484+0000 7f4c0fb77640 1 -- 172.21.15.44:0/2955950958 >> 172.21.15.44:0/2955950958 conn(0x7f4c080fbc50 msgr2=0x7f4c080fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:16.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.485+0000 7f4c0fb77640 1 -- 172.21.15.44:0/2955950958 shutdown_connections 2024-04-21T23:08:16.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.485+0000 7f4c0fb77640 1 -- 172.21.15.44:0/2955950958 wait complete. 2024-04-21T23:08:16.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.485+0000 7f4c0fb77640 1 Processor -- start 2024-04-21T23:08:16.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.485+0000 7f4c0fb77640 1 -- start start 2024-04-21T23:08:16.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.485+0000 7f4c0fb77640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c08102220 0x7f4c08101b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:16.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.485+0000 7f4c0fb77640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c08104b50 0x7f4c081001d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:16.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.485+0000 7f4c0fb77640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c08107480 0x7f4c08100710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:16.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.485+0000 7f4c0fb77640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0810d420 con 0x7f4c08102220 2024-04-21T23:08:16.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.485+0000 7f4c0fb77640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0810d2a0 con 0x7f4c08107480 2024-04-21T23:08:16.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.485+0000 7f4c0fb77640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0810d5a0 con 0x7f4c08104b50 2024-04-21T23:08:16.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.485+0000 7f4c0d913640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c08102220 0x7f4c08101b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:16.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.485+0000 7f4c0d913640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c08102220 0x7f4c08101b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40594/0 (socket says 172.21.15.44:40594) 2024-04-21T23:08:16.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.486+0000 7f4c0d913640 1 -- 172.21.15.44:0/979797818 learned_addr learned my addr 172.21.15.44:0/979797818 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:16.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.486+0000 7f4c0d112640 1 --2- 172.21.15.44:0/979797818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c08104b50 0x7f4c081001d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:16.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.486+0000 7f4c0e114640 1 --2- 172.21.15.44:0/979797818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c08107480 0x7f4c08100710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:16.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.486+0000 7f4c0d913640 1 -- 172.21.15.44:0/979797818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c08104b50 msgr2=0x7f4c081001d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.486+0000 7f4c0d913640 1 --2- 172.21.15.44:0/979797818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c08104b50 0x7f4c081001d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.486+0000 7f4c0d913640 1 -- 172.21.15.44:0/979797818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c08107480 msgr2=0x7f4c08100710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.486+0000 7f4c0d913640 1 --2- 172.21.15.44:0/979797818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c08107480 0x7f4c08100710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.486+0000 7f4c0d913640 1 -- 172.21.15.44:0/979797818 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c08100fa0 con 0x7f4c08102220 2024-04-21T23:08:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.486+0000 7f4c0d112640 1 --2- 172.21.15.44:0/979797818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c08104b50 0x7f4c081001d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:08:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.486+0000 7f4c0d913640 1 --2- 172.21.15.44:0/979797818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c08102220 0x7f4c08101b90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4c0000ebc0 tx=0x7f4c0000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.486+0000 7f4c06ffd640 1 -- 172.21.15.44:0/979797818 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c00010630 con 0x7f4c08102220 2024-04-21T23:08:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.486+0000 7f4c06ffd640 1 -- 172.21.15.44:0/979797818 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c00010cb0 con 0x7f4c08102220 2024-04-21T23:08:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.486+0000 7f4c0fb77640 1 -- 172.21.15.44:0/979797818 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c081a1b50 con 0x7f4c08102220 2024-04-21T23:08:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.486+0000 7f4c06ffd640 1 -- 172.21.15.44:0/979797818 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c00018bd0 con 0x7f4c08102220 2024-04-21T23:08:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.486+0000 7f4c0fb77640 1 -- 172.21.15.44:0/979797818 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c081a20f0 con 0x7f4c08102220 2024-04-21T23:08:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.487+0000 7f4c0fb77640 1 -- 172.21.15.44:0/979797818 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c080fd740 con 0x7f4c08102220 2024-04-21T23:08:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.488+0000 7f4c06ffd640 1 -- 172.21.15.44:0/979797818 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f4c000107d0 con 0x7f4c08102220 2024-04-21T23:08:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.489+0000 7f4c06ffd640 1 --2- 172.21.15.44:0/979797818 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f4bf00a2ca0 0x7f4bf00a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.489+0000 7f4c06ffd640 1 -- 172.21.15.44:0/979797818 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f4c000cfc10 con 0x7f4c08102220 2024-04-21T23:08:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.489+0000 7f4c0d112640 1 --2- 172.21.15.44:0/979797818 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f4bf00a2ca0 0x7f4bf00a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.489+0000 7f4c0d112640 1 --2- 172.21.15.44:0/979797818 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f4bf00a2ca0 0x7f4bf00a5160 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4c08101900 tx=0x7f4bfc005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:16.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.493+0000 7f4c06ffd640 1 -- 172.21.15.44:0/979797818 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4c0009c6d0 con 0x7f4c08102220 2024-04-21T23:08:16.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.680+0000 7f4c0fb77640 1 -- 172.21.15.44:0/979797818 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c0802d3f0 con 0x7f4c08102220 2024-04-21T23:08:16.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.682+0000 7f4c06ffd640 1 -- 172.21.15.44:0/979797818 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24530 (secure 0 0 0) 0x7f4c0009bfd0 con 0x7f4c08102220 2024-04-21T23:08:16.682 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:16.682 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":92,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:07:24.439878+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":9,"to":11},{"from":5,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:08:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.684+0000 7f4c0fb77640 1 -- 172.21.15.44:0/979797818 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f4bf00a2ca0 msgr2=0x7f4bf00a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.684+0000 7f4c0fb77640 1 --2- 172.21.15.44:0/979797818 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f4bf00a2ca0 0x7f4bf00a5160 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4c08101900 tx=0x7f4bfc005d30 comp rx=0 tx=0).stop 2024-04-21T23:08:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.684+0000 7f4c0fb77640 1 -- 172.21.15.44:0/979797818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c08102220 msgr2=0x7f4c08101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.684+0000 7f4c0fb77640 1 --2- 172.21.15.44:0/979797818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c08102220 0x7f4c08101b90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4c0000ebc0 tx=0x7f4c0000c6a0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.684+0000 7f4c0fb77640 1 -- 172.21.15.44:0/979797818 shutdown_connections 2024-04-21T23:08:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.684+0000 7f4c0fb77640 1 --2- 172.21.15.44:0/979797818 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f4bf00a2ca0 0x7f4bf00a5160 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.684+0000 7f4c0fb77640 1 --2- 172.21.15.44:0/979797818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4c08104b50 0x7f4c081001d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.684+0000 7f4c0fb77640 1 --2- 172.21.15.44:0/979797818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4c08107480 0x7f4c08100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.684+0000 7f4c0fb77640 1 --2- 172.21.15.44:0/979797818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4c08102220 0x7f4c08101b90 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.684+0000 7f4c0fb77640 1 -- 172.21.15.44:0/979797818 >> 172.21.15.44:0/979797818 conn(0x7f4c080fbc50 msgr2=0x7f4c08102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:16.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.684+0000 7f4c0fb77640 1 -- 172.21.15.44:0/979797818 shutdown_connections 2024-04-21T23:08:16.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.684+0000 7f4c0fb77640 1 -- 172.21.15.44:0/979797818 wait complete. 2024-04-21T23:08:16.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:16 smithi044 ceph-mon[27934]: pgmap v348: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:08:16.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:16 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1112624581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:16.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:16 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1774936398' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:16.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:16 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1965274930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:16.695 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:08:16.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.797+0000 7f68e443b640 1 -- 172.21.15.44:0/2036813383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68dc102330 msgr2=0x7f68dc108bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:16.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.797+0000 7f68e443b640 1 --2- 172.21.15.44:0/2036813383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68dc102330 0x7f68dc108bc0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f68d4009ca0 tx=0x7f68d402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.797+0000 7f68e443b640 1 -- 172.21.15.44:0/2036813383 shutdown_connections 2024-04-21T23:08:16.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.797+0000 7f68e443b640 1 --2- 172.21.15.44:0/2036813383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68dc101970 0x7f68dc101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.797+0000 7f68e443b640 1 --2- 172.21.15.44:0/2036813383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68dc102330 0x7f68dc108bc0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.797+0000 7f68e443b640 1 --2- 172.21.15.44:0/2036813383 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68dc100780 0x7f68dc100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.797+0000 7f68e443b640 1 -- 172.21.15.44:0/2036813383 >> 172.21.15.44:0/2036813383 conn(0x7f68dc0fbd10 msgr2=0x7f68dc0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.797+0000 7f68e443b640 1 -- 172.21.15.44:0/2036813383 shutdown_connections 2024-04-21T23:08:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.797+0000 7f68e443b640 1 -- 172.21.15.44:0/2036813383 wait complete. 2024-04-21T23:08:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.798+0000 7f68e443b640 1 Processor -- start 2024-04-21T23:08:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.798+0000 7f68e443b640 1 -- start start 2024-04-21T23:08:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.798+0000 7f68e443b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68dc100780 0x7f68dc06a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.798+0000 7f68e443b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68dc101970 0x7f68dc19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.798+0000 7f68e443b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68dc102330 0x7f68dc19eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.798+0000 7f68e443b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc10c320 con 0x7f68dc102330 2024-04-21T23:08:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.798+0000 7f68e443b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc10c1a0 con 0x7f68dc100780 2024-04-21T23:08:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.798+0000 7f68e443b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc10c4a0 con 0x7f68dc101970 2024-04-21T23:08:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.798+0000 7f68e29d8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68dc102330 0x7f68dc19eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:16.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.798+0000 7f68e29d8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68dc102330 0x7f68dc19eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40614/0 (socket says 172.21.15.44:40614) 2024-04-21T23:08:16.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.798+0000 7f68e29d8640 1 -- 172.21.15.44:0/383950291 learned_addr learned my addr 172.21.15.44:0/383950291 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:16.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.798+0000 7f68e21d7640 1 --2- 172.21.15.44:0/383950291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68dc100780 0x7f68dc06a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:16.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.798+0000 7f68e19d6640 1 --2- 172.21.15.44:0/383950291 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68dc101970 0x7f68dc19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:16.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.798+0000 7f68e29d8640 1 -- 172.21.15.44:0/383950291 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68dc101970 msgr2=0x7f68dc19c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:16.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.798+0000 7f68e29d8640 1 --2- 172.21.15.44:0/383950291 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68dc101970 0x7f68dc19c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.798+0000 7f68e29d8640 1 -- 172.21.15.44:0/383950291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68dc100780 msgr2=0x7f68dc06a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:16.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.798+0000 7f68e29d8640 1 --2- 172.21.15.44:0/383950291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68dc100780 0x7f68dc06a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:16.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.798+0000 7f68e29d8640 1 -- 172.21.15.44:0/383950291 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68dc19f6b0 con 0x7f68dc102330 2024-04-21T23:08:16.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.798+0000 7f68e21d7640 1 --2- 172.21.15.44:0/383950291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68dc100780 0x7f68dc06a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:08:16.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.799+0000 7f68e29d8640 1 --2- 172.21.15.44:0/383950291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68dc102330 0x7f68dc19eda0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f68d402c700 tx=0x7f68d4032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:16.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.799+0000 7f68d37fe640 1 -- 172.21.15.44:0/383950291 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d4039040 con 0x7f68dc102330 2024-04-21T23:08:16.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.799+0000 7f68d37fe640 1 -- 172.21.15.44:0/383950291 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68d4044070 con 0x7f68dc102330 2024-04-21T23:08:16.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.799+0000 7f68e443b640 1 -- 172.21.15.44:0/383950291 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68dc19f8e0 con 0x7f68dc102330 2024-04-21T23:08:16.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.799+0000 7f68d37fe640 1 -- 172.21.15.44:0/383950291 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d40042a0 con 0x7f68dc102330 2024-04-21T23:08:16.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.799+0000 7f68e443b640 1 -- 172.21.15.44:0/383950291 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68dc19fe20 con 0x7f68dc102330 2024-04-21T23:08:16.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.800+0000 7f68e443b640 1 -- 172.21.15.44:0/383950291 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68dc107a40 con 0x7f68dc102330 2024-04-21T23:08:16.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.801+0000 7f68d37fe640 1 -- 172.21.15.44:0/383950291 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f68d4003970 con 0x7f68dc102330 2024-04-21T23:08:16.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.802+0000 7f68d37fe640 1 --2- 172.21.15.44:0/383950291 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f68c80a2ca0 0x7f68c80a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:16.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.802+0000 7f68e21d7640 1 --2- 172.21.15.44:0/383950291 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f68c80a2ca0 0x7f68c80a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:16.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.802+0000 7f68d37fe640 1 -- 172.21.15.44:0/383950291 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f68d40f1980 con 0x7f68dc102330 2024-04-21T23:08:16.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.802+0000 7f68e21d7640 1 --2- 172.21.15.44:0/383950291 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f68c80a2ca0 0x7f68c80a5160 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f68d8005d00 tx=0x7f68d8005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:16.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:16.806+0000 7f68d37fe640 1 -- 172.21.15.44:0/383950291 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f68d40be5e0 con 0x7f68dc102330 2024-04-21T23:08:16.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:16 smithi055 ceph-mon[31117]: pgmap v348: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:08:16.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:16 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1112624581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:16.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:16 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1774936398' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:16.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:16 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1965274930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:16.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:16 smithi069 ceph-mon[31816]: pgmap v348: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:08:16.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:16 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1112624581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:16.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:16 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1774936398' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:16.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:16 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1965274930' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:17.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.033+0000 7f68e443b640 1 -- 172.21.15.44:0/383950291 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68dc02d350 con 0x7f68dc102330 2024-04-21T23:08:17.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.033+0000 7f68d37fe640 1 -- 172.21.15.44:0/383950291 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11791 (secure 0 0 0) 0x7f68d40bdee0 con 0x7f68dc102330 2024-04-21T23:08:17.033 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:17.033 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:06:50.685663+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3743898853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3743898853},{"type":"v1","addr":"172.21.15.69:6839","nonce":3743898853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:08:17.033 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2024-04-21T23:08:17.035 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.035+0000 7f68e443b640 1 -- 172.21.15.44:0/383950291 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f68c80a2ca0 msgr2=0x7f68c80a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:17.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.035+0000 7f68e443b640 1 --2- 172.21.15.44:0/383950291 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f68c80a2ca0 0x7f68c80a5160 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f68d8005d00 tx=0x7f68d8005c90 comp rx=0 tx=0).stop 2024-04-21T23:08:17.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.035+0000 7f68e443b640 1 -- 172.21.15.44:0/383950291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68dc102330 msgr2=0x7f68dc19eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:17.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.035+0000 7f68e443b640 1 --2- 172.21.15.44:0/383950291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68dc102330 0x7f68dc19eda0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f68d402c700 tx=0x7f68d4032f20 comp rx=0 tx=0).stop 2024-04-21T23:08:17.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.036+0000 7f68e443b640 1 -- 172.21.15.44:0/383950291 shutdown_connections 2024-04-21T23:08:17.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.036+0000 7f68e443b640 1 --2- 172.21.15.44:0/383950291 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f68c80a2ca0 0x7f68c80a5160 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.036+0000 7f68e443b640 1 --2- 172.21.15.44:0/383950291 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68dc101970 0x7f68dc19c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.036+0000 7f68e443b640 1 --2- 172.21.15.44:0/383950291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68dc102330 0x7f68dc19eda0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.036+0000 7f68e443b640 1 --2- 172.21.15.44:0/383950291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68dc100780 0x7f68dc06a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.036+0000 7f68e443b640 1 -- 172.21.15.44:0/383950291 >> 172.21.15.44:0/383950291 conn(0x7f68dc0fbd10 msgr2=0x7f68dc104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:17.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.036+0000 7f68e443b640 1 -- 172.21.15.44:0/383950291 shutdown_connections 2024-04-21T23:08:17.036 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.036+0000 7f68e443b640 1 -- 172.21.15.44:0/383950291 wait complete. 2024-04-21T23:08:17.046 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-21T23:06:25.562805+0000', 'modified': '2024-04-21T23:06:50.685663+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24530}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24526': {'gid': 24526, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.69:6839/3743898853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6838', 'nonce': 3743898853}, {'type': 'v1', 'addr': '172.21.15.69:6839', 'nonce': 3743898853}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24530': {'gid': 24530, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.55:6839/2066817543', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6838', 'nonce': 2066817543}, {'type': 'v1', 'addr': '172.21.15.55:6839', 'nonce': 2066817543}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24530, 'qdb_cluster': [24530]} 2024-04-21T23:08:17.047 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-21T23:08:17.047 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T23:08:17.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.150+0000 7fd66be72640 1 -- 172.21.15.44:0/135715843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6641017a0 msgr2=0x7fd664101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:17.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.150+0000 7fd66be72640 1 --2- 172.21.15.44:0/135715843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6641017a0 0x7fd664101bc0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd654009ca0 tx=0x7fd65402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.150+0000 7fd66be72640 1 -- 172.21.15.44:0/135715843 shutdown_connections 2024-04-21T23:08:17.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.150+0000 7fd66be72640 1 --2- 172.21.15.44:0/135715843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd664103350 0x7fd664109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.150+0000 7fd66be72640 1 --2- 172.21.15.44:0/135715843 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd664102990 0x7fd664102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.150+0000 7fd66be72640 1 --2- 172.21.15.44:0/135715843 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd6641017a0 0x7fd664101bc0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.150+0000 7fd66be72640 1 -- 172.21.15.44:0/135715843 >> 172.21.15.44:0/135715843 conn(0x7fd6640fcd10 msgr2=0x7fd6640ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.150+0000 7fd66be72640 1 -- 172.21.15.44:0/135715843 shutdown_connections 2024-04-21T23:08:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.151+0000 7fd66be72640 1 -- 172.21.15.44:0/135715843 wait complete. 2024-04-21T23:08:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.151+0000 7fd66be72640 1 Processor -- start 2024-04-21T23:08:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.151+0000 7fd66be72640 1 -- start start 2024-04-21T23:08:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.151+0000 7fd66be72640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6641017a0 0x7fd664197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.151+0000 7fd66be72640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd664102990 0x7fd664197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.151+0000 7fd66be72640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd664103350 0x7fd66419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.151+0000 7fd66be72640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd66410d3f0 con 0x7fd664103350 2024-04-21T23:08:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.151+0000 7fd66be72640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd66410d270 con 0x7fd664102990 2024-04-21T23:08:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.151+0000 7fd66be72640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd66410d570 con 0x7fd6641017a0 2024-04-21T23:08:17.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.151+0000 7fd66a40f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd664103350 0x7fd66419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.151+0000 7fd66a40f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd664103350 0x7fd66419f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40638/0 (socket says 172.21.15.44:40638) 2024-04-21T23:08:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.151+0000 7fd66a40f640 1 -- 172.21.15.44:0/3791972737 learned_addr learned my addr 172.21.15.44:0/3791972737 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.151+0000 7fd66940d640 1 --2- 172.21.15.44:0/3791972737 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd664102990 0x7fd664197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.151+0000 7fd669c0e640 1 --2- 172.21.15.44:0/3791972737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6641017a0 0x7fd664197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.152+0000 7fd66a40f640 1 -- 172.21.15.44:0/3791972737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6641017a0 msgr2=0x7fd664197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.152+0000 7fd66a40f640 1 --2- 172.21.15.44:0/3791972737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6641017a0 0x7fd664197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.152+0000 7fd66a40f640 1 -- 172.21.15.44:0/3791972737 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd664102990 msgr2=0x7fd664197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.152+0000 7fd66a40f640 1 --2- 172.21.15.44:0/3791972737 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd664102990 0x7fd664197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.152+0000 7fd66a40f640 1 -- 172.21.15.44:0/3791972737 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd66419f6d0 con 0x7fd664103350 2024-04-21T23:08:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.152+0000 7fd66a40f640 1 --2- 172.21.15.44:0/3791972737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd664103350 0x7fd66419f060 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00d950 tx=0x7fd65c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.152+0000 7fd662ffd640 1 -- 172.21.15.44:0/3791972737 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c017030 con 0x7fd664103350 2024-04-21T23:08:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.152+0000 7fd66be72640 1 -- 172.21.15.44:0/3791972737 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd66419f9c0 con 0x7fd664103350 2024-04-21T23:08:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.152+0000 7fd662ffd640 1 -- 172.21.15.44:0/3791972737 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd65c004020 con 0x7fd664103350 2024-04-21T23:08:17.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.152+0000 7fd662ffd640 1 -- 172.21.15.44:0/3791972737 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c003c00 con 0x7fd664103350 2024-04-21T23:08:17.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.152+0000 7fd66be72640 1 -- 172.21.15.44:0/3791972737 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd66419ff10 con 0x7fd664103350 2024-04-21T23:08:17.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.154+0000 7fd662ffd640 1 -- 172.21.15.44:0/3791972737 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd65c01a070 con 0x7fd664103350 2024-04-21T23:08:17.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.154+0000 7fd66ea58740 1 Processor -- start 2024-04-21T23:08:17.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.155+0000 7fd66ea58740 1 -- start start 2024-04-21T23:08:17.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.155+0000 7fd66ea58740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x564d051cf480 0x564d051cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.155+0000 7fd66ea58740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d051cfe70 0x564d051d4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.155+0000 7fd66ea58740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051d4fe0 0x564d051d7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.155+0000 7fd66ea58740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564d0512fdf0 con 0x564d051d4fe0 2024-04-21T23:08:17.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.155+0000 7fd66ea58740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x564d050d8640 con 0x564d051cf480 2024-04-21T23:08:17.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.155+0000 7fd66ea58740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564d0512bdb0 con 0x564d051cfe70 2024-04-21T23:08:17.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.155+0000 7fd662ffd640 1 --2- 172.21.15.44:0/3791972737 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd64c0a2c20 0x7fd64c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.155+0000 7fd66940d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051d4fe0 0x564d051d7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:17.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.155+0000 7fd66940d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051d4fe0 0x564d051d7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40652/0 (socket says 172.21.15.44:40652) 2024-04-21T23:08:17.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.155+0000 7fd66940d640 1 -- 172.21.15.44:0/4255986174 learned_addr learned my addr 172.21.15.44:0/4255986174 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:17.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.155+0000 7fd66a40f640 1 --2- 172.21.15.44:0/3791972737 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd64c0a2c20 0x7fd64c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:17.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.155+0000 7fd66940d640 1 --2- 172.21.15.44:0/4255986174 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x564d051cf480 0x564d051cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:17.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.155+0000 7fd669c0e640 1 --2- 172.21.15.44:0/4255986174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d051cfe70 0x564d051d4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:17.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.155+0000 7fd662ffd640 1 -- 172.21.15.44:0/3791972737 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fd65c0cedb0 con 0x7fd664103350 2024-04-21T23:08:17.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.155+0000 7fd66940d640 1 -- 172.21.15.44:0/4255986174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d051cfe70 msgr2=0x564d051d4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:17.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.155+0000 7fd66940d640 1 --2- 172.21.15.44:0/4255986174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d051cfe70 0x564d051d4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.155+0000 7fd66940d640 1 -- 172.21.15.44:0/4255986174 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x564d051cf480 msgr2=0x564d051cf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:17.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.155+0000 7fd66940d640 1 --2- 172.21.15.44:0/4255986174 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x564d051cf480 0x564d051cf8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.155+0000 7fd66940d640 1 -- 172.21.15.44:0/4255986174 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d04e442d0 con 0x564d051d4fe0 2024-04-21T23:08:17.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.156+0000 7fd66940d640 1 --2- 172.21.15.44:0/4255986174 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051d4fe0 0x564d051d7470 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd658012790 tx=0x7fd65800b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f050a04eb97d2b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:17.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.156+0000 7fd66a40f640 1 --2- 172.21.15.44:0/3791972737 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd64c0a2c20 0x7fd64c0a50e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd65c09b0a0 tx=0x7fd65c0905f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:17.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.156+0000 7fd64bfff640 1 -- 172.21.15.44:0/4255986174 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658012dd0 con 0x564d051d4fe0 2024-04-21T23:08:17.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.156+0000 7fd64bfff640 1 -- 172.21.15.44:0/4255986174 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd658010c50 con 0x564d051d4fe0 2024-04-21T23:08:17.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.156+0000 7fd64bfff640 1 -- 172.21.15.44:0/4255986174 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65801eb10 con 0x564d051d4fe0 2024-04-21T23:08:17.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.157+0000 7fd66ea58740 1 -- 172.21.15.44:0/4255986174 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051d4fe0 msgr2=0x564d051d7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:17.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.157+0000 7fd66ea58740 1 --2- 172.21.15.44:0/4255986174 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051d4fe0 0x564d051d7470 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd658012790 tx=0x7fd65800b7c0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.157+0000 7fd66ea58740 1 -- 172.21.15.44:0/4255986174 shutdown_connections 2024-04-21T23:08:17.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.157+0000 7fd66ea58740 1 --2- 172.21.15.44:0/4255986174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d051cfe70 0x564d051d4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.157+0000 7fd66ea58740 1 --2- 172.21.15.44:0/4255986174 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051d4fe0 0x564d051d7470 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.157+0000 7fd66ea58740 1 --2- 172.21.15.44:0/4255986174 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x564d051cf480 0x564d051cf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.157+0000 7fd66ea58740 1 -- 172.21.15.44:0/4255986174 >> 172.21.15.44:0/4255986174 conn(0x564d051c65e0 msgr2=0x564d051c69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:17.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.157+0000 7fd66ea58740 1 -- 172.21.15.44:0/4255986174 shutdown_connections 2024-04-21T23:08:17.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.157+0000 7fd66ea58740 1 -- 172.21.15.44:0/4255986174 wait complete. 2024-04-21T23:08:17.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.157+0000 7fd66ea58740 1 -- start start 2024-04-21T23:08:17.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.157+0000 7fd66ea58740 1 Processor -- start 2024-04-21T23:08:17.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.157+0000 7fd66ea58740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x564d051cf480 0x564d051d98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.157+0000 7fd66ea58740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d051cfe70 0x564d051d9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.157+0000 7fd66ea58740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051d4fe0 0x564d051e0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.157+0000 7fd66ea58740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564d051d7f90 con 0x564d051d4fe0 2024-04-21T23:08:17.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.157+0000 7fd66ea58740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x564d050ebbe0 con 0x564d051cf480 2024-04-21T23:08:17.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.157+0000 7fd66ea58740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564d051d8110 con 0x564d051cfe70 2024-04-21T23:08:17.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.158+0000 7fd66ea58740 10 client.0 _ll_register_callbacks cb 0x564d050eb5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:08:17.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.158+0000 7fd66940d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051d4fe0 0x564d051e0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:17.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.158+0000 7fd66940d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051d4fe0 0x564d051e0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40656/0 (socket says 172.21.15.44:40656) 2024-04-21T23:08:17.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.158+0000 7fd66940d640 1 -- 172.21.15.44:0/468224814 learned_addr learned my addr 172.21.15.44:0/468224814 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:17.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.158+0000 7fd66940d640 1 --2- 172.21.15.44:0/468224814 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x564d051cf480 0x564d051d98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:17.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.158+0000 7fd669c0e640 1 --2- 172.21.15.44:0/468224814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d051cfe70 0x564d051d9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:17.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.158+0000 7fd66940d640 1 -- 172.21.15.44:0/468224814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d051cfe70 msgr2=0x564d051d9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:17.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.158+0000 7fd66940d640 1 --2- 172.21.15.44:0/468224814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d051cfe70 0x564d051d9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.158+0000 7fd66940d640 1 -- 172.21.15.44:0/468224814 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x564d051cf480 msgr2=0x564d051d98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:17.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.158+0000 7fd66940d640 1 --2- 172.21.15.44:0/468224814 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x564d051cf480 0x564d051d98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.158+0000 7fd66940d640 1 -- 172.21.15.44:0/468224814 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d04f0e6b0 con 0x564d051d4fe0 2024-04-21T23:08:17.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.158+0000 7fd66940d640 1 --2- 172.21.15.44:0/468224814 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051d4fe0 0x564d051e0e60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd658012e60 tx=0x7fd658014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3264c0571f8cd523 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:17.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.158+0000 7fd6497fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:08:17.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.158+0000 7fd6497fa640 1 -- 172.21.15.44:0/468224814 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658024040 con 0x564d051d4fe0 2024-04-21T23:08:17.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.158+0000 7fd66ea58740 1 -- 172.21.15.44:0/468224814 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd658010c50 con 0x564d051d4fe0 2024-04-21T23:08:17.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.158+0000 7fd6497fa640 1 -- 172.21.15.44:0/468224814 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fd65801e8b0 con 0x564d051d4fe0 2024-04-21T23:08:17.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.159+0000 7fd6497fa640 1 -- 172.21.15.44:0/468224814 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd65801eab0 con 0x564d051d4fe0 2024-04-21T23:08:17.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.159+0000 7fd66ea58740 10 client.15234 fetch_fsmap learned FSMap version 21 2024-04-21T23:08:17.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.159+0000 7fd66ea58740 1 -- 172.21.15.44:0/468224814 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x564d051e3e90 con 0x564d051d4fe0 2024-04-21T23:08:17.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.159+0000 7fd6497fa640 1 -- 172.21.15.44:0/468224814 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd658010c50 con 0x564d051d4fe0 2024-04-21T23:08:17.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.159+0000 7fd6497fa640 1 -- 172.21.15.44:0/468224814 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fd65802a040 con 0x564d051d4fe0 2024-04-21T23:08:17.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.159+0000 7fd66ea58740 10 client.15234 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:08:17.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.159+0000 7fd66ea58740 10 client.15234 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T23:08:17.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.159+0000 7fd66ea58740 20 client.15234 populate_metadata read hostname 'smithi044' 2024-04-21T23:08:17.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.159+0000 7fd66ea58740 1 --2- 172.21.15.44:0/468224814 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x564d051e62c0 0x564d051e7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.159+0000 7fd66ea58740 4 client.15234 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:08:17.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.159+0000 7fd66ea58740 1 -- 172.21.15.44:0/468224814 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d0513d330 con 0x564d051e62c0 2024-04-21T23:08:17.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.159+0000 7fd669c0e640 1 --2- 172.21.15.44:0/468224814 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x564d051e62c0 0x564d051e7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:17.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.160+0000 7fd669c0e640 1 --2- 172.21.15.44:0/468224814 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x564d051e62c0 0x564d051e7730 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=719c20b03bdd814d server_cookie=838997faee8d7060 in_seq=0 out_seq=0 2024-04-21T23:08:17.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.160+0000 7fd6497fa640 10 client.15234 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:08:17.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.160+0000 7fd6497fa640 1 -- 172.21.15.44:0/468224814 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564d0513d330 con 0x564d051e62c0 2024-04-21T23:08:17.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.160+0000 7fd6497fa640 10 client.15234 handle_command_reply: tid=0 2024-04-21T23:08:17.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.160+0000 7fd66ea58740 1 client.15234 shutdown 2024-04-21T23:08:17.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.161+0000 7fd66ea58740 1 -- 172.21.15.44:0/468224814 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051d4fe0 msgr2=0x564d051e0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:17.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.161+0000 7fd66ea58740 1 --2- 172.21.15.44:0/468224814 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051d4fe0 0x564d051e0e60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd658012e60 tx=0x7fd658014040 comp rx=0 tx=0).stop 2024-04-21T23:08:17.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.161+0000 7fd66ea58740 1 -- 172.21.15.44:0/468224814 shutdown_connections 2024-04-21T23:08:17.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.161+0000 7fd66ea58740 1 --2- 172.21.15.44:0/468224814 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x564d051e62c0 0x564d051e7730 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.161+0000 7fd66ea58740 1 --2- 172.21.15.44:0/468224814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d051cfe70 0x564d051d9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.161+0000 7fd66ea58740 1 --2- 172.21.15.44:0/468224814 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051d4fe0 0x564d051e0e60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.161+0000 7fd6497fa640 0 client.15234 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:08:17.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.161+0000 7fd66ea58740 1 --2- 172.21.15.44:0/468224814 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x564d051cf480 0x564d051d98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.161+0000 7fd66ea58740 1 -- 172.21.15.44:0/468224814 >> 172.21.15.44:0/468224814 conn(0x564d051c65e0 msgr2=0x564d051c5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:17.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.161+0000 7fd66ea58740 1 -- 172.21.15.44:0/468224814 shutdown_connections 2024-04-21T23:08:17.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.161+0000 7fd66ea58740 1 -- 172.21.15.44:0/468224814 wait complete. 2024-04-21T23:08:17.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.161+0000 7fd66ea58740 20 client.15234 trim_cache size 0 max 16384 2024-04-21T23:08:17.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.179+0000 7fd66ea58740 1 Processor -- start 2024-04-21T23:08:17.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.179+0000 7fd66ea58740 1 -- start start 2024-04-21T23:08:17.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.179+0000 7fd66ea58740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051cf480 0x564d051e14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.179+0000 7fd66ea58740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x564d051cfe70 0x564d051e0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.179+0000 7fd66ea58740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d051d4fe0 0x564d051d12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.179+0000 7fd66ea58740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564d051e5a70 con 0x564d051cf480 2024-04-21T23:08:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.179+0000 7fd66ea58740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x564d051e6140 con 0x564d051cfe70 2024-04-21T23:08:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.179+0000 7fd66ea58740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564d051e6810 con 0x564d051d4fe0 2024-04-21T23:08:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.179+0000 7fd66940d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051cf480 0x564d051e14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.179+0000 7fd66940d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051cf480 0x564d051e14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40670/0 (socket says 172.21.15.44:40670) 2024-04-21T23:08:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.179+0000 7fd66940d640 1 -- 172.21.15.44:0/4146724061 learned_addr learned my addr 172.21.15.44:0/4146724061 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.180+0000 7fd66940d640 1 -- 172.21.15.44:0/4146724061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d051d4fe0 msgr2=0x564d051d12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.180+0000 7fd66940d640 1 --2- 172.21.15.44:0/4146724061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d051d4fe0 0x564d051d12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.180+0000 7fd66940d640 1 -- 172.21.15.44:0/4146724061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x564d051cfe70 msgr2=0x564d051e0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.180+0000 7fd66940d640 1 --2- 172.21.15.44:0/4146724061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x564d051cfe70 0x564d051e0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.180+0000 7fd66940d640 1 -- 172.21.15.44:0/4146724061 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d051d4560 con 0x564d051cf480 2024-04-21T23:08:17.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.180+0000 7fd66940d640 1 --2- 172.21.15.44:0/4146724061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051cf480 0x564d051e14f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd65802dc90 tx=0x7fd65800b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d799f1a8192c4034 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:17.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.180+0000 7fd64b7fe640 1 -- 172.21.15.44:0/4146724061 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658010c50 con 0x564d051cf480 2024-04-21T23:08:17.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.180+0000 7fd64b7fe640 1 -- 172.21.15.44:0/4146724061 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd658016030 con 0x564d051cf480 2024-04-21T23:08:17.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.180+0000 7fd64b7fe640 1 -- 172.21.15.44:0/4146724061 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65800ba60 con 0x564d051cf480 2024-04-21T23:08:17.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.180+0000 7fd66ea58740 1 -- 172.21.15.44:0/4146724061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051cf480 msgr2=0x564d051e14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:17.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.180+0000 7fd66ea58740 1 --2- 172.21.15.44:0/4146724061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051cf480 0x564d051e14f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd65802dc90 tx=0x7fd65800b9f0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.180+0000 7fd66ea58740 1 -- 172.21.15.44:0/4146724061 shutdown_connections 2024-04-21T23:08:17.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.180+0000 7fd66ea58740 1 --2- 172.21.15.44:0/4146724061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d051d4fe0 0x564d051d12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.180+0000 7fd66ea58740 1 --2- 172.21.15.44:0/4146724061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x564d051cfe70 0x564d051e0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.180+0000 7fd66ea58740 1 --2- 172.21.15.44:0/4146724061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051cf480 0x564d051e14f0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.180+0000 7fd66ea58740 1 -- 172.21.15.44:0/4146724061 >> 172.21.15.44:0/4146724061 conn(0x564d051c65e0 msgr2=0x564d051e52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:17.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.180+0000 7fd66ea58740 1 -- 172.21.15.44:0/4146724061 shutdown_connections 2024-04-21T23:08:17.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.180+0000 7fd66ea58740 1 -- 172.21.15.44:0/4146724061 wait complete. 2024-04-21T23:08:17.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.181+0000 7fd66ea58740 1 -- start start 2024-04-21T23:08:17.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.181+0000 7fd66ea58740 1 Processor -- start 2024-04-21T23:08:17.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.181+0000 7fd66ea58740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051cf480 0x564d051dc610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.181+0000 7fd66ea58740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x564d051cfe70 0x564d051dcb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.181+0000 7fd66ea58740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d051d4fe0 0x564d051df9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.181+0000 7fd66ea58740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x564d051e75b0 con 0x564d051cf480 2024-04-21T23:08:17.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.181+0000 7fd66ea58740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x564d051e6ee0 con 0x564d051cfe70 2024-04-21T23:08:17.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.181+0000 7fd66ea58740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564d051e7c80 con 0x564d051d4fe0 2024-04-21T23:08:17.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.181+0000 7fd66ea58740 10 client.0 _ll_register_callbacks cb 0x7fd658012690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:08:17.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.181+0000 7fd66940d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051cf480 0x564d051dc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:17.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.181+0000 7fd66940d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051cf480 0x564d051dc610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40674/0 (socket says 172.21.15.44:40674) 2024-04-21T23:08:17.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.181+0000 7fd66940d640 1 -- 172.21.15.44:0/3068972906 learned_addr learned my addr 172.21.15.44:0/3068972906 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:17.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.181+0000 7fd669c0e640 1 --2- 172.21.15.44:0/3068972906 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x564d051cfe70 0x564d051dcb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:17.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.182+0000 7fd66940d640 1 --2- 172.21.15.44:0/3068972906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d051d4fe0 0x564d051df9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:17.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.182+0000 7fd66940d640 1 -- 172.21.15.44:0/3068972906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d051d4fe0 msgr2=0x564d051df9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:17.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.182+0000 7fd66940d640 1 --2- 172.21.15.44:0/3068972906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d051d4fe0 0x564d051df9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.182+0000 7fd66940d640 1 -- 172.21.15.44:0/3068972906 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x564d051cfe70 msgr2=0x564d051dcb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:17.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.182+0000 7fd66940d640 1 --2- 172.21.15.44:0/3068972906 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x564d051cfe70 0x564d051dcb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.182+0000 7fd66940d640 1 -- 172.21.15.44:0/3068972906 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d051e0010 con 0x564d051cf480 2024-04-21T23:08:17.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.182+0000 7fd66940d640 1 --2- 172.21.15.44:0/3068972906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051cf480 0x564d051dc610 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd658000c00 tx=0x7fd65800b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd5559ba079980e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:17.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.182+0000 7fd6497fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:08:17.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.182+0000 7fd6497fa640 1 -- 172.21.15.44:0/3068972906 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65801edf0 con 0x564d051cf480 2024-04-21T23:08:17.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.182+0000 7fd6497fa640 1 -- 172.21.15.44:0/3068972906 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fd658010980 con 0x564d051cf480 2024-04-21T23:08:17.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.182+0000 7fd66ea58740 1 -- 172.21.15.44:0/3068972906 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd658016030 con 0x564d051cf480 2024-04-21T23:08:17.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.182+0000 7fd6497fa640 1 -- 172.21.15.44:0/3068972906 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd65801e640 con 0x564d051cf480 2024-04-21T23:08:17.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.182+0000 7fd66ea58740 10 client.15240 fetch_fsmap learned FSMap version 21 2024-04-21T23:08:17.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.182+0000 7fd66ea58740 1 -- 172.21.15.44:0/3068972906 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x564d0524c2d0 con 0x564d051cf480 2024-04-21T23:08:17.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.182+0000 7fd6497fa640 1 -- 172.21.15.44:0/3068972906 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd658016030 con 0x564d051cf480 2024-04-21T23:08:17.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.182+0000 7fd6497fa640 1 -- 172.21.15.44:0/3068972906 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fd65802a070 con 0x564d051cf480 2024-04-21T23:08:17.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.183+0000 7fd66ea58740 10 client.15240 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:08:17.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.183+0000 7fd66ea58740 10 client.15240 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T23:08:17.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.183+0000 7fd66ea58740 20 client.15240 populate_metadata read hostname 'smithi044' 2024-04-21T23:08:17.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.183+0000 7fd66ea58740 1 --2- 172.21.15.44:0/3068972906 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x564d052508c0 0x564d0524cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:17.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.183+0000 7fd66ea58740 4 client.15240 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T23:08:17.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.183+0000 7fd66ea58740 1 -- 172.21.15.44:0/3068972906 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "status"}) -- 0x564d05137680 con 0x564d052508c0 2024-04-21T23:08:17.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.183+0000 7fd669c0e640 1 --2- 172.21.15.44:0/3068972906 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x564d052508c0 0x564d0524cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:17.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.183+0000 7fd669c0e640 1 --2- 172.21.15.44:0/3068972906 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x564d052508c0 0x564d0524cd00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c91d1adc872bf805 server_cookie=3bb670a29e60b8ab in_seq=0 out_seq=0 2024-04-21T23:08:17.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.183+0000 7fd6497fa640 10 client.15240 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:08:17.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.183+0000 7fd6497fa640 1 -- 172.21.15.44:0/3068972906 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x564d05137680 con 0x564d052508c0 2024-04-21T23:08:17.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.183+0000 7fd6497fa640 10 client.15240 handle_command_reply: tid=0 2024-04-21T23:08:17.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.183+0000 7fd66ea58740 1 client.15240 shutdown 2024-04-21T23:08:17.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66ea58740 1 -- 172.21.15.44:0/3068972906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051cf480 msgr2=0x564d051dc610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:17.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66ea58740 1 --2- 172.21.15.44:0/3068972906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051cf480 0x564d051dc610 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd658000c00 tx=0x7fd65800b390 comp rx=0 tx=0).stop 2024-04-21T23:08:17.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66ea58740 1 -- 172.21.15.44:0/3068972906 shutdown_connections 2024-04-21T23:08:17.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66ea58740 1 --2- 172.21.15.44:0/3068972906 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x564d052508c0 0x564d0524cd00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66ea58740 1 --2- 172.21.15.44:0/3068972906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d051d4fe0 0x564d051df9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd6497fa640 0 client.15240 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:08:17.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66ea58740 1 --2- 172.21.15.44:0/3068972906 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x564d051cfe70 0x564d051dcb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66ea58740 1 --2- 172.21.15.44:0/3068972906 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x564d051cf480 0x564d051dc610 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66ea58740 1 -- 172.21.15.44:0/3068972906 >> 172.21.15.44:0/3068972906 conn(0x564d051c65e0 msgr2=0x564d051d12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:17.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66ea58740 1 -- 172.21.15.44:0/3068972906 shutdown_connections 2024-04-21T23:08:17.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66ea58740 1 -- 172.21.15.44:0/3068972906 wait complete. 2024-04-21T23:08:17.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66ea58740 20 client.15240 trim_cache size 0 max 16384 2024-04-21T23:08:17.191 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:08:17.191 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "80909b94-0032-11ef-bc93-c7b262605968", 2024-04-21T23:08:17.191 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-04-21T23:08:17.191 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24530, 2024-04-21T23:08:17.191 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-04-21T23:08:17.191 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-04-21T23:08:17.191 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-04-21T23:08:17.191 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 110.97687158700001, 2024-04-21T23:08:17.191 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 21, 2024-04-21T23:08:17.191 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 92, 2024-04-21T23:08:17.191 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 90, 2024-04-21T23:08:17.191 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 115.064290864 2024-04-21T23:08:17.191 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:08:17.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66be72640 1 -- 172.21.15.44:0/3791972737 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd64c0a2c20 msgr2=0x7fd64c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:17.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66be72640 1 --2- 172.21.15.44:0/3791972737 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd64c0a2c20 0x7fd64c0a50e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd65c09b0a0 tx=0x7fd65c0905f0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66be72640 1 -- 172.21.15.44:0/3791972737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd664103350 msgr2=0x7fd66419f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:17.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66be72640 1 --2- 172.21.15.44:0/3791972737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd664103350 0x7fd66419f060 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00d950 tx=0x7fd65c00de10 comp rx=0 tx=0).stop 2024-04-21T23:08:17.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66be72640 1 -- 172.21.15.44:0/3791972737 shutdown_connections 2024-04-21T23:08:17.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66be72640 1 --2- 172.21.15.44:0/3791972737 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd64c0a2c20 0x7fd64c0a50e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66be72640 1 --2- 172.21.15.44:0/3791972737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd664103350 0x7fd66419f060 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66be72640 1 --2- 172.21.15.44:0/3791972737 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd664102990 0x7fd664197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66be72640 1 --2- 172.21.15.44:0/3791972737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6641017a0 0x7fd664197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:17.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66be72640 1 -- 172.21.15.44:0/3791972737 >> 172.21.15.44:0/3791972737 conn(0x7fd6640fcd10 msgr2=0x7fd664105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:17.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66be72640 1 -- 172.21.15.44:0/3791972737 shutdown_connections 2024-04-21T23:08:17.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:17.184+0000 7fd66be72640 1 -- 172.21.15.44:0/3791972737 wait complete. 2024-04-21T23:08:17.195 INFO:tasks.cephadm:Setup complete, yielding 2024-04-21T23:08:17.195 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-21T23:08:17.205 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-04-21T23:08:17.206 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-21T23:08:17.454 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:08:17.820 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:17 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/979797818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:17.820 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:17 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/383950291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:17.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:17 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/979797818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:17.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:17 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/383950291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:17.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:17 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/979797818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:17.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:17 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/383950291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:18.880 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:18 smithi044 ceph-mon[27934]: pgmap v349: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:18.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:18 smithi055 ceph-mon[31117]: pgmap v349: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:18.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:18 smithi069 ceph-mon[31816]: pgmap v349: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:19.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.221+0000 7ff000036700 1 -- 172.21.15.44:0/49675827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff8100340 msgr2=0x7feff81007c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:19.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.221+0000 7ff000036700 1 --2- 172.21.15.44:0/49675827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff8100340 0x7feff81007c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7feff40099c0 tx=0x7feff4009cd0 comp rx=0 tx=0).stop 2024-04-21T23:08:19.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.221+0000 7ff000036700 1 -- 172.21.15.44:0/49675827 shutdown_connections 2024-04-21T23:08:19.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.221+0000 7ff000036700 1 --2- 172.21.15.44:0/49675827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feff8100d00 0x7feff810a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:19.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.221+0000 7ff000036700 1 --2- 172.21.15.44:0/49675827 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feff81038c0 0x7feff8103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:19.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.221+0000 7ff000036700 1 --2- 172.21.15.44:0/49675827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff8100340 0x7feff81007c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:19.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.221+0000 7ff000036700 1 -- 172.21.15.44:0/49675827 >> 172.21.15.44:0/49675827 conn(0x7feff80fbee0 msgr2=0x7feff80fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:19.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.221+0000 7ff000036700 1 -- 172.21.15.44:0/49675827 shutdown_connections 2024-04-21T23:08:19.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.221+0000 7ff000036700 1 -- 172.21.15.44:0/49675827 wait complete. 2024-04-21T23:08:19.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.222+0000 7ff000036700 1 Processor -- start 2024-04-21T23:08:19.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.222+0000 7ff000036700 1 -- start start 2024-04-21T23:08:19.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.222+0000 7ff000036700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff8100340 0x7feff81985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:19.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.222+0000 7ff000036700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feff8100d00 0x7feff8198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:19.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.222+0000 7ff000036700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feff81038c0 0x7feff819fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:19.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.222+0000 7ff000036700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7feff810dfc0 con 0x7feff8100340 2024-04-21T23:08:19.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.222+0000 7ff000036700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7feff810de40 con 0x7feff8100d00 2024-04-21T23:08:19.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.222+0000 7ff000036700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7feff810e140 con 0x7feff81038c0 2024-04-21T23:08:19.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.222+0000 7feffddd2700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff8100340 0x7feff81985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:19.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.222+0000 7feffddd2700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff8100340 0x7feff81985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40686/0 (socket says 172.21.15.44:40686) 2024-04-21T23:08:19.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.222+0000 7feffddd2700 1 -- 172.21.15.44:0/3671309086 learned_addr learned my addr 172.21.15.44:0/3671309086 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:19.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.222+0000 7feffe5d3700 1 --2- 172.21.15.44:0/3671309086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feff81038c0 0x7feff819fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:19.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.222+0000 7feffd5d1700 1 --2- 172.21.15.44:0/3671309086 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feff8100d00 0x7feff8198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:19.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.223+0000 7feffddd2700 1 -- 172.21.15.44:0/3671309086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feff81038c0 msgr2=0x7feff819fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:19.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.223+0000 7feffddd2700 1 --2- 172.21.15.44:0/3671309086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feff81038c0 0x7feff819fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:19.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.223+0000 7feffddd2700 1 -- 172.21.15.44:0/3671309086 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feff8100d00 msgr2=0x7feff8198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:19.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.223+0000 7feffddd2700 1 --2- 172.21.15.44:0/3671309086 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feff8100d00 0x7feff8198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:19.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.223+0000 7feffddd2700 1 -- 172.21.15.44:0/3671309086 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feff81a0230 con 0x7feff8100340 2024-04-21T23:08:19.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.223+0000 7feffd5d1700 1 --2- 172.21.15.44:0/3671309086 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feff8100d00 0x7feff8198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:08:19.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.223+0000 7feffe5d3700 1 --2- 172.21.15.44:0/3671309086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feff81038c0 0x7feff819fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:08:19.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.223+0000 7feffddd2700 1 --2- 172.21.15.44:0/3671309086 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff8100340 0x7feff81985d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fefe800b530 tx=0x7fefe800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:19.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.223+0000 7fefeeffd700 1 -- 172.21.15.44:0/3671309086 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe8015020 con 0x7feff8100340 2024-04-21T23:08:19.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.223+0000 7fefeeffd700 1 -- 172.21.15.44:0/3671309086 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefe8011d10 con 0x7feff8100340 2024-04-21T23:08:19.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.223+0000 7ff000036700 1 -- 172.21.15.44:0/3671309086 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feff81a0520 con 0x7feff8100340 2024-04-21T23:08:19.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.223+0000 7fefeeffd700 1 -- 172.21.15.44:0/3671309086 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe800f2e0 con 0x7feff8100340 2024-04-21T23:08:19.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.223+0000 7ff000036700 1 -- 172.21.15.44:0/3671309086 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feff81a0b30 con 0x7feff8100340 2024-04-21T23:08:19.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.224+0000 7ff000036700 1 -- 172.21.15.44:0/3671309086 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feff8068f20 con 0x7feff8100340 2024-04-21T23:08:19.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.226+0000 7fefeeffd700 1 -- 172.21.15.44:0/3671309086 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fefe8018020 con 0x7feff8100340 2024-04-21T23:08:19.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.226+0000 7fefeeffd700 1 --2- 172.21.15.44:0/3671309086 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fefe40a2d30 0x7fefe40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:19.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.226+0000 7feffd5d1700 1 --2- 172.21.15.44:0/3671309086 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fefe40a2d30 0x7fefe40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:19.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.226+0000 7fefeeffd700 1 -- 172.21.15.44:0/3671309086 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fefe80cf0e0 con 0x7feff8100340 2024-04-21T23:08:19.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.227+0000 7feffd5d1700 1 --2- 172.21.15.44:0/3671309086 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fefe40a2d30 0x7fefe40a51f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7feff400b580 tx=0x7feff4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:19.230 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.230+0000 7fefeeffd700 1 -- 172.21.15.44:0/3671309086 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fefe801f020 con 0x7feff8100340 2024-04-21T23:08:19.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.458+0000 7ff000036700 1 -- 172.21.15.44:0/3671309086 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7feff8062670 con 0x7fefe40a2d30 2024-04-21T23:08:19.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.459+0000 7fefeeffd700 1 -- 172.21.15.44:0/3671309086 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7feff8062670 con 0x7fefe40a2d30 2024-04-21T23:08:19.459 INFO:teuthology.orchestra.run.smithi044.stdout:Backend: cephadm 2024-04-21T23:08:19.459 INFO:teuthology.orchestra.run.smithi044.stdout:Available: Yes 2024-04-21T23:08:19.459 INFO:teuthology.orchestra.run.smithi044.stdout:Paused: No 2024-04-21T23:08:19.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.461+0000 7ff000036700 1 -- 172.21.15.44:0/3671309086 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fefe40a2d30 msgr2=0x7fefe40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:19.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.461+0000 7ff000036700 1 --2- 172.21.15.44:0/3671309086 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fefe40a2d30 0x7fefe40a51f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7feff400b580 tx=0x7feff4005fb0 comp rx=0 tx=0).stop 2024-04-21T23:08:19.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.461+0000 7ff000036700 1 -- 172.21.15.44:0/3671309086 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff8100340 msgr2=0x7feff81985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:19.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.461+0000 7ff000036700 1 --2- 172.21.15.44:0/3671309086 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff8100340 0x7feff81985d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fefe800b530 tx=0x7fefe800b8f0 comp rx=0 tx=0).stop 2024-04-21T23:08:19.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.461+0000 7ff000036700 1 -- 172.21.15.44:0/3671309086 shutdown_connections 2024-04-21T23:08:19.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.461+0000 7ff000036700 1 --2- 172.21.15.44:0/3671309086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feff81038c0 0x7feff819fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:19.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.461+0000 7ff000036700 1 --2- 172.21.15.44:0/3671309086 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fefe40a2d30 0x7fefe40a51f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:19.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.461+0000 7ff000036700 1 --2- 172.21.15.44:0/3671309086 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feff8100340 0x7feff81985d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:19.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.461+0000 7ff000036700 1 --2- 172.21.15.44:0/3671309086 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feff8100d00 0x7feff8198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:19.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.462+0000 7ff000036700 1 -- 172.21.15.44:0/3671309086 >> 172.21.15.44:0/3671309086 conn(0x7feff80fbee0 msgr2=0x7feff8106960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:19.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.462+0000 7ff000036700 1 -- 172.21.15.44:0/3671309086 shutdown_connections 2024-04-21T23:08:19.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:19.462+0000 7ff000036700 1 -- 172.21.15.44:0/3671309086 wait complete. 2024-04-21T23:08:20.201 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-21T23:08:20.454 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:08:20.855 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:20 smithi044 ceph-mon[27934]: pgmap v350: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:20.856 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:20 smithi044 ceph-mon[27934]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:20.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:20 smithi055 ceph-mon[31117]: pgmap v350: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:20.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:20 smithi055 ceph-mon[31117]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:20.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:20 smithi069 ceph-mon[31816]: pgmap v350: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:20.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:20 smithi069 ceph-mon[31816]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.315+0000 7f20f2740700 1 -- 172.21.15.44:0/2842336355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec1034b0 msgr2=0x7f20ec103910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.315+0000 7f20f2740700 1 --2- 172.21.15.44:0/2842336355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec1034b0 0x7f20ec103910 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f20e00099c0 tx=0x7f20e0009cd0 comp rx=0 tx=0).stop 2024-04-21T23:08:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.315+0000 7f20f2740700 1 -- 172.21.15.44:0/2842336355 shutdown_connections 2024-04-21T23:08:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.315+0000 7f20f2740700 1 --2- 172.21.15.44:0/2842336355 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20ec103e50 0x7f20ec10a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.315+0000 7f20f2740700 1 --2- 172.21.15.44:0/2842336355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20ec1022b0 0x7f20ec1026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.315+0000 7f20f2740700 1 --2- 172.21.15.44:0/2842336355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec1034b0 0x7f20ec103910 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.315+0000 7f20f2740700 1 -- 172.21.15.44:0/2842336355 >> 172.21.15.44:0/2842336355 conn(0x7f20ec0fd850 msgr2=0x7f20ec0ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.315+0000 7f20f2740700 1 -- 172.21.15.44:0/2842336355 shutdown_connections 2024-04-21T23:08:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.315+0000 7f20f2740700 1 -- 172.21.15.44:0/2842336355 wait complete. 2024-04-21T23:08:22.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.315+0000 7f20f2740700 1 Processor -- start 2024-04-21T23:08:22.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.315+0000 7f20f2740700 1 -- start start 2024-04-21T23:08:22.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.316+0000 7f20f2740700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec1022b0 0x7f20ec075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:22.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.316+0000 7f20f2740700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20ec1034b0 0x7f20ec074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:22.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.316+0000 7f20f2740700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20ec103e50 0x7f20ec074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:22.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.316+0000 7f20f2740700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f20ec10dea0 con 0x7f20ec1022b0 2024-04-21T23:08:22.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.316+0000 7f20f2740700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f20ec10dd20 con 0x7f20ec103e50 2024-04-21T23:08:22.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.316+0000 7f20f2740700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f20ec10e020 con 0x7f20ec1034b0 2024-04-21T23:08:22.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.316+0000 7f20ebfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec1022b0 0x7f20ec075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:22.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.316+0000 7f20ebfff700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec1022b0 0x7f20ec075fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50778/0 (socket says 172.21.15.44:50778) 2024-04-21T23:08:22.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.316+0000 7f20ebfff700 1 -- 172.21.15.44:0/3807057743 learned_addr learned my addr 172.21.15.44:0/3807057743 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:22.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.316+0000 7f20eb7fe700 1 --2- 172.21.15.44:0/3807057743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20ec1034b0 0x7f20ec074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:22.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.316+0000 7f20f0cdd700 1 --2- 172.21.15.44:0/3807057743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20ec103e50 0x7f20ec074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:22.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.316+0000 7f20ebfff700 1 -- 172.21.15.44:0/3807057743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20ec1034b0 msgr2=0x7f20ec074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:22.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.316+0000 7f20ebfff700 1 --2- 172.21.15.44:0/3807057743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20ec1034b0 0x7f20ec074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:22.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.316+0000 7f20ebfff700 1 -- 172.21.15.44:0/3807057743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20ec103e50 msgr2=0x7f20ec074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:22.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.316+0000 7f20ebfff700 1 --2- 172.21.15.44:0/3807057743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20ec103e50 0x7f20ec074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:22.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.316+0000 7f20ebfff700 1 -- 172.21.15.44:0/3807057743 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20ec075410 con 0x7f20ec1022b0 2024-04-21T23:08:22.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.316+0000 7f20ebfff700 1 --2- 172.21.15.44:0/3807057743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec1022b0 0x7f20ec075fd0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f20dc00e950 tx=0x7f20dc00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:22.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.317+0000 7f20e97fa700 1 -- 172.21.15.44:0/3807057743 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20dc00cbd0 con 0x7f20ec1022b0 2024-04-21T23:08:22.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.317+0000 7f20f2740700 1 -- 172.21.15.44:0/3807057743 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20ec1a2620 con 0x7f20ec1022b0 2024-04-21T23:08:22.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.317+0000 7f20e97fa700 1 -- 172.21.15.44:0/3807057743 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20dc004510 con 0x7f20ec1022b0 2024-04-21T23:08:22.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.317+0000 7f20e97fa700 1 -- 172.21.15.44:0/3807057743 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20dc010430 con 0x7f20ec1022b0 2024-04-21T23:08:22.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.317+0000 7f20f2740700 1 -- 172.21.15.44:0/3807057743 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20ec1a2ae0 con 0x7f20ec1022b0 2024-04-21T23:08:22.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.318+0000 7f20f2740700 1 -- 172.21.15.44:0/3807057743 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20ec068f20 con 0x7f20ec1022b0 2024-04-21T23:08:22.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.319+0000 7f20e97fa700 1 -- 172.21.15.44:0/3807057743 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f20dc003f30 con 0x7f20ec1022b0 2024-04-21T23:08:22.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.319+0000 7f20e97fa700 1 --2- 172.21.15.44:0/3807057743 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f20d40a2cb0 0x7f20d40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:22.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.319+0000 7f20e97fa700 1 -- 172.21.15.44:0/3807057743 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f20dc0cf9c0 con 0x7f20ec1022b0 2024-04-21T23:08:22.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.319+0000 7f20eb7fe700 1 --2- 172.21.15.44:0/3807057743 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f20d40a2cb0 0x7f20d40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:22.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.320+0000 7f20eb7fe700 1 --2- 172.21.15.44:0/3807057743 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f20d40a2cb0 0x7f20d40a5170 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f20e0000c00 tx=0x7f20e0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:22.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.323+0000 7f20e97fa700 1 -- 172.21.15.44:0/3807057743 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f20dc09c480 con 0x7f20ec1022b0 2024-04-21T23:08:22.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.539+0000 7f20f2740700 1 -- 172.21.15.44:0/3807057743 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f20ec062670 con 0x7f20d40a2cb0 2024-04-21T23:08:22.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.552+0000 7f20e97fa700 1 -- 172.21.15.44:0/3807057743 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4092 (secure 0 0 0) 0x7f20ec062670 con 0x7f20d40a2cb0 2024-04-21T23:08:22.552 INFO:teuthology.orchestra.run.smithi044.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-21T23:08:22.552 INFO:teuthology.orchestra.run.smithi044.stdout:mds.a smithi044 running (2m) 19s ago 2m 12.7M - 19.0.0-2405-g820a2f8c 6702148920ad 8d3075c20f53 2024-04-21T23:08:22.552 INFO:teuthology.orchestra.run.smithi044.stdout:mds.b smithi055 running (2m) 53s ago 2m 25.9M - 19.0.0-2405-g820a2f8c 6702148920ad c87ce9581cca 2024-04-21T23:08:22.552 INFO:teuthology.orchestra.run.smithi044.stdout:mds.c smithi069 running (2m) 53s ago 2m 16.2M - 19.0.0-2405-g820a2f8c 6702148920ad ce10f093697d 2024-04-21T23:08:22.553 INFO:teuthology.orchestra.run.smithi044.stdout:mds.d smithi044 running (2m) 19s ago 2m 16.2M - 19.0.0-2405-g820a2f8c 6702148920ad 126f059275fe 2024-04-21T23:08:22.553 INFO:teuthology.orchestra.run.smithi044.stdout:mds.e smithi055 running (2m) 53s ago 2m 13.9M - 19.0.0-2405-g820a2f8c 6702148920ad b2cad9b74fe4 2024-04-21T23:08:22.553 INFO:teuthology.orchestra.run.smithi044.stdout:mds.f smithi069 running (2m) 53s ago 2m 13.9M - 19.0.0-2405-g820a2f8c 6702148920ad 81754f9122c7 2024-04-21T23:08:22.553 INFO:teuthology.orchestra.run.smithi044.stdout:mds.g smithi044 running (2m) 19s ago 2m 13.9M - 19.0.0-2405-g820a2f8c 6702148920ad 0bc183c485bb 2024-04-21T23:08:22.553 INFO:teuthology.orchestra.run.smithi044.stdout:mds.h smithi055 running (2m) 53s ago 2m 13.7M - 19.0.0-2405-g820a2f8c 6702148920ad 4ecdcab93f9b 2024-04-21T23:08:22.553 INFO:teuthology.orchestra.run.smithi044.stdout:mds.i smithi069 running (2m) 53s ago 2m 18.6M - 19.0.0-2405-g820a2f8c 6702148920ad fa6d4b0214a3 2024-04-21T23:08:22.553 INFO:teuthology.orchestra.run.smithi044.stdout:mds.j smithi044 running (2m) 19s ago 2m 14.0M - 19.0.0-2405-g820a2f8c 6702148920ad a058353c7bf2 2024-04-21T23:08:22.553 INFO:teuthology.orchestra.run.smithi044.stdout:mds.k smithi055 running (2m) 53s ago 2m 13.0M - 19.0.0-2405-g820a2f8c 6702148920ad 3800890d2e1e 2024-04-21T23:08:22.553 INFO:teuthology.orchestra.run.smithi044.stdout:mds.l smithi069 running (2m) 53s ago 2m 15.4M - 19.0.0-2405-g820a2f8c 6702148920ad 4512bfa6f430 2024-04-21T23:08:22.553 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.x smithi044 *:9283,8765 running (9m) 19s ago 9m 514M - 19.0.0-2405-g820a2f8c 6702148920ad 4543c9b91d4a 2024-04-21T23:08:22.553 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.y smithi055 *:8443,8765 running (7m) 53s ago 7m 454M - 19.0.0-2405-g820a2f8c 6702148920ad 43fd150ed11a 2024-04-21T23:08:22.553 INFO:teuthology.orchestra.run.smithi044.stdout:mgr.z smithi069 *:8443,8765 running (7m) 53s ago 7m 453M - 19.0.0-2405-g820a2f8c 6702148920ad 2fc9ae974ee6 2024-04-21T23:08:22.553 INFO:teuthology.orchestra.run.smithi044.stdout:mon.a smithi044 running (9m) 19s ago 9m 49.0M 2048M 19.0.0-2405-g820a2f8c 6702148920ad 50bff5b2e0ac 2024-04-21T23:08:22.553 INFO:teuthology.orchestra.run.smithi044.stdout:mon.b smithi055 running (8m) 53s ago 8m 39.7M 2048M 19.0.0-2405-g820a2f8c 6702148920ad 64497f39ca46 2024-04-21T23:08:22.554 INFO:teuthology.orchestra.run.smithi044.stdout:mon.c smithi069 running (8m) 53s ago 8m 41.3M 2048M 19.0.0-2405-g820a2f8c 6702148920ad 3b503e5b6096 2024-04-21T23:08:22.554 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 smithi044 running (7m) 19s ago 7m 46.0M 4273M 19.0.0-2405-g820a2f8c 6702148920ad 77e17c2104ba 2024-04-21T23:08:22.554 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 smithi055 running (7m) 53s ago 7m 57.6M 4273M 19.0.0-2405-g820a2f8c 6702148920ad 2fe887551d28 2024-04-21T23:08:22.554 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 smithi069 running (6m) 53s ago 6m 46.7M 4272M 19.0.0-2405-g820a2f8c 6702148920ad d62a10db9042 2024-04-21T23:08:22.554 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 smithi044 running (6m) 19s ago 6m 47.1M 4273M 19.0.0-2405-g820a2f8c 6702148920ad 1897a0334076 2024-04-21T23:08:22.554 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 smithi055 running (5m) 53s ago 5m 45.4M 4273M 19.0.0-2405-g820a2f8c 6702148920ad 372c604fcc94 2024-04-21T23:08:22.554 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 smithi069 running (5m) 53s ago 5m 44.2M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 02df57ff1359 2024-04-21T23:08:22.554 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 smithi044 running (5m) 19s ago 5m 42.1M 4273M 19.0.0-2405-g820a2f8c 6702148920ad 13bded7a8ab4 2024-04-21T23:08:22.554 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 smithi055 running (4m) 53s ago 4m 44.8M 4273M 19.0.0-2405-g820a2f8c 6702148920ad a6eb6a3d6865 2024-04-21T23:08:22.554 INFO:teuthology.orchestra.run.smithi044.stdout:osd.8 smithi069 running (4m) 53s ago 4m 53.9M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 16976d762e55 2024-04-21T23:08:22.554 INFO:teuthology.orchestra.run.smithi044.stdout:osd.9 smithi044 running (3m) 19s ago 3m 42.9M 4273M 19.0.0-2405-g820a2f8c 6702148920ad 6c5d54bdf816 2024-04-21T23:08:22.554 INFO:teuthology.orchestra.run.smithi044.stdout:osd.10 smithi055 running (3m) 53s ago 3m 51.8M 4273M 19.0.0-2405-g820a2f8c 6702148920ad 78a7d8836a7c 2024-04-21T23:08:22.554 INFO:teuthology.orchestra.run.smithi044.stdout:osd.11 smithi069 running (2m) 53s ago 2m 51.5M 4272M 19.0.0-2405-g820a2f8c 6702148920ad 7f59ec119856 2024-04-21T23:08:22.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.555+0000 7f20f2740700 1 -- 172.21.15.44:0/3807057743 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f20d40a2cb0 msgr2=0x7f20d40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:22.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.555+0000 7f20f2740700 1 --2- 172.21.15.44:0/3807057743 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f20d40a2cb0 0x7f20d40a5170 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f20e0000c00 tx=0x7f20e0005fd0 comp rx=0 tx=0).stop 2024-04-21T23:08:22.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.555+0000 7f20f2740700 1 -- 172.21.15.44:0/3807057743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec1022b0 msgr2=0x7f20ec075fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:22.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.555+0000 7f20f2740700 1 --2- 172.21.15.44:0/3807057743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec1022b0 0x7f20ec075fd0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f20dc00e950 tx=0x7f20dc00ed10 comp rx=0 tx=0).stop 2024-04-21T23:08:22.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.555+0000 7f20f2740700 1 -- 172.21.15.44:0/3807057743 shutdown_connections 2024-04-21T23:08:22.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.555+0000 7f20f2740700 1 --2- 172.21.15.44:0/3807057743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20ec103e50 0x7f20ec074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:22.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.555+0000 7f20f2740700 1 --2- 172.21.15.44:0/3807057743 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f20d40a2cb0 0x7f20d40a5170 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:22.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.555+0000 7f20f2740700 1 --2- 172.21.15.44:0/3807057743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f20ec1022b0 0x7f20ec075fd0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:22.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.555+0000 7f20f2740700 1 --2- 172.21.15.44:0/3807057743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20ec1034b0 0x7f20ec074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:22.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.555+0000 7f20f2740700 1 -- 172.21.15.44:0/3807057743 >> 172.21.15.44:0/3807057743 conn(0x7f20ec0fd850 msgr2=0x7f20ec1067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:22.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.555+0000 7f20f2740700 1 -- 172.21.15.44:0/3807057743 shutdown_connections 2024-04-21T23:08:22.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:22.555+0000 7f20f2740700 1 -- 172.21.15.44:0/3807057743 wait complete. 2024-04-21T23:08:22.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:22 smithi044 ceph-mon[27934]: pgmap v351: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:08:22.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:22 smithi055 ceph-mon[31117]: pgmap v351: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:08:22.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:22 smithi069 ceph-mon[31816]: pgmap v351: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:08:23.348 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-21T23:08:23.596 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:08:23.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:23 smithi044 ceph-mon[27934]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:23.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-21T23:08:23.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-04-21T23:08:23.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-04-21T23:08:23.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:23 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-04-21T23:08:23.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:23 smithi055 ceph-mon[31117]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:23.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-21T23:08:23.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-04-21T23:08:23.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-04-21T23:08:23.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:23 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-04-21T23:08:23.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:23 smithi069 ceph-mon[31816]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:23.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-21T23:08:23.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-04-21T23:08:23.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-04-21T23:08:23.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:23 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-04-21T23:08:24.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:24 smithi044 ceph-mon[27934]: pgmap v352: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:24.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-21T23:08:24.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-04-21T23:08:24.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-04-21T23:08:24.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:24 smithi044 ceph-mon[27934]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-04-21T23:08:24.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:24 smithi044 ceph-mon[27934]: osdmap e93: 12 total, 12 up, 12 in 2024-04-21T23:08:24.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:24 smithi044 ceph-mon[27934]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-21T23:08:24.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:24 smithi069 ceph-mon[31816]: pgmap v352: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:24.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-21T23:08:24.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-04-21T23:08:24.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-04-21T23:08:24.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:24 smithi069 ceph-mon[31816]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-04-21T23:08:24.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:24 smithi069 ceph-mon[31816]: osdmap e93: 12 total, 12 up, 12 in 2024-04-21T23:08:24.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:24 smithi069 ceph-mon[31816]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-21T23:08:24.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:24 smithi055 ceph-mon[31117]: pgmap v352: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:08:24.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-21T23:08:24.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-04-21T23:08:24.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-04-21T23:08:24.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:24 smithi055 ceph-mon[31117]: from='mgr.14152 172.21.15.44:0/1404728053' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-04-21T23:08:24.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:24 smithi055 ceph-mon[31117]: osdmap e93: 12 total, 12 up, 12 in 2024-04-21T23:08:24.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:24 smithi055 ceph-mon[31117]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-21T23:08:25.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.365+0000 7fc7a4085700 1 -- 172.21.15.44:0/711498988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc79c108020 msgr2=0x7fc79c10a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:25.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.365+0000 7fc7a4085700 1 --2- 172.21.15.44:0/711498988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc79c108020 0x7fc79c10a410 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc79400b0a0 tx=0x7fc79400b3b0 comp rx=0 tx=0).stop 2024-04-21T23:08:25.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.365+0000 7fc7a4085700 1 -- 172.21.15.44:0/711498988 shutdown_connections 2024-04-21T23:08:25.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.365+0000 7fc7a4085700 1 --2- 172.21.15.44:0/711498988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc79c108020 0x7fc79c10a410 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:25.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.365+0000 7fc7a4085700 1 --2- 172.21.15.44:0/711498988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc79c102dc0 0x7fc79c1051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:25.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.365+0000 7fc7a4085700 1 --2- 172.21.15.44:0/711498988 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc79c1056f0 0x7fc79c107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:25.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.365+0000 7fc7a4085700 1 -- 172.21.15.44:0/711498988 >> 172.21.15.44:0/711498988 conn(0x7fc79c0fc790 msgr2=0x7fc79c0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:25.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.366+0000 7fc7a4085700 1 -- 172.21.15.44:0/711498988 shutdown_connections 2024-04-21T23:08:25.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.366+0000 7fc7a4085700 1 -- 172.21.15.44:0/711498988 wait complete. 2024-04-21T23:08:25.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.366+0000 7fc7a4085700 1 Processor -- start 2024-04-21T23:08:25.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.366+0000 7fc7a4085700 1 -- start start 2024-04-21T23:08:25.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.366+0000 7fc7a4085700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc79c102dc0 0x7fc79c196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:25.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.366+0000 7fc7a4085700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc79c1056f0 0x7fc79c196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:25.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.367+0000 7fc7a4085700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc79c108020 0x7fc79c1a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:25.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.367+0000 7fc7a4085700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc79c10df60 con 0x7fc79c108020 2024-04-21T23:08:25.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.367+0000 7fc7a4085700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc79c10dde0 con 0x7fc79c1056f0 2024-04-21T23:08:25.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.367+0000 7fc7a4085700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc79c10e0e0 con 0x7fc79c102dc0 2024-04-21T23:08:25.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.367+0000 7fc7a2622700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc79c108020 0x7fc79c1a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:25.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.367+0000 7fc7a2622700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc79c108020 0x7fc79c1a6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50784/0 (socket says 172.21.15.44:50784) 2024-04-21T23:08:25.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.367+0000 7fc7a2622700 1 -- 172.21.15.44:0/1198230249 learned_addr learned my addr 172.21.15.44:0/1198230249 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:25.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.367+0000 7fc7a1620700 1 --2- 172.21.15.44:0/1198230249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc79c1056f0 0x7fc79c196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:25.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.367+0000 7fc7a1e21700 1 --2- 172.21.15.44:0/1198230249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc79c102dc0 0x7fc79c196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:25.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.367+0000 7fc7a2622700 1 -- 172.21.15.44:0/1198230249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc79c102dc0 msgr2=0x7fc79c196310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:25.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.367+0000 7fc7a2622700 1 --2- 172.21.15.44:0/1198230249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc79c102dc0 0x7fc79c196310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:25.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.367+0000 7fc7a2622700 1 -- 172.21.15.44:0/1198230249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc79c1056f0 msgr2=0x7fc79c196850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:25.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.367+0000 7fc7a2622700 1 --2- 172.21.15.44:0/1198230249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc79c1056f0 0x7fc79c196850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:25.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.367+0000 7fc7a2622700 1 -- 172.21.15.44:0/1198230249 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc79c1a67d0 con 0x7fc79c108020 2024-04-21T23:08:25.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.367+0000 7fc7a2622700 1 --2- 172.21.15.44:0/1198230249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc79c108020 0x7fc79c1a6290 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc794007f00 tx=0x7fc794004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:25.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.367+0000 7fc792ffd700 1 -- 172.21.15.44:0/1198230249 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc794021070 con 0x7fc79c108020 2024-04-21T23:08:25.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.367+0000 7fc792ffd700 1 -- 172.21.15.44:0/1198230249 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc79401d930 con 0x7fc79c108020 2024-04-21T23:08:25.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.367+0000 7fc7a4085700 1 -- 172.21.15.44:0/1198230249 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc79c1a6a60 con 0x7fc79c108020 2024-04-21T23:08:25.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.367+0000 7fc792ffd700 1 -- 172.21.15.44:0/1198230249 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7940055a0 con 0x7fc79c108020 2024-04-21T23:08:25.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.368+0000 7fc7a4085700 1 -- 172.21.15.44:0/1198230249 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc79c1a6fe0 con 0x7fc79c108020 2024-04-21T23:08:25.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.369+0000 7fc7a4085700 1 -- 172.21.15.44:0/1198230249 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc79c068f20 con 0x7fc79c108020 2024-04-21T23:08:25.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.370+0000 7fc792ffd700 1 -- 172.21.15.44:0/1198230249 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc794024070 con 0x7fc79c108020 2024-04-21T23:08:25.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.371+0000 7fc792ffd700 1 --2- 172.21.15.44:0/1198230249 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc7880a2ca0 0x7fc7880a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:25.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.371+0000 7fc792ffd700 1 -- 172.21.15.44:0/1198230249 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7fc79400f020 con 0x7fc79c108020 2024-04-21T23:08:25.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.371+0000 7fc7a1e21700 1 --2- 172.21.15.44:0/1198230249 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc7880a2ca0 0x7fc7880a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:25.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.371+0000 7fc7a1e21700 1 --2- 172.21.15.44:0/1198230249 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc7880a2ca0 0x7fc7880a5160 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc798006fd0 tx=0x7fc798008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:25.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.375+0000 7fc792ffd700 1 -- 172.21.15.44:0/1198230249 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc79402b020 con 0x7fc79c108020 2024-04-21T23:08:25.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.592+0000 7fc7a4085700 1 -- 172.21.15.44:0/1198230249 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fc79c062670 con 0x7fc7880a2ca0 2024-04-21T23:08:25.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.598+0000 7fc792ffd700 1 -- 172.21.15.44:0/1198230249 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fc79c062670 con 0x7fc7880a2ca0 2024-04-21T23:08:25.598 INFO:teuthology.orchestra.run.smithi044.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-21T23:08:25.598 INFO:teuthology.orchestra.run.smithi044.stdout:mds.all 12/12 56s ago 2m smithi044=a;smithi044=d;smithi044=g;smithi044=j;smithi055=b;smithi055=e;smithi055=h;smithi055=k;smithi069=c;smithi069=f;smithi069=i;smithi069=l;count:12 2024-04-21T23:08:25.598 INFO:teuthology.orchestra.run.smithi044.stdout:mgr 3/3 56s ago 7m smithi044=x;smithi055=y;smithi069=z;count:3 2024-04-21T23:08:25.598 INFO:teuthology.orchestra.run.smithi044.stdout:mon 3/3 56s ago 8m smithi044:172.21.15.44=a;smithi055:172.21.15.55=b;smithi069:172.21.15.69=c;count:3 2024-04-21T23:08:25.598 INFO:teuthology.orchestra.run.smithi044.stdout:osd 12 56s ago - 2024-04-21T23:08:25.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.600+0000 7fc7a4085700 1 -- 172.21.15.44:0/1198230249 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc7880a2ca0 msgr2=0x7fc7880a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:25.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.600+0000 7fc7a4085700 1 --2- 172.21.15.44:0/1198230249 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc7880a2ca0 0x7fc7880a5160 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc798006fd0 tx=0x7fc798008040 comp rx=0 tx=0).stop 2024-04-21T23:08:25.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.600+0000 7fc7a4085700 1 -- 172.21.15.44:0/1198230249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc79c108020 msgr2=0x7fc79c1a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.600+0000 7fc7a4085700 1 --2- 172.21.15.44:0/1198230249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc79c108020 0x7fc79c1a6290 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc794007f00 tx=0x7fc794004e40 comp rx=0 tx=0).stop 2024-04-21T23:08:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.601+0000 7fc7a4085700 1 -- 172.21.15.44:0/1198230249 shutdown_connections 2024-04-21T23:08:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.601+0000 7fc7a4085700 1 --2- 172.21.15.44:0/1198230249 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc7880a2ca0 0x7fc7880a5160 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.601+0000 7fc7a4085700 1 --2- 172.21.15.44:0/1198230249 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc79c108020 0x7fc79c1a6290 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.601+0000 7fc7a4085700 1 --2- 172.21.15.44:0/1198230249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc79c102dc0 0x7fc79c196310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.601+0000 7fc7a4085700 1 --2- 172.21.15.44:0/1198230249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc79c1056f0 0x7fc79c196850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.601+0000 7fc7a4085700 1 -- 172.21.15.44:0/1198230249 >> 172.21.15.44:0/1198230249 conn(0x7fc79c0fc790 msgr2=0x7fc79c1023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.601+0000 7fc7a4085700 1 -- 172.21.15.44:0/1198230249 shutdown_connections 2024-04-21T23:08:25.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:25.601+0000 7fc7a4085700 1 -- 172.21.15.44:0/1198230249 wait complete. 2024-04-21T23:08:26.365 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:26 smithi044 ceph-mon[27934]: pgmap v354: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2024-04-21T23:08:26.365 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:26 smithi044 ceph-mon[27934]: osdmap e94: 12 total, 12 up, 12 in 2024-04-21T23:08:26.365 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:26 smithi044 ceph-mon[27934]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:26.400 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-21T23:08:26.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:26 smithi069 ceph-mon[31816]: pgmap v354: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2024-04-21T23:08:26.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:26 smithi069 ceph-mon[31816]: osdmap e94: 12 total, 12 up, 12 in 2024-04-21T23:08:26.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:26 smithi069 ceph-mon[31816]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:26.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:26 smithi055 ceph-mon[31117]: pgmap v354: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2024-04-21T23:08:26.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:26 smithi055 ceph-mon[31117]: osdmap e94: 12 total, 12 up, 12 in 2024-04-21T23:08:26.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:26 smithi055 ceph-mon[31117]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:26.655 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:08:28.402 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:28 smithi044 ceph-mon[27934]: pgmap v356: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 2 op/s 2024-04-21T23:08:28.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.411+0000 7fccc179f700 1 -- 172.21.15.44:0/4069766549 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccbc1021b0 msgr2=0x7fccbc1025d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:28.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.411+0000 7fccc179f700 1 --2- 172.21.15.44:0/4069766549 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccbc1021b0 0x7fccbc1025d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fccac0099c0 tx=0x7fccac009cd0 comp rx=0 tx=0).stop 2024-04-21T23:08:28.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.411+0000 7fccc179f700 1 -- 172.21.15.44:0/4069766549 shutdown_connections 2024-04-21T23:08:28.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.411+0000 7fccc179f700 1 --2- 172.21.15.44:0/4069766549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccbc103d50 0x7fccbc10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:28.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.411+0000 7fccc179f700 1 --2- 172.21.15.44:0/4069766549 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccbc1021b0 0x7fccbc1025d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:28.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.411+0000 7fccc179f700 1 --2- 172.21.15.44:0/4069766549 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fccbc1033b0 0x7fccbc103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:28.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.411+0000 7fccc179f700 1 -- 172.21.15.44:0/4069766549 >> 172.21.15.44:0/4069766549 conn(0x7fccbc0fd770 msgr2=0x7fccbc0ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:28.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.412+0000 7fccc179f700 1 -- 172.21.15.44:0/4069766549 shutdown_connections 2024-04-21T23:08:28.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.412+0000 7fccc179f700 1 -- 172.21.15.44:0/4069766549 wait complete. 2024-04-21T23:08:28.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.412+0000 7fccc179f700 1 Processor -- start 2024-04-21T23:08:28.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.412+0000 7fccc179f700 1 -- start start 2024-04-21T23:08:28.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.413+0000 7fccc179f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccbc1021b0 0x7fccbc198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:28.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.413+0000 7fccc179f700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccbc1033b0 0x7fccbc1987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:28.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.413+0000 7fccc179f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fccbc103d50 0x7fccbc19f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:28.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.413+0000 7fccc179f700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc10dd90 con 0x7fccbc1033b0 2024-04-21T23:08:28.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.413+0000 7fccc179f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc10dc10 con 0x7fccbc103d50 2024-04-21T23:08:28.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.413+0000 7fccc179f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc10df10 con 0x7fccbc1021b0 2024-04-21T23:08:28.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.413+0000 7fccbb7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccbc1033b0 0x7fccbc1987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:28.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.413+0000 7fccbb7fe700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccbc1033b0 0x7fccbc1987c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50794/0 (socket says 172.21.15.44:50794) 2024-04-21T23:08:28.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.413+0000 7fccbb7fe700 1 -- 172.21.15.44:0/778995428 learned_addr learned my addr 172.21.15.44:0/778995428 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:28.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.413+0000 7fccbbfff700 1 --2- 172.21.15.44:0/778995428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccbc1021b0 0x7fccbc198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:28.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.413+0000 7fccc0f9e700 1 --2- 172.21.15.44:0/778995428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fccbc103d50 0x7fccbc19f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:28.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.413+0000 7fccbb7fe700 1 -- 172.21.15.44:0/778995428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccbc1021b0 msgr2=0x7fccbc198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:28.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.413+0000 7fccbb7fe700 1 --2- 172.21.15.44:0/778995428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccbc1021b0 0x7fccbc198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:28.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.413+0000 7fccbb7fe700 1 -- 172.21.15.44:0/778995428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fccbc103d50 msgr2=0x7fccbc19f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:28.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.413+0000 7fccbb7fe700 1 --2- 172.21.15.44:0/778995428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fccbc103d50 0x7fccbc19f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:28.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.413+0000 7fccbb7fe700 1 -- 172.21.15.44:0/778995428 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccbc19fee0 con 0x7fccbc1033b0 2024-04-21T23:08:28.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.414+0000 7fccbb7fe700 1 --2- 172.21.15.44:0/778995428 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccbc1033b0 0x7fccbc1987c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fccb000c9e0 tx=0x7fccb000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:28.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.414+0000 7fccb97fa700 1 -- 172.21.15.44:0/778995428 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb0007ac0 con 0x7fccbc1033b0 2024-04-21T23:08:28.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.414+0000 7fccb97fa700 1 -- 172.21.15.44:0/778995428 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccb0007c60 con 0x7fccbc1033b0 2024-04-21T23:08:28.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.414+0000 7fccb97fa700 1 -- 172.21.15.44:0/778995428 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb0018730 con 0x7fccbc1033b0 2024-04-21T23:08:28.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.414+0000 7fccc179f700 1 -- 172.21.15.44:0/778995428 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccbc1a01d0 con 0x7fccbc1033b0 2024-04-21T23:08:28.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.414+0000 7fccc179f700 1 -- 172.21.15.44:0/778995428 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccbc1a0760 con 0x7fccbc1033b0 2024-04-21T23:08:28.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.415+0000 7fccc179f700 1 -- 172.21.15.44:0/778995428 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccbc068f20 con 0x7fccbc1033b0 2024-04-21T23:08:28.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.416+0000 7fccb97fa700 1 -- 172.21.15.44:0/778995428 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fccb00188d0 con 0x7fccbc1033b0 2024-04-21T23:08:28.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.417+0000 7fccb97fa700 1 --2- 172.21.15.44:0/778995428 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcca40a2ca0 0x7fcca40a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:28.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.417+0000 7fccbbfff700 1 --2- 172.21.15.44:0/778995428 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcca40a2ca0 0x7fcca40a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:28.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.417+0000 7fccb97fa700 1 -- 172.21.15.44:0/778995428 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7fccb00cf370 con 0x7fccbc1033b0 2024-04-21T23:08:28.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.417+0000 7fccbbfff700 1 --2- 172.21.15.44:0/778995428 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcca40a2ca0 0x7fcca40a5160 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fccac00b580 tx=0x7fccac005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:28.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.421+0000 7fccb97fa700 1 -- 172.21.15.44:0/778995428 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fccb0014030 con 0x7fccbc1033b0 2024-04-21T23:08:28.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:28 smithi055 ceph-mon[31117]: pgmap v356: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 2 op/s 2024-04-21T23:08:28.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:28 smithi069 ceph-mon[31816]: pgmap v356: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 2 op/s 2024-04-21T23:08:28.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.634+0000 7fccc179f700 1 -- 172.21.15.44:0/778995428 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fccbc062670 con 0x7fcca40a2ca0 2024-04-21T23:08:28.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.636+0000 7fccb97fa700 1 -- 172.21.15.44:0/778995428 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+187 (secure 0 0 0) 0x7fccbc062670 con 0x7fcca40a2ca0 2024-04-21T23:08:28.636 INFO:teuthology.orchestra.run.smithi044.stdout:HOST ADDR LABELS STATUS 2024-04-21T23:08:28.636 INFO:teuthology.orchestra.run.smithi044.stdout:smithi044 172.21.15.44 2024-04-21T23:08:28.636 INFO:teuthology.orchestra.run.smithi044.stdout:smithi055 172.21.15.55 2024-04-21T23:08:28.636 INFO:teuthology.orchestra.run.smithi044.stdout:smithi069 172.21.15.69 2024-04-21T23:08:28.636 INFO:teuthology.orchestra.run.smithi044.stdout:3 hosts in cluster 2024-04-21T23:08:28.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.638+0000 7fccc179f700 1 -- 172.21.15.44:0/778995428 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcca40a2ca0 msgr2=0x7fcca40a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:28.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.638+0000 7fccc179f700 1 --2- 172.21.15.44:0/778995428 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcca40a2ca0 0x7fcca40a5160 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fccac00b580 tx=0x7fccac005fb0 comp rx=0 tx=0).stop 2024-04-21T23:08:28.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.638+0000 7fccc179f700 1 -- 172.21.15.44:0/778995428 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccbc1033b0 msgr2=0x7fccbc1987c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:28.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.638+0000 7fccc179f700 1 --2- 172.21.15.44:0/778995428 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccbc1033b0 0x7fccbc1987c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fccb000c9e0 tx=0x7fccb000ccf0 comp rx=0 tx=0).stop 2024-04-21T23:08:28.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.638+0000 7fccc179f700 1 -- 172.21.15.44:0/778995428 shutdown_connections 2024-04-21T23:08:28.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.638+0000 7fccc179f700 1 --2- 172.21.15.44:0/778995428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fccbc103d50 0x7fccbc19f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:28.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.638+0000 7fccc179f700 1 --2- 172.21.15.44:0/778995428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccbc1021b0 0x7fccbc198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:28.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.638+0000 7fccc179f700 1 --2- 172.21.15.44:0/778995428 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcca40a2ca0 0x7fcca40a5160 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:28.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.638+0000 7fccc179f700 1 --2- 172.21.15.44:0/778995428 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccbc1033b0 0x7fccbc1987c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:28.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.638+0000 7fccc179f700 1 -- 172.21.15.44:0/778995428 >> 172.21.15.44:0/778995428 conn(0x7fccbc0fd770 msgr2=0x7fccbc106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:28.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.638+0000 7fccc179f700 1 -- 172.21.15.44:0/778995428 shutdown_connections 2024-04-21T23:08:28.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:28.638+0000 7fccc179f700 1 -- 172.21.15.44:0/778995428 wait complete. 2024-04-21T23:08:29.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:29 smithi044 ceph-mon[27934]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:29.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:29 smithi069 ceph-mon[31816]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:29.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:29 smithi055 ceph-mon[31117]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:29.505 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-21T23:08:29.763 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:08:30.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:30 smithi044 ceph-mon[27934]: pgmap v357: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 383 B/s wr, 1 op/s 2024-04-21T23:08:30.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:30 smithi055 ceph-mon[31117]: pgmap v357: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 383 B/s wr, 1 op/s 2024-04-21T23:08:30.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:30 smithi069 ceph-mon[31816]: pgmap v357: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 383 B/s wr, 1 op/s 2024-04-21T23:08:31.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.494+0000 7fdec3336700 1 -- 172.21.15.44:0/359194210 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdebc103e40 msgr2=0x7fdebc10a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:31.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.494+0000 7fdec3336700 1 --2- 172.21.15.44:0/359194210 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdebc103e40 0x7fdebc10a6f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400b0d0 tx=0x7fdeb400b3e0 comp rx=0 tx=0).stop 2024-04-21T23:08:31.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.494+0000 7fdec3336700 1 -- 172.21.15.44:0/359194210 shutdown_connections 2024-04-21T23:08:31.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.494+0000 7fdec3336700 1 --2- 172.21.15.44:0/359194210 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdebc103e40 0x7fdebc10a6f0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:31.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.494+0000 7fdec3336700 1 --2- 172.21.15.44:0/359194210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdebc1022a0 0x7fdebc1026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:31.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.494+0000 7fdec3336700 1 --2- 172.21.15.44:0/359194210 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdebc1034a0 0x7fdebc103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:31.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.494+0000 7fdec3336700 1 -- 172.21.15.44:0/359194210 >> 172.21.15.44:0/359194210 conn(0x7fdebc0fd840 msgr2=0x7fdebc0ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:31.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.494+0000 7fdec3336700 1 -- 172.21.15.44:0/359194210 shutdown_connections 2024-04-21T23:08:31.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.494+0000 7fdec3336700 1 -- 172.21.15.44:0/359194210 wait complete. 2024-04-21T23:08:31.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.495+0000 7fdec3336700 1 Processor -- start 2024-04-21T23:08:31.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.495+0000 7fdec3336700 1 -- start start 2024-04-21T23:08:31.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.495+0000 7fdec3336700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdebc1022a0 0x7fdebc198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:31.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.495+0000 7fdec3336700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdebc1034a0 0x7fdebc198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:31.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.495+0000 7fdec3336700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdebc103e40 0x7fdebc19f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:31.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.495+0000 7fdec3336700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc10de60 con 0x7fdebc1022a0 2024-04-21T23:08:31.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.495+0000 7fdec3336700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc10dce0 con 0x7fdebc103e40 2024-04-21T23:08:31.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.495+0000 7fdec3336700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc10dfe0 con 0x7fdebc1034a0 2024-04-21T23:08:31.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.495+0000 7fdec10d2700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdebc1022a0 0x7fdebc198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:31.497 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.495+0000 7fdec10d2700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdebc1022a0 0x7fdebc198320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51358/0 (socket says 172.21.15.44:51358) 2024-04-21T23:08:31.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.495+0000 7fdec10d2700 1 -- 172.21.15.44:0/427705559 learned_addr learned my addr 172.21.15.44:0/427705559 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:31.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.495+0000 7fdec08d1700 1 --2- 172.21.15.44:0/427705559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdebc1034a0 0x7fdebc198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:31.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.495+0000 7fdec18d3700 1 --2- 172.21.15.44:0/427705559 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdebc103e40 0x7fdebc19f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:31.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.495+0000 7fdec10d2700 1 -- 172.21.15.44:0/427705559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdebc1034a0 msgr2=0x7fdebc198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:31.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.495+0000 7fdec10d2700 1 --2- 172.21.15.44:0/427705559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdebc1034a0 0x7fdebc198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:31.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.495+0000 7fdec10d2700 1 -- 172.21.15.44:0/427705559 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdebc103e40 msgr2=0x7fdebc19f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:31.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.495+0000 7fdec10d2700 1 --2- 172.21.15.44:0/427705559 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdebc103e40 0x7fdebc19f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:31.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.496+0000 7fdec10d2700 1 -- 172.21.15.44:0/427705559 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdebc19ff80 con 0x7fdebc1022a0 2024-04-21T23:08:31.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.496+0000 7fdec18d3700 1 --2- 172.21.15.44:0/427705559 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdebc103e40 0x7fdebc19f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:08:31.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.496+0000 7fdec10d2700 1 --2- 172.21.15.44:0/427705559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdebc1022a0 0x7fdebc198320 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fdeb800d9b0 tx=0x7fdeb800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:31.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.496+0000 7fdeb27fc700 1 -- 172.21.15.44:0/427705559 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8017030 con 0x7fdebc1022a0 2024-04-21T23:08:31.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.496+0000 7fdec3336700 1 -- 172.21.15.44:0/427705559 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdebc1a0270 con 0x7fdebc1022a0 2024-04-21T23:08:31.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.496+0000 7fdeb27fc700 1 -- 172.21.15.44:0/427705559 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdeb8004500 con 0x7fdebc1022a0 2024-04-21T23:08:31.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.496+0000 7fdeb27fc700 1 -- 172.21.15.44:0/427705559 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8003aa0 con 0x7fdebc1022a0 2024-04-21T23:08:31.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.496+0000 7fdec3336700 1 -- 172.21.15.44:0/427705559 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdebc1a0800 con 0x7fdebc1022a0 2024-04-21T23:08:31.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.497+0000 7fdec3336700 1 -- 172.21.15.44:0/427705559 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdebc068f20 con 0x7fdebc1022a0 2024-04-21T23:08:31.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.498+0000 7fdeb27fc700 1 -- 172.21.15.44:0/427705559 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fdeb801a070 con 0x7fdebc1022a0 2024-04-21T23:08:31.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.499+0000 7fdeb27fc700 1 --2- 172.21.15.44:0/427705559 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdea80a2c20 0x7fdea80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:31.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.499+0000 7fdec08d1700 1 --2- 172.21.15.44:0/427705559 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdea80a2c20 0x7fdea80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:31.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.499+0000 7fdeb27fc700 1 -- 172.21.15.44:0/427705559 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7fdeb80cedc0 con 0x7fdebc1022a0 2024-04-21T23:08:31.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.499+0000 7fdec08d1700 1 --2- 172.21.15.44:0/427705559 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdea80a2c20 0x7fdea80a50e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00ac50 tx=0x7fdeac00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:31.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.503+0000 7fdeb27fc700 1 -- 172.21.15.44:0/427705559 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fdeb80232f0 con 0x7fdebc1022a0 2024-04-21T23:08:31.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.720+0000 7fdec3336700 1 -- 172.21.15.44:0/427705559 --> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fdebc062670 con 0x7fdea80a2c20 2024-04-21T23:08:31.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.721+0000 7fdeb27fc700 1 -- 172.21.15.44:0/427705559 <== mgr.14152 v2:172.21.15.44:6800/1417504446 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fdebc062670 con 0x7fdea80a2c20 2024-04-21T23:08:31.721 INFO:teuthology.orchestra.run.smithi044.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-21T23:08:31.721 INFO:teuthology.orchestra.run.smithi044.stdout:smithi044 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331001K400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-21T23:08:31.721 INFO:teuthology.orchestra.run.smithi044.stdout:smithi055 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736102VS4P0KGN 3726G No 3m ago Has a FileSystem, LVM detected 2024-04-21T23:08:31.721 INFO:teuthology.orchestra.run.smithi044.stdout:smithi069 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400Y0400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-21T23:08:31.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.723+0000 7fdec3336700 1 -- 172.21.15.44:0/427705559 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdea80a2c20 msgr2=0x7fdea80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:31.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.723+0000 7fdec3336700 1 --2- 172.21.15.44:0/427705559 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdea80a2c20 0x7fdea80a50e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00ac50 tx=0x7fdeac00a380 comp rx=0 tx=0).stop 2024-04-21T23:08:31.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.723+0000 7fdec3336700 1 -- 172.21.15.44:0/427705559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdebc1022a0 msgr2=0x7fdebc198320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:31.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.724+0000 7fdec3336700 1 --2- 172.21.15.44:0/427705559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdebc1022a0 0x7fdebc198320 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fdeb800d9b0 tx=0x7fdeb800dcc0 comp rx=0 tx=0).stop 2024-04-21T23:08:31.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.724+0000 7fdec3336700 1 -- 172.21.15.44:0/427705559 shutdown_connections 2024-04-21T23:08:31.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.724+0000 7fdec3336700 1 --2- 172.21.15.44:0/427705559 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdebc103e40 0x7fdebc19f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:31.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.724+0000 7fdec3336700 1 --2- 172.21.15.44:0/427705559 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fdea80a2c20 0x7fdea80a50e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:31.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.724+0000 7fdec3336700 1 --2- 172.21.15.44:0/427705559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdebc1022a0 0x7fdebc198320 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:31.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.724+0000 7fdec3336700 1 --2- 172.21.15.44:0/427705559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdebc1034a0 0x7fdebc198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:31.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.724+0000 7fdec3336700 1 -- 172.21.15.44:0/427705559 >> 172.21.15.44:0/427705559 conn(0x7fdebc0fd840 msgr2=0x7fdebc1066d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:31.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.724+0000 7fdec3336700 1 -- 172.21.15.44:0/427705559 shutdown_connections 2024-04-21T23:08:31.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:31.724+0000 7fdec3336700 1 -- 172.21.15.44:0/427705559 wait complete. 2024-04-21T23:08:32.380 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:32 smithi044 ceph-mon[27934]: pgmap v358: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-21T23:08:32.380 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:32 smithi044 ceph-mon[27934]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:32.382 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-21T23:08:32.391 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-04-21T23:08:32.392 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-21T23:08:32.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:32 smithi055 ceph-mon[31117]: pgmap v358: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-21T23:08:32.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:32 smithi055 ceph-mon[31117]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:32.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:32 smithi069 ceph-mon[31816]: pgmap v358: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-21T23:08:32.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:32 smithi069 ceph-mon[31816]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:32.647 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:08:34.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.312+0000 7fc59e018700 1 -- 172.21.15.44:0/3508092279 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5981005a0 msgr2=0x7fc5981009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:34.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.312+0000 7fc59e018700 1 --2- 172.21.15.44:0/3508092279 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5981005a0 0x7fc5981009c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc588009990 tx=0x7fc588009ca0 comp rx=0 tx=0).stop 2024-04-21T23:08:34.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.312+0000 7fc59e018700 1 -- 172.21.15.44:0/3508092279 shutdown_connections 2024-04-21T23:08:34.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.312+0000 7fc59e018700 1 --2- 172.21.15.44:0/3508092279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc598103f20 0x7fc59810a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:34.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.312+0000 7fc59e018700 1 --2- 172.21.15.44:0/3508092279 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc5981005a0 0x7fc5981009c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:34.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.312+0000 7fc59e018700 1 --2- 172.21.15.44:0/3508092279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc598103580 0x7fc5981039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:34.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.312+0000 7fc59e018700 1 -- 172.21.15.44:0/3508092279 >> 172.21.15.44:0/3508092279 conn(0x7fc5980fbf00 msgr2=0x7fc5980fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.312+0000 7fc59e018700 1 -- 172.21.15.44:0/3508092279 shutdown_connections 2024-04-21T23:08:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.312+0000 7fc59e018700 1 -- 172.21.15.44:0/3508092279 wait complete. 2024-04-21T23:08:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.312+0000 7fc59e018700 1 Processor -- start 2024-04-21T23:08:34.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.313+0000 7fc59e018700 1 -- start start 2024-04-21T23:08:34.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.313+0000 7fc59e018700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc5981005a0 0x7fc5981984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:34.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.313+0000 7fc59e018700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc598103580 0x7fc598198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.313+0000 7fc59e018700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc598103f20 0x7fc59819faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.313+0000 7fc59e018700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc59810def0 con 0x7fc598103580 2024-04-21T23:08:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.313+0000 7fc59e018700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc59810dd70 con 0x7fc5981005a0 2024-04-21T23:08:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.313+0000 7fc59e018700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc59810e070 con 0x7fc598103f20 2024-04-21T23:08:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.313+0000 7fc596ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc598103580 0x7fc598198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.313+0000 7fc596ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc598103580 0x7fc598198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51366/0 (socket says 172.21.15.44:51366) 2024-04-21T23:08:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.313+0000 7fc5977fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc5981005a0 0x7fc5981984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.313+0000 7fc596ffd700 1 -- 172.21.15.44:0/3739374854 learned_addr learned my addr 172.21.15.44:0/3739374854 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.313+0000 7fc597fff700 1 --2- 172.21.15.44:0/3739374854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc598103f20 0x7fc59819faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.313+0000 7fc596ffd700 1 -- 172.21.15.44:0/3739374854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc598103f20 msgr2=0x7fc59819faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.313+0000 7fc596ffd700 1 --2- 172.21.15.44:0/3739374854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc598103f20 0x7fc59819faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.313+0000 7fc596ffd700 1 -- 172.21.15.44:0/3739374854 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc5981005a0 msgr2=0x7fc5981984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.313+0000 7fc596ffd700 1 --2- 172.21.15.44:0/3739374854 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc5981005a0 0x7fc5981984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:34.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.313+0000 7fc596ffd700 1 -- 172.21.15.44:0/3739374854 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5981a0140 con 0x7fc598103580 2024-04-21T23:08:34.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.314+0000 7fc596ffd700 1 --2- 172.21.15.44:0/3739374854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc598103580 0x7fc598198a20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc58c00c9e0 tx=0x7fc58c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:34.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.314+0000 7fc594ff9700 1 -- 172.21.15.44:0/3739374854 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58c007ac0 con 0x7fc598103580 2024-04-21T23:08:34.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.314+0000 7fc594ff9700 1 -- 172.21.15.44:0/3739374854 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc58c007c60 con 0x7fc598103580 2024-04-21T23:08:34.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.314+0000 7fc59e018700 1 -- 172.21.15.44:0/3739374854 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5981013d0 con 0x7fc598103580 2024-04-21T23:08:34.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.314+0000 7fc594ff9700 1 -- 172.21.15.44:0/3739374854 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58c018660 con 0x7fc598103580 2024-04-21T23:08:34.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.314+0000 7fc59e018700 1 -- 172.21.15.44:0/3739374854 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc598101910 con 0x7fc598103580 2024-04-21T23:08:34.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.315+0000 7fc59e018700 1 -- 172.21.15.44:0/3739374854 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc598068f20 con 0x7fc598103580 2024-04-21T23:08:34.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.316+0000 7fc594ff9700 1 -- 172.21.15.44:0/3739374854 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc58c00f450 con 0x7fc598103580 2024-04-21T23:08:34.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.316+0000 7fc594ff9700 1 --2- 172.21.15.44:0/3739374854 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc5800a2c20 0x7fc5800a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:34.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.317+0000 7fc5977fe700 1 --2- 172.21.15.44:0/3739374854 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc5800a2c20 0x7fc5800a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:34.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.317+0000 7fc594ff9700 1 -- 172.21.15.44:0/3739374854 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7fc58c0cf320 con 0x7fc598103580 2024-04-21T23:08:34.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.317+0000 7fc5977fe700 1 --2- 172.21.15.44:0/3739374854 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc5800a2c20 0x7fc5800a50e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc5880039c0 tx=0x7fc588005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:34.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.321+0000 7fc594ff9700 1 -- 172.21.15.44:0/3739374854 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc58c014030 con 0x7fc598103580 2024-04-21T23:08:34.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:34 smithi044 ceph-mon[27934]: pgmap v359: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-21T23:08:34.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:34 smithi055 ceph-mon[31117]: pgmap v359: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-21T23:08:34.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:34 smithi069 ceph-mon[31816]: pgmap v359: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 341 B/s wr, 2 op/s 2024-04-21T23:08:34.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.576+0000 7fc59e018700 1 -- 172.21.15.44:0/3739374854 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fc598102d50 con 0x7fc598103580 2024-04-21T23:08:34.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.576+0000 7fc594ff9700 1 -- 172.21.15.44:0/3739374854 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) ==== 76+0+2998 (secure 0 0 0) 0x7fc58c09bec0 con 0x7fc598103580 2024-04-21T23:08:34.576 INFO:teuthology.orchestra.run.smithi044.stdout:e21 2024-04-21T23:08:34.576 INFO:teuthology.orchestra.run.smithi044.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T23:08:34.576 INFO:teuthology.orchestra.run.smithi044.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T23:08:34.576 INFO:teuthology.orchestra.run.smithi044.stdout:legacy client fscid: 1 2024-04-21T23:08:34.576 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:34.576 INFO:teuthology.orchestra.run.smithi044.stdout:Filesystem 'cephfs' (1) 2024-04-21T23:08:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:fs_name cephfs 2024-04-21T23:08:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 21 2024-04-21T23:08:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-21T23:08:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-21T23:06:25.562805+0000 2024-04-21T23:08:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-21T23:06:50.685663+0000 2024-04-21T23:08:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:tableserver 0 2024-04-21T23:08:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:root 0 2024-04-21T23:08:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:session_timeout 300 2024-04-21T23:08:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:session_autoclose 300 2024-04-21T23:08:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:max_file_size 1099511627776 2024-04-21T23:08:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:max_xattr_size 65536 2024-04-21T23:08:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:required_client_features {} 2024-04-21T23:08:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure 0 2024-04-21T23:08:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:last_failure_osd_epoch 0 2024-04-21T23:08:34.577 INFO:teuthology.orchestra.run.smithi044.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T23:08:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:max_mds 1 2024-04-21T23:08:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:in 0 2024-04-21T23:08:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:up {0=24530} 2024-04-21T23:08:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:failed 2024-04-21T23:08:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:damaged 2024-04-21T23:08:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:stopped 2024-04-21T23:08:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:data_pools [3,4] 2024-04-21T23:08:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:metadata_pool 2 2024-04-21T23:08:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:inline_data disabled 2024-04-21T23:08:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:balancer 2024-04-21T23:08:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:bal_rank_mask -1 2024-04-21T23:08:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:standby_count_wanted 0 2024-04-21T23:08:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:qdb_cluster leader: 24530 members: 24530 2024-04-21T23:08:34.578 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.b{0:24530} state up:active seq 3 addr [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:08:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.i{0:24526} state up:standby-replay seq 1 addr [v2:172.21.15.69:6838/3743898853,v1:172.21.15.69:6839/3743898853] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:08:34.579 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:34.579 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:Standby daemons: 2024-04-21T23:08:34.579 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.j{-1:14565} state up:standby seq 1 addr [v2:172.21.15.44:6834/634418901,v1:172.21.15.44:6835/634418901] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:08:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.d{-1:14577} state up:standby seq 1 addr [v2:172.21.15.44:6836/3848365642,v1:172.21.15.44:6837/3848365642] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:08:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.g{-1:14640} state up:standby seq 1 addr [v2:172.21.15.44:6838/1544956047,v1:172.21.15.44:6839/1544956047] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:08:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.a{-1:14652} state up:standby seq 1 addr [v2:172.21.15.44:6840/1863599778,v1:172.21.15.44:6841/1863599778] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:08:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.k{-1:24440} state up:standby seq 1 addr [v2:172.21.15.55:6832/1288947368,v1:172.21.15.55:6833/1288947368] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:08:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.l{-1:24448} state up:standby seq 1 addr [v2:172.21.15.69:6832/43117883,v1:172.21.15.69:6833/43117883] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:08:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.e{-1:24464} state up:standby seq 1 addr [v2:172.21.15.55:6834/1607421809,v1:172.21.15.55:6835/1607421809] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:08:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.f{-1:24487} state up:standby seq 1 addr [v2:172.21.15.69:6834/2637621987,v1:172.21.15.69:6835/2637621987] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:08:34.579 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.h{-1:24494} state up:standby seq 1 addr [v2:172.21.15.55:6836/1372428654,v1:172.21.15.55:6837/1372428654] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:08:34.580 INFO:teuthology.orchestra.run.smithi044.stdout:[mds.c{-1:24499} state up:standby seq 1 addr [v2:172.21.15.69:6836/3371833935,v1:172.21.15.69:6837/3371833935] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T23:08:34.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.578+0000 7fc59e018700 1 -- 172.21.15.44:0/3739374854 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc5800a2c20 msgr2=0x7fc5800a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:34.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.578+0000 7fc59e018700 1 --2- 172.21.15.44:0/3739374854 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc5800a2c20 0x7fc5800a50e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc5880039c0 tx=0x7fc588005fd0 comp rx=0 tx=0).stop 2024-04-21T23:08:34.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.578+0000 7fc59e018700 1 -- 172.21.15.44:0/3739374854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc598103580 msgr2=0x7fc598198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:34.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.578+0000 7fc59e018700 1 --2- 172.21.15.44:0/3739374854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc598103580 0x7fc598198a20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc58c00c9e0 tx=0x7fc58c00ccf0 comp rx=0 tx=0).stop 2024-04-21T23:08:34.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.578+0000 7fc59e018700 1 -- 172.21.15.44:0/3739374854 shutdown_connections 2024-04-21T23:08:34.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.578+0000 7fc59e018700 1 --2- 172.21.15.44:0/3739374854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc598103f20 0x7fc59819faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:34.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.578+0000 7fc59e018700 1 --2- 172.21.15.44:0/3739374854 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc5981005a0 0x7fc5981984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:34.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.578+0000 7fc59e018700 1 --2- 172.21.15.44:0/3739374854 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fc5800a2c20 0x7fc5800a50e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:34.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.578+0000 7fc59e018700 1 --2- 172.21.15.44:0/3739374854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc598103580 0x7fc598198a20 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:34.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.579+0000 7fc59e018700 1 -- 172.21.15.44:0/3739374854 >> 172.21.15.44:0/3739374854 conn(0x7fc5980fbf00 msgr2=0x7fc598106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:34.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.579+0000 7fc59e018700 1 -- 172.21.15.44:0/3739374854 shutdown_connections 2024-04-21T23:08:34.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:34.579+0000 7fc59e018700 1 -- 172.21.15.44:0/3739374854 wait complete. 2024-04-21T23:08:34.582 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2024-04-21T23:08:35.284 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:35 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3739374854' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-21T23:08:35.316 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-21T23:08:35.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:35 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3739374854' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-21T23:08:35.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:35 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3739374854' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-21T23:08:35.567 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:08:36.417 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:36 smithi044 ceph-mon[27934]: pgmap v360: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 0 B/s wr, 2 op/s 2024-04-21T23:08:36.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:36 smithi055 ceph-mon[31117]: pgmap v360: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 0 B/s wr, 2 op/s 2024-04-21T23:08:36.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:36 smithi069 ceph-mon[31816]: pgmap v360: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 0 B/s wr, 2 op/s 2024-04-21T23:08:37.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.291+0000 7f22699fa700 1 -- 172.21.15.44:0/3452179231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2264103e60 msgr2=0x7f226410a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:37.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.291+0000 7f22699fa700 1 --2- 172.21.15.44:0/3452179231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2264103e60 0x7f226410a710 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f225c00b0d0 tx=0x7f225c00b3e0 comp rx=0 tx=0).stop 2024-04-21T23:08:37.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.291+0000 7f22699fa700 1 -- 172.21.15.44:0/3452179231 shutdown_connections 2024-04-21T23:08:37.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.291+0000 7f22699fa700 1 --2- 172.21.15.44:0/3452179231 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2264103e60 0x7f226410a710 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:37.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.291+0000 7f22699fa700 1 --2- 172.21.15.44:0/3452179231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22641022c0 0x7f22641026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:37.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.291+0000 7f22699fa700 1 --2- 172.21.15.44:0/3452179231 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22641034c0 0x7f2264103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:37.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.291+0000 7f22699fa700 1 -- 172.21.15.44:0/3452179231 >> 172.21.15.44:0/3452179231 conn(0x7f22640fd880 msgr2=0x7f22640ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:37.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.291+0000 7f22699fa700 1 -- 172.21.15.44:0/3452179231 shutdown_connections 2024-04-21T23:08:37.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.291+0000 7f22699fa700 1 -- 172.21.15.44:0/3452179231 wait complete. 2024-04-21T23:08:37.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.292+0000 7f22699fa700 1 Processor -- start 2024-04-21T23:08:37.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.292+0000 7f22699fa700 1 -- start start 2024-04-21T23:08:37.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.292+0000 7f22699fa700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22641022c0 0x7f22641983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:37.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.292+0000 7f22699fa700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22641034c0 0x7f2264198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:37.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.292+0000 7f22699fa700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2264103e60 0x7f226419f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:37.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.292+0000 7f22699fa700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f226410deb0 con 0x7f22641022c0 2024-04-21T23:08:37.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.292+0000 7f22699fa700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f226410dd30 con 0x7f22641034c0 2024-04-21T23:08:37.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.292+0000 7f22699fa700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f226410e030 con 0x7f2264103e60 2024-04-21T23:08:37.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.292+0000 7f2262ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22641022c0 0x7f22641983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:37.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.292+0000 7f2262ffd700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22641022c0 0x7f22641983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51384/0 (socket says 172.21.15.44:51384) 2024-04-21T23:08:37.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.292+0000 7f2262ffd700 1 -- 172.21.15.44:0/1676491547 learned_addr learned my addr 172.21.15.44:0/1676491547 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:37.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.292+0000 7f22627fc700 1 --2- 172.21.15.44:0/1676491547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22641034c0 0x7f2264198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:37.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.292+0000 7f22637fe700 1 --2- 172.21.15.44:0/1676491547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2264103e60 0x7f226419f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:37.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.292+0000 7f2262ffd700 1 -- 172.21.15.44:0/1676491547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2264103e60 msgr2=0x7f226419f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:37.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.292+0000 7f2262ffd700 1 --2- 172.21.15.44:0/1676491547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2264103e60 0x7f226419f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:37.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.292+0000 7f2262ffd700 1 -- 172.21.15.44:0/1676491547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22641034c0 msgr2=0x7f2264198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:37.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.292+0000 7f2262ffd700 1 --2- 172.21.15.44:0/1676491547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22641034c0 0x7f2264198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:37.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.292+0000 7f2262ffd700 1 -- 172.21.15.44:0/1676491547 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22641a0030 con 0x7f22641022c0 2024-04-21T23:08:37.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.293+0000 7f2262ffd700 1 --2- 172.21.15.44:0/1676491547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22641022c0 0x7f22641983d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f225400d9b0 tx=0x7f225400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:37.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.293+0000 7f22689f8700 1 -- 172.21.15.44:0/1676491547 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2254017030 con 0x7f22641022c0 2024-04-21T23:08:37.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.293+0000 7f22699fa700 1 -- 172.21.15.44:0/1676491547 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22641a0320 con 0x7f22641022c0 2024-04-21T23:08:37.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.293+0000 7f22689f8700 1 -- 172.21.15.44:0/1676491547 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2254004500 con 0x7f22641022c0 2024-04-21T23:08:37.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.293+0000 7f22689f8700 1 -- 172.21.15.44:0/1676491547 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2254003970 con 0x7f22641022c0 2024-04-21T23:08:37.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.293+0000 7f22699fa700 1 -- 172.21.15.44:0/1676491547 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22641a08b0 con 0x7f22641022c0 2024-04-21T23:08:37.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.294+0000 7f22699fa700 1 -- 172.21.15.44:0/1676491547 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2264068f20 con 0x7f22641022c0 2024-04-21T23:08:37.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.295+0000 7f22689f8700 1 -- 172.21.15.44:0/1676491547 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f225401a070 con 0x7f22641022c0 2024-04-21T23:08:37.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.296+0000 7f22689f8700 1 --2- 172.21.15.44:0/1676491547 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f22500a2c20 0x7f22500a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:37.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.296+0000 7f22627fc700 1 --2- 172.21.15.44:0/1676491547 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f22500a2c20 0x7f22500a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:37.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.296+0000 7f22689f8700 1 -- 172.21.15.44:0/1676491547 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7f22540cecd0 con 0x7f22641022c0 2024-04-21T23:08:37.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.296+0000 7f22627fc700 1 --2- 172.21.15.44:0/1676491547 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f22500a2c20 0x7f22500a50e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f224c004630 tx=0x7f224c00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:37.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.300+0000 7f22689f8700 1 -- 172.21.15.44:0/1676491547 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f225409b810 con 0x7f22641022c0 2024-04-21T23:08:37.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.507+0000 7f22699fa700 1 -- 172.21.15.44:0/1676491547 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f2264062e40 con 0x7f22641022c0 2024-04-21T23:08:37.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.508+0000 7f22689f8700 1 -- 172.21.15.44:0/1676491547 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v94) ==== 56+0+5650 (secure 0 0 0) 0x7f225409b110 con 0x7f22641022c0 2024-04-21T23:08:37.507 INFO:teuthology.orchestra.run.smithi044.stdout:epoch 94 2024-04-21T23:08:37.507 INFO:teuthology.orchestra.run.smithi044.stdout:fsid 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T23:08:37.507 INFO:teuthology.orchestra.run.smithi044.stdout:created 2024-04-21T22:58:23.196166+0000 2024-04-21T23:08:37.507 INFO:teuthology.orchestra.run.smithi044.stdout:modified 2024-04-21T23:08:25.093542+0000 2024-04-21T23:08:37.507 INFO:teuthology.orchestra.run.smithi044.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-21T23:08:37.507 INFO:teuthology.orchestra.run.smithi044.stdout:crush_version 27 2024-04-21T23:08:37.508 INFO:teuthology.orchestra.run.smithi044.stdout:full_ratio 0.95 2024-04-21T23:08:37.508 INFO:teuthology.orchestra.run.smithi044.stdout:backfillfull_ratio 0.9 2024-04-21T23:08:37.508 INFO:teuthology.orchestra.run.smithi044.stdout:nearfull_ratio 0.85 2024-04-21T23:08:37.508 INFO:teuthology.orchestra.run.smithi044.stdout:require_min_compat_client luminous 2024-04-21T23:08:37.508 INFO:teuthology.orchestra.run.smithi044.stdout:min_compat_client luminous 2024-04-21T23:08:37.508 INFO:teuthology.orchestra.run.smithi044.stdout:require_osd_release squid 2024-04-21T23:08:37.508 INFO:teuthology.orchestra.run.smithi044.stdout:stretch_mode_enabled false 2024-04-21T23:08:37.508 INFO:teuthology.orchestra.run.smithi044.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-21T23:08:37.508 INFO:teuthology.orchestra.run.smithi044.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-21T23:08:37.508 INFO:teuthology.orchestra.run.smithi044.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-21T23:08:37.508 INFO:teuthology.orchestra.run.smithi044.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 90 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-21T23:08:37.508 INFO:teuthology.orchestra.run.smithi044.stdout:max_osd 12 2024-04-21T23:08:37.508 INFO:teuthology.orchestra.run.smithi044.stdout:osd.0 up in weight 1 up_from 8 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.44:6802/4094542096,v1:172.21.15.44:6803/4094542096] [v2:172.21.15.44:6804/4094542096,v1:172.21.15.44:6805/4094542096] exists,up aaaab74d-dfe1-4669-9839-36b2664ac353 2024-04-21T23:08:37.509 INFO:teuthology.orchestra.run.smithi044.stdout:osd.1 up in weight 1 up_from 13 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.55:6800/1988017680,v1:172.21.15.55:6801/1988017680] [v2:172.21.15.55:6802/1988017680,v1:172.21.15.55:6803/1988017680] exists,up 7950725c-5dea-4a3b-a06d-4e50b621110b 2024-04-21T23:08:37.509 INFO:teuthology.orchestra.run.smithi044.stdout:osd.2 up in weight 1 up_from 18 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6800/3485429011,v1:172.21.15.69:6801/3485429011] [v2:172.21.15.69:6802/3485429011,v1:172.21.15.69:6803/3485429011] exists,up e99888f4-9a3e-4856-89a8-1a35817893b6 2024-04-21T23:08:37.509 INFO:teuthology.orchestra.run.smithi044.stdout:osd.3 up in weight 1 up_from 25 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.44:6810/746259227,v1:172.21.15.44:6811/746259227] [v2:172.21.15.44:6812/746259227,v1:172.21.15.44:6813/746259227] exists,up 508c528e-8e90-408b-97e3-5eded769bf1b 2024-04-21T23:08:37.509 INFO:teuthology.orchestra.run.smithi044.stdout:osd.4 up in weight 1 up_from 33 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.55:6808/988958724,v1:172.21.15.55:6809/988958724] [v2:172.21.15.55:6810/988958724,v1:172.21.15.55:6811/988958724] exists,up 44a9925a-84ae-474a-8d5c-bec46baaa91c 2024-04-21T23:08:37.509 INFO:teuthology.orchestra.run.smithi044.stdout:osd.5 up in weight 1 up_from 38 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6808/2597202863,v1:172.21.15.69:6809/2597202863] [v2:172.21.15.69:6810/2597202863,v1:172.21.15.69:6811/2597202863] exists,up f219c5c9-af53-4cf0-a77a-56595b2db93b 2024-04-21T23:08:37.509 INFO:teuthology.orchestra.run.smithi044.stdout:osd.6 up in weight 1 up_from 46 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.44:6818/1221340748,v1:172.21.15.44:6819/1221340748] [v2:172.21.15.44:6820/1221340748,v1:172.21.15.44:6821/1221340748] exists,up 6c9b8cd5-e20b-49eb-af02-87709b05e2e5 2024-04-21T23:08:37.509 INFO:teuthology.orchestra.run.smithi044.stdout:osd.7 up in weight 1 up_from 51 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.55:6816/3062702978,v1:172.21.15.55:6817/3062702978] [v2:172.21.15.55:6818/3062702978,v1:172.21.15.55:6819/3062702978] exists,up 12c98dc9-81df-4a59-9e6f-e480f5162390 2024-04-21T23:08:37.509 INFO:teuthology.orchestra.run.smithi044.stdout:osd.8 up in weight 1 up_from 59 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6816/635270546,v1:172.21.15.69:6817/635270546] [v2:172.21.15.69:6818/635270546,v1:172.21.15.69:6819/635270546] exists,up d1633bc7-35b7-491c-be5c-4375af06fd67 2024-04-21T23:08:37.509 INFO:teuthology.orchestra.run.smithi044.stdout:osd.9 up in weight 1 up_from 67 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.44:6826/1043222908,v1:172.21.15.44:6827/1043222908] [v2:172.21.15.44:6828/1043222908,v1:172.21.15.44:6829/1043222908] exists,up fc0ed998-2d10-4bc6-9a3a-d6afe9135386 2024-04-21T23:08:37.509 INFO:teuthology.orchestra.run.smithi044.stdout:osd.10 up in weight 1 up_from 73 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.55:6824/3066586400,v1:172.21.15.55:6825/3066586400] [v2:172.21.15.55:6826/3066586400,v1:172.21.15.55:6827/3066586400] exists,up c8dddb89-8c25-4df2-8993-3f61e52c53c0 2024-04-21T23:08:37.509 INFO:teuthology.orchestra.run.smithi044.stdout:osd.11 up in weight 1 up_from 81 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6824/1593504610,v1:172.21.15.69:6825/1593504610] [v2:172.21.15.69:6826/1593504610,v1:172.21.15.69:6827/1593504610] exists,up 1623764d-b8ca-4222-a1e4-2714f17d64e6 2024-04-21T23:08:37.509 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 2.b [8,11] 2024-04-21T23:08:37.509 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 2.f [8,11] 2024-04-21T23:08:37.509 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 2.1b [2,1] 2024-04-21T23:08:37.509 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 2.1d [2,4] 2024-04-21T23:08:37.509 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.1 [4,9] 2024-04-21T23:08:37.510 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.2 [3,9] 2024-04-21T23:08:37.510 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.17 [4,8] 2024-04-21T23:08:37.510 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.24 [4,9] 2024-04-21T23:08:37.510 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.2a [3,8] 2024-04-21T23:08:37.510 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 3.39 [5,6] 2024-04-21T23:08:37.510 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 4.2 [9,1] 2024-04-21T23:08:37.510 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 4.e [9,11,5,8] 2024-04-21T23:08:37.510 INFO:teuthology.orchestra.run.smithi044.stdout:pg_upmap_items 4.1c [9,8] 2024-04-21T23:08:37.510 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/2729363939 expires 2024-04-22T22:59:22.969731+0000 2024-04-21T23:08:37.510 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/3462209685 expires 2024-04-22T22:59:22.969731+0000 2024-04-21T23:08:37.510 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:6801/4238737803 expires 2024-04-22T22:59:22.969731+0000 2024-04-21T23:08:37.510 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:6800/4238737803 expires 2024-04-22T22:59:22.969731+0000 2024-04-21T23:08:37.510 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:6800/2900331072 expires 2024-04-22T22:58:56.511296+0000 2024-04-21T23:08:37.510 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/908290973 expires 2024-04-22T22:58:56.511296+0000 2024-04-21T23:08:37.510 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/2994566338 expires 2024-04-22T22:58:56.511296+0000 2024-04-21T23:08:37.511 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:6801/2900331072 expires 2024-04-22T22:58:56.511296+0000 2024-04-21T23:08:37.511 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/693229699 expires 2024-04-22T22:58:56.511296+0000 2024-04-21T23:08:37.511 INFO:teuthology.orchestra.run.smithi044.stdout:blocklist 172.21.15.44:0/1931984406 expires 2024-04-22T22:59:22.969731+0000 2024-04-21T23:08:37.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.510+0000 7f22699fa700 1 -- 172.21.15.44:0/1676491547 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f22500a2c20 msgr2=0x7f22500a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:37.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.510+0000 7f22699fa700 1 --2- 172.21.15.44:0/1676491547 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f22500a2c20 0x7f22500a50e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f224c004630 tx=0x7f224c00b3c0 comp rx=0 tx=0).stop 2024-04-21T23:08:37.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.510+0000 7f22699fa700 1 -- 172.21.15.44:0/1676491547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22641022c0 msgr2=0x7f22641983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:37.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.510+0000 7f22699fa700 1 --2- 172.21.15.44:0/1676491547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22641022c0 0x7f22641983d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f225400d9b0 tx=0x7f225400dcc0 comp rx=0 tx=0).stop 2024-04-21T23:08:37.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.510+0000 7f22699fa700 1 -- 172.21.15.44:0/1676491547 shutdown_connections 2024-04-21T23:08:37.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.510+0000 7f22699fa700 1 --2- 172.21.15.44:0/1676491547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2264103e60 0x7f226419f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:37.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.510+0000 7f22699fa700 1 --2- 172.21.15.44:0/1676491547 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f22500a2c20 0x7f22500a50e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:37.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.510+0000 7f22699fa700 1 --2- 172.21.15.44:0/1676491547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f22641022c0 0x7f22641983d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:37.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.510+0000 7f22699fa700 1 --2- 172.21.15.44:0/1676491547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22641034c0 0x7f2264198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:37.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.510+0000 7f22699fa700 1 -- 172.21.15.44:0/1676491547 >> 172.21.15.44:0/1676491547 conn(0x7f22640fd880 msgr2=0x7f22641066f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:37.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.510+0000 7f22699fa700 1 -- 172.21.15.44:0/1676491547 shutdown_connections 2024-04-21T23:08:37.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:37.510+0000 7f22699fa700 1 -- 172.21.15.44:0/1676491547 wait complete. 2024-04-21T23:08:38.176 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:38 smithi044 ceph-mon[27934]: pgmap v361: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-04-21T23:08:38.177 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:38 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1676491547' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-21T23:08:38.206 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-21T23:08:38.223 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:08:38.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.332+0000 7f736c937640 1 -- 172.21.15.44:0/3572343990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73681017a0 msgr2=0x7f7368101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:38.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.332+0000 7f736c937640 1 --2- 172.21.15.44:0/3572343990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73681017a0 0x7f7368101bc0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7358009900 tx=0x7f735802bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:38.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.332+0000 7f736c937640 1 -- 172.21.15.44:0/3572343990 shutdown_connections 2024-04-21T23:08:38.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.332+0000 7f736c937640 1 --2- 172.21.15.44:0/3572343990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7368103350 0x7f7368109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.332+0000 7f736c937640 1 --2- 172.21.15.44:0/3572343990 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7368102990 0x7f7368102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.332+0000 7f736c937640 1 --2- 172.21.15.44:0/3572343990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f73681017a0 0x7f7368101bc0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.332+0000 7f736c937640 1 -- 172.21.15.44:0/3572343990 >> 172.21.15.44:0/3572343990 conn(0x7f73680fcd30 msgr2=0x7f73680ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:38.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.332+0000 7f736c937640 1 -- 172.21.15.44:0/3572343990 shutdown_connections 2024-04-21T23:08:38.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.332+0000 7f736c937640 1 -- 172.21.15.44:0/3572343990 wait complete. 2024-04-21T23:08:38.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.333+0000 7f736c937640 1 Processor -- start 2024-04-21T23:08:38.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.333+0000 7f736c937640 1 -- start start 2024-04-21T23:08:38.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.333+0000 7f736c937640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73681017a0 0x7f7368197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:38.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.333+0000 7f736c937640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7368102990 0x7f7368198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:38.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.333+0000 7f736c937640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7368103350 0x7f736819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:38.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.333+0000 7f736c937640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f736810d420 con 0x7f7368102990 2024-04-21T23:08:38.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.333+0000 7f736c937640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f736810d2a0 con 0x7f7368103350 2024-04-21T23:08:38.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.333+0000 7f736c937640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f736810d5a0 con 0x7f73681017a0 2024-04-21T23:08:38.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.333+0000 7f7365d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7368102990 0x7f7368198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:38.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.334+0000 7f7365d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7368102990 0x7f7368198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51388/0 (socket says 172.21.15.44:51388) 2024-04-21T23:08:38.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.334+0000 7f7365d9b640 1 -- 172.21.15.44:0/2146167965 learned_addr learned my addr 172.21.15.44:0/2146167965 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:38.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.334+0000 7f736659c640 1 --2- 172.21.15.44:0/2146167965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73681017a0 0x7f7368197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:38.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.334+0000 7f7366d9d640 1 --2- 172.21.15.44:0/2146167965 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7368103350 0x7f736819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:38.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.334+0000 7f7365d9b640 1 -- 172.21.15.44:0/2146167965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73681017a0 msgr2=0x7f7368197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:38.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.334+0000 7f7365d9b640 1 --2- 172.21.15.44:0/2146167965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73681017a0 0x7f7368197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.334+0000 7f7365d9b640 1 -- 172.21.15.44:0/2146167965 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7368103350 msgr2=0x7f736819f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:38.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.334+0000 7f7365d9b640 1 --2- 172.21.15.44:0/2146167965 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7368103350 0x7f736819f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.334+0000 7f7365d9b640 1 -- 172.21.15.44:0/2146167965 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f736819f6f0 con 0x7f7368102990 2024-04-21T23:08:38.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.334+0000 7f7365d9b640 1 --2- 172.21.15.44:0/2146167965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7368102990 0x7f7368198010 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f735c00c890 tx=0x7f735c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:38.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.334+0000 7f73577fe640 1 -- 172.21.15.44:0/2146167965 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c004220 con 0x7f7368102990 2024-04-21T23:08:38.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.334+0000 7f73577fe640 1 -- 172.21.15.44:0/2146167965 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f735c0043c0 con 0x7f7368102990 2024-04-21T23:08:38.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.334+0000 7f736c937640 1 -- 172.21.15.44:0/2146167965 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f736819f9e0 con 0x7f7368102990 2024-04-21T23:08:38.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.334+0000 7f73577fe640 1 -- 172.21.15.44:0/2146167965 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c0049e0 con 0x7f7368102990 2024-04-21T23:08:38.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.334+0000 7f736c937640 1 -- 172.21.15.44:0/2146167965 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f736819ff30 con 0x7f7368102990 2024-04-21T23:08:38.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.335+0000 7f736c937640 1 -- 172.21.15.44:0/2146167965 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7368108ac0 con 0x7f7368102990 2024-04-21T23:08:38.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.336+0000 7f73577fe640 1 -- 172.21.15.44:0/2146167965 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f735c0079b0 con 0x7f7368102990 2024-04-21T23:08:38.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.337+0000 7f73577fe640 1 --2- 172.21.15.44:0/2146167965 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f73500a2b90 0x7f73500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:38.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.337+0000 7f73577fe640 1 -- 172.21.15.44:0/2146167965 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7f735c0cf0b0 con 0x7f7368102990 2024-04-21T23:08:38.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.337+0000 7f736659c640 1 --2- 172.21.15.44:0/2146167965 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f73500a2b90 0x7f73500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:38.337 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.337+0000 7f736659c640 1 --2- 172.21.15.44:0/2146167965 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f73500a2b90 0x7f73500a5050 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7358000c00 tx=0x7f7358004f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:38.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.342+0000 7f73577fe640 1 -- 172.21.15.44:0/2146167965 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f735c014030 con 0x7f7368102990 2024-04-21T23:08:38.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:38 smithi069 ceph-mon[31816]: pgmap v361: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-04-21T23:08:38.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:38 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1676491547' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-21T23:08:38.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:38 smithi055 ceph-mon[31117]: pgmap v361: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-04-21T23:08:38.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:38 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1676491547' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-21T23:08:38.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.527+0000 7f736c937640 1 -- 172.21.15.44:0/2146167965 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7368062340 con 0x7f7368102990 2024-04-21T23:08:38.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.528+0000 7f73577fe640 1 -- 172.21.15.44:0/2146167965 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24716 (secure 0 0 0) 0x7f735c09bc50 con 0x7f7368102990 2024-04-21T23:08:38.527 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:38.527 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":94,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:08:25.093542+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":9,"to":11},{"from":5,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:08:38.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.530+0000 7f736c937640 1 -- 172.21.15.44:0/2146167965 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f73500a2b90 msgr2=0x7f73500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:38.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.530+0000 7f736c937640 1 --2- 172.21.15.44:0/2146167965 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f73500a2b90 0x7f73500a5050 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7358000c00 tx=0x7f7358004f10 comp rx=0 tx=0).stop 2024-04-21T23:08:38.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.530+0000 7f736c937640 1 -- 172.21.15.44:0/2146167965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7368102990 msgr2=0x7f7368198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:38.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.530+0000 7f736c937640 1 --2- 172.21.15.44:0/2146167965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7368102990 0x7f7368198010 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f735c00c890 tx=0x7f735c00cd50 comp rx=0 tx=0).stop 2024-04-21T23:08:38.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.530+0000 7f736c937640 1 -- 172.21.15.44:0/2146167965 shutdown_connections 2024-04-21T23:08:38.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.530+0000 7f736c937640 1 --2- 172.21.15.44:0/2146167965 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f73500a2b90 0x7f73500a5050 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.530+0000 7f736c937640 1 --2- 172.21.15.44:0/2146167965 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7368103350 0x7f736819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.530+0000 7f736c937640 1 --2- 172.21.15.44:0/2146167965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7368102990 0x7f7368198010 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.530+0000 7f736c937640 1 --2- 172.21.15.44:0/2146167965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73681017a0 0x7f7368197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.530+0000 7f736c937640 1 -- 172.21.15.44:0/2146167965 >> 172.21.15.44:0/2146167965 conn(0x7f73680fcd30 msgr2=0x7f7368105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:38.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.530+0000 7f736c937640 1 -- 172.21.15.44:0/2146167965 shutdown_connections 2024-04-21T23:08:38.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.530+0000 7f736c937640 1 -- 172.21.15.44:0/2146167965 wait complete. 2024-04-21T23:08:38.541 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:01:47.507503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:06:23.993760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:06:24.954661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:06:28.677618+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:08:38.541 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:08:38.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.648+0000 7fd7213b1640 1 -- 172.21.15.44:0/3013926653 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd71c102240 msgr2=0x7fd71c108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:38.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.648+0000 7fd7213b1640 1 --2- 172.21.15.44:0/3013926653 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd71c102240 0x7fd71c108ad0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd714009ca0 tx=0x7fd71402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.648+0000 7fd7213b1640 1 -- 172.21.15.44:0/3013926653 shutdown_connections 2024-04-21T23:08:38.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.648+0000 7fd7213b1640 1 --2- 172.21.15.44:0/3013926653 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd71c102240 0x7fd71c108ad0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.648+0000 7fd7213b1640 1 --2- 172.21.15.44:0/3013926653 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd71c101880 0x7fd71c101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.648+0000 7fd7213b1640 1 --2- 172.21.15.44:0/3013926653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd71c100690 0x7fd71c100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.648+0000 7fd7213b1640 1 -- 172.21.15.44:0/3013926653 >> 172.21.15.44:0/3013926653 conn(0x7fd71c0fbc20 msgr2=0x7fd71c0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:38.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.648+0000 7fd7213b1640 1 -- 172.21.15.44:0/3013926653 shutdown_connections 2024-04-21T23:08:38.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.648+0000 7fd7213b1640 1 -- 172.21.15.44:0/3013926653 wait complete. 2024-04-21T23:08:38.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.649+0000 7fd7213b1640 1 Processor -- start 2024-04-21T23:08:38.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.649+0000 7fd7213b1640 1 -- start start 2024-04-21T23:08:38.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.649+0000 7fd7213b1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd71c100690 0x7fd71c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:38.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.649+0000 7fd7213b1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd71c101880 0x7fd71c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:38.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.649+0000 7fd7213b1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd71c102240 0x7fd71c19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:38.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.649+0000 7fd7213b1640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd71c10c380 con 0x7fd71c101880 2024-04-21T23:08:38.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.649+0000 7fd7213b1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd71c10c200 con 0x7fd71c102240 2024-04-21T23:08:38.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.649+0000 7fd7213b1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd71c10c500 con 0x7fd71c100690 2024-04-21T23:08:38.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.649+0000 7fd713fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd71c101880 0x7fd71c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:38.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.649+0000 7fd713fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd71c101880 0x7fd71c198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51416/0 (socket says 172.21.15.44:51416) 2024-04-21T23:08:38.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.649+0000 7fd713fff640 1 -- 172.21.15.44:0/3055924761 learned_addr learned my addr 172.21.15.44:0/3055924761 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:38.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.649+0000 7fd71b7fe640 1 --2- 172.21.15.44:0/3055924761 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd71c102240 0x7fd71c19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:38.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.649+0000 7fd71affd640 1 --2- 172.21.15.44:0/3055924761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd71c100690 0x7fd71c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:38.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.649+0000 7fd713fff640 1 -- 172.21.15.44:0/3055924761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd71c100690 msgr2=0x7fd71c197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:38.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.649+0000 7fd713fff640 1 --2- 172.21.15.44:0/3055924761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd71c100690 0x7fd71c197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.649+0000 7fd713fff640 1 -- 172.21.15.44:0/3055924761 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd71c102240 msgr2=0x7fd71c19f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:38.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.649+0000 7fd713fff640 1 --2- 172.21.15.44:0/3055924761 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd71c102240 0x7fd71c19f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.649+0000 7fd713fff640 1 -- 172.21.15.44:0/3055924761 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71c19f700 con 0x7fd71c101880 2024-04-21T23:08:38.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.649+0000 7fd71b7fe640 1 --2- 172.21.15.44:0/3055924761 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd71c102240 0x7fd71c19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:08:38.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.650+0000 7fd713fff640 1 --2- 172.21.15.44:0/3055924761 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd71c101880 0x7fd71c198020 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd70400d950 tx=0x7fd70400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:38.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.650+0000 7fd718ff9640 1 -- 172.21.15.44:0/3055924761 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd704017070 con 0x7fd71c101880 2024-04-21T23:08:38.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.650+0000 7fd718ff9640 1 -- 172.21.15.44:0/3055924761 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd704009c30 con 0x7fd71c101880 2024-04-21T23:08:38.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.650+0000 7fd718ff9640 1 -- 172.21.15.44:0/3055924761 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70400f6b0 con 0x7fd71c101880 2024-04-21T23:08:38.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.650+0000 7fd7213b1640 1 -- 172.21.15.44:0/3055924761 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd71c19f9f0 con 0x7fd71c101880 2024-04-21T23:08:38.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.650+0000 7fd7213b1640 1 -- 172.21.15.44:0/3055924761 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd71c19ff40 con 0x7fd71c101880 2024-04-21T23:08:38.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.651+0000 7fd7213b1640 1 -- 172.21.15.44:0/3055924761 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd71c1a0170 con 0x7fd71c101880 2024-04-21T23:08:38.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.652+0000 7fd718ff9640 1 -- 172.21.15.44:0/3055924761 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd7040109d0 con 0x7fd71c101880 2024-04-21T23:08:38.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.653+0000 7fd718ff9640 1 --2- 172.21.15.44:0/3055924761 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd7080a2b90 0x7fd7080a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:38.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.653+0000 7fd71affd640 1 --2- 172.21.15.44:0/3055924761 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd7080a2b90 0x7fd7080a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:38.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.653+0000 7fd718ff9640 1 -- 172.21.15.44:0/3055924761 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7fd7040cf120 con 0x7fd71c101880 2024-04-21T23:08:38.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.653+0000 7fd71affd640 1 --2- 172.21.15.44:0/3055924761 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd7080a2b90 0x7fd7080a5050 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd71c1016e0 tx=0x7fd70c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:38.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.656+0000 7fd718ff9640 1 -- 172.21.15.44:0/3055924761 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd704009de0 con 0x7fd71c101880 2024-04-21T23:08:38.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.841+0000 7fd7213b1640 1 -- 172.21.15.44:0/3055924761 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd71c1a0340 con 0x7fd71c101880 2024-04-21T23:08:38.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.842+0000 7fd718ff9640 1 -- 172.21.15.44:0/3055924761 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fd70409bc60 con 0x7fd71c101880 2024-04-21T23:08:38.841 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-21T23:08:38.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.844+0000 7fd7213b1640 1 -- 172.21.15.44:0/3055924761 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd7080a2b90 msgr2=0x7fd7080a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:38.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.844+0000 7fd7213b1640 1 --2- 172.21.15.44:0/3055924761 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd7080a2b90 0x7fd7080a5050 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd71c1016e0 tx=0x7fd70c008040 comp rx=0 tx=0).stop 2024-04-21T23:08:38.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.844+0000 7fd7213b1640 1 -- 172.21.15.44:0/3055924761 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd71c101880 msgr2=0x7fd71c198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:38.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.844+0000 7fd7213b1640 1 --2- 172.21.15.44:0/3055924761 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd71c101880 0x7fd71c198020 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd70400d950 tx=0x7fd70400de10 comp rx=0 tx=0).stop 2024-04-21T23:08:38.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.844+0000 7fd7213b1640 1 -- 172.21.15.44:0/3055924761 shutdown_connections 2024-04-21T23:08:38.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.844+0000 7fd7213b1640 1 --2- 172.21.15.44:0/3055924761 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd7080a2b90 0x7fd7080a5050 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.844+0000 7fd7213b1640 1 --2- 172.21.15.44:0/3055924761 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd71c102240 0x7fd71c19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.844+0000 7fd7213b1640 1 --2- 172.21.15.44:0/3055924761 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd71c101880 0x7fd71c198020 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.844+0000 7fd7213b1640 1 --2- 172.21.15.44:0/3055924761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd71c100690 0x7fd71c197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.844+0000 7fd7213b1640 1 -- 172.21.15.44:0/3055924761 >> 172.21.15.44:0/3055924761 conn(0x7fd71c0fbc20 msgr2=0x7fd71c0fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:38.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.844+0000 7fd7213b1640 1 -- 172.21.15.44:0/3055924761 shutdown_connections 2024-04-21T23:08:38.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.844+0000 7fd7213b1640 1 -- 172.21.15.44:0/3055924761 wait complete. 2024-04-21T23:08:38.854 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:08:38.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.960+0000 7f0e5a89d640 1 -- 172.21.15.44:0/3599333386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e54101880 msgr2=0x7f0e54101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:38.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.960+0000 7f0e5a89d640 1 --2- 172.21.15.44:0/3599333386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e54101880 0x7f0e54101d00 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0e48009900 tx=0x7f0e4802bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:38.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.960+0000 7f0e5a89d640 1 -- 172.21.15.44:0/3599333386 shutdown_connections 2024-04-21T23:08:38.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.960+0000 7f0e5a89d640 1 --2- 172.21.15.44:0/3599333386 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e54102240 0x7f0e54108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.960+0000 7f0e5a89d640 1 --2- 172.21.15.44:0/3599333386 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e54101880 0x7f0e54101d00 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.960+0000 7f0e5a89d640 1 --2- 172.21.15.44:0/3599333386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e54100690 0x7f0e54100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.960+0000 7f0e5a89d640 1 -- 172.21.15.44:0/3599333386 >> 172.21.15.44:0/3599333386 conn(0x7f0e540fbc20 msgr2=0x7f0e540fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:38.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.960+0000 7f0e5a89d640 1 -- 172.21.15.44:0/3599333386 shutdown_connections 2024-04-21T23:08:38.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.960+0000 7f0e5a89d640 1 -- 172.21.15.44:0/3599333386 wait complete. 2024-04-21T23:08:38.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.960+0000 7f0e5a89d640 1 Processor -- start 2024-04-21T23:08:38.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.960+0000 7f0e5a89d640 1 -- start start 2024-04-21T23:08:38.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e5a89d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e54100690 0x7f0e54199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:38.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e5a89d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e54101880 0x7f0e5419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:38.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e5a89d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e54102240 0x7f0e541a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:38.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e5a89d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5410c3c0 con 0x7f0e54100690 2024-04-21T23:08:38.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e5a89d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5410c240 con 0x7f0e54101880 2024-04-21T23:08:38.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e5a89d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5410c540 con 0x7f0e54102240 2024-04-21T23:08:38.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e53fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e54100690 0x7f0e54199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:38.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e53fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e54100690 0x7f0e54199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51436/0 (socket says 172.21.15.44:51436) 2024-04-21T23:08:38.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e53fff640 1 -- 172.21.15.44:0/2801734427 learned_addr learned my addr 172.21.15.44:0/2801734427 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:38.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e537fe640 1 --2- 172.21.15.44:0/2801734427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e54101880 0x7f0e5419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:38.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e58e3a640 1 --2- 172.21.15.44:0/2801734427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e54102240 0x7f0e541a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:38.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e53fff640 1 -- 172.21.15.44:0/2801734427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e54102240 msgr2=0x7f0e541a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:38.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e53fff640 1 --2- 172.21.15.44:0/2801734427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e54102240 0x7f0e541a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e53fff640 1 -- 172.21.15.44:0/2801734427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e54101880 msgr2=0x7f0e5419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:38.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e53fff640 1 --2- 172.21.15.44:0/2801734427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e54101880 0x7f0e5419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:38.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e53fff640 1 -- 172.21.15.44:0/2801734427 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e541a1930 con 0x7f0e54100690 2024-04-21T23:08:38.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e537fe640 1 --2- 172.21.15.44:0/2801734427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e54101880 0x7f0e5419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:08:38.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e58e3a640 1 --2- 172.21.15.44:0/2801734427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e54102240 0x7f0e541a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:08:38.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e53fff640 1 --2- 172.21.15.44:0/2801734427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e54100690 0x7f0e54199d20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0e4400d770 tx=0x7f0e4400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:38.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e517fa640 1 -- 172.21.15.44:0/2801734427 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e44017070 con 0x7f0e54100690 2024-04-21T23:08:38.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e5a89d640 1 -- 172.21.15.44:0/2801734427 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e541a1c20 con 0x7f0e54100690 2024-04-21T23:08:38.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e517fa640 1 -- 172.21.15.44:0/2801734427 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e44009c30 con 0x7f0e54100690 2024-04-21T23:08:38.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.961+0000 7f0e517fa640 1 -- 172.21.15.44:0/2801734427 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e44003dd0 con 0x7f0e54100690 2024-04-21T23:08:38.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.962+0000 7f0e5a89d640 1 -- 172.21.15.44:0/2801734427 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e541a21c0 con 0x7f0e54100690 2024-04-21T23:08:38.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.962+0000 7f0e5a89d640 1 -- 172.21.15.44:0/2801734427 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e541a2390 con 0x7f0e54100690 2024-04-21T23:08:38.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.964+0000 7f0e517fa640 1 -- 172.21.15.44:0/2801734427 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0e44004000 con 0x7f0e54100690 2024-04-21T23:08:38.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.964+0000 7f0e517fa640 1 --2- 172.21.15.44:0/2801734427 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f0e3c0a2b90 0x7f0e3c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:38.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.964+0000 7f0e517fa640 1 -- 172.21.15.44:0/2801734427 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7f0e440cf300 con 0x7f0e54100690 2024-04-21T23:08:38.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.964+0000 7f0e537fe640 1 --2- 172.21.15.44:0/2801734427 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f0e3c0a2b90 0x7f0e3c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:38.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.965+0000 7f0e537fe640 1 --2- 172.21.15.44:0/2801734427 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f0e3c0a2b90 0x7f0e3c0a5050 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0e4802c240 tx=0x7f0e4802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:38.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:38.968+0000 7f0e517fa640 1 -- 172.21.15.44:0/2801734427 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0e44009de0 con 0x7f0e54100690 2024-04-21T23:08:39.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.152+0000 7f0e5a89d640 1 -- 172.21.15.44:0/2801734427 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0e541a2600 con 0x7f0e54100690 2024-04-21T23:08:39.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.153+0000 7f0e517fa640 1 -- 172.21.15.44:0/2801734427 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f0e4409bea0 con 0x7f0e54100690 2024-04-21T23:08:39.152 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:08:39.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.155+0000 7f0e5a89d640 1 -- 172.21.15.44:0/2801734427 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f0e3c0a2b90 msgr2=0x7f0e3c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:39.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.155+0000 7f0e5a89d640 1 --2- 172.21.15.44:0/2801734427 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f0e3c0a2b90 0x7f0e3c0a5050 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0e4802c240 tx=0x7f0e4802df90 comp rx=0 tx=0).stop 2024-04-21T23:08:39.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.155+0000 7f0e5a89d640 1 -- 172.21.15.44:0/2801734427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e54100690 msgr2=0x7f0e54199d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:39.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.155+0000 7f0e5a89d640 1 --2- 172.21.15.44:0/2801734427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e54100690 0x7f0e54199d20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0e4400d770 tx=0x7f0e4400dc30 comp rx=0 tx=0).stop 2024-04-21T23:08:39.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.155+0000 7f0e5a89d640 1 -- 172.21.15.44:0/2801734427 shutdown_connections 2024-04-21T23:08:39.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.155+0000 7f0e5a89d640 1 --2- 172.21.15.44:0/2801734427 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f0e3c0a2b90 0x7f0e3c0a5050 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.155+0000 7f0e5a89d640 1 --2- 172.21.15.44:0/2801734427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e54102240 0x7f0e541a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.155+0000 7f0e5a89d640 1 --2- 172.21.15.44:0/2801734427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e54101880 0x7f0e5419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.155+0000 7f0e5a89d640 1 --2- 172.21.15.44:0/2801734427 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0e54100690 0x7f0e54199d20 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.155+0000 7f0e5a89d640 1 -- 172.21.15.44:0/2801734427 >> 172.21.15.44:0/2801734427 conn(0x7f0e540fbc20 msgr2=0x7f0e540fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:39.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.155+0000 7f0e5a89d640 1 -- 172.21.15.44:0/2801734427 shutdown_connections 2024-04-21T23:08:39.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.155+0000 7f0e5a89d640 1 -- 172.21.15.44:0/2801734427 wait complete. 2024-04-21T23:08:39.165 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:08:39.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.271+0000 7fb11bb8c640 1 -- 172.21.15.44:0/2108802198 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb114102990 msgr2=0x7fb114102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:39.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.271+0000 7fb11bb8c640 1 --2- 172.21.15.44:0/2108802198 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb114102990 0x7fb114102e10 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb104009900 tx=0x7fb10402bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:39.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.271+0000 7fb11bb8c640 1 -- 172.21.15.44:0/2108802198 shutdown_connections 2024-04-21T23:08:39.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.271+0000 7fb11bb8c640 1 --2- 172.21.15.44:0/2108802198 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb114103350 0x7fb114109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.271+0000 7fb11bb8c640 1 --2- 172.21.15.44:0/2108802198 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb114102990 0x7fb114102e10 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.271+0000 7fb11bb8c640 1 --2- 172.21.15.44:0/2108802198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1141017a0 0x7fb114101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.271+0000 7fb11bb8c640 1 -- 172.21.15.44:0/2108802198 >> 172.21.15.44:0/2108802198 conn(0x7fb1140fcd10 msgr2=0x7fb1140ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:39.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.271+0000 7fb11bb8c640 1 -- 172.21.15.44:0/2108802198 shutdown_connections 2024-04-21T23:08:39.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.271+0000 7fb11bb8c640 1 -- 172.21.15.44:0/2108802198 wait complete. 2024-04-21T23:08:39.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.272+0000 7fb11bb8c640 1 Processor -- start 2024-04-21T23:08:39.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.272+0000 7fb11bb8c640 1 -- start start 2024-04-21T23:08:39.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.272+0000 7fb11bb8c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1141017a0 0x7fb114197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:39.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.272+0000 7fb11bb8c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb114102990 0x7fb114198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:39.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.272+0000 7fb11bb8c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb114103350 0x7fb11419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:39.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.272+0000 7fb11bb8c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb11410d420 con 0x7fb114103350 2024-04-21T23:08:39.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.272+0000 7fb11bb8c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb11410d2a0 con 0x7fb1141017a0 2024-04-21T23:08:39.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.272+0000 7fb11bb8c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb11410d5a0 con 0x7fb114102990 2024-04-21T23:08:39.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.272+0000 7fb11a129640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb114103350 0x7fb11419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:39.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.272+0000 7fb11a129640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb114103350 0x7fb11419f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51456/0 (socket says 172.21.15.44:51456) 2024-04-21T23:08:39.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.272+0000 7fb11a129640 1 -- 172.21.15.44:0/1523276802 learned_addr learned my addr 172.21.15.44:0/1523276802 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:39.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.272+0000 7fb119127640 1 --2- 172.21.15.44:0/1523276802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb114102990 0x7fb114198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:39.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.272+0000 7fb119928640 1 --2- 172.21.15.44:0/1523276802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1141017a0 0x7fb114197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:39.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.272+0000 7fb11a129640 1 -- 172.21.15.44:0/1523276802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb114102990 msgr2=0x7fb114198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:39.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.272+0000 7fb11a129640 1 --2- 172.21.15.44:0/1523276802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb114102990 0x7fb114198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.272+0000 7fb11a129640 1 -- 172.21.15.44:0/1523276802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1141017a0 msgr2=0x7fb114197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:39.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.272+0000 7fb11a129640 1 --2- 172.21.15.44:0/1523276802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1141017a0 0x7fb114197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.272+0000 7fb11a129640 1 -- 172.21.15.44:0/1523276802 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb11419f6e0 con 0x7fb114103350 2024-04-21T23:08:39.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.272+0000 7fb119127640 1 --2- 172.21.15.44:0/1523276802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb114102990 0x7fb114198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:08:39.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.273+0000 7fb11a129640 1 --2- 172.21.15.44:0/1523276802 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb114103350 0x7fb11419f070 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb10c00eaa0 tx=0x7fb10c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:39.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.273+0000 7fb10affd640 1 -- 172.21.15.44:0/1523276802 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10c00cdb0 con 0x7fb114103350 2024-04-21T23:08:39.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.273+0000 7fb10affd640 1 -- 172.21.15.44:0/1523276802 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb10c005020 con 0x7fb114103350 2024-04-21T23:08:39.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.273+0000 7fb11bb8c640 1 -- 172.21.15.44:0/1523276802 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb11419f9d0 con 0x7fb114103350 2024-04-21T23:08:39.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.273+0000 7fb10affd640 1 -- 172.21.15.44:0/1523276802 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10c0116b0 con 0x7fb114103350 2024-04-21T23:08:39.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.273+0000 7fb11bb8c640 1 -- 172.21.15.44:0/1523276802 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb11419fea0 con 0x7fb114103350 2024-04-21T23:08:39.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.274+0000 7fb11bb8c640 1 -- 172.21.15.44:0/1523276802 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb114108b40 con 0x7fb114103350 2024-04-21T23:08:39.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.275+0000 7fb10affd640 1 -- 172.21.15.44:0/1523276802 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb10c005560 con 0x7fb114103350 2024-04-21T23:08:39.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.276+0000 7fb10affd640 1 --2- 172.21.15.44:0/1523276802 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb1000a2b90 0x7fb1000a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:39.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.276+0000 7fb119928640 1 --2- 172.21.15.44:0/1523276802 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb1000a2b90 0x7fb1000a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:39.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.276+0000 7fb10affd640 1 -- 172.21.15.44:0/1523276802 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7fb10c0cf700 con 0x7fb114103350 2024-04-21T23:08:39.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.276+0000 7fb119928640 1 --2- 172.21.15.44:0/1523276802 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb1000a2b90 0x7fb1000a5050 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb1100058c0 tx=0x7fb11000a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:39.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.280+0000 7fb10affd640 1 -- 172.21.15.44:0/1523276802 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb10c016030 con 0x7fb114103350 2024-04-21T23:08:39.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:39 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2146167965' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:39.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:39 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3055924761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:39.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.464+0000 7fb11bb8c640 1 -- 172.21.15.44:0/1523276802 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb114062340 con 0x7fb114103350 2024-04-21T23:08:39.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.464+0000 7fb10affd640 1 -- 172.21.15.44:0/1523276802 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fb10c09c2a0 con 0x7fb114103350 2024-04-21T23:08:39.463 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-21T23:08:39.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:39 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2146167965' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:39.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:39 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3055924761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:39.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:39 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2146167965' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:39.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:39 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3055924761' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:39.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.466+0000 7fb11bb8c640 1 -- 172.21.15.44:0/1523276802 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb1000a2b90 msgr2=0x7fb1000a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:39.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.466+0000 7fb11bb8c640 1 --2- 172.21.15.44:0/1523276802 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb1000a2b90 0x7fb1000a5050 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb1100058c0 tx=0x7fb11000a4c0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.466+0000 7fb11bb8c640 1 -- 172.21.15.44:0/1523276802 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb114103350 msgr2=0x7fb11419f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:39.465 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.466+0000 7fb11bb8c640 1 --2- 172.21.15.44:0/1523276802 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb114103350 0x7fb11419f070 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb10c00eaa0 tx=0x7fb10c00ef60 comp rx=0 tx=0).stop 2024-04-21T23:08:39.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.467+0000 7fb11bb8c640 1 -- 172.21.15.44:0/1523276802 shutdown_connections 2024-04-21T23:08:39.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.467+0000 7fb11bb8c640 1 --2- 172.21.15.44:0/1523276802 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fb1000a2b90 0x7fb1000a5050 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.467+0000 7fb11bb8c640 1 --2- 172.21.15.44:0/1523276802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb114102990 0x7fb114198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.467+0000 7fb11bb8c640 1 --2- 172.21.15.44:0/1523276802 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb114103350 0x7fb11419f070 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.467+0000 7fb11bb8c640 1 --2- 172.21.15.44:0/1523276802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb1141017a0 0x7fb114197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.467+0000 7fb11bb8c640 1 -- 172.21.15.44:0/1523276802 >> 172.21.15.44:0/1523276802 conn(0x7fb1140fcd10 msgr2=0x7fb114105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:39.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.467+0000 7fb11bb8c640 1 -- 172.21.15.44:0/1523276802 shutdown_connections 2024-04-21T23:08:39.466 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.467+0000 7fb11bb8c640 1 -- 172.21.15.44:0/1523276802 wait complete. 2024-04-21T23:08:39.476 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:08:39.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.583+0000 7f5f56c6d640 1 -- 172.21.15.44:0/3626803991 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f50102240 msgr2=0x7f5f50108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:39.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.583+0000 7f5f56c6d640 1 --2- 172.21.15.44:0/3626803991 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f50102240 0x7f5f50108ad0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5f48009ca0 tx=0x7f5f4802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.583+0000 7f5f56c6d640 1 -- 172.21.15.44:0/3626803991 shutdown_connections 2024-04-21T23:08:39.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.583+0000 7f5f56c6d640 1 --2- 172.21.15.44:0/3626803991 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f50102240 0x7f5f50108ad0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.583+0000 7f5f56c6d640 1 --2- 172.21.15.44:0/3626803991 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f50101880 0x7f5f50101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.583+0000 7f5f56c6d640 1 --2- 172.21.15.44:0/3626803991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f50100690 0x7f5f50100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.583+0000 7f5f56c6d640 1 -- 172.21.15.44:0/3626803991 >> 172.21.15.44:0/3626803991 conn(0x7f5f500fbc20 msgr2=0x7f5f500fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:39.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.583+0000 7f5f56c6d640 1 -- 172.21.15.44:0/3626803991 shutdown_connections 2024-04-21T23:08:39.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.583+0000 7f5f56c6d640 1 -- 172.21.15.44:0/3626803991 wait complete. 2024-04-21T23:08:39.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.584+0000 7f5f56c6d640 1 Processor -- start 2024-04-21T23:08:39.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.584+0000 7f5f56c6d640 1 -- start start 2024-04-21T23:08:39.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.584+0000 7f5f56c6d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f50100690 0x7f5f50197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:39.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.584+0000 7f5f56c6d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f50101880 0x7f5f50198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:39.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.584+0000 7f5f56c6d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f50102240 0x7f5f5019f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:39.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.584+0000 7f5f56c6d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5010d510 con 0x7f5f50102240 2024-04-21T23:08:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.584+0000 7f5f56c6d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5010d390 con 0x7f5f50100690 2024-04-21T23:08:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.584+0000 7f5f56c6d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5010d690 con 0x7f5f50101880 2024-04-21T23:08:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.584+0000 7f5f5520a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f50102240 0x7f5f5019f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.584+0000 7f5f5520a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f50102240 0x7f5f5019f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51474/0 (socket says 172.21.15.44:51474) 2024-04-21T23:08:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.584+0000 7f5f5520a640 1 -- 172.21.15.44:0/3916382156 learned_addr learned my addr 172.21.15.44:0/3916382156 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.584+0000 7f5f54a09640 1 --2- 172.21.15.44:0/3916382156 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f50100690 0x7f5f50197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.584+0000 7f5f5520a640 1 -- 172.21.15.44:0/3916382156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f50101880 msgr2=0x7f5f50198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.584+0000 7f5f5520a640 1 --2- 172.21.15.44:0/3916382156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f50101880 0x7f5f50198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.584+0000 7f5f5520a640 1 -- 172.21.15.44:0/3916382156 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f50100690 msgr2=0x7f5f50197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.584+0000 7f5f5520a640 1 --2- 172.21.15.44:0/3916382156 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f50100690 0x7f5f50197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.584+0000 7f5f5520a640 1 -- 172.21.15.44:0/3916382156 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f5019f750 con 0x7f5f50102240 2024-04-21T23:08:39.586 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.584+0000 7f5f5520a640 1 --2- 172.21.15.44:0/3916382156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f50102240 0x7f5f5019f0e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5f48037040 tx=0x7f5f48032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:39.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.585+0000 7f5f45ffb640 1 -- 172.21.15.44:0/3916382156 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f48039040 con 0x7f5f50102240 2024-04-21T23:08:39.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.585+0000 7f5f56c6d640 1 -- 172.21.15.44:0/3916382156 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f5019f9e0 con 0x7f5f50102240 2024-04-21T23:08:39.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.585+0000 7f5f45ffb640 1 -- 172.21.15.44:0/3916382156 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f48044070 con 0x7f5f50102240 2024-04-21T23:08:39.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.585+0000 7f5f45ffb640 1 -- 172.21.15.44:0/3916382156 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f48004330 con 0x7f5f50102240 2024-04-21T23:08:39.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.585+0000 7f5f56c6d640 1 -- 172.21.15.44:0/3916382156 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f5019ffa0 con 0x7f5f50102240 2024-04-21T23:08:39.587 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.586+0000 7f5f56c6d640 1 -- 172.21.15.44:0/3916382156 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f501079b0 con 0x7f5f50102240 2024-04-21T23:08:39.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.587+0000 7f5f45ffb640 1 -- 172.21.15.44:0/3916382156 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5f48003a00 con 0x7f5f50102240 2024-04-21T23:08:39.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.587+0000 7f5f45ffb640 1 --2- 172.21.15.44:0/3916382156 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5f380a2c10 0x7f5f380a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:39.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.588+0000 7f5f54a09640 1 --2- 172.21.15.44:0/3916382156 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5f380a2c10 0x7f5f380a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:39.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.588+0000 7f5f45ffb640 1 -- 172.21.15.44:0/3916382156 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7f5f480f1d30 con 0x7f5f50102240 2024-04-21T23:08:39.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.588+0000 7f5f54a09640 1 --2- 172.21.15.44:0/3916382156 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5f380a2c10 0x7f5f380a50d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5f501016e0 tx=0x7f5f4c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:39.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.592+0000 7f5f45ffb640 1 -- 172.21.15.44:0/3916382156 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5f480f7020 con 0x7f5f50102240 2024-04-21T23:08:39.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.776+0000 7f5f56c6d640 1 -- 172.21.15.44:0/3916382156 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f5f50062340 con 0x7f5f50102240 2024-04-21T23:08:39.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.776+0000 7f5f45ffb640 1 -- 172.21.15.44:0/3916382156 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f5f480be870 con 0x7f5f50102240 2024-04-21T23:08:39.775 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:08:39.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.778+0000 7f5f56c6d640 1 -- 172.21.15.44:0/3916382156 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5f380a2c10 msgr2=0x7f5f380a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:39.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.778+0000 7f5f56c6d640 1 --2- 172.21.15.44:0/3916382156 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5f380a2c10 0x7f5f380a50d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5f501016e0 tx=0x7f5f4c008040 comp rx=0 tx=0).stop 2024-04-21T23:08:39.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.778+0000 7f5f56c6d640 1 -- 172.21.15.44:0/3916382156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f50102240 msgr2=0x7f5f5019f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:39.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.778+0000 7f5f56c6d640 1 --2- 172.21.15.44:0/3916382156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f50102240 0x7f5f5019f0e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5f48037040 tx=0x7f5f48032f90 comp rx=0 tx=0).stop 2024-04-21T23:08:39.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.778+0000 7f5f56c6d640 1 -- 172.21.15.44:0/3916382156 shutdown_connections 2024-04-21T23:08:39.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.778+0000 7f5f56c6d640 1 --2- 172.21.15.44:0/3916382156 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5f380a2c10 0x7f5f380a50d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.778+0000 7f5f56c6d640 1 --2- 172.21.15.44:0/3916382156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f50101880 0x7f5f50198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.778+0000 7f5f56c6d640 1 --2- 172.21.15.44:0/3916382156 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5f50102240 0x7f5f5019f0e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.778+0000 7f5f56c6d640 1 --2- 172.21.15.44:0/3916382156 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5f50100690 0x7f5f50197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.778+0000 7f5f56c6d640 1 -- 172.21.15.44:0/3916382156 >> 172.21.15.44:0/3916382156 conn(0x7f5f500fbc20 msgr2=0x7f5f50104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:39.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.779+0000 7f5f56c6d640 1 -- 172.21.15.44:0/3916382156 shutdown_connections 2024-04-21T23:08:39.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.779+0000 7f5f56c6d640 1 -- 172.21.15.44:0/3916382156 wait complete. 2024-04-21T23:08:39.789 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:08:39.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.891+0000 7fea3228f640 1 -- 172.21.15.44:0/370922966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea2c1017a0 msgr2=0x7fea2c101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:39.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.891+0000 7fea3228f640 1 --2- 172.21.15.44:0/370922966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea2c1017a0 0x7fea2c101bc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fea1c009900 tx=0x7fea1c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:39.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.891+0000 7fea3228f640 1 -- 172.21.15.44:0/370922966 shutdown_connections 2024-04-21T23:08:39.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.891+0000 7fea3228f640 1 --2- 172.21.15.44:0/370922966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea2c103350 0x7fea2c109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.891+0000 7fea3228f640 1 --2- 172.21.15.44:0/370922966 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea2c102990 0x7fea2c102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.891+0000 7fea3228f640 1 --2- 172.21.15.44:0/370922966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea2c1017a0 0x7fea2c101bc0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.891+0000 7fea3228f640 1 -- 172.21.15.44:0/370922966 >> 172.21.15.44:0/370922966 conn(0x7fea2c0fcd30 msgr2=0x7fea2c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:39.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.891+0000 7fea3228f640 1 -- 172.21.15.44:0/370922966 shutdown_connections 2024-04-21T23:08:39.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.891+0000 7fea3228f640 1 -- 172.21.15.44:0/370922966 wait complete. 2024-04-21T23:08:39.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.892+0000 7fea3228f640 1 Processor -- start 2024-04-21T23:08:39.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.892+0000 7fea3228f640 1 -- start start 2024-04-21T23:08:39.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.892+0000 7fea3228f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea2c1017a0 0x7fea2c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:39.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.892+0000 7fea3228f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea2c102990 0x7fea2c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:39.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.892+0000 7fea3228f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea2c103350 0x7fea2c19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:39.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.892+0000 7fea3228f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c10d420 con 0x7fea2c1017a0 2024-04-21T23:08:39.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.892+0000 7fea3228f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c10d2a0 con 0x7fea2c102990 2024-04-21T23:08:39.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.892+0000 7fea3228f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c10d5a0 con 0x7fea2c103350 2024-04-21T23:08:39.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.892+0000 7fea2bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea2c1017a0 0x7fea2c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:39.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.892+0000 7fea2bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea2c1017a0 0x7fea2c197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51492/0 (socket says 172.21.15.44:51492) 2024-04-21T23:08:39.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.892+0000 7fea2bfff640 1 -- 172.21.15.44:0/930975943 learned_addr learned my addr 172.21.15.44:0/930975943 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:39.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.892+0000 7fea3082c640 1 --2- 172.21.15.44:0/930975943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea2c103350 0x7fea2c19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:39.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.892+0000 7fea2b7fe640 1 --2- 172.21.15.44:0/930975943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea2c102990 0x7fea2c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:39.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.893+0000 7fea2bfff640 1 -- 172.21.15.44:0/930975943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea2c103350 msgr2=0x7fea2c19f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:39.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.893+0000 7fea2bfff640 1 --2- 172.21.15.44:0/930975943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea2c103350 0x7fea2c19f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.893+0000 7fea2bfff640 1 -- 172.21.15.44:0/930975943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea2c102990 msgr2=0x7fea2c197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:39.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.893+0000 7fea2bfff640 1 --2- 172.21.15.44:0/930975943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea2c102990 0x7fea2c197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:39.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.893+0000 7fea2bfff640 1 -- 172.21.15.44:0/930975943 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2c19f6c0 con 0x7fea2c1017a0 2024-04-21T23:08:39.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.893+0000 7fea2bfff640 1 --2- 172.21.15.44:0/930975943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea2c1017a0 0x7fea2c197aa0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fea1c005ab0 tx=0x7fea1c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:39.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.893+0000 7fea297fa640 1 -- 172.21.15.44:0/930975943 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c041070 con 0x7fea2c1017a0 2024-04-21T23:08:39.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.893+0000 7fea3228f640 1 -- 172.21.15.44:0/930975943 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea2c19f950 con 0x7fea2c1017a0 2024-04-21T23:08:39.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.893+0000 7fea297fa640 1 -- 172.21.15.44:0/930975943 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea1c0054e0 con 0x7fea2c1017a0 2024-04-21T23:08:39.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.893+0000 7fea297fa640 1 -- 172.21.15.44:0/930975943 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c034440 con 0x7fea2c1017a0 2024-04-21T23:08:39.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.893+0000 7fea3228f640 1 -- 172.21.15.44:0/930975943 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea2c19fe40 con 0x7fea2c1017a0 2024-04-21T23:08:39.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.894+0000 7fea3228f640 1 -- 172.21.15.44:0/930975943 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea2c108ac0 con 0x7fea2c1017a0 2024-04-21T23:08:39.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.895+0000 7fea297fa640 1 -- 172.21.15.44:0/930975943 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fea1c044070 con 0x7fea2c1017a0 2024-04-21T23:08:39.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.896+0000 7fea297fa640 1 --2- 172.21.15.44:0/930975943 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fea140a2b90 0x7fea140a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:39.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.896+0000 7fea2b7fe640 1 --2- 172.21.15.44:0/930975943 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fea140a2b90 0x7fea140a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:39.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.896+0000 7fea297fa640 1 -- 172.21.15.44:0/930975943 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7fea1c0f0fe0 con 0x7fea2c1017a0 2024-04-21T23:08:39.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.896+0000 7fea2b7fe640 1 --2- 172.21.15.44:0/930975943 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fea140a2b90 0x7fea140a5050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fea2c198fb0 tx=0x7fea2000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:39.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:39.900+0000 7fea297fa640 1 -- 172.21.15.44:0/930975943 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fea1c0bd510 con 0x7fea2c1017a0 2024-04-21T23:08:40.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.123+0000 7fea3228f640 1 -- 172.21.15.44:0/930975943 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea2c02d170 con 0x7fea2c1017a0 2024-04-21T23:08:40.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.124+0000 7fea297fa640 1 -- 172.21.15.44:0/930975943 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11791 (secure 0 0 0) 0x7fea1c0c1810 con 0x7fea2c1017a0 2024-04-21T23:08:40.123 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:40.123 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:06:50.685663+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3743898853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3743898853},{"type":"v1","addr":"172.21.15.69:6839","nonce":3743898853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:08:40.123 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2024-04-21T23:08:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.126+0000 7fea3228f640 1 -- 172.21.15.44:0/930975943 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fea140a2b90 msgr2=0x7fea140a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.126+0000 7fea3228f640 1 --2- 172.21.15.44:0/930975943 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fea140a2b90 0x7fea140a5050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fea2c198fb0 tx=0x7fea2000b410 comp rx=0 tx=0).stop 2024-04-21T23:08:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.126+0000 7fea3228f640 1 -- 172.21.15.44:0/930975943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea2c1017a0 msgr2=0x7fea2c197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.126+0000 7fea3228f640 1 --2- 172.21.15.44:0/930975943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea2c1017a0 0x7fea2c197aa0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fea1c005ab0 tx=0x7fea1c005dc0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.126+0000 7fea3228f640 1 -- 172.21.15.44:0/930975943 shutdown_connections 2024-04-21T23:08:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.126+0000 7fea3228f640 1 --2- 172.21.15.44:0/930975943 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fea140a2b90 0x7fea140a5050 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.126+0000 7fea3228f640 1 --2- 172.21.15.44:0/930975943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea2c103350 0x7fea2c19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.126+0000 7fea3228f640 1 --2- 172.21.15.44:0/930975943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fea2c102990 0x7fea2c197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.126+0000 7fea3228f640 1 --2- 172.21.15.44:0/930975943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fea2c1017a0 0x7fea2c197aa0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.126+0000 7fea3228f640 1 -- 172.21.15.44:0/930975943 >> 172.21.15.44:0/930975943 conn(0x7fea2c0fcd30 msgr2=0x7fea2c105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.126+0000 7fea3228f640 1 -- 172.21.15.44:0/930975943 shutdown_connections 2024-04-21T23:08:40.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.126+0000 7fea3228f640 1 -- 172.21.15.44:0/930975943 wait complete. 2024-04-21T23:08:40.137 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:08:40.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.241+0000 7f84f8d37640 1 -- 172.21.15.44:0/2227789530 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84f4102250 msgr2=0x7f84f4108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:40.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.241+0000 7f84f8d37640 1 --2- 172.21.15.44:0/2227789530 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84f4102250 0x7f84f4108ae0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f84ec009ca0 tx=0x7f84ec02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.241+0000 7f84f8d37640 1 -- 172.21.15.44:0/2227789530 shutdown_connections 2024-04-21T23:08:40.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.241+0000 7f84f8d37640 1 --2- 172.21.15.44:0/2227789530 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84f4102250 0x7f84f4108ae0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.241+0000 7f84f8d37640 1 --2- 172.21.15.44:0/2227789530 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84f4101890 0x7f84f4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.241+0000 7f84f8d37640 1 --2- 172.21.15.44:0/2227789530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f41006a0 0x7f84f4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.242+0000 7f84f8d37640 1 -- 172.21.15.44:0/2227789530 >> 172.21.15.44:0/2227789530 conn(0x7f84f40fbc10 msgr2=0x7f84f40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:40.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.242+0000 7f84f8d37640 1 -- 172.21.15.44:0/2227789530 shutdown_connections 2024-04-21T23:08:40.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.242+0000 7f84f8d37640 1 -- 172.21.15.44:0/2227789530 wait complete. 2024-04-21T23:08:40.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.242+0000 7f84f8d37640 1 Processor -- start 2024-04-21T23:08:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.242+0000 7f84f8d37640 1 -- start start 2024-04-21T23:08:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.242+0000 7f84f8d37640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84f41006a0 0x7f84f41045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.242+0000 7f84f8d37640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84f4101890 0x7f84f4102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.243+0000 7f84f8d37640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f4102250 0x7f84f41031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.243+0000 7f84f8d37640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f84f410c320 con 0x7f84f4101890 2024-04-21T23:08:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.243+0000 7f84f8d37640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f84f410c1a0 con 0x7f84f41006a0 2024-04-21T23:08:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.243+0000 7f84f8d37640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f84f410c4a0 con 0x7f84f4102250 2024-04-21T23:08:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.243+0000 7f84ebfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84f4101890 0x7f84f4102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.243+0000 7f84ebfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84f4101890 0x7f84f4102c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51514/0 (socket says 172.21.15.44:51514) 2024-04-21T23:08:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.243+0000 7f84ebfff640 1 -- 172.21.15.44:0/3941820431 learned_addr learned my addr 172.21.15.44:0/3941820431 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.243+0000 7f84f259c640 1 --2- 172.21.15.44:0/3941820431 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84f41006a0 0x7f84f41045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.243+0000 7f84f2d9d640 1 --2- 172.21.15.44:0/3941820431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f4102250 0x7f84f41031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:40.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.243+0000 7f84ebfff640 1 -- 172.21.15.44:0/3941820431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f4102250 msgr2=0x7f84f41031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:40.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.243+0000 7f84ebfff640 1 --2- 172.21.15.44:0/3941820431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f4102250 0x7f84f41031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.243+0000 7f84ebfff640 1 -- 172.21.15.44:0/3941820431 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84f41006a0 msgr2=0x7f84f41045e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:40.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.243+0000 7f84ebfff640 1 --2- 172.21.15.44:0/3941820431 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84f41006a0 0x7f84f41045e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.243+0000 7f84ebfff640 1 -- 172.21.15.44:0/3941820431 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84f4103a70 con 0x7f84f4101890 2024-04-21T23:08:40.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.243+0000 7f84f2d9d640 1 --2- 172.21.15.44:0/3941820431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f4102250 0x7f84f41031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:08:40.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.243+0000 7f84ebfff640 1 --2- 172.21.15.44:0/3941820431 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84f4101890 0x7f84f4102c70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00e9f0 tx=0x7f84dc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:40.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.243+0000 7f84eb7fe640 1 -- 172.21.15.44:0/3941820431 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc00cd80 con 0x7f84f4101890 2024-04-21T23:08:40.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.243+0000 7f84eb7fe640 1 -- 172.21.15.44:0/3941820431 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84dc010910 con 0x7f84f4101890 2024-04-21T23:08:40.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.243+0000 7f84f8d37640 1 -- 172.21.15.44:0/3941820431 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84f41a1b70 con 0x7f84f4101890 2024-04-21T23:08:40.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.243+0000 7f84eb7fe640 1 -- 172.21.15.44:0/3941820431 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc018990 con 0x7f84f4101890 2024-04-21T23:08:40.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.244+0000 7f84f8d37640 1 -- 172.21.15.44:0/3941820431 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84f41a2090 con 0x7f84f4101890 2024-04-21T23:08:40.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.245+0000 7f84f8d37640 1 -- 172.21.15.44:0/3941820431 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84f41a2260 con 0x7f84f4101890 2024-04-21T23:08:40.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.247+0000 7f84eb7fe640 1 -- 172.21.15.44:0/3941820431 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f84dc010ac0 con 0x7f84f4101890 2024-04-21T23:08:40.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.247+0000 7f84eb7fe640 1 --2- 172.21.15.44:0/3941820431 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f84e00a2b90 0x7f84e00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:40.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.247+0000 7f84eb7fe640 1 -- 172.21.15.44:0/3941820431 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7f84dc0cf7d0 con 0x7f84f4101890 2024-04-21T23:08:40.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.247+0000 7f84f259c640 1 --2- 172.21.15.44:0/3941820431 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f84e00a2b90 0x7f84e00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:40.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.248+0000 7f84f259c640 1 --2- 172.21.15.44:0/3941820431 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f84e00a2b90 0x7f84e00a5050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f84f41016f0 tx=0x7f84e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:40.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.252+0000 7f84eb7fe640 1 -- 172.21.15.44:0/3941820431 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f84dc09c370 con 0x7f84f4101890 2024-04-21T23:08:40.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:40 smithi044 ceph-mon[27934]: pgmap v362: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-04-21T23:08:40.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:40 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2801734427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:40.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:40 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1523276802' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:40.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:40 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3916382156' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:40.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:40 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/930975943' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:40.438 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.440+0000 7f84f8d37640 1 -- 172.21.15.44:0/3941820431 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f84f41a25f0 con 0x7f84f4101890 2024-04-21T23:08:40.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.441+0000 7f84eb7fe640 1 -- 172.21.15.44:0/3941820431 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24716 (secure 0 0 0) 0x7f84dc09bc70 con 0x7f84f4101890 2024-04-21T23:08:40.440 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:40.440 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":94,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:08:25.093542+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":9,"to":11},{"from":5,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:08:40.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.443+0000 7f84f8d37640 1 -- 172.21.15.44:0/3941820431 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f84e00a2b90 msgr2=0x7f84e00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:40.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.443+0000 7f84f8d37640 1 --2- 172.21.15.44:0/3941820431 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f84e00a2b90 0x7f84e00a5050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f84f41016f0 tx=0x7f84e4005f90 comp rx=0 tx=0).stop 2024-04-21T23:08:40.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.443+0000 7f84f8d37640 1 -- 172.21.15.44:0/3941820431 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84f4101890 msgr2=0x7f84f4102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:40.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.443+0000 7f84f8d37640 1 --2- 172.21.15.44:0/3941820431 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84f4101890 0x7f84f4102c70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00e9f0 tx=0x7f84dc00eeb0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.444+0000 7f84f8d37640 1 -- 172.21.15.44:0/3941820431 shutdown_connections 2024-04-21T23:08:40.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.444+0000 7f84f8d37640 1 --2- 172.21.15.44:0/3941820431 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f84e00a2b90 0x7f84e00a5050 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.444+0000 7f84f8d37640 1 --2- 172.21.15.44:0/3941820431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f4102250 0x7f84f41031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.444+0000 7f84f8d37640 1 --2- 172.21.15.44:0/3941820431 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84f4101890 0x7f84f4102c70 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.444+0000 7f84f8d37640 1 --2- 172.21.15.44:0/3941820431 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84f41006a0 0x7f84f41045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.444+0000 7f84f8d37640 1 -- 172.21.15.44:0/3941820431 >> 172.21.15.44:0/3941820431 conn(0x7f84f40fbc10 msgr2=0x7f84f40fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:40.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.444+0000 7f84f8d37640 1 -- 172.21.15.44:0/3941820431 shutdown_connections 2024-04-21T23:08:40.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.444+0000 7f84f8d37640 1 -- 172.21.15.44:0/3941820431 wait complete. 2024-04-21T23:08:40.454 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:01:47.507503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:06:23.993760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:06:24.954661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:06:28.677618+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:08:40.455 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:08:40.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:40 smithi069 ceph-mon[31816]: pgmap v362: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-04-21T23:08:40.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:40 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2801734427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:40.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:40 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1523276802' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:40.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:40 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3916382156' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:40.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:40 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/930975943' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:40.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:40 smithi055 ceph-mon[31117]: pgmap v362: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-04-21T23:08:40.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:40 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2801734427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:40.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:40 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1523276802' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:40.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:40 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3916382156' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:40.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:40 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/930975943' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:40.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.561+0000 7f943d537640 1 -- 172.21.15.44:0/3027354819 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9438100690 msgr2=0x7f9438100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:40.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.561+0000 7f943d537640 1 --2- 172.21.15.44:0/3027354819 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9438100690 0x7f9438100ab0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9428009900 tx=0x7f942802bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:40.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.561+0000 7f943d537640 1 -- 172.21.15.44:0/3027354819 shutdown_connections 2024-04-21T23:08:40.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.561+0000 7f943d537640 1 --2- 172.21.15.44:0/3027354819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9438102240 0x7f9438108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.561+0000 7f943d537640 1 --2- 172.21.15.44:0/3027354819 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9438101880 0x7f9438101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.561+0000 7f943d537640 1 --2- 172.21.15.44:0/3027354819 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9438100690 0x7f9438100ab0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.561+0000 7f943d537640 1 -- 172.21.15.44:0/3027354819 >> 172.21.15.44:0/3027354819 conn(0x7f94380fbc20 msgr2=0x7f94380fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:40.563 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.561+0000 7f943d537640 1 -- 172.21.15.44:0/3027354819 shutdown_connections 2024-04-21T23:08:40.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.561+0000 7f943d537640 1 -- 172.21.15.44:0/3027354819 wait complete. 2024-04-21T23:08:40.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.562+0000 7f943d537640 1 Processor -- start 2024-04-21T23:08:40.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.562+0000 7f943d537640 1 -- start start 2024-04-21T23:08:40.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.562+0000 7f943d537640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9438100690 0x7f943806a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:40.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.562+0000 7f943d537640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9438101880 0x7f943819c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:40.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.562+0000 7f943d537640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9438102240 0x7f943819ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:40.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.562+0000 7f943d537640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f943810c310 con 0x7f9438102240 2024-04-21T23:08:40.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.562+0000 7f943d537640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f943810c190 con 0x7f9438100690 2024-04-21T23:08:40.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.562+0000 7f943d537640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f943810c490 con 0x7f9438101880 2024-04-21T23:08:40.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.562+0000 7f94377fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9438102240 0x7f943819ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:40.564 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.562+0000 7f94377fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9438102240 0x7f943819ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51534/0 (socket says 172.21.15.44:51534) 2024-04-21T23:08:40.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.562+0000 7f94377fe640 1 -- 172.21.15.44:0/2453211313 learned_addr learned my addr 172.21.15.44:0/2453211313 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:40.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.562+0000 7f94367fc640 1 --2- 172.21.15.44:0/2453211313 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9438101880 0x7f943819c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:40.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.562+0000 7f9436ffd640 1 --2- 172.21.15.44:0/2453211313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9438100690 0x7f943806a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:40.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.563+0000 7f94377fe640 1 -- 172.21.15.44:0/2453211313 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9438101880 msgr2=0x7f943819c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:40.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.563+0000 7f94377fe640 1 --2- 172.21.15.44:0/2453211313 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9438101880 0x7f943819c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.563+0000 7f94377fe640 1 -- 172.21.15.44:0/2453211313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9438100690 msgr2=0x7f943806a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:40.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.563+0000 7f94377fe640 1 --2- 172.21.15.44:0/2453211313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9438100690 0x7f943806a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.563+0000 7f94377fe640 1 -- 172.21.15.44:0/2453211313 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f943819f700 con 0x7f9438102240 2024-04-21T23:08:40.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.563+0000 7f94377fe640 1 --2- 172.21.15.44:0/2453211313 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9438102240 0x7f943819ee40 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f943000eaa0 tx=0x7f943000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:40.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.563+0000 7f941ffff640 1 -- 172.21.15.44:0/2453211313 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f943000cdb0 con 0x7f9438102240 2024-04-21T23:08:40.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.563+0000 7f941ffff640 1 -- 172.21.15.44:0/2453211313 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9430005020 con 0x7f9438102240 2024-04-21T23:08:40.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.563+0000 7f941ffff640 1 -- 172.21.15.44:0/2453211313 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94300116b0 con 0x7f9438102240 2024-04-21T23:08:40.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.563+0000 7f943d537640 1 -- 172.21.15.44:0/2453211313 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f943819f9f0 con 0x7f9438102240 2024-04-21T23:08:40.565 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.563+0000 7f943d537640 1 -- 172.21.15.44:0/2453211313 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f943819ff90 con 0x7f9438102240 2024-04-21T23:08:40.566 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.564+0000 7f943d537640 1 -- 172.21.15.44:0/2453211313 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94381079d0 con 0x7f9438102240 2024-04-21T23:08:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.565+0000 7f941ffff640 1 -- 172.21.15.44:0/2453211313 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9430005560 con 0x7f9438102240 2024-04-21T23:08:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.566+0000 7f941ffff640 1 --2- 172.21.15.44:0/2453211313 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f94200a2c10 0x7f94200a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.566+0000 7f941ffff640 1 -- 172.21.15.44:0/2453211313 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7f94300cf8c0 con 0x7f9438102240 2024-04-21T23:08:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.566+0000 7f9436ffd640 1 --2- 172.21.15.44:0/2453211313 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f94200a2c10 0x7f94200a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:40.567 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.566+0000 7f9436ffd640 1 --2- 172.21.15.44:0/2453211313 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f94200a2c10 0x7f94200a50d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9428000c00 tx=0x7f9428005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:40.568 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.569+0000 7f941ffff640 1 -- 172.21.15.44:0/2453211313 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f9430016030 con 0x7f9438102240 2024-04-21T23:08:40.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.754+0000 7f943d537640 1 -- 172.21.15.44:0/2453211313 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f943802d320 con 0x7f9438102240 2024-04-21T23:08:40.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.754+0000 7f941ffff640 1 -- 172.21.15.44:0/2453211313 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f943009be90 con 0x7f9438102240 2024-04-21T23:08:40.753 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-21T23:08:40.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.756+0000 7f943d537640 1 -- 172.21.15.44:0/2453211313 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f94200a2c10 msgr2=0x7f94200a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:40.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.756+0000 7f943d537640 1 --2- 172.21.15.44:0/2453211313 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f94200a2c10 0x7f94200a50d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9428000c00 tx=0x7f9428005fd0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.756+0000 7f943d537640 1 -- 172.21.15.44:0/2453211313 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9438102240 msgr2=0x7f943819ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:40.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.756+0000 7f943d537640 1 --2- 172.21.15.44:0/2453211313 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9438102240 0x7f943819ee40 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f943000eaa0 tx=0x7f943000ef60 comp rx=0 tx=0).stop 2024-04-21T23:08:40.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.756+0000 7f943d537640 1 -- 172.21.15.44:0/2453211313 shutdown_connections 2024-04-21T23:08:40.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.756+0000 7f943d537640 1 --2- 172.21.15.44:0/2453211313 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f94200a2c10 0x7f94200a50d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.756+0000 7f943d537640 1 --2- 172.21.15.44:0/2453211313 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9438101880 0x7f943819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.756+0000 7f943d537640 1 --2- 172.21.15.44:0/2453211313 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9438102240 0x7f943819ee40 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.756+0000 7f943d537640 1 --2- 172.21.15.44:0/2453211313 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9438100690 0x7f943806a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.756+0000 7f943d537640 1 -- 172.21.15.44:0/2453211313 >> 172.21.15.44:0/2453211313 conn(0x7f94380fbc20 msgr2=0x7f9438104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:40.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.756+0000 7f943d537640 1 -- 172.21.15.44:0/2453211313 shutdown_connections 2024-04-21T23:08:40.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.756+0000 7f943d537640 1 -- 172.21.15.44:0/2453211313 wait complete. 2024-04-21T23:08:40.766 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:08:40.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.876+0000 7fd7fe478640 1 -- 172.21.15.44:0/1409935467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f8103350 msgr2=0x7fd7f8109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:40.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.876+0000 7fd7fe478640 1 --2- 172.21.15.44:0/1409935467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f8103350 0x7fd7f8109be0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd7f0009ca0 tx=0x7fd7f002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.876+0000 7fd7fe478640 1 -- 172.21.15.44:0/1409935467 shutdown_connections 2024-04-21T23:08:40.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.876+0000 7fd7fe478640 1 --2- 172.21.15.44:0/1409935467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f8103350 0x7fd7f8109be0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.876+0000 7fd7fe478640 1 --2- 172.21.15.44:0/1409935467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7f8102990 0x7fd7f8102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.876+0000 7fd7fe478640 1 --2- 172.21.15.44:0/1409935467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f81017a0 0x7fd7f8101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.876+0000 7fd7fe478640 1 -- 172.21.15.44:0/1409935467 >> 172.21.15.44:0/1409935467 conn(0x7fd7f80fcd10 msgr2=0x7fd7f80ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:40.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.876+0000 7fd7fe478640 1 -- 172.21.15.44:0/1409935467 shutdown_connections 2024-04-21T23:08:40.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.876+0000 7fd7fe478640 1 -- 172.21.15.44:0/1409935467 wait complete. 2024-04-21T23:08:40.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.877+0000 7fd7fe478640 1 Processor -- start 2024-04-21T23:08:40.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.877+0000 7fd7fe478640 1 -- start start 2024-04-21T23:08:40.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.877+0000 7fd7fe478640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f81017a0 0x7fd7f8199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:40.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.877+0000 7fd7fe478640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7f8102990 0x7fd7f819a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:40.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.877+0000 7fd7fe478640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f8103350 0x7fd7f81a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:40.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.877+0000 7fd7fe478640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f810d510 con 0x7fd7f8103350 2024-04-21T23:08:40.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.877+0000 7fd7fe478640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f810d390 con 0x7fd7f8102990 2024-04-21T23:08:40.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.877+0000 7fd7fe478640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f810d690 con 0x7fd7f81017a0 2024-04-21T23:08:40.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.877+0000 7fd7fca15640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f8103350 0x7fd7f81a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:40.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.877+0000 7fd7fca15640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f8103350 0x7fd7f81a1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40778/0 (socket says 172.21.15.44:40778) 2024-04-21T23:08:40.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.877+0000 7fd7fca15640 1 -- 172.21.15.44:0/131284949 learned_addr learned my addr 172.21.15.44:0/131284949 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:40.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.877+0000 7fd7f7fff640 1 --2- 172.21.15.44:0/131284949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f81017a0 0x7fd7f8199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:40.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.877+0000 7fd7f77fe640 1 --2- 172.21.15.44:0/131284949 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7f8102990 0x7fd7f819a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:40.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.878+0000 7fd7fca15640 1 -- 172.21.15.44:0/131284949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f81017a0 msgr2=0x7fd7f8199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:40.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.878+0000 7fd7fca15640 1 --2- 172.21.15.44:0/131284949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f81017a0 0x7fd7f8199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.878+0000 7fd7fca15640 1 -- 172.21.15.44:0/131284949 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7f8102990 msgr2=0x7fd7f819a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:40.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.878+0000 7fd7fca15640 1 --2- 172.21.15.44:0/131284949 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7f8102990 0x7fd7f819a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:40.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.878+0000 7fd7fca15640 1 -- 172.21.15.44:0/131284949 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f81a1990 con 0x7fd7f8103350 2024-04-21T23:08:40.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.878+0000 7fd7f77fe640 1 --2- 172.21.15.44:0/131284949 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7f8102990 0x7fd7f819a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:08:40.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.878+0000 7fd7f7fff640 1 --2- 172.21.15.44:0/131284949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f81017a0 0x7fd7f8199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:08:40.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.878+0000 7fd7fca15640 1 --2- 172.21.15.44:0/131284949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f8103350 0x7fd7f81a1320 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd7f0037040 tx=0x7fd7f0032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:40.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.878+0000 7fd7f57fa640 1 -- 172.21.15.44:0/131284949 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f0039040 con 0x7fd7f8103350 2024-04-21T23:08:40.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.878+0000 7fd7fe478640 1 -- 172.21.15.44:0/131284949 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7f81a1c20 con 0x7fd7f8103350 2024-04-21T23:08:40.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.878+0000 7fd7f57fa640 1 -- 172.21.15.44:0/131284949 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7f0044070 con 0x7fd7f8103350 2024-04-21T23:08:40.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.878+0000 7fd7f57fa640 1 -- 172.21.15.44:0/131284949 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f0004330 con 0x7fd7f8103350 2024-04-21T23:08:40.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.878+0000 7fd7fe478640 1 -- 172.21.15.44:0/131284949 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7f81a2110 con 0x7fd7f8103350 2024-04-21T23:08:40.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.879+0000 7fd7fe478640 1 -- 172.21.15.44:0/131284949 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7f81a2340 con 0x7fd7f8103350 2024-04-21T23:08:40.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.880+0000 7fd7f57fa640 1 -- 172.21.15.44:0/131284949 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd7f0003a00 con 0x7fd7f8103350 2024-04-21T23:08:40.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.881+0000 7fd7f57fa640 1 --2- 172.21.15.44:0/131284949 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd7e00a2b90 0x7fd7e00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:40.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.881+0000 7fd7f7fff640 1 --2- 172.21.15.44:0/131284949 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd7e00a2b90 0x7fd7e00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:40.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.881+0000 7fd7f57fa640 1 -- 172.21.15.44:0/131284949 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7fd7f00f1d30 con 0x7fd7f8103350 2024-04-21T23:08:40.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.881+0000 7fd7f7fff640 1 --2- 172.21.15.44:0/131284949 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd7e00a2b90 0x7fd7e00a5050 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd7f81027f0 tx=0x7fd7ec008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:40.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:40.885+0000 7fd7f57fa640 1 -- 172.21.15.44:0/131284949 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd7f00be870 con 0x7fd7f8103350 2024-04-21T23:08:41.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.071+0000 7fd7fe478640 1 -- 172.21.15.44:0/131284949 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd7f81a2510 con 0x7fd7f8103350 2024-04-21T23:08:41.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.071+0000 7fd7f57fa640 1 -- 172.21.15.44:0/131284949 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fd7f00be170 con 0x7fd7f8103350 2024-04-21T23:08:41.070 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:08:41.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.073+0000 7fd7fe478640 1 -- 172.21.15.44:0/131284949 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd7e00a2b90 msgr2=0x7fd7e00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:41.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.073+0000 7fd7fe478640 1 --2- 172.21.15.44:0/131284949 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd7e00a2b90 0x7fd7e00a5050 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd7f81027f0 tx=0x7fd7ec008040 comp rx=0 tx=0).stop 2024-04-21T23:08:41.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.073+0000 7fd7fe478640 1 -- 172.21.15.44:0/131284949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f8103350 msgr2=0x7fd7f81a1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:41.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.073+0000 7fd7fe478640 1 --2- 172.21.15.44:0/131284949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f8103350 0x7fd7f81a1320 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd7f0037040 tx=0x7fd7f0032f90 comp rx=0 tx=0).stop 2024-04-21T23:08:41.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.074+0000 7fd7fe478640 1 -- 172.21.15.44:0/131284949 shutdown_connections 2024-04-21T23:08:41.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.074+0000 7fd7fe478640 1 --2- 172.21.15.44:0/131284949 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fd7e00a2b90 0x7fd7e00a5050 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.074+0000 7fd7fe478640 1 --2- 172.21.15.44:0/131284949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f8103350 0x7fd7f81a1320 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.074+0000 7fd7fe478640 1 --2- 172.21.15.44:0/131284949 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7f8102990 0x7fd7f819a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.074+0000 7fd7fe478640 1 --2- 172.21.15.44:0/131284949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f81017a0 0x7fd7f8199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.074+0000 7fd7fe478640 1 -- 172.21.15.44:0/131284949 >> 172.21.15.44:0/131284949 conn(0x7fd7f80fcd10 msgr2=0x7fd7f8073970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:41.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.074+0000 7fd7fe478640 1 -- 172.21.15.44:0/131284949 shutdown_connections 2024-04-21T23:08:41.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.074+0000 7fd7fe478640 1 -- 172.21.15.44:0/131284949 wait complete. 2024-04-21T23:08:41.088 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:08:41.195 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.194+0000 7f7707ef4640 1 -- 172.21.15.44:0/3228821435 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7700102230 msgr2=0x7f7700108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.194+0000 7f7707ef4640 1 --2- 172.21.15.44:0/3228821435 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7700102230 0x7f7700108ac0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f76f0009ca0 tx=0x7f76f002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.194+0000 7f7707ef4640 1 -- 172.21.15.44:0/3228821435 shutdown_connections 2024-04-21T23:08:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.194+0000 7f7707ef4640 1 --2- 172.21.15.44:0/3228821435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7700101870 0x7f7700101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.194+0000 7f7707ef4640 1 --2- 172.21.15.44:0/3228821435 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7700102230 0x7f7700108ac0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.195+0000 7f7707ef4640 1 --2- 172.21.15.44:0/3228821435 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7700100680 0x7f7700100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.195+0000 7f7707ef4640 1 -- 172.21.15.44:0/3228821435 >> 172.21.15.44:0/3228821435 conn(0x7f77000fbc10 msgr2=0x7f77000fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.195+0000 7f7707ef4640 1 -- 172.21.15.44:0/3228821435 shutdown_connections 2024-04-21T23:08:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.195+0000 7f7707ef4640 1 -- 172.21.15.44:0/3228821435 wait complete. 2024-04-21T23:08:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.195+0000 7f7707ef4640 1 Processor -- start 2024-04-21T23:08:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.195+0000 7f7707ef4640 1 -- start start 2024-04-21T23:08:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.195+0000 7f7707ef4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7700100680 0x7f7700197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.195+0000 7f7707ef4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7700101870 0x7f7700197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.195+0000 7f7707ef4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7700102230 0x7f770019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:41.196 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.195+0000 7f7707ef4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f770010c300 con 0x7f7700102230 2024-04-21T23:08:41.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.195+0000 7f7707ef4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f770010c180 con 0x7f7700100680 2024-04-21T23:08:41.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.195+0000 7f7707ef4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f770010c480 con 0x7f7700101870 2024-04-21T23:08:41.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.195+0000 7f7706491640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7700102230 0x7f770019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:41.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.195+0000 7f7706491640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7700102230 0x7f770019f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40802/0 (socket says 172.21.15.44:40802) 2024-04-21T23:08:41.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.195+0000 7f7706491640 1 -- 172.21.15.44:0/1366101692 learned_addr learned my addr 172.21.15.44:0/1366101692 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:41.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.196+0000 7f770548f640 1 --2- 172.21.15.44:0/1366101692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7700101870 0x7f7700197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:41.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.196+0000 7f7705c90640 1 --2- 172.21.15.44:0/1366101692 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7700100680 0x7f7700197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:41.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.196+0000 7f7706491640 1 -- 172.21.15.44:0/1366101692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7700101870 msgr2=0x7f7700197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:41.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.196+0000 7f7706491640 1 --2- 172.21.15.44:0/1366101692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7700101870 0x7f7700197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.196+0000 7f7706491640 1 -- 172.21.15.44:0/1366101692 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7700100680 msgr2=0x7f7700197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:41.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.196+0000 7f7706491640 1 --2- 172.21.15.44:0/1366101692 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7700100680 0x7f7700197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.196+0000 7f7706491640 1 -- 172.21.15.44:0/1366101692 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f770019f6b0 con 0x7f7700102230 2024-04-21T23:08:41.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.196+0000 7f7705c90640 1 --2- 172.21.15.44:0/1366101692 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7700100680 0x7f7700197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:08:41.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.196+0000 7f7706491640 1 --2- 172.21.15.44:0/1366101692 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7700102230 0x7f770019f040 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f76f0005e30 tx=0x7f76f0032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:41.197 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.196+0000 7f76feffd640 1 -- 172.21.15.44:0/1366101692 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f0041020 con 0x7f7700102230 2024-04-21T23:08:41.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.196+0000 7f76feffd640 1 -- 172.21.15.44:0/1366101692 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76f0034cd0 con 0x7f7700102230 2024-04-21T23:08:41.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.196+0000 7f7707ef4640 1 -- 172.21.15.44:0/1366101692 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f770019f940 con 0x7f7700102230 2024-04-21T23:08:41.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.196+0000 7f76feffd640 1 -- 172.21.15.44:0/1366101692 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f0039040 con 0x7f7700102230 2024-04-21T23:08:41.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.196+0000 7f7707ef4640 1 -- 172.21.15.44:0/1366101692 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f770019fe30 con 0x7f7700102230 2024-04-21T23:08:41.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.197+0000 7f7707ef4640 1 -- 172.21.15.44:0/1366101692 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77001a0000 con 0x7f7700102230 2024-04-21T23:08:41.198 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.198+0000 7f76feffd640 1 -- 172.21.15.44:0/1366101692 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f76f0044020 con 0x7f7700102230 2024-04-21T23:08:41.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.199+0000 7f76feffd640 1 --2- 172.21.15.44:0/1366101692 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f76e80a2b90 0x7f76e80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:41.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.199+0000 7f7705c90640 1 --2- 172.21.15.44:0/1366101692 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f76e80a2b90 0x7f76e80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:41.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.199+0000 7f76feffd640 1 -- 172.21.15.44:0/1366101692 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7f76f00f14f0 con 0x7f7700102230 2024-04-21T23:08:41.199 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.199+0000 7f7705c90640 1 --2- 172.21.15.44:0/1366101692 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f76e80a2b90 0x7f76e80a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f77001016d0 tx=0x7f76f8005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:41.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.203+0000 7f76feffd640 1 -- 172.21.15.44:0/1366101692 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f76f004b020 con 0x7f7700102230 2024-04-21T23:08:41.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.387+0000 7f7707ef4640 1 -- 172.21.15.44:0/1366101692 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f77001a0390 con 0x7f7700102230 2024-04-21T23:08:41.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.387+0000 7f76feffd640 1 -- 172.21.15.44:0/1366101692 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f76f00be090 con 0x7f7700102230 2024-04-21T23:08:41.386 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-21T23:08:41.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.389+0000 7f7707ef4640 1 -- 172.21.15.44:0/1366101692 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f76e80a2b90 msgr2=0x7f76e80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:41.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.389+0000 7f7707ef4640 1 --2- 172.21.15.44:0/1366101692 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f76e80a2b90 0x7f76e80a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f77001016d0 tx=0x7f76f8005e70 comp rx=0 tx=0).stop 2024-04-21T23:08:41.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.389+0000 7f7707ef4640 1 -- 172.21.15.44:0/1366101692 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7700102230 msgr2=0x7f770019f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:41.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.389+0000 7f7707ef4640 1 --2- 172.21.15.44:0/1366101692 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7700102230 0x7f770019f040 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f76f0005e30 tx=0x7f76f0032ec0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.389+0000 7f7707ef4640 1 -- 172.21.15.44:0/1366101692 shutdown_connections 2024-04-21T23:08:41.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.389+0000 7f7707ef4640 1 --2- 172.21.15.44:0/1366101692 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f76e80a2b90 0x7f76e80a5050 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.389+0000 7f7707ef4640 1 --2- 172.21.15.44:0/1366101692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7700101870 0x7f7700197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.389+0000 7f7707ef4640 1 --2- 172.21.15.44:0/1366101692 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7700102230 0x7f770019f040 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.389+0000 7f7707ef4640 1 --2- 172.21.15.44:0/1366101692 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7700100680 0x7f7700197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.389+0000 7f7707ef4640 1 -- 172.21.15.44:0/1366101692 >> 172.21.15.44:0/1366101692 conn(0x7f77000fbc10 msgr2=0x7f77000fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:41.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.389+0000 7f7707ef4640 1 -- 172.21.15.44:0/1366101692 shutdown_connections 2024-04-21T23:08:41.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.389+0000 7f7707ef4640 1 -- 172.21.15.44:0/1366101692 wait complete. 2024-04-21T23:08:41.398 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:41 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3941820431' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:41.398 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:41 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2453211313' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:41.398 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:41 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/131284949' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:41.399 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:08:41.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:41 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3941820431' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:41.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:41 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2453211313' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:41.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:41 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/131284949' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:41.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:41 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3941820431' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:41.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:41 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2453211313' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:41.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:41 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/131284949' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:41.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.506+0000 7fde232b5640 1 -- 172.21.15.44:0/2112400273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde1c107460 msgr2=0x7fde1c1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:41.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.506+0000 7fde232b5640 1 --2- 172.21.15.44:0/2112400273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde1c107460 0x7fde1c1098b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fde14009900 tx=0x7fde1402bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:41.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.506+0000 7fde232b5640 1 -- 172.21.15.44:0/2112400273 shutdown_connections 2024-04-21T23:08:41.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.506+0000 7fde232b5640 1 --2- 172.21.15.44:0/2112400273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde1c107460 0x7fde1c1098b0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.506+0000 7fde232b5640 1 --2- 172.21.15.44:0/2112400273 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde1c104b30 0x7fde1c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.506+0000 7fde232b5640 1 --2- 172.21.15.44:0/2112400273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde1c102200 0x7fde1c1045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.506+0000 7fde232b5640 1 -- 172.21.15.44:0/2112400273 >> 172.21.15.44:0/2112400273 conn(0x7fde1c0fbc10 msgr2=0x7fde1c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:41.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.506+0000 7fde232b5640 1 -- 172.21.15.44:0/2112400273 shutdown_connections 2024-04-21T23:08:41.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.506+0000 7fde232b5640 1 -- 172.21.15.44:0/2112400273 wait complete. 2024-04-21T23:08:41.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.506+0000 7fde232b5640 1 Processor -- start 2024-04-21T23:08:41.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.506+0000 7fde232b5640 1 -- start start 2024-04-21T23:08:41.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.506+0000 7fde232b5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde1c102200 0x7fde1c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:41.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.506+0000 7fde232b5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde1c104b30 0x7fde1c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:41.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.506+0000 7fde232b5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde1c107460 0x7fde1c19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:41.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.506+0000 7fde232b5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fde1c10d420 con 0x7fde1c107460 2024-04-21T23:08:41.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.506+0000 7fde232b5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fde1c10d2a0 con 0x7fde1c102200 2024-04-21T23:08:41.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.506+0000 7fde232b5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fde1c10d5a0 con 0x7fde1c104b30 2024-04-21T23:08:41.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.507+0000 7fde21852640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde1c107460 0x7fde1c19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:41.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.507+0000 7fde21852640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde1c107460 0x7fde1c19f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40820/0 (socket says 172.21.15.44:40820) 2024-04-21T23:08:41.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.507+0000 7fde21852640 1 -- 172.21.15.44:0/3330664078 learned_addr learned my addr 172.21.15.44:0/3330664078 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:41.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.507+0000 7fde21051640 1 --2- 172.21.15.44:0/3330664078 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde1c102200 0x7fde1c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:41.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.507+0000 7fde20850640 1 --2- 172.21.15.44:0/3330664078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde1c104b30 0x7fde1c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:41.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.507+0000 7fde21852640 1 -- 172.21.15.44:0/3330664078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde1c104b30 msgr2=0x7fde1c197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:41.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.507+0000 7fde21852640 1 --2- 172.21.15.44:0/3330664078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde1c104b30 0x7fde1c197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.507+0000 7fde21852640 1 -- 172.21.15.44:0/3330664078 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde1c102200 msgr2=0x7fde1c197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:41.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.507+0000 7fde21852640 1 --2- 172.21.15.44:0/3330664078 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde1c102200 0x7fde1c197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.507+0000 7fde21852640 1 -- 172.21.15.44:0/3330664078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1c19f6c0 con 0x7fde1c107460 2024-04-21T23:08:41.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.507+0000 7fde20850640 1 --2- 172.21.15.44:0/3330664078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde1c104b30 0x7fde1c197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:08:41.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.507+0000 7fde21852640 1 --2- 172.21.15.44:0/3330664078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde1c107460 0x7fde1c19f050 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fde14005ab0 tx=0x7fde14004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:41.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.507+0000 7fde1a7fc640 1 -- 172.21.15.44:0/3330664078 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde14041070 con 0x7fde1c107460 2024-04-21T23:08:41.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.507+0000 7fde1a7fc640 1 -- 172.21.15.44:0/3330664078 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde140054e0 con 0x7fde1c107460 2024-04-21T23:08:41.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.507+0000 7fde232b5640 1 -- 172.21.15.44:0/3330664078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde1c19f950 con 0x7fde1c107460 2024-04-21T23:08:41.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.507+0000 7fde1a7fc640 1 -- 172.21.15.44:0/3330664078 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde140346a0 con 0x7fde1c107460 2024-04-21T23:08:41.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.507+0000 7fde232b5640 1 -- 172.21.15.44:0/3330664078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde1c19fe40 con 0x7fde1c107460 2024-04-21T23:08:41.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.508+0000 7fde232b5640 1 -- 172.21.15.44:0/3330664078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde1c0fd700 con 0x7fde1c107460 2024-04-21T23:08:41.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.510+0000 7fde1a7fc640 1 -- 172.21.15.44:0/3330664078 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fde14044070 con 0x7fde1c107460 2024-04-21T23:08:41.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.510+0000 7fde1a7fc640 1 --2- 172.21.15.44:0/3330664078 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fde040a2b90 0x7fde040a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:41.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.510+0000 7fde21051640 1 --2- 172.21.15.44:0/3330664078 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fde040a2b90 0x7fde040a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:41.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.510+0000 7fde1a7fc640 1 -- 172.21.15.44:0/3330664078 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7fde140f1790 con 0x7fde1c107460 2024-04-21T23:08:41.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.511+0000 7fde21051640 1 --2- 172.21.15.44:0/3330664078 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fde040a2b90 0x7fde040a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fde0c005c40 tx=0x7fde0c00a570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:41.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.514+0000 7fde1a7fc640 1 -- 172.21.15.44:0/3330664078 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fde140be330 con 0x7fde1c107460 2024-04-21T23:08:41.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.701+0000 7fde232b5640 1 -- 172.21.15.44:0/3330664078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fde1c02d170 con 0x7fde1c107460 2024-04-21T23:08:41.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.701+0000 7fde1a7fc640 1 -- 172.21.15.44:0/3330664078 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fde140bdc30 con 0x7fde1c107460 2024-04-21T23:08:41.700 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:08:41.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.703+0000 7fde232b5640 1 -- 172.21.15.44:0/3330664078 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fde040a2b90 msgr2=0x7fde040a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:41.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.703+0000 7fde232b5640 1 --2- 172.21.15.44:0/3330664078 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fde040a2b90 0x7fde040a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fde0c005c40 tx=0x7fde0c00a570 comp rx=0 tx=0).stop 2024-04-21T23:08:41.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.703+0000 7fde232b5640 1 -- 172.21.15.44:0/3330664078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde1c107460 msgr2=0x7fde1c19f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:41.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.703+0000 7fde232b5640 1 --2- 172.21.15.44:0/3330664078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde1c107460 0x7fde1c19f050 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fde14005ab0 tx=0x7fde14004270 comp rx=0 tx=0).stop 2024-04-21T23:08:41.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.703+0000 7fde232b5640 1 -- 172.21.15.44:0/3330664078 shutdown_connections 2024-04-21T23:08:41.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.703+0000 7fde232b5640 1 --2- 172.21.15.44:0/3330664078 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fde040a2b90 0x7fde040a5050 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.703+0000 7fde232b5640 1 --2- 172.21.15.44:0/3330664078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde1c104b30 0x7fde1c197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.703+0000 7fde232b5640 1 --2- 172.21.15.44:0/3330664078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde1c107460 0x7fde1c19f050 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.703+0000 7fde232b5640 1 --2- 172.21.15.44:0/3330664078 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde1c102200 0x7fde1c197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.703+0000 7fde232b5640 1 -- 172.21.15.44:0/3330664078 >> 172.21.15.44:0/3330664078 conn(0x7fde1c0fbc10 msgr2=0x7fde1c102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:41.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.703+0000 7fde232b5640 1 -- 172.21.15.44:0/3330664078 shutdown_connections 2024-04-21T23:08:41.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.703+0000 7fde232b5640 1 -- 172.21.15.44:0/3330664078 wait complete. 2024-04-21T23:08:41.713 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:08:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.816+0000 7f5d3f312640 1 -- 172.21.15.44:0/568280023 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d38102250 msgr2=0x7f5d38108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.816+0000 7f5d3f312640 1 --2- 172.21.15.44:0/568280023 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d38102250 0x7f5d38108ae0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5d30009ca0 tx=0x7f5d3002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.816+0000 7f5d3f312640 1 -- 172.21.15.44:0/568280023 shutdown_connections 2024-04-21T23:08:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.816+0000 7f5d3f312640 1 --2- 172.21.15.44:0/568280023 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d38102250 0x7f5d38108ae0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.816+0000 7f5d3f312640 1 --2- 172.21.15.44:0/568280023 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d38101890 0x7f5d38101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.816+0000 7f5d3f312640 1 --2- 172.21.15.44:0/568280023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d381006a0 0x7f5d38100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.816+0000 7f5d3f312640 1 -- 172.21.15.44:0/568280023 >> 172.21.15.44:0/568280023 conn(0x7f5d380fbc50 msgr2=0x7f5d380fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.816+0000 7f5d3f312640 1 -- 172.21.15.44:0/568280023 shutdown_connections 2024-04-21T23:08:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.816+0000 7f5d3f312640 1 -- 172.21.15.44:0/568280023 wait complete. 2024-04-21T23:08:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.817+0000 7f5d3f312640 1 Processor -- start 2024-04-21T23:08:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.817+0000 7f5d3f312640 1 -- start start 2024-04-21T23:08:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.817+0000 7f5d3f312640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d381006a0 0x7f5d38197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.817+0000 7f5d3f312640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d38101890 0x7f5d38197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.817+0000 7f5d3f312640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d38102250 0x7f5d3819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.817+0000 7f5d3f312640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3810c320 con 0x7f5d38102250 2024-04-21T23:08:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.817+0000 7f5d3f312640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3810c1a0 con 0x7f5d381006a0 2024-04-21T23:08:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.817+0000 7f5d3f312640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3810c4a0 con 0x7f5d38101890 2024-04-21T23:08:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.817+0000 7f5d3d8af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d38102250 0x7f5d3819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.817+0000 7f5d3d8af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d38102250 0x7f5d3819efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40844/0 (socket says 172.21.15.44:40844) 2024-04-21T23:08:41.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.817+0000 7f5d3d8af640 1 -- 172.21.15.44:0/2834383509 learned_addr learned my addr 172.21.15.44:0/2834383509 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.817+0000 7f5d3d0ae640 1 --2- 172.21.15.44:0/2834383509 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d381006a0 0x7f5d38197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.817+0000 7f5d3c8ad640 1 --2- 172.21.15.44:0/2834383509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d38101890 0x7f5d38197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.817+0000 7f5d3d8af640 1 -- 172.21.15.44:0/2834383509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d38101890 msgr2=0x7f5d38197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.817+0000 7f5d3d8af640 1 --2- 172.21.15.44:0/2834383509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d38101890 0x7f5d38197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.818+0000 7f5d3d8af640 1 -- 172.21.15.44:0/2834383509 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d381006a0 msgr2=0x7f5d38197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.818+0000 7f5d3d8af640 1 --2- 172.21.15.44:0/2834383509 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d381006a0 0x7f5d38197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.818+0000 7f5d3d8af640 1 -- 172.21.15.44:0/2834383509 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d3819f620 con 0x7f5d38102250 2024-04-21T23:08:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.818+0000 7f5d3d8af640 1 --2- 172.21.15.44:0/2834383509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d38102250 0x7f5d3819efb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5d3002c630 tx=0x7f5d30032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.818+0000 7f5d2e7fc640 1 -- 172.21.15.44:0/2834383509 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d30041020 con 0x7f5d38102250 2024-04-21T23:08:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.818+0000 7f5d2e7fc640 1 -- 172.21.15.44:0/2834383509 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d30034cd0 con 0x7f5d38102250 2024-04-21T23:08:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.818+0000 7f5d3f312640 1 -- 172.21.15.44:0/2834383509 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d3819f8b0 con 0x7f5d38102250 2024-04-21T23:08:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.818+0000 7f5d2e7fc640 1 -- 172.21.15.44:0/2834383509 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d30039040 con 0x7f5d38102250 2024-04-21T23:08:41.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.818+0000 7f5d3f312640 1 -- 172.21.15.44:0/2834383509 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d3819fda0 con 0x7f5d38102250 2024-04-21T23:08:41.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.819+0000 7f5d3f312640 1 -- 172.21.15.44:0/2834383509 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d3819ffa0 con 0x7f5d38102250 2024-04-21T23:08:41.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.821+0000 7f5d2e7fc640 1 -- 172.21.15.44:0/2834383509 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5d30044020 con 0x7f5d38102250 2024-04-21T23:08:41.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.821+0000 7f5d2e7fc640 1 --2- 172.21.15.44:0/2834383509 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5d240a2b90 0x7f5d240a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:41.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.821+0000 7f5d3d0ae640 1 --2- 172.21.15.44:0/2834383509 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5d240a2b90 0x7f5d240a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:41.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.822+0000 7f5d2e7fc640 1 -- 172.21.15.44:0/2834383509 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7f5d300f1620 con 0x7f5d38102250 2024-04-21T23:08:41.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.822+0000 7f5d3d0ae640 1 --2- 172.21.15.44:0/2834383509 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5d240a2b90 0x7f5d240a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5d381016f0 tx=0x7f5d34005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:41.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:41.825+0000 7f5d2e7fc640 1 -- 172.21.15.44:0/2834383509 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f5d3004b020 con 0x7f5d38102250 2024-04-21T23:08:42.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.055+0000 7f5d3f312640 1 -- 172.21.15.44:0/2834383509 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d381a0330 con 0x7f5d38102250 2024-04-21T23:08:42.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.056+0000 7f5d2e7fc640 1 -- 172.21.15.44:0/2834383509 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11791 (secure 0 0 0) 0x7f5d300be250 con 0x7f5d38102250 2024-04-21T23:08:42.055 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:42.055 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:06:50.685663+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3743898853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3743898853},{"type":"v1","addr":"172.21.15.69:6839","nonce":3743898853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:08:42.055 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2024-04-21T23:08:42.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.058+0000 7f5d3f312640 1 -- 172.21.15.44:0/2834383509 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5d240a2b90 msgr2=0x7f5d240a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.058+0000 7f5d3f312640 1 --2- 172.21.15.44:0/2834383509 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5d240a2b90 0x7f5d240a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5d381016f0 tx=0x7f5d34005f90 comp rx=0 tx=0).stop 2024-04-21T23:08:42.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.058+0000 7f5d3f312640 1 -- 172.21.15.44:0/2834383509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d38102250 msgr2=0x7f5d3819efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.058+0000 7f5d3f312640 1 --2- 172.21.15.44:0/2834383509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d38102250 0x7f5d3819efb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5d3002c630 tx=0x7f5d30032ea0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.058+0000 7f5d3f312640 1 -- 172.21.15.44:0/2834383509 shutdown_connections 2024-04-21T23:08:42.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.058+0000 7f5d3f312640 1 --2- 172.21.15.44:0/2834383509 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f5d240a2b90 0x7f5d240a5050 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.058+0000 7f5d3f312640 1 --2- 172.21.15.44:0/2834383509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d38101890 0x7f5d38197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.058+0000 7f5d3f312640 1 --2- 172.21.15.44:0/2834383509 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5d38102250 0x7f5d3819efb0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.058+0000 7f5d3f312640 1 --2- 172.21.15.44:0/2834383509 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5d381006a0 0x7f5d38197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.058+0000 7f5d3f312640 1 -- 172.21.15.44:0/2834383509 >> 172.21.15.44:0/2834383509 conn(0x7f5d380fbc50 msgr2=0x7f5d380fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:42.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.058+0000 7f5d3f312640 1 -- 172.21.15.44:0/2834383509 shutdown_connections 2024-04-21T23:08:42.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.058+0000 7f5d3f312640 1 -- 172.21.15.44:0/2834383509 wait complete. 2024-04-21T23:08:42.068 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:08:42.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.174+0000 7f544f5f7640 1 -- 172.21.15.44:0/141127011 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5448102280 msgr2=0x7f5448104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.174+0000 7f544f5f7640 1 --2- 172.21.15.44:0/141127011 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5448102280 0x7f5448104670 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f543c009900 tx=0x7f543c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:42.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.174+0000 7f544f5f7640 1 -- 172.21.15.44:0/141127011 shutdown_connections 2024-04-21T23:08:42.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.174+0000 7f544f5f7640 1 --2- 172.21.15.44:0/141127011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5448104bb0 0x7f5448106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.174+0000 7f544f5f7640 1 --2- 172.21.15.44:0/141127011 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f54481074e0 0x7f54481098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.174+0000 7f544f5f7640 1 --2- 172.21.15.44:0/141127011 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5448102280 0x7f5448104670 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.174+0000 7f544f5f7640 1 -- 172.21.15.44:0/141127011 >> 172.21.15.44:0/141127011 conn(0x7f54480fbc10 msgr2=0x7f54480fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:42.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.174+0000 7f544f5f7640 1 -- 172.21.15.44:0/141127011 shutdown_connections 2024-04-21T23:08:42.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.174+0000 7f544f5f7640 1 -- 172.21.15.44:0/141127011 wait complete. 2024-04-21T23:08:42.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.175+0000 7f544f5f7640 1 Processor -- start 2024-04-21T23:08:42.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.175+0000 7f544f5f7640 1 -- start start 2024-04-21T23:08:42.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.175+0000 7f544f5f7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5448102280 0x7f5448199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.175+0000 7f544f5f7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5448104bb0 0x7f544819a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.175+0000 7f544f5f7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54481074e0 0x7f54481a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.175+0000 7f544f5f7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f544810d4c0 con 0x7f54481074e0 2024-04-21T23:08:42.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.175+0000 7f544f5f7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f544810d340 con 0x7f5448104bb0 2024-04-21T23:08:42.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.175+0000 7f544f5f7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f544810d640 con 0x7f5448102280 2024-04-21T23:08:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.175+0000 7f544db94640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54481074e0 0x7f54481a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.175+0000 7f544db94640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54481074e0 0x7f54481a1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40852/0 (socket says 172.21.15.44:40852) 2024-04-21T23:08:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.175+0000 7f544db94640 1 -- 172.21.15.44:0/3901181284 learned_addr learned my addr 172.21.15.44:0/3901181284 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.175+0000 7f544d393640 1 --2- 172.21.15.44:0/3901181284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5448102280 0x7f5448199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.175+0000 7f544db94640 1 -- 172.21.15.44:0/3901181284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5448102280 msgr2=0x7f5448199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.175+0000 7f544db94640 1 --2- 172.21.15.44:0/3901181284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5448102280 0x7f5448199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.175+0000 7f544db94640 1 -- 172.21.15.44:0/3901181284 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5448104bb0 msgr2=0x7f544819a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.175+0000 7f544cb92640 1 --2- 172.21.15.44:0/3901181284 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5448104bb0 0x7f544819a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.175+0000 7f544db94640 1 --2- 172.21.15.44:0/3901181284 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5448104bb0 0x7f544819a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.175+0000 7f544db94640 1 -- 172.21.15.44:0/3901181284 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54481a1990 con 0x7f54481074e0 2024-04-21T23:08:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.176+0000 7f544db94640 1 --2- 172.21.15.44:0/3901181284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54481074e0 0x7f54481a1320 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f544000b970 tx=0x7f544000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.176+0000 7f543a7fc640 1 -- 172.21.15.44:0/3901181284 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5440019070 con 0x7f54481074e0 2024-04-21T23:08:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.176+0000 7f543a7fc640 1 -- 172.21.15.44:0/3901181284 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f544000cb40 con 0x7f54481074e0 2024-04-21T23:08:42.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.176+0000 7f544f5f7640 1 -- 172.21.15.44:0/3901181284 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54481a1c80 con 0x7f54481074e0 2024-04-21T23:08:42.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.176+0000 7f543a7fc640 1 -- 172.21.15.44:0/3901181284 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5440007650 con 0x7f54481074e0 2024-04-21T23:08:42.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.176+0000 7f544f5f7640 1 -- 172.21.15.44:0/3901181284 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54481a2180 con 0x7f54481074e0 2024-04-21T23:08:42.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.177+0000 7f544f5f7640 1 -- 172.21.15.44:0/3901181284 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5448108fd0 con 0x7f54481074e0 2024-04-21T23:08:42.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.178+0000 7f543a7fc640 1 -- 172.21.15.44:0/3901181284 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f544001c070 con 0x7f54481074e0 2024-04-21T23:08:42.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.179+0000 7f543a7fc640 1 --2- 172.21.15.44:0/3901181284 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f54340a2b90 0x7f54340a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.179+0000 7f543a7fc640 1 -- 172.21.15.44:0/3901181284 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7f54400cef10 con 0x7f54481074e0 2024-04-21T23:08:42.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.179+0000 7f544d393640 1 --2- 172.21.15.44:0/3901181284 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f54340a2b90 0x7f54340a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.179+0000 7f544d393640 1 --2- 172.21.15.44:0/3901181284 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f54340a2b90 0x7f54340a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f543c006010 tx=0x7f543c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:42.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.182+0000 7f543a7fc640 1 -- 172.21.15.44:0/3901181284 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f544009bab0 con 0x7f54481074e0 2024-04-21T23:08:42.347 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:42 smithi044 ceph-mon[27934]: pgmap v363: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:08:42.347 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:42 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1366101692' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:42.347 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:42 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3330664078' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:42.347 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:42 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2834383509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:42.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.369+0000 7f544f5f7640 1 -- 172.21.15.44:0/3901181284 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f544802d370 con 0x7f54481074e0 2024-04-21T23:08:42.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.370+0000 7f543a7fc640 1 -- 172.21.15.44:0/3901181284 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24716 (secure 0 0 0) 0x7f544009b3b0 con 0x7f54481074e0 2024-04-21T23:08:42.369 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:42.369 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":94,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:08:25.093542+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":9,"to":11},{"from":5,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:08:42.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.372+0000 7f544f5f7640 1 -- 172.21.15.44:0/3901181284 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f54340a2b90 msgr2=0x7f54340a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.372+0000 7f544f5f7640 1 --2- 172.21.15.44:0/3901181284 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f54340a2b90 0x7f54340a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f543c006010 tx=0x7f543c000bc0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.372+0000 7f544f5f7640 1 -- 172.21.15.44:0/3901181284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54481074e0 msgr2=0x7f54481a1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.372+0000 7f544f5f7640 1 --2- 172.21.15.44:0/3901181284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54481074e0 0x7f54481a1320 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f544000b970 tx=0x7f544000be30 comp rx=0 tx=0).stop 2024-04-21T23:08:42.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.372+0000 7f544f5f7640 1 -- 172.21.15.44:0/3901181284 shutdown_connections 2024-04-21T23:08:42.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.372+0000 7f544f5f7640 1 --2- 172.21.15.44:0/3901181284 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f54340a2b90 0x7f54340a5050 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.372+0000 7f544f5f7640 1 --2- 172.21.15.44:0/3901181284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f54481074e0 0x7f54481a1320 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.372+0000 7f544f5f7640 1 --2- 172.21.15.44:0/3901181284 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5448104bb0 0x7f544819a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.372+0000 7f544f5f7640 1 --2- 172.21.15.44:0/3901181284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5448102280 0x7f5448199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.372+0000 7f544f5f7640 1 -- 172.21.15.44:0/3901181284 >> 172.21.15.44:0/3901181284 conn(0x7f54480fbc10 msgr2=0x7f54481019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:42.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.372+0000 7f544f5f7640 1 -- 172.21.15.44:0/3901181284 shutdown_connections 2024-04-21T23:08:42.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.372+0000 7f544f5f7640 1 -- 172.21.15.44:0/3901181284 wait complete. 2024-04-21T23:08:42.383 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-21T23:06:25.562805+0000', 'modified': '2024-04-21T23:06:50.685663+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24530}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24526': {'gid': 24526, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.69:6839/3743898853', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6838', 'nonce': 3743898853}, {'type': 'v1', 'addr': '172.21.15.69:6839', 'nonce': 3743898853}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24530': {'gid': 24530, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.55:6839/2066817543', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.55:6838', 'nonce': 2066817543}, {'type': 'v1', 'addr': '172.21.15.55:6839', 'nonce': 2066817543}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24530, 'qdb_cluster': [24530]} 2024-04-21T23:08:42.383 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-21T23:08:42.384 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T23:08:42.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:42 smithi069 ceph-mon[31816]: pgmap v363: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:08:42.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:42 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1366101692' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:42.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:42 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3330664078' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:42.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:42 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2834383509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:42.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:42 smithi055 ceph-mon[31117]: pgmap v363: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:08:42.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:42 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1366101692' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:42.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:42 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3330664078' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:42.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:42 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2834383509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:42.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.493+0000 7f508ae2d640 1 -- 172.21.15.44:0/910665752 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5084102280 msgr2=0x7f5084104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.493+0000 7f508ae2d640 1 --2- 172.21.15.44:0/910665752 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5084102280 0x7f5084104670 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5078009900 tx=0x7f507802bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:42.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.493+0000 7f508ae2d640 1 -- 172.21.15.44:0/910665752 shutdown_connections 2024-04-21T23:08:42.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.493+0000 7f508ae2d640 1 --2- 172.21.15.44:0/910665752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50841074e0 0x7f50841098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.493+0000 7f508ae2d640 1 --2- 172.21.15.44:0/910665752 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5084104bb0 0x7f5084106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.493+0000 7f508ae2d640 1 --2- 172.21.15.44:0/910665752 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5084102280 0x7f5084104670 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.493+0000 7f508ae2d640 1 -- 172.21.15.44:0/910665752 >> 172.21.15.44:0/910665752 conn(0x7f50840fbc10 msgr2=0x7f50840fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:42.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.493+0000 7f508ae2d640 1 -- 172.21.15.44:0/910665752 shutdown_connections 2024-04-21T23:08:42.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.493+0000 7f508ae2d640 1 -- 172.21.15.44:0/910665752 wait complete. 2024-04-21T23:08:42.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.493+0000 7f508ae2d640 1 Processor -- start 2024-04-21T23:08:42.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.494+0000 7f508ae2d640 1 -- start start 2024-04-21T23:08:42.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.494+0000 7f508ae2d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5084102280 0x7f5084197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.494+0000 7f508ae2d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5084104bb0 0x7f5084197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.494+0000 7f508ae2d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50841074e0 0x7f508419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.494+0000 7f508ae2d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f508410d420 con 0x7f5084102280 2024-04-21T23:08:42.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.494+0000 7f508ae2d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f508410d2a0 con 0x7f50841074e0 2024-04-21T23:08:42.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.494+0000 7f508ae2d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f508410d5a0 con 0x7f5084104bb0 2024-04-21T23:08:42.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.494+0000 7f5088bc9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5084102280 0x7f5084197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.494+0000 7f5088bc9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5084102280 0x7f5084197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40858/0 (socket says 172.21.15.44:40858) 2024-04-21T23:08:42.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.494+0000 7f5088bc9640 1 -- 172.21.15.44:0/2569113937 learned_addr learned my addr 172.21.15.44:0/2569113937 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:42.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.494+0000 7f5083fff640 1 --2- 172.21.15.44:0/2569113937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5084104bb0 0x7f5084197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.494+0000 7f50893ca640 1 --2- 172.21.15.44:0/2569113937 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50841074e0 0x7f508419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.494+0000 7f5088bc9640 1 -- 172.21.15.44:0/2569113937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5084104bb0 msgr2=0x7f5084197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.495 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.494+0000 7f5088bc9640 1 --2- 172.21.15.44:0/2569113937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5084104bb0 0x7f5084197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.494+0000 7f5088bc9640 1 -- 172.21.15.44:0/2569113937 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50841074e0 msgr2=0x7f508419f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.494+0000 7f5088bc9640 1 --2- 172.21.15.44:0/2569113937 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50841074e0 0x7f508419f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.494+0000 7f5088bc9640 1 -- 172.21.15.44:0/2569113937 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f508419f680 con 0x7f5084102280 2024-04-21T23:08:42.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.494+0000 7f5088bc9640 1 --2- 172.21.15.44:0/2569113937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5084102280 0x7f5084197a60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5078009230 tx=0x7f5078005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:42.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.494+0000 7f5081ffb640 1 -- 172.21.15.44:0/2569113937 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5078041070 con 0x7f5084102280 2024-04-21T23:08:42.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.495+0000 7f508ae2d640 1 -- 172.21.15.44:0/2569113937 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f508419f910 con 0x7f5084102280 2024-04-21T23:08:42.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.495+0000 7f5081ffb640 1 -- 172.21.15.44:0/2569113937 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50780054e0 con 0x7f5084102280 2024-04-21T23:08:42.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.495+0000 7f5081ffb640 1 -- 172.21.15.44:0/2569113937 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5078034440 con 0x7f5084102280 2024-04-21T23:08:42.496 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.495+0000 7f508ae2d640 1 -- 172.21.15.44:0/2569113937 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f508419fe50 con 0x7f5084102280 2024-04-21T23:08:42.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.497+0000 7f5081ffb640 1 -- 172.21.15.44:0/2569113937 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5078044070 con 0x7f5084102280 2024-04-21T23:08:42.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.497+0000 7f508da0c740 1 Processor -- start 2024-04-21T23:08:42.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.498+0000 7f508da0c740 1 -- start start 2024-04-21T23:08:42.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.498+0000 7f508da0c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56273ecfd070 0x56273ecfd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.498+0000 7f508da0c740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ecfda60 0x56273ed02560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.498+0000 7f508da0c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 0x56273ed05060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.498+0000 7f508da0c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56273ec61df0 con 0x56273ed02bd0 2024-04-21T23:08:42.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.498+0000 7f508da0c740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x56273ec0a640 con 0x56273ecfda60 2024-04-21T23:08:42.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.498+0000 7f508da0c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56273ec5ddb0 con 0x56273ecfd070 2024-04-21T23:08:42.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.498+0000 7f5081ffb640 1 --2- 172.21.15.44:0/2569113937 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f50740a2b90 0x7f50740a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.498+0000 7f5083fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 0x56273ed05060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.498+0000 7f5083fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 0x56273ed05060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40862/0 (socket says 172.21.15.44:40862) 2024-04-21T23:08:42.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.498+0000 7f5083fff640 1 -- 172.21.15.44:0/1833496438 learned_addr learned my addr 172.21.15.44:0/1833496438 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:42.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.498+0000 7f50893ca640 1 --2- 172.21.15.44:0/1833496438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56273ecfd070 0x56273ecfd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.498+0000 7f5088bc9640 1 --2- 172.21.15.44:0/1833496438 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ecfda60 0x56273ed02560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.498+0000 7f50893ca640 1 --2- 172.21.15.44:0/2569113937 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f50740a2b90 0x7f50740a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.498+0000 7f5081ffb640 1 -- 172.21.15.44:0/2569113937 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7f50780f1220 con 0x7f5084102280 2024-04-21T23:08:42.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.498+0000 7f5083fff640 1 -- 172.21.15.44:0/1833496438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56273ecfd070 msgr2=0x56273ecfd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:42.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.498+0000 7f5083fff640 1 --2- 172.21.15.44:0/1833496438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56273ecfd070 0x56273ecfd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.498+0000 7f5083fff640 1 -- 172.21.15.44:0/1833496438 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ecfda60 msgr2=0x56273ed02560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:42.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.498+0000 7f5083fff640 1 --2- 172.21.15.44:0/1833496438 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ecfda60 0x56273ed02560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.498+0000 7f5083fff640 1 -- 172.21.15.44:0/1833496438 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56273e9762d0 con 0x56273ed02bd0 2024-04-21T23:08:42.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.499+0000 7f50893ca640 1 --2- 172.21.15.44:0/2569113937 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f50740a2b90 0x7f50740a5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f507c009fd0 tx=0x7f507c009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:42.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.499+0000 7f5083fff640 1 --2- 172.21.15.44:0/1833496438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 0x56273ed05060 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f507000cc00 tx=0x7f507000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d84fdc4ae4d37d0d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:42.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.499+0000 7f506effd640 1 -- 172.21.15.44:0/1833496438 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5070005220 con 0x56273ed02bd0 2024-04-21T23:08:42.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.499+0000 7f506effd640 1 -- 172.21.15.44:0/1833496438 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5070004c80 con 0x56273ed02bd0 2024-04-21T23:08:42.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.499+0000 7f506effd640 1 -- 172.21.15.44:0/1833496438 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50700107d0 con 0x56273ed02bd0 2024-04-21T23:08:42.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.500+0000 7f508da0c740 1 -- 172.21.15.44:0/1833496438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 msgr2=0x56273ed05060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.500+0000 7f508da0c740 1 --2- 172.21.15.44:0/1833496438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 0x56273ed05060 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f507000cc00 tx=0x7f507000e720 comp rx=0 tx=0).stop 2024-04-21T23:08:42.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.500+0000 7f508da0c740 1 -- 172.21.15.44:0/1833496438 shutdown_connections 2024-04-21T23:08:42.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.500+0000 7f508da0c740 1 --2- 172.21.15.44:0/1833496438 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 0x56273ed05060 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.500+0000 7f508da0c740 1 --2- 172.21.15.44:0/1833496438 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ecfda60 0x56273ed02560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.500+0000 7f508da0c740 1 --2- 172.21.15.44:0/1833496438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56273ecfd070 0x56273ecfd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.500+0000 7f508da0c740 1 -- 172.21.15.44:0/1833496438 >> 172.21.15.44:0/1833496438 conn(0x56273ecf85e0 msgr2=0x56273ecf89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:42.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.500+0000 7f508da0c740 1 -- 172.21.15.44:0/1833496438 shutdown_connections 2024-04-21T23:08:42.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.500+0000 7f508da0c740 1 -- 172.21.15.44:0/1833496438 wait complete. 2024-04-21T23:08:42.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.500+0000 7f508da0c740 1 -- start start 2024-04-21T23:08:42.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.500+0000 7f508da0c740 1 Processor -- start 2024-04-21T23:08:42.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.500+0000 7f508da0c740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ecfd070 0x56273ed07550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.500+0000 7f508da0c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56273ecfda60 0x56273ed07a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.500+0000 7f508da0c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 0x56273ed0ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.500+0000 7f508da0c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56273ed05b80 con 0x56273ed02bd0 2024-04-21T23:08:42.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.501+0000 7f508da0c740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x56273ec1dbe0 con 0x56273ecfd070 2024-04-21T23:08:42.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.501+0000 7f508da0c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56273ed05d00 con 0x56273ecfda60 2024-04-21T23:08:42.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.501+0000 7f508da0c740 10 client.0 _ll_register_callbacks cb 0x56273ec1d5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:08:42.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.501+0000 7f50893ca640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 0x56273ed0ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.501+0000 7f50893ca640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 0x56273ed0ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40878/0 (socket says 172.21.15.44:40878) 2024-04-21T23:08:42.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.501+0000 7f50893ca640 1 -- 172.21.15.44:0/1098695152 learned_addr learned my addr 172.21.15.44:0/1098695152 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:42.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.501+0000 7f5088bc9640 1 --2- 172.21.15.44:0/1098695152 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ecfd070 0x56273ed07550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.501+0000 7f5083fff640 1 --2- 172.21.15.44:0/1098695152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56273ecfda60 0x56273ed07a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.501+0000 7f50893ca640 1 -- 172.21.15.44:0/1098695152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56273ecfda60 msgr2=0x56273ed07a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:42.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.501+0000 7f50893ca640 1 --2- 172.21.15.44:0/1098695152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56273ecfda60 0x56273ed07a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.501+0000 7f50893ca640 1 -- 172.21.15.44:0/1098695152 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ecfd070 msgr2=0x56273ed07550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:42.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.501+0000 7f50893ca640 1 --2- 172.21.15.44:0/1098695152 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ecfd070 0x56273ed07550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.501+0000 7f50893ca640 1 -- 172.21.15.44:0/1098695152 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56273ea406b0 con 0x56273ed02bd0 2024-04-21T23:08:42.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.501+0000 7f50893ca640 1 --2- 172.21.15.44:0/1098695152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 0x56273ed0ead0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f507c00fd10 tx=0x7f507c01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef81534c696b8f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:42.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.501+0000 7f5063fff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:08:42.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.501+0000 7f5063fff640 1 -- 172.21.15.44:0/1098695152 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c00fe00 con 0x56273ed02bd0 2024-04-21T23:08:42.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.501+0000 7f5063fff640 1 -- 172.21.15.44:0/1098695152 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7f507c004b20 con 0x56273ed02bd0 2024-04-21T23:08:42.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.501+0000 7f508da0c740 1 -- 172.21.15.44:0/1098695152 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5070004c80 con 0x56273ed02bd0 2024-04-21T23:08:42.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.502+0000 7f5063fff640 1 -- 172.21.15.44:0/1098695152 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f507c033050 con 0x56273ed02bd0 2024-04-21T23:08:42.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.502+0000 7f508da0c740 10 client.15372 fetch_fsmap learned FSMap version 21 2024-04-21T23:08:42.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.502+0000 7f508da0c740 1 -- 172.21.15.44:0/1098695152 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56273ed11b20 con 0x56273ed02bd0 2024-04-21T23:08:42.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.502+0000 7f5063fff640 1 -- 172.21.15.44:0/1098695152 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5070004c80 con 0x56273ed02bd0 2024-04-21T23:08:42.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.502+0000 7f5063fff640 1 -- 172.21.15.44:0/1098695152 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f507c01f050 con 0x56273ed02bd0 2024-04-21T23:08:42.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.502+0000 7f508da0c740 10 client.15372 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:08:42.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.502+0000 7f508da0c740 10 client.15372 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T23:08:42.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.502+0000 7f508da0c740 20 client.15372 populate_metadata read hostname 'smithi044' 2024-04-21T23:08:42.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.502+0000 7f508da0c740 1 --2- 172.21.15.44:0/1098695152 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56273ed141e0 0x56273ed15650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.502+0000 7f508da0c740 4 client.15372 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:08:42.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.502+0000 7f508da0c740 1 -- 172.21.15.44:0/1098695152 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56273ec6f330 con 0x56273ed141e0 2024-04-21T23:08:42.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.502+0000 7f5088bc9640 1 --2- 172.21.15.44:0/1098695152 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56273ed141e0 0x56273ed15650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.503+0000 7f5088bc9640 1 --2- 172.21.15.44:0/1098695152 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56273ed141e0 0x56273ed15650 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2506112dce8be8ed server_cookie=feb181167463b8a in_seq=0 out_seq=0 2024-04-21T23:08:42.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.503+0000 7f5063fff640 10 client.15372 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:08:42.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.503+0000 7f5063fff640 1 -- 172.21.15.44:0/1098695152 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56273ec6f330 con 0x56273ed141e0 2024-04-21T23:08:42.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.503+0000 7f5063fff640 10 client.15372 handle_command_reply: tid=0 2024-04-21T23:08:42.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.504+0000 7f508da0c740 1 client.15372 shutdown 2024-04-21T23:08:42.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.504+0000 7f508da0c740 1 -- 172.21.15.44:0/1098695152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 msgr2=0x56273ed0ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.504+0000 7f508da0c740 1 --2- 172.21.15.44:0/1098695152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 0x56273ed0ead0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f507c00fd10 tx=0x7f507c01c000 comp rx=0 tx=0).stop 2024-04-21T23:08:42.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.504+0000 7f508da0c740 1 -- 172.21.15.44:0/1098695152 shutdown_connections 2024-04-21T23:08:42.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.504+0000 7f508da0c740 1 --2- 172.21.15.44:0/1098695152 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56273ed141e0 0x56273ed15650 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.504+0000 7f508da0c740 1 --2- 172.21.15.44:0/1098695152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56273ecfda60 0x56273ed07a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.504+0000 7f508da0c740 1 --2- 172.21.15.44:0/1098695152 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 0x56273ed0ead0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.504+0000 7f5063fff640 0 client.15372 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:08:42.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.504+0000 7f508da0c740 1 --2- 172.21.15.44:0/1098695152 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ecfd070 0x56273ed07550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.504+0000 7f508da0c740 1 -- 172.21.15.44:0/1098695152 >> 172.21.15.44:0/1098695152 conn(0x56273ecf85e0 msgr2=0x56273ecf7720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:42.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.504+0000 7f508da0c740 1 -- 172.21.15.44:0/1098695152 shutdown_connections 2024-04-21T23:08:42.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.504+0000 7f508da0c740 1 -- 172.21.15.44:0/1098695152 wait complete. 2024-04-21T23:08:42.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.504+0000 7f508da0c740 20 client.15372 trim_cache size 0 max 16384 2024-04-21T23:08:42.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.522+0000 7f508da0c740 1 Processor -- start 2024-04-21T23:08:42.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.522+0000 7f508da0c740 1 -- start start 2024-04-21T23:08:42.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.522+0000 7f508da0c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ecfd070 0x56273ecf89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.522+0000 7f508da0c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56273ecfda60 0x56273ed0ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.522+0000 7f508da0c740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ed02bd0 0x56273ecf6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.522+0000 7f508da0c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56273ed13990 con 0x56273ecfd070 2024-04-21T23:08:42.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.522+0000 7f508da0c740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x56273ed14060 con 0x56273ed02bd0 2024-04-21T23:08:42.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.522+0000 7f508da0c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56273ed14730 con 0x56273ecfda60 2024-04-21T23:08:42.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.523+0000 7f5088bc9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ecfd070 0x56273ecf89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.523+0000 7f5088bc9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ecfd070 0x56273ecf89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40892/0 (socket says 172.21.15.44:40892) 2024-04-21T23:08:42.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.523+0000 7f5088bc9640 1 -- 172.21.15.44:0/1166200436 learned_addr learned my addr 172.21.15.44:0/1166200436 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:42.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.523+0000 7f50893ca640 1 --2- 172.21.15.44:0/1166200436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ed02bd0 0x56273ecf6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.523+0000 7f5083fff640 1 --2- 172.21.15.44:0/1166200436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56273ecfda60 0x56273ed0ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.523+0000 7f5088bc9640 1 -- 172.21.15.44:0/1166200436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56273ecfda60 msgr2=0x56273ed0ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:42.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.523+0000 7f5088bc9640 1 --2- 172.21.15.44:0/1166200436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56273ecfda60 0x56273ed0ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.523+0000 7f5088bc9640 1 -- 172.21.15.44:0/1166200436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ed02bd0 msgr2=0x56273ecf6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:42.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.523+0000 7f5088bc9640 1 --2- 172.21.15.44:0/1166200436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ed02bd0 0x56273ecf6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.523+0000 7f5088bc9640 1 -- 172.21.15.44:0/1166200436 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56273ecf6c20 con 0x56273ecfd070 2024-04-21T23:08:42.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.523+0000 7f5088bc9640 1 --2- 172.21.15.44:0/1166200436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ecfd070 0x56273ecf89c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5078006010 tx=0x7f50780f3040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c9154a220540a9e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:42.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.523+0000 7f506e7fc640 1 -- 172.21.15.44:0/1166200436 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507804b040 con 0x56273ecfd070 2024-04-21T23:08:42.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.523+0000 7f506e7fc640 1 -- 172.21.15.44:0/1166200436 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5078005690 con 0x56273ecfd070 2024-04-21T23:08:42.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.523+0000 7f506e7fc640 1 -- 172.21.15.44:0/1166200436 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50780bd660 con 0x56273ecfd070 2024-04-21T23:08:42.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.524+0000 7f508da0c740 1 -- 172.21.15.44:0/1166200436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ecfd070 msgr2=0x56273ecf89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.524+0000 7f508da0c740 1 --2- 172.21.15.44:0/1166200436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ecfd070 0x56273ecf89c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5078006010 tx=0x7f50780f3040 comp rx=0 tx=0).stop 2024-04-21T23:08:42.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.524+0000 7f508da0c740 1 -- 172.21.15.44:0/1166200436 shutdown_connections 2024-04-21T23:08:42.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.524+0000 7f508da0c740 1 --2- 172.21.15.44:0/1166200436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56273ecfda60 0x56273ed0ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.524+0000 7f508da0c740 1 --2- 172.21.15.44:0/1166200436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ed02bd0 0x56273ecf6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.524+0000 7f508da0c740 1 --2- 172.21.15.44:0/1166200436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ecfd070 0x56273ecf89c0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.524+0000 7f508da0c740 1 -- 172.21.15.44:0/1166200436 >> 172.21.15.44:0/1166200436 conn(0x56273ecf85e0 msgr2=0x56273ecfef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:42.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.524+0000 7f508da0c740 1 -- 172.21.15.44:0/1166200436 shutdown_connections 2024-04-21T23:08:42.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.524+0000 7f508da0c740 1 -- 172.21.15.44:0/1166200436 wait complete. 2024-04-21T23:08:42.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.524+0000 7f508da0c740 1 -- start start 2024-04-21T23:08:42.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.524+0000 7f508da0c740 1 Processor -- start 2024-04-21T23:08:42.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f508da0c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56273ecfd070 0x56273ed0a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f508da0c740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ecfda60 0x56273ed0a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f508da0c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 0x56273ed0d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f508da0c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56273ed154d0 con 0x56273ed02bd0 2024-04-21T23:08:42.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f508da0c740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x56273ed14e00 con 0x56273ecfda60 2024-04-21T23:08:42.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f508da0c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56273ed15ba0 con 0x56273ecfd070 2024-04-21T23:08:42.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f508da0c740 10 client.0 _ll_register_callbacks cb 0x7f507c009940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:08:42.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f50893ca640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 0x56273ed0d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f50893ca640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 0x56273ed0d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40898/0 (socket says 172.21.15.44:40898) 2024-04-21T23:08:42.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f50893ca640 1 -- 172.21.15.44:0/4064651986 learned_addr learned my addr 172.21.15.44:0/4064651986 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:42.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f5088bc9640 1 --2- 172.21.15.44:0/4064651986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56273ecfd070 0x56273ed0a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f5083fff640 1 --2- 172.21.15.44:0/4064651986 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ecfda60 0x56273ed0a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f50893ca640 1 -- 172.21.15.44:0/4064651986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56273ecfd070 msgr2=0x56273ed0a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:42.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f50893ca640 1 --2- 172.21.15.44:0/4064651986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56273ecfd070 0x56273ed0a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f50893ca640 1 -- 172.21.15.44:0/4064651986 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ecfda60 msgr2=0x56273ed0a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:42.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f50893ca640 1 --2- 172.21.15.44:0/4064651986 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ecfda60 0x56273ed0a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f50893ca640 1 -- 172.21.15.44:0/4064651986 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56273ed0df20 con 0x56273ed02bd0 2024-04-21T23:08:42.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f5083fff640 1 --2- 172.21.15.44:0/4064651986 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ecfda60 0x56273ed0a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:08:42.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f50893ca640 1 --2- 172.21.15.44:0/4064651986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 0x56273ed0d930 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f507c012fd0 tx=0x7f507c012720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=def2c34fac21da4e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:42.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f5063fff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:08:42.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.525+0000 7f5063fff640 1 -- 172.21.15.44:0/4064651986 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c0049e0 con 0x56273ed02bd0 2024-04-21T23:08:42.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.526+0000 7f5063fff640 1 -- 172.21.15.44:0/4064651986 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7f507c004b80 con 0x56273ed02bd0 2024-04-21T23:08:42.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.526+0000 7f508da0c740 1 -- 172.21.15.44:0/4064651986 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56273ec69680 con 0x56273ed02bd0 2024-04-21T23:08:42.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.526+0000 7f5063fff640 1 -- 172.21.15.44:0/4064651986 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f507c0202c0 con 0x56273ed02bd0 2024-04-21T23:08:42.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.526+0000 7f508da0c740 10 client.15378 fetch_fsmap learned FSMap version 21 2024-04-21T23:08:42.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.526+0000 7f508da0c740 1 -- 172.21.15.44:0/4064651986 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56273ed7a3e0 con 0x56273ed02bd0 2024-04-21T23:08:42.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.526+0000 7f5063fff640 1 -- 172.21.15.44:0/4064651986 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56273ec69680 con 0x56273ed02bd0 2024-04-21T23:08:42.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.526+0000 7f5063fff640 1 -- 172.21.15.44:0/4064651986 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f507c036070 con 0x56273ed02bd0 2024-04-21T23:08:42.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.526+0000 7f508da0c740 10 client.15378 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:08:42.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.526+0000 7f508da0c740 10 client.15378 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T23:08:42.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.526+0000 7f508da0c740 20 client.15378 populate_metadata read hostname 'smithi044' 2024-04-21T23:08:42.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.526+0000 7f508da0c740 1 --2- 172.21.15.44:0/4064651986 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56273ed7f020 0x56273ed7dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.526+0000 7f508da0c740 4 client.15378 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T23:08:42.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.526+0000 7f508da0c740 1 -- 172.21.15.44:0/4064651986 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "status"}) -- 0x56273ebe7a70 con 0x56273ed7f020 2024-04-21T23:08:42.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.526+0000 7f5088bc9640 1 --2- 172.21.15.44:0/4064651986 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56273ed7f020 0x56273ed7dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.527+0000 7f5088bc9640 1 --2- 172.21.15.44:0/4064651986 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56273ed7f020 0x56273ed7dd30 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1507f366a5da265a server_cookie=3358a8e9361107f7 in_seq=0 out_seq=0 2024-04-21T23:08:42.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.527+0000 7f5063fff640 10 client.15378 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:08:42.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.527+0000 7f5063fff640 1 -- 172.21.15.44:0/4064651986 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56273ebe7a70 con 0x56273ed7f020 2024-04-21T23:08:42.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.527+0000 7f5063fff640 10 client.15378 handle_command_reply: tid=0 2024-04-21T23:08:42.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.527+0000 7f508da0c740 1 client.15378 shutdown 2024-04-21T23:08:42.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.527+0000 7f508da0c740 1 -- 172.21.15.44:0/4064651986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 msgr2=0x56273ed0d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.527+0000 7f508da0c740 1 --2- 172.21.15.44:0/4064651986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 0x56273ed0d930 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f507c012fd0 tx=0x7f507c012720 comp rx=0 tx=0).stop 2024-04-21T23:08:42.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.527+0000 7f508da0c740 1 -- 172.21.15.44:0/4064651986 shutdown_connections 2024-04-21T23:08:42.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.527+0000 7f508da0c740 1 --2- 172.21.15.44:0/4064651986 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56273ed7f020 0x56273ed7dd30 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.527+0000 7f508da0c740 1 --2- 172.21.15.44:0/4064651986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56273ed02bd0 0x56273ed0d930 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.527+0000 7f508da0c740 1 --2- 172.21.15.44:0/4064651986 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56273ecfda60 0x56273ed0a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.527+0000 7f5063fff640 0 client.15378 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:08:42.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.527+0000 7f508da0c740 1 --2- 172.21.15.44:0/4064651986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56273ecfd070 0x56273ed0a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.527+0000 7f508da0c740 1 -- 172.21.15.44:0/4064651986 >> 172.21.15.44:0/4064651986 conn(0x56273ecf85e0 msgr2=0x56273ecf7720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:42.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.527+0000 7f508da0c740 1 -- 172.21.15.44:0/4064651986 shutdown_connections 2024-04-21T23:08:42.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.527+0000 7f508da0c740 1 -- 172.21.15.44:0/4064651986 wait complete. 2024-04-21T23:08:42.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.527+0000 7f508da0c740 20 client.15378 trim_cache size 0 max 16384 2024-04-21T23:08:42.535 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:08:42.535 INFO:teuthology.orchestra.run.smithi044.stdout: "cluster_fsid": "80909b94-0032-11ef-bc93-c7b262605968", 2024-04-21T23:08:42.535 INFO:teuthology.orchestra.run.smithi044.stdout: "whoami": 0, 2024-04-21T23:08:42.535 INFO:teuthology.orchestra.run.smithi044.stdout: "id": 24530, 2024-04-21T23:08:42.536 INFO:teuthology.orchestra.run.smithi044.stdout: "want_state": "up:active", 2024-04-21T23:08:42.536 INFO:teuthology.orchestra.run.smithi044.stdout: "state": "up:active", 2024-04-21T23:08:42.536 INFO:teuthology.orchestra.run.smithi044.stdout: "fs_name": "cephfs", 2024-04-21T23:08:42.536 INFO:teuthology.orchestra.run.smithi044.stdout: "rank_uptime": 136.31947991300001, 2024-04-21T23:08:42.536 INFO:teuthology.orchestra.run.smithi044.stdout: "mdsmap_epoch": 21, 2024-04-21T23:08:42.536 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch": 94, 2024-04-21T23:08:42.536 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_epoch_barrier": 90, 2024-04-21T23:08:42.536 INFO:teuthology.orchestra.run.smithi044.stdout: "uptime": 140.40689937799999 2024-04-21T23:08:42.536 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:08:42.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.528+0000 7f508ae2d640 1 -- 172.21.15.44:0/2569113937 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f50740a2b90 msgr2=0x7f50740a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.528+0000 7f508ae2d640 1 --2- 172.21.15.44:0/2569113937 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f50740a2b90 0x7f50740a5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f507c009fd0 tx=0x7f507c009670 comp rx=0 tx=0).stop 2024-04-21T23:08:42.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.528+0000 7f508ae2d640 1 -- 172.21.15.44:0/2569113937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5084102280 msgr2=0x7f5084197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.528+0000 7f508ae2d640 1 --2- 172.21.15.44:0/2569113937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5084102280 0x7f5084197a60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5078009230 tx=0x7f5078005dd0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.537 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.528+0000 7f508ae2d640 1 -- 172.21.15.44:0/2569113937 shutdown_connections 2024-04-21T23:08:42.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.528+0000 7f508ae2d640 1 --2- 172.21.15.44:0/2569113937 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f50740a2b90 0x7f50740a5050 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.528+0000 7f508ae2d640 1 --2- 172.21.15.44:0/2569113937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5084104bb0 0x7f5084197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.528+0000 7f508ae2d640 1 --2- 172.21.15.44:0/2569113937 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50841074e0 0x7f508419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.528+0000 7f508ae2d640 1 --2- 172.21.15.44:0/2569113937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5084102280 0x7f5084197a60 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.528+0000 7f508ae2d640 1 -- 172.21.15.44:0/2569113937 >> 172.21.15.44:0/2569113937 conn(0x7f50840fbc10 msgr2=0x7f5084102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:42.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.528+0000 7f508ae2d640 1 -- 172.21.15.44:0/2569113937 shutdown_connections 2024-04-21T23:08:42.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.528+0000 7f508ae2d640 1 -- 172.21.15.44:0/2569113937 wait complete. 2024-04-21T23:08:42.539 INFO:teuthology.run_tasks:Running task sequential... 2024-04-21T23:08:42.550 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-21T23:08:42.550 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-21T23:08:42.550 INFO:teuthology.task.print:Enabling mgr modules 2024-04-21T23:08:42.550 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-21T23:08:42.551 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T23:08:42.551 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-04-21T23:08:42.552 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-21T23:08:42.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.656+0000 7f467c31e640 1 -- 172.21.15.44:0/2205675595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4674101890 msgr2=0x7f4674101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.656+0000 7f467c31e640 1 --2- 172.21.15.44:0/2205675595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4674101890 0x7f4674101d10 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f4664009900 tx=0x7f466402bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:42.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.656+0000 7f467c31e640 1 -- 172.21.15.44:0/2205675595 shutdown_connections 2024-04-21T23:08:42.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.656+0000 7f467c31e640 1 --2- 172.21.15.44:0/2205675595 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4674102250 0x7f4674108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.656+0000 7f467c31e640 1 --2- 172.21.15.44:0/2205675595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4674101890 0x7f4674101d10 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.656+0000 7f467c31e640 1 --2- 172.21.15.44:0/2205675595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46741006a0 0x7f4674100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.656+0000 7f467c31e640 1 -- 172.21.15.44:0/2205675595 >> 172.21.15.44:0/2205675595 conn(0x7f46740fbc10 msgr2=0x7f46740fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:42.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.656+0000 7f467c31e640 1 -- 172.21.15.44:0/2205675595 shutdown_connections 2024-04-21T23:08:42.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.656+0000 7f467c31e640 1 -- 172.21.15.44:0/2205675595 wait complete. 2024-04-21T23:08:42.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.657+0000 7f467c31e640 1 Processor -- start 2024-04-21T23:08:42.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.657+0000 7f467c31e640 1 -- start start 2024-04-21T23:08:42.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.657+0000 7f467c31e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46741006a0 0x7f46741969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.657+0000 7f467c31e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4674101890 0x7f4674196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.657+0000 7f467c31e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4674102250 0x7f467419df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.657+0000 7f467c31e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f467410c320 con 0x7f46741006a0 2024-04-21T23:08:42.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.657+0000 7f467c31e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f467410c1a0 con 0x7f4674101890 2024-04-21T23:08:42.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.657+0000 7f467c31e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f467410c4a0 con 0x7f4674102250 2024-04-21T23:08:42.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.658+0000 7f467a0ba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46741006a0 0x7f46741969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.658+0000 7f467a0ba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46741006a0 0x7f46741969c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40912/0 (socket says 172.21.15.44:40912) 2024-04-21T23:08:42.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.658+0000 7f467a0ba640 1 -- 172.21.15.44:0/2328011196 learned_addr learned my addr 172.21.15.44:0/2328011196 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:42.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.658+0000 7f46798b9640 1 --2- 172.21.15.44:0/2328011196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4674101890 0x7f4674196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.658+0000 7f467a8bb640 1 --2- 172.21.15.44:0/2328011196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4674102250 0x7f467419df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.658+0000 7f467a0ba640 1 -- 172.21.15.44:0/2328011196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4674102250 msgr2=0x7f467419df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.658+0000 7f467a0ba640 1 --2- 172.21.15.44:0/2328011196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4674102250 0x7f467419df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.658+0000 7f467a0ba640 1 -- 172.21.15.44:0/2328011196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4674101890 msgr2=0x7f4674196f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:42.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.658+0000 7f467a0ba640 1 --2- 172.21.15.44:0/2328011196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4674101890 0x7f4674196f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:42.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.658+0000 7f467a0ba640 1 -- 172.21.15.44:0/2328011196 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f467419e5e0 con 0x7f46741006a0 2024-04-21T23:08:42.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.658+0000 7f467a0ba640 1 --2- 172.21.15.44:0/2328011196 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46741006a0 0x7f46741969c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f467000eb70 tx=0x7f467000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:42.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.658+0000 7f466b7fe640 1 -- 172.21.15.44:0/2328011196 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4670003b70 con 0x7f46741006a0 2024-04-21T23:08:42.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.658+0000 7f467c31e640 1 -- 172.21.15.44:0/2328011196 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f467419e8d0 con 0x7f46741006a0 2024-04-21T23:08:42.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.658+0000 7f466b7fe640 1 -- 172.21.15.44:0/2328011196 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f467000ee70 con 0x7f46741006a0 2024-04-21T23:08:42.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.658+0000 7f466b7fe640 1 -- 172.21.15.44:0/2328011196 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4670010830 con 0x7f46741006a0 2024-04-21T23:08:42.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.658+0000 7f467c31e640 1 -- 172.21.15.44:0/2328011196 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f467419eda0 con 0x7f46741006a0 2024-04-21T23:08:42.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.659+0000 7f467c31e640 1 -- 172.21.15.44:0/2328011196 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4674107a10 con 0x7f46741006a0 2024-04-21T23:08:42.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.661+0000 7f466b7fe640 1 -- 172.21.15.44:0/2328011196 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4670004a40 con 0x7f46741006a0 2024-04-21T23:08:42.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.661+0000 7f466b7fe640 1 --2- 172.21.15.44:0/2328011196 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f46600a2b80 0x7f46600a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:42.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.661+0000 7f46798b9640 1 --2- 172.21.15.44:0/2328011196 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f46600a2b80 0x7f46600a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:42.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.661+0000 7f466b7fe640 1 -- 172.21.15.44:0/2328011196 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7f46700cfc20 con 0x7f46741006a0 2024-04-21T23:08:42.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.662+0000 7f46798b9640 1 --2- 172.21.15.44:0/2328011196 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f46600a2b80 0x7f46600a5040 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f466402c7b0 tx=0x7f4664005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:42.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.665+0000 7f466b7fe640 1 -- 172.21.15.44:0/2328011196 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f467009c7c0 con 0x7f46741006a0 2024-04-21T23:08:42.903 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:42.904+0000 7f467c31e640 1 -- 172.21.15.44:0/2328011196 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f467402d300 con 0x7f46741006a0 2024-04-21T23:08:43.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.153+0000 7f466b7fe640 1 -- 172.21.15.44:0/2328011196 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f467009c0c0 con 0x7f46741006a0 2024-04-21T23:08:43.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.156+0000 7f467c31e640 1 -- 172.21.15.44:0/2328011196 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f46600a2b80 msgr2=0x7f46600a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:43.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.156+0000 7f467c31e640 1 --2- 172.21.15.44:0/2328011196 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f46600a2b80 0x7f46600a5040 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f466402c7b0 tx=0x7f4664005fd0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.156+0000 7f467c31e640 1 -- 172.21.15.44:0/2328011196 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46741006a0 msgr2=0x7f46741969c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:43.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.156+0000 7f467c31e640 1 --2- 172.21.15.44:0/2328011196 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46741006a0 0x7f46741969c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f467000eb70 tx=0x7f467000c6a0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.156+0000 7f467c31e640 1 -- 172.21.15.44:0/2328011196 shutdown_connections 2024-04-21T23:08:43.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.156+0000 7f467c31e640 1 --2- 172.21.15.44:0/2328011196 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f46600a2b80 0x7f46600a5040 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.156+0000 7f467c31e640 1 --2- 172.21.15.44:0/2328011196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4674102250 0x7f467419df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.156+0000 7f467c31e640 1 --2- 172.21.15.44:0/2328011196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4674101890 0x7f4674196f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.156+0000 7f467c31e640 1 --2- 172.21.15.44:0/2328011196 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f46741006a0 0x7f46741969c0 secure :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f467000eb70 tx=0x7f467000c6a0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.156+0000 7f467c31e640 1 -- 172.21.15.44:0/2328011196 >> 172.21.15.44:0/2328011196 conn(0x7f46740fbc10 msgr2=0x7f4674104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:43.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.156+0000 7f467c31e640 1 -- 172.21.15.44:0/2328011196 shutdown_connections 2024-04-21T23:08:43.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.156+0000 7f467c31e640 1 -- 172.21.15.44:0/2328011196 wait complete. 2024-04-21T23:08:43.173 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-21T23:08:43.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.285+0000 7fca6097a640 1 -- 172.21.15.44:0/2959746714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca5c102250 msgr2=0x7fca5c108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:43.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.285+0000 7fca6097a640 1 --2- 172.21.15.44:0/2959746714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca5c102250 0x7fca5c108ae0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fca4c009900 tx=0x7fca4c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:43.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.285+0000 7fca6097a640 1 -- 172.21.15.44:0/2959746714 shutdown_connections 2024-04-21T23:08:43.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.285+0000 7fca6097a640 1 --2- 172.21.15.44:0/2959746714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca5c102250 0x7fca5c108ae0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.285+0000 7fca6097a640 1 --2- 172.21.15.44:0/2959746714 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca5c101890 0x7fca5c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.285+0000 7fca6097a640 1 --2- 172.21.15.44:0/2959746714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca5c1006a0 0x7fca5c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.285+0000 7fca6097a640 1 -- 172.21.15.44:0/2959746714 >> 172.21.15.44:0/2959746714 conn(0x7fca5c0fbc10 msgr2=0x7fca5c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:43.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.285+0000 7fca6097a640 1 -- 172.21.15.44:0/2959746714 shutdown_connections 2024-04-21T23:08:43.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.285+0000 7fca6097a640 1 -- 172.21.15.44:0/2959746714 wait complete. 2024-04-21T23:08:43.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.286+0000 7fca6097a640 1 Processor -- start 2024-04-21T23:08:43.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.286+0000 7fca6097a640 1 -- start start 2024-04-21T23:08:43.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.286+0000 7fca6097a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca5c1006a0 0x7fca5c104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:43.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.286+0000 7fca6097a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca5c101890 0x7fca5c102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:43.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.286+0000 7fca6097a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca5c102250 0x7fca5c1031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:43.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.286+0000 7fca6097a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fca5c10c300 con 0x7fca5c102250 2024-04-21T23:08:43.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.286+0000 7fca6097a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fca5c10c180 con 0x7fca5c101890 2024-04-21T23:08:43.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.286+0000 7fca6097a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fca5c10c480 con 0x7fca5c1006a0 2024-04-21T23:08:43.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.286+0000 7fca5ad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca5c102250 0x7fca5c1031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:43.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.286+0000 7fca5ad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca5c102250 0x7fca5c1031b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40932/0 (socket says 172.21.15.44:40932) 2024-04-21T23:08:43.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.286+0000 7fca5ad9d640 1 -- 172.21.15.44:0/243913415 learned_addr learned my addr 172.21.15.44:0/243913415 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:43.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.286+0000 7fca59d9b640 1 --2- 172.21.15.44:0/243913415 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca5c101890 0x7fca5c102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:43.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.286+0000 7fca5a59c640 1 --2- 172.21.15.44:0/243913415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca5c1006a0 0x7fca5c104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:43.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.286+0000 7fca5ad9d640 1 -- 172.21.15.44:0/243913415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca5c1006a0 msgr2=0x7fca5c104630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:43.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.286+0000 7fca5ad9d640 1 --2- 172.21.15.44:0/243913415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca5c1006a0 0x7fca5c104630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.286+0000 7fca5ad9d640 1 -- 172.21.15.44:0/243913415 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca5c101890 msgr2=0x7fca5c102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:43.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.286+0000 7fca5ad9d640 1 --2- 172.21.15.44:0/243913415 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca5c101890 0x7fca5c102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.286+0000 7fca5ad9d640 1 -- 172.21.15.44:0/243913415 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca5c103a40 con 0x7fca5c102250 2024-04-21T23:08:43.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.286+0000 7fca5ad9d640 1 --2- 172.21.15.44:0/243913415 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca5c102250 0x7fca5c1031b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fca4c02c7b0 tx=0x7fca4c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:43.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.287+0000 7fca4b7fe640 1 -- 172.21.15.44:0/243913415 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4c041070 con 0x7fca5c102250 2024-04-21T23:08:43.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.287+0000 7fca6097a640 1 -- 172.21.15.44:0/243913415 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca5c1a1b50 con 0x7fca5c102250 2024-04-21T23:08:43.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.287+0000 7fca4b7fe640 1 -- 172.21.15.44:0/243913415 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca4c0054e0 con 0x7fca5c102250 2024-04-21T23:08:43.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.287+0000 7fca4b7fe640 1 -- 172.21.15.44:0/243913415 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4c034440 con 0x7fca5c102250 2024-04-21T23:08:43.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.287+0000 7fca6097a640 1 -- 172.21.15.44:0/243913415 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca5c1a20a0 con 0x7fca5c102250 2024-04-21T23:08:43.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.288+0000 7fca6097a640 1 -- 172.21.15.44:0/243913415 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca5c1a22d0 con 0x7fca5c102250 2024-04-21T23:08:43.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.289+0000 7fca4b7fe640 1 -- 172.21.15.44:0/243913415 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7fca4c044070 con 0x7fca5c102250 2024-04-21T23:08:43.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.290+0000 7fca4b7fe640 1 --2- 172.21.15.44:0/243913415 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fca440a2bd0 0x7fca440a5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:43.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.290+0000 7fca5a59c640 1 -- 172.21.15.44:0/243913415 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fca440a2bd0 msgr2=0x7fca440a5090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/1417504446 2024-04-21T23:08:43.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.290+0000 7fca4b7fe640 1 -- 172.21.15.44:0/243913415 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7fca4c0f1030 con 0x7fca5c102250 2024-04-21T23:08:43.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.290+0000 7fca5a59c640 1 --2- 172.21.15.44:0/243913415 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fca440a2bd0 0x7fca440a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:08:43.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.294+0000 7fca4b7fe640 1 -- 172.21.15.44:0/243913415 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fca4c0bd580 con 0x7fca5c102250 2024-04-21T23:08:43.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:43 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3901181284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:43.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:43 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2328011196' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-21T23:08:43.436 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:43 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: ignoring --setuser ceph since I am not root 2024-04-21T23:08:43.436 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:43 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: ignoring --setgroup ceph since I am not root 2024-04-21T23:08:43.436 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:43 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:43.386+0000 7ff000fe2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T23:08:43.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:43 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3901181284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:43.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:43 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2328011196' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-21T23:08:43.464 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:43 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: ignoring --setuser ceph since I am not root 2024-04-21T23:08:43.465 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:43 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: ignoring --setgroup ceph since I am not root 2024-04-21T23:08:43.465 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:43 smithi055 ceph-mgr[32547]: -- 172.21.15.55:0/217065744 <== mon.1 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5638b3eba5a0 con 0x5638b3e8e800 2024-04-21T23:08:43.465 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:43 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:43.349+0000 7f33a5457200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T23:08:43.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:43 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3901181284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:43.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:43 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2328011196' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-21T23:08:43.466 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:43 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: ignoring --setuser ceph since I am not root 2024-04-21T23:08:43.466 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:43 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: ignoring --setgroup ceph since I am not root 2024-04-21T23:08:43.466 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:43 smithi069 ceph-mgr[33718]: -- 172.21.15.69:0/4029248415 <== mon.1 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55563e7805a0 con 0x55563e755000 2024-04-21T23:08:43.466 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:43 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:43.331+0000 7fa8c207f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T23:08:43.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.480+0000 7fca6097a640 1 -- 172.21.15.44:0/243913415 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fca5c1a24a0 con 0x7fca5c102250 2024-04-21T23:08:43.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.485+0000 7fca4b7fe640 1 -- 172.21.15.44:0/243913415 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v27)=0 v27) ==== 138+0+0 (secure 0 0 0) 0x7fca4c0c1880 con 0x7fca5c102250 2024-04-21T23:08:43.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.488+0000 7fca6097a640 1 -- 172.21.15.44:0/243913415 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fca440a2bd0 msgr2=0x7fca440a5090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:08:43.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.488+0000 7fca6097a640 1 --2- 172.21.15.44:0/243913415 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fca440a2bd0 0x7fca440a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.488+0000 7fca6097a640 1 -- 172.21.15.44:0/243913415 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca5c102250 msgr2=0x7fca5c1031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:43.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.488+0000 7fca6097a640 1 --2- 172.21.15.44:0/243913415 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca5c102250 0x7fca5c1031b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fca4c02c7b0 tx=0x7fca4c005dd0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.488+0000 7fca6097a640 1 -- 172.21.15.44:0/243913415 shutdown_connections 2024-04-21T23:08:43.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.488+0000 7fca6097a640 1 --2- 172.21.15.44:0/243913415 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fca440a2bd0 0x7fca440a5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.488+0000 7fca6097a640 1 --2- 172.21.15.44:0/243913415 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fca5c102250 0x7fca5c1031b0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.488+0000 7fca6097a640 1 --2- 172.21.15.44:0/243913415 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca5c101890 0x7fca5c102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.488+0000 7fca6097a640 1 --2- 172.21.15.44:0/243913415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca5c1006a0 0x7fca5c104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.488+0000 7fca6097a640 1 -- 172.21.15.44:0/243913415 >> 172.21.15.44:0/243913415 conn(0x7fca5c0fbc10 msgr2=0x7fca5c0fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:43.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.488+0000 7fca6097a640 1 -- 172.21.15.44:0/243913415 shutdown_connections 2024-04-21T23:08:43.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.488+0000 7fca6097a640 1 -- 172.21.15.44:0/243913415 wait complete. 2024-04-21T23:08:43.499 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-21T23:08:43.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.605+0000 7fa73f5b1640 1 -- 172.21.15.44:0/1577414789 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7381006a0 msgr2=0x7fa738100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.605+0000 7fa73f5b1640 1 --2- 172.21.15.44:0/1577414789 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7381006a0 0x7fa738100ac0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa728009900 tx=0x7fa72802bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.605+0000 7fa73f5b1640 1 -- 172.21.15.44:0/1577414789 shutdown_connections 2024-04-21T23:08:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.605+0000 7fa73f5b1640 1 --2- 172.21.15.44:0/1577414789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa738101890 0x7fa738101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.605+0000 7fa73f5b1640 1 --2- 172.21.15.44:0/1577414789 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa738102250 0x7fa738108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.605+0000 7fa73f5b1640 1 --2- 172.21.15.44:0/1577414789 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7381006a0 0x7fa738100ac0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.605+0000 7fa73f5b1640 1 -- 172.21.15.44:0/1577414789 >> 172.21.15.44:0/1577414789 conn(0x7fa7380fbc10 msgr2=0x7fa7380fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.605+0000 7fa73f5b1640 1 -- 172.21.15.44:0/1577414789 shutdown_connections 2024-04-21T23:08:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.605+0000 7fa73f5b1640 1 -- 172.21.15.44:0/1577414789 wait complete. 2024-04-21T23:08:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73f5b1640 1 Processor -- start 2024-04-21T23:08:43.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73f5b1640 1 -- start start 2024-04-21T23:08:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73f5b1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7381006a0 0x7fa738197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73f5b1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa738101890 0x7fa7381980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73f5b1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa738102250 0x7fa73819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73f5b1640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa73810c360 con 0x7fa7381006a0 2024-04-21T23:08:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73f5b1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa73810c1e0 con 0x7fa738102250 2024-04-21T23:08:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73f5b1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa73810c4e0 con 0x7fa738101890 2024-04-21T23:08:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73d34d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7381006a0 0x7fa738197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73d34d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7381006a0 0x7fa738197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40954/0 (socket says 172.21.15.44:40954) 2024-04-21T23:08:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73d34d640 1 -- 172.21.15.44:0/3703966122 learned_addr learned my addr 172.21.15.44:0/3703966122 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73db4e640 1 --2- 172.21.15.44:0/3703966122 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa738102250 0x7fa73819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73cb4c640 1 --2- 172.21.15.44:0/3703966122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa738101890 0x7fa7381980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73d34d640 1 -- 172.21.15.44:0/3703966122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa738101890 msgr2=0x7fa7381980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73d34d640 1 --2- 172.21.15.44:0/3703966122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa738101890 0x7fa7381980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73d34d640 1 -- 172.21.15.44:0/3703966122 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa738102250 msgr2=0x7fa73819f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:43.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73d34d640 1 --2- 172.21.15.44:0/3703966122 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa738102250 0x7fa73819f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73d34d640 1 -- 172.21.15.44:0/3703966122 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa73819f740 con 0x7fa7381006a0 2024-04-21T23:08:43.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73db4e640 1 --2- 172.21.15.44:0/3703966122 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa738102250 0x7fa73819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:08:43.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73cb4c640 1 --2- 172.21.15.44:0/3703966122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa738101890 0x7fa7381980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:08:43.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.606+0000 7fa73d34d640 1 --2- 172.21.15.44:0/3703966122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7381006a0 0x7fa738197b90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa728000c00 tx=0x7fa72802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:43.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.607+0000 7fa72e7fc640 1 -- 172.21.15.44:0/3703966122 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa728043020 con 0x7fa7381006a0 2024-04-21T23:08:43.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.607+0000 7fa72e7fc640 1 -- 172.21.15.44:0/3703966122 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7280054e0 con 0x7fa7381006a0 2024-04-21T23:08:43.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.607+0000 7fa72e7fc640 1 -- 172.21.15.44:0/3703966122 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa728003c40 con 0x7fa7381006a0 2024-04-21T23:08:43.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.607+0000 7fa73f5b1640 1 -- 172.21.15.44:0/3703966122 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa73819f9d0 con 0x7fa7381006a0 2024-04-21T23:08:43.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.607+0000 7fa73f5b1640 1 -- 172.21.15.44:0/3703966122 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa73819ff20 con 0x7fa7381006a0 2024-04-21T23:08:43.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.608+0000 7fa73f5b1640 1 -- 172.21.15.44:0/3703966122 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa73804fd50 con 0x7fa7381006a0 2024-04-21T23:08:43.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.609+0000 7fa72e7fc640 1 -- 172.21.15.44:0/3703966122 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7fa728046020 con 0x7fa7381006a0 2024-04-21T23:08:43.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.609+0000 7fa72e7fc640 1 --2- 172.21.15.44:0/3703966122 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa7240a2bd0 0x7fa7240a5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:43.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.610+0000 7fa73cb4c640 1 -- 172.21.15.44:0/3703966122 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa7240a2bd0 msgr2=0x7fa7240a5090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/1417504446 2024-04-21T23:08:43.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.610+0000 7fa72e7fc640 1 -- 172.21.15.44:0/3703966122 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7fa7280f1380 con 0x7fa7381006a0 2024-04-21T23:08:43.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.610+0000 7fa73cb4c640 1 --2- 172.21.15.44:0/3703966122 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa7240a2bd0 0x7fa7240a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:08:43.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.614+0000 7fa72e7fc640 1 -- 172.21.15.44:0/3703966122 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fa72804d020 con 0x7fa7381006a0 2024-04-21T23:08:43.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.811+0000 7fa73cb4c640 1 -- 172.21.15.44:0/3703966122 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa7240a2bd0 msgr2=0x7fa7240a5090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/1417504446 2024-04-21T23:08:43.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.811+0000 7fa73cb4c640 1 --2- 172.21.15.44:0/3703966122 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa7240a2bd0 0x7fa7240a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T23:08:43.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.820+0000 7fa73f5b1640 1 -- 172.21.15.44:0/3703966122 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fa7381a00f0 con 0x7fa7381006a0 2024-04-21T23:08:43.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.825+0000 7fa72e7fc640 1 -- 172.21.15.44:0/3703966122 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v28)=0 v28) ==== 133+0+0 (secure 0 0 0) 0x7fa7280bdfc0 con 0x7fa7381006a0 2024-04-21T23:08:43.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.827+0000 7fa73f5b1640 1 -- 172.21.15.44:0/3703966122 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa7240a2bd0 msgr2=0x7fa7240a5090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:08:43.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.827+0000 7fa73f5b1640 1 --2- 172.21.15.44:0/3703966122 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa7240a2bd0 0x7fa7240a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.827+0000 7fa73f5b1640 1 -- 172.21.15.44:0/3703966122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7381006a0 msgr2=0x7fa738197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:43.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.827+0000 7fa73f5b1640 1 --2- 172.21.15.44:0/3703966122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7381006a0 0x7fa738197b90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa728000c00 tx=0x7fa72802c650 comp rx=0 tx=0).stop 2024-04-21T23:08:43.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.827+0000 7fa73f5b1640 1 -- 172.21.15.44:0/3703966122 shutdown_connections 2024-04-21T23:08:43.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.827+0000 7fa73f5b1640 1 --2- 172.21.15.44:0/3703966122 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fa7240a2bd0 0x7fa7240a5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.827+0000 7fa73f5b1640 1 --2- 172.21.15.44:0/3703966122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa738101890 0x7fa7381980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.827+0000 7fa73f5b1640 1 --2- 172.21.15.44:0/3703966122 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa738102250 0x7fa73819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.827+0000 7fa73f5b1640 1 --2- 172.21.15.44:0/3703966122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa7381006a0 0x7fa738197b90 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:43.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.827+0000 7fa73f5b1640 1 -- 172.21.15.44:0/3703966122 >> 172.21.15.44:0/3703966122 conn(0x7fa7380fbc10 msgr2=0x7fa7380fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:43.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.827+0000 7fa73f5b1640 1 -- 172.21.15.44:0/3703966122 shutdown_connections 2024-04-21T23:08:43.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:43.827+0000 7fa73f5b1640 1 -- 172.21.15.44:0/3703966122 wait complete. 2024-04-21T23:08:43.836 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:43 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:43.582+0000 7ff000fe2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:08:43.836 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:43 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:43.708+0000 7ff000fe2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T23:08:43.837 INFO:teuthology.run_tasks:Running task kclient... 2024-04-21T23:08:43.851 INFO:tasks.kclient:Mounting kernel clients... 2024-04-21T23:08:43.852 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-21T23:08:43.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:08:43.852 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-21T23:08:43.867 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-21T23:08:43.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:08:43.868 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T23:08:43.922 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:43 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:43.513+0000 7fa8c207f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:08:43.923 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:43 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:43.616+0000 7fa8c207f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T23:08:43.923 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:43 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:43.923+0000 7fa8c207f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T23:08:43.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:08:43.933 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link delete ceph-brx 2024-04-21T23:08:43.964 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:43 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:43.540+0000 7f33a5457200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:08:44.009 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:43 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:43.647+0000 7f33a5457200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T23:08:44.069 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-04-21T23:08:44.069 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-21T23:08:44.069 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-21T23:08:44.070 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-21T23:08:44.093 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:44 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:44.041+0000 7ff000fe2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T23:08:44.214 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:44 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:44.008+0000 7fa8c207f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T23:08:44.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:44 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2328011196' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-21T23:08:44.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:44 smithi069 ceph-mon[31816]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-21T23:08:44.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:44 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/243913415' entity='client.admin' 2024-04-21T23:08:44.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:44 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3703966122' entity='client.admin' 2024-04-21T23:08:44.331 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:08:44.435 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:44 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:44.136+0000 7ff000fe2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T23:08:44.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:44 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2328011196' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-21T23:08:44.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:44 smithi044 ceph-mon[27934]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-21T23:08:44.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:44 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/243913415' entity='client.admin' 2024-04-21T23:08:44.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:44 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3703966122' entity='client.admin' 2024-04-21T23:08:44.464 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:43 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:43.969+0000 7f33a5457200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T23:08:44.464 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:44 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:44.058+0000 7f33a5457200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T23:08:44.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:44 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2328011196' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-21T23:08:44.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:44 smithi055 ceph-mon[31117]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-21T23:08:44.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:44 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/243913415' entity='client.admin' 2024-04-21T23:08:44.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:44 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3703966122' entity='client.admin' 2024-04-21T23:08:45.464 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:45 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:45.190+0000 7fa8c207f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T23:08:45.464 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:45 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:45.361+0000 7fa8c207f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T23:08:45.528 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:45 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:45.413+0000 7ff000fe2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T23:08:45.653 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:45 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:45.299+0000 7f33a5457200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T23:08:45.653 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:45 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:45.476+0000 7f33a5457200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T23:08:45.819 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:45 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:45.532+0000 7fa8c207f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T23:08:45.851 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:45 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:45.597+0000 7ff000fe2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T23:08:45.851 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:45 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:45.778+0000 7ff000fe2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T23:08:45.953 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:45 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:45.653+0000 7f33a5457200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T23:08:46.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.089+0000 7f0b57a8a700 1 -- 172.21.15.44:0/2901080980 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b50101120 msgr2=0x7f0b501015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:46.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.089+0000 7f0b57a8a700 1 --2- 172.21.15.44:0/2901080980 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b50101120 0x7f0b501015a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0b44009990 tx=0x7f0b44009ca0 comp rx=0 tx=0).stop 2024-04-21T23:08:46.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.089+0000 7f0b57a8a700 1 -- 172.21.15.44:0/2901080980 shutdown_connections 2024-04-21T23:08:46.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.089+0000 7f0b57a8a700 1 --2- 172.21.15.44:0/2901080980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b50101ae0 0x7f0b5010dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:46.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.089+0000 7f0b57a8a700 1 --2- 172.21.15.44:0/2901080980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b50107190 0x7f0b50107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.089+0000 7f0b57a8a700 1 --2- 172.21.15.44:0/2901080980 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b50101120 0x7f0b501015a0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.089+0000 7f0b57a8a700 1 -- 172.21.15.44:0/2901080980 >> 172.21.15.44:0/2901080980 conn(0x7f0b500780a0 msgr2=0x7f0b500784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.090+0000 7f0b57a8a700 1 -- 172.21.15.44:0/2901080980 shutdown_connections 2024-04-21T23:08:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.090+0000 7f0b57a8a700 1 -- 172.21.15.44:0/2901080980 wait complete. 2024-04-21T23:08:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.090+0000 7f0b57a8a700 1 Processor -- start 2024-04-21T23:08:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.090+0000 7f0b57a8a700 1 -- start start 2024-04-21T23:08:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.090+0000 7f0b57a8a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b50101120 0x7f0b5019e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.090+0000 7f0b57a8a700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b50101ae0 0x7f0b5019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.090+0000 7f0b57a8a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b50107190 0x7f0b50198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.090+0000 7f0b57a8a700 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0b50111400 con 0x7f0b50101ae0 2024-04-21T23:08:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.090+0000 7f0b57a8a700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0b50111280 con 0x7f0b50107190 2024-04-21T23:08:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.090+0000 7f0b57a8a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0b50111580 con 0x7f0b50101120 2024-04-21T23:08:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.091+0000 7f0b56027700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b50107190 0x7f0b50198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.091+0000 7f0b55826700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b50101120 0x7f0b5019e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.091+0000 7f0b55025700 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b50101ae0 0x7f0b5019ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.091+0000 7f0b56027700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b50107190 0x7f0b50198810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:54180/0 (socket says 172.21.15.44:54180) 2024-04-21T23:08:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.091+0000 7f0b56027700 1 -- 172.21.15.44:0/3354240977 learned_addr learned my addr 172.21.15.44:0/3354240977 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.091+0000 7f0b55826700 1 --2- 172.21.15.44:0/3354240977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b50101120 0x7f0b5019e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:49766/0 (socket says 172.21.15.44:49766) 2024-04-21T23:08:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.091+0000 7f0b56027700 1 -- 172.21.15.44:0/3354240977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b50101120 msgr2=0x7f0b5019e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.091+0000 7f0b56027700 1 --2- 172.21.15.44:0/3354240977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b50101120 0x7f0b5019e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.091+0000 7f0b56027700 1 -- 172.21.15.44:0/3354240977 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b50101ae0 msgr2=0x7f0b5019ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.091+0000 7f0b56027700 1 --2- 172.21.15.44:0/3354240977 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b50101ae0 0x7f0b5019ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.091+0000 7f0b56027700 1 -- 172.21.15.44:0/3354240977 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b50198ff0 con 0x7f0b50107190 2024-04-21T23:08:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.091+0000 7f0b55826700 1 --2- 172.21.15.44:0/3354240977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b50101120 0x7f0b5019e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:08:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.091+0000 7f0b56027700 1 --2- 172.21.15.44:0/3354240977 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b50107190 0x7f0b50198810 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0b4800ef30 tx=0x7f0b4800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.091+0000 7f0b42ffd700 1 -- 172.21.15.44:0/3354240977 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b480092e0 con 0x7f0b50107190 2024-04-21T23:08:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.091+0000 7f0b57a8a700 1 -- 172.21.15.44:0/3354240977 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b50199280 con 0x7f0b50107190 2024-04-21T23:08:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.091+0000 7f0b42ffd700 1 -- 172.21.15.44:0/3354240977 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b48004bb0 con 0x7f0b50107190 2024-04-21T23:08:46.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.091+0000 7f0b42ffd700 1 -- 172.21.15.44:0/3354240977 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b48004280 con 0x7f0b50107190 2024-04-21T23:08:46.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.092+0000 7f0b57a8a700 1 -- 172.21.15.44:0/3354240977 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b50102d30 con 0x7f0b50107190 2024-04-21T23:08:46.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.093+0000 7f0b57a8a700 1 -- 172.21.15.44:0/3354240977 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b50102840 con 0x7f0b50107190 2024-04-21T23:08:46.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.094+0000 7f0b42ffd700 1 -- 172.21.15.44:0/3354240977 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f0b4801b030 con 0x7f0b50107190 2024-04-21T23:08:46.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.094+0000 7f0b42ffd700 1 --2- 172.21.15.44:0/3354240977 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f0b3c0a2c70 0x7f0b3c0a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:46.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.094+0000 7f0b55826700 1 -- 172.21.15.44:0/3354240977 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f0b3c0a2c70 msgr2=0x7f0b3c0a5130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/1417504446 2024-04-21T23:08:46.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.094+0000 7f0b42ffd700 1 -- 172.21.15.44:0/3354240977 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7f0b480cf3f0 con 0x7f0b50107190 2024-04-21T23:08:46.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.094+0000 7f0b55826700 1 --2- 172.21.15.44:0/3354240977 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f0b3c0a2c70 0x7f0b3c0a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:08:46.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.098+0000 7f0b42ffd700 1 -- 172.21.15.44:0/3354240977 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0b48022020 con 0x7f0b50107190 2024-04-21T23:08:46.114 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:45 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:45.819+0000 7fa8c207f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T23:08:46.114 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:45 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:45.912+0000 7fa8c207f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T23:08:46.114 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:46 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:46.114+0000 7fa8c207f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T23:08:46.185 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:46 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:46.090+0000 7ff000fe2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T23:08:46.214 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:45 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:45.954+0000 7f33a5457200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T23:08:46.214 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:46 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:46.050+0000 7f33a5457200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T23:08:46.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.295+0000 7f0b55826700 1 -- 172.21.15.44:0/3354240977 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f0b3c0a2c70 msgr2=0x7f0b3c0a5130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/1417504446 2024-04-21T23:08:46.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.295+0000 7f0b55826700 1 --2- 172.21.15.44:0/3354240977 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f0b3c0a2c70 0x7f0b3c0a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T23:08:46.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.358+0000 7f0b57a8a700 1 -- 172.21.15.44:0/3354240977 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f0b50199d40 con 0x7f0b50107190 2024-04-21T23:08:46.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.358+0000 7f0b42ffd700 1 -- 172.21.15.44:0/3354240977 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) ==== 53+0+88 (secure 0 0 0) 0x7f0b48010040 con 0x7f0b50107190 2024-04-21T23:08:46.357 INFO:teuthology.orchestra.run.smithi044.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-21T23:08:46.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.360+0000 7f0b57a8a700 1 -- 172.21.15.44:0/3354240977 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f0b3c0a2c70 msgr2=0x7f0b3c0a5130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:08:46.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.360+0000 7f0b57a8a700 1 --2- 172.21.15.44:0/3354240977 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f0b3c0a2c70 0x7f0b3c0a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:46.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.360+0000 7f0b57a8a700 1 -- 172.21.15.44:0/3354240977 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b50107190 msgr2=0x7f0b50198810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:46.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.360+0000 7f0b57a8a700 1 --2- 172.21.15.44:0/3354240977 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b50107190 0x7f0b50198810 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0b4800ef30 tx=0x7f0b4800c3a0 comp rx=0 tx=0).stop 2024-04-21T23:08:46.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.361+0000 7f0b57a8a700 1 -- 172.21.15.44:0/3354240977 shutdown_connections 2024-04-21T23:08:46.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.361+0000 7f0b57a8a700 1 --2- 172.21.15.44:0/3354240977 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b50107190 0x7f0b50198810 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:46.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.361+0000 7f0b57a8a700 1 --2- 172.21.15.44:0/3354240977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b50101120 0x7f0b5019e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:46.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.361+0000 7f0b57a8a700 1 --2- 172.21.15.44:0/3354240977 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f0b3c0a2c70 0x7f0b3c0a5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:46.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.361+0000 7f0b57a8a700 1 --2- 172.21.15.44:0/3354240977 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0b50101ae0 0x7f0b5019ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:46.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.361+0000 7f0b57a8a700 1 -- 172.21.15.44:0/3354240977 >> 172.21.15.44:0/3354240977 conn(0x7f0b500780a0 msgr2=0x7f0b50100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:46.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.361+0000 7f0b57a8a700 1 -- 172.21.15.44:0/3354240977 shutdown_connections 2024-04-21T23:08:46.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:46.361+0000 7f0b57a8a700 1 -- 172.21.15.44:0/3354240977 wait complete. 2024-04-21T23:08:46.409 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:46 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:46.310+0000 7fa8c207f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:08:46.614 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:46 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3354240977' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T23:08:46.615 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:46 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:46.190+0000 7ff000fe2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T23:08:46.615 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:46 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:46.409+0000 7ff000fe2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T23:08:46.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:46 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3354240977' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T23:08:46.714 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:46 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:46.261+0000 7f33a5457200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T23:08:46.714 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:46 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:46.463+0000 7f33a5457200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:08:46.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:46 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3354240977' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T23:08:46.935 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:46 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:46.615+0000 7ff000fe2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T23:08:47.026 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-21T23:08:47.026 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-21T23:08:47.026 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi044.front.sepia.ceph.com 2024-04-21T23:08:47.027 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-21T23:08:47.027 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-21T23:08:47.027 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-21T23:08:47.027 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-21T23:08:47.027 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-21T23:08:47.027 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-21T23:08:47.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:08:47.027 DEBUG:teuthology.orchestra.run.smithi044:> ip addr 2024-04-21T23:08:47.083 INFO:teuthology.orchestra.run.smithi044.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-21T23:08:47.083 INFO:teuthology.orchestra.run.smithi044.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-21T23:08:47.083 INFO:teuthology.orchestra.run.smithi044.stdout: inet 127.0.0.1/8 scope host lo 2024-04-21T23:08:47.083 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-21T23:08:47.083 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 ::1/128 scope host 2024-04-21T23:08:47.083 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-21T23:08:47.083 INFO:teuthology.orchestra.run.smithi044.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T23:08:47.083 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8c brd ff:ff:ff:ff:ff:ff 2024-04-21T23:08:47.083 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f0 2024-04-21T23:08:47.083 INFO:teuthology.orchestra.run.smithi044.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T23:08:47.084 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:6b:f7:8d brd ff:ff:ff:ff:ff:ff 2024-04-21T23:08:47.084 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp7s0f1 2024-04-21T23:08:47.084 INFO:teuthology.orchestra.run.smithi044.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-21T23:08:47.084 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:72 brd ff:ff:ff:ff:ff:ff 2024-04-21T23:08:47.084 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f0 2024-04-21T23:08:47.084 INFO:teuthology.orchestra.run.smithi044.stdout: inet 172.21.15.44/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-21T23:08:47.084 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft 42038sec preferred_lft 42038sec 2024-04-21T23:08:47.084 INFO:teuthology.orchestra.run.smithi044.stdout: inet6 fe80::ec4:7aff:febd:1572/64 scope link noprefixroute 2024-04-21T23:08:47.084 INFO:teuthology.orchestra.run.smithi044.stdout: valid_lft forever preferred_lft forever 2024-04-21T23:08:47.084 INFO:teuthology.orchestra.run.smithi044.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T23:08:47.084 INFO:teuthology.orchestra.run.smithi044.stdout: link/ether 0c:c4:7a:bd:15:73 brd ff:ff:ff:ff:ff:ff 2024-04-21T23:08:47.084 INFO:teuthology.orchestra.run.smithi044.stdout: altname enp1s0f1 2024-04-21T23:08:47.084 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-21T23:08:47.085 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:08:47.085 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-21T23:08:47.085 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add name ceph-brx type bridge 2024-04-21T23:08:47.085 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip addr flush dev ceph-brx 2024-04-21T23:08:47.085 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set ceph-brx up 2024-04-21T23:08:47.085 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-21T23:08:47.085 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-21T23:08:47.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:08:47.214 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:46 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:46.914+0000 7fa8c207f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T23:08:47.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:08:47.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:08:47.233 DEBUG:teuthology.orchestra.run.smithi044:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-21T23:08:47.257 INFO:teuthology.orchestra.run.smithi044.stdout:1 2024-04-21T23:08:47.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:08:47.259 DEBUG:teuthology.orchestra.run.smithi044:> route 2024-04-21T23:08:47.316 INFO:teuthology.orchestra.run.smithi044.stdout:Kernel IP routing table 2024-04-21T23:08:47.316 INFO:teuthology.orchestra.run.smithi044.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-21T23:08:47.316 INFO:teuthology.orchestra.run.smithi044.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-21T23:08:47.316 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-21T23:08:47.316 INFO:teuthology.orchestra.run.smithi044.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-21T23:08:47.316 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:08:47.316 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-21T23:08:47.316 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-21T23:08:47.316 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-21T23:08:47.316 DEBUG:teuthology.orchestra.run.smithi044:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-21T23:08:47.317 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-21T23:08:47.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:08:47.439 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:08:47.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:08:47.445 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-21T23:08:47.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:08:47.461 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list-id 2024-04-21T23:08:47.463 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:47 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:47.089+0000 7f33a5457200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T23:08:47.516 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:08:47.516 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-21T23:08:47.516 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-21T23:08:47.516 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-21T23:08:47.516 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-21T23:08:47.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:08:47.614 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:47 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:47.257+0000 7ff000fe2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T23:08:47.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:08:47.635 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-21T23:08:47.636 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:08:47.636 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-21T23:08:47.636 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-21T23:08:47.636 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-21T23:08:47.636 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-21T23:08:47.636 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-21T23:08:47.636 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-21T23:08:47.636 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-21T23:08:47.637 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:47 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:47.340+0000 7fa8c207f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T23:08:47.637 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:47 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:47.436+0000 7fa8c207f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T23:08:47.637 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:47 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:47.636+0000 7fa8c207f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T23:08:47.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:08:47.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:08:47.753 DEBUG:teuthology.orchestra.run.smithi044:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T23:08:47.753 DEBUG:teuthology.orchestra.run.smithi044:> set -e 2024-04-21T23:08:47.753 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set brx.0 up 2024-04-21T23:08:47.753 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link set dev brx.0 master ceph-brx 2024-04-21T23:08:47.753 DEBUG:teuthology.orchestra.run.smithi044:> ') 2024-04-21T23:08:47.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:08:47.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:08:47.831 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T23:08:47.832 DEBUG:teuthology.orchestra.run.smithi044:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-21T23:08:47.836 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:47 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:47.528+0000 7f33a5457200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T23:08:47.836 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:47 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:47.628+0000 7f33a5457200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T23:08:47.861 INFO:teuthology.orchestra.run.smithi044.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-21T23:08:47.861 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T23:08:47.861 DEBUG:teuthology.orchestra.run.smithi044:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-21T23:08:47.916 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-21T23:08:47.916 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-21T23:08:47.917 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/52aa91e0-dc47-4a01-8a26-3fb0130744af 2024-04-21T23:08:47.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:08:47.917 DEBUG:teuthology.orchestra.run.smithi044:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/52aa91e0-dc47-4a01-8a26-3fb0130744af /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-21T23:08:47.935 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:47 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:47.719+0000 7ff000fe2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T23:08:47.935 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:47 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:47.825+0000 7ff000fe2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T23:08:47.978 INFO:teuthology.orchestra.run.smithi044.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-21T23:08:47.978 INFO:teuthology.orchestra.run.smithi044.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-04-21T23:08:48.109 INFO:teuthology.orchestra.run.smithi044.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-21T23:08:48.109 INFO:teuthology.orchestra.run.smithi044.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-04-21T23:08:48.110 INFO:teuthology.orchestra.run.smithi044.stdout:mount.ceph: resolved to: "172.21.15.44:3300,172.21.15.55:3300,172.21.15.69:3300" 2024-04-21T23:08:48.110 INFO:teuthology.orchestra.run.smithi044.stdout:mount.ceph: trying mount with new device syntax: 0@80909b94-0032-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/52aa91e0-dc47-4a01-8a26-3fb0130744af 2024-04-21T23:08:48.110 INFO:teuthology.orchestra.run.smithi044.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync,name=0,key=0,mon_addr=172.21.15.44:3300/172.21.15.55:3300/172.21.15.69:3300" will pass to kernel 2024-04-21T23:08:48.111 INFO:teuthology.orchestra.run.smithi044.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-21T23:08:48.111 INFO:teuthology.orchestra.run.smithi044.stdout: You just mounted a file system that supports labels which does not 2024-04-21T23:08:48.111 INFO:teuthology.orchestra.run.smithi044.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-21T23:08:48.111 INFO:teuthology.orchestra.run.smithi044.stdout: applications will generate AVC messages and not be allowed access to 2024-04-21T23:08:48.111 INFO:teuthology.orchestra.run.smithi044.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-21T23:08:48.112 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-21T23:08:48.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:08:48.113 DEBUG:teuthology.orchestra.run.smithi044:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-21T23:08:48.178 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-21T23:08:48.214 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:47 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:47.837+0000 7f33a5457200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T23:08:48.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T23:08:48.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:08:48.280 INFO:teuthology.orchestra.run.smithi044.stdout:client25123 2024-04-21T23:08:48.280 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T23:08:48.281 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T23:08:48.297 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-21T23:08:48.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T23:08:48.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:08:48.395 INFO:teuthology.orchestra.run.smithi044.stdout:client25123 2024-04-21T23:08:48.396 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-21T23:08:48.435 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:48 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:48.043+0000 7ff000fe2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T23:08:48.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T23:08:48.449 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:08:48.455 INFO:teuthology.orchestra.run.smithi044.stdout:client25123 2024-04-21T23:08:48.456 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/80909b94-0032-11ef-bc93-c7b262605968.client25123/status') 2024-04-21T23:08:48.464 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:48 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:48.161+0000 7fa8c207f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T23:08:48.464 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:48 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:48.254+0000 7fa8c207f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T23:08:48.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T23:08:48.555 INFO:teuthology.orchestra.run.smithi044.stdout:instance: client.25123 (3)172.21.15.44:0/4290931051 2024-04-21T23:08:48.555 INFO:teuthology.orchestra.run.smithi044.stdout:blocklisted: false 2024-04-21T23:08:48.555 INFO:teuthology.orchestra.run.smithi044.stderr:0+1 records in 2024-04-21T23:08:48.556 INFO:teuthology.orchestra.run.smithi044.stderr:0+1 records out 2024-04-21T23:08:48.556 INFO:teuthology.orchestra.run.smithi044.stderr:71 bytes copied, 2.5408e-05 s, 2.8 MB/s 2024-04-21T23:08:48.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T23:08:48.563 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-21T23:08:48.575 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-21T23:08:48.590 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:08:48.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.695+0000 7f15c0488640 1 -- 172.21.15.44:0/2312578644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15b8102250 msgr2=0x7f15b8108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:48.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.695+0000 7f15c0488640 1 --2- 172.21.15.44:0/2312578644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15b8102250 0x7f15b8108ae0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f15b0009ca0 tx=0x7f15b002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:48.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.696+0000 7f15c0488640 1 -- 172.21.15.44:0/2312578644 shutdown_connections 2024-04-21T23:08:48.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.696+0000 7f15c0488640 1 --2- 172.21.15.44:0/2312578644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15b8102250 0x7f15b8108ae0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:48.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.696+0000 7f15c0488640 1 --2- 172.21.15.44:0/2312578644 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f15b8101890 0x7f15b8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:48.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.696+0000 7f15c0488640 1 --2- 172.21.15.44:0/2312578644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b81006a0 0x7f15b8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:48.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.696+0000 7f15c0488640 1 -- 172.21.15.44:0/2312578644 >> 172.21.15.44:0/2312578644 conn(0x7f15b80fbc10 msgr2=0x7f15b80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:48.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.696+0000 7f15c0488640 1 -- 172.21.15.44:0/2312578644 shutdown_connections 2024-04-21T23:08:48.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.696+0000 7f15c0488640 1 -- 172.21.15.44:0/2312578644 wait complete. 2024-04-21T23:08:48.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.696+0000 7f15c0488640 1 Processor -- start 2024-04-21T23:08:48.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.696+0000 7f15c0488640 1 -- start start 2024-04-21T23:08:48.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.696+0000 7f15c0488640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f15b81006a0 0x7f15b8104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:48.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.696+0000 7f15c0488640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15b8101890 0x7f15b8102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:48.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.696+0000 7f15c0488640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b8102250 0x7f15b81031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:48.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.696+0000 7f15c0488640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f15b810c320 con 0x7f15b8101890 2024-04-21T23:08:48.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.696+0000 7f15c0488640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f15b810c1a0 con 0x7f15b81006a0 2024-04-21T23:08:48.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.696+0000 7f15c0488640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f15b810c4a0 con 0x7f15b8102250 2024-04-21T23:08:48.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.697+0000 7f15be224640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f15b81006a0 0x7f15b8104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:48.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.697+0000 7f15be224640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f15b81006a0 0x7f15b8104630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:54192/0 (socket says 172.21.15.44:54192) 2024-04-21T23:08:48.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.697+0000 7f15be224640 1 -- 172.21.15.44:0/1814451516 learned_addr learned my addr 172.21.15.44:0/1814451516 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:48.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.697+0000 7f15bda23640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15b8101890 0x7f15b8102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:48.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.697+0000 7f15bea25640 1 --2- 172.21.15.44:0/1814451516 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b8102250 0x7f15b81031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:48.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.697+0000 7f15be224640 1 -- 172.21.15.44:0/1814451516 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b8102250 msgr2=0x7f15b81031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:48.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.697+0000 7f15be224640 1 --2- 172.21.15.44:0/1814451516 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b8102250 0x7f15b81031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:48.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.697+0000 7f15be224640 1 -- 172.21.15.44:0/1814451516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15b8101890 msgr2=0x7f15b8102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:48.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.697+0000 7f15be224640 1 --2- 172.21.15.44:0/1814451516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15b8101890 0x7f15b8102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.697+0000 7f15be224640 1 -- 172.21.15.44:0/1814451516 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b8103a40 con 0x7f15b81006a0 2024-04-21T23:08:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.697+0000 7f15be224640 1 --2- 172.21.15.44:0/1814451516 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f15b81006a0 0x7f15b8104630 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f15a800e9c0 tx=0x7f15a800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.697+0000 7f15af7fe640 1 -- 172.21.15.44:0/1814451516 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a800cd80 con 0x7f15b81006a0 2024-04-21T23:08:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.697+0000 7f15c0488640 1 -- 172.21.15.44:0/1814451516 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b81a1b70 con 0x7f15b81006a0 2024-04-21T23:08:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.697+0000 7f15af7fe640 1 -- 172.21.15.44:0/1814451516 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15a80035f0 con 0x7f15b81006a0 2024-04-21T23:08:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.697+0000 7f15af7fe640 1 -- 172.21.15.44:0/1814451516 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a8010690 con 0x7f15b81006a0 2024-04-21T23:08:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.697+0000 7f15c0488640 1 -- 172.21.15.44:0/1814451516 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b81a20c0 con 0x7f15b81006a0 2024-04-21T23:08:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.698+0000 7f15c0488640 1 -- 172.21.15.44:0/1814451516 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15b81a22f0 con 0x7f15b81006a0 2024-04-21T23:08:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.699+0000 7f15af7fe640 1 -- 172.21.15.44:0/1814451516 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f15a80037a0 con 0x7f15b81006a0 2024-04-21T23:08:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.700+0000 7f15af7fe640 1 --2- 172.21.15.44:0/1814451516 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f15a40a2be0 0x7f15a40a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.700+0000 7f15bda23640 1 -- 172.21.15.44:0/1814451516 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f15a40a2be0 msgr2=0x7f15a40a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/1417504446 2024-04-21T23:08:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.700+0000 7f15af7fe640 1 -- 172.21.15.44:0/1814451516 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7f15a80cf820 con 0x7f15b81006a0 2024-04-21T23:08:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.700+0000 7f15bda23640 1 --2- 172.21.15.44:0/1814451516 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f15a40a2be0 0x7f15a40a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:08:48.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.704+0000 7f15af7fe640 1 -- 172.21.15.44:0/1814451516 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f15a809c3c0 con 0x7f15b81006a0 2024-04-21T23:08:48.713 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:48 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:48.383+0000 7f33a5457200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T23:08:48.714 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:48 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:48.478+0000 7f33a5457200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T23:08:48.846 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:48 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:48.497+0000 7fa8c207f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T23:08:48.846 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:48 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:48.632+0000 7fa8c207f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T23:08:48.846 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:48 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:48.728+0000 7fa8c207f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T23:08:48.846 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:48 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:48.845+0000 7fa8c207f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T23:08:48.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.889+0000 7f15c0488640 1 -- 172.21.15.44:0/1814451516 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15b81a24c0 con 0x7f15b81006a0 2024-04-21T23:08:48.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.891+0000 7f15af7fe640 1 -- 172.21.15.44:0/1814451516 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24716 (secure 0 0 0) 0x7f15a809bcc0 con 0x7f15b81006a0 2024-04-21T23:08:48.891 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:48.892 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":94,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:08:25.093542+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":9,"to":11},{"from":5,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:08:48.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.893+0000 7f15c0488640 1 -- 172.21.15.44:0/1814451516 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f15a40a2be0 msgr2=0x7f15a40a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:08:48.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.893+0000 7f15c0488640 1 --2- 172.21.15.44:0/1814451516 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f15a40a2be0 0x7f15a40a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:48.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.893+0000 7f15c0488640 1 -- 172.21.15.44:0/1814451516 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f15b81006a0 msgr2=0x7f15b8104630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:48.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.893+0000 7f15c0488640 1 --2- 172.21.15.44:0/1814451516 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f15b81006a0 0x7f15b8104630 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f15a800e9c0 tx=0x7f15a800ee80 comp rx=0 tx=0).stop 2024-04-21T23:08:48.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.893+0000 7f15c0488640 1 -- 172.21.15.44:0/1814451516 shutdown_connections 2024-04-21T23:08:48.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.893+0000 7f15c0488640 1 --2- 172.21.15.44:0/1814451516 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f15a40a2be0 0x7f15a40a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:48.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.893+0000 7f15c0488640 1 --2- 172.21.15.44:0/1814451516 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b8102250 0x7f15b81031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:48.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.893+0000 7f15c0488640 1 --2- 172.21.15.44:0/1814451516 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f15b8101890 0x7f15b8102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:48.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.893+0000 7f15c0488640 1 --2- 172.21.15.44:0/1814451516 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f15b81006a0 0x7f15b8104630 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:48.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.893+0000 7f15c0488640 1 -- 172.21.15.44:0/1814451516 >> 172.21.15.44:0/1814451516 conn(0x7f15b80fbc10 msgr2=0x7f15b80fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:48.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.893+0000 7f15c0488640 1 -- 172.21.15.44:0/1814451516 shutdown_connections 2024-04-21T23:08:48.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:48.893+0000 7f15c0488640 1 -- 172.21.15.44:0/1814451516 wait complete. 2024-04-21T23:08:48.903 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:48 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:48.604+0000 7ff000fe2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T23:08:48.903 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:48 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:48.709+0000 7ff000fe2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T23:08:48.905 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:01:47.507503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:06:23.993760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:06:24.954661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:06:28.677618+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:08:48.905 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:08:49.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.012+0000 7f447ac78640 1 -- 172.21.15.44:0/3301992327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4474100680 msgr2=0x7f4474100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:49.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.012+0000 7f447ac78640 1 --2- 172.21.15.44:0/3301992327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4474100680 0x7f4474100aa0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4468009900 tx=0x7f446802bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:49.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.012+0000 7f447ac78640 1 -- 172.21.15.44:0/3301992327 shutdown_connections 2024-04-21T23:08:49.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.012+0000 7f447ac78640 1 --2- 172.21.15.44:0/3301992327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4474102230 0x7f4474108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.012+0000 7f447ac78640 1 --2- 172.21.15.44:0/3301992327 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4474101870 0x7f4474101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.012+0000 7f447ac78640 1 --2- 172.21.15.44:0/3301992327 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4474100680 0x7f4474100aa0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.012+0000 7f447ac78640 1 -- 172.21.15.44:0/3301992327 >> 172.21.15.44:0/3301992327 conn(0x7f44740fbc10 msgr2=0x7f44740fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:49.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.013+0000 7f447ac78640 1 -- 172.21.15.44:0/3301992327 shutdown_connections 2024-04-21T23:08:49.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.013+0000 7f447ac78640 1 -- 172.21.15.44:0/3301992327 wait complete. 2024-04-21T23:08:49.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.013+0000 7f447ac78640 1 Processor -- start 2024-04-21T23:08:49.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.013+0000 7f447ac78640 1 -- start start 2024-04-21T23:08:49.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.013+0000 7f447ac78640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4474100680 0x7f4474199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:49.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.013+0000 7f447ac78640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4474101870 0x7f447419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.013+0000 7f447ac78640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4474102230 0x7f44741a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.013+0000 7f447ac78640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f447410c300 con 0x7f4474102230 2024-04-21T23:08:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.013+0000 7f447ac78640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f447410c180 con 0x7f4474100680 2024-04-21T23:08:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.013+0000 7f447ac78640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f447410c480 con 0x7f4474101870 2024-04-21T23:08:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.013+0000 7f4479215640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4474102230 0x7f44741a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.013+0000 7f4478a14640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4474100680 0x7f4474199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.013+0000 7f4479215640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4474102230 0x7f44741a1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41010/0 (socket says 172.21.15.44:41010) 2024-04-21T23:08:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.013+0000 7f4478a14640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4474100680 0x7f4474199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:54212/0 (socket says 172.21.15.44:54212) 2024-04-21T23:08:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.014+0000 7f4479215640 1 -- 172.21.15.44:0/3376508503 learned_addr learned my addr 172.21.15.44:0/3376508503 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.014+0000 7f4473fff640 1 --2- 172.21.15.44:0/3376508503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4474101870 0x7f447419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.014+0000 7f4479215640 1 -- 172.21.15.44:0/3376508503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4474101870 msgr2=0x7f447419a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.014+0000 7f4479215640 1 --2- 172.21.15.44:0/3376508503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4474101870 0x7f447419a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.014+0000 7f4479215640 1 -- 172.21.15.44:0/3376508503 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4474100680 msgr2=0x7f4474199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:49.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.014+0000 7f4479215640 1 --2- 172.21.15.44:0/3376508503 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4474100680 0x7f4474199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.014+0000 7f4479215640 1 -- 172.21.15.44:0/3376508503 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44741a18e0 con 0x7f4474102230 2024-04-21T23:08:49.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.014+0000 7f4478a14640 1 --2- 172.21.15.44:0/3376508503 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4474100680 0x7f4474199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:08:49.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.014+0000 7f4479215640 1 --2- 172.21.15.44:0/3376508503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4474102230 0x7f44741a1240 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f446c00eb10 tx=0x7f446c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:49.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.014+0000 7f4471ffb640 1 -- 172.21.15.44:0/3376508503 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c004b20 con 0x7f4474102230 2024-04-21T23:08:49.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.014+0000 7f447ac78640 1 -- 172.21.15.44:0/3376508503 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44741a1bd0 con 0x7f4474102230 2024-04-21T23:08:49.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.014+0000 7f4471ffb640 1 -- 172.21.15.44:0/3376508503 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f446c004cc0 con 0x7f4474102230 2024-04-21T23:08:49.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.014+0000 7f4471ffb640 1 -- 172.21.15.44:0/3376508503 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c010760 con 0x7f4474102230 2024-04-21T23:08:49.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.014+0000 7f447ac78640 1 -- 172.21.15.44:0/3376508503 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44741a2140 con 0x7f4474102230 2024-04-21T23:08:49.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.016+0000 7f447ac78640 1 -- 172.21.15.44:0/3376508503 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44741a2370 con 0x7f4474102230 2024-04-21T23:08:49.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.017+0000 7f4471ffb640 1 -- 172.21.15.44:0/3376508503 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f446c005000 con 0x7f4474102230 2024-04-21T23:08:49.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.018+0000 7f4471ffb640 1 --2- 172.21.15.44:0/3376508503 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f44640a2be0 0x7f44640a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:49.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.018+0000 7f4478a14640 1 -- 172.21.15.44:0/3376508503 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f44640a2be0 msgr2=0x7f44640a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/1417504446 2024-04-21T23:08:49.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.018+0000 7f4471ffb640 1 -- 172.21.15.44:0/3376508503 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7f446c0cfec0 con 0x7f4474102230 2024-04-21T23:08:49.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.018+0000 7f4478a14640 1 --2- 172.21.15.44:0/3376508503 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f44640a2be0 0x7f44640a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:08:49.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.021+0000 7f4471ffb640 1 -- 172.21.15.44:0/3376508503 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f446c09ca00 con 0x7f4474102230 2024-04-21T23:08:49.088 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:48 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1814451516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:49.089 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:48 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:48.731+0000 7f33a5457200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T23:08:49.089 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:48 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:48.870+0000 7f33a5457200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T23:08:49.089 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:48 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:48.969+0000 7f33a5457200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T23:08:49.089 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:49 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:49.089+0000 7f33a5457200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T23:08:49.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:48 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1814451516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:49.185 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:48 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:48.971+0000 7ff000fe2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T23:08:49.186 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:49 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:49.119+0000 7ff000fe2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T23:08:49.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.206+0000 7f447ac78640 1 -- 172.21.15.44:0/3376508503 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f44741a2540 con 0x7f4474102230 2024-04-21T23:08:49.205 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.206+0000 7f4471ffb640 1 -- 172.21.15.44:0/3376508503 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f446c09c300 con 0x7f4474102230 2024-04-21T23:08:49.206 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-21T23:08:49.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.208+0000 7f447ac78640 1 -- 172.21.15.44:0/3376508503 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f44640a2be0 msgr2=0x7f44640a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:08:49.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.208+0000 7f447ac78640 1 --2- 172.21.15.44:0/3376508503 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f44640a2be0 0x7f44640a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.208+0000 7f447ac78640 1 -- 172.21.15.44:0/3376508503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4474102230 msgr2=0x7f44741a1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:49.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.208+0000 7f447ac78640 1 --2- 172.21.15.44:0/3376508503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4474102230 0x7f44741a1240 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f446c00eb10 tx=0x7f446c00c2d0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.208+0000 7f447ac78640 1 -- 172.21.15.44:0/3376508503 shutdown_connections 2024-04-21T23:08:49.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.208+0000 7f447ac78640 1 --2- 172.21.15.44:0/3376508503 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f44640a2be0 0x7f44640a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.208+0000 7f447ac78640 1 --2- 172.21.15.44:0/3376508503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4474101870 0x7f447419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.208+0000 7f447ac78640 1 --2- 172.21.15.44:0/3376508503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4474102230 0x7f44741a1240 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.208+0000 7f447ac78640 1 --2- 172.21.15.44:0/3376508503 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4474100680 0x7f4474199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.208+0000 7f447ac78640 1 -- 172.21.15.44:0/3376508503 >> 172.21.15.44:0/3376508503 conn(0x7f44740fbc10 msgr2=0x7f44740fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:49.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.208+0000 7f447ac78640 1 -- 172.21.15.44:0/3376508503 shutdown_connections 2024-04-21T23:08:49.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.208+0000 7f447ac78640 1 -- 172.21.15.44:0/3376508503 wait complete. 2024-04-21T23:08:49.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:48 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1814451516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:49.214 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:48 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:48.938+0000 7fa8c207f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T23:08:49.219 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:08:49.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.326+0000 7f56fd13a640 1 -- 172.21.15.44:0/3169723254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f8104b90 msgr2=0x7f56f8106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:49.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.326+0000 7f56fd13a640 1 --2- 172.21.15.44:0/3169723254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f8104b90 0x7f56f8106f80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f56e8009900 tx=0x7f56e802bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:49.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.326+0000 7f56fd13a640 1 -- 172.21.15.44:0/3169723254 shutdown_connections 2024-04-21T23:08:49.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.326+0000 7f56fd13a640 1 --2- 172.21.15.44:0/3169723254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f81074c0 0x7f56f81098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.326+0000 7f56fd13a640 1 --2- 172.21.15.44:0/3169723254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f8104b90 0x7f56f8106f80 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.326+0000 7f56fd13a640 1 --2- 172.21.15.44:0/3169723254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56f8102260 0x7f56f8104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.326+0000 7f56fd13a640 1 -- 172.21.15.44:0/3169723254 >> 172.21.15.44:0/3169723254 conn(0x7f56f80fbc10 msgr2=0x7f56f80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:49.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.326+0000 7f56fd13a640 1 -- 172.21.15.44:0/3169723254 shutdown_connections 2024-04-21T23:08:49.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.326+0000 7f56fd13a640 1 -- 172.21.15.44:0/3169723254 wait complete. 2024-04-21T23:08:49.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.327+0000 7f56fd13a640 1 Processor -- start 2024-04-21T23:08:49.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.327+0000 7f56fd13a640 1 -- start start 2024-04-21T23:08:49.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.327+0000 7f56fd13a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f8102260 0x7f56f8197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:49.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.327+0000 7f56fd13a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56f8104b90 0x7f56f8198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:49.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.327+0000 7f56fd13a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f81074c0 0x7f56f819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:49.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.327+0000 7f56fd13a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f56f810d400 con 0x7f56f81074c0 2024-04-21T23:08:49.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.327+0000 7f56fd13a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f56f810d280 con 0x7f56f8104b90 2024-04-21T23:08:49.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.327+0000 7f56fd13a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f56f810d580 con 0x7f56f8102260 2024-04-21T23:08:49.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.328+0000 7f56f759e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f81074c0 0x7f56f819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:49.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.328+0000 7f56f759e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f81074c0 0x7f56f819f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41020/0 (socket says 172.21.15.44:41020) 2024-04-21T23:08:49.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.328+0000 7f56f759e640 1 -- 172.21.15.44:0/2735205502 learned_addr learned my addr 172.21.15.44:0/2735205502 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:49.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.328+0000 7f56f6d9d640 1 --2- 172.21.15.44:0/2735205502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f8102260 0x7f56f8197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:49.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.328+0000 7f56f659c640 1 --2- 172.21.15.44:0/2735205502 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56f8104b90 0x7f56f8198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:49.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.328+0000 7f56f759e640 1 -- 172.21.15.44:0/2735205502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f8102260 msgr2=0x7f56f8197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:49.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.328+0000 7f56f759e640 1 --2- 172.21.15.44:0/2735205502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f8102260 0x7f56f8197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.328+0000 7f56f759e640 1 -- 172.21.15.44:0/2735205502 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56f8104b90 msgr2=0x7f56f8198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:49.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.328+0000 7f56f759e640 1 --2- 172.21.15.44:0/2735205502 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56f8104b90 0x7f56f8198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.328+0000 7f56f759e640 1 -- 172.21.15.44:0/2735205502 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f819f720 con 0x7f56f81074c0 2024-04-21T23:08:49.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.328+0000 7f56f6d9d640 1 --2- 172.21.15.44:0/2735205502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f8102260 0x7f56f8197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:08:49.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.328+0000 7f56f759e640 1 --2- 172.21.15.44:0/2735205502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f81074c0 0x7f56f819f0b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f56f000eb10 tx=0x7f56f000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:49.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.328+0000 7f56effff640 1 -- 172.21.15.44:0/2735205502 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f0004b20 con 0x7f56f81074c0 2024-04-21T23:08:49.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.328+0000 7f56fd13a640 1 -- 172.21.15.44:0/2735205502 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56f819fa10 con 0x7f56f81074c0 2024-04-21T23:08:49.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.328+0000 7f56effff640 1 -- 172.21.15.44:0/2735205502 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56f0004cc0 con 0x7f56f81074c0 2024-04-21T23:08:49.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.328+0000 7f56effff640 1 -- 172.21.15.44:0/2735205502 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f0010760 con 0x7f56f81074c0 2024-04-21T23:08:49.330 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.328+0000 7f56fd13a640 1 -- 172.21.15.44:0/2735205502 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56f819ff30 con 0x7f56f81074c0 2024-04-21T23:08:49.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.330+0000 7f56fd13a640 1 -- 172.21.15.44:0/2735205502 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f80fd700 con 0x7f56f81074c0 2024-04-21T23:08:49.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.331+0000 7f56effff640 1 -- 172.21.15.44:0/2735205502 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f56f0005020 con 0x7f56f81074c0 2024-04-21T23:08:49.331 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.331+0000 7f56effff640 1 --2- 172.21.15.44:0/2735205502 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f56e40a2be0 0x7f56e40a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:49.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.331+0000 7f56f6d9d640 1 -- 172.21.15.44:0/2735205502 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f56e40a2be0 msgr2=0x7f56e40a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/1417504446 2024-04-21T23:08:49.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.332+0000 7f56effff640 1 -- 172.21.15.44:0/2735205502 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7f56f00cfb00 con 0x7f56f81074c0 2024-04-21T23:08:49.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.332+0000 7f56f6d9d640 1 --2- 172.21.15.44:0/2735205502 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f56e40a2be0 0x7f56e40a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:08:49.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.336+0000 7f56effff640 1 -- 172.21.15.44:0/2735205502 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f56f009c6a0 con 0x7f56f81074c0 2024-04-21T23:08:49.464 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:49 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:49.184+0000 7f33a5457200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T23:08:49.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.523+0000 7f56fd13a640 1 -- 172.21.15.44:0/2735205502 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f56f802d1c0 con 0x7f56f81074c0 2024-04-21T23:08:49.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.524+0000 7f56effff640 1 -- 172.21.15.44:0/2735205502 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f56f009bfa0 con 0x7f56f81074c0 2024-04-21T23:08:49.523 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:08:49.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.526+0000 7f56fd13a640 1 -- 172.21.15.44:0/2735205502 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f56e40a2be0 msgr2=0x7f56e40a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:08:49.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.526+0000 7f56fd13a640 1 --2- 172.21.15.44:0/2735205502 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f56e40a2be0 0x7f56e40a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.526+0000 7f56fd13a640 1 -- 172.21.15.44:0/2735205502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f81074c0 msgr2=0x7f56f819f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:49.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.526+0000 7f56fd13a640 1 --2- 172.21.15.44:0/2735205502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f81074c0 0x7f56f819f0b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f56f000eb10 tx=0x7f56f000c2d0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.526+0000 7f56fd13a640 1 -- 172.21.15.44:0/2735205502 shutdown_connections 2024-04-21T23:08:49.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.526+0000 7f56fd13a640 1 --2- 172.21.15.44:0/2735205502 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f56e40a2be0 0x7f56e40a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.526+0000 7f56fd13a640 1 --2- 172.21.15.44:0/2735205502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f56f81074c0 0x7f56f819f0b0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.526+0000 7f56fd13a640 1 --2- 172.21.15.44:0/2735205502 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f56f8104b90 0x7f56f8198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.526+0000 7f56fd13a640 1 --2- 172.21.15.44:0/2735205502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f56f8102260 0x7f56f8197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.526+0000 7f56fd13a640 1 -- 172.21.15.44:0/2735205502 >> 172.21.15.44:0/2735205502 conn(0x7f56f80fbc10 msgr2=0x7f56f8102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:49.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.526+0000 7f56fd13a640 1 -- 172.21.15.44:0/2735205502 shutdown_connections 2024-04-21T23:08:49.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.526+0000 7f56fd13a640 1 -- 172.21.15.44:0/2735205502 wait complete. 2024-04-21T23:08:49.536 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:49 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:49.223+0000 7ff000fe2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T23:08:49.536 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:49 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:49.350+0000 7ff000fe2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T23:08:49.536 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:49 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:49.450+0000 7ff000fe2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T23:08:49.538 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:08:49.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.647+0000 7fcc86fc9640 1 -- 172.21.15.44:0/1444982690 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc80100680 msgr2=0x7fcc80100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.647+0000 7fcc86fc9640 1 --2- 172.21.15.44:0/1444982690 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc80100680 0x7fcc80100aa0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fcc74009900 tx=0x7fcc7402bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.647+0000 7fcc86fc9640 1 -- 172.21.15.44:0/1444982690 shutdown_connections 2024-04-21T23:08:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.647+0000 7fcc86fc9640 1 --2- 172.21.15.44:0/1444982690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc80102230 0x7fcc80108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.647+0000 7fcc86fc9640 1 --2- 172.21.15.44:0/1444982690 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc80101870 0x7fcc80101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.647+0000 7fcc86fc9640 1 --2- 172.21.15.44:0/1444982690 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc80100680 0x7fcc80100aa0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.647+0000 7fcc86fc9640 1 -- 172.21.15.44:0/1444982690 >> 172.21.15.44:0/1444982690 conn(0x7fcc800fbc10 msgr2=0x7fcc800fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.647+0000 7fcc86fc9640 1 -- 172.21.15.44:0/1444982690 shutdown_connections 2024-04-21T23:08:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.647+0000 7fcc86fc9640 1 -- 172.21.15.44:0/1444982690 wait complete. 2024-04-21T23:08:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.648+0000 7fcc86fc9640 1 Processor -- start 2024-04-21T23:08:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.648+0000 7fcc86fc9640 1 -- start start 2024-04-21T23:08:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.648+0000 7fcc86fc9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc80100680 0x7fcc80197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.648+0000 7fcc86fc9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc80101870 0x7fcc801980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:49.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.648+0000 7fcc86fc9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc80102230 0x7fcc8019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.648+0000 7fcc86fc9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8010c2d0 con 0x7fcc80100680 2024-04-21T23:08:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.648+0000 7fcc86fc9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8010c150 con 0x7fcc80101870 2024-04-21T23:08:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.648+0000 7fcc86fc9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcc8010c450 con 0x7fcc80102230 2024-04-21T23:08:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.648+0000 7fcc84d65640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc80100680 0x7fcc80197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.648+0000 7fcc84d65640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc80100680 0x7fcc80197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41042/0 (socket says 172.21.15.44:41042) 2024-04-21T23:08:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.648+0000 7fcc84d65640 1 -- 172.21.15.44:0/3571483267 learned_addr learned my addr 172.21.15.44:0/3571483267 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.648+0000 7fcc85566640 1 --2- 172.21.15.44:0/3571483267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc80102230 0x7fcc8019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.648+0000 7fcc7ffff640 1 --2- 172.21.15.44:0/3571483267 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc80101870 0x7fcc801980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.649+0000 7fcc84d65640 1 -- 172.21.15.44:0/3571483267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc80102230 msgr2=0x7fcc8019f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.649+0000 7fcc84d65640 1 --2- 172.21.15.44:0/3571483267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc80102230 0x7fcc8019f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.649+0000 7fcc84d65640 1 -- 172.21.15.44:0/3571483267 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc80101870 msgr2=0x7fcc801980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.649+0000 7fcc84d65640 1 --2- 172.21.15.44:0/3571483267 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc80101870 0x7fcc801980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.649+0000 7fcc84d65640 1 -- 172.21.15.44:0/3571483267 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc8019f700 con 0x7fcc80100680 2024-04-21T23:08:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.649+0000 7fcc84d65640 1 --2- 172.21.15.44:0/3571483267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc80100680 0x7fcc80197b80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fcc74000c00 tx=0x7fcc7402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:49.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.649+0000 7fcc7dffb640 1 -- 172.21.15.44:0/3571483267 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc74041070 con 0x7fcc80100680 2024-04-21T23:08:49.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.649+0000 7fcc7dffb640 1 -- 172.21.15.44:0/3571483267 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc74004450 con 0x7fcc80100680 2024-04-21T23:08:49.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.649+0000 7fcc86fc9640 1 -- 172.21.15.44:0/3571483267 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc8019f990 con 0x7fcc80100680 2024-04-21T23:08:49.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.649+0000 7fcc7dffb640 1 -- 172.21.15.44:0/3571483267 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc74034440 con 0x7fcc80100680 2024-04-21T23:08:49.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.649+0000 7fcc86fc9640 1 -- 172.21.15.44:0/3571483267 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc8019fed0 con 0x7fcc80100680 2024-04-21T23:08:49.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.650+0000 7fcc86fc9640 1 -- 172.21.15.44:0/3571483267 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc801a00a0 con 0x7fcc80100680 2024-04-21T23:08:49.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.651+0000 7fcc7dffb640 1 -- 172.21.15.44:0/3571483267 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7fcc74044070 con 0x7fcc80100680 2024-04-21T23:08:49.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.652+0000 7fcc7dffb640 1 --2- 172.21.15.44:0/3571483267 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcc700a2c60 0x7fcc700a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:49.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.652+0000 7fcc7ffff640 1 -- 172.21.15.44:0/3571483267 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcc700a2c60 msgr2=0x7fcc700a5120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/1417504446 2024-04-21T23:08:49.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.652+0000 7fcc7dffb640 1 -- 172.21.15.44:0/3571483267 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7fcc740f0ee0 con 0x7fcc80100680 2024-04-21T23:08:49.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.652+0000 7fcc7ffff640 1 --2- 172.21.15.44:0/3571483267 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcc700a2c60 0x7fcc700a5120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:08:49.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.656+0000 7fcc7dffb640 1 -- 172.21.15.44:0/3571483267 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fcc740bd460 con 0x7fcc80100680 2024-04-21T23:08:49.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.844+0000 7fcc86fc9640 1 -- 172.21.15.44:0/3571483267 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcc801a0430 con 0x7fcc80100680 2024-04-21T23:08:49.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.844+0000 7fcc7dffb640 1 -- 172.21.15.44:0/3571483267 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fcc740c1760 con 0x7fcc80100680 2024-04-21T23:08:49.844 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-21T23:08:49.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.847+0000 7fcc86fc9640 1 -- 172.21.15.44:0/3571483267 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcc700a2c60 msgr2=0x7fcc700a5120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:08:49.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.847+0000 7fcc86fc9640 1 --2- 172.21.15.44:0/3571483267 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcc700a2c60 0x7fcc700a5120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.847+0000 7fcc86fc9640 1 -- 172.21.15.44:0/3571483267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc80100680 msgr2=0x7fcc80197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:49.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.847+0000 7fcc86fc9640 1 --2- 172.21.15.44:0/3571483267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc80100680 0x7fcc80197b80 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fcc74000c00 tx=0x7fcc7402c640 comp rx=0 tx=0).stop 2024-04-21T23:08:49.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.847+0000 7fcc86fc9640 1 -- 172.21.15.44:0/3571483267 shutdown_connections 2024-04-21T23:08:49.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.847+0000 7fcc86fc9640 1 --2- 172.21.15.44:0/3571483267 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7fcc700a2c60 0x7fcc700a5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.847+0000 7fcc86fc9640 1 --2- 172.21.15.44:0/3571483267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc80102230 0x7fcc8019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.847+0000 7fcc86fc9640 1 --2- 172.21.15.44:0/3571483267 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcc80101870 0x7fcc801980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.847+0000 7fcc86fc9640 1 --2- 172.21.15.44:0/3571483267 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcc80100680 0x7fcc80197b80 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.847+0000 7fcc86fc9640 1 -- 172.21.15.44:0/3571483267 >> 172.21.15.44:0/3571483267 conn(0x7fcc800fbc10 msgr2=0x7fcc80068a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:49.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.847+0000 7fcc86fc9640 1 -- 172.21.15.44:0/3571483267 shutdown_connections 2024-04-21T23:08:49.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.847+0000 7fcc86fc9640 1 -- 172.21.15.44:0/3571483267 wait complete. 2024-04-21T23:08:49.857 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:08:49.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.965+0000 7f70ef937640 1 -- 172.21.15.44:0/124308377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70e81009b0 msgr2=0x7f70e8100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:49.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.965+0000 7f70ef937640 1 --2- 172.21.15.44:0/124308377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70e81009b0 0x7f70e8100e10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f70e4009900 tx=0x7f70e402bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:49.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.965+0000 7f70ef937640 1 -- 172.21.15.44:0/124308377 shutdown_connections 2024-04-21T23:08:49.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.965+0000 7f70ef937640 1 --2- 172.21.15.44:0/124308377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70e81009b0 0x7f70e8100e10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.965+0000 7f70ef937640 1 --2- 172.21.15.44:0/124308377 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70e8101350 0x7f70e81017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.965+0000 7f70ef937640 1 --2- 172.21.15.44:0/124308377 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70e8100050 0x7f70e8100470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.965+0000 7f70ef937640 1 -- 172.21.15.44:0/124308377 >> 172.21.15.44:0/124308377 conn(0x7f70e80fbc10 msgr2=0x7f70e80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:49.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.965+0000 7f70ef937640 1 -- 172.21.15.44:0/124308377 shutdown_connections 2024-04-21T23:08:49.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.965+0000 7f70ef937640 1 -- 172.21.15.44:0/124308377 wait complete. 2024-04-21T23:08:49.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.965+0000 7f70ef937640 1 Processor -- start 2024-04-21T23:08:49.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.965+0000 7f70ef937640 1 -- start start 2024-04-21T23:08:49.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70ef937640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70e8100050 0x7f70e8199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:49.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70ef937640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70e81009b0 0x7f70e819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:49.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70ef937640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70e8101350 0x7f70e81a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:49.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70ef937640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f70e810f5e0 con 0x7f70e8101350 2024-04-21T23:08:49.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70ef937640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f70e810f460 con 0x7f70e8100050 2024-04-21T23:08:49.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70ef937640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f70e810f760 con 0x7f70e81009b0 2024-04-21T23:08:49.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70eded4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70e8101350 0x7f70e81a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:49.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70eced2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70e81009b0 0x7f70e819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:49.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70ed6d3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70e8100050 0x7f70e8199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:49.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70eced2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70e81009b0 0x7f70e819a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:49860/0 (socket says 172.21.15.44:49860) 2024-04-21T23:08:49.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70eced2640 1 -- 172.21.15.44:0/3122825118 learned_addr learned my addr 172.21.15.44:0/3122825118 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:49.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70eded4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70e8101350 0x7f70e81a1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41050/0 (socket says 172.21.15.44:41050) 2024-04-21T23:08:49.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70eced2640 1 -- 172.21.15.44:0/3122825118 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70e8100050 msgr2=0x7f70e8199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:49.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70eced2640 1 --2- 172.21.15.44:0/3122825118 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70e8100050 0x7f70e8199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70eced2640 1 -- 172.21.15.44:0/3122825118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70e8101350 msgr2=0x7f70e81a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:49.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70eced2640 1 --2- 172.21.15.44:0/3122825118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70e8101350 0x7f70e81a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:49.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70eced2640 1 -- 172.21.15.44:0/3122825118 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70e81a18f0 con 0x7f70e81009b0 2024-04-21T23:08:49.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70eced2640 1 --2- 172.21.15.44:0/3122825118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70e81009b0 0x7f70e819a1f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f70e4009230 tx=0x7f70e402c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:49.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70de7fc640 1 -- 172.21.15.44:0/3122825118 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e4041070 con 0x7f70e81009b0 2024-04-21T23:08:49.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70ef937640 1 -- 172.21.15.44:0/3122825118 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70e81a1b80 con 0x7f70e81009b0 2024-04-21T23:08:49.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70de7fc640 1 -- 172.21.15.44:0/3122825118 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70e4004450 con 0x7f70e81009b0 2024-04-21T23:08:49.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.966+0000 7f70de7fc640 1 -- 172.21.15.44:0/3122825118 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e4003e20 con 0x7f70e81009b0 2024-04-21T23:08:49.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.967+0000 7f70ef937640 1 -- 172.21.15.44:0/3122825118 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70e81a2070 con 0x7f70e81009b0 2024-04-21T23:08:49.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.968+0000 7f70ef937640 1 -- 172.21.15.44:0/3122825118 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70e810af30 con 0x7f70e81009b0 2024-04-21T23:08:49.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.969+0000 7f70de7fc640 1 -- 172.21.15.44:0/3122825118 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f70e4044070 con 0x7f70e81009b0 2024-04-21T23:08:49.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.969+0000 7f70de7fc640 1 --2- 172.21.15.44:0/3122825118 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f70d40a2be0 0x7f70d40a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:49.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.969+0000 7f70ed6d3640 1 -- 172.21.15.44:0/3122825118 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f70d40a2be0 msgr2=0x7f70d40a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/1417504446 2024-04-21T23:08:49.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.969+0000 7f70de7fc640 1 -- 172.21.15.44:0/3122825118 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7f70e40f0fa0 con 0x7f70e81009b0 2024-04-21T23:08:49.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.969+0000 7f70ed6d3640 1 --2- 172.21.15.44:0/3122825118 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f70d40a2be0 0x7f70d40a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:08:49.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:49.977+0000 7f70de7fc640 1 -- 172.21.15.44:0/3122825118 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f70e40bd4f0 con 0x7f70e81009b0 2024-04-21T23:08:50.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.170+0000 7f70ed6d3640 1 -- 172.21.15.44:0/3122825118 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f70d40a2be0 msgr2=0x7f70d40a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/1417504446 2024-04-21T23:08:50.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.170+0000 7f70ed6d3640 1 --2- 172.21.15.44:0/3122825118 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f70d40a2be0 0x7f70d40a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T23:08:50.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:49 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3376508503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:50.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:49 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2735205502' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:50.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:49 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3571483267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:50.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.190+0000 7f70ef937640 1 -- 172.21.15.44:0/3122825118 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f70e802d170 con 0x7f70e81009b0 2024-04-21T23:08:50.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.190+0000 7f70de7fc640 1 -- 172.21.15.44:0/3122825118 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f70e40c17f0 con 0x7f70e81009b0 2024-04-21T23:08:50.190 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:08:50.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.192+0000 7f70ef937640 1 -- 172.21.15.44:0/3122825118 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f70d40a2be0 msgr2=0x7f70d40a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:08:50.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.192+0000 7f70ef937640 1 --2- 172.21.15.44:0/3122825118 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f70d40a2be0 0x7f70d40a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.192+0000 7f70ef937640 1 -- 172.21.15.44:0/3122825118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70e81009b0 msgr2=0x7f70e819a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:50.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.192+0000 7f70ef937640 1 --2- 172.21.15.44:0/3122825118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70e81009b0 0x7f70e819a1f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f70e4009230 tx=0x7f70e402c650 comp rx=0 tx=0).stop 2024-04-21T23:08:50.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.192+0000 7f70ef937640 1 -- 172.21.15.44:0/3122825118 shutdown_connections 2024-04-21T23:08:50.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.192+0000 7f70ef937640 1 --2- 172.21.15.44:0/3122825118 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f70d40a2be0 0x7f70d40a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.192+0000 7f70ef937640 1 --2- 172.21.15.44:0/3122825118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70e81009b0 0x7f70e819a1f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.192 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.192+0000 7f70ef937640 1 --2- 172.21.15.44:0/3122825118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f70e8101350 0x7f70e81a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.192+0000 7f70ef937640 1 --2- 172.21.15.44:0/3122825118 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f70e8100050 0x7f70e8199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.192+0000 7f70ef937640 1 -- 172.21.15.44:0/3122825118 >> 172.21.15.44:0/3122825118 conn(0x7f70e80fbc10 msgr2=0x7f70e80fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:50.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.192+0000 7f70ef937640 1 -- 172.21.15.44:0/3122825118 shutdown_connections 2024-04-21T23:08:50.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.192+0000 7f70ef937640 1 -- 172.21.15.44:0/3122825118 wait complete. 2024-04-21T23:08:50.203 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:08:50.214 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:50.033+0000 7f33a5457200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T23:08:50.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:49 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3376508503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:50.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:49 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2735205502' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:50.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:49 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3571483267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:50.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:49 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3376508503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:50.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:49 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2735205502' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:50.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:49 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3571483267' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:50.215 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:49 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:49.751+0000 7fa8c207f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T23:08:50.215 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:08:49 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z[33714]: 2024-04-21T23:08:49.974+0000 7fa8c207f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T23:08:50.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.307+0000 7f381333a640 1 -- 172.21.15.44:0/3726008061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f380c101890 msgr2=0x7f380c101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:50.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.307+0000 7f381333a640 1 --2- 172.21.15.44:0/3726008061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f380c101890 0x7f380c101d10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3800009900 tx=0x7f380002bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:50.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.307+0000 7f381333a640 1 -- 172.21.15.44:0/3726008061 shutdown_connections 2024-04-21T23:08:50.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.307+0000 7f381333a640 1 --2- 172.21.15.44:0/3726008061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f380c102250 0x7f380c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.307+0000 7f381333a640 1 --2- 172.21.15.44:0/3726008061 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f380c101890 0x7f380c101d10 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.307+0000 7f381333a640 1 --2- 172.21.15.44:0/3726008061 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f380c1006a0 0x7f380c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.307+0000 7f381333a640 1 -- 172.21.15.44:0/3726008061 >> 172.21.15.44:0/3726008061 conn(0x7f380c0fbc10 msgr2=0x7f380c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.307+0000 7f381333a640 1 -- 172.21.15.44:0/3726008061 shutdown_connections 2024-04-21T23:08:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.307+0000 7f381333a640 1 -- 172.21.15.44:0/3726008061 wait complete. 2024-04-21T23:08:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.308+0000 7f381333a640 1 Processor -- start 2024-04-21T23:08:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.308+0000 7f381333a640 1 -- start start 2024-04-21T23:08:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.308+0000 7f381333a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f380c1006a0 0x7f380c0695b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.308+0000 7f381333a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f380c101890 0x7f380c19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.308+0000 7f381333a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f380c102250 0x7f380c19ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.308+0000 7f381333a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f380c10c350 con 0x7f380c102250 2024-04-21T23:08:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.308+0000 7f381333a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f380c10c1d0 con 0x7f380c1006a0 2024-04-21T23:08:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.308+0000 7f381333a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f380c10c4d0 con 0x7f380c101890 2024-04-21T23:08:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.308+0000 7f38118d7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f380c102250 0x7f380c19ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.308+0000 7f38118d7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f380c102250 0x7f380c19ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:41058/0 (socket says 172.21.15.44:41058) 2024-04-21T23:08:50.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.308+0000 7f38118d7640 1 -- 172.21.15.44:0/3232118884 learned_addr learned my addr 172.21.15.44:0/3232118884 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.308+0000 7f38108d5640 1 --2- 172.21.15.44:0/3232118884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f380c101890 0x7f380c19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.308+0000 7f38110d6640 1 --2- 172.21.15.44:0/3232118884 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f380c1006a0 0x7f380c0695b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.309+0000 7f38118d7640 1 -- 172.21.15.44:0/3232118884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f380c101890 msgr2=0x7f380c19c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.309+0000 7f38118d7640 1 --2- 172.21.15.44:0/3232118884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f380c101890 0x7f380c19c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.309+0000 7f38118d7640 1 -- 172.21.15.44:0/3232118884 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f380c1006a0 msgr2=0x7f380c0695b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.309+0000 7f38118d7640 1 --2- 172.21.15.44:0/3232118884 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f380c1006a0 0x7f380c0695b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.309+0000 7f38118d7640 1 -- 172.21.15.44:0/3232118884 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f380c19f6f0 con 0x7f380c102250 2024-04-21T23:08:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.309+0000 7f38118d7640 1 --2- 172.21.15.44:0/3232118884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f380c102250 0x7f380c19ede0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f3804009fd0 tx=0x7f380400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.309+0000 7f37fe7fc640 1 -- 172.21.15.44:0/3232118884 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3804017070 con 0x7f380c102250 2024-04-21T23:08:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.309+0000 7f381333a640 1 -- 172.21.15.44:0/3232118884 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f380c19f980 con 0x7f380c102250 2024-04-21T23:08:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.309+0000 7f37fe7fc640 1 -- 172.21.15.44:0/3232118884 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3804003e50 con 0x7f380c102250 2024-04-21T23:08:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.309+0000 7f37fe7fc640 1 -- 172.21.15.44:0/3232118884 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38040106c0 con 0x7f380c102250 2024-04-21T23:08:50.311 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.309+0000 7f381333a640 1 -- 172.21.15.44:0/3232118884 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f380c19fe50 con 0x7f380c102250 2024-04-21T23:08:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.310+0000 7f381333a640 1 -- 172.21.15.44:0/3232118884 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f380c107960 con 0x7f380c102250 2024-04-21T23:08:50.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.311+0000 7f37fe7fc640 1 -- 172.21.15.44:0/3232118884 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f3804003970 con 0x7f380c102250 2024-04-21T23:08:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.312+0000 7f37fe7fc640 1 --2- 172.21.15.44:0/3232118884 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f37f80a2be0 0x7f37f80a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.312+0000 7f38110d6640 1 -- 172.21.15.44:0/3232118884 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f37f80a2be0 msgr2=0x7f37f80a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.44:6800/1417504446 2024-04-21T23:08:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.312+0000 7f37fe7fc640 1 -- 172.21.15.44:0/3232118884 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9187+0+0 (secure 0 0 0) 0x7f38040cf1d0 con 0x7f380c102250 2024-04-21T23:08:50.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.312+0000 7f38110d6640 1 --2- 172.21.15.44:0/3232118884 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f37f80a2be0 0x7f37f80a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T23:08:50.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.317+0000 7f37fe7fc640 1 -- 172.21.15.44:0/3232118884 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f380409b6c0 con 0x7f380c102250 2024-04-21T23:08:50.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.503+0000 7f381333a640 1 -- 172.21.15.44:0/3232118884 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f380c062370 con 0x7f380c102250 2024-04-21T23:08:50.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.505+0000 7f37fe7fc640 1 -- 172.21.15.44:0/3232118884 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24716 (secure 0 0 0) 0x7f380409f9c0 con 0x7f380c102250 2024-04-21T23:08:50.505 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:50.505 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":94,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:08:25.093542+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":9,"to":11},{"from":5,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:08:50.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.508+0000 7f381333a640 1 -- 172.21.15.44:0/3232118884 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f37f80a2be0 msgr2=0x7f37f80a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:08:50.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.508+0000 7f381333a640 1 --2- 172.21.15.44:0/3232118884 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f37f80a2be0 0x7f37f80a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.508+0000 7f381333a640 1 -- 172.21.15.44:0/3232118884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f380c102250 msgr2=0x7f380c19ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:50.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.508+0000 7f381333a640 1 --2- 172.21.15.44:0/3232118884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f380c102250 0x7f380c19ede0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f3804009fd0 tx=0x7f380400b260 comp rx=0 tx=0).stop 2024-04-21T23:08:50.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.508+0000 7f381333a640 1 -- 172.21.15.44:0/3232118884 shutdown_connections 2024-04-21T23:08:50.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.508+0000 7f381333a640 1 --2- 172.21.15.44:0/3232118884 >> [v2:172.21.15.44:6800/1417504446,v1:172.21.15.44:6801/1417504446] conn(0x7f37f80a2be0 0x7f37f80a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.508+0000 7f381333a640 1 --2- 172.21.15.44:0/3232118884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f380c101890 0x7f380c19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.508+0000 7f381333a640 1 --2- 172.21.15.44:0/3232118884 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f380c102250 0x7f380c19ede0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.508+0000 7f381333a640 1 --2- 172.21.15.44:0/3232118884 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f380c1006a0 0x7f380c0695b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.508+0000 7f381333a640 1 -- 172.21.15.44:0/3232118884 >> 172.21.15.44:0/3232118884 conn(0x7f380c0fbc10 msgr2=0x7f380c104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:50.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.508+0000 7f381333a640 1 -- 172.21.15.44:0/3232118884 shutdown_connections 2024-04-21T23:08:50.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.508+0000 7f381333a640 1 -- 172.21.15.44:0/3232118884 wait complete. 2024-04-21T23:08:50.519 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:50.346+0000 7ff000fe2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T23:08:50.521 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:01:47.507503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:06:23.993760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:06:24.954661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:06:28.677618+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:08:50.521 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:08:50.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.634+0000 7f805d98a640 1 -- 172.21.15.44:0/2539481144 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8058100050 msgr2=0x7f8058100470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:50.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.634+0000 7f805d98a640 1 --2- 172.21.15.44:0/2539481144 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8058100050 0x7f8058100470 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f804c009860 tx=0x7f804c02bb70 comp rx=0 tx=0).stop 2024-04-21T23:08:50.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.634+0000 7f805d98a640 1 -- 172.21.15.44:0/2539481144 shutdown_connections 2024-04-21T23:08:50.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.634+0000 7f805d98a640 1 --2- 172.21.15.44:0/2539481144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8058101350 0x7f80581017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.634+0000 7f805d98a640 1 --2- 172.21.15.44:0/2539481144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80581009b0 0x7f8058100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.634+0000 7f805d98a640 1 --2- 172.21.15.44:0/2539481144 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8058100050 0x7f8058100470 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.634+0000 7f805d98a640 1 -- 172.21.15.44:0/2539481144 >> 172.21.15.44:0/2539481144 conn(0x7f80580fbc10 msgr2=0x7f80580fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:50.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.634+0000 7f805d98a640 1 -- 172.21.15.44:0/2539481144 shutdown_connections 2024-04-21T23:08:50.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.634+0000 7f805d98a640 1 -- 172.21.15.44:0/2539481144 wait complete. 2024-04-21T23:08:50.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.634+0000 7f805d98a640 1 Processor -- start 2024-04-21T23:08:50.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.634+0000 7f805d98a640 1 -- start start 2024-04-21T23:08:50.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.635+0000 7f805d98a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8058100050 0x7f8058199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:50.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.635+0000 7f805d98a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80581009b0 0x7f805819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:50.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.635+0000 7f805d98a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8058101350 0x7f80581a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:50.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.635+0000 7f805d98a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f805810f5e0 con 0x7f8058101350 2024-04-21T23:08:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.635+0000 7f805d98a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f805810f460 con 0x7f8058100050 2024-04-21T23:08:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.635+0000 7f805d98a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f805810f760 con 0x7f80581009b0 2024-04-21T23:08:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.635+0000 7f80577fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8058101350 0x7f80581a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.635+0000 7f80577fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8058101350 0x7f80581a1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35396/0 (socket says 172.21.15.44:35396) 2024-04-21T23:08:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.635+0000 7f80577fe640 1 -- 172.21.15.44:0/2094846664 learned_addr learned my addr 172.21.15.44:0/2094846664 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.635+0000 7f80567fc640 1 --2- 172.21.15.44:0/2094846664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80581009b0 0x7f805819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.635+0000 7f8056ffd640 1 --2- 172.21.15.44:0/2094846664 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8058100050 0x7f8058199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.635+0000 7f80577fe640 1 -- 172.21.15.44:0/2094846664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80581009b0 msgr2=0x7f805819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.635+0000 7f80577fe640 1 --2- 172.21.15.44:0/2094846664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80581009b0 0x7f805819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.635+0000 7f80577fe640 1 -- 172.21.15.44:0/2094846664 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8058100050 msgr2=0x7f8058199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.635+0000 7f80577fe640 1 --2- 172.21.15.44:0/2094846664 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8058100050 0x7f8058199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.635+0000 7f80577fe640 1 -- 172.21.15.44:0/2094846664 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80581a18c0 con 0x7f8058101350 2024-04-21T23:08:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.635+0000 7f80567fc640 1 --2- 172.21.15.44:0/2094846664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80581009b0 0x7f805819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:08:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.635+0000 7f80577fe640 1 --2- 172.21.15.44:0/2094846664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8058101350 0x7f80581a1220 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f805000eb10 tx=0x7f805000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:50.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.635+0000 7f805c988640 1 -- 172.21.15.44:0/2094846664 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805000ce20 con 0x7f8058101350 2024-04-21T23:08:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.636+0000 7f805d98a640 1 -- 172.21.15.44:0/2094846664 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80581a1bb0 con 0x7f8058101350 2024-04-21T23:08:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.636+0000 7f805c988640 1 -- 172.21.15.44:0/2094846664 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8050005020 con 0x7f8058101350 2024-04-21T23:08:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.636+0000 7f805c988640 1 -- 172.21.15.44:0/2094846664 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8050010690 con 0x7f8058101350 2024-04-21T23:08:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.636+0000 7f805d98a640 1 -- 172.21.15.44:0/2094846664 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80581a2150 con 0x7f8058101350 2024-04-21T23:08:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.636+0000 7f805d98a640 1 -- 172.21.15.44:0/2094846664 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f805810ada0 con 0x7f8058101350 2024-04-21T23:08:50.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.638+0000 7f805c988640 1 -- 172.21.15.44:0/2094846664 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 20) ==== 137518+0+0 (secure 0 0 0) 0x7f8050005540 con 0x7f8058101350 2024-04-21T23:08:50.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.639+0000 7f805c988640 1 -- 172.21.15.44:0/2094846664 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f80500cf5d0 con 0x7f8058101350 2024-04-21T23:08:50.643 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.644+0000 7f805c988640 1 -- 172.21.15.44:0/2094846664 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f805009c470 con 0x7f8058101350 2024-04-21T23:08:50.714 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y[32543]: 2024-04-21T23:08:50.255+0000 7f33a5457200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T23:08:50.861 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:50.582+0000 7ff000fe2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T23:08:50.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.889+0000 7f805d98a640 1 -- 172.21.15.44:0/2094846664 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8058062340 con 0x7f8058101350 2024-04-21T23:08:50.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.889+0000 7f805c988640 1 -- 172.21.15.44:0/2094846664 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f805009bd70 con 0x7f8058101350 2024-04-21T23:08:50.889 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-21T23:08:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.891+0000 7f805d98a640 1 -- 172.21.15.44:0/2094846664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8058101350 msgr2=0x7f80581a1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.891+0000 7f805d98a640 1 --2- 172.21.15.44:0/2094846664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8058101350 0x7f80581a1220 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f805000eb10 tx=0x7f805000ef10 comp rx=0 tx=0).stop 2024-04-21T23:08:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.891+0000 7f805d98a640 1 -- 172.21.15.44:0/2094846664 shutdown_connections 2024-04-21T23:08:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.891+0000 7f805d98a640 1 --2- 172.21.15.44:0/2094846664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80581009b0 0x7f805819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.891+0000 7f805d98a640 1 --2- 172.21.15.44:0/2094846664 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8058101350 0x7f80581a1220 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.891+0000 7f805d98a640 1 --2- 172.21.15.44:0/2094846664 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8058100050 0x7f8058199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.891+0000 7f805d98a640 1 -- 172.21.15.44:0/2094846664 >> 172.21.15.44:0/2094846664 conn(0x7f80580fbc10 msgr2=0x7f80580fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.891+0000 7f805d98a640 1 -- 172.21.15.44:0/2094846664 shutdown_connections 2024-04-21T23:08:50.891 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:50.891+0000 7f805d98a640 1 -- 172.21.15.44:0/2094846664 wait complete. 2024-04-21T23:08:50.903 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:08:51.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.015+0000 7fae2454a640 1 -- 172.21.15.44:0/735135693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fae1c102230 msgr2=0x7fae1c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:51.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.015+0000 7fae2454a640 1 --2- 172.21.15.44:0/735135693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fae1c102230 0x7fae1c108ac0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fae14009ca0 tx=0x7fae1402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.015+0000 7fae2454a640 1 -- 172.21.15.44:0/735135693 shutdown_connections 2024-04-21T23:08:51.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.015+0000 7fae2454a640 1 --2- 172.21.15.44:0/735135693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae1c101870 0x7fae1c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.015+0000 7fae2454a640 1 --2- 172.21.15.44:0/735135693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fae1c102230 0x7fae1c108ac0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.015+0000 7fae2454a640 1 --2- 172.21.15.44:0/735135693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae1c100680 0x7fae1c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.015+0000 7fae2454a640 1 -- 172.21.15.44:0/735135693 >> 172.21.15.44:0/735135693 conn(0x7fae1c0fbc10 msgr2=0x7fae1c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:51.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.015+0000 7fae2454a640 1 -- 172.21.15.44:0/735135693 shutdown_connections 2024-04-21T23:08:51.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.015+0000 7fae2454a640 1 -- 172.21.15.44:0/735135693 wait complete. 2024-04-21T23:08:51.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.016+0000 7fae2454a640 1 Processor -- start 2024-04-21T23:08:51.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.016+0000 7fae2454a640 1 -- start start 2024-04-21T23:08:51.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.016+0000 7fae2454a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae1c100680 0x7fae1c069610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:51.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.016+0000 7fae2454a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae1c101870 0x7fae1c19c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:51.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.016+0000 7fae2454a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fae1c102230 0x7fae1c19edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:51.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.016+0000 7fae2454a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c10c300 con 0x7fae1c102230 2024-04-21T23:08:51.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.016+0000 7fae2454a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c10c180 con 0x7fae1c100680 2024-04-21T23:08:51.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.016+0000 7fae2454a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c10c480 con 0x7fae1c101870 2024-04-21T23:08:51.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.017+0000 7fae22ae7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fae1c102230 0x7fae1c19edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:51.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.017+0000 7fae22ae7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fae1c102230 0x7fae1c19edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35492/0 (socket says 172.21.15.44:35492) 2024-04-21T23:08:51.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.017+0000 7fae22ae7640 1 -- 172.21.15.44:0/1328979619 learned_addr learned my addr 172.21.15.44:0/1328979619 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:51.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.017+0000 7fae222e6640 1 --2- 172.21.15.44:0/1328979619 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae1c100680 0x7fae1c069610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:51.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.017+0000 7fae21ae5640 1 --2- 172.21.15.44:0/1328979619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae1c101870 0x7fae1c19c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:51.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.017+0000 7fae22ae7640 1 -- 172.21.15.44:0/1328979619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae1c101870 msgr2=0x7fae1c19c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:51.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.017+0000 7fae22ae7640 1 --2- 172.21.15.44:0/1328979619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae1c101870 0x7fae1c19c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.017+0000 7fae22ae7640 1 -- 172.21.15.44:0/1328979619 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae1c100680 msgr2=0x7fae1c069610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:51.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.017+0000 7fae22ae7640 1 --2- 172.21.15.44:0/1328979619 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae1c100680 0x7fae1c069610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.017+0000 7fae22ae7640 1 -- 172.21.15.44:0/1328979619 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae1c19f690 con 0x7fae1c102230 2024-04-21T23:08:51.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.017+0000 7fae21ae5640 1 --2- 172.21.15.44:0/1328979619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae1c101870 0x7fae1c19c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:08:51.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.017+0000 7fae22ae7640 1 --2- 172.21.15.44:0/1328979619 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fae1c102230 0x7fae1c19edd0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fae14037040 tx=0x7fae14032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:51.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.017+0000 7fae137fe640 1 -- 172.21.15.44:0/1328979619 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae14041020 con 0x7fae1c102230 2024-04-21T23:08:51.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.017+0000 7fae137fe640 1 -- 172.21.15.44:0/1328979619 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae14034cd0 con 0x7fae1c102230 2024-04-21T23:08:51.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.017+0000 7fae137fe640 1 -- 172.21.15.44:0/1328979619 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae14039040 con 0x7fae1c102230 2024-04-21T23:08:51.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.017+0000 7fae2454a640 1 -- 172.21.15.44:0/1328979619 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae1c19f920 con 0x7fae1c102230 2024-04-21T23:08:51.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.017+0000 7fae2454a640 1 -- 172.21.15.44:0/1328979619 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae1c19fe60 con 0x7fae1c102230 2024-04-21T23:08:51.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.019+0000 7fae2454a640 1 -- 172.21.15.44:0/1328979619 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae1c1079a0 con 0x7fae1c102230 2024-04-21T23:08:51.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.021+0000 7fae137fe640 1 -- 172.21.15.44:0/1328979619 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 20) ==== 137518+0+0 (secure 0 0 0) 0x7fae14044020 con 0x7fae1c102230 2024-04-21T23:08:51.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.021+0000 7fae137fe640 1 -- 172.21.15.44:0/1328979619 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fae140f1110 con 0x7fae1c102230 2024-04-21T23:08:51.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.024+0000 7fae137fe640 1 -- 172.21.15.44:0/1328979619 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fae1404b020 con 0x7fae1c102230 2024-04-21T23:08:51.186 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:50.866+0000 7fef6396d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:08:51.186 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:50.866+0000 7fef6396d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:08:51.186 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:50.866+0000 7fef6396d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:08:51.186 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:50.866+0000 7fef6396d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:08:51.186 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:50.866+0000 7fef6396d700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:08:51.187 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:50.867+0000 7fef60967700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:08:51.187 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:50.867+0000 7fef60967700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:08:51.187 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:50.867+0000 7fef60967700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:08:51.187 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:50.867+0000 7fef60967700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:08:51.187 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:08:50.867+0000 7fef60967700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:08:51.188 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: Standby manager daemon z restarted 2024-04-21T23:08:51.189 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: Standby manager daemon z started 2024-04-21T23:08:51.189 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.? 172.21.15.69:0/3287077420' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T23:08:51.189 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.? 172.21.15.69:0/3287077420' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:08:51.189 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.? 172.21.15.69:0/3287077420' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T23:08:51.189 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.? 172.21.15.69:0/3287077420' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:08:51.190 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3122825118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:51.190 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: Standby manager daemon y restarted 2024-04-21T23:08:51.190 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: Standby manager daemon y started 2024-04-21T23:08:51.190 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.? 172.21.15.55:0/4126205062' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T23:08:51.190 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.? 172.21.15.55:0/4126205062' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:08:51.190 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.? 172.21.15.55:0/4126205062' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T23:08:51.191 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.? 172.21.15.55:0/4126205062' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:08:51.191 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3232118884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:51.191 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: Active manager daemon x restarted 2024-04-21T23:08:51.191 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: Activating manager daemon x 2024-04-21T23:08:51.191 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: osdmap e95: 12 total, 12 up, 12 in 2024-04-21T23:08:51.191 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: mgrmap e20: x(active, starting, since 0.0128817s), standbys: y, z 2024-04-21T23:08:51.192 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:08:51.192 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:08:51.192 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:08:51.192 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T23:08:51.192 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T23:08:51.192 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T23:08:51.193 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T23:08:51.193 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T23:08:51.193 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T23:08:51.193 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T23:08:51.193 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T23:08:51.193 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T23:08:51.194 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T23:08:51.194 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:08:51.194 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:08:51.194 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T23:08:51.194 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T23:08:51.194 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T23:08:51.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:08:51.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:08:51.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:08:51.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:08:51.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:08:51.195 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:08:51.196 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:08:51.196 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:08:51.196 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:08:51.196 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:08:51.196 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:08:51.196 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:08:51.197 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T23:08:51.197 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T23:08:51.197 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T23:08:51.197 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: Manager daemon x is now available 2024-04-21T23:08:51.197 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:08:51.197 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T23:08:51.197 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T23:08:51.198 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:50 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2094846664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:51.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: Standby manager daemon z restarted 2024-04-21T23:08:51.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: Standby manager daemon z started 2024-04-21T23:08:51.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.? 172.21.15.69:0/3287077420' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T23:08:51.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.? 172.21.15.69:0/3287077420' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:08:51.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.? 172.21.15.69:0/3287077420' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T23:08:51.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.? 172.21.15.69:0/3287077420' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:08:51.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3122825118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:51.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: Standby manager daemon y restarted 2024-04-21T23:08:51.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: Standby manager daemon y started 2024-04-21T23:08:51.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.? 172.21.15.55:0/4126205062' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T23:08:51.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.? 172.21.15.55:0/4126205062' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:08:51.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.? 172.21.15.55:0/4126205062' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T23:08:51.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.? 172.21.15.55:0/4126205062' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:08:51.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3232118884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:51.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: Active manager daemon x restarted 2024-04-21T23:08:51.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: Activating manager daemon x 2024-04-21T23:08:51.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: osdmap e95: 12 total, 12 up, 12 in 2024-04-21T23:08:51.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: mgrmap e20: x(active, starting, since 0.0128817s), standbys: y, z 2024-04-21T23:08:51.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:08:51.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:08:51.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:08:51.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T23:08:51.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T23:08:51.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T23:08:51.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T23:08:51.216 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T23:08:51.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T23:08:51.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T23:08:51.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T23:08:51.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T23:08:51.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T23:08:51.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:08:51.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:08:51.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T23:08:51.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T23:08:51.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T23:08:51.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:08:51.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:08:51.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:08:51.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:08:51.218 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:08:51.218 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:08:51.218 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:08:51.218 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:08:51.218 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:08:51.218 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:08:51.218 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:08:51.218 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:08:51.218 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T23:08:51.218 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T23:08:51.218 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T23:08:51.218 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: Manager daemon x is now available 2024-04-21T23:08:51.218 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:08:51.218 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T23:08:51.219 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T23:08:51.219 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:50 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2094846664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:51.219 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: Standby manager daemon z restarted 2024-04-21T23:08:51.219 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: Standby manager daemon z started 2024-04-21T23:08:51.219 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.? 172.21.15.69:0/3287077420' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T23:08:51.219 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.? 172.21.15.69:0/3287077420' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:08:51.219 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.? 172.21.15.69:0/3287077420' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T23:08:51.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.? 172.21.15.69:0/3287077420' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:08:51.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3122825118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:51.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: Standby manager daemon y restarted 2024-04-21T23:08:51.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: Standby manager daemon y started 2024-04-21T23:08:51.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.? 172.21.15.55:0/4126205062' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T23:08:51.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.? 172.21.15.55:0/4126205062' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T23:08:51.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.? 172.21.15.55:0/4126205062' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T23:08:51.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.? 172.21.15.55:0/4126205062' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T23:08:51.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3232118884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:51.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: Active manager daemon x restarted 2024-04-21T23:08:51.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: Activating manager daemon x 2024-04-21T23:08:51.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: osdmap e95: 12 total, 12 up, 12 in 2024-04-21T23:08:51.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: mgrmap e20: x(active, starting, since 0.0128817s), standbys: y, z 2024-04-21T23:08:51.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T23:08:51.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T23:08:51.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T23:08:51.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T23:08:51.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T23:08:51.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T23:08:51.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T23:08:51.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T23:08:51.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T23:08:51.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T23:08:51.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T23:08:51.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T23:08:51.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T23:08:51.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:08:51.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:08:51.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T23:08:51.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T23:08:51.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T23:08:51.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T23:08:51.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T23:08:51.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T23:08:51.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T23:08:51.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T23:08:51.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T23:08:51.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T23:08:51.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T23:08:51.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T23:08:51.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T23:08:51.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T23:08:51.222 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T23:08:51.223 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T23:08:51.223 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T23:08:51.223 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T23:08:51.223 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: Manager daemon x is now available 2024-04-21T23:08:51.223 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:08:51.223 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T23:08:51.223 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T23:08:51.223 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:50 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2094846664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:51.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.221+0000 7fae2454a640 1 -- 172.21.15.44:0/1328979619 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fae1c02d170 con 0x7fae1c102230 2024-04-21T23:08:51.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.221+0000 7fae137fe640 1 -- 172.21.15.44:0/1328979619 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7fae140be040 con 0x7fae1c102230 2024-04-21T23:08:51.224 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:08:51.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.223+0000 7fae2454a640 1 -- 172.21.15.44:0/1328979619 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fae1c102230 msgr2=0x7fae1c19edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:51.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.223+0000 7fae2454a640 1 --2- 172.21.15.44:0/1328979619 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fae1c102230 0x7fae1c19edd0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fae14037040 tx=0x7fae14032f40 comp rx=0 tx=0).stop 2024-04-21T23:08:51.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.223+0000 7fae2454a640 1 -- 172.21.15.44:0/1328979619 shutdown_connections 2024-04-21T23:08:51.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.223+0000 7fae2454a640 1 --2- 172.21.15.44:0/1328979619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae1c101870 0x7fae1c19c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.223+0000 7fae2454a640 1 --2- 172.21.15.44:0/1328979619 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fae1c102230 0x7fae1c19edd0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.223+0000 7fae2454a640 1 --2- 172.21.15.44:0/1328979619 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fae1c100680 0x7fae1c069610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.223+0000 7fae2454a640 1 -- 172.21.15.44:0/1328979619 >> 172.21.15.44:0/1328979619 conn(0x7fae1c0fbc10 msgr2=0x7fae1c104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:51.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.223+0000 7fae2454a640 1 -- 172.21.15.44:0/1328979619 shutdown_connections 2024-04-21T23:08:51.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.223+0000 7fae2454a640 1 -- 172.21.15.44:0/1328979619 wait complete. 2024-04-21T23:08:51.234 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:08:51.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.341+0000 7fbb1b15c640 1 -- 172.21.15.44:0/91665431 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb14107460 msgr2=0x7fbb141098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:51.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.341+0000 7fbb1b15c640 1 --2- 172.21.15.44:0/91665431 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb14107460 0x7fbb141098b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c009ca0 tx=0x7fbb0c02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.341+0000 7fbb1b15c640 1 -- 172.21.15.44:0/91665431 shutdown_connections 2024-04-21T23:08:51.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.341+0000 7fbb1b15c640 1 --2- 172.21.15.44:0/91665431 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb14107460 0x7fbb141098b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.341+0000 7fbb1b15c640 1 --2- 172.21.15.44:0/91665431 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb14104b30 0x7fbb14106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.341+0000 7fbb1b15c640 1 --2- 172.21.15.44:0/91665431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb14102200 0x7fbb141045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.341+0000 7fbb1b15c640 1 -- 172.21.15.44:0/91665431 >> 172.21.15.44:0/91665431 conn(0x7fbb140fbc10 msgr2=0x7fbb140fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:51.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.341+0000 7fbb1b15c640 1 -- 172.21.15.44:0/91665431 shutdown_connections 2024-04-21T23:08:51.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.341+0000 7fbb1b15c640 1 -- 172.21.15.44:0/91665431 wait complete. 2024-04-21T23:08:51.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.341+0000 7fbb1b15c640 1 Processor -- start 2024-04-21T23:08:51.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.341+0000 7fbb1b15c640 1 -- start start 2024-04-21T23:08:51.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.342+0000 7fbb1b15c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb14102200 0x7fbb14197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:51.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.342+0000 7fbb1b15c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb14104b30 0x7fbb14197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:51.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.342+0000 7fbb1b15c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb14107460 0x7fbb1419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:51.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.342+0000 7fbb1b15c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1410d420 con 0x7fbb14102200 2024-04-21T23:08:51.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.342+0000 7fbb1b15c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1410d2a0 con 0x7fbb14107460 2024-04-21T23:08:51.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.342+0000 7fbb1b15c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1410d5a0 con 0x7fbb14104b30 2024-04-21T23:08:51.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.342+0000 7fbb18ef8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb14102200 0x7fbb14197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:51.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.342+0000 7fbb18ef8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb14102200 0x7fbb14197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35512/0 (socket says 172.21.15.44:35512) 2024-04-21T23:08:51.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.342+0000 7fbb18ef8640 1 -- 172.21.15.44:0/2107378306 learned_addr learned my addr 172.21.15.44:0/2107378306 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:51.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.342+0000 7fbb13fff640 1 --2- 172.21.15.44:0/2107378306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb14104b30 0x7fbb14197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:51.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.342+0000 7fbb196f9640 1 --2- 172.21.15.44:0/2107378306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb14107460 0x7fbb1419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:51.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.342+0000 7fbb18ef8640 1 -- 172.21.15.44:0/2107378306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb14104b30 msgr2=0x7fbb14197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:51.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.342+0000 7fbb18ef8640 1 --2- 172.21.15.44:0/2107378306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb14104b30 0x7fbb14197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.342+0000 7fbb18ef8640 1 -- 172.21.15.44:0/2107378306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb14107460 msgr2=0x7fbb1419f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:51.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.342+0000 7fbb18ef8640 1 --2- 172.21.15.44:0/2107378306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb14107460 0x7fbb1419f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.342+0000 7fbb18ef8640 1 -- 172.21.15.44:0/2107378306 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb1419f6b0 con 0x7fbb14102200 2024-04-21T23:08:51.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.342+0000 7fbb18ef8640 1 --2- 172.21.15.44:0/2107378306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb14102200 0x7fbb14197a90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fbb0800e9f0 tx=0x7fbb0800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:51.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.342+0000 7fbb11ffb640 1 -- 172.21.15.44:0/2107378306 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0800cd80 con 0x7fbb14102200 2024-04-21T23:08:51.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.342+0000 7fbb11ffb640 1 -- 172.21.15.44:0/2107378306 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb08005500 con 0x7fbb14102200 2024-04-21T23:08:51.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.343+0000 7fbb1b15c640 1 -- 172.21.15.44:0/2107378306 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb1419f9a0 con 0x7fbb14102200 2024-04-21T23:08:51.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.343+0000 7fbb11ffb640 1 -- 172.21.15.44:0/2107378306 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb08010690 con 0x7fbb14102200 2024-04-21T23:08:51.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.343+0000 7fbb1b15c640 1 -- 172.21.15.44:0/2107378306 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb1419fea0 con 0x7fbb14102200 2024-04-21T23:08:51.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.344+0000 7fbb1b15c640 1 -- 172.21.15.44:0/2107378306 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb140fd700 con 0x7fbb14102200 2024-04-21T23:08:51.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.346+0000 7fbb11ffb640 1 -- 172.21.15.44:0/2107378306 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 20) ==== 137518+0+0 (secure 0 0 0) 0x7fbb080046c0 con 0x7fbb14102200 2024-04-21T23:08:51.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.346+0000 7fbb11ffb640 1 -- 172.21.15.44:0/2107378306 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fbb080cf370 con 0x7fbb14102200 2024-04-21T23:08:51.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.355+0000 7fbb11ffb640 1 -- 172.21.15.44:0/2107378306 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fbb0809c210 con 0x7fbb14102200 2024-04-21T23:08:51.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.550+0000 7fbb1b15c640 1 -- 172.21.15.44:0/2107378306 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbb1402d170 con 0x7fbb14102200 2024-04-21T23:08:51.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.550+0000 7fbb11ffb640 1 -- 172.21.15.44:0/2107378306 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7fbb0809bb10 con 0x7fbb14102200 2024-04-21T23:08:51.550 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-21T23:08:51.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.552+0000 7fbb1b15c640 1 -- 172.21.15.44:0/2107378306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb14102200 msgr2=0x7fbb14197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:51.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.552+0000 7fbb1b15c640 1 --2- 172.21.15.44:0/2107378306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb14102200 0x7fbb14197a90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fbb0800e9f0 tx=0x7fbb0800edf0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.552+0000 7fbb1b15c640 1 -- 172.21.15.44:0/2107378306 shutdown_connections 2024-04-21T23:08:51.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.552+0000 7fbb1b15c640 1 --2- 172.21.15.44:0/2107378306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb14104b30 0x7fbb14197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.552+0000 7fbb1b15c640 1 --2- 172.21.15.44:0/2107378306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbb14107460 0x7fbb1419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.552+0000 7fbb1b15c640 1 --2- 172.21.15.44:0/2107378306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbb14102200 0x7fbb14197a90 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.552+0000 7fbb1b15c640 1 -- 172.21.15.44:0/2107378306 >> 172.21.15.44:0/2107378306 conn(0x7fbb140fbc10 msgr2=0x7fbb14102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:51.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.553+0000 7fbb1b15c640 1 -- 172.21.15.44:0/2107378306 shutdown_connections 2024-04-21T23:08:51.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.553+0000 7fbb1b15c640 1 -- 172.21.15.44:0/2107378306 wait complete. 2024-04-21T23:08:51.564 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:08:51.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.689+0000 7ffaa529d640 1 -- 172.21.15.44:0/3092898305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaa00ff7f0 msgr2=0x7ffaa00ffc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:51.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.689+0000 7ffaa529d640 1 --2- 172.21.15.44:0/3092898305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaa00ff7f0 0x7ffaa00ffc70 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ffa94009900 tx=0x7ffa9402bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:51.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.689+0000 7ffaa529d640 1 -- 172.21.15.44:0/3092898305 shutdown_connections 2024-04-21T23:08:51.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.689+0000 7ffaa529d640 1 --2- 172.21.15.44:0/3092898305 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffaa01001b0 0x7ffaa0100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.689+0000 7ffaa529d640 1 --2- 172.21.15.44:0/3092898305 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaa00ff7f0 0x7ffaa00ffc70 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.689+0000 7ffaa529d640 1 --2- 172.21.15.44:0/3092898305 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffaa0101d00 0x7ffaa0102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.689+0000 7ffaa529d640 1 -- 172.21.15.44:0/3092898305 >> 172.21.15.44:0/3092898305 conn(0x7ffaa00fb390 msgr2=0x7ffaa00fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:51.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.689+0000 7ffaa529d640 1 -- 172.21.15.44:0/3092898305 shutdown_connections 2024-04-21T23:08:51.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.689+0000 7ffaa529d640 1 -- 172.21.15.44:0/3092898305 wait complete. 2024-04-21T23:08:51.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.689+0000 7ffaa529d640 1 Processor -- start 2024-04-21T23:08:51.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.690+0000 7ffaa529d640 1 -- start start 2024-04-21T23:08:51.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.690+0000 7ffaa529d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffaa00ff7f0 0x7ffaa0199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:51.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.690+0000 7ffaa529d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffaa01001b0 0x7ffaa019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:51.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.690+0000 7ffaa529d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaa0101d00 0x7ffaa01a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:51.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.690+0000 7ffaa529d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa010f660 con 0x7ffaa0101d00 2024-04-21T23:08:51.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.690+0000 7ffaa529d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa010f4e0 con 0x7ffaa01001b0 2024-04-21T23:08:51.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.690+0000 7ffaa529d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa010f7e0 con 0x7ffaa00ff7f0 2024-04-21T23:08:51.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.690+0000 7ffa9f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaa0101d00 0x7ffaa01a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:51.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.690+0000 7ffa9e7fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffaa01001b0 0x7ffaa019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:51.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.690+0000 7ffa9f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaa0101d00 0x7ffaa01a12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35532/0 (socket says 172.21.15.44:35532) 2024-04-21T23:08:51.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.690+0000 7ffa9f7fe640 1 -- 172.21.15.44:0/4293208910 learned_addr learned my addr 172.21.15.44:0/4293208910 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:51.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.690+0000 7ffa9effd640 1 --2- 172.21.15.44:0/4293208910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffaa00ff7f0 0x7ffaa0199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:51.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.690+0000 7ffa9f7fe640 1 -- 172.21.15.44:0/4293208910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffaa00ff7f0 msgr2=0x7ffaa0199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:51.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.690+0000 7ffa9f7fe640 1 --2- 172.21.15.44:0/4293208910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffaa00ff7f0 0x7ffaa0199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.690+0000 7ffa9f7fe640 1 -- 172.21.15.44:0/4293208910 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffaa01001b0 msgr2=0x7ffaa019a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:51.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.690+0000 7ffa9f7fe640 1 --2- 172.21.15.44:0/4293208910 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffaa01001b0 0x7ffaa019a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.690+0000 7ffa9f7fe640 1 -- 172.21.15.44:0/4293208910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa01a1960 con 0x7ffaa0101d00 2024-04-21T23:08:51.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.691+0000 7ffa9f7fe640 1 --2- 172.21.15.44:0/4293208910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaa0101d00 0x7ffaa01a12f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ffa9800efd0 tx=0x7ffa9800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:51.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.691+0000 7ffa87fff640 1 -- 172.21.15.44:0/4293208910 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa98003b70 con 0x7ffaa0101d00 2024-04-21T23:08:51.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.691+0000 7ffaa529d640 1 -- 172.21.15.44:0/4293208910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaa01a1c50 con 0x7ffaa0101d00 2024-04-21T23:08:51.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.691+0000 7ffa87fff640 1 -- 172.21.15.44:0/4293208910 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa98003d10 con 0x7ffaa0101d00 2024-04-21T23:08:51.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.691+0000 7ffa87fff640 1 -- 172.21.15.44:0/4293208910 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa980109f0 con 0x7ffaa0101d00 2024-04-21T23:08:51.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.691+0000 7ffaa529d640 1 -- 172.21.15.44:0/4293208910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaa01a2170 con 0x7ffaa0101d00 2024-04-21T23:08:51.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.692+0000 7ffaa529d640 1 -- 172.21.15.44:0/4293208910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaa0067d60 con 0x7ffaa0101d00 2024-04-21T23:08:51.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.694+0000 7ffa87fff640 1 -- 172.21.15.44:0/4293208910 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7ffa98010b90 con 0x7ffaa0101d00 2024-04-21T23:08:51.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.694+0000 7ffa87fff640 1 --2- 172.21.15.44:0/4293208910 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ffa880a2990 0x7ffa880a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:51.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.694+0000 7ffa87fff640 1 -- 172.21.15.44:0/4293208910 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7ffa980cf8c0 con 0x7ffaa0101d00 2024-04-21T23:08:51.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.695+0000 7ffa9effd640 1 --2- 172.21.15.44:0/4293208910 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ffa880a2990 0x7ffa880a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:51.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.695+0000 7ffa9effd640 1 --2- 172.21.15.44:0/4293208910 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ffa880a2990 0x7ffa880a4e50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ffa90005fd0 tx=0x7ffa9000b390 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:51.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.698+0000 7ffa87fff640 1 -- 172.21.15.44:0/4293208910 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ffa9809c650 con 0x7ffaa0101d00 2024-04-21T23:08:51.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.885+0000 7ffaa529d640 1 -- 172.21.15.44:0/4293208910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ffaa010b850 con 0x7ffaa0101d00 2024-04-21T23:08:51.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.886+0000 7ffa87fff640 1 -- 172.21.15.44:0/4293208910 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7ffa9809bf50 con 0x7ffaa0101d00 2024-04-21T23:08:51.885 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:08:51.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.888+0000 7ffaa529d640 1 -- 172.21.15.44:0/4293208910 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ffa880a2990 msgr2=0x7ffa880a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:51.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.888+0000 7ffaa529d640 1 --2- 172.21.15.44:0/4293208910 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ffa880a2990 0x7ffa880a4e50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ffa90005fd0 tx=0x7ffa9000b390 comp rx=0 tx=0).stop 2024-04-21T23:08:51.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.888+0000 7ffaa529d640 1 -- 172.21.15.44:0/4293208910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaa0101d00 msgr2=0x7ffaa01a12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:51.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.888+0000 7ffaa529d640 1 --2- 172.21.15.44:0/4293208910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaa0101d00 0x7ffaa01a12f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ffa9800efd0 tx=0x7ffa9800c6a0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.888+0000 7ffaa529d640 1 -- 172.21.15.44:0/4293208910 shutdown_connections 2024-04-21T23:08:51.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.888+0000 7ffaa529d640 1 --2- 172.21.15.44:0/4293208910 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ffa880a2990 0x7ffa880a4e50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.888+0000 7ffaa529d640 1 --2- 172.21.15.44:0/4293208910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffaa0101d00 0x7ffaa01a12f0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.888+0000 7ffaa529d640 1 --2- 172.21.15.44:0/4293208910 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffaa01001b0 0x7ffaa019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.888+0000 7ffaa529d640 1 --2- 172.21.15.44:0/4293208910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffaa00ff7f0 0x7ffaa0199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:51.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.888+0000 7ffaa529d640 1 -- 172.21.15.44:0/4293208910 >> 172.21.15.44:0/4293208910 conn(0x7ffaa00fb390 msgr2=0x7ffaa00fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:51.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.888+0000 7ffaa529d640 1 -- 172.21.15.44:0/4293208910 shutdown_connections 2024-04-21T23:08:51.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:51.888+0000 7ffaa529d640 1 -- 172.21.15.44:0/4293208910 wait complete. 2024-04-21T23:08:51.898 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-21T23:08:51.898 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:08:52.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.005+0000 7f26bd466640 1 -- 172.21.15.44:0/322500942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26b8101890 msgr2=0x7f26b8101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:52.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.005+0000 7f26bd466640 1 --2- 172.21.15.44:0/322500942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26b8101890 0x7f26b8101d10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f26ac009900 tx=0x7f26ac02bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:52.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.006+0000 7f26bd466640 1 -- 172.21.15.44:0/322500942 shutdown_connections 2024-04-21T23:08:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.006+0000 7f26bd466640 1 --2- 172.21.15.44:0/322500942 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26b8102250 0x7f26b8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.006+0000 7f26bd466640 1 --2- 172.21.15.44:0/322500942 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26b8101890 0x7f26b8101d10 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.006+0000 7f26bd466640 1 --2- 172.21.15.44:0/322500942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26b81006a0 0x7f26b8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.006+0000 7f26bd466640 1 -- 172.21.15.44:0/322500942 >> 172.21.15.44:0/322500942 conn(0x7f26b80fbc10 msgr2=0x7f26b80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.006+0000 7f26bd466640 1 -- 172.21.15.44:0/322500942 shutdown_connections 2024-04-21T23:08:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.006+0000 7f26bd466640 1 -- 172.21.15.44:0/322500942 wait complete. 2024-04-21T23:08:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.006+0000 7f26bd466640 1 Processor -- start 2024-04-21T23:08:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.006+0000 7f26bd466640 1 -- start start 2024-04-21T23:08:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.007+0000 7f26bd466640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26b81006a0 0x7f26b8104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.007+0000 7f26bd466640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26b8101890 0x7f26b8102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.007+0000 7f26bd466640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26b8102250 0x7f26b81031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.007+0000 7f26bd466640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f26b810c320 con 0x7f26b8101890 2024-04-21T23:08:52.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.007+0000 7f26bd466640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f26b810c1a0 con 0x7f26b81006a0 2024-04-21T23:08:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.007+0000 7f26bd466640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f26b810c4a0 con 0x7f26b8102250 2024-04-21T23:08:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.007+0000 7f26b67fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26b8101890 0x7f26b8102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.007+0000 7f26b67fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26b8101890 0x7f26b8102c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35548/0 (socket says 172.21.15.44:35548) 2024-04-21T23:08:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.007+0000 7f26b67fc640 1 -- 172.21.15.44:0/1405484650 learned_addr learned my addr 172.21.15.44:0/1405484650 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.007+0000 7f26b77fe640 1 --2- 172.21.15.44:0/1405484650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26b8102250 0x7f26b81031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.007+0000 7f26b6ffd640 1 --2- 172.21.15.44:0/1405484650 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26b81006a0 0x7f26b8104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.007+0000 7f26b67fc640 1 -- 172.21.15.44:0/1405484650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26b8102250 msgr2=0x7f26b81031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.007+0000 7f26b67fc640 1 --2- 172.21.15.44:0/1405484650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26b8102250 0x7f26b81031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.007+0000 7f26b67fc640 1 -- 172.21.15.44:0/1405484650 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26b81006a0 msgr2=0x7f26b8104630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.007+0000 7f26b67fc640 1 --2- 172.21.15.44:0/1405484650 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26b81006a0 0x7f26b8104630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.007+0000 7f26b67fc640 1 -- 172.21.15.44:0/1405484650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26b8103a40 con 0x7f26b8101890 2024-04-21T23:08:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.007+0000 7f26b67fc640 1 --2- 172.21.15.44:0/1405484650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26b8101890 0x7f26b8102c70 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f26ac009230 tx=0x7f26ac02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.007+0000 7f269ffff640 1 -- 172.21.15.44:0/1405484650 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ac041070 con 0x7f26b8101890 2024-04-21T23:08:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.008+0000 7f26bd466640 1 -- 172.21.15.44:0/1405484650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26b81a1b70 con 0x7f26b8101890 2024-04-21T23:08:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.008+0000 7f269ffff640 1 -- 172.21.15.44:0/1405484650 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26ac004450 con 0x7f26b8101890 2024-04-21T23:08:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.008+0000 7f269ffff640 1 -- 172.21.15.44:0/1405484650 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ac003e20 con 0x7f26b8101890 2024-04-21T23:08:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.008+0000 7f26bd466640 1 -- 172.21.15.44:0/1405484650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26b81a2110 con 0x7f26b8101890 2024-04-21T23:08:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.009+0000 7f26bd466640 1 -- 172.21.15.44:0/1405484650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26b81a2340 con 0x7f26b8101890 2024-04-21T23:08:52.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.010+0000 7f269ffff640 1 -- 172.21.15.44:0/1405484650 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7f26ac044070 con 0x7f26b8101890 2024-04-21T23:08:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.010+0000 7f269ffff640 1 --2- 172.21.15.44:0/1405484650 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f26a00a2990 0x7f26a00a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.011+0000 7f269ffff640 1 -- 172.21.15.44:0/1405484650 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f26ac0f0c60 con 0x7f26b8101890 2024-04-21T23:08:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.011+0000 7f26b6ffd640 1 --2- 172.21.15.44:0/1405484650 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f26a00a2990 0x7f26a00a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:52.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.011+0000 7f26b6ffd640 1 --2- 172.21.15.44:0/1405484650 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f26a00a2990 0x7f26a00a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f26b81016f0 tx=0x7f26a8005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:52.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.015+0000 7f269ffff640 1 -- 172.21.15.44:0/1405484650 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f26ac0bd3a0 con 0x7f26b8101890 2024-04-21T23:08:52.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:51 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1328979619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:52.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:51 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2107378306' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:52.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:51 smithi044 ceph-mon[27934]: mgrmap e21: x(active, since 1.01945s), standbys: y, z 2024-04-21T23:08:52.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:51 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/4293208910' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:52.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:51 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1328979619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:52.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:51 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2107378306' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:52.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:51 smithi055 ceph-mon[31117]: mgrmap e21: x(active, since 1.01945s), standbys: y, z 2024-04-21T23:08:52.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:51 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/4293208910' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:52.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:51 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1328979619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:52.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:51 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2107378306' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:52.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:51 smithi069 ceph-mon[31816]: mgrmap e21: x(active, since 1.01945s), standbys: y, z 2024-04-21T23:08:52.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:51 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/4293208910' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:52.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.243+0000 7f26bd466640 1 -- 172.21.15.44:0/1405484650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f26b81a2510 con 0x7f26b8101890 2024-04-21T23:08:52.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.243+0000 7f269ffff640 1 -- 172.21.15.44:0/1405484650 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11791 (secure 0 0 0) 0x7f26ac0c16a0 con 0x7f26b8101890 2024-04-21T23:08:52.243 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:52.243 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:06:50.685663+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3743898853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3743898853},{"type":"v1","addr":"172.21.15.69:6839","nonce":3743898853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:08:52.244 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2024-04-21T23:08:52.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.245+0000 7f26bd466640 1 -- 172.21.15.44:0/1405484650 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f26a00a2990 msgr2=0x7f26a00a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:52.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.245+0000 7f26bd466640 1 --2- 172.21.15.44:0/1405484650 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f26a00a2990 0x7f26a00a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f26b81016f0 tx=0x7f26a8005f90 comp rx=0 tx=0).stop 2024-04-21T23:08:52.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.245+0000 7f26bd466640 1 -- 172.21.15.44:0/1405484650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26b8101890 msgr2=0x7f26b8102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:52.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.245+0000 7f26bd466640 1 --2- 172.21.15.44:0/1405484650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26b8101890 0x7f26b8102c70 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f26ac009230 tx=0x7f26ac02c650 comp rx=0 tx=0).stop 2024-04-21T23:08:52.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.245+0000 7f26bd466640 1 -- 172.21.15.44:0/1405484650 shutdown_connections 2024-04-21T23:08:52.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.245+0000 7f26bd466640 1 --2- 172.21.15.44:0/1405484650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26b8102250 0x7f26b81031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.245+0000 7f26bd466640 1 --2- 172.21.15.44:0/1405484650 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f26a00a2990 0x7f26a00a4e50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.245+0000 7f26bd466640 1 --2- 172.21.15.44:0/1405484650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f26b8101890 0x7f26b8102c70 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.245+0000 7f26bd466640 1 --2- 172.21.15.44:0/1405484650 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f26b81006a0 0x7f26b8104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.245+0000 7f26bd466640 1 -- 172.21.15.44:0/1405484650 >> 172.21.15.44:0/1405484650 conn(0x7f26b80fbc10 msgr2=0x7f26b80fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:52.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.245+0000 7f26bd466640 1 -- 172.21.15.44:0/1405484650 shutdown_connections 2024-04-21T23:08:52.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.245+0000 7f26bd466640 1 -- 172.21.15.44:0/1405484650 wait complete. 2024-04-21T23:08:52.256 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-21T23:08:52.256 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-21T23:08:52.256 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T23:08:52.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.360+0000 7f632ceda640 1 -- 172.21.15.44:0/2328590631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6328103350 msgr2=0x7f6328109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:52.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.360+0000 7f632ceda640 1 --2- 172.21.15.44:0/2328590631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6328103350 0x7f6328109be0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f6320009ca0 tx=0x7f632002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.360+0000 7f632ceda640 1 -- 172.21.15.44:0/2328590631 shutdown_connections 2024-04-21T23:08:52.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.360+0000 7f632ceda640 1 --2- 172.21.15.44:0/2328590631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6328103350 0x7f6328109be0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.360+0000 7f632ceda640 1 --2- 172.21.15.44:0/2328590631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6328102990 0x7f6328102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.360+0000 7f632ceda640 1 --2- 172.21.15.44:0/2328590631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63281017a0 0x7f6328101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.360+0000 7f632ceda640 1 -- 172.21.15.44:0/2328590631 >> 172.21.15.44:0/2328590631 conn(0x7f63280fcd30 msgr2=0x7f63280ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:52.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.360+0000 7f632ceda640 1 -- 172.21.15.44:0/2328590631 shutdown_connections 2024-04-21T23:08:52.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.360+0000 7f632ceda640 1 -- 172.21.15.44:0/2328590631 wait complete. 2024-04-21T23:08:52.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.361+0000 7f632ceda640 1 Processor -- start 2024-04-21T23:08:52.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.361+0000 7f632ceda640 1 -- start start 2024-04-21T23:08:52.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.361+0000 7f632ceda640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63281017a0 0x7f6328197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:52.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.361+0000 7f632ceda640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6328102990 0x7f6328197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:52.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.361+0000 7f632ceda640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6328103350 0x7f632819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:52.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.361+0000 7f632ceda640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f632810d420 con 0x7f6328103350 2024-04-21T23:08:52.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.361+0000 7f632ceda640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f632810d2a0 con 0x7f63281017a0 2024-04-21T23:08:52.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.361+0000 7f632ceda640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f632810d5a0 con 0x7f6328102990 2024-04-21T23:08:52.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.361+0000 7f6326d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6328103350 0x7f632819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:52.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.361+0000 7f6326d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6328103350 0x7f632819eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35572/0 (socket says 172.21.15.44:35572) 2024-04-21T23:08:52.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.361+0000 7f6326d9d640 1 -- 172.21.15.44:0/1471954418 learned_addr learned my addr 172.21.15.44:0/1471954418 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:52.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.361+0000 7f631ffff640 1 --2- 172.21.15.44:0/1471954418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6328102990 0x7f6328197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:52.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.361+0000 7f632659c640 1 --2- 172.21.15.44:0/1471954418 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63281017a0 0x7f6328197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:52.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.361+0000 7f6326d9d640 1 -- 172.21.15.44:0/1471954418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6328102990 msgr2=0x7f6328197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:52.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.361+0000 7f6326d9d640 1 --2- 172.21.15.44:0/1471954418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6328102990 0x7f6328197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.362+0000 7f6326d9d640 1 -- 172.21.15.44:0/1471954418 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63281017a0 msgr2=0x7f6328197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:52.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.362+0000 7f6326d9d640 1 --2- 172.21.15.44:0/1471954418 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63281017a0 0x7f6328197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.362+0000 7f6326d9d640 1 -- 172.21.15.44:0/1471954418 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f632819f660 con 0x7f6328103350 2024-04-21T23:08:52.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.362+0000 7f631ffff640 1 --2- 172.21.15.44:0/1471954418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6328102990 0x7f6328197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:08:52.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.362+0000 7f6326d9d640 1 --2- 172.21.15.44:0/1471954418 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6328103350 0x7f632819eff0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f6320037040 tx=0x7f6320032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:52.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.362+0000 7f631f7fe640 1 -- 172.21.15.44:0/1471954418 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6320041020 con 0x7f6328103350 2024-04-21T23:08:52.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.362+0000 7f631f7fe640 1 -- 172.21.15.44:0/1471954418 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6320034d90 con 0x7f6328103350 2024-04-21T23:08:52.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.362+0000 7f631f7fe640 1 -- 172.21.15.44:0/1471954418 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6320039040 con 0x7f6328103350 2024-04-21T23:08:52.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.362+0000 7f632ceda640 1 -- 172.21.15.44:0/1471954418 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f632819f8f0 con 0x7f6328103350 2024-04-21T23:08:52.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.362+0000 7f632ceda640 1 -- 172.21.15.44:0/1471954418 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f632819fde0 con 0x7f6328103350 2024-04-21T23:08:52.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.363+0000 7f632ceda640 1 -- 172.21.15.44:0/1471954418 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6328108a60 con 0x7f6328103350 2024-04-21T23:08:52.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.364+0000 7f631f7fe640 1 -- 172.21.15.44:0/1471954418 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7f6320044020 con 0x7f6328103350 2024-04-21T23:08:52.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.365+0000 7f631f7fe640 1 --2- 172.21.15.44:0/1471954418 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f63140a2990 0x7f63140a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:52.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.365+0000 7f632659c640 1 --2- 172.21.15.44:0/1471954418 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f63140a2990 0x7f63140a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:52.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.365+0000 7f631f7fe640 1 -- 172.21.15.44:0/1471954418 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f63200f0fe0 con 0x7f6328103350 2024-04-21T23:08:52.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.365+0000 7f632659c640 1 --2- 172.21.15.44:0/1471954418 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f63140a2990 0x7f63140a4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f63281027f0 tx=0x7f6318005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:52.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.369+0000 7f631f7fe640 1 -- 172.21.15.44:0/1471954418 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f632004b020 con 0x7f6328103350 2024-04-21T23:08:52.559 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.559+0000 7f632ceda640 1 -- 172.21.15.44:0/1471954418 --> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6328061b80 con 0x7f63140a2990 2024-04-21T23:08:52.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.577+0000 7f631f7fe640 1 -- 172.21.15.44:0/1471954418 <== mgr.15396 v2:172.21.15.44:6800/3789150048 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339305 (secure 0 0 0) 0x7f6328061b80 con 0x7f63140a2990 2024-04-21T23:08:52.581 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:52.583 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-21T23:08:52.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.580+0000 7f632ceda640 1 -- 172.21.15.44:0/1471954418 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f63140a2990 msgr2=0x7f63140a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:52.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.580+0000 7f632ceda640 1 --2- 172.21.15.44:0/1471954418 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f63140a2990 0x7f63140a4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f63281027f0 tx=0x7f6318005f90 comp rx=0 tx=0).stop 2024-04-21T23:08:52.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.580+0000 7f632ceda640 1 -- 172.21.15.44:0/1471954418 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6328103350 msgr2=0x7f632819eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:52.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.580+0000 7f632ceda640 1 --2- 172.21.15.44:0/1471954418 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6328103350 0x7f632819eff0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f6320037040 tx=0x7f6320032f20 comp rx=0 tx=0).stop 2024-04-21T23:08:52.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.581+0000 7f632ceda640 1 -- 172.21.15.44:0/1471954418 shutdown_connections 2024-04-21T23:08:52.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.581+0000 7f632ceda640 1 --2- 172.21.15.44:0/1471954418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6328102990 0x7f6328197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.581+0000 7f632ceda640 1 --2- 172.21.15.44:0/1471954418 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f63140a2990 0x7f63140a4e50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.581+0000 7f632ceda640 1 --2- 172.21.15.44:0/1471954418 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6328103350 0x7f632819eff0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.581+0000 7f632ceda640 1 --2- 172.21.15.44:0/1471954418 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63281017a0 0x7f6328197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.581+0000 7f632ceda640 1 -- 172.21.15.44:0/1471954418 >> 172.21.15.44:0/1471954418 conn(0x7f63280fcd30 msgr2=0x7f6328105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:52.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.581+0000 7f632ceda640 1 -- 172.21.15.44:0/1471954418 shutdown_connections 2024-04-21T23:08:52.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.581+0000 7f632ceda640 1 -- 172.21.15.44:0/1471954418 wait complete. 2024-04-21T23:08:52.592 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":3,"stamp":"2024-04-21T23:08:51.615177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1185770,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":483,"num_read_kb":397,"num_write":384,"num_write_kb":4351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":790,"ondisk_log_size":790,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365052,"kb_used_data":15680,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124576772,"statfs":{"total":1151940427776,"available":1151566614528,"internally_reserved":0,"allocated":16056320,"data_stored":9417343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665739+0000","last_change":"2024-04-21T23:06:25.980091+0000","last_active":"2024-04-21T23:08:50.665739+0000","last_peered":"2024-04-21T23:08:50.665739+0000","last_clean":"2024-04-21T23:08:50.665739+0000","last_became_active":"2024-04-21T23:06:25.979763+0000","last_became_peered":"2024-04-21T23:06:25.979763+0000","last_unstale":"2024-04-21T23:08:50.665739+0000","last_undegraded":"2024-04-21T23:08:50.665739+0000","last_fullsized":"2024-04-21T23:08:50.665739+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:12:23.771931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735621+0000","last_change":"2024-04-21T23:06:25.992461+0000","last_active":"2024-04-21T23:08:50.735621+0000","last_peered":"2024-04-21T23:08:50.735621+0000","last_clean":"2024-04-21T23:08:50.735621+0000","last_became_active":"2024-04-21T23:06:25.992018+0000","last_became_peered":"2024-04-21T23:06:25.992018+0000","last_unstale":"2024-04-21T23:08:50.735621+0000","last_undegraded":"2024-04-21T23:08:50.735621+0000","last_fullsized":"2024-04-21T23:08:50.735621+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:46.651683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735329+0000","last_change":"2024-04-21T23:06:25.974812+0000","last_active":"2024-04-21T23:08:50.735329+0000","last_peered":"2024-04-21T23:08:50.735329+0000","last_clean":"2024-04-21T23:08:50.735329+0000","last_became_active":"2024-04-21T23:06:25.974420+0000","last_became_peered":"2024-04-21T23:06:25.974420+0000","last_unstale":"2024-04-21T23:08:50.735329+0000","last_undegraded":"2024-04-21T23:08:50.735329+0000","last_fullsized":"2024-04-21T23:08:50.735329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:00.734218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592764+0000","last_change":"2024-04-21T23:06:25.988806+0000","last_active":"2024-04-21T23:08:50.592764+0000","last_peered":"2024-04-21T23:08:50.592764+0000","last_clean":"2024-04-21T23:08:50.592764+0000","last_became_active":"2024-04-21T23:06:25.988344+0000","last_became_peered":"2024-04-21T23:06:25.988344+0000","last_unstale":"2024-04-21T23:08:50.592764+0000","last_undegraded":"2024-04-21T23:08:50.592764+0000","last_fullsized":"2024-04-21T23:08:50.592764+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:15:40.079273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592366+0000","last_change":"2024-04-21T23:06:25.979125+0000","last_active":"2024-04-21T23:08:50.592366+0000","last_peered":"2024-04-21T23:08:50.592366+0000","last_clean":"2024-04-21T23:08:50.592366+0000","last_became_active":"2024-04-21T23:06:25.978778+0000","last_became_peered":"2024-04-21T23:06:25.978778+0000","last_unstale":"2024-04-21T23:08:50.592366+0000","last_undegraded":"2024-04-21T23:08:50.592366+0000","last_fullsized":"2024-04-21T23:08:50.592366+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:44.655404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.713538+0000","last_change":"2024-04-21T23:06:25.983983+0000","last_active":"2024-04-21T23:08:50.713538+0000","last_peered":"2024-04-21T23:08:50.713538+0000","last_clean":"2024-04-21T23:08:50.713538+0000","last_became_active":"2024-04-21T23:06:25.983463+0000","last_became_peered":"2024-04-21T23:06:25.983463+0000","last_unstale":"2024-04-21T23:08:50.713538+0000","last_undegraded":"2024-04-21T23:08:50.713538+0000","last_fullsized":"2024-04-21T23:08:50.713538+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:32:12.784752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792374+0000","last_change":"2024-04-21T23:08:25.105761+0000","last_active":"2024-04-21T23:08:50.792374+0000","last_peered":"2024-04-21T23:08:50.792374+0000","last_clean":"2024-04-21T23:08:50.792374+0000","last_became_active":"2024-04-21T23:08:25.105339+0000","last_became_peered":"2024-04-21T23:08:25.105339+0000","last_unstale":"2024-04-21T23:08:50.792374+0000","last_undegraded":"2024-04-21T23:08:50.792374+0000","last_fullsized":"2024-04-21T23:08:50.792374+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:05:25.925743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.590750+0000","last_change":"2024-04-21T23:06:25.987714+0000","last_active":"2024-04-21T23:08:50.590750+0000","last_peered":"2024-04-21T23:08:50.590750+0000","last_clean":"2024-04-21T23:08:50.590750+0000","last_became_active":"2024-04-21T23:06:25.987457+0000","last_became_peered":"2024-04-21T23:06:25.987457+0000","last_unstale":"2024-04-21T23:08:50.590750+0000","last_undegraded":"2024-04-21T23:08:50.590750+0000","last_fullsized":"2024-04-21T23:08:50.590750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:06:06.184525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735072+0000","last_change":"2024-04-21T23:06:25.975957+0000","last_active":"2024-04-21T23:08:50.735072+0000","last_peered":"2024-04-21T23:08:50.735072+0000","last_clean":"2024-04-21T23:08:50.735072+0000","last_became_active":"2024-04-21T23:06:25.974710+0000","last_became_peered":"2024-04-21T23:06:25.974710+0000","last_unstale":"2024-04-21T23:08:50.735072+0000","last_undegraded":"2024-04-21T23:08:50.735072+0000","last_fullsized":"2024-04-21T23:08:50.735072+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:31:40.062273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.714037+0000","last_change":"2024-04-21T23:06:25.995085+0000","last_active":"2024-04-21T23:08:50.714037+0000","last_peered":"2024-04-21T23:08:50.714037+0000","last_clean":"2024-04-21T23:08:50.714037+0000","last_became_active":"2024-04-21T23:06:25.993552+0000","last_became_peered":"2024-04-21T23:06:25.993552+0000","last_unstale":"2024-04-21T23:08:50.714037+0000","last_undegraded":"2024-04-21T23:08:50.714037+0000","last_fullsized":"2024-04-21T23:08:50.714037+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:11.405113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.793253+0000","last_change":"2024-04-21T23:06:25.993191+0000","last_active":"2024-04-21T23:08:50.793253+0000","last_peered":"2024-04-21T23:08:50.793253+0000","last_clean":"2024-04-21T23:08:50.793253+0000","last_became_active":"2024-04-21T23:06:25.989399+0000","last_became_peered":"2024-04-21T23:06:25.989399+0000","last_unstale":"2024-04-21T23:08:50.793253+0000","last_undegraded":"2024-04-21T23:08:50.793253+0000","last_fullsized":"2024-04-21T23:08:50.793253+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:30:20.035599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592794+0000","last_change":"2024-04-21T23:06:25.994158+0000","last_active":"2024-04-21T23:08:50.592794+0000","last_peered":"2024-04-21T23:08:50.592794+0000","last_clean":"2024-04-21T23:08:50.592794+0000","last_became_active":"2024-04-21T23:06:25.993780+0000","last_became_peered":"2024-04-21T23:06:25.993780+0000","last_unstale":"2024-04-21T23:08:50.592794+0000","last_undegraded":"2024-04-21T23:08:50.592794+0000","last_fullsized":"2024-04-21T23:08:50.592794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:52:17.181898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.737142+0000","last_change":"2024-04-21T23:06:25.989274+0000","last_active":"2024-04-21T23:08:50.737142+0000","last_peered":"2024-04-21T23:08:50.737142+0000","last_clean":"2024-04-21T23:08:50.737142+0000","last_became_active":"2024-04-21T23:06:25.988931+0000","last_became_peered":"2024-04-21T23:06:25.988931+0000","last_unstale":"2024-04-21T23:08:50.737142+0000","last_undegraded":"2024-04-21T23:08:50.737142+0000","last_fullsized":"2024-04-21T23:08:50.737142+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:11:04.576136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735083+0000","last_change":"2024-04-21T23:06:25.982976+0000","last_active":"2024-04-21T23:08:50.735083+0000","last_peered":"2024-04-21T23:08:50.735083+0000","last_clean":"2024-04-21T23:08:50.735083+0000","last_became_active":"2024-04-21T23:06:25.982625+0000","last_became_peered":"2024-04-21T23:06:25.982625+0000","last_unstale":"2024-04-21T23:08:50.735083+0000","last_undegraded":"2024-04-21T23:08:50.735083+0000","last_fullsized":"2024-04-21T23:08:50.735083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:51:02.193044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735274+0000","last_change":"2024-04-21T23:06:25.994805+0000","last_active":"2024-04-21T23:08:50.735274+0000","last_peered":"2024-04-21T23:08:50.735274+0000","last_clean":"2024-04-21T23:08:50.735274+0000","last_became_active":"2024-04-21T23:06:25.994482+0000","last_became_peered":"2024-04-21T23:06:25.994482+0000","last_unstale":"2024-04-21T23:08:50.735274+0000","last_undegraded":"2024-04-21T23:08:50.735274+0000","last_fullsized":"2024-04-21T23:08:50.735274+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:24:21.792358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734531+0000","last_change":"2024-04-21T23:06:25.990840+0000","last_active":"2024-04-21T23:08:50.734531+0000","last_peered":"2024-04-21T23:08:50.734531+0000","last_clean":"2024-04-21T23:08:50.734531+0000","last_became_active":"2024-04-21T23:06:25.990517+0000","last_became_peered":"2024-04-21T23:06:25.990517+0000","last_unstale":"2024-04-21T23:08:50.734531+0000","last_undegraded":"2024-04-21T23:08:50.734531+0000","last_fullsized":"2024-04-21T23:08:50.734531+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:48:09.265280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665710+0000","last_change":"2024-04-21T23:06:25.976810+0000","last_active":"2024-04-21T23:08:50.665710+0000","last_peered":"2024-04-21T23:08:50.665710+0000","last_clean":"2024-04-21T23:08:50.665710+0000","last_became_active":"2024-04-21T23:06:25.976489+0000","last_became_peered":"2024-04-21T23:06:25.976489+0000","last_unstale":"2024-04-21T23:08:50.665710+0000","last_undegraded":"2024-04-21T23:08:50.665710+0000","last_fullsized":"2024-04-21T23:08:50.665710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:07:03.794115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700626+0000","last_change":"2024-04-21T23:06:25.988603+0000","last_active":"2024-04-21T23:08:50.700626+0000","last_peered":"2024-04-21T23:08:50.700626+0000","last_clean":"2024-04-21T23:08:50.700626+0000","last_became_active":"2024-04-21T23:06:25.988254+0000","last_became_peered":"2024-04-21T23:06:25.988254+0000","last_unstale":"2024-04-21T23:08:50.700626+0000","last_undegraded":"2024-04-21T23:08:50.700626+0000","last_fullsized":"2024-04-21T23:08:50.700626+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:25:46.099853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734889+0000","last_change":"2024-04-21T23:06:25.980831+0000","last_active":"2024-04-21T23:08:50.734889+0000","last_peered":"2024-04-21T23:08:50.734889+0000","last_clean":"2024-04-21T23:08:50.734889+0000","last_became_active":"2024-04-21T23:06:25.977051+0000","last_became_peered":"2024-04-21T23:06:25.977051+0000","last_unstale":"2024-04-21T23:08:50.734889+0000","last_undegraded":"2024-04-21T23:08:50.734889+0000","last_fullsized":"2024-04-21T23:08:50.734889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:55:03.684734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734914+0000","last_change":"2024-04-21T23:06:30.256875+0000","last_active":"2024-04-21T23:08:50.734914+0000","last_peered":"2024-04-21T23:08:50.734914+0000","last_clean":"2024-04-21T23:08:50.734914+0000","last_became_active":"2024-04-21T23:06:30.256459+0000","last_became_peered":"2024-04-21T23:06:30.256459+0000","last_unstale":"2024-04-21T23:08:50.734914+0000","last_undegraded":"2024-04-21T23:08:50.734914+0000","last_fullsized":"2024-04-21T23:08:50.734914+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:45:32.376270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700758+0000","last_change":"2024-04-21T23:06:25.209605+0000","last_active":"2024-04-21T23:08:50.700758+0000","last_peered":"2024-04-21T23:08:50.700758+0000","last_clean":"2024-04-21T23:08:50.700758+0000","last_became_active":"2024-04-21T23:06:25.209277+0000","last_became_peered":"2024-04-21T23:06:25.209277+0000","last_unstale":"2024-04-21T23:08:50.700758+0000","last_undegraded":"2024-04-21T23:08:50.700758+0000","last_fullsized":"2024-04-21T23:08:50.700758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:15:49.742940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.591263+0000","last_change":"2024-04-21T23:06:25.981774+0000","last_active":"2024-04-21T23:08:50.591263+0000","last_peered":"2024-04-21T23:08:50.591263+0000","last_clean":"2024-04-21T23:08:50.591263+0000","last_became_active":"2024-04-21T23:06:25.981432+0000","last_became_peered":"2024-04-21T23:06:25.981432+0000","last_unstale":"2024-04-21T23:08:50.591263+0000","last_undegraded":"2024-04-21T23:08:50.591263+0000","last_fullsized":"2024-04-21T23:08:50.591263+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:28:28.712149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592412+0000","last_change":"2024-04-21T23:06:30.271508+0000","last_active":"2024-04-21T23:08:50.592412+0000","last_peered":"2024-04-21T23:08:50.592412+0000","last_clean":"2024-04-21T23:08:50.592412+0000","last_became_active":"2024-04-21T23:06:30.270461+0000","last_became_peered":"2024-04-21T23:06:30.270461+0000","last_unstale":"2024-04-21T23:08:50.592412+0000","last_undegraded":"2024-04-21T23:08:50.592412+0000","last_fullsized":"2024-04-21T23:08:50.592412+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:43:07.473365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.591569+0000","last_change":"2024-04-21T23:06:25.218773+0000","last_active":"2024-04-21T23:08:50.591569+0000","last_peered":"2024-04-21T23:08:50.591569+0000","last_clean":"2024-04-21T23:08:50.591569+0000","last_became_active":"2024-04-21T23:06:25.218458+0000","last_became_peered":"2024-04-21T23:06:25.218458+0000","last_unstale":"2024-04-21T23:08:50.591569+0000","last_undegraded":"2024-04-21T23:08:50.591569+0000","last_fullsized":"2024-04-21T23:08:50.591569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:25:16.372689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735251+0000","last_change":"2024-04-21T23:06:25.977669+0000","last_active":"2024-04-21T23:08:50.735251+0000","last_peered":"2024-04-21T23:08:50.735251+0000","last_clean":"2024-04-21T23:08:50.735251+0000","last_became_active":"2024-04-21T23:06:25.977143+0000","last_became_peered":"2024-04-21T23:06:25.977143+0000","last_unstale":"2024-04-21T23:08:50.735251+0000","last_undegraded":"2024-04-21T23:08:50.735251+0000","last_fullsized":"2024-04-21T23:08:50.735251+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:39.289208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592447+0000","last_change":"2024-04-21T23:06:30.271934+0000","last_active":"2024-04-21T23:08:50.592447+0000","last_peered":"2024-04-21T23:08:50.592447+0000","last_clean":"2024-04-21T23:08:50.592447+0000","last_became_active":"2024-04-21T23:06:30.270567+0000","last_became_peered":"2024-04-21T23:06:30.270567+0000","last_unstale":"2024-04-21T23:08:50.592447+0000","last_undegraded":"2024-04-21T23:08:50.592447+0000","last_fullsized":"2024-04-21T23:08:50.592447+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:43:34.908748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592827+0000","last_change":"2024-04-21T23:07:24.462595+0000","last_active":"2024-04-21T23:08:50.592827+0000","last_peered":"2024-04-21T23:08:50.592827+0000","last_clean":"2024-04-21T23:08:50.592827+0000","last_became_active":"2024-04-21T23:07:24.454522+0000","last_became_peered":"2024-04-21T23:07:24.454522+0000","last_unstale":"2024-04-21T23:08:50.592827+0000","last_undegraded":"2024-04-21T23:08:50.592827+0000","last_fullsized":"2024-04-21T23:08:50.592827+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:07.457105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734789+0000","last_change":"2024-04-21T23:06:25.988197+0000","last_active":"2024-04-21T23:08:50.734789+0000","last_peered":"2024-04-21T23:08:50.734789+0000","last_clean":"2024-04-21T23:08:50.734789+0000","last_became_active":"2024-04-21T23:06:25.987872+0000","last_became_peered":"2024-04-21T23:06:25.987872+0000","last_unstale":"2024-04-21T23:08:50.734789+0000","last_undegraded":"2024-04-21T23:08:50.734789+0000","last_fullsized":"2024-04-21T23:08:50.734789+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:25.389020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592402+0000","last_change":"2024-04-21T23:06:30.265262+0000","last_active":"2024-04-21T23:08:50.592402+0000","last_peered":"2024-04-21T23:08:50.592402+0000","last_clean":"2024-04-21T23:08:50.592402+0000","last_became_active":"2024-04-21T23:06:30.264781+0000","last_became_peered":"2024-04-21T23:06:30.264781+0000","last_unstale":"2024-04-21T23:08:50.592402+0000","last_undegraded":"2024-04-21T23:08:50.592402+0000","last_fullsized":"2024-04-21T23:08:50.592402+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:03:30.756792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592822+0000","last_change":"2024-04-21T23:06:25.205070+0000","last_active":"2024-04-21T23:08:50.592822+0000","last_peered":"2024-04-21T23:08:50.592822+0000","last_clean":"2024-04-21T23:08:50.592822+0000","last_became_active":"2024-04-21T23:06:25.204737+0000","last_became_peered":"2024-04-21T23:06:25.204737+0000","last_unstale":"2024-04-21T23:08:50.592822+0000","last_undegraded":"2024-04-21T23:08:50.592822+0000","last_fullsized":"2024-04-21T23:08:50.592822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:37.014534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710271+0000","last_change":"2024-04-21T23:06:25.986584+0000","last_active":"2024-04-21T23:08:50.710271+0000","last_peered":"2024-04-21T23:08:50.710271+0000","last_clean":"2024-04-21T23:08:50.710271+0000","last_became_active":"2024-04-21T23:06:25.986268+0000","last_became_peered":"2024-04-21T23:06:25.986268+0000","last_unstale":"2024-04-21T23:08:50.710271+0000","last_undegraded":"2024-04-21T23:08:50.710271+0000","last_fullsized":"2024-04-21T23:08:50.710271+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:19:43.864279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734739+0000","last_change":"2024-04-21T23:06:30.261900+0000","last_active":"2024-04-21T23:08:50.734739+0000","last_peered":"2024-04-21T23:08:50.734739+0000","last_clean":"2024-04-21T23:08:50.734739+0000","last_became_active":"2024-04-21T23:06:30.261444+0000","last_became_peered":"2024-04-21T23:06:30.261444+0000","last_unstale":"2024-04-21T23:08:50.734739+0000","last_undegraded":"2024-04-21T23:08:50.734739+0000","last_fullsized":"2024-04-21T23:08:50.734739+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:10:40.810555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710271+0000","last_change":"2024-04-21T23:07:24.817616+0000","last_active":"2024-04-21T23:08:50.710271+0000","last_peered":"2024-04-21T23:08:50.710271+0000","last_clean":"2024-04-21T23:08:50.710271+0000","last_became_active":"2024-04-21T23:07:24.816888+0000","last_became_peered":"2024-04-21T23:07:24.816888+0000","last_unstale":"2024-04-21T23:08:50.710271+0000","last_undegraded":"2024-04-21T23:08:50.710271+0000","last_fullsized":"2024-04-21T23:08:50.710271+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:01:03.069832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592392+0000","last_change":"2024-04-21T23:06:25.975999+0000","last_active":"2024-04-21T23:08:50.592392+0000","last_peered":"2024-04-21T23:08:50.592392+0000","last_clean":"2024-04-21T23:08:50.592392+0000","last_became_active":"2024-04-21T23:06:25.975677+0000","last_became_peered":"2024-04-21T23:06:25.975677+0000","last_unstale":"2024-04-21T23:08:50.592392+0000","last_undegraded":"2024-04-21T23:08:50.592392+0000","last_fullsized":"2024-04-21T23:08:50.592392+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:52:03.177248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734878+0000","last_change":"2024-04-21T23:07:24.456926+0000","last_active":"2024-04-21T23:08:50.734878+0000","last_peered":"2024-04-21T23:08:50.734878+0000","last_clean":"2024-04-21T23:08:50.734878+0000","last_became_active":"2024-04-21T23:07:24.456481+0000","last_became_peered":"2024-04-21T23:07:24.456481+0000","last_unstale":"2024-04-21T23:08:50.734878+0000","last_undegraded":"2024-04-21T23:08:50.734878+0000","last_fullsized":"2024-04-21T23:08:50.734878+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:19:26.553012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.713581+0000","last_change":"2024-04-21T23:06:25.214893+0000","last_active":"2024-04-21T23:08:50.713581+0000","last_peered":"2024-04-21T23:08:50.713581+0000","last_clean":"2024-04-21T23:08:50.713581+0000","last_became_active":"2024-04-21T23:06:25.212359+0000","last_became_peered":"2024-04-21T23:06:25.212359+0000","last_unstale":"2024-04-21T23:08:50.713581+0000","last_undegraded":"2024-04-21T23:08:50.713581+0000","last_fullsized":"2024-04-21T23:08:50.713581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:18:03.278875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710458+0000","last_change":"2024-04-21T23:06:25.986849+0000","last_active":"2024-04-21T23:08:50.710458+0000","last_peered":"2024-04-21T23:08:50.710458+0000","last_clean":"2024-04-21T23:08:50.710458+0000","last_became_active":"2024-04-21T23:06:25.986334+0000","last_became_peered":"2024-04-21T23:06:25.986334+0000","last_unstale":"2024-04-21T23:08:50.710458+0000","last_undegraded":"2024-04-21T23:08:50.710458+0000","last_fullsized":"2024-04-21T23:08:50.710458+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:57:42.195629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665559+0000","last_change":"2024-04-21T23:06:30.270812+0000","last_active":"2024-04-21T23:08:50.665559+0000","last_peered":"2024-04-21T23:08:50.665559+0000","last_clean":"2024-04-21T23:08:50.665559+0000","last_became_active":"2024-04-21T23:06:30.270370+0000","last_became_peered":"2024-04-21T23:06:30.270370+0000","last_unstale":"2024-04-21T23:08:50.665559+0000","last_undegraded":"2024-04-21T23:08:50.665559+0000","last_fullsized":"2024-04-21T23:08:50.665559+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:53.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734920+0000","last_change":"2024-04-21T23:06:25.201603+0000","last_active":"2024-04-21T23:08:50.734920+0000","last_peered":"2024-04-21T23:08:50.734920+0000","last_clean":"2024-04-21T23:08:50.734920+0000","last_became_active":"2024-04-21T23:06:25.201265+0000","last_became_peered":"2024-04-21T23:06:25.201265+0000","last_unstale":"2024-04-21T23:08:50.734920+0000","last_undegraded":"2024-04-21T23:08:50.734920+0000","last_fullsized":"2024-04-21T23:08:50.734920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:23:19.134215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.590931+0000","last_change":"2024-04-21T23:06:25.982420+0000","last_active":"2024-04-21T23:08:50.590931+0000","last_peered":"2024-04-21T23:08:50.590931+0000","last_clean":"2024-04-21T23:08:50.590931+0000","last_became_active":"2024-04-21T23:06:25.981867+0000","last_became_peered":"2024-04-21T23:06:25.981867+0000","last_unstale":"2024-04-21T23:08:50.590931+0000","last_undegraded":"2024-04-21T23:08:50.590931+0000","last_fullsized":"2024-04-21T23:08:50.590931+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:10:29.433243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735614+0000","last_change":"2024-04-21T23:06:30.273907+0000","last_active":"2024-04-21T23:08:50.735614+0000","last_peered":"2024-04-21T23:08:50.735614+0000","last_clean":"2024-04-21T23:08:50.735614+0000","last_became_active":"2024-04-21T23:06:30.273472+0000","last_became_peered":"2024-04-21T23:06:30.273472+0000","last_unstale":"2024-04-21T23:08:50.735614+0000","last_undegraded":"2024-04-21T23:08:50.735614+0000","last_fullsized":"2024-04-21T23:08:50.735614+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:58:50.914940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710022+0000","last_change":"2024-04-21T23:06:25.211117+0000","last_active":"2024-04-21T23:08:50.710022+0000","last_peered":"2024-04-21T23:08:50.710022+0000","last_clean":"2024-04-21T23:08:50.710022+0000","last_became_active":"2024-04-21T23:06:25.210626+0000","last_became_peered":"2024-04-21T23:06:25.210626+0000","last_unstale":"2024-04-21T23:08:50.710022+0000","last_undegraded":"2024-04-21T23:08:50.710022+0000","last_fullsized":"2024-04-21T23:08:50.710022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:56:39.917072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700098+0000","last_change":"2024-04-21T23:06:25.980537+0000","last_active":"2024-04-21T23:08:50.700098+0000","last_peered":"2024-04-21T23:08:50.700098+0000","last_clean":"2024-04-21T23:08:50.700098+0000","last_became_active":"2024-04-21T23:06:25.980188+0000","last_became_peered":"2024-04-21T23:06:25.980188+0000","last_unstale":"2024-04-21T23:08:50.700098+0000","last_undegraded":"2024-04-21T23:08:50.700098+0000","last_fullsized":"2024-04-21T23:08:50.700098+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:58:02.828632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.590859+0000","last_change":"2024-04-21T23:06:30.263565+0000","last_active":"2024-04-21T23:08:50.590859+0000","last_peered":"2024-04-21T23:08:50.590859+0000","last_clean":"2024-04-21T23:08:50.590859+0000","last_became_active":"2024-04-21T23:06:30.263092+0000","last_became_peered":"2024-04-21T23:06:30.263092+0000","last_unstale":"2024-04-21T23:08:50.590859+0000","last_undegraded":"2024-04-21T23:08:50.590859+0000","last_fullsized":"2024-04-21T23:08:50.590859+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:58:25.709551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.714012+0000","last_change":"2024-04-21T23:06:25.212567+0000","last_active":"2024-04-21T23:08:50.714012+0000","last_peered":"2024-04-21T23:08:50.714012+0000","last_clean":"2024-04-21T23:08:50.714012+0000","last_became_active":"2024-04-21T23:06:25.210593+0000","last_became_peered":"2024-04-21T23:06:25.210593+0000","last_unstale":"2024-04-21T23:08:50.714012+0000","last_undegraded":"2024-04-21T23:08:50.714012+0000","last_fullsized":"2024-04-21T23:08:50.714012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:53:33.552742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735193+0000","last_change":"2024-04-21T23:06:25.989525+0000","last_active":"2024-04-21T23:08:50.735193+0000","last_peered":"2024-04-21T23:08:50.735193+0000","last_clean":"2024-04-21T23:08:50.735193+0000","last_became_active":"2024-04-21T23:06:25.989130+0000","last_became_peered":"2024-04-21T23:06:25.989130+0000","last_unstale":"2024-04-21T23:08:50.735193+0000","last_undegraded":"2024-04-21T23:08:50.735193+0000","last_fullsized":"2024-04-21T23:08:50.735193+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:19:31.000530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.699989+0000","last_change":"2024-04-21T23:06:30.269309+0000","last_active":"2024-04-21T23:08:50.699989+0000","last_peered":"2024-04-21T23:08:50.699989+0000","last_clean":"2024-04-21T23:08:50.699989+0000","last_became_active":"2024-04-21T23:06:30.268903+0000","last_became_peered":"2024-04-21T23:06:30.268903+0000","last_unstale":"2024-04-21T23:08:50.699989+0000","last_undegraded":"2024-04-21T23:08:50.699989+0000","last_fullsized":"2024-04-21T23:08:50.699989+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:52:16.043870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792659+0000","last_change":"2024-04-21T23:06:25.213938+0000","last_active":"2024-04-21T23:08:50.792659+0000","last_peered":"2024-04-21T23:08:50.792659+0000","last_clean":"2024-04-21T23:08:50.792659+0000","last_became_active":"2024-04-21T23:06:25.213242+0000","last_became_peered":"2024-04-21T23:06:25.213242+0000","last_unstale":"2024-04-21T23:08:50.792659+0000","last_undegraded":"2024-04-21T23:08:50.792659+0000","last_fullsized":"2024-04-21T23:08:50.792659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:27.679122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735990+0000","last_change":"2024-04-21T23:07:24.454702+0000","last_active":"2024-04-21T23:08:50.735990+0000","last_peered":"2024-04-21T23:08:50.735990+0000","last_clean":"2024-04-21T23:08:50.735990+0000","last_became_active":"2024-04-21T23:07:24.454211+0000","last_became_peered":"2024-04-21T23:07:24.454211+0000","last_unstale":"2024-04-21T23:08:50.735990+0000","last_undegraded":"2024-04-21T23:08:50.735990+0000","last_fullsized":"2024-04-21T23:08:50.735990+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:29:30.745026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592728+0000","last_change":"2024-04-21T23:06:30.272206+0000","last_active":"2024-04-21T23:08:50.592728+0000","last_peered":"2024-04-21T23:08:50.592728+0000","last_clean":"2024-04-21T23:08:50.592728+0000","last_became_active":"2024-04-21T23:06:30.271389+0000","last_became_peered":"2024-04-21T23:06:30.271389+0000","last_unstale":"2024-04-21T23:08:50.592728+0000","last_undegraded":"2024-04-21T23:08:50.592728+0000","last_fullsized":"2024-04-21T23:08:50.592728+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:54:47.996166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710352+0000","last_change":"2024-04-21T23:06:25.217655+0000","last_active":"2024-04-21T23:08:50.710352+0000","last_peered":"2024-04-21T23:08:50.710352+0000","last_clean":"2024-04-21T23:08:50.710352+0000","last_became_active":"2024-04-21T23:06:25.217337+0000","last_became_peered":"2024-04-21T23:06:25.217337+0000","last_unstale":"2024-04-21T23:08:50.710352+0000","last_undegraded":"2024-04-21T23:08:50.710352+0000","last_fullsized":"2024-04-21T23:08:50.710352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:29:28.690841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735023+0000","last_change":"2024-04-21T23:06:25.995757+0000","last_active":"2024-04-21T23:08:50.735023+0000","last_peered":"2024-04-21T23:08:50.735023+0000","last_clean":"2024-04-21T23:08:50.735023+0000","last_became_active":"2024-04-21T23:06:25.995443+0000","last_became_peered":"2024-04-21T23:06:25.995443+0000","last_unstale":"2024-04-21T23:08:50.735023+0000","last_undegraded":"2024-04-21T23:08:50.735023+0000","last_fullsized":"2024-04-21T23:08:50.735023+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:40.359633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.699925+0000","last_change":"2024-04-21T23:06:30.266441+0000","last_active":"2024-04-21T23:08:50.699925+0000","last_peered":"2024-04-21T23:08:50.699925+0000","last_clean":"2024-04-21T23:08:50.699925+0000","last_became_active":"2024-04-21T23:06:30.264065+0000","last_became_peered":"2024-04-21T23:06:30.264065+0000","last_unstale":"2024-04-21T23:08:50.699925+0000","last_undegraded":"2024-04-21T23:08:50.699925+0000","last_fullsized":"2024-04-21T23:08:50.699925+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:53:22.548096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'38","reported_seq":324,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.953842+0000","last_change":"2024-04-21T23:06:25.213919+0000","last_active":"2024-04-21T23:08:50.953842+0000","last_peered":"2024-04-21T23:08:50.953842+0000","last_clean":"2024-04-21T23:08:50.953842+0000","last_became_active":"2024-04-21T23:06:25.212244+0000","last_became_peered":"2024-04-21T23:06:25.212244+0000","last_unstale":"2024-04-21T23:08:50.953842+0000","last_undegraded":"2024-04-21T23:08:50.953842+0000","last_fullsized":"2024-04-21T23:08:50.953842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:17:25.287718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69554,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":196,"num_write":38,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592581+0000","last_change":"2024-04-21T23:06:25.993666+0000","last_active":"2024-04-21T23:08:50.592581+0000","last_peered":"2024-04-21T23:08:50.592581+0000","last_clean":"2024-04-21T23:08:50.592581+0000","last_became_active":"2024-04-21T23:06:25.993330+0000","last_became_peered":"2024-04-21T23:06:25.993330+0000","last_unstale":"2024-04-21T23:08:50.592581+0000","last_undegraded":"2024-04-21T23:08:50.592581+0000","last_fullsized":"2024-04-21T23:08:50.592581+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:11:00.109782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710648+0000","last_change":"2024-04-21T23:06:30.265807+0000","last_active":"2024-04-21T23:08:50.710648+0000","last_peered":"2024-04-21T23:08:50.710648+0000","last_clean":"2024-04-21T23:08:50.710648+0000","last_became_active":"2024-04-21T23:06:30.265116+0000","last_became_peered":"2024-04-21T23:06:30.265116+0000","last_unstale":"2024-04-21T23:08:50.710648+0000","last_undegraded":"2024-04-21T23:08:50.710648+0000","last_fullsized":"2024-04-21T23:08:50.710648+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:14:04.178281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734740+0000","last_change":"2024-04-21T23:06:25.208319+0000","last_active":"2024-04-21T23:08:50.734740+0000","last_peered":"2024-04-21T23:08:50.734740+0000","last_clean":"2024-04-21T23:08:50.734740+0000","last_became_active":"2024-04-21T23:06:25.208025+0000","last_became_peered":"2024-04-21T23:06:25.208025+0000","last_unstale":"2024-04-21T23:08:50.734740+0000","last_undegraded":"2024-04-21T23:08:50.734740+0000","last_fullsized":"2024-04-21T23:08:50.734740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:14.723170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.736590+0000","last_change":"2024-04-21T23:06:25.993604+0000","last_active":"2024-04-21T23:08:50.736590+0000","last_peered":"2024-04-21T23:08:50.736590+0000","last_clean":"2024-04-21T23:08:50.736590+0000","last_became_active":"2024-04-21T23:06:25.993217+0000","last_became_peered":"2024-04-21T23:06:25.993217+0000","last_unstale":"2024-04-21T23:08:50.736590+0000","last_undegraded":"2024-04-21T23:08:50.736590+0000","last_fullsized":"2024-04-21T23:08:50.736590+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:23:00.435688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665369+0000","last_change":"2024-04-21T23:06:30.270991+0000","last_active":"2024-04-21T23:08:50.665369+0000","last_peered":"2024-04-21T23:08:50.665369+0000","last_clean":"2024-04-21T23:08:50.665369+0000","last_became_active":"2024-04-21T23:06:30.270511+0000","last_became_peered":"2024-04-21T23:06:30.270511+0000","last_unstale":"2024-04-21T23:08:50.665369+0000","last_undegraded":"2024-04-21T23:08:50.665369+0000","last_fullsized":"2024-04-21T23:08:50.665369+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:04.265855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710232+0000","last_change":"2024-04-21T23:06:25.211787+0000","last_active":"2024-04-21T23:08:50.710232+0000","last_peered":"2024-04-21T23:08:50.710232+0000","last_clean":"2024-04-21T23:08:50.710232+0000","last_became_active":"2024-04-21T23:06:25.211473+0000","last_became_peered":"2024-04-21T23:06:25.211473+0000","last_unstale":"2024-04-21T23:08:50.710232+0000","last_undegraded":"2024-04-21T23:08:50.710232+0000","last_fullsized":"2024-04-21T23:08:50.710232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:58:51.364974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792678+0000","last_change":"2024-04-21T23:06:25.990681+0000","last_active":"2024-04-21T23:08:50.792678+0000","last_peered":"2024-04-21T23:08:50.792678+0000","last_clean":"2024-04-21T23:08:50.792678+0000","last_became_active":"2024-04-21T23:06:25.990094+0000","last_became_peered":"2024-04-21T23:06:25.990094+0000","last_unstale":"2024-04-21T23:08:50.792678+0000","last_undegraded":"2024-04-21T23:08:50.792678+0000","last_fullsized":"2024-04-21T23:08:50.792678+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:15:26.907912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700460+0000","last_change":"2024-04-21T23:06:30.271352+0000","last_active":"2024-04-21T23:08:50.700460+0000","last_peered":"2024-04-21T23:08:50.700460+0000","last_clean":"2024-04-21T23:08:50.700460+0000","last_became_active":"2024-04-21T23:06:30.270601+0000","last_became_peered":"2024-04-21T23:06:30.270601+0000","last_unstale":"2024-04-21T23:08:50.700460+0000","last_undegraded":"2024-04-21T23:08:50.700460+0000","last_fullsized":"2024-04-21T23:08:50.700460+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:37.266987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592639+0000","last_change":"2024-04-21T23:06:25.213288+0000","last_active":"2024-04-21T23:08:50.592639+0000","last_peered":"2024-04-21T23:08:50.592639+0000","last_clean":"2024-04-21T23:08:50.592639+0000","last_became_active":"2024-04-21T23:06:25.212936+0000","last_became_peered":"2024-04-21T23:06:25.212936+0000","last_unstale":"2024-04-21T23:08:50.592639+0000","last_undegraded":"2024-04-21T23:08:50.592639+0000","last_fullsized":"2024-04-21T23:08:50.592639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:35.303093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792437+0000","last_change":"2024-04-21T23:06:25.998485+0000","last_active":"2024-04-21T23:08:50.792437+0000","last_peered":"2024-04-21T23:08:50.792437+0000","last_clean":"2024-04-21T23:08:50.792437+0000","last_became_active":"2024-04-21T23:06:25.998184+0000","last_became_peered":"2024-04-21T23:06:25.998184+0000","last_unstale":"2024-04-21T23:08:50.792437+0000","last_undegraded":"2024-04-21T23:08:50.792437+0000","last_fullsized":"2024-04-21T23:08:50.792437+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:35:26.613636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.736096+0000","last_change":"2024-04-21T23:06:30.270089+0000","last_active":"2024-04-21T23:08:50.736096+0000","last_peered":"2024-04-21T23:08:50.736096+0000","last_clean":"2024-04-21T23:08:50.736096+0000","last_became_active":"2024-04-21T23:06:30.269695+0000","last_became_peered":"2024-04-21T23:06:30.269695+0000","last_unstale":"2024-04-21T23:08:50.736096+0000","last_undegraded":"2024-04-21T23:08:50.736096+0000","last_fullsized":"2024-04-21T23:08:50.736096+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:42.718580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592339+0000","last_change":"2024-04-21T23:06:25.218247+0000","last_active":"2024-04-21T23:08:50.592339+0000","last_peered":"2024-04-21T23:08:50.592339+0000","last_clean":"2024-04-21T23:08:50.592339+0000","last_became_active":"2024-04-21T23:06:25.217924+0000","last_became_peered":"2024-04-21T23:06:25.217924+0000","last_unstale":"2024-04-21T23:08:50.592339+0000","last_undegraded":"2024-04-21T23:08:50.592339+0000","last_fullsized":"2024-04-21T23:08:50.592339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:27:12.155597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710045+0000","last_change":"2024-04-21T23:06:25.994109+0000","last_active":"2024-04-21T23:08:50.710045+0000","last_peered":"2024-04-21T23:08:50.710045+0000","last_clean":"2024-04-21T23:08:50.710045+0000","last_became_active":"2024-04-21T23:06:25.993785+0000","last_became_peered":"2024-04-21T23:06:25.993785+0000","last_unstale":"2024-04-21T23:08:50.710045+0000","last_undegraded":"2024-04-21T23:08:50.710045+0000","last_fullsized":"2024-04-21T23:08:50.710045+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:12.629364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.709882+0000","last_change":"2024-04-21T23:06:30.265444+0000","last_active":"2024-04-21T23:08:50.709882+0000","last_peered":"2024-04-21T23:08:50.709882+0000","last_clean":"2024-04-21T23:08:50.709882+0000","last_became_active":"2024-04-21T23:06:30.264971+0000","last_became_peered":"2024-04-21T23:06:30.264971+0000","last_unstale":"2024-04-21T23:08:50.709882+0000","last_undegraded":"2024-04-21T23:08:50.709882+0000","last_fullsized":"2024-04-21T23:08:50.709882+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:03.596631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665514+0000","last_change":"2024-04-21T23:07:24.465377+0000","last_active":"2024-04-21T23:08:50.665514+0000","last_peered":"2024-04-21T23:08:50.665514+0000","last_clean":"2024-04-21T23:08:50.665514+0000","last_became_active":"2024-04-21T23:07:24.454409+0000","last_became_peered":"2024-04-21T23:07:24.454409+0000","last_unstale":"2024-04-21T23:08:50.665514+0000","last_undegraded":"2024-04-21T23:08:50.665514+0000","last_fullsized":"2024-04-21T23:08:50.665514+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:06:22.836124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665569+0000","last_change":"2024-04-21T23:06:25.990193+0000","last_active":"2024-04-21T23:08:50.665569+0000","last_peered":"2024-04-21T23:08:50.665569+0000","last_clean":"2024-04-21T23:08:50.665569+0000","last_became_active":"2024-04-21T23:06:25.989842+0000","last_became_peered":"2024-04-21T23:06:25.989842+0000","last_unstale":"2024-04-21T23:08:50.665569+0000","last_undegraded":"2024-04-21T23:08:50.665569+0000","last_fullsized":"2024-04-21T23:08:50.665569+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:54:53.421257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592553+0000","last_change":"2024-04-21T23:07:24.817262+0000","last_active":"2024-04-21T23:08:50.592553+0000","last_peered":"2024-04-21T23:08:50.592553+0000","last_clean":"2024-04-21T23:08:50.592553+0000","last_became_active":"2024-04-21T23:07:24.816668+0000","last_became_peered":"2024-04-21T23:07:24.816668+0000","last_unstale":"2024-04-21T23:08:50.592553+0000","last_undegraded":"2024-04-21T23:08:50.592553+0000","last_fullsized":"2024-04-21T23:08:50.592553+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:25.302151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700278+0000","last_change":"2024-04-21T23:06:25.218208+0000","last_active":"2024-04-21T23:08:50.700278+0000","last_peered":"2024-04-21T23:08:50.700278+0000","last_clean":"2024-04-21T23:08:50.700278+0000","last_became_active":"2024-04-21T23:06:25.217889+0000","last_became_peered":"2024-04-21T23:06:25.217889+0000","last_unstale":"2024-04-21T23:08:50.700278+0000","last_undegraded":"2024-04-21T23:08:50.700278+0000","last_fullsized":"2024-04-21T23:08:50.700278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:00:08.775682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.713684+0000","last_change":"2024-04-21T23:06:25.987615+0000","last_active":"2024-04-21T23:08:50.713684+0000","last_peered":"2024-04-21T23:08:50.713684+0000","last_clean":"2024-04-21T23:08:50.713684+0000","last_became_active":"2024-04-21T23:06:25.987305+0000","last_became_peered":"2024-04-21T23:06:25.987305+0000","last_unstale":"2024-04-21T23:08:50.713684+0000","last_undegraded":"2024-04-21T23:08:50.713684+0000","last_fullsized":"2024-04-21T23:08:50.713684+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:23:49.386998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735164+0000","last_change":"2024-04-21T23:06:30.268157+0000","last_active":"2024-04-21T23:08:50.735164+0000","last_peered":"2024-04-21T23:08:50.735164+0000","last_clean":"2024-04-21T23:08:50.735164+0000","last_became_active":"2024-04-21T23:06:30.267732+0000","last_became_peered":"2024-04-21T23:06:30.267732+0000","last_unstale":"2024-04-21T23:08:50.735164+0000","last_undegraded":"2024-04-21T23:08:50.735164+0000","last_fullsized":"2024-04-21T23:08:50.735164+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:29:24.250271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792667+0000","last_change":"2024-04-21T23:06:25.215401+0000","last_active":"2024-04-21T23:08:50.792667+0000","last_peered":"2024-04-21T23:08:50.792667+0000","last_clean":"2024-04-21T23:08:50.792667+0000","last_became_active":"2024-04-21T23:06:25.215067+0000","last_became_peered":"2024-04-21T23:06:25.215067+0000","last_unstale":"2024-04-21T23:08:50.792667+0000","last_undegraded":"2024-04-21T23:08:50.792667+0000","last_fullsized":"2024-04-21T23:08:50.792667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:16:58.977797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.713637+0000","last_change":"2024-04-21T23:06:25.996421+0000","last_active":"2024-04-21T23:08:50.713637+0000","last_peered":"2024-04-21T23:08:50.713637+0000","last_clean":"2024-04-21T23:08:50.713637+0000","last_became_active":"2024-04-21T23:06:25.996102+0000","last_became_peered":"2024-04-21T23:06:25.996102+0000","last_unstale":"2024-04-21T23:08:50.713637+0000","last_undegraded":"2024-04-21T23:08:50.713637+0000","last_fullsized":"2024-04-21T23:08:50.713637+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:06:42.630182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.736211+0000","last_change":"2024-04-21T23:06:30.404509+0000","last_active":"2024-04-21T23:08:50.736211+0000","last_peered":"2024-04-21T23:08:50.736211+0000","last_clean":"2024-04-21T23:08:50.736211+0000","last_became_active":"2024-04-21T23:06:30.403987+0000","last_became_peered":"2024-04-21T23:06:30.403987+0000","last_unstale":"2024-04-21T23:08:50.736211+0000","last_undegraded":"2024-04-21T23:08:50.736211+0000","last_fullsized":"2024-04-21T23:08:50.736211+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:12:28.412117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710087+0000","last_change":"2024-04-21T23:06:25.994076+0000","last_active":"2024-04-21T23:08:50.710087+0000","last_peered":"2024-04-21T23:08:50.710087+0000","last_clean":"2024-04-21T23:08:50.710087+0000","last_became_active":"2024-04-21T23:06:25.993747+0000","last_became_peered":"2024-04-21T23:06:25.993747+0000","last_unstale":"2024-04-21T23:08:50.710087+0000","last_undegraded":"2024-04-21T23:08:50.710087+0000","last_fullsized":"2024-04-21T23:08:50.710087+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:17.771165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734824+0000","last_change":"2024-04-21T23:06:25.209756+0000","last_active":"2024-04-21T23:08:50.734824+0000","last_peered":"2024-04-21T23:08:50.734824+0000","last_clean":"2024-04-21T23:08:50.734824+0000","last_became_active":"2024-04-21T23:06:25.209445+0000","last_became_peered":"2024-04-21T23:06:25.209445+0000","last_unstale":"2024-04-21T23:08:50.734824+0000","last_undegraded":"2024-04-21T23:08:50.734824+0000","last_fullsized":"2024-04-21T23:08:50.734824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:58:49.453047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792987+0000","last_change":"2024-04-21T23:06:30.261565+0000","last_active":"2024-04-21T23:08:50.792987+0000","last_peered":"2024-04-21T23:08:50.792987+0000","last_clean":"2024-04-21T23:08:50.792987+0000","last_became_active":"2024-04-21T23:06:30.261095+0000","last_became_peered":"2024-04-21T23:06:30.261095+0000","last_unstale":"2024-04-21T23:08:50.792987+0000","last_undegraded":"2024-04-21T23:08:50.792987+0000","last_fullsized":"2024-04-21T23:08:50.792987+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:37:23.174152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700060+0000","last_change":"2024-04-21T23:08:25.103635+0000","last_active":"2024-04-21T23:08:50.700060+0000","last_peered":"2024-04-21T23:08:50.700060+0000","last_clean":"2024-04-21T23:08:50.700060+0000","last_became_active":"2024-04-21T23:08:25.103318+0000","last_became_peered":"2024-04-21T23:08:25.103318+0000","last_unstale":"2024-04-21T23:08:50.700060+0000","last_undegraded":"2024-04-21T23:08:50.700060+0000","last_fullsized":"2024-04-21T23:08:50.700060+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:56:55.215016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592569+0000","last_change":"2024-04-21T23:06:25.208968+0000","last_active":"2024-04-21T23:08:50.592569+0000","last_peered":"2024-04-21T23:08:50.592569+0000","last_clean":"2024-04-21T23:08:50.592569+0000","last_became_active":"2024-04-21T23:06:25.208606+0000","last_became_peered":"2024-04-21T23:06:25.208606+0000","last_unstale":"2024-04-21T23:08:50.592569+0000","last_undegraded":"2024-04-21T23:08:50.592569+0000","last_fullsized":"2024-04-21T23:08:50.592569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:31:46.147119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":369,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792237+0000","last_change":"2024-04-21T23:05:13.077846+0000","last_active":"2024-04-21T23:08:50.792237+0000","last_peered":"2024-04-21T23:08:50.792237+0000","last_clean":"2024-04-21T23:08:50.792237+0000","last_became_active":"2024-04-21T23:05:13.077436+0000","last_became_peered":"2024-04-21T23:05:13.077436+0000","last_unstale":"2024-04-21T23:08:50.792237+0000","last_undegraded":"2024-04-21T23:08:50.792237+0000","last_fullsized":"2024-04-21T23:08:50.792237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:01:47.619212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:01:47.619212+0000","last_clean_scrub_stamp":"2024-04-21T23:01:47.619212+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:12:09.826839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710151+0000","last_change":"2024-04-21T23:06:30.405121+0000","last_active":"2024-04-21T23:08:50.710151+0000","last_peered":"2024-04-21T23:08:50.710151+0000","last_clean":"2024-04-21T23:08:50.710151+0000","last_became_active":"2024-04-21T23:06:30.404612+0000","last_became_peered":"2024-04-21T23:06:30.404612+0000","last_unstale":"2024-04-21T23:08:50.710151+0000","last_undegraded":"2024-04-21T23:08:50.710151+0000","last_fullsized":"2024-04-21T23:08:50.710151+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:59:01.737337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734745+0000","last_change":"2024-04-21T23:08:25.102015+0000","last_active":"2024-04-21T23:08:50.734745+0000","last_peered":"2024-04-21T23:08:50.734745+0000","last_clean":"2024-04-21T23:08:50.734745+0000","last_became_active":"2024-04-21T23:08:25.101674+0000","last_became_peered":"2024-04-21T23:08:25.101674+0000","last_unstale":"2024-04-21T23:08:50.734745+0000","last_undegraded":"2024-04-21T23:08:50.734745+0000","last_fullsized":"2024-04-21T23:08:50.734745+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:14:19.304795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792365+0000","last_change":"2024-04-21T23:06:25.209714+0000","last_active":"2024-04-21T23:08:50.792365+0000","last_peered":"2024-04-21T23:08:50.792365+0000","last_clean":"2024-04-21T23:08:50.792365+0000","last_became_active":"2024-04-21T23:06:25.209373+0000","last_became_peered":"2024-04-21T23:06:25.209373+0000","last_unstale":"2024-04-21T23:08:50.792365+0000","last_undegraded":"2024-04-21T23:08:50.792365+0000","last_fullsized":"2024-04-21T23:08:50.792365+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:21.654143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665293+0000","last_change":"2024-04-21T23:06:30.264873+0000","last_active":"2024-04-21T23:08:50.665293+0000","last_peered":"2024-04-21T23:08:50.665293+0000","last_clean":"2024-04-21T23:08:50.665293+0000","last_became_active":"2024-04-21T23:06:30.264318+0000","last_became_peered":"2024-04-21T23:06:30.264318+0000","last_unstale":"2024-04-21T23:08:50.665293+0000","last_undegraded":"2024-04-21T23:08:50.665293+0000","last_fullsized":"2024-04-21T23:08:50.665293+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:01:22.558782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592702+0000","last_change":"2024-04-21T23:06:25.218767+0000","last_active":"2024-04-21T23:08:50.592702+0000","last_peered":"2024-04-21T23:08:50.592702+0000","last_clean":"2024-04-21T23:08:50.592702+0000","last_became_active":"2024-04-21T23:06:25.218278+0000","last_became_peered":"2024-04-21T23:06:25.218278+0000","last_unstale":"2024-04-21T23:08:50.592702+0000","last_undegraded":"2024-04-21T23:08:50.592702+0000","last_fullsized":"2024-04-21T23:08:50.592702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:03:05.063515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735353+0000","last_change":"2024-04-21T23:06:25.968551+0000","last_active":"2024-04-21T23:08:50.735353+0000","last_peered":"2024-04-21T23:08:50.735353+0000","last_clean":"2024-04-21T23:08:50.735353+0000","last_became_active":"2024-04-21T23:06:25.968167+0000","last_became_peered":"2024-04-21T23:06:25.968167+0000","last_unstale":"2024-04-21T23:08:50.735353+0000","last_undegraded":"2024-04-21T23:08:50.735353+0000","last_fullsized":"2024-04-21T23:08:50.735353+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:08.920677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.590757+0000","last_change":"2024-04-21T23:06:30.404811+0000","last_active":"2024-04-21T23:08:50.590757+0000","last_peered":"2024-04-21T23:08:50.590757+0000","last_clean":"2024-04-21T23:08:50.590757+0000","last_became_active":"2024-04-21T23:06:30.404375+0000","last_became_peered":"2024-04-21T23:06:30.404375+0000","last_unstale":"2024-04-21T23:08:50.590757+0000","last_undegraded":"2024-04-21T23:08:50.590757+0000","last_fullsized":"2024-04-21T23:08:50.590757+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:10:15.688917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"95'43","reported_seq":100,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.832453+0000","last_change":"2024-04-21T23:06:25.204371+0000","last_active":"2024-04-21T23:08:50.832453+0000","last_peered":"2024-04-21T23:08:50.832453+0000","last_clean":"2024-04-21T23:08:50.832453+0000","last_became_active":"2024-04-21T23:06:25.204025+0000","last_became_peered":"2024-04-21T23:06:25.204025+0000","last_unstale":"2024-04-21T23:08:50.832453+0000","last_undegraded":"2024-04-21T23:08:50.832453+0000","last_fullsized":"2024-04-21T23:08:50.832453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:34:31.071013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":77,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.591302+0000","last_change":"2024-04-21T23:06:25.990800+0000","last_active":"2024-04-21T23:08:50.591302+0000","last_peered":"2024-04-21T23:08:50.591302+0000","last_clean":"2024-04-21T23:08:50.591302+0000","last_became_active":"2024-04-21T23:06:25.990477+0000","last_became_peered":"2024-04-21T23:06:25.990477+0000","last_unstale":"2024-04-21T23:08:50.591302+0000","last_undegraded":"2024-04-21T23:08:50.591302+0000","last_fullsized":"2024-04-21T23:08:50.591302+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:55:33.282303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734511+0000","last_change":"2024-04-21T23:06:30.260090+0000","last_active":"2024-04-21T23:08:50.734511+0000","last_peered":"2024-04-21T23:08:50.734511+0000","last_clean":"2024-04-21T23:08:50.734511+0000","last_became_active":"2024-04-21T23:06:30.259609+0000","last_became_peered":"2024-04-21T23:06:30.259609+0000","last_unstale":"2024-04-21T23:08:50.734511+0000","last_undegraded":"2024-04-21T23:08:50.734511+0000","last_fullsized":"2024-04-21T23:08:50.734511+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:11:17.933683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592986+0000","last_change":"2024-04-21T23:06:25.215446+0000","last_active":"2024-04-21T23:08:50.592986+0000","last_peered":"2024-04-21T23:08:50.592986+0000","last_clean":"2024-04-21T23:08:50.592986+0000","last_became_active":"2024-04-21T23:06:25.214773+0000","last_became_peered":"2024-04-21T23:06:25.214773+0000","last_unstale":"2024-04-21T23:08:50.592986+0000","last_undegraded":"2024-04-21T23:08:50.592986+0000","last_fullsized":"2024-04-21T23:08:50.592986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:25:01.090699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735730+0000","last_change":"2024-04-21T23:06:25.986640+0000","last_active":"2024-04-21T23:08:50.735730+0000","last_peered":"2024-04-21T23:08:50.735730+0000","last_clean":"2024-04-21T23:08:50.735730+0000","last_became_active":"2024-04-21T23:06:25.986285+0000","last_became_peered":"2024-04-21T23:06:25.986285+0000","last_unstale":"2024-04-21T23:08:50.735730+0000","last_undegraded":"2024-04-21T23:08:50.735730+0000","last_fullsized":"2024-04-21T23:08:50.735730+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:01:05.498199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592869+0000","last_change":"2024-04-21T23:07:24.458291+0000","last_active":"2024-04-21T23:08:50.592869+0000","last_peered":"2024-04-21T23:08:50.592869+0000","last_clean":"2024-04-21T23:08:50.592869+0000","last_became_active":"2024-04-21T23:07:24.457677+0000","last_became_peered":"2024-04-21T23:07:24.457677+0000","last_unstale":"2024-04-21T23:08:50.592869+0000","last_undegraded":"2024-04-21T23:08:50.592869+0000","last_fullsized":"2024-04-21T23:08:50.592869+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:20:30.708901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,11],"acting":[4,8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734509+0000","last_change":"2024-04-21T23:06:25.217298+0000","last_active":"2024-04-21T23:08:50.734509+0000","last_peered":"2024-04-21T23:08:50.734509+0000","last_clean":"2024-04-21T23:08:50.734509+0000","last_became_active":"2024-04-21T23:06:25.216965+0000","last_became_peered":"2024-04-21T23:06:25.216965+0000","last_unstale":"2024-04-21T23:08:50.734509+0000","last_undegraded":"2024-04-21T23:08:50.734509+0000","last_fullsized":"2024-04-21T23:08:50.734509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:56:38.543697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592266+0000","last_change":"2024-04-21T23:06:25.981400+0000","last_active":"2024-04-21T23:08:50.592266+0000","last_peered":"2024-04-21T23:08:50.592266+0000","last_clean":"2024-04-21T23:08:50.592266+0000","last_became_active":"2024-04-21T23:06:25.981118+0000","last_became_peered":"2024-04-21T23:06:25.981118+0000","last_unstale":"2024-04-21T23:08:50.592266+0000","last_undegraded":"2024-04-21T23:08:50.592266+0000","last_fullsized":"2024-04-21T23:08:50.592266+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:20:59.631062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700704+0000","last_change":"2024-04-21T23:06:30.270351+0000","last_active":"2024-04-21T23:08:50.700704+0000","last_peered":"2024-04-21T23:08:50.700704+0000","last_clean":"2024-04-21T23:08:50.700704+0000","last_became_active":"2024-04-21T23:06:30.269659+0000","last_became_peered":"2024-04-21T23:06:30.269659+0000","last_unstale":"2024-04-21T23:08:50.700704+0000","last_undegraded":"2024-04-21T23:08:50.700704+0000","last_fullsized":"2024-04-21T23:08:50.700704+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:19:14.132191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.590816+0000","last_change":"2024-04-21T23:06:25.212787+0000","last_active":"2024-04-21T23:08:50.590816+0000","last_peered":"2024-04-21T23:08:50.590816+0000","last_clean":"2024-04-21T23:08:50.590816+0000","last_became_active":"2024-04-21T23:06:25.212423+0000","last_became_peered":"2024-04-21T23:06:25.212423+0000","last_unstale":"2024-04-21T23:08:50.590816+0000","last_undegraded":"2024-04-21T23:08:50.590816+0000","last_fullsized":"2024-04-21T23:08:50.590816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:45.875664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665156+0000","last_change":"2024-04-21T23:06:25.978205+0000","last_active":"2024-04-21T23:08:50.665156+0000","last_peered":"2024-04-21T23:08:50.665156+0000","last_clean":"2024-04-21T23:08:50.665156+0000","last_became_active":"2024-04-21T23:06:25.977876+0000","last_became_peered":"2024-04-21T23:06:25.977876+0000","last_unstale":"2024-04-21T23:08:50.665156+0000","last_undegraded":"2024-04-21T23:08:50.665156+0000","last_fullsized":"2024-04-21T23:08:50.665156+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:43:29.280255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592602+0000","last_change":"2024-04-21T23:06:30.271928+0000","last_active":"2024-04-21T23:08:50.592602+0000","last_peered":"2024-04-21T23:08:50.592602+0000","last_clean":"2024-04-21T23:08:50.592602+0000","last_became_active":"2024-04-21T23:06:30.271321+0000","last_became_peered":"2024-04-21T23:06:30.271321+0000","last_unstale":"2024-04-21T23:08:50.592602+0000","last_undegraded":"2024-04-21T23:08:50.592602+0000","last_fullsized":"2024-04-21T23:08:50.592602+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:22:03.343620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734759+0000","last_change":"2024-04-21T23:06:25.204175+0000","last_active":"2024-04-21T23:08:50.734759+0000","last_peered":"2024-04-21T23:08:50.734759+0000","last_clean":"2024-04-21T23:08:50.734759+0000","last_became_active":"2024-04-21T23:06:25.203826+0000","last_became_peered":"2024-04-21T23:06:25.203826+0000","last_unstale":"2024-04-21T23:08:50.734759+0000","last_undegraded":"2024-04-21T23:08:50.734759+0000","last_fullsized":"2024-04-21T23:08:50.734759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:28:53.402395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.591046+0000","last_change":"2024-04-21T23:06:25.990559+0000","last_active":"2024-04-21T23:08:50.591046+0000","last_peered":"2024-04-21T23:08:50.591046+0000","last_clean":"2024-04-21T23:08:50.591046+0000","last_became_active":"2024-04-21T23:06:25.990239+0000","last_became_peered":"2024-04-21T23:06:25.990239+0000","last_unstale":"2024-04-21T23:08:50.591046+0000","last_undegraded":"2024-04-21T23:08:50.591046+0000","last_fullsized":"2024-04-21T23:08:50.591046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:49:26.891190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592994+0000","last_change":"2024-04-21T23:06:30.406584+0000","last_active":"2024-04-21T23:08:50.592994+0000","last_peered":"2024-04-21T23:08:50.592994+0000","last_clean":"2024-04-21T23:08:50.592994+0000","last_became_active":"2024-04-21T23:06:30.404576+0000","last_became_peered":"2024-04-21T23:06:30.404576+0000","last_unstale":"2024-04-21T23:08:50.592994+0000","last_undegraded":"2024-04-21T23:08:50.592994+0000","last_fullsized":"2024-04-21T23:08:50.592994+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:56.047199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665253+0000","last_change":"2024-04-21T23:07:24.460375+0000","last_active":"2024-04-21T23:08:50.665253+0000","last_peered":"2024-04-21T23:08:50.665253+0000","last_clean":"2024-04-21T23:08:50.665253+0000","last_became_active":"2024-04-21T23:07:24.453873+0000","last_became_peered":"2024-04-21T23:07:24.453873+0000","last_unstale":"2024-04-21T23:08:50.665253+0000","last_undegraded":"2024-04-21T23:08:50.665253+0000","last_fullsized":"2024-04-21T23:08:50.665253+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:27.709414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792771+0000","last_change":"2024-04-21T23:06:25.988925+0000","last_active":"2024-04-21T23:08:50.792771+0000","last_peered":"2024-04-21T23:08:50.792771+0000","last_clean":"2024-04-21T23:08:50.792771+0000","last_became_active":"2024-04-21T23:06:25.988480+0000","last_became_peered":"2024-04-21T23:06:25.988480+0000","last_unstale":"2024-04-21T23:08:50.792771+0000","last_undegraded":"2024-04-21T23:08:50.792771+0000","last_fullsized":"2024-04-21T23:08:50.792771+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:32:02.867858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734682+0000","last_change":"2024-04-21T23:06:30.260841+0000","last_active":"2024-04-21T23:08:50.734682+0000","last_peered":"2024-04-21T23:08:50.734682+0000","last_clean":"2024-04-21T23:08:50.734682+0000","last_became_active":"2024-04-21T23:06:30.260376+0000","last_became_peered":"2024-04-21T23:06:30.260376+0000","last_unstale":"2024-04-21T23:08:50.734682+0000","last_undegraded":"2024-04-21T23:08:50.734682+0000","last_fullsized":"2024-04-21T23:08:50.734682+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:24:57.429169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735001+0000","last_change":"2024-04-21T23:06:25.217920+0000","last_active":"2024-04-21T23:08:50.735001+0000","last_peered":"2024-04-21T23:08:50.735001+0000","last_clean":"2024-04-21T23:08:50.735001+0000","last_became_active":"2024-04-21T23:06:25.217379+0000","last_became_peered":"2024-04-21T23:06:25.217379+0000","last_unstale":"2024-04-21T23:08:50.735001+0000","last_undegraded":"2024-04-21T23:08:50.735001+0000","last_fullsized":"2024-04-21T23:08:50.735001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:09.570463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.793234+0000","last_change":"2024-04-21T23:06:25.993533+0000","last_active":"2024-04-21T23:08:50.793234+0000","last_peered":"2024-04-21T23:08:50.793234+0000","last_clean":"2024-04-21T23:08:50.793234+0000","last_became_active":"2024-04-21T23:06:25.989739+0000","last_became_peered":"2024-04-21T23:06:25.989739+0000","last_unstale":"2024-04-21T23:08:50.793234+0000","last_undegraded":"2024-04-21T23:08:50.793234+0000","last_fullsized":"2024-04-21T23:08:50.793234+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:24:05.594026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735054+0000","last_change":"2024-04-21T23:06:30.276121+0000","last_active":"2024-04-21T23:08:50.735054+0000","last_peered":"2024-04-21T23:08:50.735054+0000","last_clean":"2024-04-21T23:08:50.735054+0000","last_became_active":"2024-04-21T23:06:30.275650+0000","last_became_peered":"2024-04-21T23:06:30.275650+0000","last_unstale":"2024-04-21T23:08:50.735054+0000","last_undegraded":"2024-04-21T23:08:50.735054+0000","last_fullsized":"2024-04-21T23:08:50.735054+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:00:19.559482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"95'39","reported_seq":81,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.826456+0000","last_change":"2024-04-21T23:06:25.207079+0000","last_active":"2024-04-21T23:08:50.826456+0000","last_peered":"2024-04-21T23:08:50.826456+0000","last_clean":"2024-04-21T23:08:50.826456+0000","last_became_active":"2024-04-21T23:06:25.206724+0000","last_became_peered":"2024-04-21T23:06:25.206724+0000","last_unstale":"2024-04-21T23:08:50.826456+0000","last_undegraded":"2024-04-21T23:08:50.826456+0000","last_fullsized":"2024-04-21T23:08:50.826456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:27.221282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792921+0000","last_change":"2024-04-21T23:06:25.988144+0000","last_active":"2024-04-21T23:08:50.792921+0000","last_peered":"2024-04-21T23:08:50.792921+0000","last_clean":"2024-04-21T23:08:50.792921+0000","last_became_active":"2024-04-21T23:06:25.987813+0000","last_became_peered":"2024-04-21T23:06:25.987813+0000","last_unstale":"2024-04-21T23:08:50.792921+0000","last_undegraded":"2024-04-21T23:08:50.792921+0000","last_fullsized":"2024-04-21T23:08:50.792921+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:23.733514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.709772+0000","last_change":"2024-04-21T23:06:30.261548+0000","last_active":"2024-04-21T23:08:50.709772+0000","last_peered":"2024-04-21T23:08:50.709772+0000","last_clean":"2024-04-21T23:08:50.709772+0000","last_became_active":"2024-04-21T23:06:30.261152+0000","last_became_peered":"2024-04-21T23:06:30.261152+0000","last_unstale":"2024-04-21T23:08:50.709772+0000","last_undegraded":"2024-04-21T23:08:50.709772+0000","last_fullsized":"2024-04-21T23:08:50.709772+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:00:16.644253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735978+0000","last_change":"2024-04-21T23:06:25.208749+0000","last_active":"2024-04-21T23:08:50.735978+0000","last_peered":"2024-04-21T23:08:50.735978+0000","last_clean":"2024-04-21T23:08:50.735978+0000","last_became_active":"2024-04-21T23:06:25.201511+0000","last_became_peered":"2024-04-21T23:06:25.201511+0000","last_unstale":"2024-04-21T23:08:50.735978+0000","last_undegraded":"2024-04-21T23:08:50.735978+0000","last_fullsized":"2024-04-21T23:08:50.735978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:41.928835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.793146+0000","last_change":"2024-04-21T23:06:25.996041+0000","last_active":"2024-04-21T23:08:50.793146+0000","last_peered":"2024-04-21T23:08:50.793146+0000","last_clean":"2024-04-21T23:08:50.793146+0000","last_became_active":"2024-04-21T23:06:25.994482+0000","last_became_peered":"2024-04-21T23:06:25.994482+0000","last_unstale":"2024-04-21T23:08:50.793146+0000","last_undegraded":"2024-04-21T23:08:50.793146+0000","last_fullsized":"2024-04-21T23:08:50.793146+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:15:46.792618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665223+0000","last_change":"2024-04-21T23:06:25.974061+0000","last_active":"2024-04-21T23:08:50.665223+0000","last_peered":"2024-04-21T23:08:50.665223+0000","last_clean":"2024-04-21T23:08:50.665223+0000","last_became_active":"2024-04-21T23:06:25.973708+0000","last_became_peered":"2024-04-21T23:06:25.973708+0000","last_unstale":"2024-04-21T23:08:50.665223+0000","last_undegraded":"2024-04-21T23:08:50.665223+0000","last_fullsized":"2024-04-21T23:08:50.665223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:58.702947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735074+0000","last_change":"2024-04-21T23:06:25.988412+0000","last_active":"2024-04-21T23:08:50.735074+0000","last_peered":"2024-04-21T23:08:50.735074+0000","last_clean":"2024-04-21T23:08:50.735074+0000","last_became_active":"2024-04-21T23:06:25.988100+0000","last_became_peered":"2024-04-21T23:06:25.988100+0000","last_unstale":"2024-04-21T23:08:50.735074+0000","last_undegraded":"2024-04-21T23:08:50.735074+0000","last_fullsized":"2024-04-21T23:08:50.735074+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:16:42.626123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734567+0000","last_change":"2024-04-21T23:06:25.980144+0000","last_active":"2024-04-21T23:08:50.734567+0000","last_peered":"2024-04-21T23:08:50.734567+0000","last_clean":"2024-04-21T23:08:50.734567+0000","last_became_active":"2024-04-21T23:06:25.979760+0000","last_became_peered":"2024-04-21T23:06:25.979760+0000","last_unstale":"2024-04-21T23:08:50.734567+0000","last_undegraded":"2024-04-21T23:08:50.734567+0000","last_fullsized":"2024-04-21T23:08:50.734567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:14:32.496184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735274+0000","last_change":"2024-04-21T23:06:25.994732+0000","last_active":"2024-04-21T23:08:50.735274+0000","last_peered":"2024-04-21T23:08:50.735274+0000","last_clean":"2024-04-21T23:08:50.735274+0000","last_became_active":"2024-04-21T23:06:25.994408+0000","last_became_peered":"2024-04-21T23:06:25.994408+0000","last_unstale":"2024-04-21T23:08:50.735274+0000","last_undegraded":"2024-04-21T23:08:50.735274+0000","last_fullsized":"2024-04-21T23:08:50.735274+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:28:59.778049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735511+0000","last_change":"2024-04-21T23:07:24.455416+0000","last_active":"2024-04-21T23:08:50.735511+0000","last_peered":"2024-04-21T23:08:50.735511+0000","last_clean":"2024-04-21T23:08:50.735511+0000","last_became_active":"2024-04-21T23:07:24.454954+0000","last_became_peered":"2024-04-21T23:07:24.454954+0000","last_unstale":"2024-04-21T23:08:50.735511+0000","last_undegraded":"2024-04-21T23:08:50.735511+0000","last_fullsized":"2024-04-21T23:08:50.735511+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:29:09.706719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665537+0000","last_change":"2024-04-21T23:06:25.973850+0000","last_active":"2024-04-21T23:08:50.665537+0000","last_peered":"2024-04-21T23:08:50.665537+0000","last_clean":"2024-04-21T23:08:50.665537+0000","last_became_active":"2024-04-21T23:06:25.973522+0000","last_became_peered":"2024-04-21T23:06:25.973522+0000","last_unstale":"2024-04-21T23:08:50.665537+0000","last_undegraded":"2024-04-21T23:08:50.665537+0000","last_fullsized":"2024-04-21T23:08:50.665537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:43:58.417551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734906+0000","last_change":"2024-04-21T23:06:25.978114+0000","last_active":"2024-04-21T23:08:50.734906+0000","last_peered":"2024-04-21T23:08:50.734906+0000","last_clean":"2024-04-21T23:08:50.734906+0000","last_became_active":"2024-04-21T23:06:25.977115+0000","last_became_peered":"2024-04-21T23:06:25.977115+0000","last_unstale":"2024-04-21T23:08:50.734906+0000","last_undegraded":"2024-04-21T23:08:50.734906+0000","last_fullsized":"2024-04-21T23:08:50.734906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:26:46.756490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.591439+0000","last_change":"2024-04-21T23:06:25.994451+0000","last_active":"2024-04-21T23:08:50.591439+0000","last_peered":"2024-04-21T23:08:50.591439+0000","last_clean":"2024-04-21T23:08:50.591439+0000","last_became_active":"2024-04-21T23:06:25.994085+0000","last_became_peered":"2024-04-21T23:06:25.994085+0000","last_unstale":"2024-04-21T23:08:50.591439+0000","last_undegraded":"2024-04-21T23:08:50.591439+0000","last_fullsized":"2024-04-21T23:08:50.591439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:53.652014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.709991+0000","last_change":"2024-04-21T23:06:25.982536+0000","last_active":"2024-04-21T23:08:50.709991+0000","last_peered":"2024-04-21T23:08:50.709991+0000","last_clean":"2024-04-21T23:08:50.709991+0000","last_became_active":"2024-04-21T23:06:25.981318+0000","last_became_peered":"2024-04-21T23:06:25.981318+0000","last_unstale":"2024-04-21T23:08:50.709991+0000","last_undegraded":"2024-04-21T23:08:50.709991+0000","last_fullsized":"2024-04-21T23:08:50.709991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:35.784679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.590938+0000","last_change":"2024-04-21T23:06:25.981915+0000","last_active":"2024-04-21T23:08:50.590938+0000","last_peered":"2024-04-21T23:08:50.590938+0000","last_clean":"2024-04-21T23:08:50.590938+0000","last_became_active":"2024-04-21T23:06:25.981547+0000","last_became_peered":"2024-04-21T23:06:25.981547+0000","last_unstale":"2024-04-21T23:08:50.590938+0000","last_undegraded":"2024-04-21T23:08:50.590938+0000","last_fullsized":"2024-04-21T23:08:50.590938+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:03:11.154582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734923+0000","last_change":"2024-04-21T23:08:25.561660+0000","last_active":"2024-04-21T23:08:50.734923+0000","last_peered":"2024-04-21T23:08:50.734923+0000","last_clean":"2024-04-21T23:08:50.734923+0000","last_became_active":"2024-04-21T23:08:25.561063+0000","last_became_peered":"2024-04-21T23:08:25.561063+0000","last_unstale":"2024-04-21T23:08:50.734923+0000","last_undegraded":"2024-04-21T23:08:50.734923+0000","last_fullsized":"2024-04-21T23:08:50.734923+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:14:27.279046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.737031+0000","last_change":"2024-04-21T23:06:25.993658+0000","last_active":"2024-04-21T23:08:50.737031+0000","last_peered":"2024-04-21T23:08:50.737031+0000","last_clean":"2024-04-21T23:08:50.737031+0000","last_became_active":"2024-04-21T23:06:25.993298+0000","last_became_peered":"2024-04-21T23:06:25.993298+0000","last_unstale":"2024-04-21T23:08:50.737031+0000","last_undegraded":"2024-04-21T23:08:50.737031+0000","last_fullsized":"2024-04-21T23:08:50.737031+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:18:30.216961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665492+0000","last_change":"2024-04-21T23:06:25.993849+0000","last_active":"2024-04-21T23:08:50.665492+0000","last_peered":"2024-04-21T23:08:50.665492+0000","last_clean":"2024-04-21T23:08:50.665492+0000","last_became_active":"2024-04-21T23:06:25.993501+0000","last_became_peered":"2024-04-21T23:06:25.993501+0000","last_unstale":"2024-04-21T23:08:50.665492+0000","last_undegraded":"2024-04-21T23:08:50.665492+0000","last_fullsized":"2024-04-21T23:08:50.665492+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:40:10.461775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595168,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":347,"num_read_kb":278,"num_write":201,"num_write_kb":1411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1880064,"data_stored":1785504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351018,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1085440,"data_stored":535680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612655,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33028,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712124,"statfs":{"total":95995035648,"available":95961214976,"internally_reserved":0,"allocated":1896448,"data_stored":1323278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808884,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28468,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716684,"statfs":{"total":95995035648,"available":95965884416,"internally_reserved":0,"allocated":1482752,"data_stored":933552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36784,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708368,"statfs":{"total":95995035648,"available":95957368832,"internally_reserved":0,"allocated":1478656,"data_stored":929456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332157,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28664,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716488,"statfs":{"total":95995035648,"available":95965683712,"internally_reserved":0,"allocated":1683456,"data_stored":1130704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495683,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":531562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32308,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712844,"statfs":{"total":95995035648,"available":95961952256,"internally_reserved":0,"allocated":1155072,"data_stored":601116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1064960,"data_stored":521749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1138688,"data_stored":585092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":531666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":535680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28976,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716176,"statfs":{"total":95995035648,"available":95965364224,"internally_reserved":0,"allocated":1822720,"data_stored":1257808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":69554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":69588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":524920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":131782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":393798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":393798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":201348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T23:08:52.598 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T23:08:52.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.749+0000 7f85425b1640 1 -- 172.21.15.44:0/3750183759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f853c1018a0 msgr2=0x7f853c101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:52.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.749+0000 7f85425b1640 1 --2- 172.21.15.44:0/3750183759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f853c1018a0 0x7f853c101d20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8530009900 tx=0x7f853002bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:52.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.749+0000 7f85425b1640 1 -- 172.21.15.44:0/3750183759 shutdown_connections 2024-04-21T23:08:52.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.749+0000 7f85425b1640 1 --2- 172.21.15.44:0/3750183759 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f853c102260 0x7f853c108af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.749+0000 7f85425b1640 1 --2- 172.21.15.44:0/3750183759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f853c1018a0 0x7f853c101d20 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.749+0000 7f85425b1640 1 --2- 172.21.15.44:0/3750183759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f853c1006b0 0x7f853c100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.749+0000 7f85425b1640 1 -- 172.21.15.44:0/3750183759 >> 172.21.15.44:0/3750183759 conn(0x7f853c0fbc20 msgr2=0x7f853c0fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:52.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.749+0000 7f85425b1640 1 -- 172.21.15.44:0/3750183759 shutdown_connections 2024-04-21T23:08:52.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.749+0000 7f85425b1640 1 -- 172.21.15.44:0/3750183759 wait complete. 2024-04-21T23:08:52.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.749+0000 7f85425b1640 1 Processor -- start 2024-04-21T23:08:52.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.750+0000 7f85425b1640 1 -- start start 2024-04-21T23:08:52.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.750+0000 7f85425b1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f853c1006b0 0x7f853c06a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:52.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.750+0000 7f85425b1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f853c1018a0 0x7f853c19c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:52.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.750+0000 7f85425b1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f853c102260 0x7f853c19edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:52.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.750+0000 7f85425b1640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f853c10c330 con 0x7f853c1006b0 2024-04-21T23:08:52.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.750+0000 7f85425b1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f853c10c1b0 con 0x7f853c1018a0 2024-04-21T23:08:52.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.750+0000 7f85425b1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f853c10c4b0 con 0x7f853c102260 2024-04-21T23:08:52.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.750+0000 7f853bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f853c1006b0 0x7f853c06a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:52.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.750+0000 7f8540b4e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f853c102260 0x7f853c19edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:52.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.750+0000 7f8540b4e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f853c102260 0x7f853c19edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34572/0 (socket says 172.21.15.44:34572) 2024-04-21T23:08:52.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.750+0000 7f853bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f853c1006b0 0x7f853c06a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35600/0 (socket says 172.21.15.44:35600) 2024-04-21T23:08:52.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.750+0000 7f8540b4e640 1 -- 172.21.15.44:0/3097770887 learned_addr learned my addr 172.21.15.44:0/3097770887 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:52.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.750+0000 7f853b7fe640 1 --2- 172.21.15.44:0/3097770887 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f853c1018a0 0x7f853c19c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:52.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.751+0000 7f8540b4e640 1 -- 172.21.15.44:0/3097770887 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f853c1018a0 msgr2=0x7f853c19c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:52.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.751+0000 7f8540b4e640 1 --2- 172.21.15.44:0/3097770887 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f853c1018a0 0x7f853c19c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.751+0000 7f8540b4e640 1 -- 172.21.15.44:0/3097770887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f853c1006b0 msgr2=0x7f853c06a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:52.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.751+0000 7f8540b4e640 1 --2- 172.21.15.44:0/3097770887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f853c1006b0 0x7f853c06a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.751+0000 7f8540b4e640 1 -- 172.21.15.44:0/3097770887 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f853c19f700 con 0x7f853c102260 2024-04-21T23:08:52.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.751+0000 7f853b7fe640 1 --2- 172.21.15.44:0/3097770887 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f853c1018a0 0x7f853c19c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:08:52.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.751+0000 7f8540b4e640 1 --2- 172.21.15.44:0/3097770887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f853c102260 0x7f853c19edf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f853400eb10 tx=0x7f853400c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:52.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.751+0000 7f85397fa640 1 -- 172.21.15.44:0/3097770887 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534004b20 con 0x7f853c102260 2024-04-21T23:08:52.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.751+0000 7f85425b1640 1 -- 172.21.15.44:0/3097770887 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f853c19f990 con 0x7f853c102260 2024-04-21T23:08:52.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.751+0000 7f85397fa640 1 -- 172.21.15.44:0/3097770887 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8534004cc0 con 0x7f853c102260 2024-04-21T23:08:52.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.751+0000 7f85397fa640 1 -- 172.21.15.44:0/3097770887 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534010760 con 0x7f853c102260 2024-04-21T23:08:52.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.751+0000 7f85425b1640 1 -- 172.21.15.44:0/3097770887 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f853c19feb0 con 0x7f853c102260 2024-04-21T23:08:52.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.752+0000 7f85425b1640 1 -- 172.21.15.44:0/3097770887 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f853c1079d0 con 0x7f853c102260 2024-04-21T23:08:52.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.753+0000 7f85397fa640 1 -- 172.21.15.44:0/3097770887 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7f8534044a70 con 0x7f853c102260 2024-04-21T23:08:52.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.754+0000 7f85397fa640 1 --2- 172.21.15.44:0/3097770887 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f85240a2990 0x7f85240a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:52.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.754+0000 7f853bfff640 1 --2- 172.21.15.44:0/3097770887 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f85240a2990 0x7f85240a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:52.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.754+0000 7f85397fa640 1 -- 172.21.15.44:0/3097770887 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f85340cf6a0 con 0x7f853c102260 2024-04-21T23:08:52.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.755+0000 7f853bfff640 1 --2- 172.21.15.44:0/3097770887 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f85240a2990 0x7f85240a4e50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f853c101700 tx=0x7f852c005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:52.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.758+0000 7f85397fa640 1 -- 172.21.15.44:0/3097770887 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f853409c430 con 0x7f853c102260 2024-04-21T23:08:52.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.957+0000 7f85425b1640 1 -- 172.21.15.44:0/3097770887 --> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f853c061b80 con 0x7f85240a2990 2024-04-21T23:08:52.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.968+0000 7f85397fa640 1 -- 172.21.15.44:0/3097770887 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7f85340983a0 con 0x7f853c102260 2024-04-21T23:08:52.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.971+0000 7f85397fa640 1 -- 172.21.15.44:0/3097770887 <== mgr.15396 v2:172.21.15.44:6800/3789150048 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339305 (secure 0 0 0) 0x7f853c061b80 con 0x7f85240a2990 2024-04-21T23:08:52.974 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:52.976 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-21T23:08:52.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.975+0000 7f85425b1640 1 -- 172.21.15.44:0/3097770887 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f85240a2990 msgr2=0x7f85240a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:52.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.975+0000 7f85425b1640 1 --2- 172.21.15.44:0/3097770887 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f85240a2990 0x7f85240a4e50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f853c101700 tx=0x7f852c005f90 comp rx=0 tx=0).stop 2024-04-21T23:08:52.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.975+0000 7f85425b1640 1 -- 172.21.15.44:0/3097770887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f853c102260 msgr2=0x7f853c19edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:52.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.975+0000 7f85425b1640 1 --2- 172.21.15.44:0/3097770887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f853c102260 0x7f853c19edf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f853400eb10 tx=0x7f853400c2d0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.975+0000 7f85425b1640 1 -- 172.21.15.44:0/3097770887 shutdown_connections 2024-04-21T23:08:52.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.975+0000 7f85425b1640 1 --2- 172.21.15.44:0/3097770887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f853c102260 0x7f853c19edf0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.975+0000 7f85425b1640 1 --2- 172.21.15.44:0/3097770887 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f85240a2990 0x7f85240a4e50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.975+0000 7f85425b1640 1 --2- 172.21.15.44:0/3097770887 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f853c1018a0 0x7f853c19c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.975+0000 7f85425b1640 1 --2- 172.21.15.44:0/3097770887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f853c1006b0 0x7f853c06a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:52.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.975+0000 7f85425b1640 1 -- 172.21.15.44:0/3097770887 >> 172.21.15.44:0/3097770887 conn(0x7f853c0fbc20 msgr2=0x7f853c104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:52.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.975+0000 7f85425b1640 1 -- 172.21.15.44:0/3097770887 shutdown_connections 2024-04-21T23:08:52.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:52.975+0000 7f85425b1640 1 -- 172.21.15.44:0/3097770887 wait complete. 2024-04-21T23:08:52.986 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-04-21T23:08:52.604100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1185770,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":483,"num_read_kb":397,"num_write":384,"num_write_kb":4351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":790,"ondisk_log_size":790,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365052,"kb_used_data":15680,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124576772,"statfs":{"total":1151940427776,"available":1151566614528,"internally_reserved":0,"allocated":16056320,"data_stored":9417343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.988923"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665739+0000","last_change":"2024-04-21T23:06:25.980091+0000","last_active":"2024-04-21T23:08:50.665739+0000","last_peered":"2024-04-21T23:08:50.665739+0000","last_clean":"2024-04-21T23:08:50.665739+0000","last_became_active":"2024-04-21T23:06:25.979763+0000","last_became_peered":"2024-04-21T23:06:25.979763+0000","last_unstale":"2024-04-21T23:08:50.665739+0000","last_undegraded":"2024-04-21T23:08:50.665739+0000","last_fullsized":"2024-04-21T23:08:50.665739+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:12:23.771931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735621+0000","last_change":"2024-04-21T23:06:25.992461+0000","last_active":"2024-04-21T23:08:50.735621+0000","last_peered":"2024-04-21T23:08:50.735621+0000","last_clean":"2024-04-21T23:08:50.735621+0000","last_became_active":"2024-04-21T23:06:25.992018+0000","last_became_peered":"2024-04-21T23:06:25.992018+0000","last_unstale":"2024-04-21T23:08:50.735621+0000","last_undegraded":"2024-04-21T23:08:50.735621+0000","last_fullsized":"2024-04-21T23:08:50.735621+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:46.651683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735329+0000","last_change":"2024-04-21T23:06:25.974812+0000","last_active":"2024-04-21T23:08:50.735329+0000","last_peered":"2024-04-21T23:08:50.735329+0000","last_clean":"2024-04-21T23:08:50.735329+0000","last_became_active":"2024-04-21T23:06:25.974420+0000","last_became_peered":"2024-04-21T23:06:25.974420+0000","last_unstale":"2024-04-21T23:08:50.735329+0000","last_undegraded":"2024-04-21T23:08:50.735329+0000","last_fullsized":"2024-04-21T23:08:50.735329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:00.734218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592764+0000","last_change":"2024-04-21T23:06:25.988806+0000","last_active":"2024-04-21T23:08:50.592764+0000","last_peered":"2024-04-21T23:08:50.592764+0000","last_clean":"2024-04-21T23:08:50.592764+0000","last_became_active":"2024-04-21T23:06:25.988344+0000","last_became_peered":"2024-04-21T23:06:25.988344+0000","last_unstale":"2024-04-21T23:08:50.592764+0000","last_undegraded":"2024-04-21T23:08:50.592764+0000","last_fullsized":"2024-04-21T23:08:50.592764+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:15:40.079273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592366+0000","last_change":"2024-04-21T23:06:25.979125+0000","last_active":"2024-04-21T23:08:50.592366+0000","last_peered":"2024-04-21T23:08:50.592366+0000","last_clean":"2024-04-21T23:08:50.592366+0000","last_became_active":"2024-04-21T23:06:25.978778+0000","last_became_peered":"2024-04-21T23:06:25.978778+0000","last_unstale":"2024-04-21T23:08:50.592366+0000","last_undegraded":"2024-04-21T23:08:50.592366+0000","last_fullsized":"2024-04-21T23:08:50.592366+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:44.655404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.713538+0000","last_change":"2024-04-21T23:06:25.983983+0000","last_active":"2024-04-21T23:08:50.713538+0000","last_peered":"2024-04-21T23:08:50.713538+0000","last_clean":"2024-04-21T23:08:50.713538+0000","last_became_active":"2024-04-21T23:06:25.983463+0000","last_became_peered":"2024-04-21T23:06:25.983463+0000","last_unstale":"2024-04-21T23:08:50.713538+0000","last_undegraded":"2024-04-21T23:08:50.713538+0000","last_fullsized":"2024-04-21T23:08:50.713538+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:32:12.784752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792374+0000","last_change":"2024-04-21T23:08:25.105761+0000","last_active":"2024-04-21T23:08:50.792374+0000","last_peered":"2024-04-21T23:08:50.792374+0000","last_clean":"2024-04-21T23:08:50.792374+0000","last_became_active":"2024-04-21T23:08:25.105339+0000","last_became_peered":"2024-04-21T23:08:25.105339+0000","last_unstale":"2024-04-21T23:08:50.792374+0000","last_undegraded":"2024-04-21T23:08:50.792374+0000","last_fullsized":"2024-04-21T23:08:50.792374+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:05:25.925743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.590750+0000","last_change":"2024-04-21T23:06:25.987714+0000","last_active":"2024-04-21T23:08:50.590750+0000","last_peered":"2024-04-21T23:08:50.590750+0000","last_clean":"2024-04-21T23:08:50.590750+0000","last_became_active":"2024-04-21T23:06:25.987457+0000","last_became_peered":"2024-04-21T23:06:25.987457+0000","last_unstale":"2024-04-21T23:08:50.590750+0000","last_undegraded":"2024-04-21T23:08:50.590750+0000","last_fullsized":"2024-04-21T23:08:50.590750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:06:06.184525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735072+0000","last_change":"2024-04-21T23:06:25.975957+0000","last_active":"2024-04-21T23:08:50.735072+0000","last_peered":"2024-04-21T23:08:50.735072+0000","last_clean":"2024-04-21T23:08:50.735072+0000","last_became_active":"2024-04-21T23:06:25.974710+0000","last_became_peered":"2024-04-21T23:06:25.974710+0000","last_unstale":"2024-04-21T23:08:50.735072+0000","last_undegraded":"2024-04-21T23:08:50.735072+0000","last_fullsized":"2024-04-21T23:08:50.735072+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:31:40.062273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.714037+0000","last_change":"2024-04-21T23:06:25.995085+0000","last_active":"2024-04-21T23:08:50.714037+0000","last_peered":"2024-04-21T23:08:50.714037+0000","last_clean":"2024-04-21T23:08:50.714037+0000","last_became_active":"2024-04-21T23:06:25.993552+0000","last_became_peered":"2024-04-21T23:06:25.993552+0000","last_unstale":"2024-04-21T23:08:50.714037+0000","last_undegraded":"2024-04-21T23:08:50.714037+0000","last_fullsized":"2024-04-21T23:08:50.714037+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:11.405113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.793253+0000","last_change":"2024-04-21T23:06:25.993191+0000","last_active":"2024-04-21T23:08:50.793253+0000","last_peered":"2024-04-21T23:08:50.793253+0000","last_clean":"2024-04-21T23:08:50.793253+0000","last_became_active":"2024-04-21T23:06:25.989399+0000","last_became_peered":"2024-04-21T23:06:25.989399+0000","last_unstale":"2024-04-21T23:08:50.793253+0000","last_undegraded":"2024-04-21T23:08:50.793253+0000","last_fullsized":"2024-04-21T23:08:50.793253+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:30:20.035599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592794+0000","last_change":"2024-04-21T23:06:25.994158+0000","last_active":"2024-04-21T23:08:50.592794+0000","last_peered":"2024-04-21T23:08:50.592794+0000","last_clean":"2024-04-21T23:08:50.592794+0000","last_became_active":"2024-04-21T23:06:25.993780+0000","last_became_peered":"2024-04-21T23:06:25.993780+0000","last_unstale":"2024-04-21T23:08:50.592794+0000","last_undegraded":"2024-04-21T23:08:50.592794+0000","last_fullsized":"2024-04-21T23:08:50.592794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:52:17.181898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.737142+0000","last_change":"2024-04-21T23:06:25.989274+0000","last_active":"2024-04-21T23:08:50.737142+0000","last_peered":"2024-04-21T23:08:50.737142+0000","last_clean":"2024-04-21T23:08:50.737142+0000","last_became_active":"2024-04-21T23:06:25.988931+0000","last_became_peered":"2024-04-21T23:06:25.988931+0000","last_unstale":"2024-04-21T23:08:50.737142+0000","last_undegraded":"2024-04-21T23:08:50.737142+0000","last_fullsized":"2024-04-21T23:08:50.737142+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:11:04.576136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735083+0000","last_change":"2024-04-21T23:06:25.982976+0000","last_active":"2024-04-21T23:08:50.735083+0000","last_peered":"2024-04-21T23:08:50.735083+0000","last_clean":"2024-04-21T23:08:50.735083+0000","last_became_active":"2024-04-21T23:06:25.982625+0000","last_became_peered":"2024-04-21T23:06:25.982625+0000","last_unstale":"2024-04-21T23:08:50.735083+0000","last_undegraded":"2024-04-21T23:08:50.735083+0000","last_fullsized":"2024-04-21T23:08:50.735083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:51:02.193044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735274+0000","last_change":"2024-04-21T23:06:25.994805+0000","last_active":"2024-04-21T23:08:50.735274+0000","last_peered":"2024-04-21T23:08:50.735274+0000","last_clean":"2024-04-21T23:08:50.735274+0000","last_became_active":"2024-04-21T23:06:25.994482+0000","last_became_peered":"2024-04-21T23:06:25.994482+0000","last_unstale":"2024-04-21T23:08:50.735274+0000","last_undegraded":"2024-04-21T23:08:50.735274+0000","last_fullsized":"2024-04-21T23:08:50.735274+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:24:21.792358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734531+0000","last_change":"2024-04-21T23:06:25.990840+0000","last_active":"2024-04-21T23:08:50.734531+0000","last_peered":"2024-04-21T23:08:50.734531+0000","last_clean":"2024-04-21T23:08:50.734531+0000","last_became_active":"2024-04-21T23:06:25.990517+0000","last_became_peered":"2024-04-21T23:06:25.990517+0000","last_unstale":"2024-04-21T23:08:50.734531+0000","last_undegraded":"2024-04-21T23:08:50.734531+0000","last_fullsized":"2024-04-21T23:08:50.734531+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:48:09.265280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665710+0000","last_change":"2024-04-21T23:06:25.976810+0000","last_active":"2024-04-21T23:08:50.665710+0000","last_peered":"2024-04-21T23:08:50.665710+0000","last_clean":"2024-04-21T23:08:50.665710+0000","last_became_active":"2024-04-21T23:06:25.976489+0000","last_became_peered":"2024-04-21T23:06:25.976489+0000","last_unstale":"2024-04-21T23:08:50.665710+0000","last_undegraded":"2024-04-21T23:08:50.665710+0000","last_fullsized":"2024-04-21T23:08:50.665710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:07:03.794115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700626+0000","last_change":"2024-04-21T23:06:25.988603+0000","last_active":"2024-04-21T23:08:50.700626+0000","last_peered":"2024-04-21T23:08:50.700626+0000","last_clean":"2024-04-21T23:08:50.700626+0000","last_became_active":"2024-04-21T23:06:25.988254+0000","last_became_peered":"2024-04-21T23:06:25.988254+0000","last_unstale":"2024-04-21T23:08:50.700626+0000","last_undegraded":"2024-04-21T23:08:50.700626+0000","last_fullsized":"2024-04-21T23:08:50.700626+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:25:46.099853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734889+0000","last_change":"2024-04-21T23:06:25.980831+0000","last_active":"2024-04-21T23:08:50.734889+0000","last_peered":"2024-04-21T23:08:50.734889+0000","last_clean":"2024-04-21T23:08:50.734889+0000","last_became_active":"2024-04-21T23:06:25.977051+0000","last_became_peered":"2024-04-21T23:06:25.977051+0000","last_unstale":"2024-04-21T23:08:50.734889+0000","last_undegraded":"2024-04-21T23:08:50.734889+0000","last_fullsized":"2024-04-21T23:08:50.734889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:55:03.684734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734914+0000","last_change":"2024-04-21T23:06:30.256875+0000","last_active":"2024-04-21T23:08:50.734914+0000","last_peered":"2024-04-21T23:08:50.734914+0000","last_clean":"2024-04-21T23:08:50.734914+0000","last_became_active":"2024-04-21T23:06:30.256459+0000","last_became_peered":"2024-04-21T23:06:30.256459+0000","last_unstale":"2024-04-21T23:08:50.734914+0000","last_undegraded":"2024-04-21T23:08:50.734914+0000","last_fullsized":"2024-04-21T23:08:50.734914+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:45:32.376270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700758+0000","last_change":"2024-04-21T23:06:25.209605+0000","last_active":"2024-04-21T23:08:50.700758+0000","last_peered":"2024-04-21T23:08:50.700758+0000","last_clean":"2024-04-21T23:08:50.700758+0000","last_became_active":"2024-04-21T23:06:25.209277+0000","last_became_peered":"2024-04-21T23:06:25.209277+0000","last_unstale":"2024-04-21T23:08:50.700758+0000","last_undegraded":"2024-04-21T23:08:50.700758+0000","last_fullsized":"2024-04-21T23:08:50.700758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:15:49.742940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.591263+0000","last_change":"2024-04-21T23:06:25.981774+0000","last_active":"2024-04-21T23:08:50.591263+0000","last_peered":"2024-04-21T23:08:50.591263+0000","last_clean":"2024-04-21T23:08:50.591263+0000","last_became_active":"2024-04-21T23:06:25.981432+0000","last_became_peered":"2024-04-21T23:06:25.981432+0000","last_unstale":"2024-04-21T23:08:50.591263+0000","last_undegraded":"2024-04-21T23:08:50.591263+0000","last_fullsized":"2024-04-21T23:08:50.591263+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:28:28.712149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592412+0000","last_change":"2024-04-21T23:06:30.271508+0000","last_active":"2024-04-21T23:08:50.592412+0000","last_peered":"2024-04-21T23:08:50.592412+0000","last_clean":"2024-04-21T23:08:50.592412+0000","last_became_active":"2024-04-21T23:06:30.270461+0000","last_became_peered":"2024-04-21T23:06:30.270461+0000","last_unstale":"2024-04-21T23:08:50.592412+0000","last_undegraded":"2024-04-21T23:08:50.592412+0000","last_fullsized":"2024-04-21T23:08:50.592412+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:43:07.473365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.591569+0000","last_change":"2024-04-21T23:06:25.218773+0000","last_active":"2024-04-21T23:08:50.591569+0000","last_peered":"2024-04-21T23:08:50.591569+0000","last_clean":"2024-04-21T23:08:50.591569+0000","last_became_active":"2024-04-21T23:06:25.218458+0000","last_became_peered":"2024-04-21T23:06:25.218458+0000","last_unstale":"2024-04-21T23:08:50.591569+0000","last_undegraded":"2024-04-21T23:08:50.591569+0000","last_fullsized":"2024-04-21T23:08:50.591569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:25:16.372689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735251+0000","last_change":"2024-04-21T23:06:25.977669+0000","last_active":"2024-04-21T23:08:50.735251+0000","last_peered":"2024-04-21T23:08:50.735251+0000","last_clean":"2024-04-21T23:08:50.735251+0000","last_became_active":"2024-04-21T23:06:25.977143+0000","last_became_peered":"2024-04-21T23:06:25.977143+0000","last_unstale":"2024-04-21T23:08:50.735251+0000","last_undegraded":"2024-04-21T23:08:50.735251+0000","last_fullsized":"2024-04-21T23:08:50.735251+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:39.289208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592447+0000","last_change":"2024-04-21T23:06:30.271934+0000","last_active":"2024-04-21T23:08:50.592447+0000","last_peered":"2024-04-21T23:08:50.592447+0000","last_clean":"2024-04-21T23:08:50.592447+0000","last_became_active":"2024-04-21T23:06:30.270567+0000","last_became_peered":"2024-04-21T23:06:30.270567+0000","last_unstale":"2024-04-21T23:08:50.592447+0000","last_undegraded":"2024-04-21T23:08:50.592447+0000","last_fullsized":"2024-04-21T23:08:50.592447+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:43:34.908748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592827+0000","last_change":"2024-04-21T23:07:24.462595+0000","last_active":"2024-04-21T23:08:50.592827+0000","last_peered":"2024-04-21T23:08:50.592827+0000","last_clean":"2024-04-21T23:08:50.592827+0000","last_became_active":"2024-04-21T23:07:24.454522+0000","last_became_peered":"2024-04-21T23:07:24.454522+0000","last_unstale":"2024-04-21T23:08:50.592827+0000","last_undegraded":"2024-04-21T23:08:50.592827+0000","last_fullsized":"2024-04-21T23:08:50.592827+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:07.457105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734789+0000","last_change":"2024-04-21T23:06:25.988197+0000","last_active":"2024-04-21T23:08:50.734789+0000","last_peered":"2024-04-21T23:08:50.734789+0000","last_clean":"2024-04-21T23:08:50.734789+0000","last_became_active":"2024-04-21T23:06:25.987872+0000","last_became_peered":"2024-04-21T23:06:25.987872+0000","last_unstale":"2024-04-21T23:08:50.734789+0000","last_undegraded":"2024-04-21T23:08:50.734789+0000","last_fullsized":"2024-04-21T23:08:50.734789+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:25.389020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592402+0000","last_change":"2024-04-21T23:06:30.265262+0000","last_active":"2024-04-21T23:08:50.592402+0000","last_peered":"2024-04-21T23:08:50.592402+0000","last_clean":"2024-04-21T23:08:50.592402+0000","last_became_active":"2024-04-21T23:06:30.264781+0000","last_became_peered":"2024-04-21T23:06:30.264781+0000","last_unstale":"2024-04-21T23:08:50.592402+0000","last_undegraded":"2024-04-21T23:08:50.592402+0000","last_fullsized":"2024-04-21T23:08:50.592402+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:03:30.756792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592822+0000","last_change":"2024-04-21T23:06:25.205070+0000","last_active":"2024-04-21T23:08:50.592822+0000","last_peered":"2024-04-21T23:08:50.592822+0000","last_clean":"2024-04-21T23:08:50.592822+0000","last_became_active":"2024-04-21T23:06:25.204737+0000","last_became_peered":"2024-04-21T23:06:25.204737+0000","last_unstale":"2024-04-21T23:08:50.592822+0000","last_undegraded":"2024-04-21T23:08:50.592822+0000","last_fullsized":"2024-04-21T23:08:50.592822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:37.014534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710271+0000","last_change":"2024-04-21T23:06:25.986584+0000","last_active":"2024-04-21T23:08:50.710271+0000","last_peered":"2024-04-21T23:08:50.710271+0000","last_clean":"2024-04-21T23:08:50.710271+0000","last_became_active":"2024-04-21T23:06:25.986268+0000","last_became_peered":"2024-04-21T23:06:25.986268+0000","last_unstale":"2024-04-21T23:08:50.710271+0000","last_undegraded":"2024-04-21T23:08:50.710271+0000","last_fullsized":"2024-04-21T23:08:50.710271+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:19:43.864279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734739+0000","last_change":"2024-04-21T23:06:30.261900+0000","last_active":"2024-04-21T23:08:50.734739+0000","last_peered":"2024-04-21T23:08:50.734739+0000","last_clean":"2024-04-21T23:08:50.734739+0000","last_became_active":"2024-04-21T23:06:30.261444+0000","last_became_peered":"2024-04-21T23:06:30.261444+0000","last_unstale":"2024-04-21T23:08:50.734739+0000","last_undegraded":"2024-04-21T23:08:50.734739+0000","last_fullsized":"2024-04-21T23:08:50.734739+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:10:40.810555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710271+0000","last_change":"2024-04-21T23:07:24.817616+0000","last_active":"2024-04-21T23:08:50.710271+0000","last_peered":"2024-04-21T23:08:50.710271+0000","last_clean":"2024-04-21T23:08:50.710271+0000","last_became_active":"2024-04-21T23:07:24.816888+0000","last_became_peered":"2024-04-21T23:07:24.816888+0000","last_unstale":"2024-04-21T23:08:50.710271+0000","last_undegraded":"2024-04-21T23:08:50.710271+0000","last_fullsized":"2024-04-21T23:08:50.710271+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:01:03.069832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592392+0000","last_change":"2024-04-21T23:06:25.975999+0000","last_active":"2024-04-21T23:08:50.592392+0000","last_peered":"2024-04-21T23:08:50.592392+0000","last_clean":"2024-04-21T23:08:50.592392+0000","last_became_active":"2024-04-21T23:06:25.975677+0000","last_became_peered":"2024-04-21T23:06:25.975677+0000","last_unstale":"2024-04-21T23:08:50.592392+0000","last_undegraded":"2024-04-21T23:08:50.592392+0000","last_fullsized":"2024-04-21T23:08:50.592392+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:52:03.177248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734878+0000","last_change":"2024-04-21T23:07:24.456926+0000","last_active":"2024-04-21T23:08:50.734878+0000","last_peered":"2024-04-21T23:08:50.734878+0000","last_clean":"2024-04-21T23:08:50.734878+0000","last_became_active":"2024-04-21T23:07:24.456481+0000","last_became_peered":"2024-04-21T23:07:24.456481+0000","last_unstale":"2024-04-21T23:08:50.734878+0000","last_undegraded":"2024-04-21T23:08:50.734878+0000","last_fullsized":"2024-04-21T23:08:50.734878+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:19:26.553012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.713581+0000","last_change":"2024-04-21T23:06:25.214893+0000","last_active":"2024-04-21T23:08:50.713581+0000","last_peered":"2024-04-21T23:08:50.713581+0000","last_clean":"2024-04-21T23:08:50.713581+0000","last_became_active":"2024-04-21T23:06:25.212359+0000","last_became_peered":"2024-04-21T23:06:25.212359+0000","last_unstale":"2024-04-21T23:08:50.713581+0000","last_undegraded":"2024-04-21T23:08:50.713581+0000","last_fullsized":"2024-04-21T23:08:50.713581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:18:03.278875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710458+0000","last_change":"2024-04-21T23:06:25.986849+0000","last_active":"2024-04-21T23:08:50.710458+0000","last_peered":"2024-04-21T23:08:50.710458+0000","last_clean":"2024-04-21T23:08:50.710458+0000","last_became_active":"2024-04-21T23:06:25.986334+0000","last_became_peered":"2024-04-21T23:06:25.986334+0000","last_unstale":"2024-04-21T23:08:50.710458+0000","last_undegraded":"2024-04-21T23:08:50.710458+0000","last_fullsized":"2024-04-21T23:08:50.710458+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:57:42.195629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665559+0000","last_change":"2024-04-21T23:06:30.270812+0000","last_active":"2024-04-21T23:08:50.665559+0000","last_peered":"2024-04-21T23:08:50.665559+0000","last_clean":"2024-04-21T23:08:50.665559+0000","last_became_active":"2024-04-21T23:06:30.270370+0000","last_became_peered":"2024-04-21T23:06:30.270370+0000","last_unstale":"2024-04-21T23:08:50.665559+0000","last_undegraded":"2024-04-21T23:08:50.665559+0000","last_fullsized":"2024-04-21T23:08:50.665559+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:53.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734920+0000","last_change":"2024-04-21T23:06:25.201603+0000","last_active":"2024-04-21T23:08:50.734920+0000","last_peered":"2024-04-21T23:08:50.734920+0000","last_clean":"2024-04-21T23:08:50.734920+0000","last_became_active":"2024-04-21T23:06:25.201265+0000","last_became_peered":"2024-04-21T23:06:25.201265+0000","last_unstale":"2024-04-21T23:08:50.734920+0000","last_undegraded":"2024-04-21T23:08:50.734920+0000","last_fullsized":"2024-04-21T23:08:50.734920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:23:19.134215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.590931+0000","last_change":"2024-04-21T23:06:25.982420+0000","last_active":"2024-04-21T23:08:50.590931+0000","last_peered":"2024-04-21T23:08:50.590931+0000","last_clean":"2024-04-21T23:08:50.590931+0000","last_became_active":"2024-04-21T23:06:25.981867+0000","last_became_peered":"2024-04-21T23:06:25.981867+0000","last_unstale":"2024-04-21T23:08:50.590931+0000","last_undegraded":"2024-04-21T23:08:50.590931+0000","last_fullsized":"2024-04-21T23:08:50.590931+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:10:29.433243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735614+0000","last_change":"2024-04-21T23:06:30.273907+0000","last_active":"2024-04-21T23:08:50.735614+0000","last_peered":"2024-04-21T23:08:50.735614+0000","last_clean":"2024-04-21T23:08:50.735614+0000","last_became_active":"2024-04-21T23:06:30.273472+0000","last_became_peered":"2024-04-21T23:06:30.273472+0000","last_unstale":"2024-04-21T23:08:50.735614+0000","last_undegraded":"2024-04-21T23:08:50.735614+0000","last_fullsized":"2024-04-21T23:08:50.735614+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:58:50.914940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710022+0000","last_change":"2024-04-21T23:06:25.211117+0000","last_active":"2024-04-21T23:08:50.710022+0000","last_peered":"2024-04-21T23:08:50.710022+0000","last_clean":"2024-04-21T23:08:50.710022+0000","last_became_active":"2024-04-21T23:06:25.210626+0000","last_became_peered":"2024-04-21T23:06:25.210626+0000","last_unstale":"2024-04-21T23:08:50.710022+0000","last_undegraded":"2024-04-21T23:08:50.710022+0000","last_fullsized":"2024-04-21T23:08:50.710022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:56:39.917072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700098+0000","last_change":"2024-04-21T23:06:25.980537+0000","last_active":"2024-04-21T23:08:50.700098+0000","last_peered":"2024-04-21T23:08:50.700098+0000","last_clean":"2024-04-21T23:08:50.700098+0000","last_became_active":"2024-04-21T23:06:25.980188+0000","last_became_peered":"2024-04-21T23:06:25.980188+0000","last_unstale":"2024-04-21T23:08:50.700098+0000","last_undegraded":"2024-04-21T23:08:50.700098+0000","last_fullsized":"2024-04-21T23:08:50.700098+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:58:02.828632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.590859+0000","last_change":"2024-04-21T23:06:30.263565+0000","last_active":"2024-04-21T23:08:50.590859+0000","last_peered":"2024-04-21T23:08:50.590859+0000","last_clean":"2024-04-21T23:08:50.590859+0000","last_became_active":"2024-04-21T23:06:30.263092+0000","last_became_peered":"2024-04-21T23:06:30.263092+0000","last_unstale":"2024-04-21T23:08:50.590859+0000","last_undegraded":"2024-04-21T23:08:50.590859+0000","last_fullsized":"2024-04-21T23:08:50.590859+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:58:25.709551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.714012+0000","last_change":"2024-04-21T23:06:25.212567+0000","last_active":"2024-04-21T23:08:50.714012+0000","last_peered":"2024-04-21T23:08:50.714012+0000","last_clean":"2024-04-21T23:08:50.714012+0000","last_became_active":"2024-04-21T23:06:25.210593+0000","last_became_peered":"2024-04-21T23:06:25.210593+0000","last_unstale":"2024-04-21T23:08:50.714012+0000","last_undegraded":"2024-04-21T23:08:50.714012+0000","last_fullsized":"2024-04-21T23:08:50.714012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:53:33.552742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735193+0000","last_change":"2024-04-21T23:06:25.989525+0000","last_active":"2024-04-21T23:08:50.735193+0000","last_peered":"2024-04-21T23:08:50.735193+0000","last_clean":"2024-04-21T23:08:50.735193+0000","last_became_active":"2024-04-21T23:06:25.989130+0000","last_became_peered":"2024-04-21T23:06:25.989130+0000","last_unstale":"2024-04-21T23:08:50.735193+0000","last_undegraded":"2024-04-21T23:08:50.735193+0000","last_fullsized":"2024-04-21T23:08:50.735193+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:19:31.000530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.699989+0000","last_change":"2024-04-21T23:06:30.269309+0000","last_active":"2024-04-21T23:08:50.699989+0000","last_peered":"2024-04-21T23:08:50.699989+0000","last_clean":"2024-04-21T23:08:50.699989+0000","last_became_active":"2024-04-21T23:06:30.268903+0000","last_became_peered":"2024-04-21T23:06:30.268903+0000","last_unstale":"2024-04-21T23:08:50.699989+0000","last_undegraded":"2024-04-21T23:08:50.699989+0000","last_fullsized":"2024-04-21T23:08:50.699989+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:52:16.043870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792659+0000","last_change":"2024-04-21T23:06:25.213938+0000","last_active":"2024-04-21T23:08:50.792659+0000","last_peered":"2024-04-21T23:08:50.792659+0000","last_clean":"2024-04-21T23:08:50.792659+0000","last_became_active":"2024-04-21T23:06:25.213242+0000","last_became_peered":"2024-04-21T23:06:25.213242+0000","last_unstale":"2024-04-21T23:08:50.792659+0000","last_undegraded":"2024-04-21T23:08:50.792659+0000","last_fullsized":"2024-04-21T23:08:50.792659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:27.679122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735990+0000","last_change":"2024-04-21T23:07:24.454702+0000","last_active":"2024-04-21T23:08:50.735990+0000","last_peered":"2024-04-21T23:08:50.735990+0000","last_clean":"2024-04-21T23:08:50.735990+0000","last_became_active":"2024-04-21T23:07:24.454211+0000","last_became_peered":"2024-04-21T23:07:24.454211+0000","last_unstale":"2024-04-21T23:08:50.735990+0000","last_undegraded":"2024-04-21T23:08:50.735990+0000","last_fullsized":"2024-04-21T23:08:50.735990+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:29:30.745026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592728+0000","last_change":"2024-04-21T23:06:30.272206+0000","last_active":"2024-04-21T23:08:50.592728+0000","last_peered":"2024-04-21T23:08:50.592728+0000","last_clean":"2024-04-21T23:08:50.592728+0000","last_became_active":"2024-04-21T23:06:30.271389+0000","last_became_peered":"2024-04-21T23:06:30.271389+0000","last_unstale":"2024-04-21T23:08:50.592728+0000","last_undegraded":"2024-04-21T23:08:50.592728+0000","last_fullsized":"2024-04-21T23:08:50.592728+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:54:47.996166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710352+0000","last_change":"2024-04-21T23:06:25.217655+0000","last_active":"2024-04-21T23:08:50.710352+0000","last_peered":"2024-04-21T23:08:50.710352+0000","last_clean":"2024-04-21T23:08:50.710352+0000","last_became_active":"2024-04-21T23:06:25.217337+0000","last_became_peered":"2024-04-21T23:06:25.217337+0000","last_unstale":"2024-04-21T23:08:50.710352+0000","last_undegraded":"2024-04-21T23:08:50.710352+0000","last_fullsized":"2024-04-21T23:08:50.710352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:29:28.690841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735023+0000","last_change":"2024-04-21T23:06:25.995757+0000","last_active":"2024-04-21T23:08:50.735023+0000","last_peered":"2024-04-21T23:08:50.735023+0000","last_clean":"2024-04-21T23:08:50.735023+0000","last_became_active":"2024-04-21T23:06:25.995443+0000","last_became_peered":"2024-04-21T23:06:25.995443+0000","last_unstale":"2024-04-21T23:08:50.735023+0000","last_undegraded":"2024-04-21T23:08:50.735023+0000","last_fullsized":"2024-04-21T23:08:50.735023+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:40.359633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.699925+0000","last_change":"2024-04-21T23:06:30.266441+0000","last_active":"2024-04-21T23:08:50.699925+0000","last_peered":"2024-04-21T23:08:50.699925+0000","last_clean":"2024-04-21T23:08:50.699925+0000","last_became_active":"2024-04-21T23:06:30.264065+0000","last_became_peered":"2024-04-21T23:06:30.264065+0000","last_unstale":"2024-04-21T23:08:50.699925+0000","last_undegraded":"2024-04-21T23:08:50.699925+0000","last_fullsized":"2024-04-21T23:08:50.699925+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:53:22.548096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'38","reported_seq":324,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.953842+0000","last_change":"2024-04-21T23:06:25.213919+0000","last_active":"2024-04-21T23:08:50.953842+0000","last_peered":"2024-04-21T23:08:50.953842+0000","last_clean":"2024-04-21T23:08:50.953842+0000","last_became_active":"2024-04-21T23:06:25.212244+0000","last_became_peered":"2024-04-21T23:06:25.212244+0000","last_unstale":"2024-04-21T23:08:50.953842+0000","last_undegraded":"2024-04-21T23:08:50.953842+0000","last_fullsized":"2024-04-21T23:08:50.953842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:17:25.287718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69554,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":196,"num_write":38,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592581+0000","last_change":"2024-04-21T23:06:25.993666+0000","last_active":"2024-04-21T23:08:50.592581+0000","last_peered":"2024-04-21T23:08:50.592581+0000","last_clean":"2024-04-21T23:08:50.592581+0000","last_became_active":"2024-04-21T23:06:25.993330+0000","last_became_peered":"2024-04-21T23:06:25.993330+0000","last_unstale":"2024-04-21T23:08:50.592581+0000","last_undegraded":"2024-04-21T23:08:50.592581+0000","last_fullsized":"2024-04-21T23:08:50.592581+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:11:00.109782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710648+0000","last_change":"2024-04-21T23:06:30.265807+0000","last_active":"2024-04-21T23:08:50.710648+0000","last_peered":"2024-04-21T23:08:50.710648+0000","last_clean":"2024-04-21T23:08:50.710648+0000","last_became_active":"2024-04-21T23:06:30.265116+0000","last_became_peered":"2024-04-21T23:06:30.265116+0000","last_unstale":"2024-04-21T23:08:50.710648+0000","last_undegraded":"2024-04-21T23:08:50.710648+0000","last_fullsized":"2024-04-21T23:08:50.710648+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:14:04.178281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734740+0000","last_change":"2024-04-21T23:06:25.208319+0000","last_active":"2024-04-21T23:08:50.734740+0000","last_peered":"2024-04-21T23:08:50.734740+0000","last_clean":"2024-04-21T23:08:50.734740+0000","last_became_active":"2024-04-21T23:06:25.208025+0000","last_became_peered":"2024-04-21T23:06:25.208025+0000","last_unstale":"2024-04-21T23:08:50.734740+0000","last_undegraded":"2024-04-21T23:08:50.734740+0000","last_fullsized":"2024-04-21T23:08:50.734740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:14.723170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.736590+0000","last_change":"2024-04-21T23:06:25.993604+0000","last_active":"2024-04-21T23:08:50.736590+0000","last_peered":"2024-04-21T23:08:50.736590+0000","last_clean":"2024-04-21T23:08:50.736590+0000","last_became_active":"2024-04-21T23:06:25.993217+0000","last_became_peered":"2024-04-21T23:06:25.993217+0000","last_unstale":"2024-04-21T23:08:50.736590+0000","last_undegraded":"2024-04-21T23:08:50.736590+0000","last_fullsized":"2024-04-21T23:08:50.736590+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:23:00.435688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665369+0000","last_change":"2024-04-21T23:06:30.270991+0000","last_active":"2024-04-21T23:08:50.665369+0000","last_peered":"2024-04-21T23:08:50.665369+0000","last_clean":"2024-04-21T23:08:50.665369+0000","last_became_active":"2024-04-21T23:06:30.270511+0000","last_became_peered":"2024-04-21T23:06:30.270511+0000","last_unstale":"2024-04-21T23:08:50.665369+0000","last_undegraded":"2024-04-21T23:08:50.665369+0000","last_fullsized":"2024-04-21T23:08:50.665369+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:04.265855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710232+0000","last_change":"2024-04-21T23:06:25.211787+0000","last_active":"2024-04-21T23:08:50.710232+0000","last_peered":"2024-04-21T23:08:50.710232+0000","last_clean":"2024-04-21T23:08:50.710232+0000","last_became_active":"2024-04-21T23:06:25.211473+0000","last_became_peered":"2024-04-21T23:06:25.211473+0000","last_unstale":"2024-04-21T23:08:50.710232+0000","last_undegraded":"2024-04-21T23:08:50.710232+0000","last_fullsized":"2024-04-21T23:08:50.710232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:58:51.364974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792678+0000","last_change":"2024-04-21T23:06:25.990681+0000","last_active":"2024-04-21T23:08:50.792678+0000","last_peered":"2024-04-21T23:08:50.792678+0000","last_clean":"2024-04-21T23:08:50.792678+0000","last_became_active":"2024-04-21T23:06:25.990094+0000","last_became_peered":"2024-04-21T23:06:25.990094+0000","last_unstale":"2024-04-21T23:08:50.792678+0000","last_undegraded":"2024-04-21T23:08:50.792678+0000","last_fullsized":"2024-04-21T23:08:50.792678+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:15:26.907912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700460+0000","last_change":"2024-04-21T23:06:30.271352+0000","last_active":"2024-04-21T23:08:50.700460+0000","last_peered":"2024-04-21T23:08:50.700460+0000","last_clean":"2024-04-21T23:08:50.700460+0000","last_became_active":"2024-04-21T23:06:30.270601+0000","last_became_peered":"2024-04-21T23:06:30.270601+0000","last_unstale":"2024-04-21T23:08:50.700460+0000","last_undegraded":"2024-04-21T23:08:50.700460+0000","last_fullsized":"2024-04-21T23:08:50.700460+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:37.266987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592639+0000","last_change":"2024-04-21T23:06:25.213288+0000","last_active":"2024-04-21T23:08:50.592639+0000","last_peered":"2024-04-21T23:08:50.592639+0000","last_clean":"2024-04-21T23:08:50.592639+0000","last_became_active":"2024-04-21T23:06:25.212936+0000","last_became_peered":"2024-04-21T23:06:25.212936+0000","last_unstale":"2024-04-21T23:08:50.592639+0000","last_undegraded":"2024-04-21T23:08:50.592639+0000","last_fullsized":"2024-04-21T23:08:50.592639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:35.303093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792437+0000","last_change":"2024-04-21T23:06:25.998485+0000","last_active":"2024-04-21T23:08:50.792437+0000","last_peered":"2024-04-21T23:08:50.792437+0000","last_clean":"2024-04-21T23:08:50.792437+0000","last_became_active":"2024-04-21T23:06:25.998184+0000","last_became_peered":"2024-04-21T23:06:25.998184+0000","last_unstale":"2024-04-21T23:08:50.792437+0000","last_undegraded":"2024-04-21T23:08:50.792437+0000","last_fullsized":"2024-04-21T23:08:50.792437+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:35:26.613636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.736096+0000","last_change":"2024-04-21T23:06:30.270089+0000","last_active":"2024-04-21T23:08:50.736096+0000","last_peered":"2024-04-21T23:08:50.736096+0000","last_clean":"2024-04-21T23:08:50.736096+0000","last_became_active":"2024-04-21T23:06:30.269695+0000","last_became_peered":"2024-04-21T23:06:30.269695+0000","last_unstale":"2024-04-21T23:08:50.736096+0000","last_undegraded":"2024-04-21T23:08:50.736096+0000","last_fullsized":"2024-04-21T23:08:50.736096+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:42.718580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592339+0000","last_change":"2024-04-21T23:06:25.218247+0000","last_active":"2024-04-21T23:08:50.592339+0000","last_peered":"2024-04-21T23:08:50.592339+0000","last_clean":"2024-04-21T23:08:50.592339+0000","last_became_active":"2024-04-21T23:06:25.217924+0000","last_became_peered":"2024-04-21T23:06:25.217924+0000","last_unstale":"2024-04-21T23:08:50.592339+0000","last_undegraded":"2024-04-21T23:08:50.592339+0000","last_fullsized":"2024-04-21T23:08:50.592339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:27:12.155597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710045+0000","last_change":"2024-04-21T23:06:25.994109+0000","last_active":"2024-04-21T23:08:50.710045+0000","last_peered":"2024-04-21T23:08:50.710045+0000","last_clean":"2024-04-21T23:08:50.710045+0000","last_became_active":"2024-04-21T23:06:25.993785+0000","last_became_peered":"2024-04-21T23:06:25.993785+0000","last_unstale":"2024-04-21T23:08:50.710045+0000","last_undegraded":"2024-04-21T23:08:50.710045+0000","last_fullsized":"2024-04-21T23:08:50.710045+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:12.629364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.709882+0000","last_change":"2024-04-21T23:06:30.265444+0000","last_active":"2024-04-21T23:08:50.709882+0000","last_peered":"2024-04-21T23:08:50.709882+0000","last_clean":"2024-04-21T23:08:50.709882+0000","last_became_active":"2024-04-21T23:06:30.264971+0000","last_became_peered":"2024-04-21T23:06:30.264971+0000","last_unstale":"2024-04-21T23:08:50.709882+0000","last_undegraded":"2024-04-21T23:08:50.709882+0000","last_fullsized":"2024-04-21T23:08:50.709882+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:03.596631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665514+0000","last_change":"2024-04-21T23:07:24.465377+0000","last_active":"2024-04-21T23:08:50.665514+0000","last_peered":"2024-04-21T23:08:50.665514+0000","last_clean":"2024-04-21T23:08:50.665514+0000","last_became_active":"2024-04-21T23:07:24.454409+0000","last_became_peered":"2024-04-21T23:07:24.454409+0000","last_unstale":"2024-04-21T23:08:50.665514+0000","last_undegraded":"2024-04-21T23:08:50.665514+0000","last_fullsized":"2024-04-21T23:08:50.665514+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:06:22.836124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665569+0000","last_change":"2024-04-21T23:06:25.990193+0000","last_active":"2024-04-21T23:08:50.665569+0000","last_peered":"2024-04-21T23:08:50.665569+0000","last_clean":"2024-04-21T23:08:50.665569+0000","last_became_active":"2024-04-21T23:06:25.989842+0000","last_became_peered":"2024-04-21T23:06:25.989842+0000","last_unstale":"2024-04-21T23:08:50.665569+0000","last_undegraded":"2024-04-21T23:08:50.665569+0000","last_fullsized":"2024-04-21T23:08:50.665569+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:54:53.421257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592553+0000","last_change":"2024-04-21T23:07:24.817262+0000","last_active":"2024-04-21T23:08:50.592553+0000","last_peered":"2024-04-21T23:08:50.592553+0000","last_clean":"2024-04-21T23:08:50.592553+0000","last_became_active":"2024-04-21T23:07:24.816668+0000","last_became_peered":"2024-04-21T23:07:24.816668+0000","last_unstale":"2024-04-21T23:08:50.592553+0000","last_undegraded":"2024-04-21T23:08:50.592553+0000","last_fullsized":"2024-04-21T23:08:50.592553+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:25.302151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700278+0000","last_change":"2024-04-21T23:06:25.218208+0000","last_active":"2024-04-21T23:08:50.700278+0000","last_peered":"2024-04-21T23:08:50.700278+0000","last_clean":"2024-04-21T23:08:50.700278+0000","last_became_active":"2024-04-21T23:06:25.217889+0000","last_became_peered":"2024-04-21T23:06:25.217889+0000","last_unstale":"2024-04-21T23:08:50.700278+0000","last_undegraded":"2024-04-21T23:08:50.700278+0000","last_fullsized":"2024-04-21T23:08:50.700278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:00:08.775682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.713684+0000","last_change":"2024-04-21T23:06:25.987615+0000","last_active":"2024-04-21T23:08:50.713684+0000","last_peered":"2024-04-21T23:08:50.713684+0000","last_clean":"2024-04-21T23:08:50.713684+0000","last_became_active":"2024-04-21T23:06:25.987305+0000","last_became_peered":"2024-04-21T23:06:25.987305+0000","last_unstale":"2024-04-21T23:08:50.713684+0000","last_undegraded":"2024-04-21T23:08:50.713684+0000","last_fullsized":"2024-04-21T23:08:50.713684+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:23:49.386998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735164+0000","last_change":"2024-04-21T23:06:30.268157+0000","last_active":"2024-04-21T23:08:50.735164+0000","last_peered":"2024-04-21T23:08:50.735164+0000","last_clean":"2024-04-21T23:08:50.735164+0000","last_became_active":"2024-04-21T23:06:30.267732+0000","last_became_peered":"2024-04-21T23:06:30.267732+0000","last_unstale":"2024-04-21T23:08:50.735164+0000","last_undegraded":"2024-04-21T23:08:50.735164+0000","last_fullsized":"2024-04-21T23:08:50.735164+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:29:24.250271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792667+0000","last_change":"2024-04-21T23:06:25.215401+0000","last_active":"2024-04-21T23:08:50.792667+0000","last_peered":"2024-04-21T23:08:50.792667+0000","last_clean":"2024-04-21T23:08:50.792667+0000","last_became_active":"2024-04-21T23:06:25.215067+0000","last_became_peered":"2024-04-21T23:06:25.215067+0000","last_unstale":"2024-04-21T23:08:50.792667+0000","last_undegraded":"2024-04-21T23:08:50.792667+0000","last_fullsized":"2024-04-21T23:08:50.792667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:16:58.977797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.713637+0000","last_change":"2024-04-21T23:06:25.996421+0000","last_active":"2024-04-21T23:08:50.713637+0000","last_peered":"2024-04-21T23:08:50.713637+0000","last_clean":"2024-04-21T23:08:50.713637+0000","last_became_active":"2024-04-21T23:06:25.996102+0000","last_became_peered":"2024-04-21T23:06:25.996102+0000","last_unstale":"2024-04-21T23:08:50.713637+0000","last_undegraded":"2024-04-21T23:08:50.713637+0000","last_fullsized":"2024-04-21T23:08:50.713637+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:06:42.630182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.736211+0000","last_change":"2024-04-21T23:06:30.404509+0000","last_active":"2024-04-21T23:08:50.736211+0000","last_peered":"2024-04-21T23:08:50.736211+0000","last_clean":"2024-04-21T23:08:50.736211+0000","last_became_active":"2024-04-21T23:06:30.403987+0000","last_became_peered":"2024-04-21T23:06:30.403987+0000","last_unstale":"2024-04-21T23:08:50.736211+0000","last_undegraded":"2024-04-21T23:08:50.736211+0000","last_fullsized":"2024-04-21T23:08:50.736211+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:12:28.412117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710087+0000","last_change":"2024-04-21T23:06:25.994076+0000","last_active":"2024-04-21T23:08:50.710087+0000","last_peered":"2024-04-21T23:08:50.710087+0000","last_clean":"2024-04-21T23:08:50.710087+0000","last_became_active":"2024-04-21T23:06:25.993747+0000","last_became_peered":"2024-04-21T23:06:25.993747+0000","last_unstale":"2024-04-21T23:08:50.710087+0000","last_undegraded":"2024-04-21T23:08:50.710087+0000","last_fullsized":"2024-04-21T23:08:50.710087+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:17.771165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734824+0000","last_change":"2024-04-21T23:06:25.209756+0000","last_active":"2024-04-21T23:08:50.734824+0000","last_peered":"2024-04-21T23:08:50.734824+0000","last_clean":"2024-04-21T23:08:50.734824+0000","last_became_active":"2024-04-21T23:06:25.209445+0000","last_became_peered":"2024-04-21T23:06:25.209445+0000","last_unstale":"2024-04-21T23:08:50.734824+0000","last_undegraded":"2024-04-21T23:08:50.734824+0000","last_fullsized":"2024-04-21T23:08:50.734824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:58:49.453047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792987+0000","last_change":"2024-04-21T23:06:30.261565+0000","last_active":"2024-04-21T23:08:50.792987+0000","last_peered":"2024-04-21T23:08:50.792987+0000","last_clean":"2024-04-21T23:08:50.792987+0000","last_became_active":"2024-04-21T23:06:30.261095+0000","last_became_peered":"2024-04-21T23:06:30.261095+0000","last_unstale":"2024-04-21T23:08:50.792987+0000","last_undegraded":"2024-04-21T23:08:50.792987+0000","last_fullsized":"2024-04-21T23:08:50.792987+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:37:23.174152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700060+0000","last_change":"2024-04-21T23:08:25.103635+0000","last_active":"2024-04-21T23:08:50.700060+0000","last_peered":"2024-04-21T23:08:50.700060+0000","last_clean":"2024-04-21T23:08:50.700060+0000","last_became_active":"2024-04-21T23:08:25.103318+0000","last_became_peered":"2024-04-21T23:08:25.103318+0000","last_unstale":"2024-04-21T23:08:50.700060+0000","last_undegraded":"2024-04-21T23:08:50.700060+0000","last_fullsized":"2024-04-21T23:08:50.700060+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:56:55.215016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592569+0000","last_change":"2024-04-21T23:06:25.208968+0000","last_active":"2024-04-21T23:08:50.592569+0000","last_peered":"2024-04-21T23:08:50.592569+0000","last_clean":"2024-04-21T23:08:50.592569+0000","last_became_active":"2024-04-21T23:06:25.208606+0000","last_became_peered":"2024-04-21T23:06:25.208606+0000","last_unstale":"2024-04-21T23:08:50.592569+0000","last_undegraded":"2024-04-21T23:08:50.592569+0000","last_fullsized":"2024-04-21T23:08:50.592569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:31:46.147119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":369,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792237+0000","last_change":"2024-04-21T23:05:13.077846+0000","last_active":"2024-04-21T23:08:50.792237+0000","last_peered":"2024-04-21T23:08:50.792237+0000","last_clean":"2024-04-21T23:08:50.792237+0000","last_became_active":"2024-04-21T23:05:13.077436+0000","last_became_peered":"2024-04-21T23:05:13.077436+0000","last_unstale":"2024-04-21T23:08:50.792237+0000","last_undegraded":"2024-04-21T23:08:50.792237+0000","last_fullsized":"2024-04-21T23:08:50.792237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:01:47.619212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:01:47.619212+0000","last_clean_scrub_stamp":"2024-04-21T23:01:47.619212+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:12:09.826839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710151+0000","last_change":"2024-04-21T23:06:30.405121+0000","last_active":"2024-04-21T23:08:50.710151+0000","last_peered":"2024-04-21T23:08:50.710151+0000","last_clean":"2024-04-21T23:08:50.710151+0000","last_became_active":"2024-04-21T23:06:30.404612+0000","last_became_peered":"2024-04-21T23:06:30.404612+0000","last_unstale":"2024-04-21T23:08:50.710151+0000","last_undegraded":"2024-04-21T23:08:50.710151+0000","last_fullsized":"2024-04-21T23:08:50.710151+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:59:01.737337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734745+0000","last_change":"2024-04-21T23:08:25.102015+0000","last_active":"2024-04-21T23:08:50.734745+0000","last_peered":"2024-04-21T23:08:50.734745+0000","last_clean":"2024-04-21T23:08:50.734745+0000","last_became_active":"2024-04-21T23:08:25.101674+0000","last_became_peered":"2024-04-21T23:08:25.101674+0000","last_unstale":"2024-04-21T23:08:50.734745+0000","last_undegraded":"2024-04-21T23:08:50.734745+0000","last_fullsized":"2024-04-21T23:08:50.734745+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:14:19.304795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792365+0000","last_change":"2024-04-21T23:06:25.209714+0000","last_active":"2024-04-21T23:08:50.792365+0000","last_peered":"2024-04-21T23:08:50.792365+0000","last_clean":"2024-04-21T23:08:50.792365+0000","last_became_active":"2024-04-21T23:06:25.209373+0000","last_became_peered":"2024-04-21T23:06:25.209373+0000","last_unstale":"2024-04-21T23:08:50.792365+0000","last_undegraded":"2024-04-21T23:08:50.792365+0000","last_fullsized":"2024-04-21T23:08:50.792365+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:21.654143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665293+0000","last_change":"2024-04-21T23:06:30.264873+0000","last_active":"2024-04-21T23:08:50.665293+0000","last_peered":"2024-04-21T23:08:50.665293+0000","last_clean":"2024-04-21T23:08:50.665293+0000","last_became_active":"2024-04-21T23:06:30.264318+0000","last_became_peered":"2024-04-21T23:06:30.264318+0000","last_unstale":"2024-04-21T23:08:50.665293+0000","last_undegraded":"2024-04-21T23:08:50.665293+0000","last_fullsized":"2024-04-21T23:08:50.665293+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:01:22.558782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592702+0000","last_change":"2024-04-21T23:06:25.218767+0000","last_active":"2024-04-21T23:08:50.592702+0000","last_peered":"2024-04-21T23:08:50.592702+0000","last_clean":"2024-04-21T23:08:50.592702+0000","last_became_active":"2024-04-21T23:06:25.218278+0000","last_became_peered":"2024-04-21T23:06:25.218278+0000","last_unstale":"2024-04-21T23:08:50.592702+0000","last_undegraded":"2024-04-21T23:08:50.592702+0000","last_fullsized":"2024-04-21T23:08:50.592702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:03:05.063515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735353+0000","last_change":"2024-04-21T23:06:25.968551+0000","last_active":"2024-04-21T23:08:50.735353+0000","last_peered":"2024-04-21T23:08:50.735353+0000","last_clean":"2024-04-21T23:08:50.735353+0000","last_became_active":"2024-04-21T23:06:25.968167+0000","last_became_peered":"2024-04-21T23:06:25.968167+0000","last_unstale":"2024-04-21T23:08:50.735353+0000","last_undegraded":"2024-04-21T23:08:50.735353+0000","last_fullsized":"2024-04-21T23:08:50.735353+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:08.920677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.590757+0000","last_change":"2024-04-21T23:06:30.404811+0000","last_active":"2024-04-21T23:08:50.590757+0000","last_peered":"2024-04-21T23:08:50.590757+0000","last_clean":"2024-04-21T23:08:50.590757+0000","last_became_active":"2024-04-21T23:06:30.404375+0000","last_became_peered":"2024-04-21T23:06:30.404375+0000","last_unstale":"2024-04-21T23:08:50.590757+0000","last_undegraded":"2024-04-21T23:08:50.590757+0000","last_fullsized":"2024-04-21T23:08:50.590757+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:10:15.688917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"95'43","reported_seq":100,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.832453+0000","last_change":"2024-04-21T23:06:25.204371+0000","last_active":"2024-04-21T23:08:50.832453+0000","last_peered":"2024-04-21T23:08:50.832453+0000","last_clean":"2024-04-21T23:08:50.832453+0000","last_became_active":"2024-04-21T23:06:25.204025+0000","last_became_peered":"2024-04-21T23:06:25.204025+0000","last_unstale":"2024-04-21T23:08:50.832453+0000","last_undegraded":"2024-04-21T23:08:50.832453+0000","last_fullsized":"2024-04-21T23:08:50.832453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:34:31.071013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":77,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.591302+0000","last_change":"2024-04-21T23:06:25.990800+0000","last_active":"2024-04-21T23:08:50.591302+0000","last_peered":"2024-04-21T23:08:50.591302+0000","last_clean":"2024-04-21T23:08:50.591302+0000","last_became_active":"2024-04-21T23:06:25.990477+0000","last_became_peered":"2024-04-21T23:06:25.990477+0000","last_unstale":"2024-04-21T23:08:50.591302+0000","last_undegraded":"2024-04-21T23:08:50.591302+0000","last_fullsized":"2024-04-21T23:08:50.591302+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:55:33.282303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734511+0000","last_change":"2024-04-21T23:06:30.260090+0000","last_active":"2024-04-21T23:08:50.734511+0000","last_peered":"2024-04-21T23:08:50.734511+0000","last_clean":"2024-04-21T23:08:50.734511+0000","last_became_active":"2024-04-21T23:06:30.259609+0000","last_became_peered":"2024-04-21T23:06:30.259609+0000","last_unstale":"2024-04-21T23:08:50.734511+0000","last_undegraded":"2024-04-21T23:08:50.734511+0000","last_fullsized":"2024-04-21T23:08:50.734511+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:11:17.933683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592986+0000","last_change":"2024-04-21T23:06:25.215446+0000","last_active":"2024-04-21T23:08:50.592986+0000","last_peered":"2024-04-21T23:08:50.592986+0000","last_clean":"2024-04-21T23:08:50.592986+0000","last_became_active":"2024-04-21T23:06:25.214773+0000","last_became_peered":"2024-04-21T23:06:25.214773+0000","last_unstale":"2024-04-21T23:08:50.592986+0000","last_undegraded":"2024-04-21T23:08:50.592986+0000","last_fullsized":"2024-04-21T23:08:50.592986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:25:01.090699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735730+0000","last_change":"2024-04-21T23:06:25.986640+0000","last_active":"2024-04-21T23:08:50.735730+0000","last_peered":"2024-04-21T23:08:50.735730+0000","last_clean":"2024-04-21T23:08:50.735730+0000","last_became_active":"2024-04-21T23:06:25.986285+0000","last_became_peered":"2024-04-21T23:06:25.986285+0000","last_unstale":"2024-04-21T23:08:50.735730+0000","last_undegraded":"2024-04-21T23:08:50.735730+0000","last_fullsized":"2024-04-21T23:08:50.735730+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:01:05.498199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592869+0000","last_change":"2024-04-21T23:07:24.458291+0000","last_active":"2024-04-21T23:08:50.592869+0000","last_peered":"2024-04-21T23:08:50.592869+0000","last_clean":"2024-04-21T23:08:50.592869+0000","last_became_active":"2024-04-21T23:07:24.457677+0000","last_became_peered":"2024-04-21T23:07:24.457677+0000","last_unstale":"2024-04-21T23:08:50.592869+0000","last_undegraded":"2024-04-21T23:08:50.592869+0000","last_fullsized":"2024-04-21T23:08:50.592869+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:20:30.708901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,11],"acting":[4,8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734509+0000","last_change":"2024-04-21T23:06:25.217298+0000","last_active":"2024-04-21T23:08:50.734509+0000","last_peered":"2024-04-21T23:08:50.734509+0000","last_clean":"2024-04-21T23:08:50.734509+0000","last_became_active":"2024-04-21T23:06:25.216965+0000","last_became_peered":"2024-04-21T23:06:25.216965+0000","last_unstale":"2024-04-21T23:08:50.734509+0000","last_undegraded":"2024-04-21T23:08:50.734509+0000","last_fullsized":"2024-04-21T23:08:50.734509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:56:38.543697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592266+0000","last_change":"2024-04-21T23:06:25.981400+0000","last_active":"2024-04-21T23:08:50.592266+0000","last_peered":"2024-04-21T23:08:50.592266+0000","last_clean":"2024-04-21T23:08:50.592266+0000","last_became_active":"2024-04-21T23:06:25.981118+0000","last_became_peered":"2024-04-21T23:06:25.981118+0000","last_unstale":"2024-04-21T23:08:50.592266+0000","last_undegraded":"2024-04-21T23:08:50.592266+0000","last_fullsized":"2024-04-21T23:08:50.592266+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:20:59.631062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700704+0000","last_change":"2024-04-21T23:06:30.270351+0000","last_active":"2024-04-21T23:08:50.700704+0000","last_peered":"2024-04-21T23:08:50.700704+0000","last_clean":"2024-04-21T23:08:50.700704+0000","last_became_active":"2024-04-21T23:06:30.269659+0000","last_became_peered":"2024-04-21T23:06:30.269659+0000","last_unstale":"2024-04-21T23:08:50.700704+0000","last_undegraded":"2024-04-21T23:08:50.700704+0000","last_fullsized":"2024-04-21T23:08:50.700704+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:19:14.132191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.590816+0000","last_change":"2024-04-21T23:06:25.212787+0000","last_active":"2024-04-21T23:08:50.590816+0000","last_peered":"2024-04-21T23:08:50.590816+0000","last_clean":"2024-04-21T23:08:50.590816+0000","last_became_active":"2024-04-21T23:06:25.212423+0000","last_became_peered":"2024-04-21T23:06:25.212423+0000","last_unstale":"2024-04-21T23:08:50.590816+0000","last_undegraded":"2024-04-21T23:08:50.590816+0000","last_fullsized":"2024-04-21T23:08:50.590816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:45.875664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665156+0000","last_change":"2024-04-21T23:06:25.978205+0000","last_active":"2024-04-21T23:08:50.665156+0000","last_peered":"2024-04-21T23:08:50.665156+0000","last_clean":"2024-04-21T23:08:50.665156+0000","last_became_active":"2024-04-21T23:06:25.977876+0000","last_became_peered":"2024-04-21T23:06:25.977876+0000","last_unstale":"2024-04-21T23:08:50.665156+0000","last_undegraded":"2024-04-21T23:08:50.665156+0000","last_fullsized":"2024-04-21T23:08:50.665156+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:43:29.280255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592602+0000","last_change":"2024-04-21T23:06:30.271928+0000","last_active":"2024-04-21T23:08:50.592602+0000","last_peered":"2024-04-21T23:08:50.592602+0000","last_clean":"2024-04-21T23:08:50.592602+0000","last_became_active":"2024-04-21T23:06:30.271321+0000","last_became_peered":"2024-04-21T23:06:30.271321+0000","last_unstale":"2024-04-21T23:08:50.592602+0000","last_undegraded":"2024-04-21T23:08:50.592602+0000","last_fullsized":"2024-04-21T23:08:50.592602+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:22:03.343620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734759+0000","last_change":"2024-04-21T23:06:25.204175+0000","last_active":"2024-04-21T23:08:50.734759+0000","last_peered":"2024-04-21T23:08:50.734759+0000","last_clean":"2024-04-21T23:08:50.734759+0000","last_became_active":"2024-04-21T23:06:25.203826+0000","last_became_peered":"2024-04-21T23:06:25.203826+0000","last_unstale":"2024-04-21T23:08:50.734759+0000","last_undegraded":"2024-04-21T23:08:50.734759+0000","last_fullsized":"2024-04-21T23:08:50.734759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:28:53.402395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.591046+0000","last_change":"2024-04-21T23:06:25.990559+0000","last_active":"2024-04-21T23:08:50.591046+0000","last_peered":"2024-04-21T23:08:50.591046+0000","last_clean":"2024-04-21T23:08:50.591046+0000","last_became_active":"2024-04-21T23:06:25.990239+0000","last_became_peered":"2024-04-21T23:06:25.990239+0000","last_unstale":"2024-04-21T23:08:50.591046+0000","last_undegraded":"2024-04-21T23:08:50.591046+0000","last_fullsized":"2024-04-21T23:08:50.591046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:49:26.891190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592994+0000","last_change":"2024-04-21T23:06:30.406584+0000","last_active":"2024-04-21T23:08:50.592994+0000","last_peered":"2024-04-21T23:08:50.592994+0000","last_clean":"2024-04-21T23:08:50.592994+0000","last_became_active":"2024-04-21T23:06:30.404576+0000","last_became_peered":"2024-04-21T23:06:30.404576+0000","last_unstale":"2024-04-21T23:08:50.592994+0000","last_undegraded":"2024-04-21T23:08:50.592994+0000","last_fullsized":"2024-04-21T23:08:50.592994+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:56.047199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665253+0000","last_change":"2024-04-21T23:07:24.460375+0000","last_active":"2024-04-21T23:08:50.665253+0000","last_peered":"2024-04-21T23:08:50.665253+0000","last_clean":"2024-04-21T23:08:50.665253+0000","last_became_active":"2024-04-21T23:07:24.453873+0000","last_became_peered":"2024-04-21T23:07:24.453873+0000","last_unstale":"2024-04-21T23:08:50.665253+0000","last_undegraded":"2024-04-21T23:08:50.665253+0000","last_fullsized":"2024-04-21T23:08:50.665253+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:27.709414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792771+0000","last_change":"2024-04-21T23:06:25.988925+0000","last_active":"2024-04-21T23:08:50.792771+0000","last_peered":"2024-04-21T23:08:50.792771+0000","last_clean":"2024-04-21T23:08:50.792771+0000","last_became_active":"2024-04-21T23:06:25.988480+0000","last_became_peered":"2024-04-21T23:06:25.988480+0000","last_unstale":"2024-04-21T23:08:50.792771+0000","last_undegraded":"2024-04-21T23:08:50.792771+0000","last_fullsized":"2024-04-21T23:08:50.792771+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:32:02.867858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734682+0000","last_change":"2024-04-21T23:06:30.260841+0000","last_active":"2024-04-21T23:08:50.734682+0000","last_peered":"2024-04-21T23:08:50.734682+0000","last_clean":"2024-04-21T23:08:50.734682+0000","last_became_active":"2024-04-21T23:06:30.260376+0000","last_became_peered":"2024-04-21T23:06:30.260376+0000","last_unstale":"2024-04-21T23:08:50.734682+0000","last_undegraded":"2024-04-21T23:08:50.734682+0000","last_fullsized":"2024-04-21T23:08:50.734682+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:24:57.429169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735001+0000","last_change":"2024-04-21T23:06:25.217920+0000","last_active":"2024-04-21T23:08:50.735001+0000","last_peered":"2024-04-21T23:08:50.735001+0000","last_clean":"2024-04-21T23:08:50.735001+0000","last_became_active":"2024-04-21T23:06:25.217379+0000","last_became_peered":"2024-04-21T23:06:25.217379+0000","last_unstale":"2024-04-21T23:08:50.735001+0000","last_undegraded":"2024-04-21T23:08:50.735001+0000","last_fullsized":"2024-04-21T23:08:50.735001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:09.570463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.793234+0000","last_change":"2024-04-21T23:06:25.993533+0000","last_active":"2024-04-21T23:08:50.793234+0000","last_peered":"2024-04-21T23:08:50.793234+0000","last_clean":"2024-04-21T23:08:50.793234+0000","last_became_active":"2024-04-21T23:06:25.989739+0000","last_became_peered":"2024-04-21T23:06:25.989739+0000","last_unstale":"2024-04-21T23:08:50.793234+0000","last_undegraded":"2024-04-21T23:08:50.793234+0000","last_fullsized":"2024-04-21T23:08:50.793234+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:24:05.594026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735054+0000","last_change":"2024-04-21T23:06:30.276121+0000","last_active":"2024-04-21T23:08:50.735054+0000","last_peered":"2024-04-21T23:08:50.735054+0000","last_clean":"2024-04-21T23:08:50.735054+0000","last_became_active":"2024-04-21T23:06:30.275650+0000","last_became_peered":"2024-04-21T23:06:30.275650+0000","last_unstale":"2024-04-21T23:08:50.735054+0000","last_undegraded":"2024-04-21T23:08:50.735054+0000","last_fullsized":"2024-04-21T23:08:50.735054+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:00:19.559482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"95'39","reported_seq":81,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.826456+0000","last_change":"2024-04-21T23:06:25.207079+0000","last_active":"2024-04-21T23:08:50.826456+0000","last_peered":"2024-04-21T23:08:50.826456+0000","last_clean":"2024-04-21T23:08:50.826456+0000","last_became_active":"2024-04-21T23:06:25.206724+0000","last_became_peered":"2024-04-21T23:06:25.206724+0000","last_unstale":"2024-04-21T23:08:50.826456+0000","last_undegraded":"2024-04-21T23:08:50.826456+0000","last_fullsized":"2024-04-21T23:08:50.826456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:27.221282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792921+0000","last_change":"2024-04-21T23:06:25.988144+0000","last_active":"2024-04-21T23:08:50.792921+0000","last_peered":"2024-04-21T23:08:50.792921+0000","last_clean":"2024-04-21T23:08:50.792921+0000","last_became_active":"2024-04-21T23:06:25.987813+0000","last_became_peered":"2024-04-21T23:06:25.987813+0000","last_unstale":"2024-04-21T23:08:50.792921+0000","last_undegraded":"2024-04-21T23:08:50.792921+0000","last_fullsized":"2024-04-21T23:08:50.792921+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:23.733514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.709772+0000","last_change":"2024-04-21T23:06:30.261548+0000","last_active":"2024-04-21T23:08:50.709772+0000","last_peered":"2024-04-21T23:08:50.709772+0000","last_clean":"2024-04-21T23:08:50.709772+0000","last_became_active":"2024-04-21T23:06:30.261152+0000","last_became_peered":"2024-04-21T23:06:30.261152+0000","last_unstale":"2024-04-21T23:08:50.709772+0000","last_undegraded":"2024-04-21T23:08:50.709772+0000","last_fullsized":"2024-04-21T23:08:50.709772+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:00:16.644253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735978+0000","last_change":"2024-04-21T23:06:25.208749+0000","last_active":"2024-04-21T23:08:50.735978+0000","last_peered":"2024-04-21T23:08:50.735978+0000","last_clean":"2024-04-21T23:08:50.735978+0000","last_became_active":"2024-04-21T23:06:25.201511+0000","last_became_peered":"2024-04-21T23:06:25.201511+0000","last_unstale":"2024-04-21T23:08:50.735978+0000","last_undegraded":"2024-04-21T23:08:50.735978+0000","last_fullsized":"2024-04-21T23:08:50.735978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:41.928835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.793146+0000","last_change":"2024-04-21T23:06:25.996041+0000","last_active":"2024-04-21T23:08:50.793146+0000","last_peered":"2024-04-21T23:08:50.793146+0000","last_clean":"2024-04-21T23:08:50.793146+0000","last_became_active":"2024-04-21T23:06:25.994482+0000","last_became_peered":"2024-04-21T23:06:25.994482+0000","last_unstale":"2024-04-21T23:08:50.793146+0000","last_undegraded":"2024-04-21T23:08:50.793146+0000","last_fullsized":"2024-04-21T23:08:50.793146+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:15:46.792618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665223+0000","last_change":"2024-04-21T23:06:25.974061+0000","last_active":"2024-04-21T23:08:50.665223+0000","last_peered":"2024-04-21T23:08:50.665223+0000","last_clean":"2024-04-21T23:08:50.665223+0000","last_became_active":"2024-04-21T23:06:25.973708+0000","last_became_peered":"2024-04-21T23:06:25.973708+0000","last_unstale":"2024-04-21T23:08:50.665223+0000","last_undegraded":"2024-04-21T23:08:50.665223+0000","last_fullsized":"2024-04-21T23:08:50.665223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:58.702947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735074+0000","last_change":"2024-04-21T23:06:25.988412+0000","last_active":"2024-04-21T23:08:50.735074+0000","last_peered":"2024-04-21T23:08:50.735074+0000","last_clean":"2024-04-21T23:08:50.735074+0000","last_became_active":"2024-04-21T23:06:25.988100+0000","last_became_peered":"2024-04-21T23:06:25.988100+0000","last_unstale":"2024-04-21T23:08:50.735074+0000","last_undegraded":"2024-04-21T23:08:50.735074+0000","last_fullsized":"2024-04-21T23:08:50.735074+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:16:42.626123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734567+0000","last_change":"2024-04-21T23:06:25.980144+0000","last_active":"2024-04-21T23:08:50.734567+0000","last_peered":"2024-04-21T23:08:50.734567+0000","last_clean":"2024-04-21T23:08:50.734567+0000","last_became_active":"2024-04-21T23:06:25.979760+0000","last_became_peered":"2024-04-21T23:06:25.979760+0000","last_unstale":"2024-04-21T23:08:50.734567+0000","last_undegraded":"2024-04-21T23:08:50.734567+0000","last_fullsized":"2024-04-21T23:08:50.734567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:14:32.496184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735274+0000","last_change":"2024-04-21T23:06:25.994732+0000","last_active":"2024-04-21T23:08:50.735274+0000","last_peered":"2024-04-21T23:08:50.735274+0000","last_clean":"2024-04-21T23:08:50.735274+0000","last_became_active":"2024-04-21T23:06:25.994408+0000","last_became_peered":"2024-04-21T23:06:25.994408+0000","last_unstale":"2024-04-21T23:08:50.735274+0000","last_undegraded":"2024-04-21T23:08:50.735274+0000","last_fullsized":"2024-04-21T23:08:50.735274+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:28:59.778049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735511+0000","last_change":"2024-04-21T23:07:24.455416+0000","last_active":"2024-04-21T23:08:50.735511+0000","last_peered":"2024-04-21T23:08:50.735511+0000","last_clean":"2024-04-21T23:08:50.735511+0000","last_became_active":"2024-04-21T23:07:24.454954+0000","last_became_peered":"2024-04-21T23:07:24.454954+0000","last_unstale":"2024-04-21T23:08:50.735511+0000","last_undegraded":"2024-04-21T23:08:50.735511+0000","last_fullsized":"2024-04-21T23:08:50.735511+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:29:09.706719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665537+0000","last_change":"2024-04-21T23:06:25.973850+0000","last_active":"2024-04-21T23:08:50.665537+0000","last_peered":"2024-04-21T23:08:50.665537+0000","last_clean":"2024-04-21T23:08:50.665537+0000","last_became_active":"2024-04-21T23:06:25.973522+0000","last_became_peered":"2024-04-21T23:06:25.973522+0000","last_unstale":"2024-04-21T23:08:50.665537+0000","last_undegraded":"2024-04-21T23:08:50.665537+0000","last_fullsized":"2024-04-21T23:08:50.665537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:43:58.417551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734906+0000","last_change":"2024-04-21T23:06:25.978114+0000","last_active":"2024-04-21T23:08:50.734906+0000","last_peered":"2024-04-21T23:08:50.734906+0000","last_clean":"2024-04-21T23:08:50.734906+0000","last_became_active":"2024-04-21T23:06:25.977115+0000","last_became_peered":"2024-04-21T23:06:25.977115+0000","last_unstale":"2024-04-21T23:08:50.734906+0000","last_undegraded":"2024-04-21T23:08:50.734906+0000","last_fullsized":"2024-04-21T23:08:50.734906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:26:46.756490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.591439+0000","last_change":"2024-04-21T23:06:25.994451+0000","last_active":"2024-04-21T23:08:50.591439+0000","last_peered":"2024-04-21T23:08:50.591439+0000","last_clean":"2024-04-21T23:08:50.591439+0000","last_became_active":"2024-04-21T23:06:25.994085+0000","last_became_peered":"2024-04-21T23:06:25.994085+0000","last_unstale":"2024-04-21T23:08:50.591439+0000","last_undegraded":"2024-04-21T23:08:50.591439+0000","last_fullsized":"2024-04-21T23:08:50.591439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:53.652014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.709991+0000","last_change":"2024-04-21T23:06:25.982536+0000","last_active":"2024-04-21T23:08:50.709991+0000","last_peered":"2024-04-21T23:08:50.709991+0000","last_clean":"2024-04-21T23:08:50.709991+0000","last_became_active":"2024-04-21T23:06:25.981318+0000","last_became_peered":"2024-04-21T23:06:25.981318+0000","last_unstale":"2024-04-21T23:08:50.709991+0000","last_undegraded":"2024-04-21T23:08:50.709991+0000","last_fullsized":"2024-04-21T23:08:50.709991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:35.784679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.590938+0000","last_change":"2024-04-21T23:06:25.981915+0000","last_active":"2024-04-21T23:08:50.590938+0000","last_peered":"2024-04-21T23:08:50.590938+0000","last_clean":"2024-04-21T23:08:50.590938+0000","last_became_active":"2024-04-21T23:06:25.981547+0000","last_became_peered":"2024-04-21T23:06:25.981547+0000","last_unstale":"2024-04-21T23:08:50.590938+0000","last_undegraded":"2024-04-21T23:08:50.590938+0000","last_fullsized":"2024-04-21T23:08:50.590938+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:03:11.154582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734923+0000","last_change":"2024-04-21T23:08:25.561660+0000","last_active":"2024-04-21T23:08:50.734923+0000","last_peered":"2024-04-21T23:08:50.734923+0000","last_clean":"2024-04-21T23:08:50.734923+0000","last_became_active":"2024-04-21T23:08:25.561063+0000","last_became_peered":"2024-04-21T23:08:25.561063+0000","last_unstale":"2024-04-21T23:08:50.734923+0000","last_undegraded":"2024-04-21T23:08:50.734923+0000","last_fullsized":"2024-04-21T23:08:50.734923+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:14:27.279046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.737031+0000","last_change":"2024-04-21T23:06:25.993658+0000","last_active":"2024-04-21T23:08:50.737031+0000","last_peered":"2024-04-21T23:08:50.737031+0000","last_clean":"2024-04-21T23:08:50.737031+0000","last_became_active":"2024-04-21T23:06:25.993298+0000","last_became_peered":"2024-04-21T23:06:25.993298+0000","last_unstale":"2024-04-21T23:08:50.737031+0000","last_undegraded":"2024-04-21T23:08:50.737031+0000","last_fullsized":"2024-04-21T23:08:50.737031+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:18:30.216961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665492+0000","last_change":"2024-04-21T23:06:25.993849+0000","last_active":"2024-04-21T23:08:50.665492+0000","last_peered":"2024-04-21T23:08:50.665492+0000","last_clean":"2024-04-21T23:08:50.665492+0000","last_became_active":"2024-04-21T23:06:25.993501+0000","last_became_peered":"2024-04-21T23:06:25.993501+0000","last_unstale":"2024-04-21T23:08:50.665492+0000","last_undegraded":"2024-04-21T23:08:50.665492+0000","last_fullsized":"2024-04-21T23:08:50.665492+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:40:10.461775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595168,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":347,"num_read_kb":278,"num_write":201,"num_write_kb":1411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1880064,"data_stored":1785504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351018,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1085440,"data_stored":535680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612655,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33028,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712124,"statfs":{"total":95995035648,"available":95961214976,"internally_reserved":0,"allocated":1896448,"data_stored":1323278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808884,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28468,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716684,"statfs":{"total":95995035648,"available":95965884416,"internally_reserved":0,"allocated":1482752,"data_stored":933552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36784,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708368,"statfs":{"total":95995035648,"available":95957368832,"internally_reserved":0,"allocated":1478656,"data_stored":929456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332157,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28664,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716488,"statfs":{"total":95995035648,"available":95965683712,"internally_reserved":0,"allocated":1683456,"data_stored":1130704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495683,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":531562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32308,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712844,"statfs":{"total":95995035648,"available":95961952256,"internally_reserved":0,"allocated":1155072,"data_stored":601116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1064960,"data_stored":521749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1138688,"data_stored":585092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":531666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":535680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28976,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716176,"statfs":{"total":95995035648,"available":95965364224,"internally_reserved":0,"allocated":1822720,"data_stored":1257808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":69554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":69588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":524920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":131782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":393798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":393798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":201348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T23:08:52.992 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-21T23:08:52.993 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T23:08:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.131+0000 7f1cd1edf640 1 -- 172.21.15.44:0/3953818060 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ccc1006a0 msgr2=0x7f1ccc100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.131+0000 7f1cd1edf640 1 --2- 172.21.15.44:0/3953818060 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ccc1006a0 0x7f1ccc100ac0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc009860 tx=0x7f1cbc02bb70 comp rx=0 tx=0).stop 2024-04-21T23:08:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.131+0000 7f1cd1edf640 1 -- 172.21.15.44:0/3953818060 shutdown_connections 2024-04-21T23:08:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.131+0000 7f1cd1edf640 1 --2- 172.21.15.44:0/3953818060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ccc102250 0x7f1ccc108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.131+0000 7f1cd1edf640 1 --2- 172.21.15.44:0/3953818060 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ccc101890 0x7f1ccc101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.131+0000 7f1cd1edf640 1 --2- 172.21.15.44:0/3953818060 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ccc1006a0 0x7f1ccc100ac0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.131+0000 7f1cd1edf640 1 -- 172.21.15.44:0/3953818060 >> 172.21.15.44:0/3953818060 conn(0x7f1ccc0fbc10 msgr2=0x7f1ccc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.131+0000 7f1cd1edf640 1 -- 172.21.15.44:0/3953818060 shutdown_connections 2024-04-21T23:08:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.131+0000 7f1cd1edf640 1 -- 172.21.15.44:0/3953818060 wait complete. 2024-04-21T23:08:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.131+0000 7f1cd1edf640 1 Processor -- start 2024-04-21T23:08:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.131+0000 7f1cd1edf640 1 -- start start 2024-04-21T23:08:53.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1cd1edf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ccc1006a0 0x7f1ccc197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1cd1edf640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ccc101890 0x7f1ccc198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1cd1edf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ccc102250 0x7f1ccc19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1cd1edf640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1ccc0695d0 con 0x7f1ccc1006a0 2024-04-21T23:08:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1cd1edf640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1ccc069450 con 0x7f1ccc101890 2024-04-21T23:08:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1cd1edf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1ccc069750 con 0x7f1ccc102250 2024-04-21T23:08:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1ccb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ccc1006a0 0x7f1ccc197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1ccb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ccc1006a0 0x7f1ccc197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35610/0 (socket says 172.21.15.44:35610) 2024-04-21T23:08:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1ccb7fe640 1 -- 172.21.15.44:0/1396011879 learned_addr learned my addr 172.21.15.44:0/1396011879 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1ccbfff640 1 --2- 172.21.15.44:0/1396011879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ccc102250 0x7f1ccc19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1ccaffd640 1 --2- 172.21.15.44:0/1396011879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ccc101890 0x7f1ccc198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1ccb7fe640 1 -- 172.21.15.44:0/1396011879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ccc102250 msgr2=0x7f1ccc19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1ccb7fe640 1 --2- 172.21.15.44:0/1396011879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ccc102250 0x7f1ccc19f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1ccb7fe640 1 -- 172.21.15.44:0/1396011879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ccc101890 msgr2=0x7f1ccc198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:53.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1ccb7fe640 1 --2- 172.21.15.44:0/1396011879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ccc101890 0x7f1ccc198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1ccb7fe640 1 -- 172.21.15.44:0/1396011879 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ccc19f670 con 0x7f1ccc1006a0 2024-04-21T23:08:53.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1ccb7fe640 1 --2- 172.21.15.44:0/1396011879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ccc1006a0 0x7f1ccc197af0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc02c0b0 tx=0x7f1cbc032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:53.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1cc8ff9640 1 -- 172.21.15.44:0/1396011879 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc041020 con 0x7f1ccc1006a0 2024-04-21T23:08:53.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1cc8ff9640 1 -- 172.21.15.44:0/1396011879 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cbc02c9f0 con 0x7f1ccc1006a0 2024-04-21T23:08:53.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1cd1edf640 1 -- 172.21.15.44:0/1396011879 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ccc19f900 con 0x7f1ccc1006a0 2024-04-21T23:08:53.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.132+0000 7f1cc8ff9640 1 -- 172.21.15.44:0/1396011879 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc039040 con 0x7f1ccc1006a0 2024-04-21T23:08:53.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.133+0000 7f1cd1edf640 1 -- 172.21.15.44:0/1396011879 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ccc19fdf0 con 0x7f1ccc1006a0 2024-04-21T23:08:53.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.134+0000 7f1cd1edf640 1 -- 172.21.15.44:0/1396011879 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ccc1079c0 con 0x7f1ccc1006a0 2024-04-21T23:08:53.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.135+0000 7f1cc8ff9640 1 -- 172.21.15.44:0/1396011879 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7f1cbc044020 con 0x7f1ccc1006a0 2024-04-21T23:08:53.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.135+0000 7f1cc8ff9640 1 --2- 172.21.15.44:0/1396011879 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1cb80a2d00 0x7f1cb80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:53.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.135+0000 7f1cc8ff9640 1 -- 172.21.15.44:0/1396011879 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f1cbc0f0fd0 con 0x7f1ccc1006a0 2024-04-21T23:08:53.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.135+0000 7f1ccaffd640 1 --2- 172.21.15.44:0/1396011879 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1cb80a2d00 0x7f1cb80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:53.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.136+0000 7f1ccaffd640 1 --2- 172.21.15.44:0/1396011879 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1cb80a2d00 0x7f1cb80a51c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc198f60 tx=0x7f1cb4009260 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:53.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.140+0000 7f1cc8ff9640 1 -- 172.21.15.44:0/1396011879 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1cbc04b020 con 0x7f1ccc1006a0 2024-04-21T23:08:53.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:52 smithi055 ceph-mon[31117]: [21/Apr/2024:23:08:51] ENGINE Bus STARTING 2024-04-21T23:08:53.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:52 smithi055 ceph-mon[31117]: [21/Apr/2024:23:08:51] ENGINE Serving on https://172.21.15.44:7150 2024-04-21T23:08:53.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:52 smithi055 ceph-mon[31117]: [21/Apr/2024:23:08:51] ENGINE Serving on http://172.21.15.44:8765 2024-04-21T23:08:53.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:52 smithi055 ceph-mon[31117]: [21/Apr/2024:23:08:51] ENGINE Bus STARTED 2024-04-21T23:08:53.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:52 smithi055 ceph-mon[31117]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:08:53.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:52 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1405484650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:53.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:52 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:53.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:52 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:53.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:52 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:53.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:52 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:53.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:52 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:53.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:52 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:53.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:52 smithi069 ceph-mon[31816]: [21/Apr/2024:23:08:51] ENGINE Bus STARTING 2024-04-21T23:08:53.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:52 smithi069 ceph-mon[31816]: [21/Apr/2024:23:08:51] ENGINE Serving on https://172.21.15.44:7150 2024-04-21T23:08:53.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:52 smithi069 ceph-mon[31816]: [21/Apr/2024:23:08:51] ENGINE Serving on http://172.21.15.44:8765 2024-04-21T23:08:53.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:52 smithi069 ceph-mon[31816]: [21/Apr/2024:23:08:51] ENGINE Bus STARTED 2024-04-21T23:08:53.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:52 smithi069 ceph-mon[31816]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:08:53.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:52 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1405484650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:53.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:52 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:53.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:52 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:53.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:52 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:53.217 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:52 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:53.217 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:52 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:53.217 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:52 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:53.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.324+0000 7f1cd1edf640 1 -- 172.21.15.44:0/1396011879 --> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1ccc061b80 con 0x7f1cb80a2d00 2024-04-21T23:08:53.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.334+0000 7f1cc8ff9640 1 -- 172.21.15.44:0/1396011879 <== mgr.15396 v2:172.21.15.44:6800/3789150048 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339305 (secure 0 0 0) 0x7f1ccc061b80 con 0x7f1cb80a2d00 2024-04-21T23:08:53.339 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:53.341 INFO:teuthology.orchestra.run.smithi044.stderr:dumped all 2024-04-21T23:08:53.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.339+0000 7f1cd1edf640 1 -- 172.21.15.44:0/1396011879 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1cb80a2d00 msgr2=0x7f1cb80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:53.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.339+0000 7f1cd1edf640 1 --2- 172.21.15.44:0/1396011879 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1cb80a2d00 0x7f1cb80a51c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc198f60 tx=0x7f1cb4009260 comp rx=0 tx=0).stop 2024-04-21T23:08:53.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.339+0000 7f1cd1edf640 1 -- 172.21.15.44:0/1396011879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ccc1006a0 msgr2=0x7f1ccc197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:53.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.339+0000 7f1cd1edf640 1 --2- 172.21.15.44:0/1396011879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ccc1006a0 0x7f1ccc197af0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc02c0b0 tx=0x7f1cbc032430 comp rx=0 tx=0).stop 2024-04-21T23:08:53.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.339+0000 7f1cd1edf640 1 -- 172.21.15.44:0/1396011879 shutdown_connections 2024-04-21T23:08:53.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.339+0000 7f1cd1edf640 1 --2- 172.21.15.44:0/1396011879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ccc102250 0x7f1ccc19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.339+0000 7f1cd1edf640 1 --2- 172.21.15.44:0/1396011879 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1cb80a2d00 0x7f1cb80a51c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.339+0000 7f1cd1edf640 1 --2- 172.21.15.44:0/1396011879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ccc101890 0x7f1ccc198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.339+0000 7f1cd1edf640 1 --2- 172.21.15.44:0/1396011879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ccc1006a0 0x7f1ccc197af0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.339+0000 7f1cd1edf640 1 -- 172.21.15.44:0/1396011879 >> 172.21.15.44:0/1396011879 conn(0x7f1ccc0fbc10 msgr2=0x7f1ccc104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:53.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.339+0000 7f1cd1edf640 1 -- 172.21.15.44:0/1396011879 shutdown_connections 2024-04-21T23:08:53.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.339+0000 7f1cd1edf640 1 -- 172.21.15.44:0/1396011879 wait complete. 2024-04-21T23:08:53.349 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:52 smithi044 ceph-mon[27934]: [21/Apr/2024:23:08:51] ENGINE Bus STARTING 2024-04-21T23:08:53.350 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:52 smithi044 ceph-mon[27934]: [21/Apr/2024:23:08:51] ENGINE Serving on https://172.21.15.44:7150 2024-04-21T23:08:53.350 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:52 smithi044 ceph-mon[27934]: [21/Apr/2024:23:08:51] ENGINE Serving on http://172.21.15.44:8765 2024-04-21T23:08:53.350 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:52 smithi044 ceph-mon[27934]: [21/Apr/2024:23:08:51] ENGINE Bus STARTED 2024-04-21T23:08:53.350 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:52 smithi044 ceph-mon[27934]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:08:53.350 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:52 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1405484650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:53.350 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:52 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:53.350 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:52 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:53.350 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:52 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:53.350 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:52 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:53.350 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:52 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:53.351 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:52 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:53.351 INFO:teuthology.orchestra.run.smithi044.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-04-21T23:08:52.604100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1185770,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":483,"num_read_kb":397,"num_write":384,"num_write_kb":4351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180758,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":790,"ondisk_log_size":790,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365052,"kb_used_data":15680,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124576772,"statfs":{"total":1151940427776,"available":1151566614528,"internally_reserved":0,"allocated":16056320,"data_stored":9417343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.988923"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665739+0000","last_change":"2024-04-21T23:06:25.980091+0000","last_active":"2024-04-21T23:08:50.665739+0000","last_peered":"2024-04-21T23:08:50.665739+0000","last_clean":"2024-04-21T23:08:50.665739+0000","last_became_active":"2024-04-21T23:06:25.979763+0000","last_became_peered":"2024-04-21T23:06:25.979763+0000","last_unstale":"2024-04-21T23:08:50.665739+0000","last_undegraded":"2024-04-21T23:08:50.665739+0000","last_fullsized":"2024-04-21T23:08:50.665739+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:12:23.771931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735621+0000","last_change":"2024-04-21T23:06:25.992461+0000","last_active":"2024-04-21T23:08:50.735621+0000","last_peered":"2024-04-21T23:08:50.735621+0000","last_clean":"2024-04-21T23:08:50.735621+0000","last_became_active":"2024-04-21T23:06:25.992018+0000","last_became_peered":"2024-04-21T23:06:25.992018+0000","last_unstale":"2024-04-21T23:08:50.735621+0000","last_undegraded":"2024-04-21T23:08:50.735621+0000","last_fullsized":"2024-04-21T23:08:50.735621+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:55:46.651683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735329+0000","last_change":"2024-04-21T23:06:25.974812+0000","last_active":"2024-04-21T23:08:50.735329+0000","last_peered":"2024-04-21T23:08:50.735329+0000","last_clean":"2024-04-21T23:08:50.735329+0000","last_became_active":"2024-04-21T23:06:25.974420+0000","last_became_peered":"2024-04-21T23:06:25.974420+0000","last_unstale":"2024-04-21T23:08:50.735329+0000","last_undegraded":"2024-04-21T23:08:50.735329+0000","last_fullsized":"2024-04-21T23:08:50.735329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:13:00.734218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592764+0000","last_change":"2024-04-21T23:06:25.988806+0000","last_active":"2024-04-21T23:08:50.592764+0000","last_peered":"2024-04-21T23:08:50.592764+0000","last_clean":"2024-04-21T23:08:50.592764+0000","last_became_active":"2024-04-21T23:06:25.988344+0000","last_became_peered":"2024-04-21T23:06:25.988344+0000","last_unstale":"2024-04-21T23:08:50.592764+0000","last_undegraded":"2024-04-21T23:08:50.592764+0000","last_fullsized":"2024-04-21T23:08:50.592764+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:15:40.079273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592366+0000","last_change":"2024-04-21T23:06:25.979125+0000","last_active":"2024-04-21T23:08:50.592366+0000","last_peered":"2024-04-21T23:08:50.592366+0000","last_clean":"2024-04-21T23:08:50.592366+0000","last_became_active":"2024-04-21T23:06:25.978778+0000","last_became_peered":"2024-04-21T23:06:25.978778+0000","last_unstale":"2024-04-21T23:08:50.592366+0000","last_undegraded":"2024-04-21T23:08:50.592366+0000","last_fullsized":"2024-04-21T23:08:50.592366+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:41:44.655404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.713538+0000","last_change":"2024-04-21T23:06:25.983983+0000","last_active":"2024-04-21T23:08:50.713538+0000","last_peered":"2024-04-21T23:08:50.713538+0000","last_clean":"2024-04-21T23:08:50.713538+0000","last_became_active":"2024-04-21T23:06:25.983463+0000","last_became_peered":"2024-04-21T23:06:25.983463+0000","last_unstale":"2024-04-21T23:08:50.713538+0000","last_undegraded":"2024-04-21T23:08:50.713538+0000","last_fullsized":"2024-04-21T23:08:50.713538+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:32:12.784752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792374+0000","last_change":"2024-04-21T23:08:25.105761+0000","last_active":"2024-04-21T23:08:50.792374+0000","last_peered":"2024-04-21T23:08:50.792374+0000","last_clean":"2024-04-21T23:08:50.792374+0000","last_became_active":"2024-04-21T23:08:25.105339+0000","last_became_peered":"2024-04-21T23:08:25.105339+0000","last_unstale":"2024-04-21T23:08:50.792374+0000","last_undegraded":"2024-04-21T23:08:50.792374+0000","last_fullsized":"2024-04-21T23:08:50.792374+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:05:25.925743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.590750+0000","last_change":"2024-04-21T23:06:25.987714+0000","last_active":"2024-04-21T23:08:50.590750+0000","last_peered":"2024-04-21T23:08:50.590750+0000","last_clean":"2024-04-21T23:08:50.590750+0000","last_became_active":"2024-04-21T23:06:25.987457+0000","last_became_peered":"2024-04-21T23:06:25.987457+0000","last_unstale":"2024-04-21T23:08:50.590750+0000","last_undegraded":"2024-04-21T23:08:50.590750+0000","last_fullsized":"2024-04-21T23:08:50.590750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:06:06.184525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735072+0000","last_change":"2024-04-21T23:06:25.975957+0000","last_active":"2024-04-21T23:08:50.735072+0000","last_peered":"2024-04-21T23:08:50.735072+0000","last_clean":"2024-04-21T23:08:50.735072+0000","last_became_active":"2024-04-21T23:06:25.974710+0000","last_became_peered":"2024-04-21T23:06:25.974710+0000","last_unstale":"2024-04-21T23:08:50.735072+0000","last_undegraded":"2024-04-21T23:08:50.735072+0000","last_fullsized":"2024-04-21T23:08:50.735072+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:31:40.062273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.714037+0000","last_change":"2024-04-21T23:06:25.995085+0000","last_active":"2024-04-21T23:08:50.714037+0000","last_peered":"2024-04-21T23:08:50.714037+0000","last_clean":"2024-04-21T23:08:50.714037+0000","last_became_active":"2024-04-21T23:06:25.993552+0000","last_became_peered":"2024-04-21T23:06:25.993552+0000","last_unstale":"2024-04-21T23:08:50.714037+0000","last_undegraded":"2024-04-21T23:08:50.714037+0000","last_fullsized":"2024-04-21T23:08:50.714037+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:11.405113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.793253+0000","last_change":"2024-04-21T23:06:25.993191+0000","last_active":"2024-04-21T23:08:50.793253+0000","last_peered":"2024-04-21T23:08:50.793253+0000","last_clean":"2024-04-21T23:08:50.793253+0000","last_became_active":"2024-04-21T23:06:25.989399+0000","last_became_peered":"2024-04-21T23:06:25.989399+0000","last_unstale":"2024-04-21T23:08:50.793253+0000","last_undegraded":"2024-04-21T23:08:50.793253+0000","last_fullsized":"2024-04-21T23:08:50.793253+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:30:20.035599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592794+0000","last_change":"2024-04-21T23:06:25.994158+0000","last_active":"2024-04-21T23:08:50.592794+0000","last_peered":"2024-04-21T23:08:50.592794+0000","last_clean":"2024-04-21T23:08:50.592794+0000","last_became_active":"2024-04-21T23:06:25.993780+0000","last_became_peered":"2024-04-21T23:06:25.993780+0000","last_unstale":"2024-04-21T23:08:50.592794+0000","last_undegraded":"2024-04-21T23:08:50.592794+0000","last_fullsized":"2024-04-21T23:08:50.592794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:52:17.181898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.737142+0000","last_change":"2024-04-21T23:06:25.989274+0000","last_active":"2024-04-21T23:08:50.737142+0000","last_peered":"2024-04-21T23:08:50.737142+0000","last_clean":"2024-04-21T23:08:50.737142+0000","last_became_active":"2024-04-21T23:06:25.988931+0000","last_became_peered":"2024-04-21T23:06:25.988931+0000","last_unstale":"2024-04-21T23:08:50.737142+0000","last_undegraded":"2024-04-21T23:08:50.737142+0000","last_fullsized":"2024-04-21T23:08:50.737142+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:11:04.576136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735083+0000","last_change":"2024-04-21T23:06:25.982976+0000","last_active":"2024-04-21T23:08:50.735083+0000","last_peered":"2024-04-21T23:08:50.735083+0000","last_clean":"2024-04-21T23:08:50.735083+0000","last_became_active":"2024-04-21T23:06:25.982625+0000","last_became_peered":"2024-04-21T23:06:25.982625+0000","last_unstale":"2024-04-21T23:08:50.735083+0000","last_undegraded":"2024-04-21T23:08:50.735083+0000","last_fullsized":"2024-04-21T23:08:50.735083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:51:02.193044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735274+0000","last_change":"2024-04-21T23:06:25.994805+0000","last_active":"2024-04-21T23:08:50.735274+0000","last_peered":"2024-04-21T23:08:50.735274+0000","last_clean":"2024-04-21T23:08:50.735274+0000","last_became_active":"2024-04-21T23:06:25.994482+0000","last_became_peered":"2024-04-21T23:06:25.994482+0000","last_unstale":"2024-04-21T23:08:50.735274+0000","last_undegraded":"2024-04-21T23:08:50.735274+0000","last_fullsized":"2024-04-21T23:08:50.735274+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:24:21.792358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734531+0000","last_change":"2024-04-21T23:06:25.990840+0000","last_active":"2024-04-21T23:08:50.734531+0000","last_peered":"2024-04-21T23:08:50.734531+0000","last_clean":"2024-04-21T23:08:50.734531+0000","last_became_active":"2024-04-21T23:06:25.990517+0000","last_became_peered":"2024-04-21T23:06:25.990517+0000","last_unstale":"2024-04-21T23:08:50.734531+0000","last_undegraded":"2024-04-21T23:08:50.734531+0000","last_fullsized":"2024-04-21T23:08:50.734531+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:48:09.265280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665710+0000","last_change":"2024-04-21T23:06:25.976810+0000","last_active":"2024-04-21T23:08:50.665710+0000","last_peered":"2024-04-21T23:08:50.665710+0000","last_clean":"2024-04-21T23:08:50.665710+0000","last_became_active":"2024-04-21T23:06:25.976489+0000","last_became_peered":"2024-04-21T23:06:25.976489+0000","last_unstale":"2024-04-21T23:08:50.665710+0000","last_undegraded":"2024-04-21T23:08:50.665710+0000","last_fullsized":"2024-04-21T23:08:50.665710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:07:03.794115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700626+0000","last_change":"2024-04-21T23:06:25.988603+0000","last_active":"2024-04-21T23:08:50.700626+0000","last_peered":"2024-04-21T23:08:50.700626+0000","last_clean":"2024-04-21T23:08:50.700626+0000","last_became_active":"2024-04-21T23:06:25.988254+0000","last_became_peered":"2024-04-21T23:06:25.988254+0000","last_unstale":"2024-04-21T23:08:50.700626+0000","last_undegraded":"2024-04-21T23:08:50.700626+0000","last_fullsized":"2024-04-21T23:08:50.700626+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:25:46.099853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734889+0000","last_change":"2024-04-21T23:06:25.980831+0000","last_active":"2024-04-21T23:08:50.734889+0000","last_peered":"2024-04-21T23:08:50.734889+0000","last_clean":"2024-04-21T23:08:50.734889+0000","last_became_active":"2024-04-21T23:06:25.977051+0000","last_became_peered":"2024-04-21T23:06:25.977051+0000","last_unstale":"2024-04-21T23:08:50.734889+0000","last_undegraded":"2024-04-21T23:08:50.734889+0000","last_fullsized":"2024-04-21T23:08:50.734889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:55:03.684734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734914+0000","last_change":"2024-04-21T23:06:30.256875+0000","last_active":"2024-04-21T23:08:50.734914+0000","last_peered":"2024-04-21T23:08:50.734914+0000","last_clean":"2024-04-21T23:08:50.734914+0000","last_became_active":"2024-04-21T23:06:30.256459+0000","last_became_peered":"2024-04-21T23:06:30.256459+0000","last_unstale":"2024-04-21T23:08:50.734914+0000","last_undegraded":"2024-04-21T23:08:50.734914+0000","last_fullsized":"2024-04-21T23:08:50.734914+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:45:32.376270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700758+0000","last_change":"2024-04-21T23:06:25.209605+0000","last_active":"2024-04-21T23:08:50.700758+0000","last_peered":"2024-04-21T23:08:50.700758+0000","last_clean":"2024-04-21T23:08:50.700758+0000","last_became_active":"2024-04-21T23:06:25.209277+0000","last_became_peered":"2024-04-21T23:06:25.209277+0000","last_unstale":"2024-04-21T23:08:50.700758+0000","last_undegraded":"2024-04-21T23:08:50.700758+0000","last_fullsized":"2024-04-21T23:08:50.700758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:15:49.742940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.591263+0000","last_change":"2024-04-21T23:06:25.981774+0000","last_active":"2024-04-21T23:08:50.591263+0000","last_peered":"2024-04-21T23:08:50.591263+0000","last_clean":"2024-04-21T23:08:50.591263+0000","last_became_active":"2024-04-21T23:06:25.981432+0000","last_became_peered":"2024-04-21T23:06:25.981432+0000","last_unstale":"2024-04-21T23:08:50.591263+0000","last_undegraded":"2024-04-21T23:08:50.591263+0000","last_fullsized":"2024-04-21T23:08:50.591263+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:28:28.712149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592412+0000","last_change":"2024-04-21T23:06:30.271508+0000","last_active":"2024-04-21T23:08:50.592412+0000","last_peered":"2024-04-21T23:08:50.592412+0000","last_clean":"2024-04-21T23:08:50.592412+0000","last_became_active":"2024-04-21T23:06:30.270461+0000","last_became_peered":"2024-04-21T23:06:30.270461+0000","last_unstale":"2024-04-21T23:08:50.592412+0000","last_undegraded":"2024-04-21T23:08:50.592412+0000","last_fullsized":"2024-04-21T23:08:50.592412+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:43:07.473365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.591569+0000","last_change":"2024-04-21T23:06:25.218773+0000","last_active":"2024-04-21T23:08:50.591569+0000","last_peered":"2024-04-21T23:08:50.591569+0000","last_clean":"2024-04-21T23:08:50.591569+0000","last_became_active":"2024-04-21T23:06:25.218458+0000","last_became_peered":"2024-04-21T23:06:25.218458+0000","last_unstale":"2024-04-21T23:08:50.591569+0000","last_undegraded":"2024-04-21T23:08:50.591569+0000","last_fullsized":"2024-04-21T23:08:50.591569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:25:16.372689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735251+0000","last_change":"2024-04-21T23:06:25.977669+0000","last_active":"2024-04-21T23:08:50.735251+0000","last_peered":"2024-04-21T23:08:50.735251+0000","last_clean":"2024-04-21T23:08:50.735251+0000","last_became_active":"2024-04-21T23:06:25.977143+0000","last_became_peered":"2024-04-21T23:06:25.977143+0000","last_unstale":"2024-04-21T23:08:50.735251+0000","last_undegraded":"2024-04-21T23:08:50.735251+0000","last_fullsized":"2024-04-21T23:08:50.735251+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:39.289208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592447+0000","last_change":"2024-04-21T23:06:30.271934+0000","last_active":"2024-04-21T23:08:50.592447+0000","last_peered":"2024-04-21T23:08:50.592447+0000","last_clean":"2024-04-21T23:08:50.592447+0000","last_became_active":"2024-04-21T23:06:30.270567+0000","last_became_peered":"2024-04-21T23:06:30.270567+0000","last_unstale":"2024-04-21T23:08:50.592447+0000","last_undegraded":"2024-04-21T23:08:50.592447+0000","last_fullsized":"2024-04-21T23:08:50.592447+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:43:34.908748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592827+0000","last_change":"2024-04-21T23:07:24.462595+0000","last_active":"2024-04-21T23:08:50.592827+0000","last_peered":"2024-04-21T23:08:50.592827+0000","last_clean":"2024-04-21T23:08:50.592827+0000","last_became_active":"2024-04-21T23:07:24.454522+0000","last_became_peered":"2024-04-21T23:07:24.454522+0000","last_unstale":"2024-04-21T23:08:50.592827+0000","last_undegraded":"2024-04-21T23:08:50.592827+0000","last_fullsized":"2024-04-21T23:08:50.592827+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:18:07.457105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734789+0000","last_change":"2024-04-21T23:06:25.988197+0000","last_active":"2024-04-21T23:08:50.734789+0000","last_peered":"2024-04-21T23:08:50.734789+0000","last_clean":"2024-04-21T23:08:50.734789+0000","last_became_active":"2024-04-21T23:06:25.987872+0000","last_became_peered":"2024-04-21T23:06:25.987872+0000","last_unstale":"2024-04-21T23:08:50.734789+0000","last_undegraded":"2024-04-21T23:08:50.734789+0000","last_fullsized":"2024-04-21T23:08:50.734789+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:37:25.389020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592402+0000","last_change":"2024-04-21T23:06:30.265262+0000","last_active":"2024-04-21T23:08:50.592402+0000","last_peered":"2024-04-21T23:08:50.592402+0000","last_clean":"2024-04-21T23:08:50.592402+0000","last_became_active":"2024-04-21T23:06:30.264781+0000","last_became_peered":"2024-04-21T23:06:30.264781+0000","last_unstale":"2024-04-21T23:08:50.592402+0000","last_undegraded":"2024-04-21T23:08:50.592402+0000","last_fullsized":"2024-04-21T23:08:50.592402+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:03:30.756792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592822+0000","last_change":"2024-04-21T23:06:25.205070+0000","last_active":"2024-04-21T23:08:50.592822+0000","last_peered":"2024-04-21T23:08:50.592822+0000","last_clean":"2024-04-21T23:08:50.592822+0000","last_became_active":"2024-04-21T23:06:25.204737+0000","last_became_peered":"2024-04-21T23:06:25.204737+0000","last_unstale":"2024-04-21T23:08:50.592822+0000","last_undegraded":"2024-04-21T23:08:50.592822+0000","last_fullsized":"2024-04-21T23:08:50.592822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:43:37.014534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710271+0000","last_change":"2024-04-21T23:06:25.986584+0000","last_active":"2024-04-21T23:08:50.710271+0000","last_peered":"2024-04-21T23:08:50.710271+0000","last_clean":"2024-04-21T23:08:50.710271+0000","last_became_active":"2024-04-21T23:06:25.986268+0000","last_became_peered":"2024-04-21T23:06:25.986268+0000","last_unstale":"2024-04-21T23:08:50.710271+0000","last_undegraded":"2024-04-21T23:08:50.710271+0000","last_fullsized":"2024-04-21T23:08:50.710271+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:19:43.864279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734739+0000","last_change":"2024-04-21T23:06:30.261900+0000","last_active":"2024-04-21T23:08:50.734739+0000","last_peered":"2024-04-21T23:08:50.734739+0000","last_clean":"2024-04-21T23:08:50.734739+0000","last_became_active":"2024-04-21T23:06:30.261444+0000","last_became_peered":"2024-04-21T23:06:30.261444+0000","last_unstale":"2024-04-21T23:08:50.734739+0000","last_undegraded":"2024-04-21T23:08:50.734739+0000","last_fullsized":"2024-04-21T23:08:50.734739+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:10:40.810555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710271+0000","last_change":"2024-04-21T23:07:24.817616+0000","last_active":"2024-04-21T23:08:50.710271+0000","last_peered":"2024-04-21T23:08:50.710271+0000","last_clean":"2024-04-21T23:08:50.710271+0000","last_became_active":"2024-04-21T23:07:24.816888+0000","last_became_peered":"2024-04-21T23:07:24.816888+0000","last_unstale":"2024-04-21T23:08:50.710271+0000","last_undegraded":"2024-04-21T23:08:50.710271+0000","last_fullsized":"2024-04-21T23:08:50.710271+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:01:03.069832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592392+0000","last_change":"2024-04-21T23:06:25.975999+0000","last_active":"2024-04-21T23:08:50.592392+0000","last_peered":"2024-04-21T23:08:50.592392+0000","last_clean":"2024-04-21T23:08:50.592392+0000","last_became_active":"2024-04-21T23:06:25.975677+0000","last_became_peered":"2024-04-21T23:06:25.975677+0000","last_unstale":"2024-04-21T23:08:50.592392+0000","last_undegraded":"2024-04-21T23:08:50.592392+0000","last_fullsized":"2024-04-21T23:08:50.592392+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:52:03.177248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734878+0000","last_change":"2024-04-21T23:07:24.456926+0000","last_active":"2024-04-21T23:08:50.734878+0000","last_peered":"2024-04-21T23:08:50.734878+0000","last_clean":"2024-04-21T23:08:50.734878+0000","last_became_active":"2024-04-21T23:07:24.456481+0000","last_became_peered":"2024-04-21T23:07:24.456481+0000","last_unstale":"2024-04-21T23:08:50.734878+0000","last_undegraded":"2024-04-21T23:08:50.734878+0000","last_fullsized":"2024-04-21T23:08:50.734878+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:19:26.553012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.713581+0000","last_change":"2024-04-21T23:06:25.214893+0000","last_active":"2024-04-21T23:08:50.713581+0000","last_peered":"2024-04-21T23:08:50.713581+0000","last_clean":"2024-04-21T23:08:50.713581+0000","last_became_active":"2024-04-21T23:06:25.212359+0000","last_became_peered":"2024-04-21T23:06:25.212359+0000","last_unstale":"2024-04-21T23:08:50.713581+0000","last_undegraded":"2024-04-21T23:08:50.713581+0000","last_fullsized":"2024-04-21T23:08:50.713581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:18:03.278875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710458+0000","last_change":"2024-04-21T23:06:25.986849+0000","last_active":"2024-04-21T23:08:50.710458+0000","last_peered":"2024-04-21T23:08:50.710458+0000","last_clean":"2024-04-21T23:08:50.710458+0000","last_became_active":"2024-04-21T23:06:25.986334+0000","last_became_peered":"2024-04-21T23:06:25.986334+0000","last_unstale":"2024-04-21T23:08:50.710458+0000","last_undegraded":"2024-04-21T23:08:50.710458+0000","last_fullsized":"2024-04-21T23:08:50.710458+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:57:42.195629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665559+0000","last_change":"2024-04-21T23:06:30.270812+0000","last_active":"2024-04-21T23:08:50.665559+0000","last_peered":"2024-04-21T23:08:50.665559+0000","last_clean":"2024-04-21T23:08:50.665559+0000","last_became_active":"2024-04-21T23:06:30.270370+0000","last_became_peered":"2024-04-21T23:06:30.270370+0000","last_unstale":"2024-04-21T23:08:50.665559+0000","last_undegraded":"2024-04-21T23:08:50.665559+0000","last_fullsized":"2024-04-21T23:08:50.665559+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:02:53.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734920+0000","last_change":"2024-04-21T23:06:25.201603+0000","last_active":"2024-04-21T23:08:50.734920+0000","last_peered":"2024-04-21T23:08:50.734920+0000","last_clean":"2024-04-21T23:08:50.734920+0000","last_became_active":"2024-04-21T23:06:25.201265+0000","last_became_peered":"2024-04-21T23:06:25.201265+0000","last_unstale":"2024-04-21T23:08:50.734920+0000","last_undegraded":"2024-04-21T23:08:50.734920+0000","last_fullsized":"2024-04-21T23:08:50.734920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:23:19.134215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.590931+0000","last_change":"2024-04-21T23:06:25.982420+0000","last_active":"2024-04-21T23:08:50.590931+0000","last_peered":"2024-04-21T23:08:50.590931+0000","last_clean":"2024-04-21T23:08:50.590931+0000","last_became_active":"2024-04-21T23:06:25.981867+0000","last_became_peered":"2024-04-21T23:06:25.981867+0000","last_unstale":"2024-04-21T23:08:50.590931+0000","last_undegraded":"2024-04-21T23:08:50.590931+0000","last_fullsized":"2024-04-21T23:08:50.590931+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:10:29.433243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735614+0000","last_change":"2024-04-21T23:06:30.273907+0000","last_active":"2024-04-21T23:08:50.735614+0000","last_peered":"2024-04-21T23:08:50.735614+0000","last_clean":"2024-04-21T23:08:50.735614+0000","last_became_active":"2024-04-21T23:06:30.273472+0000","last_became_peered":"2024-04-21T23:06:30.273472+0000","last_unstale":"2024-04-21T23:08:50.735614+0000","last_undegraded":"2024-04-21T23:08:50.735614+0000","last_fullsized":"2024-04-21T23:08:50.735614+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:58:50.914940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710022+0000","last_change":"2024-04-21T23:06:25.211117+0000","last_active":"2024-04-21T23:08:50.710022+0000","last_peered":"2024-04-21T23:08:50.710022+0000","last_clean":"2024-04-21T23:08:50.710022+0000","last_became_active":"2024-04-21T23:06:25.210626+0000","last_became_peered":"2024-04-21T23:06:25.210626+0000","last_unstale":"2024-04-21T23:08:50.710022+0000","last_undegraded":"2024-04-21T23:08:50.710022+0000","last_fullsized":"2024-04-21T23:08:50.710022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:56:39.917072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700098+0000","last_change":"2024-04-21T23:06:25.980537+0000","last_active":"2024-04-21T23:08:50.700098+0000","last_peered":"2024-04-21T23:08:50.700098+0000","last_clean":"2024-04-21T23:08:50.700098+0000","last_became_active":"2024-04-21T23:06:25.980188+0000","last_became_peered":"2024-04-21T23:06:25.980188+0000","last_unstale":"2024-04-21T23:08:50.700098+0000","last_undegraded":"2024-04-21T23:08:50.700098+0000","last_fullsized":"2024-04-21T23:08:50.700098+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:58:02.828632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.590859+0000","last_change":"2024-04-21T23:06:30.263565+0000","last_active":"2024-04-21T23:08:50.590859+0000","last_peered":"2024-04-21T23:08:50.590859+0000","last_clean":"2024-04-21T23:08:50.590859+0000","last_became_active":"2024-04-21T23:06:30.263092+0000","last_became_peered":"2024-04-21T23:06:30.263092+0000","last_unstale":"2024-04-21T23:08:50.590859+0000","last_undegraded":"2024-04-21T23:08:50.590859+0000","last_fullsized":"2024-04-21T23:08:50.590859+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:58:25.709551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.714012+0000","last_change":"2024-04-21T23:06:25.212567+0000","last_active":"2024-04-21T23:08:50.714012+0000","last_peered":"2024-04-21T23:08:50.714012+0000","last_clean":"2024-04-21T23:08:50.714012+0000","last_became_active":"2024-04-21T23:06:25.210593+0000","last_became_peered":"2024-04-21T23:06:25.210593+0000","last_unstale":"2024-04-21T23:08:50.714012+0000","last_undegraded":"2024-04-21T23:08:50.714012+0000","last_fullsized":"2024-04-21T23:08:50.714012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:53:33.552742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735193+0000","last_change":"2024-04-21T23:06:25.989525+0000","last_active":"2024-04-21T23:08:50.735193+0000","last_peered":"2024-04-21T23:08:50.735193+0000","last_clean":"2024-04-21T23:08:50.735193+0000","last_became_active":"2024-04-21T23:06:25.989130+0000","last_became_peered":"2024-04-21T23:06:25.989130+0000","last_unstale":"2024-04-21T23:08:50.735193+0000","last_undegraded":"2024-04-21T23:08:50.735193+0000","last_fullsized":"2024-04-21T23:08:50.735193+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:19:31.000530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.699989+0000","last_change":"2024-04-21T23:06:30.269309+0000","last_active":"2024-04-21T23:08:50.699989+0000","last_peered":"2024-04-21T23:08:50.699989+0000","last_clean":"2024-04-21T23:08:50.699989+0000","last_became_active":"2024-04-21T23:06:30.268903+0000","last_became_peered":"2024-04-21T23:06:30.268903+0000","last_unstale":"2024-04-21T23:08:50.699989+0000","last_undegraded":"2024-04-21T23:08:50.699989+0000","last_fullsized":"2024-04-21T23:08:50.699989+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:52:16.043870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792659+0000","last_change":"2024-04-21T23:06:25.213938+0000","last_active":"2024-04-21T23:08:50.792659+0000","last_peered":"2024-04-21T23:08:50.792659+0000","last_clean":"2024-04-21T23:08:50.792659+0000","last_became_active":"2024-04-21T23:06:25.213242+0000","last_became_peered":"2024-04-21T23:06:25.213242+0000","last_unstale":"2024-04-21T23:08:50.792659+0000","last_undegraded":"2024-04-21T23:08:50.792659+0000","last_fullsized":"2024-04-21T23:08:50.792659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:27.679122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735990+0000","last_change":"2024-04-21T23:07:24.454702+0000","last_active":"2024-04-21T23:08:50.735990+0000","last_peered":"2024-04-21T23:08:50.735990+0000","last_clean":"2024-04-21T23:08:50.735990+0000","last_became_active":"2024-04-21T23:07:24.454211+0000","last_became_peered":"2024-04-21T23:07:24.454211+0000","last_unstale":"2024-04-21T23:08:50.735990+0000","last_undegraded":"2024-04-21T23:08:50.735990+0000","last_fullsized":"2024-04-21T23:08:50.735990+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:29:30.745026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592728+0000","last_change":"2024-04-21T23:06:30.272206+0000","last_active":"2024-04-21T23:08:50.592728+0000","last_peered":"2024-04-21T23:08:50.592728+0000","last_clean":"2024-04-21T23:08:50.592728+0000","last_became_active":"2024-04-21T23:06:30.271389+0000","last_became_peered":"2024-04-21T23:06:30.271389+0000","last_unstale":"2024-04-21T23:08:50.592728+0000","last_undegraded":"2024-04-21T23:08:50.592728+0000","last_fullsized":"2024-04-21T23:08:50.592728+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:54:47.996166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710352+0000","last_change":"2024-04-21T23:06:25.217655+0000","last_active":"2024-04-21T23:08:50.710352+0000","last_peered":"2024-04-21T23:08:50.710352+0000","last_clean":"2024-04-21T23:08:50.710352+0000","last_became_active":"2024-04-21T23:06:25.217337+0000","last_became_peered":"2024-04-21T23:06:25.217337+0000","last_unstale":"2024-04-21T23:08:50.710352+0000","last_undegraded":"2024-04-21T23:08:50.710352+0000","last_fullsized":"2024-04-21T23:08:50.710352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:29:28.690841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735023+0000","last_change":"2024-04-21T23:06:25.995757+0000","last_active":"2024-04-21T23:08:50.735023+0000","last_peered":"2024-04-21T23:08:50.735023+0000","last_clean":"2024-04-21T23:08:50.735023+0000","last_became_active":"2024-04-21T23:06:25.995443+0000","last_became_peered":"2024-04-21T23:06:25.995443+0000","last_unstale":"2024-04-21T23:08:50.735023+0000","last_undegraded":"2024-04-21T23:08:50.735023+0000","last_fullsized":"2024-04-21T23:08:50.735023+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:47:40.359633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.699925+0000","last_change":"2024-04-21T23:06:30.266441+0000","last_active":"2024-04-21T23:08:50.699925+0000","last_peered":"2024-04-21T23:08:50.699925+0000","last_clean":"2024-04-21T23:08:50.699925+0000","last_became_active":"2024-04-21T23:06:30.264065+0000","last_became_peered":"2024-04-21T23:06:30.264065+0000","last_unstale":"2024-04-21T23:08:50.699925+0000","last_undegraded":"2024-04-21T23:08:50.699925+0000","last_fullsized":"2024-04-21T23:08:50.699925+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:53:22.548096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'38","reported_seq":324,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.953842+0000","last_change":"2024-04-21T23:06:25.213919+0000","last_active":"2024-04-21T23:08:50.953842+0000","last_peered":"2024-04-21T23:08:50.953842+0000","last_clean":"2024-04-21T23:08:50.953842+0000","last_became_active":"2024-04-21T23:06:25.212244+0000","last_became_peered":"2024-04-21T23:06:25.212244+0000","last_unstale":"2024-04-21T23:08:50.953842+0000","last_undegraded":"2024-04-21T23:08:50.953842+0000","last_fullsized":"2024-04-21T23:08:50.953842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:17:25.287718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69554,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":196,"num_write":38,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592581+0000","last_change":"2024-04-21T23:06:25.993666+0000","last_active":"2024-04-21T23:08:50.592581+0000","last_peered":"2024-04-21T23:08:50.592581+0000","last_clean":"2024-04-21T23:08:50.592581+0000","last_became_active":"2024-04-21T23:06:25.993330+0000","last_became_peered":"2024-04-21T23:06:25.993330+0000","last_unstale":"2024-04-21T23:08:50.592581+0000","last_undegraded":"2024-04-21T23:08:50.592581+0000","last_fullsized":"2024-04-21T23:08:50.592581+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:11:00.109782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710648+0000","last_change":"2024-04-21T23:06:30.265807+0000","last_active":"2024-04-21T23:08:50.710648+0000","last_peered":"2024-04-21T23:08:50.710648+0000","last_clean":"2024-04-21T23:08:50.710648+0000","last_became_active":"2024-04-21T23:06:30.265116+0000","last_became_peered":"2024-04-21T23:06:30.265116+0000","last_unstale":"2024-04-21T23:08:50.710648+0000","last_undegraded":"2024-04-21T23:08:50.710648+0000","last_fullsized":"2024-04-21T23:08:50.710648+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:14:04.178281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734740+0000","last_change":"2024-04-21T23:06:25.208319+0000","last_active":"2024-04-21T23:08:50.734740+0000","last_peered":"2024-04-21T23:08:50.734740+0000","last_clean":"2024-04-21T23:08:50.734740+0000","last_became_active":"2024-04-21T23:06:25.208025+0000","last_became_peered":"2024-04-21T23:06:25.208025+0000","last_unstale":"2024-04-21T23:08:50.734740+0000","last_undegraded":"2024-04-21T23:08:50.734740+0000","last_fullsized":"2024-04-21T23:08:50.734740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:46:14.723170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.736590+0000","last_change":"2024-04-21T23:06:25.993604+0000","last_active":"2024-04-21T23:08:50.736590+0000","last_peered":"2024-04-21T23:08:50.736590+0000","last_clean":"2024-04-21T23:08:50.736590+0000","last_became_active":"2024-04-21T23:06:25.993217+0000","last_became_peered":"2024-04-21T23:06:25.993217+0000","last_unstale":"2024-04-21T23:08:50.736590+0000","last_undegraded":"2024-04-21T23:08:50.736590+0000","last_fullsized":"2024-04-21T23:08:50.736590+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:23:00.435688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665369+0000","last_change":"2024-04-21T23:06:30.270991+0000","last_active":"2024-04-21T23:08:50.665369+0000","last_peered":"2024-04-21T23:08:50.665369+0000","last_clean":"2024-04-21T23:08:50.665369+0000","last_became_active":"2024-04-21T23:06:30.270511+0000","last_became_peered":"2024-04-21T23:06:30.270511+0000","last_unstale":"2024-04-21T23:08:50.665369+0000","last_undegraded":"2024-04-21T23:08:50.665369+0000","last_fullsized":"2024-04-21T23:08:50.665369+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:04.265855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710232+0000","last_change":"2024-04-21T23:06:25.211787+0000","last_active":"2024-04-21T23:08:50.710232+0000","last_peered":"2024-04-21T23:08:50.710232+0000","last_clean":"2024-04-21T23:08:50.710232+0000","last_became_active":"2024-04-21T23:06:25.211473+0000","last_became_peered":"2024-04-21T23:06:25.211473+0000","last_unstale":"2024-04-21T23:08:50.710232+0000","last_undegraded":"2024-04-21T23:08:50.710232+0000","last_fullsized":"2024-04-21T23:08:50.710232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:58:51.364974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792678+0000","last_change":"2024-04-21T23:06:25.990681+0000","last_active":"2024-04-21T23:08:50.792678+0000","last_peered":"2024-04-21T23:08:50.792678+0000","last_clean":"2024-04-21T23:08:50.792678+0000","last_became_active":"2024-04-21T23:06:25.990094+0000","last_became_peered":"2024-04-21T23:06:25.990094+0000","last_unstale":"2024-04-21T23:08:50.792678+0000","last_undegraded":"2024-04-21T23:08:50.792678+0000","last_fullsized":"2024-04-21T23:08:50.792678+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:15:26.907912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700460+0000","last_change":"2024-04-21T23:06:30.271352+0000","last_active":"2024-04-21T23:08:50.700460+0000","last_peered":"2024-04-21T23:08:50.700460+0000","last_clean":"2024-04-21T23:08:50.700460+0000","last_became_active":"2024-04-21T23:06:30.270601+0000","last_became_peered":"2024-04-21T23:06:30.270601+0000","last_unstale":"2024-04-21T23:08:50.700460+0000","last_undegraded":"2024-04-21T23:08:50.700460+0000","last_fullsized":"2024-04-21T23:08:50.700460+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:37.266987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592639+0000","last_change":"2024-04-21T23:06:25.213288+0000","last_active":"2024-04-21T23:08:50.592639+0000","last_peered":"2024-04-21T23:08:50.592639+0000","last_clean":"2024-04-21T23:08:50.592639+0000","last_became_active":"2024-04-21T23:06:25.212936+0000","last_became_peered":"2024-04-21T23:06:25.212936+0000","last_unstale":"2024-04-21T23:08:50.592639+0000","last_undegraded":"2024-04-21T23:08:50.592639+0000","last_fullsized":"2024-04-21T23:08:50.592639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:35.303093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792437+0000","last_change":"2024-04-21T23:06:25.998485+0000","last_active":"2024-04-21T23:08:50.792437+0000","last_peered":"2024-04-21T23:08:50.792437+0000","last_clean":"2024-04-21T23:08:50.792437+0000","last_became_active":"2024-04-21T23:06:25.998184+0000","last_became_peered":"2024-04-21T23:06:25.998184+0000","last_unstale":"2024-04-21T23:08:50.792437+0000","last_undegraded":"2024-04-21T23:08:50.792437+0000","last_fullsized":"2024-04-21T23:08:50.792437+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:35:26.613636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.736096+0000","last_change":"2024-04-21T23:06:30.270089+0000","last_active":"2024-04-21T23:08:50.736096+0000","last_peered":"2024-04-21T23:08:50.736096+0000","last_clean":"2024-04-21T23:08:50.736096+0000","last_became_active":"2024-04-21T23:06:30.269695+0000","last_became_peered":"2024-04-21T23:06:30.269695+0000","last_unstale":"2024-04-21T23:08:50.736096+0000","last_undegraded":"2024-04-21T23:08:50.736096+0000","last_fullsized":"2024-04-21T23:08:50.736096+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:34:42.718580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592339+0000","last_change":"2024-04-21T23:06:25.218247+0000","last_active":"2024-04-21T23:08:50.592339+0000","last_peered":"2024-04-21T23:08:50.592339+0000","last_clean":"2024-04-21T23:08:50.592339+0000","last_became_active":"2024-04-21T23:06:25.217924+0000","last_became_peered":"2024-04-21T23:06:25.217924+0000","last_unstale":"2024-04-21T23:08:50.592339+0000","last_undegraded":"2024-04-21T23:08:50.592339+0000","last_fullsized":"2024-04-21T23:08:50.592339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:27:12.155597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710045+0000","last_change":"2024-04-21T23:06:25.994109+0000","last_active":"2024-04-21T23:08:50.710045+0000","last_peered":"2024-04-21T23:08:50.710045+0000","last_clean":"2024-04-21T23:08:50.710045+0000","last_became_active":"2024-04-21T23:06:25.993785+0000","last_became_peered":"2024-04-21T23:06:25.993785+0000","last_unstale":"2024-04-21T23:08:50.710045+0000","last_undegraded":"2024-04-21T23:08:50.710045+0000","last_fullsized":"2024-04-21T23:08:50.710045+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:05:12.629364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.709882+0000","last_change":"2024-04-21T23:06:30.265444+0000","last_active":"2024-04-21T23:08:50.709882+0000","last_peered":"2024-04-21T23:08:50.709882+0000","last_clean":"2024-04-21T23:08:50.709882+0000","last_became_active":"2024-04-21T23:06:30.264971+0000","last_became_peered":"2024-04-21T23:06:30.264971+0000","last_unstale":"2024-04-21T23:08:50.709882+0000","last_undegraded":"2024-04-21T23:08:50.709882+0000","last_fullsized":"2024-04-21T23:08:50.709882+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:49:03.596631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665514+0000","last_change":"2024-04-21T23:07:24.465377+0000","last_active":"2024-04-21T23:08:50.665514+0000","last_peered":"2024-04-21T23:08:50.665514+0000","last_clean":"2024-04-21T23:08:50.665514+0000","last_became_active":"2024-04-21T23:07:24.454409+0000","last_became_peered":"2024-04-21T23:07:24.454409+0000","last_unstale":"2024-04-21T23:08:50.665514+0000","last_undegraded":"2024-04-21T23:08:50.665514+0000","last_fullsized":"2024-04-21T23:08:50.665514+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:06:22.836124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665569+0000","last_change":"2024-04-21T23:06:25.990193+0000","last_active":"2024-04-21T23:08:50.665569+0000","last_peered":"2024-04-21T23:08:50.665569+0000","last_clean":"2024-04-21T23:08:50.665569+0000","last_became_active":"2024-04-21T23:06:25.989842+0000","last_became_peered":"2024-04-21T23:06:25.989842+0000","last_unstale":"2024-04-21T23:08:50.665569+0000","last_undegraded":"2024-04-21T23:08:50.665569+0000","last_fullsized":"2024-04-21T23:08:50.665569+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:54:53.421257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592553+0000","last_change":"2024-04-21T23:07:24.817262+0000","last_active":"2024-04-21T23:08:50.592553+0000","last_peered":"2024-04-21T23:08:50.592553+0000","last_clean":"2024-04-21T23:08:50.592553+0000","last_became_active":"2024-04-21T23:07:24.816668+0000","last_became_peered":"2024-04-21T23:07:24.816668+0000","last_unstale":"2024-04-21T23:08:50.592553+0000","last_undegraded":"2024-04-21T23:08:50.592553+0000","last_fullsized":"2024-04-21T23:08:50.592553+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:07:25.302151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700278+0000","last_change":"2024-04-21T23:06:25.218208+0000","last_active":"2024-04-21T23:08:50.700278+0000","last_peered":"2024-04-21T23:08:50.700278+0000","last_clean":"2024-04-21T23:08:50.700278+0000","last_became_active":"2024-04-21T23:06:25.217889+0000","last_became_peered":"2024-04-21T23:06:25.217889+0000","last_unstale":"2024-04-21T23:08:50.700278+0000","last_undegraded":"2024-04-21T23:08:50.700278+0000","last_fullsized":"2024-04-21T23:08:50.700278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:00:08.775682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.713684+0000","last_change":"2024-04-21T23:06:25.987615+0000","last_active":"2024-04-21T23:08:50.713684+0000","last_peered":"2024-04-21T23:08:50.713684+0000","last_clean":"2024-04-21T23:08:50.713684+0000","last_became_active":"2024-04-21T23:06:25.987305+0000","last_became_peered":"2024-04-21T23:06:25.987305+0000","last_unstale":"2024-04-21T23:08:50.713684+0000","last_undegraded":"2024-04-21T23:08:50.713684+0000","last_fullsized":"2024-04-21T23:08:50.713684+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:23:49.386998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735164+0000","last_change":"2024-04-21T23:06:30.268157+0000","last_active":"2024-04-21T23:08:50.735164+0000","last_peered":"2024-04-21T23:08:50.735164+0000","last_clean":"2024-04-21T23:08:50.735164+0000","last_became_active":"2024-04-21T23:06:30.267732+0000","last_became_peered":"2024-04-21T23:06:30.267732+0000","last_unstale":"2024-04-21T23:08:50.735164+0000","last_undegraded":"2024-04-21T23:08:50.735164+0000","last_fullsized":"2024-04-21T23:08:50.735164+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:29:24.250271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792667+0000","last_change":"2024-04-21T23:06:25.215401+0000","last_active":"2024-04-21T23:08:50.792667+0000","last_peered":"2024-04-21T23:08:50.792667+0000","last_clean":"2024-04-21T23:08:50.792667+0000","last_became_active":"2024-04-21T23:06:25.215067+0000","last_became_peered":"2024-04-21T23:06:25.215067+0000","last_unstale":"2024-04-21T23:08:50.792667+0000","last_undegraded":"2024-04-21T23:08:50.792667+0000","last_fullsized":"2024-04-21T23:08:50.792667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:16:58.977797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.713637+0000","last_change":"2024-04-21T23:06:25.996421+0000","last_active":"2024-04-21T23:08:50.713637+0000","last_peered":"2024-04-21T23:08:50.713637+0000","last_clean":"2024-04-21T23:08:50.713637+0000","last_became_active":"2024-04-21T23:06:25.996102+0000","last_became_peered":"2024-04-21T23:06:25.996102+0000","last_unstale":"2024-04-21T23:08:50.713637+0000","last_undegraded":"2024-04-21T23:08:50.713637+0000","last_fullsized":"2024-04-21T23:08:50.713637+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:06:42.630182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.736211+0000","last_change":"2024-04-21T23:06:30.404509+0000","last_active":"2024-04-21T23:08:50.736211+0000","last_peered":"2024-04-21T23:08:50.736211+0000","last_clean":"2024-04-21T23:08:50.736211+0000","last_became_active":"2024-04-21T23:06:30.403987+0000","last_became_peered":"2024-04-21T23:06:30.403987+0000","last_unstale":"2024-04-21T23:08:50.736211+0000","last_undegraded":"2024-04-21T23:08:50.736211+0000","last_fullsized":"2024-04-21T23:08:50.736211+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:12:28.412117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710087+0000","last_change":"2024-04-21T23:06:25.994076+0000","last_active":"2024-04-21T23:08:50.710087+0000","last_peered":"2024-04-21T23:08:50.710087+0000","last_clean":"2024-04-21T23:08:50.710087+0000","last_became_active":"2024-04-21T23:06:25.993747+0000","last_became_peered":"2024-04-21T23:06:25.993747+0000","last_unstale":"2024-04-21T23:08:50.710087+0000","last_undegraded":"2024-04-21T23:08:50.710087+0000","last_fullsized":"2024-04-21T23:08:50.710087+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:48:17.771165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734824+0000","last_change":"2024-04-21T23:06:25.209756+0000","last_active":"2024-04-21T23:08:50.734824+0000","last_peered":"2024-04-21T23:08:50.734824+0000","last_clean":"2024-04-21T23:08:50.734824+0000","last_became_active":"2024-04-21T23:06:25.209445+0000","last_became_peered":"2024-04-21T23:06:25.209445+0000","last_unstale":"2024-04-21T23:08:50.734824+0000","last_undegraded":"2024-04-21T23:08:50.734824+0000","last_fullsized":"2024-04-21T23:08:50.734824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:58:49.453047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792987+0000","last_change":"2024-04-21T23:06:30.261565+0000","last_active":"2024-04-21T23:08:50.792987+0000","last_peered":"2024-04-21T23:08:50.792987+0000","last_clean":"2024-04-21T23:08:50.792987+0000","last_became_active":"2024-04-21T23:06:30.261095+0000","last_became_peered":"2024-04-21T23:06:30.261095+0000","last_unstale":"2024-04-21T23:08:50.792987+0000","last_undegraded":"2024-04-21T23:08:50.792987+0000","last_fullsized":"2024-04-21T23:08:50.792987+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:37:23.174152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700060+0000","last_change":"2024-04-21T23:08:25.103635+0000","last_active":"2024-04-21T23:08:50.700060+0000","last_peered":"2024-04-21T23:08:50.700060+0000","last_clean":"2024-04-21T23:08:50.700060+0000","last_became_active":"2024-04-21T23:08:25.103318+0000","last_became_peered":"2024-04-21T23:08:25.103318+0000","last_unstale":"2024-04-21T23:08:50.700060+0000","last_undegraded":"2024-04-21T23:08:50.700060+0000","last_fullsized":"2024-04-21T23:08:50.700060+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:56:55.215016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592569+0000","last_change":"2024-04-21T23:06:25.208968+0000","last_active":"2024-04-21T23:08:50.592569+0000","last_peered":"2024-04-21T23:08:50.592569+0000","last_clean":"2024-04-21T23:08:50.592569+0000","last_became_active":"2024-04-21T23:06:25.208606+0000","last_became_peered":"2024-04-21T23:06:25.208606+0000","last_unstale":"2024-04-21T23:08:50.592569+0000","last_undegraded":"2024-04-21T23:08:50.592569+0000","last_fullsized":"2024-04-21T23:08:50.592569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:31:46.147119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":369,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792237+0000","last_change":"2024-04-21T23:05:13.077846+0000","last_active":"2024-04-21T23:08:50.792237+0000","last_peered":"2024-04-21T23:08:50.792237+0000","last_clean":"2024-04-21T23:08:50.792237+0000","last_became_active":"2024-04-21T23:05:13.077436+0000","last_became_peered":"2024-04-21T23:05:13.077436+0000","last_unstale":"2024-04-21T23:08:50.792237+0000","last_undegraded":"2024-04-21T23:08:50.792237+0000","last_fullsized":"2024-04-21T23:08:50.792237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:01:47.619212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:01:47.619212+0000","last_clean_scrub_stamp":"2024-04-21T23:01:47.619212+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:12:09.826839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.710151+0000","last_change":"2024-04-21T23:06:30.405121+0000","last_active":"2024-04-21T23:08:50.710151+0000","last_peered":"2024-04-21T23:08:50.710151+0000","last_clean":"2024-04-21T23:08:50.710151+0000","last_became_active":"2024-04-21T23:06:30.404612+0000","last_became_peered":"2024-04-21T23:06:30.404612+0000","last_unstale":"2024-04-21T23:08:50.710151+0000","last_undegraded":"2024-04-21T23:08:50.710151+0000","last_fullsized":"2024-04-21T23:08:50.710151+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:59:01.737337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734745+0000","last_change":"2024-04-21T23:08:25.102015+0000","last_active":"2024-04-21T23:08:50.734745+0000","last_peered":"2024-04-21T23:08:50.734745+0000","last_clean":"2024-04-21T23:08:50.734745+0000","last_became_active":"2024-04-21T23:08:25.101674+0000","last_became_peered":"2024-04-21T23:08:25.101674+0000","last_unstale":"2024-04-21T23:08:50.734745+0000","last_undegraded":"2024-04-21T23:08:50.734745+0000","last_fullsized":"2024-04-21T23:08:50.734745+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:14:19.304795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792365+0000","last_change":"2024-04-21T23:06:25.209714+0000","last_active":"2024-04-21T23:08:50.792365+0000","last_peered":"2024-04-21T23:08:50.792365+0000","last_clean":"2024-04-21T23:08:50.792365+0000","last_became_active":"2024-04-21T23:06:25.209373+0000","last_became_peered":"2024-04-21T23:06:25.209373+0000","last_unstale":"2024-04-21T23:08:50.792365+0000","last_undegraded":"2024-04-21T23:08:50.792365+0000","last_fullsized":"2024-04-21T23:08:50.792365+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:24:21.654143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665293+0000","last_change":"2024-04-21T23:06:30.264873+0000","last_active":"2024-04-21T23:08:50.665293+0000","last_peered":"2024-04-21T23:08:50.665293+0000","last_clean":"2024-04-21T23:08:50.665293+0000","last_became_active":"2024-04-21T23:06:30.264318+0000","last_became_peered":"2024-04-21T23:06:30.264318+0000","last_unstale":"2024-04-21T23:08:50.665293+0000","last_undegraded":"2024-04-21T23:08:50.665293+0000","last_fullsized":"2024-04-21T23:08:50.665293+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:01:22.558782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592702+0000","last_change":"2024-04-21T23:06:25.218767+0000","last_active":"2024-04-21T23:08:50.592702+0000","last_peered":"2024-04-21T23:08:50.592702+0000","last_clean":"2024-04-21T23:08:50.592702+0000","last_became_active":"2024-04-21T23:06:25.218278+0000","last_became_peered":"2024-04-21T23:06:25.218278+0000","last_unstale":"2024-04-21T23:08:50.592702+0000","last_undegraded":"2024-04-21T23:08:50.592702+0000","last_fullsized":"2024-04-21T23:08:50.592702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:03:05.063515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735353+0000","last_change":"2024-04-21T23:06:25.968551+0000","last_active":"2024-04-21T23:08:50.735353+0000","last_peered":"2024-04-21T23:08:50.735353+0000","last_clean":"2024-04-21T23:08:50.735353+0000","last_became_active":"2024-04-21T23:06:25.968167+0000","last_became_peered":"2024-04-21T23:06:25.968167+0000","last_unstale":"2024-04-21T23:08:50.735353+0000","last_undegraded":"2024-04-21T23:08:50.735353+0000","last_fullsized":"2024-04-21T23:08:50.735353+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:23:08.920677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.590757+0000","last_change":"2024-04-21T23:06:30.404811+0000","last_active":"2024-04-21T23:08:50.590757+0000","last_peered":"2024-04-21T23:08:50.590757+0000","last_clean":"2024-04-21T23:08:50.590757+0000","last_became_active":"2024-04-21T23:06:30.404375+0000","last_became_peered":"2024-04-21T23:06:30.404375+0000","last_unstale":"2024-04-21T23:08:50.590757+0000","last_undegraded":"2024-04-21T23:08:50.590757+0000","last_fullsized":"2024-04-21T23:08:50.590757+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:10:15.688917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"95'43","reported_seq":100,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.832453+0000","last_change":"2024-04-21T23:06:25.204371+0000","last_active":"2024-04-21T23:08:50.832453+0000","last_peered":"2024-04-21T23:08:50.832453+0000","last_clean":"2024-04-21T23:08:50.832453+0000","last_became_active":"2024-04-21T23:06:25.204025+0000","last_became_peered":"2024-04-21T23:06:25.204025+0000","last_unstale":"2024-04-21T23:08:50.832453+0000","last_undegraded":"2024-04-21T23:08:50.832453+0000","last_fullsized":"2024-04-21T23:08:50.832453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:34:31.071013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":77,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.591302+0000","last_change":"2024-04-21T23:06:25.990800+0000","last_active":"2024-04-21T23:08:50.591302+0000","last_peered":"2024-04-21T23:08:50.591302+0000","last_clean":"2024-04-21T23:08:50.591302+0000","last_became_active":"2024-04-21T23:06:25.990477+0000","last_became_peered":"2024-04-21T23:06:25.990477+0000","last_unstale":"2024-04-21T23:08:50.591302+0000","last_undegraded":"2024-04-21T23:08:50.591302+0000","last_fullsized":"2024-04-21T23:08:50.591302+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:55:33.282303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734511+0000","last_change":"2024-04-21T23:06:30.260090+0000","last_active":"2024-04-21T23:08:50.734511+0000","last_peered":"2024-04-21T23:08:50.734511+0000","last_clean":"2024-04-21T23:08:50.734511+0000","last_became_active":"2024-04-21T23:06:30.259609+0000","last_became_peered":"2024-04-21T23:06:30.259609+0000","last_unstale":"2024-04-21T23:08:50.734511+0000","last_undegraded":"2024-04-21T23:08:50.734511+0000","last_fullsized":"2024-04-21T23:08:50.734511+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:11:17.933683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592986+0000","last_change":"2024-04-21T23:06:25.215446+0000","last_active":"2024-04-21T23:08:50.592986+0000","last_peered":"2024-04-21T23:08:50.592986+0000","last_clean":"2024-04-21T23:08:50.592986+0000","last_became_active":"2024-04-21T23:06:25.214773+0000","last_became_peered":"2024-04-21T23:06:25.214773+0000","last_unstale":"2024-04-21T23:08:50.592986+0000","last_undegraded":"2024-04-21T23:08:50.592986+0000","last_fullsized":"2024-04-21T23:08:50.592986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:25:01.090699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735730+0000","last_change":"2024-04-21T23:06:25.986640+0000","last_active":"2024-04-21T23:08:50.735730+0000","last_peered":"2024-04-21T23:08:50.735730+0000","last_clean":"2024-04-21T23:08:50.735730+0000","last_became_active":"2024-04-21T23:06:25.986285+0000","last_became_peered":"2024-04-21T23:06:25.986285+0000","last_unstale":"2024-04-21T23:08:50.735730+0000","last_undegraded":"2024-04-21T23:08:50.735730+0000","last_fullsized":"2024-04-21T23:08:50.735730+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:01:05.498199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592869+0000","last_change":"2024-04-21T23:07:24.458291+0000","last_active":"2024-04-21T23:08:50.592869+0000","last_peered":"2024-04-21T23:08:50.592869+0000","last_clean":"2024-04-21T23:08:50.592869+0000","last_became_active":"2024-04-21T23:07:24.457677+0000","last_became_peered":"2024-04-21T23:07:24.457677+0000","last_unstale":"2024-04-21T23:08:50.592869+0000","last_undegraded":"2024-04-21T23:08:50.592869+0000","last_fullsized":"2024-04-21T23:08:50.592869+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:20:30.708901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,11],"acting":[4,8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734509+0000","last_change":"2024-04-21T23:06:25.217298+0000","last_active":"2024-04-21T23:08:50.734509+0000","last_peered":"2024-04-21T23:08:50.734509+0000","last_clean":"2024-04-21T23:08:50.734509+0000","last_became_active":"2024-04-21T23:06:25.216965+0000","last_became_peered":"2024-04-21T23:06:25.216965+0000","last_unstale":"2024-04-21T23:08:50.734509+0000","last_undegraded":"2024-04-21T23:08:50.734509+0000","last_fullsized":"2024-04-21T23:08:50.734509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:56:38.543697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592266+0000","last_change":"2024-04-21T23:06:25.981400+0000","last_active":"2024-04-21T23:08:50.592266+0000","last_peered":"2024-04-21T23:08:50.592266+0000","last_clean":"2024-04-21T23:08:50.592266+0000","last_became_active":"2024-04-21T23:06:25.981118+0000","last_became_peered":"2024-04-21T23:06:25.981118+0000","last_unstale":"2024-04-21T23:08:50.592266+0000","last_undegraded":"2024-04-21T23:08:50.592266+0000","last_fullsized":"2024-04-21T23:08:50.592266+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:20:59.631062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.700704+0000","last_change":"2024-04-21T23:06:30.270351+0000","last_active":"2024-04-21T23:08:50.700704+0000","last_peered":"2024-04-21T23:08:50.700704+0000","last_clean":"2024-04-21T23:08:50.700704+0000","last_became_active":"2024-04-21T23:06:30.269659+0000","last_became_peered":"2024-04-21T23:06:30.269659+0000","last_unstale":"2024-04-21T23:08:50.700704+0000","last_undegraded":"2024-04-21T23:08:50.700704+0000","last_fullsized":"2024-04-21T23:08:50.700704+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:19:14.132191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.590816+0000","last_change":"2024-04-21T23:06:25.212787+0000","last_active":"2024-04-21T23:08:50.590816+0000","last_peered":"2024-04-21T23:08:50.590816+0000","last_clean":"2024-04-21T23:08:50.590816+0000","last_became_active":"2024-04-21T23:06:25.212423+0000","last_became_peered":"2024-04-21T23:06:25.212423+0000","last_unstale":"2024-04-21T23:08:50.590816+0000","last_undegraded":"2024-04-21T23:08:50.590816+0000","last_fullsized":"2024-04-21T23:08:50.590816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:24:45.875664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665156+0000","last_change":"2024-04-21T23:06:25.978205+0000","last_active":"2024-04-21T23:08:50.665156+0000","last_peered":"2024-04-21T23:08:50.665156+0000","last_clean":"2024-04-21T23:08:50.665156+0000","last_became_active":"2024-04-21T23:06:25.977876+0000","last_became_peered":"2024-04-21T23:06:25.977876+0000","last_unstale":"2024-04-21T23:08:50.665156+0000","last_undegraded":"2024-04-21T23:08:50.665156+0000","last_fullsized":"2024-04-21T23:08:50.665156+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:43:29.280255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592602+0000","last_change":"2024-04-21T23:06:30.271928+0000","last_active":"2024-04-21T23:08:50.592602+0000","last_peered":"2024-04-21T23:08:50.592602+0000","last_clean":"2024-04-21T23:08:50.592602+0000","last_became_active":"2024-04-21T23:06:30.271321+0000","last_became_peered":"2024-04-21T23:06:30.271321+0000","last_unstale":"2024-04-21T23:08:50.592602+0000","last_undegraded":"2024-04-21T23:08:50.592602+0000","last_fullsized":"2024-04-21T23:08:50.592602+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:22:03.343620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734759+0000","last_change":"2024-04-21T23:06:25.204175+0000","last_active":"2024-04-21T23:08:50.734759+0000","last_peered":"2024-04-21T23:08:50.734759+0000","last_clean":"2024-04-21T23:08:50.734759+0000","last_became_active":"2024-04-21T23:06:25.203826+0000","last_became_peered":"2024-04-21T23:06:25.203826+0000","last_unstale":"2024-04-21T23:08:50.734759+0000","last_undegraded":"2024-04-21T23:08:50.734759+0000","last_fullsized":"2024-04-21T23:08:50.734759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:28:53.402395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.591046+0000","last_change":"2024-04-21T23:06:25.990559+0000","last_active":"2024-04-21T23:08:50.591046+0000","last_peered":"2024-04-21T23:08:50.591046+0000","last_clean":"2024-04-21T23:08:50.591046+0000","last_became_active":"2024-04-21T23:06:25.990239+0000","last_became_peered":"2024-04-21T23:06:25.990239+0000","last_unstale":"2024-04-21T23:08:50.591046+0000","last_undegraded":"2024-04-21T23:08:50.591046+0000","last_fullsized":"2024-04-21T23:08:50.591046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:49:26.891190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.592994+0000","last_change":"2024-04-21T23:06:30.406584+0000","last_active":"2024-04-21T23:08:50.592994+0000","last_peered":"2024-04-21T23:08:50.592994+0000","last_clean":"2024-04-21T23:08:50.592994+0000","last_became_active":"2024-04-21T23:06:30.404576+0000","last_became_peered":"2024-04-21T23:06:30.404576+0000","last_unstale":"2024-04-21T23:08:50.592994+0000","last_undegraded":"2024-04-21T23:08:50.592994+0000","last_fullsized":"2024-04-21T23:08:50.592994+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T02:12:56.047199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665253+0000","last_change":"2024-04-21T23:07:24.460375+0000","last_active":"2024-04-21T23:08:50.665253+0000","last_peered":"2024-04-21T23:08:50.665253+0000","last_clean":"2024-04-21T23:08:50.665253+0000","last_became_active":"2024-04-21T23:07:24.453873+0000","last_became_peered":"2024-04-21T23:07:24.453873+0000","last_unstale":"2024-04-21T23:08:50.665253+0000","last_undegraded":"2024-04-21T23:08:50.665253+0000","last_fullsized":"2024-04-21T23:08:50.665253+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:46:27.709414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792771+0000","last_change":"2024-04-21T23:06:25.988925+0000","last_active":"2024-04-21T23:08:50.792771+0000","last_peered":"2024-04-21T23:08:50.792771+0000","last_clean":"2024-04-21T23:08:50.792771+0000","last_became_active":"2024-04-21T23:06:25.988480+0000","last_became_peered":"2024-04-21T23:06:25.988480+0000","last_unstale":"2024-04-21T23:08:50.792771+0000","last_undegraded":"2024-04-21T23:08:50.792771+0000","last_fullsized":"2024-04-21T23:08:50.792771+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T01:32:02.867858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734682+0000","last_change":"2024-04-21T23:06:30.260841+0000","last_active":"2024-04-21T23:08:50.734682+0000","last_peered":"2024-04-21T23:08:50.734682+0000","last_clean":"2024-04-21T23:08:50.734682+0000","last_became_active":"2024-04-21T23:06:30.260376+0000","last_became_peered":"2024-04-21T23:06:30.260376+0000","last_unstale":"2024-04-21T23:08:50.734682+0000","last_undegraded":"2024-04-21T23:08:50.734682+0000","last_fullsized":"2024-04-21T23:08:50.734682+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:24:57.429169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735001+0000","last_change":"2024-04-21T23:06:25.217920+0000","last_active":"2024-04-21T23:08:50.735001+0000","last_peered":"2024-04-21T23:08:50.735001+0000","last_clean":"2024-04-21T23:08:50.735001+0000","last_became_active":"2024-04-21T23:06:25.217379+0000","last_became_peered":"2024-04-21T23:06:25.217379+0000","last_unstale":"2024-04-21T23:08:50.735001+0000","last_undegraded":"2024-04-21T23:08:50.735001+0000","last_fullsized":"2024-04-21T23:08:50.735001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T10:07:09.570463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.793234+0000","last_change":"2024-04-21T23:06:25.993533+0000","last_active":"2024-04-21T23:08:50.793234+0000","last_peered":"2024-04-21T23:08:50.793234+0000","last_clean":"2024-04-21T23:08:50.793234+0000","last_became_active":"2024-04-21T23:06:25.989739+0000","last_became_peered":"2024-04-21T23:06:25.989739+0000","last_unstale":"2024-04-21T23:08:50.793234+0000","last_undegraded":"2024-04-21T23:08:50.793234+0000","last_fullsized":"2024-04-21T23:08:50.793234+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:24:05.594026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735054+0000","last_change":"2024-04-21T23:06:30.276121+0000","last_active":"2024-04-21T23:08:50.735054+0000","last_peered":"2024-04-21T23:08:50.735054+0000","last_clean":"2024-04-21T23:08:50.735054+0000","last_became_active":"2024-04-21T23:06:30.275650+0000","last_became_peered":"2024-04-21T23:06:30.275650+0000","last_unstale":"2024-04-21T23:08:50.735054+0000","last_undegraded":"2024-04-21T23:08:50.735054+0000","last_fullsized":"2024-04-21T23:08:50.735054+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:00:19.559482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"95'39","reported_seq":81,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.826456+0000","last_change":"2024-04-21T23:06:25.207079+0000","last_active":"2024-04-21T23:08:50.826456+0000","last_peered":"2024-04-21T23:08:50.826456+0000","last_clean":"2024-04-21T23:08:50.826456+0000","last_became_active":"2024-04-21T23:06:25.206724+0000","last_became_peered":"2024-04-21T23:06:25.206724+0000","last_unstale":"2024-04-21T23:08:50.826456+0000","last_undegraded":"2024-04-21T23:08:50.826456+0000","last_fullsized":"2024-04-21T23:08:50.826456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:08:27.221282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.792921+0000","last_change":"2024-04-21T23:06:25.988144+0000","last_active":"2024-04-21T23:08:50.792921+0000","last_peered":"2024-04-21T23:08:50.792921+0000","last_clean":"2024-04-21T23:08:50.792921+0000","last_became_active":"2024-04-21T23:06:25.987813+0000","last_became_peered":"2024-04-21T23:06:25.987813+0000","last_unstale":"2024-04-21T23:08:50.792921+0000","last_undegraded":"2024-04-21T23:08:50.792921+0000","last_fullsized":"2024-04-21T23:08:50.792921+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:27:23.733514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.709772+0000","last_change":"2024-04-21T23:06:30.261548+0000","last_active":"2024-04-21T23:08:50.709772+0000","last_peered":"2024-04-21T23:08:50.709772+0000","last_clean":"2024-04-21T23:08:50.709772+0000","last_became_active":"2024-04-21T23:06:30.261152+0000","last_became_peered":"2024-04-21T23:06:30.261152+0000","last_unstale":"2024-04-21T23:08:50.709772+0000","last_undegraded":"2024-04-21T23:08:50.709772+0000","last_fullsized":"2024-04-21T23:08:50.709772+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:29.230523+0000","last_clean_scrub_stamp":"2024-04-21T23:06:29.230523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:00:16.644253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735978+0000","last_change":"2024-04-21T23:06:25.208749+0000","last_active":"2024-04-21T23:08:50.735978+0000","last_peered":"2024-04-21T23:08:50.735978+0000","last_clean":"2024-04-21T23:08:50.735978+0000","last_became_active":"2024-04-21T23:06:25.201511+0000","last_became_peered":"2024-04-21T23:06:25.201511+0000","last_unstale":"2024-04-21T23:08:50.735978+0000","last_undegraded":"2024-04-21T23:08:50.735978+0000","last_fullsized":"2024-04-21T23:08:50.735978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:24.179207+0000","last_clean_scrub_stamp":"2024-04-21T23:06:24.179207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:08:41.928835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.793146+0000","last_change":"2024-04-21T23:06:25.996041+0000","last_active":"2024-04-21T23:08:50.793146+0000","last_peered":"2024-04-21T23:08:50.793146+0000","last_clean":"2024-04-21T23:08:50.793146+0000","last_became_active":"2024-04-21T23:06:25.994482+0000","last_became_peered":"2024-04-21T23:06:25.994482+0000","last_unstale":"2024-04-21T23:08:50.793146+0000","last_undegraded":"2024-04-21T23:08:50.793146+0000","last_fullsized":"2024-04-21T23:08:50.793146+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:15:46.792618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665223+0000","last_change":"2024-04-21T23:06:25.974061+0000","last_active":"2024-04-21T23:08:50.665223+0000","last_peered":"2024-04-21T23:08:50.665223+0000","last_clean":"2024-04-21T23:08:50.665223+0000","last_became_active":"2024-04-21T23:06:25.973708+0000","last_became_peered":"2024-04-21T23:06:25.973708+0000","last_unstale":"2024-04-21T23:08:50.665223+0000","last_undegraded":"2024-04-21T23:08:50.665223+0000","last_fullsized":"2024-04-21T23:08:50.665223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T07:33:58.702947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735074+0000","last_change":"2024-04-21T23:06:25.988412+0000","last_active":"2024-04-21T23:08:50.735074+0000","last_peered":"2024-04-21T23:08:50.735074+0000","last_clean":"2024-04-21T23:08:50.735074+0000","last_became_active":"2024-04-21T23:06:25.988100+0000","last_became_peered":"2024-04-21T23:06:25.988100+0000","last_unstale":"2024-04-21T23:08:50.735074+0000","last_undegraded":"2024-04-21T23:08:50.735074+0000","last_fullsized":"2024-04-21T23:08:50.735074+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:16:42.626123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734567+0000","last_change":"2024-04-21T23:06:25.980144+0000","last_active":"2024-04-21T23:08:50.734567+0000","last_peered":"2024-04-21T23:08:50.734567+0000","last_clean":"2024-04-21T23:08:50.734567+0000","last_became_active":"2024-04-21T23:06:25.979760+0000","last_became_peered":"2024-04-21T23:06:25.979760+0000","last_unstale":"2024-04-21T23:08:50.734567+0000","last_undegraded":"2024-04-21T23:08:50.734567+0000","last_fullsized":"2024-04-21T23:08:50.734567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T08:14:32.496184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735274+0000","last_change":"2024-04-21T23:06:25.994732+0000","last_active":"2024-04-21T23:08:50.735274+0000","last_peered":"2024-04-21T23:08:50.735274+0000","last_clean":"2024-04-21T23:08:50.735274+0000","last_became_active":"2024-04-21T23:06:25.994408+0000","last_became_peered":"2024-04-21T23:06:25.994408+0000","last_unstale":"2024-04-21T23:08:50.735274+0000","last_undegraded":"2024-04-21T23:08:50.735274+0000","last_fullsized":"2024-04-21T23:08:50.735274+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:28:59.778049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.735511+0000","last_change":"2024-04-21T23:07:24.455416+0000","last_active":"2024-04-21T23:08:50.735511+0000","last_peered":"2024-04-21T23:08:50.735511+0000","last_clean":"2024-04-21T23:08:50.735511+0000","last_became_active":"2024-04-21T23:07:24.454954+0000","last_became_peered":"2024-04-21T23:07:24.454954+0000","last_unstale":"2024-04-21T23:08:50.735511+0000","last_undegraded":"2024-04-21T23:08:50.735511+0000","last_fullsized":"2024-04-21T23:08:50.735511+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:29:09.706719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665537+0000","last_change":"2024-04-21T23:06:25.973850+0000","last_active":"2024-04-21T23:08:50.665537+0000","last_peered":"2024-04-21T23:08:50.665537+0000","last_clean":"2024-04-21T23:08:50.665537+0000","last_became_active":"2024-04-21T23:06:25.973522+0000","last_became_peered":"2024-04-21T23:06:25.973522+0000","last_unstale":"2024-04-21T23:08:50.665537+0000","last_undegraded":"2024-04-21T23:08:50.665537+0000","last_fullsized":"2024-04-21T23:08:50.665537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T23:43:58.417551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734906+0000","last_change":"2024-04-21T23:06:25.978114+0000","last_active":"2024-04-21T23:08:50.734906+0000","last_peered":"2024-04-21T23:08:50.734906+0000","last_clean":"2024-04-21T23:08:50.734906+0000","last_became_active":"2024-04-21T23:06:25.977115+0000","last_became_peered":"2024-04-21T23:06:25.977115+0000","last_unstale":"2024-04-21T23:08:50.734906+0000","last_undegraded":"2024-04-21T23:08:50.734906+0000","last_fullsized":"2024-04-21T23:08:50.734906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T05:26:46.756490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.591439+0000","last_change":"2024-04-21T23:06:25.994451+0000","last_active":"2024-04-21T23:08:50.591439+0000","last_peered":"2024-04-21T23:08:50.591439+0000","last_clean":"2024-04-21T23:08:50.591439+0000","last_became_active":"2024-04-21T23:06:25.994085+0000","last_became_peered":"2024-04-21T23:06:25.994085+0000","last_unstale":"2024-04-21T23:08:50.591439+0000","last_undegraded":"2024-04-21T23:08:50.591439+0000","last_fullsized":"2024-04-21T23:08:50.591439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T04:33:53.652014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.709991+0000","last_change":"2024-04-21T23:06:25.982536+0000","last_active":"2024-04-21T23:08:50.709991+0000","last_peered":"2024-04-21T23:08:50.709991+0000","last_clean":"2024-04-21T23:08:50.709991+0000","last_became_active":"2024-04-21T23:06:25.981318+0000","last_became_peered":"2024-04-21T23:06:25.981318+0000","last_unstale":"2024-04-21T23:08:50.709991+0000","last_undegraded":"2024-04-21T23:08:50.709991+0000","last_fullsized":"2024-04-21T23:08:50.709991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T09:45:35.784679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.590938+0000","last_change":"2024-04-21T23:06:25.981915+0000","last_active":"2024-04-21T23:08:50.590938+0000","last_peered":"2024-04-21T23:08:50.590938+0000","last_clean":"2024-04-21T23:08:50.590938+0000","last_became_active":"2024-04-21T23:06:25.981547+0000","last_became_peered":"2024-04-21T23:06:25.981547+0000","last_unstale":"2024-04-21T23:08:50.590938+0000","last_undegraded":"2024-04-21T23:08:50.590938+0000","last_fullsized":"2024-04-21T23:08:50.590938+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T11:03:11.154582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.734923+0000","last_change":"2024-04-21T23:08:25.561660+0000","last_active":"2024-04-21T23:08:50.734923+0000","last_peered":"2024-04-21T23:08:50.734923+0000","last_clean":"2024-04-21T23:08:50.734923+0000","last_became_active":"2024-04-21T23:08:25.561063+0000","last_became_peered":"2024-04-21T23:08:25.561063+0000","last_unstale":"2024-04-21T23:08:50.734923+0000","last_undegraded":"2024-04-21T23:08:50.734923+0000","last_fullsized":"2024-04-21T23:08:50.734923+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T00:14:27.279046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.737031+0000","last_change":"2024-04-21T23:06:25.993658+0000","last_active":"2024-04-21T23:08:50.737031+0000","last_peered":"2024-04-21T23:08:50.737031+0000","last_clean":"2024-04-21T23:08:50.737031+0000","last_became_active":"2024-04-21T23:06:25.993298+0000","last_became_peered":"2024-04-21T23:06:25.993298+0000","last_unstale":"2024-04-21T23:08:50.737031+0000","last_undegraded":"2024-04-21T23:08:50.737031+0000","last_fullsized":"2024-04-21T23:08:50.737031+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T03:18:30.216961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-21T23:08:50.665492+0000","last_change":"2024-04-21T23:06:25.993849+0000","last_active":"2024-04-21T23:08:50.665492+0000","last_peered":"2024-04-21T23:08:50.665492+0000","last_clean":"2024-04-21T23:08:50.665492+0000","last_became_active":"2024-04-21T23:06:25.993501+0000","last_became_peered":"2024-04-21T23:06:25.993501+0000","last_unstale":"2024-04-21T23:08:50.665492+0000","last_undegraded":"2024-04-21T23:08:50.665492+0000","last_fullsized":"2024-04-21T23:08:50.665492+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T23:06:25.186139+0000","last_clean_scrub_stamp":"2024-04-21T23:06:25.186139+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T06:40:10.461775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595168,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":347,"num_read_kb":278,"num_write":201,"num_write_kb":1411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1880064,"data_stored":1785504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351018,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1085440,"data_stored":535680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612655,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33028,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712124,"statfs":{"total":95995035648,"available":95961214976,"internally_reserved":0,"allocated":1896448,"data_stored":1323278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808884,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28468,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716684,"statfs":{"total":95995035648,"available":95965884416,"internally_reserved":0,"allocated":1482752,"data_stored":933552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36784,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708368,"statfs":{"total":95995035648,"available":95957368832,"internally_reserved":0,"allocated":1478656,"data_stored":929456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332157,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28664,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716488,"statfs":{"total":95995035648,"available":95965683712,"internally_reserved":0,"allocated":1683456,"data_stored":1130704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495683,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":531562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32308,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712844,"statfs":{"total":95995035648,"available":95961952256,"internally_reserved":0,"allocated":1155072,"data_stored":601116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1064960,"data_stored":521749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1138688,"data_stored":585092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":531666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":535680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28976,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716176,"statfs":{"total":95995035648,"available":95965364224,"internally_reserved":0,"allocated":1822720,"data_stored":1257808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":69554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":69588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":524920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":131782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":393798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":393798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":201348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T23:08:53.357 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:08:53.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.500+0000 7f1b9bbe5640 1 -- 172.21.15.44:0/2542923807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b941074e0 msgr2=0x7f1b941098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:53.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.500+0000 7f1b9bbe5640 1 --2- 172.21.15.44:0/2542923807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b941074e0 0x7f1b941098d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c009ca0 tx=0x7f1b8c02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.500+0000 7f1b9bbe5640 1 -- 172.21.15.44:0/2542923807 shutdown_connections 2024-04-21T23:08:53.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.500+0000 7f1b9bbe5640 1 --2- 172.21.15.44:0/2542923807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94104bb0 0x7f1b94106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.500+0000 7f1b9bbe5640 1 --2- 172.21.15.44:0/2542923807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b941074e0 0x7f1b941098d0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.500+0000 7f1b9bbe5640 1 --2- 172.21.15.44:0/2542923807 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b94102280 0x7f1b94104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.500+0000 7f1b9bbe5640 1 -- 172.21.15.44:0/2542923807 >> 172.21.15.44:0/2542923807 conn(0x7f1b940fbc10 msgr2=0x7f1b940fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:53.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.500+0000 7f1b9bbe5640 1 -- 172.21.15.44:0/2542923807 shutdown_connections 2024-04-21T23:08:53.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.500+0000 7f1b9bbe5640 1 -- 172.21.15.44:0/2542923807 wait complete. 2024-04-21T23:08:53.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.501+0000 7f1b9bbe5640 1 Processor -- start 2024-04-21T23:08:53.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.501+0000 7f1b9bbe5640 1 -- start start 2024-04-21T23:08:53.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.501+0000 7f1b9bbe5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94102280 0x7f1b94197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:53.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.501+0000 7f1b9bbe5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b94104bb0 0x7f1b94197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:53.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.501+0000 7f1b9bbe5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b941074e0 0x7f1b9419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:53.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.501+0000 7f1b9bbe5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1b9410d440 con 0x7f1b941074e0 2024-04-21T23:08:53.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.501+0000 7f1b9bbe5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1b9410d2c0 con 0x7f1b94104bb0 2024-04-21T23:08:53.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.501+0000 7f1b9bbe5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1b9410d5c0 con 0x7f1b94102280 2024-04-21T23:08:53.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.501+0000 7f1b9a182640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b941074e0 0x7f1b9419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:53.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.501+0000 7f1b9a182640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b941074e0 0x7f1b9419f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35632/0 (socket says 172.21.15.44:35632) 2024-04-21T23:08:53.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.501+0000 7f1b9a182640 1 -- 172.21.15.44:0/512903977 learned_addr learned my addr 172.21.15.44:0/512903977 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:53.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.501+0000 7f1b99981640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94102280 0x7f1b94197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:53.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.501+0000 7f1b99180640 1 --2- 172.21.15.44:0/512903977 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b94104bb0 0x7f1b94197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:53.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.502+0000 7f1b9a182640 1 -- 172.21.15.44:0/512903977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94102280 msgr2=0x7f1b94197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:53.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.502+0000 7f1b9a182640 1 --2- 172.21.15.44:0/512903977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94102280 0x7f1b94197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.502+0000 7f1b9a182640 1 -- 172.21.15.44:0/512903977 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b94104bb0 msgr2=0x7f1b94197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:53.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.502+0000 7f1b9a182640 1 --2- 172.21.15.44:0/512903977 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b94104bb0 0x7f1b94197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.502+0000 7f1b9a182640 1 -- 172.21.15.44:0/512903977 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b9419f680 con 0x7f1b941074e0 2024-04-21T23:08:53.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.502+0000 7f1b99981640 1 --2- 172.21.15.44:0/512903977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94102280 0x7f1b94197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:08:53.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.502+0000 7f1b9a182640 1 --2- 172.21.15.44:0/512903977 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b941074e0 0x7f1b9419f010 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c02c630 tx=0x7f1b8c032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:53.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.502+0000 7f1b8affd640 1 -- 172.21.15.44:0/512903977 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c041020 con 0x7f1b941074e0 2024-04-21T23:08:53.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.502+0000 7f1b9bbe5640 1 -- 172.21.15.44:0/512903977 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b9419f910 con 0x7f1b941074e0 2024-04-21T23:08:53.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.502+0000 7f1b8affd640 1 -- 172.21.15.44:0/512903977 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b8c034cd0 con 0x7f1b941074e0 2024-04-21T23:08:53.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.502+0000 7f1b8affd640 1 -- 172.21.15.44:0/512903977 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c039040 con 0x7f1b941074e0 2024-04-21T23:08:53.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.502+0000 7f1b9bbe5640 1 -- 172.21.15.44:0/512903977 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b9419fe00 con 0x7f1b941074e0 2024-04-21T23:08:53.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.503+0000 7f1b9bbe5640 1 -- 172.21.15.44:0/512903977 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b940fd700 con 0x7f1b941074e0 2024-04-21T23:08:53.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.505+0000 7f1b8affd640 1 -- 172.21.15.44:0/512903977 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7f1b8c044020 con 0x7f1b941074e0 2024-04-21T23:08:53.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.505+0000 7f1b8affd640 1 --2- 172.21.15.44:0/512903977 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1b800a2d00 0x7f1b800a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:53.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.506+0000 7f1b99981640 1 --2- 172.21.15.44:0/512903977 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1b800a2d00 0x7f1b800a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:53.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.506+0000 7f1b8affd640 1 -- 172.21.15.44:0/512903977 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f1b8c0f19e0 con 0x7f1b941074e0 2024-04-21T23:08:53.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.506+0000 7f1b99981640 1 --2- 172.21.15.44:0/512903977 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1b800a2d00 0x7f1b800a51c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1b9000bfd0 tx=0x7f1b9000b390 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:53.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.509+0000 7f1b8affd640 1 -- 172.21.15.44:0/512903977 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1b8c04b020 con 0x7f1b941074e0 2024-04-21T23:08:53.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.697+0000 7f1b9bbe5640 1 -- 172.21.15.44:0/512903977 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b9402d320 con 0x7f1b941074e0 2024-04-21T23:08:53.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.700+0000 7f1b8affd640 1 -- 172.21.15.44:0/512903977 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+25092 (secure 0 0 0) 0x7f1b8c0a8020 con 0x7f1b941074e0 2024-04-21T23:08:53.700 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:53.700 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":95,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:08:50.585383+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":9,"to":11},{"from":5,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/3134653118":"2024-04-22T23:08:50.585326+0000","172.21.15.44:0/101322314":"2024-04-22T23:08:50.585326+0000","172.21.15.44:0/792524018":"2024-04-22T23:08:50.585326+0000","172.21.15.44:0/2303525779":"2024-04-22T23:08:50.585326+0000","172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/1417504446":"2024-04-22T23:08:50.585326+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/1417504446":"2024-04-22T23:08:50.585326+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:08:53.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.702+0000 7f1b9bbe5640 1 -- 172.21.15.44:0/512903977 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1b800a2d00 msgr2=0x7f1b800a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:53.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.702+0000 7f1b9bbe5640 1 --2- 172.21.15.44:0/512903977 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1b800a2d00 0x7f1b800a51c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1b9000bfd0 tx=0x7f1b9000b390 comp rx=0 tx=0).stop 2024-04-21T23:08:53.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.702+0000 7f1b9bbe5640 1 -- 172.21.15.44:0/512903977 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b941074e0 msgr2=0x7f1b9419f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:53.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.702+0000 7f1b9bbe5640 1 --2- 172.21.15.44:0/512903977 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b941074e0 0x7f1b9419f010 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c02c630 tx=0x7f1b8c032ea0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.702+0000 7f1b9bbe5640 1 -- 172.21.15.44:0/512903977 shutdown_connections 2024-04-21T23:08:53.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.702+0000 7f1b9bbe5640 1 --2- 172.21.15.44:0/512903977 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1b800a2d00 0x7f1b800a51c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.702+0000 7f1b9bbe5640 1 --2- 172.21.15.44:0/512903977 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b941074e0 0x7f1b9419f010 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.702+0000 7f1b9bbe5640 1 --2- 172.21.15.44:0/512903977 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b94104bb0 0x7f1b94197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.702+0000 7f1b9bbe5640 1 --2- 172.21.15.44:0/512903977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94102280 0x7f1b94197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.702+0000 7f1b9bbe5640 1 -- 172.21.15.44:0/512903977 >> 172.21.15.44:0/512903977 conn(0x7f1b940fbc10 msgr2=0x7f1b94102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:53.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.703+0000 7f1b9bbe5640 1 -- 172.21.15.44:0/512903977 shutdown_connections 2024-04-21T23:08:53.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.703+0000 7f1b9bbe5640 1 -- 172.21.15.44:0/512903977 wait complete. 2024-04-21T23:08:53.714 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:01:47.507503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:06:23.993760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:06:24.954661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:06:28.677618+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:08:53.714 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:08:53.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.820+0000 7efd683f2640 1 -- 172.21.15.44:0/3987638644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efd60101870 msgr2=0x7efd60101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:53.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.820+0000 7efd683f2640 1 --2- 172.21.15.44:0/3987638644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efd60101870 0x7efd60101cf0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7efd50009900 tx=0x7efd5002bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:53.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.821+0000 7efd683f2640 1 -- 172.21.15.44:0/3987638644 shutdown_connections 2024-04-21T23:08:53.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.821+0000 7efd683f2640 1 --2- 172.21.15.44:0/3987638644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd60102230 0x7efd60108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.821+0000 7efd683f2640 1 --2- 172.21.15.44:0/3987638644 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efd60101870 0x7efd60101cf0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.821+0000 7efd683f2640 1 --2- 172.21.15.44:0/3987638644 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efd60100680 0x7efd60100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.821+0000 7efd683f2640 1 -- 172.21.15.44:0/3987638644 >> 172.21.15.44:0/3987638644 conn(0x7efd600fbc10 msgr2=0x7efd600fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:53.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.821+0000 7efd683f2640 1 -- 172.21.15.44:0/3987638644 shutdown_connections 2024-04-21T23:08:53.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.821+0000 7efd683f2640 1 -- 172.21.15.44:0/3987638644 wait complete. 2024-04-21T23:08:53.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.821+0000 7efd683f2640 1 Processor -- start 2024-04-21T23:08:53.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.821+0000 7efd683f2640 1 -- start start 2024-04-21T23:08:53.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.821+0000 7efd683f2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efd60100680 0x7efd600695b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:53.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.821+0000 7efd683f2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efd60101870 0x7efd6019c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:53.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.821+0000 7efd683f2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd60102230 0x7efd6019ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:53.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.822+0000 7efd683f2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7efd6010c2e0 con 0x7efd60101870 2024-04-21T23:08:53.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.822+0000 7efd683f2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7efd6010c160 con 0x7efd60100680 2024-04-21T23:08:53.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.822+0000 7efd683f2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efd6010c460 con 0x7efd60102230 2024-04-21T23:08:53.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.822+0000 7efd6598d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efd60101870 0x7efd6019c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:53.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.822+0000 7efd6598d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efd60101870 0x7efd6019c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35642/0 (socket says 172.21.15.44:35642) 2024-04-21T23:08:53.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.822+0000 7efd6598d640 1 -- 172.21.15.44:0/659922268 learned_addr learned my addr 172.21.15.44:0/659922268 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:53.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.822+0000 7efd6618e640 1 --2- 172.21.15.44:0/659922268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efd60100680 0x7efd600695b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:53.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.822+0000 7efd6698f640 1 --2- 172.21.15.44:0/659922268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd60102230 0x7efd6019ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:53.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.822+0000 7efd6598d640 1 -- 172.21.15.44:0/659922268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd60102230 msgr2=0x7efd6019ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:53.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.822+0000 7efd6598d640 1 --2- 172.21.15.44:0/659922268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd60102230 0x7efd6019ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.822+0000 7efd6598d640 1 -- 172.21.15.44:0/659922268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efd60100680 msgr2=0x7efd600695b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:53.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.822+0000 7efd6598d640 1 --2- 172.21.15.44:0/659922268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efd60100680 0x7efd600695b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:53.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.822+0000 7efd6598d640 1 -- 172.21.15.44:0/659922268 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd6019f780 con 0x7efd60101870 2024-04-21T23:08:53.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.822+0000 7efd6618e640 1 --2- 172.21.15.44:0/659922268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efd60100680 0x7efd600695b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:08:53.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.822+0000 7efd6598d640 1 --2- 172.21.15.44:0/659922268 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efd60101870 0x7efd6019c850 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7efd50009230 tx=0x7efd5002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:53.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.822+0000 7efd577fe640 1 -- 172.21.15.44:0/659922268 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd50041070 con 0x7efd60101870 2024-04-21T23:08:53.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.822+0000 7efd683f2640 1 -- 172.21.15.44:0/659922268 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd6019f9b0 con 0x7efd60101870 2024-04-21T23:08:53.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.822+0000 7efd577fe640 1 -- 172.21.15.44:0/659922268 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd50004450 con 0x7efd60101870 2024-04-21T23:08:53.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.822+0000 7efd683f2640 1 -- 172.21.15.44:0/659922268 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd6019fef0 con 0x7efd60101870 2024-04-21T23:08:53.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.822+0000 7efd577fe640 1 -- 172.21.15.44:0/659922268 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd50003e20 con 0x7efd60101870 2024-04-21T23:08:53.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.824+0000 7efd683f2640 1 -- 172.21.15.44:0/659922268 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd60107a60 con 0x7efd60101870 2024-04-21T23:08:53.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.825+0000 7efd577fe640 1 -- 172.21.15.44:0/659922268 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7efd50044070 con 0x7efd60101870 2024-04-21T23:08:53.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.826+0000 7efd577fe640 1 --2- 172.21.15.44:0/659922268 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7efd4c0a2d00 0x7efd4c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:53.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.826+0000 7efd6618e640 1 --2- 172.21.15.44:0/659922268 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7efd4c0a2d00 0x7efd4c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:53.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.826+0000 7efd577fe640 1 -- 172.21.15.44:0/659922268 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7efd500f1270 con 0x7efd60101870 2024-04-21T23:08:53.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.827+0000 7efd6618e640 1 --2- 172.21.15.44:0/659922268 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7efd4c0a2d00 0x7efd4c0a51c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efd601016d0 tx=0x7efd5c005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:53.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:53.831+0000 7efd577fe640 1 -- 172.21.15.44:0/659922268 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7efd500bd5b0 con 0x7efd60101870 2024-04-21T23:08:54.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.021+0000 7efd683f2640 1 -- 172.21.15.44:0/659922268 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efd6002d370 con 0x7efd60101870 2024-04-21T23:08:54.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.021+0000 7efd577fe640 1 -- 172.21.15.44:0/659922268 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7efd500c18b0 con 0x7efd60101870 2024-04-21T23:08:54.021 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-21T23:08:54.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.024+0000 7efd683f2640 1 -- 172.21.15.44:0/659922268 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7efd4c0a2d00 msgr2=0x7efd4c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:54.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.024+0000 7efd683f2640 1 --2- 172.21.15.44:0/659922268 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7efd4c0a2d00 0x7efd4c0a51c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efd601016d0 tx=0x7efd5c005f90 comp rx=0 tx=0).stop 2024-04-21T23:08:54.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.024+0000 7efd683f2640 1 -- 172.21.15.44:0/659922268 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efd60101870 msgr2=0x7efd6019c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:54.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.024+0000 7efd683f2640 1 --2- 172.21.15.44:0/659922268 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efd60101870 0x7efd6019c850 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7efd50009230 tx=0x7efd5002c650 comp rx=0 tx=0).stop 2024-04-21T23:08:54.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.024+0000 7efd683f2640 1 -- 172.21.15.44:0/659922268 shutdown_connections 2024-04-21T23:08:54.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.024+0000 7efd683f2640 1 --2- 172.21.15.44:0/659922268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd60102230 0x7efd6019ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.024+0000 7efd683f2640 1 --2- 172.21.15.44:0/659922268 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7efd4c0a2d00 0x7efd4c0a51c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.024+0000 7efd683f2640 1 --2- 172.21.15.44:0/659922268 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efd60101870 0x7efd6019c850 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.024+0000 7efd683f2640 1 --2- 172.21.15.44:0/659922268 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efd60100680 0x7efd600695b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.024+0000 7efd683f2640 1 -- 172.21.15.44:0/659922268 >> 172.21.15.44:0/659922268 conn(0x7efd600fbc10 msgr2=0x7efd60104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:54.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.024+0000 7efd683f2640 1 -- 172.21.15.44:0/659922268 shutdown_connections 2024-04-21T23:08:54.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.024+0000 7efd683f2640 1 -- 172.21.15.44:0/659922268 wait complete. 2024-04-21T23:08:54.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:53 smithi044 ceph-mon[27934]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:08:54.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:53 smithi044 ceph-mon[27934]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:08:54.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:53 smithi044 ceph-mon[27934]: from='client.25240 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:08:54.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:53 smithi044 ceph-mon[27934]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-04-21T23:08:54.036 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:53 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/512903977' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:54.037 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:08:54.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.148+0000 7fe47a385640 1 -- 172.21.15.44:0/2734221905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe474101870 msgr2=0x7fe474101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:54.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.148+0000 7fe47a385640 1 --2- 172.21.15.44:0/2734221905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe474101870 0x7fe474101cf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe468009860 tx=0x7fe46802bb70 comp rx=0 tx=0).stop 2024-04-21T23:08:54.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.148+0000 7fe47a385640 1 -- 172.21.15.44:0/2734221905 shutdown_connections 2024-04-21T23:08:54.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.148+0000 7fe47a385640 1 --2- 172.21.15.44:0/2734221905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe474101870 0x7fe474101cf0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.148+0000 7fe47a385640 1 --2- 172.21.15.44:0/2734221905 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe474102230 0x7fe474108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.148+0000 7fe47a385640 1 --2- 172.21.15.44:0/2734221905 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe474100680 0x7fe474100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.148+0000 7fe47a385640 1 -- 172.21.15.44:0/2734221905 >> 172.21.15.44:0/2734221905 conn(0x7fe4740fbc10 msgr2=0x7fe4740fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:54.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.148+0000 7fe47a385640 1 -- 172.21.15.44:0/2734221905 shutdown_connections 2024-04-21T23:08:54.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.148+0000 7fe47a385640 1 -- 172.21.15.44:0/2734221905 wait complete. 2024-04-21T23:08:54.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.149+0000 7fe47a385640 1 Processor -- start 2024-04-21T23:08:54.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.149+0000 7fe47a385640 1 -- start start 2024-04-21T23:08:54.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.149+0000 7fe47a385640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe474100680 0x7fe4741969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:54.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.149+0000 7fe47a385640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe474101870 0x7fe474196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:54.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.149+0000 7fe47a385640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe474102230 0x7fe47419df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:54.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.149+0000 7fe47a385640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe47410c300 con 0x7fe474100680 2024-04-21T23:08:54.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.149+0000 7fe47a385640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe47410c180 con 0x7fe474102230 2024-04-21T23:08:54.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.149+0000 7fe47a385640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe47410c480 con 0x7fe474101870 2024-04-21T23:08:54.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.149+0000 7fe473fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe474100680 0x7fe4741969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:54.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.150+0000 7fe473fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe474100680 0x7fe4741969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35662/0 (socket says 172.21.15.44:35662) 2024-04-21T23:08:54.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.150+0000 7fe473fff640 1 -- 172.21.15.44:0/2557664369 learned_addr learned my addr 172.21.15.44:0/2557664369 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:54.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.150+0000 7fe478922640 1 --2- 172.21.15.44:0/2557664369 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe474102230 0x7fe47419df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:54.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.150+0000 7fe4737fe640 1 --2- 172.21.15.44:0/2557664369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe474101870 0x7fe474196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:54.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.150+0000 7fe473fff640 1 -- 172.21.15.44:0/2557664369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe474101870 msgr2=0x7fe474196ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:54.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.150+0000 7fe473fff640 1 --2- 172.21.15.44:0/2557664369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe474101870 0x7fe474196ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.150+0000 7fe473fff640 1 -- 172.21.15.44:0/2557664369 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe474102230 msgr2=0x7fe47419df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:54.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.150+0000 7fe473fff640 1 --2- 172.21.15.44:0/2557664369 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe474102230 0x7fe47419df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.150+0000 7fe473fff640 1 -- 172.21.15.44:0/2557664369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe47419e5c0 con 0x7fe474100680 2024-04-21T23:08:54.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.150+0000 7fe473fff640 1 --2- 172.21.15.44:0/2557664369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe474100680 0x7fe4741969a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe46400e9f0 tx=0x7fe46400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:54.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.150+0000 7fe4717fa640 1 -- 172.21.15.44:0/2557664369 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46400ccc0 con 0x7fe474100680 2024-04-21T23:08:54.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.150+0000 7fe47a385640 1 -- 172.21.15.44:0/2557664369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe47419e8b0 con 0x7fe474100680 2024-04-21T23:08:54.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.150+0000 7fe4717fa640 1 -- 172.21.15.44:0/2557664369 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4640035f0 con 0x7fe474100680 2024-04-21T23:08:54.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.150+0000 7fe4717fa640 1 -- 172.21.15.44:0/2557664369 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46400ce60 con 0x7fe474100680 2024-04-21T23:08:54.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.150+0000 7fe47a385640 1 -- 172.21.15.44:0/2557664369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe47419ee20 con 0x7fe474100680 2024-04-21T23:08:54.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.152+0000 7fe47a385640 1 -- 172.21.15.44:0/2557664369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe474107a90 con 0x7fe474100680 2024-04-21T23:08:54.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.153+0000 7fe4717fa640 1 -- 172.21.15.44:0/2557664369 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7fe4640037a0 con 0x7fe474100680 2024-04-21T23:08:54.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.153+0000 7fe4717fa640 1 --2- 172.21.15.44:0/2557664369 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe45c0a2d80 0x7fe45c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:54.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.154+0000 7fe4737fe640 1 --2- 172.21.15.44:0/2557664369 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe45c0a2d80 0x7fe45c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:54.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.154+0000 7fe4717fa640 1 -- 172.21.15.44:0/2557664369 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fe4640cf9b0 con 0x7fe474100680 2024-04-21T23:08:54.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.154+0000 7fe4737fe640 1 --2- 172.21.15.44:0/2557664369 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe45c0a2d80 0x7fe45c0a5240 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe468009830 tx=0x7fe468034ab0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:54.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.158+0000 7fe4717fa640 1 -- 172.21.15.44:0/2557664369 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe46409c230 con 0x7fe474100680 2024-04-21T23:08:54.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:53 smithi055 ceph-mon[31117]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:08:54.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:53 smithi055 ceph-mon[31117]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:08:54.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:53 smithi055 ceph-mon[31117]: from='client.25240 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:08:54.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:53 smithi055 ceph-mon[31117]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-04-21T23:08:54.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:53 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/512903977' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:54.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:53 smithi069 ceph-mon[31816]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:08:54.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:53 smithi069 ceph-mon[31816]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:08:54.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:53 smithi069 ceph-mon[31816]: from='client.25240 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:08:54.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:53 smithi069 ceph-mon[31816]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-04-21T23:08:54.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:53 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/512903977' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:54.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.395+0000 7fe47a385640 1 -- 172.21.15.44:0/2557664369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe47402d3e0 con 0x7fe474100680 2024-04-21T23:08:54.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.395+0000 7fe4717fa640 1 -- 172.21.15.44:0/2557664369 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7fe46409bb30 con 0x7fe474100680 2024-04-21T23:08:54.395 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:08:54.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.397+0000 7fe47a385640 1 -- 172.21.15.44:0/2557664369 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe45c0a2d80 msgr2=0x7fe45c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:54.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.397+0000 7fe47a385640 1 --2- 172.21.15.44:0/2557664369 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe45c0a2d80 0x7fe45c0a5240 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe468009830 tx=0x7fe468034ab0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.397+0000 7fe47a385640 1 -- 172.21.15.44:0/2557664369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe474100680 msgr2=0x7fe4741969a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:54.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.397+0000 7fe47a385640 1 --2- 172.21.15.44:0/2557664369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe474100680 0x7fe4741969a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe46400e9f0 tx=0x7fe46400eeb0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.398+0000 7fe47a385640 1 -- 172.21.15.44:0/2557664369 shutdown_connections 2024-04-21T23:08:54.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.398+0000 7fe47a385640 1 --2- 172.21.15.44:0/2557664369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe474101870 0x7fe474196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.398+0000 7fe47a385640 1 --2- 172.21.15.44:0/2557664369 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe45c0a2d80 0x7fe45c0a5240 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.398+0000 7fe47a385640 1 --2- 172.21.15.44:0/2557664369 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe474102230 0x7fe47419df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.398+0000 7fe47a385640 1 --2- 172.21.15.44:0/2557664369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe474100680 0x7fe4741969a0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.398+0000 7fe47a385640 1 -- 172.21.15.44:0/2557664369 >> 172.21.15.44:0/2557664369 conn(0x7fe4740fbc10 msgr2=0x7fe474104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:54.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.398+0000 7fe47a385640 1 -- 172.21.15.44:0/2557664369 shutdown_connections 2024-04-21T23:08:54.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.398+0000 7fe47a385640 1 -- 172.21.15.44:0/2557664369 wait complete. 2024-04-21T23:08:54.410 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:08:54.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.518+0000 7f68b046d640 1 -- 172.21.15.44:0/4178655876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8101350 msgr2=0x7f68a81017d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:54.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.518+0000 7f68b046d640 1 --2- 172.21.15.44:0/4178655876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8101350 0x7f68a81017d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f68a0009ca0 tx=0x7f68a002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.518+0000 7f68b046d640 1 -- 172.21.15.44:0/4178655876 shutdown_connections 2024-04-21T23:08:54.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.518+0000 7f68b046d640 1 --2- 172.21.15.44:0/4178655876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8101350 0x7f68a81017d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.518+0000 7f68b046d640 1 --2- 172.21.15.44:0/4178655876 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68a81009b0 0x7f68a8100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.518+0000 7f68b046d640 1 --2- 172.21.15.44:0/4178655876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68a8100050 0x7f68a8100470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.518+0000 7f68b046d640 1 -- 172.21.15.44:0/4178655876 >> 172.21.15.44:0/4178655876 conn(0x7f68a80fbc10 msgr2=0x7f68a80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:54.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.518+0000 7f68b046d640 1 -- 172.21.15.44:0/4178655876 shutdown_connections 2024-04-21T23:08:54.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.518+0000 7f68b046d640 1 -- 172.21.15.44:0/4178655876 wait complete. 2024-04-21T23:08:54.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.518+0000 7f68b046d640 1 Processor -- start 2024-04-21T23:08:54.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.518+0000 7f68b046d640 1 -- start start 2024-04-21T23:08:54.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.518+0000 7f68b046d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8100050 0x7f68a8199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:54.520 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.518+0000 7f68b046d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68a81009b0 0x7f68a819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:54.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.519+0000 7f68b046d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68a8101350 0x7f68a81a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:54.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.519+0000 7f68b046d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f68a810f5d0 con 0x7f68a8100050 2024-04-21T23:08:54.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.519+0000 7f68b046d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f68a810f450 con 0x7f68a81009b0 2024-04-21T23:08:54.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.519+0000 7f68b046d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f68a810f750 con 0x7f68a8101350 2024-04-21T23:08:54.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.519+0000 7f68ae209640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8100050 0x7f68a8199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:54.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.519+0000 7f68ae209640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8100050 0x7f68a8199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35684/0 (socket says 172.21.15.44:35684) 2024-04-21T23:08:54.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.519+0000 7f68ae209640 1 -- 172.21.15.44:0/414612482 learned_addr learned my addr 172.21.15.44:0/414612482 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:54.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.519+0000 7f68ada08640 1 --2- 172.21.15.44:0/414612482 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68a81009b0 0x7f68a819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:54.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.519+0000 7f68aea0a640 1 --2- 172.21.15.44:0/414612482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68a8101350 0x7f68a81a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:54.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.519+0000 7f68ae209640 1 -- 172.21.15.44:0/414612482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68a8101350 msgr2=0x7f68a81a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:54.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.519+0000 7f68ae209640 1 --2- 172.21.15.44:0/414612482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68a8101350 0x7f68a81a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.519+0000 7f68ae209640 1 -- 172.21.15.44:0/414612482 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68a81009b0 msgr2=0x7f68a819a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:54.521 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.519+0000 7f68ae209640 1 --2- 172.21.15.44:0/414612482 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68a81009b0 0x7f68a819a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.519+0000 7f68ae209640 1 -- 172.21.15.44:0/414612482 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a81a1910 con 0x7f68a8100050 2024-04-21T23:08:54.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.519+0000 7f68ae209640 1 --2- 172.21.15.44:0/414612482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8100050 0x7f68a8199d00 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f68a400ea60 tx=0x7f68a400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:54.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.519+0000 7f689f7fe640 1 -- 172.21.15.44:0/414612482 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a400cde0 con 0x7f68a8100050 2024-04-21T23:08:54.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.519+0000 7f68b046d640 1 -- 172.21.15.44:0/414612482 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68a81a1c00 con 0x7f68a8100050 2024-04-21T23:08:54.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.519+0000 7f689f7fe640 1 -- 172.21.15.44:0/414612482 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68a40035f0 con 0x7f68a8100050 2024-04-21T23:08:54.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.519+0000 7f68b046d640 1 -- 172.21.15.44:0/414612482 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68a81a21a0 con 0x7f68a8100050 2024-04-21T23:08:54.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.519+0000 7f689f7fe640 1 -- 172.21.15.44:0/414612482 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a4010670 con 0x7f68a8100050 2024-04-21T23:08:54.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.521+0000 7f68b046d640 1 -- 172.21.15.44:0/414612482 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68a810ada0 con 0x7f68a8100050 2024-04-21T23:08:54.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.521+0000 7f689f7fe640 1 -- 172.21.15.44:0/414612482 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7f68a40037a0 con 0x7f68a8100050 2024-04-21T23:08:54.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.522+0000 7f689f7fe640 1 --2- 172.21.15.44:0/414612482 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f68940a2d00 0x7f68940a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:54.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.522+0000 7f689f7fe640 1 -- 172.21.15.44:0/414612482 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f68a40cfbb0 con 0x7f68a8100050 2024-04-21T23:08:54.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.522+0000 7f68ada08640 1 --2- 172.21.15.44:0/414612482 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f68940a2d00 0x7f68940a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:54.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.523+0000 7f68ada08640 1 --2- 172.21.15.44:0/414612482 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f68940a2d00 0x7f68940a51c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6898009b50 tx=0x7f6898005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:54.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.527+0000 7f689f7fe640 1 -- 172.21.15.44:0/414612482 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f68a409c430 con 0x7f68a8100050 2024-04-21T23:08:54.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.726+0000 7f68b046d640 1 -- 172.21.15.44:0/414612482 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f68a802d170 con 0x7f68a8100050 2024-04-21T23:08:54.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.726+0000 7f689f7fe640 1 -- 172.21.15.44:0/414612482 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7f68a802d170 con 0x7f68a8100050 2024-04-21T23:08:54.726 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-21T23:08:54.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.728+0000 7f68b046d640 1 -- 172.21.15.44:0/414612482 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f68940a2d00 msgr2=0x7f68940a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:54.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.728+0000 7f68b046d640 1 --2- 172.21.15.44:0/414612482 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f68940a2d00 0x7f68940a51c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6898009b50 tx=0x7f6898005f90 comp rx=0 tx=0).stop 2024-04-21T23:08:54.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.728+0000 7f68b046d640 1 -- 172.21.15.44:0/414612482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8100050 msgr2=0x7f68a8199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:54.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.728+0000 7f68b046d640 1 --2- 172.21.15.44:0/414612482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8100050 0x7f68a8199d00 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f68a400ea60 tx=0x7f68a400ef20 comp rx=0 tx=0).stop 2024-04-21T23:08:54.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.728+0000 7f68b046d640 1 -- 172.21.15.44:0/414612482 shutdown_connections 2024-04-21T23:08:54.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.728+0000 7f68b046d640 1 --2- 172.21.15.44:0/414612482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68a8101350 0x7f68a81a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.728+0000 7f68b046d640 1 --2- 172.21.15.44:0/414612482 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f68940a2d00 0x7f68940a51c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.728+0000 7f68b046d640 1 --2- 172.21.15.44:0/414612482 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f68a81009b0 0x7f68a819a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.728+0000 7f68b046d640 1 --2- 172.21.15.44:0/414612482 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68a8100050 0x7f68a8199d00 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.728+0000 7f68b046d640 1 -- 172.21.15.44:0/414612482 >> 172.21.15.44:0/414612482 conn(0x7f68a80fbc10 msgr2=0x7f68a80fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:54.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.728+0000 7f68b046d640 1 -- 172.21.15.44:0/414612482 shutdown_connections 2024-04-21T23:08:54.729 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.728+0000 7f68b046d640 1 -- 172.21.15.44:0/414612482 wait complete. 2024-04-21T23:08:54.740 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:08:54.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.847+0000 7f0c7f215640 1 -- 172.21.15.44:0/400867155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c78104b30 msgr2=0x7f0c78106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:54.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.847+0000 7f0c7f215640 1 --2- 172.21.15.44:0/400867155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c78104b30 0x7f0c78106f20 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0c64009900 tx=0x7f0c6402bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:54.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.847+0000 7f0c7f215640 1 -- 172.21.15.44:0/400867155 shutdown_connections 2024-04-21T23:08:54.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.847+0000 7f0c7f215640 1 --2- 172.21.15.44:0/400867155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c78107460 0x7f0c781098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.847+0000 7f0c7f215640 1 --2- 172.21.15.44:0/400867155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c78104b30 0x7f0c78106f20 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.847+0000 7f0c7f215640 1 --2- 172.21.15.44:0/400867155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c78102200 0x7f0c781045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.847+0000 7f0c7f215640 1 -- 172.21.15.44:0/400867155 >> 172.21.15.44:0/400867155 conn(0x7f0c780fbc10 msgr2=0x7f0c780fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:54.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.847+0000 7f0c7f215640 1 -- 172.21.15.44:0/400867155 shutdown_connections 2024-04-21T23:08:54.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.847+0000 7f0c7f215640 1 -- 172.21.15.44:0/400867155 wait complete. 2024-04-21T23:08:54.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.848+0000 7f0c7f215640 1 Processor -- start 2024-04-21T23:08:54.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.848+0000 7f0c7f215640 1 -- start start 2024-04-21T23:08:54.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.848+0000 7f0c7f215640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c78102200 0x7f0c78197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:54.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.848+0000 7f0c7f215640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c78104b30 0x7f0c78198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:54.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.848+0000 7f0c7f215640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c78107460 0x7f0c7819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:54.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.848+0000 7f0c7f215640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0c7810d3f0 con 0x7f0c78104b30 2024-04-21T23:08:54.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.848+0000 7f0c7f215640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0c7810d270 con 0x7f0c78102200 2024-04-21T23:08:54.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.848+0000 7f0c7f215640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0c7810d570 con 0x7f0c78107460 2024-04-21T23:08:54.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.848+0000 7f0c77fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c78104b30 0x7f0c78198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:54.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.848+0000 7f0c7d7b2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c78107460 0x7f0c7819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:54.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.848+0000 7f0c7cfb1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c78102200 0x7f0c78197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:54.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.848+0000 7f0c77fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c78104b30 0x7f0c78198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35700/0 (socket says 172.21.15.44:35700) 2024-04-21T23:08:54.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.848+0000 7f0c77fff640 1 -- 172.21.15.44:0/2743516434 learned_addr learned my addr 172.21.15.44:0/2743516434 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:54.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.848+0000 7f0c7cfb1640 1 -- 172.21.15.44:0/2743516434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c78107460 msgr2=0x7f0c7819f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:54.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.848+0000 7f0c7cfb1640 1 --2- 172.21.15.44:0/2743516434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c78107460 0x7f0c7819f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.848+0000 7f0c7cfb1640 1 -- 172.21.15.44:0/2743516434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c78104b30 msgr2=0x7f0c78198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:54.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.848+0000 7f0c7cfb1640 1 --2- 172.21.15.44:0/2743516434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c78104b30 0x7f0c78198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:54.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.848+0000 7f0c7cfb1640 1 -- 172.21.15.44:0/2743516434 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7819f740 con 0x7f0c78102200 2024-04-21T23:08:54.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.849+0000 7f0c7cfb1640 1 --2- 172.21.15.44:0/2743516434 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c78102200 0x7f0c78197b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c00e9f0 tx=0x7f0c6c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:54.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.849+0000 7f0c75ffb640 1 -- 172.21.15.44:0/2743516434 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c00ccc0 con 0x7f0c78102200 2024-04-21T23:08:54.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.849+0000 7f0c7f215640 1 -- 172.21.15.44:0/2743516434 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c7819fa30 con 0x7f0c78102200 2024-04-21T23:08:54.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.849+0000 7f0c75ffb640 1 -- 172.21.15.44:0/2743516434 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c6c010910 con 0x7f0c78102200 2024-04-21T23:08:54.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.849+0000 7f0c75ffb640 1 -- 172.21.15.44:0/2743516434 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c018960 con 0x7f0c78102200 2024-04-21T23:08:54.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.849+0000 7f0c7f215640 1 -- 172.21.15.44:0/2743516434 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c7819ff50 con 0x7f0c78102200 2024-04-21T23:08:54.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.850+0000 7f0c7f215640 1 -- 172.21.15.44:0/2743516434 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c780fd700 con 0x7f0c78102200 2024-04-21T23:08:54.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.851+0000 7f0c75ffb640 1 -- 172.21.15.44:0/2743516434 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7f0c6c010ac0 con 0x7f0c78102200 2024-04-21T23:08:54.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.852+0000 7f0c75ffb640 1 --2- 172.21.15.44:0/2743516434 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f0c680a2d00 0x7f0c680a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:54.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.852+0000 7f0c77fff640 1 --2- 172.21.15.44:0/2743516434 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f0c680a2d00 0x7f0c680a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:54.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.852+0000 7f0c75ffb640 1 -- 172.21.15.44:0/2743516434 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f0c6c0cfee0 con 0x7f0c78102200 2024-04-21T23:08:54.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.852+0000 7f0c77fff640 1 --2- 172.21.15.44:0/2743516434 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f0c680a2d00 0x7f0c680a51c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0c6402c8e0 tx=0x7f0c64005b50 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:54.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:54.855+0000 7f0c75ffb640 1 -- 172.21.15.44:0/2743516434 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f0c6c09c0b0 con 0x7f0c78102200 2024-04-21T23:08:55.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.046+0000 7f0c7f215640 1 -- 172.21.15.44:0/2743516434 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0c78062340 con 0x7f0c78102200 2024-04-21T23:08:55.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.046+0000 7f0c75ffb640 1 -- 172.21.15.44:0/2743516434 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f0c6c0a03b0 con 0x7f0c78102200 2024-04-21T23:08:55.046 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:08:55.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.049+0000 7f0c7f215640 1 -- 172.21.15.44:0/2743516434 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f0c680a2d00 msgr2=0x7f0c680a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.049+0000 7f0c7f215640 1 --2- 172.21.15.44:0/2743516434 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f0c680a2d00 0x7f0c680a51c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0c6402c8e0 tx=0x7f0c64005b50 comp rx=0 tx=0).stop 2024-04-21T23:08:55.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.049+0000 7f0c7f215640 1 -- 172.21.15.44:0/2743516434 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c78102200 msgr2=0x7f0c78197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.049+0000 7f0c7f215640 1 --2- 172.21.15.44:0/2743516434 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c78102200 0x7f0c78197b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c00e9f0 tx=0x7f0c6c00eeb0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.049+0000 7f0c7f215640 1 -- 172.21.15.44:0/2743516434 shutdown_connections 2024-04-21T23:08:55.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.049+0000 7f0c7f215640 1 --2- 172.21.15.44:0/2743516434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c78107460 0x7f0c7819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.049+0000 7f0c7f215640 1 --2- 172.21.15.44:0/2743516434 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f0c680a2d00 0x7f0c680a51c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.049+0000 7f0c7f215640 1 --2- 172.21.15.44:0/2743516434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0c78104b30 0x7f0c78198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.049+0000 7f0c7f215640 1 --2- 172.21.15.44:0/2743516434 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0c78102200 0x7f0c78197b20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.049+0000 7f0c7f215640 1 -- 172.21.15.44:0/2743516434 >> 172.21.15.44:0/2743516434 conn(0x7f0c780fbc10 msgr2=0x7f0c78102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:55.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.049+0000 7f0c7f215640 1 -- 172.21.15.44:0/2743516434 shutdown_connections 2024-04-21T23:08:55.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.049+0000 7f0c7f215640 1 -- 172.21.15.44:0/2743516434 wait complete. 2024-04-21T23:08:55.060 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:08:55.060 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:55.061 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:55.061 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.061 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.061 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.061 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.061 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: Adjusting osd_memory_target on smithi069 to 176.1M 2024-04-21T23:08:55.061 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: Unable to set osd_memory_target on smithi069 to 184727244: error parsing value: Value '184727244' is below minimum 939524096 2024-04-21T23:08:55.061 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/659922268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:55.061 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:55.061 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:55.061 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.061 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.061 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.061 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.062 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: Adjusting osd_memory_target on smithi055 to 177.2M 2024-04-21T23:08:55.062 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: Unable to set osd_memory_target on smithi055 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-21T23:08:55.062 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:55.062 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:55.062 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.062 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.062 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.062 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.062 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: Adjusting osd_memory_target on smithi044 to 177.2M 2024-04-21T23:08:55.062 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: Unable to set osd_memory_target on smithi044 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-21T23:08:55.062 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:08:55.062 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:08:55.062 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: Updating smithi044:/etc/ceph/ceph.conf 2024-04-21T23:08:55.062 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: Updating smithi055:/etc/ceph/ceph.conf 2024-04-21T23:08:55.063 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-21T23:08:55.063 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2557664369' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:55.063 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:54 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/414612482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:55.064 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:08:55.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.185+0000 7fe7d53b8640 1 -- 172.21.15.44:0/1616119366 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7d0107800 msgr2=0x7fe7d0107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.185+0000 7fe7d53b8640 1 --2- 172.21.15.44:0/1616119366 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7d0107800 0x7fe7d0107be0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0009ca0 tx=0x7fe7c002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.185+0000 7fe7d53b8640 1 -- 172.21.15.44:0/1616119366 shutdown_connections 2024-04-21T23:08:55.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.185+0000 7fe7d53b8640 1 --2- 172.21.15.44:0/1616119366 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7d0068a40 0x7fe7d0068ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.185+0000 7fe7d53b8640 1 --2- 172.21.15.44:0/1616119366 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7d00693e0 0x7fe7d010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.185+0000 7fe7d53b8640 1 --2- 172.21.15.44:0/1616119366 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7d0107800 0x7fe7d0107be0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.185+0000 7fe7d53b8640 1 -- 172.21.15.44:0/1616119366 >> 172.21.15.44:0/1616119366 conn(0x7fe7d0077430 msgr2=0x7fe7d0077840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:55.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.185+0000 7fe7d53b8640 1 -- 172.21.15.44:0/1616119366 shutdown_connections 2024-04-21T23:08:55.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.185+0000 7fe7d53b8640 1 -- 172.21.15.44:0/1616119366 wait complete. 2024-04-21T23:08:55.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.185+0000 7fe7d53b8640 1 Processor -- start 2024-04-21T23:08:55.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.185+0000 7fe7d53b8640 1 -- start start 2024-04-21T23:08:55.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.186+0000 7fe7d53b8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7d0068a40 0x7fe7d019a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.186+0000 7fe7d53b8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7d00693e0 0x7fe7d019a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.186+0000 7fe7d53b8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7d0107800 0x7fe7d019e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.186+0000 7fe7d53b8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d0111b20 con 0x7fe7d00693e0 2024-04-21T23:08:55.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.186+0000 7fe7d53b8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d01119a0 con 0x7fe7d0068a40 2024-04-21T23:08:55.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.186+0000 7fe7d53b8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d0111ca0 con 0x7fe7d0107800 2024-04-21T23:08:55.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.186+0000 7fe7ce7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7d00693e0 0x7fe7d019a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.186+0000 7fe7ce7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7d00693e0 0x7fe7d019a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35718/0 (socket says 172.21.15.44:35718) 2024-04-21T23:08:55.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.186+0000 7fe7ce7fc640 1 -- 172.21.15.44:0/3589828188 learned_addr learned my addr 172.21.15.44:0/3589828188 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:55.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.186+0000 7fe7ceffd640 1 --2- 172.21.15.44:0/3589828188 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7d0068a40 0x7fe7d019a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.186+0000 7fe7cf7fe640 1 --2- 172.21.15.44:0/3589828188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7d0107800 0x7fe7d019e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.186+0000 7fe7ce7fc640 1 -- 172.21.15.44:0/3589828188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7d0107800 msgr2=0x7fe7d019e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.186+0000 7fe7ce7fc640 1 --2- 172.21.15.44:0/3589828188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7d0107800 0x7fe7d019e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.186+0000 7fe7ce7fc640 1 -- 172.21.15.44:0/3589828188 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7d0068a40 msgr2=0x7fe7d019a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.186+0000 7fe7ce7fc640 1 --2- 172.21.15.44:0/3589828188 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7d0068a40 0x7fe7d019a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.186+0000 7fe7ce7fc640 1 -- 172.21.15.44:0/3589828188 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d019f0d0 con 0x7fe7d00693e0 2024-04-21T23:08:55.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.186+0000 7fe7ce7fc640 1 --2- 172.21.15.44:0/3589828188 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7d00693e0 0x7fe7d019a620 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe7c400c970 tx=0x7fe7c400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:55.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.187+0000 7fe7b7fff640 1 -- 172.21.15.44:0/3589828188 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4007bb0 con 0x7fe7d00693e0 2024-04-21T23:08:55.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.187+0000 7fe7d53b8640 1 -- 172.21.15.44:0/3589828188 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7d019f3c0 con 0x7fe7d00693e0 2024-04-21T23:08:55.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.187+0000 7fe7b7fff640 1 -- 172.21.15.44:0/3589828188 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7c4007d50 con 0x7fe7d00693e0 2024-04-21T23:08:55.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.187+0000 7fe7b7fff640 1 -- 172.21.15.44:0/3589828188 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c400f450 con 0x7fe7d00693e0 2024-04-21T23:08:55.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.187+0000 7fe7d53b8640 1 -- 172.21.15.44:0/3589828188 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7d01a6c50 con 0x7fe7d00693e0 2024-04-21T23:08:55.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.188+0000 7fe7d53b8640 1 -- 172.21.15.44:0/3589828188 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7d00685f0 con 0x7fe7d00693e0 2024-04-21T23:08:55.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.189+0000 7fe7b7fff640 1 -- 172.21.15.44:0/3589828188 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7fe7c4005020 con 0x7fe7d00693e0 2024-04-21T23:08:55.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.190+0000 7fe7b7fff640 1 --2- 172.21.15.44:0/3589828188 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe7b80a2d90 0x7fe7b80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.190+0000 7fe7ceffd640 1 --2- 172.21.15.44:0/3589828188 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe7b80a2d90 0x7fe7b80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.190+0000 7fe7b7fff640 1 -- 172.21.15.44:0/3589828188 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fe7c40cf7e0 con 0x7fe7d00693e0 2024-04-21T23:08:55.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.190+0000 7fe7ceffd640 1 --2- 172.21.15.44:0/3589828188 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe7b80a2d90 0x7fe7b80a5250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0009430 tx=0x7fe7c0005c50 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:55.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.195+0000 7fe7b7fff640 1 -- 172.21.15.44:0/3589828188 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe7c4014030 con 0x7fe7d00693e0 2024-04-21T23:08:55.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.424+0000 7fe7d53b8640 1 -- 172.21.15.44:0/3589828188 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7d01a6f10 con 0x7fe7d00693e0 2024-04-21T23:08:55.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.425+0000 7fe7b7fff640 1 -- 172.21.15.44:0/3589828188 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11791 (secure 0 0 0) 0x7fe7c409be90 con 0x7fe7d00693e0 2024-04-21T23:08:55.425 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:55.425 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:06:50.685663+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3743898853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3743898853},{"type":"v1","addr":"172.21.15.69:6839","nonce":3743898853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:08:55.425 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2024-04-21T23:08:55.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.427+0000 7fe7d53b8640 1 -- 172.21.15.44:0/3589828188 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe7b80a2d90 msgr2=0x7fe7b80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.427+0000 7fe7d53b8640 1 --2- 172.21.15.44:0/3589828188 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe7b80a2d90 0x7fe7b80a5250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0009430 tx=0x7fe7c0005c50 comp rx=0 tx=0).stop 2024-04-21T23:08:55.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.427+0000 7fe7d53b8640 1 -- 172.21.15.44:0/3589828188 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7d00693e0 msgr2=0x7fe7d019a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.427+0000 7fe7d53b8640 1 --2- 172.21.15.44:0/3589828188 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7d00693e0 0x7fe7d019a620 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe7c400c970 tx=0x7fe7c400ce30 comp rx=0 tx=0).stop 2024-04-21T23:08:55.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.427+0000 7fe7d53b8640 1 -- 172.21.15.44:0/3589828188 shutdown_connections 2024-04-21T23:08:55.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.427+0000 7fe7d53b8640 1 --2- 172.21.15.44:0/3589828188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7d0107800 0x7fe7d019e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.427+0000 7fe7d53b8640 1 --2- 172.21.15.44:0/3589828188 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe7b80a2d90 0x7fe7b80a5250 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.427+0000 7fe7d53b8640 1 --2- 172.21.15.44:0/3589828188 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe7d00693e0 0x7fe7d019a620 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.427+0000 7fe7d53b8640 1 --2- 172.21.15.44:0/3589828188 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7d0068a40 0x7fe7d019a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.427+0000 7fe7d53b8640 1 -- 172.21.15.44:0/3589828188 >> 172.21.15.44:0/3589828188 conn(0x7fe7d0077430 msgr2=0x7fe7d00ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:55.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.427+0000 7fe7d53b8640 1 -- 172.21.15.44:0/3589828188 shutdown_connections 2024-04-21T23:08:55.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.427+0000 7fe7d53b8640 1 -- 172.21.15.44:0/3589828188 wait complete. 2024-04-21T23:08:55.438 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:08:55.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:08:55.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:55.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:55.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: Adjusting osd_memory_target on smithi069 to 176.1M 2024-04-21T23:08:55.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: Unable to set osd_memory_target on smithi069 to 184727244: error parsing value: Value '184727244' is below minimum 939524096 2024-04-21T23:08:55.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/659922268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:55.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:55.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:55.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: Adjusting osd_memory_target on smithi055 to 177.2M 2024-04-21T23:08:55.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: Unable to set osd_memory_target on smithi055 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-21T23:08:55.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:55.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:55.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: Adjusting osd_memory_target on smithi044 to 177.2M 2024-04-21T23:08:55.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: Unable to set osd_memory_target on smithi044 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-21T23:08:55.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:08:55.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:08:55.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: Updating smithi044:/etc/ceph/ceph.conf 2024-04-21T23:08:55.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: Updating smithi055:/etc/ceph/ceph.conf 2024-04-21T23:08:55.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-21T23:08:55.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2557664369' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:55.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:54 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/414612482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:55.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T23:08:55.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:55.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:55.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: Adjusting osd_memory_target on smithi069 to 176.1M 2024-04-21T23:08:55.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: Unable to set osd_memory_target on smithi069 to 184727244: error parsing value: Value '184727244' is below minimum 939524096 2024-04-21T23:08:55.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/659922268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:08:55.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:55.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:55.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: Adjusting osd_memory_target on smithi055 to 177.2M 2024-04-21T23:08:55.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: Unable to set osd_memory_target on smithi055 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-21T23:08:55.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:55.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:55.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:08:55.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: Adjusting osd_memory_target on smithi044 to 177.2M 2024-04-21T23:08:55.469 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: Unable to set osd_memory_target on smithi044 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-21T23:08:55.469 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:08:55.469 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:08:55.469 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: Updating smithi044:/etc/ceph/ceph.conf 2024-04-21T23:08:55.469 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: Updating smithi055:/etc/ceph/ceph.conf 2024-04-21T23:08:55.469 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-21T23:08:55.469 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2557664369' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:08:55.469 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:54 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/414612482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:08:55.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.541+0000 7f200bbf2640 1 -- 172.21.15.44:0/1063396263 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f200410bd80 msgr2=0x7f200410e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.541+0000 7f200bbf2640 1 --2- 172.21.15.44:0/1063396263 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f200410bd80 0x7f200410e1b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009900 tx=0x7f1ffc02bcd0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.541+0000 7f200bbf2640 1 -- 172.21.15.44:0/1063396263 shutdown_connections 2024-04-21T23:08:55.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.541+0000 7f200bbf2640 1 --2- 172.21.15.44:0/1063396263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20041032d0 0x7f200410b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.541+0000 7f200bbf2640 1 --2- 172.21.15.44:0/1063396263 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f200410bd80 0x7f200410e1b0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.541+0000 7f200bbf2640 1 --2- 172.21.15.44:0/1063396263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20041029b0 0x7f2004102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.541+0000 7f200bbf2640 1 -- 172.21.15.44:0/1063396263 >> 172.21.15.44:0/1063396263 conn(0x7f2004077740 msgr2=0x7f2004077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:55.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.541+0000 7f200bbf2640 1 -- 172.21.15.44:0/1063396263 shutdown_connections 2024-04-21T23:08:55.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.541+0000 7f200bbf2640 1 -- 172.21.15.44:0/1063396263 wait complete. 2024-04-21T23:08:55.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.541+0000 7f200bbf2640 1 Processor -- start 2024-04-21T23:08:55.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.542+0000 7f200bbf2640 1 -- start start 2024-04-21T23:08:55.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.542+0000 7f200bbf2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20041029b0 0x7f2004197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.542+0000 7f200bbf2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20041032d0 0x7f2004198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.542+0000 7f200bbf2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f200410bd80 0x7f200419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.542+0000 7f200bbf2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2004111b10 con 0x7f200410bd80 2024-04-21T23:08:55.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.542+0000 7f200bbf2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2004111990 con 0x7f20041032d0 2024-04-21T23:08:55.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.542+0000 7f200bbf2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2004111c90 con 0x7f20041029b0 2024-04-21T23:08:55.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.542+0000 7f200a18f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f200410bd80 0x7f200419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.542+0000 7f200a18f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f200410bd80 0x7f200419c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35732/0 (socket says 172.21.15.44:35732) 2024-04-21T23:08:55.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.542+0000 7f200a18f640 1 -- 172.21.15.44:0/2940025522 learned_addr learned my addr 172.21.15.44:0/2940025522 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:55.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.542+0000 7f200998e640 1 --2- 172.21.15.44:0/2940025522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20041029b0 0x7f2004197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.542+0000 7f200918d640 1 --2- 172.21.15.44:0/2940025522 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20041032d0 0x7f2004198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.542+0000 7f200a18f640 1 -- 172.21.15.44:0/2940025522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20041029b0 msgr2=0x7f2004197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.542+0000 7f200a18f640 1 --2- 172.21.15.44:0/2940025522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20041029b0 0x7f2004197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.542+0000 7f200a18f640 1 -- 172.21.15.44:0/2940025522 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20041032d0 msgr2=0x7f2004198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.542+0000 7f200a18f640 1 --2- 172.21.15.44:0/2940025522 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20041032d0 0x7f2004198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.542+0000 7f200a18f640 1 -- 172.21.15.44:0/2940025522 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200419cf90 con 0x7f200410bd80 2024-04-21T23:08:55.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.542+0000 7f200a18f640 1 --2- 172.21.15.44:0/2940025522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f200410bd80 0x7f200419c860 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc005940 tx=0x7f1ffc02cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:55.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.543+0000 7f1ffaffd640 1 -- 172.21.15.44:0/2940025522 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc041070 con 0x7f200410bd80 2024-04-21T23:08:55.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.543+0000 7f1ffaffd640 1 -- 172.21.15.44:0/2940025522 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ffc004450 con 0x7f200410bd80 2024-04-21T23:08:55.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.543+0000 7f200bbf2640 1 -- 172.21.15.44:0/2940025522 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f200419d220 con 0x7f200410bd80 2024-04-21T23:08:55.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.543+0000 7f1ffaffd640 1 -- 172.21.15.44:0/2940025522 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc003dc0 con 0x7f200410bd80 2024-04-21T23:08:55.546 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.543+0000 7f200bbf2640 1 -- 172.21.15.44:0/2940025522 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20041a4b10 con 0x7f200410bd80 2024-04-21T23:08:55.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.544+0000 7f200bbf2640 1 -- 172.21.15.44:0/2940025522 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2004068a40 con 0x7f200410bd80 2024-04-21T23:08:55.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.545+0000 7f1ffaffd640 1 -- 172.21.15.44:0/2940025522 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f1ffc044070 con 0x7f200410bd80 2024-04-21T23:08:55.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.545+0000 7f1ffaffd640 1 --2- 172.21.15.44:0/2940025522 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1ff00a2d90 0x7f1ff00a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.546+0000 7f200998e640 1 --2- 172.21.15.44:0/2940025522 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1ff00a2d90 0x7f1ff00a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.546+0000 7f1ffaffd640 1 -- 172.21.15.44:0/2940025522 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f1ffc0f1370 con 0x7f200410bd80 2024-04-21T23:08:55.547 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.546+0000 7f200998e640 1 --2- 172.21.15.44:0/2940025522 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1ff00a2d90 0x7f1ff00a5250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2000005fd0 tx=0x7f2000009760 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:55.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.550+0000 7f1ffaffd640 1 -- 172.21.15.44:0/2940025522 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1ffc0bd4e0 con 0x7f200410bd80 2024-04-21T23:08:55.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.735+0000 7f200bbf2640 1 -- 172.21.15.44:0/2940025522 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2004106e70 con 0x7f200410bd80 2024-04-21T23:08:55.736 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.736+0000 7f1ffaffd640 1 -- 172.21.15.44:0/2940025522 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+25092 (secure 0 0 0) 0x7f1ffc0c17e0 con 0x7f200410bd80 2024-04-21T23:08:55.737 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:08:55.737 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":95,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:08:50.585383+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":9,"to":11},{"from":5,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/3134653118":"2024-04-22T23:08:50.585326+0000","172.21.15.44:0/101322314":"2024-04-22T23:08:50.585326+0000","172.21.15.44:0/792524018":"2024-04-22T23:08:50.585326+0000","172.21.15.44:0/2303525779":"2024-04-22T23:08:50.585326+0000","172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/1417504446":"2024-04-22T23:08:50.585326+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/1417504446":"2024-04-22T23:08:50.585326+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:08:55.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.738+0000 7f200bbf2640 1 -- 172.21.15.44:0/2940025522 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1ff00a2d90 msgr2=0x7f1ff00a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.738+0000 7f200bbf2640 1 --2- 172.21.15.44:0/2940025522 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1ff00a2d90 0x7f1ff00a5250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2000005fd0 tx=0x7f2000009760 comp rx=0 tx=0).stop 2024-04-21T23:08:55.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.738+0000 7f200bbf2640 1 -- 172.21.15.44:0/2940025522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f200410bd80 msgr2=0x7f200419c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.738+0000 7f200bbf2640 1 --2- 172.21.15.44:0/2940025522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f200410bd80 0x7f200419c860 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc005940 tx=0x7f1ffc02cc40 comp rx=0 tx=0).stop 2024-04-21T23:08:55.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.738+0000 7f200bbf2640 1 -- 172.21.15.44:0/2940025522 shutdown_connections 2024-04-21T23:08:55.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.738+0000 7f200bbf2640 1 --2- 172.21.15.44:0/2940025522 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1ff00a2d90 0x7f1ff00a5250 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.738+0000 7f200bbf2640 1 --2- 172.21.15.44:0/2940025522 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f200410bd80 0x7f200419c860 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.738+0000 7f200bbf2640 1 --2- 172.21.15.44:0/2940025522 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20041032d0 0x7f2004198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.738+0000 7f200bbf2640 1 --2- 172.21.15.44:0/2940025522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20041029b0 0x7f2004197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.738+0000 7f200bbf2640 1 -- 172.21.15.44:0/2940025522 >> 172.21.15.44:0/2940025522 conn(0x7f2004077740 msgr2=0x7f20040fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:55.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.738+0000 7f200bbf2640 1 -- 172.21.15.44:0/2940025522 shutdown_connections 2024-04-21T23:08:55.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.738+0000 7f200bbf2640 1 -- 172.21.15.44:0/2940025522 wait complete. 2024-04-21T23:08:55.750 DEBUG:tasks.fwd_scrub:Yielding 2024-04-21T23:08:55.750 INFO:teuthology.run_tasks:Running task exec... 2024-04-21T23:08:55.760 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T23:08:55.760 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-04-21T23:08:55.760 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-21T23:08:55.761 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-21T23:08:55.761 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:08:55.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:08:55.761 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:08:55.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.868+0000 7fedd3c20640 1 -- 172.21.15.44:0/4144773961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedcc10bd80 msgr2=0x7fedcc10e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.868+0000 7fedd3c20640 1 --2- 172.21.15.44:0/4144773961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedcc10bd80 0x7fedcc10e1b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fedc4009ca0 tx=0x7fedc402c180 comp rx=0 tx=0).stop 2024-04-21T23:08:55.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.868+0000 7fedd3c20640 1 -- 172.21.15.44:0/4144773961 shutdown_connections 2024-04-21T23:08:55.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.868+0000 7fedd3c20640 1 --2- 172.21.15.44:0/4144773961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedcc1032d0 0x7fedcc10b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.868+0000 7fedd3c20640 1 --2- 172.21.15.44:0/4144773961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedcc10bd80 0x7fedcc10e1b0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.868+0000 7fedd3c20640 1 --2- 172.21.15.44:0/4144773961 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedcc1029b0 0x7fedcc102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.868+0000 7fedd3c20640 1 -- 172.21.15.44:0/4144773961 >> 172.21.15.44:0/4144773961 conn(0x7fedcc077760 msgr2=0x7fedcc077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:55.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.868+0000 7fedd3c20640 1 -- 172.21.15.44:0/4144773961 shutdown_connections 2024-04-21T23:08:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.868+0000 7f016c2b5640 1 -- 172.21.15.44:0/2331001210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01641066b0 msgr2=0x7f0164106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.868+0000 7f016c2b5640 1 --2- 172.21.15.44:0/2331001210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01641066b0 0x7f0164106a90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0160009930 tx=0x7f016002bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:55.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.868+0000 7fedd3c20640 1 -- 172.21.15.44:0/4144773961 wait complete. 2024-04-21T23:08:55.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.868+0000 7f016c2b5640 1 -- 172.21.15.44:0/2331001210 shutdown_connections 2024-04-21T23:08:55.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.868+0000 7f016c2b5640 1 --2- 172.21.15.44:0/2331001210 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0164100fd0 0x7f016410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.868+0000 7f016c2b5640 1 --2- 172.21.15.44:0/2331001210 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0164100610 0x7f0164100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.868+0000 7f016c2b5640 1 --2- 172.21.15.44:0/2331001210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01641066b0 0x7f0164106a90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.868+0000 7f016c2b5640 1 -- 172.21.15.44:0/2331001210 >> 172.21.15.44:0/2331001210 conn(0x7f01640777f0 msgr2=0x7f0164077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:55.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.868+0000 7f016c2b5640 1 -- 172.21.15.44:0/2331001210 shutdown_connections 2024-04-21T23:08:55.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.868+0000 7f016c2b5640 1 -- 172.21.15.44:0/2331001210 wait complete. 2024-04-21T23:08:55.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.868+0000 7fedd3c20640 1 Processor -- start 2024-04-21T23:08:55.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.868+0000 7fedd3c20640 1 -- start start 2024-04-21T23:08:55.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7f016c2b5640 1 Processor -- start 2024-04-21T23:08:55.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7fedd3c20640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedcc1029b0 0x7fedcc198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7fedd3c20640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedcc1032d0 0x7fedcc198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7fedd3c20640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedcc10bd80 0x7fedcc19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7f016c2b5640 1 -- start start 2024-04-21T23:08:55.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7fedd3c20640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fedcc111b10 con 0x7fedcc10bd80 2024-04-21T23:08:55.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7fedd3c20640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fedcc111990 con 0x7fedcc1029b0 2024-04-21T23:08:55.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7fedd3c20640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fedcc111c90 con 0x7fedcc1032d0 2024-04-21T23:08:55.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7fedd21bd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedcc10bd80 0x7fedcc19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7fedd11bb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedcc1032d0 0x7fedcc198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7fedd21bd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedcc10bd80 0x7fedcc19c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35750/0 (socket says 172.21.15.44:35750) 2024-04-21T23:08:55.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7fedd21bd640 1 -- 172.21.15.44:0/391156657 learned_addr learned my addr 172.21.15.44:0/391156657 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:55.874 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7fedd19bc640 1 --2- 172.21.15.44:0/391156657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedcc1029b0 0x7fedcc198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7f016c2b5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0164100610 0x7f0164196cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7f016c2b5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0164100fd0 0x7f0164197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7f016c2b5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f01641066b0 0x7f0164197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7f016c2b5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0164110a30 con 0x7f0164100610 2024-04-21T23:08:55.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7f016c2b5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f01641108b0 con 0x7f01641066b0 2024-04-21T23:08:55.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7f016c2b5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0164110bb0 con 0x7f0164100fd0 2024-04-21T23:08:55.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7f016a051640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0164100610 0x7f0164196cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7f016a051640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0164100610 0x7f0164196cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35758/0 (socket says 172.21.15.44:35758) 2024-04-21T23:08:55.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7f016a051640 1 -- 172.21.15.44:0/2163383124 learned_addr learned my addr 172.21.15.44:0/2163383124 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:55.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7fedd21bd640 1 -- 172.21.15.44:0/391156657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedcc1032d0 msgr2=0x7fedcc198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7fedd21bd640 1 --2- 172.21.15.44:0/391156657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedcc1032d0 0x7fedcc198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7f0169850640 1 --2- 172.21.15.44:0/2163383124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0164100fd0 0x7f0164197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7f016a852640 1 --2- 172.21.15.44:0/2163383124 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f01641066b0 0x7f0164197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7fedd21bd640 1 -- 172.21.15.44:0/391156657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedcc1029b0 msgr2=0x7fedcc198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7fedd21bd640 1 --2- 172.21.15.44:0/391156657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedcc1029b0 0x7fedcc198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7fedd21bd640 1 -- 172.21.15.44:0/391156657 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedcc19cfb0 con 0x7fedcc10bd80 2024-04-21T23:08:55.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7fedd11bb640 1 --2- 172.21.15.44:0/391156657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedcc1032d0 0x7fedcc198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:08:55.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7fedd21bd640 1 --2- 172.21.15.44:0/391156657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedcc10bd80 0x7fedcc19c880 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fedc4004af0 tx=0x7fedc4035d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:55.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7f016a051640 1 -- 172.21.15.44:0/2163383124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0164100fd0 msgr2=0x7f0164197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7f016a051640 1 --2- 172.21.15.44:0/2163383124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0164100fd0 0x7f0164197200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7f016a051640 1 -- 172.21.15.44:0/2163383124 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f01641066b0 msgr2=0x7f0164197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.869+0000 7f016a051640 1 --2- 172.21.15.44:0/2163383124 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f01641066b0 0x7f0164197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.870+0000 7f016a051640 1 -- 172.21.15.44:0/2163383124 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01641a6530 con 0x7f0164100610 2024-04-21T23:08:55.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.870+0000 7f016a852640 1 --2- 172.21.15.44:0/2163383124 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f01641066b0 0x7f0164197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:08:55.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.870+0000 7fedc2ffd640 1 -- 172.21.15.44:0/391156657 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc4039040 con 0x7fedcc10bd80 2024-04-21T23:08:55.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.870+0000 7fedc2ffd640 1 -- 172.21.15.44:0/391156657 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc4044070 con 0x7fedcc10bd80 2024-04-21T23:08:55.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.870+0000 7fedd3c20640 1 -- 172.21.15.44:0/391156657 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedcc19d240 con 0x7fedcc10bd80 2024-04-21T23:08:55.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.870+0000 7fedc2ffd640 1 -- 172.21.15.44:0/391156657 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc40328d0 con 0x7fedcc10bd80 2024-04-21T23:08:55.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.870+0000 7fedd3c20640 1 -- 172.21.15.44:0/391156657 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedcc1a4b30 con 0x7fedcc10bd80 2024-04-21T23:08:55.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.870+0000 7f016a051640 1 --2- 172.21.15.44:0/2163383124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0164100610 0x7f0164196cc0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0160009230 tx=0x7f016002cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:55.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.870+0000 7f015b7fe640 1 -- 172.21.15.44:0/2163383124 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160041070 con 0x7f0164100610 2024-04-21T23:08:55.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.870+0000 7f016c2b5640 1 -- 172.21.15.44:0/2163383124 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01641a67c0 con 0x7f0164100610 2024-04-21T23:08:55.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.870+0000 7f015b7fe640 1 -- 172.21.15.44:0/2163383124 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0160004450 con 0x7f0164100610 2024-04-21T23:08:55.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.870+0000 7f015b7fe640 1 -- 172.21.15.44:0/2163383124 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160003d90 con 0x7f0164100610 2024-04-21T23:08:55.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.871+0000 7f016c2b5640 1 -- 172.21.15.44:0/2163383124 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01641a6d00 con 0x7f0164100610 2024-04-21T23:08:55.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.871+0000 7fedd3c20640 1 -- 172.21.15.44:0/391156657 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedcc04fd50 con 0x7fedcc10bd80 2024-04-21T23:08:55.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.872+0000 7fedc2ffd640 1 -- 172.21.15.44:0/391156657 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7fedc4003b70 con 0x7fedcc10bd80 2024-04-21T23:08:55.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.873+0000 7fedc2ffd640 1 --2- 172.21.15.44:0/391156657 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fedb80a2d80 0x7fedb80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.873+0000 7f015b7fe640 1 -- 172.21.15.44:0/2163383124 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f0160044070 con 0x7f0164100610 2024-04-21T23:08:55.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.873+0000 7fedd19bc640 1 --2- 172.21.15.44:0/391156657 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fedb80a2d80 0x7fedb80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.873+0000 7fedc2ffd640 1 -- 172.21.15.44:0/391156657 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fedc40f1c20 con 0x7fedcc10bd80 2024-04-21T23:08:55.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.873+0000 7f016ef7f740 1 Processor -- start 2024-04-21T23:08:55.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.873+0000 7fedd19bc640 1 --2- 172.21.15.44:0/391156657 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fedb80a2d80 0x7fedb80a5240 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fedc8005d00 tx=0x7fedc800b410 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:55.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.874+0000 7f016ef7f740 1 -- start start 2024-04-21T23:08:55.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.874+0000 7f016ef7f740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1d98720 0x55ddf1d97470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.874+0000 7f015b7fe640 1 --2- 172.21.15.44:0/2163383124 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f01500a2e10 0x7f01500a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.874+0000 7f016ef7f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1d97ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.874+0000 7f016ef7f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1da0db0 0x55ddf1da31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.874+0000 7f016ef7f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ddf1cfb170 con 0x55ddf1d97a40 2024-04-21T23:08:55.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.874+0000 7f016ef7f740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55ddf1ca37c0 con 0x55ddf1d98720 2024-04-21T23:08:55.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.874+0000 7f016ef7f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ddf1cf7130 con 0x55ddf1da0db0 2024-04-21T23:08:55.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.874+0000 7f016a051640 1 --2- 172.21.15.44:0/2163383124 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f01500a2e10 0x7f01500a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.874+0000 7f0169850640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1d97ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.874+0000 7f0169850640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1d97ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35764/0 (socket says 172.21.15.44:35764) 2024-04-21T23:08:55.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.874+0000 7f0169850640 1 -- 172.21.15.44:0/7805011 learned_addr learned my addr 172.21.15.44:0/7805011 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:55.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.874+0000 7f016a852640 1 --2- 172.21.15.44:0/7805011 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1d98720 0x55ddf1d97470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.874+0000 7f016a852640 1 --2- 172.21.15.44:0/7805011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1da0db0 0x55ddf1da31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.874+0000 7f015b7fe640 1 -- 172.21.15.44:0/2163383124 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f01600f1350 con 0x7f0164100610 2024-04-21T23:08:55.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.874+0000 7f0169850640 1 -- 172.21.15.44:0/7805011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1da0db0 msgr2=0x55ddf1da31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:55.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.874+0000 7f0169850640 1 --2- 172.21.15.44:0/7805011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1da0db0 0x55ddf1da31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.874+0000 7f0169850640 1 -- 172.21.15.44:0/7805011 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1d98720 msgr2=0x55ddf1d97470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:55.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.874+0000 7f0169850640 1 --2- 172.21.15.44:0/7805011 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1d98720 0x55ddf1d97470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.886 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.874+0000 7f0169850640 1 -- 172.21.15.44:0/7805011 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddf1cb4e60 con 0x55ddf1d97a40 2024-04-21T23:08:55.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.875+0000 7f016a051640 1 --2- 172.21.15.44:0/2163383124 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f01500a2e10 0x7f01500a52d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f01600ee750 tx=0x7f01600b97e0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:55.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.875+0000 7f0169850640 1 --2- 172.21.15.44:0/7805011 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1d97ec0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0154009ca0 tx=0x7f0154011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72aa3d334288c940 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:55.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.878+0000 7f0158bfe640 1 -- 172.21.15.44:0/7805011 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0154009e50 con 0x55ddf1d97a40 2024-04-21T23:08:55.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.878+0000 7f0158bfe640 1 -- 172.21.15.44:0/7805011 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0154005500 con 0x55ddf1d97a40 2024-04-21T23:08:55.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.878+0000 7f0158bfe640 1 -- 172.21.15.44:0/7805011 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0154019670 con 0x55ddf1d97a40 2024-04-21T23:08:55.887 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.878+0000 7fedc2ffd640 1 -- 172.21.15.44:0/391156657 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fedc40be440 con 0x7fedcc10bd80 2024-04-21T23:08:55.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.879+0000 7f016ef7f740 1 -- 172.21.15.44:0/7805011 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 msgr2=0x55ddf1d97ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.879+0000 7f016ef7f740 1 --2- 172.21.15.44:0/7805011 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1d97ec0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0154009ca0 tx=0x7f0154011810 comp rx=0 tx=0).stop 2024-04-21T23:08:55.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.879+0000 7f016ef7f740 1 -- 172.21.15.44:0/7805011 shutdown_connections 2024-04-21T23:08:55.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.879+0000 7f016ef7f740 1 --2- 172.21.15.44:0/7805011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1da0db0 0x55ddf1da31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.879+0000 7f016ef7f740 1 --2- 172.21.15.44:0/7805011 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1d97ec0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.879+0000 7f016ef7f740 1 --2- 172.21.15.44:0/7805011 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1d98720 0x55ddf1d97470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.879+0000 7f016ef7f740 1 -- 172.21.15.44:0/7805011 >> 172.21.15.44:0/7805011 conn(0x55ddf1d5e4f0 msgr2=0x55ddf1d92490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:55.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.879+0000 7f016ef7f740 1 -- 172.21.15.44:0/7805011 shutdown_connections 2024-04-21T23:08:55.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.879+0000 7f016ef7f740 1 -- 172.21.15.44:0/7805011 wait complete. 2024-04-21T23:08:55.892 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.879+0000 7f016ef7f740 1 -- start start 2024-04-21T23:08:55.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.879+0000 7f016ef7f740 1 Processor -- start 2024-04-21T23:08:55.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.879+0000 7f016ef7f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1d9ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.879+0000 7f016ef7f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1d98720 0x55ddf1d9e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f016ef7f740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1da0db0 0x55ddf1d9e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f016ef7f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ddf1da3c20 con 0x55ddf1d97a40 2024-04-21T23:08:55.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f016ef7f740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55ddf1cb6d40 con 0x55ddf1da0db0 2024-04-21T23:08:55.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f016ef7f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ddf1da3da0 con 0x55ddf1d98720 2024-04-21T23:08:55.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f016ef7f740 10 client.0 _ll_register_callbacks cb 0x55ddf1cf2820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:08:55.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f016a852640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1d9ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f016a852640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1d9ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35774/0 (socket says 172.21.15.44:35774) 2024-04-21T23:08:55.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f016a852640 1 -- 172.21.15.44:0/2256882661 learned_addr learned my addr 172.21.15.44:0/2256882661 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:55.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f0169850640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1d98720 0x55ddf1d9e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f016a852640 1 --2- 172.21.15.44:0/2256882661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1da0db0 0x55ddf1d9e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f0169850640 1 -- 172.21.15.44:0/2256882661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1da0db0 msgr2=0x55ddf1d9e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:55.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f0169850640 1 --2- 172.21.15.44:0/2256882661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1da0db0 0x55ddf1d9e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f0169850640 1 -- 172.21.15.44:0/2256882661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 msgr2=0x55ddf1d9ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:55.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f0169850640 1 --2- 172.21.15.44:0/2256882661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1d9ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f0169850640 1 -- 172.21.15.44:0/2256882661 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddf1ad77e0 con 0x55ddf1d98720 2024-04-21T23:08:55.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f016a852640 1 --2- 172.21.15.44:0/2256882661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1d9ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:08:55.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f0169850640 1 --2- 172.21.15.44:0/2256882661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1d98720 0x55ddf1d9e2f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f015401b040 tx=0x7f0154005050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f43530324f0d92bb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:55.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f014dffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:08:55.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f014dffb640 1 -- 172.21.15.44:0/2256882661 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0154028070 con 0x55ddf1d98720 2024-04-21T23:08:55.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f014dffb640 1 -- 172.21.15.44:0/2256882661 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f0154004020 con 0x55ddf1d98720 2024-04-21T23:08:55.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.880+0000 7f016ef7f740 1 -- 172.21.15.44:0/2256882661 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0154005500 con 0x55ddf1d98720 2024-04-21T23:08:55.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.881+0000 7f014dffb640 1 -- 172.21.15.44:0/2256882661 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0154033050 con 0x55ddf1d98720 2024-04-21T23:08:55.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.881+0000 7f016ef7f740 10 client.25303 fetch_fsmap learned FSMap version 21 2024-04-21T23:08:55.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.881+0000 7f016ef7f740 1 -- 172.21.15.44:0/2256882661 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ddf1b37f70 con 0x55ddf1d98720 2024-04-21T23:08:55.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.881+0000 7f014dffb640 1 -- 172.21.15.44:0/2256882661 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0154005500 con 0x55ddf1d98720 2024-04-21T23:08:55.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.881+0000 7f014dffb640 1 -- 172.21.15.44:0/2256882661 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f0154003ad0 con 0x55ddf1d98720 2024-04-21T23:08:55.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.881+0000 7f016ef7f740 10 client.25303 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:08:55.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.881+0000 7f016ef7f740 10 client.25303 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:08:55.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.881+0000 7f016ef7f740 20 client.25303 populate_metadata read hostname 'smithi044' 2024-04-21T23:08:55.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.881+0000 7f016ef7f740 1 --2- 172.21.15.44:0/2256882661 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55ddf1db0db0 0x55ddf1db31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.881+0000 7f016ef7f740 4 client.25303 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:08:55.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.881+0000 7f016ef7f740 1 -- 172.21.15.44:0/2256882661 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ddf1ccf450 con 0x55ddf1db0db0 2024-04-21T23:08:55.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.881+0000 7f016a852640 1 --2- 172.21.15.44:0/2256882661 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55ddf1db0db0 0x55ddf1db31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.882+0000 7f016a852640 1 --2- 172.21.15.44:0/2256882661 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55ddf1db0db0 0x55ddf1db31a0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9489182917398a6 server_cookie=3c5e8c905543547e in_seq=0 out_seq=0 2024-04-21T23:08:55.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.882+0000 7f014dffb640 10 client.25303 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:08:55.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.882+0000 7f014dffb640 1 -- 172.21.15.44:0/2256882661 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ddf1ccf450 con 0x55ddf1db0db0 2024-04-21T23:08:55.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.882+0000 7f014dffb640 10 client.25303 handle_command_reply: tid=0 2024-04-21T23:08:55.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.882+0000 7f016ef7f740 1 client.25303 shutdown 2024-04-21T23:08:55.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.883+0000 7f016ef7f740 1 -- 172.21.15.44:0/2256882661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1d98720 msgr2=0x55ddf1d9e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.883+0000 7f016ef7f740 1 --2- 172.21.15.44:0/2256882661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1d98720 0x55ddf1d9e2f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f015401b040 tx=0x7f0154005050 comp rx=0 tx=0).stop 2024-04-21T23:08:55.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.883+0000 7f016ef7f740 1 -- 172.21.15.44:0/2256882661 shutdown_connections 2024-04-21T23:08:55.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.883+0000 7f016ef7f740 1 --2- 172.21.15.44:0/2256882661 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55ddf1db0db0 0x55ddf1db31a0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.883+0000 7f016ef7f740 1 --2- 172.21.15.44:0/2256882661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1d98720 0x55ddf1d9e2f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.883+0000 7f016ef7f740 1 --2- 172.21.15.44:0/2256882661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1da0db0 0x55ddf1d9e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.883+0000 7f016ef7f740 1 --2- 172.21.15.44:0/2256882661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1d9ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.883+0000 7f014dffb640 0 client.25303 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:08:55.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.883+0000 7f016ef7f740 1 -- 172.21.15.44:0/2256882661 >> 172.21.15.44:0/2256882661 conn(0x55ddf1d5e4f0 msgr2=0x55ddf1d941e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:55.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.883+0000 7f016ef7f740 1 -- 172.21.15.44:0/2256882661 shutdown_connections 2024-04-21T23:08:55.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.883+0000 7f016ef7f740 1 -- 172.21.15.44:0/2256882661 wait complete. 2024-04-21T23:08:55.900 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.883+0000 7f016ef7f740 20 client.25303 trim_cache size 0 max 16384 2024-04-21T23:08:55.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.901+0000 7f016ef7f740 1 Processor -- start 2024-04-21T23:08:55.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.901+0000 7f016ef7f740 1 -- start start 2024-04-21T23:08:55.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.901+0000 7f016ef7f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1da04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.901+0000 7f016ef7f740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1d98720 0x55ddf1d9ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.901+0000 7f016ef7f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1da0db0 0x55ddf1dacb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.901+0000 7f016ef7f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f01641aee50 con 0x55ddf1d97a40 2024-04-21T23:08:55.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.901+0000 7f016ef7f740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f01641b16d0 con 0x55ddf1d98720 2024-04-21T23:08:55.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.901+0000 7f016ef7f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f01641af4c0 con 0x55ddf1da0db0 2024-04-21T23:08:55.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.901+0000 7f016a852640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1da04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.901+0000 7f016a852640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1da04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35782/0 (socket says 172.21.15.44:35782) 2024-04-21T23:08:55.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.901+0000 7f016a852640 1 -- 172.21.15.44:0/4052750722 learned_addr learned my addr 172.21.15.44:0/4052750722 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:55.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.901+0000 7f0169850640 1 --2- 172.21.15.44:0/4052750722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1d98720 0x55ddf1d9ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.901+0000 7f016a852640 1 --2- 172.21.15.44:0/4052750722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1da0db0 0x55ddf1dacb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.901+0000 7f016a852640 1 -- 172.21.15.44:0/4052750722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1da0db0 msgr2=0x55ddf1dacb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:55.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.901+0000 7f016a852640 1 --2- 172.21.15.44:0/4052750722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1da0db0 0x55ddf1dacb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.901+0000 7f016a852640 1 -- 172.21.15.44:0/4052750722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1d98720 msgr2=0x55ddf1d9ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:55.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.901+0000 7f016a852640 1 --2- 172.21.15.44:0/4052750722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1d98720 0x55ddf1d9ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.901+0000 7f016a852640 1 -- 172.21.15.44:0/4052750722 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddf19cf170 con 0x55ddf1d97a40 2024-04-21T23:08:55.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.901+0000 7f0169850640 1 --2- 172.21.15.44:0/4052750722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1d98720 0x55ddf1d9ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:08:55.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.902+0000 7f016a852640 1 --2- 172.21.15.44:0/4052750722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1da04e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f015c0099a0 tx=0x7f015c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e3e036ad9c049e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:55.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.902+0000 7f014ffff640 1 -- 172.21.15.44:0/4052750722 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c025e60 con 0x55ddf1d97a40 2024-04-21T23:08:55.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.902+0000 7f014ffff640 1 -- 172.21.15.44:0/4052750722 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f015c025490 con 0x55ddf1d97a40 2024-04-21T23:08:55.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.902+0000 7f014ffff640 1 -- 172.21.15.44:0/4052750722 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c0055b0 con 0x55ddf1d97a40 2024-04-21T23:08:55.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.902+0000 7f016ef7f740 1 -- 172.21.15.44:0/4052750722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 msgr2=0x55ddf1da04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.902+0000 7f016ef7f740 1 --2- 172.21.15.44:0/4052750722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1da04e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f015c0099a0 tx=0x7f015c004ab0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.902+0000 7f016ef7f740 1 -- 172.21.15.44:0/4052750722 shutdown_connections 2024-04-21T23:08:55.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.902+0000 7f016ef7f740 1 --2- 172.21.15.44:0/4052750722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1da0db0 0x55ddf1dacb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.902+0000 7f016ef7f740 1 --2- 172.21.15.44:0/4052750722 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1d98720 0x55ddf1d9ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.902+0000 7f016ef7f740 1 --2- 172.21.15.44:0/4052750722 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1da04e0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.902+0000 7f016ef7f740 1 -- 172.21.15.44:0/4052750722 >> 172.21.15.44:0/4052750722 conn(0x55ddf1d5e4f0 msgr2=0x55ddf1d914f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:55.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.903+0000 7f016ef7f740 1 -- 172.21.15.44:0/4052750722 shutdown_connections 2024-04-21T23:08:55.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.903+0000 7f016ef7f740 1 -- 172.21.15.44:0/4052750722 wait complete. 2024-04-21T23:08:55.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.903+0000 7f016ef7f740 1 -- start start 2024-04-21T23:08:55.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.903+0000 7f016ef7f740 1 Processor -- start 2024-04-21T23:08:55.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.903+0000 7f016ef7f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1da9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.903+0000 7f016ef7f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1d98720 0x55ddf1da9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.903+0000 7f016ef7f740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1da0db0 0x55ddf1da6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.903+0000 7f016ef7f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f01641afb90 con 0x55ddf1d97a40 2024-04-21T23:08:55.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.903+0000 7f016ef7f740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f01641b1000 con 0x55ddf1da0db0 2024-04-21T23:08:55.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.903+0000 7f016ef7f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f01641b0260 con 0x55ddf1d98720 2024-04-21T23:08:55.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.903+0000 7f016ef7f740 10 client.0 _ll_register_callbacks cb 0x7f015c011580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:08:55.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.903+0000 7f016a852640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1da9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.903+0000 7f016a852640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1da9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35784/0 (socket says 172.21.15.44:35784) 2024-04-21T23:08:55.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.903+0000 7f016a852640 1 -- 172.21.15.44:0/3878672374 learned_addr learned my addr 172.21.15.44:0/3878672374 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:55.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.903+0000 7f0169850640 1 --2- 172.21.15.44:0/3878672374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1d98720 0x55ddf1da9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.904+0000 7f016a852640 1 --2- 172.21.15.44:0/3878672374 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1da0db0 0x55ddf1da6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.904+0000 7f016a852640 1 -- 172.21.15.44:0/3878672374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1d98720 msgr2=0x55ddf1da9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:55.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.904+0000 7f016a852640 1 --2- 172.21.15.44:0/3878672374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1d98720 0x55ddf1da9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.904+0000 7f016a852640 1 -- 172.21.15.44:0/3878672374 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1da0db0 msgr2=0x55ddf1da6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:55.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.904+0000 7f016a852640 1 --2- 172.21.15.44:0/3878672374 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1da0db0 0x55ddf1da6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.904+0000 7f016a852640 1 -- 172.21.15.44:0/3878672374 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddf1da6640 con 0x55ddf1d97a40 2024-04-21T23:08:55.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.904+0000 7f016a852640 1 --2- 172.21.15.44:0/3878672374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1da9030 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f015c005290 tx=0x7f015c009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5bdc76b45eac140 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:55.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.904+0000 7f014dffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:08:55.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.904+0000 7f014dffb640 1 -- 172.21.15.44:0/3878672374 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c025e60 con 0x55ddf1d97a40 2024-04-21T23:08:55.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.904+0000 7f016ef7f740 1 -- 172.21.15.44:0/3878672374 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f015c025490 con 0x55ddf1d97a40 2024-04-21T23:08:55.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.904+0000 7f014dffb640 1 -- 172.21.15.44:0/3878672374 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f015c004d20 con 0x55ddf1d97a40 2024-04-21T23:08:55.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.904+0000 7f014dffb640 1 -- 172.21.15.44:0/3878672374 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f015c042050 con 0x55ddf1d97a40 2024-04-21T23:08:55.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.904+0000 7f016ef7f740 10 client.15612 fetch_fsmap learned FSMap version 21 2024-04-21T23:08:55.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.904+0000 7f016ef7f740 1 -- 172.21.15.44:0/3878672374 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ddf1e17a10 con 0x55ddf1d97a40 2024-04-21T23:08:55.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.904+0000 7f014dffb640 1 -- 172.21.15.44:0/3878672374 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f015c025490 con 0x55ddf1d97a40 2024-04-21T23:08:55.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.905+0000 7f014dffb640 1 -- 172.21.15.44:0/3878672374 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f015c022600 con 0x55ddf1d97a40 2024-04-21T23:08:55.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.905+0000 7f016ef7f740 10 client.15612 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:08:55.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.905+0000 7f016ef7f740 10 client.15612 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:08:55.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.905+0000 7f016ef7f740 20 client.15612 populate_metadata read hostname 'smithi044' 2024-04-21T23:08:55.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.905+0000 7f016ef7f740 1 --2- 172.21.15.44:0/3878672374 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55ddf1e18050 0x55ddf1e1a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:55.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.905+0000 7f016ef7f740 4 client.15612 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:08:55.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.905+0000 7f016ef7f740 1 -- 172.21.15.44:0/3878672374 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ddf1d086b0 con 0x55ddf1e18050 2024-04-21T23:08:55.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.905+0000 7f0169850640 1 --2- 172.21.15.44:0/3878672374 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55ddf1e18050 0x55ddf1e1a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:55.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.905+0000 7f014dffb640 10 client.15612 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:08:55.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.905+0000 7f0169850640 1 --2- 172.21.15.44:0/3878672374 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55ddf1e18050 0x55ddf1e1a470 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28d19a328e7f1744 server_cookie=ac36f36fe6776c37 in_seq=0 out_seq=0 2024-04-21T23:08:55.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.906+0000 7f014dffb640 1 -- 172.21.15.44:0/3878672374 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ddf1d086b0 con 0x55ddf1e18050 2024-04-21T23:08:55.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.906+0000 7f014dffb640 10 client.15612 handle_command_reply: tid=0 2024-04-21T23:08:55.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.906+0000 7f016ef7f740 1 client.15612 shutdown 2024-04-21T23:08:55.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.907+0000 7f016ef7f740 1 -- 172.21.15.44:0/3878672374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 msgr2=0x55ddf1da9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.907+0000 7f016ef7f740 1 --2- 172.21.15.44:0/3878672374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1da9030 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f015c005290 tx=0x7f015c009900 comp rx=0 tx=0).stop 2024-04-21T23:08:55.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.907+0000 7f016ef7f740 1 -- 172.21.15.44:0/3878672374 shutdown_connections 2024-04-21T23:08:55.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.907+0000 7f016ef7f740 1 --2- 172.21.15.44:0/3878672374 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55ddf1e18050 0x55ddf1e1a470 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.907+0000 7f016ef7f740 1 --2- 172.21.15.44:0/3878672374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ddf1d98720 0x55ddf1da9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.907+0000 7f014dffb640 0 client.15612 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:08:55.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.907+0000 7f016ef7f740 1 --2- 172.21.15.44:0/3878672374 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ddf1da0db0 0x55ddf1da6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.907+0000 7f016ef7f740 1 --2- 172.21.15.44:0/3878672374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ddf1d97a40 0x55ddf1da9030 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.907+0000 7f016ef7f740 1 -- 172.21.15.44:0/3878672374 >> 172.21.15.44:0/3878672374 conn(0x55ddf1d5e4f0 msgr2=0x55ddf1d93850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:55.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.907+0000 7f016ef7f740 1 -- 172.21.15.44:0/3878672374 shutdown_connections 2024-04-21T23:08:55.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.907+0000 7f016ef7f740 1 -- 172.21.15.44:0/3878672374 wait complete. 2024-04-21T23:08:55.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.907+0000 7f016ef7f740 20 client.15612 trim_cache size 0 max 16384 2024-04-21T23:08:55.914 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:08:55.914 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-21T23:08:55.914 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "8b5bb175-75c0-40a3-abff-d9c051ff80ad", 2024-04-21T23:08:55.914 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-21T23:08:55.915 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:08:55.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.908+0000 7f016c2b5640 1 -- 172.21.15.44:0/2163383124 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f01500a2e10 msgr2=0x7f01500a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.908+0000 7f016c2b5640 1 --2- 172.21.15.44:0/2163383124 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f01500a2e10 0x7f01500a52d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f01600ee750 tx=0x7f01600b97e0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.908+0000 7f016c2b5640 1 -- 172.21.15.44:0/2163383124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0164100610 msgr2=0x7f0164196cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:55.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.908+0000 7f016c2b5640 1 --2- 172.21.15.44:0/2163383124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0164100610 0x7f0164196cc0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0160009230 tx=0x7f016002cb20 comp rx=0 tx=0).stop 2024-04-21T23:08:55.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.908+0000 7f016c2b5640 1 -- 172.21.15.44:0/2163383124 shutdown_connections 2024-04-21T23:08:55.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.908+0000 7f016c2b5640 1 --2- 172.21.15.44:0/2163383124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0164100fd0 0x7f0164197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.908+0000 7f016c2b5640 1 --2- 172.21.15.44:0/2163383124 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f01500a2e10 0x7f01500a52d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.908+0000 7f016c2b5640 1 --2- 172.21.15.44:0/2163383124 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f01641066b0 0x7f0164197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.908+0000 7f016c2b5640 1 --2- 172.21.15.44:0/2163383124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0164100610 0x7f0164196cc0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:55.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.908+0000 7f016c2b5640 1 -- 172.21.15.44:0/2163383124 >> 172.21.15.44:0/2163383124 conn(0x7f01640777f0 msgr2=0x7f01640ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:55.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.908+0000 7f016c2b5640 1 -- 172.21.15.44:0/2163383124 shutdown_connections 2024-04-21T23:08:55.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:55.908+0000 7f016c2b5640 1 -- 172.21.15.44:0/2163383124 wait complete. 2024-04-21T23:08:55.922 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:08:55.922 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:08:56.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.028+0000 7f8514039640 1 -- 172.21.15.44:0/2305275957 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f850c074490 msgr2=0x7f850c10da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.028+0000 7f8514039640 1 --2- 172.21.15.44:0/2305275957 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f850c074490 0x7f850c10da10 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f84fc009900 tx=0x7f84fc02bcd0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.028+0000 7f8514039640 1 -- 172.21.15.44:0/2305275957 shutdown_connections 2024-04-21T23:08:56.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.028+0000 7f8514039640 1 --2- 172.21.15.44:0/2305275957 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f850c10df50 0x7f850c1103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.028+0000 7f8514039640 1 --2- 172.21.15.44:0/2305275957 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f850c074490 0x7f850c10da10 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.028+0000 7f8514039640 1 --2- 172.21.15.44:0/2305275957 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f850c073b70 0x7f850c073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.028+0000 7f8514039640 1 -- 172.21.15.44:0/2305275957 >> 172.21.15.44:0/2305275957 conn(0x7f850c0fd200 msgr2=0x7f850c0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:56.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.028+0000 7f8514039640 1 -- 172.21.15.44:0/2305275957 shutdown_connections 2024-04-21T23:08:56.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.028+0000 7f8514039640 1 -- 172.21.15.44:0/2305275957 wait complete. 2024-04-21T23:08:56.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.029+0000 7f8514039640 1 Processor -- start 2024-04-21T23:08:56.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.029+0000 7f8514039640 1 -- start start 2024-04-21T23:08:56.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.029+0000 7f8514039640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f850c073b70 0x7f850c19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.029+0000 7f8514039640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f850c074490 0x7f850c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.029+0000 7f8514039640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f850c10df50 0x7f850c19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.029+0000 7f8514039640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f850c113cf0 con 0x7f850c073b70 2024-04-21T23:08:56.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.029+0000 7f8514039640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f850c113b70 con 0x7f850c074490 2024-04-21T23:08:56.041 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.029+0000 7f8514039640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f850c113e70 con 0x7f850c10df50 2024-04-21T23:08:56.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.029+0000 7f8511dd5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f850c073b70 0x7f850c19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.029+0000 7f8511dd5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f850c073b70 0x7f850c19a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35794/0 (socket says 172.21.15.44:35794) 2024-04-21T23:08:56.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.029+0000 7f8511dd5640 1 -- 172.21.15.44:0/2084417631 learned_addr learned my addr 172.21.15.44:0/2084417631 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:56.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.029+0000 7f85115d4640 1 --2- 172.21.15.44:0/2084417631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f850c074490 0x7f850c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.029+0000 7f85125d6640 1 --2- 172.21.15.44:0/2084417631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f850c10df50 0x7f850c19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.030+0000 7f8511dd5640 1 -- 172.21.15.44:0/2084417631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f850c10df50 msgr2=0x7f850c19eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.030+0000 7f8511dd5640 1 --2- 172.21.15.44:0/2084417631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f850c10df50 0x7f850c19eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.030+0000 7f8511dd5640 1 -- 172.21.15.44:0/2084417631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f850c074490 msgr2=0x7f850c19a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.030+0000 7f8511dd5640 1 --2- 172.21.15.44:0/2084417631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f850c074490 0x7f850c19a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.030+0000 7f8511dd5640 1 -- 172.21.15.44:0/2084417631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f850c19f1f0 con 0x7f850c073b70 2024-04-21T23:08:56.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.030+0000 7f8511dd5640 1 --2- 172.21.15.44:0/2084417631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f850c073b70 0x7f850c19a250 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f850800c890 tx=0x7f850800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:56.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.030+0000 7f8502ffd640 1 -- 172.21.15.44:0/2084417631 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8508004220 con 0x7f850c073b70 2024-04-21T23:08:56.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.030+0000 7f8514039640 1 -- 172.21.15.44:0/2084417631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f850c19f4e0 con 0x7f850c073b70 2024-04-21T23:08:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.030+0000 7f8502ffd640 1 -- 172.21.15.44:0/2084417631 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85080043c0 con 0x7f850c073b70 2024-04-21T23:08:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.030+0000 7f8502ffd640 1 -- 172.21.15.44:0/2084417631 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8508004a00 con 0x7f850c073b70 2024-04-21T23:08:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.030+0000 7f8514039640 1 -- 172.21.15.44:0/2084417631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f850c1a6d70 con 0x7f850c073b70 2024-04-21T23:08:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.032+0000 7f8502ffd640 1 -- 172.21.15.44:0/2084417631 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f85080079b0 con 0x7f850c073b70 2024-04-21T23:08:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.032+0000 7f8516c25740 1 Processor -- start 2024-04-21T23:08:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.033+0000 7f8516c25740 1 -- start start 2024-04-21T23:08:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.033+0000 7f8516c25740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73df30 0x55e3cf73e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.033+0000 7f8502ffd640 1 --2- 172.21.15.44:0/2084417631 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f84f80a2d90 0x7f84f80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.033+0000 7f8516c25740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf73e850 0x55e3cf746ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.033+0000 7f8516c25740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e3cf747430 0x55e3cf749820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.033+0000 7f8516c25740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e3cf6a2560 con 0x55e3cf73df30 2024-04-21T23:08:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.033+0000 7f8516c25740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55e3cf64a730 con 0x55e3cf73e850 2024-04-21T23:08:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.033+0000 7f8516c25740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e3cf69e520 con 0x55e3cf747430 2024-04-21T23:08:56.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.033+0000 7f8511dd5640 1 --2- 172.21.15.44:0/2084417631 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f84f80a2d90 0x7f84f80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.033+0000 7f85125d6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73df30 0x55e3cf73e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.033+0000 7f85125d6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73df30 0x55e3cf73e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35806/0 (socket says 172.21.15.44:35806) 2024-04-21T23:08:56.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.033+0000 7f85125d6640 1 -- 172.21.15.44:0/551883912 learned_addr learned my addr 172.21.15.44:0/551883912 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:56.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.033+0000 7f85115d4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf73e850 0x55e3cf746ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.033+0000 7f85125d6640 1 --2- 172.21.15.44:0/551883912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e3cf747430 0x55e3cf749820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.033+0000 7f8502ffd640 1 -- 172.21.15.44:0/2084417631 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f85080cf690 con 0x7f850c073b70 2024-04-21T23:08:56.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.033+0000 7f85115d4640 1 -- 172.21.15.44:0/551883912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e3cf747430 msgr2=0x55e3cf749820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:56.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.034+0000 7f85115d4640 1 --2- 172.21.15.44:0/551883912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e3cf747430 0x55e3cf749820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.034+0000 7f85115d4640 1 -- 172.21.15.44:0/551883912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73df30 msgr2=0x55e3cf73e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:56.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.034+0000 7f85115d4640 1 --2- 172.21.15.44:0/551883912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73df30 0x55e3cf73e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.034+0000 7f85115d4640 1 -- 172.21.15.44:0/551883912 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3cf4807d0 con 0x55e3cf73e850 2024-04-21T23:08:56.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.034+0000 7f85125d6640 1 --2- 172.21.15.44:0/551883912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73df30 0x55e3cf73e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:08:56.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.034+0000 7f8511dd5640 1 --2- 172.21.15.44:0/2084417631 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f84f80a2d90 0x7f84f80a5250 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f85080cc9e0 tx=0x7f8508005300 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:56.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.034+0000 7f85115d4640 1 --2- 172.21.15.44:0/551883912 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf73e850 0x55e3cf746ef0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f84fc000c00 tx=0x7f84fc005e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86013fb85ff9e82f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:56.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.034+0000 7f84f7fff640 1 -- 172.21.15.44:0/551883912 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc042070 con 0x55e3cf73e850 2024-04-21T23:08:56.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.034+0000 7f84f7fff640 1 -- 172.21.15.44:0/551883912 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84fc02cb20 con 0x55e3cf73e850 2024-04-21T23:08:56.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.034+0000 7f84f7fff640 1 -- 172.21.15.44:0/551883912 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc004f10 con 0x55e3cf73e850 2024-04-21T23:08:56.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.035+0000 7f8516c25740 1 -- 172.21.15.44:0/551883912 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf73e850 msgr2=0x55e3cf746ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.035+0000 7f8516c25740 1 --2- 172.21.15.44:0/551883912 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf73e850 0x55e3cf746ef0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f84fc000c00 tx=0x7f84fc005e00 comp rx=0 tx=0).stop 2024-04-21T23:08:56.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.035+0000 7f8516c25740 1 -- 172.21.15.44:0/551883912 shutdown_connections 2024-04-21T23:08:56.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.035+0000 7f8516c25740 1 --2- 172.21.15.44:0/551883912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e3cf747430 0x55e3cf749820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.035+0000 7f8516c25740 1 --2- 172.21.15.44:0/551883912 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf73e850 0x55e3cf746ef0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.035+0000 7f8516c25740 1 --2- 172.21.15.44:0/551883912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73df30 0x55e3cf73e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.035+0000 7f8516c25740 1 -- 172.21.15.44:0/551883912 >> 172.21.15.44:0/551883912 conn(0x55e3cf738660 msgr2=0x55e3cf738a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:56.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.035+0000 7f8516c25740 1 -- 172.21.15.44:0/551883912 shutdown_connections 2024-04-21T23:08:56.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.035+0000 7f8516c25740 1 -- 172.21.15.44:0/551883912 wait complete. 2024-04-21T23:08:56.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.035+0000 7f8516c25740 1 -- start start 2024-04-21T23:08:56.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.035+0000 7f8516c25740 1 Processor -- start 2024-04-21T23:08:56.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.036+0000 7f8516c25740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e3cf73df30 0x55e3cf7444c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.036+0000 7f8516c25740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73e850 0x55e3cf744a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.036+0000 7f8516c25740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf747430 0x55e3cf744f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.036+0000 7f8516c25740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e3cf74a1c0 con 0x55e3cf73e850 2024-04-21T23:08:56.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.036+0000 7f8516c25740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55e3cf65de00 con 0x55e3cf747430 2024-04-21T23:08:56.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.036+0000 7f8516c25740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e3cf74a340 con 0x55e3cf73df30 2024-04-21T23:08:56.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.036+0000 7f8516c25740 10 client.0 _ll_register_callbacks cb 0x55e3cf698eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:08:56.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.036+0000 7f85115d4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73e850 0x55e3cf744a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.036+0000 7f85115d4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73e850 0x55e3cf744a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35816/0 (socket says 172.21.15.44:35816) 2024-04-21T23:08:56.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.036+0000 7f85115d4640 1 -- 172.21.15.44:0/2930333941 learned_addr learned my addr 172.21.15.44:0/2930333941 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:56.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.036+0000 7f85125d6640 1 --2- 172.21.15.44:0/2930333941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e3cf73df30 0x55e3cf7444c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.036+0000 7f85125d6640 1 --2- 172.21.15.44:0/2930333941 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf747430 0x55e3cf744f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.036+0000 7f85115d4640 1 -- 172.21.15.44:0/2930333941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e3cf73df30 msgr2=0x55e3cf7444c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:56.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.036+0000 7f85115d4640 1 --2- 172.21.15.44:0/2930333941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e3cf73df30 0x55e3cf7444c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.036+0000 7f85115d4640 1 -- 172.21.15.44:0/2930333941 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf747430 msgr2=0x55e3cf744f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:56.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.036+0000 7f85115d4640 1 --2- 172.21.15.44:0/2930333941 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf747430 0x55e3cf744f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.036+0000 7f85115d4640 1 -- 172.21.15.44:0/2930333941 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3cf3b63f0 con 0x55e3cf73e850 2024-04-21T23:08:56.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.036+0000 7f85115d4640 1 --2- 172.21.15.44:0/2930333941 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73e850 0x55e3cf744a00 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f84fc02c8c0 tx=0x7f84fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db2cf306db5e4655 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:56.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.036+0000 7f84f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:08:56.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.036+0000 7f84f57fa640 1 -- 172.21.15.44:0/2930333941 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc04c070 con 0x55e3cf73e850 2024-04-21T23:08:56.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.037+0000 7f84f57fa640 1 -- 172.21.15.44:0/2930333941 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f84fc005110 con 0x55e3cf73e850 2024-04-21T23:08:56.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.037+0000 7f8516c25740 1 -- 172.21.15.44:0/2930333941 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84fc02cb20 con 0x55e3cf73e850 2024-04-21T23:08:56.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.037+0000 7f84f57fa640 1 -- 172.21.15.44:0/2930333941 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f84fc058050 con 0x55e3cf73e850 2024-04-21T23:08:56.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.037+0000 7f8516c25740 10 client.15624 fetch_fsmap learned FSMap version 21 2024-04-21T23:08:56.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.037+0000 7f8516c25740 1 -- 172.21.15.44:0/2930333941 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e3cf3760f0 con 0x55e3cf73e850 2024-04-21T23:08:56.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.037+0000 7f84f57fa640 1 -- 172.21.15.44:0/2930333941 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84fc02cb20 con 0x55e3cf73e850 2024-04-21T23:08:56.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.037+0000 7f84f57fa640 1 -- 172.21.15.44:0/2930333941 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f84fc034440 con 0x55e3cf73e850 2024-04-21T23:08:56.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.037+0000 7f8516c25740 10 client.15624 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:08:56.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.037+0000 7f8516c25740 10 client.15624 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:08:56.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.037+0000 7f8516c25740 20 client.15624 populate_metadata read hostname 'smithi044' 2024-04-21T23:08:56.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.037+0000 7f8516c25740 1 --2- 172.21.15.44:0/2930333941 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e3cf757640 0x55e3cf759a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.037+0000 7f8516c25740 4 client.15624 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:08:56.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.037+0000 7f8516c25740 1 -- 172.21.15.44:0/2930333941 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3cf6afaa0 con 0x55e3cf757640 2024-04-21T23:08:56.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.037+0000 7f85125d6640 1 --2- 172.21.15.44:0/2930333941 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e3cf757640 0x55e3cf759a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.038+0000 7f85125d6640 1 --2- 172.21.15.44:0/2930333941 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e3cf757640 0x55e3cf759a30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1efc212b79db3a82 server_cookie=f032a30258112ebe in_seq=0 out_seq=0 2024-04-21T23:08:56.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.038+0000 7f84f57fa640 10 client.15624 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:08:56.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.038+0000 7f84f57fa640 1 -- 172.21.15.44:0/2930333941 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e3cf6afaa0 con 0x55e3cf757640 2024-04-21T23:08:56.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.039+0000 7f84f57fa640 10 client.15624 handle_command_reply: tid=0 2024-04-21T23:08:56.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.039+0000 7f8516c25740 1 client.15624 shutdown 2024-04-21T23:08:56.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.039+0000 7f8516c25740 1 -- 172.21.15.44:0/2930333941 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73e850 msgr2=0x55e3cf744a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.039+0000 7f8516c25740 1 --2- 172.21.15.44:0/2930333941 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73e850 0x55e3cf744a00 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f84fc02c8c0 tx=0x7f84fc0046c0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.039+0000 7f8516c25740 1 -- 172.21.15.44:0/2930333941 shutdown_connections 2024-04-21T23:08:56.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.039+0000 7f8516c25740 1 --2- 172.21.15.44:0/2930333941 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e3cf757640 0x55e3cf759a30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.039+0000 7f8516c25740 1 --2- 172.21.15.44:0/2930333941 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf747430 0x55e3cf744f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.039+0000 7f8516c25740 1 --2- 172.21.15.44:0/2930333941 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73e850 0x55e3cf744a00 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.039+0000 7f84f57fa640 0 client.15624 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:08:56.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.039+0000 7f8516c25740 1 --2- 172.21.15.44:0/2930333941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e3cf73df30 0x55e3cf7444c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.039+0000 7f8516c25740 1 -- 172.21.15.44:0/2930333941 >> 172.21.15.44:0/2930333941 conn(0x55e3cf738660 msgr2=0x55e3cf73a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:56.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.039+0000 7f8516c25740 1 -- 172.21.15.44:0/2930333941 shutdown_connections 2024-04-21T23:08:56.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.039+0000 7f8516c25740 1 -- 172.21.15.44:0/2930333941 wait complete. 2024-04-21T23:08:56.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.039+0000 7f8516c25740 20 client.15624 trim_cache size 0 max 16384 2024-04-21T23:08:56.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.057+0000 7f8516c25740 1 Processor -- start 2024-04-21T23:08:56.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.057+0000 7f8516c25740 1 -- start start 2024-04-21T23:08:56.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.057+0000 7f8516c25740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73df30 0x55e3cf753080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.057+0000 7f8516c25740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf73e850 0x55e3cf744d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.057+0000 7f8516c25740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e3cf747430 0x55e3cf737560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.057+0000 7f8516c25740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f850c1aedb0 con 0x55e3cf73df30 2024-04-21T23:08:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.057+0000 7f8516c25740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f850c1b1630 con 0x55e3cf73e850 2024-04-21T23:08:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.057+0000 7f8516c25740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f850c1af420 con 0x55e3cf747430 2024-04-21T23:08:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.057+0000 7f85125d6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73df30 0x55e3cf753080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.057+0000 7f85125d6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73df30 0x55e3cf753080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35826/0 (socket says 172.21.15.44:35826) 2024-04-21T23:08:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.057+0000 7f85125d6640 1 -- 172.21.15.44:0/3400548758 learned_addr learned my addr 172.21.15.44:0/3400548758 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.057+0000 7f85115d4640 1 --2- 172.21.15.44:0/3400548758 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf73e850 0x55e3cf744d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.057+0000 7f85125d6640 1 -- 172.21.15.44:0/3400548758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e3cf747430 msgr2=0x55e3cf737560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.057+0000 7f85125d6640 1 --2- 172.21.15.44:0/3400548758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e3cf747430 0x55e3cf737560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.057+0000 7f85125d6640 1 -- 172.21.15.44:0/3400548758 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf73e850 msgr2=0x55e3cf744d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.057+0000 7f85125d6640 1 --2- 172.21.15.44:0/3400548758 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf73e850 0x55e3cf744d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.057+0000 7f85125d6640 1 -- 172.21.15.44:0/3400548758 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3cf759da0 con 0x55e3cf73df30 2024-04-21T23:08:56.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.057+0000 7f85115d4640 1 --2- 172.21.15.44:0/3400548758 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf73e850 0x55e3cf744d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:08:56.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.057+0000 7f85125d6640 1 --2- 172.21.15.44:0/3400548758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73df30 0x55e3cf753080 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f85040147d0 tx=0x7f85040096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=470a3266c0be5498 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:56.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.057+0000 7f84f77fe640 1 -- 172.21.15.44:0/3400548758 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8504004b20 con 0x55e3cf73df30 2024-04-21T23:08:56.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.058+0000 7f84f77fe640 1 -- 172.21.15.44:0/3400548758 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8504004cc0 con 0x55e3cf73df30 2024-04-21T23:08:56.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.058+0000 7f84f77fe640 1 -- 172.21.15.44:0/3400548758 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85040248f0 con 0x55e3cf73df30 2024-04-21T23:08:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.058+0000 7f8516c25740 1 -- 172.21.15.44:0/3400548758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73df30 msgr2=0x55e3cf753080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.058+0000 7f8516c25740 1 --2- 172.21.15.44:0/3400548758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73df30 0x55e3cf753080 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f85040147d0 tx=0x7f85040096d0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.059+0000 7f8516c25740 1 -- 172.21.15.44:0/3400548758 shutdown_connections 2024-04-21T23:08:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.059+0000 7f8516c25740 1 --2- 172.21.15.44:0/3400548758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e3cf747430 0x55e3cf737560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.059+0000 7f8516c25740 1 --2- 172.21.15.44:0/3400548758 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf73e850 0x55e3cf744d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.059+0000 7f8516c25740 1 --2- 172.21.15.44:0/3400548758 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf73df30 0x55e3cf753080 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.059+0000 7f8516c25740 1 -- 172.21.15.44:0/3400548758 >> 172.21.15.44:0/3400548758 conn(0x55e3cf738660 msgr2=0x55e3cf736e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.059+0000 7f8516c25740 1 -- 172.21.15.44:0/3400548758 shutdown_connections 2024-04-21T23:08:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.059+0000 7f8516c25740 1 -- 172.21.15.44:0/3400548758 wait complete. 2024-04-21T23:08:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.059+0000 7f8516c25740 1 -- start start 2024-04-21T23:08:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.059+0000 7f8516c25740 1 Processor -- start 2024-04-21T23:08:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.060+0000 7f8516c25740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e3cf73df30 0x55e3cf7410c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.060+0000 7f8516c25740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf73e850 0x55e3cf741600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.060+0000 7f8516c25740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf747430 0x55e3cf741b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.060+0000 7f8516c25740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f850c1afaf0 con 0x55e3cf747430 2024-04-21T23:08:56.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.060+0000 7f8516c25740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f850c1b0f60 con 0x55e3cf73e850 2024-04-21T23:08:56.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.060+0000 7f8516c25740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f850c1b01c0 con 0x55e3cf73df30 2024-04-21T23:08:56.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.060+0000 7f8516c25740 10 client.0 _ll_register_callbacks cb 0x7f85040098c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:08:56.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.060+0000 7f85125d6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf747430 0x55e3cf741b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.060+0000 7f85125d6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf747430 0x55e3cf741b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35832/0 (socket says 172.21.15.44:35832) 2024-04-21T23:08:56.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.060+0000 7f85125d6640 1 -- 172.21.15.44:0/1372827927 learned_addr learned my addr 172.21.15.44:0/1372827927 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:56.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.060+0000 7f85125d6640 1 --2- 172.21.15.44:0/1372827927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e3cf73df30 0x55e3cf7410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.060+0000 7f85115d4640 1 --2- 172.21.15.44:0/1372827927 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf73e850 0x55e3cf741600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.060+0000 7f85125d6640 1 -- 172.21.15.44:0/1372827927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e3cf73df30 msgr2=0x55e3cf7410c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:56.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.060+0000 7f85125d6640 1 --2- 172.21.15.44:0/1372827927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e3cf73df30 0x55e3cf7410c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.060+0000 7f85125d6640 1 -- 172.21.15.44:0/1372827927 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf73e850 msgr2=0x55e3cf741600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:08:56.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.060+0000 7f85125d6640 1 --2- 172.21.15.44:0/1372827927 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf73e850 0x55e3cf741600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.060+0000 7f85125d6640 1 -- 172.21.15.44:0/1372827927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3cf7bc8a0 con 0x55e3cf747430 2024-04-21T23:08:56.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.060+0000 7f85125d6640 1 --2- 172.21.15.44:0/1372827927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf747430 0x55e3cf741b40 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x55e3cf742700 tx=0x7f8504017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb56bc242993104 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:56.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.060+0000 7f84f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:08:56.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.061+0000 7f84f57fa640 1 -- 172.21.15.44:0/1372827927 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8504024030 con 0x55e3cf747430 2024-04-21T23:08:56.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.061+0000 7f84f57fa640 1 -- 172.21.15.44:0/1372827927 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f85040241d0 con 0x55e3cf747430 2024-04-21T23:08:56.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.061+0000 7f8516c25740 1 -- 172.21.15.44:0/1372827927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8504004cc0 con 0x55e3cf747430 2024-04-21T23:08:56.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.061+0000 7f84f57fa640 1 -- 172.21.15.44:0/1372827927 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8504012b20 con 0x55e3cf747430 2024-04-21T23:08:56.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.061+0000 7f8516c25740 10 client.15630 fetch_fsmap learned FSMap version 21 2024-04-21T23:08:56.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.061+0000 7f8516c25740 1 -- 172.21.15.44:0/1372827927 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e3cf7bd9d0 con 0x55e3cf747430 2024-04-21T23:08:56.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.061+0000 7f84f57fa640 1 -- 172.21.15.44:0/1372827927 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8504004cc0 con 0x55e3cf747430 2024-04-21T23:08:56.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.061+0000 7f84f57fa640 1 -- 172.21.15.44:0/1372827927 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f850400f040 con 0x55e3cf747430 2024-04-21T23:08:56.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.061+0000 7f8516c25740 10 client.15630 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:08:56.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.061+0000 7f8516c25740 10 client.15630 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:08:56.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.061+0000 7f8516c25740 20 client.15630 populate_metadata read hostname 'smithi044' 2024-04-21T23:08:56.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.061+0000 7f8516c25740 1 --2- 172.21.15.44:0/1372827927 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e3cf757640 0x55e3cf7c0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.061+0000 7f8516c25740 4 client.15630 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:08:56.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.061+0000 7f8516c25740 1 -- 172.21.15.44:0/1372827927 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e3cf6a9df0 con 0x55e3cf757640 2024-04-21T23:08:56.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.062+0000 7f85115d4640 1 --2- 172.21.15.44:0/1372827927 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e3cf757640 0x55e3cf7c0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.062+0000 7f84f57fa640 10 client.15630 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:08:56.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.062+0000 7f85115d4640 1 --2- 172.21.15.44:0/1372827927 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e3cf757640 0x55e3cf7c0080 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=745bab420b2406ad server_cookie=20ac38ce92dabc01 in_seq=0 out_seq=0 2024-04-21T23:08:56.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.062+0000 7f84f57fa640 1 -- 172.21.15.44:0/1372827927 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e3cf6a9df0 con 0x55e3cf757640 2024-04-21T23:08:56.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.062+0000 7f84f57fa640 10 client.15630 handle_command_reply: tid=0 2024-04-21T23:08:56.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.062+0000 7f8516c25740 1 client.15630 shutdown 2024-04-21T23:08:56.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.062+0000 7f8516c25740 1 -- 172.21.15.44:0/1372827927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf747430 msgr2=0x55e3cf741b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.062+0000 7f8516c25740 1 --2- 172.21.15.44:0/1372827927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf747430 0x55e3cf741b40 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x55e3cf742700 tx=0x7f8504017040 comp rx=0 tx=0).stop 2024-04-21T23:08:56.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.062+0000 7f8516c25740 1 -- 172.21.15.44:0/1372827927 shutdown_connections 2024-04-21T23:08:56.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.062+0000 7f8516c25740 1 --2- 172.21.15.44:0/1372827927 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e3cf757640 0x55e3cf7c0080 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.062+0000 7f8516c25740 1 --2- 172.21.15.44:0/1372827927 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e3cf747430 0x55e3cf741b40 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.062+0000 7f8516c25740 1 --2- 172.21.15.44:0/1372827927 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e3cf73e850 0x55e3cf741600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.062+0000 7f8516c25740 1 --2- 172.21.15.44:0/1372827927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e3cf73df30 0x55e3cf7410c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.062+0000 7f84f57fa640 0 client.15630 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:08:56.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.062+0000 7f8516c25740 1 -- 172.21.15.44:0/1372827927 >> 172.21.15.44:0/1372827927 conn(0x55e3cf738660 msgr2=0x55e3cf7376e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:56.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.062+0000 7f8516c25740 1 -- 172.21.15.44:0/1372827927 shutdown_connections 2024-04-21T23:08:56.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.062+0000 7f8516c25740 1 -- 172.21.15.44:0/1372827927 wait complete. 2024-04-21T23:08:56.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.063+0000 7f8516c25740 20 client.15630 trim_cache size 0 max 16384 2024-04-21T23:08:56.070 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:08:56.070 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:08:56.070 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-21T23:08:56.070 INFO:teuthology.orchestra.run.smithi044.stdout: "8b5bb175-75c0-40a3-abff-d9c051ff80ad": { 2024-04-21T23:08:56.070 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-21T23:08:56.070 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "8b5bb175-75c0-40a3-abff-d9c051ff80ad", 2024-04-21T23:08:56.070 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-04-21T23:08:56.070 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:08:56.071 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:08:56.071 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:08:56.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.063+0000 7f8514039640 1 -- 172.21.15.44:0/2084417631 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f84f80a2d90 msgr2=0x7f84f80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.063+0000 7f8514039640 1 --2- 172.21.15.44:0/2084417631 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f84f80a2d90 0x7f84f80a5250 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f85080cc9e0 tx=0x7f8508005300 comp rx=0 tx=0).stop 2024-04-21T23:08:56.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.063+0000 7f8514039640 1 -- 172.21.15.44:0/2084417631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f850c073b70 msgr2=0x7f850c19a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.063+0000 7f8514039640 1 --2- 172.21.15.44:0/2084417631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f850c073b70 0x7f850c19a250 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f850800c890 tx=0x7f850800cd50 comp rx=0 tx=0).stop 2024-04-21T23:08:56.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.063+0000 7f8514039640 1 -- 172.21.15.44:0/2084417631 shutdown_connections 2024-04-21T23:08:56.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.063+0000 7f8514039640 1 --2- 172.21.15.44:0/2084417631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f850c10df50 0x7f850c19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.063+0000 7f8514039640 1 --2- 172.21.15.44:0/2084417631 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f84f80a2d90 0x7f84f80a5250 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.063+0000 7f8514039640 1 --2- 172.21.15.44:0/2084417631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f850c074490 0x7f850c19a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.063+0000 7f8514039640 1 --2- 172.21.15.44:0/2084417631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f850c073b70 0x7f850c19a250 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.063+0000 7f8514039640 1 -- 172.21.15.44:0/2084417631 >> 172.21.15.44:0/2084417631 conn(0x7f850c0fd200 msgr2=0x7f850c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:56.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.063+0000 7f8514039640 1 -- 172.21.15.44:0/2084417631 shutdown_connections 2024-04-21T23:08:56.073 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.063+0000 7f8514039640 1 -- 172.21.15.44:0/2084417631 wait complete. 2024-04-21T23:08:56.073 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: Updating smithi044:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:08:56.073 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: Updating smithi055:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:08:56.073 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: Updating smithi069:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:08:56.073 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:08:56.073 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:08:56.073 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:08:56.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:08:56.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: Updating smithi055:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:08:56.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: Updating smithi069:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:08:56.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: Updating smithi044:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:08:56.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2743516434' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:56.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-04-21T23:08:56.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.074 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:08:56.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:08:56.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:08:56.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3589828188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:56.075 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:55 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2940025522' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:56.076 INFO:tasks.cephfs.filesystem:scrub status for tag:8b5bb175-75c0-40a3-abff-d9c051ff80ad - {'path': '/', 'tag': '8b5bb175-75c0-40a3-abff-d9c051ff80ad', 'options': 'recursive,force'} 2024-04-21T23:08:56.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.104+0000 7fedd3c20640 1 -- 172.21.15.44:0/391156657 --> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fedcc106ee0 con 0x7fedb80a2d80 2024-04-21T23:08:56.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.165+0000 7fedc2ffd640 1 -- 172.21.15.44:0/391156657 <== mgr.15396 v2:172.21.15.44:6800/3789150048 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fedc800bc40 con 0x7fedb80a2d80 2024-04-21T23:08:56.165 INFO:teuthology.orchestra.run.smithi044.stdout:Schedule set for path / 2024-04-21T23:08:56.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.167+0000 7fedd3c20640 1 -- 172.21.15.44:0/391156657 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fedb80a2d80 msgr2=0x7fedb80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.167+0000 7fedd3c20640 1 --2- 172.21.15.44:0/391156657 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fedb80a2d80 0x7fedb80a5240 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fedc8005d00 tx=0x7fedc800b410 comp rx=0 tx=0).stop 2024-04-21T23:08:56.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.167+0000 7fedd3c20640 1 -- 172.21.15.44:0/391156657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedcc10bd80 msgr2=0x7fedcc19c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.167+0000 7fedd3c20640 1 --2- 172.21.15.44:0/391156657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedcc10bd80 0x7fedcc19c880 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fedc4004af0 tx=0x7fedc4035d80 comp rx=0 tx=0).stop 2024-04-21T23:08:56.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.167+0000 7fedd3c20640 1 -- 172.21.15.44:0/391156657 shutdown_connections 2024-04-21T23:08:56.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.167+0000 7fedd3c20640 1 --2- 172.21.15.44:0/391156657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedcc1032d0 0x7fedcc198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.168+0000 7fedd3c20640 1 --2- 172.21.15.44:0/391156657 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fedb80a2d80 0x7fedb80a5240 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.168+0000 7fedd3c20640 1 --2- 172.21.15.44:0/391156657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fedcc10bd80 0x7fedcc19c880 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.168+0000 7fedd3c20640 1 --2- 172.21.15.44:0/391156657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fedcc1029b0 0x7fedcc198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.168+0000 7fedd3c20640 1 -- 172.21.15.44:0/391156657 >> 172.21.15.44:0/391156657 conn(0x7fedcc077760 msgr2=0x7fedcc0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:56.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.168+0000 7fedd3c20640 1 -- 172.21.15.44:0/391156657 shutdown_connections 2024-04-21T23:08:56.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.168+0000 7fedd3c20640 1 -- 172.21.15.44:0/391156657 wait complete. 2024-04-21T23:08:56.178 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-21T23:08:56.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.324+0000 7f68460ae640 1 -- 172.21.15.44:0/1709226924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68401006a0 msgr2=0x7f6840100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.324+0000 7f68460ae640 1 --2- 172.21.15.44:0/1709226924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68401006a0 0x7f6840100b20 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f6834009930 tx=0x7f683402bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:56.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.324+0000 7f68460ae640 1 -- 172.21.15.44:0/1709226924 shutdown_connections 2024-04-21T23:08:56.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.324+0000 7f68460ae640 1 --2- 172.21.15.44:0/1709226924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6840101060 0x7f684010d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.324+0000 7f68460ae640 1 --2- 172.21.15.44:0/1709226924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f68401006a0 0x7f6840100b20 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.324+0000 7f68460ae640 1 --2- 172.21.15.44:0/1709226924 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6840106660 0x7f6840106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.324+0000 7f68460ae640 1 -- 172.21.15.44:0/1709226924 >> 172.21.15.44:0/1709226924 conn(0x7f6840077790 msgr2=0x7f6840077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:56.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.324+0000 7f68460ae640 1 -- 172.21.15.44:0/1709226924 shutdown_connections 2024-04-21T23:08:56.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.324+0000 7f68460ae640 1 -- 172.21.15.44:0/1709226924 wait complete. 2024-04-21T23:08:56.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.324+0000 7f68460ae640 1 Processor -- start 2024-04-21T23:08:56.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.324+0000 7f68460ae640 1 -- start start 2024-04-21T23:08:56.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.325+0000 7f68460ae640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68401006a0 0x7f6840198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.325+0000 7f68460ae640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6840101060 0x7f6840198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.325+0000 7f68460ae640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6840106660 0x7f684019c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.325+0000 7f68460ae640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6840110a70 con 0x7f6840106660 2024-04-21T23:08:56.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.325+0000 7f68460ae640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f68401108f0 con 0x7f6840101060 2024-04-21T23:08:56.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.325+0000 7f68460ae640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6840110bf0 con 0x7f68401006a0 2024-04-21T23:08:56.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.325+0000 7f683ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6840106660 0x7f684019c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.325+0000 7f683ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6840106660 0x7f684019c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35858/0 (socket says 172.21.15.44:35858) 2024-04-21T23:08:56.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.325+0000 7f683ffff640 1 -- 172.21.15.44:0/9496225 learned_addr learned my addr 172.21.15.44:0/9496225 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:56.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.325+0000 7f683effd640 1 --2- 172.21.15.44:0/9496225 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6840101060 0x7f6840198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.325+0000 7f683f7fe640 1 --2- 172.21.15.44:0/9496225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68401006a0 0x7f6840198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.325+0000 7f683ffff640 1 -- 172.21.15.44:0/9496225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68401006a0 msgr2=0x7f6840198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.325+0000 7f683ffff640 1 --2- 172.21.15.44:0/9496225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68401006a0 0x7f6840198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.325+0000 7f683ffff640 1 -- 172.21.15.44:0/9496225 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6840101060 msgr2=0x7f6840198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.325+0000 7f683ffff640 1 --2- 172.21.15.44:0/9496225 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6840101060 0x7f6840198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.325+0000 7f683ffff640 1 -- 172.21.15.44:0/9496225 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f684019d040 con 0x7f6840106660 2024-04-21T23:08:56.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.325+0000 7f683ffff640 1 --2- 172.21.15.44:0/9496225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6840106660 0x7f684019c910 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f683800eaa0 tx=0x7f683800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:56.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.325+0000 7f683cff9640 1 -- 172.21.15.44:0/9496225 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683800cdb0 con 0x7f6840106660 2024-04-21T23:08:56.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.326+0000 7f68460ae640 1 -- 172.21.15.44:0/9496225 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f684019d330 con 0x7f6840106660 2024-04-21T23:08:56.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.326+0000 7f683cff9640 1 -- 172.21.15.44:0/9496225 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6838005020 con 0x7f6840106660 2024-04-21T23:08:56.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.326+0000 7f683cff9640 1 -- 172.21.15.44:0/9496225 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68380116d0 con 0x7f6840106660 2024-04-21T23:08:56.328 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.326+0000 7f68460ae640 1 -- 172.21.15.44:0/9496225 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68401a4bc0 con 0x7f6840106660 2024-04-21T23:08:56.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.327+0000 7f68460ae640 1 -- 172.21.15.44:0/9496225 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6840101d90 con 0x7f6840106660 2024-04-21T23:08:56.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.328+0000 7f683cff9640 1 -- 172.21.15.44:0/9496225 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f68380054d0 con 0x7f6840106660 2024-04-21T23:08:56.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.328+0000 7f683cff9640 1 --2- 172.21.15.44:0/9496225 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f68280a2d80 0x7f68280a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.329+0000 7f683f7fe640 1 --2- 172.21.15.44:0/9496225 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f68280a2d80 0x7f68280a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.329+0000 7f683cff9640 1 -- 172.21.15.44:0/9496225 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f68380cfb00 con 0x7f6840106660 2024-04-21T23:08:56.329 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.329+0000 7f683f7fe640 1 --2- 172.21.15.44:0/9496225 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f68280a2d80 0x7f68280a5240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f683000bb60 tx=0x7f683000b390 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:56.332 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.333+0000 7f683cff9640 1 -- 172.21.15.44:0/9496225 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6838016030 con 0x7f6840106660 2024-04-21T23:08:56.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: Updating smithi044:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:08:56.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: Updating smithi055:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:08:56.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: Updating smithi069:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:08:56.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:08:56.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:08:56.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:08:56.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:08:56.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: Updating smithi055:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:08:56.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: Updating smithi069:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:08:56.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: Updating smithi044:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:08:56.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2743516434' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:56.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-04-21T23:08:56.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:08:56.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:08:56.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:08:56.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3589828188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:56.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:55 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2940025522' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:56.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: Updating smithi044:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:08:56.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: Updating smithi055:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:08:56.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: Updating smithi069:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-21T23:08:56.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:08:56.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: Updating smithi044:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:08:56.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:08:56.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-21T23:08:56.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: Updating smithi055:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:08:56.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: Updating smithi069:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:08:56.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: Updating smithi044:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T23:08:56.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2743516434' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:08:56.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-04-21T23:08:56.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:08:56.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:08:56.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:08:56.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:08:56.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3589828188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:08:56.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:55 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2940025522' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:08:56.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.539+0000 7f68460ae640 1 -- 172.21.15.44:0/9496225 --> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f6840061b80 con 0x7f68280a2d80 2024-04-21T23:08:56.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.598+0000 7f683cff9640 1 -- 172.21.15.44:0/9496225 <== mgr.15396 v2:172.21.15.44:6800/3789150048 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f6840061b80 con 0x7f68280a2d80 2024-04-21T23:08:56.598 INFO:teuthology.orchestra.run.smithi044.stdout:Retention added to path / 2024-04-21T23:08:56.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.600+0000 7f68460ae640 1 -- 172.21.15.44:0/9496225 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f68280a2d80 msgr2=0x7f68280a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.600+0000 7f68460ae640 1 --2- 172.21.15.44:0/9496225 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f68280a2d80 0x7f68280a5240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f683000bb60 tx=0x7f683000b390 comp rx=0 tx=0).stop 2024-04-21T23:08:56.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.600+0000 7f68460ae640 1 -- 172.21.15.44:0/9496225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6840106660 msgr2=0x7f684019c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.600+0000 7f68460ae640 1 --2- 172.21.15.44:0/9496225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6840106660 0x7f684019c910 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f683800eaa0 tx=0x7f683800ef60 comp rx=0 tx=0).stop 2024-04-21T23:08:56.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.600+0000 7f68460ae640 1 -- 172.21.15.44:0/9496225 shutdown_connections 2024-04-21T23:08:56.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.600+0000 7f68460ae640 1 --2- 172.21.15.44:0/9496225 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f68280a2d80 0x7f68280a5240 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.600+0000 7f68460ae640 1 --2- 172.21.15.44:0/9496225 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6840106660 0x7f684019c910 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.600+0000 7f68460ae640 1 --2- 172.21.15.44:0/9496225 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6840101060 0x7f6840198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.600+0000 7f68460ae640 1 --2- 172.21.15.44:0/9496225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68401006a0 0x7f6840198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.600+0000 7f68460ae640 1 -- 172.21.15.44:0/9496225 >> 172.21.15.44:0/9496225 conn(0x7f6840077790 msgr2=0x7f684006a0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:56.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.600+0000 7f68460ae640 1 -- 172.21.15.44:0/9496225 shutdown_connections 2024-04-21T23:08:56.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.601+0000 7f68460ae640 1 -- 172.21.15.44:0/9496225 wait complete. 2024-04-21T23:08:56.612 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-21T23:08:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.715+0000 7f39ae1c8640 1 -- 172.21.15.44:0/466477258 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39a8100fe0 msgr2=0x7f39a810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.715+0000 7f39ae1c8640 1 --2- 172.21.15.44:0/466477258 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39a8100fe0 0x7f39a810d4f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f39a0009cd0 tx=0x7f39a002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.715+0000 7f39ae1c8640 1 -- 172.21.15.44:0/466477258 shutdown_connections 2024-04-21T23:08:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.715+0000 7f39ae1c8640 1 --2- 172.21.15.44:0/466477258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39a8100620 0x7f39a8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.715+0000 7f39ae1c8640 1 --2- 172.21.15.44:0/466477258 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39a8100fe0 0x7f39a810d4f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.715+0000 7f39ae1c8640 1 --2- 172.21.15.44:0/466477258 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39a8106630 0x7f39a8106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.715+0000 7f39ae1c8640 1 -- 172.21.15.44:0/466477258 >> 172.21.15.44:0/466477258 conn(0x7f39a8077760 msgr2=0x7f39a8077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.715+0000 7f39ae1c8640 1 -- 172.21.15.44:0/466477258 shutdown_connections 2024-04-21T23:08:56.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.715+0000 7f39ae1c8640 1 -- 172.21.15.44:0/466477258 wait complete. 2024-04-21T23:08:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.715+0000 7f39ae1c8640 1 Processor -- start 2024-04-21T23:08:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.715+0000 7f39ae1c8640 1 -- start start 2024-04-21T23:08:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39ae1c8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39a8100620 0x7f39a8196fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39ae1c8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39a8100fe0 0x7f39a81974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39ae1c8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39a8106630 0x7f39a819b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39ae1c8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f39a8110a60 con 0x7f39a8100fe0 2024-04-21T23:08:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39ae1c8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f39a81108e0 con 0x7f39a8100620 2024-04-21T23:08:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39ae1c8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f39a8110be0 con 0x7f39a8106630 2024-04-21T23:08:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39a6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39a8100fe0 0x7f39a81974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39a6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39a8100fe0 0x7f39a81974e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35872/0 (socket says 172.21.15.44:35872) 2024-04-21T23:08:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39a6ffd640 1 -- 172.21.15.44:0/1573227932 learned_addr learned my addr 172.21.15.44:0/1573227932 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:56.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39a7fff640 1 --2- 172.21.15.44:0/1573227932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39a8106630 0x7f39a819b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39a77fe640 1 --2- 172.21.15.44:0/1573227932 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39a8100620 0x7f39a8196fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39a6ffd640 1 -- 172.21.15.44:0/1573227932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39a8106630 msgr2=0x7f39a819b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39a6ffd640 1 --2- 172.21.15.44:0/1573227932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39a8106630 0x7f39a819b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39a6ffd640 1 -- 172.21.15.44:0/1573227932 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39a8100620 msgr2=0x7f39a8196fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39a6ffd640 1 --2- 172.21.15.44:0/1573227932 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39a8100620 0x7f39a8196fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39a6ffd640 1 -- 172.21.15.44:0/1573227932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39a819bf90 con 0x7f39a8100fe0 2024-04-21T23:08:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39a77fe640 1 --2- 172.21.15.44:0/1573227932 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39a8100620 0x7f39a8196fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:08:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39a6ffd640 1 --2- 172.21.15.44:0/1573227932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39a8100fe0 0x7f39a81974e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f399c00c970 tx=0x7f399c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39a4ff9640 1 -- 172.21.15.44:0/1573227932 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399c007bb0 con 0x7f39a8100fe0 2024-04-21T23:08:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39ae1c8640 1 -- 172.21.15.44:0/1573227932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39a819c280 con 0x7f39a8100fe0 2024-04-21T23:08:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39a4ff9640 1 -- 172.21.15.44:0/1573227932 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f399c007d50 con 0x7f39a8100fe0 2024-04-21T23:08:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39a4ff9640 1 -- 172.21.15.44:0/1573227932 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399c00f450 con 0x7f39a8100fe0 2024-04-21T23:08:56.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.716+0000 7f39ae1c8640 1 -- 172.21.15.44:0/1573227932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39a8068f60 con 0x7f39a8100fe0 2024-04-21T23:08:56.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.717+0000 7f39ae1c8640 1 -- 172.21.15.44:0/1573227932 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39a804fd50 con 0x7f39a8100fe0 2024-04-21T23:08:56.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.718+0000 7f39a4ff9640 1 -- 172.21.15.44:0/1573227932 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f399c005020 con 0x7f39a8100fe0 2024-04-21T23:08:56.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.719+0000 7f39a4ff9640 1 --2- 172.21.15.44:0/1573227932 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f39900a2d80 0x7f39900a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:56.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.719+0000 7f39a77fe640 1 --2- 172.21.15.44:0/1573227932 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f39900a2d80 0x7f39900a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:56.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.719+0000 7f39a4ff9640 1 -- 172.21.15.44:0/1573227932 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f399c0cf820 con 0x7f39a8100fe0 2024-04-21T23:08:56.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.720+0000 7f39a77fe640 1 --2- 172.21.15.44:0/1573227932 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f39900a2d80 0x7f39900a5240 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3998006fd0 tx=0x7f3998008040 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:56.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.723+0000 7f39a4ff9640 1 -- 172.21.15.44:0/1573227932 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f399c014030 con 0x7f39a8100fe0 2024-04-21T23:08:56.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.927+0000 7f39ae1c8640 1 -- 172.21.15.44:0/1573227932 --> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f39a8069220 con 0x7f39900a2d80 2024-04-21T23:08:56.935 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.935+0000 7f39a4ff9640 1 -- 172.21.15.44:0/1573227932 <== mgr.15396 v2:172.21.15.44:6800/3789150048 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f39a8069220 con 0x7f39900a2d80 2024-04-21T23:08:56.935 INFO:teuthology.orchestra.run.smithi044.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-21T00:00:00", "created": "2024-04-21T23:08:56", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-21T23:08:56.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.937+0000 7f39ae1c8640 1 -- 172.21.15.44:0/1573227932 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f39900a2d80 msgr2=0x7f39900a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.937+0000 7f39ae1c8640 1 --2- 172.21.15.44:0/1573227932 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f39900a2d80 0x7f39900a5240 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3998006fd0 tx=0x7f3998008040 comp rx=0 tx=0).stop 2024-04-21T23:08:56.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.937+0000 7f39ae1c8640 1 -- 172.21.15.44:0/1573227932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39a8100fe0 msgr2=0x7f39a81974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:56.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.937+0000 7f39ae1c8640 1 --2- 172.21.15.44:0/1573227932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39a8100fe0 0x7f39a81974e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f399c00c970 tx=0x7f399c00ce30 comp rx=0 tx=0).stop 2024-04-21T23:08:56.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.937+0000 7f39ae1c8640 1 -- 172.21.15.44:0/1573227932 shutdown_connections 2024-04-21T23:08:56.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.937+0000 7f39ae1c8640 1 --2- 172.21.15.44:0/1573227932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39a8106630 0x7f39a819b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.937+0000 7f39ae1c8640 1 --2- 172.21.15.44:0/1573227932 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f39900a2d80 0x7f39900a5240 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.937+0000 7f39ae1c8640 1 --2- 172.21.15.44:0/1573227932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f39a8100fe0 0x7f39a81974e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.937+0000 7f39ae1c8640 1 --2- 172.21.15.44:0/1573227932 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39a8100620 0x7f39a8196fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:56.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.937+0000 7f39ae1c8640 1 -- 172.21.15.44:0/1573227932 >> 172.21.15.44:0/1573227932 conn(0x7f39a8077760 msgr2=0x7f39a8103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:56.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.937+0000 7f39ae1c8640 1 -- 172.21.15.44:0/1573227932 shutdown_connections 2024-04-21T23:08:56.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:56.937+0000 7f39ae1c8640 1 -- 172.21.15.44:0/1573227932 wait complete. 2024-04-21T23:08:56.948 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-21T23:08:57.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.053+0000 7f370c35d640 1 -- 172.21.15.44:0/3995258374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37041006a0 msgr2=0x7f3704100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:57.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.053+0000 7f370c35d640 1 --2- 172.21.15.44:0/3995258374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37041006a0 0x7f3704100b20 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f36f4009930 tx=0x7f36f402bc30 comp rx=0 tx=0).stop 2024-04-21T23:08:57.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.053+0000 7f370c35d640 1 -- 172.21.15.44:0/3995258374 shutdown_connections 2024-04-21T23:08:57.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.053+0000 7f370c35d640 1 --2- 172.21.15.44:0/3995258374 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3704101060 0x7f370410d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:57.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.053+0000 7f370c35d640 1 --2- 172.21.15.44:0/3995258374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f37041006a0 0x7f3704100b20 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:57.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.053+0000 7f370c35d640 1 --2- 172.21.15.44:0/3995258374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3704106660 0x7f3704106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:57.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.053+0000 7f370c35d640 1 -- 172.21.15.44:0/3995258374 >> 172.21.15.44:0/3995258374 conn(0x7f3704077790 msgr2=0x7f3704077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:57.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.053+0000 7f370c35d640 1 -- 172.21.15.44:0/3995258374 shutdown_connections 2024-04-21T23:08:57.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.053+0000 7f370c35d640 1 -- 172.21.15.44:0/3995258374 wait complete. 2024-04-21T23:08:57.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.053+0000 7f370c35d640 1 Processor -- start 2024-04-21T23:08:57.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.054+0000 7f370c35d640 1 -- start start 2024-04-21T23:08:57.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.054+0000 7f370c35d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37041006a0 0x7f3704198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:57.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.054+0000 7f370c35d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3704101060 0x7f3704199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:57.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.054+0000 7f370c35d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3704106660 0x7f37041a3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:57.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.054+0000 7f370c35d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3704110a70 con 0x7f3704101060 2024-04-21T23:08:57.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.054+0000 7f370c35d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f37041108f0 con 0x7f3704106660 2024-04-21T23:08:57.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.054+0000 7f370c35d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3704110bf0 con 0x7f37041006a0 2024-04-21T23:08:57.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.054+0000 7f37098f8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3704101060 0x7f3704199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:57.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.054+0000 7f37098f8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3704101060 0x7f3704199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35896/0 (socket says 172.21.15.44:35896) 2024-04-21T23:08:57.060 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.054+0000 7f37098f8640 1 -- 172.21.15.44:0/1823190875 learned_addr learned my addr 172.21.15.44:0/1823190875 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:08:57.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.054+0000 7f370a0f9640 1 --2- 172.21.15.44:0/1823190875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37041006a0 0x7f3704198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:57.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.054+0000 7f370a8fa640 1 --2- 172.21.15.44:0/1823190875 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3704106660 0x7f37041a3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:57.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.054+0000 7f37098f8640 1 -- 172.21.15.44:0/1823190875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37041006a0 msgr2=0x7f3704198f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:57.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.054+0000 7f37098f8640 1 --2- 172.21.15.44:0/1823190875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37041006a0 0x7f3704198f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:57.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.054+0000 7f37098f8640 1 -- 172.21.15.44:0/1823190875 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3704106660 msgr2=0x7f37041a3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:57.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.054+0000 7f37098f8640 1 --2- 172.21.15.44:0/1823190875 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3704106660 0x7f37041a3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:57.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.054+0000 7f37098f8640 1 -- 172.21.15.44:0/1823190875 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37041a44d0 con 0x7f3704101060 2024-04-21T23:08:57.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.054+0000 7f37098f8640 1 --2- 172.21.15.44:0/1823190875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3704101060 0x7f3704199450 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f36f4009230 tx=0x7f36f402cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:57.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.055+0000 7f36fb7fe640 1 -- 172.21.15.44:0/1823190875 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f4041070 con 0x7f3704101060 2024-04-21T23:08:57.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.055+0000 7f36fb7fe640 1 -- 172.21.15.44:0/1823190875 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36f4004450 con 0x7f3704101060 2024-04-21T23:08:57.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.055+0000 7f370c35d640 1 -- 172.21.15.44:0/1823190875 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37041a46a0 con 0x7f3704101060 2024-04-21T23:08:57.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.055+0000 7f36fb7fe640 1 -- 172.21.15.44:0/1823190875 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f4034440 con 0x7f3704101060 2024-04-21T23:08:57.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.055+0000 7f370c35d640 1 -- 172.21.15.44:0/1823190875 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37041a4aa0 con 0x7f3704101060 2024-04-21T23:08:57.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.056+0000 7f370c35d640 1 -- 172.21.15.44:0/1823190875 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3704101d90 con 0x7f3704101060 2024-04-21T23:08:57.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.057+0000 7f36fb7fe640 1 -- 172.21.15.44:0/1823190875 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f36f4044070 con 0x7f3704101060 2024-04-21T23:08:57.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.058+0000 7f36fb7fe640 1 --2- 172.21.15.44:0/1823190875 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f36f00a2d80 0x7f36f00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:08:57.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.058+0000 7f370a0f9640 1 --2- 172.21.15.44:0/1823190875 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f36f00a2d80 0x7f36f00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:08:57.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.058+0000 7f36fb7fe640 1 -- 172.21.15.44:0/1823190875 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f36f40f1680 con 0x7f3704101060 2024-04-21T23:08:57.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.058+0000 7f370a0f9640 1 --2- 172.21.15.44:0/1823190875 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f36f00a2d80 0x7f36f00a5240 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3700005fd0 tx=0x7f370000b390 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:08:57.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.061+0000 7f36fb7fe640 1 -- 172.21.15.44:0/1823190875 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f36f40bd850 con 0x7f3704101060 2024-04-21T23:08:57.250 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:56 smithi044 ceph-mon[27934]: scrub queued for path: / 2024-04-21T23:08:57.250 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:56 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:08:57.250 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:56 smithi044 ceph-mon[27934]: scrub summary: active paths [/] 2024-04-21T23:08:57.250 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:56 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:08:57.250 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:56 smithi044 ceph-mon[27934]: from='client.15597 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:57.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.272+0000 7f370c35d640 1 -- 172.21.15.44:0/1823190875 --> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f3704061b80 con 0x7f36f00a2d80 2024-04-21T23:08:57.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.279+0000 7f36fb7fe640 1 -- 172.21.15.44:0/1823190875 <== mgr.15396 v2:172.21.15.44:6800/3789150048 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f3704061b80 con 0x7f36f00a2d80 2024-04-21T23:08:57.279 INFO:teuthology.orchestra.run.smithi044.stdout:/ 1m 6m3h 2024-04-21T23:08:57.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.281+0000 7f370c35d640 1 -- 172.21.15.44:0/1823190875 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f36f00a2d80 msgr2=0x7f36f00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:57.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.281+0000 7f370c35d640 1 --2- 172.21.15.44:0/1823190875 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f36f00a2d80 0x7f36f00a5240 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3700005fd0 tx=0x7f370000b390 comp rx=0 tx=0).stop 2024-04-21T23:08:57.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.281+0000 7f370c35d640 1 -- 172.21.15.44:0/1823190875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3704101060 msgr2=0x7f3704199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:08:57.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.281+0000 7f370c35d640 1 --2- 172.21.15.44:0/1823190875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3704101060 0x7f3704199450 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f36f4009230 tx=0x7f36f402cc70 comp rx=0 tx=0).stop 2024-04-21T23:08:57.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.281+0000 7f370c35d640 1 -- 172.21.15.44:0/1823190875 shutdown_connections 2024-04-21T23:08:57.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.281+0000 7f370c35d640 1 --2- 172.21.15.44:0/1823190875 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f36f00a2d80 0x7f36f00a5240 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:57.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.281+0000 7f370c35d640 1 --2- 172.21.15.44:0/1823190875 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3704106660 0x7f37041a3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:57.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.281+0000 7f370c35d640 1 --2- 172.21.15.44:0/1823190875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3704101060 0x7f3704199450 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:57.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.281+0000 7f370c35d640 1 --2- 172.21.15.44:0/1823190875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37041006a0 0x7f3704198f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:08:57.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.281+0000 7f370c35d640 1 -- 172.21.15.44:0/1823190875 >> 172.21.15.44:0/1823190875 conn(0x7f3704077790 msgr2=0x7f3704102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:08:57.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.281+0000 7f370c35d640 1 -- 172.21.15.44:0/1823190875 shutdown_connections 2024-04-21T23:08:57.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:08:57.281+0000 7f370c35d640 1 -- 172.21.15.44:0/1823190875 wait complete. 2024-04-21T23:08:57.292 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-21T23:08:57.317 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-21T23:08:57.330 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-21T23:08:57.363 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-04-21T23:08:57.363 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-21T23:08:57.366 INFO:teuthology.run_tasks:Running task workunit... 2024-04-21T23:08:57.383 INFO:tasks.workunit:Pulling workunits from ref 430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-21T23:08:57.383 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-21T23:08:57.383 INFO:tasks.workunit:timeout=3h 2024-04-21T23:08:57.383 INFO:tasks.workunit:cleanup=True 2024-04-21T23:08:57.384 DEBUG:teuthology.orchestra.run.smithi044:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-21T23:08:57.399 INFO:teuthology.orchestra.run.smithi044.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-21T23:08:57.400 INFO:teuthology.orchestra.run.smithi044.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-21T23:08:57.400 INFO:teuthology.orchestra.run.smithi044.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-21T23:08:57.400 INFO:teuthology.orchestra.run.smithi044.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T23:08:57.400 INFO:teuthology.orchestra.run.smithi044.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-21T23:08:57.400 INFO:teuthology.orchestra.run.smithi044.stdout:Access: 2024-04-21 23:06:52.765638751 +0000 2024-04-21T23:08:57.400 INFO:teuthology.orchestra.run.smithi044.stdout:Modify: 2024-04-21 23:06:52.765638751 +0000 2024-04-21T23:08:57.400 INFO:teuthology.orchestra.run.smithi044.stdout:Change: 2024-04-21 23:08:48.177011771 +0000 2024-04-21T23:08:57.400 INFO:teuthology.orchestra.run.smithi044.stdout: Birth: 2024-04-21 23:06:52.765638751 +0000 2024-04-21T23:08:57.400 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-21T23:08:57.400 DEBUG:teuthology.orchestra.run.smithi044:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-21T23:08:57.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:56 smithi055 ceph-mon[31117]: scrub queued for path: / 2024-04-21T23:08:57.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:56 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:08:57.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:56 smithi055 ceph-mon[31117]: scrub summary: active paths [/] 2024-04-21T23:08:57.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:56 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:08:57.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:56 smithi055 ceph-mon[31117]: from='client.15597 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:57.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:56 smithi069 ceph-mon[31816]: scrub queued for path: / 2024-04-21T23:08:57.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:56 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:08:57.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:56 smithi069 ceph-mon[31816]: scrub summary: active paths [/] 2024-04-21T23:08:57.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:56 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:08:57.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:56 smithi069 ceph-mon[31816]: from='client.15597 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:57.471 DEBUG:teuthology.orchestra.run.smithi044:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-21T23:08:57.528 INFO:tasks.workunit.client.0.smithi044.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-21T23:08:58.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:57 smithi044 ceph-mon[27934]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:58.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:57 smithi044 ceph-mon[27934]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:08:58.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:57 smithi044 ceph-mon[27934]: from='client.15642 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:58.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:57 smithi044 ceph-mon[27934]: scrub summary: idle 2024-04-21T23:08:58.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:57 smithi055 ceph-mon[31117]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:58.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:57 smithi055 ceph-mon[31117]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:08:58.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:57 smithi055 ceph-mon[31117]: from='client.15642 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:58.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:57 smithi055 ceph-mon[31117]: scrub summary: idle 2024-04-21T23:08:58.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:57 smithi069 ceph-mon[31816]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:58.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:57 smithi069 ceph-mon[31816]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T23:08:58.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:57 smithi069 ceph-mon[31816]: from='client.15642 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:58.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:57 smithi069 ceph-mon[31816]: scrub summary: idle 2024-04-21T23:08:59.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:08:58 smithi044 ceph-mon[27934]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:59.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:08:59 smithi055 ceph-mon[31117]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:08:59.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:08:59 smithi069 ceph-mon[31816]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:09:00.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:00 smithi044 ceph-mon[27934]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 131 KiB/s wr, 68 op/s 2024-04-21T23:09:00.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:00 smithi055 ceph-mon[31117]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 131 KiB/s wr, 68 op/s 2024-04-21T23:09:00.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:00 smithi069 ceph-mon[31816]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 131 KiB/s wr, 68 op/s 2024-04-21T23:09:00.935 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:09:00 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:09:00.609+0000 7fef3d421700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:09:00.935 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:09:00 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:09:00.609+0000 7fef3d421700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:09:00.935 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:09:00 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:09:00.609+0000 7fef3d421700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:09:00.935 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:09:00 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:09:00.609+0000 7fef3d421700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:09:00.936 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:09:00 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:09:00.609+0000 7fef3d421700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T23:09:02.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:02 smithi044 ceph-mon[27934]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 102 KiB/s wr, 53 op/s 2024-04-21T23:09:02.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:02 smithi055 ceph-mon[31117]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 102 KiB/s wr, 53 op/s 2024-04-21T23:09:02.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:02 smithi069 ceph-mon[31816]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 102 KiB/s wr, 53 op/s 2024-04-21T23:09:04.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:04 smithi044 ceph-mon[27934]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 108 KiB/s wr, 64 op/s 2024-04-21T23:09:04.461 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:04 smithi044 ceph-mon[27934]: mgrmap e24: x(active, since 12s), standbys: y, z 2024-04-21T23:09:04.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:04 smithi055 ceph-mon[31117]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 108 KiB/s wr, 64 op/s 2024-04-21T23:09:04.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:04 smithi055 ceph-mon[31117]: mgrmap e24: x(active, since 12s), standbys: y, z 2024-04-21T23:09:04.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:04 smithi069 ceph-mon[31816]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 108 KiB/s wr, 64 op/s 2024-04-21T23:09:04.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:04 smithi069 ceph-mon[31816]: mgrmap e24: x(active, since 12s), standbys: y, z 2024-04-21T23:09:06.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:06 smithi044 ceph-mon[27934]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 99 KiB/s wr, 58 op/s 2024-04-21T23:09:06.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:06 smithi055 ceph-mon[31117]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 99 KiB/s wr, 58 op/s 2024-04-21T23:09:06.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:06 smithi069 ceph-mon[31816]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 99 KiB/s wr, 58 op/s 2024-04-21T23:09:08.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:08 smithi044 ceph-mon[27934]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 99 KiB/s wr, 58 op/s 2024-04-21T23:09:08.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:08 smithi055 ceph-mon[31117]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 99 KiB/s wr, 58 op/s 2024-04-21T23:09:08.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:08 smithi069 ceph-mon[31816]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 99 KiB/s wr, 58 op/s 2024-04-21T23:09:10.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:10 smithi044 ceph-mon[27934]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 99 KiB/s wr, 59 op/s 2024-04-21T23:09:10.460 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:10 smithi069 ceph-mon[31816]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 99 KiB/s wr, 59 op/s 2024-04-21T23:09:10.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:10 smithi055 ceph-mon[31117]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 99 KiB/s wr, 59 op/s 2024-04-21T23:09:10.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.507+0000 7fdbbfd65640 1 -- 172.21.15.44:0/3200777717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdbb81029b0 msgr2=0x7fdbb8102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:10.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.507+0000 7fdbbfd65640 1 --2- 172.21.15.44:0/3200777717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdbb81029b0 0x7fdbb8102d90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4009900 tx=0x7fdbb402bcd0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.507+0000 7fdbbfd65640 1 -- 172.21.15.44:0/3200777717 shutdown_connections 2024-04-21T23:09:10.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.507+0000 7fdbbfd65640 1 --2- 172.21.15.44:0/3200777717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdbb810bd80 0x7fdbb810e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.507+0000 7fdbbfd65640 1 --2- 172.21.15.44:0/3200777717 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdbb81032d0 0x7fdbb810b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.507+0000 7fdbbfd65640 1 --2- 172.21.15.44:0/3200777717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdbb81029b0 0x7fdbb8102d90 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.507+0000 7fdbbfd65640 1 -- 172.21.15.44:0/3200777717 >> 172.21.15.44:0/3200777717 conn(0x7fdbb8077760 msgr2=0x7fdbb8077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:10.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.507+0000 7fdbbfd65640 1 -- 172.21.15.44:0/3200777717 shutdown_connections 2024-04-21T23:09:10.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.507+0000 7fdbbfd65640 1 -- 172.21.15.44:0/3200777717 wait complete. 2024-04-21T23:09:10.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.508+0000 7fdbbfd65640 1 Processor -- start 2024-04-21T23:09:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.508+0000 7fdbbfd65640 1 -- start start 2024-04-21T23:09:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.508+0000 7fdbbfd65640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdbb81029b0 0x7fdbb8198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.508+0000 7fdbbfd65640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdbb81032d0 0x7fdbb8198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.508+0000 7fdbbfd65640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdbb810bd80 0x7fdbb819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.508+0000 7fdbbfd65640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb8111b10 con 0x7fdbb81032d0 2024-04-21T23:09:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.508+0000 7fdbbfd65640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb8111990 con 0x7fdbb81029b0 2024-04-21T23:09:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.508+0000 7fdbbfd65640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb8111c90 con 0x7fdbb810bd80 2024-04-21T23:09:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.508+0000 7fdbbd300640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdbb81032d0 0x7fdbb8198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.508+0000 7fdbbd300640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdbb81032d0 0x7fdbb8198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52312/0 (socket says 172.21.15.44:52312) 2024-04-21T23:09:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.508+0000 7fdbbd300640 1 -- 172.21.15.44:0/3424884135 learned_addr learned my addr 172.21.15.44:0/3424884135 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.508+0000 7fdbbe302640 1 --2- 172.21.15.44:0/3424884135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdbb810bd80 0x7fdbb819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.509+0000 7fdbbdb01640 1 --2- 172.21.15.44:0/3424884135 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdbb81029b0 0x7fdbb8198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.509+0000 7fdbbd300640 1 -- 172.21.15.44:0/3424884135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdbb810bd80 msgr2=0x7fdbb819c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.509+0000 7fdbbd300640 1 --2- 172.21.15.44:0/3424884135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdbb810bd80 0x7fdbb819c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.509+0000 7fdbbd300640 1 -- 172.21.15.44:0/3424884135 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdbb81029b0 msgr2=0x7fdbb8198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.509+0000 7fdbbd300640 1 --2- 172.21.15.44:0/3424884135 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdbb81029b0 0x7fdbb8198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.509+0000 7fdbbd300640 1 -- 172.21.15.44:0/3424884135 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb819d000 con 0x7fdbb81032d0 2024-04-21T23:09:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.509+0000 7fdbbd300640 1 --2- 172.21.15.44:0/3424884135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdbb81032d0 0x7fdbb8198550 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fdba800c890 tx=0x7fdba800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.509+0000 7fdbaeffd640 1 -- 172.21.15.44:0/3424884135 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba8004220 con 0x7fdbb81032d0 2024-04-21T23:09:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.509+0000 7fdbaeffd640 1 -- 172.21.15.44:0/3424884135 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdba80043c0 con 0x7fdbb81032d0 2024-04-21T23:09:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.509+0000 7fdbbfd65640 1 -- 172.21.15.44:0/3424884135 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbb819d2f0 con 0x7fdbb81032d0 2024-04-21T23:09:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.509+0000 7fdbaeffd640 1 -- 172.21.15.44:0/3424884135 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba80049e0 con 0x7fdbb81032d0 2024-04-21T23:09:10.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.509+0000 7fdbbfd65640 1 -- 172.21.15.44:0/3424884135 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbb81a4b80 con 0x7fdbb81032d0 2024-04-21T23:09:10.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.511+0000 7fdbaeffd640 1 -- 172.21.15.44:0/3424884135 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7fdba80079b0 con 0x7fdbb81032d0 2024-04-21T23:09:10.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.512+0000 7fdbc2a2d740 1 Processor -- start 2024-04-21T23:09:10.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.512+0000 7fdbc2a2d740 1 -- start start 2024-04-21T23:09:10.517 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.512+0000 7fdbc2a2d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5089600 0x5619d50899e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.512+0000 7fdbc2a2d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5089f20 0x5619d50970e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.512+0000 7fdbc2a2d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5097620 0x5619d5099a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.512+0000 7fdbc2a2d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5619d4fedb70 con 0x5619d5089600 2024-04-21T23:09:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.512+0000 7fdbc2a2d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5619d4f96470 con 0x5619d5097620 2024-04-21T23:09:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.512+0000 7fdbc2a2d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5619d4fe9b30 con 0x5619d5089f20 2024-04-21T23:09:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.512+0000 7fdbaeffd640 1 --2- 172.21.15.44:0/3424884135 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fdba40a2e10 0x7fdba40a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.513+0000 7fdbbd300640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5097620 0x5619d5099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.513+0000 7fdbbd300640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5097620 0x5619d5099a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:53694/0 (socket says 172.21.15.44:53694) 2024-04-21T23:09:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.513+0000 7fdbbdb01640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5089f20 0x5619d50970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.513+0000 7fdbbd300640 1 -- 172.21.15.44:0/3127862064 learned_addr learned my addr 172.21.15.44:0/3127862064 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.513+0000 7fdbbe302640 1 --2- 172.21.15.44:0/3127862064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5089600 0x5619d50899e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.513+0000 7fdbbe302640 1 --2- 172.21.15.44:0/3424884135 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fdba40a2e10 0x7fdba40a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:10.518 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.513+0000 7fdbaeffd640 1 -- 172.21.15.44:0/3424884135 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fdba80cfb70 con 0x7fdbb81032d0 2024-04-21T23:09:10.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.513+0000 7fdbbdb01640 1 -- 172.21.15.44:0/3127862064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5097620 msgr2=0x5619d5099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:10.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.513+0000 7fdbbdb01640 1 --2- 172.21.15.44:0/3127862064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5097620 0x5619d5099a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.513+0000 7fdbbdb01640 1 -- 172.21.15.44:0/3127862064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5089600 msgr2=0x5619d50899e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:10.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.513+0000 7fdbbdb01640 1 --2- 172.21.15.44:0/3127862064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5089600 0x5619d50899e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.513+0000 7fdbbdb01640 1 -- 172.21.15.44:0/3127862064 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619d4d02210 con 0x5619d5089f20 2024-04-21T23:09:10.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.513+0000 7fdbbd300640 1 --2- 172.21.15.44:0/3127862064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5097620 0x5619d5099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:09:10.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.513+0000 7fdbbdb01640 1 --2- 172.21.15.44:0/3127862064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5089f20 0x5619d50970e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4009230 tx=0x7fdbb402c6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9793189e16205eb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:10.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.513+0000 7fdbbe302640 1 --2- 172.21.15.44:0/3424884135 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fdba40a2e10 0x7fdba40a52d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0009fd0 tx=0x7fdbb0009600 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:10.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.514+0000 7fdba3fff640 1 -- 172.21.15.44:0/3127862064 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4042070 con 0x5619d5089f20 2024-04-21T23:09:10.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.514+0000 7fdba3fff640 1 -- 172.21.15.44:0/3127862064 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbb402cba0 con 0x5619d5089f20 2024-04-21T23:09:10.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.514+0000 7fdba3fff640 1 -- 172.21.15.44:0/3127862064 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb40046c0 con 0x5619d5089f20 2024-04-21T23:09:10.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.514+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/3127862064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5089f20 msgr2=0x5619d50970e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:10.519 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.514+0000 7fdbc2a2d740 1 --2- 172.21.15.44:0/3127862064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5089f20 0x5619d50970e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4009230 tx=0x7fdbb402c6f0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.514+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/3127862064 shutdown_connections 2024-04-21T23:09:10.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.514+0000 7fdbc2a2d740 1 --2- 172.21.15.44:0/3127862064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5089f20 0x5619d50970e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.514+0000 7fdbc2a2d740 1 --2- 172.21.15.44:0/3127862064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5097620 0x5619d5099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.514+0000 7fdbc2a2d740 1 --2- 172.21.15.44:0/3127862064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5089600 0x5619d50899e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.514+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/3127862064 >> 172.21.15.44:0/3127862064 conn(0x5619d5084440 msgr2=0x5619d5084820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:10.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.514+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/3127862064 shutdown_connections 2024-04-21T23:09:10.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.514+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/3127862064 wait complete. 2024-04-21T23:09:10.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.515+0000 7fdbc2a2d740 1 -- start start 2024-04-21T23:09:10.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.515+0000 7fdbc2a2d740 1 Processor -- start 2024-04-21T23:09:10.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.515+0000 7fdbc2a2d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5089600 0x5619d5094630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:10.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.515+0000 7fdbc2a2d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5089f20 0x5619d5094b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:10.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.515+0000 7fdbc2a2d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5097620 0x5619d50950b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:10.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.515+0000 7fdbc2a2d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5619d509a190 con 0x5619d5097620 2024-04-21T23:09:10.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.515+0000 7fdbc2a2d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5619d4fa9a10 con 0x5619d5089f20 2024-04-21T23:09:10.520 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.515+0000 7fdbc2a2d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5619d509a310 con 0x5619d5089600 2024-04-21T23:09:10.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.515+0000 7fdbc2a2d740 10 client.0 _ll_register_callbacks cb 0x5619d4fe4b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:10.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.515+0000 7fdbbe302640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5097620 0x5619d50950b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:10.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.516+0000 7fdbbe302640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5097620 0x5619d50950b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52320/0 (socket says 172.21.15.44:52320) 2024-04-21T23:09:10.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.516+0000 7fdbbdb01640 1 --2- 172.21.15.44:0/823615122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5089600 0x5619d5094630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:10.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.516+0000 7fdbbe302640 1 -- 172.21.15.44:0/823615122 learned_addr learned my addr 172.21.15.44:0/823615122 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:10.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.516+0000 7fdbbd300640 1 --2- 172.21.15.44:0/823615122 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5089f20 0x5619d5094b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:10.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.516+0000 7fdbbe302640 1 -- 172.21.15.44:0/823615122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5089600 msgr2=0x5619d5094630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:10.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.516+0000 7fdbbe302640 1 --2- 172.21.15.44:0/823615122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5089600 0x5619d5094630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.516+0000 7fdbbe302640 1 -- 172.21.15.44:0/823615122 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5089f20 msgr2=0x5619d5094b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:10.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.516+0000 7fdbbe302640 1 --2- 172.21.15.44:0/823615122 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5089f20 0x5619d5094b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.516+0000 7fdbbe302640 1 -- 172.21.15.44:0/823615122 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619d4dce620 con 0x5619d5097620 2024-04-21T23:09:10.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.516+0000 7fdbbdb01640 1 --2- 172.21.15.44:0/823615122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5089600 0x5619d5094630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:09:10.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.516+0000 7fdbbd300640 1 --2- 172.21.15.44:0/823615122 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5089f20 0x5619d5094b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:09:10.521 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.516+0000 7fdbbe302640 1 --2- 172.21.15.44:0/823615122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5097620 0x5619d50950b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fdbb000ab90 tx=0x7fdbb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd4055d120a82f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:10.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.516+0000 7fdba17fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:09:10.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.516+0000 7fdba17fa640 1 -- 172.21.15.44:0/823615122 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb002d060 con 0x5619d5097620 2024-04-21T23:09:10.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.516+0000 7fdba17fa640 1 -- 172.21.15.44:0/823615122 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fdbb001eb30 con 0x5619d5097620 2024-04-21T23:09:10.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.516+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/823615122 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbb402cba0 con 0x5619d5097620 2024-04-21T23:09:10.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.516+0000 7fdba17fa640 1 -- 172.21.15.44:0/823615122 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fdbb0034050 con 0x5619d5097620 2024-04-21T23:09:10.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.516+0000 7fdbc2a2d740 10 client.15666 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:10.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.517+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/823615122 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5619d50a64b0 con 0x5619d5097620 2024-04-21T23:09:10.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.517+0000 7fdba17fa640 1 -- 172.21.15.44:0/823615122 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbb402cba0 con 0x5619d5097620 2024-04-21T23:09:10.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.517+0000 7fdba17fa640 1 -- 172.21.15.44:0/823615122 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fdbb000f040 con 0x5619d5097620 2024-04-21T23:09:10.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.517+0000 7fdbc2a2d740 10 client.15666 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:10.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.517+0000 7fdbc2a2d740 10 client.15666 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:10.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.517+0000 7fdbc2a2d740 20 client.15666 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:10.522 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.517+0000 7fdbc2a2d740 1 --2- 172.21.15.44:0/823615122 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5619d50a7970 0x5619d50a9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:10.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.517+0000 7fdbc2a2d740 4 client.15666 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:09:10.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.517+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/823615122 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5619d4ffb0b0 con 0x5619d50a7970 2024-04-21T23:09:10.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.517+0000 7fdbbdb01640 1 --2- 172.21.15.44:0/823615122 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5619d50a7970 0x5619d50a9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:10.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.518+0000 7fdbbdb01640 1 --2- 172.21.15.44:0/823615122 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5619d50a7970 0x5619d50a9d60 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cde10855aa66c89 server_cookie=a810d809336dc72f in_seq=0 out_seq=0 2024-04-21T23:09:10.523 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.518+0000 7fdba17fa640 10 client.15666 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:10.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.518+0000 7fdba17fa640 1 -- 172.21.15.44:0/823615122 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5619d4ffb0b0 con 0x5619d50a7970 2024-04-21T23:09:10.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.518+0000 7fdba17fa640 10 client.15666 handle_command_reply: tid=0 2024-04-21T23:09:10.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.518+0000 7fdbc2a2d740 1 client.15666 shutdown 2024-04-21T23:09:10.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.519+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/823615122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5097620 msgr2=0x5619d50950b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:10.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.519+0000 7fdbc2a2d740 1 --2- 172.21.15.44:0/823615122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5097620 0x5619d50950b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fdbb000ab90 tx=0x7fdbb0004a40 comp rx=0 tx=0).stop 2024-04-21T23:09:10.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.519+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/823615122 shutdown_connections 2024-04-21T23:09:10.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.519+0000 7fdbc2a2d740 1 --2- 172.21.15.44:0/823615122 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5619d50a7970 0x5619d50a9d60 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.524 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.519+0000 7fdbc2a2d740 1 --2- 172.21.15.44:0/823615122 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5097620 0x5619d50950b0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.519+0000 7fdbc2a2d740 1 --2- 172.21.15.44:0/823615122 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5089f20 0x5619d5094b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.519+0000 7fdbc2a2d740 1 --2- 172.21.15.44:0/823615122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5089600 0x5619d5094630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.519+0000 7fdba17fa640 0 client.15666 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:10.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.519+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/823615122 >> 172.21.15.44:0/823615122 conn(0x5619d5084440 msgr2=0x5619d5086190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:10.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.519+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/823615122 shutdown_connections 2024-04-21T23:09:10.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.519+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/823615122 wait complete. 2024-04-21T23:09:10.525 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.519+0000 7fdbc2a2d740 20 client.15666 trim_cache size 0 max 16384 2024-04-21T23:09:10.535 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.534+0000 7fdbc2a2d740 1 Processor -- start 2024-04-21T23:09:10.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.534+0000 7fdbc2a2d740 1 -- start start 2024-04-21T23:09:10.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.534+0000 7fdbc2a2d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5089600 0x5619d5096d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:10.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.534+0000 7fdbc2a2d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5089f20 0x5619d50a3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:10.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.534+0000 7fdbc2a2d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5097620 0x5619d50af670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:10.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.534+0000 7fdbc2a2d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb81aaa30 con 0x5619d5089f20 2024-04-21T23:09:10.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.534+0000 7fdbc2a2d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb81ad2b0 con 0x5619d5089600 2024-04-21T23:09:10.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.534+0000 7fdbc2a2d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb81ab0a0 con 0x5619d5097620 2024-04-21T23:09:10.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.534+0000 7fdbbd300640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5089f20 0x5619d50a3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:10.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.534+0000 7fdbbd300640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5089f20 0x5619d50a3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52328/0 (socket says 172.21.15.44:52328) 2024-04-21T23:09:10.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.534+0000 7fdbbd300640 1 -- 172.21.15.44:0/3914999498 learned_addr learned my addr 172.21.15.44:0/3914999498 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:10.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.534+0000 7fdbbe302640 1 --2- 172.21.15.44:0/3914999498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5097620 0x5619d50af670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:10.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.534+0000 7fdbbdb01640 1 --2- 172.21.15.44:0/3914999498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5089600 0x5619d5096d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:10.536 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.535+0000 7fdbbd300640 1 -- 172.21.15.44:0/3914999498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5097620 msgr2=0x5619d50af670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:10.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.535+0000 7fdbbd300640 1 --2- 172.21.15.44:0/3914999498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5097620 0x5619d50af670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.535+0000 7fdbbd300640 1 -- 172.21.15.44:0/3914999498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5089600 msgr2=0x5619d5096d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:10.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.535+0000 7fdbbd300640 1 --2- 172.21.15.44:0/3914999498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5089600 0x5619d5096d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.535+0000 7fdbbd300640 1 -- 172.21.15.44:0/3914999498 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619d5092b80 con 0x5619d5089f20 2024-04-21T23:09:10.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.535+0000 7fdbbdb01640 1 --2- 172.21.15.44:0/3914999498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5089600 0x5619d5096d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:09:10.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.535+0000 7fdbbe302640 1 --2- 172.21.15.44:0/3914999498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5097620 0x5619d50af670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:09:10.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.535+0000 7fdbbd300640 1 --2- 172.21.15.44:0/3914999498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5089f20 0x5619d50a3090 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fdba8000c00 tx=0x7fdba8086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d176ca9fd609d999 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:10.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.535+0000 7fdba37fe640 1 -- 172.21.15.44:0/3914999498 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba80b4070 con 0x5619d5089f20 2024-04-21T23:09:10.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.535+0000 7fdba37fe640 1 -- 172.21.15.44:0/3914999498 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdba80a46b0 con 0x5619d5089f20 2024-04-21T23:09:10.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.535+0000 7fdba37fe640 1 -- 172.21.15.44:0/3914999498 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba801d040 con 0x5619d5089f20 2024-04-21T23:09:10.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.536+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/3914999498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5089f20 msgr2=0x5619d50a3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:10.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.536+0000 7fdbc2a2d740 1 --2- 172.21.15.44:0/3914999498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5089f20 0x5619d50a3090 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fdba8000c00 tx=0x7fdba8086040 comp rx=0 tx=0).stop 2024-04-21T23:09:10.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.536+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/3914999498 shutdown_connections 2024-04-21T23:09:10.537 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.536+0000 7fdbc2a2d740 1 --2- 172.21.15.44:0/3914999498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5097620 0x5619d50af670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.536+0000 7fdbc2a2d740 1 --2- 172.21.15.44:0/3914999498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5089f20 0x5619d50a3090 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.536+0000 7fdbc2a2d740 1 --2- 172.21.15.44:0/3914999498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5089600 0x5619d5096d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.536+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/3914999498 >> 172.21.15.44:0/3914999498 conn(0x5619d5084440 msgr2=0x5619d5084820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:10.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.536+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/3914999498 shutdown_connections 2024-04-21T23:09:10.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.536+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/3914999498 wait complete. 2024-04-21T23:09:10.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.536+0000 7fdbc2a2d740 1 -- start start 2024-04-21T23:09:10.538 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.536+0000 7fdbc2a2d740 1 Processor -- start 2024-04-21T23:09:10.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.536+0000 7fdbc2a2d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5089600 0x5619d50912c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:10.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.536+0000 7fdbc2a2d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5089f20 0x5619d5091800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:10.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdbc2a2d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5097620 0x5619d5091d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:10.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdbc2a2d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb81ab770 con 0x5619d5097620 2024-04-21T23:09:10.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdbc2a2d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb81acbe0 con 0x5619d5089600 2024-04-21T23:09:10.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdbc2a2d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb81abe40 con 0x5619d5089f20 2024-04-21T23:09:10.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdbc2a2d740 10 client.0 _ll_register_callbacks cb 0x7fdbb0012760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:10.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdbbe302640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5097620 0x5619d5091d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:10.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdbbe302640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5097620 0x5619d5091d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52334/0 (socket says 172.21.15.44:52334) 2024-04-21T23:09:10.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdbbe302640 1 -- 172.21.15.44:0/3210946591 learned_addr learned my addr 172.21.15.44:0/3210946591 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:10.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdbbd300640 1 --2- 172.21.15.44:0/3210946591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5089f20 0x5619d5091800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:10.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdbbdb01640 1 --2- 172.21.15.44:0/3210946591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5089600 0x5619d50912c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:10.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdbbe302640 1 -- 172.21.15.44:0/3210946591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5089f20 msgr2=0x5619d5091800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:10.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdbbe302640 1 --2- 172.21.15.44:0/3210946591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5089f20 0x5619d5091800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdbbe302640 1 -- 172.21.15.44:0/3210946591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5089600 msgr2=0x5619d50912c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:10.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdbbe302640 1 --2- 172.21.15.44:0/3210946591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5089600 0x5619d50912c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdbbe302640 1 -- 172.21.15.44:0/3210946591 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619d510cf10 con 0x5619d5097620 2024-04-21T23:09:10.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdbbdb01640 1 --2- 172.21.15.44:0/3210946591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5089600 0x5619d50912c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:09:10.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdbbd300640 1 --2- 172.21.15.44:0/3210946591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5089f20 0x5619d5091800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:09:10.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdbbe302640 1 --2- 172.21.15.44:0/3210946591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5097620 0x5619d5091d40 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0012810 tx=0x7fdbb000a840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c754faea1bbf18 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:10.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdba17fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:09:10.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdba17fa640 1 -- 172.21.15.44:0/3210946591 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb003d060 con 0x5619d5097620 2024-04-21T23:09:10.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdba17fa640 1 -- 172.21.15.44:0/3210946591 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fdbb00120c0 con 0x5619d5097620 2024-04-21T23:09:10.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/3210946591 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdba80a46b0 con 0x5619d5097620 2024-04-21T23:09:10.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdba17fa640 1 -- 172.21.15.44:0/3210946591 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fdbb0044050 con 0x5619d5097620 2024-04-21T23:09:10.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.537+0000 7fdbc2a2d740 10 client.15672 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:10.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.538+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/3210946591 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5619d510e200 con 0x5619d5097620 2024-04-21T23:09:10.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.538+0000 7fdba17fa640 1 -- 172.21.15.44:0/3210946591 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdba80a46b0 con 0x5619d5097620 2024-04-21T23:09:10.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.538+0000 7fdba17fa640 1 -- 172.21.15.44:0/3210946591 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fdbb00109e0 con 0x5619d5097620 2024-04-21T23:09:10.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.538+0000 7fdbc2a2d740 10 client.15672 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:10.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.538+0000 7fdbc2a2d740 10 client.15672 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:10.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.538+0000 7fdbc2a2d740 20 client.15672 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:10.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.538+0000 7fdbc2a2d740 1 --2- 172.21.15.44:0/3210946591 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5619d510e950 0x5619d5110d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:10.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.538+0000 7fdbc2a2d740 4 client.15672 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:09:10.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.538+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/3210946591 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5619d507f900 con 0x5619d510e950 2024-04-21T23:09:10.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.538+0000 7fdbbdb01640 1 --2- 172.21.15.44:0/3210946591 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5619d510e950 0x5619d5110d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:10.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.538+0000 7fdba17fa640 10 client.15672 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:10.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.538+0000 7fdbbdb01640 1 --2- 172.21.15.44:0/3210946591 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5619d510e950 0x5619d5110d70 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=364866883d6637f0 server_cookie=17b8ac175b721f47 in_seq=0 out_seq=0 2024-04-21T23:09:10.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.551+0000 7fdba17fa640 1 -- 172.21.15.44:0/3210946591 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5619d507f900 con 0x5619d510e950 2024-04-21T23:09:10.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.552+0000 7fdba17fa640 10 client.15672 handle_command_reply: tid=0 2024-04-21T23:09:10.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.552+0000 7fdbc2a2d740 1 client.15672 shutdown 2024-04-21T23:09:10.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.552+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/3210946591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5097620 msgr2=0x5619d5091d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:10.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.552+0000 7fdbc2a2d740 1 --2- 172.21.15.44:0/3210946591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5097620 0x5619d5091d40 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0012810 tx=0x7fdbb000a840 comp rx=0 tx=0).stop 2024-04-21T23:09:10.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.552+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/3210946591 shutdown_connections 2024-04-21T23:09:10.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.552+0000 7fdbc2a2d740 1 --2- 172.21.15.44:0/3210946591 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5619d510e950 0x5619d5110d70 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.552+0000 7fdbc2a2d740 1 --2- 172.21.15.44:0/3210946591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5619d5089f20 0x5619d5091800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.552+0000 7fdbc2a2d740 1 --2- 172.21.15.44:0/3210946591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5619d5097620 0x5619d5091d40 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.552+0000 7fdba17fa640 0 client.15672 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:10.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.552+0000 7fdbc2a2d740 1 --2- 172.21.15.44:0/3210946591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5619d5089600 0x5619d50912c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.552+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/3210946591 >> 172.21.15.44:0/3210946591 conn(0x5619d5084440 msgr2=0x5619d50945f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:10.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.552+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/3210946591 shutdown_connections 2024-04-21T23:09:10.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.552+0000 7fdbc2a2d740 1 -- 172.21.15.44:0/3210946591 wait complete. 2024-04-21T23:09:10.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.552+0000 7fdbc2a2d740 20 client.15672 trim_cache size 0 max 16384 2024-04-21T23:09:10.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:09:10.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:09:10.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:09:10.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:09:10.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.553+0000 7fdbbfd65640 1 -- 172.21.15.44:0/3424884135 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fdba40a2e10 msgr2=0x7fdba40a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:10.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.553+0000 7fdbbfd65640 1 --2- 172.21.15.44:0/3424884135 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fdba40a2e10 0x7fdba40a52d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0009fd0 tx=0x7fdbb0009600 comp rx=0 tx=0).stop 2024-04-21T23:09:10.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.553+0000 7fdbbfd65640 1 -- 172.21.15.44:0/3424884135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdbb81032d0 msgr2=0x7fdbb8198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:10.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.553+0000 7fdbbfd65640 1 --2- 172.21.15.44:0/3424884135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdbb81032d0 0x7fdbb8198550 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fdba800c890 tx=0x7fdba800cd50 comp rx=0 tx=0).stop 2024-04-21T23:09:10.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.553+0000 7fdbbfd65640 1 -- 172.21.15.44:0/3424884135 shutdown_connections 2024-04-21T23:09:10.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.553+0000 7fdbbfd65640 1 --2- 172.21.15.44:0/3424884135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdbb810bd80 0x7fdbb819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.553+0000 7fdbbfd65640 1 --2- 172.21.15.44:0/3424884135 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fdba40a2e10 0x7fdba40a52d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.553+0000 7fdbbfd65640 1 --2- 172.21.15.44:0/3424884135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdbb81032d0 0x7fdbb8198550 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.553+0000 7fdbbfd65640 1 --2- 172.21.15.44:0/3424884135 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdbb81029b0 0x7fdbb8198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:10.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.553+0000 7fdbbfd65640 1 -- 172.21.15.44:0/3424884135 >> 172.21.15.44:0/3424884135 conn(0x7fdbb8077760 msgr2=0x7fdbb80fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:10.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.553+0000 7fdbbfd65640 1 -- 172.21.15.44:0/3424884135 shutdown_connections 2024-04-21T23:09:10.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:10.553+0000 7fdbbfd65640 1 -- 172.21.15.44:0/3424884135 wait complete. 2024-04-21T23:09:12.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:12 smithi044 ceph-mon[27934]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-04-21T23:09:12.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:12 smithi069 ceph-mon[31816]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-04-21T23:09:12.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:12 smithi055 ceph-mon[31117]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-04-21T23:09:14.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:14 smithi044 ceph-mon[27934]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-04-21T23:09:14.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:14 smithi055 ceph-mon[31117]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-04-21T23:09:14.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:14 smithi069 ceph-mon[31816]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-04-21T23:09:16.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:16 smithi044 ceph-mon[27934]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-04-21T23:09:16.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:16 smithi055 ceph-mon[31117]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-04-21T23:09:16.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:16 smithi069 ceph-mon[31816]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-04-21T23:09:18.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:18 smithi044 ceph-mon[27934]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-04-21T23:09:18.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:18 smithi055 ceph-mon[31117]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-04-21T23:09:18.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:18 smithi069 ceph-mon[31816]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-04-21T23:09:20.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:20 smithi069 ceph-mon[31816]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 2.6 KiB/s wr, 7 op/s 2024-04-21T23:09:20.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:20 smithi044 ceph-mon[27934]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 2.6 KiB/s wr, 7 op/s 2024-04-21T23:09:20.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:20 smithi055 ceph-mon[31117]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 2.6 KiB/s wr, 7 op/s 2024-04-21T23:09:22.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:22 smithi044 ceph-mon[27934]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1.9 KiB/s wr, 6 op/s 2024-04-21T23:09:22.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:22 smithi069 ceph-mon[31816]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1.9 KiB/s wr, 6 op/s 2024-04-21T23:09:22.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:22 smithi055 ceph-mon[31117]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1.9 KiB/s wr, 6 op/s 2024-04-21T23:09:23.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.647+0000 7f4379842640 1 -- 172.21.15.44:0/1375632343 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4374107740 msgr2=0x7f4374107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:23.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.647+0000 7f4379842640 1 --2- 172.21.15.44:0/1375632343 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4374107740 0x7f4374107b20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f4364009930 tx=0x7f436402bc30 comp rx=0 tx=0).stop 2024-04-21T23:09:23.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.647+0000 7f4379842640 1 -- 172.21.15.44:0/1375632343 shutdown_connections 2024-04-21T23:09:23.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.647+0000 7f4379842640 1 --2- 172.21.15.44:0/1375632343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43741020f0 0x7f437410e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.647+0000 7f4379842640 1 --2- 172.21.15.44:0/1375632343 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4374101730 0x7f4374101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.647+0000 7f4379842640 1 --2- 172.21.15.44:0/1375632343 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4374107740 0x7f4374107b20 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.647+0000 7f4379842640 1 -- 172.21.15.44:0/1375632343 >> 172.21.15.44:0/1375632343 conn(0x7f43740fd200 msgr2=0x7f43740ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.647+0000 7f4379842640 1 -- 172.21.15.44:0/1375632343 shutdown_connections 2024-04-21T23:09:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.647+0000 7f4379842640 1 -- 172.21.15.44:0/1375632343 wait complete. 2024-04-21T23:09:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.648+0000 7f4379842640 1 Processor -- start 2024-04-21T23:09:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.648+0000 7f4379842640 1 -- start start 2024-04-21T23:09:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.648+0000 7f4379842640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4374101730 0x7f43741980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.648+0000 7f4379842640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43741020f0 0x7f43741985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.648+0000 7f4379842640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4374107740 0x7f437419c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.648+0000 7f4379842640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4374111b50 con 0x7f43741020f0 2024-04-21T23:09:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.648+0000 7f4379842640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f43741119d0 con 0x7f4374101730 2024-04-21T23:09:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.648+0000 7f4379842640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4374111cd0 con 0x7f4374107740 2024-04-21T23:09:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.648+0000 7f43727fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43741020f0 0x7f43741985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.648+0000 7f43727fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43741020f0 0x7f43741985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59764/0 (socket says 172.21.15.44:59764) 2024-04-21T23:09:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.648+0000 7f43727fc640 1 -- 172.21.15.44:0/2608118879 learned_addr learned my addr 172.21.15.44:0/2608118879 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.648+0000 7f4372ffd640 1 --2- 172.21.15.44:0/2608118879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4374101730 0x7f43741980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.648+0000 7f43737fe640 1 --2- 172.21.15.44:0/2608118879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4374107740 0x7f437419c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.648+0000 7f43727fc640 1 -- 172.21.15.44:0/2608118879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4374107740 msgr2=0x7f437419c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.648+0000 7f43727fc640 1 --2- 172.21.15.44:0/2608118879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4374107740 0x7f437419c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.648+0000 7f43727fc640 1 -- 172.21.15.44:0/2608118879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4374101730 msgr2=0x7f43741980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.648+0000 7f43727fc640 1 --2- 172.21.15.44:0/2608118879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4374101730 0x7f43741980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.648+0000 7f43727fc640 1 -- 172.21.15.44:0/2608118879 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f437419d0a0 con 0x7f43741020f0 2024-04-21T23:09:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.649+0000 7f43727fc640 1 --2- 172.21.15.44:0/2608118879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43741020f0 0x7f43741985f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f436800d950 tx=0x7f436800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.649+0000 7f4378840640 1 -- 172.21.15.44:0/2608118879 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4368017070 con 0x7f43741020f0 2024-04-21T23:09:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.649+0000 7f4378840640 1 -- 172.21.15.44:0/2608118879 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4368009c30 con 0x7f43741020f0 2024-04-21T23:09:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.649+0000 7f4379842640 1 -- 172.21.15.44:0/2608118879 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f437419d390 con 0x7f43741020f0 2024-04-21T23:09:23.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.649+0000 7f4378840640 1 -- 172.21.15.44:0/2608118879 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436800f6b0 con 0x7f43741020f0 2024-04-21T23:09:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.649+0000 7f4379842640 1 -- 172.21.15.44:0/2608118879 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43741a4c20 con 0x7f43741020f0 2024-04-21T23:09:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.651+0000 7f4378840640 1 -- 172.21.15.44:0/2608118879 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f43680109d0 con 0x7f43741020f0 2024-04-21T23:09:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.651+0000 7f437c422740 1 Processor -- start 2024-04-21T23:09:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f437c422740 1 -- start start 2024-04-21T23:09:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f437c422740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e028b20 0x55652e028f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f437c422740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e029440 0x55652e031cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f437c422740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55652e032210 0x55652e034600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f437c422740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55652df8cb70 con 0x55652e028b20 2024-04-21T23:09:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f437c422740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55652df35470 con 0x55652e032210 2024-04-21T23:09:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f437c422740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55652df88b30 con 0x55652e029440 2024-04-21T23:09:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f4378840640 1 --2- 172.21.15.44:0/2608118879 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f435c0a2e90 0x7f435c0a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f43737fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e028b20 0x55652e028f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f43737fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e028b20 0x55652e028f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59770/0 (socket says 172.21.15.44:59770) 2024-04-21T23:09:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f43737fe640 1 -- 172.21.15.44:0/3950708812 learned_addr learned my addr 172.21.15.44:0/3950708812 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f4372ffd640 1 --2- 172.21.15.44:0/3950708812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e029440 0x55652e031cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f43737fe640 1 --2- 172.21.15.44:0/2608118879 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f435c0a2e90 0x7f435c0a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f4378840640 1 -- 172.21.15.44:0/2608118879 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f43680cf640 con 0x7f43741020f0 2024-04-21T23:09:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f43727fc640 1 --2- 172.21.15.44:0/3950708812 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55652e032210 0x55652e034600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f43737fe640 1 -- 172.21.15.44:0/3950708812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e029440 msgr2=0x55652e031cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f43737fe640 1 --2- 172.21.15.44:0/3950708812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e029440 0x55652e031cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f43737fe640 1 -- 172.21.15.44:0/3950708812 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55652e032210 msgr2=0x55652e034600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f43737fe640 1 --2- 172.21.15.44:0/3950708812 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55652e032210 0x55652e034600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f43737fe640 1 -- 172.21.15.44:0/3950708812 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55652dca1210 con 0x55652e028b20 2024-04-21T23:09:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.652+0000 7f43727fc640 1 --2- 172.21.15.44:0/3950708812 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55652e032210 0x55652e034600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:09:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.653+0000 7f43737fe640 1 --2- 172.21.15.44:0/3950708812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e028b20 0x55652e028f00 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f436c00cb20 tx=0x7f436c0072b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d572ce99f2ebf1f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.653+0000 7f43737fe640 1 --2- 172.21.15.44:0/2608118879 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f435c0a2e90 0x7f435c0a5350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f436c00c000 tx=0x7f436c01b000 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.653+0000 7f4359ffb640 1 -- 172.21.15.44:0/3950708812 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c015070 con 0x55652e028b20 2024-04-21T23:09:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.653+0000 7f4359ffb640 1 -- 172.21.15.44:0/3950708812 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f436c01e070 con 0x55652e028b20 2024-04-21T23:09:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.653+0000 7f4359ffb640 1 -- 172.21.15.44:0/3950708812 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c00a6d0 con 0x55652e028b20 2024-04-21T23:09:23.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.653+0000 7f437c422740 1 -- 172.21.15.44:0/3950708812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e028b20 msgr2=0x55652e028f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.653+0000 7f437c422740 1 --2- 172.21.15.44:0/3950708812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e028b20 0x55652e028f00 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f436c00cb20 tx=0x7f436c0072b0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.653+0000 7f437c422740 1 -- 172.21.15.44:0/3950708812 shutdown_connections 2024-04-21T23:09:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.653+0000 7f437c422740 1 --2- 172.21.15.44:0/3950708812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e029440 0x55652e031cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.653+0000 7f437c422740 1 --2- 172.21.15.44:0/3950708812 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55652e032210 0x55652e034600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.653+0000 7f437c422740 1 --2- 172.21.15.44:0/3950708812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e028b20 0x55652e028f00 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.654+0000 7f437c422740 1 -- 172.21.15.44:0/3950708812 >> 172.21.15.44:0/3950708812 conn(0x55652e023440 msgr2=0x55652e023820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.654+0000 7f437c422740 1 -- 172.21.15.44:0/3950708812 shutdown_connections 2024-04-21T23:09:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.654+0000 7f437c422740 1 -- 172.21.15.44:0/3950708812 wait complete. 2024-04-21T23:09:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.654+0000 7f437c422740 1 -- start start 2024-04-21T23:09:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.654+0000 7f437c422740 1 Processor -- start 2024-04-21T23:09:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.654+0000 7f437c422740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e028b20 0x55652e02f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.654+0000 7f437c422740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e029440 0x55652e02f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.654+0000 7f437c422740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55652e032210 0x55652e02fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.654+0000 7f437c422740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55652e034de0 con 0x55652e028b20 2024-04-21T23:09:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.654+0000 7f437c422740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55652df48a10 con 0x55652e032210 2024-04-21T23:09:23.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.654+0000 7f437c422740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55652e034f60 con 0x55652e029440 2024-04-21T23:09:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.654+0000 7f437c422740 10 client.0 _ll_register_callbacks cb 0x55652df83b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.655+0000 7f4372ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e028b20 0x55652e02f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.655+0000 7f4372ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e028b20 0x55652e02f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59772/0 (socket says 172.21.15.44:59772) 2024-04-21T23:09:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.655+0000 7f4372ffd640 1 -- 172.21.15.44:0/3235003964 learned_addr learned my addr 172.21.15.44:0/3235003964 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.655+0000 7f43737fe640 1 --2- 172.21.15.44:0/3235003964 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55652e032210 0x55652e02fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.655+0000 7f43727fc640 1 --2- 172.21.15.44:0/3235003964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e029440 0x55652e02f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.655+0000 7f4372ffd640 1 -- 172.21.15.44:0/3235003964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e029440 msgr2=0x55652e02f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.655+0000 7f4372ffd640 1 --2- 172.21.15.44:0/3235003964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e029440 0x55652e02f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.655+0000 7f4372ffd640 1 -- 172.21.15.44:0/3235003964 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55652e032210 msgr2=0x55652e02fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.655+0000 7f4372ffd640 1 --2- 172.21.15.44:0/3235003964 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55652e032210 0x55652e02fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.655+0000 7f4372ffd640 1 -- 172.21.15.44:0/3235003964 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55652dd6d620 con 0x55652e028b20 2024-04-21T23:09:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.655+0000 7f4372ffd640 1 --2- 172.21.15.44:0/3235003964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e028b20 0x55652e02f2f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f436402cfd0 tx=0x7f4364004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc9482de01e94364 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.655+0000 7f4352ffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:09:23.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.655+0000 7f4352ffd640 1 -- 172.21.15.44:0/3235003964 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43640043b0 con 0x55652e028b20 2024-04-21T23:09:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.655+0000 7f4352ffd640 1 -- 172.21.15.44:0/3235003964 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f43640046c0 con 0x55652e028b20 2024-04-21T23:09:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.655+0000 7f437c422740 1 -- 172.21.15.44:0/3235003964 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f436c01e070 con 0x55652e028b20 2024-04-21T23:09:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.655+0000 7f4352ffd640 1 -- 172.21.15.44:0/3235003964 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f43640048c0 con 0x55652e028b20 2024-04-21T23:09:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.655+0000 7f437c422740 10 client.15684 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.655+0000 7f437c422740 1 -- 172.21.15.44:0/3235003964 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55652e0410c0 con 0x55652e028b20 2024-04-21T23:09:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.656+0000 7f4352ffd640 1 -- 172.21.15.44:0/3235003964 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f436c01e070 con 0x55652e028b20 2024-04-21T23:09:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.656+0000 7f4352ffd640 1 -- 172.21.15.44:0/3235003964 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f43640035f0 con 0x55652e028b20 2024-04-21T23:09:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.656+0000 7f437c422740 10 client.15684 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.656+0000 7f437c422740 10 client.15684 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.656+0000 7f437c422740 20 client.15684 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.656+0000 7f437c422740 1 --2- 172.21.15.44:0/3235003964 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55652e042580 0x55652e044970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.656+0000 7f437c422740 4 client.15684 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:09:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.656+0000 7f437c422740 1 -- 172.21.15.44:0/3235003964 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55652df9a0b0 con 0x55652e042580 2024-04-21T23:09:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.656+0000 7f43727fc640 1 --2- 172.21.15.44:0/3235003964 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55652e042580 0x55652e044970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:23.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.657+0000 7f43727fc640 1 --2- 172.21.15.44:0/3235003964 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55652e042580 0x55652e044970 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84f732bc6e6d73ec server_cookie=63e8b784f7300be in_seq=0 out_seq=0 2024-04-21T23:09:23.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.657+0000 7f4352ffd640 10 client.15684 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:23.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.657+0000 7f4352ffd640 1 -- 172.21.15.44:0/3235003964 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55652df9a0b0 con 0x55652e042580 2024-04-21T23:09:23.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.657+0000 7f4352ffd640 10 client.15684 handle_command_reply: tid=0 2024-04-21T23:09:23.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.658+0000 7f437c422740 1 client.15684 shutdown 2024-04-21T23:09:23.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.658+0000 7f437c422740 1 -- 172.21.15.44:0/3235003964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e028b20 msgr2=0x55652e02f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:23.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.658+0000 7f437c422740 1 --2- 172.21.15.44:0/3235003964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e028b20 0x55652e02f2f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f436402cfd0 tx=0x7f4364004040 comp rx=0 tx=0).stop 2024-04-21T23:09:23.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.658+0000 7f437c422740 1 -- 172.21.15.44:0/3235003964 shutdown_connections 2024-04-21T23:09:23.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.658+0000 7f437c422740 1 --2- 172.21.15.44:0/3235003964 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55652e042580 0x55652e044970 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.658+0000 7f437c422740 1 --2- 172.21.15.44:0/3235003964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e029440 0x55652e02f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.658+0000 7f437c422740 1 --2- 172.21.15.44:0/3235003964 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55652e032210 0x55652e02fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.658+0000 7f437c422740 1 --2- 172.21.15.44:0/3235003964 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e028b20 0x55652e02f2f0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.658+0000 7f4352ffd640 0 client.15684 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:23.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.658+0000 7f437c422740 1 -- 172.21.15.44:0/3235003964 >> 172.21.15.44:0/3235003964 conn(0x55652e023440 msgr2=0x55652e025190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:23.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.658+0000 7f437c422740 1 -- 172.21.15.44:0/3235003964 shutdown_connections 2024-04-21T23:09:23.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.658+0000 7f437c422740 1 -- 172.21.15.44:0/3235003964 wait complete. 2024-04-21T23:09:23.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.658+0000 7f437c422740 20 client.15684 trim_cache size 0 max 16384 2024-04-21T23:09:23.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.673+0000 7f437c422740 1 Processor -- start 2024-04-21T23:09:23.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.673+0000 7f437c422740 1 -- start start 2024-04-21T23:09:23.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.673+0000 7f437c422740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55652e028b20 0x55652e031a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:23.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.673+0000 7f437c422740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e029440 0x55652e03dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:23.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.673+0000 7f437c422740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e032210 0x55652e04a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:23.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.673+0000 7f437c422740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f43741acc40 con 0x55652e032210 2024-04-21T23:09:23.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.673+0000 7f437c422740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f43741af4c0 con 0x55652e028b20 2024-04-21T23:09:23.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.673+0000 7f437c422740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f43741ad2b0 con 0x55652e029440 2024-04-21T23:09:23.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.673+0000 7f43737fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e032210 0x55652e04a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:23.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.673+0000 7f43737fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e032210 0x55652e04a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59782/0 (socket says 172.21.15.44:59782) 2024-04-21T23:09:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.673+0000 7f43737fe640 1 -- 172.21.15.44:0/4218466304 learned_addr learned my addr 172.21.15.44:0/4218466304 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.673+0000 7f43727fc640 1 --2- 172.21.15.44:0/4218466304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e029440 0x55652e03dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.673+0000 7f43737fe640 1 -- 172.21.15.44:0/4218466304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e029440 msgr2=0x55652e03dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.673+0000 7f43737fe640 1 --2- 172.21.15.44:0/4218466304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e029440 0x55652e03dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.673+0000 7f43737fe640 1 -- 172.21.15.44:0/4218466304 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55652e028b20 msgr2=0x55652e031a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.673+0000 7f43737fe640 1 --2- 172.21.15.44:0/4218466304 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55652e028b20 0x55652e031a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.673+0000 7f43737fe640 1 -- 172.21.15.44:0/4218466304 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55652e022c40 con 0x55652e032210 2024-04-21T23:09:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.673+0000 7f43727fc640 1 --2- 172.21.15.44:0/4218466304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e029440 0x55652e03dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:09:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.674+0000 7f43737fe640 1 --2- 172.21.15.44:0/4218466304 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e032210 0x55652e04a290 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f436c00c070 tx=0x7f436c01ae20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=695eb7af6d619b8f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.674+0000 7f43597fa640 1 -- 172.21.15.44:0/4218466304 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c032070 con 0x55652e032210 2024-04-21T23:09:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.674+0000 7f43597fa640 1 -- 172.21.15.44:0/4218466304 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f436c01e050 con 0x55652e032210 2024-04-21T23:09:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.674+0000 7f43597fa640 1 -- 172.21.15.44:0/4218466304 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c015040 con 0x55652e032210 2024-04-21T23:09:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.674+0000 7f437c422740 1 -- 172.21.15.44:0/4218466304 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e032210 msgr2=0x55652e04a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.674+0000 7f437c422740 1 --2- 172.21.15.44:0/4218466304 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e032210 0x55652e04a290 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f436c00c070 tx=0x7f436c01ae20 comp rx=0 tx=0).stop 2024-04-21T23:09:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.674+0000 7f437c422740 1 -- 172.21.15.44:0/4218466304 shutdown_connections 2024-04-21T23:09:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.674+0000 7f437c422740 1 --2- 172.21.15.44:0/4218466304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e029440 0x55652e03dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.674+0000 7f437c422740 1 --2- 172.21.15.44:0/4218466304 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e032210 0x55652e04a290 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.674+0000 7f437c422740 1 --2- 172.21.15.44:0/4218466304 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55652e028b20 0x55652e031a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.674+0000 7f437c422740 1 -- 172.21.15.44:0/4218466304 >> 172.21.15.44:0/4218466304 conn(0x55652e023440 msgr2=0x55652e01e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.674+0000 7f437c422740 1 -- 172.21.15.44:0/4218466304 shutdown_connections 2024-04-21T23:09:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.674+0000 7f437c422740 1 -- 172.21.15.44:0/4218466304 wait complete. 2024-04-21T23:09:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.675+0000 7f437c422740 1 -- start start 2024-04-21T23:09:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.675+0000 7f437c422740 1 Processor -- start 2024-04-21T23:09:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.675+0000 7f437c422740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e028b20 0x55652e03a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.675+0000 7f437c422740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e029440 0x55652e02bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.675+0000 7f437c422740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55652e032210 0x55652e02c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:23.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.675+0000 7f437c422740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f43741ad980 con 0x55652e029440 2024-04-21T23:09:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.675+0000 7f437c422740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f43741aedf0 con 0x55652e032210 2024-04-21T23:09:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.675+0000 7f437c422740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f43741ae050 con 0x55652e028b20 2024-04-21T23:09:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.675+0000 7f437c422740 10 client.0 _ll_register_callbacks cb 0x7f4364009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.675+0000 7f43727fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e029440 0x55652e02bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.675+0000 7f43727fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e029440 0x55652e02bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59784/0 (socket says 172.21.15.44:59784) 2024-04-21T23:09:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.675+0000 7f43727fc640 1 -- 172.21.15.44:0/1151040547 learned_addr learned my addr 172.21.15.44:0/1151040547 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.675+0000 7f4372ffd640 1 --2- 172.21.15.44:0/1151040547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e028b20 0x55652e03a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f43737fe640 1 --2- 172.21.15.44:0/1151040547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55652e032210 0x55652e02c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f43727fc640 1 -- 172.21.15.44:0/1151040547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e028b20 msgr2=0x55652e03a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f43727fc640 1 --2- 172.21.15.44:0/1151040547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e028b20 0x55652e03a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f43727fc640 1 -- 172.21.15.44:0/1151040547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55652e032210 msgr2=0x55652e02c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f43727fc640 1 --2- 172.21.15.44:0/1151040547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55652e032210 0x55652e02c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f43727fc640 1 -- 172.21.15.44:0/1151040547 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55652e02c9a0 con 0x55652e029440 2024-04-21T23:09:23.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f43727fc640 1 --2- 172.21.15.44:0/1151040547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e029440 0x55652e02bc80 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f43680979d0 tx=0x7f4368005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e111d38f2e80490 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f4352ffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:09:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f4352ffd640 1 -- 172.21.15.44:0/1151040547 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4368020040 con 0x55652e029440 2024-04-21T23:09:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f4352ffd640 1 -- 172.21.15.44:0/1151040547 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f43680a3d50 con 0x55652e029440 2024-04-21T23:09:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f437c422740 1 -- 172.21.15.44:0/1151040547 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f436c01e050 con 0x55652e029440 2024-04-21T23:09:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f4352ffd640 1 -- 172.21.15.44:0/1151040547 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f436809f920 con 0x55652e029440 2024-04-21T23:09:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f437c422740 10 client.15690 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f437c422740 1 -- 172.21.15.44:0/1151040547 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55652e0ace40 con 0x55652e029440 2024-04-21T23:09:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f4352ffd640 1 -- 172.21.15.44:0/1151040547 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f436c01e050 con 0x55652e029440 2024-04-21T23:09:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f4352ffd640 1 -- 172.21.15.44:0/1151040547 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f436809a9c0 con 0x55652e029440 2024-04-21T23:09:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f437c422740 10 client.15690 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f437c422740 10 client.15690 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f437c422740 20 client.15690 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f437c422740 1 --2- 172.21.15.44:0/1151040547 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55652e0ad510 0x55652e0af930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f437c422740 4 client.15690 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:09:23.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.676+0000 7f437c422740 1 -- 172.21.15.44:0/1151040547 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55652df94400 con 0x55652e0ad510 2024-04-21T23:09:23.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.677+0000 7f4372ffd640 1 --2- 172.21.15.44:0/1151040547 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55652e0ad510 0x55652e0af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:23.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.677+0000 7f4372ffd640 1 --2- 172.21.15.44:0/1151040547 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55652e0ad510 0x55652e0af930 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c14e20645b2bdfad server_cookie=f8948d915f6631ad in_seq=0 out_seq=0 2024-04-21T23:09:23.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.677+0000 7f4352ffd640 10 client.15690 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:23.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.681+0000 7f4352ffd640 1 -- 172.21.15.44:0/1151040547 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55652df94400 con 0x55652e0ad510 2024-04-21T23:09:23.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.681+0000 7f4352ffd640 10 client.15690 handle_command_reply: tid=0 2024-04-21T23:09:23.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.681+0000 7f437c422740 1 client.15690 shutdown 2024-04-21T23:09:23.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.681+0000 7f437c422740 1 -- 172.21.15.44:0/1151040547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e029440 msgr2=0x55652e02bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:23.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.681+0000 7f437c422740 1 --2- 172.21.15.44:0/1151040547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e029440 0x55652e02bc80 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f43680979d0 tx=0x7f4368005f90 comp rx=0 tx=0).stop 2024-04-21T23:09:23.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.681+0000 7f437c422740 1 -- 172.21.15.44:0/1151040547 shutdown_connections 2024-04-21T23:09:23.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.681+0000 7f437c422740 1 --2- 172.21.15.44:0/1151040547 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55652e0ad510 0x55652e0af930 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.681+0000 7f437c422740 1 --2- 172.21.15.44:0/1151040547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55652e032210 0x55652e02c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.681+0000 7f4352ffd640 0 client.15690 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:23.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.681+0000 7f437c422740 1 --2- 172.21.15.44:0/1151040547 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55652e029440 0x55652e02bc80 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.681+0000 7f437c422740 1 --2- 172.21.15.44:0/1151040547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55652e028b20 0x55652e03a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.681+0000 7f437c422740 1 -- 172.21.15.44:0/1151040547 >> 172.21.15.44:0/1151040547 conn(0x55652e023440 msgr2=0x55652e02dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:23.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.681+0000 7f437c422740 1 -- 172.21.15.44:0/1151040547 shutdown_connections 2024-04-21T23:09:23.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.681+0000 7f437c422740 1 -- 172.21.15.44:0/1151040547 wait complete. 2024-04-21T23:09:23.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.681+0000 7f437c422740 20 client.15690 trim_cache size 0 max 16384 2024-04-21T23:09:23.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:09:23.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:09:23.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:09:23.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:09:23.685 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.682+0000 7f4379842640 1 -- 172.21.15.44:0/2608118879 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f435c0a2e90 msgr2=0x7f435c0a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:23.685 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.682+0000 7f4379842640 1 --2- 172.21.15.44:0/2608118879 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f435c0a2e90 0x7f435c0a5350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f436c00c000 tx=0x7f436c01b000 comp rx=0 tx=0).stop 2024-04-21T23:09:23.685 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.682+0000 7f4379842640 1 -- 172.21.15.44:0/2608118879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43741020f0 msgr2=0x7f43741985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:23.685 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.682+0000 7f4379842640 1 --2- 172.21.15.44:0/2608118879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43741020f0 0x7f43741985f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f436800d950 tx=0x7f436800de10 comp rx=0 tx=0).stop 2024-04-21T23:09:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.682+0000 7f4379842640 1 -- 172.21.15.44:0/2608118879 shutdown_connections 2024-04-21T23:09:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.682+0000 7f4379842640 1 --2- 172.21.15.44:0/2608118879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4374107740 0x7f437419c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.682+0000 7f4379842640 1 --2- 172.21.15.44:0/2608118879 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f435c0a2e90 0x7f435c0a5350 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.682+0000 7f4379842640 1 --2- 172.21.15.44:0/2608118879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43741020f0 0x7f43741985f0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.682+0000 7f4379842640 1 --2- 172.21.15.44:0/2608118879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4374101730 0x7f43741980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.682+0000 7f4379842640 1 -- 172.21.15.44:0/2608118879 >> 172.21.15.44:0/2608118879 conn(0x7f43740fd200 msgr2=0x7f43740fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.682+0000 7f4379842640 1 -- 172.21.15.44:0/2608118879 shutdown_connections 2024-04-21T23:09:23.686 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:23.682+0000 7f4379842640 1 -- 172.21.15.44:0/2608118879 wait complete. 2024-04-21T23:09:24.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:24 smithi044 ceph-mon[27934]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-04-21T23:09:24.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:24 smithi055 ceph-mon[31117]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-04-21T23:09:24.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:24 smithi069 ceph-mon[31816]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-04-21T23:09:26.077 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:09:26.077 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:09:26.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.197+0000 7f7dfc80e640 1 -- 172.21.15.44:0/2917412609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7df810e0e0 msgr2=0x7f7df81104d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.197+0000 7f7dfc80e640 1 --2- 172.21.15.44:0/2917412609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7df810e0e0 0x7f7df81104d0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f7df0009ca0 tx=0x7f7df002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.197+0000 7f7dfc80e640 1 -- 172.21.15.44:0/2917412609 shutdown_connections 2024-04-21T23:09:26.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.197+0000 7f7dfc80e640 1 --2- 172.21.15.44:0/2917412609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7df8100f70 0x7f7df810dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.197+0000 7f7dfc80e640 1 --2- 172.21.15.44:0/2917412609 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7df810e0e0 0x7f7df81104d0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.197+0000 7f7dfc80e640 1 --2- 172.21.15.44:0/2917412609 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7df81005c0 0x7f7df81009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.197+0000 7f7dfc80e640 1 -- 172.21.15.44:0/2917412609 >> 172.21.15.44:0/2917412609 conn(0x7f7df8077760 msgr2=0x7f7df8077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:26.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.198+0000 7f7dfc80e640 1 -- 172.21.15.44:0/2917412609 shutdown_connections 2024-04-21T23:09:26.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.198+0000 7f7dfc80e640 1 -- 172.21.15.44:0/2917412609 wait complete. 2024-04-21T23:09:26.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.198+0000 7f7dfc80e640 1 Processor -- start 2024-04-21T23:09:26.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.198+0000 7f7dfc80e640 1 -- start start 2024-04-21T23:09:26.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7dfc80e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7df81005c0 0x7f7df819a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7dfc80e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7df8100f70 0x7f7df819a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7dfc80e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7df810e0e0 0x7f7df819ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7dfc80e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7df8113e10 con 0x7f7df8100f70 2024-04-21T23:09:26.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7dfc80e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7df8113c90 con 0x7f7df810e0e0 2024-04-21T23:09:26.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7dfc80e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7df8113f90 con 0x7f7df81005c0 2024-04-21T23:09:26.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7deffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7df8100f70 0x7f7df819a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7deffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7df8100f70 0x7f7df819a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59812/0 (socket says 172.21.15.44:59812) 2024-04-21T23:09:26.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7deffff640 1 -- 172.21.15.44:0/284501915 learned_addr learned my addr 172.21.15.44:0/284501915 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:26.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7df659c640 1 --2- 172.21.15.44:0/284501915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7df81005c0 0x7f7df819a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.201 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7df6d9d640 1 --2- 172.21.15.44:0/284501915 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7df810e0e0 0x7f7df819ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7deffff640 1 -- 172.21.15.44:0/284501915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7df81005c0 msgr2=0x7f7df819a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7deffff640 1 --2- 172.21.15.44:0/284501915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7df81005c0 0x7f7df819a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7deffff640 1 -- 172.21.15.44:0/284501915 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7df810e0e0 msgr2=0x7f7df819ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7deffff640 1 --2- 172.21.15.44:0/284501915 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7df810e0e0 0x7f7df819ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7deffff640 1 -- 172.21.15.44:0/284501915 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df819f310 con 0x7f7df8100f70 2024-04-21T23:09:26.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7df659c640 1 --2- 172.21.15.44:0/284501915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7df81005c0 0x7f7df819a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:09:26.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7df6d9d640 1 --2- 172.21.15.44:0/284501915 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7df810e0e0 0x7f7df819ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:09:26.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7deffff640 1 --2- 172.21.15.44:0/284501915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7df8100f70 0x7f7df819a8b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f7de000d770 tx=0x7f7de000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:26.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7def7fe640 1 -- 172.21.15.44:0/284501915 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de0017070 con 0x7f7df8100f70 2024-04-21T23:09:26.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7dfc80e640 1 -- 172.21.15.44:0/284501915 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7df819f600 con 0x7f7df8100f70 2024-04-21T23:09:26.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7def7fe640 1 -- 172.21.15.44:0/284501915 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7de0005500 con 0x7f7df8100f70 2024-04-21T23:09:26.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7def7fe640 1 -- 172.21.15.44:0/284501915 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de0004e00 con 0x7f7df8100f70 2024-04-21T23:09:26.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.199+0000 7f7dfc80e640 1 -- 172.21.15.44:0/284501915 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7df81a6e90 con 0x7f7df8100f70 2024-04-21T23:09:26.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.202+0000 7f7def7fe640 1 -- 172.21.15.44:0/284501915 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f7de0004fa0 con 0x7f7df8100f70 2024-04-21T23:09:26.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.202+0000 7f7dff3f9740 1 Processor -- start 2024-04-21T23:09:26.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.202+0000 7f7dff3f9740 1 -- start start 2024-04-21T23:09:26.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.203+0000 7f7dff3f9740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636aac7ef30 0x5636aac7f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.203+0000 7f7dff3f9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7f850 0x5636aac87ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.203+0000 7f7dff3f9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636aac88430 0x5636aac8a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.203+0000 7f7dff3f9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5636aabe3560 con 0x5636aac7f850 2024-04-21T23:09:26.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.203+0000 7f7dff3f9740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5636aab8b730 con 0x5636aac7ef30 2024-04-21T23:09:26.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.203+0000 7f7dff3f9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5636aabdf520 con 0x5636aac88430 2024-04-21T23:09:26.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.203+0000 7f7def7fe640 1 --2- 172.21.15.44:0/284501915 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7de40a2e20 0x7f7de40a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.203+0000 7f7df659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7f850 0x5636aac87ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.203+0000 7f7df6d9d640 1 --2- 172.21.15.44:0/284501915 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7de40a2e20 0x7f7de40a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.203+0000 7f7df659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7f850 0x5636aac87ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59816/0 (socket says 172.21.15.44:59816) 2024-04-21T23:09:26.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.203+0000 7f7df659c640 1 -- 172.21.15.44:0/2630477120 learned_addr learned my addr 172.21.15.44:0/2630477120 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:26.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.203+0000 7f7def7fe640 1 -- 172.21.15.44:0/284501915 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f7de00cf4b0 con 0x7f7df8100f70 2024-04-21T23:09:26.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.203+0000 7f7deffff640 1 --2- 172.21.15.44:0/2630477120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636aac88430 0x5636aac8a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.203+0000 7f7df6d9d640 1 --2- 172.21.15.44:0/2630477120 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636aac7ef30 0x5636aac7f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.203+0000 7f7df659c640 1 -- 172.21.15.44:0/2630477120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636aac88430 msgr2=0x5636aac8a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:26.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.203+0000 7f7df659c640 1 --2- 172.21.15.44:0/2630477120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636aac88430 0x5636aac8a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.203+0000 7f7df659c640 1 -- 172.21.15.44:0/2630477120 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636aac7ef30 msgr2=0x5636aac7f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:26.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.203+0000 7f7df659c640 1 --2- 172.21.15.44:0/2630477120 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636aac7ef30 0x5636aac7f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.203+0000 7f7df659c640 1 -- 172.21.15.44:0/2630477120 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636aa9c17d0 con 0x5636aac7f850 2024-04-21T23:09:26.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.203+0000 7f7df659c640 1 --2- 172.21.15.44:0/2630477120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7f850 0x5636aac87ef0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7de800bb70 tx=0x7f7de800d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bec03dec8344bcb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:26.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.204+0000 7f7df6d9d640 1 --2- 172.21.15.44:0/284501915 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7de40a2e20 0x7f7de40a52e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7df0004af0 tx=0x7f7df00324f0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:26.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.204+0000 7f7decff9640 1 -- 172.21.15.44:0/2630477120 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de8017620 con 0x5636aac7f850 2024-04-21T23:09:26.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.204+0000 7f7decff9640 1 -- 172.21.15.44:0/2630477120 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7de8017ca0 con 0x5636aac7f850 2024-04-21T23:09:26.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.204+0000 7f7decff9640 1 -- 172.21.15.44:0/2630477120 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de800fbd0 con 0x5636aac7f850 2024-04-21T23:09:26.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.204+0000 7f7dff3f9740 1 -- 172.21.15.44:0/2630477120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7f850 msgr2=0x5636aac87ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.204+0000 7f7dff3f9740 1 --2- 172.21.15.44:0/2630477120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7f850 0x5636aac87ef0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7de800bb70 tx=0x7f7de800d710 comp rx=0 tx=0).stop 2024-04-21T23:09:26.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.204+0000 7f7dff3f9740 1 -- 172.21.15.44:0/2630477120 shutdown_connections 2024-04-21T23:09:26.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.204+0000 7f7dff3f9740 1 --2- 172.21.15.44:0/2630477120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636aac88430 0x5636aac8a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.204+0000 7f7dff3f9740 1 --2- 172.21.15.44:0/2630477120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7f850 0x5636aac87ef0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.204+0000 7f7dff3f9740 1 --2- 172.21.15.44:0/2630477120 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636aac7ef30 0x5636aac7f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.204+0000 7f7dff3f9740 1 -- 172.21.15.44:0/2630477120 >> 172.21.15.44:0/2630477120 conn(0x5636aac79660 msgr2=0x5636aac79a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:26.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.205+0000 7f7dff3f9740 1 -- 172.21.15.44:0/2630477120 shutdown_connections 2024-04-21T23:09:26.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.205+0000 7f7dff3f9740 1 -- 172.21.15.44:0/2630477120 wait complete. 2024-04-21T23:09:26.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.205+0000 7f7dff3f9740 1 -- start start 2024-04-21T23:09:26.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.205+0000 7f7dff3f9740 1 Processor -- start 2024-04-21T23:09:26.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.205+0000 7f7dff3f9740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636aac7ef30 0x5636aac85450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.205+0000 7f7dff3f9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636aac7f850 0x5636aac85990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.205+0000 7f7dff3f9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac88430 0x5636aac85ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.205+0000 7f7dff3f9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5636aac8b220 con 0x5636aac88430 2024-04-21T23:09:26.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.205+0000 7f7dff3f9740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5636aab9ee00 con 0x5636aac7ef30 2024-04-21T23:09:26.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.205+0000 7f7dff3f9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5636aac8b3a0 con 0x5636aac7f850 2024-04-21T23:09:26.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.205+0000 7f7dff3f9740 10 client.0 _ll_register_callbacks cb 0x5636aabd9eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:26.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.205+0000 7f7df6d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac88430 0x5636aac85ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.205+0000 7f7df6d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac88430 0x5636aac85ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59820/0 (socket says 172.21.15.44:59820) 2024-04-21T23:09:26.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.205+0000 7f7df6d9d640 1 -- 172.21.15.44:0/940339147 learned_addr learned my addr 172.21.15.44:0/940339147 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:26.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.205+0000 7f7deffff640 1 --2- 172.21.15.44:0/940339147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636aac7f850 0x5636aac85990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.205+0000 7f7df659c640 1 --2- 172.21.15.44:0/940339147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636aac7ef30 0x5636aac85450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.206+0000 7f7df6d9d640 1 -- 172.21.15.44:0/940339147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636aac7f850 msgr2=0x5636aac85990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:26.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.206+0000 7f7df6d9d640 1 --2- 172.21.15.44:0/940339147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636aac7f850 0x5636aac85990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.206+0000 7f7df6d9d640 1 -- 172.21.15.44:0/940339147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636aac7ef30 msgr2=0x5636aac85450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:26.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.206+0000 7f7df6d9d640 1 --2- 172.21.15.44:0/940339147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636aac7ef30 0x5636aac85450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.206+0000 7f7df6d9d640 1 -- 172.21.15.44:0/940339147 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636aa8f73f0 con 0x5636aac88430 2024-04-21T23:09:26.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.206+0000 7f7df6d9d640 1 --2- 172.21.15.44:0/940339147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac88430 0x5636aac85ed0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7df8069a50 tx=0x7f7df0045070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=227e30848b4e7a7b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:26.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.206+0000 7f7dddffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:09:26.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.206+0000 7f7dddffb640 1 -- 172.21.15.44:0/940339147 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df0049040 con 0x5636aac88430 2024-04-21T23:09:26.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.206+0000 7f7dddffb640 1 -- 172.21.15.44:0/940339147 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f7df0004040 con 0x5636aac88430 2024-04-21T23:09:26.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.206+0000 7f7dff3f9740 1 -- 172.21.15.44:0/940339147 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7de8017ca0 con 0x5636aac88430 2024-04-21T23:09:26.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.206+0000 7f7dddffb640 1 -- 172.21.15.44:0/940339147 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7df005b050 con 0x5636aac88430 2024-04-21T23:09:26.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.206+0000 7f7dff3f9740 10 client.15702 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:26.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.206+0000 7f7dff3f9740 1 -- 172.21.15.44:0/940339147 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5636aa8b70f0 con 0x5636aac88430 2024-04-21T23:09:26.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.206+0000 7f7dddffb640 1 -- 172.21.15.44:0/940339147 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7de8017ca0 con 0x5636aac88430 2024-04-21T23:09:26.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.207+0000 7f7dddffb640 1 -- 172.21.15.44:0/940339147 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f7df004c070 con 0x5636aac88430 2024-04-21T23:09:26.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.207+0000 7f7dff3f9740 10 client.15702 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:26.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.207+0000 7f7dff3f9740 10 client.15702 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:26.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.207+0000 7f7dff3f9740 20 client.15702 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:26.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.207+0000 7f7dff3f9740 1 --2- 172.21.15.44:0/940339147 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5636aac987e0 0x5636aac9ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.207+0000 7f7dff3f9740 4 client.15702 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:09:26.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.207+0000 7f7dff3f9740 1 -- 172.21.15.44:0/940339147 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636aabf0aa0 con 0x5636aac987e0 2024-04-21T23:09:26.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.207+0000 7f7df659c640 1 --2- 172.21.15.44:0/940339147 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5636aac987e0 0x5636aac9ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.207+0000 7f7df659c640 1 --2- 172.21.15.44:0/940339147 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5636aac987e0 0x5636aac9ac10 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6567576531d1483 server_cookie=57d2973e26461440 in_seq=0 out_seq=0 2024-04-21T23:09:26.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.207+0000 7f7dddffb640 10 client.15702 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:26.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.208+0000 7f7dddffb640 1 -- 172.21.15.44:0/940339147 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5636aabf0aa0 con 0x5636aac987e0 2024-04-21T23:09:26.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.208+0000 7f7dddffb640 10 client.15702 handle_command_reply: tid=0 2024-04-21T23:09:26.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.208+0000 7f7dff3f9740 1 client.15702 shutdown 2024-04-21T23:09:26.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.208+0000 7f7dff3f9740 1 -- 172.21.15.44:0/940339147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac88430 msgr2=0x5636aac85ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.208+0000 7f7dff3f9740 1 --2- 172.21.15.44:0/940339147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac88430 0x5636aac85ed0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7df8069a50 tx=0x7f7df0045070 comp rx=0 tx=0).stop 2024-04-21T23:09:26.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.208+0000 7f7dff3f9740 1 -- 172.21.15.44:0/940339147 shutdown_connections 2024-04-21T23:09:26.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.208+0000 7f7dff3f9740 1 --2- 172.21.15.44:0/940339147 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5636aac987e0 0x5636aac9ac10 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.208+0000 7f7dff3f9740 1 --2- 172.21.15.44:0/940339147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636aac7f850 0x5636aac85990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.208+0000 7f7dff3f9740 1 --2- 172.21.15.44:0/940339147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac88430 0x5636aac85ed0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.208+0000 7f7dddffb640 0 client.15702 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:26.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.208+0000 7f7dff3f9740 1 --2- 172.21.15.44:0/940339147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636aac7ef30 0x5636aac85450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.208+0000 7f7dff3f9740 1 -- 172.21.15.44:0/940339147 >> 172.21.15.44:0/940339147 conn(0x5636aac79660 msgr2=0x5636aac7b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:26.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.208+0000 7f7dff3f9740 1 -- 172.21.15.44:0/940339147 shutdown_connections 2024-04-21T23:09:26.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.208+0000 7f7dff3f9740 1 -- 172.21.15.44:0/940339147 wait complete. 2024-04-21T23:09:26.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.208+0000 7f7dff3f9740 20 client.15702 trim_cache size 0 max 16384 2024-04-21T23:09:26.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.226+0000 7f7dff3f9740 1 Processor -- start 2024-04-21T23:09:26.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.226+0000 7f7dff3f9740 1 -- start start 2024-04-21T23:09:26.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.226+0000 7f7dff3f9740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636aac7ef30 0x5636aac87b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.226+0000 7f7dff3f9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7f850 0x5636aac940e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.226+0000 7f7dff3f9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636aac88430 0x5636aac79a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.226+0000 7f7dff3f9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7df81aeda0 con 0x5636aac7f850 2024-04-21T23:09:26.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.226+0000 7f7dff3f9740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7df81b1620 con 0x5636aac7ef30 2024-04-21T23:09:26.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.226+0000 7f7dff3f9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7df81af410 con 0x5636aac88430 2024-04-21T23:09:26.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.226+0000 7f7deffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7f850 0x5636aac940e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.226+0000 7f7deffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7f850 0x5636aac940e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59832/0 (socket says 172.21.15.44:59832) 2024-04-21T23:09:26.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.226+0000 7f7deffff640 1 -- 172.21.15.44:0/1782151945 learned_addr learned my addr 172.21.15.44:0/1782151945 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:26.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.227+0000 7f7df6d9d640 1 --2- 172.21.15.44:0/1782151945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636aac88430 0x5636aac79a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.227+0000 7f7deffff640 1 -- 172.21.15.44:0/1782151945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636aac88430 msgr2=0x5636aac79a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:26.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.227+0000 7f7deffff640 1 --2- 172.21.15.44:0/1782151945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636aac88430 0x5636aac79a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.227+0000 7f7deffff640 1 -- 172.21.15.44:0/1782151945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636aac7ef30 msgr2=0x5636aac87b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:26.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.227+0000 7f7deffff640 1 --2- 172.21.15.44:0/1782151945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636aac7ef30 0x5636aac87b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.227+0000 7f7deffff640 1 -- 172.21.15.44:0/1782151945 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636aac74cf0 con 0x5636aac7f850 2024-04-21T23:09:26.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.227+0000 7f7deffff640 1 --2- 172.21.15.44:0/1782151945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7f850 0x5636aac940e0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f7de0009610 tx=0x7f7de00048a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4af219388ce4874e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:26.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.227+0000 7f7ddffff640 1 -- 172.21.15.44:0/1782151945 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de00b4070 con 0x5636aac7f850 2024-04-21T23:09:26.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.227+0000 7f7ddffff640 1 -- 172.21.15.44:0/1782151945 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7de009b480 con 0x5636aac7f850 2024-04-21T23:09:26.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.227+0000 7f7ddffff640 1 -- 172.21.15.44:0/1782151945 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de0020040 con 0x5636aac7f850 2024-04-21T23:09:26.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.227+0000 7f7dff3f9740 1 -- 172.21.15.44:0/1782151945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7f850 msgr2=0x5636aac940e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.227+0000 7f7dff3f9740 1 --2- 172.21.15.44:0/1782151945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7f850 0x5636aac940e0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f7de0009610 tx=0x7f7de00048a0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.227+0000 7f7dff3f9740 1 -- 172.21.15.44:0/1782151945 shutdown_connections 2024-04-21T23:09:26.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.227+0000 7f7dff3f9740 1 --2- 172.21.15.44:0/1782151945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636aac88430 0x5636aac79a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.228+0000 7f7dff3f9740 1 --2- 172.21.15.44:0/1782151945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7f850 0x5636aac940e0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.228+0000 7f7dff3f9740 1 --2- 172.21.15.44:0/1782151945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636aac7ef30 0x5636aac87b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.228+0000 7f7dff3f9740 1 -- 172.21.15.44:0/1782151945 >> 172.21.15.44:0/1782151945 conn(0x5636aac79660 msgr2=0x5636aac7aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:26.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.228+0000 7f7dff3f9740 1 -- 172.21.15.44:0/1782151945 shutdown_connections 2024-04-21T23:09:26.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.228+0000 7f7dff3f9740 1 -- 172.21.15.44:0/1782151945 wait complete. 2024-04-21T23:09:26.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.228+0000 7f7dff3f9740 1 -- start start 2024-04-21T23:09:26.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.228+0000 7f7dff3f9740 1 Processor -- start 2024-04-21T23:09:26.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.228+0000 7f7dff3f9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7ef30 0x5636aac8bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.228+0000 7f7dff3f9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636aac7f850 0x5636aac8c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.228+0000 7f7dff3f9740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636aac88430 0x5636aac8c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.228+0000 7f7dff3f9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7df81afae0 con 0x5636aac7ef30 2024-04-21T23:09:26.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.228+0000 7f7dff3f9740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7df81b0f50 con 0x5636aac88430 2024-04-21T23:09:26.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.228+0000 7f7dff3f9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7df81b01b0 con 0x5636aac7f850 2024-04-21T23:09:26.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.228+0000 7f7dff3f9740 10 client.0 _ll_register_callbacks cb 0x7f7df0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:26.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.228+0000 7f7df659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7ef30 0x5636aac8bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.228+0000 7f7df659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7ef30 0x5636aac8bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59834/0 (socket says 172.21.15.44:59834) 2024-04-21T23:09:26.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.228+0000 7f7df659c640 1 -- 172.21.15.44:0/2342946517 learned_addr learned my addr 172.21.15.44:0/2342946517 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:26.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.228+0000 7f7deffff640 1 --2- 172.21.15.44:0/2342946517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636aac7f850 0x5636aac8c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.228+0000 7f7df6d9d640 1 --2- 172.21.15.44:0/2342946517 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636aac88430 0x5636aac8c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7df659c640 1 -- 172.21.15.44:0/2342946517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636aac7f850 msgr2=0x5636aac8c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:26.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7df659c640 1 --2- 172.21.15.44:0/2342946517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636aac7f850 0x5636aac8c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7df659c640 1 -- 172.21.15.44:0/2342946517 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636aac88430 msgr2=0x5636aac8c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:26.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7df659c640 1 --2- 172.21.15.44:0/2342946517 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636aac88430 0x5636aac8c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7df659c640 1 -- 172.21.15.44:0/2342946517 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636aacfdd50 con 0x5636aac7ef30 2024-04-21T23:09:26.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7df6d9d640 1 --2- 172.21.15.44:0/2342946517 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636aac88430 0x5636aac8c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:09:26.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7df659c640 1 --2- 172.21.15.44:0/2342946517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7ef30 0x5636aac8bf50 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7de800ba90 tx=0x7f7de800df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c1a7a1f8c2a15c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:26.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7dddffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:09:26.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7dddffb640 1 -- 172.21.15.44:0/2342946517 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de800f650 con 0x5636aac7ef30 2024-04-21T23:09:26.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7dddffb640 1 -- 172.21.15.44:0/2342946517 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f7de8005de0 con 0x5636aac7ef30 2024-04-21T23:09:26.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7dff3f9740 1 -- 172.21.15.44:0/2342946517 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7de009b480 con 0x5636aac7ef30 2024-04-21T23:09:26.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7dddffb640 1 -- 172.21.15.44:0/2342946517 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7de8038050 con 0x5636aac7ef30 2024-04-21T23:09:26.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7dff3f9740 10 client.15708 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:26.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7dff3f9740 1 -- 172.21.15.44:0/2342946517 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5636aacff000 con 0x5636aac7ef30 2024-04-21T23:09:26.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7dddffb640 1 -- 172.21.15.44:0/2342946517 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7de009b480 con 0x5636aac7ef30 2024-04-21T23:09:26.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7dddffb640 1 -- 172.21.15.44:0/2342946517 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f7de8017420 con 0x5636aac7ef30 2024-04-21T23:09:26.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7dff3f9740 10 client.15708 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:26.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7dff3f9740 10 client.15708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:26.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7dff3f9740 20 client.15708 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:26.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7dff3f9740 1 --2- 172.21.15.44:0/2342946517 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5636aacff5e0 0x5636aad019d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7dff3f9740 4 client.15708 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:09:26.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.229+0000 7f7dff3f9740 1 -- 172.21.15.44:0/2342946517 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5636aabeadf0 con 0x5636aacff5e0 2024-04-21T23:09:26.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.230+0000 7f7deffff640 1 --2- 172.21.15.44:0/2342946517 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5636aacff5e0 0x5636aad019d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.230+0000 7f7dddffb640 10 client.15708 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:26.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.230+0000 7f7deffff640 1 --2- 172.21.15.44:0/2342946517 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5636aacff5e0 0x5636aad019d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84677fa73c3faea1 server_cookie=c09902c378a7eb69 in_seq=0 out_seq=0 2024-04-21T23:09:26.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.230+0000 7f7dddffb640 1 -- 172.21.15.44:0/2342946517 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5636aabeadf0 con 0x5636aacff5e0 2024-04-21T23:09:26.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.230+0000 7f7dddffb640 10 client.15708 handle_command_reply: tid=0 2024-04-21T23:09:26.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.230+0000 7f7dff3f9740 1 client.15708 shutdown 2024-04-21T23:09:26.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.230+0000 7f7dff3f9740 1 -- 172.21.15.44:0/2342946517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7ef30 msgr2=0x5636aac8bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.230+0000 7f7dff3f9740 1 --2- 172.21.15.44:0/2342946517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7ef30 0x5636aac8bf50 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7de800ba90 tx=0x7f7de800df80 comp rx=0 tx=0).stop 2024-04-21T23:09:26.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.230+0000 7f7dff3f9740 1 -- 172.21.15.44:0/2342946517 shutdown_connections 2024-04-21T23:09:26.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.230+0000 7f7dff3f9740 1 --2- 172.21.15.44:0/2342946517 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5636aacff5e0 0x5636aad019d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.230+0000 7f7dff3f9740 1 --2- 172.21.15.44:0/2342946517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636aac7f850 0x5636aac8c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.230+0000 7f7dff3f9740 1 --2- 172.21.15.44:0/2342946517 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636aac88430 0x5636aac8c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.230+0000 7f7dddffb640 0 client.15708 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:26.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.230+0000 7f7dff3f9740 1 --2- 172.21.15.44:0/2342946517 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636aac7ef30 0x5636aac8bf50 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.230+0000 7f7dff3f9740 1 -- 172.21.15.44:0/2342946517 >> 172.21.15.44:0/2342946517 conn(0x5636aac79660 msgr2=0x5636aac7b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:26.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.231+0000 7f7dff3f9740 1 -- 172.21.15.44:0/2342946517 shutdown_connections 2024-04-21T23:09:26.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.231+0000 7f7dff3f9740 1 -- 172.21.15.44:0/2342946517 wait complete. 2024-04-21T23:09:26.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.231+0000 7f7dff3f9740 20 client.15708 trim_cache size 0 max 16384 2024-04-21T23:09:26.245 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:09:26.245 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-21T23:09:26.245 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-21T23:09:26.245 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:09:26.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.231+0000 7f7dfc80e640 1 -- 172.21.15.44:0/284501915 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7de40a2e20 msgr2=0x7f7de40a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.231+0000 7f7dfc80e640 1 --2- 172.21.15.44:0/284501915 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7de40a2e20 0x7f7de40a52e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7df0004af0 tx=0x7f7df00324f0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.231+0000 7f7dfc80e640 1 -- 172.21.15.44:0/284501915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7df8100f70 msgr2=0x7f7df819a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.231+0000 7f7dfc80e640 1 --2- 172.21.15.44:0/284501915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7df8100f70 0x7f7df819a8b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f7de000d770 tx=0x7f7de000dc30 comp rx=0 tx=0).stop 2024-04-21T23:09:26.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.231+0000 7f7dfc80e640 1 -- 172.21.15.44:0/284501915 shutdown_connections 2024-04-21T23:09:26.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.231+0000 7f7dfc80e640 1 --2- 172.21.15.44:0/284501915 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7de40a2e20 0x7f7de40a52e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.231+0000 7f7dfc80e640 1 --2- 172.21.15.44:0/284501915 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7df810e0e0 0x7f7df819ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.231+0000 7f7dfc80e640 1 --2- 172.21.15.44:0/284501915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7df8100f70 0x7f7df819a8b0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.231+0000 7f7dfc80e640 1 --2- 172.21.15.44:0/284501915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7df81005c0 0x7f7df819a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.231+0000 7f7dfc80e640 1 -- 172.21.15.44:0/284501915 >> 172.21.15.44:0/284501915 conn(0x7f7df8077760 msgr2=0x7f7df810f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:26.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.231+0000 7f7dfc80e640 1 -- 172.21.15.44:0/284501915 shutdown_connections 2024-04-21T23:09:26.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.231+0000 7f7dfc80e640 1 -- 172.21.15.44:0/284501915 wait complete. 2024-04-21T23:09:26.248 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:09:26.248 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:09:26.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.353+0000 7f4e240f4640 1 -- 172.21.15.44:0/315686865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e1c101110 msgr2=0x7f4e1c101570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.353+0000 7f4e240f4640 1 --2- 172.21.15.44:0/315686865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e1c101110 0x7f4e1c101570 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c009930 tx=0x7f4e0c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:09:26.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.353+0000 7f4e240f4640 1 -- 172.21.15.44:0/315686865 shutdown_connections 2024-04-21T23:09:26.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.353+0000 7f4e240f4640 1 --2- 172.21.15.44:0/315686865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e1c101ab0 0x7f4e1c10d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.353+0000 7f4e240f4640 1 --2- 172.21.15.44:0/315686865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e1c101110 0x7f4e1c101570 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.353+0000 7f4e240f4640 1 --2- 172.21.15.44:0/315686865 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e1c1067b0 0x7f4e1c106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.354+0000 7f4e240f4640 1 -- 172.21.15.44:0/315686865 >> 172.21.15.44:0/315686865 conn(0x7f4e1c076c00 msgr2=0x7f4e1c077010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:26.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.354+0000 7f4e240f4640 1 -- 172.21.15.44:0/315686865 shutdown_connections 2024-04-21T23:09:26.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.354+0000 7f4e240f4640 1 -- 172.21.15.44:0/315686865 wait complete. 2024-04-21T23:09:26.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.354+0000 7f4e240f4640 1 Processor -- start 2024-04-21T23:09:26.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.354+0000 7f4e240f4640 1 -- start start 2024-04-21T23:09:26.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.354+0000 7f4e240f4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e1c101110 0x7f4e1c198fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.354+0000 7f4e240f4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e1c101ab0 0x7f4e1c1994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.354+0000 7f4e240f4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e1c1067b0 0x7f4e1c1a4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.354+0000 7f4e240f4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4e1c110b90 con 0x7f4e1c1067b0 2024-04-21T23:09:26.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.354+0000 7f4e240f4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4e1c110a10 con 0x7f4e1c101ab0 2024-04-21T23:09:26.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.354+0000 7f4e240f4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4e1c110d10 con 0x7f4e1c101110 2024-04-21T23:09:26.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.355+0000 7f4e22691640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e1c1067b0 0x7f4e1c1a4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.355+0000 7f4e22691640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e1c1067b0 0x7f4e1c1a4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59866/0 (socket says 172.21.15.44:59866) 2024-04-21T23:09:26.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.355+0000 7f4e22691640 1 -- 172.21.15.44:0/1733949219 learned_addr learned my addr 172.21.15.44:0/1733949219 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:26.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.355+0000 7f4e2168f640 1 --2- 172.21.15.44:0/1733949219 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e1c101ab0 0x7f4e1c1994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.355+0000 7f4e21e90640 1 --2- 172.21.15.44:0/1733949219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e1c101110 0x7f4e1c198fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.355+0000 7f4e22691640 1 -- 172.21.15.44:0/1733949219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e1c101110 msgr2=0x7f4e1c198fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.355+0000 7f4e22691640 1 --2- 172.21.15.44:0/1733949219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e1c101110 0x7f4e1c198fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.355+0000 7f4e22691640 1 -- 172.21.15.44:0/1733949219 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e1c101ab0 msgr2=0x7f4e1c1994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.355+0000 7f4e22691640 1 --2- 172.21.15.44:0/1733949219 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e1c101ab0 0x7f4e1c1994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.355+0000 7f4e22691640 1 -- 172.21.15.44:0/1733949219 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e1c1a4570 con 0x7f4e1c1067b0 2024-04-21T23:09:26.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.355+0000 7f4e22691640 1 --2- 172.21.15.44:0/1733949219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e1c1067b0 0x7f4e1c1a4030 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4e1400d750 tx=0x7f4e1400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:26.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.355+0000 7f4e12ffd640 1 -- 172.21.15.44:0/1733949219 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e14017070 con 0x7f4e1c1067b0 2024-04-21T23:09:26.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.355+0000 7f4e240f4640 1 -- 172.21.15.44:0/1733949219 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e1c1a4740 con 0x7f4e1c1067b0 2024-04-21T23:09:26.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.355+0000 7f4e12ffd640 1 -- 172.21.15.44:0/1733949219 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e14005500 con 0x7f4e1c1067b0 2024-04-21T23:09:26.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.355+0000 7f4e12ffd640 1 -- 172.21.15.44:0/1733949219 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e14004d90 con 0x7f4e1c1067b0 2024-04-21T23:09:26.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.355+0000 7f4e240f4640 1 -- 172.21.15.44:0/1733949219 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e1c1a4c00 con 0x7f4e1c1067b0 2024-04-21T23:09:26.358 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.356+0000 7f4e240f4640 1 -- 172.21.15.44:0/1733949219 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e1c068a40 con 0x7f4e1c1067b0 2024-04-21T23:09:26.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.357+0000 7f4e12ffd640 1 -- 172.21.15.44:0/1733949219 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f4e14004f30 con 0x7f4e1c1067b0 2024-04-21T23:09:26.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.358+0000 7f4e12ffd640 1 --2- 172.21.15.44:0/1733949219 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4e080a2e20 0x7f4e080a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.358+0000 7f4e21e90640 1 --2- 172.21.15.44:0/1733949219 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4e080a2e20 0x7f4e080a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.358+0000 7f4e12ffd640 1 -- 172.21.15.44:0/1733949219 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f4e140cf880 con 0x7f4e1c1067b0 2024-04-21T23:09:26.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.359+0000 7f4e21e90640 1 --2- 172.21.15.44:0/1733949219 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4e080a2e20 0x7f4e080a52e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4e18009370 tx=0x7f4e180092a0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:26.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.362+0000 7f4e12ffd640 1 -- 172.21.15.44:0/1733949219 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4e1409beb0 con 0x7f4e1c1067b0 2024-04-21T23:09:26.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:26 smithi044 ceph-mon[27934]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:09:26.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:26 smithi055 ceph-mon[31117]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:09:26.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:26 smithi069 ceph-mon[31816]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:09:26.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.590+0000 7f4e240f4640 1 -- 172.21.15.44:0/1733949219 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e1c108c80 con 0x7f4e1c1067b0 2024-04-21T23:09:26.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.591+0000 7f4e12ffd640 1 -- 172.21.15.44:0/1733949219 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11791 (secure 0 0 0) 0x7f4e1409b7b0 con 0x7f4e1c1067b0 2024-04-21T23:09:26.591 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:09:26.591 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:06:50.685663+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3743898853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3743898853},{"type":"v1","addr":"172.21.15.69:6839","nonce":3743898853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:09:26.591 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2024-04-21T23:09:26.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.593+0000 7f4e240f4640 1 -- 172.21.15.44:0/1733949219 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4e080a2e20 msgr2=0x7f4e080a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.593+0000 7f4e240f4640 1 --2- 172.21.15.44:0/1733949219 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4e080a2e20 0x7f4e080a52e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4e18009370 tx=0x7f4e180092a0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.593+0000 7f4e240f4640 1 -- 172.21.15.44:0/1733949219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e1c1067b0 msgr2=0x7f4e1c1a4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.593+0000 7f4e240f4640 1 --2- 172.21.15.44:0/1733949219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e1c1067b0 0x7f4e1c1a4030 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4e1400d750 tx=0x7f4e1400dc10 comp rx=0 tx=0).stop 2024-04-21T23:09:26.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.593+0000 7f4e240f4640 1 -- 172.21.15.44:0/1733949219 shutdown_connections 2024-04-21T23:09:26.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.593+0000 7f4e240f4640 1 --2- 172.21.15.44:0/1733949219 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4e080a2e20 0x7f4e080a52e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.593+0000 7f4e240f4640 1 --2- 172.21.15.44:0/1733949219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e1c1067b0 0x7f4e1c1a4030 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.593+0000 7f4e240f4640 1 --2- 172.21.15.44:0/1733949219 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e1c101ab0 0x7f4e1c1994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.593+0000 7f4e240f4640 1 --2- 172.21.15.44:0/1733949219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e1c101110 0x7f4e1c198fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.593+0000 7f4e240f4640 1 -- 172.21.15.44:0/1733949219 >> 172.21.15.44:0/1733949219 conn(0x7f4e1c076c00 msgr2=0x7f4e1c0fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:26.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.594+0000 7f4e240f4640 1 -- 172.21.15.44:0/1733949219 shutdown_connections 2024-04-21T23:09:26.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.594+0000 7f4e240f4640 1 -- 172.21.15.44:0/1733949219 wait complete. 2024-04-21T23:09:26.604 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:09:26.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.709+0000 7f8d04bbf640 1 -- 172.21.15.44:0/3382120439 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d001029b0 msgr2=0x7f8d00102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.709+0000 7f8d04bbf640 1 --2- 172.21.15.44:0/3382120439 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d001029b0 0x7f8d00102d90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0009900 tx=0x7f8cf002bcd0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.710+0000 7f8d04bbf640 1 -- 172.21.15.44:0/3382120439 shutdown_connections 2024-04-21T23:09:26.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.710+0000 7f8d04bbf640 1 --2- 172.21.15.44:0/3382120439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d0010bcf0 0x7f8d0010e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.710+0000 7f8d04bbf640 1 --2- 172.21.15.44:0/3382120439 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d001032d0 0x7f8d0010b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.710+0000 7f8d04bbf640 1 --2- 172.21.15.44:0/3382120439 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d001029b0 0x7f8d00102d90 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.710+0000 7f8d04bbf640 1 -- 172.21.15.44:0/3382120439 >> 172.21.15.44:0/3382120439 conn(0x7f8d000777f0 msgr2=0x7f8d00077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.710+0000 7f8d04bbf640 1 -- 172.21.15.44:0/3382120439 shutdown_connections 2024-04-21T23:09:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.710+0000 7f8d04bbf640 1 -- 172.21.15.44:0/3382120439 wait complete. 2024-04-21T23:09:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.710+0000 7f8d04bbf640 1 Processor -- start 2024-04-21T23:09:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.710+0000 7f8d04bbf640 1 -- start start 2024-04-21T23:09:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8d04bbf640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d001029b0 0x7f8d00197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8d04bbf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d001032d0 0x7f8d001984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8d04bbf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d0010bcf0 0x7f8d0019c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8d04bbf640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8d00111af0 con 0x7f8d001032d0 2024-04-21T23:09:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8d04bbf640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8d00111970 con 0x7f8d001029b0 2024-04-21T23:09:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8d04bbf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8d00111c70 con 0x7f8d0010bcf0 2024-04-21T23:09:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8cf7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d001032d0 0x7f8d001984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8cf7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d001032d0 0x7f8d001984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59892/0 (socket says 172.21.15.44:59892) 2024-04-21T23:09:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8cf7fff640 1 -- 172.21.15.44:0/1982808317 learned_addr learned my addr 172.21.15.44:0/1982808317 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8cfe59c640 1 --2- 172.21.15.44:0/1982808317 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d001029b0 0x7f8d00197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8cfed9d640 1 --2- 172.21.15.44:0/1982808317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d0010bcf0 0x7f8d0019c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8cf7fff640 1 -- 172.21.15.44:0/1982808317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d0010bcf0 msgr2=0x7f8d0019c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8cf7fff640 1 --2- 172.21.15.44:0/1982808317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d0010bcf0 0x7f8d0019c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8cf7fff640 1 -- 172.21.15.44:0/1982808317 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d001029b0 msgr2=0x7f8d00197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8cf7fff640 1 --2- 172.21.15.44:0/1982808317 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d001029b0 0x7f8d00197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8cf7fff640 1 -- 172.21.15.44:0/1982808317 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d0019cf60 con 0x7f8d001032d0 2024-04-21T23:09:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8cf7fff640 1 --2- 172.21.15.44:0/1982808317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d001032d0 0x7f8d001984b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8ce800d950 tx=0x7f8ce800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8cf77fe640 1 -- 172.21.15.44:0/1982808317 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8017070 con 0x7f8d001032d0 2024-04-21T23:09:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8d04bbf640 1 -- 172.21.15.44:0/1982808317 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d0019d250 con 0x7f8d001032d0 2024-04-21T23:09:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8cf77fe640 1 -- 172.21.15.44:0/1982808317 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ce8009c30 con 0x7f8d001032d0 2024-04-21T23:09:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8cf77fe640 1 -- 172.21.15.44:0/1982808317 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce800f6b0 con 0x7f8d001032d0 2024-04-21T23:09:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.711+0000 7f8d04bbf640 1 -- 172.21.15.44:0/1982808317 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d001a4ae0 con 0x7f8d001032d0 2024-04-21T23:09:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.714+0000 7f8cf77fe640 1 -- 172.21.15.44:0/1982808317 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f8ce80109d0 con 0x7f8d001032d0 2024-04-21T23:09:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.714+0000 7f8d077a6740 1 Processor -- start 2024-04-21T23:09:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.714+0000 7f8d077a6740 1 -- start start 2024-04-21T23:09:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8d077a6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b204d50 0x55f39b205130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8d077a6740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b205670 0x55f39b2123f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8d077a6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b212930 0x55f39b214d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8d077a6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f39b169530 con 0x55f39b212930 2024-04-21T23:09:26.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8d077a6740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55f39b111830 con 0x55f39b205670 2024-04-21T23:09:26.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8d077a6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f39b1654f0 con 0x55f39b204d50 2024-04-21T23:09:26.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8cf77fe640 1 --2- 172.21.15.44:0/1982808317 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f8cec0a2e20 0x7f8cec0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8cf7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b212930 0x55f39b214d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8cfed9d640 1 --2- 172.21.15.44:0/1982808317 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f8cec0a2e20 0x7f8cec0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8cfe59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b205670 0x55f39b2123f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8cf7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b212930 0x55f39b214d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59902/0 (socket says 172.21.15.44:59902) 2024-04-21T23:09:26.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8cf7fff640 1 -- 172.21.15.44:0/2752200955 learned_addr learned my addr 172.21.15.44:0/2752200955 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:26.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8cf77fe640 1 -- 172.21.15.44:0/1982808317 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f8ce80cf6c0 con 0x7f8d001032d0 2024-04-21T23:09:26.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8cfed9d640 1 --2- 172.21.15.44:0/2752200955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b204d50 0x55f39b205130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8cf7fff640 1 -- 172.21.15.44:0/2752200955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b204d50 msgr2=0x55f39b205130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:26.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8cf7fff640 1 --2- 172.21.15.44:0/2752200955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b204d50 0x55f39b205130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8cf7fff640 1 -- 172.21.15.44:0/2752200955 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b205670 msgr2=0x55f39b2123f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:26.720 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8cf7fff640 1 --2- 172.21.15.44:0/2752200955 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b205670 0x55f39b2123f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8cf7fff640 1 -- 172.21.15.44:0/2752200955 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f39b1352c0 con 0x55f39b212930 2024-04-21T23:09:26.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8cfe59c640 1 --2- 172.21.15.44:0/2752200955 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b205670 0x55f39b2123f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:09:26.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8cfed9d640 1 --2- 172.21.15.44:0/1982808317 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f8cec0a2e20 0x7f8cec0a52e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8005fd0 tx=0x7f8cf800a920 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:26.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.715+0000 7f8cf7fff640 1 --2- 172.21.15.44:0/2752200955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b212930 0x55f39b214d20 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8ce80cca00 tx=0x7f8ce8090a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=232e80487716d5d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:26.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.716+0000 7f8cf4ff9640 1 -- 172.21.15.44:0/2752200955 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8020070 con 0x55f39b212930 2024-04-21T23:09:26.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.716+0000 7f8cf4ff9640 1 -- 172.21.15.44:0/2752200955 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ce8009de0 con 0x55f39b212930 2024-04-21T23:09:26.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.716+0000 7f8cf4ff9640 1 -- 172.21.15.44:0/2752200955 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce809f9a0 con 0x55f39b212930 2024-04-21T23:09:26.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.716+0000 7f8d077a6740 1 -- 172.21.15.44:0/2752200955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b212930 msgr2=0x55f39b214d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.716+0000 7f8d077a6740 1 --2- 172.21.15.44:0/2752200955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b212930 0x55f39b214d20 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8ce80cca00 tx=0x7f8ce8090a80 comp rx=0 tx=0).stop 2024-04-21T23:09:26.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.716+0000 7f8d077a6740 1 -- 172.21.15.44:0/2752200955 shutdown_connections 2024-04-21T23:09:26.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.716+0000 7f8d077a6740 1 --2- 172.21.15.44:0/2752200955 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b212930 0x55f39b214d20 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.716+0000 7f8d077a6740 1 --2- 172.21.15.44:0/2752200955 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b205670 0x55f39b2123f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.716+0000 7f8d077a6740 1 --2- 172.21.15.44:0/2752200955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b204d50 0x55f39b205130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.721 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.716+0000 7f8d077a6740 1 -- 172.21.15.44:0/2752200955 >> 172.21.15.44:0/2752200955 conn(0x55f39b1ff750 msgr2=0x55f39b1ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:26.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.717+0000 7f8d077a6740 1 -- 172.21.15.44:0/2752200955 shutdown_connections 2024-04-21T23:09:26.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.717+0000 7f8d077a6740 1 -- 172.21.15.44:0/2752200955 wait complete. 2024-04-21T23:09:26.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.717+0000 7f8d077a6740 1 -- start start 2024-04-21T23:09:26.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.717+0000 7f8d077a6740 1 Processor -- start 2024-04-21T23:09:26.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.717+0000 7f8d077a6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b204d50 0x55f39b212320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.717+0000 7f8d077a6740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b205670 0x55f39b20f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.717+0000 7f8d077a6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b212930 0x55f39b20fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.717+0000 7f8d077a6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f39b215690 con 0x55f39b204d50 2024-04-21T23:09:26.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.717+0000 7f8d077a6740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55f39b124f10 con 0x55f39b205670 2024-04-21T23:09:26.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.717+0000 7f8d077a6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f39b215810 con 0x55f39b212930 2024-04-21T23:09:26.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.717+0000 7f8d077a6740 10 client.0 _ll_register_callbacks cb 0x55f39ae7cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:26.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.717+0000 7f8cfe59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b204d50 0x55f39b212320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.717+0000 7f8cfe59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b204d50 0x55f39b212320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59914/0 (socket says 172.21.15.44:59914) 2024-04-21T23:09:26.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.717+0000 7f8cfe59c640 1 -- 172.21.15.44:0/1797535094 learned_addr learned my addr 172.21.15.44:0/1797535094 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:26.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.717+0000 7f8cf7fff640 1 --2- 172.21.15.44:0/1797535094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b205670 0x55f39b20f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.717+0000 7f8cfed9d640 1 --2- 172.21.15.44:0/1797535094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b212930 0x55f39b20fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.718+0000 7f8cfe59c640 1 -- 172.21.15.44:0/1797535094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b212930 msgr2=0x55f39b20fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:26.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.718+0000 7f8cfe59c640 1 --2- 172.21.15.44:0/1797535094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b212930 0x55f39b20fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.718+0000 7f8cfe59c640 1 -- 172.21.15.44:0/1797535094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b205670 msgr2=0x55f39b20f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:26.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.718+0000 7f8cfe59c640 1 --2- 172.21.15.44:0/1797535094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b205670 0x55f39b20f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.718+0000 7f8cfe59c640 1 -- 172.21.15.44:0/1797535094 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f39b0f6c80 con 0x55f39b204d50 2024-04-21T23:09:26.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.718+0000 7f8cfed9d640 1 --2- 172.21.15.44:0/1797535094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b212930 0x55f39b20fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:09:26.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.718+0000 7f8cfe59c640 1 --2- 172.21.15.44:0/1797535094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b204d50 0x55f39b212320 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8cf002c210 tx=0x7f8cf002ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf1bc6a874bb712 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:26.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.718+0000 7f8ce5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:09:26.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.718+0000 7f8ce5ffb640 1 -- 172.21.15.44:0/1797535094 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf0004400 con 0x55f39b204d50 2024-04-21T23:09:26.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.718+0000 7f8ce5ffb640 1 -- 172.21.15.44:0/1797535094 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f8cf00046c0 con 0x55f39b204d50 2024-04-21T23:09:26.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.718+0000 7f8d077a6740 1 -- 172.21.15.44:0/1797535094 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ce8009de0 con 0x55f39b204d50 2024-04-21T23:09:26.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.718+0000 7f8ce5ffb640 1 -- 172.21.15.44:0/1797535094 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8cf00048c0 con 0x55f39b204d50 2024-04-21T23:09:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.718+0000 7f8d077a6740 10 client.15726 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.718+0000 7f8d077a6740 1 -- 172.21.15.44:0/1797535094 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f39ae7d3f0 con 0x55f39b204d50 2024-04-21T23:09:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.718+0000 7f8ce5ffb640 1 -- 172.21.15.44:0/1797535094 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ce8009de0 con 0x55f39b204d50 2024-04-21T23:09:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.719+0000 7f8ce5ffb640 1 -- 172.21.15.44:0/1797535094 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f8cf00035f0 con 0x55f39b204d50 2024-04-21T23:09:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.719+0000 7f8d077a6740 10 client.15726 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.719+0000 7f8d077a6740 10 client.15726 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.719+0000 7f8d077a6740 20 client.15726 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.719+0000 7f8d077a6740 1 --2- 172.21.15.44:0/1797535094 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f39b222a00 0x55f39b224e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.719+0000 7f8d077a6740 4 client.15726 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:09:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.719+0000 7f8d077a6740 1 -- 172.21.15.44:0/1797535094 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f39b176a70 con 0x55f39b222a00 2024-04-21T23:09:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.719+0000 7f8cf7fff640 1 --2- 172.21.15.44:0/1797535094 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f39b222a00 0x55f39b224e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.719+0000 7f8ce5ffb640 10 client.15726 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.719+0000 7f8cf7fff640 1 --2- 172.21.15.44:0/1797535094 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f39b222a00 0x55f39b224e30 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28ffdecb48f8f508 server_cookie=de10dc6e51afb7b7 in_seq=0 out_seq=0 2024-04-21T23:09:26.725 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.720+0000 7f8ce5ffb640 1 -- 172.21.15.44:0/1797535094 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f39b176a70 con 0x55f39b222a00 2024-04-21T23:09:26.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.720+0000 7f8ce5ffb640 10 client.15726 handle_command_reply: tid=0 2024-04-21T23:09:26.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.720+0000 7f8d077a6740 1 client.15726 shutdown 2024-04-21T23:09:26.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.720+0000 7f8d077a6740 1 -- 172.21.15.44:0/1797535094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b204d50 msgr2=0x55f39b212320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.720+0000 7f8d077a6740 1 --2- 172.21.15.44:0/1797535094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b204d50 0x55f39b212320 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8cf002c210 tx=0x7f8cf002ce60 comp rx=0 tx=0).stop 2024-04-21T23:09:26.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.720+0000 7f8d077a6740 1 -- 172.21.15.44:0/1797535094 shutdown_connections 2024-04-21T23:09:26.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.720+0000 7f8d077a6740 1 --2- 172.21.15.44:0/1797535094 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f39b222a00 0x55f39b224e30 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.720+0000 7f8d077a6740 1 --2- 172.21.15.44:0/1797535094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b212930 0x55f39b20fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.720+0000 7f8d077a6740 1 --2- 172.21.15.44:0/1797535094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b205670 0x55f39b20f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.720+0000 7f8ce5ffb640 0 client.15726 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:26.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.720+0000 7f8d077a6740 1 --2- 172.21.15.44:0/1797535094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b204d50 0x55f39b212320 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.720+0000 7f8d077a6740 1 -- 172.21.15.44:0/1797535094 >> 172.21.15.44:0/1797535094 conn(0x55f39b1ff750 msgr2=0x55f39b2014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:26.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.720+0000 7f8d077a6740 1 -- 172.21.15.44:0/1797535094 shutdown_connections 2024-04-21T23:09:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.720+0000 7f8d077a6740 1 -- 172.21.15.44:0/1797535094 wait complete. 2024-04-21T23:09:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.721+0000 7f8d077a6740 20 client.15726 trim_cache size 0 max 16384 2024-04-21T23:09:26.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.734+0000 7f8d077a6740 1 Processor -- start 2024-04-21T23:09:26.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.734+0000 7f8d077a6740 1 -- start start 2024-04-21T23:09:26.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.734+0000 7f8d077a6740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b204d50 0x55f39b21e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.735+0000 7f8d077a6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b205670 0x55f39b211de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.735+0000 7f8d077a6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b212930 0x55f39b212320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.735+0000 7f8d077a6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8d001aa990 con 0x55f39b212930 2024-04-21T23:09:26.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.735+0000 7f8d077a6740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8d001ad210 con 0x55f39b204d50 2024-04-21T23:09:26.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.735+0000 7f8d077a6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8d001ab000 con 0x55f39b205670 2024-04-21T23:09:26.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.735+0000 7f8cfed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b212930 0x55f39b212320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.735+0000 7f8cfed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b212930 0x55f39b212320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59918/0 (socket says 172.21.15.44:59918) 2024-04-21T23:09:26.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.735+0000 7f8cfed9d640 1 -- 172.21.15.44:0/2749369270 learned_addr learned my addr 172.21.15.44:0/2749369270 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:26.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.735+0000 7f8cfe59c640 1 --2- 172.21.15.44:0/2749369270 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b204d50 0x55f39b21e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.735+0000 7f8cf7fff640 1 --2- 172.21.15.44:0/2749369270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b205670 0x55f39b211de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.735+0000 7f8cfed9d640 1 -- 172.21.15.44:0/2749369270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b205670 msgr2=0x55f39b211de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:26.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.735+0000 7f8cfed9d640 1 --2- 172.21.15.44:0/2749369270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b205670 0x55f39b211de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.735+0000 7f8cfed9d640 1 -- 172.21.15.44:0/2749369270 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b204d50 msgr2=0x55f39b21e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:26.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.735+0000 7f8cfed9d640 1 --2- 172.21.15.44:0/2749369270 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b204d50 0x55f39b21e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.735+0000 7f8cfed9d640 1 -- 172.21.15.44:0/2749369270 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f39b2054a0 con 0x55f39b212930 2024-04-21T23:09:26.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.735+0000 7f8cfe59c640 1 --2- 172.21.15.44:0/2749369270 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b204d50 0x55f39b21e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:09:26.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.735+0000 7f8cf7fff640 1 --2- 172.21.15.44:0/2749369270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b205670 0x55f39b211de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:09:26.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.735+0000 7f8cfed9d640 1 --2- 172.21.15.44:0/2749369270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b212930 0x55f39b212320 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8005f90 tx=0x7f8cf801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a65dc70013f33dd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:26.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.735+0000 7f8ce7fff640 1 -- 172.21.15.44:0/2749369270 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf8024070 con 0x55f39b212930 2024-04-21T23:09:26.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.736+0000 7f8ce7fff640 1 -- 172.21.15.44:0/2749369270 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cf8004d20 con 0x55f39b212930 2024-04-21T23:09:26.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.736+0000 7f8ce7fff640 1 -- 172.21.15.44:0/2749369270 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf8005590 con 0x55f39b212930 2024-04-21T23:09:26.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.736+0000 7f8d077a6740 1 -- 172.21.15.44:0/2749369270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b212930 msgr2=0x55f39b212320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.736+0000 7f8d077a6740 1 --2- 172.21.15.44:0/2749369270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b212930 0x55f39b212320 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8005f90 tx=0x7f8cf801a000 comp rx=0 tx=0).stop 2024-04-21T23:09:26.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.736+0000 7f8d077a6740 1 -- 172.21.15.44:0/2749369270 shutdown_connections 2024-04-21T23:09:26.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.736+0000 7f8d077a6740 1 --2- 172.21.15.44:0/2749369270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b205670 0x55f39b211de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.736+0000 7f8d077a6740 1 --2- 172.21.15.44:0/2749369270 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b212930 0x55f39b212320 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.736+0000 7f8d077a6740 1 --2- 172.21.15.44:0/2749369270 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b204d50 0x55f39b21e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.736+0000 7f8d077a6740 1 -- 172.21.15.44:0/2749369270 >> 172.21.15.44:0/2749369270 conn(0x55f39b1ff750 msgr2=0x55f39b20ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:26.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.736+0000 7f8d077a6740 1 -- 172.21.15.44:0/2749369270 shutdown_connections 2024-04-21T23:09:26.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.736+0000 7f8d077a6740 1 -- 172.21.15.44:0/2749369270 wait complete. 2024-04-21T23:09:26.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.737+0000 7f8d077a6740 1 -- start start 2024-04-21T23:09:26.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.737+0000 7f8d077a6740 1 Processor -- start 2024-04-21T23:09:26.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.737+0000 7f8d077a6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b204d50 0x55f39b216ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.737+0000 7f8d077a6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b205670 0x55f39b217020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.737+0000 7f8d077a6740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b212930 0x55f39b287f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.737+0000 7f8d077a6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8d001ab6d0 con 0x55f39b204d50 2024-04-21T23:09:26.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.737+0000 7f8d077a6740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8d001acb40 con 0x55f39b212930 2024-04-21T23:09:26.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.737+0000 7f8d077a6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8d001abda0 con 0x55f39b205670 2024-04-21T23:09:26.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.737+0000 7f8d077a6740 10 client.0 _ll_register_callbacks cb 0x7f8cf0005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:26.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.737+0000 7f8cfe59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b204d50 0x55f39b216ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.737+0000 7f8cfe59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b204d50 0x55f39b216ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59928/0 (socket says 172.21.15.44:59928) 2024-04-21T23:09:26.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.737+0000 7f8cfe59c640 1 -- 172.21.15.44:0/3249140168 learned_addr learned my addr 172.21.15.44:0/3249140168 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:26.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.737+0000 7f8cfed9d640 1 --2- 172.21.15.44:0/3249140168 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b212930 0x55f39b287f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.737+0000 7f8cf7fff640 1 --2- 172.21.15.44:0/3249140168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b205670 0x55f39b217020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.737+0000 7f8cfe59c640 1 -- 172.21.15.44:0/3249140168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b205670 msgr2=0x55f39b217020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:26.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.737+0000 7f8cfe59c640 1 --2- 172.21.15.44:0/3249140168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b205670 0x55f39b217020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.737+0000 7f8cfe59c640 1 -- 172.21.15.44:0/3249140168 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b212930 msgr2=0x55f39b287f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:26.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.737+0000 7f8cfe59c640 1 --2- 172.21.15.44:0/3249140168 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b212930 0x55f39b287f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.737+0000 7f8cfe59c640 1 -- 172.21.15.44:0/3249140168 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f39af477d0 con 0x55f39b204d50 2024-04-21T23:09:26.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.737+0000 7f8cfed9d640 1 --2- 172.21.15.44:0/3249140168 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b212930 0x55f39b287f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:09:26.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.738+0000 7f8cfe59c640 1 --2- 172.21.15.44:0/3249140168 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b204d50 0x55f39b216ae0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f8cf002bd10 tx=0x7f8cf002c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b53e7e635ee14f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:26.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.738+0000 7f8ce5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:09:26.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.738+0000 7f8ce5ffb640 1 -- 172.21.15.44:0/3249140168 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf005a020 con 0x55f39b204d50 2024-04-21T23:09:26.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.738+0000 7f8ce5ffb640 1 -- 172.21.15.44:0/3249140168 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f8cf0003f10 con 0x55f39b204d50 2024-04-21T23:09:26.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.738+0000 7f8d077a6740 1 -- 172.21.15.44:0/3249140168 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8cf8004d20 con 0x55f39b204d50 2024-04-21T23:09:26.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.738+0000 7f8ce5ffb640 1 -- 172.21.15.44:0/3249140168 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8cf0065050 con 0x55f39b204d50 2024-04-21T23:09:26.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.738+0000 7f8d077a6740 10 client.15732 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:26.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.738+0000 7f8d077a6740 1 -- 172.21.15.44:0/3249140168 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f39b289540 con 0x55f39b204d50 2024-04-21T23:09:26.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.738+0000 7f8ce5ffb640 1 -- 172.21.15.44:0/3249140168 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cf8004d20 con 0x55f39b204d50 2024-04-21T23:09:26.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.738+0000 7f8ce5ffb640 1 -- 172.21.15.44:0/3249140168 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f8cf0004a60 con 0x55f39b204d50 2024-04-21T23:09:26.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.738+0000 7f8d077a6740 10 client.15732 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:26.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.738+0000 7f8d077a6740 10 client.15732 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:26.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.738+0000 7f8d077a6740 20 client.15732 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:26.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.738+0000 7f8d077a6740 1 --2- 172.21.15.44:0/3249140168 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f39b289ce0 0x55f39b28c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:26.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.738+0000 7f8d077a6740 4 client.15732 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:09:26.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.738+0000 7f8d077a6740 1 -- 172.21.15.44:0/3249140168 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f39b170dc0 con 0x55f39b289ce0 2024-04-21T23:09:26.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.739+0000 7f8cf7fff640 1 --2- 172.21.15.44:0/3249140168 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f39b289ce0 0x55f39b28c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:26.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.739+0000 7f8ce5ffb640 10 client.15732 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:26.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.739+0000 7f8cf7fff640 1 --2- 172.21.15.44:0/3249140168 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f39b289ce0 0x55f39b28c100 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d63a7305d5d7b80 server_cookie=e89139100a316f51 in_seq=0 out_seq=0 2024-04-21T23:09:26.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.739+0000 7f8ce5ffb640 1 -- 172.21.15.44:0/3249140168 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f39b170dc0 con 0x55f39b289ce0 2024-04-21T23:09:26.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.739+0000 7f8ce5ffb640 10 client.15732 handle_command_reply: tid=0 2024-04-21T23:09:26.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.739+0000 7f8d077a6740 1 client.15732 shutdown 2024-04-21T23:09:26.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.739+0000 7f8d077a6740 1 -- 172.21.15.44:0/3249140168 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b204d50 msgr2=0x55f39b216ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.739+0000 7f8d077a6740 1 --2- 172.21.15.44:0/3249140168 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b204d50 0x55f39b216ae0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f8cf002bd10 tx=0x7f8cf002c440 comp rx=0 tx=0).stop 2024-04-21T23:09:26.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.739+0000 7f8d077a6740 1 -- 172.21.15.44:0/3249140168 shutdown_connections 2024-04-21T23:09:26.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.739+0000 7f8d077a6740 1 --2- 172.21.15.44:0/3249140168 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f39b289ce0 0x55f39b28c100 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.739+0000 7f8d077a6740 1 --2- 172.21.15.44:0/3249140168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f39b205670 0x55f39b217020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.739+0000 7f8d077a6740 1 --2- 172.21.15.44:0/3249140168 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f39b212930 0x55f39b287f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.739+0000 7f8ce5ffb640 0 client.15732 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:26.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.739+0000 7f8d077a6740 1 --2- 172.21.15.44:0/3249140168 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f39b204d50 0x55f39b216ae0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.739+0000 7f8d077a6740 1 -- 172.21.15.44:0/3249140168 >> 172.21.15.44:0/3249140168 conn(0x55f39b1ff750 msgr2=0x55f39b1fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:26.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.739+0000 7f8d077a6740 1 -- 172.21.15.44:0/3249140168 shutdown_connections 2024-04-21T23:09:26.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.739+0000 7f8d077a6740 1 -- 172.21.15.44:0/3249140168 wait complete. 2024-04-21T23:09:26.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.740+0000 7f8d077a6740 20 client.15732 trim_cache size 0 max 16384 2024-04-21T23:09:26.746 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-21T23:09:26.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.740+0000 7f8d04bbf640 1 -- 172.21.15.44:0/1982808317 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f8cec0a2e20 msgr2=0x7f8cec0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.740+0000 7f8d04bbf640 1 --2- 172.21.15.44:0/1982808317 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f8cec0a2e20 0x7f8cec0a52e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8005fd0 tx=0x7f8cf800a920 comp rx=0 tx=0).stop 2024-04-21T23:09:26.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.740+0000 7f8d04bbf640 1 -- 172.21.15.44:0/1982808317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d001032d0 msgr2=0x7f8d001984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:26.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.740+0000 7f8d04bbf640 1 --2- 172.21.15.44:0/1982808317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d001032d0 0x7f8d001984b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8ce800d950 tx=0x7f8ce800de10 comp rx=0 tx=0).stop 2024-04-21T23:09:26.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.740+0000 7f8d04bbf640 1 -- 172.21.15.44:0/1982808317 shutdown_connections 2024-04-21T23:09:26.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.740+0000 7f8d04bbf640 1 --2- 172.21.15.44:0/1982808317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d0010bcf0 0x7f8d0019c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.740+0000 7f8d04bbf640 1 --2- 172.21.15.44:0/1982808317 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f8cec0a2e20 0x7f8cec0a52e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.740+0000 7f8d04bbf640 1 --2- 172.21.15.44:0/1982808317 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8d001032d0 0x7f8d001984b0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.740+0000 7f8d04bbf640 1 --2- 172.21.15.44:0/1982808317 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d001029b0 0x7f8d00197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:26.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.740+0000 7f8d04bbf640 1 -- 172.21.15.44:0/1982808317 >> 172.21.15.44:0/1982808317 conn(0x7f8d000777f0 msgr2=0x7f8d000fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:26.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.740+0000 7f8d04bbf640 1 -- 172.21.15.44:0/1982808317 shutdown_connections 2024-04-21T23:09:26.748 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:26.740+0000 7f8d04bbf640 1 -- 172.21.15.44:0/1982808317 wait complete. 2024-04-21T23:09:27.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:27 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1733949219' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:09:27.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:27 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1733949219' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:09:27.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:27 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1733949219' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:09:27.754 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:09:27.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:09:27.755 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:09:27.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.865+0000 7fc4f9b5c640 1 -- 172.21.15.44:0/975736873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4f4107720 msgr2=0x7fc4f4107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:27.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.865+0000 7fc4f9b5c640 1 --2- 172.21.15.44:0/975736873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4f4107720 0x7fc4f4107b00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009930 tx=0x7fc4e402bc30 comp rx=0 tx=0).stop 2024-04-21T23:09:27.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.865+0000 7fc4f9b5c640 1 -- 172.21.15.44:0/975736873 shutdown_connections 2024-04-21T23:09:27.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.865+0000 7fc4f9b5c640 1 --2- 172.21.15.44:0/975736873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4f4101710 0x7fc4f4101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.865+0000 7fc4f9b5c640 1 --2- 172.21.15.44:0/975736873 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4f41020d0 0x7fc4f410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.865+0000 7fc4f9b5c640 1 --2- 172.21.15.44:0/975736873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4f4107720 0x7fc4f4107b00 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.865+0000 7fc4f9b5c640 1 -- 172.21.15.44:0/975736873 >> 172.21.15.44:0/975736873 conn(0x7fc4f40fd200 msgr2=0x7fc4f40ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:27.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.865+0000 7fc4f9b5c640 1 -- 172.21.15.44:0/975736873 shutdown_connections 2024-04-21T23:09:27.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.865+0000 7fc4f9b5c640 1 -- 172.21.15.44:0/975736873 wait complete. 2024-04-21T23:09:27.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.866+0000 7fc4f9b5c640 1 Processor -- start 2024-04-21T23:09:27.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.866+0000 7fc4f9b5c640 1 -- start start 2024-04-21T23:09:27.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.866+0000 7fc4f9b5c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4f4101710 0x7fc4f4073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:27.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.866+0000 7fc4f9b5c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4f41020d0 0x7fc4f4073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:27.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.866+0000 7fc4f9b5c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4f4107720 0x7fc4f40744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:27.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.866+0000 7fc4f9b5c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f4111b30 con 0x7fc4f4101710 2024-04-21T23:09:27.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.866+0000 7fc4f9b5c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f41119b0 con 0x7fc4f41020d0 2024-04-21T23:09:27.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.866+0000 7fc4f9b5c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f4111cb0 con 0x7fc4f4107720 2024-04-21T23:09:27.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.866+0000 7fc4f37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4f4101710 0x7fc4f4073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:27.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.866+0000 7fc4f37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4f4101710 0x7fc4f4073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59936/0 (socket says 172.21.15.44:59936) 2024-04-21T23:09:27.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.866+0000 7fc4f37fe640 1 -- 172.21.15.44:0/3794833999 learned_addr learned my addr 172.21.15.44:0/3794833999 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:27.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.866+0000 7fc4f3fff640 1 --2- 172.21.15.44:0/3794833999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4f4107720 0x7fc4f40744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:27.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.866+0000 7fc4f2ffd640 1 --2- 172.21.15.44:0/3794833999 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4f41020d0 0x7fc4f4073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:27.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.866+0000 7fc4f37fe640 1 -- 172.21.15.44:0/3794833999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4f4107720 msgr2=0x7fc4f40744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:27.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.866+0000 7fc4f37fe640 1 --2- 172.21.15.44:0/3794833999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4f4107720 0x7fc4f40744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.866+0000 7fc4f37fe640 1 -- 172.21.15.44:0/3794833999 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4f41020d0 msgr2=0x7fc4f4073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:27.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.866+0000 7fc4f37fe640 1 --2- 172.21.15.44:0/3794833999 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4f41020d0 0x7fc4f4073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.867+0000 7fc4f37fe640 1 -- 172.21.15.44:0/3794833999 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f41a64d0 con 0x7fc4f4101710 2024-04-21T23:09:27.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.867+0000 7fc4f37fe640 1 --2- 172.21.15.44:0/3794833999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4f4101710 0x7fc4f4073a40 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009230 tx=0x7fc4e402cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:27.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.867+0000 7fc4f0ff9640 1 -- 172.21.15.44:0/3794833999 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4041070 con 0x7fc4f4101710 2024-04-21T23:09:27.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.867+0000 7fc4f0ff9640 1 -- 172.21.15.44:0/3794833999 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4e4004450 con 0x7fc4f4101710 2024-04-21T23:09:27.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.867+0000 7fc4f9b5c640 1 -- 172.21.15.44:0/3794833999 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4f41a6760 con 0x7fc4f4101710 2024-04-21T23:09:27.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.867+0000 7fc4f0ff9640 1 -- 172.21.15.44:0/3794833999 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4034440 con 0x7fc4f4101710 2024-04-21T23:09:27.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.867+0000 7fc4f9b5c640 1 -- 172.21.15.44:0/3794833999 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4f41a6ca0 con 0x7fc4f4101710 2024-04-21T23:09:27.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.869+0000 7fc4f0ff9640 1 -- 172.21.15.44:0/3794833999 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7fc4e4044070 con 0x7fc4f4101710 2024-04-21T23:09:27.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.870+0000 7fc4f0ff9640 1 --2- 172.21.15.44:0/3794833999 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc4dc0a2ea0 0x7fc4dc0a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:27.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.870+0000 7fc4f2ffd640 1 --2- 172.21.15.44:0/3794833999 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc4dc0a2ea0 0x7fc4dc0a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:27.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.870+0000 7fc4f0ff9640 1 -- 172.21.15.44:0/3794833999 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fc4e40f1410 con 0x7fc4f4101710 2024-04-21T23:09:27.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.870+0000 7fc4f2ffd640 1 --2- 172.21.15.44:0/3794833999 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc4dc0a2ea0 0x7fc4dc0a5360 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8005fd0 tx=0x7fc4e8005850 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:27.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.870+0000 7fc4fc817740 1 Processor -- start 2024-04-21T23:09:27.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.871+0000 7fc4fc817740 1 -- start start 2024-04-21T23:09:27.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.871+0000 7fc4fc817740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556603987720 0x556603986470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:27.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.871+0000 7fc4fc817740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556603986a40 0x556603986ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:27.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.871+0000 7fc4fc817740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55660398fdb0 0x5566039921a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:27.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.871+0000 7fc4fc817740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5566038ea170 con 0x55660398fdb0 2024-04-21T23:09:27.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.871+0000 7fc4fc817740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5566038927c0 con 0x556603986a40 2024-04-21T23:09:27.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.871+0000 7fc4fc817740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5566038e6130 con 0x556603987720 2024-04-21T23:09:27.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.871+0000 7fc4f3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55660398fdb0 0x5566039921a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:27.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.871+0000 7fc4f3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55660398fdb0 0x5566039921a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59940/0 (socket says 172.21.15.44:59940) 2024-04-21T23:09:27.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.871+0000 7fc4f3fff640 1 -- 172.21.15.44:0/2009981724 learned_addr learned my addr 172.21.15.44:0/2009981724 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:27.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.871+0000 7fc4f2ffd640 1 --2- 172.21.15.44:0/2009981724 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556603986a40 0x556603986ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:27.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.871+0000 7fc4f37fe640 1 --2- 172.21.15.44:0/2009981724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556603987720 0x556603986470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:27.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.871+0000 7fc4f3fff640 1 -- 172.21.15.44:0/2009981724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556603987720 msgr2=0x556603986470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:27.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.871+0000 7fc4f3fff640 1 --2- 172.21.15.44:0/2009981724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556603987720 0x556603986470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.871+0000 7fc4f3fff640 1 -- 172.21.15.44:0/2009981724 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556603986a40 msgr2=0x556603986ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:27.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.871+0000 7fc4f3fff640 1 --2- 172.21.15.44:0/2009981724 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556603986a40 0x556603986ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.871+0000 7fc4f3fff640 1 -- 172.21.15.44:0/2009981724 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566038a3e60 con 0x55660398fdb0 2024-04-21T23:09:27.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.871+0000 7fc4f3fff640 1 --2- 172.21.15.44:0/2009981724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55660398fdb0 0x5566039921a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec007ec0 tx=0x7fc4ec00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4b2b7670fb93bf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:27.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.872+0000 7fc4d9ffb640 1 -- 172.21.15.44:0/2009981724 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec010840 con 0x55660398fdb0 2024-04-21T23:09:27.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.872+0000 7fc4d9ffb640 1 -- 172.21.15.44:0/2009981724 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4ec0183f0 con 0x55660398fdb0 2024-04-21T23:09:27.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.872+0000 7fc4d9ffb640 1 -- 172.21.15.44:0/2009981724 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec0175a0 con 0x55660398fdb0 2024-04-21T23:09:27.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.872+0000 7fc4fc817740 1 -- 172.21.15.44:0/2009981724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55660398fdb0 msgr2=0x5566039921a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:27.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.872+0000 7fc4fc817740 1 --2- 172.21.15.44:0/2009981724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55660398fdb0 0x5566039921a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec007ec0 tx=0x7fc4ec00f790 comp rx=0 tx=0).stop 2024-04-21T23:09:27.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.872+0000 7fc4fc817740 1 -- 172.21.15.44:0/2009981724 shutdown_connections 2024-04-21T23:09:27.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.872+0000 7fc4fc817740 1 --2- 172.21.15.44:0/2009981724 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55660398fdb0 0x5566039921a0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.872+0000 7fc4fc817740 1 --2- 172.21.15.44:0/2009981724 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556603986a40 0x556603986ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.872+0000 7fc4fc817740 1 --2- 172.21.15.44:0/2009981724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556603987720 0x556603986470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.872+0000 7fc4fc817740 1 -- 172.21.15.44:0/2009981724 >> 172.21.15.44:0/2009981724 conn(0x55660394d4f0 msgr2=0x556603981490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:27.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.872+0000 7fc4fc817740 1 -- 172.21.15.44:0/2009981724 shutdown_connections 2024-04-21T23:09:27.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.872+0000 7fc4fc817740 1 -- 172.21.15.44:0/2009981724 wait complete. 2024-04-21T23:09:27.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.873+0000 7fc4fc817740 1 -- start start 2024-04-21T23:09:27.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.873+0000 7fc4fc817740 1 Processor -- start 2024-04-21T23:09:27.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.873+0000 7fc4fc817740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556603986a40 0x55660398ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:27.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.873+0000 7fc4fc817740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556603987720 0x55660398d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:27.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.873+0000 7fc4fc817740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55660398fdb0 0x55660398d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:27.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.873+0000 7fc4fc817740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x556603992c20 con 0x556603987720 2024-04-21T23:09:27.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.873+0000 7fc4fc817740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5566038a5d40 con 0x55660398fdb0 2024-04-21T23:09:27.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.873+0000 7fc4fc817740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556603992da0 con 0x556603986a40 2024-04-21T23:09:27.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.874+0000 7fc4fc817740 10 client.0 _ll_register_callbacks cb 0x5566038e1820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:27.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.874+0000 7fc4f2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556603987720 0x55660398d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:27.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.874+0000 7fc4f2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556603987720 0x55660398d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59944/0 (socket says 172.21.15.44:59944) 2024-04-21T23:09:27.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.874+0000 7fc4f2ffd640 1 -- 172.21.15.44:0/1937744052 learned_addr learned my addr 172.21.15.44:0/1937744052 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:27.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.874+0000 7fc4f37fe640 1 --2- 172.21.15.44:0/1937744052 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556603986a40 0x55660398ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:27.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.874+0000 7fc4f3fff640 1 --2- 172.21.15.44:0/1937744052 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55660398fdb0 0x55660398d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:27.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.874+0000 7fc4f2ffd640 1 -- 172.21.15.44:0/1937744052 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556603986a40 msgr2=0x55660398ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:27.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.874+0000 7fc4f2ffd640 1 --2- 172.21.15.44:0/1937744052 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556603986a40 0x55660398ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.874+0000 7fc4f2ffd640 1 -- 172.21.15.44:0/1937744052 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55660398fdb0 msgr2=0x55660398d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:27.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.874+0000 7fc4f2ffd640 1 --2- 172.21.15.44:0/1937744052 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55660398fdb0 0x55660398d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.874+0000 7fc4f2ffd640 1 -- 172.21.15.44:0/1937744052 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566036c67e0 con 0x556603987720 2024-04-21T23:09:27.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.874+0000 7fc4f3fff640 1 --2- 172.21.15.44:0/1937744052 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55660398fdb0 0x55660398d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:09:27.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.874+0000 7fc4f2ffd640 1 --2- 172.21.15.44:0/1937744052 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556603987720 0x55660398d230 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc4e800ff80 tx=0x7fc4e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=904f7e10647f41d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:27.879 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.874+0000 7fc4d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:09:27.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.874+0000 7fc4d2ffd640 1 -- 172.21.15.44:0/1937744052 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e801e830 con 0x556603987720 2024-04-21T23:09:27.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.874+0000 7fc4d2ffd640 1 -- 172.21.15.44:0/1937744052 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fc4e801e9d0 con 0x556603987720 2024-04-21T23:09:27.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.874+0000 7fc4fc817740 1 -- 172.21.15.44:0/1937744052 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4ec0183f0 con 0x556603987720 2024-04-21T23:09:27.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.874+0000 7fc4d2ffd640 1 -- 172.21.15.44:0/1937744052 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc4e801ebd0 con 0x556603987720 2024-04-21T23:09:27.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.874+0000 7fc4fc817740 10 client.15744 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:27.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.875+0000 7fc4fc817740 1 -- 172.21.15.44:0/1937744052 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556603726f70 con 0x556603987720 2024-04-21T23:09:27.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.875+0000 7fc4d2ffd640 1 -- 172.21.15.44:0/1937744052 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4ec0183f0 con 0x556603987720 2024-04-21T23:09:27.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.875+0000 7fc4d2ffd640 1 -- 172.21.15.44:0/1937744052 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fc4e80103d0 con 0x556603987720 2024-04-21T23:09:27.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.875+0000 7fc4fc817740 10 client.15744 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:27.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.875+0000 7fc4fc817740 10 client.15744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:27.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.875+0000 7fc4fc817740 20 client.15744 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:27.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.875+0000 7fc4fc817740 1 --2- 172.21.15.44:0/1937744052 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55660399fe60 0x5566039a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:27.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.875+0000 7fc4fc817740 4 client.15744 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:09:27.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.875+0000 7fc4fc817740 1 -- 172.21.15.44:0/1937744052 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566038be450 con 0x55660399fe60 2024-04-21T23:09:27.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.875+0000 7fc4f37fe640 1 --2- 172.21.15.44:0/1937744052 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55660399fe60 0x5566039a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:27.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.876+0000 7fc4f37fe640 1 --2- 172.21.15.44:0/1937744052 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55660399fe60 0x5566039a22c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3066221c4132cc7 server_cookie=38cf9b7aa895ad6 in_seq=0 out_seq=0 2024-04-21T23:09:27.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.876+0000 7fc4d2ffd640 10 client.15744 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.877+0000 7fc4d2ffd640 1 -- 172.21.15.44:0/1937744052 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5566038be450 con 0x55660399fe60 2024-04-21T23:09:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.877+0000 7fc4d2ffd640 10 client.15744 handle_command_reply: tid=0 2024-04-21T23:09:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.877+0000 7fc4fc817740 1 client.15744 shutdown 2024-04-21T23:09:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.877+0000 7fc4fc817740 1 -- 172.21.15.44:0/1937744052 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556603987720 msgr2=0x55660398d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.877+0000 7fc4fc817740 1 --2- 172.21.15.44:0/1937744052 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556603987720 0x55660398d230 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc4e800ff80 tx=0x7fc4e80046c0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.877+0000 7fc4fc817740 1 -- 172.21.15.44:0/1937744052 shutdown_connections 2024-04-21T23:09:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.877+0000 7fc4fc817740 1 --2- 172.21.15.44:0/1937744052 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55660399fe60 0x5566039a22c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.877+0000 7fc4fc817740 1 --2- 172.21.15.44:0/1937744052 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55660398fdb0 0x55660398d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.877+0000 7fc4fc817740 1 --2- 172.21.15.44:0/1937744052 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556603987720 0x55660398d230 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.877+0000 7fc4d2ffd640 0 client.15744 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.877+0000 7fc4fc817740 1 --2- 172.21.15.44:0/1937744052 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556603986a40 0x55660398ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.882 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.877+0000 7fc4fc817740 1 -- 172.21.15.44:0/1937744052 >> 172.21.15.44:0/1937744052 conn(0x55660394d4f0 msgr2=0x5566039831e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:27.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.877+0000 7fc4fc817740 1 -- 172.21.15.44:0/1937744052 shutdown_connections 2024-04-21T23:09:27.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.877+0000 7fc4fc817740 1 -- 172.21.15.44:0/1937744052 wait complete. 2024-04-21T23:09:27.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.877+0000 7fc4fc817740 20 client.15744 trim_cache size 0 max 16384 2024-04-21T23:09:27.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.912+0000 7fc4fc817740 1 Processor -- start 2024-04-21T23:09:27.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.912+0000 7fc4fc817740 1 -- start start 2024-04-21T23:09:27.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.913+0000 7fc4fc817740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556603986a40 0x55660399bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:27.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.913+0000 7fc4fc817740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556603987720 0x55660398eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:27.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.913+0000 7fc4fc817740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55660398fdb0 0x5566039a7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:27.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.913+0000 7fc4fc817740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f41aedf0 con 0x556603987720 2024-04-21T23:09:27.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.913+0000 7fc4fc817740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f41b1670 con 0x55660398fdb0 2024-04-21T23:09:27.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.913+0000 7fc4fc817740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f41af460 con 0x556603986a40 2024-04-21T23:09:27.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.913+0000 7fc4f2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556603987720 0x55660398eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:27.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.913+0000 7fc4f2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556603987720 0x55660398eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59956/0 (socket says 172.21.15.44:59956) 2024-04-21T23:09:27.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.913+0000 7fc4f2ffd640 1 -- 172.21.15.44:0/353249703 learned_addr learned my addr 172.21.15.44:0/353249703 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:27.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.913+0000 7fc4f37fe640 1 --2- 172.21.15.44:0/353249703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556603986a40 0x55660399bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:27.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.913+0000 7fc4f3fff640 1 --2- 172.21.15.44:0/353249703 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55660398fdb0 0x5566039a7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:27.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.913+0000 7fc4f2ffd640 1 -- 172.21.15.44:0/353249703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556603986a40 msgr2=0x55660399bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:27.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.913+0000 7fc4f2ffd640 1 --2- 172.21.15.44:0/353249703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556603986a40 0x55660399bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.913+0000 7fc4f2ffd640 1 -- 172.21.15.44:0/353249703 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55660398fdb0 msgr2=0x5566039a7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:27.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.913+0000 7fc4f2ffd640 1 --2- 172.21.15.44:0/353249703 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55660398fdb0 0x5566039a7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.913+0000 7fc4f2ffd640 1 -- 172.21.15.44:0/353249703 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566035be170 con 0x556603987720 2024-04-21T23:09:27.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.913+0000 7fc4f2ffd640 1 --2- 172.21.15.44:0/353249703 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556603987720 0x55660398eb20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8014040 tx=0x7fc4e8011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b388a591d23c578d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:27.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.913+0000 7fc4d97fa640 1 -- 172.21.15.44:0/353249703 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8038070 con 0x556603987720 2024-04-21T23:09:27.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.913+0000 7fc4d97fa640 1 -- 172.21.15.44:0/353249703 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4e8011be0 con 0x556603987720 2024-04-21T23:09:27.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.913+0000 7fc4d97fa640 1 -- 172.21.15.44:0/353249703 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8005280 con 0x556603987720 2024-04-21T23:09:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.914+0000 7fc4fc817740 1 -- 172.21.15.44:0/353249703 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556603987720 msgr2=0x55660398eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.914+0000 7fc4fc817740 1 --2- 172.21.15.44:0/353249703 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556603987720 0x55660398eb20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8014040 tx=0x7fc4e8011f90 comp rx=0 tx=0).stop 2024-04-21T23:09:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.914+0000 7fc4fc817740 1 -- 172.21.15.44:0/353249703 shutdown_connections 2024-04-21T23:09:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.914+0000 7fc4fc817740 1 --2- 172.21.15.44:0/353249703 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55660398fdb0 0x5566039a7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.914+0000 7fc4fc817740 1 --2- 172.21.15.44:0/353249703 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556603987720 0x55660398eb20 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.914+0000 7fc4fc817740 1 --2- 172.21.15.44:0/353249703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556603986a40 0x55660399bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.914+0000 7fc4fc817740 1 -- 172.21.15.44:0/353249703 >> 172.21.15.44:0/353249703 conn(0x55660394d4f0 msgr2=0x55660398e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.914+0000 7fc4fc817740 1 -- 172.21.15.44:0/353249703 shutdown_connections 2024-04-21T23:09:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.914+0000 7fc4fc817740 1 -- 172.21.15.44:0/353249703 wait complete. 2024-04-21T23:09:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.915+0000 7fc4fc817740 1 -- start start 2024-04-21T23:09:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.915+0000 7fc4fc817740 1 Processor -- start 2024-04-21T23:09:27.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.915+0000 7fc4fc817740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556603986a40 0x556603998310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:27.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.915+0000 7fc4fc817740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556603987720 0x556603992ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:27.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.915+0000 7fc4fc817740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55660398fdb0 0x556603993530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:27.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.915+0000 7fc4fc817740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f41afb30 con 0x55660398fdb0 2024-04-21T23:09:27.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.915+0000 7fc4fc817740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f41b0fa0 con 0x556603987720 2024-04-21T23:09:27.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.915+0000 7fc4fc817740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f41b0200 con 0x556603986a40 2024-04-21T23:09:27.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.916+0000 7fc4fc817740 10 client.0 _ll_register_callbacks cb 0x7fc4e8015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:27.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.916+0000 7fc4f3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55660398fdb0 0x556603993530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:27.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.916+0000 7fc4f3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55660398fdb0 0x556603993530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59964/0 (socket says 172.21.15.44:59964) 2024-04-21T23:09:27.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.916+0000 7fc4f3fff640 1 -- 172.21.15.44:0/4021352519 learned_addr learned my addr 172.21.15.44:0/4021352519 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:27.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.916+0000 7fc4f37fe640 1 --2- 172.21.15.44:0/4021352519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556603986a40 0x556603998310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:27.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.916+0000 7fc4f2ffd640 1 --2- 172.21.15.44:0/4021352519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556603987720 0x556603992ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:27.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.916+0000 7fc4f3fff640 1 -- 172.21.15.44:0/4021352519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556603986a40 msgr2=0x556603998310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:27.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.916+0000 7fc4f3fff640 1 --2- 172.21.15.44:0/4021352519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556603986a40 0x556603998310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.916+0000 7fc4f3fff640 1 -- 172.21.15.44:0/4021352519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556603987720 msgr2=0x556603992ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:27.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.916+0000 7fc4f3fff640 1 --2- 172.21.15.44:0/4021352519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556603987720 0x556603992ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.916+0000 7fc4f3fff640 1 -- 172.21.15.44:0/4021352519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556603993c20 con 0x55660398fdb0 2024-04-21T23:09:27.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.916+0000 7fc4f3fff640 1 --2- 172.21.15.44:0/4021352519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55660398fdb0 0x556603993530 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec000f80 tx=0x7fc4ec003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f16a494f2c56b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:27.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.916+0000 7fc4d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:09:27.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.916+0000 7fc4d2ffd640 1 -- 172.21.15.44:0/4021352519 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec027070 con 0x55660398fdb0 2024-04-21T23:09:27.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.916+0000 7fc4fc817740 1 -- 172.21.15.44:0/4021352519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4e8011be0 con 0x55660398fdb0 2024-04-21T23:09:27.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.916+0000 7fc4d2ffd640 1 -- 172.21.15.44:0/4021352519 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fc4ec010a50 con 0x55660398fdb0 2024-04-21T23:09:27.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.917+0000 7fc4d2ffd640 1 -- 172.21.15.44:0/4021352519 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc4ec033050 con 0x55660398fdb0 2024-04-21T23:09:27.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.917+0000 7fc4fc817740 10 client.15750 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:27.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.917+0000 7fc4fc817740 1 -- 172.21.15.44:0/4021352519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556603a073a0 con 0x55660398fdb0 2024-04-21T23:09:27.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.917+0000 7fc4d2ffd640 1 -- 172.21.15.44:0/4021352519 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4e8011be0 con 0x55660398fdb0 2024-04-21T23:09:27.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.917+0000 7fc4d2ffd640 1 -- 172.21.15.44:0/4021352519 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fc4ec0234d0 con 0x55660398fdb0 2024-04-21T23:09:27.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.917+0000 7fc4fc817740 10 client.15750 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:27.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.917+0000 7fc4fc817740 10 client.15750 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:27.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.917+0000 7fc4fc817740 20 client.15750 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:27.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.917+0000 7fc4fc817740 1 --2- 172.21.15.44:0/4021352519 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556603a07840 0x556603a09c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:27.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.917+0000 7fc4fc817740 4 client.15750 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:09:27.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.917+0000 7fc4fc817740 1 -- 172.21.15.44:0/4021352519 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5566038f76b0 con 0x556603a07840 2024-04-21T23:09:27.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.917+0000 7fc4f37fe640 1 --2- 172.21.15.44:0/4021352519 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556603a07840 0x556603a09c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:27.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.918+0000 7fc4d2ffd640 10 client.15750 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:27.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.918+0000 7fc4f37fe640 1 --2- 172.21.15.44:0/4021352519 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556603a07840 0x556603a09c30 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ac80d3ddc9c541c server_cookie=3f7e2098e5247ba7 in_seq=0 out_seq=0 2024-04-21T23:09:27.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.919+0000 7fc4d2ffd640 1 -- 172.21.15.44:0/4021352519 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5566038f76b0 con 0x556603a07840 2024-04-21T23:09:27.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.919+0000 7fc4d2ffd640 10 client.15750 handle_command_reply: tid=0 2024-04-21T23:09:27.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.919+0000 7fc4fc817740 1 client.15750 shutdown 2024-04-21T23:09:27.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.919+0000 7fc4fc817740 1 -- 172.21.15.44:0/4021352519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55660398fdb0 msgr2=0x556603993530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:27.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.919+0000 7fc4fc817740 1 --2- 172.21.15.44:0/4021352519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55660398fdb0 0x556603993530 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec000f80 tx=0x7fc4ec003d70 comp rx=0 tx=0).stop 2024-04-21T23:09:27.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.919+0000 7fc4fc817740 1 -- 172.21.15.44:0/4021352519 shutdown_connections 2024-04-21T23:09:27.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.919+0000 7fc4fc817740 1 --2- 172.21.15.44:0/4021352519 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556603a07840 0x556603a09c30 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.919+0000 7fc4fc817740 1 --2- 172.21.15.44:0/4021352519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55660398fdb0 0x556603993530 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.919+0000 7fc4d2ffd640 0 client.15750 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:27.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.919+0000 7fc4fc817740 1 --2- 172.21.15.44:0/4021352519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556603987720 0x556603992ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.919+0000 7fc4fc817740 1 --2- 172.21.15.44:0/4021352519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556603986a40 0x556603998310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.919+0000 7fc4fc817740 1 -- 172.21.15.44:0/4021352519 >> 172.21.15.44:0/4021352519 conn(0x55660394d4f0 msgr2=0x5566039804f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:27.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.919+0000 7fc4fc817740 1 -- 172.21.15.44:0/4021352519 shutdown_connections 2024-04-21T23:09:27.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.919+0000 7fc4fc817740 1 -- 172.21.15.44:0/4021352519 wait complete. 2024-04-21T23:09:27.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.920+0000 7fc4fc817740 20 client.15750 trim_cache size 0 max 16384 2024-04-21T23:09:27.925 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:09:27.925 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-21T23:09:27.925 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "957d333d-96dd-4b61-a8f2-a096b26542e5", 2024-04-21T23:09:27.925 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-21T23:09:27.925 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:09:27.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.920+0000 7fc4f9b5c640 1 -- 172.21.15.44:0/3794833999 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc4dc0a2ea0 msgr2=0x7fc4dc0a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:27.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.920+0000 7fc4f9b5c640 1 --2- 172.21.15.44:0/3794833999 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc4dc0a2ea0 0x7fc4dc0a5360 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8005fd0 tx=0x7fc4e8005850 comp rx=0 tx=0).stop 2024-04-21T23:09:27.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.920+0000 7fc4f9b5c640 1 -- 172.21.15.44:0/3794833999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4f4101710 msgr2=0x7fc4f4073a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:27.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.920+0000 7fc4f9b5c640 1 --2- 172.21.15.44:0/3794833999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4f4101710 0x7fc4f4073a40 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009230 tx=0x7fc4e402cc70 comp rx=0 tx=0).stop 2024-04-21T23:09:27.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.920+0000 7fc4f9b5c640 1 -- 172.21.15.44:0/3794833999 shutdown_connections 2024-04-21T23:09:27.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.920+0000 7fc4f9b5c640 1 --2- 172.21.15.44:0/3794833999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4f4107720 0x7fc4f40744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.920+0000 7fc4f9b5c640 1 --2- 172.21.15.44:0/3794833999 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc4dc0a2ea0 0x7fc4dc0a5360 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.920+0000 7fc4f9b5c640 1 --2- 172.21.15.44:0/3794833999 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4f41020d0 0x7fc4f4073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.920+0000 7fc4f9b5c640 1 --2- 172.21.15.44:0/3794833999 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4f4101710 0x7fc4f4073a40 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:27.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.920+0000 7fc4f9b5c640 1 -- 172.21.15.44:0/3794833999 >> 172.21.15.44:0/3794833999 conn(0x7fc4f40fd200 msgr2=0x7fc4f40fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:27.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.920+0000 7fc4f9b5c640 1 -- 172.21.15.44:0/3794833999 shutdown_connections 2024-04-21T23:09:27.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:27.920+0000 7fc4f9b5c640 1 -- 172.21.15.44:0/3794833999 wait complete. 2024-04-21T23:09:27.931 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:09:27.932 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:09:28.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.041+0000 7f89f942e640 1 -- 172.21.15.44:0/1498835729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89f41029b0 msgr2=0x7f89f4102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:28.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.041+0000 7f89f942e640 1 --2- 172.21.15.44:0/1498835729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89f41029b0 0x7f89f4102d90 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009900 tx=0x7f89e402bcd0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.041+0000 7f89f942e640 1 -- 172.21.15.44:0/1498835729 shutdown_connections 2024-04-21T23:09:28.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.041+0000 7f89f942e640 1 --2- 172.21.15.44:0/1498835729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89f41032d0 0x7f89f410b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.041+0000 7f89f942e640 1 --2- 172.21.15.44:0/1498835729 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89f410bcf0 0x7f89f410e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.041+0000 7f89f942e640 1 --2- 172.21.15.44:0/1498835729 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89f41029b0 0x7f89f4102d90 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.041+0000 7f89f942e640 1 -- 172.21.15.44:0/1498835729 >> 172.21.15.44:0/1498835729 conn(0x7f89f40777f0 msgr2=0x7f89f4077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:28.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.041+0000 7f89f942e640 1 -- 172.21.15.44:0/1498835729 shutdown_connections 2024-04-21T23:09:28.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.041+0000 7f89f942e640 1 -- 172.21.15.44:0/1498835729 wait complete. 2024-04-21T23:09:28.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.041+0000 7f89f942e640 1 Processor -- start 2024-04-21T23:09:28.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.041+0000 7f89f942e640 1 -- start start 2024-04-21T23:09:28.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89f942e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89f41029b0 0x7f89f4197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:28.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89f942e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89f41032d0 0x7f89f4198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:28.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89f942e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89f410bcf0 0x7f89f419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:28.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89f942e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f89f4111af0 con 0x7f89f41032d0 2024-04-21T23:09:28.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89f942e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f89f4111970 con 0x7f89f41029b0 2024-04-21T23:09:28.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89f942e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f89f4111c70 con 0x7f89f410bcf0 2024-04-21T23:09:28.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89ebfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89f41032d0 0x7f89f4198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:28.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89ebfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89f41032d0 0x7f89f4198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59984/0 (socket says 172.21.15.44:59984) 2024-04-21T23:09:28.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89ebfff640 1 -- 172.21.15.44:0/2422414401 learned_addr learned my addr 172.21.15.44:0/2422414401 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:28.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89f2ffd640 1 --2- 172.21.15.44:0/2422414401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89f41029b0 0x7f89f4197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:28.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89f37fe640 1 --2- 172.21.15.44:0/2422414401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89f410bcf0 0x7f89f419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:28.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89ebfff640 1 -- 172.21.15.44:0/2422414401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89f410bcf0 msgr2=0x7f89f419c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:28.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89ebfff640 1 --2- 172.21.15.44:0/2422414401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89f410bcf0 0x7f89f419c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89ebfff640 1 -- 172.21.15.44:0/2422414401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89f41029b0 msgr2=0x7f89f4197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:28.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89ebfff640 1 --2- 172.21.15.44:0/2422414401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89f41029b0 0x7f89f4197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89ebfff640 1 -- 172.21.15.44:0/2422414401 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89f419cfc0 con 0x7f89f41032d0 2024-04-21T23:09:28.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89f37fe640 1 --2- 172.21.15.44:0/2422414401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89f410bcf0 0x7f89f419c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:09:28.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89f2ffd640 1 --2- 172.21.15.44:0/2422414401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89f41029b0 0x7f89f4197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:09:28.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89ebfff640 1 --2- 172.21.15.44:0/2422414401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89f41032d0 0x7f89f4198510 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00d950 tx=0x7f89dc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:28.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89f0ff9640 1 -- 172.21.15.44:0/2422414401 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc017030 con 0x7f89f41032d0 2024-04-21T23:09:28.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89f0ff9640 1 -- 172.21.15.44:0/2422414401 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89dc004020 con 0x7f89f41032d0 2024-04-21T23:09:28.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89f942e640 1 -- 172.21.15.44:0/2422414401 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89f419d2b0 con 0x7f89f41032d0 2024-04-21T23:09:28.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89f0ff9640 1 -- 172.21.15.44:0/2422414401 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc0047a0 con 0x7f89f41032d0 2024-04-21T23:09:28.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.042+0000 7f89f942e640 1 -- 172.21.15.44:0/2422414401 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89f41a4b40 con 0x7f89f41032d0 2024-04-21T23:09:28.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.045+0000 7f89f0ff9640 1 -- 172.21.15.44:0/2422414401 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f89dc0041d0 con 0x7f89f41032d0 2024-04-21T23:09:28.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.046+0000 7f89fc015740 1 Processor -- start 2024-04-21T23:09:28.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.046+0000 7f89fc015740 1 -- start start 2024-04-21T23:09:28.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.046+0000 7f89fc015740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5555e8743f30 0x5555e8744310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:28.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.046+0000 7f89fc015740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e8744850 0x5555e874cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:28.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.046+0000 7f89fc015740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e874d430 0x5555e874f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:28.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.046+0000 7f89fc015740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5555e86a8560 con 0x5555e874d430 2024-04-21T23:09:28.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.046+0000 7f89fc015740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5555e8650730 con 0x5555e8744850 2024-04-21T23:09:28.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.046+0000 7f89fc015740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5555e86a4520 con 0x5555e8743f30 2024-04-21T23:09:28.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.046+0000 7f89ebfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e874d430 0x5555e874f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:28.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.046+0000 7f89f0ff9640 1 --2- 172.21.15.44:0/2422414401 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f89e00a2e20 0x7f89e00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:28.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.046+0000 7f89f2ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e8744850 0x5555e874cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:28.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.046+0000 7f89f2ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e8744850 0x5555e874cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:42172/0 (socket says 172.21.15.44:42172) 2024-04-21T23:09:28.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.046+0000 7f89f2ffd640 1 -- 172.21.15.44:0/515715078 learned_addr learned my addr 172.21.15.44:0/515715078 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.046+0000 7f89f37fe640 1 --2- 172.21.15.44:0/515715078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5555e8743f30 0x5555e8744310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.047+0000 7f89f37fe640 1 --2- 172.21.15.44:0/2422414401 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f89e00a2e20 0x7f89e00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.047+0000 7f89f0ff9640 1 -- 172.21.15.44:0/2422414401 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f89dc0cf570 con 0x7f89f41032d0 2024-04-21T23:09:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.047+0000 7f89f2ffd640 1 -- 172.21.15.44:0/515715078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5555e8743f30 msgr2=0x5555e8744310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.047+0000 7f89f2ffd640 1 --2- 172.21.15.44:0/515715078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5555e8743f30 0x5555e8744310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.047+0000 7f89f2ffd640 1 -- 172.21.15.44:0/515715078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e874d430 msgr2=0x5555e874f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.047+0000 7f89f2ffd640 1 --2- 172.21.15.44:0/515715078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e874d430 0x5555e874f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.047+0000 7f89f2ffd640 1 -- 172.21.15.44:0/515715078 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555e84867d0 con 0x5555e8744850 2024-04-21T23:09:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.047+0000 7f89f2ffd640 1 --2- 172.21.15.44:0/515715078 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e8744850 0x5555e874cef0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009900 tx=0x7f89e4005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecc56887759152ac server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.047+0000 7f89f37fe640 1 --2- 172.21.15.44:0/2422414401 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f89e00a2e20 0x7f89e00a52e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f89ec009fd0 tx=0x7f89ec0093c0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.047+0000 7f89e97fa640 1 -- 172.21.15.44:0/515715078 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e4042070 con 0x5555e8744850 2024-04-21T23:09:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.047+0000 7f89e97fa640 1 -- 172.21.15.44:0/515715078 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89e402cb20 con 0x5555e8744850 2024-04-21T23:09:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.047+0000 7f89e97fa640 1 -- 172.21.15.44:0/515715078 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e40046c0 con 0x5555e8744850 2024-04-21T23:09:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.048+0000 7f89fc015740 1 -- 172.21.15.44:0/515715078 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e8744850 msgr2=0x5555e874cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.048+0000 7f89fc015740 1 --2- 172.21.15.44:0/515715078 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e8744850 0x5555e874cef0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009900 tx=0x7f89e4005e80 comp rx=0 tx=0).stop 2024-04-21T23:09:28.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.048+0000 7f89fc015740 1 -- 172.21.15.44:0/515715078 shutdown_connections 2024-04-21T23:09:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.048+0000 7f89fc015740 1 --2- 172.21.15.44:0/515715078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e874d430 0x5555e874f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.048+0000 7f89fc015740 1 --2- 172.21.15.44:0/515715078 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e8744850 0x5555e874cef0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.048+0000 7f89fc015740 1 --2- 172.21.15.44:0/515715078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5555e8743f30 0x5555e8744310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.048+0000 7f89fc015740 1 -- 172.21.15.44:0/515715078 >> 172.21.15.44:0/515715078 conn(0x5555e873e660 msgr2=0x5555e873ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.048+0000 7f89fc015740 1 -- 172.21.15.44:0/515715078 shutdown_connections 2024-04-21T23:09:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.048+0000 7f89fc015740 1 -- 172.21.15.44:0/515715078 wait complete. 2024-04-21T23:09:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.049+0000 7f89fc015740 1 -- start start 2024-04-21T23:09:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.049+0000 7f89fc015740 1 Processor -- start 2024-04-21T23:09:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.049+0000 7f89fc015740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5555e8743f30 0x5555e874a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.049+0000 7f89fc015740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e8744850 0x5555e874a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.049+0000 7f89fc015740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e874d430 0x5555e874af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.049+0000 7f89fc015740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5555e87501c0 con 0x5555e874d430 2024-04-21T23:09:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.049+0000 7f89fc015740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5555e8663e00 con 0x5555e8744850 2024-04-21T23:09:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.049+0000 7f89fc015740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5555e8750340 con 0x5555e8743f30 2024-04-21T23:09:28.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.049+0000 7f89fc015740 10 client.0 _ll_register_callbacks cb 0x5555e869eeb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:28.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.049+0000 7f89f37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e874d430 0x5555e874af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:28.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.049+0000 7f89f37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e874d430 0x5555e874af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60000/0 (socket says 172.21.15.44:60000) 2024-04-21T23:09:28.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.049+0000 7f89f37fe640 1 -- 172.21.15.44:0/2875366500 learned_addr learned my addr 172.21.15.44:0/2875366500 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:28.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.049+0000 7f89ebfff640 1 --2- 172.21.15.44:0/2875366500 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e8744850 0x5555e874a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:28.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.049+0000 7f89f2ffd640 1 --2- 172.21.15.44:0/2875366500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5555e8743f30 0x5555e874a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:28.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.049+0000 7f89f37fe640 1 -- 172.21.15.44:0/2875366500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5555e8743f30 msgr2=0x5555e874a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:28.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.049+0000 7f89f37fe640 1 --2- 172.21.15.44:0/2875366500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5555e8743f30 0x5555e874a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.049+0000 7f89f37fe640 1 -- 172.21.15.44:0/2875366500 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e8744850 msgr2=0x5555e874a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:28.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.049+0000 7f89f37fe640 1 --2- 172.21.15.44:0/2875366500 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e8744850 0x5555e874a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.049+0000 7f89f37fe640 1 -- 172.21.15.44:0/2875366500 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555e83bc3f0 con 0x5555e874d430 2024-04-21T23:09:28.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.049+0000 7f89ebfff640 1 --2- 172.21.15.44:0/2875366500 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e8744850 0x5555e874a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:09:28.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.050+0000 7f89f37fe640 1 --2- 172.21.15.44:0/2875366500 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e874d430 0x5555e874af00 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f89ec009d40 tx=0x7f89ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68d161b6f18897a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:28.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.050+0000 7f89da7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:09:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.050+0000 7f89da7fc640 1 -- 172.21.15.44:0/2875366500 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ec004d20 con 0x5555e874d430 2024-04-21T23:09:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.050+0000 7f89da7fc640 1 -- 172.21.15.44:0/2875366500 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f89ec00f960 con 0x5555e874d430 2024-04-21T23:09:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.050+0000 7f89fc015740 1 -- 172.21.15.44:0/2875366500 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89e402cb20 con 0x5555e874d430 2024-04-21T23:09:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.050+0000 7f89da7fc640 1 -- 172.21.15.44:0/2875366500 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f89ec032050 con 0x5555e874d430 2024-04-21T23:09:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.050+0000 7f89fc015740 10 client.15762 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.050+0000 7f89fc015740 1 -- 172.21.15.44:0/2875366500 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5555e837c0f0 con 0x5555e874d430 2024-04-21T23:09:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.050+0000 7f89da7fc640 1 -- 172.21.15.44:0/2875366500 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89e402cb20 con 0x5555e874d430 2024-04-21T23:09:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.051+0000 7f89da7fc640 1 -- 172.21.15.44:0/2875366500 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f89ec01d040 con 0x5555e874d430 2024-04-21T23:09:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.051+0000 7f89fc015740 10 client.15762 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.051+0000 7f89fc015740 10 client.15762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.051+0000 7f89fc015740 20 client.15762 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.051+0000 7f89fc015740 1 --2- 172.21.15.44:0/2875366500 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5555e875d770 0x5555e875fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.051+0000 7f89fc015740 4 client.15762 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:09:28.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.051+0000 7f89fc015740 1 -- 172.21.15.44:0/2875366500 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555e86b5aa0 con 0x5555e875d770 2024-04-21T23:09:28.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.051+0000 7f89f2ffd640 1 --2- 172.21.15.44:0/2875366500 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5555e875d770 0x5555e875fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:28.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.051+0000 7f89f2ffd640 1 --2- 172.21.15.44:0/2875366500 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5555e875d770 0x5555e875fbf0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f1db92614585554 server_cookie=de0cf0d0cefd0a5 in_seq=0 out_seq=0 2024-04-21T23:09:28.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.051+0000 7f89da7fc640 10 client.15762 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:28.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.052+0000 7f89da7fc640 1 -- 172.21.15.44:0/2875366500 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5555e86b5aa0 con 0x5555e875d770 2024-04-21T23:09:28.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.052+0000 7f89da7fc640 10 client.15762 handle_command_reply: tid=0 2024-04-21T23:09:28.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.052+0000 7f89fc015740 1 client.15762 shutdown 2024-04-21T23:09:28.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.052+0000 7f89fc015740 1 -- 172.21.15.44:0/2875366500 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e874d430 msgr2=0x5555e874af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:28.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.052+0000 7f89fc015740 1 --2- 172.21.15.44:0/2875366500 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e874d430 0x5555e874af00 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f89ec009d40 tx=0x7f89ec004a40 comp rx=0 tx=0).stop 2024-04-21T23:09:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.052+0000 7f89fc015740 1 -- 172.21.15.44:0/2875366500 shutdown_connections 2024-04-21T23:09:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.052+0000 7f89fc015740 1 --2- 172.21.15.44:0/2875366500 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5555e875d770 0x5555e875fbf0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.052+0000 7f89fc015740 1 --2- 172.21.15.44:0/2875366500 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e874d430 0x5555e874af00 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.052+0000 7f89da7fc640 0 client.15762 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.052+0000 7f89fc015740 1 --2- 172.21.15.44:0/2875366500 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e8744850 0x5555e874a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.052+0000 7f89fc015740 1 --2- 172.21.15.44:0/2875366500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5555e8743f30 0x5555e874a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.052+0000 7f89fc015740 1 -- 172.21.15.44:0/2875366500 >> 172.21.15.44:0/2875366500 conn(0x5555e873e660 msgr2=0x5555e87403b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.052+0000 7f89fc015740 1 -- 172.21.15.44:0/2875366500 shutdown_connections 2024-04-21T23:09:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.053+0000 7f89fc015740 1 -- 172.21.15.44:0/2875366500 wait complete. 2024-04-21T23:09:28.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.053+0000 7f89fc015740 20 client.15762 trim_cache size 0 max 16384 2024-04-21T23:09:28.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.071+0000 7f89fc015740 1 Processor -- start 2024-04-21T23:09:28.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.071+0000 7f89fc015740 1 -- start start 2024-04-21T23:09:28.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.071+0000 7f89fc015740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e8743f30 0x5555e8759080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:28.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.071+0000 7f89fc015740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e8744850 0x5555e87654c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:28.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.071+0000 7f89fc015740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5555e874d430 0x5555e8765a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:28.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.071+0000 7f89fc015740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41acc80 con 0x5555e8743f30 2024-04-21T23:09:28.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.071+0000 7f89fc015740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41af500 con 0x5555e8744850 2024-04-21T23:09:28.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.071+0000 7f89fc015740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41ad2f0 con 0x5555e874d430 2024-04-21T23:09:28.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.071+0000 7f89f2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e8743f30 0x5555e8759080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:28.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.071+0000 7f89f2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e8743f30 0x5555e8759080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60012/0 (socket says 172.21.15.44:60012) 2024-04-21T23:09:28.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.071+0000 7f89f2ffd640 1 -- 172.21.15.44:0/490103601 learned_addr learned my addr 172.21.15.44:0/490103601 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:28.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.071+0000 7f89f37fe640 1 --2- 172.21.15.44:0/490103601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5555e874d430 0x5555e8765a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:28.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.071+0000 7f89ebfff640 1 --2- 172.21.15.44:0/490103601 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e8744850 0x5555e87654c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:28.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.071+0000 7f89f2ffd640 1 -- 172.21.15.44:0/490103601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5555e874d430 msgr2=0x5555e8765a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.071+0000 7f89f2ffd640 1 --2- 172.21.15.44:0/490103601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5555e874d430 0x5555e8765a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.071+0000 7f89f2ffd640 1 -- 172.21.15.44:0/490103601 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e8744850 msgr2=0x5555e87654c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.071+0000 7f89f2ffd640 1 --2- 172.21.15.44:0/490103601 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e8744850 0x5555e87654c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.071+0000 7f89f2ffd640 1 -- 172.21.15.44:0/490103601 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555e8746040 con 0x5555e8743f30 2024-04-21T23:09:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.071+0000 7f89ebfff640 1 --2- 172.21.15.44:0/490103601 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e8744850 0x5555e87654c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:09:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.072+0000 7f89f2ffd640 1 --2- 172.21.15.44:0/490103601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e8743f30 0x5555e8759080 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009900 tx=0x7f89e402e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df3529e939d40f71 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.072+0000 7f89e8ff9640 1 -- 172.21.15.44:0/490103601 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e4056070 con 0x5555e8743f30 2024-04-21T23:09:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.072+0000 7f89e8ff9640 1 -- 172.21.15.44:0/490103601 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89e402cb00 con 0x5555e8743f30 2024-04-21T23:09:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.072+0000 7f89e8ff9640 1 -- 172.21.15.44:0/490103601 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e4042040 con 0x5555e8743f30 2024-04-21T23:09:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.073+0000 7f89fc015740 1 -- 172.21.15.44:0/490103601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e8743f30 msgr2=0x5555e8759080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.073+0000 7f89fc015740 1 --2- 172.21.15.44:0/490103601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e8743f30 0x5555e8759080 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009900 tx=0x7f89e402e040 comp rx=0 tx=0).stop 2024-04-21T23:09:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.073+0000 7f89fc015740 1 -- 172.21.15.44:0/490103601 shutdown_connections 2024-04-21T23:09:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.073+0000 7f89fc015740 1 --2- 172.21.15.44:0/490103601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5555e874d430 0x5555e8765a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.073+0000 7f89fc015740 1 --2- 172.21.15.44:0/490103601 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e8744850 0x5555e87654c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.073+0000 7f89fc015740 1 --2- 172.21.15.44:0/490103601 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e8743f30 0x5555e8759080 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.073+0000 7f89fc015740 1 -- 172.21.15.44:0/490103601 >> 172.21.15.44:0/490103601 conn(0x5555e873e660 msgr2=0x5555e8748d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.073+0000 7f89fc015740 1 -- 172.21.15.44:0/490103601 shutdown_connections 2024-04-21T23:09:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.073+0000 7f89fc015740 1 -- 172.21.15.44:0/490103601 wait complete. 2024-04-21T23:09:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.073+0000 7f89fc015740 1 -- start start 2024-04-21T23:09:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.073+0000 7f89fc015740 1 Processor -- start 2024-04-21T23:09:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.073+0000 7f89fc015740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5555e8743f30 0x5555e8755780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.073+0000 7f89fc015740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e8744850 0x5555e8746ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.073+0000 7f89fc015740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e874d430 0x5555e8747000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.073+0000 7f89fc015740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41ad9c0 con 0x5555e8744850 2024-04-21T23:09:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.073+0000 7f89fc015740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41aee30 con 0x5555e874d430 2024-04-21T23:09:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.073+0000 7f89fc015740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f89f41ae090 con 0x5555e8743f30 2024-04-21T23:09:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.073+0000 7f89fc015740 10 client.0 _ll_register_callbacks cb 0x7f89ec014750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.074+0000 7f89ebfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e8744850 0x5555e8746ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.074+0000 7f89ebfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e8744850 0x5555e8746ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60018/0 (socket says 172.21.15.44:60018) 2024-04-21T23:09:28.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.074+0000 7f89ebfff640 1 -- 172.21.15.44:0/1915139242 learned_addr learned my addr 172.21.15.44:0/1915139242 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:28.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.074+0000 7f89f2ffd640 1 --2- 172.21.15.44:0/1915139242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5555e8743f30 0x5555e8755780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:28.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.074+0000 7f89f37fe640 1 --2- 172.21.15.44:0/1915139242 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e874d430 0x5555e8747000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:28.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.074+0000 7f89ebfff640 1 -- 172.21.15.44:0/1915139242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5555e8743f30 msgr2=0x5555e8755780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:28.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.074+0000 7f89ebfff640 1 --2- 172.21.15.44:0/1915139242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5555e8743f30 0x5555e8755780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.074+0000 7f89ebfff640 1 -- 172.21.15.44:0/1915139242 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e874d430 msgr2=0x5555e8747000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:28.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.074+0000 7f89ebfff640 1 --2- 172.21.15.44:0/1915139242 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e874d430 0x5555e8747000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.074+0000 7f89ebfff640 1 -- 172.21.15.44:0/1915139242 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555e8747540 con 0x5555e8744850 2024-04-21T23:09:28.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.074+0000 7f89ebfff640 1 --2- 172.21.15.44:0/1915139242 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e8744850 0x5555e8746ac0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f89dc009610 tx=0x7f89dc097110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fad1a6c3c8f8b02 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:28.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.074+0000 7f89da7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:09:28.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.074+0000 7f89da7fc640 1 -- 172.21.15.44:0/1915139242 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc0cf390 con 0x5555e8744850 2024-04-21T23:09:28.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.074+0000 7f89fc015740 1 -- 172.21.15.44:0/1915139242 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89e402cb00 con 0x5555e8744850 2024-04-21T23:09:28.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.074+0000 7f89da7fc640 1 -- 172.21.15.44:0/1915139242 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f89dc0d2bd0 con 0x5555e8744850 2024-04-21T23:09:28.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.074+0000 7f89da7fc640 1 -- 172.21.15.44:0/1915139242 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f89dc09b4a0 con 0x5555e8744850 2024-04-21T23:09:28.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.074+0000 7f89fc015740 10 client.15768 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:28.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.074+0000 7f89fc015740 1 -- 172.21.15.44:0/1915139242 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5555e87c7e50 con 0x5555e8744850 2024-04-21T23:09:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.075+0000 7f89da7fc640 1 -- 172.21.15.44:0/1915139242 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89e402cb00 con 0x5555e8744850 2024-04-21T23:09:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.075+0000 7f89da7fc640 1 -- 172.21.15.44:0/1915139242 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f89dc0cca80 con 0x5555e8744850 2024-04-21T23:09:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.075+0000 7f89fc015740 10 client.15768 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.075+0000 7f89fc015740 10 client.15768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.075+0000 7f89fc015740 20 client.15768 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.075+0000 7f89fc015740 1 --2- 172.21.15.44:0/1915139242 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5555e875d770 0x5555e87ca590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.075+0000 7f89fc015740 4 client.15768 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:09:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.075+0000 7f89fc015740 1 -- 172.21.15.44:0/1915139242 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5555e86afdf0 con 0x5555e875d770 2024-04-21T23:09:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.075+0000 7f89f2ffd640 1 --2- 172.21.15.44:0/1915139242 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5555e875d770 0x5555e87ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.075+0000 7f89da7fc640 10 client.15768 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.075+0000 7f89f2ffd640 1 --2- 172.21.15.44:0/1915139242 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5555e875d770 0x5555e87ca590 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e189bd9aadda97b server_cookie=ebcf99d9e5c0cb09 in_seq=0 out_seq=0 2024-04-21T23:09:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.076+0000 7f89da7fc640 1 -- 172.21.15.44:0/1915139242 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5555e86afdf0 con 0x5555e875d770 2024-04-21T23:09:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.076+0000 7f89da7fc640 10 client.15768 handle_command_reply: tid=0 2024-04-21T23:09:28.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.076+0000 7f89fc015740 1 client.15768 shutdown 2024-04-21T23:09:28.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.076+0000 7f89fc015740 1 -- 172.21.15.44:0/1915139242 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e8744850 msgr2=0x5555e8746ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:28.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.076+0000 7f89fc015740 1 --2- 172.21.15.44:0/1915139242 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e8744850 0x5555e8746ac0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f89dc009610 tx=0x7f89dc097110 comp rx=0 tx=0).stop 2024-04-21T23:09:28.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.076+0000 7f89fc015740 1 -- 172.21.15.44:0/1915139242 shutdown_connections 2024-04-21T23:09:28.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.076+0000 7f89fc015740 1 --2- 172.21.15.44:0/1915139242 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5555e875d770 0x5555e87ca590 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.076+0000 7f89fc015740 1 --2- 172.21.15.44:0/1915139242 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5555e874d430 0x5555e8747000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.076+0000 7f89fc015740 1 --2- 172.21.15.44:0/1915139242 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5555e8744850 0x5555e8746ac0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.076+0000 7f89fc015740 1 --2- 172.21.15.44:0/1915139242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5555e8743f30 0x5555e8755780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.076+0000 7f89fc015740 1 -- 172.21.15.44:0/1915139242 >> 172.21.15.44:0/1915139242 conn(0x5555e873e660 msgr2=0x5555e873ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:28.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.076+0000 7f89da7fc640 0 client.15768 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:28.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.076+0000 7f89fc015740 1 -- 172.21.15.44:0/1915139242 shutdown_connections 2024-04-21T23:09:28.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.076+0000 7f89fc015740 1 -- 172.21.15.44:0/1915139242 wait complete. 2024-04-21T23:09:28.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.076+0000 7f89fc015740 20 client.15768 trim_cache size 0 max 16384 2024-04-21T23:09:28.084 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:09:28.084 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:09:28.084 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-21T23:09:28.084 INFO:teuthology.orchestra.run.smithi044.stdout: "957d333d-96dd-4b61-a8f2-a096b26542e5": { 2024-04-21T23:09:28.084 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-21T23:09:28.084 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "957d333d-96dd-4b61-a8f2-a096b26542e5", 2024-04-21T23:09:28.084 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-04-21T23:09:28.084 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:09:28.084 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:09:28.084 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:09:28.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.077+0000 7f89f942e640 1 -- 172.21.15.44:0/2422414401 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f89e00a2e20 msgr2=0x7f89e00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:28.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.077+0000 7f89f942e640 1 --2- 172.21.15.44:0/2422414401 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f89e00a2e20 0x7f89e00a52e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f89ec009fd0 tx=0x7f89ec0093c0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.077+0000 7f89f942e640 1 -- 172.21.15.44:0/2422414401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89f41032d0 msgr2=0x7f89f4198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:28.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.077+0000 7f89f942e640 1 --2- 172.21.15.44:0/2422414401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89f41032d0 0x7f89f4198510 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00d950 tx=0x7f89dc00de10 comp rx=0 tx=0).stop 2024-04-21T23:09:28.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.077+0000 7f89f942e640 1 -- 172.21.15.44:0/2422414401 shutdown_connections 2024-04-21T23:09:28.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.077+0000 7f89f942e640 1 --2- 172.21.15.44:0/2422414401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89f410bcf0 0x7f89f419c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.077+0000 7f89f942e640 1 --2- 172.21.15.44:0/2422414401 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f89e00a2e20 0x7f89e00a52e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.077+0000 7f89f942e640 1 --2- 172.21.15.44:0/2422414401 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89f41032d0 0x7f89f4198510 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.077+0000 7f89f942e640 1 --2- 172.21.15.44:0/2422414401 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89f41029b0 0x7f89f4197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:28.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.077+0000 7f89f942e640 1 -- 172.21.15.44:0/2422414401 >> 172.21.15.44:0/2422414401 conn(0x7f89f40777f0 msgr2=0x7f89f40fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:28.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.077+0000 7f89f942e640 1 -- 172.21.15.44:0/2422414401 shutdown_connections 2024-04-21T23:09:28.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:28.077+0000 7f89f942e640 1 -- 172.21.15.44:0/2422414401 wait complete. 2024-04-21T23:09:28.087 INFO:tasks.cephfs.filesystem:scrub status for tag:957d333d-96dd-4b61-a8f2-a096b26542e5 - {'path': '/', 'tag': '957d333d-96dd-4b61-a8f2-a096b26542e5', 'options': 'recursive,force'} 2024-04-21T23:09:28.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:28 smithi044 ceph-mon[27934]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:09:28.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:28 smithi069 ceph-mon[31816]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:09:28.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:28 smithi055 ceph-mon[31117]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:09:29.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:29 smithi044 ceph-mon[27934]: scrub queued for path: / 2024-04-21T23:09:29.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:29 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:09:29.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:29 smithi044 ceph-mon[27934]: scrub summary: active paths [/] 2024-04-21T23:09:29.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:29 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:09:29.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:29 smithi055 ceph-mon[31117]: scrub queued for path: / 2024-04-21T23:09:29.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:29 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:09:29.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:29 smithi055 ceph-mon[31117]: scrub summary: active paths [/] 2024-04-21T23:09:29.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:29 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:09:29.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:29 smithi069 ceph-mon[31816]: scrub queued for path: / 2024-04-21T23:09:29.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:29 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:09:29.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:29 smithi069 ceph-mon[31816]: scrub summary: active paths [/] 2024-04-21T23:09:29.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:29 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:09:30.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:30 smithi044 ceph-mon[27934]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-21T23:09:30.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:30 smithi055 ceph-mon[31117]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-21T23:09:30.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:30 smithi069 ceph-mon[31816]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-21T23:09:32.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:32 smithi044 ceph-mon[27934]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-21T23:09:32.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:32 smithi055 ceph-mon[31117]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-21T23:09:32.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:32 smithi069 ceph-mon[31816]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-21T23:09:33.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:33 smithi044 ceph-mon[27934]: scrub summary: idle 2024-04-21T23:09:33.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:33 smithi055 ceph-mon[31117]: scrub summary: idle 2024-04-21T23:09:33.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:33 smithi069 ceph-mon[31816]: scrub summary: idle 2024-04-21T23:09:34.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:34 smithi044 ceph-mon[27934]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-21T23:09:34.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:34 smithi055 ceph-mon[31117]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-21T23:09:34.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:34 smithi069 ceph-mon[31816]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-21T23:09:36.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:36 smithi044 ceph-mon[27934]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-21T23:09:36.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:36 smithi055 ceph-mon[31117]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-21T23:09:36.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:36 smithi069 ceph-mon[31816]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-21T23:09:36.773 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.773+0000 7f1ff7bfe640 1 -- 172.21.15.44:0/2707806125 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ff01029b0 msgr2=0x7f1ff0102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:36.774 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.773+0000 7f1ff7bfe640 1 --2- 172.21.15.44:0/2707806125 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ff01029b0 0x7f1ff0102d90 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1fec009900 tx=0x7f1fec02bcd0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.774 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.773+0000 7f1ff7bfe640 1 -- 172.21.15.44:0/2707806125 shutdown_connections 2024-04-21T23:09:36.774 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.773+0000 7f1ff7bfe640 1 --2- 172.21.15.44:0/2707806125 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ff010bd80 0x7f1ff010e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.774 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.773+0000 7f1ff7bfe640 1 --2- 172.21.15.44:0/2707806125 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ff01032d0 0x7f1ff010b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.774 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.773+0000 7f1ff7bfe640 1 --2- 172.21.15.44:0/2707806125 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ff01029b0 0x7f1ff0102d90 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.774 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.773+0000 7f1ff7bfe640 1 -- 172.21.15.44:0/2707806125 >> 172.21.15.44:0/2707806125 conn(0x7f1ff0077760 msgr2=0x7f1ff0077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:36.774 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.773+0000 7f1ff7bfe640 1 -- 172.21.15.44:0/2707806125 shutdown_connections 2024-04-21T23:09:36.774 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.773+0000 7f1ff7bfe640 1 -- 172.21.15.44:0/2707806125 wait complete. 2024-04-21T23:09:36.774 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.773+0000 7f1ff7bfe640 1 Processor -- start 2024-04-21T23:09:36.774 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.773+0000 7f1ff7bfe640 1 -- start start 2024-04-21T23:09:36.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.774+0000 7f1ff7bfe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ff01029b0 0x7f1ff0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:36.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.774+0000 7f1ff7bfe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ff01032d0 0x7f1ff0198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:36.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.774+0000 7f1ff7bfe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ff010bd80 0x7f1ff019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:36.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.774+0000 7f1ff7bfe640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff0111b10 con 0x7f1ff01029b0 2024-04-21T23:09:36.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.774+0000 7f1ff7bfe640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff0111990 con 0x7f1ff010bd80 2024-04-21T23:09:36.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.774+0000 7f1ff7bfe640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff0111c90 con 0x7f1ff01032d0 2024-04-21T23:09:36.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.774+0000 7f1ff599a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ff01029b0 0x7f1ff0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:36.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.774+0000 7f1ff599a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ff01029b0 0x7f1ff0197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57618/0 (socket says 172.21.15.44:57618) 2024-04-21T23:09:36.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.774+0000 7f1ff599a640 1 -- 172.21.15.44:0/2599852966 learned_addr learned my addr 172.21.15.44:0/2599852966 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:36.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.774+0000 7f1ff619b640 1 --2- 172.21.15.44:0/2599852966 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ff010bd80 0x7f1ff019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:36.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.774+0000 7f1ff5199640 1 --2- 172.21.15.44:0/2599852966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ff01032d0 0x7f1ff0198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:36.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.774+0000 7f1ff599a640 1 -- 172.21.15.44:0/2599852966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ff01032d0 msgr2=0x7f1ff0198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:36.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.774+0000 7f1ff599a640 1 --2- 172.21.15.44:0/2599852966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ff01032d0 0x7f1ff0198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.775 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.774+0000 7f1ff599a640 1 -- 172.21.15.44:0/2599852966 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ff010bd80 msgr2=0x7f1ff019c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.774+0000 7f1ff599a640 1 --2- 172.21.15.44:0/2599852966 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ff010bd80 0x7f1ff019c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.774+0000 7f1ff599a640 1 -- 172.21.15.44:0/2599852966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ff019cf70 con 0x7f1ff01029b0 2024-04-21T23:09:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.774+0000 7f1ff599a640 1 --2- 172.21.15.44:0/2599852966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ff01029b0 0x7f1ff0197fd0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f1fec02efd0 tx=0x7f1fec02ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.775+0000 7f1fe6ffd640 1 -- 172.21.15.44:0/2599852966 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec041070 con 0x7f1ff01029b0 2024-04-21T23:09:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.775+0000 7f1ff7bfe640 1 -- 172.21.15.44:0/2599852966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ff019d200 con 0x7f1ff01029b0 2024-04-21T23:09:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.775+0000 7f1fe6ffd640 1 -- 172.21.15.44:0/2599852966 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fec0054e0 con 0x7f1ff01029b0 2024-04-21T23:09:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.775+0000 7f1fe6ffd640 1 -- 172.21.15.44:0/2599852966 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec034440 con 0x7f1ff01029b0 2024-04-21T23:09:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.775+0000 7f1ff7bfe640 1 -- 172.21.15.44:0/2599852966 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ff01a4af0 con 0x7f1ff01029b0 2024-04-21T23:09:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.777+0000 7f1fe6ffd640 1 -- 172.21.15.44:0/2599852966 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f1fec044070 con 0x7f1ff01029b0 2024-04-21T23:09:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.778+0000 7f1ffa7e1740 1 Processor -- start 2024-04-21T23:09:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.778+0000 7f1ffa7e1740 1 -- start start 2024-04-21T23:09:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.778+0000 7f1ffa7e1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612bf76cb20 0x5612bf76cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.778+0000 7f1ffa7e1740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 0x5612bf775cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.778+0000 7f1ffa7e1740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5612bf776210 0x5612bf778600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.778+0000 7f1ffa7e1740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5612bf6d0b70 con 0x5612bf76d440 2024-04-21T23:09:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.778+0000 7f1ffa7e1740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5612bf679470 con 0x5612bf776210 2024-04-21T23:09:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.778+0000 7f1ffa7e1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5612bf6ccb30 con 0x5612bf76cb20 2024-04-21T23:09:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.778+0000 7f1fe6ffd640 1 --2- 172.21.15.44:0/2599852966 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1fdc0a2e10 0x7f1fdc0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.778+0000 7f1ff599a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 0x5612bf775cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.778+0000 7f1ff619b640 1 --2- 172.21.15.44:0/2599852966 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1fdc0a2e10 0x7f1fdc0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.778+0000 7f1ff599a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 0x5612bf775cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57634/0 (socket says 172.21.15.44:57634) 2024-04-21T23:09:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.778+0000 7f1ff599a640 1 -- 172.21.15.44:0/421185595 learned_addr learned my addr 172.21.15.44:0/421185595 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.778+0000 7f1ff619b640 1 --2- 172.21.15.44:0/421185595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612bf76cb20 0x5612bf76cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.778+0000 7f1fe6ffd640 1 -- 172.21.15.44:0/2599852966 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f1fec0f1670 con 0x7f1ff01029b0 2024-04-21T23:09:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.779+0000 7f1ff5199640 1 --2- 172.21.15.44:0/421185595 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5612bf776210 0x5612bf778600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.779+0000 7f1ff599a640 1 -- 172.21.15.44:0/421185595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612bf76cb20 msgr2=0x5612bf76cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.779+0000 7f1ff599a640 1 --2- 172.21.15.44:0/421185595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612bf76cb20 0x5612bf76cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.779+0000 7f1ff599a640 1 -- 172.21.15.44:0/421185595 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5612bf776210 msgr2=0x5612bf778600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.779+0000 7f1ff599a640 1 --2- 172.21.15.44:0/421185595 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5612bf776210 0x5612bf778600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.779+0000 7f1ff599a640 1 -- 172.21.15.44:0/421185595 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612bf3e5210 con 0x5612bf76d440 2024-04-21T23:09:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.779+0000 7f1ff619b640 1 --2- 172.21.15.44:0/2599852966 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1fdc0a2e10 0x7f1fdc0a52d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8007fd0 tx=0x7f1fe8007d60 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.779+0000 7f1ff599a640 1 --2- 172.21.15.44:0/421185595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 0x5612bf775cd0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0bd640 tx=0x7f1fec08f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d59c4cb3ba71b3a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.779+0000 7f1fdbfff640 1 -- 172.21.15.44:0/421185595 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec0f0070 con 0x5612bf76d440 2024-04-21T23:09:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.779+0000 7f1fdbfff640 1 -- 172.21.15.44:0/421185595 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fec005690 con 0x5612bf76d440 2024-04-21T23:09:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.779+0000 7f1fdbfff640 1 -- 172.21.15.44:0/421185595 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec04b070 con 0x5612bf76d440 2024-04-21T23:09:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.780+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/421185595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 msgr2=0x5612bf775cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.780+0000 7f1ffa7e1740 1 --2- 172.21.15.44:0/421185595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 0x5612bf775cd0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0bd640 tx=0x7f1fec08f670 comp rx=0 tx=0).stop 2024-04-21T23:09:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.780+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/421185595 shutdown_connections 2024-04-21T23:09:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.780+0000 7f1ffa7e1740 1 --2- 172.21.15.44:0/421185595 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5612bf776210 0x5612bf778600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.780+0000 7f1ffa7e1740 1 --2- 172.21.15.44:0/421185595 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 0x5612bf775cd0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.780+0000 7f1ffa7e1740 1 --2- 172.21.15.44:0/421185595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612bf76cb20 0x5612bf76cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.780+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/421185595 >> 172.21.15.44:0/421185595 conn(0x5612bf767440 msgr2=0x5612bf767820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.780+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/421185595 shutdown_connections 2024-04-21T23:09:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.780+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/421185595 wait complete. 2024-04-21T23:09:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.780+0000 7f1ffa7e1740 1 -- start start 2024-04-21T23:09:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.781+0000 7f1ffa7e1740 1 Processor -- start 2024-04-21T23:09:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.781+0000 7f1ffa7e1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612bf76cb20 0x5612bf7732d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.781+0000 7f1ffa7e1740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 0x5612bf773810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.781+0000 7f1ffa7e1740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5612bf776210 0x5612bf773d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.781+0000 7f1ffa7e1740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5612bf6d0b70 con 0x5612bf76d440 2024-04-21T23:09:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.781+0000 7f1ffa7e1740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5612bf68ca10 con 0x5612bf776210 2024-04-21T23:09:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.781+0000 7f1ffa7e1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5612bf778c00 con 0x5612bf76cb20 2024-04-21T23:09:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.781+0000 7f1ffa7e1740 10 client.0 _ll_register_callbacks cb 0x5612bf6c7b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.781+0000 7f1ff5199640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 0x5612bf773810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.781+0000 7f1ff5199640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 0x5612bf773810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57640/0 (socket says 172.21.15.44:57640) 2024-04-21T23:09:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.781+0000 7f1ff5199640 1 -- 172.21.15.44:0/1508974779 learned_addr learned my addr 172.21.15.44:0/1508974779 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.781+0000 7f1ff599a640 1 --2- 172.21.15.44:0/1508974779 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612bf76cb20 0x5612bf7732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.781+0000 7f1ff619b640 1 --2- 172.21.15.44:0/1508974779 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5612bf776210 0x5612bf773d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.782+0000 7f1ff5199640 1 -- 172.21.15.44:0/1508974779 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612bf76cb20 msgr2=0x5612bf7732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.782+0000 7f1ff5199640 1 --2- 172.21.15.44:0/1508974779 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612bf76cb20 0x5612bf7732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.782+0000 7f1ff5199640 1 -- 172.21.15.44:0/1508974779 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5612bf776210 msgr2=0x5612bf773d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.782+0000 7f1ff5199640 1 --2- 172.21.15.44:0/1508974779 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5612bf776210 0x5612bf773d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.782+0000 7f1ff5199640 1 -- 172.21.15.44:0/1508974779 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612bf4b1620 con 0x5612bf76d440 2024-04-21T23:09:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.782+0000 7f1ff5199640 1 --2- 172.21.15.44:0/1508974779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 0x5612bf773810 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1fe000ca40 tx=0x7f1fe000ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fce7d46fbcd4674 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.782+0000 7f1fd97fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:09:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.782+0000 7f1fd97fa640 1 -- 172.21.15.44:0/1508974779 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe0021020 con 0x5612bf76d440 2024-04-21T23:09:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.782+0000 7f1fd97fa640 1 -- 172.21.15.44:0/1508974779 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f1fe0005020 con 0x5612bf76d440 2024-04-21T23:09:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.782+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/1508974779 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5612bf6de0b0 con 0x5612bf76d440 2024-04-21T23:09:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.782+0000 7f1fd97fa640 1 -- 172.21.15.44:0/1508974779 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1fe0029050 con 0x5612bf76d440 2024-04-21T23:09:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.782+0000 7f1ffa7e1740 10 client.15780 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.782+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/1508974779 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5612bf784dd0 con 0x5612bf76d440 2024-04-21T23:09:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.782+0000 7f1fd97fa640 1 -- 172.21.15.44:0/1508974779 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5612bf6de0b0 con 0x5612bf76d440 2024-04-21T23:09:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.782+0000 7f1fd97fa640 1 -- 172.21.15.44:0/1508974779 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f1fe001c070 con 0x5612bf76d440 2024-04-21T23:09:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.782+0000 7f1ffa7e1740 10 client.15780 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.783+0000 7f1ffa7e1740 10 client.15780 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.783+0000 7f1ffa7e1740 20 client.15780 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.783+0000 7f1ffa7e1740 1 --2- 172.21.15.44:0/1508974779 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5612bf786380 0x5612bf7887b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.783+0000 7f1ffa7e1740 4 client.15780 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:09:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.783+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/1508974779 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612bf6d8400 con 0x5612bf786380 2024-04-21T23:09:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.783+0000 7f1ff599a640 1 --2- 172.21.15.44:0/1508974779 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5612bf786380 0x5612bf7887b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.783+0000 7f1ff599a640 1 --2- 172.21.15.44:0/1508974779 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5612bf786380 0x5612bf7887b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4846de67284c4f82 server_cookie=35d4494830de02f9 in_seq=0 out_seq=0 2024-04-21T23:09:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.783+0000 7f1fd97fa640 10 client.15780 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.784+0000 7f1fd97fa640 1 -- 172.21.15.44:0/1508974779 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5612bf6d8400 con 0x5612bf786380 2024-04-21T23:09:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.784+0000 7f1fd97fa640 10 client.15780 handle_command_reply: tid=0 2024-04-21T23:09:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.784+0000 7f1ffa7e1740 1 client.15780 shutdown 2024-04-21T23:09:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.784+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/1508974779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 msgr2=0x5612bf773810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.784+0000 7f1ffa7e1740 1 --2- 172.21.15.44:0/1508974779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 0x5612bf773810 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1fe000ca40 tx=0x7f1fe000ce40 comp rx=0 tx=0).stop 2024-04-21T23:09:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.784+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/1508974779 shutdown_connections 2024-04-21T23:09:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.784+0000 7f1ffa7e1740 1 --2- 172.21.15.44:0/1508974779 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5612bf786380 0x5612bf7887b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.784+0000 7f1ffa7e1740 1 --2- 172.21.15.44:0/1508974779 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5612bf776210 0x5612bf773d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.784+0000 7f1ffa7e1740 1 --2- 172.21.15.44:0/1508974779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 0x5612bf773810 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.784+0000 7f1ffa7e1740 1 --2- 172.21.15.44:0/1508974779 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612bf76cb20 0x5612bf7732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.790 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.784+0000 7f1fd97fa640 0 client.15780 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:36.790 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.784+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/1508974779 >> 172.21.15.44:0/1508974779 conn(0x5612bf767440 msgr2=0x5612bf769190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:36.790 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.784+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/1508974779 shutdown_connections 2024-04-21T23:09:36.790 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.784+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/1508974779 wait complete. 2024-04-21T23:09:36.790 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.784+0000 7f1ffa7e1740 20 client.15780 trim_cache size 0 max 16384 2024-04-21T23:09:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.800+0000 7f1ffa7e1740 1 Processor -- start 2024-04-21T23:09:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.800+0000 7f1ffa7e1740 1 -- start start 2024-04-21T23:09:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.800+0000 7f1ffa7e1740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5612bf76cb20 0x5612bf775a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.800+0000 7f1ffa7e1740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 0x5612bf781940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.800+0000 7f1ffa7e1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612bf776210 0x5612bf771ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.800+0000 7f1ffa7e1740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff01aa970 con 0x5612bf76d440 2024-04-21T23:09:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.800+0000 7f1ffa7e1740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff01ad1f0 con 0x5612bf76cb20 2024-04-21T23:09:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.800+0000 7f1ffa7e1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff01aafe0 con 0x5612bf776210 2024-04-21T23:09:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.800+0000 7f1ff5199640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 0x5612bf781940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.800+0000 7f1ff5199640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 0x5612bf781940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57652/0 (socket says 172.21.15.44:57652) 2024-04-21T23:09:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.800+0000 7f1ff5199640 1 -- 172.21.15.44:0/3728878209 learned_addr learned my addr 172.21.15.44:0/3728878209 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.800+0000 7f1ff599a640 1 --2- 172.21.15.44:0/3728878209 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5612bf76cb20 0x5612bf775a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.800+0000 7f1ff619b640 1 --2- 172.21.15.44:0/3728878209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612bf776210 0x5612bf771ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.801+0000 7f1ff5199640 1 -- 172.21.15.44:0/3728878209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612bf776210 msgr2=0x5612bf771ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.801+0000 7f1ff5199640 1 --2- 172.21.15.44:0/3728878209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612bf776210 0x5612bf771ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.801+0000 7f1ff5199640 1 -- 172.21.15.44:0/3728878209 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5612bf76cb20 msgr2=0x5612bf775a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.801+0000 7f1ff5199640 1 --2- 172.21.15.44:0/3728878209 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5612bf76cb20 0x5612bf775a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.801+0000 7f1ff5199640 1 -- 172.21.15.44:0/3728878209 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612bf7720e0 con 0x5612bf76d440 2024-04-21T23:09:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.801+0000 7f1ff5199640 1 --2- 172.21.15.44:0/3728878209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 0x5612bf781940 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f1fe000cfb0 tx=0x7f1fe0005a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8236a09817fa850c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.801+0000 7f1fdb7fe640 1 -- 172.21.15.44:0/3728878209 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe0021030 con 0x5612bf76d440 2024-04-21T23:09:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.801+0000 7f1fdb7fe640 1 -- 172.21.15.44:0/3728878209 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fe0031070 con 0x5612bf76d440 2024-04-21T23:09:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.801+0000 7f1fdb7fe640 1 -- 172.21.15.44:0/3728878209 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe0005260 con 0x5612bf76d440 2024-04-21T23:09:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.802+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/3728878209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 msgr2=0x5612bf781940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.802+0000 7f1ffa7e1740 1 --2- 172.21.15.44:0/3728878209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 0x5612bf781940 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f1fe000cfb0 tx=0x7f1fe0005a90 comp rx=0 tx=0).stop 2024-04-21T23:09:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.802+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/3728878209 shutdown_connections 2024-04-21T23:09:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.802+0000 7f1ffa7e1740 1 --2- 172.21.15.44:0/3728878209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612bf776210 0x5612bf771ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.802+0000 7f1ffa7e1740 1 --2- 172.21.15.44:0/3728878209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf76d440 0x5612bf781940 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.802+0000 7f1ffa7e1740 1 --2- 172.21.15.44:0/3728878209 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5612bf76cb20 0x5612bf775a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.802+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/3728878209 >> 172.21.15.44:0/3728878209 conn(0x5612bf767440 msgr2=0x5612bf767820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.802+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/3728878209 shutdown_connections 2024-04-21T23:09:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.802+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/3728878209 wait complete. 2024-04-21T23:09:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.802+0000 7f1ffa7e1740 1 -- start start 2024-04-21T23:09:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.802+0000 7f1ffa7e1740 1 Processor -- start 2024-04-21T23:09:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1ffa7e1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612bf76cb20 0x5612bf77e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1ffa7e1740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5612bf76d440 0x5612bf76fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1ffa7e1740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf776210 0x5612bf770560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1ffa7e1740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff01ab6b0 con 0x5612bf776210 2024-04-21T23:09:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1ffa7e1740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff01acb20 con 0x5612bf76d440 2024-04-21T23:09:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1ffa7e1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff01abd80 con 0x5612bf76cb20 2024-04-21T23:09:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1ffa7e1740 10 client.0 _ll_register_callbacks cb 0x7f1fe00127a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1ff619b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf776210 0x5612bf770560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1ff619b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf776210 0x5612bf770560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57662/0 (socket says 172.21.15.44:57662) 2024-04-21T23:09:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1ff619b640 1 -- 172.21.15.44:0/1984915732 learned_addr learned my addr 172.21.15.44:0/1984915732 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1ff599a640 1 --2- 172.21.15.44:0/1984915732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612bf76cb20 0x5612bf77e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1ff5199640 1 --2- 172.21.15.44:0/1984915732 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5612bf76d440 0x5612bf76fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1ff619b640 1 -- 172.21.15.44:0/1984915732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612bf76cb20 msgr2=0x5612bf77e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1ff619b640 1 --2- 172.21.15.44:0/1984915732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612bf76cb20 0x5612bf77e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1ff619b640 1 -- 172.21.15.44:0/1984915732 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5612bf76d440 msgr2=0x5612bf76fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1ff619b640 1 --2- 172.21.15.44:0/1984915732 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5612bf76d440 0x5612bf76fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1ff619b640 1 -- 172.21.15.44:0/1984915732 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612bf7eb930 con 0x5612bf776210 2024-04-21T23:09:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1ff619b640 1 --2- 172.21.15.44:0/1984915732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf776210 0x5612bf770560 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80144a0 tx=0x7f1fe8007520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bab2e045807b3149 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1fd97fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:09:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1fd97fa640 1 -- 172.21.15.44:0/1984915732 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8012e60 con 0x5612bf776210 2024-04-21T23:09:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1fd97fa640 1 -- 172.21.15.44:0/1984915732 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f1fe8004d20 con 0x5612bf776210 2024-04-21T23:09:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.803+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/1984915732 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fe0031070 con 0x5612bf776210 2024-04-21T23:09:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.804+0000 7f1fd97fa640 1 -- 172.21.15.44:0/1984915732 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1fe8034050 con 0x5612bf776210 2024-04-21T23:09:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.804+0000 7f1ffa7e1740 10 client.15786 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.804+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/1984915732 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5612bf7ecab0 con 0x5612bf776210 2024-04-21T23:09:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.804+0000 7f1fd97fa640 1 -- 172.21.15.44:0/1984915732 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fe0031070 con 0x5612bf776210 2024-04-21T23:09:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.804+0000 7f1fd97fa640 1 -- 172.21.15.44:0/1984915732 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f1fe802b6e0 con 0x5612bf776210 2024-04-21T23:09:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.804+0000 7f1ffa7e1740 10 client.15786 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.804+0000 7f1ffa7e1740 10 client.15786 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.804+0000 7f1ffa7e1740 20 client.15786 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.804+0000 7f1ffa7e1740 1 --2- 172.21.15.44:0/1984915732 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5612bf7ed230 0x5612bf7ef650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.804+0000 7f1ffa7e1740 4 client.15786 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:09:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.804+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/1984915732 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5612bf6567a0 con 0x5612bf7ed230 2024-04-21T23:09:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.804+0000 7f1ff599a640 1 --2- 172.21.15.44:0/1984915732 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5612bf7ed230 0x5612bf7ef650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.805+0000 7f1ff599a640 1 --2- 172.21.15.44:0/1984915732 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5612bf7ed230 0x5612bf7ef650 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfe049cddc9d2544 server_cookie=de66766f383cbf59 in_seq=0 out_seq=0 2024-04-21T23:09:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.805+0000 7f1fd97fa640 10 client.15786 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.812+0000 7f1fd97fa640 1 -- 172.21.15.44:0/1984915732 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5612bf6567a0 con 0x5612bf7ed230 2024-04-21T23:09:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.812+0000 7f1fd97fa640 10 client.15786 handle_command_reply: tid=0 2024-04-21T23:09:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.812+0000 7f1ffa7e1740 1 client.15786 shutdown 2024-04-21T23:09:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.812+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/1984915732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf776210 msgr2=0x5612bf770560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.812+0000 7f1ffa7e1740 1 --2- 172.21.15.44:0/1984915732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf776210 0x5612bf770560 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80144a0 tx=0x7f1fe8007520 comp rx=0 tx=0).stop 2024-04-21T23:09:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.812+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/1984915732 shutdown_connections 2024-04-21T23:09:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.812+0000 7f1ffa7e1740 1 --2- 172.21.15.44:0/1984915732 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5612bf7ed230 0x5612bf7ef650 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.812+0000 7f1ffa7e1740 1 --2- 172.21.15.44:0/1984915732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5612bf776210 0x5612bf770560 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.812+0000 7f1ffa7e1740 1 --2- 172.21.15.44:0/1984915732 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5612bf76d440 0x5612bf76fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.812+0000 7f1fd97fa640 0 client.15786 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.812+0000 7f1ffa7e1740 1 --2- 172.21.15.44:0/1984915732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5612bf76cb20 0x5612bf77e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.812+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/1984915732 >> 172.21.15.44:0/1984915732 conn(0x5612bf767440 msgr2=0x5612bf773270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.813+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/1984915732 shutdown_connections 2024-04-21T23:09:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.813+0000 7f1ffa7e1740 1 -- 172.21.15.44:0/1984915732 wait complete. 2024-04-21T23:09:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.813+0000 7f1ffa7e1740 20 client.15786 trim_cache size 0 max 16384 2024-04-21T23:09:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:09:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:09:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:09:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:09:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.813+0000 7f1ff7bfe640 1 -- 172.21.15.44:0/2599852966 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1fdc0a2e10 msgr2=0x7f1fdc0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.813+0000 7f1ff7bfe640 1 --2- 172.21.15.44:0/2599852966 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1fdc0a2e10 0x7f1fdc0a52d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8007fd0 tx=0x7f1fe8007d60 comp rx=0 tx=0).stop 2024-04-21T23:09:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.813+0000 7f1ff7bfe640 1 -- 172.21.15.44:0/2599852966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ff01029b0 msgr2=0x7f1ff0197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.813+0000 7f1ff7bfe640 1 --2- 172.21.15.44:0/2599852966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ff01029b0 0x7f1ff0197fd0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f1fec02efd0 tx=0x7f1fec02ce50 comp rx=0 tx=0).stop 2024-04-21T23:09:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.813+0000 7f1ff7bfe640 1 -- 172.21.15.44:0/2599852966 shutdown_connections 2024-04-21T23:09:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.813+0000 7f1ff7bfe640 1 --2- 172.21.15.44:0/2599852966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ff01032d0 0x7f1ff0198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.813+0000 7f1ff7bfe640 1 --2- 172.21.15.44:0/2599852966 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1fdc0a2e10 0x7f1fdc0a52d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.813+0000 7f1ff7bfe640 1 --2- 172.21.15.44:0/2599852966 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1ff010bd80 0x7f1ff019c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.813+0000 7f1ff7bfe640 1 --2- 172.21.15.44:0/2599852966 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1ff01029b0 0x7f1ff0197fd0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.813+0000 7f1ff7bfe640 1 -- 172.21.15.44:0/2599852966 >> 172.21.15.44:0/2599852966 conn(0x7f1ff0077760 msgr2=0x7f1ff00fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.813+0000 7f1ff7bfe640 1 -- 172.21.15.44:0/2599852966 shutdown_connections 2024-04-21T23:09:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:36.813+0000 7f1ff7bfe640 1 -- 172.21.15.44:0/2599852966 wait complete. 2024-04-21T23:09:38.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:38 smithi044 ceph-mon[27934]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 341 B/s wr, 3 op/s 2024-04-21T23:09:38.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:38 smithi069 ceph-mon[31816]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 341 B/s wr, 3 op/s 2024-04-21T23:09:38.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:38 smithi055 ceph-mon[31117]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 341 B/s wr, 3 op/s 2024-04-21T23:09:40.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:40 smithi044 ceph-mon[27934]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-21T23:09:40.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:40 smithi055 ceph-mon[31117]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-21T23:09:40.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:40 smithi069 ceph-mon[31816]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-21T23:09:42.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:42 smithi044 ceph-mon[27934]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-04-21T23:09:42.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:42 smithi055 ceph-mon[31117]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-04-21T23:09:42.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:42 smithi069 ceph-mon[31816]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-04-21T23:09:44.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:44 smithi044 ceph-mon[27934]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2024-04-21T23:09:44.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:44 smithi055 ceph-mon[31117]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2024-04-21T23:09:44.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:44 smithi069 ceph-mon[31816]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2024-04-21T23:09:46.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:46 smithi044 ceph-mon[27934]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-04-21T23:09:46.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:46 smithi069 ceph-mon[31816]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-04-21T23:09:46.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:46 smithi055 ceph-mon[31117]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-04-21T23:09:48.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:48 smithi044 ceph-mon[27934]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-04-21T23:09:48.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:48 smithi055 ceph-mon[31117]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-04-21T23:09:48.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:48 smithi069 ceph-mon[31816]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-04-21T23:09:49.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.949+0000 7fe38c0fa640 1 -- 172.21.15.44:0/338979961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe384107740 msgr2=0x7fe384107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:49.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.949+0000 7fe38c0fa640 1 --2- 172.21.15.44:0/338979961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe384107740 0x7fe384107b20 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe380009930 tx=0x7fe38002bc30 comp rx=0 tx=0).stop 2024-04-21T23:09:49.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.949+0000 7fe38c0fa640 1 -- 172.21.15.44:0/338979961 shutdown_connections 2024-04-21T23:09:49.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.949+0000 7fe38c0fa640 1 --2- 172.21.15.44:0/338979961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe384101730 0x7fe384101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.949+0000 7fe38c0fa640 1 --2- 172.21.15.44:0/338979961 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe3841020f0 0x7fe38410e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.949+0000 7fe38c0fa640 1 --2- 172.21.15.44:0/338979961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe384107740 0x7fe384107b20 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.949+0000 7fe38c0fa640 1 -- 172.21.15.44:0/338979961 >> 172.21.15.44:0/338979961 conn(0x7fe3840fd220 msgr2=0x7fe3840ff640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:49.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.949+0000 7fe38c0fa640 1 -- 172.21.15.44:0/338979961 shutdown_connections 2024-04-21T23:09:49.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.949+0000 7fe38c0fa640 1 -- 172.21.15.44:0/338979961 wait complete. 2024-04-21T23:09:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.950+0000 7fe38c0fa640 1 Processor -- start 2024-04-21T23:09:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.950+0000 7fe38c0fa640 1 -- start start 2024-04-21T23:09:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.950+0000 7fe38c0fa640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe384101730 0x7fe384195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.950+0000 7fe38c0fa640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3841020f0 0x7fe384196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.950+0000 7fe38c0fa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe384107740 0x7fe38419a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.950+0000 7fe38c0fa640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe38410f9a0 con 0x7fe384107740 2024-04-21T23:09:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.950+0000 7fe38c0fa640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe38410f820 con 0x7fe384101730 2024-04-21T23:09:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.950+0000 7fe38c0fa640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe38410fb20 con 0x7fe3841020f0 2024-04-21T23:09:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.951+0000 7fe38a697640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe384107740 0x7fe38419a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.951+0000 7fe389e96640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe384101730 0x7fe384195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.951+0000 7fe38a697640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe384107740 0x7fe38419a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51470/0 (socket says 172.21.15.44:51470) 2024-04-21T23:09:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.951+0000 7fe389695640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3841020f0 0x7fe384196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:49.952 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.951+0000 7fe38a697640 1 -- 172.21.15.44:0/199740851 learned_addr learned my addr 172.21.15.44:0/199740851 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.951+0000 7fe38a697640 1 -- 172.21.15.44:0/199740851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3841020f0 msgr2=0x7fe384196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.951+0000 7fe38a697640 1 --2- 172.21.15.44:0/199740851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3841020f0 0x7fe384196380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.951+0000 7fe38a697640 1 -- 172.21.15.44:0/199740851 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe384101730 msgr2=0x7fe384195e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.951+0000 7fe38a697640 1 --2- 172.21.15.44:0/199740851 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe384101730 0x7fe384195e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.951+0000 7fe38a697640 1 -- 172.21.15.44:0/199740851 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe38419ae30 con 0x7fe384107740 2024-04-21T23:09:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.951+0000 7fe389e96640 1 --2- 172.21.15.44:0/199740851 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe384101730 0x7fe384195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:09:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.951+0000 7fe38a697640 1 --2- 172.21.15.44:0/199740851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe384107740 0x7fe38419a700 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe37c00edb0 tx=0x7fe37c012490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.951+0000 7fe37affd640 1 -- 172.21.15.44:0/199740851 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37c01c030 con 0x7fe384107740 2024-04-21T23:09:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.952+0000 7fe37affd640 1 -- 172.21.15.44:0/199740851 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe37c012db0 con 0x7fe384107740 2024-04-21T23:09:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.952+0000 7fe38c0fa640 1 -- 172.21.15.44:0/199740851 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe38419b120 con 0x7fe384107740 2024-04-21T23:09:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.952+0000 7fe37affd640 1 -- 172.21.15.44:0/199740851 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37c018400 con 0x7fe384107740 2024-04-21T23:09:49.953 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.952+0000 7fe38c0fa640 1 -- 172.21.15.44:0/199740851 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3841a29b0 con 0x7fe384107740 2024-04-21T23:09:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.955+0000 7fe38edb6740 1 Processor -- start 2024-04-21T23:09:49.955 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.956+0000 7fe37affd640 1 -- 172.21.15.44:0/199740851 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7fe37c01f070 con 0x7fe384107740 2024-04-21T23:09:49.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.956+0000 7fe38edb6740 1 -- start start 2024-04-21T23:09:49.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.956+0000 7fe38edb6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf80b20 0x563cadf80f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:49.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.956+0000 7fe38edb6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf81440 0x563cadf89cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:49.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.956+0000 7fe38edb6740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563cadf8a210 0x563cadf8c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:49.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.956+0000 7fe38edb6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563cadee4b70 con 0x563cadf80b20 2024-04-21T23:09:49.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.956+0000 7fe38edb6740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x563cade8d470 con 0x563cadf8a210 2024-04-21T23:09:49.956 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.956+0000 7fe38edb6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563cadee0b30 con 0x563cadf81440 2024-04-21T23:09:49.957 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.957+0000 7fe389695640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf80b20 0x563cadf80f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:49.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.957+0000 7fe389695640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf80b20 0x563cadf80f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51482/0 (socket says 172.21.15.44:51482) 2024-04-21T23:09:49.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.957+0000 7fe389695640 1 -- 172.21.15.44:0/2598895072 learned_addr learned my addr 172.21.15.44:0/2598895072 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:49.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.957+0000 7fe37affd640 1 --2- 172.21.15.44:0/199740851 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe3700a2e90 0x7fe3700a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:49.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.957+0000 7fe389e96640 1 --2- 172.21.15.44:0/2598895072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf81440 0x563cadf89cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:49.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.957+0000 7fe37affd640 1 -- 172.21.15.44:0/199740851 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fe37c0d3680 con 0x7fe384107740 2024-04-21T23:09:49.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.957+0000 7fe389695640 1 -- 172.21.15.44:0/2598895072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf81440 msgr2=0x563cadf89cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:49.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.957+0000 7fe389695640 1 --2- 172.21.15.44:0/2598895072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf81440 0x563cadf89cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.957+0000 7fe389695640 1 -- 172.21.15.44:0/2598895072 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563cadf8a210 msgr2=0x563cadf8c600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T23:09:49.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.957+0000 7fe389695640 1 --2- 172.21.15.44:0/2598895072 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563cadf8a210 0x563cadf8c600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.957+0000 7fe389695640 1 -- 172.21.15.44:0/2598895072 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cadbf9210 con 0x563cadf80b20 2024-04-21T23:09:49.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.957+0000 7fe38a697640 1 --2- 172.21.15.44:0/199740851 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe3700a2e90 0x7fe3700a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:49.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.957+0000 7fe389e96640 1 --2- 172.21.15.44:0/2598895072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf81440 0x563cadf89cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:09:49.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.958+0000 7fe389695640 1 --2- 172.21.15.44:0/2598895072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf80b20 0x563cadf80f00 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe37400bc30 tx=0x7fe3740107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d9993b0c4f9bb2f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:49.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.958+0000 7fe38a697640 1 --2- 172.21.15.44:0/199740851 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe3700a2e90 0x7fe3700a5350 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0d0910 tx=0x7fe37c005b40 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:49.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.958+0000 7fe36ffff640 1 -- 172.21.15.44:0/2598895072 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37401c480 con 0x563cadf80b20 2024-04-21T23:09:49.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.958+0000 7fe36ffff640 1 -- 172.21.15.44:0/2598895072 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe37401cb00 con 0x563cadf80b20 2024-04-21T23:09:49.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.958+0000 7fe36ffff640 1 -- 172.21.15.44:0/2598895072 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe374011c30 con 0x563cadf80b20 2024-04-21T23:09:49.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.959+0000 7fe38edb6740 1 -- 172.21.15.44:0/2598895072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf80b20 msgr2=0x563cadf80f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:49.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.959+0000 7fe38edb6740 1 --2- 172.21.15.44:0/2598895072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf80b20 0x563cadf80f00 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe37400bc30 tx=0x7fe3740107a0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.959+0000 7fe38edb6740 1 -- 172.21.15.44:0/2598895072 shutdown_connections 2024-04-21T23:09:49.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.959+0000 7fe38edb6740 1 --2- 172.21.15.44:0/2598895072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf81440 0x563cadf89cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.959+0000 7fe38edb6740 1 --2- 172.21.15.44:0/2598895072 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563cadf8a210 0x563cadf8c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.959+0000 7fe38edb6740 1 --2- 172.21.15.44:0/2598895072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf80b20 0x563cadf80f00 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.959+0000 7fe38edb6740 1 -- 172.21.15.44:0/2598895072 >> 172.21.15.44:0/2598895072 conn(0x563cadf7b440 msgr2=0x563cadf7b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:49.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.959+0000 7fe38edb6740 1 -- 172.21.15.44:0/2598895072 shutdown_connections 2024-04-21T23:09:49.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.959+0000 7fe38edb6740 1 -- 172.21.15.44:0/2598895072 wait complete. 2024-04-21T23:09:49.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.960+0000 7fe38edb6740 1 -- start start 2024-04-21T23:09:49.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.960+0000 7fe38edb6740 1 Processor -- start 2024-04-21T23:09:49.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.960+0000 7fe38edb6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf81440 0x563cadf89a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:49.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.961+0000 7fe38edb6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf8a210 0x563cadf86d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:49.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.961+0000 7fe38edb6740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563cadf872b0 0x563cadf87730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:49.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.961+0000 7fe38edb6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563cadf8ce40 con 0x563cadf81440 2024-04-21T23:09:49.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.961+0000 7fe389695640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf81440 0x563cadf89a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:49.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.961+0000 7fe38edb6740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x563cadea0a10 con 0x563cadf872b0 2024-04-21T23:09:49.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.961+0000 7fe38edb6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563cadf8cfc0 con 0x563cadf8a210 2024-04-21T23:09:49.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.961+0000 7fe389695640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf81440 0x563cadf89a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51494/0 (socket says 172.21.15.44:51494) 2024-04-21T23:09:49.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.961+0000 7fe389695640 1 -- 172.21.15.44:0/1398376002 learned_addr learned my addr 172.21.15.44:0/1398376002 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.961+0000 7fe389e96640 1 --2- 172.21.15.44:0/1398376002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf8a210 0x563cadf86d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.961+0000 7fe38edb6740 10 client.0 _ll_register_callbacks cb 0x563cadedbb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.961+0000 7fe389695640 1 -- 172.21.15.44:0/1398376002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf8a210 msgr2=0x563cadf86d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.961+0000 7fe389695640 1 --2- 172.21.15.44:0/1398376002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf8a210 0x563cadf86d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.961+0000 7fe389695640 1 -- 172.21.15.44:0/1398376002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563cadf872b0 msgr2=0x563cadf87730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T23:09:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.961+0000 7fe389695640 1 --2- 172.21.15.44:0/1398376002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563cadf872b0 0x563cadf87730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.961+0000 7fe389695640 1 -- 172.21.15.44:0/1398376002 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cadcc5620 con 0x563cadf81440 2024-04-21T23:09:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.961+0000 7fe38edb6740 1 -- 172.21.15.44:0/1398376002 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe37401cb00 con 0x563cadf81440 2024-04-21T23:09:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.962+0000 7fe389695640 1 --2- 172.21.15.44:0/1398376002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf81440 0x563cadf89a80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe374011010 tx=0x7fe37401cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30182325c0f74f9e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.962+0000 7fe36d7fa640 10 client.15798 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:09:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.962+0000 7fe36d7fa640 1 -- 172.21.15.44:0/1398376002 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37401c680 con 0x563cadf81440 2024-04-21T23:09:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.962+0000 7fe36d7fa640 1 -- 172.21.15.44:0/1398376002 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fe37401c820 con 0x563cadf81440 2024-04-21T23:09:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.962+0000 7fe36d7fa640 1 -- 172.21.15.44:0/1398376002 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe374011930 con 0x563cadf81440 2024-04-21T23:09:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.962+0000 7fe38edb6740 10 client.15798 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:49.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.962+0000 7fe38edb6740 1 -- 172.21.15.44:0/1398376002 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({fsmap=21}) -- 0x563cadf98f50 con 0x563cadf81440 2024-04-21T23:09:49.966 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.963+0000 7fe36d7fa640 1 -- 172.21.15.44:0/1398376002 <== mon.0 v2:172.21.15.44:3300/0 4 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fe374011ad0 con 0x563cadf81440 2024-04-21T23:09:49.966 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.963+0000 7fe38edb6740 10 client.15798 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:49.966 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.963+0000 7fe38edb6740 10 client.15798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:49.966 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.963+0000 7fe38edb6740 20 client.15798 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:49.966 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.963+0000 7fe38edb6740 1 --2- 172.21.15.44:0/1398376002 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563cadf9a2a0 0x563cadf9c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:49.966 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.963+0000 7fe38edb6740 4 client.15798 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:09:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.963+0000 7fe38edb6740 1 -- 172.21.15.44:0/1398376002 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563cadef20b0 con 0x563cadf9a2a0 2024-04-21T23:09:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.963+0000 7fe389e96640 1 --2- 172.21.15.44:0/1398376002 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563cadf9a2a0 0x563cadf9c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.964+0000 7fe36d7fa640 10 client.15798 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.964+0000 7fe389e96640 1 --2- 172.21.15.44:0/1398376002 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563cadf9a2a0 0x563cadf9c690 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4e50d8a802796df server_cookie=437ebfa75eb7271a in_seq=0 out_seq=0 2024-04-21T23:09:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.965+0000 7fe36d7fa640 1 -- 172.21.15.44:0/1398376002 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563cadef20b0 con 0x563cadf9a2a0 2024-04-21T23:09:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.965+0000 7fe36d7fa640 10 client.15798 handle_command_reply: tid=0 2024-04-21T23:09:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.965+0000 7fe38edb6740 1 client.15798 shutdown 2024-04-21T23:09:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.965+0000 7fe38edb6740 1 -- 172.21.15.44:0/1398376002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf81440 msgr2=0x563cadf89a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.965+0000 7fe38edb6740 1 --2- 172.21.15.44:0/1398376002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf81440 0x563cadf89a80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe374011010 tx=0x7fe37401cec0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.965+0000 7fe38edb6740 1 -- 172.21.15.44:0/1398376002 shutdown_connections 2024-04-21T23:09:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.965+0000 7fe38edb6740 1 --2- 172.21.15.44:0/1398376002 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563cadf9a2a0 0x563cadf9c690 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.965+0000 7fe38edb6740 1 --2- 172.21.15.44:0/1398376002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf8a210 0x563cadf86d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.965+0000 7fe36d7fa640 0 client.15798 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.965+0000 7fe38edb6740 1 --2- 172.21.15.44:0/1398376002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563cadf872b0 0x563cadf87730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.965+0000 7fe38edb6740 1 --2- 172.21.15.44:0/1398376002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf81440 0x563cadf89a80 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:49.968 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.965+0000 7fe38edb6740 1 -- 172.21.15.44:0/1398376002 >> 172.21.15.44:0/1398376002 conn(0x563cadf7b440 msgr2=0x563cadf7c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:49.968 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.965+0000 7fe38edb6740 1 -- 172.21.15.44:0/1398376002 shutdown_connections 2024-04-21T23:09:49.968 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.965+0000 7fe38edb6740 1 -- 172.21.15.44:0/1398376002 wait complete. 2024-04-21T23:09:49.968 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.965+0000 7fe38edb6740 20 client.15798 trim_cache size 0 max 16384 2024-04-21T23:09:50.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.995+0000 7fe38edb6740 1 Processor -- start 2024-04-21T23:09:50.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.995+0000 7fe38edb6740 1 -- start start 2024-04-21T23:09:50.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.996+0000 7fe38edb6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf81440 0x563cadf895b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:50.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.996+0000 7fe38edb6740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563cadf872b0 0x563cadf9f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:50.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.996+0000 7fe38edb6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf8a210 0x563cadf9fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:50.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.996+0000 7fe38edb6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe3841aaae0 con 0x563cadf81440 2024-04-21T23:09:50.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.996+0000 7fe38edb6740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe3841ad360 con 0x563cadf872b0 2024-04-21T23:09:50.002 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.996+0000 7fe38edb6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe3841ab150 con 0x563cadf8a210 2024-04-21T23:09:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.996+0000 7fe389695640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf81440 0x563cadf895b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.996+0000 7fe389695640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf81440 0x563cadf895b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51502/0 (socket says 172.21.15.44:51502) 2024-04-21T23:09:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.996+0000 7fe389695640 1 -- 172.21.15.44:0/540683076 learned_addr learned my addr 172.21.15.44:0/540683076 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.996+0000 7fe389e96640 1 --2- 172.21.15.44:0/540683076 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563cadf872b0 0x563cadf9f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.997+0000 7fe389695640 1 -- 172.21.15.44:0/540683076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf8a210 msgr2=0x563cadf9fbf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T23:09:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.997+0000 7fe389695640 1 --2- 172.21.15.44:0/540683076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf8a210 0x563cadf9fbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.997+0000 7fe389695640 1 -- 172.21.15.44:0/540683076 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563cadf872b0 msgr2=0x563cadf9f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.997+0000 7fe389695640 1 --2- 172.21.15.44:0/540683076 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563cadf872b0 0x563cadf9f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.997+0000 7fe389695640 1 -- 172.21.15.44:0/540683076 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cadf81270 con 0x563cadf81440 2024-04-21T23:09:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.997+0000 7fe389e96640 1 --2- 172.21.15.44:0/540683076 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563cadf872b0 0x563cadf9f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:09:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.997+0000 7fe389695640 1 --2- 172.21.15.44:0/540683076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf81440 0x563cadf895b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe37400bb60 tx=0x7fe374022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=823c877a1ed4e866 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.997+0000 7fe36f7fe640 1 -- 172.21.15.44:0/540683076 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe374011640 con 0x563cadf81440 2024-04-21T23:09:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.997+0000 7fe36f7fe640 1 -- 172.21.15.44:0/540683076 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe37401cb00 con 0x563cadf81440 2024-04-21T23:09:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.997+0000 7fe36f7fe640 1 -- 172.21.15.44:0/540683076 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe374004bc0 con 0x563cadf81440 2024-04-21T23:09:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.998+0000 7fe38edb6740 1 -- 172.21.15.44:0/540683076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf81440 msgr2=0x563cadf895b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.998+0000 7fe38edb6740 1 --2- 172.21.15.44:0/540683076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf81440 0x563cadf895b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe37400bb60 tx=0x7fe374022000 comp rx=0 tx=0).stop 2024-04-21T23:09:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.998+0000 7fe38edb6740 1 -- 172.21.15.44:0/540683076 shutdown_connections 2024-04-21T23:09:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.998+0000 7fe38edb6740 1 --2- 172.21.15.44:0/540683076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf8a210 0x563cadf9fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.998+0000 7fe38edb6740 1 --2- 172.21.15.44:0/540683076 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563cadf872b0 0x563cadf9f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.998+0000 7fe38edb6740 1 --2- 172.21.15.44:0/540683076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf81440 0x563cadf895b0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.998+0000 7fe38edb6740 1 -- 172.21.15.44:0/540683076 >> 172.21.15.44:0/540683076 conn(0x563cadf7b440 msgr2=0x563cadf76220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.998+0000 7fe38edb6740 1 -- 172.21.15.44:0/540683076 shutdown_connections 2024-04-21T23:09:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.998+0000 7fe38edb6740 1 -- 172.21.15.44:0/540683076 wait complete. 2024-04-21T23:09:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.999+0000 7fe38edb6740 1 -- start start 2024-04-21T23:09:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.999+0000 7fe38edb6740 1 Processor -- start 2024-04-21T23:09:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.999+0000 7fe38edb6740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563cadf81440 0x563cadf9f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.999+0000 7fe38edb6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf872b0 0x563cadf9ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.999+0000 7fe38edb6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf8a210 0x563cadf9b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.999+0000 7fe38edb6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe3841ab820 con 0x563cadf872b0 2024-04-21T23:09:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.999+0000 7fe389e96640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf872b0 0x563cadf9ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.999+0000 7fe389e96640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf872b0 0x563cadf9ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51516/0 (socket says 172.21.15.44:51516) 2024-04-21T23:09:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.999+0000 7fe389e96640 1 -- 172.21.15.44:0/98641960 learned_addr learned my addr 172.21.15.44:0/98641960 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.999+0000 7fe38edb6740 1 -- 172.21.15.44:0/98641960 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe3841acc90 con 0x563cadf81440 2024-04-21T23:09:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:49.999+0000 7fe38edb6740 1 -- 172.21.15.44:0/98641960 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe3841abef0 con 0x563cadf8a210 2024-04-21T23:09:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.000+0000 7fe389695640 1 --2- 172.21.15.44:0/98641960 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563cadf81440 0x563cadf9f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.000+0000 7fe38edb6740 10 client.0 _ll_register_callbacks cb 0x7fe380009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.000+0000 7fe389695640 1 --2- 172.21.15.44:0/98641960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf8a210 0x563cadf9b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:50.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.000+0000 7fe389e96640 1 -- 172.21.15.44:0/98641960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf8a210 msgr2=0x563cadf9b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:50.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.000+0000 7fe389e96640 1 --2- 172.21.15.44:0/98641960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf8a210 0x563cadf9b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:50.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.000+0000 7fe389e96640 1 -- 172.21.15.44:0/98641960 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563cadf81440 msgr2=0x563cadf9f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:50.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.000+0000 7fe389e96640 1 --2- 172.21.15.44:0/98641960 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563cadf81440 0x563cadf9f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:50.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.000+0000 7fe389e96640 1 -- 172.21.15.44:0/98641960 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cadf9b950 con 0x563cadf872b0 2024-04-21T23:09:50.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.000+0000 7fe389695640 1 --2- 172.21.15.44:0/98641960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf8a210 0x563cadf9b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:09:50.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.000+0000 7fe389e96640 1 --2- 172.21.15.44:0/98641960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf872b0 0x563cadf9ad20 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fe38002c240 tx=0x7fe38002c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=587802defc3a3a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:50.007 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.000+0000 7fe36d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:09:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.000+0000 7fe36d7fa640 1 -- 172.21.15.44:0/98641960 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe380004250 con 0x563cadf872b0 2024-04-21T23:09:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.000+0000 7fe36d7fa640 1 -- 172.21.15.44:0/98641960 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fe3800043f0 con 0x563cadf872b0 2024-04-21T23:09:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.000+0000 7fe38edb6740 1 -- 172.21.15.44:0/98641960 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe37401cb00 con 0x563cadf872b0 2024-04-21T23:09:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.000+0000 7fe36d7fa640 1 -- 172.21.15.44:0/98641960 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe38005b050 con 0x563cadf872b0 2024-04-21T23:09:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.000+0000 7fe38edb6740 10 client.15804 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.001+0000 7fe38edb6740 1 -- 172.21.15.44:0/98641960 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563cae0003c0 con 0x563cadf872b0 2024-04-21T23:09:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.001+0000 7fe36d7fa640 1 -- 172.21.15.44:0/98641960 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe38004a410 con 0x563cadf872b0 2024-04-21T23:09:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.001+0000 7fe36d7fa640 1 -- 172.21.15.44:0/98641960 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fe380005050 con 0x563cadf872b0 2024-04-21T23:09:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.001+0000 7fe38edb6740 10 client.15804 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.001+0000 7fe38edb6740 10 client.15804 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.001+0000 7fe38edb6740 20 client.15804 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.001+0000 7fe38edb6740 1 --2- 172.21.15.44:0/98641960 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563cadf9a2a0 0x563cae002b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.001+0000 7fe38edb6740 4 client.15804 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:09:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.001+0000 7fe38edb6740 1 -- 172.21.15.44:0/98641960 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fe37401cb00 con 0x563cadf9a2a0 2024-04-21T23:09:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.001+0000 7fe389695640 1 --2- 172.21.15.44:0/98641960 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563cadf9a2a0 0x563cae002b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.002+0000 7fe36d7fa640 10 client.15804 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.002+0000 7fe389695640 1 --2- 172.21.15.44:0/98641960 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563cadf9a2a0 0x563cae002b00 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1ffc2115440d7ed server_cookie=7404ec578e5ed70b in_seq=0 out_seq=0 2024-04-21T23:09:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.006+0000 7fe36d7fa640 1 -- 172.21.15.44:0/98641960 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fe37401cb00 con 0x563cadf9a2a0 2024-04-21T23:09:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.006+0000 7fe36d7fa640 10 client.15804 handle_command_reply: tid=0 2024-04-21T23:09:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.006+0000 7fe38edb6740 1 client.15804 shutdown 2024-04-21T23:09:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.006+0000 7fe38edb6740 1 -- 172.21.15.44:0/98641960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf872b0 msgr2=0x563cadf9ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.006+0000 7fe38edb6740 1 --2- 172.21.15.44:0/98641960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf872b0 0x563cadf9ad20 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fe38002c240 tx=0x7fe38002c3d0 comp rx=0 tx=0).stop 2024-04-21T23:09:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.006+0000 7fe38edb6740 1 -- 172.21.15.44:0/98641960 shutdown_connections 2024-04-21T23:09:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.006+0000 7fe38edb6740 1 --2- 172.21.15.44:0/98641960 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563cadf9a2a0 0x563cae002b00 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.006+0000 7fe38edb6740 1 --2- 172.21.15.44:0/98641960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563cadf8a210 0x563cadf9b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.006+0000 7fe38edb6740 1 --2- 172.21.15.44:0/98641960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563cadf872b0 0x563cadf9ad20 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.006+0000 7fe38edb6740 1 --2- 172.21.15.44:0/98641960 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563cadf81440 0x563cadf9f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:50.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.006+0000 7fe38edb6740 1 -- 172.21.15.44:0/98641960 >> 172.21.15.44:0/98641960 conn(0x563cadf7b440 msgr2=0x563cadf855a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:50.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.006+0000 7fe36d7fa640 0 client.15804 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:50.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.006+0000 7fe38edb6740 1 -- 172.21.15.44:0/98641960 shutdown_connections 2024-04-21T23:09:50.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.006+0000 7fe38edb6740 1 -- 172.21.15.44:0/98641960 wait complete. 2024-04-21T23:09:50.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.006+0000 7fe38edb6740 20 client.15804 trim_cache size 0 max 16384 2024-04-21T23:09:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:09:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:09:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:09:50.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:09:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.007+0000 7fe38c0fa640 1 -- 172.21.15.44:0/199740851 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe3700a2e90 msgr2=0x7fe3700a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.007+0000 7fe38c0fa640 1 --2- 172.21.15.44:0/199740851 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe3700a2e90 0x7fe3700a5350 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0d0910 tx=0x7fe37c005b40 comp rx=0 tx=0).stop 2024-04-21T23:09:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.007+0000 7fe38c0fa640 1 -- 172.21.15.44:0/199740851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe384107740 msgr2=0x7fe38419a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.007+0000 7fe38c0fa640 1 --2- 172.21.15.44:0/199740851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe384107740 0x7fe38419a700 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe37c00edb0 tx=0x7fe37c012490 comp rx=0 tx=0).stop 2024-04-21T23:09:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.007+0000 7fe38c0fa640 1 -- 172.21.15.44:0/199740851 shutdown_connections 2024-04-21T23:09:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.007+0000 7fe38c0fa640 1 --2- 172.21.15.44:0/199740851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3841020f0 0x7fe384196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.007+0000 7fe38c0fa640 1 --2- 172.21.15.44:0/199740851 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe3700a2e90 0x7fe3700a5350 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.007+0000 7fe38c0fa640 1 --2- 172.21.15.44:0/199740851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe384107740 0x7fe38419a700 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.007+0000 7fe38c0fa640 1 --2- 172.21.15.44:0/199740851 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe384101730 0x7fe384195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.007+0000 7fe38c0fa640 1 -- 172.21.15.44:0/199740851 >> 172.21.15.44:0/199740851 conn(0x7fe3840fd220 msgr2=0x7fe3840fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.007+0000 7fe38c0fa640 1 -- 172.21.15.44:0/199740851 shutdown_connections 2024-04-21T23:09:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:09:50.007+0000 7fe38c0fa640 1 -- 172.21.15.44:0/199740851 wait complete. 2024-04-21T23:09:50.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:50 smithi044 ceph-mon[27934]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.4 KiB/s wr, 4 op/s 2024-04-21T23:09:50.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:50 smithi055 ceph-mon[31117]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.4 KiB/s wr, 4 op/s 2024-04-21T23:09:50.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:50 smithi069 ceph-mon[31816]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.4 KiB/s wr, 4 op/s 2024-04-21T23:09:52.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:52 smithi044 ceph-mon[27934]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-21T23:09:52.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:52 smithi055 ceph-mon[31117]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-21T23:09:52.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:52 smithi069 ceph-mon[31816]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-21T23:09:54.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:54 smithi044 ceph-mon[27934]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-21T23:09:54.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:54 smithi069 ceph-mon[31816]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-21T23:09:54.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:54 smithi055 ceph-mon[31117]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-21T23:09:56.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:56 smithi044 ceph-mon[27934]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-21T23:09:56.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:56 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:09:56.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:56 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:56.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:56 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:56.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:56 smithi055 ceph-mon[31117]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-21T23:09:56.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:56 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:09:56.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:56 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:56.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:56 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:56.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:56 smithi069 ceph-mon[31816]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-21T23:09:56.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:56 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:09:56.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:56 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:56.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:56 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:57.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:57 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:57.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:57 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:57.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:57 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:57.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:57 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:57.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:57 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:57.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:57 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:09:57.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:57 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:57.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:57 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:57.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:57 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:57.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:57 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:57.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:57 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:57.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:57 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:57.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:57 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:09:57.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:57 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:57.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:57 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:57.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:57 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:57.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:57 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:57.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:57 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:57.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:57 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:09:57.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:57 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:09:57.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:57 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:09:58.088 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:09:58.089 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:09:58.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.200+0000 7fbd0e3a7640 1 -- 172.21.15.44:0/31395345 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd08100fd0 msgr2=0x7fbd0810d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.200+0000 7fbd0e3a7640 1 --2- 172.21.15.44:0/31395345 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd08100fd0 0x7fbd0810d4e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fbd00009cd0 tx=0x7fbd0002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.200+0000 7fbd0e3a7640 1 -- 172.21.15.44:0/31395345 shutdown_connections 2024-04-21T23:09:58.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.200+0000 7fbd0e3a7640 1 --2- 172.21.15.44:0/31395345 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd08100fd0 0x7fbd0810d4e0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.200+0000 7fbd0e3a7640 1 --2- 172.21.15.44:0/31395345 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd08100610 0x7fbd08100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.200+0000 7fbd0e3a7640 1 --2- 172.21.15.44:0/31395345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd081066b0 0x7fbd08106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.200+0000 7fbd0e3a7640 1 -- 172.21.15.44:0/31395345 >> 172.21.15.44:0/31395345 conn(0x7fbd080777f0 msgr2=0x7fbd08077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:58.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.200+0000 7fbd0e3a7640 1 -- 172.21.15.44:0/31395345 shutdown_connections 2024-04-21T23:09:58.200 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.200+0000 7fbd0e3a7640 1 -- 172.21.15.44:0/31395345 wait complete. 2024-04-21T23:09:58.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.201+0000 7fbd0e3a7640 1 Processor -- start 2024-04-21T23:09:58.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.201+0000 7fbd0e3a7640 1 -- start start 2024-04-21T23:09:58.202 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.201+0000 7fbd0e3a7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd08100610 0x7fbd08197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.201+0000 7fbd0e3a7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd08100fd0 0x7fbd081984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.201+0000 7fbd0e3a7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd081066b0 0x7fbd0819c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.201+0000 7fbd0e3a7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbd08111b30 con 0x7fbd08100fd0 2024-04-21T23:09:58.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.201+0000 7fbd0e3a7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbd081119b0 con 0x7fbd081066b0 2024-04-21T23:09:58.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.201+0000 7fbd0e3a7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbd08111cb0 con 0x7fbd08100610 2024-04-21T23:09:58.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.201+0000 7fbd077fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd08100fd0 0x7fbd081984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.201+0000 7fbd077fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd08100fd0 0x7fbd081984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53292/0 (socket says 172.21.15.44:53292) 2024-04-21T23:09:58.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.201+0000 7fbd077fe640 1 -- 172.21.15.44:0/516506957 learned_addr learned my addr 172.21.15.44:0/516506957 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:58.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.201+0000 7fbd07fff640 1 --2- 172.21.15.44:0/516506957 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd08100610 0x7fbd08197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.201+0000 7fbd0c944640 1 --2- 172.21.15.44:0/516506957 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd081066b0 0x7fbd0819c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.202+0000 7fbd077fe640 1 -- 172.21.15.44:0/516506957 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd08100610 msgr2=0x7fbd08197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.202+0000 7fbd077fe640 1 --2- 172.21.15.44:0/516506957 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd08100610 0x7fbd08197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.202+0000 7fbd077fe640 1 -- 172.21.15.44:0/516506957 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd081066b0 msgr2=0x7fbd0819c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.202+0000 7fbd077fe640 1 --2- 172.21.15.44:0/516506957 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd081066b0 0x7fbd0819c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.202+0000 7fbd077fe640 1 -- 172.21.15.44:0/516506957 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd0819cf10 con 0x7fbd08100fd0 2024-04-21T23:09:58.203 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.202+0000 7fbd077fe640 1 --2- 172.21.15.44:0/516506957 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd08100fd0 0x7fbd081984b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc00c920 tx=0x7fbcfc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:58.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.202+0000 7fbd057fa640 1 -- 172.21.15.44:0/516506957 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc007c70 con 0x7fbd08100fd0 2024-04-21T23:09:58.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.202+0000 7fbd0e3a7640 1 -- 172.21.15.44:0/516506957 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd0819d200 con 0x7fbd08100fd0 2024-04-21T23:09:58.204 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.203+0000 7fbd0e3a7640 1 -- 172.21.15.44:0/516506957 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd0810dea0 con 0x7fbd08100fd0 2024-04-21T23:09:58.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.206+0000 7fbd10f8a740 1 Processor -- start 2024-04-21T23:09:58.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.206+0000 7fbd057fa640 1 -- 172.21.15.44:0/516506957 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcfc007e10 con 0x7fbd08100fd0 2024-04-21T23:09:58.206 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.206+0000 7fbd057fa640 1 -- 172.21.15.44:0/516506957 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc00f450 con 0x7fbd08100fd0 2024-04-21T23:09:58.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.206+0000 7fbd10f8a740 1 -- start start 2024-04-21T23:09:58.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.207+0000 7fbd10f8a740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 0x562fd8485120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.208+0000 7fbd10f8a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd84856f0 0x562fd8496720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.208+0000 7fbd10f8a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd8496c60 0x562fd8499050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.208+0000 7fbd10f8a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562fd83e9560 con 0x562fd84856f0 2024-04-21T23:09:58.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.208+0000 7fbd10f8a740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x562fd8391730 con 0x562fd8484d40 2024-04-21T23:09:58.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.208+0000 7fbd10f8a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562fd83e5520 con 0x562fd8496c60 2024-04-21T23:09:58.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.208+0000 7fbd07fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd84856f0 0x562fd8496720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.208+0000 7fbd0c944640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 0x562fd8485120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.208+0000 7fbd0c944640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 0x562fd8485120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:60178/0 (socket says 172.21.15.44:60178) 2024-04-21T23:09:58.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.208+0000 7fbd0c944640 1 -- 172.21.15.44:0/982853646 learned_addr learned my addr 172.21.15.44:0/982853646 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:58.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.208+0000 7fbd0c944640 1 -- 172.21.15.44:0/982853646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd8496c60 msgr2=0x562fd8499050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T23:09:58.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.208+0000 7fbd0c944640 1 --2- 172.21.15.44:0/982853646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd8496c60 0x562fd8499050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.208+0000 7fbd0c944640 1 -- 172.21.15.44:0/982853646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd84856f0 msgr2=0x562fd8496720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:58.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.208+0000 7fbd0c944640 1 --2- 172.21.15.44:0/982853646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd84856f0 0x562fd8496720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.208+0000 7fbd0c944640 1 -- 172.21.15.44:0/982853646 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fd81c77d0 con 0x562fd8484d40 2024-04-21T23:09:58.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.208+0000 7fbd0c944640 1 --2- 172.21.15.44:0/982853646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 0x562fd8485120 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbd0002c700 tx=0x7fbd00009980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=635cf4bfca1e6c05 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:58.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.209+0000 7fbcee7fc640 1 -- 172.21.15.44:0/982853646 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd00043070 con 0x562fd8484d40 2024-04-21T23:09:58.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.209+0000 7fbcee7fc640 1 -- 172.21.15.44:0/982853646 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd00032c90 con 0x562fd8484d40 2024-04-21T23:09:58.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.209+0000 7fbcee7fc640 1 -- 172.21.15.44:0/982853646 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd00039040 con 0x562fd8484d40 2024-04-21T23:09:58.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.209+0000 7fbd057fa640 1 -- 172.21.15.44:0/516506957 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7fbcfc005020 con 0x7fbd08100fd0 2024-04-21T23:09:58.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.209+0000 7fbd10f8a740 1 -- 172.21.15.44:0/982853646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 msgr2=0x562fd8485120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.209+0000 7fbd10f8a740 1 --2- 172.21.15.44:0/982853646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 0x562fd8485120 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbd0002c700 tx=0x7fbd00009980 comp rx=0 tx=0).stop 2024-04-21T23:09:58.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.209+0000 7fbd10f8a740 1 -- 172.21.15.44:0/982853646 shutdown_connections 2024-04-21T23:09:58.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.209+0000 7fbd10f8a740 1 --2- 172.21.15.44:0/982853646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd8496c60 0x562fd8499050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.210+0000 7fbd10f8a740 1 --2- 172.21.15.44:0/982853646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd84856f0 0x562fd8496720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.210+0000 7fbd10f8a740 1 --2- 172.21.15.44:0/982853646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 0x562fd8485120 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.210+0000 7fbd10f8a740 1 -- 172.21.15.44:0/982853646 >> 172.21.15.44:0/982853646 conn(0x562fd847f5a0 msgr2=0x562fd847f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:58.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.210+0000 7fbd10f8a740 1 -- 172.21.15.44:0/982853646 shutdown_connections 2024-04-21T23:09:58.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.210+0000 7fbd10f8a740 1 -- 172.21.15.44:0/982853646 wait complete. 2024-04-21T23:09:58.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.210+0000 7fbd10f8a740 1 -- start start 2024-04-21T23:09:58.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.210+0000 7fbd057fa640 1 --2- 172.21.15.44:0/516506957 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbcf00a2d30 0x7fbcf00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.210+0000 7fbd0c944640 1 --2- 172.21.15.44:0/516506957 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbcf00a2d30 0x7fbcf00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.210+0000 7fbd10f8a740 1 Processor -- start 2024-04-21T23:09:58.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.210+0000 7fbd057fa640 1 -- 172.21.15.44:0/516506957 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fbcfc0cfa90 con 0x7fbd08100fd0 2024-04-21T23:09:58.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.211+0000 7fbd10f8a740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 0x562fd8493c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.211+0000 7fbd10f8a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd84856f0 0x562fd8494180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.211+0000 7fbd10f8a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd8496c60 0x562fd84946c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.211+0000 7fbd10f8a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562fd8499990 con 0x562fd84856f0 2024-04-21T23:09:58.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.211+0000 7fbd10f8a740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x562fd83a4e00 con 0x562fd8484d40 2024-04-21T23:09:58.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.211+0000 7fbd10f8a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562fd8499b10 con 0x562fd8496c60 2024-04-21T23:09:58.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.211+0000 7fbd10f8a740 10 client.0 _ll_register_callbacks cb 0x562fd83dfeb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:58.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.211+0000 7fbd0c944640 1 --2- 172.21.15.44:0/516506957 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbcf00a2d30 0x7fbcf00a51f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbd00009bf0 tx=0x7fbd0002c0f0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:58.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.211+0000 7fbd077fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd84856f0 0x562fd8494180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.211+0000 7fbd077fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd84856f0 0x562fd8494180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53310/0 (socket says 172.21.15.44:53310) 2024-04-21T23:09:58.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.211+0000 7fbd077fe640 1 -- 172.21.15.44:0/3413590971 learned_addr learned my addr 172.21.15.44:0/3413590971 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:58.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.211+0000 7fbd07fff640 1 --2- 172.21.15.44:0/3413590971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 0x562fd8493c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.211+0000 7fbd0c944640 1 --2- 172.21.15.44:0/3413590971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd8496c60 0x562fd84946c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.211+0000 7fbd077fe640 1 -- 172.21.15.44:0/3413590971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd8496c60 msgr2=0x562fd84946c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:58.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.211+0000 7fbd077fe640 1 --2- 172.21.15.44:0/3413590971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd8496c60 0x562fd84946c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.211+0000 7fbd077fe640 1 -- 172.21.15.44:0/3413590971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 msgr2=0x562fd8493c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:58.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.211+0000 7fbd077fe640 1 --2- 172.21.15.44:0/3413590971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 0x562fd8493c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.211+0000 7fbd077fe640 1 -- 172.21.15.44:0/3413590971 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fd80fd3f0 con 0x562fd84856f0 2024-04-21T23:09:58.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.212+0000 7fbd0c944640 1 --2- 172.21.15.44:0/3413590971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd8496c60 0x562fd84946c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:09:58.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.212+0000 7fbd077fe640 1 --2- 172.21.15.44:0/3413590971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd84856f0 0x562fd8494180 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc0ccdd0 tx=0x7fbcfc09c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4872dbdfb7bb938 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:58.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.212+0000 7fbce77fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:09:58.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.212+0000 7fbce77fe640 1 -- 172.21.15.44:0/3413590971 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc09fd70 con 0x562fd84856f0 2024-04-21T23:09:58.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.212+0000 7fbd10f8a740 1 -- 172.21.15.44:0/3413590971 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd00032c90 con 0x562fd84856f0 2024-04-21T23:09:58.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.212+0000 7fbce77fe640 1 -- 172.21.15.44:0/3413590971 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fbcfc0a41a0 con 0x562fd84856f0 2024-04-21T23:09:58.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.212+0000 7fbce77fe640 1 -- 172.21.15.44:0/3413590971 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbcfc0a4630 con 0x562fd84856f0 2024-04-21T23:09:58.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.212+0000 7fbd10f8a740 10 client.15816 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:58.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.212+0000 7fbd10f8a740 1 -- 172.21.15.44:0/3413590971 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562fd80bd0f0 con 0x562fd84856f0 2024-04-21T23:09:58.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.212+0000 7fbce77fe640 1 -- 172.21.15.44:0/3413590971 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd00032c90 con 0x562fd84856f0 2024-04-21T23:09:58.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.213+0000 7fbce77fe640 1 -- 172.21.15.44:0/3413590971 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fbcfc0ba120 con 0x562fd84856f0 2024-04-21T23:09:58.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.213+0000 7fbd10f8a740 10 client.15816 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:58.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.213+0000 7fbd10f8a740 10 client.15816 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:58.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.213+0000 7fbd10f8a740 20 client.15816 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:58.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.213+0000 7fbd10f8a740 1 --2- 172.21.15.44:0/3413590971 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562fd84a74d0 0x562fd84a98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.213+0000 7fbd10f8a740 4 client.15816 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:09:58.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.213+0000 7fbd10f8a740 1 -- 172.21.15.44:0/3413590971 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562fd83f6aa0 con 0x562fd84a74d0 2024-04-21T23:09:58.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.213+0000 7fbd07fff640 1 --2- 172.21.15.44:0/3413590971 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562fd84a74d0 0x562fd84a98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.213+0000 7fbce77fe640 10 client.15816 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:58.217 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.213+0000 7fbd07fff640 1 --2- 172.21.15.44:0/3413590971 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562fd84a74d0 0x562fd84a98f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8edeec915f8134a7 server_cookie=72f8fd402592b2ad in_seq=0 out_seq=0 2024-04-21T23:09:58.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.214+0000 7fbce77fe640 1 -- 172.21.15.44:0/3413590971 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562fd83f6aa0 con 0x562fd84a74d0 2024-04-21T23:09:58.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.214+0000 7fbce77fe640 10 client.15816 handle_command_reply: tid=0 2024-04-21T23:09:58.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.214+0000 7fbd10f8a740 1 client.15816 shutdown 2024-04-21T23:09:58.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.214+0000 7fbd10f8a740 1 -- 172.21.15.44:0/3413590971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd84856f0 msgr2=0x562fd8494180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.214+0000 7fbd10f8a740 1 --2- 172.21.15.44:0/3413590971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd84856f0 0x562fd8494180 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc0ccdd0 tx=0x7fbcfc09c0c0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.214+0000 7fbd10f8a740 1 -- 172.21.15.44:0/3413590971 shutdown_connections 2024-04-21T23:09:58.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.214+0000 7fbd10f8a740 1 --2- 172.21.15.44:0/3413590971 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562fd84a74d0 0x562fd84a98f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.214+0000 7fbd10f8a740 1 --2- 172.21.15.44:0/3413590971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd8496c60 0x562fd84946c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.214+0000 7fbd10f8a740 1 --2- 172.21.15.44:0/3413590971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd84856f0 0x562fd8494180 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.214+0000 7fbd10f8a740 1 --2- 172.21.15.44:0/3413590971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 0x562fd8493c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.214+0000 7fbd10f8a740 1 -- 172.21.15.44:0/3413590971 >> 172.21.15.44:0/3413590971 conn(0x562fd847f5a0 msgr2=0x562fd84813b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:58.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.214+0000 7fbce77fe640 0 client.15816 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:58.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.215+0000 7fbd10f8a740 1 -- 172.21.15.44:0/3413590971 shutdown_connections 2024-04-21T23:09:58.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.215+0000 7fbd10f8a740 1 -- 172.21.15.44:0/3413590971 wait complete. 2024-04-21T23:09:58.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.215+0000 7fbd10f8a740 20 client.15816 trim_cache size 0 max 16384 2024-04-21T23:09:58.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.232+0000 7fbd10f8a740 1 Processor -- start 2024-04-21T23:09:58.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.232+0000 7fbd10f8a740 1 -- start start 2024-04-21T23:09:58.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.233+0000 7fbd10f8a740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 0x562fd8496370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.233+0000 7fbd10f8a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd84856f0 0x562fd84a2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.233+0000 7fbd10f8a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd8496c60 0x562fd8494500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.233+0000 7fbd10f8a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf400de80 con 0x562fd84856f0 2024-04-21T23:09:58.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.233+0000 7fbd10f8a740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf4010700 con 0x562fd8484d40 2024-04-21T23:09:58.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.233+0000 7fbd10f8a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf400e4f0 con 0x562fd8496c60 2024-04-21T23:09:58.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.233+0000 7fbd077fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd84856f0 0x562fd84a2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.233+0000 7fbd077fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd84856f0 0x562fd84a2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53318/0 (socket says 172.21.15.44:53318) 2024-04-21T23:09:58.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.233+0000 7fbd077fe640 1 -- 172.21.15.44:0/4189481600 learned_addr learned my addr 172.21.15.44:0/4189481600 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:58.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.233+0000 7fbd07fff640 1 --2- 172.21.15.44:0/4189481600 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 0x562fd8496370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.233+0000 7fbd0c944640 1 --2- 172.21.15.44:0/4189481600 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd8496c60 0x562fd8494500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.233+0000 7fbd077fe640 1 -- 172.21.15.44:0/4189481600 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd8496c60 msgr2=0x562fd8494500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:58.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.233+0000 7fbd077fe640 1 --2- 172.21.15.44:0/4189481600 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd8496c60 0x562fd8494500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.233+0000 7fbd077fe640 1 -- 172.21.15.44:0/4189481600 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 msgr2=0x562fd8496370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:58.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.233+0000 7fbd077fe640 1 --2- 172.21.15.44:0/4189481600 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 0x562fd8496370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.233+0000 7fbd077fe640 1 -- 172.21.15.44:0/4189481600 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fd848ea30 con 0x562fd84856f0 2024-04-21T23:09:58.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.233+0000 7fbd077fe640 1 --2- 172.21.15.44:0/4189481600 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd84856f0 0x562fd84a2920 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc09b950 tx=0x7fbcfc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1004c5e892a780ca server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:58.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.234+0000 7fbcedffb640 1 -- 172.21.15.44:0/4189481600 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc09fd70 con 0x562fd84856f0 2024-04-21T23:09:58.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.234+0000 7fbcedffb640 1 -- 172.21.15.44:0/4189481600 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcfc0ccfa0 con 0x562fd84856f0 2024-04-21T23:09:58.234 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.234+0000 7fbcedffb640 1 -- 172.21.15.44:0/4189481600 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc005310 con 0x562fd84856f0 2024-04-21T23:09:58.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.235+0000 7fbd10f8a740 1 -- 172.21.15.44:0/4189481600 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd84856f0 msgr2=0x562fd84a2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.235+0000 7fbd10f8a740 1 --2- 172.21.15.44:0/4189481600 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd84856f0 0x562fd84a2920 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc09b950 tx=0x7fbcfc000bc0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.235+0000 7fbd10f8a740 1 -- 172.21.15.44:0/4189481600 shutdown_connections 2024-04-21T23:09:58.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.235+0000 7fbd10f8a740 1 --2- 172.21.15.44:0/4189481600 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd8496c60 0x562fd8494500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.235+0000 7fbd10f8a740 1 --2- 172.21.15.44:0/4189481600 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd84856f0 0x562fd84a2920 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.235+0000 7fbd10f8a740 1 --2- 172.21.15.44:0/4189481600 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 0x562fd8496370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.235+0000 7fbd10f8a740 1 -- 172.21.15.44:0/4189481600 >> 172.21.15.44:0/4189481600 conn(0x562fd847f5a0 msgr2=0x562fd848ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:58.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.235+0000 7fbd10f8a740 1 -- 172.21.15.44:0/4189481600 shutdown_connections 2024-04-21T23:09:58.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.235+0000 7fbd10f8a740 1 -- 172.21.15.44:0/4189481600 wait complete. 2024-04-21T23:09:58.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.235+0000 7fbd10f8a740 1 -- start start 2024-04-21T23:09:58.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.235+0000 7fbd10f8a740 1 Processor -- start 2024-04-21T23:09:58.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.236+0000 7fbd10f8a740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 0x562fd849edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.236+0000 7fbd10f8a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd8496c60 0x562fd849f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.236+0000 7fbd10f8a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd849a360 0x562fd849a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.236+0000 7fbd10f8a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf400ebc0 con 0x562fd8496c60 2024-04-21T23:09:58.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.236+0000 7fbd10f8a740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf4010030 con 0x562fd8484d40 2024-04-21T23:09:58.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.236+0000 7fbd10f8a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf400f290 con 0x562fd849a360 2024-04-21T23:09:58.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.236+0000 7fbd10f8a740 10 client.0 _ll_register_callbacks cb 0x7fbcfc00cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:58.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.236+0000 7fbd077fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd8496c60 0x562fd849f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.236+0000 7fbd077fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd8496c60 0x562fd849f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53326/0 (socket says 172.21.15.44:53326) 2024-04-21T23:09:58.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.236+0000 7fbd077fe640 1 -- 172.21.15.44:0/329253789 learned_addr learned my addr 172.21.15.44:0/329253789 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:58.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.236+0000 7fbd07fff640 1 --2- 172.21.15.44:0/329253789 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 0x562fd849edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.236+0000 7fbd0c944640 1 --2- 172.21.15.44:0/329253789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd849a360 0x562fd849a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.236+0000 7fbd077fe640 1 -- 172.21.15.44:0/329253789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd849a360 msgr2=0x562fd849a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:58.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.236+0000 7fbd077fe640 1 --2- 172.21.15.44:0/329253789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd849a360 0x562fd849a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.236+0000 7fbd077fe640 1 -- 172.21.15.44:0/329253789 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 msgr2=0x562fd849edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:58.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.236+0000 7fbd077fe640 1 --2- 172.21.15.44:0/329253789 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 0x562fd849edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.236+0000 7fbd077fe640 1 -- 172.21.15.44:0/329253789 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fd849afe0 con 0x562fd8496c60 2024-04-21T23:09:58.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.236+0000 7fbd07fff640 1 --2- 172.21.15.44:0/329253789 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 0x562fd849edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:09:58.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.236+0000 7fbd0c944640 1 --2- 172.21.15.44:0/329253789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd849a360 0x562fd849a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:09:58.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.237+0000 7fbd077fe640 1 --2- 172.21.15.44:0/329253789 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd8496c60 0x562fd849f330 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc0a4100 tx=0x7fbcfc0cd330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b73aeee81f4c9033 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:58.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.237+0000 7fbce77fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:09:58.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.237+0000 7fbce77fe640 1 -- 172.21.15.44:0/329253789 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc09fd70 con 0x562fd8496c60 2024-04-21T23:09:58.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.237+0000 7fbce77fe640 1 -- 172.21.15.44:0/329253789 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fbcfc0a43a0 con 0x562fd8496c60 2024-04-21T23:09:58.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.237+0000 7fbd10f8a740 1 -- 172.21.15.44:0/329253789 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcfc0ccfa0 con 0x562fd8496c60 2024-04-21T23:09:58.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.237+0000 7fbce77fe640 1 -- 172.21.15.44:0/329253789 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbcfc0b4bc0 con 0x562fd8496c60 2024-04-21T23:09:58.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.237+0000 7fbd10f8a740 10 client.15822 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:58.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.237+0000 7fbd10f8a740 1 -- 172.21.15.44:0/329253789 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562fd850da40 con 0x562fd8496c60 2024-04-21T23:09:58.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.238+0000 7fbce77fe640 1 -- 172.21.15.44:0/329253789 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcfc0ccfa0 con 0x562fd8496c60 2024-04-21T23:09:58.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.238+0000 7fbce77fe640 1 -- 172.21.15.44:0/329253789 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fbcfc0861c0 con 0x562fd8496c60 2024-04-21T23:09:58.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.238+0000 7fbd10f8a740 10 client.15822 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:58.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.238+0000 7fbd10f8a740 10 client.15822 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:58.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.238+0000 7fbd10f8a740 20 client.15822 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:58.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.238+0000 7fbd10f8a740 1 --2- 172.21.15.44:0/329253789 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562fd850dfc0 0x562fd85103e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.238+0000 7fbd10f8a740 4 client.15822 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:09:58.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.238+0000 7fbd10f8a740 1 -- 172.21.15.44:0/329253789 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562fd83f0df0 con 0x562fd850dfc0 2024-04-21T23:09:58.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.238+0000 7fbd07fff640 1 --2- 172.21.15.44:0/329253789 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562fd850dfc0 0x562fd85103e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.239+0000 7fbce77fe640 10 client.15822 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:58.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.239+0000 7fbd07fff640 1 --2- 172.21.15.44:0/329253789 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562fd850dfc0 0x562fd85103e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2437ce2d06daac9 server_cookie=50a09489fa6ee731 in_seq=0 out_seq=0 2024-04-21T23:09:58.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.239+0000 7fbce77fe640 1 -- 172.21.15.44:0/329253789 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562fd83f0df0 con 0x562fd850dfc0 2024-04-21T23:09:58.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.239+0000 7fbce77fe640 10 client.15822 handle_command_reply: tid=0 2024-04-21T23:09:58.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.239+0000 7fbd10f8a740 1 client.15822 shutdown 2024-04-21T23:09:58.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.239+0000 7fbd10f8a740 1 -- 172.21.15.44:0/329253789 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd8496c60 msgr2=0x562fd849f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.239+0000 7fbd10f8a740 1 --2- 172.21.15.44:0/329253789 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd8496c60 0x562fd849f330 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc0a4100 tx=0x7fbcfc0cd330 comp rx=0 tx=0).stop 2024-04-21T23:09:58.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.239+0000 7fbd10f8a740 1 -- 172.21.15.44:0/329253789 shutdown_connections 2024-04-21T23:09:58.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.239+0000 7fbd10f8a740 1 --2- 172.21.15.44:0/329253789 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562fd850dfc0 0x562fd85103e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.239+0000 7fbce77fe640 0 client.15822 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:58.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.239+0000 7fbd10f8a740 1 --2- 172.21.15.44:0/329253789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562fd849a360 0x562fd849a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.239+0000 7fbd10f8a740 1 --2- 172.21.15.44:0/329253789 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562fd8496c60 0x562fd849f330 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.239+0000 7fbd10f8a740 1 --2- 172.21.15.44:0/329253789 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562fd8484d40 0x562fd849edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.239+0000 7fbd10f8a740 1 -- 172.21.15.44:0/329253789 >> 172.21.15.44:0/329253789 conn(0x562fd847f5a0 msgr2=0x562fd8480940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:58.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.240+0000 7fbd10f8a740 1 -- 172.21.15.44:0/329253789 shutdown_connections 2024-04-21T23:09:58.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.240+0000 7fbd10f8a740 1 -- 172.21.15.44:0/329253789 wait complete. 2024-04-21T23:09:58.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.240+0000 7fbd10f8a740 20 client.15822 trim_cache size 0 max 16384 2024-04-21T23:09:58.246 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:09:58.246 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-21T23:09:58.246 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-21T23:09:58.246 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:09:58.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.240+0000 7fbd0e3a7640 1 -- 172.21.15.44:0/516506957 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbcf00a2d30 msgr2=0x7fbcf00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.240+0000 7fbd0e3a7640 1 --2- 172.21.15.44:0/516506957 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbcf00a2d30 0x7fbcf00a51f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbd00009bf0 tx=0x7fbd0002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.240+0000 7fbd0e3a7640 1 -- 172.21.15.44:0/516506957 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd08100fd0 msgr2=0x7fbd081984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.240+0000 7fbd0e3a7640 1 --2- 172.21.15.44:0/516506957 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd08100fd0 0x7fbd081984b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc00c920 tx=0x7fbcfc00cde0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.240+0000 7fbd0e3a7640 1 -- 172.21.15.44:0/516506957 shutdown_connections 2024-04-21T23:09:58.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.240+0000 7fbd0e3a7640 1 --2- 172.21.15.44:0/516506957 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbcf00a2d30 0x7fbcf00a51f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.240+0000 7fbd0e3a7640 1 --2- 172.21.15.44:0/516506957 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd081066b0 0x7fbd0819c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.240+0000 7fbd0e3a7640 1 --2- 172.21.15.44:0/516506957 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd08100fd0 0x7fbd081984b0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.240+0000 7fbd0e3a7640 1 --2- 172.21.15.44:0/516506957 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd08100610 0x7fbd08197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.240+0000 7fbd0e3a7640 1 -- 172.21.15.44:0/516506957 >> 172.21.15.44:0/516506957 conn(0x7fbd080777f0 msgr2=0x7fbd080ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:58.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.240+0000 7fbd0e3a7640 1 -- 172.21.15.44:0/516506957 shutdown_connections 2024-04-21T23:09:58.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.240+0000 7fbd0e3a7640 1 -- 172.21.15.44:0/516506957 wait complete. 2024-04-21T23:09:58.251 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:09:58.251 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:09:58.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.360+0000 7f750ad65640 1 -- 172.21.15.44:0/1406770455 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7504068a40 msgr2=0x7f7504068ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.360+0000 7f750ad65640 1 --2- 172.21.15.44:0/1406770455 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7504068a40 0x7f7504068ea0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009900 tx=0x7f74f002bc30 comp rx=0 tx=0).stop 2024-04-21T23:09:58.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.360+0000 7f750ad65640 1 -- 172.21.15.44:0/1406770455 shutdown_connections 2024-04-21T23:09:58.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.360+0000 7f750ad65640 1 --2- 172.21.15.44:0/1406770455 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75040693e0 0x7f750410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.360+0000 7f750ad65640 1 --2- 172.21.15.44:0/1406770455 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7504068a40 0x7f7504068ea0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.360+0000 7f750ad65640 1 --2- 172.21.15.44:0/1406770455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7504107800 0x7f7504107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.360+0000 7f750ad65640 1 -- 172.21.15.44:0/1406770455 >> 172.21.15.44:0/1406770455 conn(0x7f7504077430 msgr2=0x7f7504077840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:58.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.360+0000 7f750ad65640 1 -- 172.21.15.44:0/1406770455 shutdown_connections 2024-04-21T23:09:58.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.360+0000 7f750ad65640 1 -- 172.21.15.44:0/1406770455 wait complete. 2024-04-21T23:09:58.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.361+0000 7f750ad65640 1 Processor -- start 2024-04-21T23:09:58.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.361+0000 7f750ad65640 1 -- start start 2024-04-21T23:09:58.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.361+0000 7f750ad65640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7504068a40 0x7f7504197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.361+0000 7f750ad65640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75040693e0 0x7f75041984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.361+0000 7f750ad65640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7504107800 0x7f750419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.361+0000 7f750ad65640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7504111b20 con 0x7f75040693e0 2024-04-21T23:09:58.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.361+0000 7f750ad65640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f75041119a0 con 0x7f7504068a40 2024-04-21T23:09:58.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.361+0000 7f750ad65640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7504111ca0 con 0x7f7504107800 2024-04-21T23:09:58.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.361+0000 7f7503fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75040693e0 0x7f75041984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.361+0000 7f7503fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75040693e0 0x7f75041984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53348/0 (socket says 172.21.15.44:53348) 2024-04-21T23:09:58.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.361+0000 7f7503fff640 1 -- 172.21.15.44:0/796923793 learned_addr learned my addr 172.21.15.44:0/796923793 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:58.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.361+0000 7f7508b01640 1 --2- 172.21.15.44:0/796923793 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7504068a40 0x7f7504197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.361+0000 7f7509302640 1 --2- 172.21.15.44:0/796923793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7504107800 0x7f750419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.362+0000 7f7508b01640 1 -- 172.21.15.44:0/796923793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7504107800 msgr2=0x7f750419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.362+0000 7f7508b01640 1 --2- 172.21.15.44:0/796923793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7504107800 0x7f750419c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.362+0000 7f7508b01640 1 -- 172.21.15.44:0/796923793 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75040693e0 msgr2=0x7f75041984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.362+0000 7f7508b01640 1 --2- 172.21.15.44:0/796923793 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75040693e0 0x7f75041984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.362+0000 7f7508b01640 1 -- 172.21.15.44:0/796923793 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750419cf50 con 0x7f7504068a40 2024-04-21T23:09:58.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.362+0000 7f7503fff640 1 --2- 172.21.15.44:0/796923793 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75040693e0 0x7f75041984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:09:58.364 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.362+0000 7f7508b01640 1 --2- 172.21.15.44:0/796923793 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7504068a40 0x7f7504197f60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f74f800e9f0 tx=0x7f74f800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:58.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.362+0000 7f7501ffb640 1 -- 172.21.15.44:0/796923793 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f800ccc0 con 0x7f7504068a40 2024-04-21T23:09:58.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.362+0000 7f7501ffb640 1 -- 172.21.15.44:0/796923793 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f8005020 con 0x7f7504068a40 2024-04-21T23:09:58.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.362+0000 7f750ad65640 1 -- 172.21.15.44:0/796923793 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f750419d240 con 0x7f7504068a40 2024-04-21T23:09:58.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.362+0000 7f7501ffb640 1 -- 172.21.15.44:0/796923793 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f80049e0 con 0x7f7504068a40 2024-04-21T23:09:58.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.362+0000 7f750ad65640 1 -- 172.21.15.44:0/796923793 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75041a4ad0 con 0x7f7504068a40 2024-04-21T23:09:58.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.364+0000 7f750ad65640 1 -- 172.21.15.44:0/796923793 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75040685f0 con 0x7f7504068a40 2024-04-21T23:09:58.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.365+0000 7f7501ffb640 1 -- 172.21.15.44:0/796923793 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f74f8005560 con 0x7f7504068a40 2024-04-21T23:09:58.365 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.366+0000 7f7501ffb640 1 --2- 172.21.15.44:0/796923793 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f74f40a2f20 0x7f74f40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.366+0000 7f7501ffb640 1 -- 172.21.15.44:0/796923793 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f74f80cfa20 con 0x7f7504068a40 2024-04-21T23:09:58.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.367+0000 7f7503fff640 1 --2- 172.21.15.44:0/796923793 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f74f40a2f20 0x7f74f40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.367+0000 7f7503fff640 1 --2- 172.21.15.44:0/796923793 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f74f40a2f20 0x7f74f40a53e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7504106d90 tx=0x7f74f00058d0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:58.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.371+0000 7f7501ffb640 1 -- 172.21.15.44:0/796923793 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f74f80051d0 con 0x7f7504068a40 2024-04-21T23:09:58.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:58 smithi044 ceph-mon[27934]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-21T23:09:58.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:58 smithi055 ceph-mon[31117]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-21T23:09:58.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:58 smithi069 ceph-mon[31816]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-21T23:09:58.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.596+0000 7f750ad65640 1 -- 172.21.15.44:0/796923793 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7504109c30 con 0x7f7504068a40 2024-04-21T23:09:58.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.597+0000 7f7501ffb640 1 -- 172.21.15.44:0/796923793 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11791 (secure 0 0 0) 0x7f74f809bff0 con 0x7f7504068a40 2024-04-21T23:09:58.598 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:09:58.598 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:06:50.685663+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3743898853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3743898853},{"type":"v1","addr":"172.21.15.69:6839","nonce":3743898853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:09:58.598 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2024-04-21T23:09:58.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.600+0000 7f750ad65640 1 -- 172.21.15.44:0/796923793 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f74f40a2f20 msgr2=0x7f74f40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.600+0000 7f750ad65640 1 --2- 172.21.15.44:0/796923793 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f74f40a2f20 0x7f74f40a53e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7504106d90 tx=0x7f74f00058d0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.600+0000 7f750ad65640 1 -- 172.21.15.44:0/796923793 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7504068a40 msgr2=0x7f7504197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.600+0000 7f750ad65640 1 --2- 172.21.15.44:0/796923793 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7504068a40 0x7f7504197f60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f74f800e9f0 tx=0x7f74f800eeb0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.600+0000 7f750ad65640 1 -- 172.21.15.44:0/796923793 shutdown_connections 2024-04-21T23:09:58.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.600+0000 7f750ad65640 1 --2- 172.21.15.44:0/796923793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7504107800 0x7f750419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.600+0000 7f750ad65640 1 --2- 172.21.15.44:0/796923793 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f74f40a2f20 0x7f74f40a53e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.600+0000 7f750ad65640 1 --2- 172.21.15.44:0/796923793 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f75040693e0 0x7f75041984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.600+0000 7f750ad65640 1 --2- 172.21.15.44:0/796923793 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7504068a40 0x7f7504197f60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.600+0000 7f750ad65640 1 -- 172.21.15.44:0/796923793 >> 172.21.15.44:0/796923793 conn(0x7f7504077430 msgr2=0x7f75040ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:58.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.600+0000 7f750ad65640 1 -- 172.21.15.44:0/796923793 shutdown_connections 2024-04-21T23:09:58.602 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.601+0000 7f750ad65640 1 -- 172.21.15.44:0/796923793 wait complete. 2024-04-21T23:09:58.615 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:09:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.723+0000 7f62288a7640 1 -- 172.21.15.44:0/505312363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62241029b0 msgr2=0x7f6224102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.723+0000 7f62288a7640 1 --2- 172.21.15.44:0/505312363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62241029b0 0x7f6224102d90 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f6214009900 tx=0x7f621402bcd0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.723+0000 7f62288a7640 1 -- 172.21.15.44:0/505312363 shutdown_connections 2024-04-21T23:09:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.723+0000 7f62288a7640 1 --2- 172.21.15.44:0/505312363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f622410bcf0 0x7f622410e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.723+0000 7f62288a7640 1 --2- 172.21.15.44:0/505312363 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62241032d0 0x7f622410b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.723+0000 7f62288a7640 1 --2- 172.21.15.44:0/505312363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f62241029b0 0x7f6224102d90 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.723+0000 7f62288a7640 1 -- 172.21.15.44:0/505312363 >> 172.21.15.44:0/505312363 conn(0x7f62240777f0 msgr2=0x7f6224077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:58.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.723+0000 7f62288a7640 1 -- 172.21.15.44:0/505312363 shutdown_connections 2024-04-21T23:09:58.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.723+0000 7f62288a7640 1 -- 172.21.15.44:0/505312363 wait complete. 2024-04-21T23:09:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.724+0000 7f62288a7640 1 Processor -- start 2024-04-21T23:09:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.724+0000 7f62288a7640 1 -- start start 2024-04-21T23:09:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.724+0000 7f62288a7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62241032d0 0x7f6224100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.724+0000 7f62288a7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f622410bcf0 0x7f6224100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.724+0000 7f62288a7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6224101080 0x7f62241a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.724+0000 7f62288a7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6224111af0 con 0x7f6224101080 2024-04-21T23:09:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.724+0000 7f62288a7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6224111970 con 0x7f62241032d0 2024-04-21T23:09:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.724+0000 7f62288a7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6224111c70 con 0x7f622410bcf0 2024-04-21T23:09:58.726 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.724+0000 7f6222d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6224101080 0x7f62241a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.724+0000 7f6222d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6224101080 0x7f62241a63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53364/0 (socket says 172.21.15.44:53364) 2024-04-21T23:09:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.724+0000 7f6222d9d640 1 -- 172.21.15.44:0/210761058 learned_addr learned my addr 172.21.15.44:0/210761058 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.725+0000 7f6221d9b640 1 --2- 172.21.15.44:0/210761058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f622410bcf0 0x7f6224100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.725+0000 7f622259c640 1 --2- 172.21.15.44:0/210761058 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62241032d0 0x7f6224100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.725+0000 7f6222d9d640 1 -- 172.21.15.44:0/210761058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f622410bcf0 msgr2=0x7f6224100b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.725+0000 7f6222d9d640 1 --2- 172.21.15.44:0/210761058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f622410bcf0 0x7f6224100b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.725+0000 7f6222d9d640 1 -- 172.21.15.44:0/210761058 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62241032d0 msgr2=0x7f6224100600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.725+0000 7f6222d9d640 1 --2- 172.21.15.44:0/210761058 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62241032d0 0x7f6224100600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.725+0000 7f6222d9d640 1 -- 172.21.15.44:0/210761058 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62241a6ac0 con 0x7f6224101080 2024-04-21T23:09:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.725+0000 7f622259c640 1 --2- 172.21.15.44:0/210761058 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62241032d0 0x7f6224100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:09:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.725+0000 7f6222d9d640 1 --2- 172.21.15.44:0/210761058 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6224101080 0x7f62241a63c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f621c00e960 tx=0x7f621c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.725+0000 7f62137fe640 1 -- 172.21.15.44:0/210761058 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c004d20 con 0x7f6224101080 2024-04-21T23:09:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.725+0000 7f62137fe640 1 -- 172.21.15.44:0/210761058 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f621c005020 con 0x7f6224101080 2024-04-21T23:09:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.725+0000 7f62288a7640 1 -- 172.21.15.44:0/210761058 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62241a6db0 con 0x7f6224101080 2024-04-21T23:09:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.725+0000 7f62137fe640 1 -- 172.21.15.44:0/210761058 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c0116d0 con 0x7f6224101080 2024-04-21T23:09:58.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.726+0000 7f62288a7640 1 -- 172.21.15.44:0/210761058 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62241a7270 con 0x7f6224101080 2024-04-21T23:09:58.728 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.729+0000 7f62137fe640 1 -- 172.21.15.44:0/210761058 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f621c005560 con 0x7f6224101080 2024-04-21T23:09:58.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.730+0000 7f62137fe640 1 --2- 172.21.15.44:0/210761058 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f620c0a2f50 0x7f620c0a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.730+0000 7f622b56c740 1 Processor -- start 2024-04-21T23:09:58.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.730+0000 7f62137fe640 1 -- 172.21.15.44:0/210761058 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f621c0cf810 con 0x7f6224101080 2024-04-21T23:09:58.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.730+0000 7f622b56c740 1 -- start start 2024-04-21T23:09:58.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.730+0000 7f622259c640 1 --2- 172.21.15.44:0/210761058 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f620c0a2f50 0x7f620c0a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.730+0000 7f622b56c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bc2d50 0x562933bc3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.730+0000 7f622b56c740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562933bc3670 0x562933bd47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.732 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.730+0000 7f622b56c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bd4d30 0x562933bd7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.730+0000 7f622b56c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562933b27530 con 0x562933bc2d50 2024-04-21T23:09:58.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.730+0000 7f622b56c740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x562933acf830 con 0x562933bc3670 2024-04-21T23:09:58.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.730+0000 7f622b56c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562933b234f0 con 0x562933bd4d30 2024-04-21T23:09:58.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.731+0000 7f6221d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562933bc3670 0x562933bd47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.731+0000 7f6221d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562933bc3670 0x562933bd47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:60266/0 (socket says 172.21.15.44:60266) 2024-04-21T23:09:58.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.731+0000 7f6221d9b640 1 -- 172.21.15.44:0/735017223 learned_addr learned my addr 172.21.15.44:0/735017223 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:58.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.731+0000 7f6222d9d640 1 --2- 172.21.15.44:0/735017223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bd4d30 0x562933bd7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.731+0000 7f622259c640 1 --2- 172.21.15.44:0/210761058 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f620c0a2f50 0x7f620c0a5410 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6214009230 tx=0x7f6214005a20 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:58.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.731+0000 7f6221d9b640 1 -- 172.21.15.44:0/735017223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bd4d30 msgr2=0x562933bd7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:58.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.731+0000 7f6221d9b640 1 --2- 172.21.15.44:0/735017223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bd4d30 0x562933bd7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.731+0000 7f6221d9b640 1 -- 172.21.15.44:0/735017223 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bc2d50 msgr2=0x562933bc3130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T23:09:58.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.731+0000 7f622259c640 1 --2- 172.21.15.44:0/735017223 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bc2d50 0x562933bc3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.731+0000 7f6221d9b640 1 --2- 172.21.15.44:0/735017223 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bc2d50 0x562933bc3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.733 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.731+0000 7f6221d9b640 1 -- 172.21.15.44:0/735017223 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562933af32c0 con 0x562933bc3670 2024-04-21T23:09:58.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.731+0000 7f622259c640 1 --2- 172.21.15.44:0/735017223 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bc2d50 0x562933bc3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:09:58.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.731+0000 7f6221d9b640 1 --2- 172.21.15.44:0/735017223 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562933bc3670 0x562933bd47f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6218009c90 tx=0x7f6218011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c41d3d7da7029ab server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:58.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.731+0000 7f6210ff9640 1 -- 172.21.15.44:0/735017223 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621801e070 con 0x562933bc3670 2024-04-21T23:09:58.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.732+0000 7f6210ff9640 1 -- 172.21.15.44:0/735017223 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6218019b10 con 0x562933bc3670 2024-04-21T23:09:58.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.732+0000 7f6210ff9640 1 -- 172.21.15.44:0/735017223 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621800ea70 con 0x562933bc3670 2024-04-21T23:09:58.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.732+0000 7f622b56c740 1 -- 172.21.15.44:0/735017223 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562933bc3670 msgr2=0x562933bd47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.732+0000 7f622b56c740 1 --2- 172.21.15.44:0/735017223 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562933bc3670 0x562933bd47f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6218009c90 tx=0x7f6218011810 comp rx=0 tx=0).stop 2024-04-21T23:09:58.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.732+0000 7f622b56c740 1 -- 172.21.15.44:0/735017223 shutdown_connections 2024-04-21T23:09:58.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.732+0000 7f622b56c740 1 --2- 172.21.15.44:0/735017223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bd4d30 0x562933bd7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.732+0000 7f622b56c740 1 --2- 172.21.15.44:0/735017223 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562933bc3670 0x562933bd47f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.732+0000 7f622b56c740 1 --2- 172.21.15.44:0/735017223 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bc2d50 0x562933bc3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.732+0000 7f622b56c740 1 -- 172.21.15.44:0/735017223 >> 172.21.15.44:0/735017223 conn(0x562933bbd750 msgr2=0x562933bbdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:58.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.732+0000 7f622b56c740 1 -- 172.21.15.44:0/735017223 shutdown_connections 2024-04-21T23:09:58.734 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.732+0000 7f622b56c740 1 -- 172.21.15.44:0/735017223 wait complete. 2024-04-21T23:09:58.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.733+0000 7f622b56c740 1 -- start start 2024-04-21T23:09:58.735 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.733+0000 7f622b56c740 1 Processor -- start 2024-04-21T23:09:58.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.733+0000 7f622b56c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bc2d50 0x562933bd45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.733+0000 7f622b56c740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562933bc3670 0x562933bd1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.733+0000 7f622b56c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bd4d30 0x562933bd1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.733+0000 7f622b56c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562933bd7af0 con 0x562933bc2d50 2024-04-21T23:09:58.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.733+0000 7f622b56c740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x562933ae2f10 con 0x562933bc3670 2024-04-21T23:09:58.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.733+0000 7f622b56c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562933bd7c70 con 0x562933bd4d30 2024-04-21T23:09:58.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.733+0000 7f622b56c740 10 client.0 _ll_register_callbacks cb 0x56293383abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:58.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.733+0000 7f622259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bc2d50 0x562933bd45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.738 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.733+0000 7f6222d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bd4d30 0x562933bd1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.733+0000 7f6221d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562933bc3670 0x562933bd1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.733+0000 7f6222d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bd4d30 0x562933bd1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:54184/0 (socket says 172.21.15.44:54184) 2024-04-21T23:09:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.733+0000 7f6222d9d640 1 -- 172.21.15.44:0/3824548984 learned_addr learned my addr 172.21.15.44:0/3824548984 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.734+0000 7f6222d9d640 1 -- 172.21.15.44:0/3824548984 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562933bc3670 msgr2=0x562933bd1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.734+0000 7f6222d9d640 1 --2- 172.21.15.44:0/3824548984 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562933bc3670 0x562933bd1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.734+0000 7f6222d9d640 1 -- 172.21.15.44:0/3824548984 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bc2d50 msgr2=0x562933bd45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.734+0000 7f6222d9d640 1 --2- 172.21.15.44:0/3824548984 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bc2d50 0x562933bd45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.734+0000 7f6222d9d640 1 -- 172.21.15.44:0/3824548984 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562933ab4c80 con 0x562933bd4d30 2024-04-21T23:09:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.734+0000 7f622259c640 1 --2- 172.21.15.44:0/3824548984 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bc2d50 0x562933bd45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:09:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.734+0000 7f6222d9d640 1 --2- 172.21.15.44:0/3824548984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bd4d30 0x562933bd1ea0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f621c09b3e0 tx=0x7f621c09b310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3a52a15d24ae65b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.734+0000 7f6209ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:09:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.734+0000 7f6209ffb640 1 -- 172.21.15.44:0/3824548984 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c0b3020 con 0x562933bd4d30 2024-04-21T23:09:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.734+0000 7f622b56c740 1 -- 172.21.15.44:0/3824548984 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6218019b10 con 0x562933bd4d30 2024-04-21T23:09:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.734+0000 7f6209ffb640 1 -- 172.21.15.44:0/3824548984 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f621c0cf5a0 con 0x562933bd4d30 2024-04-21T23:09:58.739 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.734+0000 7f6209ffb640 1 -- 172.21.15.44:0/3824548984 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f621c01f070 con 0x562933bd4d30 2024-04-21T23:09:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.735+0000 7f622b56c740 10 client.25486 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.735+0000 7f622b56c740 1 -- 172.21.15.44:0/3824548984 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56293383b3f0 con 0x562933bd4d30 2024-04-21T23:09:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.735+0000 7f6209ffb640 1 -- 172.21.15.44:0/3824548984 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6218019b10 con 0x562933bd4d30 2024-04-21T23:09:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.735+0000 7f6209ffb640 1 -- 172.21.15.44:0/3824548984 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f621c022060 con 0x562933bd4d30 2024-04-21T23:09:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.735+0000 7f622b56c740 10 client.25486 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.735+0000 7f622b56c740 10 client.25486 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.735+0000 7f622b56c740 20 client.25486 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.735+0000 7f622b56c740 1 --2- 172.21.15.44:0/3824548984 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562933be4b50 0x562933be6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.735+0000 7f622b56c740 4 client.25486 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:09:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.735+0000 7f622b56c740 1 -- 172.21.15.44:0/3824548984 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562933b34a70 con 0x562933be4b50 2024-04-21T23:09:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.735+0000 7f622259c640 1 --2- 172.21.15.44:0/3824548984 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562933be4b50 0x562933be6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.736+0000 7f622259c640 1 --2- 172.21.15.44:0/3824548984 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562933be4b50 0x562933be6fd0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90309c6a8e1b1c6 server_cookie=28ec1c8926db07e0 in_seq=0 out_seq=0 2024-04-21T23:09:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.736+0000 7f6209ffb640 10 client.25486 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.737+0000 7f6209ffb640 1 -- 172.21.15.44:0/3824548984 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562933b34a70 con 0x562933be4b50 2024-04-21T23:09:58.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.737+0000 7f6209ffb640 10 client.25486 handle_command_reply: tid=0 2024-04-21T23:09:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.737+0000 7f622b56c740 1 client.25486 shutdown 2024-04-21T23:09:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.737+0000 7f622b56c740 1 -- 172.21.15.44:0/3824548984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bd4d30 msgr2=0x562933bd1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.737+0000 7f622b56c740 1 --2- 172.21.15.44:0/3824548984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bd4d30 0x562933bd1ea0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f621c09b3e0 tx=0x7f621c09b310 comp rx=0 tx=0).stop 2024-04-21T23:09:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.737+0000 7f622b56c740 1 -- 172.21.15.44:0/3824548984 shutdown_connections 2024-04-21T23:09:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.737+0000 7f622b56c740 1 --2- 172.21.15.44:0/3824548984 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562933be4b50 0x562933be6fd0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.737+0000 7f622b56c740 1 --2- 172.21.15.44:0/3824548984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bd4d30 0x562933bd1ea0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.737+0000 7f622b56c740 1 --2- 172.21.15.44:0/3824548984 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562933bc3670 0x562933bd1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.737+0000 7f6209ffb640 0 client.25486 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.737+0000 7f622b56c740 1 --2- 172.21.15.44:0/3824548984 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bc2d50 0x562933bd45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.737+0000 7f622b56c740 1 -- 172.21.15.44:0/3824548984 >> 172.21.15.44:0/3824548984 conn(0x562933bbd750 msgr2=0x562933bbf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.737+0000 7f622b56c740 1 -- 172.21.15.44:0/3824548984 shutdown_connections 2024-04-21T23:09:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.737+0000 7f622b56c740 1 -- 172.21.15.44:0/3824548984 wait complete. 2024-04-21T23:09:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.737+0000 7f622b56c740 20 client.25486 trim_cache size 0 max 16384 2024-04-21T23:09:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.750+0000 7f622b56c740 1 Processor -- start 2024-04-21T23:09:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.750+0000 7f622b56c740 1 -- start start 2024-04-21T23:09:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.751+0000 7f622b56c740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562933bc2d50 0x562933be09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.751+0000 7f622b56c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bc3670 0x562933bccea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.751+0000 7f622b56c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bd4d30 0x562933bd4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.751+0000 7f622b56c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f62241ab050 con 0x562933bc3670 2024-04-21T23:09:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.751+0000 7f622b56c740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f62241ad8d0 con 0x562933bc2d50 2024-04-21T23:09:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.751+0000 7f622b56c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f62241ab6c0 con 0x562933bd4d30 2024-04-21T23:09:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.751+0000 7f6221d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bc3670 0x562933bccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.751+0000 7f6221d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bc3670 0x562933bccea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53402/0 (socket says 172.21.15.44:53402) 2024-04-21T23:09:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.751+0000 7f6221d9b640 1 -- 172.21.15.44:0/343579635 learned_addr learned my addr 172.21.15.44:0/343579635 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.751+0000 7f6221d9b640 1 -- 172.21.15.44:0/343579635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bd4d30 msgr2=0x562933bd4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.751+0000 7f6221d9b640 1 --2- 172.21.15.44:0/343579635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bd4d30 0x562933bd4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.751+0000 7f6221d9b640 1 -- 172.21.15.44:0/343579635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562933bc2d50 msgr2=0x562933be09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.751+0000 7f6221d9b640 1 --2- 172.21.15.44:0/343579635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562933bc2d50 0x562933be09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.751+0000 7f6221d9b640 1 -- 172.21.15.44:0/343579635 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629339057d0 con 0x562933bc3670 2024-04-21T23:09:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.751+0000 7f6221d9b640 1 --2- 172.21.15.44:0/343579635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bc3670 0x562933bccea0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f6218009be0 tx=0x7f62180097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45247d6ba8bb9118 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.752+0000 7f620bfff640 1 -- 172.21.15.44:0/343579635 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6218029020 con 0x562933bc3670 2024-04-21T23:09:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.752+0000 7f620bfff640 1 -- 172.21.15.44:0/343579635 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6218009890 con 0x562933bc3670 2024-04-21T23:09:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.752+0000 7f620bfff640 1 -- 172.21.15.44:0/343579635 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621801e040 con 0x562933bc3670 2024-04-21T23:09:58.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.752+0000 7f622b56c740 1 -- 172.21.15.44:0/343579635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bc3670 msgr2=0x562933bccea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.752+0000 7f622b56c740 1 --2- 172.21.15.44:0/343579635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bc3670 0x562933bccea0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f6218009be0 tx=0x7f62180097a0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.752+0000 7f622b56c740 1 -- 172.21.15.44:0/343579635 shutdown_connections 2024-04-21T23:09:58.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.752+0000 7f622b56c740 1 --2- 172.21.15.44:0/343579635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bd4d30 0x562933bd4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.752+0000 7f622b56c740 1 --2- 172.21.15.44:0/343579635 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bc3670 0x562933bccea0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.752+0000 7f622b56c740 1 --2- 172.21.15.44:0/343579635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562933bc2d50 0x562933be09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.752+0000 7f622b56c740 1 -- 172.21.15.44:0/343579635 >> 172.21.15.44:0/343579635 conn(0x562933bbd750 msgr2=0x562933bcc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:58.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.753+0000 7f622b56c740 1 -- 172.21.15.44:0/343579635 shutdown_connections 2024-04-21T23:09:58.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.753+0000 7f622b56c740 1 -- 172.21.15.44:0/343579635 wait complete. 2024-04-21T23:09:58.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.753+0000 7f622b56c740 1 -- start start 2024-04-21T23:09:58.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.753+0000 7f622b56c740 1 Processor -- start 2024-04-21T23:09:58.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.753+0000 7f622b56c740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562933bc2d50 0x562933bdcf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.753+0000 7f622b56c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bc3670 0x562933bdd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.753+0000 7f622b56c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bd4d30 0x562933bcdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.753+0000 7f622b56c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f62241abd90 con 0x562933bd4d30 2024-04-21T23:09:58.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.753+0000 7f622b56c740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f62241ad200 con 0x562933bc2d50 2024-04-21T23:09:58.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.753+0000 7f622b56c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f62241ac460 con 0x562933bc3670 2024-04-21T23:09:58.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.753+0000 7f622b56c740 10 client.0 _ll_register_callbacks cb 0x7f621c004f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:58.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.753+0000 7f6222d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bd4d30 0x562933bcdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.753+0000 7f6222d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bd4d30 0x562933bcdd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53416/0 (socket says 172.21.15.44:53416) 2024-04-21T23:09:58.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.753+0000 7f6222d9d640 1 -- 172.21.15.44:0/1787705285 learned_addr learned my addr 172.21.15.44:0/1787705285 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:58.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.753+0000 7f6221d9b640 1 --2- 172.21.15.44:0/1787705285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bc3670 0x562933bdd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.754+0000 7f6221d9b640 1 -- 172.21.15.44:0/1787705285 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562933bc2d50 msgr2=0x562933bdcf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:58.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.754+0000 7f6221d9b640 1 --2- 172.21.15.44:0/1787705285 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562933bc2d50 0x562933bdcf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.754+0000 7f6221d9b640 1 -- 172.21.15.44:0/1787705285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bd4d30 msgr2=0x562933bcdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:58.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.754+0000 7f6221d9b640 1 --2- 172.21.15.44:0/1787705285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bd4d30 0x562933bcdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.754+0000 7f6221d9b640 1 -- 172.21.15.44:0/1787705285 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562933bce4e0 con 0x562933bc3670 2024-04-21T23:09:58.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.754+0000 7f6222d9d640 1 --2- 172.21.15.44:0/1787705285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bd4d30 0x562933bcdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:09:58.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.754+0000 7f6221d9b640 1 --2- 172.21.15.44:0/1787705285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bc3670 0x562933bdd440 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f621801c040 tx=0x7f6218005b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d105ebd97369e044 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:58.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.754+0000 7f6209ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:09:58.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.754+0000 7f6209ffb640 1 -- 172.21.15.44:0/1787705285 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621801e040 con 0x562933bc3670 2024-04-21T23:09:58.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.754+0000 7f622b56c740 1 -- 172.21.15.44:0/1787705285 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6218009890 con 0x562933bc3670 2024-04-21T23:09:58.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.754+0000 7f6209ffb640 1 -- 172.21.15.44:0/1787705285 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f6218005510 con 0x562933bc3670 2024-04-21T23:09:58.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.754+0000 7f6209ffb640 1 -- 172.21.15.44:0/1787705285 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f621803f050 con 0x562933bc3670 2024-04-21T23:09:58.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.754+0000 7f622b56c740 10 client.25489 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:58.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.754+0000 7f622b56c740 1 -- 172.21.15.44:0/1787705285 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562933c4ab70 con 0x562933bc3670 2024-04-21T23:09:58.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.755+0000 7f6209ffb640 1 -- 172.21.15.44:0/1787705285 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6218009890 con 0x562933bc3670 2024-04-21T23:09:58.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.755+0000 7f6209ffb640 1 -- 172.21.15.44:0/1787705285 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f6218025690 con 0x562933bc3670 2024-04-21T23:09:58.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.755+0000 7f622b56c740 10 client.25489 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:58.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.755+0000 7f622b56c740 10 client.25489 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:58.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.755+0000 7f622b56c740 20 client.25489 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:58.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.755+0000 7f622b56c740 1 --2- 172.21.15.44:0/1787705285 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562933c4b110 0x562933c4d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:58.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.755+0000 7f622b56c740 4 client.25489 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:09:58.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.755+0000 7f622b56c740 1 -- 172.21.15.44:0/1787705285 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562933b2edc0 con 0x562933c4b110 2024-04-21T23:09:58.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.755+0000 7f622259c640 1 --2- 172.21.15.44:0/1787705285 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562933c4b110 0x562933c4d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:58.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.756+0000 7f6209ffb640 10 client.25489 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:58.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.756+0000 7f622259c640 1 --2- 172.21.15.44:0/1787705285 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562933c4b110 0x562933c4d530 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12353386fcbf4e5a server_cookie=18e2b73435492ddb in_seq=0 out_seq=0 2024-04-21T23:09:58.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.756+0000 7f6209ffb640 1 -- 172.21.15.44:0/1787705285 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562933b2edc0 con 0x562933c4b110 2024-04-21T23:09:58.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.756+0000 7f6209ffb640 10 client.25489 handle_command_reply: tid=0 2024-04-21T23:09:58.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.756+0000 7f622b56c740 1 client.25489 shutdown 2024-04-21T23:09:58.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.756+0000 7f622b56c740 1 -- 172.21.15.44:0/1787705285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bc3670 msgr2=0x562933bdd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.756+0000 7f622b56c740 1 --2- 172.21.15.44:0/1787705285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bc3670 0x562933bdd440 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f621801c040 tx=0x7f6218005b30 comp rx=0 tx=0).stop 2024-04-21T23:09:58.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.756+0000 7f622b56c740 1 -- 172.21.15.44:0/1787705285 shutdown_connections 2024-04-21T23:09:58.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.756+0000 7f622b56c740 1 --2- 172.21.15.44:0/1787705285 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562933c4b110 0x562933c4d530 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.756+0000 7f622b56c740 1 --2- 172.21.15.44:0/1787705285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562933bc3670 0x562933bdd440 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.756+0000 7f6209ffb640 0 client.25489 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:58.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.756+0000 7f622b56c740 1 --2- 172.21.15.44:0/1787705285 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562933bd4d30 0x562933bcdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.756+0000 7f622b56c740 1 --2- 172.21.15.44:0/1787705285 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562933bc2d50 0x562933bdcf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.756+0000 7f622b56c740 1 -- 172.21.15.44:0/1787705285 >> 172.21.15.44:0/1787705285 conn(0x562933bbd750 msgr2=0x562933bcd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:58.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.756+0000 7f622b56c740 1 -- 172.21.15.44:0/1787705285 shutdown_connections 2024-04-21T23:09:58.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.756+0000 7f622b56c740 1 -- 172.21.15.44:0/1787705285 wait complete. 2024-04-21T23:09:58.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.756+0000 7f622b56c740 20 client.25489 trim_cache size 0 max 16384 2024-04-21T23:09:58.761 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-21T23:09:58.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.757+0000 7f62288a7640 1 -- 172.21.15.44:0/210761058 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f620c0a2f50 msgr2=0x7f620c0a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.757+0000 7f62288a7640 1 --2- 172.21.15.44:0/210761058 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f620c0a2f50 0x7f620c0a5410 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6214009230 tx=0x7f6214005a20 comp rx=0 tx=0).stop 2024-04-21T23:09:58.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.757+0000 7f62288a7640 1 -- 172.21.15.44:0/210761058 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6224101080 msgr2=0x7f62241a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:58.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.757+0000 7f62288a7640 1 --2- 172.21.15.44:0/210761058 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6224101080 0x7f62241a63c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f621c00e960 tx=0x7f621c00ee20 comp rx=0 tx=0).stop 2024-04-21T23:09:58.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.757+0000 7f62288a7640 1 -- 172.21.15.44:0/210761058 shutdown_connections 2024-04-21T23:09:58.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.757+0000 7f62288a7640 1 --2- 172.21.15.44:0/210761058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f622410bcf0 0x7f6224100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.757+0000 7f62288a7640 1 --2- 172.21.15.44:0/210761058 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f620c0a2f50 0x7f620c0a5410 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.757+0000 7f62288a7640 1 --2- 172.21.15.44:0/210761058 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6224101080 0x7f62241a63c0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.757+0000 7f62288a7640 1 --2- 172.21.15.44:0/210761058 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f62241032d0 0x7f6224100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:58.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.757+0000 7f62288a7640 1 -- 172.21.15.44:0/210761058 >> 172.21.15.44:0/210761058 conn(0x7f62240777f0 msgr2=0x7f62240fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:58.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.757+0000 7f62288a7640 1 -- 172.21.15.44:0/210761058 shutdown_connections 2024-04-21T23:09:58.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:58.757+0000 7f62288a7640 1 -- 172.21.15.44:0/210761058 wait complete. 2024-04-21T23:09:59.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:09:59 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/796923793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:09:59.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:09:59 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/796923793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:09:59.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:09:59 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/796923793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:09:59.769 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:09:59.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:09:59.769 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:09:59.880 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.880+0000 7f388a65f640 1 -- 172.21.15.44:0/3448148434 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3884101710 msgr2=0x7f3884101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:59.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.880+0000 7f388a65f640 1 --2- 172.21.15.44:0/3448148434 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3884101710 0x7f3884101b90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3878009860 tx=0x7f387802bb70 comp rx=0 tx=0).stop 2024-04-21T23:09:59.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.880+0000 7f388a65f640 1 -- 172.21.15.44:0/3448148434 shutdown_connections 2024-04-21T23:09:59.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.880+0000 7f388a65f640 1 --2- 172.21.15.44:0/3448148434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38841020d0 0x7f388410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.880+0000 7f388a65f640 1 --2- 172.21.15.44:0/3448148434 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3884101710 0x7f3884101b90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.880+0000 7f388a65f640 1 --2- 172.21.15.44:0/3448148434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3884107720 0x7f3884107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.880+0000 7f388a65f640 1 -- 172.21.15.44:0/3448148434 >> 172.21.15.44:0/3448148434 conn(0x7f38840fd200 msgr2=0x7f38840ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:59.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.880+0000 7f388a65f640 1 -- 172.21.15.44:0/3448148434 shutdown_connections 2024-04-21T23:09:59.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.880+0000 7f388a65f640 1 -- 172.21.15.44:0/3448148434 wait complete. 2024-04-21T23:09:59.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.881+0000 7f388a65f640 1 Processor -- start 2024-04-21T23:09:59.881 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.881+0000 7f388a65f640 1 -- start start 2024-04-21T23:09:59.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.882+0000 7f388a65f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3884101710 0x7f38841980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:59.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.882+0000 7f388a65f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38841020d0 0x7f38841985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:59.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.882+0000 7f388a65f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3884107720 0x7f388419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:59.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.882+0000 7f388a65f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3884111b30 con 0x7f3884107720 2024-04-21T23:09:59.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.882+0000 7f388a65f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f38841119b0 con 0x7f38841020d0 2024-04-21T23:09:59.883 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.882+0000 7f388a65f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3884111cb0 con 0x7f3884101710 2024-04-21T23:09:59.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.882+0000 7f3888bfc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3884107720 0x7f388419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:59.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.882+0000 7f3888bfc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3884107720 0x7f388419c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53428/0 (socket says 172.21.15.44:53428) 2024-04-21T23:09:59.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.882+0000 7f3888bfc640 1 -- 172.21.15.44:0/4016642155 learned_addr learned my addr 172.21.15.44:0/4016642155 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:59.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.882+0000 7f38837fe640 1 --2- 172.21.15.44:0/4016642155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38841020d0 0x7f38841985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:59.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.882+0000 7f3883fff640 1 --2- 172.21.15.44:0/4016642155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3884101710 0x7f38841980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:59.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.882+0000 7f3888bfc640 1 -- 172.21.15.44:0/4016642155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3884101710 msgr2=0x7f38841980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:59.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.882+0000 7f3888bfc640 1 --2- 172.21.15.44:0/4016642155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3884101710 0x7f38841980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.882+0000 7f3888bfc640 1 -- 172.21.15.44:0/4016642155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38841020d0 msgr2=0x7f38841985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:59.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.882+0000 7f3888bfc640 1 --2- 172.21.15.44:0/4016642155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38841020d0 0x7f38841985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.882+0000 7f3888bfc640 1 -- 172.21.15.44:0/4016642155 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f388419d050 con 0x7f3884107720 2024-04-21T23:09:59.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.882+0000 7f3888bfc640 1 --2- 172.21.15.44:0/4016642155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3884107720 0x7f388419c920 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f387c00efc0 tx=0x7f387c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:59.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.883+0000 7f38817fa640 1 -- 172.21.15.44:0/4016642155 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c003970 con 0x7f3884107720 2024-04-21T23:09:59.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.883+0000 7f388a65f640 1 -- 172.21.15.44:0/4016642155 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f388419d340 con 0x7f3884107720 2024-04-21T23:09:59.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.883+0000 7f38817fa640 1 -- 172.21.15.44:0/4016642155 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f387c004020 con 0x7f3884107720 2024-04-21T23:09:59.884 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.883+0000 7f38817fa640 1 -- 172.21.15.44:0/4016642155 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c00b9f0 con 0x7f3884107720 2024-04-21T23:09:59.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.883+0000 7f388a65f640 1 -- 172.21.15.44:0/4016642155 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38841a4bd0 con 0x7f3884107720 2024-04-21T23:09:59.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.885+0000 7f38817fa640 1 -- 172.21.15.44:0/4016642155 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f387c01d020 con 0x7f3884107720 2024-04-21T23:09:59.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.886+0000 7f38817fa640 1 --2- 172.21.15.44:0/4016642155 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f386c0a2e20 0x7f386c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:59.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.886+0000 7f38817fa640 1 -- 172.21.15.44:0/4016642155 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f387c0cf9c0 con 0x7f3884107720 2024-04-21T23:09:59.885 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.886+0000 7f3883fff640 1 --2- 172.21.15.44:0/4016642155 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f386c0a2e20 0x7f386c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:59.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.886+0000 7f388d240740 1 Processor -- start 2024-04-21T23:09:59.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.886+0000 7f388d240740 1 -- start start 2024-04-21T23:09:59.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.886+0000 7f3883fff640 1 --2- 172.21.15.44:0/4016642155 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f386c0a2e20 0x7f386c0a52e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3874009b50 tx=0x7f3874005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:59.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.887+0000 7f388d240740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32e2b30 0x55e4e32e1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:59.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.887+0000 7f388d240740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e1e50 0x55e4e32e22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:59.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.887+0000 7f388d240740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32eb1c0 0x55e4e32ed5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:59.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.887+0000 7f388d240740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e4e3241170 con 0x55e4e32e1e50 2024-04-21T23:09:59.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.887+0000 7f388d240740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55e4e31e97c0 con 0x55e4e32eb1c0 2024-04-21T23:09:59.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.887+0000 7f388d240740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e4e323d130 con 0x55e4e32e2b30 2024-04-21T23:09:59.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.887+0000 7f38837fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e1e50 0x55e4e32e22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:59.888 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.887+0000 7f3883fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32e2b30 0x55e4e32e1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:59.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.887+0000 7f38837fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e1e50 0x55e4e32e22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53432/0 (socket says 172.21.15.44:53432) 2024-04-21T23:09:59.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.887+0000 7f3883fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32e2b30 0x55e4e32e1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:54216/0 (socket says 172.21.15.44:54216) 2024-04-21T23:09:59.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.887+0000 7f38837fe640 1 -- 172.21.15.44:0/3814550980 learned_addr learned my addr 172.21.15.44:0/3814550980 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:59.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.887+0000 7f3888bfc640 1 --2- 172.21.15.44:0/3814550980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32eb1c0 0x55e4e32ed5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:59.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.887+0000 7f3883fff640 1 -- 172.21.15.44:0/3814550980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32eb1c0 msgr2=0x55e4e32ed5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:59.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.887+0000 7f3883fff640 1 --2- 172.21.15.44:0/3814550980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32eb1c0 0x55e4e32ed5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.887+0000 7f3883fff640 1 -- 172.21.15.44:0/3814550980 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e1e50 msgr2=0x55e4e32e22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:59.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.887+0000 7f3883fff640 1 --2- 172.21.15.44:0/3814550980 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e1e50 0x55e4e32e22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.887+0000 7f3883fff640 1 -- 172.21.15.44:0/3814550980 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4e31fae60 con 0x55e4e32e2b30 2024-04-21T23:09:59.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.887+0000 7f3888bfc640 1 --2- 172.21.15.44:0/3814550980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32eb1c0 0x55e4e32ed5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:09:59.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.887+0000 7f38837fe640 1 --2- 172.21.15.44:0/3814550980 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e1e50 0x55e4e32e22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:09:59.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.888+0000 7f3883fff640 1 --2- 172.21.15.44:0/3814550980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32e2b30 0x55e4e32e1880 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3874009fd0 tx=0x7f3874014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa7ac126985544b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:59.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.888+0000 7f386a7fc640 1 -- 172.21.15.44:0/3814550980 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3874005020 con 0x55e4e32e2b30 2024-04-21T23:09:59.889 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.888+0000 7f386a7fc640 1 -- 172.21.15.44:0/3814550980 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38740047a0 con 0x55e4e32e2b30 2024-04-21T23:09:59.890 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.888+0000 7f386a7fc640 1 -- 172.21.15.44:0/3814550980 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387401e6c0 con 0x55e4e32e2b30 2024-04-21T23:09:59.893 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.889+0000 7f388d240740 1 -- 172.21.15.44:0/3814550980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32e2b30 msgr2=0x55e4e32e1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.889+0000 7f388d240740 1 --2- 172.21.15.44:0/3814550980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32e2b30 0x55e4e32e1880 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3874009fd0 tx=0x7f3874014000 comp rx=0 tx=0).stop 2024-04-21T23:09:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.889+0000 7f388d240740 1 -- 172.21.15.44:0/3814550980 shutdown_connections 2024-04-21T23:09:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.889+0000 7f388d240740 1 --2- 172.21.15.44:0/3814550980 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32eb1c0 0x55e4e32ed5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.889+0000 7f388d240740 1 --2- 172.21.15.44:0/3814550980 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e1e50 0x55e4e32e22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.889+0000 7f388d240740 1 --2- 172.21.15.44:0/3814550980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32e2b30 0x55e4e32e1880 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.889+0000 7f388d240740 1 -- 172.21.15.44:0/3814550980 >> 172.21.15.44:0/3814550980 conn(0x55e4e32a44f0 msgr2=0x55e4e32dc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.889+0000 7f388d240740 1 -- 172.21.15.44:0/3814550980 shutdown_connections 2024-04-21T23:09:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.889+0000 7f388d240740 1 -- 172.21.15.44:0/3814550980 wait complete. 2024-04-21T23:09:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.890+0000 7f388d240740 1 -- start start 2024-04-21T23:09:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.890+0000 7f388d240740 1 Processor -- start 2024-04-21T23:09:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.891+0000 7f388d240740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e1e50 0x55e4e32e8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:59.894 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.891+0000 7f388d240740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32e2b30 0x55e4e32e8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.891+0000 7f388d240740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32eb1c0 0x55e4e32e8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.891+0000 7f388d240740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e4e32ee030 con 0x55e4e32e1e50 2024-04-21T23:09:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.891+0000 7f388d240740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55e4e31fcd40 con 0x55e4e32e2b30 2024-04-21T23:09:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.891+0000 7f3883fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e1e50 0x55e4e32e8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.891+0000 7f388d240740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e4e32ee1b0 con 0x55e4e32eb1c0 2024-04-21T23:09:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.891+0000 7f3883fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e1e50 0x55e4e32e8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53440/0 (socket says 172.21.15.44:53440) 2024-04-21T23:09:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.891+0000 7f3883fff640 1 -- 172.21.15.44:0/3733717765 learned_addr learned my addr 172.21.15.44:0/3733717765 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.891+0000 7f388d240740 10 client.0 _ll_register_callbacks cb 0x55e4e3238820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.891+0000 7f38837fe640 1 --2- 172.21.15.44:0/3733717765 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32e2b30 0x55e4e32e8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.891+0000 7f3888bfc640 1 --2- 172.21.15.44:0/3733717765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32eb1c0 0x55e4e32e8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.891+0000 7f3883fff640 1 -- 172.21.15.44:0/3733717765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32eb1c0 msgr2=0x55e4e32e8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.891+0000 7f3883fff640 1 --2- 172.21.15.44:0/3733717765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32eb1c0 0x55e4e32e8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.891+0000 7f3883fff640 1 -- 172.21.15.44:0/3733717765 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32e2b30 msgr2=0x55e4e32e8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.891+0000 7f3883fff640 1 --2- 172.21.15.44:0/3733717765 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32e2b30 0x55e4e32e8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.895 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.891+0000 7f3883fff640 1 -- 172.21.15.44:0/3733717765 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4e301d7e0 con 0x55e4e32e1e50 2024-04-21T23:09:59.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.891+0000 7f3888bfc640 1 --2- 172.21.15.44:0/3733717765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32eb1c0 0x55e4e32e8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:09:59.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.892+0000 7f3883fff640 1 --2- 172.21.15.44:0/3733717765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e1e50 0x55e4e32e8100 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f38740057c0 tx=0x7f3874004070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab81593cc441790 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:59.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.892+0000 7f38637fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:09:59.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.892+0000 7f38637fe640 1 -- 172.21.15.44:0/3733717765 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387401e420 con 0x55e4e32e1e50 2024-04-21T23:09:59.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.892+0000 7f38637fe640 1 -- 172.21.15.44:0/3733717765 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f387401e5c0 con 0x55e4e32e1e50 2024-04-21T23:09:59.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.892+0000 7f388d240740 1 -- 172.21.15.44:0/3733717765 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38740047a0 con 0x55e4e32e1e50 2024-04-21T23:09:59.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.892+0000 7f38637fe640 1 -- 172.21.15.44:0/3733717765 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f387401e7c0 con 0x55e4e32e1e50 2024-04-21T23:09:59.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.892+0000 7f388d240740 10 client.15855 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:59.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.892+0000 7f388d240740 1 -- 172.21.15.44:0/3733717765 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e4e307df70 con 0x55e4e32e1e50 2024-04-21T23:09:59.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.892+0000 7f38637fe640 1 -- 172.21.15.44:0/3733717765 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38740047a0 con 0x55e4e32e1e50 2024-04-21T23:09:59.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.893+0000 7f38637fe640 1 -- 172.21.15.44:0/3733717765 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f387403d020 con 0x55e4e32e1e50 2024-04-21T23:09:59.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.893+0000 7f388d240740 10 client.15855 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:59.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.893+0000 7f388d240740 10 client.15855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:59.896 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.893+0000 7f388d240740 20 client.15855 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:59.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.893+0000 7f388d240740 1 --2- 172.21.15.44:0/3733717765 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e4e32fb100 0x55e4e32fd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:59.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.893+0000 7f388d240740 4 client.15855 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:09:59.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.893+0000 7f388d240740 1 -- 172.21.15.44:0/3733717765 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e4e3215450 con 0x55e4e32fb100 2024-04-21T23:09:59.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.893+0000 7f38837fe640 1 --2- 172.21.15.44:0/3733717765 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e4e32fb100 0x55e4e32fd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:59.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.894+0000 7f38637fe640 10 client.15855 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:59.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.894+0000 7f38837fe640 1 --2- 172.21.15.44:0/3733717765 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e4e32fb100 0x55e4e32fd4f0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b6b0e319536c2c4 server_cookie=14f5a710c3805511 in_seq=0 out_seq=0 2024-04-21T23:09:59.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.894+0000 7f38637fe640 1 -- 172.21.15.44:0/3733717765 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e4e3215450 con 0x55e4e32fb100 2024-04-21T23:09:59.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.894+0000 7f38637fe640 10 client.15855 handle_command_reply: tid=0 2024-04-21T23:09:59.897 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.894+0000 7f388d240740 1 client.15855 shutdown 2024-04-21T23:09:59.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.895+0000 7f388d240740 1 -- 172.21.15.44:0/3733717765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e1e50 msgr2=0x55e4e32e8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:59.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.895+0000 7f388d240740 1 --2- 172.21.15.44:0/3733717765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e1e50 0x55e4e32e8100 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f38740057c0 tx=0x7f3874004070 comp rx=0 tx=0).stop 2024-04-21T23:09:59.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.895+0000 7f388d240740 1 -- 172.21.15.44:0/3733717765 shutdown_connections 2024-04-21T23:09:59.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.895+0000 7f388d240740 1 --2- 172.21.15.44:0/3733717765 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e4e32fb100 0x55e4e32fd4f0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.895+0000 7f38637fe640 0 client.15855 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:59.898 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.895+0000 7f388d240740 1 --2- 172.21.15.44:0/3733717765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32eb1c0 0x55e4e32e8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.895+0000 7f388d240740 1 --2- 172.21.15.44:0/3733717765 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32e2b30 0x55e4e32e8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.895+0000 7f388d240740 1 --2- 172.21.15.44:0/3733717765 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e1e50 0x55e4e32e8100 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.895+0000 7f388d240740 1 -- 172.21.15.44:0/3733717765 >> 172.21.15.44:0/3733717765 conn(0x55e4e32a44f0 msgr2=0x55e4e32de5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:59.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.895+0000 7f388d240740 1 -- 172.21.15.44:0/3733717765 shutdown_connections 2024-04-21T23:09:59.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.895+0000 7f388d240740 1 -- 172.21.15.44:0/3733717765 wait complete. 2024-04-21T23:09:59.899 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.895+0000 7f388d240740 20 client.15855 trim_cache size 0 max 16384 2024-04-21T23:09:59.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.930+0000 7f388d240740 1 Processor -- start 2024-04-21T23:09:59.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.930+0000 7f388d240740 1 -- start start 2024-04-21T23:09:59.931 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.931+0000 7f388d240740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32e1e50 0x55e4e32f6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:59.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.931+0000 7f388d240740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e2b30 0x55e4e32e79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:59.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.931+0000 7f388d240740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32eb1c0 0x55e4e3302c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:59.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.931+0000 7f388d240740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f38841aaa90 con 0x55e4e32e2b30 2024-04-21T23:09:59.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.931+0000 7f388d240740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f38841ad310 con 0x55e4e32eb1c0 2024-04-21T23:09:59.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.931+0000 7f388d240740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f38841ab100 con 0x55e4e32e1e50 2024-04-21T23:09:59.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.931+0000 7f38837fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e2b30 0x55e4e32e79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:59.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.931+0000 7f38837fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e2b30 0x55e4e32e79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53456/0 (socket says 172.21.15.44:53456) 2024-04-21T23:09:59.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.931+0000 7f38837fe640 1 -- 172.21.15.44:0/4175310128 learned_addr learned my addr 172.21.15.44:0/4175310128 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:59.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.931+0000 7f3883fff640 1 --2- 172.21.15.44:0/4175310128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32e1e50 0x55e4e32f6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:59.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.931+0000 7f3888bfc640 1 --2- 172.21.15.44:0/4175310128 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32eb1c0 0x55e4e3302c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:59.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.931+0000 7f3883fff640 1 -- 172.21.15.44:0/4175310128 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32eb1c0 msgr2=0x55e4e3302c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:59.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.931+0000 7f3883fff640 1 --2- 172.21.15.44:0/4175310128 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32eb1c0 0x55e4e3302c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.932 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.931+0000 7f3883fff640 1 -- 172.21.15.44:0/4175310128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e2b30 msgr2=0x55e4e32e79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:59.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.931+0000 7f3883fff640 1 --2- 172.21.15.44:0/4175310128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e2b30 0x55e4e32e79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.931+0000 7f3883fff640 1 -- 172.21.15.44:0/4175310128 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4e2f15170 con 0x55e4e32e1e50 2024-04-21T23:09:59.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.932+0000 7f3883fff640 1 --2- 172.21.15.44:0/4175310128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32e1e50 0x55e4e32f6f90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3874004e20 tx=0x7f3874026590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f170b574f7731c9d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:59.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.932+0000 7f3869ffb640 1 -- 172.21.15.44:0/4175310128 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3874042020 con 0x55e4e32e1e50 2024-04-21T23:09:59.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.932+0000 7f3869ffb640 1 -- 172.21.15.44:0/4175310128 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f387402a050 con 0x55e4e32e1e50 2024-04-21T23:09:59.933 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.932+0000 7f3869ffb640 1 -- 172.21.15.44:0/4175310128 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387401e420 con 0x55e4e32e1e50 2024-04-21T23:09:59.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.933+0000 7f388d240740 1 -- 172.21.15.44:0/4175310128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32e1e50 msgr2=0x55e4e32f6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:59.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.933+0000 7f388d240740 1 --2- 172.21.15.44:0/4175310128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32e1e50 0x55e4e32f6f90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3874004e20 tx=0x7f3874026590 comp rx=0 tx=0).stop 2024-04-21T23:09:59.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.933+0000 7f388d240740 1 -- 172.21.15.44:0/4175310128 shutdown_connections 2024-04-21T23:09:59.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.933+0000 7f388d240740 1 --2- 172.21.15.44:0/4175310128 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32eb1c0 0x55e4e3302c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.933+0000 7f388d240740 1 --2- 172.21.15.44:0/4175310128 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e2b30 0x55e4e32e79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.933+0000 7f388d240740 1 --2- 172.21.15.44:0/4175310128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32e1e50 0x55e4e32f6f90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.933+0000 7f388d240740 1 -- 172.21.15.44:0/4175310128 >> 172.21.15.44:0/4175310128 conn(0x55e4e32a44f0 msgr2=0x55e4e32e8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:59.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.933+0000 7f388d240740 1 -- 172.21.15.44:0/4175310128 shutdown_connections 2024-04-21T23:09:59.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.933+0000 7f388d240740 1 -- 172.21.15.44:0/4175310128 wait complete. 2024-04-21T23:09:59.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.933+0000 7f388d240740 1 -- start start 2024-04-21T23:09:59.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.934+0000 7f388d240740 1 Processor -- start 2024-04-21T23:09:59.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.934+0000 7f388d240740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32e1e50 0x55e4e32f3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:59.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.934+0000 7f388d240740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e2b30 0x55e4e32eed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:59.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.934+0000 7f388d240740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32eb1c0 0x55e4e32ef3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:59.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.934+0000 7f388d240740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f38841ab7d0 con 0x55e4e32e2b30 2024-04-21T23:09:59.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.934+0000 7f388d240740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f38841acc40 con 0x55e4e32eb1c0 2024-04-21T23:09:59.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.934+0000 7f388d240740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f38841abea0 con 0x55e4e32e1e50 2024-04-21T23:09:59.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.934+0000 7f38837fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e2b30 0x55e4e32eed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:59.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.934+0000 7f38837fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e2b30 0x55e4e32eed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53458/0 (socket says 172.21.15.44:53458) 2024-04-21T23:09:59.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.934+0000 7f38837fe640 1 -- 172.21.15.44:0/2619858937 learned_addr learned my addr 172.21.15.44:0/2619858937 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:09:59.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.934+0000 7f388d240740 10 client.0 _ll_register_callbacks cb 0x7f387c009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:09:59.937 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.934+0000 7f3888bfc640 1 --2- 172.21.15.44:0/2619858937 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32eb1c0 0x55e4e32ef3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:59.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.934+0000 7f3883fff640 1 --2- 172.21.15.44:0/2619858937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32e1e50 0x55e4e32f3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:59.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.935+0000 7f38837fe640 1 -- 172.21.15.44:0/2619858937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32e1e50 msgr2=0x55e4e32f3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:59.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.935+0000 7f38837fe640 1 --2- 172.21.15.44:0/2619858937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32e1e50 0x55e4e32f3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.935+0000 7f38837fe640 1 -- 172.21.15.44:0/2619858937 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32eb1c0 msgr2=0x55e4e32ef3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:09:59.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.935+0000 7f38837fe640 1 --2- 172.21.15.44:0/2619858937 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32eb1c0 0x55e4e32ef3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.935+0000 7f38837fe640 1 -- 172.21.15.44:0/2619858937 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4e32efbd0 con 0x55e4e32e2b30 2024-04-21T23:09:59.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.935+0000 7f3888bfc640 1 --2- 172.21.15.44:0/2619858937 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32eb1c0 0x55e4e32ef3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:09:59.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.935+0000 7f38837fe640 1 --2- 172.21.15.44:0/2619858937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e2b30 0x55e4e32eed90 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3878009780 tx=0x7f3878009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fbde29a57d8f25f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:09:59.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.935+0000 7f38637fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:09:59.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.935+0000 7f38637fe640 1 -- 172.21.15.44:0/2619858937 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387804f060 con 0x55e4e32e2b30 2024-04-21T23:09:59.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.935+0000 7f38637fe640 1 -- 172.21.15.44:0/2619858937 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f38780042c0 con 0x55e4e32e2b30 2024-04-21T23:09:59.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.935+0000 7f388d240740 1 -- 172.21.15.44:0/2619858937 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f387402a050 con 0x55e4e32e2b30 2024-04-21T23:09:59.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.935+0000 7f38637fe640 1 -- 172.21.15.44:0/2619858937 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3878039040 con 0x55e4e32e2b30 2024-04-21T23:09:59.938 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.935+0000 7f388d240740 10 client.15861 fetch_fsmap learned FSMap version 21 2024-04-21T23:09:59.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.935+0000 7f388d240740 1 -- 172.21.15.44:0/2619858937 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e4e3361670 con 0x55e4e32e2b30 2024-04-21T23:09:59.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.936+0000 7f38637fe640 1 -- 172.21.15.44:0/2619858937 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f387402a050 con 0x55e4e32e2b30 2024-04-21T23:09:59.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.936+0000 7f38637fe640 1 -- 172.21.15.44:0/2619858937 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f3878032640 con 0x55e4e32e2b30 2024-04-21T23:09:59.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.936+0000 7f388d240740 10 client.15861 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:09:59.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.936+0000 7f388d240740 10 client.15861 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:09:59.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.936+0000 7f388d240740 20 client.15861 populate_metadata read hostname 'smithi044' 2024-04-21T23:09:59.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.936+0000 7f388d240740 1 --2- 172.21.15.44:0/2619858937 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e4e3361d20 0x55e4e3364140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:09:59.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.936+0000 7f388d240740 4 client.15861 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:09:59.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.936+0000 7f388d240740 1 -- 172.21.15.44:0/2619858937 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e4e324e6b0 con 0x55e4e3361d20 2024-04-21T23:09:59.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.936+0000 7f3883fff640 1 --2- 172.21.15.44:0/2619858937 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e4e3361d20 0x55e4e3364140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:09:59.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.937+0000 7f38637fe640 10 client.15861 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:59.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.937+0000 7f3883fff640 1 --2- 172.21.15.44:0/2619858937 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e4e3361d20 0x55e4e3364140 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5b19e71496f5d68 server_cookie=ddd3309e30f09d58 in_seq=0 out_seq=0 2024-04-21T23:09:59.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.938+0000 7f38637fe640 1 -- 172.21.15.44:0/2619858937 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e4e324e6b0 con 0x55e4e3361d20 2024-04-21T23:09:59.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.938+0000 7f38637fe640 10 client.15861 handle_command_reply: tid=0 2024-04-21T23:09:59.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.938+0000 7f388d240740 1 client.15861 shutdown 2024-04-21T23:09:59.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.938+0000 7f388d240740 1 -- 172.21.15.44:0/2619858937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e2b30 msgr2=0x55e4e32eed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:59.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.938+0000 7f388d240740 1 --2- 172.21.15.44:0/2619858937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e2b30 0x55e4e32eed90 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3878009780 tx=0x7f3878009540 comp rx=0 tx=0).stop 2024-04-21T23:09:59.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.938+0000 7f388d240740 1 -- 172.21.15.44:0/2619858937 shutdown_connections 2024-04-21T23:09:59.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.938+0000 7f388d240740 1 --2- 172.21.15.44:0/2619858937 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e4e3361d20 0x55e4e3364140 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.938+0000 7f388d240740 1 --2- 172.21.15.44:0/2619858937 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e4e32eb1c0 0x55e4e32ef3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.938+0000 7f38637fe640 0 client.15861 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:09:59.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.938+0000 7f388d240740 1 --2- 172.21.15.44:0/2619858937 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e4e32e2b30 0x55e4e32eed90 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.939+0000 7f388d240740 1 --2- 172.21.15.44:0/2619858937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e4e32e1e50 0x55e4e32f3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.941 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.939+0000 7f388d240740 1 -- 172.21.15.44:0/2619858937 >> 172.21.15.44:0/2619858937 conn(0x55e4e32a44f0 msgr2=0x55e4e32f5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:59.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.939+0000 7f388d240740 1 -- 172.21.15.44:0/2619858937 shutdown_connections 2024-04-21T23:09:59.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.939+0000 7f388d240740 1 -- 172.21.15.44:0/2619858937 wait complete. 2024-04-21T23:09:59.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.939+0000 7f388d240740 20 client.15861 trim_cache size 0 max 16384 2024-04-21T23:09:59.942 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:09:59.942 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-21T23:09:59.942 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "34792b20-b675-4fbe-8b80-345e21f4a8c5", 2024-04-21T23:09:59.942 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-21T23:09:59.942 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:09:59.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.940+0000 7f388a65f640 1 -- 172.21.15.44:0/4016642155 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f386c0a2e20 msgr2=0x7f386c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:59.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.940+0000 7f388a65f640 1 --2- 172.21.15.44:0/4016642155 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f386c0a2e20 0x7f386c0a52e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3874009b50 tx=0x7f3874005f90 comp rx=0 tx=0).stop 2024-04-21T23:09:59.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.940+0000 7f388a65f640 1 -- 172.21.15.44:0/4016642155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3884107720 msgr2=0x7f388419c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:09:59.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.940+0000 7f388a65f640 1 --2- 172.21.15.44:0/4016642155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3884107720 0x7f388419c920 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f387c00efc0 tx=0x7f387c00c800 comp rx=0 tx=0).stop 2024-04-21T23:09:59.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.940+0000 7f388a65f640 1 -- 172.21.15.44:0/4016642155 shutdown_connections 2024-04-21T23:09:59.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.940+0000 7f388a65f640 1 --2- 172.21.15.44:0/4016642155 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f386c0a2e20 0x7f386c0a52e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.940+0000 7f388a65f640 1 --2- 172.21.15.44:0/4016642155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3884107720 0x7f388419c920 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.940+0000 7f388a65f640 1 --2- 172.21.15.44:0/4016642155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38841020d0 0x7f38841985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.940+0000 7f388a65f640 1 --2- 172.21.15.44:0/4016642155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3884101710 0x7f38841980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:09:59.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.940+0000 7f388a65f640 1 -- 172.21.15.44:0/4016642155 >> 172.21.15.44:0/4016642155 conn(0x7f38840fd200 msgr2=0x7f38840fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:09:59.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.940+0000 7f388a65f640 1 -- 172.21.15.44:0/4016642155 shutdown_connections 2024-04-21T23:09:59.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:09:59.940+0000 7f388a65f640 1 -- 172.21.15.44:0/4016642155 wait complete. 2024-04-21T23:09:59.954 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:09:59.954 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:10:00.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.062+0000 7f94ea66d640 1 -- 172.21.15.44:0/2914696478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94e410e0e0 msgr2=0x7f94e41104d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:00.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.062+0000 7f94ea66d640 1 --2- 172.21.15.44:0/2914696478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94e410e0e0 0x7f94e41104d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f94dc009ca0 tx=0x7f94dc02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.062+0000 7f94ea66d640 1 -- 172.21.15.44:0/2914696478 shutdown_connections 2024-04-21T23:10:00.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.062+0000 7f94ea66d640 1 --2- 172.21.15.44:0/2914696478 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94e410e0e0 0x7f94e41104d0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.062+0000 7f94ea66d640 1 --2- 172.21.15.44:0/2914696478 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94e4100e50 0x7f94e410dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.062+0000 7f94ea66d640 1 --2- 172.21.15.44:0/2914696478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94e4100530 0x7f94e4100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.062 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.062+0000 7f94ea66d640 1 -- 172.21.15.44:0/2914696478 >> 172.21.15.44:0/2914696478 conn(0x7f94e40777f0 msgr2=0x7f94e4077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:00.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.062+0000 7f94ea66d640 1 -- 172.21.15.44:0/2914696478 shutdown_connections 2024-04-21T23:10:00.063 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.062+0000 7f94ea66d640 1 -- 172.21.15.44:0/2914696478 wait complete. 2024-04-21T23:10:00.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.063+0000 7f94ea66d640 1 Processor -- start 2024-04-21T23:10:00.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.063+0000 7f94ea66d640 1 -- start start 2024-04-21T23:10:00.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.064+0000 7f94ea66d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94e4100530 0x7f94e419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:00.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.064+0000 7f94ea66d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94e4100e50 0x7f94e419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:00.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.064+0000 7f94e3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94e4100530 0x7f94e419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:00.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.064+0000 7f94e3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94e4100530 0x7f94e419a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53482/0 (socket says 172.21.15.44:53482) 2024-04-21T23:10:00.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.064+0000 7f94ea66d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94e410e0e0 0x7f94e419ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:00.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.064+0000 7f94e3fff640 1 -- 172.21.15.44:0/3873771358 learned_addr learned my addr 172.21.15.44:0/3873771358 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:00.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.064+0000 7f94ea66d640 1 -- 172.21.15.44:0/3873771358 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f94e4113d10 con 0x7f94e4100530 2024-04-21T23:10:00.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.064+0000 7f94ea66d640 1 -- 172.21.15.44:0/3873771358 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f94e4113b90 con 0x7f94e4100e50 2024-04-21T23:10:00.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.064+0000 7f94ea66d640 1 -- 172.21.15.44:0/3873771358 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f94e4113e90 con 0x7f94e410e0e0 2024-04-21T23:10:00.065 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.064+0000 7f94e37fe640 1 --2- 172.21.15.44:0/3873771358 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94e4100e50 0x7f94e419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:00.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.064+0000 7f94e37fe640 1 -- 172.21.15.44:0/3873771358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94e410e0e0 msgr2=0x7f94e419ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:00.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.064+0000 7f94e8c0a640 1 --2- 172.21.15.44:0/3873771358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94e410e0e0 0x7f94e419ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:00.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.064+0000 7f94e37fe640 1 --2- 172.21.15.44:0/3873771358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94e410e0e0 0x7f94e419ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.064+0000 7f94e37fe640 1 -- 172.21.15.44:0/3873771358 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94e4100530 msgr2=0x7f94e419a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:00.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.064+0000 7f94e37fe640 1 --2- 172.21.15.44:0/3873771358 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94e4100530 0x7f94e419a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.064+0000 7f94e37fe640 1 -- 172.21.15.44:0/3873771358 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e419f1a0 con 0x7f94e4100e50 2024-04-21T23:10:00.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.064+0000 7f94e3fff640 1 --2- 172.21.15.44:0/3873771358 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94e4100530 0x7f94e419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:00.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.065+0000 7f94e37fe640 1 --2- 172.21.15.44:0/3873771358 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94e4100e50 0x7f94e419a6f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f94d800e9f0 tx=0x7f94d800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:00.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.065+0000 7f94e17fa640 1 -- 172.21.15.44:0/3873771358 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d800ccc0 con 0x7f94e4100e50 2024-04-21T23:10:00.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.065+0000 7f94e17fa640 1 -- 172.21.15.44:0/3873771358 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94d8010910 con 0x7f94e4100e50 2024-04-21T23:10:00.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.065+0000 7f94ea66d640 1 -- 172.21.15.44:0/3873771358 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94e419f490 con 0x7f94e4100e50 2024-04-21T23:10:00.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.065+0000 7f94e17fa640 1 -- 172.21.15.44:0/3873771358 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d8018960 con 0x7f94e4100e50 2024-04-21T23:10:00.066 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.065+0000 7f94ea66d640 1 -- 172.21.15.44:0/3873771358 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94e41a6d20 con 0x7f94e4100e50 2024-04-21T23:10:00.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.068+0000 7f94ed24d740 1 Processor -- start 2024-04-21T23:10:00.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.068+0000 7f94ed24d740 1 -- start start 2024-04-21T23:10:00.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.069+0000 7f94ed24d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3253f30 0x559bb3254310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:00.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.069+0000 7f94ed24d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb3254850 0x559bb325cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:00.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.069+0000 7f94ed24d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb325d430 0x559bb325f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:00.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.069+0000 7f94ed24d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x559bb31b8560 con 0x559bb3253f30 2024-04-21T23:10:00.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.069+0000 7f94ed24d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x559bb3160730 con 0x559bb325d430 2024-04-21T23:10:00.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.069+0000 7f94ed24d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559bb31b4520 con 0x559bb3254850 2024-04-21T23:10:00.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.069+0000 7f94e8c0a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3253f30 0x559bb3254310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:00.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.069+0000 7f94e8c0a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3253f30 0x559bb3254310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53484/0 (socket says 172.21.15.44:53484) 2024-04-21T23:10:00.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.069+0000 7f94e8c0a640 1 -- 172.21.15.44:0/1498914094 learned_addr learned my addr 172.21.15.44:0/1498914094 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:00.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.069+0000 7f94e3fff640 1 --2- 172.21.15.44:0/1498914094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb3254850 0x559bb325cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:00.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.069+0000 7f94e8c0a640 1 -- 172.21.15.44:0/1498914094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb3254850 msgr2=0x559bb325cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:00.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.069+0000 7f94e8c0a640 1 --2- 172.21.15.44:0/1498914094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb3254850 0x559bb325cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.069+0000 7f94e8c0a640 1 -- 172.21.15.44:0/1498914094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb325d430 msgr2=0x559bb325f820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T23:10:00.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.069+0000 7f94e8c0a640 1 --2- 172.21.15.44:0/1498914094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb325d430 0x559bb325f820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.069+0000 7f94e8c0a640 1 -- 172.21.15.44:0/1498914094 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bb2f967d0 con 0x559bb3253f30 2024-04-21T23:10:00.070 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.069+0000 7f94e17fa640 1 -- 172.21.15.44:0/3873771358 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f94d8010ac0 con 0x7f94e4100e50 2024-04-21T23:10:00.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.069+0000 7f94e8c0a640 1 --2- 172.21.15.44:0/1498914094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3253f30 0x559bb3254310 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f94dc004af0 tx=0x7f94dc005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b72a53cd9b212af9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:00.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.070+0000 7f94e3fff640 1 --2- 172.21.15.44:0/1498914094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb3254850 0x559bb325cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:10:00.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.070+0000 7f94ca7fc640 1 -- 172.21.15.44:0/1498914094 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94dc039050 con 0x559bb3253f30 2024-04-21T23:10:00.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.070+0000 7f94ca7fc640 1 -- 172.21.15.44:0/1498914094 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94dc045070 con 0x559bb3253f30 2024-04-21T23:10:00.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.070+0000 7f94ca7fc640 1 -- 172.21.15.44:0/1498914094 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94dc004040 con 0x559bb3253f30 2024-04-21T23:10:00.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.070+0000 7f94e17fa640 1 --2- 172.21.15.44:0/3873771358 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f94cc0ab6b0 0x7f94cc0adb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:00.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.070+0000 7f94e3fff640 1 --2- 172.21.15.44:0/3873771358 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f94cc0ab6b0 0x7f94cc0adb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:00.071 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.070+0000 7f94e17fa640 1 -- 172.21.15.44:0/3873771358 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f94d80cfe50 con 0x7f94e4100e50 2024-04-21T23:10:00.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.071+0000 7f94e3fff640 1 --2- 172.21.15.44:0/3873771358 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f94cc0ab6b0 0x7f94cc0adb70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f94d400bfd0 tx=0x7f94d400b4a0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:00.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.071+0000 7f94ed24d740 1 -- 172.21.15.44:0/1498914094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3253f30 msgr2=0x559bb3254310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:00.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.071+0000 7f94ed24d740 1 --2- 172.21.15.44:0/1498914094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3253f30 0x559bb3254310 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f94dc004af0 tx=0x7f94dc005fd0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.071+0000 7f94ed24d740 1 -- 172.21.15.44:0/1498914094 shutdown_connections 2024-04-21T23:10:00.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.071+0000 7f94ed24d740 1 --2- 172.21.15.44:0/1498914094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb3254850 0x559bb325cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.071+0000 7f94ed24d740 1 --2- 172.21.15.44:0/1498914094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb325d430 0x559bb325f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.071+0000 7f94ed24d740 1 --2- 172.21.15.44:0/1498914094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3253f30 0x559bb3254310 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.071+0000 7f94ed24d740 1 -- 172.21.15.44:0/1498914094 >> 172.21.15.44:0/1498914094 conn(0x559bb324e660 msgr2=0x559bb324ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:00.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.071+0000 7f94ed24d740 1 -- 172.21.15.44:0/1498914094 shutdown_connections 2024-04-21T23:10:00.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.072+0000 7f94ed24d740 1 -- 172.21.15.44:0/1498914094 wait complete. 2024-04-21T23:10:00.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.072+0000 7f94ed24d740 1 -- start start 2024-04-21T23:10:00.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.072+0000 7f94ed24d740 1 Processor -- start 2024-04-21T23:10:00.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.073+0000 7f94ed24d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb3253f30 0x559bb325a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:00.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.073+0000 7f94ed24d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3254850 0x559bb325aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:00.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.073+0000 7f94ed24d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb325d430 0x559bb325af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:00.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.073+0000 7f94ed24d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x559bb32601c0 con 0x559bb3254850 2024-04-21T23:10:00.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.073+0000 7f94ed24d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x559bb3173e00 con 0x559bb3253f30 2024-04-21T23:10:00.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.073+0000 7f94ed24d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559bb3260340 con 0x559bb325d430 2024-04-21T23:10:00.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.073+0000 7f94ed24d740 10 client.0 _ll_register_callbacks cb 0x559bb31aeeb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:00.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.073+0000 7f94e3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb3253f30 0x559bb325a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:00.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.073+0000 7f94e3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb3253f30 0x559bb325a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:60382/0 (socket says 172.21.15.44:60382) 2024-04-21T23:10:00.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.073+0000 7f94e37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3254850 0x559bb325aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:00.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.073+0000 7f94e3fff640 1 -- 172.21.15.44:0/253628394 learned_addr learned my addr 172.21.15.44:0/253628394 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:00.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.073+0000 7f94e8c0a640 1 --2- 172.21.15.44:0/253628394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb325d430 0x559bb325af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:00.075 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.073+0000 7f94e3fff640 1 -- 172.21.15.44:0/253628394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb325d430 msgr2=0x559bb325af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:00.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.073+0000 7f94e3fff640 1 --2- 172.21.15.44:0/253628394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb325d430 0x559bb325af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.073+0000 7f94e3fff640 1 -- 172.21.15.44:0/253628394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3254850 msgr2=0x559bb325aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:00.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.073+0000 7f94e3fff640 1 --2- 172.21.15.44:0/253628394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3254850 0x559bb325aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.073+0000 7f94e3fff640 1 -- 172.21.15.44:0/253628394 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bb2ecc3f0 con 0x559bb3253f30 2024-04-21T23:10:00.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.073+0000 7f94e37fe640 1 --2- 172.21.15.44:0/253628394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3254850 0x559bb325aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:00.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.074+0000 7f94e3fff640 1 --2- 172.21.15.44:0/253628394 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb3253f30 0x559bb325a500 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f94d4010a90 tx=0x7f94d40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac2ebdcf30a688a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:00.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.074+0000 7f94c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:10:00.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.074+0000 7f94c37fe640 1 -- 172.21.15.44:0/253628394 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d401db10 con 0x559bb3253f30 2024-04-21T23:10:00.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.074+0000 7f94c37fe640 1 -- 172.21.15.44:0/253628394 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f94d401dcb0 con 0x559bb3253f30 2024-04-21T23:10:00.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.074+0000 7f94ed24d740 1 -- 172.21.15.44:0/253628394 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94dc045070 con 0x559bb3253f30 2024-04-21T23:10:00.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.074+0000 7f94c37fe640 1 -- 172.21.15.44:0/253628394 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f94d4032050 con 0x559bb3253f30 2024-04-21T23:10:00.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.074+0000 7f94ed24d740 10 client.25454 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:00.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.074+0000 7f94ed24d740 1 -- 172.21.15.44:0/253628394 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559bb2e8c0f0 con 0x559bb3253f30 2024-04-21T23:10:00.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.075+0000 7f94c37fe640 1 -- 172.21.15.44:0/253628394 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94dc045070 con 0x559bb3253f30 2024-04-21T23:10:00.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.075+0000 7f94c37fe640 1 -- 172.21.15.44:0/253628394 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f94d401b550 con 0x559bb3253f30 2024-04-21T23:10:00.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.075+0000 7f94ed24d740 10 client.25454 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:00.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.075+0000 7f94ed24d740 10 client.25454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:00.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.075+0000 7f94ed24d740 20 client.25454 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:00.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.075+0000 7f94ed24d740 1 --2- 172.21.15.44:0/253628394 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x559bb326dc10 0x559bb32700b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:00.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.075+0000 7f94ed24d740 4 client.25454 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:10:00.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.075+0000 7f94ed24d740 1 -- 172.21.15.44:0/253628394 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559bb31c5aa0 con 0x559bb326dc10 2024-04-21T23:10:00.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.076+0000 7f94e37fe640 1 --2- 172.21.15.44:0/253628394 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x559bb326dc10 0x559bb32700b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:00.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.076+0000 7f94e37fe640 1 --2- 172.21.15.44:0/253628394 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x559bb326dc10 0x559bb32700b0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6fb0d759c787178 server_cookie=e92f488799cc0efe in_seq=0 out_seq=0 2024-04-21T23:10:00.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.076+0000 7f94c37fe640 10 client.25454 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:00.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.077+0000 7f94c37fe640 1 -- 172.21.15.44:0/253628394 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559bb31c5aa0 con 0x559bb326dc10 2024-04-21T23:10:00.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.077+0000 7f94c37fe640 10 client.25454 handle_command_reply: tid=0 2024-04-21T23:10:00.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.077+0000 7f94ed24d740 1 client.25454 shutdown 2024-04-21T23:10:00.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.077+0000 7f94ed24d740 1 -- 172.21.15.44:0/253628394 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb3253f30 msgr2=0x559bb325a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:00.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.077+0000 7f94ed24d740 1 --2- 172.21.15.44:0/253628394 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb3253f30 0x559bb325a500 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f94d4010a90 tx=0x7f94d40046c0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.077+0000 7f94ed24d740 1 -- 172.21.15.44:0/253628394 shutdown_connections 2024-04-21T23:10:00.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.078+0000 7f94ed24d740 1 --2- 172.21.15.44:0/253628394 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x559bb326dc10 0x559bb32700b0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.078+0000 7f94ed24d740 1 --2- 172.21.15.44:0/253628394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb325d430 0x559bb325af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.078+0000 7f94c37fe640 0 client.25454 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:00.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.078+0000 7f94ed24d740 1 --2- 172.21.15.44:0/253628394 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3254850 0x559bb325aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.078+0000 7f94ed24d740 1 --2- 172.21.15.44:0/253628394 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb3253f30 0x559bb325a500 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.078+0000 7f94ed24d740 1 -- 172.21.15.44:0/253628394 >> 172.21.15.44:0/253628394 conn(0x559bb324e660 msgr2=0x559bb32503b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:00.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.078+0000 7f94ed24d740 1 -- 172.21.15.44:0/253628394 shutdown_connections 2024-04-21T23:10:00.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.078+0000 7f94ed24d740 1 -- 172.21.15.44:0/253628394 wait complete. 2024-04-21T23:10:00.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.078+0000 7f94ed24d740 20 client.25454 trim_cache size 0 max 16384 2024-04-21T23:10:00.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.114+0000 7f94ed24d740 1 Processor -- start 2024-04-21T23:10:00.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.114+0000 7f94ed24d740 1 -- start start 2024-04-21T23:10:00.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.114+0000 7f94ed24d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb3253f30 0x559bb325cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:00.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.114+0000 7f94ed24d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3254850 0x559bb3269080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:00.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.114+0000 7f94ed24d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb325d430 0x559bb324d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:00.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.115+0000 7f94ed24d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f94e41aa9c0 con 0x559bb3254850 2024-04-21T23:10:00.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.115+0000 7f94ed24d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f94e41ad240 con 0x559bb325d430 2024-04-21T23:10:00.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.115+0000 7f94ed24d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f94e41ab030 con 0x559bb3253f30 2024-04-21T23:10:00.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.115+0000 7f94e37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3254850 0x559bb3269080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:00.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.115+0000 7f94e3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb3253f30 0x559bb325cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:00.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.115+0000 7f94e3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb3253f30 0x559bb325cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:54282/0 (socket says 172.21.15.44:54282) 2024-04-21T23:10:00.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.115+0000 7f94e3fff640 1 -- 172.21.15.44:0/1214897936 learned_addr learned my addr 172.21.15.44:0/1214897936 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:00.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.115+0000 7f94e8c0a640 1 --2- 172.21.15.44:0/1214897936 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb325d430 0x559bb324d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:00.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.115+0000 7f94e37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3254850 0x559bb3269080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53502/0 (socket says 172.21.15.44:53502) 2024-04-21T23:10:00.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.115+0000 7f94e3fff640 1 -- 172.21.15.44:0/1214897936 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb325d430 msgr2=0x559bb324d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:00.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.115+0000 7f94e3fff640 1 --2- 172.21.15.44:0/1214897936 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb325d430 0x559bb324d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.115+0000 7f94e3fff640 1 -- 172.21.15.44:0/1214897936 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3254850 msgr2=0x559bb3269080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:00.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.115+0000 7f94e3fff640 1 --2- 172.21.15.44:0/1214897936 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3254850 0x559bb3269080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.115+0000 7f94e3fff640 1 -- 172.21.15.44:0/1214897936 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bb324e190 con 0x559bb3253f30 2024-04-21T23:10:00.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.115+0000 7f94e3fff640 1 --2- 172.21.15.44:0/1214897936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb3253f30 0x559bb325cc30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f94d4000c00 tx=0x7f94d40104b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ecf6cd84caa0a67 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:00.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.115+0000 7f94c9ffb640 1 -- 172.21.15.44:0/1214897936 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d401bbe0 con 0x559bb3253f30 2024-04-21T23:10:00.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.116+0000 7f94c9ffb640 1 -- 172.21.15.44:0/1214897936 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94d401bd80 con 0x559bb3253f30 2024-04-21T23:10:00.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.116+0000 7f94c9ffb640 1 -- 172.21.15.44:0/1214897936 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d4005570 con 0x559bb3253f30 2024-04-21T23:10:00.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.117+0000 7f94ed24d740 1 -- 172.21.15.44:0/1214897936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb3253f30 msgr2=0x559bb325cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:00.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.117+0000 7f94ed24d740 1 --2- 172.21.15.44:0/1214897936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb3253f30 0x559bb325cc30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f94d4000c00 tx=0x7f94d40104b0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.117+0000 7f94ed24d740 1 -- 172.21.15.44:0/1214897936 shutdown_connections 2024-04-21T23:10:00.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.117+0000 7f94ed24d740 1 --2- 172.21.15.44:0/1214897936 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb325d430 0x559bb324d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.117+0000 7f94ed24d740 1 --2- 172.21.15.44:0/1214897936 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3254850 0x559bb3269080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.117+0000 7f94ed24d740 1 --2- 172.21.15.44:0/1214897936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb3253f30 0x559bb325cc30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.117+0000 7f94ed24d740 1 -- 172.21.15.44:0/1214897936 >> 172.21.15.44:0/1214897936 conn(0x559bb324e660 msgr2=0x559bb324ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:00.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.117+0000 7f94ed24d740 1 -- 172.21.15.44:0/1214897936 shutdown_connections 2024-04-21T23:10:00.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.117+0000 7f94ed24d740 1 -- 172.21.15.44:0/1214897936 wait complete. 2024-04-21T23:10:00.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.117+0000 7f94ed24d740 1 -- start start 2024-04-21T23:10:00.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.117+0000 7f94ed24d740 1 Processor -- start 2024-04-21T23:10:00.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.118+0000 7f94ed24d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3253f30 0x559bb3261020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:00.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.118+0000 7f94ed24d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb3254850 0x559bb3261560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:00.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.118+0000 7f94ed24d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb325d430 0x559bb3261aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:00.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.118+0000 7f94ed24d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f94e41ab700 con 0x559bb3253f30 2024-04-21T23:10:00.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.118+0000 7f94ed24d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f94e41acb70 con 0x559bb325d430 2024-04-21T23:10:00.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.118+0000 7f94ed24d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f94e41abdd0 con 0x559bb3254850 2024-04-21T23:10:00.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.118+0000 7f94ed24d740 10 client.0 _ll_register_callbacks cb 0x7f94d400bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:00.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.118+0000 7f94e3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3253f30 0x559bb3261020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:00.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.118+0000 7f94e3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3253f30 0x559bb3261020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53514/0 (socket says 172.21.15.44:53514) 2024-04-21T23:10:00.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.118+0000 7f94e3fff640 1 -- 172.21.15.44:0/1737974918 learned_addr learned my addr 172.21.15.44:0/1737974918 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:00.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.118+0000 7f94e8c0a640 1 --2- 172.21.15.44:0/1737974918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb325d430 0x559bb3261aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:00.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.118+0000 7f94e37fe640 1 --2- 172.21.15.44:0/1737974918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb3254850 0x559bb3261560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:00.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.118+0000 7f94e3fff640 1 -- 172.21.15.44:0/1737974918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb3254850 msgr2=0x559bb3261560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:00.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.118+0000 7f94e3fff640 1 --2- 172.21.15.44:0/1737974918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb3254850 0x559bb3261560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.118+0000 7f94e3fff640 1 -- 172.21.15.44:0/1737974918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb325d430 msgr2=0x559bb3261aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:00.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.118+0000 7f94e3fff640 1 --2- 172.21.15.44:0/1737974918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb325d430 0x559bb3261aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.119+0000 7f94e3fff640 1 -- 172.21.15.44:0/1737974918 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bb32d32b0 con 0x559bb3253f30 2024-04-21T23:10:00.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.119+0000 7f94e8c0a640 1 --2- 172.21.15.44:0/1737974918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb325d430 0x559bb3261aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:10:00.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.119+0000 7f94e3fff640 1 --2- 172.21.15.44:0/1737974918 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3253f30 0x559bb3261020 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f94d4017040 tx=0x7f94d401b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1a5b359e4b04d8d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:00.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.119+0000 7f94c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:10:00.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.119+0000 7f94c37fe640 1 -- 172.21.15.44:0/1737974918 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d4045060 con 0x559bb3253f30 2024-04-21T23:10:00.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.119+0000 7f94c37fe640 1 -- 172.21.15.44:0/1737974918 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f94d4004230 con 0x559bb3253f30 2024-04-21T23:10:00.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.119+0000 7f94ed24d740 1 -- 172.21.15.44:0/1737974918 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94d401bd80 con 0x559bb3253f30 2024-04-21T23:10:00.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.119+0000 7f94c37fe640 1 -- 172.21.15.44:0/1737974918 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f94d404c050 con 0x559bb3253f30 2024-04-21T23:10:00.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.119+0000 7f94ed24d740 10 client.15879 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:00.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.119+0000 7f94ed24d740 1 -- 172.21.15.44:0/1737974918 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559bb32d45a0 con 0x559bb3253f30 2024-04-21T23:10:00.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.119+0000 7f94c37fe640 1 -- 172.21.15.44:0/1737974918 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94d401d7e0 con 0x559bb3253f30 2024-04-21T23:10:00.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.120+0000 7f94c37fe640 1 -- 172.21.15.44:0/1737974918 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f94d4003680 con 0x559bb3253f30 2024-04-21T23:10:00.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.120+0000 7f94ed24d740 10 client.15879 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:00.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.120+0000 7f94ed24d740 10 client.15879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:00.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.120+0000 7f94ed24d740 20 client.15879 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:00.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.120+0000 7f94ed24d740 1 --2- 172.21.15.44:0/1737974918 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x559bb326dc10 0x559bb32d6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:00.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.120+0000 7f94ed24d740 4 client.15879 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:10:00.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.120+0000 7f94ed24d740 1 -- 172.21.15.44:0/1737974918 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f94d401bd80 con 0x559bb326dc10 2024-04-21T23:10:00.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.120+0000 7f94e37fe640 1 --2- 172.21.15.44:0/1737974918 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x559bb326dc10 0x559bb32d6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:00.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.121+0000 7f94c37fe640 10 client.15879 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:00.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.121+0000 7f94e37fe640 1 --2- 172.21.15.44:0/1737974918 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x559bb326dc10 0x559bb32d6c30 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e38e46d81564bdfd server_cookie=5ad04e69d8584c64 in_seq=0 out_seq=0 2024-04-21T23:10:00.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.121+0000 7f94c37fe640 1 -- 172.21.15.44:0/1737974918 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f94d401bd80 con 0x559bb326dc10 2024-04-21T23:10:00.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.121+0000 7f94c37fe640 10 client.15879 handle_command_reply: tid=0 2024-04-21T23:10:00.123 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.121+0000 7f94ed24d740 1 client.15879 shutdown 2024-04-21T23:10:00.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.121+0000 7f94ed24d740 1 -- 172.21.15.44:0/1737974918 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3253f30 msgr2=0x559bb3261020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:00.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.121+0000 7f94ed24d740 1 --2- 172.21.15.44:0/1737974918 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3253f30 0x559bb3261020 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f94d4017040 tx=0x7f94d401b610 comp rx=0 tx=0).stop 2024-04-21T23:10:00.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.121+0000 7f94ed24d740 1 -- 172.21.15.44:0/1737974918 shutdown_connections 2024-04-21T23:10:00.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.121+0000 7f94ed24d740 1 --2- 172.21.15.44:0/1737974918 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x559bb326dc10 0x559bb32d6c30 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.121+0000 7f94c37fe640 0 client.15879 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:00.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.121+0000 7f94ed24d740 1 --2- 172.21.15.44:0/1737974918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559bb3254850 0x559bb3261560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.121+0000 7f94ed24d740 1 --2- 172.21.15.44:0/1737974918 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559bb325d430 0x559bb3261aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.121+0000 7f94ed24d740 1 --2- 172.21.15.44:0/1737974918 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559bb3253f30 0x559bb3261020 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.121+0000 7f94ed24d740 1 -- 172.21.15.44:0/1737974918 >> 172.21.15.44:0/1737974918 conn(0x559bb324e660 msgr2=0x559bb324d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:00.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.121+0000 7f94ed24d740 1 -- 172.21.15.44:0/1737974918 shutdown_connections 2024-04-21T23:10:00.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.121+0000 7f94ed24d740 1 -- 172.21.15.44:0/1737974918 wait complete. 2024-04-21T23:10:00.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.121+0000 7f94ed24d740 20 client.15879 trim_cache size 0 max 16384 2024-04-21T23:10:00.125 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:10:00.125 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:10:00.125 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-21T23:10:00.126 INFO:teuthology.orchestra.run.smithi044.stdout: "34792b20-b675-4fbe-8b80-345e21f4a8c5": { 2024-04-21T23:10:00.126 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-21T23:10:00.126 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "34792b20-b675-4fbe-8b80-345e21f4a8c5", 2024-04-21T23:10:00.126 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-04-21T23:10:00.126 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:10:00.126 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:10:00.126 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:10:00.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.122+0000 7f94ea66d640 1 -- 172.21.15.44:0/3873771358 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f94cc0ab6b0 msgr2=0x7f94cc0adb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:00.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.122+0000 7f94ea66d640 1 --2- 172.21.15.44:0/3873771358 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f94cc0ab6b0 0x7f94cc0adb70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f94d400bfd0 tx=0x7f94d400b4a0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.122+0000 7f94ea66d640 1 -- 172.21.15.44:0/3873771358 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94e4100e50 msgr2=0x7f94e419a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:00.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.122+0000 7f94ea66d640 1 --2- 172.21.15.44:0/3873771358 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94e4100e50 0x7f94e419a6f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f94d800e9f0 tx=0x7f94d800eeb0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.122+0000 7f94ea66d640 1 -- 172.21.15.44:0/3873771358 shutdown_connections 2024-04-21T23:10:00.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.122+0000 7f94ea66d640 1 --2- 172.21.15.44:0/3873771358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94e410e0e0 0x7f94e419ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.122+0000 7f94ea66d640 1 --2- 172.21.15.44:0/3873771358 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f94cc0ab6b0 0x7f94cc0adb70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.122+0000 7f94ea66d640 1 --2- 172.21.15.44:0/3873771358 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f94e4100e50 0x7f94e419a6f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.122+0000 7f94ea66d640 1 --2- 172.21.15.44:0/3873771358 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f94e4100530 0x7f94e419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:00.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.122+0000 7f94ea66d640 1 -- 172.21.15.44:0/3873771358 >> 172.21.15.44:0/3873771358 conn(0x7f94e40777f0 msgr2=0x7f94e410f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:00.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.122+0000 7f94ea66d640 1 -- 172.21.15.44:0/3873771358 shutdown_connections 2024-04-21T23:10:00.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:00.122+0000 7f94ea66d640 1 -- 172.21.15.44:0/3873771358 wait complete. 2024-04-21T23:10:00.133 INFO:tasks.cephfs.filesystem:scrub status for tag:34792b20-b675-4fbe-8b80-345e21f4a8c5 - {'path': '/', 'tag': '34792b20-b675-4fbe-8b80-345e21f4a8c5', 'options': 'recursive,force'} 2024-04-21T23:10:00.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:00 smithi044 ceph-mon[27934]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-21T23:10:00.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:00 smithi044 ceph-mon[27934]: scrub queued for path: / 2024-04-21T23:10:00.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:00 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:10:00.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:00 smithi044 ceph-mon[27934]: scrub summary: active paths [/] 2024-04-21T23:10:00.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:00 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:10:00.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:00 smithi044 ceph-mon[27934]: overall HEALTH_OK 2024-04-21T23:10:00.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:00 smithi069 ceph-mon[31816]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-21T23:10:00.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:00 smithi069 ceph-mon[31816]: scrub queued for path: / 2024-04-21T23:10:00.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:00 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:10:00.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:00 smithi069 ceph-mon[31816]: scrub summary: active paths [/] 2024-04-21T23:10:00.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:00 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:10:00.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:00 smithi069 ceph-mon[31816]: overall HEALTH_OK 2024-04-21T23:10:00.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:00 smithi055 ceph-mon[31117]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-21T23:10:00.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:00 smithi055 ceph-mon[31117]: scrub queued for path: / 2024-04-21T23:10:00.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:00 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:10:00.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:00 smithi055 ceph-mon[31117]: scrub summary: active paths [/] 2024-04-21T23:10:00.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:00 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:10:00.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:00 smithi055 ceph-mon[31117]: overall HEALTH_OK 2024-04-21T23:10:02.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:02 smithi055 ceph-mon[31117]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-21T23:10:02.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:02 smithi055 ceph-mon[31117]: scrub summary: idle 2024-04-21T23:10:02.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:02 smithi069 ceph-mon[31816]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-21T23:10:02.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:02 smithi069 ceph-mon[31816]: scrub summary: idle 2024-04-21T23:10:02.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:02 smithi044 ceph-mon[27934]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-21T23:10:02.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:02 smithi044 ceph-mon[27934]: scrub summary: idle 2024-04-21T23:10:03.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.147+0000 7f12efaed640 1 -- 172.21.15.44:0/29589949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12e8100fe0 msgr2=0x7f12e810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:03.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.147+0000 7f12efaed640 1 --2- 172.21.15.44:0/29589949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12e8100fe0 0x7f12e810d4f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f12e0009cd0 tx=0x7f12e002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.147 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.147+0000 7f12efaed640 1 -- 172.21.15.44:0/29589949 shutdown_connections 2024-04-21T23:10:03.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.147+0000 7f12efaed640 1 --2- 172.21.15.44:0/29589949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12e8100620 0x7f12e8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.147+0000 7f12efaed640 1 --2- 172.21.15.44:0/29589949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12e8100fe0 0x7f12e810d4f0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.147+0000 7f12efaed640 1 --2- 172.21.15.44:0/29589949 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12e8106630 0x7f12e8106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.147+0000 7f12efaed640 1 -- 172.21.15.44:0/29589949 >> 172.21.15.44:0/29589949 conn(0x7f12e8077760 msgr2=0x7f12e8077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:03.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.148+0000 7f12efaed640 1 -- 172.21.15.44:0/29589949 shutdown_connections 2024-04-21T23:10:03.148 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.148+0000 7f12efaed640 1 -- 172.21.15.44:0/29589949 wait complete. 2024-04-21T23:10:03.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.148+0000 7f12efaed640 1 Processor -- start 2024-04-21T23:10:03.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.148+0000 7f12efaed640 1 -- start start 2024-04-21T23:10:03.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.149+0000 7f12efaed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12e8100620 0x7f12e80fd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.149+0000 7f12efaed640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12e8100fe0 0x7f12e80fdb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.149+0000 7f12efaed640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12e8106630 0x7f12e80fe060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.149+0000 7f12efaed640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f12e8110a00 con 0x7f12e8100620 2024-04-21T23:10:03.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.149+0000 7f12efaed640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f12e8110880 con 0x7f12e8100fe0 2024-04-21T23:10:03.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.149+0000 7f12efaed640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f12e8110b80 con 0x7f12e8106630 2024-04-21T23:10:03.150 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.149+0000 7f12ed889640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12e8100620 0x7f12e80fd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.149+0000 7f12ed889640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12e8100620 0x7f12e80fd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36654/0 (socket says 172.21.15.44:36654) 2024-04-21T23:10:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.149+0000 7f12ed889640 1 -- 172.21.15.44:0/3693802425 learned_addr learned my addr 172.21.15.44:0/3693802425 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.149+0000 7f12ed088640 1 --2- 172.21.15.44:0/3693802425 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12e8100fe0 0x7f12e80fdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.149+0000 7f12ed889640 1 -- 172.21.15.44:0/3693802425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12e8106630 msgr2=0x7f12e80fe060 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T23:10:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.149+0000 7f12ee08a640 1 --2- 172.21.15.44:0/3693802425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12e8106630 0x7f12e80fe060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.149+0000 7f12ed889640 1 --2- 172.21.15.44:0/3693802425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12e8106630 0x7f12e80fe060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.149+0000 7f12ed889640 1 -- 172.21.15.44:0/3693802425 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12e8100fe0 msgr2=0x7f12e80fdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.149+0000 7f12ed889640 1 --2- 172.21.15.44:0/3693802425 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12e8100fe0 0x7f12e80fdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.149+0000 7f12ed889640 1 -- 172.21.15.44:0/3693802425 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e81a6590 con 0x7f12e8100620 2024-04-21T23:10:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.150+0000 7f12ed889640 1 --2- 172.21.15.44:0/3693802425 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12e8100620 0x7f12e80fd5e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f12e400e9f0 tx=0x7f12e400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.150+0000 7f12deffd640 1 -- 172.21.15.44:0/3693802425 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e400cd80 con 0x7f12e8100620 2024-04-21T23:10:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.150+0000 7f12deffd640 1 -- 172.21.15.44:0/3693802425 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12e4010910 con 0x7f12e8100620 2024-04-21T23:10:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.150+0000 7f12deffd640 1 -- 172.21.15.44:0/3693802425 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e4018990 con 0x7f12e8100620 2024-04-21T23:10:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.150+0000 7f12efaed640 1 -- 172.21.15.44:0/3693802425 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12e81a6880 con 0x7f12e8100620 2024-04-21T23:10:03.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.150+0000 7f12efaed640 1 -- 172.21.15.44:0/3693802425 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12e81a6d80 con 0x7f12e8100620 2024-04-21T23:10:03.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.154+0000 7f12f27b1740 1 Processor -- start 2024-04-21T23:10:03.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.154+0000 7f12f27b1740 1 -- start start 2024-04-21T23:10:03.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.154+0000 7f12deffd640 1 -- 172.21.15.44:0/3693802425 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f12e4010ac0 con 0x7f12e8100620 2024-04-21T23:10:03.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.154+0000 7f12f27b1740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec8b20 0x563818ec8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.154+0000 7f12f27b1740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec9440 0x563818ed1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.154+0000 7f12f27b1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563818ed2210 0x563818ed4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.154+0000 7f12f27b1740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563818e2cb70 con 0x563818ec9440 2024-04-21T23:10:03.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.154+0000 7f12f27b1740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x563818dd5470 con 0x563818ec8b20 2024-04-21T23:10:03.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.154+0000 7f12f27b1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563818e28b30 con 0x563818ed2210 2024-04-21T23:10:03.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.155+0000 7f12ed889640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec9440 0x563818ed1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:03.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.155+0000 7f12ed889640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec9440 0x563818ed1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36664/0 (socket says 172.21.15.44:36664) 2024-04-21T23:10:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.155+0000 7f12ed889640 1 -- 172.21.15.44:0/2236564856 learned_addr learned my addr 172.21.15.44:0/2236564856 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.155+0000 7f12ee08a640 1 --2- 172.21.15.44:0/2236564856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec8b20 0x563818ec8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.155+0000 7f12ee08a640 1 -- 172.21.15.44:0/2236564856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563818ed2210 msgr2=0x563818ed4600 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T23:10:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.155+0000 7f12ee08a640 1 --2- 172.21.15.44:0/2236564856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563818ed2210 0x563818ed4600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.155+0000 7f12ee08a640 1 -- 172.21.15.44:0/2236564856 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec9440 msgr2=0x563818ed1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.155+0000 7f12ee08a640 1 --2- 172.21.15.44:0/2236564856 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec9440 0x563818ed1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.155+0000 7f12ee08a640 1 -- 172.21.15.44:0/2236564856 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563818b41210 con 0x563818ec8b20 2024-04-21T23:10:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.155+0000 7f12ed889640 1 --2- 172.21.15.44:0/2236564856 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec9440 0x563818ed1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.155+0000 7f12ee08a640 1 --2- 172.21.15.44:0/2236564856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec8b20 0x563818ec8f00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f12e0037040 tx=0x7f12e0032340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66e5d4b35a48c688 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.156+0000 7f12d3fff640 1 -- 172.21.15.44:0/2236564856 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e0039050 con 0x563818ec8b20 2024-04-21T23:10:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.156+0000 7f12d3fff640 1 -- 172.21.15.44:0/2236564856 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12e003a040 con 0x563818ec8b20 2024-04-21T23:10:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.156+0000 7f12d3fff640 1 -- 172.21.15.44:0/2236564856 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e0003c70 con 0x563818ec8b20 2024-04-21T23:10:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.156+0000 7f12deffd640 1 --2- 172.21.15.44:0/3693802425 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f12d40a2e90 0x7f12d40a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.156+0000 7f12f27b1740 1 -- 172.21.15.44:0/2236564856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec8b20 msgr2=0x563818ec8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.156+0000 7f12f27b1740 1 --2- 172.21.15.44:0/2236564856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec8b20 0x563818ec8f00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f12e0037040 tx=0x7f12e0032340 comp rx=0 tx=0).stop 2024-04-21T23:10:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.157+0000 7f12f27b1740 1 -- 172.21.15.44:0/2236564856 shutdown_connections 2024-04-21T23:10:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.157+0000 7f12f27b1740 1 --2- 172.21.15.44:0/2236564856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563818ed2210 0x563818ed4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.157+0000 7f12f27b1740 1 --2- 172.21.15.44:0/2236564856 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec9440 0x563818ed1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.157+0000 7f12f27b1740 1 --2- 172.21.15.44:0/2236564856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec8b20 0x563818ec8f00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.157+0000 7f12f27b1740 1 -- 172.21.15.44:0/2236564856 >> 172.21.15.44:0/2236564856 conn(0x563818ec3440 msgr2=0x563818ec3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.157+0000 7f12ed889640 1 --2- 172.21.15.44:0/3693802425 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f12d40a2e90 0x7f12d40a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.157+0000 7f12deffd640 1 -- 172.21.15.44:0/3693802425 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f12e40cff40 con 0x7f12e8100620 2024-04-21T23:10:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.157+0000 7f12ed889640 1 --2- 172.21.15.44:0/3693802425 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f12d40a2e90 0x7f12d40a5350 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f12e40a4550 tx=0x7f12e40d39d0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.157+0000 7f12f27b1740 1 -- 172.21.15.44:0/2236564856 shutdown_connections 2024-04-21T23:10:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.157+0000 7f12f27b1740 1 -- 172.21.15.44:0/2236564856 wait complete. 2024-04-21T23:10:03.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.158+0000 7f12f27b1740 1 -- start start 2024-04-21T23:10:03.159 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.158+0000 7f12f27b1740 1 Processor -- start 2024-04-21T23:10:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.159+0000 7f12f27b1740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec8b20 0x563818ecf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.159+0000 7f12f27b1740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec9440 0x563818ecf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.159+0000 7f12f27b1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563818ed2210 0x563818ecfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.159+0000 7f12f27b1740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563818ed4de0 con 0x563818ec9440 2024-04-21T23:10:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.159+0000 7f12f27b1740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x563818de8a10 con 0x563818ec8b20 2024-04-21T23:10:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.159+0000 7f12f27b1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563818ed4f60 con 0x563818ed2210 2024-04-21T23:10:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.159+0000 7f12ed088640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec9440 0x563818ecf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.159+0000 7f12ed088640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec9440 0x563818ecf830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36668/0 (socket says 172.21.15.44:36668) 2024-04-21T23:10:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.159+0000 7f12ed088640 1 -- 172.21.15.44:0/2923051945 learned_addr learned my addr 172.21.15.44:0/2923051945 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.159+0000 7f12f27b1740 10 client.0 _ll_register_callbacks cb 0x563818e23b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.159+0000 7f12ed088640 1 -- 172.21.15.44:0/2923051945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563818ed2210 msgr2=0x563818ecfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.159+0000 7f12ed088640 1 --2- 172.21.15.44:0/2923051945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563818ed2210 0x563818ecfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.159+0000 7f12ed088640 1 -- 172.21.15.44:0/2923051945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec8b20 msgr2=0x563818ecf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.159+0000 7f12ee08a640 1 --2- 172.21.15.44:0/2923051945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec8b20 0x563818ecf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.159+0000 7f12ed088640 1 --2- 172.21.15.44:0/2923051945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec8b20 0x563818ecf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.159+0000 7f12ed088640 1 -- 172.21.15.44:0/2923051945 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563818c0d620 con 0x563818ec9440 2024-04-21T23:10:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.159+0000 7f12ee08a640 1 --2- 172.21.15.44:0/2923051945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec8b20 0x563818ecf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:10:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.160+0000 7f12ed088640 1 --2- 172.21.15.44:0/2923051945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec9440 0x563818ecf830 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f12d800be50 tx=0x7f12d80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd82a30ea1405d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.160+0000 7f12d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:10:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.160+0000 7f12d17fa640 1 -- 172.21.15.44:0/2923051945 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d8011850 con 0x563818ec9440 2024-04-21T23:10:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.160+0000 7f12d17fa640 1 -- 172.21.15.44:0/2923051945 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f12d80119f0 con 0x563818ec9440 2024-04-21T23:10:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.160+0000 7f12f27b1740 1 -- 172.21.15.44:0/2923051945 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12e003a040 con 0x563818ec9440 2024-04-21T23:10:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.160+0000 7f12d17fa640 1 -- 172.21.15.44:0/2923051945 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f12d8025050 con 0x563818ec9440 2024-04-21T23:10:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.160+0000 7f12f27b1740 10 client.15891 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.160+0000 7f12f27b1740 1 -- 172.21.15.44:0/2923051945 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563818ee1180 con 0x563818ec9440 2024-04-21T23:10:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.160+0000 7f12d17fa640 1 -- 172.21.15.44:0/2923051945 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12e003a040 con 0x563818ec9440 2024-04-21T23:10:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.160+0000 7f12d17fa640 1 -- 172.21.15.44:0/2923051945 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f12d801a380 con 0x563818ec9440 2024-04-21T23:10:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.160+0000 7f12f27b1740 10 client.15891 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.160+0000 7f12f27b1740 10 client.15891 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.161+0000 7f12f27b1740 20 client.15891 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.161+0000 7f12f27b1740 1 --2- 172.21.15.44:0/2923051945 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563818ee2690 0x563818ee4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.161+0000 7f12f27b1740 4 client.15891 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:10:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.161+0000 7f12f27b1740 1 -- 172.21.15.44:0/2923051945 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563818e3a0b0 con 0x563818ee2690 2024-04-21T23:10:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.161+0000 7f12ee08a640 1 --2- 172.21.15.44:0/2923051945 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563818ee2690 0x563818ee4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.161+0000 7f12ee08a640 1 --2- 172.21.15.44:0/2923051945 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563818ee2690 0x563818ee4b00 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f39c6e6f77e123b server_cookie=31594cf6f6335a6 in_seq=0 out_seq=0 2024-04-21T23:10:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.161+0000 7f12d17fa640 10 client.15891 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.162+0000 7f12d17fa640 1 -- 172.21.15.44:0/2923051945 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563818e3a0b0 con 0x563818ee2690 2024-04-21T23:10:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.162+0000 7f12d17fa640 10 client.15891 handle_command_reply: tid=0 2024-04-21T23:10:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.162+0000 7f12f27b1740 1 client.15891 shutdown 2024-04-21T23:10:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.162+0000 7f12f27b1740 1 -- 172.21.15.44:0/2923051945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec9440 msgr2=0x563818ecf830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.162+0000 7f12f27b1740 1 --2- 172.21.15.44:0/2923051945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec9440 0x563818ecf830 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f12d800be50 tx=0x7f12d80107a0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.162+0000 7f12f27b1740 1 -- 172.21.15.44:0/2923051945 shutdown_connections 2024-04-21T23:10:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.162+0000 7f12f27b1740 1 --2- 172.21.15.44:0/2923051945 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563818ee2690 0x563818ee4b00 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.162+0000 7f12d17fa640 0 client.15891 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.162+0000 7f12f27b1740 1 --2- 172.21.15.44:0/2923051945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563818ed2210 0x563818ecfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.162+0000 7f12f27b1740 1 --2- 172.21.15.44:0/2923051945 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec9440 0x563818ecf830 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.162+0000 7f12f27b1740 1 --2- 172.21.15.44:0/2923051945 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec8b20 0x563818ecf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.162+0000 7f12f27b1740 1 -- 172.21.15.44:0/2923051945 >> 172.21.15.44:0/2923051945 conn(0x563818ec3440 msgr2=0x563818ec5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.162+0000 7f12f27b1740 1 -- 172.21.15.44:0/2923051945 shutdown_connections 2024-04-21T23:10:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.163+0000 7f12f27b1740 1 -- 172.21.15.44:0/2923051945 wait complete. 2024-04-21T23:10:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.163+0000 7f12f27b1740 20 client.15891 trim_cache size 0 max 16384 2024-04-21T23:10:03.179 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.178+0000 7f12f27b1740 1 Processor -- start 2024-04-21T23:10:03.179 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.178+0000 7f12f27b1740 1 -- start start 2024-04-21T23:10:03.179 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.178+0000 7f12f27b1740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec8b20 0x563818ed1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.179 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.178+0000 7f12f27b1740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec9440 0x563818eddca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.179 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.179+0000 7f12f27b1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563818ed2210 0x563818ec4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.180 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.179+0000 7f12f27b1740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f12e81aee20 con 0x563818ec8b20 2024-04-21T23:10:03.180 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.179+0000 7f12f27b1740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f12e81b16a0 con 0x563818ec9440 2024-04-21T23:10:03.180 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.179+0000 7f12f27b1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f12e81af490 con 0x563818ed2210 2024-04-21T23:10:03.180 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.179+0000 7f12ee08a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec8b20 0x563818ed1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:03.180 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.179+0000 7f12ee08a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec8b20 0x563818ed1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36670/0 (socket says 172.21.15.44:36670) 2024-04-21T23:10:03.180 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.179+0000 7f12ee08a640 1 -- 172.21.15.44:0/2969722925 learned_addr learned my addr 172.21.15.44:0/2969722925 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:03.180 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.179+0000 7f12ee08a640 1 -- 172.21.15.44:0/2969722925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563818ed2210 msgr2=0x563818ec4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:03.180 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.179+0000 7f12ee08a640 1 --2- 172.21.15.44:0/2969722925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563818ed2210 0x563818ec4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.180 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.179+0000 7f12ee08a640 1 -- 172.21.15.44:0/2969722925 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec9440 msgr2=0x563818eddca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:03.180 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.179+0000 7f12ee08a640 1 --2- 172.21.15.44:0/2969722925 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec9440 0x563818eddca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.180 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.179+0000 7f12ee08a640 1 -- 172.21.15.44:0/2969722925 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563818ed4970 con 0x563818ec8b20 2024-04-21T23:10:03.180 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.179+0000 7f12ee08a640 1 --2- 172.21.15.44:0/2969722925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec8b20 0x563818ed1a20 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f12e0009cd0 tx=0x7f12e0005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c76267347d450faf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:03.180 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.179+0000 7f12d37fe640 1 -- 172.21.15.44:0/2969722925 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e004f020 con 0x563818ec8b20 2024-04-21T23:10:03.180 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.180+0000 7f12d37fe640 1 -- 172.21.15.44:0/2969722925 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12e004fe50 con 0x563818ec8b20 2024-04-21T23:10:03.181 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.180+0000 7f12d37fe640 1 -- 172.21.15.44:0/2969722925 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e0039030 con 0x563818ec8b20 2024-04-21T23:10:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.180+0000 7f12f27b1740 1 -- 172.21.15.44:0/2969722925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec8b20 msgr2=0x563818ed1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.180+0000 7f12f27b1740 1 --2- 172.21.15.44:0/2969722925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec8b20 0x563818ed1a20 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f12e0009cd0 tx=0x7f12e0005c50 comp rx=0 tx=0).stop 2024-04-21T23:10:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.181+0000 7f12f27b1740 1 -- 172.21.15.44:0/2969722925 shutdown_connections 2024-04-21T23:10:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.181+0000 7f12f27b1740 1 --2- 172.21.15.44:0/2969722925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563818ed2210 0x563818ec4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.181+0000 7f12f27b1740 1 --2- 172.21.15.44:0/2969722925 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec9440 0x563818eddca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.181+0000 7f12f27b1740 1 --2- 172.21.15.44:0/2969722925 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ec8b20 0x563818ed1a20 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.181+0000 7f12f27b1740 1 -- 172.21.15.44:0/2969722925 >> 172.21.15.44:0/2969722925 conn(0x563818ec3440 msgr2=0x563818ec4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.181+0000 7f12f27b1740 1 -- 172.21.15.44:0/2969722925 shutdown_connections 2024-04-21T23:10:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.181+0000 7f12f27b1740 1 -- 172.21.15.44:0/2969722925 wait complete. 2024-04-21T23:10:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.181+0000 7f12f27b1740 1 -- start start 2024-04-21T23:10:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.181+0000 7f12f27b1740 1 Processor -- start 2024-04-21T23:10:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.182+0000 7f12f27b1740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec8b20 0x563818ed5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.182+0000 7f12f27b1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563818ec9440 0x563818ed64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.182+0000 7f12f27b1740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ed2210 0x563818f47c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.182+0000 7f12f27b1740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f12e81afb60 con 0x563818ed2210 2024-04-21T23:10:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.182+0000 7f12f27b1740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f12e81b0fd0 con 0x563818ec8b20 2024-04-21T23:10:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.182+0000 7f12f27b1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f12e81b0230 con 0x563818ec9440 2024-04-21T23:10:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.182+0000 7f12f27b1740 10 client.0 _ll_register_callbacks cb 0x7f12d800b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.182+0000 7f12ee08a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ed2210 0x563818f47c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.182+0000 7f12ee08a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ed2210 0x563818f47c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36672/0 (socket says 172.21.15.44:36672) 2024-04-21T23:10:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.182+0000 7f12ee08a640 1 -- 172.21.15.44:0/2525219441 learned_addr learned my addr 172.21.15.44:0/2525219441 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.182+0000 7f12ed088640 1 --2- 172.21.15.44:0/2525219441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563818ec9440 0x563818ed64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.182+0000 7f12ee08a640 1 --2- 172.21.15.44:0/2525219441 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec8b20 0x563818ed5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.182+0000 7f12ee08a640 1 -- 172.21.15.44:0/2525219441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563818ec9440 msgr2=0x563818ed64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.182+0000 7f12ee08a640 1 --2- 172.21.15.44:0/2525219441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563818ec9440 0x563818ed64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.182+0000 7f12ee08a640 1 -- 172.21.15.44:0/2525219441 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec8b20 msgr2=0x563818ed5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.182+0000 7f12ee08a640 1 --2- 172.21.15.44:0/2525219441 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec8b20 0x563818ed5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.182+0000 7f12ee08a640 1 -- 172.21.15.44:0/2525219441 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563818f481c0 con 0x563818ed2210 2024-04-21T23:10:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.182+0000 7f12ee08a640 1 --2- 172.21.15.44:0/2525219441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ed2210 0x563818f47c80 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f12e0004af0 tx=0x7f12e0032340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57e8eb09199aec5f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.182+0000 7f12d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:10:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.183+0000 7f12d17fa640 1 -- 172.21.15.44:0/2525219441 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e004f040 con 0x563818ed2210 2024-04-21T23:10:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.183+0000 7f12d17fa640 1 -- 172.21.15.44:0/2525219441 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f12e00327e0 con 0x563818ed2210 2024-04-21T23:10:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.183+0000 7f12f27b1740 1 -- 172.21.15.44:0/2525219441 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12e004fe50 con 0x563818ed2210 2024-04-21T23:10:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.183+0000 7f12d17fa640 1 -- 172.21.15.44:0/2525219441 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f12e0039040 con 0x563818ed2210 2024-04-21T23:10:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.183+0000 7f12f27b1740 10 client.15897 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.183+0000 7f12f27b1740 1 -- 172.21.15.44:0/2525219441 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563818f49300 con 0x563818ed2210 2024-04-21T23:10:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.183+0000 7f12d17fa640 1 -- 172.21.15.44:0/2525219441 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12e0003e50 con 0x563818ed2210 2024-04-21T23:10:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.183+0000 7f12d17fa640 1 -- 172.21.15.44:0/2525219441 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f12e0004040 con 0x563818ed2210 2024-04-21T23:10:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.183+0000 7f12f27b1740 10 client.15897 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.183+0000 7f12f27b1740 10 client.15897 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.183+0000 7f12f27b1740 20 client.15897 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.183+0000 7f12f27b1740 1 --2- 172.21.15.44:0/2525219441 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563818f49520 0x563818f4b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.183+0000 7f12f27b1740 4 client.15897 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:10:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.183+0000 7f12f27b1740 1 -- 172.21.15.44:0/2525219441 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f12e004fe50 con 0x563818f49520 2024-04-21T23:10:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.183+0000 7f12ed088640 1 --2- 172.21.15.44:0/2525219441 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563818f49520 0x563818f4b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.184+0000 7f12d17fa640 10 client.15897 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.184+0000 7f12ed088640 1 --2- 172.21.15.44:0/2525219441 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563818f49520 0x563818f4b910 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ea51f38dab79fc0 server_cookie=cfefaa7dbf88e68e in_seq=0 out_seq=0 2024-04-21T23:10:03.191 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.190+0000 7f12d17fa640 1 -- 172.21.15.44:0/2525219441 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f12e004fe50 con 0x563818f49520 2024-04-21T23:10:03.191 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.191+0000 7f12d17fa640 10 client.15897 handle_command_reply: tid=0 2024-04-21T23:10:03.191 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.191+0000 7f12f27b1740 1 client.15897 shutdown 2024-04-21T23:10:03.191 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.191+0000 7f12f27b1740 1 -- 172.21.15.44:0/2525219441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ed2210 msgr2=0x563818f47c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:03.191 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.191+0000 7f12f27b1740 1 --2- 172.21.15.44:0/2525219441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ed2210 0x563818f47c80 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f12e0004af0 tx=0x7f12e0032340 comp rx=0 tx=0).stop 2024-04-21T23:10:03.191 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.191+0000 7f12f27b1740 1 -- 172.21.15.44:0/2525219441 shutdown_connections 2024-04-21T23:10:03.191 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.191+0000 7f12f27b1740 1 --2- 172.21.15.44:0/2525219441 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563818f49520 0x563818f4b910 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.192 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.191+0000 7f12f27b1740 1 --2- 172.21.15.44:0/2525219441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563818ec9440 0x563818ed64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.192 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.191+0000 7f12f27b1740 1 --2- 172.21.15.44:0/2525219441 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563818ed2210 0x563818f47c80 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.192 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.191+0000 7f12d17fa640 0 client.15897 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:03.192 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.191+0000 7f12f27b1740 1 --2- 172.21.15.44:0/2525219441 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563818ec8b20 0x563818ed5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.192 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.191+0000 7f12f27b1740 1 -- 172.21.15.44:0/2525219441 >> 172.21.15.44:0/2525219441 conn(0x563818ec3440 msgr2=0x563818ec50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:03.192 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.191+0000 7f12f27b1740 1 -- 172.21.15.44:0/2525219441 shutdown_connections 2024-04-21T23:10:03.192 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.191+0000 7f12f27b1740 1 -- 172.21.15.44:0/2525219441 wait complete. 2024-04-21T23:10:03.192 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.191+0000 7f12f27b1740 20 client.15897 trim_cache size 0 max 16384 2024-04-21T23:10:03.192 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:10:03.192 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:10:03.192 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:10:03.193 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:10:03.194 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.192+0000 7f12efaed640 1 -- 172.21.15.44:0/3693802425 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f12d40a2e90 msgr2=0x7f12d40a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:03.194 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.192+0000 7f12efaed640 1 --2- 172.21.15.44:0/3693802425 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f12d40a2e90 0x7f12d40a5350 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f12e40a4550 tx=0x7f12e40d39d0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.194 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.192+0000 7f12efaed640 1 -- 172.21.15.44:0/3693802425 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12e8100620 msgr2=0x7f12e80fd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:03.194 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.192+0000 7f12efaed640 1 --2- 172.21.15.44:0/3693802425 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12e8100620 0x7f12e80fd5e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f12e400e9f0 tx=0x7f12e400eeb0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.194 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.192+0000 7f12efaed640 1 -- 172.21.15.44:0/3693802425 shutdown_connections 2024-04-21T23:10:03.194 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.192+0000 7f12efaed640 1 --2- 172.21.15.44:0/3693802425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12e8106630 0x7f12e80fe060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.194 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.192+0000 7f12efaed640 1 --2- 172.21.15.44:0/3693802425 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f12d40a2e90 0x7f12d40a5350 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.194 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.192+0000 7f12efaed640 1 --2- 172.21.15.44:0/3693802425 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12e8100fe0 0x7f12e80fdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.194 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.192+0000 7f12efaed640 1 --2- 172.21.15.44:0/3693802425 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12e8100620 0x7f12e80fd5e0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:03.194 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.192+0000 7f12efaed640 1 -- 172.21.15.44:0/3693802425 >> 172.21.15.44:0/3693802425 conn(0x7f12e8077760 msgr2=0x7f12e8069290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:03.194 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.192+0000 7f12efaed640 1 -- 172.21.15.44:0/3693802425 shutdown_connections 2024-04-21T23:10:03.195 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:03.192+0000 7f12efaed640 1 -- 172.21.15.44:0/3693802425 wait complete. 2024-04-21T23:10:04.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:04 smithi044 ceph-mon[27934]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 18 op/s 2024-04-21T23:10:04.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:04 smithi055 ceph-mon[31117]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 18 op/s 2024-04-21T23:10:04.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:04 smithi069 ceph-mon[31816]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 18 op/s 2024-04-21T23:10:06.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:06 smithi044 ceph-mon[27934]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-21T23:10:06.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:06 smithi069 ceph-mon[31816]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-21T23:10:06.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:06 smithi055 ceph-mon[31117]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-21T23:10:08.516 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:08 smithi069 ceph-mon[31816]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-04-21T23:10:08.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:08 smithi044 ceph-mon[27934]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-04-21T23:10:08.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:08 smithi055 ceph-mon[31117]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-04-21T23:10:08.832 INFO:tasks.workunit.client.0.smithi044.stderr:Updating files: 95% (11946/12485) Updating files: 96% (11986/12485) Updating files: 97% (12111/12485) Updating files: 98% (12236/12485) Updating files: 99% (12361/12485) Updating files: 100% (12485/12485) Updating files: 100% (12485/12485), done. 2024-04-21T23:10:09.586 INFO:tasks.workunit.client.0.smithi044.stderr:Note: switching to '430e09df97c8fc7dc2b2ae424f68ed11366c540f'. 2024-04-21T23:10:09.587 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-04-21T23:10:09.587 INFO:tasks.workunit.client.0.smithi044.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-21T23:10:09.587 INFO:tasks.workunit.client.0.smithi044.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-21T23:10:09.587 INFO:tasks.workunit.client.0.smithi044.stderr:state without impacting any branches by switching back to a branch. 2024-04-21T23:10:09.587 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-04-21T23:10:09.587 INFO:tasks.workunit.client.0.smithi044.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-21T23:10:09.587 INFO:tasks.workunit.client.0.smithi044.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-21T23:10:09.587 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-04-21T23:10:09.587 INFO:tasks.workunit.client.0.smithi044.stderr: git switch -c 2024-04-21T23:10:09.587 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-04-21T23:10:09.587 INFO:tasks.workunit.client.0.smithi044.stderr:Or undo this operation with: 2024-04-21T23:10:09.587 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-04-21T23:10:09.587 INFO:tasks.workunit.client.0.smithi044.stderr: git switch - 2024-04-21T23:10:09.587 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-04-21T23:10:09.587 INFO:tasks.workunit.client.0.smithi044.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-21T23:10:09.588 INFO:tasks.workunit.client.0.smithi044.stderr: 2024-04-21T23:10:09.588 INFO:tasks.workunit.client.0.smithi044.stderr:HEAD is now at 430e09df97c Merge pull request #57017 from zdover23/wip-doc-2024-04-21-backport-56904-to-squid 2024-04-21T23:10:09.590 DEBUG:teuthology.orchestra.run.smithi044:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-21T23:10:09.651 INFO:tasks.workunit.client.0.smithi044.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-21T23:10:09.653 INFO:tasks.workunit.client.0.smithi044.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-21T23:10:09.654 INFO:tasks.workunit.client.0.smithi044.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-21T23:10:10.316 INFO:tasks.workunit.client.0.smithi044.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-21T23:10:10.373 INFO:tasks.workunit.client.0.smithi044.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-21T23:10:10.414 INFO:tasks.workunit.client.0.smithi044.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-21T23:10:10.416 INFO:tasks.workunit.client.0.smithi044.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-21T23:10:10.416 INFO:tasks.workunit.client.0.smithi044.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-21T23:10:10.455 INFO:tasks.workunit.client.0.smithi044.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-21T23:10:10.459 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T23:10:10.459 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-21T23:10:10.515 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-04-21T23:10:10.516 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-04-21T23:10:10.517 DEBUG:teuthology.orchestra.run.smithi044:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=430e09df97c8fc7dc2b2ae424f68ed11366c540f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-04-21T23:10:10.517 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:10 smithi055 ceph-mon[31117]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-04-21T23:10:10.577 INFO:tasks.workunit.client.0.smithi044.stderr:+ dbench 1 2024-04-21T23:10:10.589 INFO:tasks.workunit.client.0.smithi044.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-21T23:10:10.589 INFO:tasks.workunit.client.0.smithi044.stdout: 2024-04-21T23:10:10.589 INFO:tasks.workunit.client.0.smithi044.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-21T23:10:10.590 INFO:tasks.workunit.client.0.smithi044.stdout:0 of 1 processes prepared for launch 0 sec 2024-04-21T23:10:10.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:10 smithi044 ceph-mon[27934]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-04-21T23:10:10.690 INFO:tasks.workunit.client.0.smithi044.stdout:1 of 1 processes prepared for launch 0 sec 2024-04-21T23:10:10.690 INFO:tasks.workunit.client.0.smithi044.stdout:releasing clients 2024-04-21T23:10:10.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:10 smithi069 ceph-mon[31816]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-04-21T23:10:11.615 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:11 smithi044 ceph-mon[27934]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-04-21T23:10:11.690 INFO:tasks.workunit.client.0.smithi044.stdout: 1 3365 35.44 MB/sec warmup 1 sec latency 8.698 ms 2024-04-21T23:10:11.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:11 smithi055 ceph-mon[31117]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-04-21T23:10:11.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:11 smithi069 ceph-mon[31816]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-04-21T23:10:12.690 INFO:tasks.workunit.client.0.smithi044.stdout: 1 6154 24.06 MB/sec warmup 2 sec latency 15.411 ms 2024-04-21T23:10:13.690 INFO:tasks.workunit.client.0.smithi044.stdout: 1 8761 21.48 MB/sec warmup 3 sec latency 15.175 ms 2024-04-21T23:10:13.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:13 smithi044 ceph-mon[27934]: pgmap v44: 129 pgs: 129 active+clean; 2.8 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 159 KiB/s wr, 36 op/s 2024-04-21T23:10:13.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:13 smithi055 ceph-mon[31117]: pgmap v44: 129 pgs: 129 active+clean; 2.8 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 159 KiB/s wr, 36 op/s 2024-04-21T23:10:13.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:13 smithi069 ceph-mon[31816]: pgmap v44: 129 pgs: 129 active+clean; 2.8 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 159 KiB/s wr, 36 op/s 2024-04-21T23:10:14.690 INFO:tasks.workunit.client.0.smithi044.stdout: 1 11561 19.76 MB/sec warmup 4 sec latency 21.980 ms 2024-04-21T23:10:15.690 INFO:tasks.workunit.client.0.smithi044.stdout: 1 14862 19.51 MB/sec warmup 5 sec latency 10.817 ms 2024-04-21T23:10:15.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:15 smithi044 ceph-mon[27934]: pgmap v45: 129 pgs: 129 active+clean; 2.8 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 137 KiB/s wr, 19 op/s 2024-04-21T23:10:15.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:15 smithi055 ceph-mon[31117]: pgmap v45: 129 pgs: 129 active+clean; 2.8 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 137 KiB/s wr, 19 op/s 2024-04-21T23:10:15.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:15 smithi069 ceph-mon[31816]: pgmap v45: 129 pgs: 129 active+clean; 2.8 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 137 KiB/s wr, 19 op/s 2024-04-21T23:10:16.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.284+0000 7f0252ba1640 1 -- 172.21.15.44:0/1865483725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f024c10bd80 msgr2=0x7f024c10e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:16.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.284+0000 7f0252ba1640 1 --2- 172.21.15.44:0/1865483725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f024c10bd80 0x7f024c10e1b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f0244009ca0 tx=0x7f024402c180 comp rx=0 tx=0).stop 2024-04-21T23:10:16.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.284+0000 7f0252ba1640 1 -- 172.21.15.44:0/1865483725 shutdown_connections 2024-04-21T23:10:16.286 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.284+0000 7f0252ba1640 1 --2- 172.21.15.44:0/1865483725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f024c1032d0 0x7f024c10b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.287 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.284+0000 7f0252ba1640 1 --2- 172.21.15.44:0/1865483725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f024c10bd80 0x7f024c10e1b0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.287 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.284+0000 7f0252ba1640 1 --2- 172.21.15.44:0/1865483725 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f024c1029b0 0x7f024c102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.287 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.284+0000 7f0252ba1640 1 -- 172.21.15.44:0/1865483725 >> 172.21.15.44:0/1865483725 conn(0x7f024c077760 msgr2=0x7f024c077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:16.287 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.284+0000 7f0252ba1640 1 -- 172.21.15.44:0/1865483725 shutdown_connections 2024-04-21T23:10:16.287 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.284+0000 7f0252ba1640 1 -- 172.21.15.44:0/1865483725 wait complete. 2024-04-21T23:10:16.287 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.285+0000 7f0252ba1640 1 Processor -- start 2024-04-21T23:10:16.287 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.285+0000 7f0252ba1640 1 -- start start 2024-04-21T23:10:16.287 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.285+0000 7f0252ba1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f024c1029b0 0x7f024c198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:16.287 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.285+0000 7f0252ba1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f024c1032d0 0x7f024c198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:16.287 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.285+0000 7f0252ba1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f024c10bd80 0x7f024c19c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:16.287 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.285+0000 7f0252ba1640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f024c111b10 con 0x7f024c1032d0 2024-04-21T23:10:16.287 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.285+0000 7f0252ba1640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f024c111990 con 0x7f024c1029b0 2024-04-21T23:10:16.287 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.285+0000 7f0252ba1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f024c111c90 con 0x7f024c10bd80 2024-04-21T23:10:16.287 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.285+0000 7f024bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f024c1032d0 0x7f024c198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:16.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.285+0000 7f024bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f024c1032d0 0x7f024c198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51222/0 (socket says 172.21.15.44:51222) 2024-04-21T23:10:16.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.285+0000 7f024bfff640 1 -- 172.21.15.44:0/1646169094 learned_addr learned my addr 172.21.15.44:0/1646169094 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:16.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.285+0000 7f025093d640 1 --2- 172.21.15.44:0/1646169094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f024c1029b0 0x7f024c198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:16.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.286+0000 7f025113e640 1 --2- 172.21.15.44:0/1646169094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f024c10bd80 0x7f024c19c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:16.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.286+0000 7f024bfff640 1 -- 172.21.15.44:0/1646169094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f024c10bd80 msgr2=0x7f024c19c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:16.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.286+0000 7f024bfff640 1 --2- 172.21.15.44:0/1646169094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f024c10bd80 0x7f024c19c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.286+0000 7f024bfff640 1 -- 172.21.15.44:0/1646169094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f024c1029b0 msgr2=0x7f024c198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:16.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.286+0000 7f024bfff640 1 --2- 172.21.15.44:0/1646169094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f024c1029b0 0x7f024c198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.286+0000 7f024bfff640 1 -- 172.21.15.44:0/1646169094 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f024c19d000 con 0x7f024c1032d0 2024-04-21T23:10:16.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.286+0000 7f024bfff640 1 --2- 172.21.15.44:0/1646169094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f024c1032d0 0x7f024c198550 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f023800e9f0 tx=0x7f023800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:16.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.286+0000 7f0249ffb640 1 -- 172.21.15.44:0/1646169094 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023800cd80 con 0x7f024c1032d0 2024-04-21T23:10:16.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.286+0000 7f0249ffb640 1 -- 172.21.15.44:0/1646169094 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0238005500 con 0x7f024c1032d0 2024-04-21T23:10:16.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.286+0000 7f0252ba1640 1 -- 172.21.15.44:0/1646169094 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f024c19d2f0 con 0x7f024c1032d0 2024-04-21T23:10:16.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.286+0000 7f0249ffb640 1 -- 172.21.15.44:0/1646169094 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0238010690 con 0x7f024c1032d0 2024-04-21T23:10:16.288 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.286+0000 7f0252ba1640 1 -- 172.21.15.44:0/1646169094 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f024c1a4b80 con 0x7f024c1032d0 2024-04-21T23:10:16.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.288+0000 7f0249ffb640 1 -- 172.21.15.44:0/1646169094 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f02380046c0 con 0x7f024c1032d0 2024-04-21T23:10:16.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.289+0000 7f025578f740 1 Processor -- start 2024-04-21T23:10:16.295 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.289+0000 7f025578f740 1 -- start start 2024-04-21T23:10:16.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.289+0000 7f0249ffb640 1 --2- 172.21.15.44:0/1646169094 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f023c0a2e90 0x7f023c0a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:16.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.289+0000 7f025113e640 1 --2- 172.21.15.44:0/1646169094 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f023c0a2e90 0x7f023c0a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:16.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.289+0000 7f0249ffb640 1 -- 172.21.15.44:0/1646169094 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f02380cfe00 con 0x7f024c1032d0 2024-04-21T23:10:16.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.289+0000 7f025578f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecae4b20 0x5616ecae4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:16.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.289+0000 7f025578f740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecae5440 0x5616ecaedcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:16.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.289+0000 7f025578f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecaee210 0x5616ecaf0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:16.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.289+0000 7f025578f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5616eca48b70 con 0x5616ecaee210 2024-04-21T23:10:16.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.289+0000 7f025578f740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5616ec9f1470 con 0x5616ecae5440 2024-04-21T23:10:16.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.289+0000 7f025578f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5616eca44b30 con 0x5616ecae4b20 2024-04-21T23:10:16.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.289+0000 7f025093d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecae4b20 0x5616ecae4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:16.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.289+0000 7f025113e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecaee210 0x5616ecaf0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:16.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.290+0000 7f024bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecae5440 0x5616ecaedcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:16.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.290+0000 7f025093d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecae4b20 0x5616ecae4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:35918/0 (socket says 172.21.15.44:35918) 2024-04-21T23:10:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.290+0000 7f025093d640 1 -- 172.21.15.44:0/1562058086 learned_addr learned my addr 172.21.15.44:0/1562058086 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.290+0000 7f025093d640 1 -- 172.21.15.44:0/1562058086 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecae5440 msgr2=0x5616ecaedcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.290+0000 7f025093d640 1 --2- 172.21.15.44:0/1562058086 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecae5440 0x5616ecaedcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.290+0000 7f025093d640 1 -- 172.21.15.44:0/1562058086 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecaee210 msgr2=0x5616ecaf0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.290+0000 7f025093d640 1 --2- 172.21.15.44:0/1562058086 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecaee210 0x5616ecaf0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.290+0000 7f025093d640 1 -- 172.21.15.44:0/1562058086 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616ec75d210 con 0x5616ecae4b20 2024-04-21T23:10:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.290+0000 7f024bfff640 1 --2- 172.21.15.44:0/1562058086 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecae5440 0x5616ecaedcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.290+0000 7f025113e640 1 --2- 172.21.15.44:0/1646169094 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f023c0a2e90 0x7f023c0a5350 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0244005cc0 tx=0x7f0244003a70 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.290+0000 7f025093d640 1 --2- 172.21.15.44:0/1562058086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecae4b20 0x5616ecae4f00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0240009cf0 tx=0x7f0240011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b50b2c9f0d2d320 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.290+0000 7f0236ffd640 1 -- 172.21.15.44:0/1562058086 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0240004230 con 0x5616ecae4b20 2024-04-21T23:10:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.290+0000 7f0236ffd640 1 -- 172.21.15.44:0/1562058086 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02400043d0 con 0x5616ecae4b20 2024-04-21T23:10:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.290+0000 7f0236ffd640 1 -- 172.21.15.44:0/1562058086 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0240019840 con 0x5616ecae4b20 2024-04-21T23:10:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.291+0000 7f025578f740 1 -- 172.21.15.44:0/1562058086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecae4b20 msgr2=0x5616ecae4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.291+0000 7f025578f740 1 --2- 172.21.15.44:0/1562058086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecae4b20 0x5616ecae4f00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0240009cf0 tx=0x7f0240011810 comp rx=0 tx=0).stop 2024-04-21T23:10:16.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.291+0000 7f025578f740 1 -- 172.21.15.44:0/1562058086 shutdown_connections 2024-04-21T23:10:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.291+0000 7f025578f740 1 --2- 172.21.15.44:0/1562058086 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecaee210 0x5616ecaf0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.291+0000 7f025578f740 1 --2- 172.21.15.44:0/1562058086 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecae5440 0x5616ecaedcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.291+0000 7f025578f740 1 --2- 172.21.15.44:0/1562058086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecae4b20 0x5616ecae4f00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.291+0000 7f025578f740 1 -- 172.21.15.44:0/1562058086 >> 172.21.15.44:0/1562058086 conn(0x5616ecadf440 msgr2=0x5616ecadf820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.291+0000 7f025578f740 1 -- 172.21.15.44:0/1562058086 shutdown_connections 2024-04-21T23:10:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.291+0000 7f025578f740 1 -- 172.21.15.44:0/1562058086 wait complete. 2024-04-21T23:10:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.291+0000 7f025578f740 1 -- start start 2024-04-21T23:10:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.291+0000 7f025578f740 1 Processor -- start 2024-04-21T23:10:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f025578f740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecae4b20 0x5616ecaeb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f025578f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecae5440 0x5616ecaeb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f025578f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecaee210 0x5616ecaebca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f025578f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5616ecaf0e40 con 0x5616ecae5440 2024-04-21T23:10:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f025578f740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5616eca04a10 con 0x5616ecae4b20 2024-04-21T23:10:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f025578f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5616ecaf0fc0 con 0x5616ecaee210 2024-04-21T23:10:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f025578f740 10 client.0 _ll_register_callbacks cb 0x5616eca3fb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f024bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecae5440 0x5616ecaeb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f025113e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecaee210 0x5616ecaebca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f025093d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecae4b20 0x5616ecaeb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f024bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecae5440 0x5616ecaeb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51246/0 (socket says 172.21.15.44:51246) 2024-04-21T23:10:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f025113e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecaee210 0x5616ecaebca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:35922/0 (socket says 172.21.15.44:35922) 2024-04-21T23:10:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f024bfff640 1 -- 172.21.15.44:0/3824480291 learned_addr learned my addr 172.21.15.44:0/3824480291 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f024bfff640 1 -- 172.21.15.44:0/3824480291 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecaee210 msgr2=0x5616ecaebca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f024bfff640 1 --2- 172.21.15.44:0/3824480291 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecaee210 0x5616ecaebca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f024bfff640 1 -- 172.21.15.44:0/3824480291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecae4b20 msgr2=0x5616ecaeb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f024bfff640 1 --2- 172.21.15.44:0/3824480291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecae4b20 0x5616ecaeb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f024bfff640 1 -- 172.21.15.44:0/3824480291 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616ec829620 con 0x5616ecae5440 2024-04-21T23:10:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f025093d640 1 --2- 172.21.15.44:0/3824480291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecae4b20 0x5616ecaeb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f024bfff640 1 --2- 172.21.15.44:0/3824480291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecae5440 0x5616ecaeb760 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f02380986a0 tx=0x7f023809b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4a517b892e00180 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.292+0000 7f022bfff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:10:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.293+0000 7f022bfff640 1 -- 172.21.15.44:0/3824480291 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02380a4940 con 0x5616ecae5440 2024-04-21T23:10:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.293+0000 7f022bfff640 1 -- 172.21.15.44:0/3824480291 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f02380cd2b0 con 0x5616ecae5440 2024-04-21T23:10:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.293+0000 7f025578f740 1 -- 172.21.15.44:0/3824480291 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02400043d0 con 0x5616ecae5440 2024-04-21T23:10:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.293+0000 7f022bfff640 1 -- 172.21.15.44:0/3824480291 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f02380cd4b0 con 0x5616ecae5440 2024-04-21T23:10:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.293+0000 7f025578f740 10 client.15909 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.293+0000 7f025578f740 1 -- 172.21.15.44:0/3824480291 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5616ecafd080 con 0x5616ecae5440 2024-04-21T23:10:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.293+0000 7f022bfff640 1 -- 172.21.15.44:0/3824480291 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02400043d0 con 0x5616ecae5440 2024-04-21T23:10:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.293+0000 7f022bfff640 1 -- 172.21.15.44:0/3824480291 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f02380b3070 con 0x5616ecae5440 2024-04-21T23:10:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.293+0000 7f025578f740 10 client.15909 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.293+0000 7f025578f740 10 client.15909 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.293+0000 7f025578f740 20 client.15909 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.293+0000 7f025578f740 1 --2- 172.21.15.44:0/3824480291 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5616ecafe4f0 0x5616ecb00920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.293+0000 7f025578f740 4 client.15909 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:10:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.293+0000 7f025578f740 1 -- 172.21.15.44:0/3824480291 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616eca560b0 con 0x5616ecafe4f0 2024-04-21T23:10:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.293+0000 7f025093d640 1 --2- 172.21.15.44:0/3824480291 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5616ecafe4f0 0x5616ecb00920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.294+0000 7f025093d640 1 --2- 172.21.15.44:0/3824480291 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5616ecafe4f0 0x5616ecb00920 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9191669d7bb83b2 server_cookie=b91ed7ec4b6b3af1 in_seq=0 out_seq=0 2024-04-21T23:10:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.294+0000 7f022bfff640 10 client.15909 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.295+0000 7f022bfff640 1 -- 172.21.15.44:0/3824480291 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5616eca560b0 con 0x5616ecafe4f0 2024-04-21T23:10:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.295+0000 7f022bfff640 10 client.15909 handle_command_reply: tid=0 2024-04-21T23:10:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.295+0000 7f025578f740 1 client.15909 shutdown 2024-04-21T23:10:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.295+0000 7f025578f740 1 -- 172.21.15.44:0/3824480291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecae5440 msgr2=0x5616ecaeb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.295+0000 7f025578f740 1 --2- 172.21.15.44:0/3824480291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecae5440 0x5616ecaeb760 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f02380986a0 tx=0x7f023809b760 comp rx=0 tx=0).stop 2024-04-21T23:10:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.295+0000 7f025578f740 1 -- 172.21.15.44:0/3824480291 shutdown_connections 2024-04-21T23:10:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.295+0000 7f025578f740 1 --2- 172.21.15.44:0/3824480291 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5616ecafe4f0 0x5616ecb00920 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.295+0000 7f025578f740 1 --2- 172.21.15.44:0/3824480291 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecaee210 0x5616ecaebca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.295+0000 7f022bfff640 0 client.15909 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.295+0000 7f025578f740 1 --2- 172.21.15.44:0/3824480291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecae5440 0x5616ecaeb760 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.295+0000 7f025578f740 1 --2- 172.21.15.44:0/3824480291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecae4b20 0x5616ecaeb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.295+0000 7f025578f740 1 -- 172.21.15.44:0/3824480291 >> 172.21.15.44:0/3824480291 conn(0x5616ecadf440 msgr2=0x5616ecae1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:16.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.295+0000 7f025578f740 1 -- 172.21.15.44:0/3824480291 shutdown_connections 2024-04-21T23:10:16.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.295+0000 7f025578f740 1 -- 172.21.15.44:0/3824480291 wait complete. 2024-04-21T23:10:16.302 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.295+0000 7f025578f740 20 client.15909 trim_cache size 0 max 16384 2024-04-21T23:10:16.316 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.311+0000 7f025578f740 1 Processor -- start 2024-04-21T23:10:16.316 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.311+0000 7f025578f740 1 -- start start 2024-04-21T23:10:16.316 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.311+0000 7f025578f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecae4b20 0x5616ecaed950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:16.316 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.311+0000 7f025578f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecae5440 0x5616ecaf9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:16.316 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.311+0000 7f025578f740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecaee210 0x5616ecb06230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:16.316 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.311+0000 7f025578f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f024c1acc00 con 0x5616ecae5440 2024-04-21T23:10:16.316 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.311+0000 7f025578f740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f024c1af480 con 0x5616ecaee210 2024-04-21T23:10:16.316 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.311+0000 7f025578f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f024c1ad270 con 0x5616ecae4b20 2024-04-21T23:10:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.311+0000 7f024bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecae5440 0x5616ecaf9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.311+0000 7f024bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecae5440 0x5616ecaf9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51260/0 (socket says 172.21.15.44:51260) 2024-04-21T23:10:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.311+0000 7f024bfff640 1 -- 172.21.15.44:0/949929879 learned_addr learned my addr 172.21.15.44:0/949929879 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.311+0000 7f025113e640 1 --2- 172.21.15.44:0/949929879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecaee210 0x5616ecb06230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.311+0000 7f025093d640 1 --2- 172.21.15.44:0/949929879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecae4b20 0x5616ecaed950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.312+0000 7f024bfff640 1 -- 172.21.15.44:0/949929879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecae4b20 msgr2=0x5616ecaed950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.312+0000 7f024bfff640 1 --2- 172.21.15.44:0/949929879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecae4b20 0x5616ecaed950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.312+0000 7f024bfff640 1 -- 172.21.15.44:0/949929879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecaee210 msgr2=0x5616ecb06230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.312+0000 7f024bfff640 1 --2- 172.21.15.44:0/949929879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecaee210 0x5616ecb06230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.312+0000 7f024bfff640 1 -- 172.21.15.44:0/949929879 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616ecae9770 con 0x5616ecae5440 2024-04-21T23:10:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.312+0000 7f025093d640 1 --2- 172.21.15.44:0/949929879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecae4b20 0x5616ecaed950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:10:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.312+0000 7f025113e640 1 --2- 172.21.15.44:0/949929879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecaee210 0x5616ecb06230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:10:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.312+0000 7f024bfff640 1 --2- 172.21.15.44:0/949929879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecae5440 0x5616ecaf9d00 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f023800c6a0 tx=0x7f0238097f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2c7608a340c92c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.312+0000 7f02367fc640 1 -- 172.21.15.44:0/949929879 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02380ccf10 con 0x5616ecae5440 2024-04-21T23:10:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.312+0000 7f02367fc640 1 -- 172.21.15.44:0/949929879 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02380ab020 con 0x5616ecae5440 2024-04-21T23:10:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.312+0000 7f02367fc640 1 -- 172.21.15.44:0/949929879 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02380a4940 con 0x5616ecae5440 2024-04-21T23:10:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.312+0000 7f025578f740 1 -- 172.21.15.44:0/949929879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecae5440 msgr2=0x5616ecaf9d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.312+0000 7f025578f740 1 --2- 172.21.15.44:0/949929879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecae5440 0x5616ecaf9d00 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f023800c6a0 tx=0x7f0238097f90 comp rx=0 tx=0).stop 2024-04-21T23:10:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.313+0000 7f025578f740 1 -- 172.21.15.44:0/949929879 shutdown_connections 2024-04-21T23:10:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.313+0000 7f025578f740 1 --2- 172.21.15.44:0/949929879 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecaee210 0x5616ecb06230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.313+0000 7f025578f740 1 --2- 172.21.15.44:0/949929879 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecae5440 0x5616ecaf9d00 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.313+0000 7f025578f740 1 --2- 172.21.15.44:0/949929879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecae4b20 0x5616ecaed950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.313+0000 7f025578f740 1 -- 172.21.15.44:0/949929879 >> 172.21.15.44:0/949929879 conn(0x5616ecadf440 msgr2=0x5616ecada220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.313+0000 7f025578f740 1 -- 172.21.15.44:0/949929879 shutdown_connections 2024-04-21T23:10:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.313+0000 7f025578f740 1 -- 172.21.15.44:0/949929879 wait complete. 2024-04-21T23:10:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.313+0000 7f025578f740 1 -- start start 2024-04-21T23:10:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.313+0000 7f025578f740 1 Processor -- start 2024-04-21T23:10:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.313+0000 7f025578f740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecae4b20 0x5616ecae7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.313+0000 7f025578f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecae5440 0x5616ecae8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.313+0000 7f025578f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecaee210 0x5616ecae8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.313+0000 7f025578f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f024c1ad940 con 0x5616ecaee210 2024-04-21T23:10:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.313+0000 7f025578f740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f024c1aedb0 con 0x5616ecae4b20 2024-04-21T23:10:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.313+0000 7f025578f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f024c1ae010 con 0x5616ecae5440 2024-04-21T23:10:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.313+0000 7f025578f740 10 client.0 _ll_register_callbacks cb 0x7f023800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.313+0000 7f025113e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecaee210 0x5616ecae8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.313+0000 7f025113e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecaee210 0x5616ecae8870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51264/0 (socket says 172.21.15.44:51264) 2024-04-21T23:10:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.313+0000 7f025113e640 1 -- 172.21.15.44:0/2185425319 learned_addr learned my addr 172.21.15.44:0/2185425319 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.313+0000 7f025093d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecae4b20 0x5616ecae7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.314+0000 7f024bfff640 1 --2- 172.21.15.44:0/2185425319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecae5440 0x5616ecae8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.314+0000 7f025113e640 1 -- 172.21.15.44:0/2185425319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecae5440 msgr2=0x5616ecae8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.314+0000 7f025113e640 1 --2- 172.21.15.44:0/2185425319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecae5440 0x5616ecae8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.314+0000 7f025113e640 1 -- 172.21.15.44:0/2185425319 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecae4b20 msgr2=0x5616ecae7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:16.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.314+0000 7f025113e640 1 --2- 172.21.15.44:0/2185425319 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecae4b20 0x5616ecae7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.314+0000 7f025113e640 1 -- 172.21.15.44:0/2185425319 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616ecb63aa0 con 0x5616ecaee210 2024-04-21T23:10:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.314+0000 7f024bfff640 1 --2- 172.21.15.44:0/2185425319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecae5440 0x5616ecae8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.314+0000 7f025113e640 1 --2- 172.21.15.44:0/2185425319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecaee210 0x5616ecae8870 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f024c06a040 tx=0x7f024402c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e33b2f7bf86be910 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.314+0000 7f022bfff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:10:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.314+0000 7f022bfff640 1 -- 172.21.15.44:0/2185425319 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0244039040 con 0x5616ecaee210 2024-04-21T23:10:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.314+0000 7f022bfff640 1 -- 172.21.15.44:0/2185425319 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f0244034d20 con 0x5616ecaee210 2024-04-21T23:10:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.314+0000 7f025578f740 1 -- 172.21.15.44:0/2185425319 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02380ab020 con 0x5616ecaee210 2024-04-21T23:10:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.314+0000 7f022bfff640 1 -- 172.21.15.44:0/2185425319 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f024405f050 con 0x5616ecaee210 2024-04-21T23:10:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.314+0000 7f025578f740 10 client.15915 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.314+0000 7f025578f740 1 -- 172.21.15.44:0/2185425319 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5616ecb64c20 con 0x5616ecaee210 2024-04-21T23:10:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.315+0000 7f022bfff640 1 -- 172.21.15.44:0/2185425319 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02380ab020 con 0x5616ecaee210 2024-04-21T23:10:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.315+0000 7f022bfff640 1 -- 172.21.15.44:0/2185425319 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f024403f040 con 0x5616ecaee210 2024-04-21T23:10:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.315+0000 7f025578f740 10 client.15915 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.315+0000 7f025578f740 10 client.15915 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.315+0000 7f025578f740 20 client.15915 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.315+0000 7f025578f740 1 --2- 172.21.15.44:0/2185425319 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5616ecb653a0 0x5616ecb677c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.315+0000 7f025578f740 4 client.15915 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:10:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.315+0000 7f025578f740 1 -- 172.21.15.44:0/2185425319 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5616eca50400 con 0x5616ecb653a0 2024-04-21T23:10:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.315+0000 7f025093d640 1 --2- 172.21.15.44:0/2185425319 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5616ecb653a0 0x5616ecb677c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.315+0000 7f022bfff640 10 client.15915 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.315+0000 7f025093d640 1 --2- 172.21.15.44:0/2185425319 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5616ecb653a0 0x5616ecb677c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fce986209b3ed42 server_cookie=bd06e6e02c266d7b in_seq=0 out_seq=0 2024-04-21T23:10:16.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.507+0000 7f022bfff640 1 -- 172.21.15.44:0/2185425319 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5616eca50400 con 0x5616ecb653a0 2024-04-21T23:10:16.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.507+0000 7f022bfff640 10 client.15915 handle_command_reply: tid=0 2024-04-21T23:10:16.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.507+0000 7f025578f740 1 client.15915 shutdown 2024-04-21T23:10:16.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.507+0000 7f025578f740 1 -- 172.21.15.44:0/2185425319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecaee210 msgr2=0x5616ecae8870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:16.507 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.507+0000 7f025578f740 1 --2- 172.21.15.44:0/2185425319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecaee210 0x5616ecae8870 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f024c06a040 tx=0x7f024402c180 comp rx=0 tx=0).stop 2024-04-21T23:10:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.507+0000 7f025578f740 1 -- 172.21.15.44:0/2185425319 shutdown_connections 2024-04-21T23:10:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.507+0000 7f025578f740 1 --2- 172.21.15.44:0/2185425319 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5616ecb653a0 0x5616ecb677c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.507+0000 7f025578f740 1 --2- 172.21.15.44:0/2185425319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616ecae5440 0x5616ecae8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.507+0000 7f025578f740 1 --2- 172.21.15.44:0/2185425319 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5616ecaee210 0x5616ecae8870 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.507+0000 7f025578f740 1 --2- 172.21.15.44:0/2185425319 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5616ecae4b20 0x5616ecae7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.507+0000 7f022bfff640 0 client.15915 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.507+0000 7f025578f740 1 -- 172.21.15.44:0/2185425319 >> 172.21.15.44:0/2185425319 conn(0x5616ecadf440 msgr2=0x5616ecae0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.507+0000 7f025578f740 1 -- 172.21.15.44:0/2185425319 shutdown_connections 2024-04-21T23:10:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.507+0000 7f025578f740 1 -- 172.21.15.44:0/2185425319 wait complete. 2024-04-21T23:10:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.507+0000 7f025578f740 20 client.15915 trim_cache size 0 max 16384 2024-04-21T23:10:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:10:16.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:10:16.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:10:16.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:10:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.508+0000 7f0252ba1640 1 -- 172.21.15.44:0/1646169094 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f023c0a2e90 msgr2=0x7f023c0a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.508+0000 7f0252ba1640 1 --2- 172.21.15.44:0/1646169094 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f023c0a2e90 0x7f023c0a5350 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0244005cc0 tx=0x7f0244003a70 comp rx=0 tx=0).stop 2024-04-21T23:10:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.508+0000 7f0252ba1640 1 -- 172.21.15.44:0/1646169094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f024c1032d0 msgr2=0x7f024c198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.508+0000 7f0252ba1640 1 --2- 172.21.15.44:0/1646169094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f024c1032d0 0x7f024c198550 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f023800e9f0 tx=0x7f023800edf0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.508+0000 7f0252ba1640 1 -- 172.21.15.44:0/1646169094 shutdown_connections 2024-04-21T23:10:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.508+0000 7f0252ba1640 1 --2- 172.21.15.44:0/1646169094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f024c10bd80 0x7f024c19c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.508+0000 7f0252ba1640 1 --2- 172.21.15.44:0/1646169094 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f023c0a2e90 0x7f023c0a5350 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.508+0000 7f0252ba1640 1 --2- 172.21.15.44:0/1646169094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f024c1032d0 0x7f024c198550 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.508+0000 7f0252ba1640 1 --2- 172.21.15.44:0/1646169094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f024c1029b0 0x7f024c198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.508+0000 7f0252ba1640 1 -- 172.21.15.44:0/1646169094 >> 172.21.15.44:0/1646169094 conn(0x7f024c077760 msgr2=0x7f024c0fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.508+0000 7f0252ba1640 1 -- 172.21.15.44:0/1646169094 shutdown_connections 2024-04-21T23:10:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:16.508+0000 7f0252ba1640 1 -- 172.21.15.44:0/1646169094 wait complete. 2024-04-21T23:10:16.690 INFO:tasks.workunit.client.0.smithi044.stdout: 1 17890 19.21 MB/sec warmup 6 sec latency 76.748 ms 2024-04-21T23:10:17.690 INFO:tasks.workunit.client.0.smithi044.stdout: 1 20753 18.45 MB/sec warmup 7 sec latency 32.579 ms 2024-04-21T23:10:17.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:17 smithi044 ceph-mon[27934]: pgmap v46: 129 pgs: 129 active+clean; 6.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 903 KiB/s wr, 223 op/s 2024-04-21T23:10:17.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:17 smithi055 ceph-mon[31117]: pgmap v46: 129 pgs: 129 active+clean; 6.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 903 KiB/s wr, 223 op/s 2024-04-21T23:10:17.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:17 smithi069 ceph-mon[31816]: pgmap v46: 129 pgs: 129 active+clean; 6.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 903 KiB/s wr, 223 op/s 2024-04-21T23:10:18.690 INFO:tasks.workunit.client.0.smithi044.stdout: 1 23671 17.83 MB/sec warmup 8 sec latency 14.731 ms 2024-04-21T23:10:19.690 INFO:tasks.workunit.client.0.smithi044.stdout: 1 25818 17.30 MB/sec warmup 9 sec latency 45.225 ms 2024-04-21T23:10:19.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:19 smithi044 ceph-mon[27934]: pgmap v47: 129 pgs: 129 active+clean; 6.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 902 KiB/s wr, 221 op/s 2024-04-21T23:10:19.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:19 smithi055 ceph-mon[31117]: pgmap v47: 129 pgs: 129 active+clean; 6.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 902 KiB/s wr, 221 op/s 2024-04-21T23:10:19.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:19 smithi069 ceph-mon[31816]: pgmap v47: 129 pgs: 129 active+clean; 6.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 902 KiB/s wr, 221 op/s 2024-04-21T23:10:20.690 INFO:tasks.workunit.client.0.smithi044.stdout: 1 29102 17.42 MB/sec warmup 10 sec latency 16.213 ms 2024-04-21T23:10:21.691 INFO:tasks.workunit.client.0.smithi044.stdout: 1 32358 17.55 MB/sec warmup 11 sec latency 11.657 ms 2024-04-21T23:10:21.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:21 smithi044 ceph-mon[27934]: pgmap v48: 129 pgs: 129 active+clean; 6.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 902 KiB/s wr, 221 op/s 2024-04-21T23:10:21.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:21 smithi055 ceph-mon[31117]: pgmap v48: 129 pgs: 129 active+clean; 6.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 902 KiB/s wr, 221 op/s 2024-04-21T23:10:21.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:21 smithi069 ceph-mon[31816]: pgmap v48: 129 pgs: 129 active+clean; 6.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 902 KiB/s wr, 221 op/s 2024-04-21T23:10:22.691 INFO:tasks.workunit.client.0.smithi044.stdout: 1 35152 17.20 MB/sec warmup 12 sec latency 26.218 ms 2024-04-21T23:10:23.691 INFO:tasks.workunit.client.0.smithi044.stdout: 1 38103 16.94 MB/sec warmup 13 sec latency 14.719 ms 2024-04-21T23:10:23.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:23 smithi069 ceph-mon[31816]: pgmap v49: 129 pgs: 129 active+clean; 13 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 1.6 MiB/s wr, 331 op/s 2024-04-21T23:10:23.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:23 smithi069 ceph-mon[31816]: mgrmap e25: x(active, since 92s), standbys: y, z 2024-04-21T23:10:23.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:23 smithi055 ceph-mon[31117]: pgmap v49: 129 pgs: 129 active+clean; 13 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 1.6 MiB/s wr, 331 op/s 2024-04-21T23:10:23.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:23 smithi055 ceph-mon[31117]: mgrmap e25: x(active, since 92s), standbys: y, z 2024-04-21T23:10:24.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:23 smithi044 ceph-mon[27934]: pgmap v49: 129 pgs: 129 active+clean; 13 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 1.6 MiB/s wr, 331 op/s 2024-04-21T23:10:24.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:23 smithi044 ceph-mon[27934]: mgrmap e25: x(active, since 92s), standbys: y, z 2024-04-21T23:10:24.691 INFO:tasks.workunit.client.0.smithi044.stdout: 1 40951 16.91 MB/sec warmup 14 sec latency 14.538 ms 2024-04-21T23:10:25.691 INFO:tasks.workunit.client.0.smithi044.stdout: 1 43577 16.75 MB/sec warmup 15 sec latency 22.780 ms 2024-04-21T23:10:25.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:25 smithi055 ceph-mon[31117]: pgmap v50: 129 pgs: 129 active+clean; 13 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 1.5 MiB/s wr, 313 op/s 2024-04-21T23:10:25.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:25 smithi069 ceph-mon[31816]: pgmap v50: 129 pgs: 129 active+clean; 13 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 1.5 MiB/s wr, 313 op/s 2024-04-21T23:10:26.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:25 smithi044 ceph-mon[27934]: pgmap v50: 129 pgs: 129 active+clean; 13 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 1.5 MiB/s wr, 313 op/s 2024-04-21T23:10:26.691 INFO:tasks.workunit.client.0.smithi044.stdout: 1 46836 16.86 MB/sec warmup 16 sec latency 14.692 ms 2024-04-21T23:10:27.691 INFO:tasks.workunit.client.0.smithi044.stdout: 1 49975 16.96 MB/sec warmup 17 sec latency 26.566 ms 2024-04-21T23:10:27.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:27 smithi055 ceph-mon[31117]: pgmap v51: 129 pgs: 129 active+clean; 19 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 400 op/s 2024-04-21T23:10:27.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:27 smithi069 ceph-mon[31816]: pgmap v51: 129 pgs: 129 active+clean; 19 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 400 op/s 2024-04-21T23:10:28.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:27 smithi044 ceph-mon[27934]: pgmap v51: 129 pgs: 129 active+clean; 19 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 400 op/s 2024-04-21T23:10:28.691 INFO:tasks.workunit.client.0.smithi044.stdout: 1 53052 16.84 MB/sec warmup 18 sec latency 16.113 ms 2024-04-21T23:10:29.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.600+0000 7fd3af44e640 1 -- 172.21.15.44:0/1865537292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3a8100b40 msgr2=0x7fd3a8100f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:29.602 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.600+0000 7fd3af44e640 1 --2- 172.21.15.44:0/1865537292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3a8100b40 0x7fd3a8100f20 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4009930 tx=0x7fd3a402bc80 comp rx=0 tx=0).stop 2024-04-21T23:10:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.600+0000 7fd3af44e640 1 -- 172.21.15.44:0/1865537292 shutdown_connections 2024-04-21T23:10:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.600+0000 7fd3af44e640 1 --2- 172.21.15.44:0/1865537292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3a8105a10 0x7fd3a810d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.600+0000 7fd3af44e640 1 --2- 172.21.15.44:0/1865537292 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3a81014f0 0x7fd3a81052e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.600+0000 7fd3af44e640 1 --2- 172.21.15.44:0/1865537292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3a8100b40 0x7fd3a8100f20 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.600+0000 7fd3af44e640 1 -- 172.21.15.44:0/1865537292 >> 172.21.15.44:0/1865537292 conn(0x7fd3a8077500 msgr2=0x7fd3a8077910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.600+0000 7fd3af44e640 1 -- 172.21.15.44:0/1865537292 shutdown_connections 2024-04-21T23:10:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.600+0000 7fd3af44e640 1 -- 172.21.15.44:0/1865537292 wait complete. 2024-04-21T23:10:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.601+0000 7fd3af44e640 1 Processor -- start 2024-04-21T23:10:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.601+0000 7fd3af44e640 1 -- start start 2024-04-21T23:10:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.601+0000 7fd3af44e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3a8100b40 0x7fd3a8196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.601+0000 7fd3af44e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3a8105a10 0x7fd3a8197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.601+0000 7fd3af44e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3a819b790 0x7fd3a819bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.601+0000 7fd3af44e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a8110a40 con 0x7fd3a8100b40 2024-04-21T23:10:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.601+0000 7fd3af44e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a81108c0 con 0x7fd3a8105a10 2024-04-21T23:10:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.601+0000 7fd3af44e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a8110bc0 con 0x7fd3a819b790 2024-04-21T23:10:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.601+0000 7fd3ad1ea640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3a8100b40 0x7fd3a8196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.601+0000 7fd3ad1ea640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3a8100b40 0x7fd3a8196f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59428/0 (socket says 172.21.15.44:59428) 2024-04-21T23:10:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.601+0000 7fd3ad1ea640 1 -- 172.21.15.44:0/690832215 learned_addr learned my addr 172.21.15.44:0/690832215 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.601+0000 7fd3ac9e9640 1 --2- 172.21.15.44:0/690832215 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3a8105a10 0x7fd3a8197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.601+0000 7fd3ad9eb640 1 --2- 172.21.15.44:0/690832215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3a819b790 0x7fd3a819bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.602+0000 7fd3ad1ea640 1 -- 172.21.15.44:0/690832215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3a819b790 msgr2=0x7fd3a819bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.602+0000 7fd3ad1ea640 1 --2- 172.21.15.44:0/690832215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3a819b790 0x7fd3a819bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.602+0000 7fd3ad1ea640 1 -- 172.21.15.44:0/690832215 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3a8105a10 msgr2=0x7fd3a8197460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.602+0000 7fd3ad1ea640 1 --2- 172.21.15.44:0/690832215 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3a8105a10 0x7fd3a8197460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.602+0000 7fd3ad1ea640 1 -- 172.21.15.44:0/690832215 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3a819c280 con 0x7fd3a8100b40 2024-04-21T23:10:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.602+0000 7fd3ad1ea640 1 --2- 172.21.15.44:0/690832215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3a8100b40 0x7fd3a8196f20 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4009430 tx=0x7fd3a402cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.602+0000 7fd39e7fc640 1 -- 172.21.15.44:0/690832215 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4041070 con 0x7fd3a8100b40 2024-04-21T23:10:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.602+0000 7fd39e7fc640 1 -- 172.21.15.44:0/690832215 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3a4004450 con 0x7fd3a8100b40 2024-04-21T23:10:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.602+0000 7fd3af44e640 1 -- 172.21.15.44:0/690832215 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3a819c510 con 0x7fd3a8100b40 2024-04-21T23:10:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.602+0000 7fd39e7fc640 1 -- 172.21.15.44:0/690832215 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4003dc0 con 0x7fd3a8100b40 2024-04-21T23:10:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.602+0000 7fd3af44e640 1 -- 172.21.15.44:0/690832215 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3a81a3d30 con 0x7fd3a8100b40 2024-04-21T23:10:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.605+0000 7fd39e7fc640 1 -- 172.21.15.44:0/690832215 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd3a4044070 con 0x7fd3a8100b40 2024-04-21T23:10:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.605+0000 7fd3b203a740 1 Processor -- start 2024-04-21T23:10:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.605+0000 7fd3b203a740 1 -- start start 2024-04-21T23:10:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.605+0000 7fd3b203a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098b8d30 0x5559098b9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.605+0000 7fd3b203a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b9650 0x5559098c1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.605+0000 7fd3b203a740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098c2420 0x5559098c4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.605+0000 7fd3b203a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55590981ab70 con 0x5559098b9650 2024-04-21T23:10:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.605+0000 7fd3b203a740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5559097c3470 con 0x5559098c2420 2024-04-21T23:10:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.605+0000 7fd3b203a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555909816b30 con 0x5559098b8d30 2024-04-21T23:10:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.605+0000 7fd3ad1ea640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b9650 0x5559098c1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.605+0000 7fd3ad1ea640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b9650 0x5559098c1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59442/0 (socket says 172.21.15.44:59442) 2024-04-21T23:10:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.605+0000 7fd3ad1ea640 1 -- 172.21.15.44:0/345618483 learned_addr learned my addr 172.21.15.44:0/345618483 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.605+0000 7fd3ac9e9640 1 --2- 172.21.15.44:0/345618483 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098c2420 0x5559098c4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.605+0000 7fd39e7fc640 1 --2- 172.21.15.44:0/690832215 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd3940a2d70 0x7fd3940a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.605+0000 7fd3ad9eb640 1 --2- 172.21.15.44:0/345618483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098b8d30 0x5559098b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.606+0000 7fd3ac9e9640 1 -- 172.21.15.44:0/345618483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098b8d30 msgr2=0x5559098b9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.606+0000 7fd3ac9e9640 1 --2- 172.21.15.44:0/345618483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098b8d30 0x5559098b9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.606+0000 7fd3ac9e9640 1 -- 172.21.15.44:0/345618483 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b9650 msgr2=0x5559098c1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.606+0000 7fd3ac9e9640 1 --2- 172.21.15.44:0/345618483 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b9650 0x5559098c1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.606+0000 7fd39e7fc640 1 -- 172.21.15.44:0/690832215 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fd3a40f1030 con 0x7fd3a8100b40 2024-04-21T23:10:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.606+0000 7fd3ad9eb640 1 --2- 172.21.15.44:0/690832215 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd3940a2d70 0x7fd3940a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.606+0000 7fd3ac9e9640 1 -- 172.21.15.44:0/345618483 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55590952f210 con 0x5559098c2420 2024-04-21T23:10:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.606+0000 7fd3ad1ea640 1 --2- 172.21.15.44:0/345618483 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b9650 0x5559098c1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.606+0000 7fd3ac9e9640 1 --2- 172.21.15.44:0/345618483 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098c2420 0x5559098c4810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd3a80789c0 tx=0x7fd398011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=278750442b30ab80 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.606+0000 7fd3ad9eb640 1 --2- 172.21.15.44:0/690832215 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd3940a2d70 0x7fd3940a5230 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0005c40 tx=0x7fd3a0004ab0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.606+0000 7fd3937fe640 1 -- 172.21.15.44:0/345618483 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd398009d70 con 0x5559098c2420 2024-04-21T23:10:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.606+0000 7fd3937fe640 1 -- 172.21.15.44:0/345618483 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd398005020 con 0x5559098c2420 2024-04-21T23:10:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.606+0000 7fd3937fe640 1 -- 172.21.15.44:0/345618483 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd398004b90 con 0x5559098c2420 2024-04-21T23:10:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.607+0000 7fd3b203a740 1 -- 172.21.15.44:0/345618483 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098c2420 msgr2=0x5559098c4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.607+0000 7fd3b203a740 1 --2- 172.21.15.44:0/345618483 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098c2420 0x5559098c4810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd3a80789c0 tx=0x7fd398011810 comp rx=0 tx=0).stop 2024-04-21T23:10:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.607+0000 7fd3b203a740 1 -- 172.21.15.44:0/345618483 shutdown_connections 2024-04-21T23:10:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.607+0000 7fd3b203a740 1 --2- 172.21.15.44:0/345618483 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098c2420 0x5559098c4810 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.607+0000 7fd3b203a740 1 --2- 172.21.15.44:0/345618483 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b9650 0x5559098c1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.607+0000 7fd3b203a740 1 --2- 172.21.15.44:0/345618483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098b8d30 0x5559098b9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.607+0000 7fd3b203a740 1 -- 172.21.15.44:0/345618483 >> 172.21.15.44:0/345618483 conn(0x5559098b3650 msgr2=0x5559098b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.607+0000 7fd3b203a740 1 -- 172.21.15.44:0/345618483 shutdown_connections 2024-04-21T23:10:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.607+0000 7fd3b203a740 1 -- 172.21.15.44:0/345618483 wait complete. 2024-04-21T23:10:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.607+0000 7fd3b203a740 1 -- start start 2024-04-21T23:10:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.607+0000 7fd3b203a740 1 Processor -- start 2024-04-21T23:10:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd3b203a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b8d30 0x5559098b2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd3b203a740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098b9650 0x5559098b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd3b203a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098c2420 0x5559098b1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd3b203a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5559098c5050 con 0x5559098b8d30 2024-04-21T23:10:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd3b203a740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5559097d6a10 con 0x5559098b9650 2024-04-21T23:10:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd3b203a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5559098c51d0 con 0x5559098c2420 2024-04-21T23:10:29.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd3b203a740 10 client.0 _ll_register_callbacks cb 0x555909811b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:29.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd3ad1ea640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b8d30 0x5559098b2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:29.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd3ad1ea640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b8d30 0x5559098b2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59452/0 (socket says 172.21.15.44:59452) 2024-04-21T23:10:29.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd3ac9e9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098b9650 0x5559098b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:29.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd3ad1ea640 1 -- 172.21.15.44:0/126320146 learned_addr learned my addr 172.21.15.44:0/126320146 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:29.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd3ad9eb640 1 --2- 172.21.15.44:0/126320146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098c2420 0x5559098b1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:29.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd3ad1ea640 1 -- 172.21.15.44:0/126320146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098c2420 msgr2=0x5559098b1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:29.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd3ad1ea640 1 --2- 172.21.15.44:0/126320146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098c2420 0x5559098b1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd3ad1ea640 1 -- 172.21.15.44:0/126320146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098b9650 msgr2=0x5559098b0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:29.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd3ad1ea640 1 --2- 172.21.15.44:0/126320146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098b9650 0x5559098b0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd3ad1ea640 1 -- 172.21.15.44:0/126320146 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559095fb620 con 0x5559098b8d30 2024-04-21T23:10:29.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd3ad9eb640 1 --2- 172.21.15.44:0/126320146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098c2420 0x5559098b1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:29.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd3ac9e9640 1 --2- 172.21.15.44:0/126320146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098b9650 0x5559098b0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:10:29.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd3ad1ea640 1 --2- 172.21.15.44:0/126320146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b8d30 0x5559098b2930 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd3a40ee9e0 tx=0x7fd3a4003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=290525d47dd95a4d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.608+0000 7fd390ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:10:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.609+0000 7fd390ff9640 1 -- 172.21.15.44:0/126320146 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a404b040 con 0x5559098b8d30 2024-04-21T23:10:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.609+0000 7fd390ff9640 1 -- 172.21.15.44:0/126320146 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fd3a40ee0b0 con 0x5559098b8d30 2024-04-21T23:10:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.609+0000 7fd3b203a740 1 -- 172.21.15.44:0/126320146 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd398005020 con 0x5559098b8d30 2024-04-21T23:10:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.609+0000 7fd390ff9640 1 -- 172.21.15.44:0/126320146 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd3a40bd140 con 0x5559098b8d30 2024-04-21T23:10:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.609+0000 7fd3b203a740 10 client.15927 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.609+0000 7fd3b203a740 1 -- 172.21.15.44:0/126320146 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5559098d3520 con 0x5559098b8d30 2024-04-21T23:10:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.609+0000 7fd390ff9640 1 -- 172.21.15.44:0/126320146 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd398005020 con 0x5559098b8d30 2024-04-21T23:10:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.609+0000 7fd390ff9640 1 -- 172.21.15.44:0/126320146 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fd3a40ee660 con 0x5559098b8d30 2024-04-21T23:10:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.610+0000 7fd3b203a740 10 client.15927 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.610+0000 7fd3b203a740 10 client.15927 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.610+0000 7fd3b203a740 20 client.15927 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.610+0000 7fd3b203a740 1 --2- 172.21.15.44:0/126320146 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5559098d4980 0x5559098d6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.610+0000 7fd3b203a740 4 client.15927 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:10:29.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.610+0000 7fd3b203a740 1 -- 172.21.15.44:0/126320146 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559098280b0 con 0x5559098d4980 2024-04-21T23:10:29.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.610+0000 7fd3ac9e9640 1 --2- 172.21.15.44:0/126320146 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5559098d4980 0x5559098d6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:29.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.610+0000 7fd390ff9640 10 client.15927 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:29.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.610+0000 7fd3ac9e9640 1 --2- 172.21.15.44:0/126320146 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5559098d4980 0x5559098d6dd0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ad29b83e676d381 server_cookie=ea2729f5b9ce83c8 in_seq=0 out_seq=0 2024-04-21T23:10:29.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.611+0000 7fd390ff9640 1 -- 172.21.15.44:0/126320146 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5559098280b0 con 0x5559098d4980 2024-04-21T23:10:29.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.611+0000 7fd390ff9640 10 client.15927 handle_command_reply: tid=0 2024-04-21T23:10:29.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.611+0000 7fd3b203a740 1 client.15927 shutdown 2024-04-21T23:10:29.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.612+0000 7fd3b203a740 1 -- 172.21.15.44:0/126320146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b8d30 msgr2=0x5559098b2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:29.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.612+0000 7fd3b203a740 1 --2- 172.21.15.44:0/126320146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b8d30 0x5559098b2930 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd3a40ee9e0 tx=0x7fd3a4003f60 comp rx=0 tx=0).stop 2024-04-21T23:10:29.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.612+0000 7fd3b203a740 1 -- 172.21.15.44:0/126320146 shutdown_connections 2024-04-21T23:10:29.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.612+0000 7fd3b203a740 1 --2- 172.21.15.44:0/126320146 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5559098d4980 0x5559098d6dd0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.612+0000 7fd3b203a740 1 --2- 172.21.15.44:0/126320146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098c2420 0x5559098b1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.612+0000 7fd3b203a740 1 --2- 172.21.15.44:0/126320146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098b9650 0x5559098b0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.612+0000 7fd390ff9640 0 client.15927 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:29.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.612+0000 7fd3b203a740 1 --2- 172.21.15.44:0/126320146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b8d30 0x5559098b2930 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.612+0000 7fd3b203a740 1 -- 172.21.15.44:0/126320146 >> 172.21.15.44:0/126320146 conn(0x5559098b3650 msgr2=0x5559098b53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:29.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.612+0000 7fd3b203a740 1 -- 172.21.15.44:0/126320146 shutdown_connections 2024-04-21T23:10:29.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.612+0000 7fd3b203a740 1 -- 172.21.15.44:0/126320146 wait complete. 2024-04-21T23:10:29.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.612+0000 7fd3b203a740 20 client.15927 trim_cache size 0 max 16384 2024-04-21T23:10:29.628 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.627+0000 7fd3b203a740 1 Processor -- start 2024-04-21T23:10:29.628 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.627+0000 7fd3b203a740 1 -- start start 2024-04-21T23:10:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.627+0000 7fd3b203a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b8d30 0x5559098c1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.627+0000 7fd3b203a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098b9650 0x5559098dc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.627+0000 7fd3b203a740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098c2420 0x5559098dcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.627+0000 7fd3b203a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a81aa900 con 0x5559098b8d30 2024-04-21T23:10:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.627+0000 7fd3b203a740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a81ad180 con 0x5559098c2420 2024-04-21T23:10:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.627+0000 7fd3b203a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a81aaf70 con 0x5559098b9650 2024-04-21T23:10:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.627+0000 7fd3ad1ea640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b8d30 0x5559098c1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.627+0000 7fd3ad1ea640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b8d30 0x5559098c1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59466/0 (socket says 172.21.15.44:59466) 2024-04-21T23:10:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.627+0000 7fd3ad9eb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098c2420 0x5559098dcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.627+0000 7fd3ad1ea640 1 -- 172.21.15.44:0/3903278610 learned_addr learned my addr 172.21.15.44:0/3903278610 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.627+0000 7fd3ac9e9640 1 --2- 172.21.15.44:0/3903278610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098b9650 0x5559098dc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.627+0000 7fd3ad1ea640 1 -- 172.21.15.44:0/3903278610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098b9650 msgr2=0x5559098dc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.627+0000 7fd3ad1ea640 1 --2- 172.21.15.44:0/3903278610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098b9650 0x5559098dc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.627+0000 7fd3ad1ea640 1 -- 172.21.15.44:0/3903278610 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098c2420 msgr2=0x5559098dcbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:29.629 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.627+0000 7fd3ad1ea640 1 --2- 172.21.15.44:0/3903278610 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098c2420 0x5559098dcbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.627+0000 7fd3ad1ea640 1 -- 172.21.15.44:0/3903278610 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559098bd9f0 con 0x5559098b8d30 2024-04-21T23:10:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.627+0000 7fd3ac9e9640 1 --2- 172.21.15.44:0/3903278610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098b9650 0x5559098dc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:10:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.628+0000 7fd3ad1ea640 1 --2- 172.21.15.44:0/3903278610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b8d30 0x5559098c1a00 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4003fd0 tx=0x7fd3a40bb880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3439b06c1df4fc16 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.628+0000 7fd392ffd640 1 -- 172.21.15.44:0/3903278610 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a404b070 con 0x5559098b8d30 2024-04-21T23:10:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.628+0000 7fd392ffd640 1 -- 172.21.15.44:0/3903278610 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3a40bd370 con 0x5559098b8d30 2024-04-21T23:10:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.628+0000 7fd392ffd640 1 -- 172.21.15.44:0/3903278610 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a40bccd0 con 0x5559098b8d30 2024-04-21T23:10:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.628+0000 7fd3b203a740 1 -- 172.21.15.44:0/3903278610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b8d30 msgr2=0x5559098c1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.628+0000 7fd3b203a740 1 --2- 172.21.15.44:0/3903278610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b8d30 0x5559098c1a00 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4003fd0 tx=0x7fd3a40bb880 comp rx=0 tx=0).stop 2024-04-21T23:10:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.628+0000 7fd3b203a740 1 -- 172.21.15.44:0/3903278610 shutdown_connections 2024-04-21T23:10:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.628+0000 7fd3b203a740 1 --2- 172.21.15.44:0/3903278610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098b9650 0x5559098dc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.628+0000 7fd3b203a740 1 --2- 172.21.15.44:0/3903278610 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098c2420 0x5559098dcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.628+0000 7fd3b203a740 1 --2- 172.21.15.44:0/3903278610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b8d30 0x5559098c1a00 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.628+0000 7fd3b203a740 1 -- 172.21.15.44:0/3903278610 >> 172.21.15.44:0/3903278610 conn(0x5559098b3650 msgr2=0x5559098ac220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:29.630 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.629+0000 7fd3b203a740 1 -- 172.21.15.44:0/3903278610 shutdown_connections 2024-04-21T23:10:29.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.629+0000 7fd3b203a740 1 -- 172.21.15.44:0/3903278610 wait complete. 2024-04-21T23:10:29.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.629+0000 7fd3b203a740 1 -- start start 2024-04-21T23:10:29.631 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.629+0000 7fd3b203a740 1 Processor -- start 2024-04-21T23:10:29.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.629+0000 7fd3b203a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098b8d30 0x5559098cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:29.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.629+0000 7fd3b203a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b9650 0x5559098cc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:29.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.629+0000 7fd3b203a740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098c2420 0x5559098cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:29.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.629+0000 7fd3b203a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a81ab640 con 0x5559098b9650 2024-04-21T23:10:29.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.629+0000 7fd3b203a740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a81acab0 con 0x5559098c2420 2024-04-21T23:10:29.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.629+0000 7fd3b203a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a81abd10 con 0x5559098b8d30 2024-04-21T23:10:29.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.629+0000 7fd3b203a740 10 client.0 _ll_register_callbacks cb 0x7fd3a4009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:29.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.629+0000 7fd3ac9e9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b9650 0x5559098cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:29.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.629+0000 7fd3ac9e9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b9650 0x5559098cc110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59478/0 (socket says 172.21.15.44:59478) 2024-04-21T23:10:29.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.629+0000 7fd3ac9e9640 1 -- 172.21.15.44:0/3037342684 learned_addr learned my addr 172.21.15.44:0/3037342684 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:29.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.630+0000 7fd3ad9eb640 1 --2- 172.21.15.44:0/3037342684 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098c2420 0x5559098cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:29.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.630+0000 7fd3ad1ea640 1 --2- 172.21.15.44:0/3037342684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098b8d30 0x5559098cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:29.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.630+0000 7fd3ac9e9640 1 -- 172.21.15.44:0/3037342684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098b8d30 msgr2=0x5559098cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:29.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.630+0000 7fd3ac9e9640 1 --2- 172.21.15.44:0/3037342684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098b8d30 0x5559098cbbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.630+0000 7fd3ac9e9640 1 -- 172.21.15.44:0/3037342684 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098c2420 msgr2=0x5559098cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:29.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.630+0000 7fd3ac9e9640 1 --2- 172.21.15.44:0/3037342684 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098c2420 0x5559098cc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.630+0000 7fd3ac9e9640 1 -- 172.21.15.44:0/3037342684 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55590993a170 con 0x5559098b9650 2024-04-21T23:10:29.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.630+0000 7fd3ad9eb640 1 --2- 172.21.15.44:0/3037342684 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098c2420 0x5559098cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:29.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.630+0000 7fd3ad1ea640 1 --2- 172.21.15.44:0/3037342684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098b8d30 0x5559098cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:29.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.630+0000 7fd3ac9e9640 1 --2- 172.21.15.44:0/3037342684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b9650 0x5559098cc110 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd398009380 tx=0x7fd398009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a7fb0752036964e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:29.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.630+0000 7fd390ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:10:29.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.630+0000 7fd390ff9640 1 -- 172.21.15.44:0/3037342684 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd398005460 con 0x5559098b9650 2024-04-21T23:10:29.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.630+0000 7fd390ff9640 1 -- 172.21.15.44:0/3037342684 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fd39802bb50 con 0x5559098b9650 2024-04-21T23:10:29.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.630+0000 7fd3b203a740 1 -- 172.21.15.44:0/3037342684 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3a40bd370 con 0x5559098b9650 2024-04-21T23:10:29.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.630+0000 7fd390ff9640 1 -- 172.21.15.44:0/3037342684 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd398039050 con 0x5559098b9650 2024-04-21T23:10:29.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.630+0000 7fd3b203a740 10 client.15933 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:29.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.630+0000 7fd3b203a740 1 -- 172.21.15.44:0/3037342684 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55590993b3d0 con 0x5559098b9650 2024-04-21T23:10:29.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.630+0000 7fd390ff9640 1 -- 172.21.15.44:0/3037342684 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3a40bd370 con 0x5559098b9650 2024-04-21T23:10:29.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.631+0000 7fd390ff9640 1 -- 172.21.15.44:0/3037342684 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fd39802b370 con 0x5559098b9650 2024-04-21T23:10:29.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.631+0000 7fd3b203a740 10 client.15933 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:29.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.631+0000 7fd3b203a740 10 client.15933 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:29.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.631+0000 7fd3b203a740 20 client.15933 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:29.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.631+0000 7fd3b203a740 1 --2- 172.21.15.44:0/3037342684 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55590993ba40 0x55590993de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:29.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.631+0000 7fd3b203a740 4 client.15933 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:10:29.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.631+0000 7fd3b203a740 1 -- 172.21.15.44:0/3037342684 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555909822400 con 0x55590993ba40 2024-04-21T23:10:29.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.631+0000 7fd3ad1ea640 1 --2- 172.21.15.44:0/3037342684 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55590993ba40 0x55590993de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:29.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.631+0000 7fd3ad1ea640 1 --2- 172.21.15.44:0/3037342684 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55590993ba40 0x55590993de60 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=474c11b5890e4eb2 server_cookie=c93fa68fbea4f2e6 in_seq=0 out_seq=0 2024-04-21T23:10:29.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.631+0000 7fd390ff9640 10 client.15933 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:29.691 INFO:tasks.workunit.client.0.smithi044.stdout: 1 55819 16.62 MB/sec warmup 19 sec latency 48.192 ms 2024-04-21T23:10:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.900+0000 7fd390ff9640 1 -- 172.21.15.44:0/3037342684 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555909822400 con 0x55590993ba40 2024-04-21T23:10:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.900+0000 7fd390ff9640 10 client.15933 handle_command_reply: tid=0 2024-04-21T23:10:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.901+0000 7fd3b203a740 1 client.15933 shutdown 2024-04-21T23:10:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.901+0000 7fd3b203a740 1 -- 172.21.15.44:0/3037342684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b9650 msgr2=0x5559098cc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.901+0000 7fd3b203a740 1 --2- 172.21.15.44:0/3037342684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b9650 0x5559098cc110 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd398009380 tx=0x7fd398009b20 comp rx=0 tx=0).stop 2024-04-21T23:10:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.901+0000 7fd3b203a740 1 -- 172.21.15.44:0/3037342684 shutdown_connections 2024-04-21T23:10:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.901+0000 7fd3b203a740 1 --2- 172.21.15.44:0/3037342684 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55590993ba40 0x55590993de60 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.901+0000 7fd3b203a740 1 --2- 172.21.15.44:0/3037342684 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559098c2420 0x5559098cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.901+0000 7fd3b203a740 1 --2- 172.21.15.44:0/3037342684 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559098b9650 0x5559098cc110 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.901+0000 7fd3b203a740 1 --2- 172.21.15.44:0/3037342684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559098b8d30 0x5559098cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.901+0000 7fd390ff9640 0 client.15933 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:29.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.901+0000 7fd3b203a740 1 -- 172.21.15.44:0/3037342684 >> 172.21.15.44:0/3037342684 conn(0x5559098b3650 msgr2=0x5559098bdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:29.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.901+0000 7fd3b203a740 1 -- 172.21.15.44:0/3037342684 shutdown_connections 2024-04-21T23:10:29.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.901+0000 7fd3b203a740 1 -- 172.21.15.44:0/3037342684 wait complete. 2024-04-21T23:10:29.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.901+0000 7fd3b203a740 20 client.15933 trim_cache size 0 max 16384 2024-04-21T23:10:29.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:10:29.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:10:29.902 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:10:29.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:10:29.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.902+0000 7fd3af44e640 1 -- 172.21.15.44:0/690832215 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd3940a2d70 msgr2=0x7fd3940a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:29.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.902+0000 7fd3af44e640 1 --2- 172.21.15.44:0/690832215 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd3940a2d70 0x7fd3940a5230 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0005c40 tx=0x7fd3a0004ab0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.903 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.902+0000 7fd3af44e640 1 -- 172.21.15.44:0/690832215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3a8100b40 msgr2=0x7fd3a8196f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.902+0000 7fd3af44e640 1 --2- 172.21.15.44:0/690832215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3a8100b40 0x7fd3a8196f20 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4009430 tx=0x7fd3a402cb10 comp rx=0 tx=0).stop 2024-04-21T23:10:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.902+0000 7fd3af44e640 1 -- 172.21.15.44:0/690832215 shutdown_connections 2024-04-21T23:10:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.902+0000 7fd3af44e640 1 --2- 172.21.15.44:0/690832215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3a819b790 0x7fd3a819bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.902+0000 7fd3af44e640 1 --2- 172.21.15.44:0/690832215 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd3940a2d70 0x7fd3940a5230 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.902+0000 7fd3af44e640 1 --2- 172.21.15.44:0/690832215 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd3a8105a10 0x7fd3a8197460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.902+0000 7fd3af44e640 1 --2- 172.21.15.44:0/690832215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd3a8100b40 0x7fd3a8196f20 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.902+0000 7fd3af44e640 1 -- 172.21.15.44:0/690832215 >> 172.21.15.44:0/690832215 conn(0x7fd3a8077500 msgr2=0x7fd3a80feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.902+0000 7fd3af44e640 1 -- 172.21.15.44:0/690832215 shutdown_connections 2024-04-21T23:10:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:29.902+0000 7fd3af44e640 1 -- 172.21.15.44:0/690832215 wait complete. 2024-04-21T23:10:30.134 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:10:30.134 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:10:30.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.245+0000 7feb5c4e6640 1 -- 172.21.15.44:0/569462551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb541009e0 msgr2=0x7feb54100dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.245+0000 7feb5c4e6640 1 --2- 172.21.15.44:0/569462551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb541009e0 0x7feb54100dc0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7feb48009900 tx=0x7feb4802bd10 comp rx=0 tx=0).stop 2024-04-21T23:10:30.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.245+0000 7feb5c4e6640 1 -- 172.21.15.44:0/569462551 shutdown_connections 2024-04-21T23:10:30.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.245+0000 7feb5c4e6640 1 --2- 172.21.15.44:0/569462551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb54102ac0 0x7feb5410d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.245+0000 7feb5c4e6640 1 --2- 172.21.15.44:0/569462551 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb54102120 0x7feb54102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.245+0000 7feb5c4e6640 1 --2- 172.21.15.44:0/569462551 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb541009e0 0x7feb54100dc0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.245+0000 7feb5c4e6640 1 -- 172.21.15.44:0/569462551 >> 172.21.15.44:0/569462551 conn(0x7feb54077760 msgr2=0x7feb54077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:30.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.245+0000 7feb5c4e6640 1 -- 172.21.15.44:0/569462551 shutdown_connections 2024-04-21T23:10:30.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.245+0000 7feb5c4e6640 1 -- 172.21.15.44:0/569462551 wait complete. 2024-04-21T23:10:30.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.246+0000 7feb5c4e6640 1 Processor -- start 2024-04-21T23:10:30.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.246+0000 7feb5c4e6640 1 -- start start 2024-04-21T23:10:30.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.246+0000 7feb5c4e6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb541009e0 0x7feb54197030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.246+0000 7feb5c4e6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb54102120 0x7feb54197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.246+0000 7feb5c4e6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb54102ac0 0x7feb5419b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.246+0000 7feb5c4e6640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7feb54110af0 con 0x7feb54102120 2024-04-21T23:10:30.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.246+0000 7feb5c4e6640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7feb54110970 con 0x7feb54102ac0 2024-04-21T23:10:30.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.246+0000 7feb5c4e6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7feb54110c70 con 0x7feb541009e0 2024-04-21T23:10:30.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.246+0000 7feb59a81640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb54102120 0x7feb54197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.246+0000 7feb59a81640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb54102120 0x7feb54197570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59494/0 (socket says 172.21.15.44:59494) 2024-04-21T23:10:30.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.246+0000 7feb59a81640 1 -- 172.21.15.44:0/3373171154 learned_addr learned my addr 172.21.15.44:0/3373171154 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:30.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.246+0000 7feb5a282640 1 --2- 172.21.15.44:0/3373171154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb541009e0 0x7feb54197030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.246+0000 7feb5aa83640 1 --2- 172.21.15.44:0/3373171154 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb54102ac0 0x7feb5419b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.247+0000 7feb59a81640 1 -- 172.21.15.44:0/3373171154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb541009e0 msgr2=0x7feb54197030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.247+0000 7feb59a81640 1 --2- 172.21.15.44:0/3373171154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb541009e0 0x7feb54197030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.247+0000 7feb59a81640 1 -- 172.21.15.44:0/3373171154 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb54102ac0 msgr2=0x7feb5419b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.247+0000 7feb59a81640 1 --2- 172.21.15.44:0/3373171154 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb54102ac0 0x7feb5419b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.247+0000 7feb59a81640 1 -- 172.21.15.44:0/3373171154 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb5419bfd0 con 0x7feb54102120 2024-04-21T23:10:30.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.247+0000 7feb59a81640 1 --2- 172.21.15.44:0/3373171154 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb54102120 0x7feb54197570 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7feb5000ebc0 tx=0x7feb5000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:30.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.247+0000 7feb477fe640 1 -- 172.21.15.44:0/3373171154 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb50010630 con 0x7feb54102120 2024-04-21T23:10:30.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.247+0000 7feb477fe640 1 -- 172.21.15.44:0/3373171154 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb50010cb0 con 0x7feb54102120 2024-04-21T23:10:30.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.247+0000 7feb5c4e6640 1 -- 172.21.15.44:0/3373171154 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb5419c2c0 con 0x7feb54102120 2024-04-21T23:10:30.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.247+0000 7feb477fe640 1 -- 172.21.15.44:0/3373171154 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb50018bd0 con 0x7feb54102120 2024-04-21T23:10:30.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.247+0000 7feb5c4e6640 1 -- 172.21.15.44:0/3373171154 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb541a3b50 con 0x7feb54102120 2024-04-21T23:10:30.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.250+0000 7feb477fe640 1 -- 172.21.15.44:0/3373171154 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7feb500107d0 con 0x7feb54102120 2024-04-21T23:10:30.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.250+0000 7feb5f1ae740 1 Processor -- start 2024-04-21T23:10:30.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.250+0000 7feb5f1ae740 1 -- start start 2024-04-21T23:10:30.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.250+0000 7feb5f1ae740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b05066a140 0x55b05066a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.250+0000 7feb5f1ae740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066aa60 0x55b050673100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.250+0000 7feb5f1ae740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b050673640 0x55b050675a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.250+0000 7feb5f1ae740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b0505cc560 con 0x55b05066aa60 2024-04-21T23:10:30.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.250+0000 7feb5f1ae740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55b050574730 con 0x55b05066a140 2024-04-21T23:10:30.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.250+0000 7feb5f1ae740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b0505c8520 con 0x55b050673640 2024-04-21T23:10:30.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.250+0000 7feb59a81640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b050673640 0x55b050675a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.250+0000 7feb5a282640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066aa60 0x55b050673100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.250+0000 7feb5aa83640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b05066a140 0x55b05066a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.250+0000 7feb59a81640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b050673640 0x55b050675a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:43686/0 (socket says 172.21.15.44:43686) 2024-04-21T23:10:30.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.250+0000 7feb5a282640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066aa60 0x55b050673100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59508/0 (socket says 172.21.15.44:59508) 2024-04-21T23:10:30.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.250+0000 7feb59a81640 1 -- 172.21.15.44:0/3881145593 learned_addr learned my addr 172.21.15.44:0/3881145593 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:30.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.250+0000 7feb5aa83640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b05066a140 0x55b05066a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:36686/0 (socket says 172.21.15.44:36686) 2024-04-21T23:10:30.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.251+0000 7feb477fe640 1 --2- 172.21.15.44:0/3373171154 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7feb400a2d80 0x7feb400a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.251+0000 7feb59a81640 1 -- 172.21.15.44:0/3881145593 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b05066a140 msgr2=0x55b05066a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:30.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.251+0000 7feb59a81640 1 --2- 172.21.15.44:0/3881145593 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b05066a140 0x55b05066a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.251+0000 7feb59a81640 1 -- 172.21.15.44:0/3881145593 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066aa60 msgr2=0x55b050673100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:30.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.251+0000 7feb59a81640 1 --2- 172.21.15.44:0/3881145593 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066aa60 0x55b050673100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.251+0000 7feb59a81640 1 -- 172.21.15.44:0/3881145593 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0503aa7d0 con 0x55b050673640 2024-04-21T23:10:30.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.251+0000 7feb5aa83640 1 --2- 172.21.15.44:0/3881145593 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b05066a140 0x55b05066a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:30.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.251+0000 7feb5aa83640 1 --2- 172.21.15.44:0/3373171154 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7feb400a2d80 0x7feb400a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.251+0000 7feb477fe640 1 -- 172.21.15.44:0/3373171154 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7feb500cfd90 con 0x7feb54102120 2024-04-21T23:10:30.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.251+0000 7feb59a81640 1 --2- 172.21.15.44:0/3881145593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b050673640 0x55b050675a30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7feb500cd0b0 tx=0x7feb50066bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=255bf0385880c1ce server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:30.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.251+0000 7feb44bfe640 1 -- 172.21.15.44:0/3881145593 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb50010e60 con 0x55b050673640 2024-04-21T23:10:30.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.251+0000 7feb5aa83640 1 --2- 172.21.15.44:0/3373171154 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7feb400a2d80 0x7feb400a5240 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feb4c004650 tx=0x7feb4c0046c0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:30.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.251+0000 7feb44bfe640 1 -- 172.21.15.44:0/3881145593 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb5006ad00 con 0x55b050673640 2024-04-21T23:10:30.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.251+0000 7feb44bfe640 1 -- 172.21.15.44:0/3881145593 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb5009bb00 con 0x55b050673640 2024-04-21T23:10:30.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.252+0000 7feb5f1ae740 1 -- 172.21.15.44:0/3881145593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b050673640 msgr2=0x55b050675a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.252+0000 7feb5f1ae740 1 --2- 172.21.15.44:0/3881145593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b050673640 0x55b050675a30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7feb500cd0b0 tx=0x7feb50066bf0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.252+0000 7feb5f1ae740 1 -- 172.21.15.44:0/3881145593 shutdown_connections 2024-04-21T23:10:30.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.252+0000 7feb5f1ae740 1 --2- 172.21.15.44:0/3881145593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b050673640 0x55b050675a30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.252+0000 7feb5f1ae740 1 --2- 172.21.15.44:0/3881145593 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066aa60 0x55b050673100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.252+0000 7feb5f1ae740 1 --2- 172.21.15.44:0/3881145593 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b05066a140 0x55b05066a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.252+0000 7feb5f1ae740 1 -- 172.21.15.44:0/3881145593 >> 172.21.15.44:0/3881145593 conn(0x55b050664870 msgr2=0x55b050664c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:30.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.252+0000 7feb5f1ae740 1 -- 172.21.15.44:0/3881145593 shutdown_connections 2024-04-21T23:10:30.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.252+0000 7feb5f1ae740 1 -- 172.21.15.44:0/3881145593 wait complete. 2024-04-21T23:10:30.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.252+0000 7feb5f1ae740 1 -- start start 2024-04-21T23:10:30.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.252+0000 7feb5f1ae740 1 Processor -- start 2024-04-21T23:10:30.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.253+0000 7feb5f1ae740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066a140 0x55b050670660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.253+0000 7feb5f1ae740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b05066aa60 0x55b050670ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.253+0000 7feb5f1ae740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b050673640 0x55b0506710e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.253+0000 7feb5f1ae740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b0506763d0 con 0x55b05066a140 2024-04-21T23:10:30.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.253+0000 7feb5f1ae740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55b050587e00 con 0x55b05066aa60 2024-04-21T23:10:30.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.253+0000 7feb5f1ae740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b050676550 con 0x55b050673640 2024-04-21T23:10:30.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.253+0000 7feb5f1ae740 10 client.0 _ll_register_callbacks cb 0x55b0505c2eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:30.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.253+0000 7feb5a282640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066a140 0x55b050670660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.253+0000 7feb5a282640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066a140 0x55b050670660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59524/0 (socket says 172.21.15.44:59524) 2024-04-21T23:10:30.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.253+0000 7feb5a282640 1 -- 172.21.15.44:0/2566056436 learned_addr learned my addr 172.21.15.44:0/2566056436 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:30.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.253+0000 7feb5aa83640 1 --2- 172.21.15.44:0/2566056436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b050673640 0x55b0506710e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.253+0000 7feb59a81640 1 --2- 172.21.15.44:0/2566056436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b05066aa60 0x55b050670ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.253+0000 7feb5a282640 1 -- 172.21.15.44:0/2566056436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b050673640 msgr2=0x55b0506710e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:30.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.253+0000 7feb5a282640 1 --2- 172.21.15.44:0/2566056436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b050673640 0x55b0506710e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.253+0000 7feb5a282640 1 -- 172.21.15.44:0/2566056436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b05066aa60 msgr2=0x55b050670ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:30.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.253+0000 7feb5a282640 1 --2- 172.21.15.44:0/2566056436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b05066aa60 0x55b050670ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.253+0000 7feb5a282640 1 -- 172.21.15.44:0/2566056436 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0502e03f0 con 0x55b05066a140 2024-04-21T23:10:30.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.253+0000 7feb59a81640 1 --2- 172.21.15.44:0/2566056436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b05066aa60 0x55b050670ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:30.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.253+0000 7feb5a282640 1 --2- 172.21.15.44:0/2566056436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066a140 0x55b050670660 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7feb48009900 tx=0x7feb4802c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e90d2ae59ac59807 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:30.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.253+0000 7feb3dffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:10:30.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.254+0000 7feb3dffb640 1 -- 172.21.15.44:0/2566056436 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb48044070 con 0x55b05066a140 2024-04-21T23:10:30.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.254+0000 7feb5f1ae740 1 -- 172.21.15.44:0/2566056436 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb5006ad00 con 0x55b05066a140 2024-04-21T23:10:30.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.254+0000 7feb3dffb640 1 -- 172.21.15.44:0/2566056436 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7feb480046c0 con 0x55b05066a140 2024-04-21T23:10:30.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.254+0000 7feb3dffb640 1 -- 172.21.15.44:0/2566056436 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7feb480048c0 con 0x55b05066a140 2024-04-21T23:10:30.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.254+0000 7feb5f1ae740 10 client.15945 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:30.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.254+0000 7feb5f1ae740 1 -- 172.21.15.44:0/2566056436 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b0502a00f0 con 0x55b05066a140 2024-04-21T23:10:30.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.254+0000 7feb3dffb640 1 -- 172.21.15.44:0/2566056436 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb5006ad00 con 0x55b05066a140 2024-04-21T23:10:30.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.254+0000 7feb3dffb640 1 -- 172.21.15.44:0/2566056436 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7feb48003820 con 0x55b05066a140 2024-04-21T23:10:30.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.254+0000 7feb5f1ae740 10 client.15945 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:30.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.254+0000 7feb5f1ae740 10 client.15945 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:30.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.254+0000 7feb5f1ae740 20 client.15945 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:30.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.254+0000 7feb5f1ae740 1 --2- 172.21.15.44:0/2566056436 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b050683990 0x55b050685de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.254+0000 7feb5f1ae740 4 client.15945 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:10:30.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.254+0000 7feb5f1ae740 1 -- 172.21.15.44:0/2566056436 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0505d9aa0 con 0x55b050683990 2024-04-21T23:10:30.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.255+0000 7feb59a81640 1 --2- 172.21.15.44:0/2566056436 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b050683990 0x55b050685de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.255+0000 7feb3dffb640 10 client.15945 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:30.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.255+0000 7feb59a81640 1 --2- 172.21.15.44:0/2566056436 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b050683990 0x55b050685de0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fe853399bc12c56 server_cookie=3717ff9234b893a9 in_seq=0 out_seq=0 2024-04-21T23:10:30.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.256+0000 7feb3dffb640 1 -- 172.21.15.44:0/2566056436 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b0505d9aa0 con 0x55b050683990 2024-04-21T23:10:30.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.256+0000 7feb3dffb640 10 client.15945 handle_command_reply: tid=0 2024-04-21T23:10:30.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.256+0000 7feb5f1ae740 1 client.15945 shutdown 2024-04-21T23:10:30.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.256+0000 7feb5f1ae740 1 -- 172.21.15.44:0/2566056436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066a140 msgr2=0x55b050670660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.256+0000 7feb5f1ae740 1 --2- 172.21.15.44:0/2566056436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066a140 0x55b050670660 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7feb48009900 tx=0x7feb4802c880 comp rx=0 tx=0).stop 2024-04-21T23:10:30.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.256+0000 7feb5f1ae740 1 -- 172.21.15.44:0/2566056436 shutdown_connections 2024-04-21T23:10:30.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.256+0000 7feb5f1ae740 1 --2- 172.21.15.44:0/2566056436 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b050683990 0x55b050685de0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.256+0000 7feb5f1ae740 1 --2- 172.21.15.44:0/2566056436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b050673640 0x55b0506710e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.256+0000 7feb5f1ae740 1 --2- 172.21.15.44:0/2566056436 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b05066aa60 0x55b050670ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.256+0000 7feb3dffb640 0 client.15945 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:30.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.256+0000 7feb5f1ae740 1 --2- 172.21.15.44:0/2566056436 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066a140 0x55b050670660 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.256+0000 7feb5f1ae740 1 -- 172.21.15.44:0/2566056436 >> 172.21.15.44:0/2566056436 conn(0x55b050664870 msgr2=0x55b0506665c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:30.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.256+0000 7feb5f1ae740 1 -- 172.21.15.44:0/2566056436 shutdown_connections 2024-04-21T23:10:30.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.256+0000 7feb5f1ae740 1 -- 172.21.15.44:0/2566056436 wait complete. 2024-04-21T23:10:30.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.256+0000 7feb5f1ae740 20 client.15945 trim_cache size 0 max 16384 2024-04-21T23:10:30.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.276+0000 7feb5f1ae740 1 Processor -- start 2024-04-21T23:10:30.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.276+0000 7feb5f1ae740 1 -- start start 2024-04-21T23:10:30.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.276+0000 7feb5f1ae740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b05066a140 0x55b050672d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.276+0000 7feb5f1ae740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066aa60 0x55b05067f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.276+0000 7feb5f1ae740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b050673640 0x55b050661560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.276+0000 7feb5f1ae740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7feb541aa980 con 0x55b05066aa60 2024-04-21T23:10:30.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.276+0000 7feb5f1ae740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7feb541ad200 con 0x55b050673640 2024-04-21T23:10:30.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.276+0000 7feb5f1ae740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7feb541aaff0 con 0x55b05066a140 2024-04-21T23:10:30.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.276+0000 7feb59a81640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066aa60 0x55b05067f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.276+0000 7feb59a81640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066aa60 0x55b05067f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59540/0 (socket says 172.21.15.44:59540) 2024-04-21T23:10:30.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.276+0000 7feb5a282640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b05066a140 0x55b050672d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.276+0000 7feb59a81640 1 -- 172.21.15.44:0/547250537 learned_addr learned my addr 172.21.15.44:0/547250537 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:30.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.276+0000 7feb5aa83640 1 --2- 172.21.15.44:0/547250537 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b050673640 0x55b050661560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.277+0000 7feb5a282640 1 -- 172.21.15.44:0/547250537 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b050673640 msgr2=0x55b050661560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:30.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.277+0000 7feb5a282640 1 --2- 172.21.15.44:0/547250537 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b050673640 0x55b050661560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.277+0000 7feb5a282640 1 -- 172.21.15.44:0/547250537 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066aa60 msgr2=0x55b05067f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:30.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.277+0000 7feb5a282640 1 --2- 172.21.15.44:0/547250537 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066aa60 0x55b05067f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.277+0000 7feb5a282640 1 -- 172.21.15.44:0/547250537 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b050665aa0 con 0x55b05066a140 2024-04-21T23:10:30.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.277+0000 7feb59a81640 1 --2- 172.21.15.44:0/547250537 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066aa60 0x55b05067f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:30.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.277+0000 7feb5a282640 1 --2- 172.21.15.44:0/547250537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b05066a140 0x55b050672d90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7feb48009850 tx=0x7feb48000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b15f01eae374920f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:30.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.277+0000 7feb3ffff640 1 -- 172.21.15.44:0/547250537 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb48000e60 con 0x55b05066a140 2024-04-21T23:10:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.277+0000 7feb3ffff640 1 -- 172.21.15.44:0/547250537 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb48034630 con 0x55b05066a140 2024-04-21T23:10:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.277+0000 7feb3ffff640 1 -- 172.21.15.44:0/547250537 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb48044070 con 0x55b05066a140 2024-04-21T23:10:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.278+0000 7feb5f1ae740 1 -- 172.21.15.44:0/547250537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b05066a140 msgr2=0x55b050672d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.278+0000 7feb5f1ae740 1 --2- 172.21.15.44:0/547250537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b05066a140 0x55b050672d90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7feb48009850 tx=0x7feb48000bc0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.278+0000 7feb5f1ae740 1 -- 172.21.15.44:0/547250537 shutdown_connections 2024-04-21T23:10:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.278+0000 7feb5f1ae740 1 --2- 172.21.15.44:0/547250537 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b050673640 0x55b050661560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.278+0000 7feb5f1ae740 1 --2- 172.21.15.44:0/547250537 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066aa60 0x55b05067f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.278+0000 7feb5f1ae740 1 --2- 172.21.15.44:0/547250537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b05066a140 0x55b050672d90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.278+0000 7feb5f1ae740 1 -- 172.21.15.44:0/547250537 >> 172.21.15.44:0/547250537 conn(0x55b050664870 msgr2=0x55b050660e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.278+0000 7feb5f1ae740 1 -- 172.21.15.44:0/547250537 shutdown_connections 2024-04-21T23:10:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.278+0000 7feb5f1ae740 1 -- 172.21.15.44:0/547250537 wait complete. 2024-04-21T23:10:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.278+0000 7feb5f1ae740 1 -- start start 2024-04-21T23:10:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.278+0000 7feb5f1ae740 1 Processor -- start 2024-04-21T23:10:30.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb5f1ae740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b05066a140 0x55b05067b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb5f1ae740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066aa60 0x55b05066ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb5f1ae740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b050673640 0x55b05066d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb5f1ae740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7feb541ab6c0 con 0x55b05066aa60 2024-04-21T23:10:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb5f1ae740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7feb541acb30 con 0x55b05066a140 2024-04-21T23:10:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb5f1ae740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7feb541abd90 con 0x55b050673640 2024-04-21T23:10:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb5f1ae740 10 client.0 _ll_register_callbacks cb 0x7feb48009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb59a81640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066aa60 0x55b05066ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb5a282640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b05066a140 0x55b05067b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb59a81640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066aa60 0x55b05066ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59556/0 (socket says 172.21.15.44:59556) 2024-04-21T23:10:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb59a81640 1 -- 172.21.15.44:0/3908590558 learned_addr learned my addr 172.21.15.44:0/3908590558 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb5aa83640 1 --2- 172.21.15.44:0/3908590558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b050673640 0x55b05066d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb5a282640 1 -- 172.21.15.44:0/3908590558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b050673640 msgr2=0x55b05066d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb5a282640 1 --2- 172.21.15.44:0/3908590558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b050673640 0x55b05066d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb5a282640 1 -- 172.21.15.44:0/3908590558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066aa60 msgr2=0x55b05066ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb5a282640 1 --2- 172.21.15.44:0/3908590558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066aa60 0x55b05066ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb5a282640 1 -- 172.21.15.44:0/3908590558 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b05066da30 con 0x55b05066a140 2024-04-21T23:10:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb59a81640 1 --2- 172.21.15.44:0/3908590558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066aa60 0x55b05066ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb5aa83640 1 --2- 172.21.15.44:0/3908590558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b050673640 0x55b05066d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb5a282640 1 --2- 172.21.15.44:0/3908590558 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b05066a140 0x55b05067b990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feb4802c250 tx=0x7feb48034a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1089894128fe4267 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.279+0000 7feb3dffb640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:10:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.280+0000 7feb3dffb640 1 -- 172.21.15.44:0/3908590558 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb48044070 con 0x55b05066a140 2024-04-21T23:10:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.280+0000 7feb3dffb640 1 -- 172.21.15.44:0/3908590558 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7feb48055440 con 0x55b05066a140 2024-04-21T23:10:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.280+0000 7feb5f1ae740 1 -- 172.21.15.44:0/3908590558 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb48034630 con 0x55b05066a140 2024-04-21T23:10:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.280+0000 7feb3dffb640 1 -- 172.21.15.44:0/3908590558 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7feb480043e0 con 0x55b05066a140 2024-04-21T23:10:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.280+0000 7feb5f1ae740 10 client.25526 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.280+0000 7feb5f1ae740 1 -- 172.21.15.44:0/3908590558 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b0506e9d60 con 0x55b05066a140 2024-04-21T23:10:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.280+0000 7feb3dffb640 1 -- 172.21.15.44:0/3908590558 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb48034630 con 0x55b05066a140 2024-04-21T23:10:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.280+0000 7feb3dffb640 1 -- 172.21.15.44:0/3908590558 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7feb48040420 con 0x55b05066a140 2024-04-21T23:10:30.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.280+0000 7feb5f1ae740 10 client.25526 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:30.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.280+0000 7feb5f1ae740 10 client.25526 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:30.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.280+0000 7feb5f1ae740 20 client.25526 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:30.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.280+0000 7feb5f1ae740 1 --2- 172.21.15.44:0/3908590558 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b0506ea230 0x55b0506ec620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.280+0000 7feb5f1ae740 4 client.25526 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:10:30.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.280+0000 7feb5f1ae740 1 -- 172.21.15.44:0/3908590558 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b0505d3df0 con 0x55b0506ea230 2024-04-21T23:10:30.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.280+0000 7feb59a81640 1 --2- 172.21.15.44:0/3908590558 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b0506ea230 0x55b0506ec620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.281+0000 7feb3dffb640 10 client.25526 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:30.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.281+0000 7feb59a81640 1 --2- 172.21.15.44:0/3908590558 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b0506ea230 0x55b0506ec620 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a16799b0abbe167 server_cookie=ff204c63c67118bd in_seq=0 out_seq=0 2024-04-21T23:10:30.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.283+0000 7feb3dffb640 1 -- 172.21.15.44:0/3908590558 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b0505d3df0 con 0x55b0506ea230 2024-04-21T23:10:30.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.283+0000 7feb3dffb640 10 client.25526 handle_command_reply: tid=0 2024-04-21T23:10:30.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.283+0000 7feb5f1ae740 1 client.25526 shutdown 2024-04-21T23:10:30.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.283+0000 7feb5f1ae740 1 -- 172.21.15.44:0/3908590558 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b05066a140 msgr2=0x55b05067b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.283+0000 7feb5f1ae740 1 --2- 172.21.15.44:0/3908590558 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b05066a140 0x55b05067b990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feb4802c250 tx=0x7feb48034a10 comp rx=0 tx=0).stop 2024-04-21T23:10:30.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.283+0000 7feb5f1ae740 1 -- 172.21.15.44:0/3908590558 shutdown_connections 2024-04-21T23:10:30.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.283+0000 7feb5f1ae740 1 --2- 172.21.15.44:0/3908590558 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b0506ea230 0x55b0506ec620 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.283+0000 7feb5f1ae740 1 --2- 172.21.15.44:0/3908590558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b050673640 0x55b05066d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.283+0000 7feb3dffb640 0 client.25526 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:30.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.283+0000 7feb5f1ae740 1 --2- 172.21.15.44:0/3908590558 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b05066aa60 0x55b05066ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.283+0000 7feb5f1ae740 1 --2- 172.21.15.44:0/3908590558 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b05066a140 0x55b05067b990 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.283+0000 7feb5f1ae740 1 -- 172.21.15.44:0/3908590558 >> 172.21.15.44:0/3908590558 conn(0x55b050664870 msgr2=0x55b0506616e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:30.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.283+0000 7feb5f1ae740 1 -- 172.21.15.44:0/3908590558 shutdown_connections 2024-04-21T23:10:30.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.283+0000 7feb5f1ae740 1 -- 172.21.15.44:0/3908590558 wait complete. 2024-04-21T23:10:30.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.283+0000 7feb5f1ae740 20 client.25526 trim_cache size 0 max 16384 2024-04-21T23:10:30.290 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:10:30.290 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-21T23:10:30.290 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-21T23:10:30.290 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:10:30.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.284+0000 7feb5c4e6640 1 -- 172.21.15.44:0/3373171154 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7feb400a2d80 msgr2=0x7feb400a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.284+0000 7feb5c4e6640 1 --2- 172.21.15.44:0/3373171154 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7feb400a2d80 0x7feb400a5240 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feb4c004650 tx=0x7feb4c0046c0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.284+0000 7feb5c4e6640 1 -- 172.21.15.44:0/3373171154 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb54102120 msgr2=0x7feb54197570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.284+0000 7feb5c4e6640 1 --2- 172.21.15.44:0/3373171154 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb54102120 0x7feb54197570 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7feb5000ebc0 tx=0x7feb5000c6a0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.284+0000 7feb5c4e6640 1 -- 172.21.15.44:0/3373171154 shutdown_connections 2024-04-21T23:10:30.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.284+0000 7feb5c4e6640 1 --2- 172.21.15.44:0/3373171154 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7feb400a2d80 0x7feb400a5240 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.284+0000 7feb5c4e6640 1 --2- 172.21.15.44:0/3373171154 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb54102ac0 0x7feb5419b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.284+0000 7feb5c4e6640 1 --2- 172.21.15.44:0/3373171154 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb54102120 0x7feb54197570 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.284+0000 7feb5c4e6640 1 --2- 172.21.15.44:0/3373171154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb541009e0 0x7feb54197030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.284+0000 7feb5c4e6640 1 -- 172.21.15.44:0/3373171154 >> 172.21.15.44:0/3373171154 conn(0x7feb54077760 msgr2=0x7feb54105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:30.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.284+0000 7feb5c4e6640 1 -- 172.21.15.44:0/3373171154 shutdown_connections 2024-04-21T23:10:30.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.284+0000 7feb5c4e6640 1 -- 172.21.15.44:0/3373171154 wait complete. 2024-04-21T23:10:30.295 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:10:30.295 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:10:30.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.401+0000 7fb40d03b640 1 -- 172.21.15.44:0/3330054580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4080693e0 msgr2=0x7fb40810e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.401+0000 7fb40d03b640 1 --2- 172.21.15.44:0/3330054580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4080693e0 0x7fb40810e5e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb400009be0 tx=0x7fb40002bef0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.401+0000 7fb40d03b640 1 -- 172.21.15.44:0/3330054580 shutdown_connections 2024-04-21T23:10:30.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.401+0000 7fb40d03b640 1 --2- 172.21.15.44:0/3330054580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4080693e0 0x7fb40810e5e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.401+0000 7fb40d03b640 1 --2- 172.21.15.44:0/3330054580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb408068a40 0x7fb408068ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.401+0000 7fb40d03b640 1 --2- 172.21.15.44:0/3330054580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb408107800 0x7fb408107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.401+0000 7fb40d03b640 1 -- 172.21.15.44:0/3330054580 >> 172.21.15.44:0/3330054580 conn(0x7fb408077430 msgr2=0x7fb408077840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:30.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.401+0000 7fb40d03b640 1 -- 172.21.15.44:0/3330054580 shutdown_connections 2024-04-21T23:10:30.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.401+0000 7fb40d03b640 1 -- 172.21.15.44:0/3330054580 wait complete. 2024-04-21T23:10:30.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.402+0000 7fb40d03b640 1 Processor -- start 2024-04-21T23:10:30.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.402+0000 7fb40d03b640 1 -- start start 2024-04-21T23:10:30.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.402+0000 7fb40d03b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb408068a40 0x7fb408197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.402+0000 7fb40d03b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4080693e0 0x7fb408198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.402+0000 7fb40d03b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb408107800 0x7fb40819c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.402+0000 7fb40d03b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb408111b20 con 0x7fb4080693e0 2024-04-21T23:10:30.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.402+0000 7fb40d03b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb4081119a0 con 0x7fb408107800 2024-04-21T23:10:30.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.402+0000 7fb40d03b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb408111ca0 con 0x7fb408068a40 2024-04-21T23:10:30.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.402+0000 7fb3fffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4080693e0 0x7fb408198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.402+0000 7fb3fffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4080693e0 0x7fb408198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59576/0 (socket says 172.21.15.44:59576) 2024-04-21T23:10:30.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.402+0000 7fb3fffff640 1 -- 172.21.15.44:0/3872135227 learned_addr learned my addr 172.21.15.44:0/3872135227 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:30.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.403+0000 7fb40759e640 1 --2- 172.21.15.44:0/3872135227 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb408107800 0x7fb40819c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.403+0000 7fb406d9d640 1 --2- 172.21.15.44:0/3872135227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb408068a40 0x7fb408197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.403+0000 7fb3fffff640 1 -- 172.21.15.44:0/3872135227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb408068a40 msgr2=0x7fb408197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.403+0000 7fb3fffff640 1 --2- 172.21.15.44:0/3872135227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb408068a40 0x7fb408197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.403+0000 7fb3fffff640 1 -- 172.21.15.44:0/3872135227 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb408107800 msgr2=0x7fb40819c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.403+0000 7fb3fffff640 1 --2- 172.21.15.44:0/3872135227 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb408107800 0x7fb40819c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.403+0000 7fb3fffff640 1 -- 172.21.15.44:0/3872135227 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40819cef0 con 0x7fb4080693e0 2024-04-21T23:10:30.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.403+0000 7fb3fffff640 1 --2- 172.21.15.44:0/3872135227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4080693e0 0x7fb408198440 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000e9f0 tx=0x7fb3f000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:30.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.403+0000 7fb404d99640 1 -- 172.21.15.44:0/3872135227 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f000ccc0 con 0x7fb4080693e0 2024-04-21T23:10:30.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.403+0000 7fb40d03b640 1 -- 172.21.15.44:0/3872135227 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb40819d1e0 con 0x7fb4080693e0 2024-04-21T23:10:30.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.403+0000 7fb404d99640 1 -- 172.21.15.44:0/3872135227 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3f000ce60 con 0x7fb4080693e0 2024-04-21T23:10:30.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.403+0000 7fb40d03b640 1 -- 172.21.15.44:0/3872135227 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4081a4a70 con 0x7fb4080693e0 2024-04-21T23:10:30.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.403+0000 7fb404d99640 1 -- 172.21.15.44:0/3872135227 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0010690 con 0x7fb4080693e0 2024-04-21T23:10:30.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.404+0000 7fb40d03b640 1 -- 172.21.15.44:0/3872135227 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4080685f0 con 0x7fb4080693e0 2024-04-21T23:10:30.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.405+0000 7fb404d99640 1 -- 172.21.15.44:0/3872135227 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb3f00046c0 con 0x7fb4080693e0 2024-04-21T23:10:30.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.406+0000 7fb404d99640 1 --2- 172.21.15.44:0/3872135227 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb3f40a2d00 0x7fb3f40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.406+0000 7fb404d99640 1 -- 172.21.15.44:0/3872135227 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fb3f00cfbd0 con 0x7fb4080693e0 2024-04-21T23:10:30.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.406+0000 7fb406d9d640 1 --2- 172.21.15.44:0/3872135227 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb3f40a2d00 0x7fb3f40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.407 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.406+0000 7fb406d9d640 1 --2- 172.21.15.44:0/3872135227 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb3f40a2d00 0x7fb3f40a51c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8005fd0 tx=0x7fb3f8009260 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:30.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.411+0000 7fb404d99640 1 -- 172.21.15.44:0/3872135227 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb3f009c450 con 0x7fb4080693e0 2024-04-21T23:10:30.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:30 smithi044 ceph-mon[27934]: pgmap v52: 129 pgs: 129 active+clean; 19 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 1.4 MiB/s wr, 196 op/s 2024-04-21T23:10:30.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:30 smithi069 ceph-mon[31816]: pgmap v52: 129 pgs: 129 active+clean; 19 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 1.4 MiB/s wr, 196 op/s 2024-04-21T23:10:30.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:30 smithi055 ceph-mon[31117]: pgmap v52: 129 pgs: 129 active+clean; 19 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 1.4 MiB/s wr, 196 op/s 2024-04-21T23:10:30.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.685+0000 7fb40d03b640 1 -- 172.21.15.44:0/3872135227 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4081a4d60 con 0x7fb4080693e0 2024-04-21T23:10:30.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.686+0000 7fb404d99640 1 -- 172.21.15.44:0/3872135227 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11791 (secure 0 0 0) 0x7fb3f009bd50 con 0x7fb4080693e0 2024-04-21T23:10:30.686 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:10:30.686 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:06:50.685663+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3743898853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3743898853},{"type":"v1","addr":"172.21.15.69:6839","nonce":3743898853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:10:30.686 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2024-04-21T23:10:30.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.688+0000 7fb40d03b640 1 -- 172.21.15.44:0/3872135227 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb3f40a2d00 msgr2=0x7fb3f40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.688+0000 7fb40d03b640 1 --2- 172.21.15.44:0/3872135227 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb3f40a2d00 0x7fb3f40a51c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8005fd0 tx=0x7fb3f8009260 comp rx=0 tx=0).stop 2024-04-21T23:10:30.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.688+0000 7fb40d03b640 1 -- 172.21.15.44:0/3872135227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4080693e0 msgr2=0x7fb408198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.688+0000 7fb40d03b640 1 --2- 172.21.15.44:0/3872135227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4080693e0 0x7fb408198440 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000e9f0 tx=0x7fb3f000edf0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.688+0000 7fb40d03b640 1 -- 172.21.15.44:0/3872135227 shutdown_connections 2024-04-21T23:10:30.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.688+0000 7fb40d03b640 1 --2- 172.21.15.44:0/3872135227 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb3f40a2d00 0x7fb3f40a51c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.688+0000 7fb40d03b640 1 --2- 172.21.15.44:0/3872135227 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb408107800 0x7fb40819c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.688+0000 7fb40d03b640 1 --2- 172.21.15.44:0/3872135227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4080693e0 0x7fb408198440 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.688+0000 7fb40d03b640 1 --2- 172.21.15.44:0/3872135227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb408068a40 0x7fb408197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.688+0000 7fb40d03b640 1 -- 172.21.15.44:0/3872135227 >> 172.21.15.44:0/3872135227 conn(0x7fb408077430 msgr2=0x7fb4080ff2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:30.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.688+0000 7fb40d03b640 1 -- 172.21.15.44:0/3872135227 shutdown_connections 2024-04-21T23:10:30.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.688+0000 7fb40d03b640 1 -- 172.21.15.44:0/3872135227 wait complete. 2024-04-21T23:10:30.691 INFO:tasks.workunit.client.0.smithi044.stdout: 1 57667 16.38 MB/sec warmup 20 sec latency 124.494 ms 2024-04-21T23:10:30.699 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:10:30.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.810+0000 7ff7b3066640 1 -- 172.21.15.44:0/1248294453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7ac100a10 msgr2=0x7ff7ac10cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.810+0000 7ff7b3066640 1 --2- 172.21.15.44:0/1248294453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7ac100a10 0x7ff7ac10cd50 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009ca0 tx=0x7ff7a402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.810+0000 7ff7b3066640 1 -- 172.21.15.44:0/1248294453 shutdown_connections 2024-04-21T23:10:30.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.810+0000 7ff7b3066640 1 --2- 172.21.15.44:0/1248294453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7ac100070 0x7ff7ac1004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.810+0000 7ff7b3066640 1 --2- 172.21.15.44:0/1248294453 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7ac100a10 0x7ff7ac10cd50 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.810+0000 7ff7b3066640 1 --2- 172.21.15.44:0/1248294453 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7ac101d00 0x7ff7ac1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.810+0000 7ff7b3066640 1 -- 172.21.15.44:0/1248294453 >> 172.21.15.44:0/1248294453 conn(0x7ff7ac0758c0 msgr2=0x7ff7ac075cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:30.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.810+0000 7ff7b3066640 1 -- 172.21.15.44:0/1248294453 shutdown_connections 2024-04-21T23:10:30.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.811+0000 7ff7b3066640 1 -- 172.21.15.44:0/1248294453 wait complete. 2024-04-21T23:10:30.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.811+0000 7ff7b3066640 1 Processor -- start 2024-04-21T23:10:30.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.811+0000 7ff7b3066640 1 -- start start 2024-04-21T23:10:30.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.811+0000 7ff7b3066640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7ac100070 0x7ff7ac198e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.811+0000 7ff7b3066640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7ac100a10 0x7ff7ac199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.811+0000 7ff7b3066640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7ac101d00 0x7ff7ac1a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.812+0000 7ff7b3066640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac110b30 con 0x7ff7ac101d00 2024-04-21T23:10:30.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.812+0000 7ff7b3066640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac1109b0 con 0x7ff7ac100070 2024-04-21T23:10:30.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.812+0000 7ff7b3066640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac110cb0 con 0x7ff7ac100a10 2024-04-21T23:10:30.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.812+0000 7ff7b1603640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7ac101d00 0x7ff7ac1a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.812+0000 7ff7b1603640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7ac101d00 0x7ff7ac1a3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50076/0 (socket says 172.21.15.44:50076) 2024-04-21T23:10:30.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.812+0000 7ff7b0e02640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7ac100070 0x7ff7ac198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.812+0000 7ff7b1603640 1 -- 172.21.15.44:0/2236989091 learned_addr learned my addr 172.21.15.44:0/2236989091 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:30.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.812+0000 7ff7abfff640 1 --2- 172.21.15.44:0/2236989091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7ac100a10 0x7ff7ac199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.812+0000 7ff7b0e02640 1 -- 172.21.15.44:0/2236989091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7ac100a10 msgr2=0x7ff7ac199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.812+0000 7ff7b0e02640 1 --2- 172.21.15.44:0/2236989091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7ac100a10 0x7ff7ac199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.812+0000 7ff7b0e02640 1 -- 172.21.15.44:0/2236989091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7ac101d00 msgr2=0x7ff7ac1a3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.812+0000 7ff7b0e02640 1 --2- 172.21.15.44:0/2236989091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7ac101d00 0x7ff7ac1a3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.812+0000 7ff7b0e02640 1 -- 172.21.15.44:0/2236989091 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ac1a4410 con 0x7ff7ac100070 2024-04-21T23:10:30.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.812+0000 7ff7b1603640 1 --2- 172.21.15.44:0/2236989091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7ac101d00 0x7ff7ac1a3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:30.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.812+0000 7ff7b0e02640 1 --2- 172.21.15.44:0/2236989091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7ac100070 0x7ff7ac198e50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff79c00e9f0 tx=0x7ff79c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:30.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.812+0000 7ff7a9ffb640 1 -- 172.21.15.44:0/2236989091 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79c00ccc0 con 0x7ff7ac100070 2024-04-21T23:10:30.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.812+0000 7ff7b3066640 1 -- 172.21.15.44:0/2236989091 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7ac1a4640 con 0x7ff7ac100070 2024-04-21T23:10:30.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.812+0000 7ff7a9ffb640 1 -- 172.21.15.44:0/2236989091 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff79c010910 con 0x7ff7ac100070 2024-04-21T23:10:30.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.812+0000 7ff7a9ffb640 1 -- 172.21.15.44:0/2236989091 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79c018960 con 0x7ff7ac100070 2024-04-21T23:10:30.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.813+0000 7ff7b3066640 1 -- 172.21.15.44:0/2236989091 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7ac1a4bb0 con 0x7ff7ac100070 2024-04-21T23:10:30.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.815+0000 7ff7a9ffb640 1 -- 172.21.15.44:0/2236989091 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff79c010ac0 con 0x7ff7ac100070 2024-04-21T23:10:30.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.815+0000 7ff7b5c48740 1 Processor -- start 2024-04-21T23:10:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.815+0000 7ff7b5c48740 1 -- start start 2024-04-21T23:10:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.815+0000 7ff7a9ffb640 1 --2- 172.21.15.44:0/2236989091 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff7940a2d00 0x7ff7940a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7b5c48740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a277142020 0x55a277142400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7b5c48740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a277142940 0x55a27714afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7b5c48740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a27714b520 0x55a27714d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7b5c48740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a2770a6530 con 0x55a27714b520 2024-04-21T23:10:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7b5c48740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55a27704e830 con 0x55a277142020 2024-04-21T23:10:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7b5c48740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a2770a24f0 con 0x55a277142940 2024-04-21T23:10:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7a9ffb640 1 -- 172.21.15.44:0/2236989091 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7ff79c0cfb00 con 0x7ff7ac100070 2024-04-21T23:10:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7b0e02640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a277142020 0x55a277142400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7b1603640 1 --2- 172.21.15.44:0/2236989091 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff7940a2d00 0x7ff7940a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7abfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a277142940 0x55a27714afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7b0e02640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a277142020 0x55a277142400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:57394/0 (socket says 172.21.15.44:57394) 2024-04-21T23:10:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7b0e02640 1 -- 172.21.15.44:0/60455114 learned_addr learned my addr 172.21.15.44:0/60455114 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7b1603640 1 --2- 172.21.15.44:0/60455114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a27714b520 0x55a27714d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7b0e02640 1 -- 172.21.15.44:0/60455114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a277142940 msgr2=0x55a27714afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7b0e02640 1 --2- 172.21.15.44:0/60455114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a277142940 0x55a27714afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7b0e02640 1 -- 172.21.15.44:0/60455114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a27714b520 msgr2=0x55a27714d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7b0e02640 1 --2- 172.21.15.44:0/60455114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a27714b520 0x55a27714d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7b0e02640 1 -- 172.21.15.44:0/60455114 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2770722c0 con 0x55a277142020 2024-04-21T23:10:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7abfff640 1 --2- 172.21.15.44:0/60455114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a277142940 0x55a27714afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7b1603640 1 --2- 172.21.15.44:0/60455114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a27714b520 0x55a27714d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:10:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7b0e02640 1 --2- 172.21.15.44:0/60455114 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a277142020 0x55a277142400 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff79c098700 tx=0x7ff79c09bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65dd948dd5130a22 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff7b1603640 1 --2- 172.21.15.44:0/2236989091 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff7940a2d00 0x7ff7940a51c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55a27713d8e0 tx=0x7ff7a402e040 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff792ffd640 1 -- 172.21.15.44:0/60455114 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79c07bf20 con 0x55a277142020 2024-04-21T23:10:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff792ffd640 1 -- 172.21.15.44:0/60455114 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff79c078f40 con 0x55a277142020 2024-04-21T23:10:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.816+0000 7ff792ffd640 1 -- 172.21.15.44:0/60455114 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79c0983a0 con 0x55a277142020 2024-04-21T23:10:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.817+0000 7ff7b5c48740 1 -- 172.21.15.44:0/60455114 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a277142020 msgr2=0x55a277142400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.817+0000 7ff7b5c48740 1 --2- 172.21.15.44:0/60455114 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a277142020 0x55a277142400 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff79c098700 tx=0x7ff79c09bc30 comp rx=0 tx=0).stop 2024-04-21T23:10:30.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.817+0000 7ff7b5c48740 1 -- 172.21.15.44:0/60455114 shutdown_connections 2024-04-21T23:10:30.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.817+0000 7ff7b5c48740 1 --2- 172.21.15.44:0/60455114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a277142940 0x55a27714afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.817+0000 7ff7b5c48740 1 --2- 172.21.15.44:0/60455114 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a27714b520 0x55a27714d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.817+0000 7ff7b5c48740 1 --2- 172.21.15.44:0/60455114 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a277142020 0x55a277142400 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.817+0000 7ff7b5c48740 1 -- 172.21.15.44:0/60455114 >> 172.21.15.44:0/60455114 conn(0x55a27713c750 msgr2=0x55a27713cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:30.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.817+0000 7ff7b5c48740 1 -- 172.21.15.44:0/60455114 shutdown_connections 2024-04-21T23:10:30.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.817+0000 7ff7b5c48740 1 -- 172.21.15.44:0/60455114 wait complete. 2024-04-21T23:10:30.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.817+0000 7ff7b5c48740 1 -- start start 2024-04-21T23:10:30.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.818+0000 7ff7b5c48740 1 Processor -- start 2024-04-21T23:10:30.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.818+0000 7ff7b5c48740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142020 0x55a27714aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.818+0000 7ff7b5c48740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a277142940 0x55a277148250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.818+0000 7ff7b5c48740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a27714b520 0x55a2771487f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.818+0000 7ff7b5c48740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a27714e2b0 con 0x55a277142020 2024-04-21T23:10:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.818+0000 7ff7b5c48740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55a277061f10 con 0x55a27714b520 2024-04-21T23:10:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.818+0000 7ff7b5c48740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a27714e430 con 0x55a277142940 2024-04-21T23:10:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.818+0000 7ff7b5c48740 10 client.0 _ll_register_callbacks cb 0x55a276db9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.818+0000 7ff7b0e02640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142020 0x55a27714aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.818+0000 7ff7abfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a277142940 0x55a277148250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.818+0000 7ff7b0e02640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142020 0x55a27714aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50098/0 (socket says 172.21.15.44:50098) 2024-04-21T23:10:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.818+0000 7ff7b1603640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a27714b520 0x55a2771487f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.818+0000 7ff7b0e02640 1 -- 172.21.15.44:0/2869584147 learned_addr learned my addr 172.21.15.44:0/2869584147 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.818+0000 7ff7b0e02640 1 -- 172.21.15.44:0/2869584147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a277142940 msgr2=0x55a277148250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.818+0000 7ff7b0e02640 1 --2- 172.21.15.44:0/2869584147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a277142940 0x55a277148250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.818+0000 7ff7b0e02640 1 -- 172.21.15.44:0/2869584147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a27714b520 msgr2=0x55a2771487f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.818+0000 7ff7b0e02640 1 --2- 172.21.15.44:0/2869584147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a27714b520 0x55a2771487f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.819+0000 7ff7b0e02640 1 -- 172.21.15.44:0/2869584147 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a277033c80 con 0x55a277142020 2024-04-21T23:10:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.819+0000 7ff7b0e02640 1 --2- 172.21.15.44:0/2869584147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142020 0x55a27714aec0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ff79c09bcc0 tx=0x7ff79c0cd130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=174c9d7e70bd9b9e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.819+0000 7ff78bfff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:10:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.819+0000 7ff78bfff640 1 -- 172.21.15.44:0/2869584147 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79c09b380 con 0x55a277142020 2024-04-21T23:10:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.819+0000 7ff78bfff640 1 -- 172.21.15.44:0/2869584147 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7ff79c010430 con 0x55a277142020 2024-04-21T23:10:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.819+0000 7ff7b5c48740 1 -- 172.21.15.44:0/2869584147 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff79c078f40 con 0x55a277142020 2024-04-21T23:10:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.819+0000 7ff78bfff640 1 -- 172.21.15.44:0/2869584147 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff79c0a0030 con 0x55a277142020 2024-04-21T23:10:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.819+0000 7ff7b5c48740 10 client.15969 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.819+0000 7ff7b5c48740 1 -- 172.21.15.44:0/2869584147 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a276dba3f0 con 0x55a277142020 2024-04-21T23:10:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.819+0000 7ff78bfff640 1 -- 172.21.15.44:0/2869584147 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff79c078f40 con 0x55a277142020 2024-04-21T23:10:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.819+0000 7ff78bfff640 1 -- 172.21.15.44:0/2869584147 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7ff79c0b4060 con 0x55a277142020 2024-04-21T23:10:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.819+0000 7ff7b5c48740 10 client.15969 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.819+0000 7ff7b5c48740 10 client.15969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.819+0000 7ff7b5c48740 20 client.15969 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.819+0000 7ff7b5c48740 1 --2- 172.21.15.44:0/2869584147 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a27715b3e0 0x55a27715d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.819+0000 7ff7b5c48740 4 client.15969 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:10:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.819+0000 7ff7b5c48740 1 -- 172.21.15.44:0/2869584147 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2770b3a70 con 0x55a27715b3e0 2024-04-21T23:10:30.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.820+0000 7ff7abfff640 1 --2- 172.21.15.44:0/2869584147 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a27715b3e0 0x55a27715d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.820+0000 7ff7abfff640 1 --2- 172.21.15.44:0/2869584147 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a27715b3e0 0x55a27715d7d0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b46332fee7c800ef server_cookie=65e8796beb570dff in_seq=0 out_seq=0 2024-04-21T23:10:30.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.820+0000 7ff78bfff640 10 client.15969 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:30.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.822+0000 7ff78bfff640 1 -- 172.21.15.44:0/2869584147 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a2770b3a70 con 0x55a27715b3e0 2024-04-21T23:10:30.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.822+0000 7ff78bfff640 10 client.15969 handle_command_reply: tid=0 2024-04-21T23:10:30.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.822+0000 7ff7b5c48740 1 client.15969 shutdown 2024-04-21T23:10:30.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.822+0000 7ff7b5c48740 1 -- 172.21.15.44:0/2869584147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142020 msgr2=0x55a27714aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.822+0000 7ff7b5c48740 1 --2- 172.21.15.44:0/2869584147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142020 0x55a27714aec0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ff79c09bcc0 tx=0x7ff79c0cd130 comp rx=0 tx=0).stop 2024-04-21T23:10:30.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.822+0000 7ff7b5c48740 1 -- 172.21.15.44:0/2869584147 shutdown_connections 2024-04-21T23:10:30.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.822+0000 7ff7b5c48740 1 --2- 172.21.15.44:0/2869584147 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a27715b3e0 0x55a27715d7d0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.822+0000 7ff7b5c48740 1 --2- 172.21.15.44:0/2869584147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a277142940 0x55a277148250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.822+0000 7ff78bfff640 0 client.15969 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:30.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.822+0000 7ff7b5c48740 1 --2- 172.21.15.44:0/2869584147 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a27714b520 0x55a2771487f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.822+0000 7ff7b5c48740 1 --2- 172.21.15.44:0/2869584147 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142020 0x55a27714aec0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.822+0000 7ff7b5c48740 1 -- 172.21.15.44:0/2869584147 >> 172.21.15.44:0/2869584147 conn(0x55a27713c750 msgr2=0x55a27713e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:30.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.822+0000 7ff7b5c48740 1 -- 172.21.15.44:0/2869584147 shutdown_connections 2024-04-21T23:10:30.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.822+0000 7ff7b5c48740 1 -- 172.21.15.44:0/2869584147 wait complete. 2024-04-21T23:10:30.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.822+0000 7ff7b5c48740 20 client.15969 trim_cache size 0 max 16384 2024-04-21T23:10:30.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.835+0000 7ff7b5c48740 1 Processor -- start 2024-04-21T23:10:30.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.835+0000 7ff7b5c48740 1 -- start start 2024-04-21T23:10:30.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.835+0000 7ff7b5c48740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142020 0x55a277157170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.835+0000 7ff7b5c48740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a277142940 0x55a277162d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.835+0000 7ff7b5c48740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a27714b520 0x55a2771632b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.835+0000 7ff7b5c48740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac1aaab0 con 0x55a277142020 2024-04-21T23:10:30.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.835+0000 7ff7b5c48740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac1ad330 con 0x55a27714b520 2024-04-21T23:10:30.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.835+0000 7ff7b5c48740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac1ab120 con 0x55a277142940 2024-04-21T23:10:30.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.836+0000 7ff7b0e02640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142020 0x55a277157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.836+0000 7ff7b1603640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a27714b520 0x55a2771632b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.836+0000 7ff7b0e02640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142020 0x55a277157170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50110/0 (socket says 172.21.15.44:50110) 2024-04-21T23:10:30.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.836+0000 7ff7b0e02640 1 -- 172.21.15.44:0/2966147549 learned_addr learned my addr 172.21.15.44:0/2966147549 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:30.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.836+0000 7ff7abfff640 1 --2- 172.21.15.44:0/2966147549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a277142940 0x55a277162d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.836+0000 7ff7b0e02640 1 -- 172.21.15.44:0/2966147549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a277142940 msgr2=0x55a277162d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:30.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.836+0000 7ff7b0e02640 1 --2- 172.21.15.44:0/2966147549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a277142940 0x55a277162d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.836+0000 7ff7b0e02640 1 -- 172.21.15.44:0/2966147549 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a27714b520 msgr2=0x55a2771632b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:30.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.836+0000 7ff7b0e02640 1 --2- 172.21.15.44:0/2966147549 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a27714b520 0x55a2771632b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.836+0000 7ff7b0e02640 1 -- 172.21.15.44:0/2966147549 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a276e847d0 con 0x55a277142020 2024-04-21T23:10:30.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.836+0000 7ff7b1603640 1 --2- 172.21.15.44:0/2966147549 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a27714b520 0x55a2771632b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:30.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.836+0000 7ff7abfff640 1 --2- 172.21.15.44:0/2966147549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a277142940 0x55a277162d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:30.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.836+0000 7ff7b0e02640 1 --2- 172.21.15.44:0/2966147549 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142020 0x55a277157170 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7ff79c0ccf60 tx=0x7ff79c09a240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d3315bcf512ba server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:30.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.836+0000 7ff7927fc640 1 -- 172.21.15.44:0/2966147549 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79c078f00 con 0x55a277142020 2024-04-21T23:10:30.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.836+0000 7ff7927fc640 1 -- 172.21.15.44:0/2966147549 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff79c0a48f0 con 0x55a277142020 2024-04-21T23:10:30.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.836+0000 7ff7927fc640 1 -- 172.21.15.44:0/2966147549 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79c09b3e0 con 0x55a277142020 2024-04-21T23:10:30.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.837+0000 7ff7b5c48740 1 -- 172.21.15.44:0/2966147549 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142020 msgr2=0x55a277157170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.837+0000 7ff7b5c48740 1 --2- 172.21.15.44:0/2966147549 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142020 0x55a277157170 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7ff79c0ccf60 tx=0x7ff79c09a240 comp rx=0 tx=0).stop 2024-04-21T23:10:30.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.837+0000 7ff7b5c48740 1 -- 172.21.15.44:0/2966147549 shutdown_connections 2024-04-21T23:10:30.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.837+0000 7ff7b5c48740 1 --2- 172.21.15.44:0/2966147549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a277142940 0x55a277162d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.837+0000 7ff7b5c48740 1 --2- 172.21.15.44:0/2966147549 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a27714b520 0x55a2771632b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.837+0000 7ff7b5c48740 1 --2- 172.21.15.44:0/2966147549 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142020 0x55a277157170 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.837+0000 7ff7b5c48740 1 -- 172.21.15.44:0/2966147549 >> 172.21.15.44:0/2966147549 conn(0x55a27713c750 msgr2=0x55a27713b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:30.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.837+0000 7ff7b5c48740 1 -- 172.21.15.44:0/2966147549 shutdown_connections 2024-04-21T23:10:30.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.837+0000 7ff7b5c48740 1 -- 172.21.15.44:0/2966147549 wait complete. 2024-04-21T23:10:30.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.838+0000 7ff7b5c48740 1 -- start start 2024-04-21T23:10:30.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.838+0000 7ff7b5c48740 1 Processor -- start 2024-04-21T23:10:30.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.838+0000 7ff7b5c48740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a277142020 0x55a2771538a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.838+0000 7ff7b5c48740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142940 0x55a27714ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.838+0000 7ff7b5c48740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a27714b520 0x55a27714f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.838+0000 7ff7b5c48740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac1ab7f0 con 0x55a277142940 2024-04-21T23:10:30.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.838+0000 7ff7b5c48740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac1acc60 con 0x55a277142020 2024-04-21T23:10:30.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.838+0000 7ff7b5c48740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac1abec0 con 0x55a27714b520 2024-04-21T23:10:30.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.838+0000 7ff7b5c48740 10 client.0 _ll_register_callbacks cb 0x7ff79c0cce20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:30.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.838+0000 7ff7abfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142940 0x55a27714ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.838+0000 7ff7b0e02640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a277142020 0x55a2771538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.838+0000 7ff7abfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142940 0x55a27714ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50112/0 (socket says 172.21.15.44:50112) 2024-04-21T23:10:30.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.838+0000 7ff7abfff640 1 -- 172.21.15.44:0/3997924344 learned_addr learned my addr 172.21.15.44:0/3997924344 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:30.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.838+0000 7ff7b1603640 1 --2- 172.21.15.44:0/3997924344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a27714b520 0x55a27714f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.838+0000 7ff7abfff640 1 -- 172.21.15.44:0/3997924344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a27714b520 msgr2=0x55a27714f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:30.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.838+0000 7ff7abfff640 1 --2- 172.21.15.44:0/3997924344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a27714b520 0x55a27714f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.838+0000 7ff7abfff640 1 -- 172.21.15.44:0/3997924344 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a277142020 msgr2=0x55a2771538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:30.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.838+0000 7ff7abfff640 1 --2- 172.21.15.44:0/3997924344 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a277142020 0x55a2771538a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.838+0000 7ff7abfff640 1 -- 172.21.15.44:0/3997924344 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a27714f7c0 con 0x55a277142940 2024-04-21T23:10:30.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.838+0000 7ff7b0e02640 1 --2- 172.21.15.44:0/3997924344 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a277142020 0x55a2771538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:30.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.839+0000 7ff7abfff640 1 --2- 172.21.15.44:0/3997924344 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142940 0x55a27714ec40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x55a277141af0 tx=0x7ff7a0014ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9fe3670d96313b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:30.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.839+0000 7ff78bfff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:10:30.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.839+0000 7ff78bfff640 1 -- 172.21.15.44:0/3997924344 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0027070 con 0x55a277142940 2024-04-21T23:10:30.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.839+0000 7ff78bfff640 1 -- 172.21.15.44:0/3997924344 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7ff7a0004050 con 0x55a277142940 2024-04-21T23:10:30.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.839+0000 7ff7b5c48740 1 -- 172.21.15.44:0/3997924344 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a2770addc0 con 0x55a277142940 2024-04-21T23:10:30.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.839+0000 7ff78bfff640 1 -- 172.21.15.44:0/3997924344 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff7a0033050 con 0x55a277142940 2024-04-21T23:10:30.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.839+0000 7ff7b5c48740 10 client.15975 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:30.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.839+0000 7ff7b5c48740 1 -- 172.21.15.44:0/3997924344 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a2771c14e0 con 0x55a277142940 2024-04-21T23:10:30.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.839+0000 7ff78bfff640 1 -- 172.21.15.44:0/3997924344 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a2770addc0 con 0x55a277142940 2024-04-21T23:10:30.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.839+0000 7ff78bfff640 1 -- 172.21.15.44:0/3997924344 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7ff7a0003790 con 0x55a277142940 2024-04-21T23:10:30.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.839+0000 7ff7b5c48740 10 client.15975 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:30.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.839+0000 7ff7b5c48740 10 client.15975 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:30.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.839+0000 7ff7b5c48740 20 client.15975 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:30.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.839+0000 7ff7b5c48740 1 --2- 172.21.15.44:0/3997924344 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a27715b3e0 0x55a2771c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:30.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.839+0000 7ff7b5c48740 4 client.15975 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:10:30.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.839+0000 7ff7b5c48740 1 -- 172.21.15.44:0/3997924344 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a27702b6e0 con 0x55a27715b3e0 2024-04-21T23:10:30.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.840+0000 7ff7b0e02640 1 --2- 172.21.15.44:0/3997924344 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a27715b3e0 0x55a2771c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:30.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.840+0000 7ff78bfff640 10 client.15975 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:30.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.840+0000 7ff7b0e02640 1 --2- 172.21.15.44:0/3997924344 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a27715b3e0 0x55a2771c3cb0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac0e4b4d0cac65b server_cookie=5ea343b5e6d91c48 in_seq=0 out_seq=0 2024-04-21T23:10:30.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.842+0000 7ff78bfff640 1 -- 172.21.15.44:0/3997924344 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a27702b6e0 con 0x55a27715b3e0 2024-04-21T23:10:30.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.842+0000 7ff78bfff640 10 client.15975 handle_command_reply: tid=0 2024-04-21T23:10:30.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.842+0000 7ff7b5c48740 1 client.15975 shutdown 2024-04-21T23:10:30.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff7b5c48740 1 -- 172.21.15.44:0/3997924344 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142940 msgr2=0x55a27714ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff7b5c48740 1 --2- 172.21.15.44:0/3997924344 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142940 0x55a27714ec40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x55a277141af0 tx=0x7ff7a0014ea0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff7b5c48740 1 -- 172.21.15.44:0/3997924344 shutdown_connections 2024-04-21T23:10:30.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff7b5c48740 1 --2- 172.21.15.44:0/3997924344 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a27715b3e0 0x55a2771c3cb0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff7b5c48740 1 --2- 172.21.15.44:0/3997924344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a27714b520 0x55a27714f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff7b5c48740 1 --2- 172.21.15.44:0/3997924344 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a277142940 0x55a27714ec40 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff78bfff640 0 client.15975 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:30.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff7b5c48740 1 --2- 172.21.15.44:0/3997924344 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a277142020 0x55a2771538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff7b5c48740 1 -- 172.21.15.44:0/3997924344 >> 172.21.15.44:0/3997924344 conn(0x55a27713c750 msgr2=0x55a27713db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:30.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff7b5c48740 1 -- 172.21.15.44:0/3997924344 shutdown_connections 2024-04-21T23:10:30.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff7b5c48740 1 -- 172.21.15.44:0/3997924344 wait complete. 2024-04-21T23:10:30.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff7b5c48740 20 client.15975 trim_cache size 0 max 16384 2024-04-21T23:10:30.848 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-21T23:10:30.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff7b3066640 1 -- 172.21.15.44:0/2236989091 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff7940a2d00 msgr2=0x7ff7940a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff7b3066640 1 --2- 172.21.15.44:0/2236989091 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff7940a2d00 0x7ff7940a51c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55a27713d8e0 tx=0x7ff7a402e040 comp rx=0 tx=0).stop 2024-04-21T23:10:30.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff7b3066640 1 -- 172.21.15.44:0/2236989091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7ac100070 msgr2=0x7ff7ac198e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:30.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff7b3066640 1 --2- 172.21.15.44:0/2236989091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7ac100070 0x7ff7ac198e50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff79c00e9f0 tx=0x7ff79c00eeb0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff7b3066640 1 -- 172.21.15.44:0/2236989091 shutdown_connections 2024-04-21T23:10:30.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff7b3066640 1 --2- 172.21.15.44:0/2236989091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7ac100a10 0x7ff7ac199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff7b3066640 1 --2- 172.21.15.44:0/2236989091 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff7940a2d00 0x7ff7940a51c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff7b3066640 1 --2- 172.21.15.44:0/2236989091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff7ac101d00 0x7ff7ac1a3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.843+0000 7ff7b3066640 1 --2- 172.21.15.44:0/2236989091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff7ac100070 0x7ff7ac198e50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:30.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.844+0000 7ff7b3066640 1 -- 172.21.15.44:0/2236989091 >> 172.21.15.44:0/2236989091 conn(0x7ff7ac0758c0 msgr2=0x7ff7ac0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:30.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.844+0000 7ff7b3066640 1 -- 172.21.15.44:0/2236989091 shutdown_connections 2024-04-21T23:10:30.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:30.844+0000 7ff7b3066640 1 -- 172.21.15.44:0/2236989091 wait complete. 2024-04-21T23:10:31.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:31 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3872135227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:10:31.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:31 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3872135227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:10:31.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:31 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3872135227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:10:31.691 INFO:tasks.workunit.client.0.smithi044.stdout: 1 60908 16.48 MB/sec warmup 21 sec latency 6.897 ms 2024-04-21T23:10:31.855 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:10:31.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:10:31.855 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:10:31.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.969+0000 7f328720e640 1 -- 172.21.15.44:0/1408594532 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3280100e50 msgr2=0x7f328010dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:31.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.969+0000 7f328720e640 1 --2- 172.21.15.44:0/1408594532 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3280100e50 0x7f328010dba0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f326c009900 tx=0x7f326c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:10:31.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.969+0000 7f328720e640 1 -- 172.21.15.44:0/1408594532 shutdown_connections 2024-04-21T23:10:31.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.969+0000 7f328720e640 1 --2- 172.21.15.44:0/1408594532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f328010e0e0 0x7f32801104d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:31.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.969+0000 7f328720e640 1 --2- 172.21.15.44:0/1408594532 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3280100e50 0x7f328010dba0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:31.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.969+0000 7f328720e640 1 --2- 172.21.15.44:0/1408594532 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3280100530 0x7f3280100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:31.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.969+0000 7f328720e640 1 -- 172.21.15.44:0/1408594532 >> 172.21.15.44:0/1408594532 conn(0x7f32800777f0 msgr2=0x7f3280077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:31.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.969+0000 7f328720e640 1 -- 172.21.15.44:0/1408594532 shutdown_connections 2024-04-21T23:10:31.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.969+0000 7f328720e640 1 -- 172.21.15.44:0/1408594532 wait complete. 2024-04-21T23:10:31.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.970+0000 7f328720e640 1 Processor -- start 2024-04-21T23:10:31.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.970+0000 7f328720e640 1 -- start start 2024-04-21T23:10:31.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.970+0000 7f328720e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3280100530 0x7f328019a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:31.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.970+0000 7f328720e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3280100e50 0x7f328019a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:31.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.970+0000 7f328720e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f328010e0e0 0x7f328019eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:31.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.970+0000 7f328720e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3280113d10 con 0x7f328010e0e0 2024-04-21T23:10:31.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.970+0000 7f328720e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3280113b90 con 0x7f3280100530 2024-04-21T23:10:31.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.970+0000 7f328720e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3280113e90 con 0x7f3280100e50 2024-04-21T23:10:31.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.970+0000 7f32857ab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f328010e0e0 0x7f328019eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:31.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.970+0000 7f3284faa640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3280100530 0x7f328019a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:31.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.970+0000 7f32857ab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f328010e0e0 0x7f328019eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50128/0 (socket says 172.21.15.44:50128) 2024-04-21T23:10:31.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.970+0000 7f32857ab640 1 -- 172.21.15.44:0/1991159889 learned_addr learned my addr 172.21.15.44:0/1991159889 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:31.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.970+0000 7f327ffff640 1 --2- 172.21.15.44:0/1991159889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3280100e50 0x7f328019a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:31.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.971+0000 7f32857ab640 1 -- 172.21.15.44:0/1991159889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3280100e50 msgr2=0x7f328019a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:31.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.971+0000 7f32857ab640 1 --2- 172.21.15.44:0/1991159889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3280100e50 0x7f328019a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:31.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.971+0000 7f32857ab640 1 -- 172.21.15.44:0/1991159889 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3280100530 msgr2=0x7f328019a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:31.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.971+0000 7f32857ab640 1 --2- 172.21.15.44:0/1991159889 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3280100530 0x7f328019a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:31.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.971+0000 7f32857ab640 1 -- 172.21.15.44:0/1991159889 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f328019f240 con 0x7f328010e0e0 2024-04-21T23:10:31.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.971+0000 7f3284faa640 1 --2- 172.21.15.44:0/1991159889 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3280100530 0x7f328019a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:31.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.971+0000 7f32857ab640 1 --2- 172.21.15.44:0/1991159889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f328010e0e0 0x7f328019eb10 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f327800eab0 tx=0x7f3278009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:31.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.971+0000 7f327dffb640 1 -- 172.21.15.44:0/1991159889 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327800ce40 con 0x7f328010e0e0 2024-04-21T23:10:31.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.971+0000 7f328720e640 1 -- 172.21.15.44:0/1991159889 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f328019f530 con 0x7f328010e0e0 2024-04-21T23:10:31.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.971+0000 7f327dffb640 1 -- 172.21.15.44:0/1991159889 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3278005020 con 0x7f328010e0e0 2024-04-21T23:10:31.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.971+0000 7f327dffb640 1 -- 172.21.15.44:0/1991159889 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3278010760 con 0x7f328010e0e0 2024-04-21T23:10:31.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.971+0000 7f328720e640 1 -- 172.21.15.44:0/1991159889 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32801a6dc0 con 0x7f328010e0e0 2024-04-21T23:10:31.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.974+0000 7f327dffb640 1 -- 172.21.15.44:0/1991159889 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3278004a40 con 0x7f328010e0e0 2024-04-21T23:10:31.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.974+0000 7f3289df9740 1 Processor -- start 2024-04-21T23:10:31.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.974+0000 7f3289df9740 1 -- start start 2024-04-21T23:10:31.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.974+0000 7f3289df9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501d720 0x55722501c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:31.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.974+0000 7f3289df9740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55722501ca40 0x55722501cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:31.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.974+0000 7f3289df9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557225025db0 0x5572250281a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:31.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.974+0000 7f3289df9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557224f80170 con 0x557225025db0 2024-04-21T23:10:31.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.974+0000 7f3289df9740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x557224f287c0 con 0x55722501ca40 2024-04-21T23:10:31.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.974+0000 7f3289df9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557224f7c130 con 0x55722501d720 2024-04-21T23:10:31.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.974+0000 7f327dffb640 1 --2- 172.21.15.44:0/1991159889 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f32700a2d00 0x7f32700a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:31.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.974+0000 7f327ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501d720 0x55722501c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:31.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.974+0000 7f3284faa640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55722501ca40 0x55722501cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:31.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.974+0000 7f327ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501d720 0x55722501c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:42350/0 (socket says 172.21.15.44:42350) 2024-04-21T23:10:31.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.974+0000 7f327ffff640 1 -- 172.21.15.44:0/1185053353 learned_addr learned my addr 172.21.15.44:0/1185053353 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:31.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.974+0000 7f3284faa640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55722501ca40 0x55722501cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:57444/0 (socket says 172.21.15.44:57444) 2024-04-21T23:10:31.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.974+0000 7f327dffb640 1 -- 172.21.15.44:0/1991159889 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f32780cfdb0 con 0x7f328010e0e0 2024-04-21T23:10:31.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.974+0000 7f32857ab640 1 --2- 172.21.15.44:0/1991159889 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f32700a2d00 0x7f32700a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:31.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.975+0000 7f327ffff640 1 --2- 172.21.15.44:0/1185053353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557225025db0 0x5572250281a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:31.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.975+0000 7f3284faa640 1 -- 172.21.15.44:0/1185053353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501d720 msgr2=0x55722501c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:31.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.975+0000 7f3284faa640 1 --2- 172.21.15.44:0/1185053353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501d720 0x55722501c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:31.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.975+0000 7f3284faa640 1 -- 172.21.15.44:0/1185053353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557225025db0 msgr2=0x5572250281a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:31.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.975+0000 7f3284faa640 1 --2- 172.21.15.44:0/1185053353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557225025db0 0x5572250281a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:31.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.975+0000 7f3284faa640 1 -- 172.21.15.44:0/1185053353 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557224f39e60 con 0x55722501ca40 2024-04-21T23:10:31.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.975+0000 7f32857ab640 1 --2- 172.21.15.44:0/1991159889 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f32700a2d00 0x7f32700a51c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f327800ea80 tx=0x7f3278005b40 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:31.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.975+0000 7f3284faa640 1 --2- 172.21.15.44:0/1185053353 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55722501ca40 0x55722501cec0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3274009fd0 tx=0x7f32740118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=352a6b87934a2580 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:31.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.975+0000 7f326affd640 1 -- 172.21.15.44:0/1185053353 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3274004030 con 0x55722501ca40 2024-04-21T23:10:31.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.975+0000 7f326affd640 1 -- 172.21.15.44:0/1185053353 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3274005500 con 0x55722501ca40 2024-04-21T23:10:31.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.975+0000 7f326affd640 1 -- 172.21.15.44:0/1185053353 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3274004d80 con 0x55722501ca40 2024-04-21T23:10:31.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.976+0000 7f3289df9740 1 -- 172.21.15.44:0/1185053353 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55722501ca40 msgr2=0x55722501cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:31.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.976+0000 7f3289df9740 1 --2- 172.21.15.44:0/1185053353 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55722501ca40 0x55722501cec0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3274009fd0 tx=0x7f32740118d0 comp rx=0 tx=0).stop 2024-04-21T23:10:31.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.976+0000 7f3289df9740 1 -- 172.21.15.44:0/1185053353 shutdown_connections 2024-04-21T23:10:31.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.976+0000 7f3289df9740 1 --2- 172.21.15.44:0/1185053353 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557225025db0 0x5572250281a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:31.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.976+0000 7f3289df9740 1 --2- 172.21.15.44:0/1185053353 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55722501ca40 0x55722501cec0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:31.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.976+0000 7f3289df9740 1 --2- 172.21.15.44:0/1185053353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501d720 0x55722501c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:31.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.976+0000 7f3289df9740 1 -- 172.21.15.44:0/1185053353 >> 172.21.15.44:0/1185053353 conn(0x557224fe34f0 msgr2=0x557225017490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:31.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.976+0000 7f3289df9740 1 -- 172.21.15.44:0/1185053353 shutdown_connections 2024-04-21T23:10:31.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.976+0000 7f3289df9740 1 -- 172.21.15.44:0/1185053353 wait complete. 2024-04-21T23:10:31.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.976+0000 7f3289df9740 1 -- start start 2024-04-21T23:10:31.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.976+0000 7f3289df9740 1 Processor -- start 2024-04-21T23:10:31.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.977+0000 7f3289df9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55722501ca40 0x557225022d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:31.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.977+0000 7f3289df9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501d720 0x557225023280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:31.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.977+0000 7f3289df9740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557225025db0 0x5572250237c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:31.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.977+0000 7f3289df9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557225028c20 con 0x55722501ca40 2024-04-21T23:10:31.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.977+0000 7f3289df9740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x557224f3bd40 con 0x557225025db0 2024-04-21T23:10:31.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.977+0000 7f3289df9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557225028da0 con 0x55722501d720 2024-04-21T23:10:31.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.977+0000 7f3289df9740 10 client.0 _ll_register_callbacks cb 0x557224f77820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:31.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.977+0000 7f327ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55722501ca40 0x557225022d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:31.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.977+0000 7f3284faa640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501d720 0x557225023280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:31.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.977+0000 7f327ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55722501ca40 0x557225022d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50146/0 (socket says 172.21.15.44:50146) 2024-04-21T23:10:31.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.977+0000 7f327ffff640 1 -- 172.21.15.44:0/168013780 learned_addr learned my addr 172.21.15.44:0/168013780 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:31.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.977+0000 7f327ffff640 1 --2- 172.21.15.44:0/168013780 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557225025db0 0x5572250237c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:31.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.977+0000 7f327ffff640 1 -- 172.21.15.44:0/168013780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501d720 msgr2=0x557225023280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:31.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.977+0000 7f327ffff640 1 --2- 172.21.15.44:0/168013780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501d720 0x557225023280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:31.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.977+0000 7f327ffff640 1 -- 172.21.15.44:0/168013780 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557225025db0 msgr2=0x5572250237c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:31.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.977+0000 7f327ffff640 1 --2- 172.21.15.44:0/168013780 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557225025db0 0x5572250237c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:31.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.977+0000 7f327ffff640 1 -- 172.21.15.44:0/168013780 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557224d5c7e0 con 0x55722501ca40 2024-04-21T23:10:31.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.977+0000 7f3284faa640 1 --2- 172.21.15.44:0/168013780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501d720 0x557225023280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:31.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.978+0000 7f327ffff640 1 --2- 172.21.15.44:0/168013780 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55722501ca40 0x557225022d40 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f326c02c820 tx=0x7f326c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fc324798a135b80 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:31.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.978+0000 7f325ffff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:10:31.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.978+0000 7f325ffff640 1 -- 172.21.15.44:0/168013780 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c004ad0 con 0x55722501ca40 2024-04-21T23:10:31.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.978+0000 7f325ffff640 1 -- 172.21.15.44:0/168013780 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f326c004c70 con 0x55722501ca40 2024-04-21T23:10:31.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.978+0000 7f3289df9740 1 -- 172.21.15.44:0/168013780 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3274005500 con 0x55722501ca40 2024-04-21T23:10:31.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.978+0000 7f325ffff640 1 -- 172.21.15.44:0/168013780 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f326c053050 con 0x55722501ca40 2024-04-21T23:10:31.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.978+0000 7f3289df9740 10 client.15987 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:31.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.978+0000 7f3289df9740 1 -- 172.21.15.44:0/168013780 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557224dbcf70 con 0x55722501ca40 2024-04-21T23:10:31.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.978+0000 7f325ffff640 1 -- 172.21.15.44:0/168013780 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3274005500 con 0x55722501ca40 2024-04-21T23:10:31.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.978+0000 7f325ffff640 1 -- 172.21.15.44:0/168013780 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f326c003850 con 0x55722501ca40 2024-04-21T23:10:31.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.978+0000 7f3289df9740 10 client.15987 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:31.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.978+0000 7f3289df9740 10 client.15987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:31.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.978+0000 7f3289df9740 20 client.15987 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:31.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.978+0000 7f3289df9740 1 --2- 172.21.15.44:0/168013780 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557225035ea0 0x557225038290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:31.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.979+0000 7f3289df9740 4 client.15987 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:10:31.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.979+0000 7f3289df9740 1 -- 172.21.15.44:0/168013780 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557224f54450 con 0x557225035ea0 2024-04-21T23:10:31.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.979+0000 7f3284faa640 1 --2- 172.21.15.44:0/168013780 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557225035ea0 0x557225038290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:31.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.979+0000 7f3284faa640 1 --2- 172.21.15.44:0/168013780 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557225035ea0 0x557225038290 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=278d60054700fc50 server_cookie=ccf46990d0af19a3 in_seq=0 out_seq=0 2024-04-21T23:10:31.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.979+0000 7f325ffff640 10 client.15987 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:31.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.980+0000 7f325ffff640 1 -- 172.21.15.44:0/168013780 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557224f54450 con 0x557225035ea0 2024-04-21T23:10:31.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.980+0000 7f325ffff640 10 client.15987 handle_command_reply: tid=0 2024-04-21T23:10:31.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.980+0000 7f3289df9740 1 client.15987 shutdown 2024-04-21T23:10:31.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.981+0000 7f3289df9740 1 -- 172.21.15.44:0/168013780 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55722501ca40 msgr2=0x557225022d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:31.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.981+0000 7f3289df9740 1 --2- 172.21.15.44:0/168013780 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55722501ca40 0x557225022d40 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f326c02c820 tx=0x7f326c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:10:31.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.981+0000 7f3289df9740 1 -- 172.21.15.44:0/168013780 shutdown_connections 2024-04-21T23:10:31.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.981+0000 7f3289df9740 1 --2- 172.21.15.44:0/168013780 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557225035ea0 0x557225038290 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:31.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.981+0000 7f3289df9740 1 --2- 172.21.15.44:0/168013780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501d720 0x557225023280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:31.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.981+0000 7f3289df9740 1 --2- 172.21.15.44:0/168013780 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557225025db0 0x5572250237c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:31.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.981+0000 7f325ffff640 0 client.15987 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:31.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.981+0000 7f3289df9740 1 --2- 172.21.15.44:0/168013780 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55722501ca40 0x557225022d40 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:31.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.981+0000 7f3289df9740 1 -- 172.21.15.44:0/168013780 >> 172.21.15.44:0/168013780 conn(0x557224fe34f0 msgr2=0x5572250191e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:31.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.981+0000 7f3289df9740 1 -- 172.21.15.44:0/168013780 shutdown_connections 2024-04-21T23:10:31.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.981+0000 7f3289df9740 1 -- 172.21.15.44:0/168013780 wait complete. 2024-04-21T23:10:31.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.981+0000 7f3289df9740 20 client.15987 trim_cache size 0 max 16384 2024-04-21T23:10:32.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.999+0000 7f3289df9740 1 Processor -- start 2024-04-21T23:10:32.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.999+0000 7f3289df9740 1 -- start start 2024-04-21T23:10:32.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.999+0000 7f3289df9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501ca40 0x557225031b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.999+0000 7f3289df9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55722501d720 0x557225022620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:31.999+0000 7f3289df9740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557225025db0 0x557225018ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.000+0000 7f3289df9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f32801aeed0 con 0x55722501d720 2024-04-21T23:10:32.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.000+0000 7f3289df9740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f32801b1750 con 0x557225025db0 2024-04-21T23:10:32.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.000+0000 7f3289df9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f32801af540 con 0x55722501ca40 2024-04-21T23:10:32.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.000+0000 7f3284faa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55722501d720 0x557225022620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.000+0000 7f327ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501ca40 0x557225031b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.000+0000 7f3284faa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55722501d720 0x557225022620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50156/0 (socket says 172.21.15.44:50156) 2024-04-21T23:10:32.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.000+0000 7f3284faa640 1 -- 172.21.15.44:0/1319628938 learned_addr learned my addr 172.21.15.44:0/1319628938 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:32.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.000+0000 7f327ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501ca40 0x557225031b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:42376/0 (socket says 172.21.15.44:42376) 2024-04-21T23:10:32.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.000+0000 7f327ffff640 1 --2- 172.21.15.44:0/1319628938 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557225025db0 0x557225018ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.000+0000 7f327ffff640 1 -- 172.21.15.44:0/1319628938 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557225025db0 msgr2=0x557225018ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:32.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.000+0000 7f327ffff640 1 --2- 172.21.15.44:0/1319628938 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557225025db0 0x557225018ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.000+0000 7f327ffff640 1 -- 172.21.15.44:0/1319628938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55722501d720 msgr2=0x557225022620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:32.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.000+0000 7f327ffff640 1 --2- 172.21.15.44:0/1319628938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55722501d720 0x557225022620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.000+0000 7f327ffff640 1 -- 172.21.15.44:0/1319628938 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557224c54170 con 0x55722501ca40 2024-04-21T23:10:32.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.000+0000 7f3284faa640 1 --2- 172.21.15.44:0/1319628938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55722501d720 0x557225022620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:32.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.000+0000 7f327ffff640 1 --2- 172.21.15.44:0/1319628938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501ca40 0x557225031b80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f326c0096c0 tx=0x7f326c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1abbb419c76bba97 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:32.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.000+0000 7f326a7fc640 1 -- 172.21.15.44:0/1319628938 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c034440 con 0x55722501ca40 2024-04-21T23:10:32.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.000+0000 7f326a7fc640 1 -- 172.21.15.44:0/1319628938 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f326c0048d0 con 0x55722501ca40 2024-04-21T23:10:32.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.000+0000 7f326a7fc640 1 -- 172.21.15.44:0/1319628938 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c034960 con 0x55722501ca40 2024-04-21T23:10:32.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.001+0000 7f3289df9740 1 -- 172.21.15.44:0/1319628938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501ca40 msgr2=0x557225031b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.001+0000 7f3289df9740 1 --2- 172.21.15.44:0/1319628938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501ca40 0x557225031b80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f326c0096c0 tx=0x7f326c005fd0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.001+0000 7f3289df9740 1 -- 172.21.15.44:0/1319628938 shutdown_connections 2024-04-21T23:10:32.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.001+0000 7f3289df9740 1 --2- 172.21.15.44:0/1319628938 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557225025db0 0x557225018ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.001+0000 7f3289df9740 1 --2- 172.21.15.44:0/1319628938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55722501d720 0x557225022620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.001+0000 7f3289df9740 1 --2- 172.21.15.44:0/1319628938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501ca40 0x557225031b80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.001+0000 7f3289df9740 1 -- 172.21.15.44:0/1319628938 >> 172.21.15.44:0/1319628938 conn(0x557224fe34f0 msgr2=0x5572250188e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:32.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.001+0000 7f3289df9740 1 -- 172.21.15.44:0/1319628938 shutdown_connections 2024-04-21T23:10:32.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.001+0000 7f3289df9740 1 -- 172.21.15.44:0/1319628938 wait complete. 2024-04-21T23:10:32.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.001+0000 7f3289df9740 1 -- start start 2024-04-21T23:10:32.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.002+0000 7f3289df9740 1 Processor -- start 2024-04-21T23:10:32.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.002+0000 7f3289df9740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55722501ca40 0x5572250299f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.002+0000 7f3289df9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501d720 0x557225029f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.002+0000 7f3289df9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557225025db0 0x55722502a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.002+0000 7f3289df9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f32801afc10 con 0x557225025db0 2024-04-21T23:10:32.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.002+0000 7f3289df9740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f32801b1080 con 0x55722501ca40 2024-04-21T23:10:32.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.002+0000 7f3289df9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f32801b02e0 con 0x55722501d720 2024-04-21T23:10:32.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.002+0000 7f3289df9740 10 client.0 _ll_register_callbacks cb 0x7f326c02c910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.002+0000 7f327ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55722501ca40 0x5572250299f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.002+0000 7f3284faa640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501d720 0x557225029f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.002+0000 7f327ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55722501ca40 0x5572250299f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:57468/0 (socket says 172.21.15.44:57468) 2024-04-21T23:10:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.002+0000 7f3284faa640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501d720 0x557225029f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:42392/0 (socket says 172.21.15.44:42392) 2024-04-21T23:10:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.002+0000 7f327ffff640 1 -- 172.21.15.44:0/2881381029 learned_addr learned my addr 172.21.15.44:0/2881381029 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.002+0000 7f327ffff640 1 --2- 172.21.15.44:0/2881381029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557225025db0 0x55722502a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.002+0000 7f3284faa640 1 -- 172.21.15.44:0/2881381029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55722501ca40 msgr2=0x5572250299f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.002+0000 7f3284faa640 1 --2- 172.21.15.44:0/2881381029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55722501ca40 0x5572250299f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.002+0000 7f3284faa640 1 -- 172.21.15.44:0/2881381029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557225025db0 msgr2=0x55722502a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.002+0000 7f3284faa640 1 --2- 172.21.15.44:0/2881381029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557225025db0 0x55722502a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.002+0000 7f3284faa640 1 -- 172.21.15.44:0/2881381029 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55722509b360 con 0x55722501d720 2024-04-21T23:10:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.003+0000 7f3284faa640 1 --2- 172.21.15.44:0/2881381029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501d720 0x557225029f30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3274009380 tx=0x7f32740096e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62a3adc2c53c6f25 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.003+0000 7f325ffff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:10:32.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.003+0000 7f325ffff640 1 -- 172.21.15.44:0/2881381029 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32740035f0 con 0x55722501d720 2024-04-21T23:10:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.003+0000 7f3289df9740 1 -- 172.21.15.44:0/2881381029 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f326c0048d0 con 0x55722501d720 2024-04-21T23:10:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.003+0000 7f325ffff640 1 -- 172.21.15.44:0/2881381029 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f3274016040 con 0x55722501d720 2024-04-21T23:10:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.003+0000 7f325ffff640 1 -- 172.21.15.44:0/2881381029 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f327403d050 con 0x55722501d720 2024-04-21T23:10:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.003+0000 7f3289df9740 10 client.25615 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.003+0000 7f3289df9740 1 -- 172.21.15.44:0/2881381029 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55722509c5e0 con 0x55722501d720 2024-04-21T23:10:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.003+0000 7f325ffff640 1 -- 172.21.15.44:0/2881381029 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f326c0048d0 con 0x55722501d720 2024-04-21T23:10:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.003+0000 7f325ffff640 1 -- 172.21.15.44:0/2881381029 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f3274019820 con 0x55722501d720 2024-04-21T23:10:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.003+0000 7f3289df9740 10 client.25615 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.003+0000 7f3289df9740 10 client.25615 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.003+0000 7f3289df9740 20 client.25615 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.003+0000 7f3289df9740 1 --2- 172.21.15.44:0/2881381029 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55722509cb70 0x55722509ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.003+0000 7f3289df9740 4 client.25615 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:10:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.003+0000 7f3289df9740 1 -- 172.21.15.44:0/2881381029 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557224f8d6b0 con 0x55722509cb70 2024-04-21T23:10:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.004+0000 7f327ffff640 1 --2- 172.21.15.44:0/2881381029 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55722509cb70 0x55722509ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.004+0000 7f325ffff640 10 client.25615 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:32.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.004+0000 7f327ffff640 1 --2- 172.21.15.44:0/2881381029 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55722509cb70 0x55722509ef90 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=296a3eced6da80f8 server_cookie=4735b1880f5f38f4 in_seq=0 out_seq=0 2024-04-21T23:10:32.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.007+0000 7f325ffff640 1 -- 172.21.15.44:0/2881381029 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557224f8d6b0 con 0x55722509cb70 2024-04-21T23:10:32.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.007+0000 7f325ffff640 10 client.25615 handle_command_reply: tid=0 2024-04-21T23:10:32.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.007+0000 7f3289df9740 1 client.25615 shutdown 2024-04-21T23:10:32.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.007+0000 7f3289df9740 1 -- 172.21.15.44:0/2881381029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501d720 msgr2=0x557225029f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.007+0000 7f3289df9740 1 --2- 172.21.15.44:0/2881381029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501d720 0x557225029f30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3274009380 tx=0x7f32740096e0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.007+0000 7f3289df9740 1 -- 172.21.15.44:0/2881381029 shutdown_connections 2024-04-21T23:10:32.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.007+0000 7f3289df9740 1 --2- 172.21.15.44:0/2881381029 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55722509cb70 0x55722509ef90 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.007+0000 7f3289df9740 1 --2- 172.21.15.44:0/2881381029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55722501d720 0x557225029f30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.007+0000 7f3289df9740 1 --2- 172.21.15.44:0/2881381029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557225025db0 0x55722502a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.007+0000 7f325ffff640 0 client.25615 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:32.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.007+0000 7f3289df9740 1 --2- 172.21.15.44:0/2881381029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55722501ca40 0x5572250299f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.007+0000 7f3289df9740 1 -- 172.21.15.44:0/2881381029 >> 172.21.15.44:0/2881381029 conn(0x557224fe34f0 msgr2=0x557225023590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:32.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.007+0000 7f3289df9740 1 -- 172.21.15.44:0/2881381029 shutdown_connections 2024-04-21T23:10:32.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.007+0000 7f3289df9740 1 -- 172.21.15.44:0/2881381029 wait complete. 2024-04-21T23:10:32.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.007+0000 7f3289df9740 20 client.25615 trim_cache size 0 max 16384 2024-04-21T23:10:32.013 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:10:32.013 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-21T23:10:32.013 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "12e1aa69-6141-4eb5-b0e8-5f554ac257f2", 2024-04-21T23:10:32.013 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-21T23:10:32.013 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:10:32.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.008+0000 7f328720e640 1 -- 172.21.15.44:0/1991159889 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f32700a2d00 msgr2=0x7f32700a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.008+0000 7f328720e640 1 --2- 172.21.15.44:0/1991159889 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f32700a2d00 0x7f32700a51c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f327800ea80 tx=0x7f3278005b40 comp rx=0 tx=0).stop 2024-04-21T23:10:32.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.008+0000 7f328720e640 1 -- 172.21.15.44:0/1991159889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f328010e0e0 msgr2=0x7f328019eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.008+0000 7f328720e640 1 --2- 172.21.15.44:0/1991159889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f328010e0e0 0x7f328019eb10 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f327800eab0 tx=0x7f3278009f80 comp rx=0 tx=0).stop 2024-04-21T23:10:32.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.008+0000 7f328720e640 1 -- 172.21.15.44:0/1991159889 shutdown_connections 2024-04-21T23:10:32.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.008+0000 7f328720e640 1 --2- 172.21.15.44:0/1991159889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3280100e50 0x7f328019a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.008+0000 7f328720e640 1 --2- 172.21.15.44:0/1991159889 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f32700a2d00 0x7f32700a51c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.008+0000 7f328720e640 1 --2- 172.21.15.44:0/1991159889 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f328010e0e0 0x7f328019eb10 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.008+0000 7f328720e640 1 --2- 172.21.15.44:0/1991159889 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3280100530 0x7f328019a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.008+0000 7f328720e640 1 -- 172.21.15.44:0/1991159889 >> 172.21.15.44:0/1991159889 conn(0x7f32800777f0 msgr2=0x7f328010f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:32.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.008+0000 7f328720e640 1 -- 172.21.15.44:0/1991159889 shutdown_connections 2024-04-21T23:10:32.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.008+0000 7f328720e640 1 -- 172.21.15.44:0/1991159889 wait complete. 2024-04-21T23:10:32.018 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:10:32.019 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:10:32.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.131+0000 7fd84800e640 1 -- 172.21.15.44:0/1114469661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd840100e50 msgr2=0x7fd84010dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.131+0000 7fd84800e640 1 --2- 172.21.15.44:0/1114469661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd840100e50 0x7fd84010dba0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd830009900 tx=0x7fd83002bc30 comp rx=0 tx=0).stop 2024-04-21T23:10:32.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.131+0000 7fd84800e640 1 -- 172.21.15.44:0/1114469661 shutdown_connections 2024-04-21T23:10:32.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.131+0000 7fd84800e640 1 --2- 172.21.15.44:0/1114469661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd84010e0e0 0x7fd8401104d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.131+0000 7fd84800e640 1 --2- 172.21.15.44:0/1114469661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd840100e50 0x7fd84010dba0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.131+0000 7fd84800e640 1 --2- 172.21.15.44:0/1114469661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd840100530 0x7fd840100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.131+0000 7fd84800e640 1 -- 172.21.15.44:0/1114469661 >> 172.21.15.44:0/1114469661 conn(0x7fd8400777f0 msgr2=0x7fd840077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:32.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.131+0000 7fd84800e640 1 -- 172.21.15.44:0/1114469661 shutdown_connections 2024-04-21T23:10:32.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.131+0000 7fd84800e640 1 -- 172.21.15.44:0/1114469661 wait complete. 2024-04-21T23:10:32.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.132+0000 7fd84800e640 1 Processor -- start 2024-04-21T23:10:32.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.132+0000 7fd84800e640 1 -- start start 2024-04-21T23:10:32.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.132+0000 7fd84800e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd840100530 0x7fd84019a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.132+0000 7fd84800e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd840100e50 0x7fd84019a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.132+0000 7fd84800e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd84010e0e0 0x7fd84019eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.132+0000 7fd84800e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd840113d30 con 0x7fd840100530 2024-04-21T23:10:32.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.132+0000 7fd84800e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd840113bb0 con 0x7fd840100e50 2024-04-21T23:10:32.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.132+0000 7fd84800e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd840113eb0 con 0x7fd84010e0e0 2024-04-21T23:10:32.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.132+0000 7fd845daa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd840100530 0x7fd84019a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.132+0000 7fd845daa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd840100530 0x7fd84019a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50174/0 (socket says 172.21.15.44:50174) 2024-04-21T23:10:32.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.132+0000 7fd845daa640 1 -- 172.21.15.44:0/3266084553 learned_addr learned my addr 172.21.15.44:0/3266084553 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:32.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.132+0000 7fd8455a9640 1 --2- 172.21.15.44:0/3266084553 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd840100e50 0x7fd84019a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.132+0000 7fd8465ab640 1 --2- 172.21.15.44:0/3266084553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd84010e0e0 0x7fd84019eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.132+0000 7fd845daa640 1 -- 172.21.15.44:0/3266084553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd84010e0e0 msgr2=0x7fd84019eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.132+0000 7fd845daa640 1 --2- 172.21.15.44:0/3266084553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd84010e0e0 0x7fd84019eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.132+0000 7fd845daa640 1 -- 172.21.15.44:0/3266084553 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd840100e50 msgr2=0x7fd84019a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.132+0000 7fd845daa640 1 --2- 172.21.15.44:0/3266084553 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd840100e50 0x7fd84019a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.132+0000 7fd845daa640 1 -- 172.21.15.44:0/3266084553 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd84019f1f0 con 0x7fd840100530 2024-04-21T23:10:32.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.132+0000 7fd8455a9640 1 --2- 172.21.15.44:0/3266084553 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd840100e50 0x7fd84019a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:32.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.133+0000 7fd845daa640 1 --2- 172.21.15.44:0/3266084553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd840100530 0x7fd84019a250 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fd83c00d950 tx=0x7fd83c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:32.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.133+0000 7fd836ffd640 1 -- 172.21.15.44:0/3266084553 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c017030 con 0x7fd840100530 2024-04-21T23:10:32.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.133+0000 7fd836ffd640 1 -- 172.21.15.44:0/3266084553 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd83c004020 con 0x7fd840100530 2024-04-21T23:10:32.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.133+0000 7fd84800e640 1 -- 172.21.15.44:0/3266084553 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd84019f4e0 con 0x7fd840100530 2024-04-21T23:10:32.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.133+0000 7fd836ffd640 1 -- 172.21.15.44:0/3266084553 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0047a0 con 0x7fd840100530 2024-04-21T23:10:32.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.133+0000 7fd84800e640 1 -- 172.21.15.44:0/3266084553 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8401a6d70 con 0x7fd840100530 2024-04-21T23:10:32.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.135+0000 7fd836ffd640 1 -- 172.21.15.44:0/3266084553 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd83c0041d0 con 0x7fd840100530 2024-04-21T23:10:32.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.135+0000 7fd84abf6740 1 Processor -- start 2024-04-21T23:10:32.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.136+0000 7fd84abf6740 1 -- start start 2024-04-21T23:10:32.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.136+0000 7fd84abf6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe90f30 0x557bafe91310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.136+0000 7fd84abf6740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe91850 0x557bafe99ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.136+0000 7fd84abf6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe9a430 0x557bafe9c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.136+0000 7fd84abf6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557bafdf5560 con 0x557bafe90f30 2024-04-21T23:10:32.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.136+0000 7fd84abf6740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x557bafd9d730 con 0x557bafe91850 2024-04-21T23:10:32.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.136+0000 7fd84abf6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557bafdf1520 con 0x557bafe9a430 2024-04-21T23:10:32.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.136+0000 7fd8465ab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe90f30 0x557bafe91310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.136+0000 7fd8465ab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe90f30 0x557bafe91310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50178/0 (socket says 172.21.15.44:50178) 2024-04-21T23:10:32.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.136+0000 7fd8465ab640 1 -- 172.21.15.44:0/2129803379 learned_addr learned my addr 172.21.15.44:0/2129803379 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:32.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.136+0000 7fd845daa640 1 --2- 172.21.15.44:0/2129803379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe91850 0x557bafe99ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.136+0000 7fd8455a9640 1 --2- 172.21.15.44:0/2129803379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe9a430 0x557bafe9c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.136+0000 7fd836ffd640 1 --2- 172.21.15.44:0/3266084553 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd82c0a2d00 0x7fd82c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.136+0000 7fd8465ab640 1 -- 172.21.15.44:0/2129803379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe9a430 msgr2=0x557bafe9c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:32.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.136+0000 7fd8465ab640 1 --2- 172.21.15.44:0/2129803379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe9a430 0x557bafe9c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.136+0000 7fd8465ab640 1 -- 172.21.15.44:0/2129803379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe91850 msgr2=0x557bafe99ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:32.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.136+0000 7fd8465ab640 1 --2- 172.21.15.44:0/2129803379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe91850 0x557bafe99ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.136+0000 7fd8465ab640 1 -- 172.21.15.44:0/2129803379 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bafbd37d0 con 0x557bafe90f30 2024-04-21T23:10:32.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.137+0000 7fd836ffd640 1 -- 172.21.15.44:0/3266084553 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fd83c0cf1d0 con 0x7fd840100530 2024-04-21T23:10:32.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.137+0000 7fd845daa640 1 --2- 172.21.15.44:0/2129803379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe91850 0x557bafe99ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:32.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.137+0000 7fd8465ab640 1 --2- 172.21.15.44:0/3266084553 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd82c0a2d00 0x7fd82c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.137+0000 7fd8465ab640 1 --2- 172.21.15.44:0/2129803379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe90f30 0x557bafe91310 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd838011410 tx=0x7fd8380118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81bc8690f95d6989 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:32.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.137+0000 7fd82bfff640 1 -- 172.21.15.44:0/2129803379 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83801c6d0 con 0x557bafe90f30 2024-04-21T23:10:32.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.137+0000 7fd82bfff640 1 -- 172.21.15.44:0/2129803379 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd83801cd50 con 0x557bafe90f30 2024-04-21T23:10:32.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.137+0000 7fd8465ab640 1 --2- 172.21.15.44:0/3266084553 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd82c0a2d00 0x7fd82c0a51c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd838022000 tx=0x7fd83801f000 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:32.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.137+0000 7fd82bfff640 1 -- 172.21.15.44:0/2129803379 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd838012bc0 con 0x557bafe90f30 2024-04-21T23:10:32.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.138+0000 7fd84abf6740 1 -- 172.21.15.44:0/2129803379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe90f30 msgr2=0x557bafe91310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.138+0000 7fd84abf6740 1 --2- 172.21.15.44:0/2129803379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe90f30 0x557bafe91310 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd838011410 tx=0x7fd8380118d0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.138+0000 7fd84abf6740 1 -- 172.21.15.44:0/2129803379 shutdown_connections 2024-04-21T23:10:32.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.138+0000 7fd84abf6740 1 --2- 172.21.15.44:0/2129803379 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe9a430 0x557bafe9c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.138+0000 7fd84abf6740 1 --2- 172.21.15.44:0/2129803379 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe91850 0x557bafe99ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.138+0000 7fd84abf6740 1 --2- 172.21.15.44:0/2129803379 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe90f30 0x557bafe91310 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.138+0000 7fd84abf6740 1 -- 172.21.15.44:0/2129803379 >> 172.21.15.44:0/2129803379 conn(0x557bafe8b660 msgr2=0x557bafe8ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:32.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.138+0000 7fd84abf6740 1 -- 172.21.15.44:0/2129803379 shutdown_connections 2024-04-21T23:10:32.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.138+0000 7fd84abf6740 1 -- 172.21.15.44:0/2129803379 wait complete. 2024-04-21T23:10:32.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.138+0000 7fd84abf6740 1 -- start start 2024-04-21T23:10:32.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.138+0000 7fd84abf6740 1 Processor -- start 2024-04-21T23:10:32.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd84abf6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe90f30 0x557bafe97470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd84abf6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe91850 0x557bafe979b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd84abf6740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe9a430 0x557bafe97ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd84abf6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557bafe9d220 con 0x557bafe90f30 2024-04-21T23:10:32.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd84abf6740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x557bafdb0e00 con 0x557bafe9a430 2024-04-21T23:10:32.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd84abf6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557bafe9d3a0 con 0x557bafe91850 2024-04-21T23:10:32.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd84abf6740 10 client.0 _ll_register_callbacks cb 0x557bafdebeb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:32.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd845daa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe90f30 0x557bafe97470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd845daa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe90f30 0x557bafe97470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50190/0 (socket says 172.21.15.44:50190) 2024-04-21T23:10:32.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd845daa640 1 -- 172.21.15.44:0/1972992256 learned_addr learned my addr 172.21.15.44:0/1972992256 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:32.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd8465ab640 1 --2- 172.21.15.44:0/1972992256 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe9a430 0x557bafe97ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd8455a9640 1 --2- 172.21.15.44:0/1972992256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe91850 0x557bafe979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd845daa640 1 -- 172.21.15.44:0/1972992256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe91850 msgr2=0x557bafe979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:32.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd845daa640 1 --2- 172.21.15.44:0/1972992256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe91850 0x557bafe979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd845daa640 1 -- 172.21.15.44:0/1972992256 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe9a430 msgr2=0x557bafe97ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:32.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd845daa640 1 --2- 172.21.15.44:0/1972992256 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe9a430 0x557bafe97ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd845daa640 1 -- 172.21.15.44:0/1972992256 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bafb093f0 con 0x557bafe90f30 2024-04-21T23:10:32.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd8455a9640 1 --2- 172.21.15.44:0/1972992256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe91850 0x557bafe979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:10:32.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd845daa640 1 --2- 172.21.15.44:0/1972992256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe90f30 0x557bafe97470 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0cc5e0 tx=0x7fd83c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78e125e6277bac5e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:32.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd8297fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:10:32.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.139+0000 7fd8297fa640 1 -- 172.21.15.44:0/1972992256 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0975f0 con 0x557bafe90f30 2024-04-21T23:10:32.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.140+0000 7fd8297fa640 1 -- 172.21.15.44:0/1972992256 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fd83c09a7a0 con 0x557bafe90f30 2024-04-21T23:10:32.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.140+0000 7fd84abf6740 1 -- 172.21.15.44:0/1972992256 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd83801cd50 con 0x557bafe90f30 2024-04-21T23:10:32.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.140+0000 7fd8297fa640 1 -- 172.21.15.44:0/1972992256 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd83c09acb0 con 0x557bafe90f30 2024-04-21T23:10:32.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.140+0000 7fd84abf6740 10 client.16005 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:32.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.140+0000 7fd84abf6740 1 -- 172.21.15.44:0/1972992256 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557bafac90f0 con 0x557bafe90f30 2024-04-21T23:10:32.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.140+0000 7fd8297fa640 1 -- 172.21.15.44:0/1972992256 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd83801cd50 con 0x557bafe90f30 2024-04-21T23:10:32.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.140+0000 7fd8297fa640 1 -- 172.21.15.44:0/1972992256 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fd83c0bb0b0 con 0x557bafe90f30 2024-04-21T23:10:32.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.140+0000 7fd84abf6740 10 client.16005 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:32.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.140+0000 7fd84abf6740 10 client.16005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:32.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.140+0000 7fd84abf6740 20 client.16005 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:32.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.140+0000 7fd84abf6740 1 --2- 172.21.15.44:0/1972992256 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557bafeaa7d0 0x557bafeacc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.140+0000 7fd84abf6740 4 client.16005 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:10:32.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.140+0000 7fd84abf6740 1 -- 172.21.15.44:0/1972992256 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557bafe02aa0 con 0x557bafeaa7d0 2024-04-21T23:10:32.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.140+0000 7fd8455a9640 1 --2- 172.21.15.44:0/1972992256 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557bafeaa7d0 0x557bafeacc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.141+0000 7fd8455a9640 1 --2- 172.21.15.44:0/1972992256 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557bafeaa7d0 0x557bafeacc20 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24fcc16708084603 server_cookie=f40aa0dffb6a358 in_seq=0 out_seq=0 2024-04-21T23:10:32.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.141+0000 7fd8297fa640 10 client.16005 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:32.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.142+0000 7fd8297fa640 1 -- 172.21.15.44:0/1972992256 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557bafe02aa0 con 0x557bafeaa7d0 2024-04-21T23:10:32.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.142+0000 7fd8297fa640 10 client.16005 handle_command_reply: tid=0 2024-04-21T23:10:32.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.142+0000 7fd84abf6740 1 client.16005 shutdown 2024-04-21T23:10:32.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.143+0000 7fd84abf6740 1 -- 172.21.15.44:0/1972992256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe90f30 msgr2=0x557bafe97470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.143+0000 7fd84abf6740 1 --2- 172.21.15.44:0/1972992256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe90f30 0x557bafe97470 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0cc5e0 tx=0x7fd83c000bc0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.143+0000 7fd84abf6740 1 -- 172.21.15.44:0/1972992256 shutdown_connections 2024-04-21T23:10:32.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.143+0000 7fd84abf6740 1 --2- 172.21.15.44:0/1972992256 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557bafeaa7d0 0x557bafeacc20 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.143+0000 7fd84abf6740 1 --2- 172.21.15.44:0/1972992256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe91850 0x557bafe979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.143+0000 7fd84abf6740 1 --2- 172.21.15.44:0/1972992256 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe9a430 0x557bafe97ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.143+0000 7fd8297fa640 0 client.16005 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:32.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.143+0000 7fd84abf6740 1 --2- 172.21.15.44:0/1972992256 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe90f30 0x557bafe97470 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.143+0000 7fd84abf6740 1 -- 172.21.15.44:0/1972992256 >> 172.21.15.44:0/1972992256 conn(0x557bafe8b660 msgr2=0x557bafe8d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:32.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.143+0000 7fd84abf6740 1 -- 172.21.15.44:0/1972992256 shutdown_connections 2024-04-21T23:10:32.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.143+0000 7fd84abf6740 1 -- 172.21.15.44:0/1972992256 wait complete. 2024-04-21T23:10:32.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.143+0000 7fd84abf6740 20 client.16005 trim_cache size 0 max 16384 2024-04-21T23:10:32.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.162+0000 7fd84abf6740 1 Processor -- start 2024-04-21T23:10:32.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.162+0000 7fd84abf6740 1 -- start start 2024-04-21T23:10:32.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.162+0000 7fd84abf6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe90f30 0x557bafe99ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.162+0000 7fd84abf6740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe91850 0x557bafea60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.162+0000 7fd84abf6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe9a430 0x557bafe8a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.162+0000 7fd84abf6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd8401aee80 con 0x557bafe90f30 2024-04-21T23:10:32.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.162+0000 7fd84abf6740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd8401b1700 con 0x557bafe91850 2024-04-21T23:10:32.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.162+0000 7fd84abf6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd8401af4f0 con 0x557bafe9a430 2024-04-21T23:10:32.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.162+0000 7fd8465ab640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe9a430 0x557bafe8a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.162+0000 7fd845daa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe90f30 0x557bafe99ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.162+0000 7fd8455a9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe91850 0x557bafea60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.162+0000 7fd8465ab640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe9a430 0x557bafe8a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:42434/0 (socket says 172.21.15.44:42434) 2024-04-21T23:10:32.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.162+0000 7fd8465ab640 1 -- 172.21.15.44:0/35153273 learned_addr learned my addr 172.21.15.44:0/35153273 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:32.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.162+0000 7fd845daa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe90f30 0x557bafe99ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50192/0 (socket says 172.21.15.44:50192) 2024-04-21T23:10:32.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.163+0000 7fd8465ab640 1 -- 172.21.15.44:0/35153273 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe91850 msgr2=0x557bafea60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:32.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.163+0000 7fd8465ab640 1 --2- 172.21.15.44:0/35153273 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe91850 0x557bafea60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.163+0000 7fd8465ab640 1 -- 172.21.15.44:0/35153273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe90f30 msgr2=0x557bafe99ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:32.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.163+0000 7fd8465ab640 1 --2- 172.21.15.44:0/35153273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe90f30 0x557bafe99ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.163+0000 7fd8465ab640 1 -- 172.21.15.44:0/35153273 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bafe91680 con 0x557bafe9a430 2024-04-21T23:10:32.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.163+0000 7fd8465ab640 1 --2- 172.21.15.44:0/35153273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe9a430 0x557bafe8a560 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd838012790 tx=0x7fd838015f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6dc26bb6378a1fe server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:32.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.163+0000 7fd82b7fe640 1 -- 172.21.15.44:0/35153273 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd838036070 con 0x557bafe9a430 2024-04-21T23:10:32.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.163+0000 7fd82b7fe640 1 -- 172.21.15.44:0/35153273 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8380147b0 con 0x557bafe9a430 2024-04-21T23:10:32.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.163+0000 7fd82b7fe640 1 -- 172.21.15.44:0/35153273 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83801c730 con 0x557bafe9a430 2024-04-21T23:10:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.164+0000 7fd84abf6740 1 -- 172.21.15.44:0/35153273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe9a430 msgr2=0x557bafe8a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.164+0000 7fd84abf6740 1 --2- 172.21.15.44:0/35153273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe9a430 0x557bafe8a560 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd838012790 tx=0x7fd838015f50 comp rx=0 tx=0).stop 2024-04-21T23:10:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.164+0000 7fd84abf6740 1 -- 172.21.15.44:0/35153273 shutdown_connections 2024-04-21T23:10:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.164+0000 7fd84abf6740 1 --2- 172.21.15.44:0/35153273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe9a430 0x557bafe8a560 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.164+0000 7fd84abf6740 1 --2- 172.21.15.44:0/35153273 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe91850 0x557bafea60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.164+0000 7fd84abf6740 1 --2- 172.21.15.44:0/35153273 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe90f30 0x557bafe99ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.164+0000 7fd84abf6740 1 -- 172.21.15.44:0/35153273 >> 172.21.15.44:0/35153273 conn(0x557bafe8b660 msgr2=0x557bafe89e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.164+0000 7fd84abf6740 1 -- 172.21.15.44:0/35153273 shutdown_connections 2024-04-21T23:10:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.164+0000 7fd84abf6740 1 -- 172.21.15.44:0/35153273 wait complete. 2024-04-21T23:10:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.165+0000 7fd84abf6740 1 -- start start 2024-04-21T23:10:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.165+0000 7fd84abf6740 1 Processor -- start 2024-04-21T23:10:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.165+0000 7fd84abf6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe90f30 0x557bafe9e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.165+0000 7fd84abf6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe91850 0x557bafe9e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.165+0000 7fd84abf6740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe9a430 0x557bafe9ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.165+0000 7fd84abf6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd8401afbc0 con 0x557bafe91850 2024-04-21T23:10:32.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.165+0000 7fd84abf6740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd8401b1030 con 0x557bafe9a430 2024-04-21T23:10:32.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.165+0000 7fd84abf6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd8401b0290 con 0x557bafe90f30 2024-04-21T23:10:32.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.165+0000 7fd84abf6740 10 client.0 _ll_register_callbacks cb 0x7fd83c0046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:32.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.165+0000 7fd8455a9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe91850 0x557bafe9e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.165+0000 7fd845daa640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe90f30 0x557bafe9e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.165+0000 7fd8455a9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe91850 0x557bafe9e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50206/0 (socket says 172.21.15.44:50206) 2024-04-21T23:10:32.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.166+0000 7fd8455a9640 1 -- 172.21.15.44:0/748410456 learned_addr learned my addr 172.21.15.44:0/748410456 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:32.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.165+0000 7fd845daa640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe90f30 0x557bafe9e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:42438/0 (socket says 172.21.15.44:42438) 2024-04-21T23:10:32.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.166+0000 7fd8465ab640 1 --2- 172.21.15.44:0/748410456 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe9a430 0x557bafe9ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.166+0000 7fd845daa640 1 -- 172.21.15.44:0/748410456 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe9a430 msgr2=0x557bafe9ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:32.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.166+0000 7fd845daa640 1 --2- 172.21.15.44:0/748410456 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe9a430 0x557bafe9ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.166+0000 7fd845daa640 1 -- 172.21.15.44:0/748410456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe91850 msgr2=0x557bafe9e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:32.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.166+0000 7fd845daa640 1 --2- 172.21.15.44:0/748410456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe91850 0x557bafe9e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.166+0000 7fd845daa640 1 -- 172.21.15.44:0/748410456 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557baff0fd60 con 0x557bafe90f30 2024-04-21T23:10:32.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.166+0000 7fd8455a9640 1 --2- 172.21.15.44:0/748410456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe91850 0x557bafe9e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:32.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.166+0000 7fd845daa640 1 --2- 172.21.15.44:0/748410456 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe90f30 0x557bafe9e000 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0bb420 tx=0x7fd83c09ab50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd9261a13449b36c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:32.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.166+0000 7fd8297fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:10:32.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.166+0000 7fd8297fa640 1 -- 172.21.15.44:0/748410456 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c09ae90 con 0x557bafe90f30 2024-04-21T23:10:32.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.166+0000 7fd8297fa640 1 -- 172.21.15.44:0/748410456 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fd83c09a7a0 con 0x557bafe90f30 2024-04-21T23:10:32.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.166+0000 7fd84abf6740 1 -- 172.21.15.44:0/748410456 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557bafdfcdf0 con 0x557bafe90f30 2024-04-21T23:10:32.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.166+0000 7fd8297fa640 1 -- 172.21.15.44:0/748410456 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd83c09b120 con 0x557bafe90f30 2024-04-21T23:10:32.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.166+0000 7fd84abf6740 10 client.25633 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:32.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.166+0000 7fd84abf6740 1 -- 172.21.15.44:0/748410456 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557baff11050 con 0x557bafe90f30 2024-04-21T23:10:32.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.167+0000 7fd8297fa640 1 -- 172.21.15.44:0/748410456 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x557bafdfcdf0 con 0x557bafe90f30 2024-04-21T23:10:32.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.167+0000 7fd8297fa640 1 -- 172.21.15.44:0/748410456 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fd83c0970a0 con 0x557bafe90f30 2024-04-21T23:10:32.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.167+0000 7fd84abf6740 10 client.25633 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:32.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.167+0000 7fd84abf6740 10 client.25633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:32.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.167+0000 7fd84abf6740 20 client.25633 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:32.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.167+0000 7fd84abf6740 1 --2- 172.21.15.44:0/748410456 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557baff117d0 0x557baff13bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.167+0000 7fd84abf6740 4 client.25633 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:10:32.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.167+0000 7fd84abf6740 1 -- 172.21.15.44:0/748410456 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557bafd7a5a0 con 0x557baff117d0 2024-04-21T23:10:32.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.167+0000 7fd8455a9640 1 --2- 172.21.15.44:0/748410456 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557baff117d0 0x557baff13bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.167+0000 7fd8297fa640 10 client.25633 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:32.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.167+0000 7fd8455a9640 1 --2- 172.21.15.44:0/748410456 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557baff117d0 0x557baff13bf0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b480cda4167923bf server_cookie=6bf5c6ddf9d0cd28 in_seq=0 out_seq=0 2024-04-21T23:10:32.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.168+0000 7fd8297fa640 1 -- 172.21.15.44:0/748410456 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557bafd7a5a0 con 0x557baff117d0 2024-04-21T23:10:32.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.168+0000 7fd8297fa640 10 client.25633 handle_command_reply: tid=0 2024-04-21T23:10:32.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.168+0000 7fd84abf6740 1 client.25633 shutdown 2024-04-21T23:10:32.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.168+0000 7fd84abf6740 1 -- 172.21.15.44:0/748410456 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe90f30 msgr2=0x557bafe9e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.168+0000 7fd84abf6740 1 --2- 172.21.15.44:0/748410456 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe90f30 0x557bafe9e000 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0bb420 tx=0x7fd83c09ab50 comp rx=0 tx=0).stop 2024-04-21T23:10:32.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.168+0000 7fd84abf6740 1 -- 172.21.15.44:0/748410456 shutdown_connections 2024-04-21T23:10:32.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.168+0000 7fd84abf6740 1 --2- 172.21.15.44:0/748410456 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557baff117d0 0x557baff13bf0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.168+0000 7fd84abf6740 1 --2- 172.21.15.44:0/748410456 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557bafe9a430 0x557bafe9ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.168+0000 7fd84abf6740 1 --2- 172.21.15.44:0/748410456 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557bafe91850 0x557bafe9e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.168+0000 7fd8297fa640 0 client.25633 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:32.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.168+0000 7fd84abf6740 1 --2- 172.21.15.44:0/748410456 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bafe90f30 0x557bafe9e000 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.168+0000 7fd84abf6740 1 -- 172.21.15.44:0/748410456 >> 172.21.15.44:0/748410456 conn(0x557bafe8b660 msgr2=0x557bafe97cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:32.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.168+0000 7fd84abf6740 1 -- 172.21.15.44:0/748410456 shutdown_connections 2024-04-21T23:10:32.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.168+0000 7fd84abf6740 1 -- 172.21.15.44:0/748410456 wait complete. 2024-04-21T23:10:32.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.168+0000 7fd84abf6740 20 client.25633 trim_cache size 0 max 16384 2024-04-21T23:10:32.173 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:10:32.173 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-21T23:10:32.173 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-21T23:10:32.173 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:10:32.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.169+0000 7fd84800e640 1 -- 172.21.15.44:0/3266084553 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd82c0a2d00 msgr2=0x7fd82c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.169+0000 7fd84800e640 1 --2- 172.21.15.44:0/3266084553 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd82c0a2d00 0x7fd82c0a51c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd838022000 tx=0x7fd83801f000 comp rx=0 tx=0).stop 2024-04-21T23:10:32.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.169+0000 7fd84800e640 1 -- 172.21.15.44:0/3266084553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd840100530 msgr2=0x7fd84019a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.169+0000 7fd84800e640 1 --2- 172.21.15.44:0/3266084553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd840100530 0x7fd84019a250 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fd83c00d950 tx=0x7fd83c00de10 comp rx=0 tx=0).stop 2024-04-21T23:10:32.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.169+0000 7fd84800e640 1 -- 172.21.15.44:0/3266084553 shutdown_connections 2024-04-21T23:10:32.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.169+0000 7fd84800e640 1 --2- 172.21.15.44:0/3266084553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd84010e0e0 0x7fd84019eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.169+0000 7fd84800e640 1 --2- 172.21.15.44:0/3266084553 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd82c0a2d00 0x7fd82c0a51c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.169+0000 7fd84800e640 1 --2- 172.21.15.44:0/3266084553 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd840100e50 0x7fd84019a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.169+0000 7fd84800e640 1 --2- 172.21.15.44:0/3266084553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd840100530 0x7fd84019a250 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.169+0000 7fd84800e640 1 -- 172.21.15.44:0/3266084553 >> 172.21.15.44:0/3266084553 conn(0x7fd8400777f0 msgr2=0x7fd84010f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:32.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.169+0000 7fd84800e640 1 -- 172.21.15.44:0/3266084553 shutdown_connections 2024-04-21T23:10:32.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.169+0000 7fd84800e640 1 -- 172.21.15.44:0/3266084553 wait complete. 2024-04-21T23:10:32.178 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:32 smithi044 ceph-mon[27934]: pgmap v53: 129 pgs: 129 active+clean; 19 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 1.4 MiB/s wr, 196 op/s 2024-04-21T23:10:32.179 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:10:32.180 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:10:32.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.286+0000 7f4378b5e640 1 -- 172.21.15.44:0/4170358948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43740693e0 msgr2=0x7f437410e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.286+0000 7f4378b5e640 1 --2- 172.21.15.44:0/4170358948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43740693e0 0x7f437410e5d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f436c009ca0 tx=0x7f436c02c180 comp rx=0 tx=0).stop 2024-04-21T23:10:32.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.286+0000 7f4378b5e640 1 -- 172.21.15.44:0/4170358948 shutdown_connections 2024-04-21T23:10:32.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.286+0000 7f4378b5e640 1 --2- 172.21.15.44:0/4170358948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4374068a40 0x7f4374068ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.286+0000 7f4378b5e640 1 --2- 172.21.15.44:0/4170358948 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f43740693e0 0x7f437410e5d0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.286+0000 7f4378b5e640 1 --2- 172.21.15.44:0/4170358948 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4374107810 0x7f4374107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.286+0000 7f4378b5e640 1 -- 172.21.15.44:0/4170358948 >> 172.21.15.44:0/4170358948 conn(0x7f4374077430 msgr2=0x7f4374077840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:32.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.286+0000 7f4378b5e640 1 -- 172.21.15.44:0/4170358948 shutdown_connections 2024-04-21T23:10:32.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.286+0000 7f4378b5e640 1 -- 172.21.15.44:0/4170358948 wait complete. 2024-04-21T23:10:32.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.287+0000 7f4378b5e640 1 Processor -- start 2024-04-21T23:10:32.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.287+0000 7f4378b5e640 1 -- start start 2024-04-21T23:10:32.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.287+0000 7f4378b5e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4374068a40 0x7f4374100960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.287+0000 7f4378b5e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43740693e0 0x7f4374100ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.287+0000 7f4378b5e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4374107810 0x7f43741013e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.287+0000 7f4378b5e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4374111b50 con 0x7f4374068a40 2024-04-21T23:10:32.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.287+0000 7f4378b5e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f43741119d0 con 0x7f4374107810 2024-04-21T23:10:32.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.287+0000 7f4378b5e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4374111cd0 con 0x7f43740693e0 2024-04-21T23:10:32.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.287+0000 7f437259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4374068a40 0x7f4374100960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.287+0000 7f437259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4374068a40 0x7f4374100960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50232/0 (socket says 172.21.15.44:50232) 2024-04-21T23:10:32.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.287+0000 7f437259c640 1 -- 172.21.15.44:0/1036793658 learned_addr learned my addr 172.21.15.44:0/1036793658 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:32.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.287+0000 7f436bfff640 1 --2- 172.21.15.44:0/1036793658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43740693e0 0x7f4374100ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.288+0000 7f4372d9d640 1 --2- 172.21.15.44:0/1036793658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4374107810 0x7f43741013e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.288+0000 7f437259c640 1 -- 172.21.15.44:0/1036793658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43740693e0 msgr2=0x7f4374100ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.288+0000 7f437259c640 1 --2- 172.21.15.44:0/1036793658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43740693e0 0x7f4374100ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.288+0000 7f437259c640 1 -- 172.21.15.44:0/1036793658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4374107810 msgr2=0x7f43741013e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.288+0000 7f437259c640 1 --2- 172.21.15.44:0/1036793658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4374107810 0x7f43741013e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.288+0000 7f437259c640 1 -- 172.21.15.44:0/1036793658 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43741a6430 con 0x7f4374068a40 2024-04-21T23:10:32.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.288+0000 7f436bfff640 1 --2- 172.21.15.44:0/1036793658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43740693e0 0x7f4374100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:32.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.288+0000 7f4372d9d640 1 --2- 172.21.15.44:0/1036793658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4374107810 0x7f43741013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:10:32.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.288+0000 7f437259c640 1 --2- 172.21.15.44:0/1036793658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4374068a40 0x7f4374100960 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f436400e9f0 tx=0x7f436400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:32.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.288+0000 7f436b7fe640 1 -- 172.21.15.44:0/1036793658 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436400cd80 con 0x7f4374068a40 2024-04-21T23:10:32.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.288+0000 7f4378b5e640 1 -- 172.21.15.44:0/1036793658 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43741a6720 con 0x7f4374068a40 2024-04-21T23:10:32.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.288+0000 7f436b7fe640 1 -- 172.21.15.44:0/1036793658 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4364010910 con 0x7f4374068a40 2024-04-21T23:10:32.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.288+0000 7f436b7fe640 1 -- 172.21.15.44:0/1036793658 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43640188f0 con 0x7f4374068a40 2024-04-21T23:10:32.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.288+0000 7f4378b5e640 1 -- 172.21.15.44:0/1036793658 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43741a6cc0 con 0x7f4374068a40 2024-04-21T23:10:32.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.289+0000 7f4378b5e640 1 -- 172.21.15.44:0/1036793658 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43740685f0 con 0x7f4374068a40 2024-04-21T23:10:32.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.290+0000 7f436b7fe640 1 -- 172.21.15.44:0/1036793658 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f4364010ac0 con 0x7f4374068a40 2024-04-21T23:10:32.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.291+0000 7f436b7fe640 1 --2- 172.21.15.44:0/1036793658 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f43600a2d00 0x7f43600a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.291+0000 7f436bfff640 1 --2- 172.21.15.44:0/1036793658 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f43600a2d00 0x7f43600a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.291+0000 7f436b7fe640 1 -- 172.21.15.44:0/1036793658 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f43640cfc10 con 0x7f4374068a40 2024-04-21T23:10:32.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.292+0000 7f436bfff640 1 --2- 172.21.15.44:0/1036793658 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f43600a2d00 0x7f43600a51c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f435c0095f0 tx=0x7f435c005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:32.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.296+0000 7f436b7fe640 1 -- 172.21.15.44:0/1036793658 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f436409c490 con 0x7f4374068a40 2024-04-21T23:10:32.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:32 smithi055 ceph-mon[31117]: pgmap v53: 129 pgs: 129 active+clean; 19 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 1.4 MiB/s wr, 196 op/s 2024-04-21T23:10:32.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:32 smithi069 ceph-mon[31816]: pgmap v53: 129 pgs: 129 active+clean; 19 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 1.4 MiB/s wr, 196 op/s 2024-04-21T23:10:32.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.532+0000 7f4378b5e640 1 -- 172.21.15.44:0/1036793658 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43741a6fe0 con 0x7f4374068a40 2024-04-21T23:10:32.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.533+0000 7f436b7fe640 1 -- 172.21.15.44:0/1036793658 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11791 (secure 0 0 0) 0x7f436409bd90 con 0x7f4374068a40 2024-04-21T23:10:32.533 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:10:32.533 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:06:50.685663+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3743898853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3743898853},{"type":"v1","addr":"172.21.15.69:6839","nonce":3743898853}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:10:32.533 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 21 2024-04-21T23:10:32.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.535+0000 7f4378b5e640 1 -- 172.21.15.44:0/1036793658 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f43600a2d00 msgr2=0x7f43600a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.535+0000 7f4378b5e640 1 --2- 172.21.15.44:0/1036793658 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f43600a2d00 0x7f43600a51c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f435c0095f0 tx=0x7f435c005f90 comp rx=0 tx=0).stop 2024-04-21T23:10:32.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.535+0000 7f4378b5e640 1 -- 172.21.15.44:0/1036793658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4374068a40 msgr2=0x7f4374100960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.535+0000 7f4378b5e640 1 --2- 172.21.15.44:0/1036793658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4374068a40 0x7f4374100960 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f436400e9f0 tx=0x7f436400eeb0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.535+0000 7f4378b5e640 1 -- 172.21.15.44:0/1036793658 shutdown_connections 2024-04-21T23:10:32.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.536+0000 7f4378b5e640 1 --2- 172.21.15.44:0/1036793658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43740693e0 0x7f4374100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.536+0000 7f4378b5e640 1 --2- 172.21.15.44:0/1036793658 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f43600a2d00 0x7f43600a51c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.536+0000 7f4378b5e640 1 --2- 172.21.15.44:0/1036793658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4374107810 0x7f43741013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.536+0000 7f4378b5e640 1 --2- 172.21.15.44:0/1036793658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4374068a40 0x7f4374100960 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.536+0000 7f4378b5e640 1 -- 172.21.15.44:0/1036793658 >> 172.21.15.44:0/1036793658 conn(0x7f4374077430 msgr2=0x7f43740fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:32.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.536+0000 7f4378b5e640 1 -- 172.21.15.44:0/1036793658 shutdown_connections 2024-04-21T23:10:32.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.536+0000 7f4378b5e640 1 -- 172.21.15.44:0/1036793658 wait complete. 2024-04-21T23:10:32.546 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:10:32.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.656+0000 7fcb88240640 1 -- 172.21.15.44:0/2686507178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb8010bec0 msgr2=0x7fcb8010e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.656+0000 7fcb88240640 1 --2- 172.21.15.44:0/2686507178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb8010bec0 0x7fcb8010e350 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fcb78009900 tx=0x7fcb7802bc30 comp rx=0 tx=0).stop 2024-04-21T23:10:32.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.656+0000 7fcb88240640 1 -- 172.21.15.44:0/2686507178 shutdown_connections 2024-04-21T23:10:32.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.656+0000 7fcb88240640 1 --2- 172.21.15.44:0/2686507178 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb8010bec0 0x7fcb8010e350 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.656+0000 7fcb88240640 1 --2- 172.21.15.44:0/2686507178 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcb80103360 0x7fcb8010b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.656+0000 7fcb88240640 1 --2- 172.21.15.44:0/2686507178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb801029b0 0x7fcb80102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.656+0000 7fcb88240640 1 -- 172.21.15.44:0/2686507178 >> 172.21.15.44:0/2686507178 conn(0x7fcb80077760 msgr2=0x7fcb80077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:32.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.656+0000 7fcb88240640 1 -- 172.21.15.44:0/2686507178 shutdown_connections 2024-04-21T23:10:32.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.656+0000 7fcb88240640 1 -- 172.21.15.44:0/2686507178 wait complete. 2024-04-21T23:10:32.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.656+0000 7fcb88240640 1 Processor -- start 2024-04-21T23:10:32.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.656+0000 7fcb88240640 1 -- start start 2024-04-21T23:10:32.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.657+0000 7fcb88240640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb801029b0 0x7fcb80100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.657+0000 7fcb88240640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcb8010bec0 0x7fcb80100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.657+0000 7fcb88240640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb80101080 0x7fcb801a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.657+0000 7fcb88240640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcb80111cd0 con 0x7fcb801029b0 2024-04-21T23:10:32.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.657+0000 7fcb88240640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcb80111b50 con 0x7fcb8010bec0 2024-04-21T23:10:32.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.657+0000 7fcb88240640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcb80111e50 con 0x7fcb80101080 2024-04-21T23:10:32.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.657+0000 7fcb85fdc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb801029b0 0x7fcb80100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.657+0000 7fcb85fdc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb801029b0 0x7fcb80100600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50248/0 (socket says 172.21.15.44:50248) 2024-04-21T23:10:32.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.657+0000 7fcb85fdc640 1 -- 172.21.15.44:0/3270021080 learned_addr learned my addr 172.21.15.44:0/3270021080 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:32.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.657+0000 7fcb857db640 1 --2- 172.21.15.44:0/3270021080 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcb8010bec0 0x7fcb80100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.657+0000 7fcb867dd640 1 --2- 172.21.15.44:0/3270021080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb80101080 0x7fcb801a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.657+0000 7fcb85fdc640 1 -- 172.21.15.44:0/3270021080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb80101080 msgr2=0x7fcb801a63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.657+0000 7fcb85fdc640 1 --2- 172.21.15.44:0/3270021080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb80101080 0x7fcb801a63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.657+0000 7fcb85fdc640 1 -- 172.21.15.44:0/3270021080 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcb8010bec0 msgr2=0x7fcb80100b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.657+0000 7fcb85fdc640 1 --2- 172.21.15.44:0/3270021080 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcb8010bec0 0x7fcb80100b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.657+0000 7fcb85fdc640 1 -- 172.21.15.44:0/3270021080 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb801a6ae0 con 0x7fcb801029b0 2024-04-21T23:10:32.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.657+0000 7fcb857db640 1 --2- 172.21.15.44:0/3270021080 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcb8010bec0 0x7fcb80100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:32.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.657+0000 7fcb85fdc640 1 --2- 172.21.15.44:0/3270021080 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb801029b0 0x7fcb80100600 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c00ea80 tx=0x7fcb7c00ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:32.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.657+0000 7fcb76ffd640 1 -- 172.21.15.44:0/3270021080 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c00ce40 con 0x7fcb801029b0 2024-04-21T23:10:32.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.658+0000 7fcb76ffd640 1 -- 172.21.15.44:0/3270021080 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb7c005020 con 0x7fcb801029b0 2024-04-21T23:10:32.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.658+0000 7fcb88240640 1 -- 172.21.15.44:0/3270021080 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb801a6dd0 con 0x7fcb801029b0 2024-04-21T23:10:32.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.658+0000 7fcb76ffd640 1 -- 172.21.15.44:0/3270021080 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c011760 con 0x7fcb801029b0 2024-04-21T23:10:32.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.658+0000 7fcb88240640 1 -- 172.21.15.44:0/3270021080 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb801a7240 con 0x7fcb801029b0 2024-04-21T23:10:32.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.660+0000 7fcb76ffd640 1 -- 172.21.15.44:0/3270021080 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fcb7c0051d0 con 0x7fcb801029b0 2024-04-21T23:10:32.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.660+0000 7fcb8ae2e740 1 Processor -- start 2024-04-21T23:10:32.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.660+0000 7fcb8ae2e740 1 -- start start 2024-04-21T23:10:32.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.660+0000 7fcb76ffd640 1 --2- 172.21.15.44:0/3270021080 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcb6c0a2d30 0x7fcb6c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb8ae2e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f143020 0x56326f143400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb8ae2e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143940 0x56326f14bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb8ae2e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f14c520 0x56326f14e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb867dd640 1 --2- 172.21.15.44:0/3270021080 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcb6c0a2d30 0x7fcb6c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb8ae2e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56326f0a7530 con 0x56326f143020 2024-04-21T23:10:32.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb8ae2e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x56326f04f830 con 0x56326f14c520 2024-04-21T23:10:32.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb8ae2e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56326f0a34f0 con 0x56326f143940 2024-04-21T23:10:32.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb76ffd640 1 -- 172.21.15.44:0/3270021080 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fcb7c0cfd80 con 0x7fcb801029b0 2024-04-21T23:10:32.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb85fdc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f143020 0x56326f143400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb85fdc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f143020 0x56326f143400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50252/0 (socket says 172.21.15.44:50252) 2024-04-21T23:10:32.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb85fdc640 1 -- 172.21.15.44:0/2006476691 learned_addr learned my addr 172.21.15.44:0/2006476691 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:32.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb857db640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143940 0x56326f14bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb867dd640 1 --2- 172.21.15.44:0/2006476691 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f14c520 0x56326f14e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb857db640 1 -- 172.21.15.44:0/2006476691 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f14c520 msgr2=0x56326f14e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:32.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb857db640 1 --2- 172.21.15.44:0/2006476691 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f14c520 0x56326f14e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb857db640 1 -- 172.21.15.44:0/2006476691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f143020 msgr2=0x56326f143400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:32.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb857db640 1 --2- 172.21.15.44:0/2006476691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f143020 0x56326f143400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb857db640 1 -- 172.21.15.44:0/2006476691 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56326f0732c0 con 0x56326f143940 2024-04-21T23:10:32.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb85fdc640 1 --2- 172.21.15.44:0/2006476691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f143020 0x56326f143400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:32.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb867dd640 1 --2- 172.21.15.44:0/3270021080 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcb6c0a2d30 0x7fcb6c0a51f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcb78005ab0 tx=0x7fcb7802c7e0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:32.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb857db640 1 --2- 172.21.15.44:0/2006476691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143940 0x56326f14bfe0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcb8006a150 tx=0x7fcb70011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4af48a2d4d664b7d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:32.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.661+0000 7fcb6bfff640 1 -- 172.21.15.44:0/2006476691 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb70004030 con 0x56326f143940 2024-04-21T23:10:32.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.662+0000 7fcb6bfff640 1 -- 172.21.15.44:0/2006476691 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb70005500 con 0x56326f143940 2024-04-21T23:10:32.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.662+0000 7fcb6bfff640 1 -- 172.21.15.44:0/2006476691 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb70004d90 con 0x56326f143940 2024-04-21T23:10:32.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.662+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/2006476691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143940 msgr2=0x56326f14bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.662+0000 7fcb8ae2e740 1 --2- 172.21.15.44:0/2006476691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143940 0x56326f14bfe0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcb8006a150 tx=0x7fcb70011810 comp rx=0 tx=0).stop 2024-04-21T23:10:32.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.662+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/2006476691 shutdown_connections 2024-04-21T23:10:32.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.662+0000 7fcb8ae2e740 1 --2- 172.21.15.44:0/2006476691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143940 0x56326f14bfe0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.662+0000 7fcb8ae2e740 1 --2- 172.21.15.44:0/2006476691 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f14c520 0x56326f14e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.662+0000 7fcb8ae2e740 1 --2- 172.21.15.44:0/2006476691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f143020 0x56326f143400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.662+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/2006476691 >> 172.21.15.44:0/2006476691 conn(0x56326f13d750 msgr2=0x56326f13db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:32.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.662+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/2006476691 shutdown_connections 2024-04-21T23:10:32.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.662+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/2006476691 wait complete. 2024-04-21T23:10:32.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.663+0000 7fcb8ae2e740 1 -- start start 2024-04-21T23:10:32.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.663+0000 7fcb8ae2e740 1 Processor -- start 2024-04-21T23:10:32.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.663+0000 7fcb8ae2e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143020 0x56326f14be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.663+0000 7fcb8ae2e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f143940 0x56326f149210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.663+0000 7fcb8ae2e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f14c520 0x56326f149770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.663+0000 7fcb8ae2e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56326f14f310 con 0x56326f143940 2024-04-21T23:10:32.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.663+0000 7fcb8ae2e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x56326f062f10 con 0x56326f14c520 2024-04-21T23:10:32.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.663+0000 7fcb8ae2e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56326f14f490 con 0x56326f143020 2024-04-21T23:10:32.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.663+0000 7fcb8ae2e740 10 client.0 _ll_register_callbacks cb 0x56326edbabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:32.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.663+0000 7fcb85fdc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143020 0x56326f14be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.663+0000 7fcb857db640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f143940 0x56326f149210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.663+0000 7fcb85fdc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143020 0x56326f14be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:42500/0 (socket says 172.21.15.44:42500) 2024-04-21T23:10:32.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.663+0000 7fcb85fdc640 1 -- 172.21.15.44:0/2198649393 learned_addr learned my addr 172.21.15.44:0/2198649393 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:32.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.663+0000 7fcb867dd640 1 --2- 172.21.15.44:0/2198649393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f14c520 0x56326f149770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.663+0000 7fcb85fdc640 1 -- 172.21.15.44:0/2198649393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f14c520 msgr2=0x56326f149770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:32.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.663+0000 7fcb85fdc640 1 --2- 172.21.15.44:0/2198649393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f14c520 0x56326f149770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.663+0000 7fcb85fdc640 1 -- 172.21.15.44:0/2198649393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f143940 msgr2=0x56326f149210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:32.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.663+0000 7fcb85fdc640 1 --2- 172.21.15.44:0/2198649393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f143940 0x56326f149210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.663+0000 7fcb85fdc640 1 -- 172.21.15.44:0/2198649393 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56326f034c80 con 0x56326f143020 2024-04-21T23:10:32.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.664+0000 7fcb85fdc640 1 --2- 172.21.15.44:0/2198649393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143020 0x56326f14be80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c0983c0 tx=0x7fcb7c0a03e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fc41441450c2424 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:32.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.664+0000 7fcb697fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:10:32.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.664+0000 7fcb697fa640 1 -- 172.21.15.44:0/2198649393 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c01f070 con 0x56326f143020 2024-04-21T23:10:32.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.664+0000 7fcb697fa640 1 -- 172.21.15.44:0/2198649393 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fcb7c0a48e0 con 0x56326f143020 2024-04-21T23:10:32.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.664+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/2198649393 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb70005500 con 0x56326f143020 2024-04-21T23:10:32.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.664+0000 7fcb697fa640 1 -- 172.21.15.44:0/2198649393 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcb7c0bab40 con 0x56326f143020 2024-04-21T23:10:32.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.664+0000 7fcb8ae2e740 10 client.25651 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:32.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.664+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/2198649393 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56326edbb3f0 con 0x56326f143020 2024-04-21T23:10:32.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.664+0000 7fcb697fa640 1 -- 172.21.15.44:0/2198649393 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb70005500 con 0x56326f143020 2024-04-21T23:10:32.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.664+0000 7fcb697fa640 1 -- 172.21.15.44:0/2198649393 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fcb7c00f040 con 0x56326f143020 2024-04-21T23:10:32.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.664+0000 7fcb8ae2e740 10 client.25651 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:32.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.664+0000 7fcb8ae2e740 10 client.25651 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:32.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.664+0000 7fcb8ae2e740 20 client.25651 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:32.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.664+0000 7fcb8ae2e740 1 --2- 172.21.15.44:0/2198649393 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56326f15c5d0 0x56326f15e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.664+0000 7fcb8ae2e740 4 client.25651 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:10:32.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.664+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/2198649393 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56326f0b4a70 con 0x56326f15c5d0 2024-04-21T23:10:32.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.665+0000 7fcb857db640 1 --2- 172.21.15.44:0/2198649393 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56326f15c5d0 0x56326f15e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.665+0000 7fcb857db640 1 --2- 172.21.15.44:0/2198649393 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56326f15c5d0 0x56326f15e9c0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e747bd0e907ed110 server_cookie=d1032a9a89bac70b in_seq=0 out_seq=0 2024-04-21T23:10:32.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.665+0000 7fcb697fa640 10 client.25651 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:32.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.668+0000 7fcb697fa640 1 -- 172.21.15.44:0/2198649393 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56326f0b4a70 con 0x56326f15c5d0 2024-04-21T23:10:32.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.668+0000 7fcb697fa640 10 client.25651 handle_command_reply: tid=0 2024-04-21T23:10:32.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.668+0000 7fcb8ae2e740 1 client.25651 shutdown 2024-04-21T23:10:32.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.668+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/2198649393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143020 msgr2=0x56326f14be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.668+0000 7fcb8ae2e740 1 --2- 172.21.15.44:0/2198649393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143020 0x56326f14be80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c0983c0 tx=0x7fcb7c0a03e0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.668+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/2198649393 shutdown_connections 2024-04-21T23:10:32.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.668+0000 7fcb8ae2e740 1 --2- 172.21.15.44:0/2198649393 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56326f15c5d0 0x56326f15e9c0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.668+0000 7fcb8ae2e740 1 --2- 172.21.15.44:0/2198649393 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f14c520 0x56326f149770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.668+0000 7fcb697fa640 0 client.25651 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:32.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.668+0000 7fcb8ae2e740 1 --2- 172.21.15.44:0/2198649393 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f143940 0x56326f149210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.668+0000 7fcb8ae2e740 1 --2- 172.21.15.44:0/2198649393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143020 0x56326f14be80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.668+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/2198649393 >> 172.21.15.44:0/2198649393 conn(0x56326f13d750 msgr2=0x56326f13f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:32.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.669+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/2198649393 shutdown_connections 2024-04-21T23:10:32.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.669+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/2198649393 wait complete. 2024-04-21T23:10:32.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.669+0000 7fcb8ae2e740 20 client.25651 trim_cache size 0 max 16384 2024-04-21T23:10:32.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.683+0000 7fcb8ae2e740 1 Processor -- start 2024-04-21T23:10:32.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.683+0000 7fcb8ae2e740 1 -- start start 2024-04-21T23:10:32.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.683+0000 7fcb8ae2e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f143020 0x56326f1581d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.683+0000 7fcb8ae2e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143940 0x56326f14b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.683+0000 7fcb8ae2e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f14c520 0x56326f14be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.683+0000 7fcb8ae2e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcb801aee30 con 0x56326f14c520 2024-04-21T23:10:32.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.683+0000 7fcb8ae2e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcb801b16b0 con 0x56326f143020 2024-04-21T23:10:32.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.683+0000 7fcb8ae2e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcb801af4a0 con 0x56326f143940 2024-04-21T23:10:32.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.683+0000 7fcb857db640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143940 0x56326f14b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.683+0000 7fcb867dd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f14c520 0x56326f14be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.683+0000 7fcb857db640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143940 0x56326f14b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:42512/0 (socket says 172.21.15.44:42512) 2024-04-21T23:10:32.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.683+0000 7fcb85fdc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f143020 0x56326f1581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.683+0000 7fcb857db640 1 -- 172.21.15.44:0/4187484300 learned_addr learned my addr 172.21.15.44:0/4187484300 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:32.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.684+0000 7fcb857db640 1 -- 172.21.15.44:0/4187484300 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f143020 msgr2=0x56326f1581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:32.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.684+0000 7fcb857db640 1 --2- 172.21.15.44:0/4187484300 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f143020 0x56326f1581d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.684+0000 7fcb857db640 1 -- 172.21.15.44:0/4187484300 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f14c520 msgr2=0x56326f14be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:32.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.684+0000 7fcb857db640 1 --2- 172.21.15.44:0/4187484300 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f14c520 0x56326f14be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.684+0000 7fcb857db640 1 -- 172.21.15.44:0/4187484300 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56326ee857d0 con 0x56326f143940 2024-04-21T23:10:32.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.684+0000 7fcb857db640 1 --2- 172.21.15.44:0/4187484300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143940 0x56326f14b940 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcb70004960 tx=0x7fcb70013040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6b959432a4420a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:32.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.684+0000 7fcb6b7fe640 1 -- 172.21.15.44:0/4187484300 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7002f070 con 0x56326f143940 2024-04-21T23:10:32.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.684+0000 7fcb6b7fe640 1 -- 172.21.15.44:0/4187484300 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb70004530 con 0x56326f143940 2024-04-21T23:10:32.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.684+0000 7fcb6b7fe640 1 -- 172.21.15.44:0/4187484300 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb70003b20 con 0x56326f143940 2024-04-21T23:10:32.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.685+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/4187484300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143940 msgr2=0x56326f14b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.685+0000 7fcb8ae2e740 1 --2- 172.21.15.44:0/4187484300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143940 0x56326f14b940 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcb70004960 tx=0x7fcb70013040 comp rx=0 tx=0).stop 2024-04-21T23:10:32.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.685+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/4187484300 shutdown_connections 2024-04-21T23:10:32.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.685+0000 7fcb8ae2e740 1 --2- 172.21.15.44:0/4187484300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143940 0x56326f14b940 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.685+0000 7fcb8ae2e740 1 --2- 172.21.15.44:0/4187484300 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f14c520 0x56326f14be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.685+0000 7fcb8ae2e740 1 --2- 172.21.15.44:0/4187484300 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f143020 0x56326f1581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.685+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/4187484300 >> 172.21.15.44:0/4187484300 conn(0x56326f13d750 msgr2=0x56326f13c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:32.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.685+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/4187484300 shutdown_connections 2024-04-21T23:10:32.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.685+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/4187484300 wait complete. 2024-04-21T23:10:32.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.685+0000 7fcb8ae2e740 1 -- start start 2024-04-21T23:10:32.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.685+0000 7fcb8ae2e740 1 Processor -- start 2024-04-21T23:10:32.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.685+0000 7fcb8ae2e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143020 0x56326f154b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.685+0000 7fcb8ae2e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f143940 0x56326f14feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.685+0000 7fcb8ae2e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f14c520 0x56326f150430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.685+0000 7fcb8ae2e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcb801afb70 con 0x56326f143940 2024-04-21T23:10:32.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.685+0000 7fcb8ae2e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcb801b0fe0 con 0x56326f14c520 2024-04-21T23:10:32.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.685+0000 7fcb8ae2e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcb801b0240 con 0x56326f143020 2024-04-21T23:10:32.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb8ae2e740 10 client.0 _ll_register_callbacks cb 0x7fcb7c0cd0d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:32.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb857db640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f143940 0x56326f14feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb857db640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f143940 0x56326f14feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50292/0 (socket says 172.21.15.44:50292) 2024-04-21T23:10:32.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb857db640 1 -- 172.21.15.44:0/3432662469 learned_addr learned my addr 172.21.15.44:0/3432662469 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:32.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb867dd640 1 --2- 172.21.15.44:0/3432662469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f14c520 0x56326f150430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb85fdc640 1 --2- 172.21.15.44:0/3432662469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143020 0x56326f154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb857db640 1 -- 172.21.15.44:0/3432662469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143020 msgr2=0x56326f154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:32.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb857db640 1 --2- 172.21.15.44:0/3432662469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143020 0x56326f154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb857db640 1 -- 172.21.15.44:0/3432662469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f14c520 msgr2=0x56326f150430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:32.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb857db640 1 --2- 172.21.15.44:0/3432662469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f14c520 0x56326f150430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb857db640 1 -- 172.21.15.44:0/3432662469 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56326f150c60 con 0x56326f143940 2024-04-21T23:10:32.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb867dd640 1 --2- 172.21.15.44:0/3432662469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f14c520 0x56326f150430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:32.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb85fdc640 1 --2- 172.21.15.44:0/3432662469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143020 0x56326f154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:32.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb857db640 1 --2- 172.21.15.44:0/3432662469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f143940 0x56326f14feb0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fcb70009bd0 tx=0x7fcb70005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee0b1643fb9d01f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:32.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb697fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:10:32.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb697fa640 1 -- 172.21.15.44:0/3432662469 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7002f070 con 0x56326f143940 2024-04-21T23:10:32.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/3432662469 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb70004530 con 0x56326f143940 2024-04-21T23:10:32.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb697fa640 1 -- 172.21.15.44:0/3432662469 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fcb7001e070 con 0x56326f143940 2024-04-21T23:10:32.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb697fa640 1 -- 172.21.15.44:0/3432662469 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fcb7003d050 con 0x56326f143940 2024-04-21T23:10:32.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb8ae2e740 10 client.16035 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:32.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.686+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/3432662469 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56326f1c2b30 con 0x56326f143940 2024-04-21T23:10:32.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.687+0000 7fcb697fa640 1 -- 172.21.15.44:0/3432662469 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb70004530 con 0x56326f143940 2024-04-21T23:10:32.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.687+0000 7fcb697fa640 1 -- 172.21.15.44:0/3432662469 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fcb70034460 con 0x56326f143940 2024-04-21T23:10:32.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.687+0000 7fcb8ae2e740 10 client.16035 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:32.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.687+0000 7fcb8ae2e740 10 client.16035 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:32.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.687+0000 7fcb8ae2e740 20 client.16035 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:32.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.687+0000 7fcb8ae2e740 1 --2- 172.21.15.44:0/3432662469 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56326f1c3100 0x56326f1c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:32.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.687+0000 7fcb8ae2e740 4 client.16035 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:10:32.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.687+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/3432662469 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56326f0aedc0 con 0x56326f1c3100 2024-04-21T23:10:32.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.687+0000 7fcb85fdc640 1 --2- 172.21.15.44:0/3432662469 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56326f1c3100 0x56326f1c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:32.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.687+0000 7fcb85fdc640 1 --2- 172.21.15.44:0/3432662469 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56326f1c3100 0x56326f1c5520 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=991ed4f109e34945 server_cookie=f24517ea6339b3ee in_seq=0 out_seq=0 2024-04-21T23:10:32.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.687+0000 7fcb697fa640 10 client.16035 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:32.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.688+0000 7fcb697fa640 1 -- 172.21.15.44:0/3432662469 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56326f0aedc0 con 0x56326f1c3100 2024-04-21T23:10:32.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.688+0000 7fcb697fa640 10 client.16035 handle_command_reply: tid=0 2024-04-21T23:10:32.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.688+0000 7fcb8ae2e740 1 client.16035 shutdown 2024-04-21T23:10:32.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.688+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/3432662469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f143940 msgr2=0x56326f14feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.688+0000 7fcb8ae2e740 1 --2- 172.21.15.44:0/3432662469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f143940 0x56326f14feb0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fcb70009bd0 tx=0x7fcb70005b20 comp rx=0 tx=0).stop 2024-04-21T23:10:32.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.688+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/3432662469 shutdown_connections 2024-04-21T23:10:32.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.688+0000 7fcb8ae2e740 1 --2- 172.21.15.44:0/3432662469 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56326f1c3100 0x56326f1c5520 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.688+0000 7fcb8ae2e740 1 --2- 172.21.15.44:0/3432662469 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56326f14c520 0x56326f150430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.688+0000 7fcb8ae2e740 1 --2- 172.21.15.44:0/3432662469 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56326f143940 0x56326f14feb0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.688+0000 7fcb8ae2e740 1 --2- 172.21.15.44:0/3432662469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56326f143020 0x56326f154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.688+0000 7fcb697fa640 0 client.16035 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:32.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.688+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/3432662469 >> 172.21.15.44:0/3432662469 conn(0x56326f13d750 msgr2=0x56326f143e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:32.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.688+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/3432662469 shutdown_connections 2024-04-21T23:10:32.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.688+0000 7fcb8ae2e740 1 -- 172.21.15.44:0/3432662469 wait complete. 2024-04-21T23:10:32.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.688+0000 7fcb8ae2e740 20 client.16035 trim_cache size 0 max 16384 2024-04-21T23:10:32.695 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-21T23:10:32.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.689+0000 7fcb88240640 1 -- 172.21.15.44:0/3270021080 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcb6c0a2d30 msgr2=0x7fcb6c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.689+0000 7fcb88240640 1 --2- 172.21.15.44:0/3270021080 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcb6c0a2d30 0x7fcb6c0a51f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcb78005ab0 tx=0x7fcb7802c7e0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.689+0000 7fcb88240640 1 -- 172.21.15.44:0/3270021080 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb801029b0 msgr2=0x7fcb80100600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:32.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.689+0000 7fcb88240640 1 --2- 172.21.15.44:0/3270021080 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb801029b0 0x7fcb80100600 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c00ea80 tx=0x7fcb7c00ef40 comp rx=0 tx=0).stop 2024-04-21T23:10:32.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.689+0000 7fcb88240640 1 -- 172.21.15.44:0/3270021080 shutdown_connections 2024-04-21T23:10:32.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.689+0000 7fcb88240640 1 --2- 172.21.15.44:0/3270021080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb80101080 0x7fcb801a63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.689+0000 7fcb88240640 1 --2- 172.21.15.44:0/3270021080 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcb6c0a2d30 0x7fcb6c0a51f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.689+0000 7fcb88240640 1 --2- 172.21.15.44:0/3270021080 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcb8010bec0 0x7fcb80100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.689+0000 7fcb88240640 1 --2- 172.21.15.44:0/3270021080 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcb801029b0 0x7fcb80100600 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:32.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.689+0000 7fcb88240640 1 -- 172.21.15.44:0/3270021080 >> 172.21.15.44:0/3270021080 conn(0x7fcb80077760 msgr2=0x7fcb800fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:32.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.689+0000 7fcb88240640 1 -- 172.21.15.44:0/3270021080 shutdown_connections 2024-04-21T23:10:32.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:32.689+0000 7fcb88240640 1 -- 172.21.15.44:0/3270021080 wait complete. 2024-04-21T23:10:32.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 64172 16.58 MB/sec warmup 22 sec latency 15.583 ms 2024-04-21T23:10:33.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:33 smithi044 ceph-mon[27934]: scrub queued for path: / 2024-04-21T23:10:33.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:33 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:10:33.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:33 smithi044 ceph-mon[27934]: scrub summary: active paths [/] 2024-04-21T23:10:33.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:33 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:10:33.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:33 smithi044 ceph-mon[27934]: scrub summary: idle 2024-04-21T23:10:33.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:33 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1036793658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:10:33.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:33 smithi055 ceph-mon[31117]: scrub queued for path: / 2024-04-21T23:10:33.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:33 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:10:33.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:33 smithi055 ceph-mon[31117]: scrub summary: active paths [/] 2024-04-21T23:10:33.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:33 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:10:33.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:33 smithi055 ceph-mon[31117]: scrub summary: idle 2024-04-21T23:10:33.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:33 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1036793658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:10:33.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:33 smithi069 ceph-mon[31816]: scrub queued for path: / 2024-04-21T23:10:33.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:33 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:10:33.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:33 smithi069 ceph-mon[31816]: scrub summary: active paths [/] 2024-04-21T23:10:33.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:33 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:10:33.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:33 smithi069 ceph-mon[31816]: scrub summary: idle 2024-04-21T23:10:33.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:33 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1036793658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:10:33.691 INFO:tasks.workunit.client.0.smithi044.stdout: 1 67065 16.46 MB/sec warmup 23 sec latency 24.854 ms 2024-04-21T23:10:33.700 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:10:33.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:10:33.700 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:10:33.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.817+0000 7fbd9f7a7640 1 -- 172.21.15.44:0/3959587820 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd981029b0 msgr2=0x7fbd98102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:33.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.817+0000 7fbd9f7a7640 1 --2- 172.21.15.44:0/3959587820 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd981029b0 0x7fbd98102d90 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fbd94009900 tx=0x7fbd9402bc30 comp rx=0 tx=0).stop 2024-04-21T23:10:33.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.817+0000 7fbd9f7a7640 1 -- 172.21.15.44:0/3959587820 shutdown_connections 2024-04-21T23:10:33.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.817+0000 7fbd9f7a7640 1 --2- 172.21.15.44:0/3959587820 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd9810bec0 0x7fbd9810e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.817+0000 7fbd9f7a7640 1 --2- 172.21.15.44:0/3959587820 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd98103360 0x7fbd9810b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.817+0000 7fbd9f7a7640 1 --2- 172.21.15.44:0/3959587820 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd981029b0 0x7fbd98102d90 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.817+0000 7fbd9f7a7640 1 -- 172.21.15.44:0/3959587820 >> 172.21.15.44:0/3959587820 conn(0x7fbd98077760 msgr2=0x7fbd98077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:33.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.817+0000 7fbd9f7a7640 1 -- 172.21.15.44:0/3959587820 shutdown_connections 2024-04-21T23:10:33.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.817+0000 7fbd9f7a7640 1 -- 172.21.15.44:0/3959587820 wait complete. 2024-04-21T23:10:33.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.818+0000 7fbd9f7a7640 1 Processor -- start 2024-04-21T23:10:33.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.818+0000 7fbd9f7a7640 1 -- start start 2024-04-21T23:10:33.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.818+0000 7fbd9f7a7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd981029b0 0x7fbd98198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.818+0000 7fbd9f7a7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd9810bec0 0x7fbd98198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.818+0000 7fbd9f7a7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd9819caf0 0x7fbd9819cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.818+0000 7fbd9f7a7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbd98111cd0 con 0x7fbd981029b0 2024-04-21T23:10:33.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.818+0000 7fbd9f7a7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbd98111b50 con 0x7fbd9819caf0 2024-04-21T23:10:33.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.818+0000 7fbd9f7a7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbd98111e50 con 0x7fbd9810bec0 2024-04-21T23:10:33.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.818+0000 7fbd9d543640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd981029b0 0x7fbd98198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.818+0000 7fbd9d543640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd981029b0 0x7fbd98198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50310/0 (socket says 172.21.15.44:50310) 2024-04-21T23:10:33.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.818+0000 7fbd9d543640 1 -- 172.21.15.44:0/569866010 learned_addr learned my addr 172.21.15.44:0/569866010 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:33.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.819+0000 7fbd9dd44640 1 --2- 172.21.15.44:0/569866010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd9819caf0 0x7fbd9819cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.819+0000 7fbd9d543640 1 -- 172.21.15.44:0/569866010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd9810bec0 msgr2=0x7fbd98198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:33.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.819+0000 7fbd9d543640 1 --2- 172.21.15.44:0/569866010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd9810bec0 0x7fbd98198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.819+0000 7fbd9d543640 1 -- 172.21.15.44:0/569866010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd9819caf0 msgr2=0x7fbd9819cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:33.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.819+0000 7fbd9d543640 1 --2- 172.21.15.44:0/569866010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd9819caf0 0x7fbd9819cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.819+0000 7fbd9d543640 1 -- 172.21.15.44:0/569866010 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd9819d5e0 con 0x7fbd981029b0 2024-04-21T23:10:33.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.819+0000 7fbd9d543640 1 --2- 172.21.15.44:0/569866010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd981029b0 0x7fbd98198230 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fbd94006010 tx=0x7fbd9402cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:33.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.819+0000 7fbd8e7fc640 1 -- 172.21.15.44:0/569866010 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd94041070 con 0x7fbd981029b0 2024-04-21T23:10:33.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.819+0000 7fbd9f7a7640 1 -- 172.21.15.44:0/569866010 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd9819d870 con 0x7fbd981029b0 2024-04-21T23:10:33.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.819+0000 7fbd8e7fc640 1 -- 172.21.15.44:0/569866010 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd94004450 con 0x7fbd981029b0 2024-04-21T23:10:33.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.819+0000 7fbd8e7fc640 1 -- 172.21.15.44:0/569866010 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9404b060 con 0x7fbd981029b0 2024-04-21T23:10:33.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.819+0000 7fbd9f7a7640 1 -- 172.21.15.44:0/569866010 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd981a5040 con 0x7fbd981029b0 2024-04-21T23:10:33.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.822+0000 7fbd8e7fc640 1 -- 172.21.15.44:0/569866010 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fbd94044070 con 0x7fbd981029b0 2024-04-21T23:10:33.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.822+0000 7fbda238b740 1 Processor -- start 2024-04-21T23:10:33.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.822+0000 7fbda238b740 1 -- start start 2024-04-21T23:10:33.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.822+0000 7fbda238b740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 0x559c46cf1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.822+0000 7fbd8e7fc640 1 --2- 172.21.15.44:0/569866010 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbd840a2d00 0x7fbd840a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.822+0000 7fbda238b740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cf1a40 0x559c46cf1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.822+0000 7fbda238b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cfadb0 0x559c46cfd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.822+0000 7fbda238b740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x559c46c55170 con 0x559c46cf2720 2024-04-21T23:10:33.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.822+0000 7fbda238b740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x559c46bfd7c0 con 0x559c46cf1a40 2024-04-21T23:10:33.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.822+0000 7fbda238b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559c46c51130 con 0x559c46cfadb0 2024-04-21T23:10:33.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.823+0000 7fbd9dd44640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 0x559c46cf1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.823+0000 7fbd9d543640 1 --2- 172.21.15.44:0/569866010 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbd840a2d00 0x7fbd840a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.823+0000 7fbd9cd42640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cf1a40 0x559c46cf1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.823+0000 7fbd9dd44640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 0x559c46cf1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50314/0 (socket says 172.21.15.44:50314) 2024-04-21T23:10:33.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.823+0000 7fbd9dd44640 1 -- 172.21.15.44:0/3403146497 learned_addr learned my addr 172.21.15.44:0/3403146497 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:33.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.823+0000 7fbd8e7fc640 1 -- 172.21.15.44:0/569866010 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fbd940f1490 con 0x7fbd981029b0 2024-04-21T23:10:33.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.823+0000 7fbd9dd44640 1 --2- 172.21.15.44:0/3403146497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cfadb0 0x559c46cfd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.823+0000 7fbd9cd42640 1 -- 172.21.15.44:0/3403146497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cfadb0 msgr2=0x559c46cfd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:33.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.823+0000 7fbd9cd42640 1 --2- 172.21.15.44:0/3403146497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cfadb0 0x559c46cfd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.823+0000 7fbd9cd42640 1 -- 172.21.15.44:0/3403146497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 msgr2=0x559c46cf1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:33.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.823+0000 7fbd9cd42640 1 --2- 172.21.15.44:0/3403146497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 0x559c46cf1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.823+0000 7fbd9cd42640 1 -- 172.21.15.44:0/3403146497 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c46c0ee60 con 0x559c46cf1a40 2024-04-21T23:10:33.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.823+0000 7fbd9dd44640 1 --2- 172.21.15.44:0/3403146497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 0x559c46cf1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:33.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.823+0000 7fbd9d543640 1 --2- 172.21.15.44:0/569866010 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbd840a2d00 0x7fbd840a51c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbd940eeaa0 tx=0x7fbd9407a040 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:33.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.823+0000 7fbd9cd42640 1 --2- 172.21.15.44:0/3403146497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cf1a40 0x559c46cf1ec0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbd9806a150 tx=0x7fbd8800e350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=346944898480e54e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:33.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.823+0000 7fbd837fe640 1 -- 172.21.15.44:0/3403146497 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8800ecf0 con 0x559c46cf1a40 2024-04-21T23:10:33.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.823+0000 7fbd837fe640 1 -- 172.21.15.44:0/3403146497 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd8800ee90 con 0x559c46cf1a40 2024-04-21T23:10:33.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.823+0000 7fbd837fe640 1 -- 172.21.15.44:0/3403146497 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd88004ad0 con 0x559c46cf1a40 2024-04-21T23:10:33.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.824+0000 7fbda238b740 1 -- 172.21.15.44:0/3403146497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cf1a40 msgr2=0x559c46cf1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:33.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.824+0000 7fbda238b740 1 --2- 172.21.15.44:0/3403146497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cf1a40 0x559c46cf1ec0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbd9806a150 tx=0x7fbd8800e350 comp rx=0 tx=0).stop 2024-04-21T23:10:33.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.824+0000 7fbda238b740 1 -- 172.21.15.44:0/3403146497 shutdown_connections 2024-04-21T23:10:33.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.824+0000 7fbda238b740 1 --2- 172.21.15.44:0/3403146497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cfadb0 0x559c46cfd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.824+0000 7fbda238b740 1 --2- 172.21.15.44:0/3403146497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cf1a40 0x559c46cf1ec0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.824+0000 7fbda238b740 1 --2- 172.21.15.44:0/3403146497 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 0x559c46cf1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.824+0000 7fbda238b740 1 -- 172.21.15.44:0/3403146497 >> 172.21.15.44:0/3403146497 conn(0x559c46cb84f0 msgr2=0x559c46cec490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:33.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.824+0000 7fbda238b740 1 -- 172.21.15.44:0/3403146497 shutdown_connections 2024-04-21T23:10:33.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.824+0000 7fbda238b740 1 -- 172.21.15.44:0/3403146497 wait complete. 2024-04-21T23:10:33.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.825+0000 7fbda238b740 1 -- start start 2024-04-21T23:10:33.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.825+0000 7fbda238b740 1 Processor -- start 2024-04-21T23:10:33.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.825+0000 7fbda238b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cf1a40 0x559c46cf7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.825+0000 7fbda238b740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 0x559c46cf8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.825+0000 7fbda238b740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cfadb0 0x559c46cf8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.825+0000 7fbda238b740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x559c46cfdc20 con 0x559c46cf2720 2024-04-21T23:10:33.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.825+0000 7fbda238b740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x559c46c10d40 con 0x559c46cfadb0 2024-04-21T23:10:33.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.825+0000 7fbda238b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559c46cfdda0 con 0x559c46cf1a40 2024-04-21T23:10:33.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.825+0000 7fbda238b740 10 client.0 _ll_register_callbacks cb 0x559c46c4c820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:33.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.825+0000 7fbd9cd42640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 0x559c46cf8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.825+0000 7fbd9dd44640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cf1a40 0x559c46cf7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.825+0000 7fbd9cd42640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 0x559c46cf8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50324/0 (socket says 172.21.15.44:50324) 2024-04-21T23:10:33.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.825+0000 7fbd9cd42640 1 -- 172.21.15.44:0/3416152922 learned_addr learned my addr 172.21.15.44:0/3416152922 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:33.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.825+0000 7fbd9dd44640 1 --2- 172.21.15.44:0/3416152922 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cfadb0 0x559c46cf8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.825+0000 7fbd9cd42640 1 -- 172.21.15.44:0/3416152922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cf1a40 msgr2=0x559c46cf7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:33.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.826+0000 7fbd9cd42640 1 --2- 172.21.15.44:0/3416152922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cf1a40 0x559c46cf7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.826+0000 7fbd9cd42640 1 -- 172.21.15.44:0/3416152922 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cfadb0 msgr2=0x559c46cf8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:33.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.826+0000 7fbd9cd42640 1 --2- 172.21.15.44:0/3416152922 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cfadb0 0x559c46cf8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.826+0000 7fbd9cd42640 1 -- 172.21.15.44:0/3416152922 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c46a317e0 con 0x559c46cf2720 2024-04-21T23:10:33.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.826+0000 7fbd9cd42640 1 --2- 172.21.15.44:0/3416152922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 0x559c46cf8340 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fbd88004700 tx=0x7fbd880052d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35452b1dcb37088 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:33.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.826+0000 7fbd80ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:10:33.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.826+0000 7fbd80ff9640 1 -- 172.21.15.44:0/3416152922 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd88004440 con 0x559c46cf2720 2024-04-21T23:10:33.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.826+0000 7fbd80ff9640 1 -- 172.21.15.44:0/3416152922 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fbd88004810 con 0x559c46cf2720 2024-04-21T23:10:33.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.826+0000 7fbda238b740 1 -- 172.21.15.44:0/3416152922 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd8800ee90 con 0x559c46cf2720 2024-04-21T23:10:33.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.826+0000 7fbd80ff9640 1 -- 172.21.15.44:0/3416152922 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbd88004a10 con 0x559c46cf2720 2024-04-21T23:10:33.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.826+0000 7fbda238b740 10 client.16047 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:33.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.826+0000 7fbda238b740 1 -- 172.21.15.44:0/3416152922 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559c46a91f70 con 0x559c46cf2720 2024-04-21T23:10:33.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.826+0000 7fbd80ff9640 1 -- 172.21.15.44:0/3416152922 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd8800ee90 con 0x559c46cf2720 2024-04-21T23:10:33.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.827+0000 7fbd80ff9640 1 -- 172.21.15.44:0/3416152922 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fbd88018470 con 0x559c46cf2720 2024-04-21T23:10:33.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.827+0000 7fbda238b740 10 client.16047 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:33.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.827+0000 7fbda238b740 10 client.16047 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:33.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.827+0000 7fbda238b740 20 client.16047 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:33.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.827+0000 7fbda238b740 1 --2- 172.21.15.44:0/3416152922 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x559c46d0ade0 0x559c46d0d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.827+0000 7fbda238b740 4 client.16047 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:10:33.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.827+0000 7fbda238b740 1 -- 172.21.15.44:0/3416152922 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c46c29450 con 0x559c46d0ade0 2024-04-21T23:10:33.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.827+0000 7fbd9dd44640 1 --2- 172.21.15.44:0/3416152922 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x559c46d0ade0 0x559c46d0d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.827+0000 7fbd9dd44640 1 --2- 172.21.15.44:0/3416152922 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x559c46d0ade0 0x559c46d0d1d0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=918c650cea4d3d99 server_cookie=9c4602507f6e489c in_seq=0 out_seq=0 2024-04-21T23:10:33.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.827+0000 7fbd80ff9640 10 client.16047 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:33.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.828+0000 7fbd80ff9640 1 -- 172.21.15.44:0/3416152922 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559c46c29450 con 0x559c46d0ade0 2024-04-21T23:10:33.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.828+0000 7fbd80ff9640 10 client.16047 handle_command_reply: tid=0 2024-04-21T23:10:33.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.828+0000 7fbda238b740 1 client.16047 shutdown 2024-04-21T23:10:33.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.828+0000 7fbda238b740 1 -- 172.21.15.44:0/3416152922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 msgr2=0x559c46cf8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:33.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.828+0000 7fbda238b740 1 --2- 172.21.15.44:0/3416152922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 0x559c46cf8340 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fbd88004700 tx=0x7fbd880052d0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.828+0000 7fbda238b740 1 -- 172.21.15.44:0/3416152922 shutdown_connections 2024-04-21T23:10:33.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.828+0000 7fbda238b740 1 --2- 172.21.15.44:0/3416152922 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x559c46d0ade0 0x559c46d0d1d0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.828+0000 7fbda238b740 1 --2- 172.21.15.44:0/3416152922 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cfadb0 0x559c46cf8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.828+0000 7fbda238b740 1 --2- 172.21.15.44:0/3416152922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 0x559c46cf8340 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.828+0000 7fbd80ff9640 0 client.16047 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:33.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.828+0000 7fbda238b740 1 --2- 172.21.15.44:0/3416152922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cf1a40 0x559c46cf7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.828+0000 7fbda238b740 1 -- 172.21.15.44:0/3416152922 >> 172.21.15.44:0/3416152922 conn(0x559c46cb84f0 msgr2=0x559c46cee1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:33.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.829+0000 7fbda238b740 1 -- 172.21.15.44:0/3416152922 shutdown_connections 2024-04-21T23:10:33.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.829+0000 7fbda238b740 1 -- 172.21.15.44:0/3416152922 wait complete. 2024-04-21T23:10:33.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.829+0000 7fbda238b740 20 client.16047 trim_cache size 0 max 16384 2024-04-21T23:10:33.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.847+0000 7fbda238b740 1 Processor -- start 2024-04-21T23:10:33.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.847+0000 7fbda238b740 1 -- start start 2024-04-21T23:10:33.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.847+0000 7fbda238b740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cf1a40 0x559c46cfa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.847+0000 7fbda238b740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 0x559c46d06b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.847+0000 7fbda238b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cfadb0 0x559c46ceb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.847+0000 7fbda238b740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbd981acc80 con 0x559c46cf2720 2024-04-21T23:10:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.847+0000 7fbda238b740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbd981af500 con 0x559c46cf1a40 2024-04-21T23:10:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.847+0000 7fbda238b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbd981ad2f0 con 0x559c46cfadb0 2024-04-21T23:10:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.847+0000 7fbd9cd42640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 0x559c46d06b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.847+0000 7fbd9cd42640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 0x559c46d06b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50334/0 (socket says 172.21.15.44:50334) 2024-04-21T23:10:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.848+0000 7fbd9cd42640 1 -- 172.21.15.44:0/4259868117 learned_addr learned my addr 172.21.15.44:0/4259868117 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.848+0000 7fbd9dd44640 1 --2- 172.21.15.44:0/4259868117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cfadb0 0x559c46ceb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.848+0000 7fbd9dd44640 1 --2- 172.21.15.44:0/4259868117 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cf1a40 0x559c46cfa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.848+0000 7fbd9cd42640 1 -- 172.21.15.44:0/4259868117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cfadb0 msgr2=0x559c46ceb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.848+0000 7fbd9cd42640 1 --2- 172.21.15.44:0/4259868117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cfadb0 0x559c46ceb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.848+0000 7fbd9cd42640 1 -- 172.21.15.44:0/4259868117 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cf1a40 msgr2=0x559c46cfa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.848+0000 7fbd9cd42640 1 --2- 172.21.15.44:0/4259868117 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cf1a40 0x559c46cfa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.848+0000 7fbd9cd42640 1 -- 172.21.15.44:0/4259868117 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c46929170 con 0x559c46cf2720 2024-04-21T23:10:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.848+0000 7fbd9cd42640 1 --2- 172.21.15.44:0/4259868117 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 0x559c46d06b50 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fbd8800a010 tx=0x7fbd88009a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96535dfe1d348b29 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:33.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.848+0000 7fbd82ffd640 1 -- 172.21.15.44:0/4259868117 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd88036070 con 0x559c46cf2720 2024-04-21T23:10:33.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.848+0000 7fbd82ffd640 1 -- 172.21.15.44:0/4259868117 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd88018b00 con 0x559c46cf2720 2024-04-21T23:10:33.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.848+0000 7fbd82ffd640 1 -- 172.21.15.44:0/4259868117 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd88004bb0 con 0x559c46cf2720 2024-04-21T23:10:33.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.849+0000 7fbda238b740 1 -- 172.21.15.44:0/4259868117 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 msgr2=0x559c46d06b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:33.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.849+0000 7fbda238b740 1 --2- 172.21.15.44:0/4259868117 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 0x559c46d06b50 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fbd8800a010 tx=0x7fbd88009a90 comp rx=0 tx=0).stop 2024-04-21T23:10:33.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.849+0000 7fbda238b740 1 -- 172.21.15.44:0/4259868117 shutdown_connections 2024-04-21T23:10:33.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.849+0000 7fbda238b740 1 --2- 172.21.15.44:0/4259868117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cfadb0 0x559c46ceb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.849+0000 7fbda238b740 1 --2- 172.21.15.44:0/4259868117 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf2720 0x559c46d06b50 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.849+0000 7fbda238b740 1 --2- 172.21.15.44:0/4259868117 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cf1a40 0x559c46cfa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.849+0000 7fbda238b740 1 -- 172.21.15.44:0/4259868117 >> 172.21.15.44:0/4259868117 conn(0x559c46cb84f0 msgr2=0x559c46ceac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.849+0000 7fbda238b740 1 -- 172.21.15.44:0/4259868117 shutdown_connections 2024-04-21T23:10:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.849+0000 7fbda238b740 1 -- 172.21.15.44:0/4259868117 wait complete. 2024-04-21T23:10:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.849+0000 7fbda238b740 1 -- start start 2024-04-21T23:10:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.849+0000 7fbda238b740 1 Processor -- start 2024-04-21T23:10:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.850+0000 7fbda238b740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf1a40 0x559c46cfbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.850+0000 7fbda238b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cf2720 0x559c46cfc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.850+0000 7fbda238b740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cfadb0 0x559c46cfc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.850+0000 7fbda238b740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbd981ad9c0 con 0x559c46cf1a40 2024-04-21T23:10:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.850+0000 7fbda238b740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbd981aee30 con 0x559c46cfadb0 2024-04-21T23:10:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.850+0000 7fbda238b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbd981ae090 con 0x559c46cf2720 2024-04-21T23:10:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.850+0000 7fbda238b740 10 client.0 _ll_register_callbacks cb 0x7fbd90009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.850+0000 7fbd9dd44640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf1a40 0x559c46cfbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.850+0000 7fbd9dd44640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf1a40 0x559c46cfbbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50344/0 (socket says 172.21.15.44:50344) 2024-04-21T23:10:33.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.850+0000 7fbd9dd44640 1 -- 172.21.15.44:0/3137478076 learned_addr learned my addr 172.21.15.44:0/3137478076 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:33.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.850+0000 7fbd9dd44640 1 --2- 172.21.15.44:0/3137478076 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cfadb0 0x559c46cfc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.850+0000 7fbd9cd42640 1 --2- 172.21.15.44:0/3137478076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cf2720 0x559c46cfc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.850+0000 7fbd9dd44640 1 -- 172.21.15.44:0/3137478076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cf2720 msgr2=0x559c46cfc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:33.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.850+0000 7fbd9dd44640 1 --2- 172.21.15.44:0/3137478076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cf2720 0x559c46cfc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.850+0000 7fbd9dd44640 1 -- 172.21.15.44:0/3137478076 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cfadb0 msgr2=0x559c46cfc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:33.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.850+0000 7fbd9dd44640 1 --2- 172.21.15.44:0/3137478076 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cfadb0 0x559c46cfc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.850+0000 7fbd9dd44640 1 -- 172.21.15.44:0/3137478076 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c46d01670 con 0x559c46cf1a40 2024-04-21T23:10:33.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.850+0000 7fbd9cd42640 1 --2- 172.21.15.44:0/3137478076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cf2720 0x559c46cfc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:33.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.850+0000 7fbd9dd44640 1 --2- 172.21.15.44:0/3137478076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf1a40 0x559c46cfbbd0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fbd900147b0 tx=0x7fbd90004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=147fd4f356ec7f91 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:33.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.850+0000 7fbd80ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:10:33.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.851+0000 7fbd80ff9640 1 -- 172.21.15.44:0/3137478076 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9001c040 con 0x559c46cf1a40 2024-04-21T23:10:33.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.851+0000 7fbda238b740 1 -- 172.21.15.44:0/3137478076 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd88018b00 con 0x559c46cf1a40 2024-04-21T23:10:33.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.851+0000 7fbd80ff9640 1 -- 172.21.15.44:0/3137478076 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7fbd90014050 con 0x559c46cf1a40 2024-04-21T23:10:33.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.851+0000 7fbd80ff9640 1 -- 172.21.15.44:0/3137478076 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbd90037050 con 0x559c46cf1a40 2024-04-21T23:10:33.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.851+0000 7fbda238b740 10 client.16053 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:33.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.851+0000 7fbda238b740 1 -- 172.21.15.44:0/3137478076 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559c46d71790 con 0x559c46cf1a40 2024-04-21T23:10:33.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.851+0000 7fbd80ff9640 1 -- 172.21.15.44:0/3137478076 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd88018b00 con 0x559c46cf1a40 2024-04-21T23:10:33.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.851+0000 7fbd80ff9640 1 -- 172.21.15.44:0/3137478076 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fbd90022040 con 0x559c46cf1a40 2024-04-21T23:10:33.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.851+0000 7fbda238b740 10 client.16053 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:33.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.851+0000 7fbda238b740 10 client.16053 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:33.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.851+0000 7fbda238b740 20 client.16053 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:33.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.851+0000 7fbda238b740 1 --2- 172.21.15.44:0/3137478076 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x559c46d71e60 0x559c46d74250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.851+0000 7fbda238b740 4 client.16053 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:10:33.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.851+0000 7fbda238b740 1 -- 172.21.15.44:0/3137478076 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559c46c626b0 con 0x559c46d71e60 2024-04-21T23:10:33.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.851+0000 7fbd9cd42640 1 --2- 172.21.15.44:0/3137478076 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x559c46d71e60 0x559c46d74250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.852+0000 7fbd9cd42640 1 --2- 172.21.15.44:0/3137478076 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x559c46d71e60 0x559c46d74250 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae220190bca42513 server_cookie=21092a113d2d8119 in_seq=0 out_seq=0 2024-04-21T23:10:33.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.852+0000 7fbd80ff9640 10 client.16053 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:33.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.853+0000 7fbd80ff9640 1 -- 172.21.15.44:0/3137478076 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559c46c626b0 con 0x559c46d71e60 2024-04-21T23:10:33.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.853+0000 7fbd80ff9640 10 client.16053 handle_command_reply: tid=0 2024-04-21T23:10:33.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.853+0000 7fbda238b740 1 client.16053 shutdown 2024-04-21T23:10:33.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.853+0000 7fbda238b740 1 -- 172.21.15.44:0/3137478076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf1a40 msgr2=0x559c46cfbbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:33.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.853+0000 7fbda238b740 1 --2- 172.21.15.44:0/3137478076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf1a40 0x559c46cfbbd0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fbd900147b0 tx=0x7fbd90004a40 comp rx=0 tx=0).stop 2024-04-21T23:10:33.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.853+0000 7fbda238b740 1 -- 172.21.15.44:0/3137478076 shutdown_connections 2024-04-21T23:10:33.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.853+0000 7fbda238b740 1 --2- 172.21.15.44:0/3137478076 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x559c46d71e60 0x559c46d74250 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.853+0000 7fbda238b740 1 --2- 172.21.15.44:0/3137478076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559c46cf2720 0x559c46cfc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.853+0000 7fbda238b740 1 --2- 172.21.15.44:0/3137478076 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559c46cfadb0 0x559c46cfc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.853+0000 7fbda238b740 1 --2- 172.21.15.44:0/3137478076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559c46cf1a40 0x559c46cfbbd0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.853+0000 7fbd80ff9640 0 client.16053 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:33.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.853+0000 7fbda238b740 1 -- 172.21.15.44:0/3137478076 >> 172.21.15.44:0/3137478076 conn(0x559c46cb84f0 msgr2=0x559c46ceb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:33.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.853+0000 7fbda238b740 1 -- 172.21.15.44:0/3137478076 shutdown_connections 2024-04-21T23:10:33.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.854+0000 7fbda238b740 1 -- 172.21.15.44:0/3137478076 wait complete. 2024-04-21T23:10:33.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.854+0000 7fbda238b740 20 client.16053 trim_cache size 0 max 16384 2024-04-21T23:10:33.859 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:10:33.859 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-21T23:10:33.859 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "a33bfafe-00bf-43aa-8883-4d78ececd8c2", 2024-04-21T23:10:33.859 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-21T23:10:33.859 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:10:33.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.854+0000 7fbd9f7a7640 1 -- 172.21.15.44:0/569866010 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbd840a2d00 msgr2=0x7fbd840a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:33.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.854+0000 7fbd9f7a7640 1 --2- 172.21.15.44:0/569866010 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbd840a2d00 0x7fbd840a51c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbd940eeaa0 tx=0x7fbd9407a040 comp rx=0 tx=0).stop 2024-04-21T23:10:33.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.854+0000 7fbd9f7a7640 1 -- 172.21.15.44:0/569866010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd981029b0 msgr2=0x7fbd98198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:33.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.854+0000 7fbd9f7a7640 1 --2- 172.21.15.44:0/569866010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd981029b0 0x7fbd98198230 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fbd94006010 tx=0x7fbd9402cb30 comp rx=0 tx=0).stop 2024-04-21T23:10:33.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.854+0000 7fbd9f7a7640 1 -- 172.21.15.44:0/569866010 shutdown_connections 2024-04-21T23:10:33.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.854+0000 7fbd9f7a7640 1 --2- 172.21.15.44:0/569866010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd9810bec0 0x7fbd98198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.854+0000 7fbd9f7a7640 1 --2- 172.21.15.44:0/569866010 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbd840a2d00 0x7fbd840a51c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.854+0000 7fbd9f7a7640 1 --2- 172.21.15.44:0/569866010 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbd9819caf0 0x7fbd9819cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.854+0000 7fbd9f7a7640 1 --2- 172.21.15.44:0/569866010 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbd981029b0 0x7fbd98198230 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.854+0000 7fbd9f7a7640 1 -- 172.21.15.44:0/569866010 >> 172.21.15.44:0/569866010 conn(0x7fbd98077760 msgr2=0x7fbd980fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:33.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.854+0000 7fbd9f7a7640 1 -- 172.21.15.44:0/569866010 shutdown_connections 2024-04-21T23:10:33.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.854+0000 7fbd9f7a7640 1 -- 172.21.15.44:0/569866010 wait complete. 2024-04-21T23:10:33.865 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:10:33.865 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:10:33.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.973+0000 7faa3fb5d640 1 -- 172.21.15.44:0/2210215929 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa381066b0 msgr2=0x7faa38106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:33.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.973+0000 7faa3fb5d640 1 --2- 172.21.15.44:0/2210215929 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa381066b0 0x7faa38106a90 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7faa30009cd0 tx=0x7faa3002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.973+0000 7faa3fb5d640 1 -- 172.21.15.44:0/2210215929 shutdown_connections 2024-04-21T23:10:33.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.973+0000 7faa3fb5d640 1 --2- 172.21.15.44:0/2210215929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa38100fd0 0x7faa3810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.973+0000 7faa3fb5d640 1 --2- 172.21.15.44:0/2210215929 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa38100610 0x7faa38100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.973+0000 7faa3fb5d640 1 --2- 172.21.15.44:0/2210215929 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa381066b0 0x7faa38106a90 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.973+0000 7faa3fb5d640 1 -- 172.21.15.44:0/2210215929 >> 172.21.15.44:0/2210215929 conn(0x7faa380777f0 msgr2=0x7faa38077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:33.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.973+0000 7faa3fb5d640 1 -- 172.21.15.44:0/2210215929 shutdown_connections 2024-04-21T23:10:33.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.973+0000 7faa3fb5d640 1 -- 172.21.15.44:0/2210215929 wait complete. 2024-04-21T23:10:33.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.973+0000 7faa3fb5d640 1 Processor -- start 2024-04-21T23:10:33.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.973+0000 7faa3fb5d640 1 -- start start 2024-04-21T23:10:33.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.974+0000 7faa3fb5d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa38100610 0x7faa38198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.974+0000 7faa3fb5d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa38100fd0 0x7faa38198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.974+0000 7faa3fb5d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa381066b0 0x7faa3819c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.974+0000 7faa3fb5d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faa38111b30 con 0x7faa38100fd0 2024-04-21T23:10:33.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.974+0000 7faa3fb5d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7faa381119b0 con 0x7faa381066b0 2024-04-21T23:10:33.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.974+0000 7faa3fb5d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faa38111cb0 con 0x7faa38100610 2024-04-21T23:10:33.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.974+0000 7faa3d0f8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa38100fd0 0x7faa38198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.974+0000 7faa3d0f8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa38100fd0 0x7faa38198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50362/0 (socket says 172.21.15.44:50362) 2024-04-21T23:10:33.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.974+0000 7faa3d8f9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa38100610 0x7faa38198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.974+0000 7faa3d0f8640 1 -- 172.21.15.44:0/1805993255 learned_addr learned my addr 172.21.15.44:0/1805993255 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:33.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.974+0000 7faa3e0fa640 1 --2- 172.21.15.44:0/1805993255 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa381066b0 0x7faa3819c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.974+0000 7faa3d8f9640 1 -- 172.21.15.44:0/1805993255 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa381066b0 msgr2=0x7faa3819c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:33.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.974+0000 7faa3d8f9640 1 --2- 172.21.15.44:0/1805993255 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa381066b0 0x7faa3819c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.974+0000 7faa3d8f9640 1 -- 172.21.15.44:0/1805993255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa38100fd0 msgr2=0x7faa38198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:33.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.974+0000 7faa3d8f9640 1 --2- 172.21.15.44:0/1805993255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa38100fd0 0x7faa38198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.974+0000 7faa3d8f9640 1 -- 172.21.15.44:0/1805993255 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa3819d040 con 0x7faa38100610 2024-04-21T23:10:33.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.974+0000 7faa3d0f8640 1 --2- 172.21.15.44:0/1805993255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa38100fd0 0x7faa38198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:33.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.974+0000 7faa3d8f9640 1 --2- 172.21.15.44:0/1805993255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa38100610 0x7faa38198050 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7faa3002c630 tx=0x7faa30032c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:33.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.974+0000 7faa2effd640 1 -- 172.21.15.44:0/1805993255 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa30041020 con 0x7faa38100610 2024-04-21T23:10:33.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.975+0000 7faa3fb5d640 1 -- 172.21.15.44:0/1805993255 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa3819d2d0 con 0x7faa38100610 2024-04-21T23:10:33.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.975+0000 7faa3fb5d640 1 -- 172.21.15.44:0/1805993255 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa381a4bc0 con 0x7faa38100610 2024-04-21T23:10:33.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.975+0000 7faa2effd640 1 -- 172.21.15.44:0/1805993255 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa30034cd0 con 0x7faa38100610 2024-04-21T23:10:33.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.975+0000 7faa2effd640 1 -- 172.21.15.44:0/1805993255 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa30039040 con 0x7faa38100610 2024-04-21T23:10:33.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.977+0000 7faa2effd640 1 -- 172.21.15.44:0/1805993255 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7faa30044020 con 0x7faa38100610 2024-04-21T23:10:33.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.978+0000 7faa4281e740 1 Processor -- start 2024-04-21T23:10:33.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.978+0000 7faa4281e740 1 -- start start 2024-04-21T23:10:33.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.978+0000 7faa2effd640 1 --2- 172.21.15.44:0/1805993255 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faa240a2d00 0x7faa240a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.978+0000 7faa4281e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe64f30 0x55c7dfe65310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.978+0000 7faa4281e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe65850 0x55c7dfe6def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.978+0000 7faa4281e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe6e430 0x55c7dfe70820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.978+0000 7faa4281e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55c7dfdc9560 con 0x55c7dfe64f30 2024-04-21T23:10:33.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.978+0000 7faa4281e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55c7dfd71730 con 0x55c7dfe65850 2024-04-21T23:10:33.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.978+0000 7faa4281e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c7dfdc5520 con 0x55c7dfe6e430 2024-04-21T23:10:33.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.978+0000 7faa3e0fa640 1 --2- 172.21.15.44:0/1805993255 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faa240a2d00 0x7faa240a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.978+0000 7faa3d8f9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe64f30 0x55c7dfe65310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.978+0000 7faa2effd640 1 -- 172.21.15.44:0/1805993255 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7faa300f0d50 con 0x7faa38100610 2024-04-21T23:10:33.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.978+0000 7faa3d8f9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe64f30 0x55c7dfe65310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50370/0 (socket says 172.21.15.44:50370) 2024-04-21T23:10:33.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.978+0000 7faa3d8f9640 1 -- 172.21.15.44:0/1872485639 learned_addr learned my addr 172.21.15.44:0/1872485639 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:33.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.978+0000 7faa3d0f8640 1 --2- 172.21.15.44:0/1872485639 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe65850 0x55c7dfe6def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.979+0000 7faa3e0fa640 1 --2- 172.21.15.44:0/1872485639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe6e430 0x55c7dfe70820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.979+0000 7faa3d8f9640 1 -- 172.21.15.44:0/1872485639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe6e430 msgr2=0x55c7dfe70820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:33.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.979+0000 7faa3d8f9640 1 --2- 172.21.15.44:0/1872485639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe6e430 0x55c7dfe70820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.979+0000 7faa3d8f9640 1 -- 172.21.15.44:0/1872485639 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe65850 msgr2=0x55c7dfe6def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:33.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.979+0000 7faa3d8f9640 1 --2- 172.21.15.44:0/1872485639 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe65850 0x55c7dfe6def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.979+0000 7faa3d8f9640 1 -- 172.21.15.44:0/1872485639 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7dfba77d0 con 0x55c7dfe64f30 2024-04-21T23:10:33.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.979+0000 7faa3e0fa640 1 --2- 172.21.15.44:0/1805993255 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faa240a2d00 0x7faa240a51c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faa34009f70 tx=0x7faa34011810 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:33.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.979+0000 7faa3d8f9640 1 --2- 172.21.15.44:0/1872485639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe64f30 0x55c7dfe65310 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7faa300c56c0 tx=0x7faa30003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e48872abe222216 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:33.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.979+0000 7faa23fff640 1 -- 172.21.15.44:0/1872485639 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa300bcd50 con 0x55c7dfe64f30 2024-04-21T23:10:33.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.979+0000 7faa23fff640 1 -- 172.21.15.44:0/1872485639 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa30034920 con 0x55c7dfe64f30 2024-04-21T23:10:33.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.979+0000 7faa23fff640 1 -- 172.21.15.44:0/1872485639 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3008fc60 con 0x55c7dfe64f30 2024-04-21T23:10:33.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.980+0000 7faa4281e740 1 -- 172.21.15.44:0/1872485639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe64f30 msgr2=0x55c7dfe65310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:33.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.980+0000 7faa4281e740 1 --2- 172.21.15.44:0/1872485639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe64f30 0x55c7dfe65310 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7faa300c56c0 tx=0x7faa30003c30 comp rx=0 tx=0).stop 2024-04-21T23:10:33.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.980+0000 7faa4281e740 1 -- 172.21.15.44:0/1872485639 shutdown_connections 2024-04-21T23:10:33.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.980+0000 7faa4281e740 1 --2- 172.21.15.44:0/1872485639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe6e430 0x55c7dfe70820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.980+0000 7faa4281e740 1 --2- 172.21.15.44:0/1872485639 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe65850 0x55c7dfe6def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.980+0000 7faa4281e740 1 --2- 172.21.15.44:0/1872485639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe64f30 0x55c7dfe65310 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.980+0000 7faa4281e740 1 -- 172.21.15.44:0/1872485639 >> 172.21.15.44:0/1872485639 conn(0x55c7dfe5f660 msgr2=0x55c7dfe5fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:33.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.980+0000 7faa4281e740 1 -- 172.21.15.44:0/1872485639 shutdown_connections 2024-04-21T23:10:33.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.980+0000 7faa4281e740 1 -- 172.21.15.44:0/1872485639 wait complete. 2024-04-21T23:10:33.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.980+0000 7faa4281e740 1 -- start start 2024-04-21T23:10:33.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.981+0000 7faa4281e740 1 Processor -- start 2024-04-21T23:10:33.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.981+0000 7faa4281e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe64f30 0x55c7dfe6b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.981+0000 7faa4281e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe65850 0x55c7dfe6b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.981+0000 7faa4281e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe6e430 0x55c7dfe6bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.981+0000 7faa3d8f9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe64f30 0x55c7dfe6b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.981+0000 7faa3d8f9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe64f30 0x55c7dfe6b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50376/0 (socket says 172.21.15.44:50376) 2024-04-21T23:10:33.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.981+0000 7faa3d8f9640 1 -- 172.21.15.44:0/13130689 learned_addr learned my addr 172.21.15.44:0/13130689 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:33.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.981+0000 7faa3d0f8640 1 --2- 172.21.15.44:0/13130689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe65850 0x55c7dfe6b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.982+0000 7faa3e0fa640 1 --2- 172.21.15.44:0/13130689 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe6e430 0x55c7dfe6bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.982+0000 7faa4281e740 1 -- 172.21.15.44:0/13130689 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55c7dfe70fe0 con 0x55c7dfe64f30 2024-04-21T23:10:33.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.982+0000 7faa4281e740 1 -- 172.21.15.44:0/13130689 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55c7dfd84e00 con 0x55c7dfe6e430 2024-04-21T23:10:33.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.982+0000 7faa4281e740 1 -- 172.21.15.44:0/13130689 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c7dfe71160 con 0x55c7dfe65850 2024-04-21T23:10:33.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.982+0000 7faa4281e740 10 client.0 _ll_register_callbacks cb 0x55c7dfdbfeb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:33.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.982+0000 7faa3e0fa640 1 -- 172.21.15.44:0/13130689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe65850 msgr2=0x55c7dfe6b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:33.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.982+0000 7faa3e0fa640 1 --2- 172.21.15.44:0/13130689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe65850 0x55c7dfe6b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.982+0000 7faa3e0fa640 1 -- 172.21.15.44:0/13130689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe64f30 msgr2=0x55c7dfe6b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:33.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.982+0000 7faa3e0fa640 1 --2- 172.21.15.44:0/13130689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe64f30 0x55c7dfe6b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.982+0000 7faa3e0fa640 1 -- 172.21.15.44:0/13130689 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7dfadd3f0 con 0x55c7dfe6e430 2024-04-21T23:10:33.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.982+0000 7faa3d8f9640 1 --2- 172.21.15.44:0/13130689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe64f30 0x55c7dfe6b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:33.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.982+0000 7faa3d0f8640 1 --2- 172.21.15.44:0/13130689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe65850 0x55c7dfe6b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:33.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.982+0000 7faa3e0fa640 1 --2- 172.21.15.44:0/13130689 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe6e430 0x55c7dfe6bf10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x55c7dfe607f0 tx=0x7faa34004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8d12f7e77c9476d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:33.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.982+0000 7faa217fa640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:10:33.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.982+0000 7faa217fa640 1 -- 172.21.15.44:0/13130689 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3401a040 con 0x55c7dfe6e430 2024-04-21T23:10:33.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.982+0000 7faa4281e740 1 -- 172.21.15.44:0/13130689 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa30034920 con 0x55c7dfe6e430 2024-04-21T23:10:33.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.982+0000 7faa217fa640 1 -- 172.21.15.44:0/13130689 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7faa3400f680 con 0x55c7dfe6e430 2024-04-21T23:10:33.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.982+0000 7faa217fa640 1 -- 172.21.15.44:0/13130689 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7faa3400f8c0 con 0x55c7dfe6e430 2024-04-21T23:10:33.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.983+0000 7faa4281e740 10 client.25640 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:33.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.983+0000 7faa4281e740 1 -- 172.21.15.44:0/13130689 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c7dfa9d0f0 con 0x55c7dfe6e430 2024-04-21T23:10:33.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.983+0000 7faa217fa640 1 -- 172.21.15.44:0/13130689 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa30034920 con 0x55c7dfe6e430 2024-04-21T23:10:33.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.983+0000 7faa217fa640 1 -- 172.21.15.44:0/13130689 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7faa34020ac0 con 0x55c7dfe6e430 2024-04-21T23:10:33.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.983+0000 7faa4281e740 10 client.25640 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:33.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.983+0000 7faa4281e740 10 client.25640 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:33.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.983+0000 7faa4281e740 20 client.25640 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:33.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.983+0000 7faa4281e740 1 --2- 172.21.15.44:0/13130689 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55c7dfe7e4b0 0x55c7dfe80910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:33.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.983+0000 7faa4281e740 4 client.25640 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:10:33.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.983+0000 7faa4281e740 1 -- 172.21.15.44:0/13130689 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7dfdd6aa0 con 0x55c7dfe7e4b0 2024-04-21T23:10:33.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.983+0000 7faa3d8f9640 1 --2- 172.21.15.44:0/13130689 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55c7dfe7e4b0 0x55c7dfe80910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:33.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.984+0000 7faa217fa640 10 client.25640 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:33.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.984+0000 7faa3d8f9640 1 --2- 172.21.15.44:0/13130689 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55c7dfe7e4b0 0x55c7dfe80910 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cde6d8f325c272ef server_cookie=74f9585ee84de8b1 in_seq=0 out_seq=0 2024-04-21T23:10:33.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.985+0000 7faa217fa640 1 -- 172.21.15.44:0/13130689 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c7dfdd6aa0 con 0x55c7dfe7e4b0 2024-04-21T23:10:33.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.985+0000 7faa217fa640 10 client.25640 handle_command_reply: tid=0 2024-04-21T23:10:33.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.985+0000 7faa4281e740 1 client.25640 shutdown 2024-04-21T23:10:33.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.985+0000 7faa4281e740 1 -- 172.21.15.44:0/13130689 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe6e430 msgr2=0x55c7dfe6bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:33.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.985+0000 7faa4281e740 1 --2- 172.21.15.44:0/13130689 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe6e430 0x55c7dfe6bf10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x55c7dfe607f0 tx=0x7faa34004730 comp rx=0 tx=0).stop 2024-04-21T23:10:33.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.985+0000 7faa4281e740 1 -- 172.21.15.44:0/13130689 shutdown_connections 2024-04-21T23:10:33.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.985+0000 7faa4281e740 1 --2- 172.21.15.44:0/13130689 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55c7dfe7e4b0 0x55c7dfe80910 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.985+0000 7faa4281e740 1 --2- 172.21.15.44:0/13130689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe65850 0x55c7dfe6b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.985+0000 7faa4281e740 1 --2- 172.21.15.44:0/13130689 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe6e430 0x55c7dfe6bf10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.985+0000 7faa217fa640 0 client.25640 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:33.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.985+0000 7faa4281e740 1 --2- 172.21.15.44:0/13130689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe64f30 0x55c7dfe6b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:33.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.985+0000 7faa4281e740 1 -- 172.21.15.44:0/13130689 >> 172.21.15.44:0/13130689 conn(0x55c7dfe5f660 msgr2=0x55c7dfe613b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:33.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.985+0000 7faa4281e740 1 -- 172.21.15.44:0/13130689 shutdown_connections 2024-04-21T23:10:33.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.986+0000 7faa4281e740 1 -- 172.21.15.44:0/13130689 wait complete. 2024-04-21T23:10:33.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:33.986+0000 7faa4281e740 20 client.25640 trim_cache size 0 max 16384 2024-04-21T23:10:34.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.005+0000 7faa4281e740 1 Processor -- start 2024-04-21T23:10:34.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.005+0000 7faa4281e740 1 -- start start 2024-04-21T23:10:34.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.005+0000 7faa4281e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe64f30 0x55c7dfe6ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:34.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.005+0000 7faa4281e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe65850 0x55c7dfe79d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:34.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.005+0000 7faa4281e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe6e430 0x55c7dfe861e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:34.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.005+0000 7faa4281e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faa381aa9e0 con 0x55c7dfe65850 2024-04-21T23:10:34.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.005+0000 7faa4281e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7faa381ad260 con 0x55c7dfe6e430 2024-04-21T23:10:34.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.005+0000 7faa4281e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faa381ab050 con 0x55c7dfe64f30 2024-04-21T23:10:34.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.005+0000 7faa3d0f8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe65850 0x55c7dfe79d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:34.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.005+0000 7faa3d0f8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe65850 0x55c7dfe79d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50392/0 (socket says 172.21.15.44:50392) 2024-04-21T23:10:34.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.005+0000 7faa3d0f8640 1 -- 172.21.15.44:0/3729926687 learned_addr learned my addr 172.21.15.44:0/3729926687 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:34.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.005+0000 7faa3d8f9640 1 --2- 172.21.15.44:0/3729926687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe64f30 0x55c7dfe6ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:34.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.006+0000 7faa3e0fa640 1 --2- 172.21.15.44:0/3729926687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe6e430 0x55c7dfe861e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:34.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.006+0000 7faa3d0f8640 1 -- 172.21.15.44:0/3729926687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe64f30 msgr2=0x55c7dfe6ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:34.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.006+0000 7faa3d0f8640 1 --2- 172.21.15.44:0/3729926687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe64f30 0x55c7dfe6ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:34.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.006+0000 7faa3d0f8640 1 -- 172.21.15.44:0/3729926687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe6e430 msgr2=0x55c7dfe861e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:34.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.006+0000 7faa3d0f8640 1 --2- 172.21.15.44:0/3729926687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe6e430 0x55c7dfe861e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:34.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.006+0000 7faa3d0f8640 1 -- 172.21.15.44:0/3729926687 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7dfe65680 con 0x55c7dfe65850 2024-04-21T23:10:34.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.006+0000 7faa3d0f8640 1 --2- 172.21.15.44:0/3729926687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe65850 0x55c7dfe79d30 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7faa28012a80 tx=0x7faa28005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a748fa0bb30ee29b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:34.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.006+0000 7faa237fe640 1 -- 172.21.15.44:0/3729926687 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa28012190 con 0x55c7dfe65850 2024-04-21T23:10:34.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.006+0000 7faa237fe640 1 -- 172.21.15.44:0/3729926687 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa28005020 con 0x55c7dfe65850 2024-04-21T23:10:34.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.006+0000 7faa237fe640 1 -- 172.21.15.44:0/3729926687 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa28004ae0 con 0x55c7dfe65850 2024-04-21T23:10:34.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.007+0000 7faa4281e740 1 -- 172.21.15.44:0/3729926687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe65850 msgr2=0x55c7dfe79d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:34.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.007+0000 7faa4281e740 1 --2- 172.21.15.44:0/3729926687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe65850 0x55c7dfe79d30 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7faa28012a80 tx=0x7faa28005f90 comp rx=0 tx=0).stop 2024-04-21T23:10:34.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.007+0000 7faa4281e740 1 -- 172.21.15.44:0/3729926687 shutdown_connections 2024-04-21T23:10:34.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.007+0000 7faa4281e740 1 --2- 172.21.15.44:0/3729926687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe6e430 0x55c7dfe861e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:34.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.007+0000 7faa4281e740 1 --2- 172.21.15.44:0/3729926687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe65850 0x55c7dfe79d30 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:34.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.007+0000 7faa4281e740 1 --2- 172.21.15.44:0/3729926687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe64f30 0x55c7dfe6ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:34.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.007+0000 7faa4281e740 1 -- 172.21.15.44:0/3729926687 >> 172.21.15.44:0/3729926687 conn(0x55c7dfe5f660 msgr2=0x55c7dfe61340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:34.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.007+0000 7faa4281e740 1 -- 172.21.15.44:0/3729926687 shutdown_connections 2024-04-21T23:10:34.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.007+0000 7faa4281e740 1 -- 172.21.15.44:0/3729926687 wait complete. 2024-04-21T23:10:34.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.007+0000 7faa4281e740 1 -- start start 2024-04-21T23:10:34.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.007+0000 7faa4281e740 1 Processor -- start 2024-04-21T23:10:34.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.008+0000 7faa4281e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe64f30 0x55c7dfedcd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:34.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.008+0000 7faa4281e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe65850 0x55c7dfedd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:34.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.008+0000 7faa4281e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe6e430 0x55c7dfe713e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.008+0000 7faa4281e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faa381ab720 con 0x55c7dfe65850 2024-04-21T23:10:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.008+0000 7faa4281e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7faa381acb90 con 0x55c7dfe64f30 2024-04-21T23:10:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.008+0000 7faa4281e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faa381abdf0 con 0x55c7dfe6e430 2024-04-21T23:10:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.008+0000 7faa4281e740 10 client.0 _ll_register_callbacks cb 0x7faa34004360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.008+0000 7faa3d0f8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe65850 0x55c7dfedd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.008+0000 7faa3d0f8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe65850 0x55c7dfedd290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50406/0 (socket says 172.21.15.44:50406) 2024-04-21T23:10:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.008+0000 7faa3d0f8640 1 -- 172.21.15.44:0/1632963790 learned_addr learned my addr 172.21.15.44:0/1632963790 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.008+0000 7faa3d8f9640 1 --2- 172.21.15.44:0/1632963790 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe64f30 0x55c7dfedcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.008+0000 7faa3e0fa640 1 --2- 172.21.15.44:0/1632963790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe6e430 0x55c7dfe713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.008+0000 7faa3d0f8640 1 -- 172.21.15.44:0/1632963790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe6e430 msgr2=0x55c7dfe713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.008+0000 7faa3d0f8640 1 --2- 172.21.15.44:0/1632963790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe6e430 0x55c7dfe713e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.008+0000 7faa3d0f8640 1 -- 172.21.15.44:0/1632963790 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe64f30 msgr2=0x55c7dfedcd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:34.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.008+0000 7faa3d0f8640 1 --2- 172.21.15.44:0/1632963790 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe64f30 0x55c7dfedcd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.008+0000 7faa3d0f8640 1 -- 172.21.15.44:0/1632963790 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7dfe71aa0 con 0x55c7dfe65850 2024-04-21T23:10:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.008+0000 7faa3d0f8640 1 --2- 172.21.15.44:0/1632963790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe65850 0x55c7dfedd290 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7faa2800f790 tx=0x7faa28005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8f27b22ab1d9e15 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.008+0000 7faa217fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:10:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.009+0000 7faa217fa640 1 -- 172.21.15.44:0/1632963790 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2802a070 con 0x55c7dfe65850 2024-04-21T23:10:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.009+0000 7faa217fa640 1 -- 172.21.15.44:0/1632963790 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7faa28004880 con 0x55c7dfe65850 2024-04-21T23:10:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.009+0000 7faa4281e740 1 -- 172.21.15.44:0/1632963790 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa28005020 con 0x55c7dfe65850 2024-04-21T23:10:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.009+0000 7faa217fa640 1 -- 172.21.15.44:0/1632963790 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7faa28005520 con 0x55c7dfe65850 2024-04-21T23:10:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.009+0000 7faa4281e740 10 client.16071 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.009+0000 7faa4281e740 1 -- 172.21.15.44:0/1632963790 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c7dfee4490 con 0x55c7dfe65850 2024-04-21T23:10:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.009+0000 7faa217fa640 1 -- 172.21.15.44:0/1632963790 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa28005020 con 0x55c7dfe65850 2024-04-21T23:10:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.009+0000 7faa217fa640 1 -- 172.21.15.44:0/1632963790 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7faa2802fa60 con 0x55c7dfe65850 2024-04-21T23:10:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.009+0000 7faa4281e740 10 client.16071 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.009+0000 7faa4281e740 10 client.16071 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.009+0000 7faa4281e740 20 client.16071 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:34.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.009+0000 7faa4281e740 1 --2- 172.21.15.44:0/1632963790 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55c7dfee4c10 0x55c7dfee7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.009+0000 7faa4281e740 4 client.16071 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:10:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.009+0000 7faa4281e740 1 -- 172.21.15.44:0/1632963790 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c7dfdd0df0 con 0x55c7dfee4c10 2024-04-21T23:10:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.009+0000 7faa3d8f9640 1 --2- 172.21.15.44:0/1632963790 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55c7dfee4c10 0x55c7dfee7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.010+0000 7faa3d8f9640 1 --2- 172.21.15.44:0/1632963790 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55c7dfee4c10 0x55c7dfee7030 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94869af15034c234 server_cookie=84d2c95cc1fade06 in_seq=0 out_seq=0 2024-04-21T23:10:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.010+0000 7faa217fa640 10 client.16071 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.011+0000 7faa217fa640 1 -- 172.21.15.44:0/1632963790 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c7dfdd0df0 con 0x55c7dfee4c10 2024-04-21T23:10:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.011+0000 7faa217fa640 10 client.16071 handle_command_reply: tid=0 2024-04-21T23:10:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.011+0000 7faa4281e740 1 client.16071 shutdown 2024-04-21T23:10:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.011+0000 7faa4281e740 1 -- 172.21.15.44:0/1632963790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe65850 msgr2=0x55c7dfedd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.011+0000 7faa4281e740 1 --2- 172.21.15.44:0/1632963790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe65850 0x55c7dfedd290 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7faa2800f790 tx=0x7faa28005730 comp rx=0 tx=0).stop 2024-04-21T23:10:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.011+0000 7faa4281e740 1 -- 172.21.15.44:0/1632963790 shutdown_connections 2024-04-21T23:10:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.011+0000 7faa4281e740 1 --2- 172.21.15.44:0/1632963790 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55c7dfee4c10 0x55c7dfee7030 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.011+0000 7faa4281e740 1 --2- 172.21.15.44:0/1632963790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7dfe6e430 0x55c7dfe713e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:34.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.011+0000 7faa4281e740 1 --2- 172.21.15.44:0/1632963790 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55c7dfe65850 0x55c7dfedd290 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.011+0000 7faa217fa640 0 client.16071 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.011+0000 7faa4281e740 1 --2- 172.21.15.44:0/1632963790 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55c7dfe64f30 0x55c7dfedcd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.011+0000 7faa4281e740 1 -- 172.21.15.44:0/1632963790 >> 172.21.15.44:0/1632963790 conn(0x55c7dfe5f660 msgr2=0x55c7dfe5de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.011+0000 7faa4281e740 1 -- 172.21.15.44:0/1632963790 shutdown_connections 2024-04-21T23:10:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.011+0000 7faa4281e740 1 -- 172.21.15.44:0/1632963790 wait complete. 2024-04-21T23:10:34.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.011+0000 7faa4281e740 20 client.16071 trim_cache size 0 max 16384 2024-04-21T23:10:34.016 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:10:34.016 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:10:34.016 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-21T23:10:34.016 INFO:teuthology.orchestra.run.smithi044.stdout: "a33bfafe-00bf-43aa-8883-4d78ececd8c2": { 2024-04-21T23:10:34.016 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-21T23:10:34.017 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "a33bfafe-00bf-43aa-8883-4d78ececd8c2", 2024-04-21T23:10:34.017 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-04-21T23:10:34.017 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:10:34.017 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:10:34.017 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:10:34.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.011+0000 7faa3fb5d640 1 -- 172.21.15.44:0/1805993255 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faa240a2d00 msgr2=0x7faa240a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.011+0000 7faa3fb5d640 1 --2- 172.21.15.44:0/1805993255 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faa240a2d00 0x7faa240a51c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faa34009f70 tx=0x7faa34011810 comp rx=0 tx=0).stop 2024-04-21T23:10:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.011+0000 7faa3fb5d640 1 -- 172.21.15.44:0/1805993255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa38100610 msgr2=0x7faa38198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.011+0000 7faa3fb5d640 1 --2- 172.21.15.44:0/1805993255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa38100610 0x7faa38198050 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7faa3002c630 tx=0x7faa30032c90 comp rx=0 tx=0).stop 2024-04-21T23:10:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.011+0000 7faa3fb5d640 1 -- 172.21.15.44:0/1805993255 shutdown_connections 2024-04-21T23:10:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.012+0000 7faa3fb5d640 1 --2- 172.21.15.44:0/1805993255 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faa240a2d00 0x7faa240a51c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.012+0000 7faa3fb5d640 1 --2- 172.21.15.44:0/1805993255 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faa381066b0 0x7faa3819c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.012+0000 7faa3fb5d640 1 --2- 172.21.15.44:0/1805993255 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faa38100fd0 0x7faa38198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.012+0000 7faa3fb5d640 1 --2- 172.21.15.44:0/1805993255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa38100610 0x7faa38198050 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.012+0000 7faa3fb5d640 1 -- 172.21.15.44:0/1805993255 >> 172.21.15.44:0/1805993255 conn(0x7faa380777f0 msgr2=0x7faa380ff2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.012+0000 7faa3fb5d640 1 -- 172.21.15.44:0/1805993255 shutdown_connections 2024-04-21T23:10:34.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:10:34.012+0000 7faa3fb5d640 1 -- 172.21.15.44:0/1805993255 wait complete. 2024-04-21T23:10:34.022 INFO:tasks.cephfs.filesystem:scrub status for tag:a33bfafe-00bf-43aa-8883-4d78ececd8c2 - {'path': '/', 'tag': 'a33bfafe-00bf-43aa-8883-4d78ececd8c2', 'options': 'recursive,force'} 2024-04-21T23:10:34.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:34 smithi044 ceph-mon[27934]: pgmap v54: 129 pgs: 129 active+clean; 14 MiB data, 450 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 495 op/s 2024-04-21T23:10:34.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:34 smithi055 ceph-mon[31117]: pgmap v54: 129 pgs: 129 active+clean; 14 MiB data, 450 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 495 op/s 2024-04-21T23:10:34.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:34 smithi069 ceph-mon[31816]: pgmap v54: 129 pgs: 129 active+clean; 14 MiB data, 450 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 495 op/s 2024-04-21T23:10:34.692 INFO:tasks.workunit.client.0.smithi044.stdout: 1 69888 16.33 MB/sec warmup 24 sec latency 18.506 ms 2024-04-21T23:10:35.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:35 smithi044 ceph-mon[27934]: scrub queued for path: / 2024-04-21T23:10:35.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:35 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:10:35.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:35 smithi044 ceph-mon[27934]: scrub summary: active paths [/] 2024-04-21T23:10:35.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:35 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:10:35.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:35 smithi055 ceph-mon[31117]: scrub queued for path: / 2024-04-21T23:10:35.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:35 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:10:35.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:35 smithi055 ceph-mon[31117]: scrub summary: active paths [/] 2024-04-21T23:10:35.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:35 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:10:35.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:35 smithi069 ceph-mon[31816]: scrub queued for path: / 2024-04-21T23:10:35.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:35 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:10:35.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:35 smithi069 ceph-mon[31816]: scrub summary: active paths [/] 2024-04-21T23:10:35.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:35 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:10:35.692 INFO:tasks.workunit.client.0.smithi044.stdout: 1 72582 16.34 MB/sec warmup 25 sec latency 18.573 ms 2024-04-21T23:10:36.438 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:36 smithi044 ceph-mon[27934]: pgmap v55: 129 pgs: 129 active+clean; 14 MiB data, 450 MiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 1.5 MiB/s wr, 385 op/s 2024-04-21T23:10:36.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:36 smithi069 ceph-mon[31816]: pgmap v55: 129 pgs: 129 active+clean; 14 MiB data, 450 MiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 1.5 MiB/s wr, 385 op/s 2024-04-21T23:10:36.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:36 smithi055 ceph-mon[31117]: pgmap v55: 129 pgs: 129 active+clean; 14 MiB data, 450 MiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 1.5 MiB/s wr, 385 op/s 2024-04-21T23:10:36.692 INFO:tasks.workunit.client.0.smithi044.stdout: 1 75148 16.25 MB/sec warmup 26 sec latency 29.546 ms 2024-04-21T23:10:37.692 INFO:tasks.workunit.client.0.smithi044.stdout: 1 75281 15.66 MB/sec warmup 27 sec latency 486.746 ms 2024-04-21T23:10:38.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:38 smithi044 ceph-mon[27934]: pgmap v56: 129 pgs: 129 active+clean; 21 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 2.1 MiB/s wr, 424 op/s 2024-04-21T23:10:38.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:38 smithi044 ceph-mon[27934]: scrub summary: idle 2024-04-21T23:10:38.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:38 smithi069 ceph-mon[31816]: pgmap v56: 129 pgs: 129 active+clean; 21 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 2.1 MiB/s wr, 424 op/s 2024-04-21T23:10:38.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:38 smithi069 ceph-mon[31816]: scrub summary: idle 2024-04-21T23:10:38.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:38 smithi055 ceph-mon[31117]: pgmap v56: 129 pgs: 129 active+clean; 21 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 2.1 MiB/s wr, 424 op/s 2024-04-21T23:10:38.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:38 smithi055 ceph-mon[31117]: scrub summary: idle 2024-04-21T23:10:38.692 INFO:tasks.workunit.client.0.smithi044.stdout: 1 78070 15.71 MB/sec warmup 28 sec latency 19.321 ms 2024-04-21T23:10:39.692 INFO:tasks.workunit.client.0.smithi044.stdout: 1 80644 15.51 MB/sec warmup 29 sec latency 26.523 ms 2024-04-21T23:10:40.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:40 smithi044 ceph-mon[27934]: pgmap v57: 129 pgs: 129 active+clean; 34 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 2.6 MiB/s wr, 404 op/s 2024-04-21T23:10:40.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:40 smithi055 ceph-mon[31117]: pgmap v57: 129 pgs: 129 active+clean; 34 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 2.6 MiB/s wr, 404 op/s 2024-04-21T23:10:40.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:40 smithi069 ceph-mon[31816]: pgmap v57: 129 pgs: 129 active+clean; 34 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 2.6 MiB/s wr, 404 op/s 2024-04-21T23:10:40.692 INFO:tasks.workunit.client.0.smithi044.stdout: 1 83045 15.53 MB/sec warmup 30 sec latency 27.498 ms 2024-04-21T23:10:41.692 INFO:tasks.workunit.client.0.smithi044.stdout: 1 85796 15.48 MB/sec warmup 31 sec latency 22.889 ms 2024-04-21T23:10:42.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:42 smithi044 ceph-mon[27934]: pgmap v58: 129 pgs: 129 active+clean; 34 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 2.6 MiB/s wr, 404 op/s 2024-04-21T23:10:42.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:42 smithi069 ceph-mon[31816]: pgmap v58: 129 pgs: 129 active+clean; 34 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 2.6 MiB/s wr, 404 op/s 2024-04-21T23:10:42.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:42 smithi055 ceph-mon[31117]: pgmap v58: 129 pgs: 129 active+clean; 34 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 2.6 MiB/s wr, 404 op/s 2024-04-21T23:10:42.692 INFO:tasks.workunit.client.0.smithi044.stdout: 1 88934 15.59 MB/sec warmup 32 sec latency 21.126 ms 2024-04-21T23:10:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.992+0000 7f6e35096640 1 -- 172.21.15.44:0/418602279 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e30100fe0 msgr2=0x7f6e3010d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.992+0000 7f6e35096640 1 --2- 172.21.15.44:0/418602279 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e30100fe0 0x7f6e3010d4f0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f6e28009cd0 tx=0x7f6e2802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.992+0000 7f6e35096640 1 -- 172.21.15.44:0/418602279 shutdown_connections 2024-04-21T23:10:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.992+0000 7f6e35096640 1 --2- 172.21.15.44:0/418602279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e30100620 0x7f6e30100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.992+0000 7f6e35096640 1 --2- 172.21.15.44:0/418602279 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e30100fe0 0x7f6e3010d4f0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.992+0000 7f6e35096640 1 --2- 172.21.15.44:0/418602279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e30106630 0x7f6e30106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.992+0000 7f6e35096640 1 -- 172.21.15.44:0/418602279 >> 172.21.15.44:0/418602279 conn(0x7f6e30077760 msgr2=0x7f6e30077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:43.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.992+0000 7f6e35096640 1 -- 172.21.15.44:0/418602279 shutdown_connections 2024-04-21T23:10:43.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.992+0000 7f6e35096640 1 -- 172.21.15.44:0/418602279 wait complete. 2024-04-21T23:10:43.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.993+0000 7f6e35096640 1 Processor -- start 2024-04-21T23:10:43.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.993+0000 7f6e35096640 1 -- start start 2024-04-21T23:10:43.011 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.993+0000 7f6e35096640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e30100620 0x7f6e30197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:43.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.993+0000 7f6e35096640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e30100fe0 0x7f6e30198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:43.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.993+0000 7f6e35096640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e30106630 0x7f6e3019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:43.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.993+0000 7f6e35096640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6e30110a40 con 0x7f6e30100620 2024-04-21T23:10:43.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.993+0000 7f6e35096640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6e301108c0 con 0x7f6e30100fe0 2024-04-21T23:10:43.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.993+0000 7f6e35096640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6e30110bc0 con 0x7f6e30106630 2024-04-21T23:10:43.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.993+0000 7f6e2ed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e30100620 0x7f6e30197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:43.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.993+0000 7f6e2ed9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e30100620 0x7f6e30197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37426/0 (socket says 172.21.15.44:37426) 2024-04-21T23:10:43.012 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.993+0000 7f6e2ed9d640 1 -- 172.21.15.44:0/728677653 learned_addr learned my addr 172.21.15.44:0/728677653 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.993+0000 7f6e27fff640 1 --2- 172.21.15.44:0/728677653 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e30100fe0 0x7f6e30198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.993+0000 7f6e2ed9d640 1 -- 172.21.15.44:0/728677653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e30106630 msgr2=0x7f6e3019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.993+0000 7f6e2ed9d640 1 --2- 172.21.15.44:0/728677653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e30106630 0x7f6e3019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.993+0000 7f6e2ed9d640 1 -- 172.21.15.44:0/728677653 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e30100fe0 msgr2=0x7f6e30198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.993+0000 7f6e2ed9d640 1 --2- 172.21.15.44:0/728677653 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e30100fe0 0x7f6e30198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.993+0000 7f6e2ed9d640 1 -- 172.21.15.44:0/728677653 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e3019cfb0 con 0x7f6e30100620 2024-04-21T23:10:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.993+0000 7f6e2ed9d640 1 --2- 172.21.15.44:0/728677653 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e30100620 0x7f6e30197fc0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f6e2000e9f0 tx=0x7f6e2000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.993+0000 7f6e2cd99640 1 -- 172.21.15.44:0/728677653 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e2000cd80 con 0x7f6e30100620 2024-04-21T23:10:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.994+0000 7f6e35096640 1 -- 172.21.15.44:0/728677653 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e3019d2a0 con 0x7f6e30100620 2024-04-21T23:10:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.994+0000 7f6e2cd99640 1 -- 172.21.15.44:0/728677653 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e20010910 con 0x7f6e30100620 2024-04-21T23:10:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.994+0000 7f6e2cd99640 1 -- 172.21.15.44:0/728677653 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e200188f0 con 0x7f6e30100620 2024-04-21T23:10:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.994+0000 7f6e35096640 1 -- 172.21.15.44:0/728677653 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e301a4b30 con 0x7f6e30100620 2024-04-21T23:10:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.996+0000 7f6e2cd99640 1 -- 172.21.15.44:0/728677653 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6e20010ac0 con 0x7f6e30100620 2024-04-21T23:10:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.996+0000 7f6e37922740 1 Processor -- start 2024-04-21T23:10:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.996+0000 7f6e37922740 1 -- start start 2024-04-21T23:10:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e37922740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a7f8b20 0x56270a7f8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e37922740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f9440 0x56270a801cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e37922740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a802210 0x56270a804600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e37922740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56270a75cb70 con 0x56270a7f8b20 2024-04-21T23:10:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e37922740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x56270a705470 con 0x56270a802210 2024-04-21T23:10:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e37922740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56270a758b30 con 0x56270a7f9440 2024-04-21T23:10:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e2f59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a7f8b20 0x56270a7f8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e2f59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a7f8b20 0x56270a7f8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37436/0 (socket says 172.21.15.44:37436) 2024-04-21T23:10:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e2f59e640 1 -- 172.21.15.44:0/632551159 learned_addr learned my addr 172.21.15.44:0/632551159 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e27fff640 1 --2- 172.21.15.44:0/632551159 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a802210 0x56270a804600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e2cd99640 1 --2- 172.21.15.44:0/728677653 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6e1c0a2cf0 0x7f6e1c0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e2ed9d640 1 --2- 172.21.15.44:0/632551159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f9440 0x56270a801cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e2f59e640 1 -- 172.21.15.44:0/632551159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f9440 msgr2=0x56270a801cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e2f59e640 1 --2- 172.21.15.44:0/632551159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f9440 0x56270a801cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e2f59e640 1 -- 172.21.15.44:0/632551159 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a802210 msgr2=0x56270a804600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e2f59e640 1 --2- 172.21.15.44:0/632551159 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a802210 0x56270a804600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e2f59e640 1 -- 172.21.15.44:0/632551159 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56270a471210 con 0x56270a7f8b20 2024-04-21T23:10:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e2ed9d640 1 --2- 172.21.15.44:0/632551159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f9440 0x56270a801cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:10:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e2cd99640 1 -- 172.21.15.44:0/728677653 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f6e200cfbe0 con 0x7f6e30100620 2024-04-21T23:10:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e2f59e640 1 --2- 172.21.15.44:0/728677653 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6e1c0a2cf0 0x7f6e1c0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.997+0000 7f6e2f59e640 1 --2- 172.21.15.44:0/632551159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a7f8b20 0x56270a7f8f00 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f6e28037040 tx=0x7f6e280324c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26632f7ed9fa20bd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.998+0000 7f6e257fa640 1 -- 172.21.15.44:0/632551159 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e28043040 con 0x56270a7f8b20 2024-04-21T23:10:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.998+0000 7f6e257fa640 1 -- 172.21.15.44:0/632551159 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e28003b70 con 0x56270a7f8b20 2024-04-21T23:10:43.017 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.998+0000 7f6e257fa640 1 -- 172.21.15.44:0/632551159 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e28039040 con 0x56270a7f8b20 2024-04-21T23:10:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.998+0000 7f6e37922740 1 -- 172.21.15.44:0/632551159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a7f8b20 msgr2=0x56270a7f8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.998+0000 7f6e2f59e640 1 --2- 172.21.15.44:0/728677653 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6e1c0a2cf0 0x7f6e1c0a51b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6e28049000 tx=0x7f6e2804a000 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.998+0000 7f6e37922740 1 --2- 172.21.15.44:0/632551159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a7f8b20 0x56270a7f8f00 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f6e28037040 tx=0x7f6e280324c0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.998+0000 7f6e37922740 1 -- 172.21.15.44:0/632551159 shutdown_connections 2024-04-21T23:10:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.998+0000 7f6e37922740 1 --2- 172.21.15.44:0/632551159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f9440 0x56270a801cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.998+0000 7f6e37922740 1 --2- 172.21.15.44:0/632551159 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a802210 0x56270a804600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.998+0000 7f6e37922740 1 --2- 172.21.15.44:0/632551159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a7f8b20 0x56270a7f8f00 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.998+0000 7f6e37922740 1 -- 172.21.15.44:0/632551159 >> 172.21.15.44:0/632551159 conn(0x56270a7f3440 msgr2=0x56270a7f3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.998+0000 7f6e37922740 1 -- 172.21.15.44:0/632551159 shutdown_connections 2024-04-21T23:10:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.999+0000 7f6e37922740 1 -- 172.21.15.44:0/632551159 wait complete. 2024-04-21T23:10:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.999+0000 7f6e37922740 1 -- start start 2024-04-21T23:10:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.999+0000 7f6e37922740 1 Processor -- start 2024-04-21T23:10:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.999+0000 7f6e37922740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f8b20 0x56270a7ff200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.999+0000 7f6e37922740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a7f9440 0x56270a7ff740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.999+0000 7f6e37922740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a802210 0x56270a7ffc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.999+0000 7f6e37922740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56270a804e40 con 0x56270a802210 2024-04-21T23:10:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.999+0000 7f6e37922740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x56270a718a10 con 0x56270a7f9440 2024-04-21T23:10:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.999+0000 7f6e37922740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56270a804fc0 con 0x56270a7f8b20 2024-04-21T23:10:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.999+0000 7f6e37922740 10 client.0 _ll_register_callbacks cb 0x56270a753b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:43.019 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.999+0000 7f6e2f59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a802210 0x56270a7ffc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.999+0000 7f6e2f59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a802210 0x56270a7ffc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37450/0 (socket says 172.21.15.44:37450) 2024-04-21T23:10:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.999+0000 7f6e2f59e640 1 -- 172.21.15.44:0/3843622171 learned_addr learned my addr 172.21.15.44:0/3843622171 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.999+0000 7f6e27fff640 1 --2- 172.21.15.44:0/3843622171 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a7f9440 0x56270a7ff740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:42.999+0000 7f6e2ed9d640 1 --2- 172.21.15.44:0/3843622171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f8b20 0x56270a7ff200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.000+0000 7f6e2f59e640 1 -- 172.21.15.44:0/3843622171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f8b20 msgr2=0x56270a7ff200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.000+0000 7f6e2f59e640 1 --2- 172.21.15.44:0/3843622171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f8b20 0x56270a7ff200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.000+0000 7f6e2f59e640 1 -- 172.21.15.44:0/3843622171 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a7f9440 msgr2=0x56270a7ff740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:43.020 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.000+0000 7f6e2f59e640 1 --2- 172.21.15.44:0/3843622171 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a7f9440 0x56270a7ff740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.000+0000 7f6e2f59e640 1 -- 172.21.15.44:0/3843622171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56270a53d620 con 0x56270a802210 2024-04-21T23:10:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.000+0000 7f6e2f59e640 1 --2- 172.21.15.44:0/3843622171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a802210 0x56270a7ffc80 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f6e28004710 tx=0x7f6e28004560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb546f3c7b577e0b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.000+0000 7f6e167fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:10:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.000+0000 7f6e167fc640 1 -- 172.21.15.44:0/3843622171 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e28057060 con 0x56270a802210 2024-04-21T23:10:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.000+0000 7f6e167fc640 1 -- 172.21.15.44:0/3843622171 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f6e28003970 con 0x56270a802210 2024-04-21T23:10:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.000+0000 7f6e37922740 1 -- 172.21.15.44:0/3843622171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e28003b70 con 0x56270a802210 2024-04-21T23:10:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.000+0000 7f6e167fc640 1 -- 172.21.15.44:0/3843622171 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6e2805e050 con 0x56270a802210 2024-04-21T23:10:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.000+0000 7f6e37922740 10 client.16083 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.000+0000 7f6e37922740 1 -- 172.21.15.44:0/3843622171 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56270a811000 con 0x56270a802210 2024-04-21T23:10:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.000+0000 7f6e167fc640 1 -- 172.21.15.44:0/3843622171 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e28003b70 con 0x56270a802210 2024-04-21T23:10:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.000+0000 7f6e167fc640 1 -- 172.21.15.44:0/3843622171 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f6e28047040 con 0x56270a802210 2024-04-21T23:10:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.001+0000 7f6e37922740 10 client.16083 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:43.021 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.001+0000 7f6e37922740 10 client.16083 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.001+0000 7f6e37922740 20 client.16083 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.001+0000 7f6e37922740 1 --2- 172.21.15.44:0/3843622171 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56270a812350 0x56270a814740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.001+0000 7f6e37922740 4 client.16083 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:10:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.001+0000 7f6e37922740 1 -- 172.21.15.44:0/3843622171 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56270a76a0b0 con 0x56270a812350 2024-04-21T23:10:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.001+0000 7f6e2ed9d640 1 --2- 172.21.15.44:0/3843622171 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56270a812350 0x56270a814740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.001+0000 7f6e2ed9d640 1 --2- 172.21.15.44:0/3843622171 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56270a812350 0x56270a814740 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43afb4adfa409053 server_cookie=332174d76676c48a in_seq=0 out_seq=0 2024-04-21T23:10:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.001+0000 7f6e167fc640 10 client.16083 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.003+0000 7f6e167fc640 1 -- 172.21.15.44:0/3843622171 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56270a76a0b0 con 0x56270a812350 2024-04-21T23:10:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.003+0000 7f6e167fc640 10 client.16083 handle_command_reply: tid=0 2024-04-21T23:10:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.003+0000 7f6e37922740 1 client.16083 shutdown 2024-04-21T23:10:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.003+0000 7f6e37922740 1 -- 172.21.15.44:0/3843622171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a802210 msgr2=0x56270a7ffc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.003+0000 7f6e37922740 1 --2- 172.21.15.44:0/3843622171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a802210 0x56270a7ffc80 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f6e28004710 tx=0x7f6e28004560 comp rx=0 tx=0).stop 2024-04-21T23:10:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.003+0000 7f6e37922740 1 -- 172.21.15.44:0/3843622171 shutdown_connections 2024-04-21T23:10:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.003+0000 7f6e37922740 1 --2- 172.21.15.44:0/3843622171 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56270a812350 0x56270a814740 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.003+0000 7f6e37922740 1 --2- 172.21.15.44:0/3843622171 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a802210 0x56270a7ffc80 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.003+0000 7f6e37922740 1 --2- 172.21.15.44:0/3843622171 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a7f9440 0x56270a7ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.003+0000 7f6e37922740 1 --2- 172.21.15.44:0/3843622171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f8b20 0x56270a7ff200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.003+0000 7f6e167fc640 0 client.16083 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.003+0000 7f6e37922740 1 -- 172.21.15.44:0/3843622171 >> 172.21.15.44:0/3843622171 conn(0x56270a7f3440 msgr2=0x56270a7f5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.003+0000 7f6e37922740 1 -- 172.21.15.44:0/3843622171 shutdown_connections 2024-04-21T23:10:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.003+0000 7f6e37922740 1 -- 172.21.15.44:0/3843622171 wait complete. 2024-04-21T23:10:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.003+0000 7f6e37922740 20 client.16083 trim_cache size 0 max 16384 2024-04-21T23:10:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.019+0000 7f6e37922740 1 Processor -- start 2024-04-21T23:10:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.019+0000 7f6e37922740 1 -- start start 2024-04-21T23:10:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.019+0000 7f6e37922740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f8b20 0x56270a80dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.019+0000 7f6e37922740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a7f9440 0x56270a7ff8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.019+0000 7f6e37922740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a802210 0x56270a819da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.019+0000 7f6e37922740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6e301acca0 con 0x56270a7f9440 2024-04-21T23:10:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.019+0000 7f6e37922740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6e301af520 con 0x56270a802210 2024-04-21T23:10:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.019+0000 7f6e37922740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6e301ad310 con 0x56270a7f8b20 2024-04-21T23:10:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.019+0000 7f6e27fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a7f9440 0x56270a7ff8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.019+0000 7f6e2ed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f8b20 0x56270a80dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.019+0000 7f6e27fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a7f9440 0x56270a7ff8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37464/0 (socket says 172.21.15.44:37464) 2024-04-21T23:10:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.019+0000 7f6e27fff640 1 -- 172.21.15.44:0/1390736467 learned_addr learned my addr 172.21.15.44:0/1390736467 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.020+0000 7f6e2f59e640 1 --2- 172.21.15.44:0/1390736467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a802210 0x56270a819da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.020+0000 7f6e27fff640 1 -- 172.21.15.44:0/1390736467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f8b20 msgr2=0x56270a80dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.020+0000 7f6e27fff640 1 --2- 172.21.15.44:0/1390736467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f8b20 0x56270a80dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.020+0000 7f6e27fff640 1 -- 172.21.15.44:0/1390736467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a802210 msgr2=0x56270a819da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.020+0000 7f6e27fff640 1 --2- 172.21.15.44:0/1390736467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a802210 0x56270a819da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.020+0000 7f6e27fff640 1 -- 172.21.15.44:0/1390736467 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56270a7f9270 con 0x56270a7f9440 2024-04-21T23:10:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.020+0000 7f6e27fff640 1 --2- 172.21.15.44:0/1390736467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a7f9440 0x56270a7ff8d0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f6e18005fd0 tx=0x7f6e1800bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=630feac4c7be1828 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.020+0000 7f6e24ff9640 1 -- 172.21.15.44:0/1390736467 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e18003b50 con 0x56270a7f9440 2024-04-21T23:10:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.020+0000 7f6e24ff9640 1 -- 172.21.15.44:0/1390736467 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e18005500 con 0x56270a7f9440 2024-04-21T23:10:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.020+0000 7f6e24ff9640 1 -- 172.21.15.44:0/1390736467 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e18004dd0 con 0x56270a7f9440 2024-04-21T23:10:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.021+0000 7f6e37922740 1 -- 172.21.15.44:0/1390736467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a7f9440 msgr2=0x56270a7ff8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.021+0000 7f6e37922740 1 --2- 172.21.15.44:0/1390736467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a7f9440 0x56270a7ff8d0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f6e18005fd0 tx=0x7f6e1800bc90 comp rx=0 tx=0).stop 2024-04-21T23:10:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.021+0000 7f6e37922740 1 -- 172.21.15.44:0/1390736467 shutdown_connections 2024-04-21T23:10:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.021+0000 7f6e37922740 1 --2- 172.21.15.44:0/1390736467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a802210 0x56270a819da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.021+0000 7f6e37922740 1 --2- 172.21.15.44:0/1390736467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a7f9440 0x56270a7ff8d0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.021+0000 7f6e37922740 1 --2- 172.21.15.44:0/1390736467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f8b20 0x56270a80dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.021+0000 7f6e37922740 1 -- 172.21.15.44:0/1390736467 >> 172.21.15.44:0/1390736467 conn(0x56270a7f3440 msgr2=0x56270a7ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.021+0000 7f6e37922740 1 -- 172.21.15.44:0/1390736467 shutdown_connections 2024-04-21T23:10:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.021+0000 7f6e37922740 1 -- 172.21.15.44:0/1390736467 wait complete. 2024-04-21T23:10:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.021+0000 7f6e37922740 1 -- start start 2024-04-21T23:10:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e37922740 1 Processor -- start 2024-04-21T23:10:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e37922740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a7f8b20 0x56270a80a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e37922740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f9440 0x56270a80a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e37922740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a802210 0x56270a807220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e37922740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6e301ad9e0 con 0x56270a802210 2024-04-21T23:10:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e37922740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6e301aee50 con 0x56270a7f8b20 2024-04-21T23:10:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e37922740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6e301ae0b0 con 0x56270a7f9440 2024-04-21T23:10:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e37922740 10 client.0 _ll_register_callbacks cb 0x7f6e18004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e2f59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a802210 0x56270a807220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e2f59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a802210 0x56270a807220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37466/0 (socket says 172.21.15.44:37466) 2024-04-21T23:10:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e2ed9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a7f8b20 0x56270a80a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e27fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f9440 0x56270a80a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e2f59e640 1 -- 172.21.15.44:0/3130796204 learned_addr learned my addr 172.21.15.44:0/3130796204 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e2f59e640 1 -- 172.21.15.44:0/3130796204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f9440 msgr2=0x56270a80a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e2f59e640 1 --2- 172.21.15.44:0/3130796204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f9440 0x56270a80a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e2f59e640 1 -- 172.21.15.44:0/3130796204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a7f8b20 msgr2=0x56270a80a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e2f59e640 1 --2- 172.21.15.44:0/3130796204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a7f8b20 0x56270a80a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e2f59e640 1 -- 172.21.15.44:0/3130796204 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56270a807a50 con 0x56270a802210 2024-04-21T23:10:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e2ed9d640 1 --2- 172.21.15.44:0/3130796204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a7f8b20 0x56270a80a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e2f59e640 1 --2- 172.21.15.44:0/3130796204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a802210 0x56270a807220 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6e28009620 tx=0x7f6e280324a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c34c2a5e6db5b1fc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.022+0000 7f6e167fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:10:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.023+0000 7f6e167fc640 1 -- 172.21.15.44:0/3130796204 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e28057040 con 0x56270a802210 2024-04-21T23:10:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.023+0000 7f6e37922740 1 -- 172.21.15.44:0/3130796204 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e18005500 con 0x56270a802210 2024-04-21T23:10:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.023+0000 7f6e167fc640 1 -- 172.21.15.44:0/3130796204 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9445+0+0 (secure 0 0 0) 0x7f6e28034540 con 0x56270a802210 2024-04-21T23:10:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.023+0000 7f6e167fc640 1 -- 172.21.15.44:0/3130796204 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6e28039070 con 0x56270a802210 2024-04-21T23:10:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.023+0000 7f6e37922740 10 client.16089 fetch_fsmap learned FSMap version 21 2024-04-21T23:10:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.023+0000 7f6e37922740 1 -- 172.21.15.44:0/3130796204 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56270a879190 con 0x56270a802210 2024-04-21T23:10:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.023+0000 7f6e167fc640 1 -- 172.21.15.44:0/3130796204 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e18005500 con 0x56270a802210 2024-04-21T23:10:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.023+0000 7f6e167fc640 1 -- 172.21.15.44:0/3130796204 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f6e28047070 con 0x56270a802210 2024-04-21T23:10:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.023+0000 7f6e37922740 10 client.16089 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T23:10:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.023+0000 7f6e37922740 10 client.16089 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.023+0000 7f6e37922740 20 client.16089 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.023+0000 7f6e37922740 1 --2- 172.21.15.44:0/3130796204 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56270a879800 0x56270a87bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.023+0000 7f6e37922740 4 client.16089 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:10:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.023+0000 7f6e37922740 1 -- 172.21.15.44:0/3130796204 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56270a764400 con 0x56270a879800 2024-04-21T23:10:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.024+0000 7f6e2ed9d640 1 --2- 172.21.15.44:0/3130796204 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56270a879800 0x56270a87bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.024+0000 7f6e2ed9d640 1 --2- 172.21.15.44:0/3130796204 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56270a879800 0x56270a87bc20 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3340052c935c8f server_cookie=ef437800afecb934 in_seq=0 out_seq=0 2024-04-21T23:10:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.024+0000 7f6e167fc640 10 client.16089 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.165+0000 7f6e167fc640 1 -- 172.21.15.44:0/3130796204 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56270a764400 con 0x56270a879800 2024-04-21T23:10:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.165+0000 7f6e167fc640 10 client.16089 handle_command_reply: tid=0 2024-04-21T23:10:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.165+0000 7f6e37922740 1 client.16089 shutdown 2024-04-21T23:10:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.166+0000 7f6e37922740 1 -- 172.21.15.44:0/3130796204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a802210 msgr2=0x56270a807220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.166+0000 7f6e37922740 1 --2- 172.21.15.44:0/3130796204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a802210 0x56270a807220 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6e28009620 tx=0x7f6e280324a0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.166+0000 7f6e37922740 1 -- 172.21.15.44:0/3130796204 shutdown_connections 2024-04-21T23:10:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.166+0000 7f6e37922740 1 --2- 172.21.15.44:0/3130796204 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56270a879800 0x56270a87bc20 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.166+0000 7f6e167fc640 0 client.16089 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.166+0000 7f6e37922740 1 --2- 172.21.15.44:0/3130796204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56270a7f9440 0x56270a80a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.166+0000 7f6e37922740 1 --2- 172.21.15.44:0/3130796204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56270a802210 0x56270a807220 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.166+0000 7f6e37922740 1 --2- 172.21.15.44:0/3130796204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56270a7f8b20 0x56270a80a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.166+0000 7f6e37922740 1 -- 172.21.15.44:0/3130796204 >> 172.21.15.44:0/3130796204 conn(0x56270a7f3440 msgr2=0x56270a7ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.166+0000 7f6e37922740 1 -- 172.21.15.44:0/3130796204 shutdown_connections 2024-04-21T23:10:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.166+0000 7f6e37922740 1 -- 172.21.15.44:0/3130796204 wait complete. 2024-04-21T23:10:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.166+0000 7f6e37922740 20 client.16089 trim_cache size 0 max 16384 2024-04-21T23:10:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:10:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:10:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:10:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:10:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.167+0000 7f6e35096640 1 -- 172.21.15.44:0/728677653 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6e1c0a2cf0 msgr2=0x7f6e1c0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.167+0000 7f6e35096640 1 --2- 172.21.15.44:0/728677653 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6e1c0a2cf0 0x7f6e1c0a51b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6e28049000 tx=0x7f6e2804a000 comp rx=0 tx=0).stop 2024-04-21T23:10:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.167+0000 7f6e35096640 1 -- 172.21.15.44:0/728677653 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e30100620 msgr2=0x7f6e30197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.167+0000 7f6e35096640 1 --2- 172.21.15.44:0/728677653 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e30100620 0x7f6e30197fc0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f6e2000e9f0 tx=0x7f6e2000eeb0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.167+0000 7f6e35096640 1 -- 172.21.15.44:0/728677653 shutdown_connections 2024-04-21T23:10:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.167+0000 7f6e35096640 1 --2- 172.21.15.44:0/728677653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e30106630 0x7f6e3019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.167+0000 7f6e35096640 1 --2- 172.21.15.44:0/728677653 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6e1c0a2cf0 0x7f6e1c0a51b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.167+0000 7f6e35096640 1 --2- 172.21.15.44:0/728677653 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6e30100fe0 0x7f6e30198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.167+0000 7f6e35096640 1 --2- 172.21.15.44:0/728677653 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6e30100620 0x7f6e30197fc0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.167+0000 7f6e35096640 1 -- 172.21.15.44:0/728677653 >> 172.21.15.44:0/728677653 conn(0x7f6e30077760 msgr2=0x7f6e300691e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:43.169 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.167+0000 7f6e35096640 1 -- 172.21.15.44:0/728677653 shutdown_connections 2024-04-21T23:10:43.169 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:43.167+0000 7f6e35096640 1 -- 172.21.15.44:0/728677653 wait complete. 2024-04-21T23:10:43.692 INFO:tasks.workunit.client.0.smithi044.stdout: 1 91308 15.37 MB/sec warmup 33 sec latency 79.255 ms 2024-04-21T23:10:43.964 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:10:43 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: ignoring --setuser ceph since I am not root 2024-04-21T23:10:43.964 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:10:43 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: ignoring --setgroup ceph since I am not root 2024-04-21T23:10:43.964 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:10:43 smithi069 ceph-mds[63204]: -- 172.21.15.69:0/3392530218 <== mon.1 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a02816fa40 con 0x55a028142c00 2024-04-21T23:10:43.964 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:10:43 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: starting mds.i at 2024-04-21T23:10:44.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:44 smithi044 ceph-mon[27934]: pgmap v59: 129 pgs: 129 active+clean; 35 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 8.6 MiB/s wr, 713 op/s 2024-04-21T23:10:44.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:44 smithi044 ceph-mon[27934]: daemon mds.i restarted 2024-04-21T23:10:44.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:44 smithi069 ceph-mon[31816]: pgmap v59: 129 pgs: 129 active+clean; 35 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 8.6 MiB/s wr, 713 op/s 2024-04-21T23:10:44.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:44 smithi069 ceph-mon[31816]: daemon mds.i restarted 2024-04-21T23:10:44.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:44 smithi055 ceph-mon[31117]: pgmap v59: 129 pgs: 129 active+clean; 35 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 8.6 MiB/s wr, 713 op/s 2024-04-21T23:10:44.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:44 smithi055 ceph-mon[31117]: daemon mds.i restarted 2024-04-21T23:10:44.692 INFO:tasks.workunit.client.0.smithi044.stdout: 1 93520 15.30 MB/sec warmup 34 sec latency 19.108 ms 2024-04-21T23:10:45.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:45 smithi044 ceph-mon[27934]: osdmap e96: 12 total, 12 up, 12 in 2024-04-21T23:10:45.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:45 smithi044 ceph-mon[27934]: mds.? [v2:172.21.15.69:6838/2494217106,v1:172.21.15.69:6839/2494217106] up:boot 2024-04-21T23:10:45.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:45 smithi044 ceph-mon[27934]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:10:45.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:45 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:10:45.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:45 smithi044 ceph-mon[27934]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:10:45.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:45 smithi069 ceph-mon[31816]: osdmap e96: 12 total, 12 up, 12 in 2024-04-21T23:10:45.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:45 smithi069 ceph-mon[31816]: mds.? [v2:172.21.15.69:6838/2494217106,v1:172.21.15.69:6839/2494217106] up:boot 2024-04-21T23:10:45.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:45 smithi069 ceph-mon[31816]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:10:45.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:45 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:10:45.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:45 smithi069 ceph-mon[31816]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:10:45.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:45 smithi055 ceph-mon[31117]: osdmap e96: 12 total, 12 up, 12 in 2024-04-21T23:10:45.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:45 smithi055 ceph-mon[31117]: mds.? [v2:172.21.15.69:6838/2494217106,v1:172.21.15.69:6839/2494217106] up:boot 2024-04-21T23:10:45.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:45 smithi055 ceph-mon[31117]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:10:45.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:45 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:10:45.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:45 smithi055 ceph-mon[31117]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:10:45.692 INFO:tasks.workunit.client.0.smithi044.stdout: 1 96549 15.36 MB/sec warmup 35 sec latency 15.040 ms 2024-04-21T23:10:46.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:46 smithi044 ceph-mon[27934]: pgmap v61: 129 pgs: 129 active+clean; 35 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 9.4 MiB/s wr, 496 op/s 2024-04-21T23:10:46.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:46 smithi069 ceph-mon[31816]: pgmap v61: 129 pgs: 129 active+clean; 35 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 9.4 MiB/s wr, 496 op/s 2024-04-21T23:10:46.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:46 smithi055 ceph-mon[31117]: pgmap v61: 129 pgs: 129 active+clean; 35 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 9.4 MiB/s wr, 496 op/s 2024-04-21T23:10:46.692 INFO:tasks.workunit.client.0.smithi044.stdout: 1 99753 15.46 MB/sec warmup 36 sec latency 9.529 ms 2024-04-21T23:10:47.692 INFO:tasks.workunit.client.0.smithi044.stdout: 1 102561 15.40 MB/sec warmup 37 sec latency 62.006 ms 2024-04-21T23:10:48.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:48 smithi044 ceph-mon[27934]: pgmap v62: 129 pgs: 129 active+clean; 31 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 9.3 MiB/s wr, 584 op/s 2024-04-21T23:10:48.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:48 smithi069 ceph-mon[31816]: pgmap v62: 129 pgs: 129 active+clean; 31 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 9.3 MiB/s wr, 584 op/s 2024-04-21T23:10:48.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:48 smithi055 ceph-mon[31117]: pgmap v62: 129 pgs: 129 active+clean; 31 MiB data, 529 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 9.3 MiB/s wr, 584 op/s 2024-04-21T23:10:48.692 INFO:tasks.workunit.client.0.smithi044.stdout: 1 105245 15.33 MB/sec warmup 38 sec latency 18.234 ms 2024-04-21T23:10:49.692 INFO:tasks.workunit.client.0.smithi044.stdout: 1 107709 15.30 MB/sec warmup 39 sec latency 19.001 ms 2024-04-21T23:10:50.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:50 smithi044 ceph-mon[27934]: pgmap v63: 129 pgs: 129 active+clean; 29 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 8.0 MiB/s wr, 587 op/s 2024-04-21T23:10:50.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:50 smithi055 ceph-mon[31117]: pgmap v63: 129 pgs: 129 active+clean; 29 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 8.0 MiB/s wr, 587 op/s 2024-04-21T23:10:50.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:50 smithi069 ceph-mon[31816]: pgmap v63: 129 pgs: 129 active+clean; 29 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 8.0 MiB/s wr, 587 op/s 2024-04-21T23:10:50.692 INFO:tasks.workunit.client.0.smithi044.stdout: 1 110760 15.35 MB/sec warmup 40 sec latency 13.808 ms 2024-04-21T23:10:51.692 INFO:tasks.workunit.client.0.smithi044.stdout: 1 113904 15.44 MB/sec warmup 41 sec latency 15.468 ms 2024-04-21T23:10:52.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:52 smithi044 ceph-mon[27934]: pgmap v64: 129 pgs: 129 active+clean; 29 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 8.0 MiB/s wr, 587 op/s 2024-04-21T23:10:52.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:52 smithi055 ceph-mon[31117]: pgmap v64: 129 pgs: 129 active+clean; 29 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 8.0 MiB/s wr, 587 op/s 2024-04-21T23:10:52.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:52 smithi069 ceph-mon[31816]: pgmap v64: 129 pgs: 129 active+clean; 29 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 8.0 MiB/s wr, 587 op/s 2024-04-21T23:10:52.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 116892 15.41 MB/sec warmup 42 sec latency 15.415 ms 2024-04-21T23:10:53.692 INFO:tasks.workunit.client.0.smithi044.stdout: 1 119533 15.33 MB/sec warmup 43 sec latency 14.466 ms 2024-04-21T23:10:54.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:54 smithi044 ceph-mon[27934]: pgmap v65: 129 pgs: 129 active+clean; 35 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 324 op/s 2024-04-21T23:10:54.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:54 smithi055 ceph-mon[31117]: pgmap v65: 129 pgs: 129 active+clean; 35 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 324 op/s 2024-04-21T23:10:54.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:54 smithi069 ceph-mon[31816]: pgmap v65: 129 pgs: 129 active+clean; 35 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 324 op/s 2024-04-21T23:10:54.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 121865 15.30 MB/sec warmup 44 sec latency 27.439 ms 2024-04-21T23:10:55.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 124861 15.34 MB/sec warmup 45 sec latency 15.115 ms 2024-04-21T23:10:56.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.261+0000 7f25d9b7a640 1 -- 172.21.15.44:0/1062780355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25d4100fe0 msgr2=0x7f25d410d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:56.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.261+0000 7f25d9b7a640 1 --2- 172.21.15.44:0/1062780355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25d4100fe0 0x7f25d410d4f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f25cc009cd0 tx=0x7f25cc02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.261+0000 7f25d9b7a640 1 -- 172.21.15.44:0/1062780355 shutdown_connections 2024-04-21T23:10:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.261+0000 7f25d9b7a640 1 --2- 172.21.15.44:0/1062780355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d4100620 0x7f25d4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.261+0000 7f25d9b7a640 1 --2- 172.21.15.44:0/1062780355 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25d4100fe0 0x7f25d410d4f0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.261+0000 7f25d9b7a640 1 --2- 172.21.15.44:0/1062780355 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25d4106630 0x7f25d4106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.261+0000 7f25d9b7a640 1 -- 172.21.15.44:0/1062780355 >> 172.21.15.44:0/1062780355 conn(0x7f25d4077760 msgr2=0x7f25d4077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.261+0000 7f25d9b7a640 1 -- 172.21.15.44:0/1062780355 shutdown_connections 2024-04-21T23:10:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.261+0000 7f25d9b7a640 1 -- 172.21.15.44:0/1062780355 wait complete. 2024-04-21T23:10:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.261+0000 7f25d9b7a640 1 Processor -- start 2024-04-21T23:10:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.261+0000 7f25d9b7a640 1 -- start start 2024-04-21T23:10:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d9b7a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d4100620 0x7f25d4198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d9b7a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25d4100fe0 0x7f25d41985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d9b7a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25d4106630 0x7f25d419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d9b7a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f25d4111b40 con 0x7f25d4106630 2024-04-21T23:10:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d9b7a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f25d41119c0 con 0x7f25d4100fe0 2024-04-21T23:10:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d9b7a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f25d4111cc0 con 0x7f25d4100620 2024-04-21T23:10:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25d4106630 0x7f25d419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25d4106630 0x7f25d419c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36454/0 (socket says 172.21.15.44:36454) 2024-04-21T23:10:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d3fff640 1 -- 172.21.15.44:0/2209105398 learned_addr learned my addr 172.21.15.44:0/2209105398 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d2ffd640 1 --2- 172.21.15.44:0/2209105398 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25d4100fe0 0x7f25d41985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d37fe640 1 --2- 172.21.15.44:0/2209105398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d4100620 0x7f25d4198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d3fff640 1 -- 172.21.15.44:0/2209105398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d4100620 msgr2=0x7f25d4198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d3fff640 1 --2- 172.21.15.44:0/2209105398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d4100620 0x7f25d4198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d3fff640 1 -- 172.21.15.44:0/2209105398 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25d4100fe0 msgr2=0x7f25d41985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d3fff640 1 --2- 172.21.15.44:0/2209105398 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25d4100fe0 0x7f25d41985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d3fff640 1 -- 172.21.15.44:0/2209105398 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25d419d050 con 0x7f25d4106630 2024-04-21T23:10:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d3fff640 1 --2- 172.21.15.44:0/2209105398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25d4106630 0x7f25d419c920 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f25cc005e30 tx=0x7f25cc032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d0ff9640 1 -- 172.21.15.44:0/2209105398 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc041020 con 0x7f25d4106630 2024-04-21T23:10:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d9b7a640 1 -- 172.21.15.44:0/2209105398 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25d419d2e0 con 0x7f25d4106630 2024-04-21T23:10:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d0ff9640 1 -- 172.21.15.44:0/2209105398 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25cc034cd0 con 0x7f25d4106630 2024-04-21T23:10:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.262+0000 7f25d0ff9640 1 -- 172.21.15.44:0/2209105398 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc039040 con 0x7f25d4106630 2024-04-21T23:10:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.263+0000 7f25d9b7a640 1 -- 172.21.15.44:0/2209105398 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25d41a4bd0 con 0x7f25d4106630 2024-04-21T23:10:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.265+0000 7f25d0ff9640 1 -- 172.21.15.44:0/2209105398 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f25cc044020 con 0x7f25d4106630 2024-04-21T23:10:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.265+0000 7f25dc763740 1 Processor -- start 2024-04-21T23:10:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.265+0000 7f25dc763740 1 -- start start 2024-04-21T23:10:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.265+0000 7f25dc763740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b84b20 0x55d609b84f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.265+0000 7f25dc763740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b85440 0x55d609b8dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.265+0000 7f25dc763740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b8e210 0x55d609b90600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.265+0000 7f25dc763740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d609ae8b70 con 0x55d609b84b20 2024-04-21T23:10:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.265+0000 7f25dc763740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55d609a91470 con 0x55d609b85440 2024-04-21T23:10:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.265+0000 7f25dc763740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d609ae4b30 con 0x55d609b8e210 2024-04-21T23:10:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.265+0000 7f25d0ff9640 1 --2- 172.21.15.44:0/2209105398 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f25bc0a2cf0 0x7f25bc0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.266+0000 7f25d37fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b85440 0x55d609b8dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.266+0000 7f25d2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b84b20 0x55d609b84f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.266+0000 7f25d37fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b85440 0x55d609b8dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:60220/0 (socket says 172.21.15.44:60220) 2024-04-21T23:10:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.266+0000 7f25d37fe640 1 -- 172.21.15.44:0/3083307210 learned_addr learned my addr 172.21.15.44:0/3083307210 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.266+0000 7f25d3fff640 1 --2- 172.21.15.44:0/2209105398 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f25bc0a2cf0 0x7f25bc0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.266+0000 7f25d2ffd640 1 --2- 172.21.15.44:0/3083307210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b8e210 0x55d609b90600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.266+0000 7f25d0ff9640 1 -- 172.21.15.44:0/2209105398 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7f25cc0f1390 con 0x7f25d4106630 2024-04-21T23:10:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.266+0000 7f25d2ffd640 1 -- 172.21.15.44:0/3083307210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b8e210 msgr2=0x55d609b90600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.266+0000 7f25d2ffd640 1 --2- 172.21.15.44:0/3083307210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b8e210 0x55d609b90600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.266+0000 7f25d2ffd640 1 -- 172.21.15.44:0/3083307210 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b85440 msgr2=0x55d609b8dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.266+0000 7f25d2ffd640 1 --2- 172.21.15.44:0/3083307210 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b85440 0x55d609b8dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.266+0000 7f25d2ffd640 1 -- 172.21.15.44:0/3083307210 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6097fd210 con 0x55d609b84b20 2024-04-21T23:10:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.266+0000 7f25d37fe640 1 --2- 172.21.15.44:0/3083307210 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b85440 0x55d609b8dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:10:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.266+0000 7f25d2ffd640 1 --2- 172.21.15.44:0/3083307210 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b84b20 0x55d609b84f00 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f25c800b6c0 tx=0x7f25c800b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65cbbd66aed1e3ca server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.266+0000 7f25d3fff640 1 --2- 172.21.15.44:0/2209105398 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f25bc0a2cf0 0x7f25bc0a51b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f25cc007f00 tx=0x7f25cc02e040 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.266+0000 7f25b9ffb640 1 -- 172.21.15.44:0/3083307210 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c8004020 con 0x55d609b84b20 2024-04-21T23:10:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.266+0000 7f25b9ffb640 1 -- 172.21.15.44:0/3083307210 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25c8004c80 con 0x55d609b84b20 2024-04-21T23:10:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.266+0000 7f25b9ffb640 1 -- 172.21.15.44:0/3083307210 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c80053a0 con 0x55d609b84b20 2024-04-21T23:10:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.267+0000 7f25dc763740 1 -- 172.21.15.44:0/3083307210 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b84b20 msgr2=0x55d609b84f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.267+0000 7f25dc763740 1 --2- 172.21.15.44:0/3083307210 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b84b20 0x55d609b84f00 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f25c800b6c0 tx=0x7f25c800b650 comp rx=0 tx=0).stop 2024-04-21T23:10:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.267+0000 7f25dc763740 1 -- 172.21.15.44:0/3083307210 shutdown_connections 2024-04-21T23:10:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.267+0000 7f25dc763740 1 --2- 172.21.15.44:0/3083307210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b8e210 0x55d609b90600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.267+0000 7f25dc763740 1 --2- 172.21.15.44:0/3083307210 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b85440 0x55d609b8dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.267+0000 7f25dc763740 1 --2- 172.21.15.44:0/3083307210 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b84b20 0x55d609b84f00 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.267+0000 7f25dc763740 1 -- 172.21.15.44:0/3083307210 >> 172.21.15.44:0/3083307210 conn(0x55d609b7f440 msgr2=0x55d609b7f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.267+0000 7f25dc763740 1 -- 172.21.15.44:0/3083307210 shutdown_connections 2024-04-21T23:10:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.267+0000 7f25dc763740 1 -- 172.21.15.44:0/3083307210 wait complete. 2024-04-21T23:10:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.268+0000 7f25dc763740 1 -- start start 2024-04-21T23:10:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.268+0000 7f25dc763740 1 Processor -- start 2024-04-21T23:10:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.268+0000 7f25dc763740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b84b20 0x55d609b8b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.268+0000 7f25dc763740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b85440 0x55d609b8b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.268+0000 7f25dc763740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b8e210 0x55d609b8bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.268+0000 7f25dc763740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d609b90e40 con 0x55d609b85440 2024-04-21T23:10:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.268+0000 7f25dc763740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55d609aa4a10 con 0x55d609b8e210 2024-04-21T23:10:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.268+0000 7f25dc763740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d609b90fc0 con 0x55d609b84b20 2024-04-21T23:10:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.268+0000 7f25dc763740 10 client.0 _ll_register_callbacks cb 0x55d609adfb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.268+0000 7f25d37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b85440 0x55d609b8b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.268+0000 7f25d37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b85440 0x55d609b8b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36474/0 (socket says 172.21.15.44:36474) 2024-04-21T23:10:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.268+0000 7f25d37fe640 1 -- 172.21.15.44:0/4093064703 learned_addr learned my addr 172.21.15.44:0/4093064703 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.268+0000 7f25d2ffd640 1 --2- 172.21.15.44:0/4093064703 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b8e210 0x55d609b8bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.268+0000 7f25d2ffd640 1 --2- 172.21.15.44:0/4093064703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b84b20 0x55d609b8b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.268+0000 7f25d37fe640 1 -- 172.21.15.44:0/4093064703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b84b20 msgr2=0x55d609b8b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.268+0000 7f25d37fe640 1 --2- 172.21.15.44:0/4093064703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b84b20 0x55d609b8b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.268+0000 7f25d37fe640 1 -- 172.21.15.44:0/4093064703 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b8e210 msgr2=0x55d609b8bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.268+0000 7f25d37fe640 1 --2- 172.21.15.44:0/4093064703 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b8e210 0x55d609b8bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.268+0000 7f25d37fe640 1 -- 172.21.15.44:0/4093064703 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6098c9620 con 0x55d609b85440 2024-04-21T23:10:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.268+0000 7f25d2ffd640 1 --2- 172.21.15.44:0/4093064703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b84b20 0x55d609b8b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:10:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.269+0000 7f25d37fe640 1 --2- 172.21.15.44:0/4093064703 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b85440 0x55d609b8b750 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f25c4012740 tx=0x7f25c4012c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac736318e54a10 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.269+0000 7f25b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:10:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.269+0000 7f25b2ffd640 1 -- 172.21.15.44:0/4093064703 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c400f8c0 con 0x55d609b85440 2024-04-21T23:10:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.269+0000 7f25b2ffd640 1 -- 172.21.15.44:0/4093064703 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7f25c400fa60 con 0x55d609b85440 2024-04-21T23:10:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.269+0000 7f25dc763740 1 -- 172.21.15.44:0/4093064703 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25c8004c80 con 0x55d609b85440 2024-04-21T23:10:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.269+0000 7f25b2ffd640 1 -- 172.21.15.44:0/4093064703 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f25c400fc60 con 0x55d609b85440 2024-04-21T23:10:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.269+0000 7f25dc763740 10 client.16107 fetch_fsmap learned FSMap version 23 2024-04-21T23:10:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.269+0000 7f25dc763740 1 -- 172.21.15.44:0/4093064703 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d609b9d100 con 0x55d609b85440 2024-04-21T23:10:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.269+0000 7f25b2ffd640 1 -- 172.21.15.44:0/4093064703 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25c8004c80 con 0x55d609b85440 2024-04-21T23:10:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.270+0000 7f25b2ffd640 1 -- 172.21.15.44:0/4093064703 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f25c401c420 con 0x55d609b85440 2024-04-21T23:10:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.270+0000 7f25dc763740 10 client.16107 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T23:10:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.270+0000 7f25dc763740 10 client.16107 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.270+0000 7f25dc763740 20 client.16107 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.270+0000 7f25dc763740 1 --2- 172.21.15.44:0/4093064703 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d609b9e660 0x55d609ba0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.270+0000 7f25dc763740 4 client.16107 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:10:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.270+0000 7f25dc763740 1 -- 172.21.15.44:0/4093064703 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d609af60b0 con 0x55d609b9e660 2024-04-21T23:10:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.270+0000 7f25d2ffd640 1 --2- 172.21.15.44:0/4093064703 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d609b9e660 0x55d609ba0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.270+0000 7f25b2ffd640 10 client.16107 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.270+0000 7f25d2ffd640 1 --2- 172.21.15.44:0/4093064703 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d609b9e660 0x55d609ba0a50 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c93768fb84c06a16 server_cookie=a5e8b4a489b71c6c in_seq=0 out_seq=0 2024-04-21T23:10:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.272+0000 7f25b2ffd640 1 -- 172.21.15.44:0/4093064703 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d609af60b0 con 0x55d609b9e660 2024-04-21T23:10:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.272+0000 7f25b2ffd640 10 client.16107 handle_command_reply: tid=0 2024-04-21T23:10:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.272+0000 7f25dc763740 1 client.16107 shutdown 2024-04-21T23:10:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.273+0000 7f25dc763740 1 -- 172.21.15.44:0/4093064703 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b85440 msgr2=0x55d609b8b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.273+0000 7f25dc763740 1 --2- 172.21.15.44:0/4093064703 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b85440 0x55d609b8b750 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f25c4012740 tx=0x7f25c4012c00 comp rx=0 tx=0).stop 2024-04-21T23:10:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.273+0000 7f25dc763740 1 -- 172.21.15.44:0/4093064703 shutdown_connections 2024-04-21T23:10:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.273+0000 7f25dc763740 1 --2- 172.21.15.44:0/4093064703 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d609b9e660 0x55d609ba0a50 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.273+0000 7f25dc763740 1 --2- 172.21.15.44:0/4093064703 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b8e210 0x55d609b8bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.273+0000 7f25dc763740 1 --2- 172.21.15.44:0/4093064703 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b85440 0x55d609b8b750 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.273+0000 7f25b2ffd640 0 client.16107 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.273+0000 7f25dc763740 1 --2- 172.21.15.44:0/4093064703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b84b20 0x55d609b8b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.273+0000 7f25dc763740 1 -- 172.21.15.44:0/4093064703 >> 172.21.15.44:0/4093064703 conn(0x55d609b7f440 msgr2=0x55d609b81190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.273+0000 7f25dc763740 1 -- 172.21.15.44:0/4093064703 shutdown_connections 2024-04-21T23:10:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.273+0000 7f25dc763740 1 -- 172.21.15.44:0/4093064703 wait complete. 2024-04-21T23:10:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.273+0000 7f25dc763740 20 client.16107 trim_cache size 0 max 16384 2024-04-21T23:10:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.288+0000 7f25dc763740 1 Processor -- start 2024-04-21T23:10:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.288+0000 7f25dc763740 1 -- start start 2024-04-21T23:10:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.288+0000 7f25dc763740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b84b20 0x55d609b99d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.288+0000 7f25dc763740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b85440 0x55d609b8bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.288+0000 7f25dc763740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b8e210 0x55d609ba6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.288+0000 7f25dc763740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f25d41acbb0 con 0x55d609b85440 2024-04-21T23:10:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.288+0000 7f25dc763740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f25d41af430 con 0x55d609b8e210 2024-04-21T23:10:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.288+0000 7f25dc763740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f25d41ad220 con 0x55d609b84b20 2024-04-21T23:10:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.288+0000 7f25d37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b85440 0x55d609b8bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.288+0000 7f25d37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b85440 0x55d609b8bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36490/0 (socket says 172.21.15.44:36490) 2024-04-21T23:10:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.288+0000 7f25d37fe640 1 -- 172.21.15.44:0/1354182596 learned_addr learned my addr 172.21.15.44:0/1354182596 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.288+0000 7f25d2ffd640 1 --2- 172.21.15.44:0/1354182596 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b8e210 0x55d609ba6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.288+0000 7f25d2ffd640 1 --2- 172.21.15.44:0/1354182596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b84b20 0x55d609b99d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.289+0000 7f25d37fe640 1 -- 172.21.15.44:0/1354182596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b84b20 msgr2=0x55d609b99d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.289+0000 7f25d37fe640 1 --2- 172.21.15.44:0/1354182596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b84b20 0x55d609b99d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.289+0000 7f25d37fe640 1 -- 172.21.15.44:0/1354182596 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b8e210 msgr2=0x55d609ba6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.289+0000 7f25d37fe640 1 --2- 172.21.15.44:0/1354182596 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b8e210 0x55d609ba6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.289+0000 7f25d37fe640 1 -- 172.21.15.44:0/1354182596 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d609ba10d0 con 0x55d609b85440 2024-04-21T23:10:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.289+0000 7f25d37fe640 1 --2- 172.21.15.44:0/1354182596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b85440 0x55d609b8bab0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f25c401ca80 tx=0x7f25c400ff00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d857da057df9447e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.289+0000 7f25b97fa640 1 -- 172.21.15.44:0/1354182596 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c401cb00 con 0x55d609b85440 2024-04-21T23:10:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.289+0000 7f25b97fa640 1 -- 172.21.15.44:0/1354182596 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25c4012e50 con 0x55d609b85440 2024-04-21T23:10:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.289+0000 7f25b97fa640 1 -- 172.21.15.44:0/1354182596 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c402a910 con 0x55d609b85440 2024-04-21T23:10:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.290+0000 7f25dc763740 1 -- 172.21.15.44:0/1354182596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b85440 msgr2=0x55d609b8bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.290+0000 7f25dc763740 1 --2- 172.21.15.44:0/1354182596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b85440 0x55d609b8bab0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f25c401ca80 tx=0x7f25c400ff00 comp rx=0 tx=0).stop 2024-04-21T23:10:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.290+0000 7f25dc763740 1 -- 172.21.15.44:0/1354182596 shutdown_connections 2024-04-21T23:10:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.290+0000 7f25dc763740 1 --2- 172.21.15.44:0/1354182596 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b8e210 0x55d609ba6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.290+0000 7f25dc763740 1 --2- 172.21.15.44:0/1354182596 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b85440 0x55d609b8bab0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.290+0000 7f25dc763740 1 --2- 172.21.15.44:0/1354182596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b84b20 0x55d609b99d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.290+0000 7f25dc763740 1 -- 172.21.15.44:0/1354182596 >> 172.21.15.44:0/1354182596 conn(0x55d609b7f440 msgr2=0x55d609b80fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.290+0000 7f25dc763740 1 -- 172.21.15.44:0/1354182596 shutdown_connections 2024-04-21T23:10:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.290+0000 7f25dc763740 1 -- 172.21.15.44:0/1354182596 wait complete. 2024-04-21T23:10:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.290+0000 7f25dc763740 1 -- start start 2024-04-21T23:10:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.290+0000 7f25dc763740 1 Processor -- start 2024-04-21T23:10:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25dc763740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b84b20 0x55d609b91e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25dc763740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b85440 0x55d609b92380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25dc763740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b8e210 0x55d609b928c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25dc763740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f25d41ad8f0 con 0x55d609b8e210 2024-04-21T23:10:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25dc763740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f25d41aed60 con 0x55d609b84b20 2024-04-21T23:10:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25dc763740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f25d41adfc0 con 0x55d609b85440 2024-04-21T23:10:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25dc763740 10 client.0 _ll_register_callbacks cb 0x7f25c4009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:10:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25d2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b8e210 0x55d609b928c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25d2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b8e210 0x55d609b928c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36502/0 (socket says 172.21.15.44:36502) 2024-04-21T23:10:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25d2ffd640 1 -- 172.21.15.44:0/633265588 learned_addr learned my addr 172.21.15.44:0/633265588 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:10:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25d2ffd640 1 --2- 172.21.15.44:0/633265588 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b84b20 0x55d609b91e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25d37fe640 1 --2- 172.21.15.44:0/633265588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b85440 0x55d609b92380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25d2ffd640 1 -- 172.21.15.44:0/633265588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b85440 msgr2=0x55d609b92380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25d2ffd640 1 --2- 172.21.15.44:0/633265588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b85440 0x55d609b92380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25d2ffd640 1 -- 172.21.15.44:0/633265588 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b84b20 msgr2=0x55d609b91e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:10:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25d2ffd640 1 --2- 172.21.15.44:0/633265588 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b84b20 0x55d609b91e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25d2ffd640 1 -- 172.21.15.44:0/633265588 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d609c03e20 con 0x55d609b8e210 2024-04-21T23:10:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25d2ffd640 1 --2- 172.21.15.44:0/633265588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b8e210 0x55d609b928c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f25c8005d00 tx=0x7f25c800b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dee8fb9aa81c80f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:10:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:10:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25b2ffd640 1 -- 172.21.15.44:0/633265588 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c802e780 con 0x55d609b8e210 2024-04-21T23:10:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25b2ffd640 1 -- 172.21.15.44:0/633265588 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7f25c802eae0 con 0x55d609b8e210 2024-04-21T23:10:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25dc763740 1 -- 172.21.15.44:0/633265588 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25c4012e50 con 0x55d609b8e210 2024-04-21T23:10:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.291+0000 7f25b2ffd640 1 -- 172.21.15.44:0/633265588 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f25c802e920 con 0x55d609b8e210 2024-04-21T23:10:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.292+0000 7f25dc763740 10 client.16113 fetch_fsmap learned FSMap version 23 2024-04-21T23:10:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.292+0000 7f25dc763740 1 -- 172.21.15.44:0/633265588 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d609c050f0 con 0x55d609b8e210 2024-04-21T23:10:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.292+0000 7f25b2ffd640 1 -- 172.21.15.44:0/633265588 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25c4012e50 con 0x55d609b8e210 2024-04-21T23:10:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.292+0000 7f25b2ffd640 1 -- 172.21.15.44:0/633265588 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f25c802ece0 con 0x55d609b8e210 2024-04-21T23:10:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.292+0000 7f25dc763740 10 client.16113 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T23:10:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.292+0000 7f25dc763740 10 client.16113 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:10:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.292+0000 7f25dc763740 20 client.16113 populate_metadata read hostname 'smithi044' 2024-04-21T23:10:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.292+0000 7f25dc763740 1 --2- 172.21.15.44:0/633265588 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d609c05600 0x55d609c07a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:10:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.292+0000 7f25dc763740 4 client.16113 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:10:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.292+0000 7f25dc763740 1 -- 172.21.15.44:0/633265588 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d609af0400 con 0x55d609c05600 2024-04-21T23:10:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.292+0000 7f25d37fe640 1 --2- 172.21.15.44:0/633265588 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d609c05600 0x55d609c07a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:10:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.292+0000 7f25b2ffd640 10 client.16113 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.292+0000 7f25d37fe640 1 --2- 172.21.15.44:0/633265588 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d609c05600 0x55d609c07a20 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85ed16c8093342bb server_cookie=d72730f73f6806d in_seq=0 out_seq=0 2024-04-21T23:10:56.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:56 smithi044 ceph-mon[27934]: pgmap v66: 129 pgs: 129 active+clean; 35 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 308 op/s 2024-04-21T23:10:56.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:56 smithi069 ceph-mon[31816]: pgmap v66: 129 pgs: 129 active+clean; 35 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 308 op/s 2024-04-21T23:10:56.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:56 smithi055 ceph-mon[31117]: pgmap v66: 129 pgs: 129 active+clean; 35 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 308 op/s 2024-04-21T23:10:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.554+0000 7f25b2ffd640 1 -- 172.21.15.44:0/633265588 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d609af0400 con 0x55d609c05600 2024-04-21T23:10:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.554+0000 7f25b2ffd640 10 client.16113 handle_command_reply: tid=0 2024-04-21T23:10:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.554+0000 7f25dc763740 1 client.16113 shutdown 2024-04-21T23:10:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.554+0000 7f25dc763740 1 -- 172.21.15.44:0/633265588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b8e210 msgr2=0x55d609b928c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.554+0000 7f25dc763740 1 --2- 172.21.15.44:0/633265588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b8e210 0x55d609b928c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f25c8005d00 tx=0x7f25c800b650 comp rx=0 tx=0).stop 2024-04-21T23:10:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.554+0000 7f25dc763740 1 -- 172.21.15.44:0/633265588 shutdown_connections 2024-04-21T23:10:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.554+0000 7f25dc763740 1 --2- 172.21.15.44:0/633265588 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d609c05600 0x55d609c07a20 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.554+0000 7f25dc763740 1 --2- 172.21.15.44:0/633265588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d609b85440 0x55d609b92380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.554+0000 7f25dc763740 1 --2- 172.21.15.44:0/633265588 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d609b8e210 0x55d609b928c0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.554+0000 7f25dc763740 1 --2- 172.21.15.44:0/633265588 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d609b84b20 0x55d609b91e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.554+0000 7f25dc763740 1 -- 172.21.15.44:0/633265588 >> 172.21.15.44:0/633265588 conn(0x55d609b7f440 msgr2=0x55d609b80600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.554+0000 7f25b2ffd640 0 client.16113 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:10:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.554+0000 7f25dc763740 1 -- 172.21.15.44:0/633265588 shutdown_connections 2024-04-21T23:10:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.554+0000 7f25dc763740 1 -- 172.21.15.44:0/633265588 wait complete. 2024-04-21T23:10:56.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.555+0000 7f25dc763740 20 client.16113 trim_cache size 0 max 16384 2024-04-21T23:10:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:10:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:10:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:10:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:10:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.555+0000 7f25d9b7a640 1 -- 172.21.15.44:0/2209105398 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f25bc0a2cf0 msgr2=0x7f25bc0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.555+0000 7f25d9b7a640 1 --2- 172.21.15.44:0/2209105398 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f25bc0a2cf0 0x7f25bc0a51b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f25cc007f00 tx=0x7f25cc02e040 comp rx=0 tx=0).stop 2024-04-21T23:10:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.555+0000 7f25d9b7a640 1 -- 172.21.15.44:0/2209105398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25d4106630 msgr2=0x7f25d419c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:10:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.555+0000 7f25d9b7a640 1 --2- 172.21.15.44:0/2209105398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25d4106630 0x7f25d419c920 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f25cc005e30 tx=0x7f25cc032ea0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.555+0000 7f25d9b7a640 1 -- 172.21.15.44:0/2209105398 shutdown_connections 2024-04-21T23:10:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.555+0000 7f25d9b7a640 1 --2- 172.21.15.44:0/2209105398 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f25bc0a2cf0 0x7f25bc0a51b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.555+0000 7f25d9b7a640 1 --2- 172.21.15.44:0/2209105398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25d4106630 0x7f25d419c920 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.555+0000 7f25d9b7a640 1 --2- 172.21.15.44:0/2209105398 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25d4100fe0 0x7f25d41985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.555+0000 7f25d9b7a640 1 --2- 172.21.15.44:0/2209105398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d4100620 0x7f25d4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:10:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.555+0000 7f25d9b7a640 1 -- 172.21.15.44:0/2209105398 >> 172.21.15.44:0/2209105398 conn(0x7f25d4077760 msgr2=0x7f25d41031f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:10:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.555+0000 7f25d9b7a640 1 -- 172.21.15.44:0/2209105398 shutdown_connections 2024-04-21T23:10:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:10:56.555+0000 7f25d9b7a640 1 -- 172.21.15.44:0/2209105398 wait complete. 2024-04-21T23:10:56.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 127395 15.30 MB/sec warmup 46 sec latency 102.799 ms 2024-04-21T23:10:57.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:57 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:57.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:57 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:57.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:57 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:10:57.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 129588 15.21 MB/sec warmup 47 sec latency 23.587 ms 2024-04-21T23:10:58.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:58 smithi044 ceph-mon[27934]: pgmap v67: 129 pgs: 129 active+clean; 32 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 418 op/s 2024-04-21T23:10:58.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:58 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:58.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:58 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:58.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:10:58 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:10:58.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:58 smithi055 ceph-mon[31117]: pgmap v67: 129 pgs: 129 active+clean; 32 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 418 op/s 2024-04-21T23:10:58.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:58 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:58.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:58 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:58.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:10:58 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:10:58.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:58 smithi069 ceph-mon[31816]: pgmap v67: 129 pgs: 129 active+clean; 32 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 418 op/s 2024-04-21T23:10:58.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:58 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:10:58.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:58 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:10:58.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:10:58 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:10:58.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 132194 15.19 MB/sec warmup 48 sec latency 22.124 ms 2024-04-21T23:10:59.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 135355 15.26 MB/sec warmup 49 sec latency 14.346 ms 2024-04-21T23:11:00.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:00 smithi044 ceph-mon[27934]: pgmap v68: 129 pgs: 129 active+clean; 34 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 406 op/s 2024-04-21T23:11:00.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:00 smithi055 ceph-mon[31117]: pgmap v68: 129 pgs: 129 active+clean; 34 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 406 op/s 2024-04-21T23:11:00.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:00 smithi069 ceph-mon[31816]: pgmap v68: 129 pgs: 129 active+clean; 34 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 406 op/s 2024-04-21T23:11:00.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 138504 15.31 MB/sec warmup 50 sec latency 15.360 ms 2024-04-21T23:11:01.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 139396 15.06 MB/sec warmup 51 sec latency 70.805 ms 2024-04-21T23:11:02.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:02 smithi044 ceph-mon[27934]: pgmap v69: 129 pgs: 129 active+clean; 34 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 338 op/s 2024-04-21T23:11:02.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:02 smithi055 ceph-mon[31117]: pgmap v69: 129 pgs: 129 active+clean; 34 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 338 op/s 2024-04-21T23:11:02.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:02 smithi069 ceph-mon[31816]: pgmap v69: 129 pgs: 129 active+clean; 34 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 338 op/s 2024-04-21T23:11:02.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 141378 14.99 MB/sec warmup 52 sec latency 102.854 ms 2024-04-21T23:11:03.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 143615 14.93 MB/sec warmup 53 sec latency 23.172 ms 2024-04-21T23:11:04.024 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:11:04.024 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:11:04.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.136+0000 7f061db4b640 1 -- 172.21.15.44:0/2102889866 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06181032d0 msgr2=0x7f061810b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.136+0000 7f061db4b640 1 --2- 172.21.15.44:0/2102889866 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06181032d0 0x7f061810b7b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f060c009900 tx=0x7f060c02bcd0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.136+0000 7f061db4b640 1 -- 172.21.15.44:0/2102889866 shutdown_connections 2024-04-21T23:11:04.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.136+0000 7f061db4b640 1 --2- 172.21.15.44:0/2102889866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f061810bcf0 0x7f061810e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.136+0000 7f061db4b640 1 --2- 172.21.15.44:0/2102889866 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06181032d0 0x7f061810b7b0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.136+0000 7f061db4b640 1 --2- 172.21.15.44:0/2102889866 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06181029b0 0x7f0618102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.136+0000 7f061db4b640 1 -- 172.21.15.44:0/2102889866 >> 172.21.15.44:0/2102889866 conn(0x7f06180777f0 msgr2=0x7f0618077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:04.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.136+0000 7f061db4b640 1 -- 172.21.15.44:0/2102889866 shutdown_connections 2024-04-21T23:11:04.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.136+0000 7f061db4b640 1 -- 172.21.15.44:0/2102889866 wait complete. 2024-04-21T23:11:04.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.136+0000 7f061db4b640 1 Processor -- start 2024-04-21T23:11:04.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.137+0000 7f061db4b640 1 -- start start 2024-04-21T23:11:04.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.137+0000 7f061db4b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06181029b0 0x7f0618198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.137+0000 7f061db4b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06181032d0 0x7f0618198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.137+0000 7f061db4b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f061810bcf0 0x7f061819c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.137+0000 7f061db4b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f0618111af0 con 0x7f06181029b0 2024-04-21T23:11:04.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.137+0000 7f061db4b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0618111970 con 0x7f06181032d0 2024-04-21T23:11:04.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.137+0000 7f061db4b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0618111c70 con 0x7f061810bcf0 2024-04-21T23:11:04.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.137+0000 7f06177fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06181029b0 0x7f0618198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.137+0000 7f06177fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06181029b0 0x7f0618198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43460/0 (socket says 172.21.15.44:43460) 2024-04-21T23:11:04.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.137+0000 7f06177fe640 1 -- 172.21.15.44:0/346014901 learned_addr learned my addr 172.21.15.44:0/346014901 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:04.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.137+0000 7f0616ffd640 1 --2- 172.21.15.44:0/346014901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06181032d0 0x7f0618198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.137+0000 7f0617fff640 1 --2- 172.21.15.44:0/346014901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f061810bcf0 0x7f061819c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.137+0000 7f06177fe640 1 -- 172.21.15.44:0/346014901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f061810bcf0 msgr2=0x7f061819c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.137+0000 7f06177fe640 1 --2- 172.21.15.44:0/346014901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f061810bcf0 0x7f061819c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.137+0000 7f06177fe640 1 -- 172.21.15.44:0/346014901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06181032d0 msgr2=0x7f0618198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.137+0000 7f06177fe640 1 --2- 172.21.15.44:0/346014901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06181032d0 0x7f0618198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.137+0000 7f06177fe640 1 -- 172.21.15.44:0/346014901 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061819cfb0 con 0x7f06181029b0 2024-04-21T23:11:04.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.138+0000 7f06177fe640 1 --2- 172.21.15.44:0/346014901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06181029b0 0x7f0618198010 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f060800ea60 tx=0x7f060800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:04.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.138+0000 7f0614ff9640 1 -- 172.21.15.44:0/346014901 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060800cde0 con 0x7f06181029b0 2024-04-21T23:11:04.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.138+0000 7f061db4b640 1 -- 172.21.15.44:0/346014901 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f061819d2a0 con 0x7f06181029b0 2024-04-21T23:11:04.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.138+0000 7f0614ff9640 1 -- 172.21.15.44:0/346014901 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06080035f0 con 0x7f06181029b0 2024-04-21T23:11:04.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.138+0000 7f0614ff9640 1 -- 172.21.15.44:0/346014901 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0608010670 con 0x7f06181029b0 2024-04-21T23:11:04.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.138+0000 7f061db4b640 1 -- 172.21.15.44:0/346014901 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06181a4b30 con 0x7f06181029b0 2024-04-21T23:11:04.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.140+0000 7f0614ff9640 1 -- 172.21.15.44:0/346014901 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f06080037a0 con 0x7f06181029b0 2024-04-21T23:11:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.140+0000 7f062080b740 1 Processor -- start 2024-04-21T23:11:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.140+0000 7f062080b740 1 -- start start 2024-04-21T23:11:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.140+0000 7f062080b740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77d7f30 0x5559a77d8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.140+0000 7f062080b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77d8850 0x5559a77e0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.140+0000 7f062080b740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77e1430 0x5559a77e3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.140+0000 7f062080b740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5559a773c560 con 0x5559a77d7f30 2024-04-21T23:11:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.140+0000 7f062080b740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5559a76e4730 con 0x5559a77e1430 2024-04-21T23:11:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.140+0000 7f062080b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5559a7738520 con 0x5559a77d8850 2024-04-21T23:11:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.141+0000 7f0617fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77d7f30 0x5559a77d8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.141+0000 7f0617fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77d7f30 0x5559a77d8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43464/0 (socket says 172.21.15.44:43464) 2024-04-21T23:11:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.141+0000 7f0617fff640 1 -- 172.21.15.44:0/1171730292 learned_addr learned my addr 172.21.15.44:0/1171730292 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.141+0000 7f0616ffd640 1 --2- 172.21.15.44:0/1171730292 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77e1430 0x5559a77e3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.141+0000 7f06177fe640 1 --2- 172.21.15.44:0/1171730292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77d8850 0x5559a77e0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.141+0000 7f0614ff9640 1 --2- 172.21.15.44:0/346014901 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f06000a2d00 0x7f06000a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.141+0000 7f0617fff640 1 --2- 172.21.15.44:0/346014901 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f06000a2d00 0x7f06000a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.141+0000 7f0616ffd640 1 -- 172.21.15.44:0/1171730292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77d8850 msgr2=0x5559a77e0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.141+0000 7f0616ffd640 1 --2- 172.21.15.44:0/1171730292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77d8850 0x5559a77e0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.141+0000 7f0616ffd640 1 -- 172.21.15.44:0/1171730292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77d7f30 msgr2=0x5559a77d8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.141+0000 7f0616ffd640 1 --2- 172.21.15.44:0/1171730292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77d7f30 0x5559a77d8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.141+0000 7f0616ffd640 1 -- 172.21.15.44:0/1171730292 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559a751a7d0 con 0x5559a77e1430 2024-04-21T23:11:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.141+0000 7f0617fff640 1 --2- 172.21.15.44:0/1171730292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77d7f30 0x5559a77d8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.141+0000 7f0614ff9640 1 -- 172.21.15.44:0/346014901 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7f06080cfac0 con 0x7f06181029b0 2024-04-21T23:11:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.141+0000 7f0616ffd640 1 --2- 172.21.15.44:0/1171730292 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77e1430 0x5559a77e3820 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f060c000c00 tx=0x7f060c009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0d53cf89bda8217 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.141+0000 7f0617fff640 1 --2- 172.21.15.44:0/346014901 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f06000a2d00 0x7f06000a51c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0610009fd0 tx=0x7f0610004ab0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.141+0000 7f05fdffb640 1 -- 172.21.15.44:0/1171730292 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c042070 con 0x5559a77e1430 2024-04-21T23:11:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.141+0000 7f05fdffb640 1 -- 172.21.15.44:0/1171730292 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f060c004040 con 0x5559a77e1430 2024-04-21T23:11:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.141+0000 7f05fdffb640 1 -- 172.21.15.44:0/1171730292 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c004c90 con 0x5559a77e1430 2024-04-21T23:11:04.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.142+0000 7f062080b740 1 -- 172.21.15.44:0/1171730292 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77e1430 msgr2=0x5559a77e3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.142+0000 7f062080b740 1 --2- 172.21.15.44:0/1171730292 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77e1430 0x5559a77e3820 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f060c000c00 tx=0x7f060c009560 comp rx=0 tx=0).stop 2024-04-21T23:11:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.142+0000 7f062080b740 1 -- 172.21.15.44:0/1171730292 shutdown_connections 2024-04-21T23:11:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.142+0000 7f062080b740 1 --2- 172.21.15.44:0/1171730292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77d8850 0x5559a77e0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.142+0000 7f062080b740 1 --2- 172.21.15.44:0/1171730292 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77e1430 0x5559a77e3820 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.142+0000 7f062080b740 1 --2- 172.21.15.44:0/1171730292 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77d7f30 0x5559a77d8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.142+0000 7f062080b740 1 -- 172.21.15.44:0/1171730292 >> 172.21.15.44:0/1171730292 conn(0x5559a77d2660 msgr2=0x5559a77d2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.142+0000 7f062080b740 1 -- 172.21.15.44:0/1171730292 shutdown_connections 2024-04-21T23:11:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.142+0000 7f062080b740 1 -- 172.21.15.44:0/1171730292 wait complete. 2024-04-21T23:11:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.142+0000 7f062080b740 1 -- start start 2024-04-21T23:11:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.143+0000 7f062080b740 1 Processor -- start 2024-04-21T23:11:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.143+0000 7f062080b740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77d7f30 0x5559a77de4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.143+0000 7f062080b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77d8850 0x5559a77de9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.143+0000 7f062080b740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77e1430 0x5559a77def30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.143+0000 7f062080b740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5559a77e41c0 con 0x5559a77e1430 2024-04-21T23:11:04.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.143+0000 7f062080b740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5559a76f7e00 con 0x5559a77d7f30 2024-04-21T23:11:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.143+0000 7f062080b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5559a77e4340 con 0x5559a77d8850 2024-04-21T23:11:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.143+0000 7f062080b740 10 client.0 _ll_register_callbacks cb 0x5559a7732eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.143+0000 7f0616ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77d8850 0x5559a77de9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.143+0000 7f0617fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77e1430 0x5559a77def30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.143+0000 7f0616ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77d8850 0x5559a77de9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:40292/0 (socket says 172.21.15.44:40292) 2024-04-21T23:11:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.143+0000 7f0616ffd640 1 -- 172.21.15.44:0/2173022534 learned_addr learned my addr 172.21.15.44:0/2173022534 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.143+0000 7f06177fe640 1 --2- 172.21.15.44:0/2173022534 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77d7f30 0x5559a77de4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.143+0000 7f0616ffd640 1 -- 172.21.15.44:0/2173022534 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77d7f30 msgr2=0x5559a77de4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.143+0000 7f0616ffd640 1 --2- 172.21.15.44:0/2173022534 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77d7f30 0x5559a77de4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.143+0000 7f0616ffd640 1 -- 172.21.15.44:0/2173022534 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77e1430 msgr2=0x5559a77def30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.143+0000 7f0616ffd640 1 --2- 172.21.15.44:0/2173022534 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77e1430 0x5559a77def30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.143+0000 7f0616ffd640 1 -- 172.21.15.44:0/2173022534 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559a74503f0 con 0x5559a77d8850 2024-04-21T23:11:04.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.144+0000 7f0616ffd640 1 --2- 172.21.15.44:0/2173022534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77d8850 0x5559a77de9f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f060c02c210 tx=0x7f060c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce8f0605656ff275 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:04.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.144+0000 7f05f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:11:04.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.144+0000 7f05f6ffd640 1 -- 172.21.15.44:0/2173022534 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c042070 con 0x5559a77d8850 2024-04-21T23:11:04.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.144+0000 7f062080b740 1 -- 172.21.15.44:0/2173022534 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f060c004040 con 0x5559a77d8850 2024-04-21T23:11:04.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.144+0000 7f05f6ffd640 1 -- 172.21.15.44:0/2173022534 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7f060c02cb30 con 0x5559a77d8850 2024-04-21T23:11:04.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.144+0000 7f05f6ffd640 1 -- 172.21.15.44:0/2173022534 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f060c0046c0 con 0x5559a77d8850 2024-04-21T23:11:04.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.144+0000 7f062080b740 10 client.25735 fetch_fsmap learned FSMap version 23 2024-04-21T23:11:04.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.144+0000 7f062080b740 1 -- 172.21.15.44:0/2173022534 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5559a74100f0 con 0x5559a77d8850 2024-04-21T23:11:04.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.144+0000 7f05f6ffd640 1 -- 172.21.15.44:0/2173022534 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f060c004040 con 0x5559a77d8850 2024-04-21T23:11:04.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.144+0000 7f05f6ffd640 1 -- 172.21.15.44:0/2173022534 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f060c04f030 con 0x5559a77d8850 2024-04-21T23:11:04.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.144+0000 7f062080b740 10 client.25735 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T23:11:04.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.144+0000 7f062080b740 10 client.25735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:04.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.144+0000 7f062080b740 20 client.25735 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:04.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.144+0000 7f062080b740 1 --2- 172.21.15.44:0/2173022534 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5559a77f1630 0x5559a77f3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.144+0000 7f062080b740 4 client.25735 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:11:04.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.144+0000 7f062080b740 1 -- 172.21.15.44:0/2173022534 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559a7749aa0 con 0x5559a77f1630 2024-04-21T23:11:04.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.145+0000 7f06177fe640 1 --2- 172.21.15.44:0/2173022534 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5559a77f1630 0x5559a77f3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.145+0000 7f06177fe640 1 --2- 172.21.15.44:0/2173022534 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5559a77f1630 0x5559a77f3a20 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d6383178adaa3be server_cookie=15eb6aa44d9f5930 in_seq=0 out_seq=0 2024-04-21T23:11:04.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.145+0000 7f05f6ffd640 10 client.25735 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:04.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.145+0000 7f05f6ffd640 1 -- 172.21.15.44:0/2173022534 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5559a7749aa0 con 0x5559a77f1630 2024-04-21T23:11:04.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.145+0000 7f05f6ffd640 10 client.25735 handle_command_reply: tid=0 2024-04-21T23:11:04.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.146+0000 7f062080b740 1 client.25735 shutdown 2024-04-21T23:11:04.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.146+0000 7f062080b740 1 -- 172.21.15.44:0/2173022534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77d8850 msgr2=0x5559a77de9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.146+0000 7f062080b740 1 --2- 172.21.15.44:0/2173022534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77d8850 0x5559a77de9f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f060c02c210 tx=0x7f060c005fd0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.146+0000 7f062080b740 1 -- 172.21.15.44:0/2173022534 shutdown_connections 2024-04-21T23:11:04.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.146+0000 7f062080b740 1 --2- 172.21.15.44:0/2173022534 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5559a77f1630 0x5559a77f3a20 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.146+0000 7f062080b740 1 --2- 172.21.15.44:0/2173022534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77d8850 0x5559a77de9f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.146+0000 7f05f6ffd640 0 client.25735 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:04.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.146+0000 7f062080b740 1 --2- 172.21.15.44:0/2173022534 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77e1430 0x5559a77def30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.146+0000 7f062080b740 1 --2- 172.21.15.44:0/2173022534 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77d7f30 0x5559a77de4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.146+0000 7f062080b740 1 -- 172.21.15.44:0/2173022534 >> 172.21.15.44:0/2173022534 conn(0x5559a77d2660 msgr2=0x5559a77d43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:04.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.146+0000 7f062080b740 1 -- 172.21.15.44:0/2173022534 shutdown_connections 2024-04-21T23:11:04.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.146+0000 7f062080b740 1 -- 172.21.15.44:0/2173022534 wait complete. 2024-04-21T23:11:04.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.146+0000 7f062080b740 20 client.25735 trim_cache size 0 max 16384 2024-04-21T23:11:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.164+0000 7f062080b740 1 Processor -- start 2024-04-21T23:11:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.164+0000 7f062080b740 1 -- start start 2024-04-21T23:11:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.164+0000 7f062080b740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77d7f30 0x5559a77e0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.164+0000 7f062080b740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77d8850 0x5559a77ed080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.164+0000 7f062080b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77e1430 0x5559a77d1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.164+0000 7f062080b740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f06181acb50 con 0x5559a77d8850 2024-04-21T23:11:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.164+0000 7f062080b740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f06181af3d0 con 0x5559a77d7f30 2024-04-21T23:11:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.164+0000 7f062080b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f06181ad1c0 con 0x5559a77e1430 2024-04-21T23:11:04.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.164+0000 7f0616ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77d8850 0x5559a77ed080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.164+0000 7f0616ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77d8850 0x5559a77ed080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43492/0 (socket says 172.21.15.44:43492) 2024-04-21T23:11:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.164+0000 7f0616ffd640 1 -- 172.21.15.44:0/2924149102 learned_addr learned my addr 172.21.15.44:0/2924149102 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.164+0000 7f06177fe640 1 --2- 172.21.15.44:0/2924149102 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77d7f30 0x5559a77e0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.164+0000 7f0617fff640 1 --2- 172.21.15.44:0/2924149102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77e1430 0x5559a77d1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.164+0000 7f0616ffd640 1 -- 172.21.15.44:0/2924149102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77e1430 msgr2=0x5559a77d1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.164+0000 7f0616ffd640 1 --2- 172.21.15.44:0/2924149102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77e1430 0x5559a77d1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.164+0000 7f0616ffd640 1 -- 172.21.15.44:0/2924149102 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77d7f30 msgr2=0x5559a77e0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.164+0000 7f0616ffd640 1 --2- 172.21.15.44:0/2924149102 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77d7f30 0x5559a77e0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.164+0000 7f0616ffd640 1 -- 172.21.15.44:0/2924149102 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559a77e3b50 con 0x5559a77d8850 2024-04-21T23:11:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.165+0000 7f0616ffd640 1 --2- 172.21.15.44:0/2924149102 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77d8850 0x5559a77ed080 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f060c003a10 tx=0x7f060c0049c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=225b0248007a275a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.165+0000 7f05fd7fa640 1 -- 172.21.15.44:0/2924149102 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c042040 con 0x5559a77d8850 2024-04-21T23:11:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.165+0000 7f05fd7fa640 1 -- 172.21.15.44:0/2924149102 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f060c009520 con 0x5559a77d8850 2024-04-21T23:11:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.165+0000 7f05fd7fa640 1 -- 172.21.15.44:0/2924149102 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c003a90 con 0x5559a77d8850 2024-04-21T23:11:04.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.165+0000 7f062080b740 1 -- 172.21.15.44:0/2924149102 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77d8850 msgr2=0x5559a77ed080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.165+0000 7f062080b740 1 --2- 172.21.15.44:0/2924149102 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77d8850 0x5559a77ed080 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f060c003a10 tx=0x7f060c0049c0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.165+0000 7f062080b740 1 -- 172.21.15.44:0/2924149102 shutdown_connections 2024-04-21T23:11:04.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.165+0000 7f062080b740 1 --2- 172.21.15.44:0/2924149102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77e1430 0x5559a77d1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.165+0000 7f062080b740 1 --2- 172.21.15.44:0/2924149102 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77d8850 0x5559a77ed080 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.165+0000 7f062080b740 1 --2- 172.21.15.44:0/2924149102 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77d7f30 0x5559a77e0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.165+0000 7f062080b740 1 -- 172.21.15.44:0/2924149102 >> 172.21.15.44:0/2924149102 conn(0x5559a77d2660 msgr2=0x5559a77d0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:04.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.165+0000 7f062080b740 1 -- 172.21.15.44:0/2924149102 shutdown_connections 2024-04-21T23:11:04.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.165+0000 7f062080b740 1 -- 172.21.15.44:0/2924149102 wait complete. 2024-04-21T23:11:04.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.166+0000 7f062080b740 1 -- start start 2024-04-21T23:11:04.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.166+0000 7f062080b740 1 Processor -- start 2024-04-21T23:11:04.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.166+0000 7f062080b740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77d7f30 0x5559a77db2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.166+0000 7f062080b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77e1430 0x5559a77db810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.166+0000 7f062080b740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77dbd50 0x5559a785ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.166+0000 7f062080b740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f06181ad890 con 0x5559a77dbd50 2024-04-21T23:11:04.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.166+0000 7f062080b740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f06181aed00 con 0x5559a77d7f30 2024-04-21T23:11:04.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.166+0000 7f062080b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f06181adf60 con 0x5559a77e1430 2024-04-21T23:11:04.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.166+0000 7f062080b740 10 client.0 _ll_register_callbacks cb 0x7f0610009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:04.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.166+0000 7f06177fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77d7f30 0x5559a77db2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.166+0000 7f0617fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77dbd50 0x5559a785ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.166+0000 7f06177fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77d7f30 0x5559a77db2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:48776/0 (socket says 172.21.15.44:48776) 2024-04-21T23:11:04.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.166+0000 7f06177fe640 1 -- 172.21.15.44:0/3976745520 learned_addr learned my addr 172.21.15.44:0/3976745520 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:04.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.166+0000 7f0616ffd640 1 --2- 172.21.15.44:0/3976745520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77e1430 0x5559a77db810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.167+0000 7f06177fe640 1 -- 172.21.15.44:0/3976745520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77e1430 msgr2=0x5559a77db810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:04.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.167+0000 7f06177fe640 1 --2- 172.21.15.44:0/3976745520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77e1430 0x5559a77db810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.167+0000 7f06177fe640 1 -- 172.21.15.44:0/3976745520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77dbd50 msgr2=0x5559a785ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:04.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.167+0000 7f06177fe640 1 --2- 172.21.15.44:0/3976745520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77dbd50 0x5559a785ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.167+0000 7f06177fe640 1 -- 172.21.15.44:0/3976745520 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559a785b320 con 0x5559a77d7f30 2024-04-21T23:11:04.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.167+0000 7f0617fff640 1 --2- 172.21.15.44:0/3976745520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77dbd50 0x5559a785ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:04.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.167+0000 7f06177fe640 1 --2- 172.21.15.44:0/3976745520 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77d7f30 0x5559a77db2d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0608098610 tx=0x7f06080af040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e19c52fe35bf6ff server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:04.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.167+0000 7f05f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:11:04.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.167+0000 7f05f6ffd640 1 -- 172.21.15.44:0/3976745520 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060809fff0 con 0x5559a77d7f30 2024-04-21T23:11:04.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.167+0000 7f05f6ffd640 1 -- 172.21.15.44:0/3976745520 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7f06080cd0f0 con 0x5559a77d7f30 2024-04-21T23:11:04.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.167+0000 7f062080b740 1 -- 172.21.15.44:0/3976745520 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f060c009520 con 0x5559a77d7f30 2024-04-21T23:11:04.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.167+0000 7f05f6ffd640 1 -- 172.21.15.44:0/3976745520 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f06080a48b0 con 0x5559a77d7f30 2024-04-21T23:11:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.167+0000 7f062080b740 10 client.25703 fetch_fsmap learned FSMap version 23 2024-04-21T23:11:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.167+0000 7f062080b740 1 -- 172.21.15.44:0/3976745520 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5559a785c670 con 0x5559a77d7f30 2024-04-21T23:11:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.167+0000 7f05f6ffd640 1 -- 172.21.15.44:0/3976745520 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f060c009520 con 0x5559a77d7f30 2024-04-21T23:11:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f05f6ffd640 1 -- 172.21.15.44:0/3976745520 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f06080b4060 con 0x5559a77d7f30 2024-04-21T23:11:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f062080b740 10 client.25703 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T23:11:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f062080b740 10 client.25703 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f062080b740 20 client.25703 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f062080b740 1 --2- 172.21.15.44:0/3976745520 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5559a785cd70 0x5559a785f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f062080b740 4 client.25703 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:11:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f062080b740 1 -- 172.21.15.44:0/3976745520 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5559a7743df0 con 0x5559a785cd70 2024-04-21T23:11:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f0616ffd640 1 --2- 172.21.15.44:0/3976745520 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5559a785cd70 0x5559a785f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f0616ffd640 1 --2- 172.21.15.44:0/3976745520 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5559a785cd70 0x5559a785f190 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a720095bb40f3f91 server_cookie=ecb745fdb5ceb8b9 in_seq=0 out_seq=0 2024-04-21T23:11:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f05f6ffd640 10 client.25703 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f05f6ffd640 1 -- 172.21.15.44:0/3976745520 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5559a7743df0 con 0x5559a785cd70 2024-04-21T23:11:04.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f05f6ffd640 10 client.25703 handle_command_reply: tid=0 2024-04-21T23:11:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f062080b740 1 client.25703 shutdown 2024-04-21T23:11:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f062080b740 1 -- 172.21.15.44:0/3976745520 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77d7f30 msgr2=0x5559a77db2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f062080b740 1 --2- 172.21.15.44:0/3976745520 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77d7f30 0x5559a77db2d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0608098610 tx=0x7f06080af040 comp rx=0 tx=0).stop 2024-04-21T23:11:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f062080b740 1 -- 172.21.15.44:0/3976745520 shutdown_connections 2024-04-21T23:11:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f062080b740 1 --2- 172.21.15.44:0/3976745520 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5559a785cd70 0x5559a785f190 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f062080b740 1 --2- 172.21.15.44:0/3976745520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5559a77e1430 0x5559a77db810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f05f6ffd640 0 client.25703 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f062080b740 1 --2- 172.21.15.44:0/3976745520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5559a77dbd50 0x5559a785ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.168+0000 7f062080b740 1 --2- 172.21.15.44:0/3976745520 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5559a77d7f30 0x5559a77db2d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.169+0000 7f062080b740 1 -- 172.21.15.44:0/3976745520 >> 172.21.15.44:0/3976745520 conn(0x5559a77d2660 msgr2=0x5559a77d16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.169+0000 7f062080b740 1 -- 172.21.15.44:0/3976745520 shutdown_connections 2024-04-21T23:11:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.169+0000 7f062080b740 1 -- 172.21.15.44:0/3976745520 wait complete. 2024-04-21T23:11:04.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.169+0000 7f062080b740 20 client.25703 trim_cache size 0 max 16384 2024-04-21T23:11:04.175 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:11:04.175 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-21T23:11:04.175 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-21T23:11:04.175 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:11:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.169+0000 7f061db4b640 1 -- 172.21.15.44:0/346014901 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f06000a2d00 msgr2=0x7f06000a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.169+0000 7f061db4b640 1 --2- 172.21.15.44:0/346014901 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f06000a2d00 0x7f06000a51c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0610009fd0 tx=0x7f0610004ab0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.169+0000 7f061db4b640 1 -- 172.21.15.44:0/346014901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06181029b0 msgr2=0x7f0618198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.169+0000 7f061db4b640 1 --2- 172.21.15.44:0/346014901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06181029b0 0x7f0618198010 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f060800ea60 tx=0x7f060800ef20 comp rx=0 tx=0).stop 2024-04-21T23:11:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.169+0000 7f061db4b640 1 -- 172.21.15.44:0/346014901 shutdown_connections 2024-04-21T23:11:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.169+0000 7f061db4b640 1 --2- 172.21.15.44:0/346014901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f061810bcf0 0x7f061819c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.169+0000 7f061db4b640 1 --2- 172.21.15.44:0/346014901 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f06000a2d00 0x7f06000a51c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.169+0000 7f061db4b640 1 --2- 172.21.15.44:0/346014901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f06181032d0 0x7f0618198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.169+0000 7f061db4b640 1 --2- 172.21.15.44:0/346014901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f06181029b0 0x7f0618198010 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.169+0000 7f061db4b640 1 -- 172.21.15.44:0/346014901 >> 172.21.15.44:0/346014901 conn(0x7f06180777f0 msgr2=0x7f06180fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:04.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.169+0000 7f061db4b640 1 -- 172.21.15.44:0/346014901 shutdown_connections 2024-04-21T23:11:04.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.169+0000 7f061db4b640 1 -- 172.21.15.44:0/346014901 wait complete. 2024-04-21T23:11:04.180 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:11:04.180 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:11:04.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.284+0000 7f6752e8d640 1 -- 172.21.15.44:0/1251483713 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f674c068a40 msgr2=0x7f674c068e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.284+0000 7f6752e8d640 1 --2- 172.21.15.44:0/1251483713 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f674c068a40 0x7f674c068e20 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6748009900 tx=0x7f674802bc30 comp rx=0 tx=0).stop 2024-04-21T23:11:04.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.284+0000 7f6752e8d640 1 -- 172.21.15.44:0/1251483713 shutdown_connections 2024-04-21T23:11:04.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.284+0000 7f6752e8d640 1 --2- 172.21.15.44:0/1251483713 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f674c069360 0x7f674c10df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.284+0000 7f6752e8d640 1 --2- 172.21.15.44:0/1251483713 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f674c10e440 0x7f674c110830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.284+0000 7f6752e8d640 1 --2- 172.21.15.44:0/1251483713 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f674c068a40 0x7f674c068e20 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.284+0000 7f6752e8d640 1 -- 172.21.15.44:0/1251483713 >> 172.21.15.44:0/1251483713 conn(0x7f674c077430 msgr2=0x7f674c077840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:04.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.284+0000 7f6752e8d640 1 -- 172.21.15.44:0/1251483713 shutdown_connections 2024-04-21T23:11:04.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.284+0000 7f6752e8d640 1 -- 172.21.15.44:0/1251483713 wait complete. 2024-04-21T23:11:04.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.284+0000 7f6752e8d640 1 Processor -- start 2024-04-21T23:11:04.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.284+0000 7f6752e8d640 1 -- start start 2024-04-21T23:11:04.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.284+0000 7f6752e8d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f674c068a40 0x7f674c19a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.285+0000 7f6752e8d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f674c069360 0x7f674c19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.285+0000 7f6752e8d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f674c10e440 0x7f674c19ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.285+0000 7f6752e8d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f674c113d10 con 0x7f674c10e440 2024-04-21T23:11:04.286 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.285+0000 7f6752e8d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f674c113b90 con 0x7f674c068a40 2024-04-21T23:11:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.285+0000 7f6752e8d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f674c113e90 con 0x7f674c069360 2024-04-21T23:11:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.285+0000 7f675142a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f674c10e440 0x7f674c19ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.285+0000 7f6743fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f674c069360 0x7f674c19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.285+0000 7f675142a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f674c10e440 0x7f674c19ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43518/0 (socket says 172.21.15.44:43518) 2024-04-21T23:11:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.285+0000 7f675142a640 1 -- 172.21.15.44:0/1346790671 learned_addr learned my addr 172.21.15.44:0/1346790671 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.285+0000 7f6750c29640 1 --2- 172.21.15.44:0/1346790671 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f674c068a40 0x7f674c19a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.285+0000 7f675142a640 1 -- 172.21.15.44:0/1346790671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f674c069360 msgr2=0x7f674c19a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.285+0000 7f675142a640 1 --2- 172.21.15.44:0/1346790671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f674c069360 0x7f674c19a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.285+0000 7f675142a640 1 -- 172.21.15.44:0/1346790671 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f674c068a40 msgr2=0x7f674c19a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.285+0000 7f675142a640 1 --2- 172.21.15.44:0/1346790671 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f674c068a40 0x7f674c19a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.285+0000 7f675142a640 1 -- 172.21.15.44:0/1346790671 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674c19f180 con 0x7f674c10e440 2024-04-21T23:11:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.285+0000 7f675142a640 1 --2- 172.21.15.44:0/1346790671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f674c10e440 0x7f674c19ea50 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f674400ead0 tx=0x7f674400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.285+0000 7f6741ffb640 1 -- 172.21.15.44:0/1346790671 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6744004b20 con 0x7f674c10e440 2024-04-21T23:11:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.285+0000 7f6741ffb640 1 -- 172.21.15.44:0/1346790671 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6744004cc0 con 0x7f674c10e440 2024-04-21T23:11:04.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.285+0000 7f6752e8d640 1 -- 172.21.15.44:0/1346790671 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f674c19f470 con 0x7f674c10e440 2024-04-21T23:11:04.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.285+0000 7f6741ffb640 1 -- 172.21.15.44:0/1346790671 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6744010850 con 0x7f674c10e440 2024-04-21T23:11:04.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.286+0000 7f6752e8d640 1 -- 172.21.15.44:0/1346790671 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f674c1a6d00 con 0x7f674c10e440 2024-04-21T23:11:04.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.287+0000 7f6752e8d640 1 -- 172.21.15.44:0/1346790671 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f674c0685f0 con 0x7f674c10e440 2024-04-21T23:11:04.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.288+0000 7f6741ffb640 1 -- 172.21.15.44:0/1346790671 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6744005020 con 0x7f674c10e440 2024-04-21T23:11:04.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.288+0000 7f6741ffb640 1 --2- 172.21.15.44:0/1346790671 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f67340a2d00 0x7f67340a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.288+0000 7f6741ffb640 1 -- 172.21.15.44:0/1346790671 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7f67440cffa0 con 0x7f674c10e440 2024-04-21T23:11:04.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.289+0000 7f6750c29640 1 --2- 172.21.15.44:0/1346790671 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f67340a2d00 0x7f67340a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.289+0000 7f6750c29640 1 --2- 172.21.15.44:0/1346790671 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f67340a2d00 0x7f67340a51c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6748006010 tx=0x7f6748005dd0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:04.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.293+0000 7f6741ffb640 1 -- 172.21.15.44:0/1346790671 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f674409c820 con 0x7f674c10e440 2024-04-21T23:11:04.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:04 smithi044 ceph-mon[27934]: pgmap v70: 129 pgs: 129 active+clean; 40 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.1 MiB/s wr, 453 op/s 2024-04-21T23:11:04.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:04 smithi055 ceph-mon[31117]: pgmap v70: 129 pgs: 129 active+clean; 40 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.1 MiB/s wr, 453 op/s 2024-04-21T23:11:04.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:04 smithi069 ceph-mon[31816]: pgmap v70: 129 pgs: 129 active+clean; 40 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.1 MiB/s wr, 453 op/s 2024-04-21T23:11:04.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.519+0000 7f6752e8d640 1 -- 172.21.15.44:0/1346790671 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f674c02d170 con 0x7f674c10e440 2024-04-21T23:11:04.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.519+0000 7f6741ffb640 1 -- 172.21.15.44:0/1346790671 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11791 (secure 0 0 0) 0x7f674409c120 con 0x7f674c10e440 2024-04-21T23:11:04.519 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:11:04.519 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:10:44.145121+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25708":{"gid":25708,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/2494217106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":2494217106},{"type":"v1","addr":"172.21.15.69:6839","nonce":2494217106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:11:04.520 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 23 2024-04-21T23:11:04.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.522+0000 7f6752e8d640 1 -- 172.21.15.44:0/1346790671 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f67340a2d00 msgr2=0x7f67340a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.522+0000 7f6752e8d640 1 --2- 172.21.15.44:0/1346790671 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f67340a2d00 0x7f67340a51c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6748006010 tx=0x7f6748005dd0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.522+0000 7f6752e8d640 1 -- 172.21.15.44:0/1346790671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f674c10e440 msgr2=0x7f674c19ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.522+0000 7f6752e8d640 1 --2- 172.21.15.44:0/1346790671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f674c10e440 0x7f674c19ea50 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f674400ead0 tx=0x7f674400ef90 comp rx=0 tx=0).stop 2024-04-21T23:11:04.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.522+0000 7f6752e8d640 1 -- 172.21.15.44:0/1346790671 shutdown_connections 2024-04-21T23:11:04.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.522+0000 7f6752e8d640 1 --2- 172.21.15.44:0/1346790671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f674c069360 0x7f674c19a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.522+0000 7f6752e8d640 1 --2- 172.21.15.44:0/1346790671 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f67340a2d00 0x7f67340a51c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.522+0000 7f6752e8d640 1 --2- 172.21.15.44:0/1346790671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f674c10e440 0x7f674c19ea50 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.522+0000 7f6752e8d640 1 --2- 172.21.15.44:0/1346790671 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f674c068a40 0x7f674c19a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.522+0000 7f6752e8d640 1 -- 172.21.15.44:0/1346790671 >> 172.21.15.44:0/1346790671 conn(0x7f674c077430 msgr2=0x7f674c10f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:04.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.522+0000 7f6752e8d640 1 -- 172.21.15.44:0/1346790671 shutdown_connections 2024-04-21T23:11:04.522 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.522+0000 7f6752e8d640 1 -- 172.21.15.44:0/1346790671 wait complete. 2024-04-21T23:11:04.536 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:11:04.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.651+0000 7fabeaa40640 1 -- 172.21.15.44:0/4030550874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabe4101380 msgr2=0x7fabe4101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.651+0000 7fabeaa40640 1 --2- 172.21.15.44:0/4030550874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabe4101380 0x7fabe4101760 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fabd4009900 tx=0x7fabd402bc30 comp rx=0 tx=0).stop 2024-04-21T23:11:04.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.651+0000 7fabeaa40640 1 -- 172.21.15.44:0/4030550874 shutdown_connections 2024-04-21T23:11:04.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.651+0000 7fabeaa40640 1 --2- 172.21.15.44:0/4030550874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabe4101d30 0x7fabe41052d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.651+0000 7fabeaa40640 1 --2- 172.21.15.44:0/4030550874 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabe4105a00 0x7fabe410d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.651+0000 7fabeaa40640 1 --2- 172.21.15.44:0/4030550874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabe4101380 0x7fabe4101760 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.651+0000 7fabeaa40640 1 -- 172.21.15.44:0/4030550874 >> 172.21.15.44:0/4030550874 conn(0x7fabe4075960 msgr2=0x7fabe4075d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:04.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.651+0000 7fabeaa40640 1 -- 172.21.15.44:0/4030550874 shutdown_connections 2024-04-21T23:11:04.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.651+0000 7fabeaa40640 1 -- 172.21.15.44:0/4030550874 wait complete. 2024-04-21T23:11:04.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.652+0000 7fabeaa40640 1 Processor -- start 2024-04-21T23:11:04.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.652+0000 7fabeaa40640 1 -- start start 2024-04-21T23:11:04.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.652+0000 7fabeaa40640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabe4101380 0x7fabe4197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.652+0000 7fabeaa40640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabe4105a00 0x7fabe41984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.652+0000 7fabeaa40640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabe419c850 0x7fabe419cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.652+0000 7fabeaa40640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fabe4110a20 con 0x7fabe4101380 2024-04-21T23:11:04.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.652+0000 7fabeaa40640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fabe41108a0 con 0x7fabe4105a00 2024-04-21T23:11:04.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.652+0000 7fabeaa40640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fabe4110ba0 con 0x7fabe419c850 2024-04-21T23:11:04.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.652+0000 7fabe3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabe4101380 0x7fabe4197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.652+0000 7fabe3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabe4101380 0x7fabe4197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43536/0 (socket says 172.21.15.44:43536) 2024-04-21T23:11:04.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.652+0000 7fabe3fff640 1 -- 172.21.15.44:0/1513930675 learned_addr learned my addr 172.21.15.44:0/1513930675 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:04.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.652+0000 7fabe8fdd640 1 --2- 172.21.15.44:0/1513930675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabe419c850 0x7fabe419cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.652+0000 7fabe37fe640 1 --2- 172.21.15.44:0/1513930675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabe4105a00 0x7fabe41984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.653+0000 7fabe3fff640 1 -- 172.21.15.44:0/1513930675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabe419c850 msgr2=0x7fabe419cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.653+0000 7fabe3fff640 1 --2- 172.21.15.44:0/1513930675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabe419c850 0x7fabe419cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.653+0000 7fabe3fff640 1 -- 172.21.15.44:0/1513930675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabe4105a00 msgr2=0x7fabe41984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.653+0000 7fabe3fff640 1 --2- 172.21.15.44:0/1513930675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabe4105a00 0x7fabe41984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.653+0000 7fabe3fff640 1 -- 172.21.15.44:0/1513930675 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe419d340 con 0x7fabe4101380 2024-04-21T23:11:04.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.653+0000 7fabe8fdd640 1 --2- 172.21.15.44:0/1513930675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabe419c850 0x7fabe419cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:04.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.653+0000 7fabe37fe640 1 --2- 172.21.15.44:0/1513930675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabe4105a00 0x7fabe41984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:04.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.653+0000 7fabe3fff640 1 --2- 172.21.15.44:0/1513930675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabe4101380 0x7fabe4197f90 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fabd4006010 tx=0x7fabd402cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:04.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.653+0000 7fabe17fa640 1 -- 172.21.15.44:0/1513930675 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd4041070 con 0x7fabe4101380 2024-04-21T23:11:04.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.653+0000 7fabe17fa640 1 -- 172.21.15.44:0/1513930675 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabd4004450 con 0x7fabe4101380 2024-04-21T23:11:04.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.653+0000 7fabeaa40640 1 -- 172.21.15.44:0/1513930675 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabe419d5d0 con 0x7fabe4101380 2024-04-21T23:11:04.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.653+0000 7fabe17fa640 1 -- 172.21.15.44:0/1513930675 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd4003e00 con 0x7fabe4101380 2024-04-21T23:11:04.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.653+0000 7fabeaa40640 1 -- 172.21.15.44:0/1513930675 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabe41a4d90 con 0x7fabe4101380 2024-04-21T23:11:04.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.656+0000 7fabed62d740 1 Processor -- start 2024-04-21T23:11:04.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.656+0000 7fabe17fa640 1 -- 172.21.15.44:0/1513930675 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fabd4044070 con 0x7fabe4101380 2024-04-21T23:11:04.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.656+0000 7fabed62d740 1 -- start start 2024-04-21T23:11:04.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.656+0000 7fabed62d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55717e78d020 0x55717e78d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.656+0000 7fabed62d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d940 0x55717e795fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.656+0000 7fabed62d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e796520 0x55717e798910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.656+0000 7fabed62d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55717e6f1530 con 0x55717e796520 2024-04-21T23:11:04.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.656+0000 7fabed62d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55717e699830 con 0x55717e78d940 2024-04-21T23:11:04.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.656+0000 7fabed62d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55717e6ed4f0 con 0x55717e78d020 2024-04-21T23:11:04.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.657+0000 7fabe37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e796520 0x55717e798910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.657+0000 7fabe3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d940 0x55717e795fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.657+0000 7fabe37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e796520 0x55717e798910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43544/0 (socket says 172.21.15.44:43544) 2024-04-21T23:11:04.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.657+0000 7fabe37fe640 1 -- 172.21.15.44:0/2401628965 learned_addr learned my addr 172.21.15.44:0/2401628965 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:04.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.657+0000 7fabe8fdd640 1 --2- 172.21.15.44:0/2401628965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55717e78d020 0x55717e78d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.657+0000 7fabe37fe640 1 -- 172.21.15.44:0/2401628965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55717e78d020 msgr2=0x55717e78d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:04.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.657+0000 7fabe37fe640 1 --2- 172.21.15.44:0/2401628965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55717e78d020 0x55717e78d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.657+0000 7fabe37fe640 1 -- 172.21.15.44:0/2401628965 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d940 msgr2=0x55717e795fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:04.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.657+0000 7fabe37fe640 1 --2- 172.21.15.44:0/2401628965 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d940 0x55717e795fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.657+0000 7fabe37fe640 1 -- 172.21.15.44:0/2401628965 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55717e6bd2c0 con 0x55717e796520 2024-04-21T23:11:04.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.657+0000 7fabe3fff640 1 --2- 172.21.15.44:0/2401628965 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d940 0x55717e795fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:04.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.657+0000 7fabe17fa640 1 --2- 172.21.15.44:0/1513930675 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fabcc0a2d00 0x7fabcc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.657+0000 7fabe8fdd640 1 --2- 172.21.15.44:0/1513930675 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fabcc0a2d00 0x7fabcc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.657+0000 7fabe37fe640 1 --2- 172.21.15.44:0/2401628965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e796520 0x55717e798910 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fabd8011470 tx=0x7fabd8011930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=604ea1ba937eda71 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:04.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.657+0000 7fabe17fa640 1 -- 172.21.15.44:0/1513930675 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7fabd40f1150 con 0x7fabe4101380 2024-04-21T23:11:04.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.657+0000 7fabca7fc640 1 -- 172.21.15.44:0/2401628965 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd801e070 con 0x55717e796520 2024-04-21T23:11:04.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.657+0000 7fabca7fc640 1 -- 172.21.15.44:0/2401628965 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabd8004310 con 0x55717e796520 2024-04-21T23:11:04.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.657+0000 7fabca7fc640 1 -- 172.21.15.44:0/2401628965 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd800f9a0 con 0x55717e796520 2024-04-21T23:11:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.658+0000 7fabe8fdd640 1 --2- 172.21.15.44:0/1513930675 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fabcc0a2d00 0x7fabcc0a51c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fabdc009fd0 tx=0x7fabdc009500 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.658+0000 7fabed62d740 1 -- 172.21.15.44:0/2401628965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e796520 msgr2=0x55717e798910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.658+0000 7fabed62d740 1 --2- 172.21.15.44:0/2401628965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e796520 0x55717e798910 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fabd8011470 tx=0x7fabd8011930 comp rx=0 tx=0).stop 2024-04-21T23:11:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.658+0000 7fabed62d740 1 -- 172.21.15.44:0/2401628965 shutdown_connections 2024-04-21T23:11:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.658+0000 7fabed62d740 1 --2- 172.21.15.44:0/2401628965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e796520 0x55717e798910 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.658+0000 7fabed62d740 1 --2- 172.21.15.44:0/2401628965 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d940 0x55717e795fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.658+0000 7fabed62d740 1 --2- 172.21.15.44:0/2401628965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55717e78d020 0x55717e78d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.658+0000 7fabed62d740 1 -- 172.21.15.44:0/2401628965 >> 172.21.15.44:0/2401628965 conn(0x55717e787750 msgr2=0x55717e787b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.658+0000 7fabed62d740 1 -- 172.21.15.44:0/2401628965 shutdown_connections 2024-04-21T23:11:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.658+0000 7fabed62d740 1 -- 172.21.15.44:0/2401628965 wait complete. 2024-04-21T23:11:04.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.658+0000 7fabed62d740 1 -- start start 2024-04-21T23:11:04.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.659+0000 7fabed62d740 1 Processor -- start 2024-04-21T23:11:04.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.659+0000 7fabed62d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d020 0x55717e795ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.659+0000 7fabed62d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e78d940 0x55717e793230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.659+0000 7fabed62d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55717e796520 0x55717e7937b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.659+0000 7fabed62d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55717e799310 con 0x55717e78d940 2024-04-21T23:11:04.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.659+0000 7fabed62d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55717e6acf10 con 0x55717e78d020 2024-04-21T23:11:04.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.659+0000 7fabed62d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55717e799490 con 0x55717e796520 2024-04-21T23:11:04.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.659+0000 7fabed62d740 10 client.0 _ll_register_callbacks cb 0x55717e404be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:04.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.659+0000 7fabe37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e78d940 0x55717e793230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.659+0000 7fabe3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d020 0x55717e795ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.659+0000 7fabe37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e78d940 0x55717e793230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43560/0 (socket says 172.21.15.44:43560) 2024-04-21T23:11:04.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.659+0000 7fabe37fe640 1 -- 172.21.15.44:0/2820235878 learned_addr learned my addr 172.21.15.44:0/2820235878 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:04.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.659+0000 7fabe8fdd640 1 --2- 172.21.15.44:0/2820235878 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55717e796520 0x55717e7937b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.659+0000 7fabe3fff640 1 -- 172.21.15.44:0/2820235878 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55717e796520 msgr2=0x55717e7937b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:04.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.659+0000 7fabe3fff640 1 --2- 172.21.15.44:0/2820235878 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55717e796520 0x55717e7937b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.659+0000 7fabe3fff640 1 -- 172.21.15.44:0/2820235878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e78d940 msgr2=0x55717e793230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:04.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.659+0000 7fabe3fff640 1 --2- 172.21.15.44:0/2820235878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e78d940 0x55717e793230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.659+0000 7fabe3fff640 1 -- 172.21.15.44:0/2820235878 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55717e67ec80 con 0x55717e78d020 2024-04-21T23:11:04.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.660+0000 7fabe3fff640 1 --2- 172.21.15.44:0/2820235878 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d020 0x55717e795ea0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fabd40ee610 tx=0x7fabd40f4990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0c7be77b037edbf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:04.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.660+0000 7fabc37fe640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:11:04.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.660+0000 7fabc37fe640 1 -- 172.21.15.44:0/2820235878 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd404b040 con 0x55717e78d020 2024-04-21T23:11:04.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.660+0000 7fabed62d740 1 -- 172.21.15.44:0/2820235878 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabd8004310 con 0x55717e78d020 2024-04-21T23:11:04.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.660+0000 7fabc37fe640 1 -- 172.21.15.44:0/2820235878 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7fabd40b9b00 con 0x55717e78d020 2024-04-21T23:11:04.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.660+0000 7fabc37fe640 1 -- 172.21.15.44:0/2820235878 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fabd40b9730 con 0x55717e78d020 2024-04-21T23:11:04.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.660+0000 7fabed62d740 10 client.25721 fetch_fsmap learned FSMap version 23 2024-04-21T23:11:04.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.660+0000 7fabed62d740 1 -- 172.21.15.44:0/2820235878 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55717e4053f0 con 0x55717e78d020 2024-04-21T23:11:04.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.660+0000 7fabc37fe640 1 -- 172.21.15.44:0/2820235878 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabd8004310 con 0x55717e78d020 2024-04-21T23:11:04.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.660+0000 7fabc37fe640 1 -- 172.21.15.44:0/2820235878 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fabd40c5bc0 con 0x55717e78d020 2024-04-21T23:11:04.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.660+0000 7fabed62d740 10 client.25721 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T23:11:04.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.660+0000 7fabed62d740 10 client.25721 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:04.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.660+0000 7fabed62d740 20 client.25721 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:04.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.660+0000 7fabed62d740 1 --2- 172.21.15.44:0/2820235878 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55717e7a6510 0x55717e7a89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.660+0000 7fabed62d740 4 client.25721 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:11:04.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.660+0000 7fabed62d740 1 -- 172.21.15.44:0/2820235878 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55717e6fea70 con 0x55717e7a6510 2024-04-21T23:11:04.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.661+0000 7fabe37fe640 1 --2- 172.21.15.44:0/2820235878 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55717e7a6510 0x55717e7a89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.661+0000 7fabc37fe640 10 client.25721 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:04.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.661+0000 7fabe37fe640 1 --2- 172.21.15.44:0/2820235878 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55717e7a6510 0x55717e7a89a0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4461c2c9cb2442e server_cookie=28ce013eb63004b8 in_seq=0 out_seq=0 2024-04-21T23:11:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.662+0000 7fabc37fe640 1 -- 172.21.15.44:0/2820235878 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55717e6fea70 con 0x55717e7a6510 2024-04-21T23:11:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.662+0000 7fabc37fe640 10 client.25721 handle_command_reply: tid=0 2024-04-21T23:11:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.662+0000 7fabed62d740 1 client.25721 shutdown 2024-04-21T23:11:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.662+0000 7fabed62d740 1 -- 172.21.15.44:0/2820235878 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d020 msgr2=0x55717e795ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.662+0000 7fabed62d740 1 --2- 172.21.15.44:0/2820235878 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d020 0x55717e795ea0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fabd40ee610 tx=0x7fabd40f4990 comp rx=0 tx=0).stop 2024-04-21T23:11:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.662+0000 7fabed62d740 1 -- 172.21.15.44:0/2820235878 shutdown_connections 2024-04-21T23:11:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.662+0000 7fabed62d740 1 --2- 172.21.15.44:0/2820235878 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55717e7a6510 0x55717e7a89a0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.662+0000 7fabed62d740 1 --2- 172.21.15.44:0/2820235878 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55717e796520 0x55717e7937b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.662+0000 7fabed62d740 1 --2- 172.21.15.44:0/2820235878 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e78d940 0x55717e793230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.662+0000 7fabed62d740 1 --2- 172.21.15.44:0/2820235878 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d020 0x55717e795ea0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.662+0000 7fabc37fe640 0 client.25721 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:04.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.662+0000 7fabed62d740 1 -- 172.21.15.44:0/2820235878 >> 172.21.15.44:0/2820235878 conn(0x55717e787750 msgr2=0x55717e7894a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:04.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.662+0000 7fabed62d740 1 -- 172.21.15.44:0/2820235878 shutdown_connections 2024-04-21T23:11:04.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.662+0000 7fabed62d740 1 -- 172.21.15.44:0/2820235878 wait complete. 2024-04-21T23:11:04.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.662+0000 7fabed62d740 20 client.25721 trim_cache size 0 max 16384 2024-04-21T23:11:04.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.675+0000 7fabed62d740 1 Processor -- start 2024-04-21T23:11:04.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.675+0000 7fabed62d740 1 -- start start 2024-04-21T23:11:04.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.676+0000 7fabed62d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d020 0x55717e7a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.676+0000 7fabed62d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55717e78d940 0x55717e795960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.676+0000 7fabed62d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e796520 0x55717e795ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.676+0000 7fabed62d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fabe41aca40 con 0x55717e796520 2024-04-21T23:11:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.676+0000 7fabed62d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fabe41af2c0 con 0x55717e78d020 2024-04-21T23:11:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.676+0000 7fabed62d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fabe41ad0b0 con 0x55717e78d940 2024-04-21T23:11:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.676+0000 7fabe8fdd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e796520 0x55717e795ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.676+0000 7fabe8fdd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e796520 0x55717e795ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43568/0 (socket says 172.21.15.44:43568) 2024-04-21T23:11:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.676+0000 7fabe8fdd640 1 -- 172.21.15.44:0/3306960093 learned_addr learned my addr 172.21.15.44:0/3306960093 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.676+0000 7fabe37fe640 1 --2- 172.21.15.44:0/3306960093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55717e78d940 0x55717e795960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.676+0000 7fabe3fff640 1 --2- 172.21.15.44:0/3306960093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d020 0x55717e7a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.676+0000 7fabe8fdd640 1 -- 172.21.15.44:0/3306960093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55717e78d940 msgr2=0x55717e795960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.676+0000 7fabe8fdd640 1 --2- 172.21.15.44:0/3306960093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55717e78d940 0x55717e795960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.676+0000 7fabe8fdd640 1 -- 172.21.15.44:0/3306960093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d020 msgr2=0x55717e7a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.676+0000 7fabe8fdd640 1 --2- 172.21.15.44:0/3306960093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d020 0x55717e7a21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.676+0000 7fabe8fdd640 1 -- 172.21.15.44:0/3306960093 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55717e4cf7d0 con 0x55717e796520 2024-04-21T23:11:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.676+0000 7fabe8fdd640 1 --2- 172.21.15.44:0/3306960093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e796520 0x55717e795ea0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fabdc003a70 tx=0x7fabdc012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf347cc1210c4f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.676+0000 7fabc9ffb640 1 -- 172.21.15.44:0/3306960093 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabdc004b20 con 0x55717e796520 2024-04-21T23:11:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.676+0000 7fabc9ffb640 1 -- 172.21.15.44:0/3306960093 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabdc004cc0 con 0x55717e796520 2024-04-21T23:11:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.676+0000 7fabc9ffb640 1 -- 172.21.15.44:0/3306960093 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabdc005530 con 0x55717e796520 2024-04-21T23:11:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.677+0000 7fabed62d740 1 -- 172.21.15.44:0/3306960093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e796520 msgr2=0x55717e795ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.677+0000 7fabed62d740 1 --2- 172.21.15.44:0/3306960093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e796520 0x55717e795ea0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fabdc003a70 tx=0x7fabdc012000 comp rx=0 tx=0).stop 2024-04-21T23:11:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.677+0000 7fabed62d740 1 -- 172.21.15.44:0/3306960093 shutdown_connections 2024-04-21T23:11:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.677+0000 7fabed62d740 1 --2- 172.21.15.44:0/3306960093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55717e78d940 0x55717e795960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.677+0000 7fabed62d740 1 --2- 172.21.15.44:0/3306960093 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e796520 0x55717e795ea0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.677+0000 7fabed62d740 1 --2- 172.21.15.44:0/3306960093 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d020 0x55717e7a21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.677+0000 7fabed62d740 1 -- 172.21.15.44:0/3306960093 >> 172.21.15.44:0/3306960093 conn(0x55717e787750 msgr2=0x55717e78e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.677+0000 7fabed62d740 1 -- 172.21.15.44:0/3306960093 shutdown_connections 2024-04-21T23:11:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.677+0000 7fabed62d740 1 -- 172.21.15.44:0/3306960093 wait complete. 2024-04-21T23:11:04.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.677+0000 7fabed62d740 1 -- start start 2024-04-21T23:11:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.678+0000 7fabed62d740 1 Processor -- start 2024-04-21T23:11:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.678+0000 7fabed62d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e78d020 0x55717e7900f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.678+0000 7fabed62d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d940 0x55717e790630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.678+0000 7fabed62d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55717e796520 0x55717e790b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.678+0000 7fabed62d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fabe41ad780 con 0x55717e78d020 2024-04-21T23:11:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.678+0000 7fabed62d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fabe41aebf0 con 0x55717e78d940 2024-04-21T23:11:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.678+0000 7fabed62d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fabe41ade50 con 0x55717e796520 2024-04-21T23:11:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.678+0000 7fabed62d740 10 client.0 _ll_register_callbacks cb 0x55717e404be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.678+0000 7fabe3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e78d020 0x55717e7900f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.678+0000 7fabe3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e78d020 0x55717e7900f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43576/0 (socket says 172.21.15.44:43576) 2024-04-21T23:11:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.678+0000 7fabe3fff640 1 -- 172.21.15.44:0/2675170858 learned_addr learned my addr 172.21.15.44:0/2675170858 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.678+0000 7fabe37fe640 1 --2- 172.21.15.44:0/2675170858 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d940 0x55717e790630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.678+0000 7fabe8fdd640 1 --2- 172.21.15.44:0/2675170858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55717e796520 0x55717e790b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.678+0000 7fabe3fff640 1 -- 172.21.15.44:0/2675170858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55717e796520 msgr2=0x55717e790b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.678+0000 7fabe3fff640 1 --2- 172.21.15.44:0/2675170858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55717e796520 0x55717e790b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.678+0000 7fabe3fff640 1 -- 172.21.15.44:0/2675170858 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d940 msgr2=0x55717e790630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.678+0000 7fabe3fff640 1 --2- 172.21.15.44:0/2675170858 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d940 0x55717e790630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.678+0000 7fabe3fff640 1 -- 172.21.15.44:0/2675170858 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55717e80bc90 con 0x55717e78d020 2024-04-21T23:11:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.678+0000 7fabe3fff640 1 --2- 172.21.15.44:0/2675170858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e78d020 0x55717e7900f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fabd40bd190 tx=0x7fabd4005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27980fe5c217b4c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.678+0000 7fabc37fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:11:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.679+0000 7fabc37fe640 1 -- 172.21.15.44:0/2675170858 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd404b040 con 0x55717e78d020 2024-04-21T23:11:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.679+0000 7fabc37fe640 1 -- 172.21.15.44:0/2675170858 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7fabd40bc980 con 0x55717e78d020 2024-04-21T23:11:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.679+0000 7fabed62d740 1 -- 172.21.15.44:0/2675170858 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabdc004cc0 con 0x55717e78d020 2024-04-21T23:11:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.679+0000 7fabc37fe640 1 -- 172.21.15.44:0/2675170858 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fabd40bcdf0 con 0x55717e78d020 2024-04-21T23:11:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.679+0000 7fabed62d740 10 client.16155 fetch_fsmap learned FSMap version 23 2024-04-21T23:11:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.679+0000 7fabed62d740 1 -- 172.21.15.44:0/2675170858 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55717e80cf70 con 0x55717e78d020 2024-04-21T23:11:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.679+0000 7fabc37fe640 1 -- 172.21.15.44:0/2675170858 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabdc004cc0 con 0x55717e78d020 2024-04-21T23:11:04.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.679+0000 7fabc37fe640 1 -- 172.21.15.44:0/2675170858 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7fabd40d6120 con 0x55717e78d020 2024-04-21T23:11:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.679+0000 7fabed62d740 10 client.16155 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T23:11:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.679+0000 7fabed62d740 10 client.16155 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.679+0000 7fabed62d740 20 client.16155 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.679+0000 7fabed62d740 1 --2- 172.21.15.44:0/2675170858 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55717e80d510 0x55717e80f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.679+0000 7fabed62d740 4 client.16155 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:11:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.679+0000 7fabed62d740 1 -- 172.21.15.44:0/2675170858 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55717e6f8dc0 con 0x55717e80d510 2024-04-21T23:11:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.679+0000 7fabe37fe640 1 --2- 172.21.15.44:0/2675170858 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55717e80d510 0x55717e80f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.680+0000 7fabe37fe640 1 --2- 172.21.15.44:0/2675170858 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55717e80d510 0x55717e80f930 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4672cc651bfffa7 server_cookie=92af475e98325ed2 in_seq=0 out_seq=0 2024-04-21T23:11:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.680+0000 7fabc37fe640 10 client.16155 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.680+0000 7fabc37fe640 1 -- 172.21.15.44:0/2675170858 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55717e6f8dc0 con 0x55717e80d510 2024-04-21T23:11:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.680+0000 7fabc37fe640 10 client.16155 handle_command_reply: tid=0 2024-04-21T23:11:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.680+0000 7fabed62d740 1 client.16155 shutdown 2024-04-21T23:11:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.680+0000 7fabed62d740 1 -- 172.21.15.44:0/2675170858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e78d020 msgr2=0x55717e7900f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.680+0000 7fabed62d740 1 --2- 172.21.15.44:0/2675170858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e78d020 0x55717e7900f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fabd40bd190 tx=0x7fabd4005bd0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.680+0000 7fabed62d740 1 -- 172.21.15.44:0/2675170858 shutdown_connections 2024-04-21T23:11:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.680+0000 7fabed62d740 1 --2- 172.21.15.44:0/2675170858 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55717e80d510 0x55717e80f930 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.680+0000 7fabed62d740 1 --2- 172.21.15.44:0/2675170858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55717e796520 0x55717e790b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.680+0000 7fabed62d740 1 --2- 172.21.15.44:0/2675170858 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55717e78d940 0x55717e790630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.680+0000 7fabed62d740 1 --2- 172.21.15.44:0/2675170858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55717e78d020 0x55717e7900f0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.680+0000 7fabc37fe640 0 client.16155 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.680+0000 7fabed62d740 1 -- 172.21.15.44:0/2675170858 >> 172.21.15.44:0/2675170858 conn(0x55717e787750 msgr2=0x55717e785fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.681+0000 7fabed62d740 1 -- 172.21.15.44:0/2675170858 shutdown_connections 2024-04-21T23:11:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.681+0000 7fabed62d740 1 -- 172.21.15.44:0/2675170858 wait complete. 2024-04-21T23:11:04.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.681+0000 7fabed62d740 20 client.16155 trim_cache size 0 max 16384 2024-04-21T23:11:04.689 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-21T23:11:04.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.681+0000 7fabeaa40640 1 -- 172.21.15.44:0/1513930675 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fabcc0a2d00 msgr2=0x7fabcc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.681+0000 7fabeaa40640 1 --2- 172.21.15.44:0/1513930675 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fabcc0a2d00 0x7fabcc0a51c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fabdc009fd0 tx=0x7fabdc009500 comp rx=0 tx=0).stop 2024-04-21T23:11:04.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.681+0000 7fabeaa40640 1 -- 172.21.15.44:0/1513930675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabe4101380 msgr2=0x7fabe4197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:04.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.681+0000 7fabeaa40640 1 --2- 172.21.15.44:0/1513930675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabe4101380 0x7fabe4197f90 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fabd4006010 tx=0x7fabd402cb30 comp rx=0 tx=0).stop 2024-04-21T23:11:04.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.681+0000 7fabeaa40640 1 -- 172.21.15.44:0/1513930675 shutdown_connections 2024-04-21T23:11:04.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.681+0000 7fabeaa40640 1 --2- 172.21.15.44:0/1513930675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabe419c850 0x7fabe419cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.681+0000 7fabeaa40640 1 --2- 172.21.15.44:0/1513930675 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fabcc0a2d00 0x7fabcc0a51c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.681+0000 7fabeaa40640 1 --2- 172.21.15.44:0/1513930675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fabe4105a00 0x7fabe41984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.681+0000 7fabeaa40640 1 --2- 172.21.15.44:0/1513930675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fabe4101380 0x7fabe4197f90 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:04.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.681+0000 7fabeaa40640 1 -- 172.21.15.44:0/1513930675 >> 172.21.15.44:0/1513930675 conn(0x7fabe4075960 msgr2=0x7fabe4104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:04.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.681+0000 7fabeaa40640 1 -- 172.21.15.44:0/1513930675 shutdown_connections 2024-04-21T23:11:04.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:04.681+0000 7fabeaa40640 1 -- 172.21.15.44:0/1513930675 wait complete. 2024-04-21T23:11:04.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 146383 14.92 MB/sec warmup 54 sec latency 16.851 ms 2024-04-21T23:11:05.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:05 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1346790671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:11:05.463 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:05 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1346790671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:11:05.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:05 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1346790671' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:11:05.692 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:11:05.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:11:05.693 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:11:05.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 149486 14.98 MB/sec warmup 55 sec latency 24.416 ms 2024-04-21T23:11:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.806+0000 7f9091172640 1 -- 172.21.15.44:0/4007585949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f908c10bcf0 msgr2=0x7f908c10e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.806+0000 7f9091172640 1 --2- 172.21.15.44:0/4007585949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f908c10bcf0 0x7f908c10e190 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f9084009ca0 tx=0x7f908402c180 comp rx=0 tx=0).stop 2024-04-21T23:11:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.806+0000 7f9091172640 1 -- 172.21.15.44:0/4007585949 shutdown_connections 2024-04-21T23:11:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.806+0000 7f9091172640 1 --2- 172.21.15.44:0/4007585949 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f908c10bcf0 0x7f908c10e190 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.806+0000 7f9091172640 1 --2- 172.21.15.44:0/4007585949 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f908c1032d0 0x7f908c10b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.806+0000 7f9091172640 1 --2- 172.21.15.44:0/4007585949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f908c1029b0 0x7f908c102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.806+0000 7f9091172640 1 -- 172.21.15.44:0/4007585949 >> 172.21.15.44:0/4007585949 conn(0x7f908c0777f0 msgr2=0x7f908c077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:05.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.806+0000 7f9091172640 1 -- 172.21.15.44:0/4007585949 shutdown_connections 2024-04-21T23:11:05.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.806+0000 7f9091172640 1 -- 172.21.15.44:0/4007585949 wait complete. 2024-04-21T23:11:05.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.807+0000 7f9091172640 1 Processor -- start 2024-04-21T23:11:05.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.807+0000 7f9091172640 1 -- start start 2024-04-21T23:11:05.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.807+0000 7f9091172640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f908c1029b0 0x7f908c198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.807+0000 7f9091172640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f908c1032d0 0x7f908c198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.807+0000 7f9091172640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f908c10bcf0 0x7f908c19c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.807+0000 7f9091172640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f908c111af0 con 0x7f908c1029b0 2024-04-21T23:11:05.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.807+0000 7f9091172640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f908c111970 con 0x7f908c10bcf0 2024-04-21T23:11:05.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.807+0000 7f9091172640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f908c111c70 con 0x7f908c1032d0 2024-04-21T23:11:05.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.808+0000 7f908ad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f908c1029b0 0x7f908c198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.808+0000 7f908ad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f908c1029b0 0x7f908c198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43596/0 (socket says 172.21.15.44:43596) 2024-04-21T23:11:05.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.808+0000 7f908ad9d640 1 -- 172.21.15.44:0/4037520254 learned_addr learned my addr 172.21.15.44:0/4037520254 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:05.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.808+0000 7f908b59e640 1 --2- 172.21.15.44:0/4037520254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f908c10bcf0 0x7f908c19c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.808+0000 7f9083fff640 1 --2- 172.21.15.44:0/4037520254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f908c1032d0 0x7f908c198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.808+0000 7f908b59e640 1 -- 172.21.15.44:0/4037520254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f908c1032d0 msgr2=0x7f908c198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:05.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.808+0000 7f908b59e640 1 --2- 172.21.15.44:0/4037520254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f908c1032d0 0x7f908c198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.808+0000 7f908b59e640 1 -- 172.21.15.44:0/4037520254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f908c1029b0 msgr2=0x7f908c198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:05.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.808+0000 7f908b59e640 1 --2- 172.21.15.44:0/4037520254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f908c1029b0 0x7f908c198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.808+0000 7f908b59e640 1 -- 172.21.15.44:0/4037520254 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f908c19cfb0 con 0x7f908c10bcf0 2024-04-21T23:11:05.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.808+0000 7f9083fff640 1 --2- 172.21.15.44:0/4037520254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f908c1032d0 0x7f908c198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:05.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.808+0000 7f908ad9d640 1 --2- 172.21.15.44:0/4037520254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f908c1029b0 0x7f908c198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:05.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.808+0000 7f908b59e640 1 --2- 172.21.15.44:0/4037520254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f908c10bcf0 0x7f908c19c880 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f908402c790 tx=0x7f9084035e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:05.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.809+0000 7f9088d99640 1 -- 172.21.15.44:0/4037520254 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9084039040 con 0x7f908c10bcf0 2024-04-21T23:11:05.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.809+0000 7f9088d99640 1 -- 172.21.15.44:0/4037520254 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9084044070 con 0x7f908c10bcf0 2024-04-21T23:11:05.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.809+0000 7f9091172640 1 -- 172.21.15.44:0/4037520254 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f908c19d240 con 0x7f908c10bcf0 2024-04-21T23:11:05.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.809+0000 7f9088d99640 1 -- 172.21.15.44:0/4037520254 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90840328d0 con 0x7f908c10bcf0 2024-04-21T23:11:05.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.809+0000 7f9091172640 1 -- 172.21.15.44:0/4037520254 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f908c1a4b30 con 0x7f908c10bcf0 2024-04-21T23:11:05.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.811+0000 7f9088d99640 1 -- 172.21.15.44:0/4037520254 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f9084032a70 con 0x7f908c10bcf0 2024-04-21T23:11:05.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.811+0000 7f9093e31740 1 Processor -- start 2024-04-21T23:11:05.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.811+0000 7f9093e31740 1 -- start start 2024-04-21T23:11:05.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.811+0000 7f9093e31740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7f4720 0x55e8cb7f3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.811+0000 7f9093e31740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f3a40 0x55e8cb7f3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.811+0000 7f9093e31740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7fcdb0 0x55e8cb7ff1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.811+0000 7f9093e31740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e8cb757170 con 0x55e8cb7fcdb0 2024-04-21T23:11:05.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.811+0000 7f9093e31740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55e8cb6ff7c0 con 0x55e8cb7f4720 2024-04-21T23:11:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.811+0000 7f9093e31740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e8cb753130 con 0x55e8cb7f3a40 2024-04-21T23:11:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.812+0000 7f9083fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7f4720 0x55e8cb7f3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.812+0000 7f9083fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7f4720 0x55e8cb7f3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:48866/0 (socket says 172.21.15.44:48866) 2024-04-21T23:11:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.812+0000 7f9083fff640 1 -- 172.21.15.44:0/4043169342 learned_addr learned my addr 172.21.15.44:0/4043169342 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.812+0000 7f908ad9d640 1 --2- 172.21.15.44:0/4043169342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f3a40 0x55e8cb7f3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.812+0000 7f9083fff640 1 --2- 172.21.15.44:0/4043169342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7fcdb0 0x55e8cb7ff1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.812+0000 7f9088d99640 1 --2- 172.21.15.44:0/4037520254 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f90780a2d00 0x7f90780a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.812+0000 7f908ad9d640 1 -- 172.21.15.44:0/4043169342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7f4720 msgr2=0x55e8cb7f3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.812+0000 7f908ad9d640 1 --2- 172.21.15.44:0/4043169342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7f4720 0x55e8cb7f3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.812+0000 7f908ad9d640 1 -- 172.21.15.44:0/4043169342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7fcdb0 msgr2=0x55e8cb7ff1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.812+0000 7f908ad9d640 1 --2- 172.21.15.44:0/4043169342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7fcdb0 0x55e8cb7ff1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.812+0000 7f908ad9d640 1 -- 172.21.15.44:0/4043169342 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8cb710e60 con 0x55e8cb7f3a40 2024-04-21T23:11:05.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.812+0000 7f9083fff640 1 --2- 172.21.15.44:0/4043169342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7fcdb0 0x55e8cb7ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:11:05.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.812+0000 7f908b59e640 1 --2- 172.21.15.44:0/4037520254 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f90780a2d00 0x7f90780a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.812+0000 7f9088d99640 1 -- 172.21.15.44:0/4037520254 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7f90840f1a60 con 0x7f908c10bcf0 2024-04-21T23:11:05.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.813+0000 7f908ad9d640 1 --2- 172.21.15.44:0/4043169342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f3a40 0x55e8cb7f3ec0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f907c011510 tx=0x7f907c0119d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad082dc733cccf33 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:05.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.813+0000 7f908b59e640 1 --2- 172.21.15.44:0/4037520254 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f90780a2d00 0x7f90780a51c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f90840ef410 tx=0x7f9084044880 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:05.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.813+0000 7f90817fa640 1 -- 172.21.15.44:0/4043169342 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c01e070 con 0x55e8cb7f3a40 2024-04-21T23:11:05.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.813+0000 7f90817fa640 1 -- 172.21.15.44:0/4043169342 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f907c00faa0 con 0x55e8cb7f3a40 2024-04-21T23:11:05.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.813+0000 7f90817fa640 1 -- 172.21.15.44:0/4043169342 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c019bd0 con 0x55e8cb7f3a40 2024-04-21T23:11:05.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.814+0000 7f9093e31740 1 -- 172.21.15.44:0/4043169342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f3a40 msgr2=0x55e8cb7f3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:05.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.814+0000 7f9093e31740 1 --2- 172.21.15.44:0/4043169342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f3a40 0x55e8cb7f3ec0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f907c011510 tx=0x7f907c0119d0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.814+0000 7f9093e31740 1 -- 172.21.15.44:0/4043169342 shutdown_connections 2024-04-21T23:11:05.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.814+0000 7f9093e31740 1 --2- 172.21.15.44:0/4043169342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f3a40 0x55e8cb7f3ec0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.814+0000 7f9093e31740 1 --2- 172.21.15.44:0/4043169342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7fcdb0 0x55e8cb7ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.814+0000 7f9093e31740 1 --2- 172.21.15.44:0/4043169342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7f4720 0x55e8cb7f3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.814+0000 7f9093e31740 1 -- 172.21.15.44:0/4043169342 >> 172.21.15.44:0/4043169342 conn(0x55e8cb7ba4f0 msgr2=0x55e8cb7ee490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:05.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.814+0000 7f9093e31740 1 -- 172.21.15.44:0/4043169342 shutdown_connections 2024-04-21T23:11:05.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.814+0000 7f9093e31740 1 -- 172.21.15.44:0/4043169342 wait complete. 2024-04-21T23:11:05.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.814+0000 7f9093e31740 1 -- start start 2024-04-21T23:11:05.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.815+0000 7f9093e31740 1 Processor -- start 2024-04-21T23:11:05.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.815+0000 7f9093e31740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f3a40 0x55e8cb7f9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.815+0000 7f9093e31740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7f4720 0x55e8cb7fa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.815+0000 7f9093e31740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7fcdb0 0x55e8cb7fa820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.815+0000 7f9093e31740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e8cb7ffc20 con 0x55e8cb7fcdb0 2024-04-21T23:11:05.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.815+0000 7f9093e31740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55e8cb712d40 con 0x55e8cb7f4720 2024-04-21T23:11:05.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.815+0000 7f9093e31740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e8cb7ffda0 con 0x55e8cb7f3a40 2024-04-21T23:11:05.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.815+0000 7f9093e31740 10 client.0 _ll_register_callbacks cb 0x55e8cb74e820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:05.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.815+0000 7f9083fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7fcdb0 0x55e8cb7fa820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.815+0000 7f908ad9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7f4720 0x55e8cb7fa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.815+0000 7f9083fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7fcdb0 0x55e8cb7fa820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43624/0 (socket says 172.21.15.44:43624) 2024-04-21T23:11:05.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.815+0000 7f9083fff640 1 -- 172.21.15.44:0/2708632580 learned_addr learned my addr 172.21.15.44:0/2708632580 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:05.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.815+0000 7f9083fff640 1 --2- 172.21.15.44:0/2708632580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f3a40 0x55e8cb7f9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.815+0000 7f908ad9d640 1 -- 172.21.15.44:0/2708632580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f3a40 msgr2=0x55e8cb7f9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:05.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.815+0000 7f908ad9d640 1 --2- 172.21.15.44:0/2708632580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f3a40 0x55e8cb7f9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.815+0000 7f908ad9d640 1 -- 172.21.15.44:0/2708632580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7fcdb0 msgr2=0x55e8cb7fa820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:05.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.815+0000 7f908ad9d640 1 --2- 172.21.15.44:0/2708632580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7fcdb0 0x55e8cb7fa820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.815+0000 7f908ad9d640 1 -- 172.21.15.44:0/2708632580 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8cb5337e0 con 0x55e8cb7f4720 2024-04-21T23:11:05.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.815+0000 7f9083fff640 1 --2- 172.21.15.44:0/2708632580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7fcdb0 0x55e8cb7fa820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:05.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.816+0000 7f908ad9d640 1 --2- 172.21.15.44:0/2708632580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7f4720 0x55e8cb7fa2e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f907c011ed0 tx=0x7f907c00f870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1207b2bea6b17ab6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:05.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.816+0000 7f90727fc640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:11:05.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.816+0000 7f90727fc640 1 -- 172.21.15.44:0/2708632580 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c01e040 con 0x55e8cb7f4720 2024-04-21T23:11:05.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.816+0000 7f90727fc640 1 -- 172.21.15.44:0/2708632580 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7f907c019930 con 0x55e8cb7f4720 2024-04-21T23:11:05.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.816+0000 7f9093e31740 1 -- 172.21.15.44:0/2708632580 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f907c00faa0 con 0x55e8cb7f4720 2024-04-21T23:11:05.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.816+0000 7f90727fc640 1 -- 172.21.15.44:0/2708632580 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f907c032050 con 0x55e8cb7f4720 2024-04-21T23:11:05.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.816+0000 7f9093e31740 10 client.25739 fetch_fsmap learned FSMap version 23 2024-04-21T23:11:05.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.816+0000 7f9093e31740 1 -- 172.21.15.44:0/2708632580 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e8cb593f70 con 0x55e8cb7f4720 2024-04-21T23:11:05.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.816+0000 7f90727fc640 1 -- 172.21.15.44:0/2708632580 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f907c00faa0 con 0x55e8cb7f4720 2024-04-21T23:11:05.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.817+0000 7f90727fc640 1 -- 172.21.15.44:0/2708632580 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f907c021030 con 0x55e8cb7f4720 2024-04-21T23:11:05.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.817+0000 7f9093e31740 10 client.25739 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T23:11:05.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.817+0000 7f9093e31740 10 client.25739 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:05.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.817+0000 7f9093e31740 20 client.25739 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:05.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.817+0000 7f9093e31740 1 --2- 172.21.15.44:0/2708632580 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e8cb80ce20 0x55e8cb80f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.817+0000 7f9093e31740 4 client.25739 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:11:05.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.817+0000 7f9093e31740 1 -- 172.21.15.44:0/2708632580 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8cb72b450 con 0x55e8cb80ce20 2024-04-21T23:11:05.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.817+0000 7f9083fff640 1 --2- 172.21.15.44:0/2708632580 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e8cb80ce20 0x55e8cb80f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.817+0000 7f90727fc640 10 client.25739 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:05.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.817+0000 7f9083fff640 1 --2- 172.21.15.44:0/2708632580 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e8cb80ce20 0x55e8cb80f210 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54996cb476fb270d server_cookie=d06d4c422da1375c in_seq=0 out_seq=0 2024-04-21T23:11:05.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.819+0000 7f90727fc640 1 -- 172.21.15.44:0/2708632580 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e8cb72b450 con 0x55e8cb80ce20 2024-04-21T23:11:05.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.819+0000 7f90727fc640 10 client.25739 handle_command_reply: tid=0 2024-04-21T23:11:05.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.819+0000 7f9093e31740 1 client.25739 shutdown 2024-04-21T23:11:05.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.819+0000 7f9093e31740 1 -- 172.21.15.44:0/2708632580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7f4720 msgr2=0x55e8cb7fa2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:05.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.819+0000 7f9093e31740 1 --2- 172.21.15.44:0/2708632580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7f4720 0x55e8cb7fa2e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f907c011ed0 tx=0x7f907c00f870 comp rx=0 tx=0).stop 2024-04-21T23:11:05.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.819+0000 7f9093e31740 1 -- 172.21.15.44:0/2708632580 shutdown_connections 2024-04-21T23:11:05.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.819+0000 7f9093e31740 1 --2- 172.21.15.44:0/2708632580 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e8cb80ce20 0x55e8cb80f210 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.819+0000 7f9093e31740 1 --2- 172.21.15.44:0/2708632580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7fcdb0 0x55e8cb7fa820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.819+0000 7f9093e31740 1 --2- 172.21.15.44:0/2708632580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7f4720 0x55e8cb7fa2e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.819+0000 7f9093e31740 1 --2- 172.21.15.44:0/2708632580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f3a40 0x55e8cb7f9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.819+0000 7f90727fc640 0 client.25739 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:05.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.819+0000 7f9093e31740 1 -- 172.21.15.44:0/2708632580 >> 172.21.15.44:0/2708632580 conn(0x55e8cb7ba4f0 msgr2=0x55e8cb7f01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:05.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.819+0000 7f9093e31740 1 -- 172.21.15.44:0/2708632580 shutdown_connections 2024-04-21T23:11:05.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.819+0000 7f9093e31740 1 -- 172.21.15.44:0/2708632580 wait complete. 2024-04-21T23:11:05.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.819+0000 7f9093e31740 20 client.25739 trim_cache size 0 max 16384 2024-04-21T23:11:05.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.838+0000 7f9093e31740 1 Processor -- start 2024-04-21T23:11:05.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.838+0000 7f9093e31740 1 -- start start 2024-04-21T23:11:05.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.839+0000 7f9093e31740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7f3a40 0x55e8cb7fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.839+0000 7f9093e31740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f4720 0x55e8cb808b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.839+0000 7f9093e31740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7fcdb0 0x55e8cb814980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.839+0000 7f9093e31740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f908c1acb90 con 0x55e8cb7f3a40 2024-04-21T23:11:05.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.839+0000 7f9093e31740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f908c1af410 con 0x55e8cb7fcdb0 2024-04-21T23:11:05.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.839+0000 7f9093e31740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f908c1ad200 con 0x55e8cb7f4720 2024-04-21T23:11:05.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.839+0000 7f9083fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7f3a40 0x55e8cb7fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.839+0000 7f9083fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7f3a40 0x55e8cb7fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43634/0 (socket says 172.21.15.44:43634) 2024-04-21T23:11:05.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.839+0000 7f9083fff640 1 -- 172.21.15.44:0/3881906272 learned_addr learned my addr 172.21.15.44:0/3881906272 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:05.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.839+0000 7f908ad9d640 1 --2- 172.21.15.44:0/3881906272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f4720 0x55e8cb808b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.839+0000 7f9083fff640 1 -- 172.21.15.44:0/3881906272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f4720 msgr2=0x55e8cb808b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:05.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.839+0000 7f9083fff640 1 --2- 172.21.15.44:0/3881906272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f4720 0x55e8cb808b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.839+0000 7f9083fff640 1 -- 172.21.15.44:0/3881906272 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7fcdb0 msgr2=0x55e8cb814980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T23:11:05.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.839+0000 7f9083fff640 1 --2- 172.21.15.44:0/3881906272 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7fcdb0 0x55e8cb814980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.839+0000 7f9083fff640 1 -- 172.21.15.44:0/3881906272 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8cb42b170 con 0x55e8cb7f3a40 2024-04-21T23:11:05.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.840+0000 7f9083fff640 1 --2- 172.21.15.44:0/3881906272 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7f3a40 0x55e8cb7fc4d0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x55e8cb80f920 tx=0x7f90740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cbb3db01eb392d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.840+0000 7f9080ff9640 1 -- 172.21.15.44:0/3881906272 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907402b070 con 0x55e8cb7f3a40 2024-04-21T23:11:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.840+0000 7f9080ff9640 1 -- 172.21.15.44:0/3881906272 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9074023c40 con 0x55e8cb7f3a40 2024-04-21T23:11:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.840+0000 7f9080ff9640 1 -- 172.21.15.44:0/3881906272 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9074025e60 con 0x55e8cb7f3a40 2024-04-21T23:11:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.841+0000 7f9093e31740 1 -- 172.21.15.44:0/3881906272 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7f3a40 msgr2=0x55e8cb7fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.841+0000 7f9093e31740 1 --2- 172.21.15.44:0/3881906272 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7f3a40 0x55e8cb7fc4d0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x55e8cb80f920 tx=0x7f90740046c0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.841+0000 7f9093e31740 1 -- 172.21.15.44:0/3881906272 shutdown_connections 2024-04-21T23:11:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.841+0000 7f9093e31740 1 --2- 172.21.15.44:0/3881906272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f4720 0x55e8cb808b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.841+0000 7f9093e31740 1 --2- 172.21.15.44:0/3881906272 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7fcdb0 0x55e8cb814980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.841+0000 7f9093e31740 1 --2- 172.21.15.44:0/3881906272 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7f3a40 0x55e8cb7fc4d0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.841+0000 7f9093e31740 1 -- 172.21.15.44:0/3881906272 >> 172.21.15.44:0/3881906272 conn(0x55e8cb7ba4f0 msgr2=0x55e8cb7f8670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.841+0000 7f9093e31740 1 -- 172.21.15.44:0/3881906272 shutdown_connections 2024-04-21T23:11:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.841+0000 7f9093e31740 1 -- 172.21.15.44:0/3881906272 wait complete. 2024-04-21T23:11:05.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.842+0000 7f9093e31740 1 -- start start 2024-04-21T23:11:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.842+0000 7f9093e31740 1 Processor -- start 2024-04-21T23:11:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.842+0000 7f9093e31740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f3a40 0x55e8cb8053c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.842+0000 7f9093e31740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7f4720 0x55e8cb7fd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.842+0000 7f9093e31740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7fcdb0 0x55e8cb7fdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.842+0000 7f9093e31740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f908c1ad8d0 con 0x55e8cb7fcdb0 2024-04-21T23:11:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.842+0000 7f9093e31740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f908c1aed40 con 0x55e8cb7f4720 2024-04-21T23:11:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.842+0000 7f9093e31740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f908c1adfa0 con 0x55e8cb7f3a40 2024-04-21T23:11:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.842+0000 7f9093e31740 10 client.0 _ll_register_callbacks cb 0x7f907c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.842+0000 7f9083fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7fcdb0 0x55e8cb7fdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.842+0000 7f908ad9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7f4720 0x55e8cb7fd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.842+0000 7f9083fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7fcdb0 0x55e8cb7fdcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43646/0 (socket says 172.21.15.44:43646) 2024-04-21T23:11:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.842+0000 7f9083fff640 1 -- 172.21.15.44:0/2878997574 learned_addr learned my addr 172.21.15.44:0/2878997574 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.842+0000 7f9083fff640 1 --2- 172.21.15.44:0/2878997574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f3a40 0x55e8cb8053c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.842+0000 7f9083fff640 1 -- 172.21.15.44:0/2878997574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f3a40 msgr2=0x55e8cb8053c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.842+0000 7f9083fff640 1 --2- 172.21.15.44:0/2878997574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f3a40 0x55e8cb8053c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.842+0000 7f9083fff640 1 -- 172.21.15.44:0/2878997574 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7f4720 msgr2=0x55e8cb7fd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.842+0000 7f9083fff640 1 --2- 172.21.15.44:0/2878997574 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7f4720 0x55e8cb7fd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.842+0000 7f9083fff640 1 -- 172.21.15.44:0/2878997574 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8cb7fe4f0 con 0x55e8cb7fcdb0 2024-04-21T23:11:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.843+0000 7f9083fff640 1 --2- 172.21.15.44:0/2878997574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7fcdb0 0x55e8cb7fdcf0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f9074005780 tx=0x7f9074013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a8c4e0c78b7bcb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.843+0000 7f90727fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:11:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.843+0000 7f90727fc640 1 -- 172.21.15.44:0/2878997574 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907402b040 con 0x55e8cb7fcdb0 2024-04-21T23:11:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.843+0000 7f9093e31740 1 -- 172.21.15.44:0/2878997574 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9074023c40 con 0x55e8cb7fcdb0 2024-04-21T23:11:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.843+0000 7f90727fc640 1 -- 172.21.15.44:0/2878997574 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7f9074004340 con 0x55e8cb7fcdb0 2024-04-21T23:11:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.843+0000 7f90727fc640 1 -- 172.21.15.44:0/2878997574 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f9074025e60 con 0x55e8cb7fcdb0 2024-04-21T23:11:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.843+0000 7f9093e31740 10 client.16173 fetch_fsmap learned FSMap version 23 2024-04-21T23:11:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.843+0000 7f9093e31740 1 -- 172.21.15.44:0/2878997574 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e8cb8731f0 con 0x55e8cb7fcdb0 2024-04-21T23:11:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.843+0000 7f90727fc640 1 -- 172.21.15.44:0/2878997574 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9074023c40 con 0x55e8cb7fcdb0 2024-04-21T23:11:05.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.843+0000 7f90727fc640 1 -- 172.21.15.44:0/2878997574 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f90740257a0 con 0x55e8cb7fcdb0 2024-04-21T23:11:05.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.843+0000 7f9093e31740 10 client.16173 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T23:11:05.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.843+0000 7f9093e31740 10 client.16173 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:05.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.843+0000 7f9093e31740 20 client.16173 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:05.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.844+0000 7f9093e31740 1 --2- 172.21.15.44:0/2878997574 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e8cb873800 0x55e8cb875bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.844+0000 7f9093e31740 4 client.16173 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:11:05.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.844+0000 7f9093e31740 1 -- 172.21.15.44:0/2878997574 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e8cb7646b0 con 0x55e8cb873800 2024-04-21T23:11:05.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.844+0000 7f908ad9d640 1 --2- 172.21.15.44:0/2878997574 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e8cb873800 0x55e8cb875bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.844+0000 7f908ad9d640 1 --2- 172.21.15.44:0/2878997574 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e8cb873800 0x55e8cb875bf0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bca6f179f5d609f7 server_cookie=20236b1bea0cacb6 in_seq=0 out_seq=0 2024-04-21T23:11:05.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.844+0000 7f90727fc640 10 client.16173 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:05.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.845+0000 7f90727fc640 1 -- 172.21.15.44:0/2878997574 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e8cb7646b0 con 0x55e8cb873800 2024-04-21T23:11:05.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.845+0000 7f90727fc640 10 client.16173 handle_command_reply: tid=0 2024-04-21T23:11:05.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.845+0000 7f9093e31740 1 client.16173 shutdown 2024-04-21T23:11:05.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.845+0000 7f9093e31740 1 -- 172.21.15.44:0/2878997574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7fcdb0 msgr2=0x55e8cb7fdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:05.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.845+0000 7f9093e31740 1 --2- 172.21.15.44:0/2878997574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7fcdb0 0x55e8cb7fdcf0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f9074005780 tx=0x7f9074013040 comp rx=0 tx=0).stop 2024-04-21T23:11:05.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.845+0000 7f9093e31740 1 -- 172.21.15.44:0/2878997574 shutdown_connections 2024-04-21T23:11:05.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.845+0000 7f9093e31740 1 --2- 172.21.15.44:0/2878997574 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e8cb873800 0x55e8cb875bf0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.845+0000 7f9093e31740 1 --2- 172.21.15.44:0/2878997574 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e8cb7fcdb0 0x55e8cb7fdcf0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.845+0000 7f90727fc640 0 client.16173 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:05.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.845+0000 7f9093e31740 1 --2- 172.21.15.44:0/2878997574 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e8cb7f4720 0x55e8cb7fd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.845+0000 7f9093e31740 1 --2- 172.21.15.44:0/2878997574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e8cb7f3a40 0x55e8cb8053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.845+0000 7f9093e31740 1 -- 172.21.15.44:0/2878997574 >> 172.21.15.44:0/2878997574 conn(0x55e8cb7ba4f0 msgr2=0x55e8cb7ecc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:05.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.845+0000 7f9093e31740 1 -- 172.21.15.44:0/2878997574 shutdown_connections 2024-04-21T23:11:05.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.845+0000 7f9093e31740 1 -- 172.21.15.44:0/2878997574 wait complete. 2024-04-21T23:11:05.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.846+0000 7f9093e31740 20 client.16173 trim_cache size 0 max 16384 2024-04-21T23:11:05.850 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:11:05.850 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-21T23:11:05.850 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "0010b4be-e845-4cd3-a88a-211615c2b3ba", 2024-04-21T23:11:05.850 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-21T23:11:05.850 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:11:05.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.846+0000 7f9091172640 1 -- 172.21.15.44:0/4037520254 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f90780a2d00 msgr2=0x7f90780a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:05.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.846+0000 7f9091172640 1 --2- 172.21.15.44:0/4037520254 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f90780a2d00 0x7f90780a51c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f90840ef410 tx=0x7f9084044880 comp rx=0 tx=0).stop 2024-04-21T23:11:05.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.846+0000 7f9091172640 1 -- 172.21.15.44:0/4037520254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f908c10bcf0 msgr2=0x7f908c19c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:05.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.846+0000 7f9091172640 1 --2- 172.21.15.44:0/4037520254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f908c10bcf0 0x7f908c19c880 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f908402c790 tx=0x7f9084035e00 comp rx=0 tx=0).stop 2024-04-21T23:11:05.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.846+0000 7f9091172640 1 -- 172.21.15.44:0/4037520254 shutdown_connections 2024-04-21T23:11:05.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.847+0000 7f9091172640 1 --2- 172.21.15.44:0/4037520254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f908c1032d0 0x7f908c198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.847+0000 7f9091172640 1 --2- 172.21.15.44:0/4037520254 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f90780a2d00 0x7f90780a51c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.847+0000 7f9091172640 1 --2- 172.21.15.44:0/4037520254 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f908c10bcf0 0x7f908c19c880 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.847+0000 7f9091172640 1 --2- 172.21.15.44:0/4037520254 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f908c1029b0 0x7f908c198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.847+0000 7f9091172640 1 -- 172.21.15.44:0/4037520254 >> 172.21.15.44:0/4037520254 conn(0x7f908c0777f0 msgr2=0x7f908c0fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:05.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.847+0000 7f9091172640 1 -- 172.21.15.44:0/4037520254 shutdown_connections 2024-04-21T23:11:05.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.847+0000 7f9091172640 1 -- 172.21.15.44:0/4037520254 wait complete. 2024-04-21T23:11:05.859 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:11:05.859 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:11:05.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.969+0000 7f55e8fc9640 1 -- 172.21.15.44:0/220019242 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55e41032d0 msgr2=0x7f55e410b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:05.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.969+0000 7f55e8fc9640 1 --2- 172.21.15.44:0/220019242 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55e41032d0 0x7f55e410b7b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f55d8009900 tx=0x7f55d802bcd0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.969+0000 7f55e8fc9640 1 -- 172.21.15.44:0/220019242 shutdown_connections 2024-04-21T23:11:05.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.969+0000 7f55e8fc9640 1 --2- 172.21.15.44:0/220019242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55e410bcf0 0x7f55e410e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.969+0000 7f55e8fc9640 1 --2- 172.21.15.44:0/220019242 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55e41032d0 0x7f55e410b7b0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.969+0000 7f55e8fc9640 1 --2- 172.21.15.44:0/220019242 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55e41029b0 0x7f55e4102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.969+0000 7f55e8fc9640 1 -- 172.21.15.44:0/220019242 >> 172.21.15.44:0/220019242 conn(0x7f55e40777f0 msgr2=0x7f55e4077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:05.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.969+0000 7f55e8fc9640 1 -- 172.21.15.44:0/220019242 shutdown_connections 2024-04-21T23:11:05.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.969+0000 7f55e8fc9640 1 -- 172.21.15.44:0/220019242 wait complete. 2024-04-21T23:11:05.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.970+0000 7f55e8fc9640 1 Processor -- start 2024-04-21T23:11:05.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.970+0000 7f55e8fc9640 1 -- start start 2024-04-21T23:11:05.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.970+0000 7f55e8fc9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55e41029b0 0x7f55e4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.970+0000 7f55e8fc9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55e41032d0 0x7f55e4198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.970+0000 7f55e8fc9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55e410bcf0 0x7f55e419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.970+0000 7f55e8fc9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f55e4111b40 con 0x7f55e41029b0 2024-04-21T23:11:05.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.970+0000 7f55e8fc9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f55e41119c0 con 0x7f55e410bcf0 2024-04-21T23:11:05.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.970+0000 7f55e8fc9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f55e4111cc0 con 0x7f55e41032d0 2024-04-21T23:11:05.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.970+0000 7f55e259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55e41029b0 0x7f55e4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.970+0000 7f55e259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55e41029b0 0x7f55e4197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43652/0 (socket says 172.21.15.44:43652) 2024-04-21T23:11:05.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.970+0000 7f55e259c640 1 -- 172.21.15.44:0/3188823520 learned_addr learned my addr 172.21.15.44:0/3188823520 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:05.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.970+0000 7f55e1d9b640 1 --2- 172.21.15.44:0/3188823520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55e41032d0 0x7f55e4198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.970+0000 7f55e2d9d640 1 --2- 172.21.15.44:0/3188823520 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55e410bcf0 0x7f55e419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.970+0000 7f55e259c640 1 -- 172.21.15.44:0/3188823520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55e41032d0 msgr2=0x7f55e4198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:05.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.970+0000 7f55e259c640 1 --2- 172.21.15.44:0/3188823520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55e41032d0 0x7f55e4198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.970+0000 7f55e259c640 1 -- 172.21.15.44:0/3188823520 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55e410bcf0 msgr2=0x7f55e419c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:05.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.970+0000 7f55e259c640 1 --2- 172.21.15.44:0/3188823520 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55e410bcf0 0x7f55e419c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.970+0000 7f55e259c640 1 -- 172.21.15.44:0/3188823520 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55e419cfd0 con 0x7f55e41029b0 2024-04-21T23:11:05.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.970+0000 7f55e1d9b640 1 --2- 172.21.15.44:0/3188823520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55e41032d0 0x7f55e4198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:05.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.971+0000 7f55e259c640 1 --2- 172.21.15.44:0/3188823520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55e41029b0 0x7f55e4197fe0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f55d400e9f0 tx=0x7f55d400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:05.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.971+0000 7f55d37fe640 1 -- 172.21.15.44:0/3188823520 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d400cd80 con 0x7f55e41029b0 2024-04-21T23:11:05.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.971+0000 7f55d37fe640 1 -- 172.21.15.44:0/3188823520 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55d4010910 con 0x7f55e41029b0 2024-04-21T23:11:05.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.971+0000 7f55e8fc9640 1 -- 172.21.15.44:0/3188823520 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55e419d2c0 con 0x7f55e41029b0 2024-04-21T23:11:05.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.971+0000 7f55d37fe640 1 -- 172.21.15.44:0/3188823520 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d4018990 con 0x7f55e41029b0 2024-04-21T23:11:05.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.971+0000 7f55e8fc9640 1 -- 172.21.15.44:0/3188823520 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55e41a4b50 con 0x7f55e41029b0 2024-04-21T23:11:05.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.973+0000 7f55d37fe640 1 -- 172.21.15.44:0/3188823520 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f55d4010ac0 con 0x7f55e41029b0 2024-04-21T23:11:05.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.973+0000 7f55ebbb3740 1 Processor -- start 2024-04-21T23:11:05.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.973+0000 7f55d37fe640 1 --2- 172.21.15.44:0/3188823520 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f55cc0a2d00 0x7f55cc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.974+0000 7f55e2d9d640 1 --2- 172.21.15.44:0/3188823520 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f55cc0a2d00 0x7f55cc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.974+0000 7f55ebbb3740 1 -- start start 2024-04-21T23:11:05.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.974+0000 7f55d37fe640 1 -- 172.21.15.44:0/3188823520 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7f55d40cfae0 con 0x7f55e41029b0 2024-04-21T23:11:05.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.974+0000 7f55ebbb3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5df30 0x5584b2f5e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.974+0000 7f55ebbb3740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5584b2f5e850 0x5584b2f66ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.974+0000 7f55ebbb3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f67430 0x5584b2f69820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.974+0000 7f55ebbb3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5584b2ec2560 con 0x5584b2f67430 2024-04-21T23:11:05.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.974+0000 7f55ebbb3740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5584b2e6a730 con 0x5584b2f5e850 2024-04-21T23:11:05.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.974+0000 7f55ebbb3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5584b2ebe520 con 0x5584b2f5df30 2024-04-21T23:11:05.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.974+0000 7f55e259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5df30 0x5584b2f5e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.974+0000 7f55e1d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5584b2f5e850 0x5584b2f66ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.974+0000 7f55e259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5df30 0x5584b2f5e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:40416/0 (socket says 172.21.15.44:40416) 2024-04-21T23:11:05.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.974+0000 7f55e259c640 1 -- 172.21.15.44:0/2280732154 learned_addr learned my addr 172.21.15.44:0/2280732154 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:05.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.974+0000 7f55e2d9d640 1 --2- 172.21.15.44:0/3188823520 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f55cc0a2d00 0x7f55cc0a51c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f55dc005fd0 tx=0x7f55dc009500 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:05.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.974+0000 7f55e2d9d640 1 --2- 172.21.15.44:0/2280732154 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f67430 0x5584b2f69820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.974+0000 7f55e259c640 1 -- 172.21.15.44:0/2280732154 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5584b2f5e850 msgr2=0x5584b2f66ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:05.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.974+0000 7f55e259c640 1 --2- 172.21.15.44:0/2280732154 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5584b2f5e850 0x5584b2f66ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.974+0000 7f55e259c640 1 -- 172.21.15.44:0/2280732154 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f67430 msgr2=0x5584b2f69820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:05.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.974+0000 7f55e259c640 1 --2- 172.21.15.44:0/2280732154 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f67430 0x5584b2f69820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.974+0000 7f55e259c640 1 -- 172.21.15.44:0/2280732154 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584b2ca07d0 con 0x5584b2f5df30 2024-04-21T23:11:05.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.975+0000 7f55e259c640 1 --2- 172.21.15.44:0/2280732154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5df30 0x5584b2f5e310 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f55d40986e0 tx=0x7f55d409bd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd8c343829432384 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:05.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.975+0000 7f55d0ff9640 1 -- 172.21.15.44:0/2280732154 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d40cd110 con 0x5584b2f5df30 2024-04-21T23:11:05.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.975+0000 7f55d0ff9640 1 -- 172.21.15.44:0/2280732154 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55d40b1070 con 0x5584b2f5df30 2024-04-21T23:11:05.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.975+0000 7f55d0ff9640 1 -- 172.21.15.44:0/2280732154 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d409b3c0 con 0x5584b2f5df30 2024-04-21T23:11:05.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.975+0000 7f55ebbb3740 1 -- 172.21.15.44:0/2280732154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5df30 msgr2=0x5584b2f5e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:05.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.975+0000 7f55ebbb3740 1 --2- 172.21.15.44:0/2280732154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5df30 0x5584b2f5e310 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f55d40986e0 tx=0x7f55d409bd10 comp rx=0 tx=0).stop 2024-04-21T23:11:05.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.976+0000 7f55ebbb3740 1 -- 172.21.15.44:0/2280732154 shutdown_connections 2024-04-21T23:11:05.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.976+0000 7f55ebbb3740 1 --2- 172.21.15.44:0/2280732154 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f67430 0x5584b2f69820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.976+0000 7f55ebbb3740 1 --2- 172.21.15.44:0/2280732154 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5584b2f5e850 0x5584b2f66ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.976+0000 7f55ebbb3740 1 --2- 172.21.15.44:0/2280732154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5df30 0x5584b2f5e310 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.976+0000 7f55ebbb3740 1 -- 172.21.15.44:0/2280732154 >> 172.21.15.44:0/2280732154 conn(0x5584b2f58660 msgr2=0x5584b2f58a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.976+0000 7f55ebbb3740 1 -- 172.21.15.44:0/2280732154 shutdown_connections 2024-04-21T23:11:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.976+0000 7f55ebbb3740 1 -- 172.21.15.44:0/2280732154 wait complete. 2024-04-21T23:11:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.976+0000 7f55ebbb3740 1 -- start start 2024-04-21T23:11:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.976+0000 7f55ebbb3740 1 Processor -- start 2024-04-21T23:11:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.976+0000 7f55ebbb3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5df30 0x5584b2f64550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.976+0000 7f55ebbb3740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5584b2f5e850 0x5584b2f64a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.976+0000 7f55ebbb3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f67430 0x5584b2f64fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.976+0000 7f55ebbb3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5584b2f6a220 con 0x5584b2f67430 2024-04-21T23:11:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.976+0000 7f55ebbb3740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5584b2e7de00 con 0x5584b2f5e850 2024-04-21T23:11:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.976+0000 7f55ebbb3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5584b2f6a3a0 con 0x5584b2f5df30 2024-04-21T23:11:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.976+0000 7f55ebbb3740 10 client.0 _ll_register_callbacks cb 0x5584b2eb8eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:05.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.977+0000 7f55e2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f67430 0x5584b2f64fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.977+0000 7f55e259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5df30 0x5584b2f64550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.977+0000 7f55e1d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5584b2f5e850 0x5584b2f64a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.977+0000 7f55e259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5df30 0x5584b2f64550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:40418/0 (socket says 172.21.15.44:40418) 2024-04-21T23:11:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.977+0000 7f55e259c640 1 -- 172.21.15.44:0/1711385936 learned_addr learned my addr 172.21.15.44:0/1711385936 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.977+0000 7f55e259c640 1 -- 172.21.15.44:0/1711385936 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5584b2f5e850 msgr2=0x5584b2f64a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.977+0000 7f55e259c640 1 --2- 172.21.15.44:0/1711385936 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5584b2f5e850 0x5584b2f64a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.977+0000 7f55e259c640 1 -- 172.21.15.44:0/1711385936 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f67430 msgr2=0x5584b2f64fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.977+0000 7f55e259c640 1 --2- 172.21.15.44:0/1711385936 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f67430 0x5584b2f64fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.977+0000 7f55e259c640 1 -- 172.21.15.44:0/1711385936 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584b2bd63f0 con 0x5584b2f5df30 2024-04-21T23:11:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.977+0000 7f55e259c640 1 --2- 172.21.15.44:0/1711385936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5df30 0x5584b2f64550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f55d40cfe00 tx=0x7f55d409a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e30cf97a456d464c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.977+0000 7f55c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:11:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.977+0000 7f55c9ffb640 1 -- 172.21.15.44:0/1711385936 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d40b4e60 con 0x5584b2f5df30 2024-04-21T23:11:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.977+0000 7f55c9ffb640 1 -- 172.21.15.44:0/1711385936 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7f55d409b360 con 0x5584b2f5df30 2024-04-21T23:11:05.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.977+0000 7f55ebbb3740 1 -- 172.21.15.44:0/1711385936 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55d40b1070 con 0x5584b2f5df30 2024-04-21T23:11:05.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.977+0000 7f55c9ffb640 1 -- 172.21.15.44:0/1711385936 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f55d4010430 con 0x5584b2f5df30 2024-04-21T23:11:05.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.977+0000 7f55ebbb3740 10 client.25795 fetch_fsmap learned FSMap version 23 2024-04-21T23:11:05.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.977+0000 7f55ebbb3740 1 -- 172.21.15.44:0/1711385936 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5584b2b960f0 con 0x5584b2f5df30 2024-04-21T23:11:05.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.977+0000 7f55c9ffb640 1 -- 172.21.15.44:0/1711385936 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55d40b1070 con 0x5584b2f5df30 2024-04-21T23:11:05.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.978+0000 7f55c9ffb640 1 -- 172.21.15.44:0/1711385936 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f55d40a11a0 con 0x5584b2f5df30 2024-04-21T23:11:05.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.978+0000 7f55ebbb3740 10 client.25795 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T23:11:05.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.978+0000 7f55ebbb3740 10 client.25795 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:05.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.978+0000 7f55ebbb3740 20 client.25795 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:05.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.978+0000 7f55ebbb3740 1 --2- 172.21.15.44:0/1711385936 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5584b2f77730 0x5584b2f79b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:05.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.978+0000 7f55ebbb3740 4 client.25795 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:11:05.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.978+0000 7f55ebbb3740 1 -- 172.21.15.44:0/1711385936 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584b2ecfaa0 con 0x5584b2f77730 2024-04-21T23:11:05.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.978+0000 7f55e1d9b640 1 --2- 172.21.15.44:0/1711385936 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5584b2f77730 0x5584b2f79b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:05.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.978+0000 7f55c9ffb640 10 client.25795 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:05.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.978+0000 7f55e1d9b640 1 --2- 172.21.15.44:0/1711385936 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5584b2f77730 0x5584b2f79b20 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4a22c7f33a9e86d server_cookie=a3a51f824cdd56a0 in_seq=0 out_seq=0 2024-04-21T23:11:05.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.979+0000 7f55c9ffb640 1 -- 172.21.15.44:0/1711385936 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5584b2ecfaa0 con 0x5584b2f77730 2024-04-21T23:11:05.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.979+0000 7f55c9ffb640 10 client.25795 handle_command_reply: tid=0 2024-04-21T23:11:05.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.979+0000 7f55ebbb3740 1 client.25795 shutdown 2024-04-21T23:11:05.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.979+0000 7f55ebbb3740 1 -- 172.21.15.44:0/1711385936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5df30 msgr2=0x5584b2f64550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:05.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.979+0000 7f55ebbb3740 1 --2- 172.21.15.44:0/1711385936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5df30 0x5584b2f64550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f55d40cfe00 tx=0x7f55d409a100 comp rx=0 tx=0).stop 2024-04-21T23:11:05.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.979+0000 7f55ebbb3740 1 -- 172.21.15.44:0/1711385936 shutdown_connections 2024-04-21T23:11:05.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.979+0000 7f55ebbb3740 1 --2- 172.21.15.44:0/1711385936 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5584b2f77730 0x5584b2f79b20 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.979+0000 7f55ebbb3740 1 --2- 172.21.15.44:0/1711385936 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f67430 0x5584b2f64fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.979+0000 7f55ebbb3740 1 --2- 172.21.15.44:0/1711385936 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5584b2f5e850 0x5584b2f64a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.979+0000 7f55c9ffb640 0 client.25795 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:05.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.979+0000 7f55ebbb3740 1 --2- 172.21.15.44:0/1711385936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5df30 0x5584b2f64550 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:05.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.979+0000 7f55ebbb3740 1 -- 172.21.15.44:0/1711385936 >> 172.21.15.44:0/1711385936 conn(0x5584b2f58660 msgr2=0x5584b2f5a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:05.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.979+0000 7f55ebbb3740 1 -- 172.21.15.44:0/1711385936 shutdown_connections 2024-04-21T23:11:05.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.979+0000 7f55ebbb3740 1 -- 172.21.15.44:0/1711385936 wait complete. 2024-04-21T23:11:05.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:05.979+0000 7f55ebbb3740 20 client.25795 trim_cache size 0 max 16384 2024-04-21T23:11:06.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.001+0000 7f55ebbb3740 1 Processor -- start 2024-04-21T23:11:06.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.001+0000 7f55ebbb3740 1 -- start start 2024-04-21T23:11:06.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.001+0000 7f55ebbb3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f5df30 0x5584b2f66c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:06.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.001+0000 7f55ebbb3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5e850 0x5584b2f63400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:06.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.001+0000 7f55ebbb3740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5584b2f67430 0x5584b2f7cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:06.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.001+0000 7f55ebbb3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f55e41acba0 con 0x5584b2f5df30 2024-04-21T23:11:06.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.001+0000 7f55ebbb3740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f55e41af420 con 0x5584b2f67430 2024-04-21T23:11:06.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.001+0000 7f55ebbb3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f55e41ad210 con 0x5584b2f5e850 2024-04-21T23:11:06.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.001+0000 7f55e259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f5df30 0x5584b2f66c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:06.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.001+0000 7f55e1d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5e850 0x5584b2f63400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:06.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.001+0000 7f55e259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f5df30 0x5584b2f66c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43688/0 (socket says 172.21.15.44:43688) 2024-04-21T23:11:06.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.001+0000 7f55e259c640 1 -- 172.21.15.44:0/32803454 learned_addr learned my addr 172.21.15.44:0/32803454 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:06.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.001+0000 7f55e2d9d640 1 --2- 172.21.15.44:0/32803454 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5584b2f67430 0x5584b2f7cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:06.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.001+0000 7f55e259c640 1 -- 172.21.15.44:0/32803454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5e850 msgr2=0x5584b2f63400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:06.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.001+0000 7f55e259c640 1 --2- 172.21.15.44:0/32803454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5e850 0x5584b2f63400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:06.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.001+0000 7f55e259c640 1 -- 172.21.15.44:0/32803454 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5584b2f67430 msgr2=0x5584b2f7cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:06.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.001+0000 7f55e259c640 1 --2- 172.21.15.44:0/32803454 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5584b2f67430 0x5584b2f7cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:06.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.001+0000 7f55e259c640 1 -- 172.21.15.44:0/32803454 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584b2f58fa0 con 0x5584b2f5df30 2024-04-21T23:11:06.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.001+0000 7f55e1d9b640 1 --2- 172.21.15.44:0/32803454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5e850 0x5584b2f63400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:06.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.001+0000 7f55e259c640 1 --2- 172.21.15.44:0/32803454 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f5df30 0x5584b2f66c80 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f55d409ba10 tx=0x7f55d409bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb4d7dd9abd070de server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:06.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.002+0000 7f55cbfff640 1 -- 172.21.15.44:0/32803454 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d40b4e60 con 0x5584b2f5df30 2024-04-21T23:11:06.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.002+0000 7f55cbfff640 1 -- 172.21.15.44:0/32803454 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55d40a8070 con 0x5584b2f5df30 2024-04-21T23:11:06.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.002+0000 7f55cbfff640 1 -- 172.21.15.44:0/32803454 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d409b160 con 0x5584b2f5df30 2024-04-21T23:11:06.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.002+0000 7f55ebbb3740 1 -- 172.21.15.44:0/32803454 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f5df30 msgr2=0x5584b2f66c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:06.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.002+0000 7f55ebbb3740 1 --2- 172.21.15.44:0/32803454 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f5df30 0x5584b2f66c80 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f55d409ba10 tx=0x7f55d409bd10 comp rx=0 tx=0).stop 2024-04-21T23:11:06.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55ebbb3740 1 -- 172.21.15.44:0/32803454 shutdown_connections 2024-04-21T23:11:06.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55ebbb3740 1 --2- 172.21.15.44:0/32803454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5e850 0x5584b2f63400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:06.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55ebbb3740 1 --2- 172.21.15.44:0/32803454 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5584b2f67430 0x5584b2f7cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:06.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55ebbb3740 1 --2- 172.21.15.44:0/32803454 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f5df30 0x5584b2f66c80 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:06.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55ebbb3740 1 -- 172.21.15.44:0/32803454 >> 172.21.15.44:0/32803454 conn(0x5584b2f58660 msgr2=0x5584b2f62cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:06.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55ebbb3740 1 -- 172.21.15.44:0/32803454 shutdown_connections 2024-04-21T23:11:06.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55ebbb3740 1 -- 172.21.15.44:0/32803454 wait complete. 2024-04-21T23:11:06.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55ebbb3740 1 -- start start 2024-04-21T23:11:06.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55ebbb3740 1 Processor -- start 2024-04-21T23:11:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55ebbb3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5df30 0x5584b2f7a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55ebbb3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f5e850 0x5584b2f7aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55ebbb3740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5584b2f67430 0x5584b2f7af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55ebbb3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f55e41ad8e0 con 0x5584b2f5e850 2024-04-21T23:11:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55ebbb3740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f55e41aed50 con 0x5584b2f67430 2024-04-21T23:11:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55ebbb3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f55e41adfb0 con 0x5584b2f5df30 2024-04-21T23:11:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55ebbb3740 10 client.0 _ll_register_callbacks cb 0x7f55dc0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55e1d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f5e850 0x5584b2f7aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55e1d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f5e850 0x5584b2f7aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43702/0 (socket says 172.21.15.44:43702) 2024-04-21T23:11:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55e1d9b640 1 -- 172.21.15.44:0/1826017035 learned_addr learned my addr 172.21.15.44:0/1826017035 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55e2d9d640 1 --2- 172.21.15.44:0/1826017035 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5584b2f67430 0x5584b2f7af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.003+0000 7f55e259c640 1 --2- 172.21.15.44:0/1826017035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5df30 0x5584b2f7a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.004+0000 7f55e1d9b640 1 -- 172.21.15.44:0/1826017035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5df30 msgr2=0x5584b2f7a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:06.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.004+0000 7f55e1d9b640 1 --2- 172.21.15.44:0/1826017035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5df30 0x5584b2f7a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:06.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.004+0000 7f55e1d9b640 1 -- 172.21.15.44:0/1826017035 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5584b2f67430 msgr2=0x5584b2f7af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:06.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.004+0000 7f55e1d9b640 1 --2- 172.21.15.44:0/1826017035 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5584b2f67430 0x5584b2f7af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:06.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.004+0000 7f55e1d9b640 1 -- 172.21.15.44:0/1826017035 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584b2fdc910 con 0x5584b2f5e850 2024-04-21T23:11:06.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.004+0000 7f55e1d9b640 1 --2- 172.21.15.44:0/1826017035 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f5e850 0x5584b2f7aa00 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f55d8000c00 tx=0x7f55d8005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=630acdeaa880331f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:06.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.004+0000 7f55c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:11:06.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.004+0000 7f55c9ffb640 1 -- 172.21.15.44:0/1826017035 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d8048d60 con 0x5584b2f5e850 2024-04-21T23:11:06.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.004+0000 7f55c9ffb640 1 -- 172.21.15.44:0/1826017035 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7f55d8004040 con 0x5584b2f5e850 2024-04-21T23:11:06.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.004+0000 7f55ebbb3740 1 -- 172.21.15.44:0/1826017035 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55d40a8070 con 0x5584b2f5e850 2024-04-21T23:11:06.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.004+0000 7f55c9ffb640 1 -- 172.21.15.44:0/1826017035 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f55d8004240 con 0x5584b2f5e850 2024-04-21T23:11:06.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.004+0000 7f55ebbb3740 10 client.16191 fetch_fsmap learned FSMap version 23 2024-04-21T23:11:06.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.004+0000 7f55ebbb3740 1 -- 172.21.15.44:0/1826017035 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5584b2fddbd0 con 0x5584b2f5e850 2024-04-21T23:11:06.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.004+0000 7f55c9ffb640 1 -- 172.21.15.44:0/1826017035 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55d40a8070 con 0x5584b2f5e850 2024-04-21T23:11:06.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.004+0000 7f55c9ffb640 1 -- 172.21.15.44:0/1826017035 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f55d8048610 con 0x5584b2f5e850 2024-04-21T23:11:06.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.004+0000 7f55ebbb3740 10 client.16191 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T23:11:06.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.004+0000 7f55ebbb3740 10 client.16191 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:06.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.004+0000 7f55ebbb3740 20 client.16191 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:06.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.004+0000 7f55ebbb3740 1 --2- 172.21.15.44:0/1826017035 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5584b2fde300 0x5584b2fe0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:06.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.005+0000 7f55ebbb3740 4 client.16191 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:11:06.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.005+0000 7f55ebbb3740 1 -- 172.21.15.44:0/1826017035 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5584b2ec9df0 con 0x5584b2fde300 2024-04-21T23:11:06.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.005+0000 7f55e259c640 1 --2- 172.21.15.44:0/1826017035 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5584b2fde300 0x5584b2fe0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:06.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.005+0000 7f55e259c640 1 --2- 172.21.15.44:0/1826017035 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5584b2fde300 0x5584b2fe0720 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91764af4148ee748 server_cookie=2c966b485c61cd7e in_seq=0 out_seq=0 2024-04-21T23:11:06.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.005+0000 7f55c9ffb640 10 client.16191 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:06.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.009+0000 7f55c9ffb640 1 -- 172.21.15.44:0/1826017035 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5584b2ec9df0 con 0x5584b2fde300 2024-04-21T23:11:06.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.009+0000 7f55c9ffb640 10 client.16191 handle_command_reply: tid=0 2024-04-21T23:11:06.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.009+0000 7f55ebbb3740 1 client.16191 shutdown 2024-04-21T23:11:06.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.009+0000 7f55ebbb3740 1 -- 172.21.15.44:0/1826017035 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f5e850 msgr2=0x5584b2f7aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:06.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.009+0000 7f55ebbb3740 1 --2- 172.21.15.44:0/1826017035 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f5e850 0x5584b2f7aa00 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f55d8000c00 tx=0x7f55d8005a20 comp rx=0 tx=0).stop 2024-04-21T23:11:06.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.009+0000 7f55ebbb3740 1 -- 172.21.15.44:0/1826017035 shutdown_connections 2024-04-21T23:11:06.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.009+0000 7f55ebbb3740 1 --2- 172.21.15.44:0/1826017035 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5584b2fde300 0x5584b2fe0720 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:06.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.009+0000 7f55ebbb3740 1 --2- 172.21.15.44:0/1826017035 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5584b2f67430 0x5584b2f7af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:06.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.009+0000 7f55c9ffb640 0 client.16191 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:06.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.009+0000 7f55ebbb3740 1 --2- 172.21.15.44:0/1826017035 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5584b2f5e850 0x5584b2f7aa00 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:06.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.009+0000 7f55ebbb3740 1 --2- 172.21.15.44:0/1826017035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5584b2f5df30 0x5584b2f7a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:06.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.009+0000 7f55ebbb3740 1 -- 172.21.15.44:0/1826017035 >> 172.21.15.44:0/1826017035 conn(0x5584b2f58660 msgr2=0x5584b2f72e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:06.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.009+0000 7f55ebbb3740 1 -- 172.21.15.44:0/1826017035 shutdown_connections 2024-04-21T23:11:06.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.009+0000 7f55ebbb3740 1 -- 172.21.15.44:0/1826017035 wait complete. 2024-04-21T23:11:06.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.009+0000 7f55ebbb3740 20 client.16191 trim_cache size 0 max 16384 2024-04-21T23:11:06.013 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:11:06.013 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:11:06.013 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-21T23:11:06.013 INFO:teuthology.orchestra.run.smithi044.stdout: "0010b4be-e845-4cd3-a88a-211615c2b3ba": { 2024-04-21T23:11:06.013 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-21T23:11:06.014 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "0010b4be-e845-4cd3-a88a-211615c2b3ba", 2024-04-21T23:11:06.014 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-04-21T23:11:06.014 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:11:06.014 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:11:06.014 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:11:06.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.009+0000 7f55e8fc9640 1 -- 172.21.15.44:0/3188823520 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f55cc0a2d00 msgr2=0x7f55cc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:06.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.009+0000 7f55e8fc9640 1 --2- 172.21.15.44:0/3188823520 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f55cc0a2d00 0x7f55cc0a51c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f55dc005fd0 tx=0x7f55dc009500 comp rx=0 tx=0).stop 2024-04-21T23:11:06.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.009+0000 7f55e8fc9640 1 -- 172.21.15.44:0/3188823520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55e41029b0 msgr2=0x7f55e4197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:06.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.009+0000 7f55e8fc9640 1 --2- 172.21.15.44:0/3188823520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55e41029b0 0x7f55e4197fe0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f55d400e9f0 tx=0x7f55d400eeb0 comp rx=0 tx=0).stop 2024-04-21T23:11:06.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.010+0000 7f55e8fc9640 1 -- 172.21.15.44:0/3188823520 shutdown_connections 2024-04-21T23:11:06.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.010+0000 7f55e8fc9640 1 --2- 172.21.15.44:0/3188823520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55e41032d0 0x7f55e4198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:06.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.010+0000 7f55e8fc9640 1 --2- 172.21.15.44:0/3188823520 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f55cc0a2d00 0x7f55cc0a51c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:06.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.010+0000 7f55e8fc9640 1 --2- 172.21.15.44:0/3188823520 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55e410bcf0 0x7f55e419c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:06.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.010+0000 7f55e8fc9640 1 --2- 172.21.15.44:0/3188823520 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f55e41029b0 0x7f55e4197fe0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:06.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.010+0000 7f55e8fc9640 1 -- 172.21.15.44:0/3188823520 >> 172.21.15.44:0/3188823520 conn(0x7f55e40777f0 msgr2=0x7f55e40fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:06.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.010+0000 7f55e8fc9640 1 -- 172.21.15.44:0/3188823520 shutdown_connections 2024-04-21T23:11:06.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:06.010+0000 7f55e8fc9640 1 -- 172.21.15.44:0/3188823520 wait complete. 2024-04-21T23:11:06.020 INFO:tasks.cephfs.filesystem:scrub status for tag:0010b4be-e845-4cd3-a88a-211615c2b3ba - {'path': '/', 'tag': '0010b4be-e845-4cd3-a88a-211615c2b3ba', 'options': 'recursive,force'} 2024-04-21T23:11:06.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:06 smithi044 ceph-mon[27934]: pgmap v71: 129 pgs: 129 active+clean; 40 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 363 op/s 2024-04-21T23:11:06.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:06 smithi044 ceph-mon[27934]: scrub queued for path: / 2024-04-21T23:11:06.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:06 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:11:06.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:06 smithi044 ceph-mon[27934]: scrub summary: active paths [/] 2024-04-21T23:11:06.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:06 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:11:06.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:06 smithi069 ceph-mon[31816]: pgmap v71: 129 pgs: 129 active+clean; 40 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 363 op/s 2024-04-21T23:11:06.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:06 smithi069 ceph-mon[31816]: scrub queued for path: / 2024-04-21T23:11:06.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:06 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:11:06.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:06 smithi069 ceph-mon[31816]: scrub summary: active paths [/] 2024-04-21T23:11:06.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:06 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:11:06.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:06 smithi055 ceph-mon[31117]: pgmap v71: 129 pgs: 129 active+clean; 40 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 363 op/s 2024-04-21T23:11:06.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:06 smithi055 ceph-mon[31117]: scrub queued for path: / 2024-04-21T23:11:06.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:06 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:11:06.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:06 smithi055 ceph-mon[31117]: scrub summary: active paths [/] 2024-04-21T23:11:06.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:06 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:11:06.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 152759 15.05 MB/sec warmup 56 sec latency 16.595 ms 2024-04-21T23:11:07.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:07 smithi055 ceph-mon[31117]: scrub summary: idle 2024-04-21T23:11:07.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:07 smithi069 ceph-mon[31816]: scrub summary: idle 2024-04-21T23:11:07.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:07 smithi044 ceph-mon[27934]: scrub summary: idle 2024-04-21T23:11:07.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 155553 15.01 MB/sec warmup 57 sec latency 29.625 ms 2024-04-21T23:11:08.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:08 smithi055 ceph-mon[31117]: pgmap v72: 129 pgs: 129 active+clean; 45 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 444 op/s 2024-04-21T23:11:08.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:08 smithi069 ceph-mon[31816]: pgmap v72: 129 pgs: 129 active+clean; 45 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 444 op/s 2024-04-21T23:11:08.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:08 smithi044 ceph-mon[27934]: pgmap v72: 129 pgs: 129 active+clean; 45 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 444 op/s 2024-04-21T23:11:08.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 157652 14.96 MB/sec warmup 58 sec latency 47.285 ms 2024-04-21T23:11:09.647 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.645+0000 7f3baf5e5640 1 -- 172.21.15.44:0/1090131091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ba81032d0 msgr2=0x7f3ba810b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.645+0000 7f3baf5e5640 1 --2- 172.21.15.44:0/1090131091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ba81032d0 0x7f3ba810b7b0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f3b98009900 tx=0x7f3b9802bcd0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.646+0000 7f3baf5e5640 1 -- 172.21.15.44:0/1090131091 shutdown_connections 2024-04-21T23:11:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.646+0000 7f3baf5e5640 1 --2- 172.21.15.44:0/1090131091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ba810bd80 0x7f3ba810e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.646+0000 7f3baf5e5640 1 --2- 172.21.15.44:0/1090131091 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ba81032d0 0x7f3ba810b7b0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.646+0000 7f3baf5e5640 1 --2- 172.21.15.44:0/1090131091 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ba81029b0 0x7f3ba8102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.646+0000 7f3baf5e5640 1 -- 172.21.15.44:0/1090131091 >> 172.21.15.44:0/1090131091 conn(0x7f3ba8077760 msgr2=0x7f3ba8077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.646+0000 7f3baf5e5640 1 -- 172.21.15.44:0/1090131091 shutdown_connections 2024-04-21T23:11:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.646+0000 7f3baf5e5640 1 -- 172.21.15.44:0/1090131091 wait complete. 2024-04-21T23:11:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.646+0000 7f3baf5e5640 1 Processor -- start 2024-04-21T23:11:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.646+0000 7f3baf5e5640 1 -- start start 2024-04-21T23:11:09.648 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.646+0000 7f3baf5e5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ba81029b0 0x7f3ba8100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.646+0000 7f3baf5e5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ba81032d0 0x7f3ba8100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.647+0000 7f3baf5e5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ba810bd80 0x7f3ba8101080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.647+0000 7f3baf5e5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba8111b10 con 0x7f3ba810bd80 2024-04-21T23:11:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.647+0000 7f3baf5e5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba8111990 con 0x7f3ba81032d0 2024-04-21T23:11:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.647+0000 7f3baf5e5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba8111c90 con 0x7f3ba81029b0 2024-04-21T23:11:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.647+0000 7f3badb82640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ba810bd80 0x7f3ba8101080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.647+0000 7f3bad381640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ba81029b0 0x7f3ba8100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.647+0000 7f3badb82640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ba810bd80 0x7f3ba8101080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43728/0 (socket says 172.21.15.44:43728) 2024-04-21T23:11:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.647+0000 7f3badb82640 1 -- 172.21.15.44:0/1489977560 learned_addr learned my addr 172.21.15.44:0/1489977560 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.647+0000 7f3bad381640 1 --2- 172.21.15.44:0/1489977560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ba81029b0 0x7f3ba8100600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:40452/0 (socket says 172.21.15.44:40452) 2024-04-21T23:11:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.647+0000 7f3bacb80640 1 --2- 172.21.15.44:0/1489977560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ba81032d0 0x7f3ba8100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.647+0000 7f3badb82640 1 -- 172.21.15.44:0/1489977560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ba81029b0 msgr2=0x7f3ba8100600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.647+0000 7f3badb82640 1 --2- 172.21.15.44:0/1489977560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ba81029b0 0x7f3ba8100600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.647+0000 7f3badb82640 1 -- 172.21.15.44:0/1489977560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ba81032d0 msgr2=0x7f3ba8100b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:09.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.647+0000 7f3badb82640 1 --2- 172.21.15.44:0/1489977560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ba81032d0 0x7f3ba8100b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.647+0000 7f3badb82640 1 -- 172.21.15.44:0/1489977560 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ba81a6590 con 0x7f3ba810bd80 2024-04-21T23:11:09.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.647+0000 7f3badb82640 1 --2- 172.21.15.44:0/1489977560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ba810bd80 0x7f3ba8101080 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3ba000e9d0 tx=0x7f3ba000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:09.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.647+0000 7f3b9e7fc640 1 -- 172.21.15.44:0/1489977560 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba000cd80 con 0x7f3ba810bd80 2024-04-21T23:11:09.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.647+0000 7f3b9e7fc640 1 -- 172.21.15.44:0/1489977560 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ba0005020 con 0x7f3ba810bd80 2024-04-21T23:11:09.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.647+0000 7f3baf5e5640 1 -- 172.21.15.44:0/1489977560 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ba81a6880 con 0x7f3ba810bd80 2024-04-21T23:11:09.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.647+0000 7f3b9e7fc640 1 -- 172.21.15.44:0/1489977560 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba00116d0 con 0x7f3ba810bd80 2024-04-21T23:11:09.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.648+0000 7f3baf5e5640 1 -- 172.21.15.44:0/1489977560 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ba81a6da0 con 0x7f3ba810bd80 2024-04-21T23:11:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.650+0000 7f3b9e7fc640 1 -- 172.21.15.44:0/1489977560 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3ba00051d0 con 0x7f3ba810bd80 2024-04-21T23:11:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.650+0000 7f3bb21d1740 1 Processor -- start 2024-04-21T23:11:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.650+0000 7f3bb21d1740 1 -- start start 2024-04-21T23:11:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.650+0000 7f3b9e7fc640 1 --2- 172.21.15.44:0/1489977560 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3b940a2d70 0x7f3b940a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.650+0000 7f3b9e7fc640 1 -- 172.21.15.44:0/1489977560 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7f3ba00cfc10 con 0x7f3ba810bd80 2024-04-21T23:11:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3bacb80640 1 --2- 172.21.15.44:0/1489977560 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3b940a2d70 0x7f3b940a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3bb21d1740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99cb20 0x561bfa99cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3bb21d1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99d440 0x561bfa9a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3bb21d1740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9a8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3bb21d1740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561bfa900b70 con 0x561bfa9a6210 2024-04-21T23:11:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3bb21d1740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x561bfa8a9470 con 0x561bfa99cb20 2024-04-21T23:11:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3bb21d1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x561bfa8fcb30 con 0x561bfa99d440 2024-04-21T23:11:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3badb82640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9a8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3bad381640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99cb20 0x561bfa99cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3bad381640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99cb20 0x561bfa99cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:48972/0 (socket says 172.21.15.44:48972) 2024-04-21T23:11:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3bad381640 1 -- 172.21.15.44:0/890143116 learned_addr learned my addr 172.21.15.44:0/890143116 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3badb82640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9a8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43732/0 (socket says 172.21.15.44:43732) 2024-04-21T23:11:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3bacb80640 1 --2- 172.21.15.44:0/890143116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99d440 0x561bfa9a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3bad381640 1 -- 172.21.15.44:0/890143116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99d440 msgr2=0x561bfa9a5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3bad381640 1 --2- 172.21.15.44:0/890143116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99d440 0x561bfa9a5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3bad381640 1 -- 172.21.15.44:0/890143116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 msgr2=0x561bfa9a8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3bad381640 1 --2- 172.21.15.44:0/890143116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9a8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3bacb80640 1 --2- 172.21.15.44:0/1489977560 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3b940a2d70 0x7f3b940a5230 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3b9802efd0 tx=0x7f3b98005e70 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3bad381640 1 -- 172.21.15.44:0/890143116 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bfa615210 con 0x561bfa99cb20 2024-04-21T23:11:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3badb82640 1 --2- 172.21.15.44:0/890143116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9a8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3bad381640 1 --2- 172.21.15.44:0/890143116 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99cb20 0x561bfa99cf00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4009cf0 tx=0x7f3ba4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9330f0fa97a55c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.651+0000 7f3b937fe640 1 -- 172.21.15.44:0/890143116 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4004230 con 0x561bfa99cb20 2024-04-21T23:11:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.652+0000 7f3b937fe640 1 -- 172.21.15.44:0/890143116 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ba40043d0 con 0x561bfa99cb20 2024-04-21T23:11:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.652+0000 7f3b937fe640 1 -- 172.21.15.44:0/890143116 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba40198c0 con 0x561bfa99cb20 2024-04-21T23:11:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.652+0000 7f3bb21d1740 1 -- 172.21.15.44:0/890143116 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99cb20 msgr2=0x561bfa99cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.652+0000 7f3bb21d1740 1 --2- 172.21.15.44:0/890143116 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99cb20 0x561bfa99cf00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4009cf0 tx=0x7f3ba4011810 comp rx=0 tx=0).stop 2024-04-21T23:11:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.652+0000 7f3bb21d1740 1 -- 172.21.15.44:0/890143116 shutdown_connections 2024-04-21T23:11:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.652+0000 7f3bb21d1740 1 --2- 172.21.15.44:0/890143116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99d440 0x561bfa9a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.652+0000 7f3bb21d1740 1 --2- 172.21.15.44:0/890143116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9a8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.652+0000 7f3bb21d1740 1 --2- 172.21.15.44:0/890143116 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99cb20 0x561bfa99cf00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.652+0000 7f3bb21d1740 1 -- 172.21.15.44:0/890143116 >> 172.21.15.44:0/890143116 conn(0x561bfa997440 msgr2=0x561bfa997820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.652+0000 7f3bb21d1740 1 -- 172.21.15.44:0/890143116 shutdown_connections 2024-04-21T23:11:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.652+0000 7f3bb21d1740 1 -- 172.21.15.44:0/890143116 wait complete. 2024-04-21T23:11:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.653+0000 7f3bb21d1740 1 -- start start 2024-04-21T23:11:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.653+0000 7f3bb21d1740 1 Processor -- start 2024-04-21T23:11:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.653+0000 7f3bb21d1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99cb20 0x561bfa9a31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.653+0000 7f3bb21d1740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99d440 0x561bfa9a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.653+0000 7f3bb21d1740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9a3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.653+0000 7f3bb21d1740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x561bfa9a8e40 con 0x561bfa9a6210 2024-04-21T23:11:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.653+0000 7f3bb21d1740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x561bfa8bca10 con 0x561bfa99d440 2024-04-21T23:11:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.653+0000 7f3bb21d1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x561bfa9a8fc0 con 0x561bfa99cb20 2024-04-21T23:11:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.653+0000 7f3bb21d1740 10 client.0 _ll_register_callbacks cb 0x561bfa8f7b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.653+0000 7f3badb82640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9a3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.653+0000 7f3bad381640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99cb20 0x561bfa9a31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.653+0000 7f3badb82640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9a3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43738/0 (socket says 172.21.15.44:43738) 2024-04-21T23:11:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.653+0000 7f3badb82640 1 -- 172.21.15.44:0/2548728580 learned_addr learned my addr 172.21.15.44:0/2548728580 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.653+0000 7f3bacb80640 1 --2- 172.21.15.44:0/2548728580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99d440 0x561bfa9a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3badb82640 1 -- 172.21.15.44:0/2548728580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99cb20 msgr2=0x561bfa9a31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3badb82640 1 --2- 172.21.15.44:0/2548728580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99cb20 0x561bfa9a31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3badb82640 1 -- 172.21.15.44:0/2548728580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99d440 msgr2=0x561bfa9a36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3badb82640 1 --2- 172.21.15.44:0/2548728580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99d440 0x561bfa9a36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3badb82640 1 -- 172.21.15.44:0/2548728580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bfa6e1620 con 0x561bfa9a6210 2024-04-21T23:11:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3badb82640 1 --2- 172.21.15.44:0/2548728580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9a3c30 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3ba000c640 tx=0x7f3ba00a4ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=791ab6cdad460604 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3b90ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:11:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3b90ff9640 1 -- 172.21.15.44:0/2548728580 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba001f030 con 0x561bfa9a6210 2024-04-21T23:11:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3b90ff9640 1 -- 172.21.15.44:0/2548728580 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7f3ba00a4770 con 0x561bfa9a6210 2024-04-21T23:11:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3bb21d1740 1 -- 172.21.15.44:0/2548728580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ba40043d0 con 0x561bfa9a6210 2024-04-21T23:11:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3b90ff9640 1 -- 172.21.15.44:0/2548728580 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3ba009b290 con 0x561bfa9a6210 2024-04-21T23:11:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3bb21d1740 10 client.16203 fetch_fsmap learned FSMap version 23 2024-04-21T23:11:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3bb21d1740 1 -- 172.21.15.44:0/2548728580 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561bfa9b5000 con 0x561bfa9a6210 2024-04-21T23:11:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3b90ff9640 1 -- 172.21.15.44:0/2548728580 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ba40043d0 con 0x561bfa9a6210 2024-04-21T23:11:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3b90ff9640 1 -- 172.21.15.44:0/2548728580 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f3ba000f040 con 0x561bfa9a6210 2024-04-21T23:11:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3bb21d1740 10 client.16203 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T23:11:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3bb21d1740 10 client.16203 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3bb21d1740 20 client.16203 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3bb21d1740 1 --2- 172.21.15.44:0/2548728580 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x561bfa9b6470 0x561bfa9b88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3bb21d1740 4 client.16203 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:11:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.654+0000 7f3bb21d1740 1 -- 172.21.15.44:0/2548728580 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561bfa90e0b0 con 0x561bfa9b6470 2024-04-21T23:11:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.655+0000 7f3bad381640 1 --2- 172.21.15.44:0/2548728580 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x561bfa9b6470 0x561bfa9b88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.655+0000 7f3b90ff9640 10 client.16203 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.655+0000 7f3bad381640 1 --2- 172.21.15.44:0/2548728580 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x561bfa9b6470 0x561bfa9b88c0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9dbfe5d8cc18e915 server_cookie=bed19f026cf90f25 in_seq=0 out_seq=0 2024-04-21T23:11:09.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.657+0000 7f3b90ff9640 1 -- 172.21.15.44:0/2548728580 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561bfa90e0b0 con 0x561bfa9b6470 2024-04-21T23:11:09.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.657+0000 7f3b90ff9640 10 client.16203 handle_command_reply: tid=0 2024-04-21T23:11:09.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.657+0000 7f3bb21d1740 1 client.16203 shutdown 2024-04-21T23:11:09.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.657+0000 7f3bb21d1740 1 -- 172.21.15.44:0/2548728580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 msgr2=0x561bfa9a3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.657+0000 7f3bb21d1740 1 --2- 172.21.15.44:0/2548728580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9a3c30 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3ba000c640 tx=0x7f3ba00a4ac0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.657+0000 7f3bb21d1740 1 -- 172.21.15.44:0/2548728580 shutdown_connections 2024-04-21T23:11:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.657+0000 7f3bb21d1740 1 --2- 172.21.15.44:0/2548728580 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x561bfa9b6470 0x561bfa9b88c0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.657+0000 7f3bb21d1740 1 --2- 172.21.15.44:0/2548728580 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9a3c30 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.657+0000 7f3bb21d1740 1 --2- 172.21.15.44:0/2548728580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99d440 0x561bfa9a36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.657+0000 7f3b90ff9640 0 client.16203 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.657+0000 7f3bb21d1740 1 --2- 172.21.15.44:0/2548728580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99cb20 0x561bfa9a31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.657+0000 7f3bb21d1740 1 -- 172.21.15.44:0/2548728580 >> 172.21.15.44:0/2548728580 conn(0x561bfa997440 msgr2=0x561bfa999190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.657+0000 7f3bb21d1740 1 -- 172.21.15.44:0/2548728580 shutdown_connections 2024-04-21T23:11:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.657+0000 7f3bb21d1740 1 -- 172.21.15.44:0/2548728580 wait complete. 2024-04-21T23:11:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.657+0000 7f3bb21d1740 20 client.16203 trim_cache size 0 max 16384 2024-04-21T23:11:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.673+0000 7f3bb21d1740 1 Processor -- start 2024-04-21T23:11:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.673+0000 7f3bb21d1740 1 -- start start 2024-04-21T23:11:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.673+0000 7f3bb21d1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99cb20 0x561bfa9a58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.673+0000 7f3bb21d1740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99d440 0x561bfa9b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.673+0000 7f3bb21d1740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9be220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.673+0000 7f3bb21d1740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba81aee70 con 0x561bfa9a6210 2024-04-21T23:11:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.673+0000 7f3bb21d1740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba81b16f0 con 0x561bfa99d440 2024-04-21T23:11:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.673+0000 7f3bb21d1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba81af4e0 con 0x561bfa99cb20 2024-04-21T23:11:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.673+0000 7f3badb82640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9be220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.673+0000 7f3bacb80640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99d440 0x561bfa9b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.673+0000 7f3badb82640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9be220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43748/0 (socket says 172.21.15.44:43748) 2024-04-21T23:11:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.673+0000 7f3badb82640 1 -- 172.21.15.44:0/2791975737 learned_addr learned my addr 172.21.15.44:0/2791975737 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.673+0000 7f3bacb80640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99d440 0x561bfa9b1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:48998/0 (socket says 172.21.15.44:48998) 2024-04-21T23:11:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.673+0000 7f3bad381640 1 --2- 172.21.15.44:0/2791975737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99cb20 0x561bfa9a58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.673+0000 7f3badb82640 1 -- 172.21.15.44:0/2791975737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99cb20 msgr2=0x561bfa9a58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.673+0000 7f3badb82640 1 --2- 172.21.15.44:0/2791975737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99cb20 0x561bfa9a58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.673+0000 7f3badb82640 1 -- 172.21.15.44:0/2791975737 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99d440 msgr2=0x561bfa9b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.673+0000 7f3badb82640 1 --2- 172.21.15.44:0/2791975737 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99d440 0x561bfa9b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.673+0000 7f3badb82640 1 -- 172.21.15.44:0/2791975737 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bfa9a8ee0 con 0x561bfa9a6210 2024-04-21T23:11:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.673+0000 7f3bacb80640 1 --2- 172.21.15.44:0/2791975737 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99d440 0x561bfa9b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.674+0000 7f3badb82640 1 --2- 172.21.15.44:0/2791975737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9be220 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3ba009bb40 tx=0x7f3ba009bc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=459ce272673b18f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.674+0000 7f3b92ffd640 1 -- 172.21.15.44:0/2791975737 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0081030 con 0x561bfa9a6210 2024-04-21T23:11:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.674+0000 7f3b92ffd640 1 -- 172.21.15.44:0/2791975737 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ba00a1380 con 0x561bfa9a6210 2024-04-21T23:11:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.674+0000 7f3b92ffd640 1 -- 172.21.15.44:0/2791975737 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba001f070 con 0x561bfa9a6210 2024-04-21T23:11:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.674+0000 7f3bb21d1740 1 -- 172.21.15.44:0/2791975737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 msgr2=0x561bfa9be220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.674+0000 7f3bb21d1740 1 --2- 172.21.15.44:0/2791975737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9be220 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3ba009bb40 tx=0x7f3ba009bc10 comp rx=0 tx=0).stop 2024-04-21T23:11:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.674+0000 7f3bb21d1740 1 -- 172.21.15.44:0/2791975737 shutdown_connections 2024-04-21T23:11:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.674+0000 7f3bb21d1740 1 --2- 172.21.15.44:0/2791975737 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9be220 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.674+0000 7f3bb21d1740 1 --2- 172.21.15.44:0/2791975737 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99d440 0x561bfa9b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.675+0000 7f3bb21d1740 1 --2- 172.21.15.44:0/2791975737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99cb20 0x561bfa9a58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.675+0000 7f3bb21d1740 1 -- 172.21.15.44:0/2791975737 >> 172.21.15.44:0/2791975737 conn(0x561bfa997440 msgr2=0x561bfa9a1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.675+0000 7f3bb21d1740 1 -- 172.21.15.44:0/2791975737 shutdown_connections 2024-04-21T23:11:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.675+0000 7f3bb21d1740 1 -- 172.21.15.44:0/2791975737 wait complete. 2024-04-21T23:11:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.675+0000 7f3bb21d1740 1 -- start start 2024-04-21T23:11:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.675+0000 7f3bb21d1740 1 Processor -- start 2024-04-21T23:11:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.675+0000 7f3bb21d1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99cb20 0x561bfa9a9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.675+0000 7f3bb21d1740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99d440 0x561bfa9aa180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.675+0000 7f3bb21d1740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9aa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.675+0000 7f3bb21d1740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba81afbb0 con 0x561bfa9a6210 2024-04-21T23:11:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.675+0000 7f3bb21d1740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba81b1020 con 0x561bfa99d440 2024-04-21T23:11:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.675+0000 7f3bb21d1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba81b0280 con 0x561bfa99cb20 2024-04-21T23:11:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.675+0000 7f3bb21d1740 10 client.0 _ll_register_callbacks cb 0x7f3ba00a01b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.675+0000 7f3badb82640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9aa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:09.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.675+0000 7f3badb82640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9aa6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43756/0 (socket says 172.21.15.44:43756) 2024-04-21T23:11:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.675+0000 7f3badb82640 1 -- 172.21.15.44:0/4257430398 learned_addr learned my addr 172.21.15.44:0/4257430398 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.675+0000 7f3bacb80640 1 --2- 172.21.15.44:0/4257430398 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99d440 0x561bfa9aa180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.676+0000 7f3bad381640 1 --2- 172.21.15.44:0/4257430398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99cb20 0x561bfa9a9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.676+0000 7f3bacb80640 1 -- 172.21.15.44:0/4257430398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99cb20 msgr2=0x561bfa9a9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.676+0000 7f3bacb80640 1 --2- 172.21.15.44:0/4257430398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99cb20 0x561bfa9a9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.676+0000 7f3bacb80640 1 -- 172.21.15.44:0/4257430398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 msgr2=0x561bfa9aa6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.676+0000 7f3bacb80640 1 --2- 172.21.15.44:0/4257430398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9aa6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.676+0000 7f3bacb80640 1 -- 172.21.15.44:0/4257430398 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bfaa1bac0 con 0x561bfa99d440 2024-04-21T23:11:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.676+0000 7f3bad381640 1 --2- 172.21.15.44:0/4257430398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99cb20 0x561bfa9a9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.676+0000 7f3badb82640 1 --2- 172.21.15.44:0/4257430398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9aa6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.676+0000 7f3bacb80640 1 --2- 172.21.15.44:0/4257430398 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99d440 0x561bfa9aa180 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3b98000c00 tx=0x7f3b9802bcd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f41120d150deb2e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.676+0000 7f3b90ff9640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:11:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.676+0000 7f3b90ff9640 1 -- 172.21.15.44:0/4257430398 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b98004b90 con 0x561bfa99d440 2024-04-21T23:11:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.676+0000 7f3bb21d1740 1 -- 172.21.15.44:0/4257430398 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ba00a1380 con 0x561bfa99d440 2024-04-21T23:11:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.676+0000 7f3b90ff9640 1 -- 172.21.15.44:0/4257430398 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7f3b98004f10 con 0x561bfa99d440 2024-04-21T23:11:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.676+0000 7f3b90ff9640 1 -- 172.21.15.44:0/4257430398 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3b98005110 con 0x561bfa99d440 2024-04-21T23:11:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.676+0000 7f3bb21d1740 10 client.25778 fetch_fsmap learned FSMap version 23 2024-04-21T23:11:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.676+0000 7f3bb21d1740 1 -- 172.21.15.44:0/4257430398 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x561bfaa1cde0 con 0x561bfa99d440 2024-04-21T23:11:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.677+0000 7f3b90ff9640 1 -- 172.21.15.44:0/4257430398 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ba00a1380 con 0x561bfa99d440 2024-04-21T23:11:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.677+0000 7f3b90ff9640 1 -- 172.21.15.44:0/4257430398 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f3b98053020 con 0x561bfa99d440 2024-04-21T23:11:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.677+0000 7f3bb21d1740 10 client.25778 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T23:11:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.677+0000 7f3bb21d1740 10 client.25778 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.677+0000 7f3bb21d1740 20 client.25778 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.677+0000 7f3bb21d1740 1 --2- 172.21.15.44:0/4257430398 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x561bfaa1d4b0 0x561bfaa1f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.677+0000 7f3bb21d1740 4 client.25778 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:11:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.677+0000 7f3bb21d1740 1 -- 172.21.15.44:0/4257430398 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561bfa908400 con 0x561bfaa1d4b0 2024-04-21T23:11:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.677+0000 7f3bad381640 1 --2- 172.21.15.44:0/4257430398 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x561bfaa1d4b0 0x561bfaa1f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.677+0000 7f3b90ff9640 10 client.25778 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.677+0000 7f3bad381640 1 --2- 172.21.15.44:0/4257430398 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x561bfaa1d4b0 0x561bfaa1f8d0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c684202b7eaaeb31 server_cookie=445cb324b278c2c5 in_seq=0 out_seq=0 2024-04-21T23:11:09.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 160318 14.94 MB/sec warmup 59 sec latency 19.851 ms 2024-04-21T23:11:09.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.926+0000 7f3b90ff9640 1 -- 172.21.15.44:0/4257430398 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561bfa908400 con 0x561bfaa1d4b0 2024-04-21T23:11:09.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.926+0000 7f3b90ff9640 10 client.25778 handle_command_reply: tid=0 2024-04-21T23:11:09.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.926+0000 7f3bb21d1740 1 client.25778 shutdown 2024-04-21T23:11:09.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.927+0000 7f3bb21d1740 1 -- 172.21.15.44:0/4257430398 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99d440 msgr2=0x561bfa9aa180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:09.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.927+0000 7f3bb21d1740 1 --2- 172.21.15.44:0/4257430398 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99d440 0x561bfa9aa180 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3b98000c00 tx=0x7f3b9802bcd0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.927 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.927+0000 7f3bb21d1740 1 -- 172.21.15.44:0/4257430398 shutdown_connections 2024-04-21T23:11:09.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.927+0000 7f3bb21d1740 1 --2- 172.21.15.44:0/4257430398 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x561bfaa1d4b0 0x561bfaa1f8d0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.927+0000 7f3b90ff9640 0 client.25778 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:09.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.927+0000 7f3bb21d1740 1 --2- 172.21.15.44:0/4257430398 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x561bfa9a6210 0x561bfa9aa6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.927+0000 7f3bb21d1740 1 --2- 172.21.15.44:0/4257430398 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x561bfa99d440 0x561bfa9aa180 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.927+0000 7f3bb21d1740 1 --2- 172.21.15.44:0/4257430398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561bfa99cb20 0x561bfa9a9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.927+0000 7f3bb21d1740 1 -- 172.21.15.44:0/4257430398 >> 172.21.15.44:0/4257430398 conn(0x561bfa997440 msgr2=0x561bfa9a3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:09.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.927+0000 7f3bb21d1740 1 -- 172.21.15.44:0/4257430398 shutdown_connections 2024-04-21T23:11:09.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.927+0000 7f3bb21d1740 1 -- 172.21.15.44:0/4257430398 wait complete. 2024-04-21T23:11:09.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.927+0000 7f3bb21d1740 20 client.25778 trim_cache size 0 max 16384 2024-04-21T23:11:09.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:11:09.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:11:09.928 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:11:09.929 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:11:09.929 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.928+0000 7f3baf5e5640 1 -- 172.21.15.44:0/1489977560 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3b940a2d70 msgr2=0x7f3b940a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:09.929 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.928+0000 7f3baf5e5640 1 --2- 172.21.15.44:0/1489977560 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3b940a2d70 0x7f3b940a5230 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3b9802efd0 tx=0x7f3b98005e70 comp rx=0 tx=0).stop 2024-04-21T23:11:09.929 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.928+0000 7f3baf5e5640 1 -- 172.21.15.44:0/1489977560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ba810bd80 msgr2=0x7f3ba8101080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:09.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.928+0000 7f3baf5e5640 1 --2- 172.21.15.44:0/1489977560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ba810bd80 0x7f3ba8101080 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3ba000e9d0 tx=0x7f3ba000ee90 comp rx=0 tx=0).stop 2024-04-21T23:11:09.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.928+0000 7f3baf5e5640 1 -- 172.21.15.44:0/1489977560 shutdown_connections 2024-04-21T23:11:09.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.928+0000 7f3baf5e5640 1 --2- 172.21.15.44:0/1489977560 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3b940a2d70 0x7f3b940a5230 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.928+0000 7f3baf5e5640 1 --2- 172.21.15.44:0/1489977560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3ba810bd80 0x7f3ba8101080 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.928+0000 7f3baf5e5640 1 --2- 172.21.15.44:0/1489977560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3ba81032d0 0x7f3ba8100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.928+0000 7f3baf5e5640 1 --2- 172.21.15.44:0/1489977560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ba81029b0 0x7f3ba8100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:09.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.928+0000 7f3baf5e5640 1 -- 172.21.15.44:0/1489977560 >> 172.21.15.44:0/1489977560 conn(0x7f3ba8077760 msgr2=0x7f3ba80fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:09.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.928+0000 7f3baf5e5640 1 -- 172.21.15.44:0/1489977560 shutdown_connections 2024-04-21T23:11:09.930 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:09.928+0000 7f3baf5e5640 1 -- 172.21.15.44:0/1489977560 wait complete. 2024-04-21T23:11:10.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:10 smithi055 ceph-mon[31117]: pgmap v73: 129 pgs: 129 active+clean; 46 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 313 op/s 2024-04-21T23:11:10.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:10 smithi069 ceph-mon[31816]: pgmap v73: 129 pgs: 129 active+clean; 46 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 313 op/s 2024-04-21T23:11:10.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:10 smithi044 ceph-mon[27934]: pgmap v73: 129 pgs: 129 active+clean; 46 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 313 op/s 2024-04-21T23:11:10.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 163195 14.98 MB/sec warmup 60 sec latency 123.747 ms 2024-04-21T23:11:11.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 165567 14.88 MB/sec warmup 61 sec latency 25.606 ms 2024-04-21T23:11:12.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:12 smithi069 ceph-mon[31816]: pgmap v74: 129 pgs: 129 active+clean; 46 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 213 op/s 2024-04-21T23:11:12.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:12 smithi055 ceph-mon[31117]: pgmap v74: 129 pgs: 129 active+clean; 46 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 213 op/s 2024-04-21T23:11:12.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:12 smithi044 ceph-mon[27934]: pgmap v74: 129 pgs: 129 active+clean; 46 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 213 op/s 2024-04-21T23:11:12.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 167474 14.84 MB/sec warmup 62 sec latency 65.892 ms 2024-04-21T23:11:13.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 170347 14.88 MB/sec warmup 63 sec latency 77.110 ms 2024-04-21T23:11:14.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:14 smithi055 ceph-mon[31117]: pgmap v75: 129 pgs: 129 active+clean; 42 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.3 MiB/s wr, 457 op/s 2024-04-21T23:11:14.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:14 smithi069 ceph-mon[31816]: pgmap v75: 129 pgs: 129 active+clean; 42 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.3 MiB/s wr, 457 op/s 2024-04-21T23:11:14.486 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:14 smithi044 ceph-mon[27934]: pgmap v75: 129 pgs: 129 active+clean; 42 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.3 MiB/s wr, 457 op/s 2024-04-21T23:11:14.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 172771 14.79 MB/sec warmup 64 sec latency 23.488 ms 2024-04-21T23:11:15.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 174926 14.77 MB/sec warmup 65 sec latency 16.357 ms 2024-04-21T23:11:16.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:16 smithi069 ceph-mon[31816]: pgmap v76: 129 pgs: 129 active+clean; 42 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 342 op/s 2024-04-21T23:11:16.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:16 smithi055 ceph-mon[31117]: pgmap v76: 129 pgs: 129 active+clean; 42 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 342 op/s 2024-04-21T23:11:16.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:16 smithi044 ceph-mon[27934]: pgmap v76: 129 pgs: 129 active+clean; 42 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 342 op/s 2024-04-21T23:11:16.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 177914 14.80 MB/sec warmup 66 sec latency 21.561 ms 2024-04-21T23:11:17.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 181056 14.85 MB/sec warmup 67 sec latency 15.712 ms 2024-04-21T23:11:18.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:18 smithi055 ceph-mon[31117]: pgmap v77: 129 pgs: 129 active+clean; 45 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 397 op/s 2024-04-21T23:11:18.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:18 smithi069 ceph-mon[31816]: pgmap v77: 129 pgs: 129 active+clean; 45 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 397 op/s 2024-04-21T23:11:18.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:18 smithi044 ceph-mon[27934]: pgmap v77: 129 pgs: 129 active+clean; 45 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 397 op/s 2024-04-21T23:11:18.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 183210 14.75 MB/sec warmup 68 sec latency 61.329 ms 2024-04-21T23:11:19.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 185215 14.72 MB/sec warmup 69 sec latency 63.817 ms 2024-04-21T23:11:20.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:20 smithi055 ceph-mon[31117]: pgmap v78: 129 pgs: 129 active+clean; 49 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 398 op/s 2024-04-21T23:11:20.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:20 smithi069 ceph-mon[31816]: pgmap v78: 129 pgs: 129 active+clean; 49 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 398 op/s 2024-04-21T23:11:20.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:20 smithi044 ceph-mon[27934]: pgmap v78: 129 pgs: 129 active+clean; 49 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 398 op/s 2024-04-21T23:11:20.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 188404 14.78 MB/sec warmup 70 sec latency 18.736 ms 2024-04-21T23:11:21.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 191636 14.82 MB/sec warmup 71 sec latency 14.877 ms 2024-04-21T23:11:22.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:22 smithi055 ceph-mon[31117]: pgmap v79: 129 pgs: 129 active+clean; 49 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 380 op/s 2024-04-21T23:11:22.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:22 smithi069 ceph-mon[31816]: pgmap v79: 129 pgs: 129 active+clean; 49 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 380 op/s 2024-04-21T23:11:22.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:22 smithi044 ceph-mon[27934]: pgmap v79: 129 pgs: 129 active+clean; 49 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 380 op/s 2024-04-21T23:11:22.693 INFO:tasks.workunit.client.0.smithi044.stdout: 1 194045 14.74 MB/sec warmup 72 sec latency 21.581 ms 2024-04-21T23:11:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.020+0000 7f6eba2a4640 1 -- 172.21.15.44:0/2941126959 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb4100620 msgr2=0x7f6eb4100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.020+0000 7f6eba2a4640 1 --2- 172.21.15.44:0/2941126959 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb4100620 0x7f6eb4100aa0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8009930 tx=0x7f6ea802bc30 comp rx=0 tx=0).stop 2024-04-21T23:11:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.020+0000 7f6eba2a4640 1 -- 172.21.15.44:0/2941126959 shutdown_connections 2024-04-21T23:11:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.020+0000 7f6eba2a4640 1 --2- 172.21.15.44:0/2941126959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6eb4100fe0 0x7f6eb410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.020+0000 7f6eba2a4640 1 --2- 172.21.15.44:0/2941126959 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb4100620 0x7f6eb4100aa0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.020+0000 7f6eba2a4640 1 --2- 172.21.15.44:0/2941126959 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6eb4106630 0x7f6eb4106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.020+0000 7f6eba2a4640 1 -- 172.21.15.44:0/2941126959 >> 172.21.15.44:0/2941126959 conn(0x7f6eb4077760 msgr2=0x7f6eb4077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.020+0000 7f6eba2a4640 1 -- 172.21.15.44:0/2941126959 shutdown_connections 2024-04-21T23:11:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.020+0000 7f6eba2a4640 1 -- 172.21.15.44:0/2941126959 wait complete. 2024-04-21T23:11:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.020+0000 7f6eba2a4640 1 Processor -- start 2024-04-21T23:11:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.020+0000 7f6eba2a4640 1 -- start start 2024-04-21T23:11:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.021+0000 7f6eba2a4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6eb4100620 0x7f6eb4197040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.021+0000 7f6eba2a4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb4100fe0 0x7f6eb4197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.021+0000 7f6eba2a4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6eb4106630 0x7f6eb419b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.021+0000 7f6eba2a4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb4110a40 con 0x7f6eb4100fe0 2024-04-21T23:11:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.021+0000 7f6eba2a4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb41108c0 con 0x7f6eb4106630 2024-04-21T23:11:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.021+0000 7f6eba2a4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb4110bc0 con 0x7f6eb4100620 2024-04-21T23:11:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.021+0000 7f6eb37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb4100fe0 0x7f6eb4197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.021+0000 7f6eb3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6eb4100620 0x7f6eb4197040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.021+0000 7f6eb37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb4100fe0 0x7f6eb4197580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58736/0 (socket says 172.21.15.44:58736) 2024-04-21T23:11:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.021+0000 7f6eb37fe640 1 -- 172.21.15.44:0/295522629 learned_addr learned my addr 172.21.15.44:0/295522629 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.021+0000 7f6eb8841640 1 --2- 172.21.15.44:0/295522629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6eb4106630 0x7f6eb419b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.021+0000 7f6eb3fff640 1 -- 172.21.15.44:0/295522629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6eb4106630 msgr2=0x7f6eb419b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.021+0000 7f6eb3fff640 1 --2- 172.21.15.44:0/295522629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6eb4106630 0x7f6eb419b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.021+0000 7f6eb3fff640 1 -- 172.21.15.44:0/295522629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb4100fe0 msgr2=0x7f6eb4197580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.021+0000 7f6eb3fff640 1 --2- 172.21.15.44:0/295522629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb4100fe0 0x7f6eb4197580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.021+0000 7f6eb3fff640 1 -- 172.21.15.44:0/295522629 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb419c030 con 0x7f6eb4100620 2024-04-21T23:11:23.024 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.021+0000 7f6eb37fe640 1 --2- 172.21.15.44:0/295522629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb4100fe0 0x7f6eb4197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.021+0000 7f6eb3fff640 1 --2- 172.21.15.44:0/295522629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6eb4100620 0x7f6eb4197040 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6ea400e9f0 tx=0x7f6ea400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.022+0000 7f6eb17fa640 1 -- 172.21.15.44:0/295522629 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea400ccc0 con 0x7f6eb4100620 2024-04-21T23:11:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.022+0000 7f6eb17fa640 1 -- 172.21.15.44:0/295522629 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea4005020 con 0x7f6eb4100620 2024-04-21T23:11:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.022+0000 7f6eba2a4640 1 -- 172.21.15.44:0/295522629 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eb419c320 con 0x7f6eb4100620 2024-04-21T23:11:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.022+0000 7f6eb17fa640 1 -- 172.21.15.44:0/295522629 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea40049e0 con 0x7f6eb4100620 2024-04-21T23:11:23.025 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.022+0000 7f6eba2a4640 1 -- 172.21.15.44:0/295522629 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eb4068f60 con 0x7f6eb4100620 2024-04-21T23:11:23.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.024+0000 7f6eb17fa640 1 -- 172.21.15.44:0/295522629 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6ea4005560 con 0x7f6eb4100620 2024-04-21T23:11:23.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.024+0000 7f6ebcf6a740 1 Processor -- start 2024-04-21T23:11:23.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.024+0000 7f6ebcf6a740 1 -- start start 2024-04-21T23:11:23.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.024+0000 7f6ebcf6a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c88640 0x555f95c88a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:23.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.024+0000 7f6ebcf6a740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c88f60 0x555f95c960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:23.032 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.024+0000 7f6ebcf6a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c96620 0x555f95c98a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.024+0000 7f6ebcf6a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x555f95becb70 con 0x555f95c96620 2024-04-21T23:11:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.024+0000 7f6ebcf6a740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x555f95b95470 con 0x555f95c88f60 2024-04-21T23:11:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.024+0000 7f6ebcf6a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555f95be8b30 con 0x555f95c88640 2024-04-21T23:11:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.024+0000 7f6eb37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c96620 0x555f95c98a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.024+0000 7f6eb37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c96620 0x555f95c98a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58744/0 (socket says 172.21.15.44:58744) 2024-04-21T23:11:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.024+0000 7f6eb37fe640 1 -- 172.21.15.44:0/264685671 learned_addr learned my addr 172.21.15.44:0/264685671 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.025+0000 7f6eb3fff640 1 --2- 172.21.15.44:0/264685671 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c88f60 0x555f95c960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.025+0000 7f6eb8841640 1 --2- 172.21.15.44:0/264685671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c88640 0x555f95c88a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.025+0000 7f6eb17fa640 1 --2- 172.21.15.44:0/295522629 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6e9c0a2cf0 0x7f6e9c0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.025+0000 7f6eb37fe640 1 -- 172.21.15.44:0/264685671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c88640 msgr2=0x555f95c88a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.025+0000 7f6eb37fe640 1 --2- 172.21.15.44:0/264685671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c88640 0x555f95c88a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.025+0000 7f6eb37fe640 1 -- 172.21.15.44:0/264685671 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c88f60 msgr2=0x555f95c960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.025+0000 7f6eb37fe640 1 --2- 172.21.15.44:0/264685671 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c88f60 0x555f95c960e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.025+0000 7f6eb37fe640 1 -- 172.21.15.44:0/264685671 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f95901210 con 0x555f95c96620 2024-04-21T23:11:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.025+0000 7f6eb3fff640 1 --2- 172.21.15.44:0/264685671 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c88f60 0x555f95c960e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:11:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.025+0000 7f6eb8841640 1 --2- 172.21.15.44:0/264685671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c88640 0x555f95c88a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:11:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.025+0000 7f6eb8841640 1 --2- 172.21.15.44:0/295522629 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6e9c0a2cf0 0x7f6e9c0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.025+0000 7f6eb17fa640 1 -- 172.21.15.44:0/295522629 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7f6ea40cf6e0 con 0x7f6eb4100620 2024-04-21T23:11:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.025+0000 7f6eb37fe640 1 --2- 172.21.15.44:0/264685671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c96620 0x555f95c98a10 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8009230 tx=0x7f6ea8009670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da44c2624771d48 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.025+0000 7f6e9a7fc640 1 -- 172.21.15.44:0/264685671 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8004a70 con 0x555f95c96620 2024-04-21T23:11:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.025+0000 7f6e9a7fc640 1 -- 172.21.15.44:0/264685671 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea8004c10 con 0x555f95c96620 2024-04-21T23:11:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.025+0000 7f6e9a7fc640 1 -- 172.21.15.44:0/264685671 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8005320 con 0x555f95c96620 2024-04-21T23:11:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.026+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/264685671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c96620 msgr2=0x555f95c98a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:23.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.026+0000 7f6ebcf6a740 1 --2- 172.21.15.44:0/264685671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c96620 0x555f95c98a10 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8009230 tx=0x7f6ea8009670 comp rx=0 tx=0).stop 2024-04-21T23:11:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.026+0000 7f6eb8841640 1 --2- 172.21.15.44:0/295522629 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6e9c0a2cf0 0x7f6e9c0a51b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6eac005fd0 tx=0x7f6eac014040 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.026+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/264685671 shutdown_connections 2024-04-21T23:11:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.026+0000 7f6ebcf6a740 1 --2- 172.21.15.44:0/264685671 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c96620 0x555f95c98a10 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.026+0000 7f6ebcf6a740 1 --2- 172.21.15.44:0/264685671 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c88f60 0x555f95c960e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.026+0000 7f6ebcf6a740 1 --2- 172.21.15.44:0/264685671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c88640 0x555f95c88a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.026+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/264685671 >> 172.21.15.44:0/264685671 conn(0x555f95c83440 msgr2=0x555f95c83820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.026+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/264685671 shutdown_connections 2024-04-21T23:11:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.026+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/264685671 wait complete. 2024-04-21T23:11:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.026+0000 7f6ebcf6a740 1 -- start start 2024-04-21T23:11:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.027+0000 7f6ebcf6a740 1 Processor -- start 2024-04-21T23:11:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.027+0000 7f6ebcf6a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c88640 0x555f95c93650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.027+0000 7f6ebcf6a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c88f60 0x555f95c93b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.027+0000 7f6ebcf6a740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c96620 0x555f95c940d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.027+0000 7f6ebcf6a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x555f95c99180 con 0x555f95c88f60 2024-04-21T23:11:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.027+0000 7f6ebcf6a740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x555f95ba8a10 con 0x555f95c96620 2024-04-21T23:11:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.027+0000 7f6ebcf6a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555f95c99300 con 0x555f95c88640 2024-04-21T23:11:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.027+0000 7f6ebcf6a740 10 client.0 _ll_register_callbacks cb 0x555f95be3b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.027+0000 7f6eb37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c88f60 0x555f95c93b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.027+0000 7f6eb37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c88f60 0x555f95c93b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58758/0 (socket says 172.21.15.44:58758) 2024-04-21T23:11:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.027+0000 7f6eb37fe640 1 -- 172.21.15.44:0/2491747485 learned_addr learned my addr 172.21.15.44:0/2491747485 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.027+0000 7f6eb8841640 1 --2- 172.21.15.44:0/2491747485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c96620 0x555f95c940d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.027+0000 7f6eb3fff640 1 --2- 172.21.15.44:0/2491747485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c88640 0x555f95c93650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.027+0000 7f6eb37fe640 1 -- 172.21.15.44:0/2491747485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c88640 msgr2=0x555f95c93650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.027+0000 7f6eb37fe640 1 --2- 172.21.15.44:0/2491747485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c88640 0x555f95c93650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.027+0000 7f6eb37fe640 1 -- 172.21.15.44:0/2491747485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c96620 msgr2=0x555f95c940d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:23.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.027+0000 7f6eb37fe640 1 --2- 172.21.15.44:0/2491747485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c96620 0x555f95c940d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.027+0000 7f6eb37fe640 1 -- 172.21.15.44:0/2491747485 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f959cd620 con 0x555f95c88f60 2024-04-21T23:11:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.027+0000 7f6eb8841640 1 --2- 172.21.15.44:0/2491747485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c96620 0x555f95c940d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.027+0000 7f6eb37fe640 1 --2- 172.21.15.44:0/2491747485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c88f60 0x555f95c93b90 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8005940 tx=0x7f6ea8004ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca9f10dbf7a371a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.028+0000 7f6e937fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:11:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.028+0000 7f6e937fe640 1 -- 172.21.15.44:0/2491747485 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8053060 con 0x555f95c88f60 2024-04-21T23:11:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.028+0000 7f6e937fe640 1 -- 172.21.15.44:0/2491747485 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7f6ea80055a0 con 0x555f95c88f60 2024-04-21T23:11:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.028+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/2491747485 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ea8004c10 con 0x555f95c88f60 2024-04-21T23:11:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.028+0000 7f6e937fe640 1 -- 172.21.15.44:0/2491747485 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6ea804a050 con 0x555f95c88f60 2024-04-21T23:11:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.028+0000 7f6ebcf6a740 10 client.16221 fetch_fsmap learned FSMap version 23 2024-04-21T23:11:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.028+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/2491747485 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x555f95c99180 con 0x555f95c88f60 2024-04-21T23:11:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.028+0000 7f6e937fe640 1 -- 172.21.15.44:0/2491747485 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea8004c10 con 0x555f95c88f60 2024-04-21T23:11:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.028+0000 7f6e937fe640 1 -- 172.21.15.44:0/2491747485 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f6ea803b440 con 0x555f95c88f60 2024-04-21T23:11:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.028+0000 7f6ebcf6a740 10 client.16221 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T23:11:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.028+0000 7f6ebcf6a740 10 client.16221 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:23.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.028+0000 7f6ebcf6a740 20 client.16221 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.028+0000 7f6ebcf6a740 1 --2- 172.21.15.44:0/2491747485 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x555f95ca65e0 0x555f95ca89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.028+0000 7f6ebcf6a740 4 client.16221 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:11:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.028+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/2491747485 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f95bfa0b0 con 0x555f95ca65e0 2024-04-21T23:11:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.029+0000 7f6eb3fff640 1 --2- 172.21.15.44:0/2491747485 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x555f95ca65e0 0x555f95ca89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.029+0000 7f6eb3fff640 1 --2- 172.21.15.44:0/2491747485 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x555f95ca65e0 0x555f95ca89d0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17dac8d9da05f723 server_cookie=6f1ed8d4e7e166cc in_seq=0 out_seq=0 2024-04-21T23:11:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.029+0000 7f6e937fe640 10 client.16221 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.030+0000 7f6e937fe640 1 -- 172.21.15.44:0/2491747485 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555f95bfa0b0 con 0x555f95ca65e0 2024-04-21T23:11:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.030+0000 7f6e937fe640 10 client.16221 handle_command_reply: tid=0 2024-04-21T23:11:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.030+0000 7f6ebcf6a740 1 client.16221 shutdown 2024-04-21T23:11:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.031+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/2491747485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c88f60 msgr2=0x555f95c93b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.031+0000 7f6ebcf6a740 1 --2- 172.21.15.44:0/2491747485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c88f60 0x555f95c93b90 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8005940 tx=0x7f6ea8004ec0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.031+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/2491747485 shutdown_connections 2024-04-21T23:11:23.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.031+0000 7f6ebcf6a740 1 --2- 172.21.15.44:0/2491747485 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x555f95ca65e0 0x555f95ca89d0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.031+0000 7f6ebcf6a740 1 --2- 172.21.15.44:0/2491747485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c96620 0x555f95c940d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.031+0000 7f6ebcf6a740 1 --2- 172.21.15.44:0/2491747485 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c88f60 0x555f95c93b90 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.031+0000 7f6e937fe640 0 client.16221 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:23.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.031+0000 7f6ebcf6a740 1 --2- 172.21.15.44:0/2491747485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c88640 0x555f95c93650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.031+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/2491747485 >> 172.21.15.44:0/2491747485 conn(0x555f95c83440 msgr2=0x555f95c85190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:23.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.031+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/2491747485 shutdown_connections 2024-04-21T23:11:23.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.031+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/2491747485 wait complete. 2024-04-21T23:11:23.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.031+0000 7f6ebcf6a740 20 client.16221 trim_cache size 0 max 16384 2024-04-21T23:11:23.050 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.046+0000 7f6ebcf6a740 1 Processor -- start 2024-04-21T23:11:23.050 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.046+0000 7f6ebcf6a740 1 -- start start 2024-04-21T23:11:23.050 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.046+0000 7f6ebcf6a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c88640 0x555f95c95d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:23.050 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.046+0000 7f6ebcf6a740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c88f60 0x555f95ca2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:23.050 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.046+0000 7f6ebcf6a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c96620 0x555f95cae0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:23.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.046+0000 7f6ebcf6a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb41aa960 con 0x555f95c96620 2024-04-21T23:11:23.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.046+0000 7f6ebcf6a740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb41ad1e0 con 0x555f95c88f60 2024-04-21T23:11:23.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.046+0000 7f6ebcf6a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb41aafd0 con 0x555f95c88640 2024-04-21T23:11:23.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.046+0000 7f6eb8841640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c96620 0x555f95cae0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:23.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.046+0000 7f6eb3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c88640 0x555f95c95d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:23.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.046+0000 7f6eb8841640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c96620 0x555f95cae0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58770/0 (socket says 172.21.15.44:58770) 2024-04-21T23:11:23.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.046+0000 7f6eb8841640 1 -- 172.21.15.44:0/2072907475 learned_addr learned my addr 172.21.15.44:0/2072907475 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:23.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.046+0000 7f6eb37fe640 1 --2- 172.21.15.44:0/2072907475 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c88f60 0x555f95ca2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:23.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.046+0000 7f6eb3fff640 1 -- 172.21.15.44:0/2072907475 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c88f60 msgr2=0x555f95ca2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:23.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.046+0000 7f6eb3fff640 1 --2- 172.21.15.44:0/2072907475 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c88f60 0x555f95ca2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.046+0000 7f6eb3fff640 1 -- 172.21.15.44:0/2072907475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c96620 msgr2=0x555f95cae0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:23.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.046+0000 7f6eb3fff640 1 --2- 172.21.15.44:0/2072907475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c96620 0x555f95cae0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.046+0000 7f6eb3fff640 1 -- 172.21.15.44:0/2072907475 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f95c8e3f0 con 0x555f95c88640 2024-04-21T23:11:23.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.047+0000 7f6eb3fff640 1 --2- 172.21.15.44:0/2072907475 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c88640 0x555f95c95d80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6ea40cca90 tx=0x7f6ea4086040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=124f1eaa47907138 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:23.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.047+0000 7f6e99ffb640 1 -- 172.21.15.44:0/2072907475 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea401c040 con 0x555f95c88640 2024-04-21T23:11:23.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.047+0000 7f6e99ffb640 1 -- 172.21.15.44:0/2072907475 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea40a44d0 con 0x555f95c88640 2024-04-21T23:11:23.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.047+0000 7f6e99ffb640 1 -- 172.21.15.44:0/2072907475 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea4097ed0 con 0x555f95c88640 2024-04-21T23:11:23.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.047+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/2072907475 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c88640 msgr2=0x555f95c95d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:23.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.047+0000 7f6ebcf6a740 1 --2- 172.21.15.44:0/2072907475 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c88640 0x555f95c95d80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6ea40cca90 tx=0x7f6ea4086040 comp rx=0 tx=0).stop 2024-04-21T23:11:23.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.047+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/2072907475 shutdown_connections 2024-04-21T23:11:23.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.047+0000 7f6ebcf6a740 1 --2- 172.21.15.44:0/2072907475 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c96620 0x555f95cae0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.047+0000 7f6ebcf6a740 1 --2- 172.21.15.44:0/2072907475 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c88f60 0x555f95ca2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.047+0000 7f6ebcf6a740 1 --2- 172.21.15.44:0/2072907475 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c88640 0x555f95c95d80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.047+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/2072907475 >> 172.21.15.44:0/2072907475 conn(0x555f95c83440 msgr2=0x555f95c91f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:23.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.047+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/2072907475 shutdown_connections 2024-04-21T23:11:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.048+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/2072907475 wait complete. 2024-04-21T23:11:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.048+0000 7f6ebcf6a740 1 -- start start 2024-04-21T23:11:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.048+0000 7f6ebcf6a740 1 Processor -- start 2024-04-21T23:11:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.048+0000 7f6ebcf6a740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c88640 0x555f95c9e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.048+0000 7f6ebcf6a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c88f60 0x555f95c99d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.048+0000 7f6ebcf6a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c96620 0x555f95c9a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.048+0000 7f6ebcf6a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb41ab6a0 con 0x555f95c88f60 2024-04-21T23:11:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.048+0000 7f6ebcf6a740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb41acb10 con 0x555f95c88640 2024-04-21T23:11:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.048+0000 7f6ebcf6a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb41abd70 con 0x555f95c96620 2024-04-21T23:11:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.048+0000 7f6ebcf6a740 10 client.0 _ll_register_callbacks cb 0x7f6ea8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.048+0000 7f6eb37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c88f60 0x555f95c99d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.048+0000 7f6eb3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c88640 0x555f95c9e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.048+0000 7f6eb37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c88f60 0x555f95c99d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58786/0 (socket says 172.21.15.44:58786) 2024-04-21T23:11:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.048+0000 7f6eb37fe640 1 -- 172.21.15.44:0/4027126002 learned_addr learned my addr 172.21.15.44:0/4027126002 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.049+0000 7f6eb8841640 1 --2- 172.21.15.44:0/4027126002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c96620 0x555f95c9a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.049+0000 7f6eb37fe640 1 -- 172.21.15.44:0/4027126002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c96620 msgr2=0x555f95c9a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.049+0000 7f6eb37fe640 1 --2- 172.21.15.44:0/4027126002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c96620 0x555f95c9a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.049+0000 7f6eb37fe640 1 -- 172.21.15.44:0/4027126002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c88640 msgr2=0x555f95c9e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.049+0000 7f6eb37fe640 1 --2- 172.21.15.44:0/4027126002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c88640 0x555f95c9e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.049+0000 7f6eb37fe640 1 -- 172.21.15.44:0/4027126002 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f95c9aad0 con 0x555f95c88f60 2024-04-21T23:11:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.049+0000 7f6eb8841640 1 --2- 172.21.15.44:0/4027126002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c96620 0x555f95c9a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:11:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.049+0000 7f6eb37fe640 1 --2- 172.21.15.44:0/4027126002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c88f60 0x555f95c99d40 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f6ea803b890 tx=0x7f6ea8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c4da437c985b70 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.049+0000 7f6e937fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:11:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.049+0000 7f6e937fe640 1 -- 172.21.15.44:0/4027126002 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8053040 con 0x555f95c88f60 2024-04-21T23:11:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.049+0000 7f6e937fe640 1 -- 172.21.15.44:0/4027126002 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9445+0+0 (secure 0 0 0) 0x7f6ea803ba80 con 0x555f95c88f60 2024-04-21T23:11:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.049+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/4027126002 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ea40a44d0 con 0x555f95c88f60 2024-04-21T23:11:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.049+0000 7f6e937fe640 1 -- 172.21.15.44:0/4027126002 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6ea802cd90 con 0x555f95c88f60 2024-04-21T23:11:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.049+0000 7f6ebcf6a740 10 client.16227 fetch_fsmap learned FSMap version 23 2024-04-21T23:11:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.049+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/4027126002 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x555f95d0c9d0 con 0x555f95c88f60 2024-04-21T23:11:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.050+0000 7f6e937fe640 1 -- 172.21.15.44:0/4027126002 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea40a44d0 con 0x555f95c88f60 2024-04-21T23:11:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.050+0000 7f6e937fe640 1 -- 172.21.15.44:0/4027126002 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 23) ==== 8330+0+0 (secure 0 0 0) 0x7f6ea8003780 con 0x555f95c88f60 2024-04-21T23:11:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.050+0000 7f6ebcf6a740 10 client.16227 fetch_fsmap finished waiting for FSMap version 23 2024-04-21T23:11:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.050+0000 7f6ebcf6a740 10 client.16227 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.050+0000 7f6ebcf6a740 20 client.16227 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.050+0000 7f6ebcf6a740 1 --2- 172.21.15.44:0/4027126002 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x555f95d0d0c0 0x555f95d0f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.050+0000 7f6ebcf6a740 4 client.16227 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:11:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.050+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/4027126002 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555f95bf4400 con 0x555f95d0d0c0 2024-04-21T23:11:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.050+0000 7f6eb3fff640 1 --2- 172.21.15.44:0/4027126002 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x555f95d0d0c0 0x555f95d0f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.050+0000 7f6e937fe640 10 client.16227 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.050+0000 7f6eb3fff640 1 --2- 172.21.15.44:0/4027126002 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x555f95d0d0c0 0x555f95d0f4e0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dff95c38d84949b5 server_cookie=90d9af70c3b5e9ca in_seq=0 out_seq=0 2024-04-21T23:11:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.295+0000 7f6e937fe640 1 -- 172.21.15.44:0/4027126002 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555f95bf4400 con 0x555f95d0d0c0 2024-04-21T23:11:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.295+0000 7f6e937fe640 10 client.16227 handle_command_reply: tid=0 2024-04-21T23:11:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.295+0000 7f6ebcf6a740 1 client.16227 shutdown 2024-04-21T23:11:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.296+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/4027126002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c88f60 msgr2=0x555f95c99d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.296+0000 7f6ebcf6a740 1 --2- 172.21.15.44:0/4027126002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c88f60 0x555f95c99d40 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f6ea803b890 tx=0x7f6ea8005fd0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.296+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/4027126002 shutdown_connections 2024-04-21T23:11:23.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.296+0000 7f6ebcf6a740 1 --2- 172.21.15.44:0/4027126002 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x555f95d0d0c0 0x555f95d0f4e0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.296+0000 7f6e937fe640 0 client.16227 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:23.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.296+0000 7f6ebcf6a740 1 --2- 172.21.15.44:0/4027126002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555f95c96620 0x555f95c9a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.296+0000 7f6ebcf6a740 1 --2- 172.21.15.44:0/4027126002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555f95c88f60 0x555f95c99d40 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.296+0000 7f6ebcf6a740 1 --2- 172.21.15.44:0/4027126002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555f95c88640 0x555f95c9e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.296+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/4027126002 >> 172.21.15.44:0/4027126002 conn(0x555f95c83440 msgr2=0x555f95c81c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:23.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.296+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/4027126002 shutdown_connections 2024-04-21T23:11:23.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.296+0000 7f6ebcf6a740 1 -- 172.21.15.44:0/4027126002 wait complete. 2024-04-21T23:11:23.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.296+0000 7f6ebcf6a740 20 client.16227 trim_cache size 0 max 16384 2024-04-21T23:11:23.297 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:11:23.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:11:23.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:11:23.298 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:11:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.297+0000 7f6eba2a4640 1 -- 172.21.15.44:0/295522629 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6e9c0a2cf0 msgr2=0x7f6e9c0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.297+0000 7f6eba2a4640 1 --2- 172.21.15.44:0/295522629 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6e9c0a2cf0 0x7f6e9c0a51b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6eac005fd0 tx=0x7f6eac014040 comp rx=0 tx=0).stop 2024-04-21T23:11:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.297+0000 7f6eba2a4640 1 -- 172.21.15.44:0/295522629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6eb4100620 msgr2=0x7f6eb4197040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.297+0000 7f6eba2a4640 1 --2- 172.21.15.44:0/295522629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6eb4100620 0x7f6eb4197040 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6ea400e9f0 tx=0x7f6ea400eeb0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.297+0000 7f6eba2a4640 1 -- 172.21.15.44:0/295522629 shutdown_connections 2024-04-21T23:11:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.297+0000 7f6eba2a4640 1 --2- 172.21.15.44:0/295522629 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6e9c0a2cf0 0x7f6e9c0a51b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.297+0000 7f6eba2a4640 1 --2- 172.21.15.44:0/295522629 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6eb4106630 0x7f6eb419b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.297+0000 7f6eba2a4640 1 --2- 172.21.15.44:0/295522629 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6eb4100fe0 0x7f6eb4197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.297+0000 7f6eba2a4640 1 --2- 172.21.15.44:0/295522629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6eb4100620 0x7f6eb4197040 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.297+0000 7f6eba2a4640 1 -- 172.21.15.44:0/295522629 >> 172.21.15.44:0/295522629 conn(0x7f6eb4077760 msgr2=0x7f6eb41032f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.297+0000 7f6eba2a4640 1 -- 172.21.15.44:0/295522629 shutdown_connections 2024-04-21T23:11:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:23.297+0000 7f6eba2a4640 1 -- 172.21.15.44:0/295522629 wait complete. 2024-04-21T23:11:23.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 195782 14.70 MB/sec warmup 73 sec latency 145.390 ms 2024-04-21T23:11:24.214 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:11:23 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: ignoring --setuser ceph since I am not root 2024-04-21T23:11:24.214 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:11:23 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: ignoring --setgroup ceph since I am not root 2024-04-21T23:11:24.214 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:11:23 smithi069 ceph-mds[63204]: -- 172.21.15.69:0/1315735616 <== mon.1 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5654e1625a40 con 0x5654e15f8c00 2024-04-21T23:11:24.214 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:11:23 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: starting mds.i at 2024-04-21T23:11:24.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:24 smithi044 ceph-mon[27934]: pgmap v80: 129 pgs: 129 active+clean; 54 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.4 MiB/s wr, 518 op/s 2024-04-21T23:11:24.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:24 smithi044 ceph-mon[27934]: daemon mds.i restarted 2024-04-21T23:11:24.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 198470 14.69 MB/sec warmup 74 sec latency 59.109 ms 2024-04-21T23:11:24.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:24 smithi055 ceph-mon[31117]: pgmap v80: 129 pgs: 129 active+clean; 54 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.4 MiB/s wr, 518 op/s 2024-04-21T23:11:24.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:24 smithi055 ceph-mon[31117]: daemon mds.i restarted 2024-04-21T23:11:24.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:24 smithi069 ceph-mon[31816]: pgmap v80: 129 pgs: 129 active+clean; 54 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.4 MiB/s wr, 518 op/s 2024-04-21T23:11:24.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:24 smithi069 ceph-mon[31816]: daemon mds.i restarted 2024-04-21T23:11:25.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:25 smithi044 ceph-mon[27934]: osdmap e97: 12 total, 12 up, 12 in 2024-04-21T23:11:25.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:25 smithi044 ceph-mon[27934]: mds.? [v2:172.21.15.69:6838/3162569119,v1:172.21.15.69:6839/3162569119] up:boot 2024-04-21T23:11:25.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:25 smithi044 ceph-mon[27934]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:11:25.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:25 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:11:25.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:25 smithi044 ceph-mon[27934]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:11:25.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 201017 14.65 MB/sec warmup 75 sec latency 26.743 ms 2024-04-21T23:11:25.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:25 smithi055 ceph-mon[31117]: osdmap e97: 12 total, 12 up, 12 in 2024-04-21T23:11:25.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:25 smithi055 ceph-mon[31117]: mds.? [v2:172.21.15.69:6838/3162569119,v1:172.21.15.69:6839/3162569119] up:boot 2024-04-21T23:11:25.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:25 smithi055 ceph-mon[31117]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:11:25.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:25 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:11:25.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:25 smithi055 ceph-mon[31117]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:11:25.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:25 smithi069 ceph-mon[31816]: osdmap e97: 12 total, 12 up, 12 in 2024-04-21T23:11:25.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:25 smithi069 ceph-mon[31816]: mds.? [v2:172.21.15.69:6838/3162569119,v1:172.21.15.69:6839/3162569119] up:boot 2024-04-21T23:11:25.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:25 smithi069 ceph-mon[31816]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:11:25.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:25 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:11:25.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:25 smithi069 ceph-mon[31816]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:11:26.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:26 smithi044 ceph-mon[27934]: pgmap v82: 129 pgs: 129 active+clean; 54 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 329 op/s 2024-04-21T23:11:26.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 203165 14.63 MB/sec warmup 76 sec latency 20.113 ms 2024-04-21T23:11:26.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:26 smithi055 ceph-mon[31117]: pgmap v82: 129 pgs: 129 active+clean; 54 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 329 op/s 2024-04-21T23:11:26.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:26 smithi069 ceph-mon[31816]: pgmap v82: 129 pgs: 129 active+clean; 54 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 329 op/s 2024-04-21T23:11:27.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 206237 14.68 MB/sec warmup 77 sec latency 10.030 ms 2024-04-21T23:11:28.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:28 smithi044 ceph-mon[27934]: pgmap v83: 129 pgs: 129 active+clean; 50 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.4 MiB/s wr, 444 op/s 2024-04-21T23:11:28.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 209337 14.71 MB/sec warmup 78 sec latency 18.936 ms 2024-04-21T23:11:28.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:28 smithi069 ceph-mon[31816]: pgmap v83: 129 pgs: 129 active+clean; 50 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.4 MiB/s wr, 444 op/s 2024-04-21T23:11:28.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:28 smithi055 ceph-mon[31117]: pgmap v83: 129 pgs: 129 active+clean; 50 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.4 MiB/s wr, 444 op/s 2024-04-21T23:11:29.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 210918 14.61 MB/sec warmup 79 sec latency 44.907 ms 2024-04-21T23:11:30.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:30 smithi044 ceph-mon[27934]: pgmap v84: 129 pgs: 129 active+clean; 43 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 526 op/s 2024-04-21T23:11:30.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 213500 14.61 MB/sec warmup 80 sec latency 32.067 ms 2024-04-21T23:11:30.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:30 smithi069 ceph-mon[31816]: pgmap v84: 129 pgs: 129 active+clean; 43 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 526 op/s 2024-04-21T23:11:30.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:30 smithi055 ceph-mon[31117]: pgmap v84: 129 pgs: 129 active+clean; 43 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 526 op/s 2024-04-21T23:11:31.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 216647 14.66 MB/sec warmup 81 sec latency 20.547 ms 2024-04-21T23:11:32.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:32 smithi044 ceph-mon[27934]: pgmap v85: 129 pgs: 129 active+clean; 43 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 526 op/s 2024-04-21T23:11:32.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 218840 14.57 MB/sec warmup 82 sec latency 17.865 ms 2024-04-21T23:11:32.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:32 smithi055 ceph-mon[31117]: pgmap v85: 129 pgs: 129 active+clean; 43 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 526 op/s 2024-04-21T23:11:32.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:32 smithi069 ceph-mon[31816]: pgmap v85: 129 pgs: 129 active+clean; 43 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 526 op/s 2024-04-21T23:11:33.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 220849 14.55 MB/sec warmup 83 sec latency 22.413 ms 2024-04-21T23:11:34.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:34 smithi044 ceph-mon[27934]: pgmap v86: 129 pgs: 129 active+clean; 48 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.9 MiB/s wr, 513 op/s 2024-04-21T23:11:34.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 223755 14.59 MB/sec warmup 84 sec latency 51.846 ms 2024-04-21T23:11:34.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:34 smithi069 ceph-mon[31816]: pgmap v86: 129 pgs: 129 active+clean; 48 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.9 MiB/s wr, 513 op/s 2024-04-21T23:11:34.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:34 smithi055 ceph-mon[31117]: pgmap v86: 129 pgs: 129 active+clean; 48 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.9 MiB/s wr, 513 op/s 2024-04-21T23:11:35.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 226390 14.55 MB/sec warmup 85 sec latency 30.819 ms 2024-04-21T23:11:36.020 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:11:36.021 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:11:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.127+0000 7f44a7840640 1 -- 172.21.15.44:0/760040882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f44a01032d0 msgr2=0x7f44a010b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.127+0000 7f44a7840640 1 --2- 172.21.15.44:0/760040882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f44a01032d0 0x7f44a010b7b0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f4490009900 tx=0x7f449002bcd0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.127+0000 7f44a7840640 1 -- 172.21.15.44:0/760040882 shutdown_connections 2024-04-21T23:11:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.127+0000 7f44a7840640 1 --2- 172.21.15.44:0/760040882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44a010bcf0 0x7f44a010e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.127+0000 7f44a7840640 1 --2- 172.21.15.44:0/760040882 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f44a01032d0 0x7f44a010b7b0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.127+0000 7f44a7840640 1 --2- 172.21.15.44:0/760040882 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44a01029b0 0x7f44a0102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.127+0000 7f44a7840640 1 -- 172.21.15.44:0/760040882 >> 172.21.15.44:0/760040882 conn(0x7f44a00777f0 msgr2=0x7f44a0077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.127+0000 7f44a7840640 1 -- 172.21.15.44:0/760040882 shutdown_connections 2024-04-21T23:11:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.127+0000 7f44a7840640 1 -- 172.21.15.44:0/760040882 wait complete. 2024-04-21T23:11:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.127+0000 7f44a7840640 1 Processor -- start 2024-04-21T23:11:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.127+0000 7f44a7840640 1 -- start start 2024-04-21T23:11:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44a7840640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44a01029b0 0x7f44a0198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44a7840640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f44a01032d0 0x7f44a0198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44a7840640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44a010bcf0 0x7f44a019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44a7840640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f44a0111af0 con 0x7f44a01032d0 2024-04-21T23:11:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44a7840640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f44a0111970 con 0x7f44a01029b0 2024-04-21T23:11:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44a7840640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f44a0111c70 con 0x7f44a010bcf0 2024-04-21T23:11:36.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44a4ddb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f44a01032d0 0x7f44a0198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44a55dc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44a01029b0 0x7f44a0198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44a4ddb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f44a01032d0 0x7f44a0198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38518/0 (socket says 172.21.15.44:38518) 2024-04-21T23:11:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44a4ddb640 1 -- 172.21.15.44:0/1507829975 learned_addr learned my addr 172.21.15.44:0/1507829975 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44a5ddd640 1 --2- 172.21.15.44:0/1507829975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44a010bcf0 0x7f44a019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44a4ddb640 1 -- 172.21.15.44:0/1507829975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44a010bcf0 msgr2=0x7f44a019c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44a4ddb640 1 --2- 172.21.15.44:0/1507829975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44a010bcf0 0x7f44a019c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44a4ddb640 1 -- 172.21.15.44:0/1507829975 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44a01029b0 msgr2=0x7f44a0198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44a4ddb640 1 --2- 172.21.15.44:0/1507829975 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44a01029b0 0x7f44a0198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44a4ddb640 1 -- 172.21.15.44:0/1507829975 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44a019d000 con 0x7f44a01032d0 2024-04-21T23:11:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44a4ddb640 1 --2- 172.21.15.44:0/1507829975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f44a01032d0 0x7f44a0198550 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f4490005940 tx=0x7f449002cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44967fc640 1 -- 172.21.15.44:0/1507829975 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4490041070 con 0x7f44a01032d0 2024-04-21T23:11:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44967fc640 1 -- 172.21.15.44:0/1507829975 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4490004450 con 0x7f44a01032d0 2024-04-21T23:11:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44a7840640 1 -- 172.21.15.44:0/1507829975 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44a019d290 con 0x7f44a01032d0 2024-04-21T23:11:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44967fc640 1 -- 172.21.15.44:0/1507829975 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4490003d80 con 0x7f44a01032d0 2024-04-21T23:11:36.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.128+0000 7f44a7840640 1 -- 172.21.15.44:0/1507829975 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44a0100530 con 0x7f44a01032d0 2024-04-21T23:11:36.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.130+0000 7f44967fc640 1 -- 172.21.15.44:0/1507829975 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f4490044070 con 0x7f44a01032d0 2024-04-21T23:11:36.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.131+0000 7f44967fc640 1 --2- 172.21.15.44:0/1507829975 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f448c0a2d00 0x7f448c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.131+0000 7f44aa42c740 1 Processor -- start 2024-04-21T23:11:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.131+0000 7f44aa42c740 1 -- start start 2024-04-21T23:11:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.131+0000 7f44967fc640 1 -- 172.21.15.44:0/1507829975 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f44900f11d0 con 0x7f44a01032d0 2024-04-21T23:11:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.131+0000 7f44a55dc640 1 --2- 172.21.15.44:0/1507829975 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f448c0a2d00 0x7f448c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f44aa42c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c2995cf30 0x560c2995d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f44aa42c740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995d850 0x560c29965ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f44aa42c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c29966430 0x560c29968820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f44aa42c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560c298c1560 con 0x560c29966430 2024-04-21T23:11:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f44aa42c740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x560c29869730 con 0x560c2995d850 2024-04-21T23:11:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f44aa42c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560c298bd520 con 0x560c2995cf30 2024-04-21T23:11:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f44a5ddd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c29966430 0x560c29968820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f44a4ddb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995d850 0x560c29965ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f44a5ddd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c29966430 0x560c29968820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38534/0 (socket says 172.21.15.44:38534) 2024-04-21T23:11:36.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f44a5ddd640 1 -- 172.21.15.44:0/3571600003 learned_addr learned my addr 172.21.15.44:0/3571600003 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:36.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f44a55dc640 1 --2- 172.21.15.44:0/1507829975 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f448c0a2d00 0x7f448c0a51c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4498009c20 tx=0x7f4498005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:36.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f44a55dc640 1 --2- 172.21.15.44:0/3571600003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c2995cf30 0x560c2995d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f44a5ddd640 1 -- 172.21.15.44:0/3571600003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c2995cf30 msgr2=0x560c2995d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f44a5ddd640 1 --2- 172.21.15.44:0/3571600003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c2995cf30 0x560c2995d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f44a5ddd640 1 -- 172.21.15.44:0/3571600003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995d850 msgr2=0x560c29965ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f44a5ddd640 1 --2- 172.21.15.44:0/3571600003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995d850 0x560c29965ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f44a5ddd640 1 -- 172.21.15.44:0/3571600003 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c2969f7d0 con 0x560c29966430 2024-04-21T23:11:36.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f44a4ddb640 1 --2- 172.21.15.44:0/3571600003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995d850 0x560c29965ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:36.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f44a5ddd640 1 --2- 172.21.15.44:0/3571600003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c29966430 0x560c29968820 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f449c009f40 tx=0x7f449c009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adaf28ce5686368a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:36.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f448b7fe640 1 -- 172.21.15.44:0/3571600003 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f449c003c30 con 0x560c29966430 2024-04-21T23:11:36.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.132+0000 7f448b7fe640 1 -- 172.21.15.44:0/3571600003 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f449c01f070 con 0x560c29966430 2024-04-21T23:11:36.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.133+0000 7f448b7fe640 1 -- 172.21.15.44:0/3571600003 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f449c00ba00 con 0x560c29966430 2024-04-21T23:11:36.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.133+0000 7f44aa42c740 1 -- 172.21.15.44:0/3571600003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c29966430 msgr2=0x560c29968820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.133+0000 7f44aa42c740 1 --2- 172.21.15.44:0/3571600003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c29966430 0x560c29968820 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f449c009f40 tx=0x7f449c009500 comp rx=0 tx=0).stop 2024-04-21T23:11:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.133+0000 7f44aa42c740 1 -- 172.21.15.44:0/3571600003 shutdown_connections 2024-04-21T23:11:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.133+0000 7f44aa42c740 1 --2- 172.21.15.44:0/3571600003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c29966430 0x560c29968820 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.133+0000 7f44aa42c740 1 --2- 172.21.15.44:0/3571600003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995d850 0x560c29965ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.133+0000 7f44aa42c740 1 --2- 172.21.15.44:0/3571600003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c2995cf30 0x560c2995d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.133+0000 7f44aa42c740 1 -- 172.21.15.44:0/3571600003 >> 172.21.15.44:0/3571600003 conn(0x560c29957660 msgr2=0x560c29957a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.133+0000 7f44aa42c740 1 -- 172.21.15.44:0/3571600003 shutdown_connections 2024-04-21T23:11:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.133+0000 7f44aa42c740 1 -- 172.21.15.44:0/3571600003 wait complete. 2024-04-21T23:11:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.134+0000 7f44aa42c740 1 -- start start 2024-04-21T23:11:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.134+0000 7f44aa42c740 1 Processor -- start 2024-04-21T23:11:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.134+0000 7f44aa42c740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995cf30 0x560c29963490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.134+0000 7f44aa42c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995d850 0x560c299639d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.134+0000 7f44aa42c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c29966430 0x560c29963f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.134+0000 7f44aa42c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560c29969220 con 0x560c2995d850 2024-04-21T23:11:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.134+0000 7f44aa42c740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x560c2987ce00 con 0x560c2995cf30 2024-04-21T23:11:36.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.134+0000 7f44aa42c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560c299693a0 con 0x560c29966430 2024-04-21T23:11:36.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.134+0000 7f44aa42c740 10 client.0 _ll_register_callbacks cb 0x560c298b7eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:36.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.134+0000 7f44a4ddb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995d850 0x560c299639d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.134+0000 7f44a4ddb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995d850 0x560c299639d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38548/0 (socket says 172.21.15.44:38548) 2024-04-21T23:11:36.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.134+0000 7f44a55dc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995cf30 0x560c29963490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.134+0000 7f44a4ddb640 1 -- 172.21.15.44:0/1994822235 learned_addr learned my addr 172.21.15.44:0/1994822235 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:36.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.134+0000 7f44a5ddd640 1 --2- 172.21.15.44:0/1994822235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c29966430 0x560c29963f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f44a4ddb640 1 -- 172.21.15.44:0/1994822235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c29966430 msgr2=0x560c29963f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f44a4ddb640 1 --2- 172.21.15.44:0/1994822235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c29966430 0x560c29963f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.142 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f44a4ddb640 1 -- 172.21.15.44:0/1994822235 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995cf30 msgr2=0x560c29963490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f44a4ddb640 1 --2- 172.21.15.44:0/1994822235 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995cf30 0x560c29963490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f44a4ddb640 1 -- 172.21.15.44:0/1994822235 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c295d53f0 con 0x560c2995d850 2024-04-21T23:11:36.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f44a55dc640 1 --2- 172.21.15.44:0/1994822235 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995cf30 0x560c29963490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:36.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f44a4ddb640 1 --2- 172.21.15.44:0/1994822235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995d850 0x560c299639d0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f4490005d00 tx=0x7f44900bd2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=722ff00089ad5f04 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:36.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f4488ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:11:36.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f4488ff9640 1 -- 172.21.15.44:0/1994822235 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f449004b040 con 0x560c2995d850 2024-04-21T23:11:36.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f4488ff9640 1 -- 172.21.15.44:0/1994822235 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f44900c5e40 con 0x560c2995d850 2024-04-21T23:11:36.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f44aa42c740 1 -- 172.21.15.44:0/1994822235 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f449c01f070 con 0x560c2995d850 2024-04-21T23:11:36.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f4488ff9640 1 -- 172.21.15.44:0/1994822235 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4490004c80 con 0x560c2995d850 2024-04-21T23:11:36.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f44aa42c740 10 client.16245 fetch_fsmap learned FSMap version 25 2024-04-21T23:11:36.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f44aa42c740 1 -- 172.21.15.44:0/1994822235 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560c295950f0 con 0x560c2995d850 2024-04-21T23:11:36.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f4488ff9640 1 -- 172.21.15.44:0/1994822235 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f449c01f070 con 0x560c2995d850 2024-04-21T23:11:36.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f4488ff9640 1 -- 172.21.15.44:0/1994822235 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f44900d6290 con 0x560c2995d850 2024-04-21T23:11:36.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f44aa42c740 10 client.16245 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:11:36.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f44aa42c740 10 client.16245 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:36.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f44aa42c740 20 client.16245 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:36.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f44aa42c740 1 --2- 172.21.15.44:0/1994822235 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560c29976670 0x560c29978ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f44aa42c740 4 client.16245 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:11:36.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.135+0000 7f44aa42c740 1 -- 172.21.15.44:0/1994822235 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c298ceaa0 con 0x560c29976670 2024-04-21T23:11:36.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.136+0000 7f44a55dc640 1 --2- 172.21.15.44:0/1994822235 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560c29976670 0x560c29978ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.136+0000 7f4488ff9640 10 client.16245 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:36.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.136+0000 7f44a55dc640 1 --2- 172.21.15.44:0/1994822235 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560c29976670 0x560c29978ac0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec8fc562f3947c68 server_cookie=ff6fc8d70a1a2cf2 in_seq=0 out_seq=0 2024-04-21T23:11:36.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.137+0000 7f4488ff9640 1 -- 172.21.15.44:0/1994822235 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560c298ceaa0 con 0x560c29976670 2024-04-21T23:11:36.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.137+0000 7f4488ff9640 10 client.16245 handle_command_reply: tid=0 2024-04-21T23:11:36.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.137+0000 7f44aa42c740 1 client.16245 shutdown 2024-04-21T23:11:36.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.137+0000 7f44aa42c740 1 -- 172.21.15.44:0/1994822235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995d850 msgr2=0x560c299639d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.137+0000 7f44aa42c740 1 --2- 172.21.15.44:0/1994822235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995d850 0x560c299639d0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f4490005d00 tx=0x7f44900bd2e0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.137+0000 7f44aa42c740 1 -- 172.21.15.44:0/1994822235 shutdown_connections 2024-04-21T23:11:36.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.137+0000 7f44aa42c740 1 --2- 172.21.15.44:0/1994822235 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560c29976670 0x560c29978ac0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.137+0000 7f44aa42c740 1 --2- 172.21.15.44:0/1994822235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c29966430 0x560c29963f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.137+0000 7f44aa42c740 1 --2- 172.21.15.44:0/1994822235 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995d850 0x560c299639d0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.137+0000 7f44aa42c740 1 --2- 172.21.15.44:0/1994822235 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995cf30 0x560c29963490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.137+0000 7f4488ff9640 0 client.16245 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:36.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.137+0000 7f44aa42c740 1 -- 172.21.15.44:0/1994822235 >> 172.21.15.44:0/1994822235 conn(0x560c29957660 msgr2=0x560c299593b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.137+0000 7f44aa42c740 1 -- 172.21.15.44:0/1994822235 shutdown_connections 2024-04-21T23:11:36.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.137+0000 7f44aa42c740 1 -- 172.21.15.44:0/1994822235 wait complete. 2024-04-21T23:11:36.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.138+0000 7f44aa42c740 20 client.16245 trim_cache size 0 max 16384 2024-04-21T23:11:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.155+0000 7f44aa42c740 1 Processor -- start 2024-04-21T23:11:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.155+0000 7f44aa42c740 1 -- start start 2024-04-21T23:11:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.155+0000 7f44aa42c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995cf30 0x560c29965bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.155+0000 7f44aa42c740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995d850 0x560c299623d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.155+0000 7f44aa42c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c29966430 0x560c299720e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.155+0000 7f44aa42c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f44a01aa990 con 0x560c2995cf30 2024-04-21T23:11:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.155+0000 7f44aa42c740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f44a01ad210 con 0x560c2995d850 2024-04-21T23:11:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.155+0000 7f44aa42c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f44a01ab000 con 0x560c29966430 2024-04-21T23:11:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.156+0000 7f44a55dc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995cf30 0x560c29965bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.156+0000 7f44a55dc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995cf30 0x560c29965bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38564/0 (socket says 172.21.15.44:38564) 2024-04-21T23:11:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.156+0000 7f44a55dc640 1 -- 172.21.15.44:0/2239692444 learned_addr learned my addr 172.21.15.44:0/2239692444 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.156+0000 7f44a4ddb640 1 --2- 172.21.15.44:0/2239692444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995d850 0x560c299623d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.156+0000 7f44a5ddd640 1 --2- 172.21.15.44:0/2239692444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c29966430 0x560c299720e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.156+0000 7f44a55dc640 1 -- 172.21.15.44:0/2239692444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c29966430 msgr2=0x560c299720e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.156+0000 7f44a55dc640 1 --2- 172.21.15.44:0/2239692444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c29966430 0x560c299720e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.156+0000 7f44a55dc640 1 -- 172.21.15.44:0/2239692444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995d850 msgr2=0x560c299623d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.156+0000 7f44a55dc640 1 --2- 172.21.15.44:0/2239692444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995d850 0x560c299623d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.156+0000 7f44a55dc640 1 -- 172.21.15.44:0/2239692444 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c29957190 con 0x560c2995cf30 2024-04-21T23:11:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.156+0000 7f44a4ddb640 1 --2- 172.21.15.44:0/2239692444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995d850 0x560c299623d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:11:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.156+0000 7f44a5ddd640 1 --2- 172.21.15.44:0/2239692444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c29966430 0x560c299720e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:11:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.156+0000 7f44a55dc640 1 --2- 172.21.15.44:0/2239692444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995cf30 0x560c29965bc0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f4498011ca0 tx=0x7f4498014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ac54e0e142c084 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.156+0000 7f448affd640 1 -- 172.21.15.44:0/2239692444 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f449802a360 con 0x560c2995cf30 2024-04-21T23:11:36.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.156+0000 7f448affd640 1 -- 172.21.15.44:0/2239692444 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44980035f0 con 0x560c2995cf30 2024-04-21T23:11:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.156+0000 7f448affd640 1 -- 172.21.15.44:0/2239692444 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4498027650 con 0x560c2995cf30 2024-04-21T23:11:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.157+0000 7f44aa42c740 1 -- 172.21.15.44:0/2239692444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995cf30 msgr2=0x560c29965bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.157+0000 7f44aa42c740 1 --2- 172.21.15.44:0/2239692444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995cf30 0x560c29965bc0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f4498011ca0 tx=0x7f4498014000 comp rx=0 tx=0).stop 2024-04-21T23:11:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.157+0000 7f44aa42c740 1 -- 172.21.15.44:0/2239692444 shutdown_connections 2024-04-21T23:11:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.157+0000 7f44aa42c740 1 --2- 172.21.15.44:0/2239692444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c29966430 0x560c299720e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.157+0000 7f44aa42c740 1 --2- 172.21.15.44:0/2239692444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995d850 0x560c299623d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.157+0000 7f44aa42c740 1 --2- 172.21.15.44:0/2239692444 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995cf30 0x560c29965bc0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.157+0000 7f44aa42c740 1 -- 172.21.15.44:0/2239692444 >> 172.21.15.44:0/2239692444 conn(0x560c29957660 msgr2=0x560c299592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.157+0000 7f44aa42c740 1 -- 172.21.15.44:0/2239692444 shutdown_connections 2024-04-21T23:11:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.157+0000 7f44aa42c740 1 -- 172.21.15.44:0/2239692444 wait complete. 2024-04-21T23:11:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.157+0000 7f44aa42c740 1 -- start start 2024-04-21T23:11:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.157+0000 7f44aa42c740 1 Processor -- start 2024-04-21T23:11:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f44aa42c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995cf30 0x560c2996e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f44aa42c740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995d850 0x560c2996eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f44aa42c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c29966430 0x560c2996abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f44aa42c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f44a01ab6d0 con 0x560c2995cf30 2024-04-21T23:11:36.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f44aa42c740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f44a01acb40 con 0x560c2995d850 2024-04-21T23:11:36.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f44aa42c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f44a01abda0 con 0x560c29966430 2024-04-21T23:11:36.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f44aa42c740 10 client.0 _ll_register_callbacks cb 0x7f44900b9040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:36.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f44a55dc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995cf30 0x560c2996e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f44a55dc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995cf30 0x560c2996e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38572/0 (socket says 172.21.15.44:38572) 2024-04-21T23:11:36.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f44a55dc640 1 -- 172.21.15.44:0/548018009 learned_addr learned my addr 172.21.15.44:0/548018009 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:36.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f44a4ddb640 1 --2- 172.21.15.44:0/548018009 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995d850 0x560c2996eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f44a5ddd640 1 --2- 172.21.15.44:0/548018009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c29966430 0x560c2996abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f44a55dc640 1 -- 172.21.15.44:0/548018009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c29966430 msgr2=0x560c2996abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f44a55dc640 1 --2- 172.21.15.44:0/548018009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c29966430 0x560c2996abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f44a55dc640 1 -- 172.21.15.44:0/548018009 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995d850 msgr2=0x560c2996eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f44a55dc640 1 --2- 172.21.15.44:0/548018009 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995d850 0x560c2996eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f44a55dc640 1 -- 172.21.15.44:0/548018009 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c2996b3b0 con 0x560c2995cf30 2024-04-21T23:11:36.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f44a55dc640 1 --2- 172.21.15.44:0/548018009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995cf30 0x560c2996e5e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f4498003fa0 tx=0x7f4498003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=887ec640d41aa7b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:36.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f4488ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:11:36.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f4488ff9640 1 -- 172.21.15.44:0/548018009 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4498003820 con 0x560c2995cf30 2024-04-21T23:11:36.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f44aa42c740 1 -- 172.21.15.44:0/548018009 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44980035f0 con 0x560c2995cf30 2024-04-21T23:11:36.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f4488ff9640 1 -- 172.21.15.44:0/548018009 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f44980039c0 con 0x560c2995cf30 2024-04-21T23:11:36.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.158+0000 7f4488ff9640 1 -- 172.21.15.44:0/548018009 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4498044050 con 0x560c2995cf30 2024-04-21T23:11:36.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.159+0000 7f44aa42c740 10 client.16251 fetch_fsmap learned FSMap version 25 2024-04-21T23:11:36.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.159+0000 7f44aa42c740 1 -- 172.21.15.44:0/548018009 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560c299dd7c0 con 0x560c2995cf30 2024-04-21T23:11:36.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.159+0000 7f4488ff9640 1 -- 172.21.15.44:0/548018009 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44980035f0 con 0x560c2995cf30 2024-04-21T23:11:36.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.159+0000 7f4488ff9640 1 -- 172.21.15.44:0/548018009 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f449803b870 con 0x560c2995cf30 2024-04-21T23:11:36.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.159+0000 7f44aa42c740 10 client.16251 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:11:36.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.159+0000 7f44aa42c740 10 client.16251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:36.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.159+0000 7f44aa42c740 20 client.16251 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:36.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.159+0000 7f44aa42c740 1 --2- 172.21.15.44:0/548018009 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560c299dde90 0x560c299e02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.159+0000 7f44aa42c740 4 client.16251 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:11:36.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.159+0000 7f44aa42c740 1 -- 172.21.15.44:0/548018009 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560c298c8df0 con 0x560c299dde90 2024-04-21T23:11:36.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.159+0000 7f44a4ddb640 1 --2- 172.21.15.44:0/548018009 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560c299dde90 0x560c299e02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.159+0000 7f44a4ddb640 1 --2- 172.21.15.44:0/548018009 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560c299dde90 0x560c299e02b0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69d21f491cc11ae4 server_cookie=97b320b803880456 in_seq=0 out_seq=0 2024-04-21T23:11:36.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.159+0000 7f4488ff9640 10 client.16251 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:36.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f4488ff9640 1 -- 172.21.15.44:0/548018009 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560c298c8df0 con 0x560c299dde90 2024-04-21T23:11:36.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f4488ff9640 10 client.16251 handle_command_reply: tid=0 2024-04-21T23:11:36.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44aa42c740 1 client.16251 shutdown 2024-04-21T23:11:36.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44aa42c740 1 -- 172.21.15.44:0/548018009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995cf30 msgr2=0x560c2996e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44aa42c740 1 --2- 172.21.15.44:0/548018009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995cf30 0x560c2996e5e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f4498003fa0 tx=0x7f4498003dc0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44aa42c740 1 -- 172.21.15.44:0/548018009 shutdown_connections 2024-04-21T23:11:36.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44aa42c740 1 --2- 172.21.15.44:0/548018009 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560c299dde90 0x560c299e02b0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44aa42c740 1 --2- 172.21.15.44:0/548018009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560c29966430 0x560c2996abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44aa42c740 1 --2- 172.21.15.44:0/548018009 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560c2995d850 0x560c2996eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f4488ff9640 0 client.16251 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:36.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44aa42c740 1 --2- 172.21.15.44:0/548018009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560c2995cf30 0x560c2996e5e0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44aa42c740 1 -- 172.21.15.44:0/548018009 >> 172.21.15.44:0/548018009 conn(0x560c29957660 msgr2=0x560c29955e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44aa42c740 1 -- 172.21.15.44:0/548018009 shutdown_connections 2024-04-21T23:11:36.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44aa42c740 1 -- 172.21.15.44:0/548018009 wait complete. 2024-04-21T23:11:36.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44aa42c740 20 client.16251 trim_cache size 0 max 16384 2024-04-21T23:11:36.168 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:11:36.168 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-21T23:11:36.168 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-21T23:11:36.168 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:11:36.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44a7840640 1 -- 172.21.15.44:0/1507829975 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f448c0a2d00 msgr2=0x7f448c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44a7840640 1 --2- 172.21.15.44:0/1507829975 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f448c0a2d00 0x7f448c0a51c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4498009c20 tx=0x7f4498005f90 comp rx=0 tx=0).stop 2024-04-21T23:11:36.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44a7840640 1 -- 172.21.15.44:0/1507829975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f44a01032d0 msgr2=0x7f44a0198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44a7840640 1 --2- 172.21.15.44:0/1507829975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f44a01032d0 0x7f44a0198550 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f4490005940 tx=0x7f449002cbd0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44a7840640 1 -- 172.21.15.44:0/1507829975 shutdown_connections 2024-04-21T23:11:36.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44a7840640 1 --2- 172.21.15.44:0/1507829975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44a010bcf0 0x7f44a019c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44a7840640 1 --2- 172.21.15.44:0/1507829975 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f448c0a2d00 0x7f448c0a51c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44a7840640 1 --2- 172.21.15.44:0/1507829975 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f44a01032d0 0x7f44a0198550 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44a7840640 1 --2- 172.21.15.44:0/1507829975 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f44a01029b0 0x7f44a0198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.160+0000 7f44a7840640 1 -- 172.21.15.44:0/1507829975 >> 172.21.15.44:0/1507829975 conn(0x7f44a00777f0 msgr2=0x7f44a00fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.161+0000 7f44a7840640 1 -- 172.21.15.44:0/1507829975 shutdown_connections 2024-04-21T23:11:36.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.161+0000 7f44a7840640 1 -- 172.21.15.44:0/1507829975 wait complete. 2024-04-21T23:11:36.171 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:11:36.171 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:11:36.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.274+0000 7f38b4eb5640 1 -- 172.21.15.44:0/2247093679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38b00693e0 msgr2=0x7f38b010e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.274+0000 7f38b4eb5640 1 --2- 172.21.15.44:0/2247093679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38b00693e0 0x7f38b010e5e0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f38a0009ca0 tx=0x7f38a002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.274+0000 7f38b4eb5640 1 -- 172.21.15.44:0/2247093679 shutdown_connections 2024-04-21T23:11:36.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.274+0000 7f38b4eb5640 1 --2- 172.21.15.44:0/2247093679 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b0068a40 0x7f38b0068ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.274+0000 7f38b4eb5640 1 --2- 172.21.15.44:0/2247093679 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38b00693e0 0x7f38b010e5e0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.274+0000 7f38b4eb5640 1 --2- 172.21.15.44:0/2247093679 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38b0107800 0x7f38b0107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.274+0000 7f38b4eb5640 1 -- 172.21.15.44:0/2247093679 >> 172.21.15.44:0/2247093679 conn(0x7f38b0077430 msgr2=0x7f38b0077840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.274+0000 7f38b4eb5640 1 -- 172.21.15.44:0/2247093679 shutdown_connections 2024-04-21T23:11:36.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.274+0000 7f38b4eb5640 1 -- 172.21.15.44:0/2247093679 wait complete. 2024-04-21T23:11:36.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.275+0000 7f38b4eb5640 1 Processor -- start 2024-04-21T23:11:36.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.275+0000 7f38b4eb5640 1 -- start start 2024-04-21T23:11:36.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.275+0000 7f38b4eb5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38b0068a40 0x7f38b0197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.275+0000 7f38b4eb5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b00693e0 0x7f38b0198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.275+0000 7f38b4eb5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38b0107800 0x7f38b019c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.275+0000 7f38b4eb5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f38b0111b20 con 0x7f38b0068a40 2024-04-21T23:11:36.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.275+0000 7f38b4eb5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f38b01119a0 con 0x7f38b0107800 2024-04-21T23:11:36.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.275+0000 7f38b4eb5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f38b0111ca0 con 0x7f38b00693e0 2024-04-21T23:11:36.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.275+0000 7f38ae59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38b0068a40 0x7f38b0197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.275+0000 7f38ae59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38b0068a40 0x7f38b0197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38582/0 (socket says 172.21.15.44:38582) 2024-04-21T23:11:36.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.275+0000 7f38ae59c640 1 -- 172.21.15.44:0/2220455827 learned_addr learned my addr 172.21.15.44:0/2220455827 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:36.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.275+0000 7f38add9b640 1 --2- 172.21.15.44:0/2220455827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b00693e0 0x7f38b0198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.275+0000 7f38aed9d640 1 --2- 172.21.15.44:0/2220455827 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38b0107800 0x7f38b019c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.276+0000 7f38ae59c640 1 -- 172.21.15.44:0/2220455827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b00693e0 msgr2=0x7f38b0198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.276+0000 7f38ae59c640 1 --2- 172.21.15.44:0/2220455827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b00693e0 0x7f38b0198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.276+0000 7f38ae59c640 1 -- 172.21.15.44:0/2220455827 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38b0107800 msgr2=0x7f38b019c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.276+0000 7f38ae59c640 1 --2- 172.21.15.44:0/2220455827 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38b0107800 0x7f38b019c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.276+0000 7f38ae59c640 1 -- 172.21.15.44:0/2220455827 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38b019cf10 con 0x7f38b0068a40 2024-04-21T23:11:36.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.276+0000 7f38ae59c640 1 --2- 172.21.15.44:0/2220455827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38b0068a40 0x7f38b0197f20 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f38a800e9f0 tx=0x7f38a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:36.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.276+0000 7f389f7fe640 1 -- 172.21.15.44:0/2220455827 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a800cd80 con 0x7f38b0068a40 2024-04-21T23:11:36.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.276+0000 7f38b4eb5640 1 -- 172.21.15.44:0/2220455827 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38b019d200 con 0x7f38b0068a40 2024-04-21T23:11:36.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.276+0000 7f389f7fe640 1 -- 172.21.15.44:0/2220455827 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38a8010910 con 0x7f38b0068a40 2024-04-21T23:11:36.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.276+0000 7f389f7fe640 1 -- 172.21.15.44:0/2220455827 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a80188f0 con 0x7f38b0068a40 2024-04-21T23:11:36.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.276+0000 7f38b4eb5640 1 -- 172.21.15.44:0/2220455827 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38b01a4a90 con 0x7f38b0068a40 2024-04-21T23:11:36.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.277+0000 7f38b4eb5640 1 -- 172.21.15.44:0/2220455827 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38b00685f0 con 0x7f38b0068a40 2024-04-21T23:11:36.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.278+0000 7f389f7fe640 1 -- 172.21.15.44:0/2220455827 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f38a8010ac0 con 0x7f38b0068a40 2024-04-21T23:11:36.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.279+0000 7f389f7fe640 1 --2- 172.21.15.44:0/2220455827 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f38980a2d00 0x7f38980a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.279+0000 7f38add9b640 1 --2- 172.21.15.44:0/2220455827 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f38980a2d00 0x7f38980a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.279+0000 7f389f7fe640 1 -- 172.21.15.44:0/2220455827 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f38a80cff00 con 0x7f38b0068a40 2024-04-21T23:11:36.280 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.279+0000 7f38add9b640 1 --2- 172.21.15.44:0/2220455827 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f38980a2d00 0x7f38980a51c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f38b0106430 tx=0x7f38a4005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:36.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.283+0000 7f389f7fe640 1 -- 172.21.15.44:0/2220455827 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f38a809c130 con 0x7f38b0068a40 2024-04-21T23:11:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.393+0000 7f255837a640 1 -- 172.21.15.44:0/150396661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25501029b0 msgr2=0x7f2550102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.393+0000 7f255837a640 1 --2- 172.21.15.44:0/150396661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25501029b0 0x7f2550102d90 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f2540009900 tx=0x7f254002bcd0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.393+0000 7f255837a640 1 -- 172.21.15.44:0/150396661 shutdown_connections 2024-04-21T23:11:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.393+0000 7f255837a640 1 --2- 172.21.15.44:0/150396661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25501032d0 0x7f255010b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.393+0000 7f255837a640 1 --2- 172.21.15.44:0/150396661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f255010bd80 0x7f255010e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.393+0000 7f255837a640 1 --2- 172.21.15.44:0/150396661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25501029b0 0x7f2550102d90 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.393+0000 7f255837a640 1 -- 172.21.15.44:0/150396661 >> 172.21.15.44:0/150396661 conn(0x7f2550077760 msgr2=0x7f2550077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.393+0000 7f255837a640 1 -- 172.21.15.44:0/150396661 shutdown_connections 2024-04-21T23:11:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.393+0000 7f255837a640 1 -- 172.21.15.44:0/150396661 wait complete. 2024-04-21T23:11:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.394+0000 7f255837a640 1 Processor -- start 2024-04-21T23:11:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.394+0000 7f255837a640 1 -- start start 2024-04-21T23:11:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.394+0000 7f255837a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25501029b0 0x7f2550100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.394+0000 7f255837a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25501032d0 0x7f2550100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.394+0000 7f255837a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f255010bd80 0x7f2550101080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.394+0000 7f255837a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2550111b10 con 0x7f255010bd80 2024-04-21T23:11:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.394+0000 7f255837a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2550111990 con 0x7f25501032d0 2024-04-21T23:11:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.394+0000 7f255837a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2550111c90 con 0x7f25501029b0 2024-04-21T23:11:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.394+0000 7f2556917640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f255010bd80 0x7f2550101080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.394+0000 7f2556917640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f255010bd80 0x7f2550101080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38602/0 (socket says 172.21.15.44:38602) 2024-04-21T23:11:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.394+0000 7f2556116640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25501029b0 0x7f2550100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.394+0000 7f2556917640 1 -- 172.21.15.44:0/1485015637 learned_addr learned my addr 172.21.15.44:0/1485015637 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.394+0000 7f2555915640 1 --2- 172.21.15.44:0/1485015637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25501032d0 0x7f2550100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.394+0000 7f2556917640 1 -- 172.21.15.44:0/1485015637 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25501029b0 msgr2=0x7f2550100600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.394+0000 7f2556917640 1 --2- 172.21.15.44:0/1485015637 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25501029b0 0x7f2550100600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.394+0000 7f2556917640 1 -- 172.21.15.44:0/1485015637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25501032d0 msgr2=0x7f2550100b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.394+0000 7f2556917640 1 --2- 172.21.15.44:0/1485015637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25501032d0 0x7f2550100b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.394+0000 7f2556917640 1 -- 172.21.15.44:0/1485015637 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25501a6530 con 0x7f255010bd80 2024-04-21T23:11:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.395+0000 7f2556917640 1 --2- 172.21.15.44:0/1485015637 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f255010bd80 0x7f2550101080 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f2548009c90 tx=0x7f254800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.395+0000 7f25477fe640 1 -- 172.21.15.44:0/1485015637 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2548003b70 con 0x7f255010bd80 2024-04-21T23:11:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.395+0000 7f25477fe640 1 -- 172.21.15.44:0/1485015637 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2548003d10 con 0x7f255010bd80 2024-04-21T23:11:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.395+0000 7f255837a640 1 -- 172.21.15.44:0/1485015637 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25501a6820 con 0x7f255010bd80 2024-04-21T23:11:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.395+0000 7f25477fe640 1 -- 172.21.15.44:0/1485015637 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2548010960 con 0x7f255010bd80 2024-04-21T23:11:36.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.395+0000 7f255837a640 1 -- 172.21.15.44:0/1485015637 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25501a6d20 con 0x7f255010bd80 2024-04-21T23:11:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.397+0000 7f25477fe640 1 -- 172.21.15.44:0/1485015637 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f2548004020 con 0x7f255010bd80 2024-04-21T23:11:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.398+0000 7f25477fe640 1 --2- 172.21.15.44:0/1485015637 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f253c0a2cf0 0x7f253c0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.398+0000 7f2556116640 1 --2- 172.21.15.44:0/1485015637 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f253c0a2cf0 0x7f253c0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.398+0000 7f25477fe640 1 -- 172.21.15.44:0/1485015637 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f25480cfdd0 con 0x7f255010bd80 2024-04-21T23:11:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.398+0000 7f2556116640 1 --2- 172.21.15.44:0/1485015637 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f253c0a2cf0 0x7f253c0a51b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f254002efd0 tx=0x7f254002cf90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.399+0000 7f255af64740 1 Processor -- start 2024-04-21T23:11:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.399+0000 7f255af64740 1 -- start start 2024-04-21T23:11:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.399+0000 7f255af64740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1ca2b20 0x5591d1ca2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.399+0000 7f255af64740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1ca3440 0x5591d1cabcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.399+0000 7f255af64740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1cac210 0x5591d1cae600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.399+0000 7f255af64740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5591d1c06b70 con 0x5591d1cac210 2024-04-21T23:11:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.399+0000 7f255af64740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5591d1baf470 con 0x5591d1ca2b20 2024-04-21T23:11:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.399+0000 7f255af64740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5591d1c02b30 con 0x5591d1ca3440 2024-04-21T23:11:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.399+0000 7f2556917640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1cac210 0x5591d1cae600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.399+0000 7f2555915640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1ca3440 0x5591d1cabcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.399+0000 7f2556116640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1ca2b20 0x5591d1ca2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.399+0000 7f2555915640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1ca3440 0x5591d1cabcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:59168/0 (socket says 172.21.15.44:59168) 2024-04-21T23:11:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.399+0000 7f2555915640 1 -- 172.21.15.44:0/27607505 learned_addr learned my addr 172.21.15.44:0/27607505 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.400+0000 7f2556917640 1 -- 172.21.15.44:0/27607505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1ca3440 msgr2=0x5591d1cabcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.400+0000 7f2556917640 1 --2- 172.21.15.44:0/27607505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1ca3440 0x5591d1cabcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.400+0000 7f2556917640 1 -- 172.21.15.44:0/27607505 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1ca2b20 msgr2=0x5591d1ca2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.400+0000 7f2556917640 1 --2- 172.21.15.44:0/27607505 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1ca2b20 0x5591d1ca2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.400+0000 7f2556917640 1 -- 172.21.15.44:0/27607505 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591d191b210 con 0x5591d1cac210 2024-04-21T23:11:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.400+0000 7f2556917640 1 --2- 172.21.15.44:0/27607505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1cac210 0x5591d1cae600 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f25480cd120 tx=0x7f254809ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4334f7cb884651b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.400+0000 7f2544bfe640 1 -- 172.21.15.44:0/27607505 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254809c5f0 con 0x5591d1cac210 2024-04-21T23:11:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.400+0000 7f2544bfe640 1 -- 172.21.15.44:0/27607505 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f254809bef0 con 0x5591d1cac210 2024-04-21T23:11:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.400+0000 7f2544bfe640 1 -- 172.21.15.44:0/27607505 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25480046b0 con 0x5591d1cac210 2024-04-21T23:11:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.401+0000 7f255af64740 1 -- 172.21.15.44:0/27607505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1cac210 msgr2=0x5591d1cae600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.401+0000 7f255af64740 1 --2- 172.21.15.44:0/27607505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1cac210 0x5591d1cae600 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f25480cd120 tx=0x7f254809ba30 comp rx=0 tx=0).stop 2024-04-21T23:11:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.401+0000 7f255af64740 1 -- 172.21.15.44:0/27607505 shutdown_connections 2024-04-21T23:11:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.401+0000 7f255af64740 1 --2- 172.21.15.44:0/27607505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1ca3440 0x5591d1cabcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.401+0000 7f255af64740 1 --2- 172.21.15.44:0/27607505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1cac210 0x5591d1cae600 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.401+0000 7f255af64740 1 --2- 172.21.15.44:0/27607505 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1ca2b20 0x5591d1ca2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.401+0000 7f255af64740 1 -- 172.21.15.44:0/27607505 >> 172.21.15.44:0/27607505 conn(0x5591d1c9d440 msgr2=0x5591d1c9d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.401+0000 7f255af64740 1 -- 172.21.15.44:0/27607505 shutdown_connections 2024-04-21T23:11:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.401+0000 7f255af64740 1 -- 172.21.15.44:0/27607505 wait complete. 2024-04-21T23:11:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.401+0000 7f255af64740 1 -- start start 2024-04-21T23:11:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f255af64740 1 Processor -- start 2024-04-21T23:11:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f255af64740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1ca2b20 0x5591d1ca9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f255af64740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1ca3440 0x5591d1ca9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f255af64740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1cac210 0x5591d1ca9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f255af64740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5591d1caede0 con 0x5591d1ca2b20 2024-04-21T23:11:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f255af64740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5591d1bc2a10 con 0x5591d1ca3440 2024-04-21T23:11:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f255af64740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5591d1caef60 con 0x5591d1cac210 2024-04-21T23:11:36.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f255af64740 10 client.0 _ll_register_callbacks cb 0x5591d1bfdb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:36.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f2556116640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1ca2b20 0x5591d1ca9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f2556116640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1ca2b20 0x5591d1ca9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38610/0 (socket says 172.21.15.44:38610) 2024-04-21T23:11:36.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f2555915640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1ca3440 0x5591d1ca9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f2556116640 1 -- 172.21.15.44:0/1954377829 learned_addr learned my addr 172.21.15.44:0/1954377829 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:36.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f2556917640 1 --2- 172.21.15.44:0/1954377829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1cac210 0x5591d1ca9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f2556917640 1 -- 172.21.15.44:0/1954377829 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1ca3440 msgr2=0x5591d1ca9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f2556917640 1 --2- 172.21.15.44:0/1954377829 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1ca3440 0x5591d1ca9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f2556917640 1 -- 172.21.15.44:0/1954377829 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1ca2b20 msgr2=0x5591d1ca9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f2556917640 1 --2- 172.21.15.44:0/1954377829 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1ca2b20 0x5591d1ca9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f2556917640 1 -- 172.21.15.44:0/1954377829 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591d19e7620 con 0x5591d1cac210 2024-04-21T23:11:36.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f2556116640 1 --2- 172.21.15.44:0/1954377829 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1ca2b20 0x5591d1ca9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:36.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f2556917640 1 --2- 172.21.15.44:0/1954377829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1cac210 0x5591d1ca9cd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x5591d1ca39d0 tx=0x7f25480bc0c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c08dd7903e916c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:36.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.402+0000 7f2539ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:11:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.403+0000 7f2539ffb640 1 -- 172.21.15.44:0/1954377829 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254807f060 con 0x5591d1cac210 2024-04-21T23:11:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.403+0000 7f2539ffb640 1 -- 172.21.15.44:0/1954377829 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f2548098560 con 0x5591d1cac210 2024-04-21T23:11:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.403+0000 7f255af64740 1 -- 172.21.15.44:0/1954377829 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f254809bef0 con 0x5591d1cac210 2024-04-21T23:11:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.403+0000 7f2539ffb640 1 -- 172.21.15.44:0/1954377829 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f25480a4b60 con 0x5591d1cac210 2024-04-21T23:11:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.403+0000 7f255af64740 10 client.25876 fetch_fsmap learned FSMap version 25 2024-04-21T23:11:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.403+0000 7f255af64740 1 -- 172.21.15.44:0/1954377829 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5591d1caede0 con 0x5591d1cac210 2024-04-21T23:11:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.403+0000 7f2539ffb640 1 -- 172.21.15.44:0/1954377829 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f254809bef0 con 0x5591d1cac210 2024-04-21T23:11:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.403+0000 7f2539ffb640 1 -- 172.21.15.44:0/1954377829 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f2548004c60 con 0x5591d1cac210 2024-04-21T23:11:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.403+0000 7f255af64740 10 client.25876 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:11:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.403+0000 7f255af64740 10 client.25876 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.403+0000 7f255af64740 20 client.25876 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.403+0000 7f255af64740 1 --2- 172.21.15.44:0/1954377829 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5591d1cbc1e0 0x5591d1cbe5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.403+0000 7f255af64740 4 client.25876 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:11:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.403+0000 7f255af64740 1 -- 172.21.15.44:0/1954377829 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591d1c140b0 con 0x5591d1cbc1e0 2024-04-21T23:11:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.403+0000 7f2556116640 1 --2- 172.21.15.44:0/1954377829 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5591d1cbc1e0 0x5591d1cbe5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.404+0000 7f2539ffb640 10 client.25876 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.404+0000 7f2556116640 1 --2- 172.21.15.44:0/1954377829 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5591d1cbc1e0 0x5591d1cbe5d0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bd61d4efce53a39 server_cookie=597d406cf62dc8e6 in_seq=0 out_seq=0 2024-04-21T23:11:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.405+0000 7f2539ffb640 1 -- 172.21.15.44:0/1954377829 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5591d1c140b0 con 0x5591d1cbc1e0 2024-04-21T23:11:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.405+0000 7f2539ffb640 10 client.25876 handle_command_reply: tid=0 2024-04-21T23:11:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.405+0000 7f255af64740 1 client.25876 shutdown 2024-04-21T23:11:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.405+0000 7f255af64740 1 -- 172.21.15.44:0/1954377829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1cac210 msgr2=0x5591d1ca9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.405+0000 7f255af64740 1 --2- 172.21.15.44:0/1954377829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1cac210 0x5591d1ca9cd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x5591d1ca39d0 tx=0x7f25480bc0c0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.405+0000 7f255af64740 1 -- 172.21.15.44:0/1954377829 shutdown_connections 2024-04-21T23:11:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.405+0000 7f255af64740 1 --2- 172.21.15.44:0/1954377829 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5591d1cbc1e0 0x5591d1cbe5d0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.405+0000 7f255af64740 1 --2- 172.21.15.44:0/1954377829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1cac210 0x5591d1ca9cd0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.405+0000 7f2539ffb640 0 client.25876 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.405+0000 7f255af64740 1 --2- 172.21.15.44:0/1954377829 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1ca3440 0x5591d1ca9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.405+0000 7f255af64740 1 --2- 172.21.15.44:0/1954377829 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1ca2b20 0x5591d1ca9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.405+0000 7f255af64740 1 -- 172.21.15.44:0/1954377829 >> 172.21.15.44:0/1954377829 conn(0x5591d1c9d440 msgr2=0x5591d1c9f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.405+0000 7f255af64740 1 -- 172.21.15.44:0/1954377829 shutdown_connections 2024-04-21T23:11:36.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.405+0000 7f255af64740 1 -- 172.21.15.44:0/1954377829 wait complete. 2024-04-21T23:11:36.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.406+0000 7f255af64740 20 client.25876 trim_cache size 0 max 16384 2024-04-21T23:11:36.422 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.422+0000 7f255af64740 1 Processor -- start 2024-04-21T23:11:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.422+0000 7f255af64740 1 -- start start 2024-04-21T23:11:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.422+0000 7f255af64740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1ca2b20 0x5591d1cb7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.422+0000 7f255af64740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1ca3440 0x5591d1ca9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.422+0000 7f255af64740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1cac210 0x5591d1c9ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.422+0000 7f255af64740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f25501aee90 con 0x5591d1ca3440 2024-04-21T23:11:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.422+0000 7f255af64740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f25501b1710 con 0x5591d1ca2b20 2024-04-21T23:11:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.422+0000 7f255af64740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f25501af500 con 0x5591d1cac210 2024-04-21T23:11:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.422+0000 7f2556116640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1ca2b20 0x5591d1cb7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.422+0000 7f2555915640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1ca3440 0x5591d1ca9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.422+0000 7f2556917640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1cac210 0x5591d1c9ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.422+0000 7f2556116640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1ca2b20 0x5591d1cb7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:33384/0 (socket says 172.21.15.44:33384) 2024-04-21T23:11:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.422+0000 7f2556116640 1 -- 172.21.15.44:0/1848373560 learned_addr learned my addr 172.21.15.44:0/1848373560 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.422+0000 7f2556917640 1 -- 172.21.15.44:0/1848373560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1ca2b20 msgr2=0x5591d1cb7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.422+0000 7f2556917640 1 --2- 172.21.15.44:0/1848373560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1ca2b20 0x5591d1cb7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.422+0000 7f2556917640 1 -- 172.21.15.44:0/1848373560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1ca3440 msgr2=0x5591d1ca9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.422+0000 7f2556917640 1 --2- 172.21.15.44:0/1848373560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1ca3440 0x5591d1ca9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.422+0000 7f2556917640 1 -- 172.21.15.44:0/1848373560 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591d1cae930 con 0x5591d1cac210 2024-04-21T23:11:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.422+0000 7f2556116640 1 --2- 172.21.15.44:0/1848373560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1ca2b20 0x5591d1cb7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.422+0000 7f2555915640 1 --2- 172.21.15.44:0/1848373560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1ca3440 0x5591d1ca9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.423+0000 7f2556917640 1 --2- 172.21.15.44:0/1848373560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1cac210 0x5591d1c9ef70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f25480a4d20 tx=0x7f254800cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=843357befcc06d5c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.423+0000 7f253bfff640 1 -- 172.21.15.44:0/1848373560 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254807f070 con 0x5591d1cac210 2024-04-21T23:11:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.423+0000 7f253bfff640 1 -- 172.21.15.44:0/1848373560 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25480851a0 con 0x5591d1cac210 2024-04-21T23:11:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.423+0000 7f253bfff640 1 -- 172.21.15.44:0/1848373560 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254809bef0 con 0x5591d1cac210 2024-04-21T23:11:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.424+0000 7f255af64740 1 -- 172.21.15.44:0/1848373560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1cac210 msgr2=0x5591d1c9ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.424+0000 7f255af64740 1 --2- 172.21.15.44:0/1848373560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1cac210 0x5591d1c9ef70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f25480a4d20 tx=0x7f254800cf90 comp rx=0 tx=0).stop 2024-04-21T23:11:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.424+0000 7f255af64740 1 -- 172.21.15.44:0/1848373560 shutdown_connections 2024-04-21T23:11:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.424+0000 7f255af64740 1 --2- 172.21.15.44:0/1848373560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1cac210 0x5591d1c9ef70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.424+0000 7f255af64740 1 --2- 172.21.15.44:0/1848373560 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1ca3440 0x5591d1ca9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.424+0000 7f255af64740 1 --2- 172.21.15.44:0/1848373560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1ca2b20 0x5591d1cb7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.424+0000 7f255af64740 1 -- 172.21.15.44:0/1848373560 >> 172.21.15.44:0/1848373560 conn(0x5591d1c9d440 msgr2=0x5591d1c9e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.424+0000 7f255af64740 1 -- 172.21.15.44:0/1848373560 shutdown_connections 2024-04-21T23:11:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.424+0000 7f255af64740 1 -- 172.21.15.44:0/1848373560 wait complete. 2024-04-21T23:11:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.424+0000 7f255af64740 1 -- start start 2024-04-21T23:11:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.424+0000 7f255af64740 1 Processor -- start 2024-04-21T23:11:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.425+0000 7f255af64740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1ca2b20 0x5591d1ca5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.425+0000 7f255af64740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1ca3440 0x5591d1ca6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.425+0000 7f255af64740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1cac210 0x5591d1ca6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.425+0000 7f255af64740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f25501afbd0 con 0x5591d1ca3440 2024-04-21T23:11:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.425+0000 7f255af64740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f25501b1040 con 0x5591d1cac210 2024-04-21T23:11:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.425+0000 7f255af64740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f25501b02a0 con 0x5591d1ca2b20 2024-04-21T23:11:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.425+0000 7f2556116640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1ca2b20 0x5591d1ca5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.425+0000 7f2556116640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1ca2b20 0x5591d1ca5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:59194/0 (socket says 172.21.15.44:59194) 2024-04-21T23:11:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.425+0000 7f2556116640 1 -- 172.21.15.44:0/1732909031 learned_addr learned my addr 172.21.15.44:0/1732909031 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.425+0000 7f2555915640 1 --2- 172.21.15.44:0/1732909031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1ca3440 0x5591d1ca6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.425+0000 7f2556917640 1 --2- 172.21.15.44:0/1732909031 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1cac210 0x5591d1ca6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.425+0000 7f2556116640 1 -- 172.21.15.44:0/1732909031 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1cac210 msgr2=0x5591d1ca6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.425+0000 7f2556116640 1 --2- 172.21.15.44:0/1732909031 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1cac210 0x5591d1ca6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.425+0000 7f2556116640 1 -- 172.21.15.44:0/1732909031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1ca3440 msgr2=0x5591d1ca6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.425+0000 7f2556116640 1 --2- 172.21.15.44:0/1732909031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1ca3440 0x5591d1ca6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.425+0000 7f2556116640 1 -- 172.21.15.44:0/1732909031 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591d1d214a0 con 0x5591d1ca2b20 2024-04-21T23:11:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.425+0000 7f255af64740 10 client.0 _ll_register_callbacks cb 0x7f2540042070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.425+0000 7f255af64740 1 -- 172.21.15.44:0/1732909031 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5591d1c0e400 con 0x5591d1ca2b20 2024-04-21T23:11:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.425+0000 7f2556116640 1 --2- 172.21.15.44:0/1732909031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1ca2b20 0x5591d1ca5ee0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2540005ec0 tx=0x7f254002e050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5d27b5aa3fab589 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.425+0000 7f2539ffb640 10 client.25882 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:11:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.426+0000 7f2539ffb640 1 -- 172.21.15.44:0/1732909031 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2540063050 con 0x5591d1ca2b20 2024-04-21T23:11:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.426+0000 7f2539ffb640 1 -- 172.21.15.44:0/1732909031 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f2540055750 con 0x5591d1ca2b20 2024-04-21T23:11:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.426+0000 7f2539ffb640 1 -- 172.21.15.44:0/1732909031 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f25400044a0 con 0x5591d1ca2b20 2024-04-21T23:11:36.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.426+0000 7f255af64740 10 client.25882 fetch_fsmap learned FSMap version 25 2024-04-21T23:11:36.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.426+0000 7f255af64740 1 -- 172.21.15.44:0/1732909031 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=25}) -- 0x5591d1d22500 con 0x5591d1ca2b20 2024-04-21T23:11:36.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.426+0000 7f2539ffb640 1 -- 172.21.15.44:0/1732909031 <== mon.1 v2:172.21.15.69:3300/0 4 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f2540066020 con 0x5591d1ca2b20 2024-04-21T23:11:36.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.426+0000 7f255af64740 10 client.25882 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:11:36.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.426+0000 7f255af64740 10 client.25882 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:36.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.426+0000 7f255af64740 20 client.25882 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:36.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.426+0000 7f255af64740 1 --2- 172.21.15.44:0/1732909031 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5591d1d22c80 0x5591d1d25140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.426+0000 7f255af64740 4 client.25882 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:11:36.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.426+0000 7f255af64740 1 -- 172.21.15.44:0/1732909031 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5591d1b8c7a0 con 0x5591d1d22c80 2024-04-21T23:11:36.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.427+0000 7f2555915640 1 --2- 172.21.15.44:0/1732909031 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5591d1d22c80 0x5591d1d25140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.427+0000 7f2555915640 1 --2- 172.21.15.44:0/1732909031 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5591d1d22c80 0x5591d1d25140 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d471bb151b9f3d79 server_cookie=fb965dd02b7ca3f9 in_seq=0 out_seq=0 2024-04-21T23:11:36.431 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.427+0000 7f2539ffb640 10 client.25882 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:36.434 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:36 smithi044 ceph-mon[27934]: pgmap v87: 129 pgs: 129 active+clean; 48 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 492 op/s 2024-04-21T23:11:36.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.596+0000 7f38b4eb5640 1 -- 172.21.15.44:0/2220455827 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38b0199270 con 0x7f38b0068a40 2024-04-21T23:11:36.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.598+0000 7f389f7fe640 1 -- 172.21.15.44:0/2220455827 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11791 (secure 0 0 0) 0x7f38a80a0430 con 0x7f38b0068a40 2024-04-21T23:11:36.598 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:11:36.598 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:11:24.214378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25843":{"gid":25843,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3162569119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3162569119},{"type":"v1","addr":"172.21.15.69:6839","nonce":3162569119}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:11:36.598 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-04-21T23:11:36.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.603+0000 7f38b4eb5640 1 -- 172.21.15.44:0/2220455827 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f38980a2d00 msgr2=0x7f38980a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.603 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.603+0000 7f38b4eb5640 1 --2- 172.21.15.44:0/2220455827 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f38980a2d00 0x7f38980a51c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f38b0106430 tx=0x7f38a4005f90 comp rx=0 tx=0).stop 2024-04-21T23:11:36.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.603+0000 7f38b4eb5640 1 -- 172.21.15.44:0/2220455827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38b0068a40 msgr2=0x7f38b0197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.603+0000 7f38b4eb5640 1 --2- 172.21.15.44:0/2220455827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38b0068a40 0x7f38b0197f20 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f38a800e9f0 tx=0x7f38a800eeb0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.603+0000 7f38b4eb5640 1 -- 172.21.15.44:0/2220455827 shutdown_connections 2024-04-21T23:11:36.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.603+0000 7f38b4eb5640 1 --2- 172.21.15.44:0/2220455827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b00693e0 0x7f38b0198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.603+0000 7f38b4eb5640 1 --2- 172.21.15.44:0/2220455827 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f38980a2d00 0x7f38980a51c0 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f38b0106430 tx=0x7f38a4005f90 comp rx=0 tx=0).stop 2024-04-21T23:11:36.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.603+0000 7f38b4eb5640 1 --2- 172.21.15.44:0/2220455827 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f38b0107800 0x7f38b019c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.603+0000 7f38b4eb5640 1 --2- 172.21.15.44:0/2220455827 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f38b0068a40 0x7f38b0197f20 secure :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f38a800e9f0 tx=0x7f38a800eeb0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.603+0000 7f38b4eb5640 1 -- 172.21.15.44:0/2220455827 >> 172.21.15.44:0/2220455827 conn(0x7f38b0077430 msgr2=0x7f38b00ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.604 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.604+0000 7f38b4eb5640 1 -- 172.21.15.44:0/2220455827 shutdown_connections 2024-04-21T23:11:36.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.604+0000 7f38b4eb5640 1 -- 172.21.15.44:0/2220455827 wait complete. 2024-04-21T23:11:36.623 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:11:36.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.686+0000 7f2539ffb640 1 -- 172.21.15.44:0/1732909031 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5591d1b8c7a0 con 0x5591d1d22c80 2024-04-21T23:11:36.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.686+0000 7f2539ffb640 10 client.25882 handle_command_reply: tid=0 2024-04-21T23:11:36.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.686+0000 7f255af64740 1 client.25882 shutdown 2024-04-21T23:11:36.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.686+0000 7f255af64740 1 -- 172.21.15.44:0/1732909031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1ca2b20 msgr2=0x5591d1ca5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.688 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.687+0000 7f255af64740 1 --2- 172.21.15.44:0/1732909031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1ca2b20 0x5591d1ca5ee0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2540005ec0 tx=0x7f254002e050 comp rx=0 tx=0).stop 2024-04-21T23:11:36.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.687+0000 7f255af64740 1 -- 172.21.15.44:0/1732909031 shutdown_connections 2024-04-21T23:11:36.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.687+0000 7f255af64740 1 --2- 172.21.15.44:0/1732909031 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5591d1d22c80 0x5591d1d25140 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.687+0000 7f255af64740 1 --2- 172.21.15.44:0/1732909031 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5591d1cac210 0x5591d1ca6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.687+0000 7f2539ffb640 0 client.25882 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:36.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.687+0000 7f255af64740 1 --2- 172.21.15.44:0/1732909031 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5591d1ca3440 0x5591d1ca6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.687+0000 7f255af64740 1 --2- 172.21.15.44:0/1732909031 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5591d1ca2b20 0x5591d1ca5ee0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.687+0000 7f255af64740 1 -- 172.21.15.44:0/1732909031 >> 172.21.15.44:0/1732909031 conn(0x5591d1c9d440 msgr2=0x5591d1c9c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.687+0000 7f255af64740 1 -- 172.21.15.44:0/1732909031 shutdown_connections 2024-04-21T23:11:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.687+0000 7f255af64740 1 -- 172.21.15.44:0/1732909031 wait complete. 2024-04-21T23:11:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.687+0000 7f255af64740 20 client.25882 trim_cache size 0 max 16384 2024-04-21T23:11:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:11:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:11:36.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:11:36.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:11:36.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.688+0000 7f255837a640 1 -- 172.21.15.44:0/1485015637 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f253c0a2cf0 msgr2=0x7f253c0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.688+0000 7f255837a640 1 --2- 172.21.15.44:0/1485015637 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f253c0a2cf0 0x7f253c0a51b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f254002efd0 tx=0x7f254002cf90 comp rx=0 tx=0).stop 2024-04-21T23:11:36.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.688+0000 7f255837a640 1 -- 172.21.15.44:0/1485015637 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f255010bd80 msgr2=0x7f2550101080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.688+0000 7f255837a640 1 --2- 172.21.15.44:0/1485015637 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f255010bd80 0x7f2550101080 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f2548009c90 tx=0x7f254800c6a0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.688+0000 7f255837a640 1 -- 172.21.15.44:0/1485015637 shutdown_connections 2024-04-21T23:11:36.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.688+0000 7f255837a640 1 --2- 172.21.15.44:0/1485015637 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f253c0a2cf0 0x7f253c0a51b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.688+0000 7f255837a640 1 --2- 172.21.15.44:0/1485015637 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f255010bd80 0x7f2550101080 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.688+0000 7f255837a640 1 --2- 172.21.15.44:0/1485015637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25501032d0 0x7f2550100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.688+0000 7f255837a640 1 --2- 172.21.15.44:0/1485015637 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25501029b0 0x7f2550100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.688+0000 7f255837a640 1 -- 172.21.15.44:0/1485015637 >> 172.21.15.44:0/1485015637 conn(0x7f2550077760 msgr2=0x7f25500fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.688+0000 7f255837a640 1 -- 172.21.15.44:0/1485015637 shutdown_connections 2024-04-21T23:11:36.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:36.688+0000 7f255837a640 1 -- 172.21.15.44:0/1485015637 wait complete. 2024-04-21T23:11:36.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 227927 14.48 MB/sec warmup 86 sec latency 73.482 ms 2024-04-21T23:11:36.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:36 smithi069 ceph-mon[31816]: pgmap v87: 129 pgs: 129 active+clean; 48 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 492 op/s 2024-04-21T23:11:36.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:36 smithi055 ceph-mon[31117]: pgmap v87: 129 pgs: 129 active+clean; 48 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 492 op/s 2024-04-21T23:11:36.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.755+0000 7f12f9c59640 1 -- 172.21.15.44:0/2161556385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12f4101700 msgr2=0x7f12f4101b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.755+0000 7f12f9c59640 1 --2- 172.21.15.44:0/2161556385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12f4101700 0x7f12f4101b80 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f12e8009930 tx=0x7f12e802bc30 comp rx=0 tx=0).stop 2024-04-21T23:11:36.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.755+0000 7f12f9c59640 1 -- 172.21.15.44:0/2161556385 shutdown_connections 2024-04-21T23:11:36.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.755+0000 7f12f9c59640 1 --2- 172.21.15.44:0/2161556385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12f41020c0 0x7f12f410e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.755+0000 7f12f9c59640 1 --2- 172.21.15.44:0/2161556385 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12f4101700 0x7f12f4101b80 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.758 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.755+0000 7f12f9c59640 1 --2- 172.21.15.44:0/2161556385 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12f4107710 0x7f12f4107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.755+0000 7f12f9c59640 1 -- 172.21.15.44:0/2161556385 >> 172.21.15.44:0/2161556385 conn(0x7f12f40fd1f0 msgr2=0x7f12f40ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.755+0000 7f12f9c59640 1 -- 172.21.15.44:0/2161556385 shutdown_connections 2024-04-21T23:11:36.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.755+0000 7f12f9c59640 1 -- 172.21.15.44:0/2161556385 wait complete. 2024-04-21T23:11:36.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.756+0000 7f12f9c59640 1 Processor -- start 2024-04-21T23:11:36.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.756+0000 7f12f9c59640 1 -- start start 2024-04-21T23:11:36.759 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.756+0000 7f12f9c59640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12f4101700 0x7f12f4198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.756+0000 7f12f9c59640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12f41020c0 0x7f12f4198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.756+0000 7f12f9c59640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12f4107710 0x7f12f419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.756+0000 7f12f9c59640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f12f4111b20 con 0x7f12f4107710 2024-04-21T23:11:36.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.756+0000 7f12f9c59640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f12f41119a0 con 0x7f12f4101700 2024-04-21T23:11:36.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.756+0000 7f12f9c59640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f12f4111ca0 con 0x7f12f41020c0 2024-04-21T23:11:36.760 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.756+0000 7f12f37fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12f4101700 0x7f12f4198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.756+0000 7f12f3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12f4107710 0x7f12f419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.756+0000 7f12f2ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12f41020c0 0x7f12f4198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.756+0000 7f12f37fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12f4101700 0x7f12f4198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:33422/0 (socket says 172.21.15.44:33422) 2024-04-21T23:11:36.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.756+0000 7f12f37fe640 1 -- 172.21.15.44:0/2466993875 learned_addr learned my addr 172.21.15.44:0/2466993875 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:36.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.757+0000 7f12f37fe640 1 -- 172.21.15.44:0/2466993875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12f41020c0 msgr2=0x7f12f4198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.757+0000 7f12f37fe640 1 --2- 172.21.15.44:0/2466993875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12f41020c0 0x7f12f4198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.757+0000 7f12f37fe640 1 -- 172.21.15.44:0/2466993875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12f4107710 msgr2=0x7f12f419c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.757+0000 7f12f37fe640 1 --2- 172.21.15.44:0/2466993875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12f4107710 0x7f12f419c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.757+0000 7f12f37fe640 1 -- 172.21.15.44:0/2466993875 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f419d030 con 0x7f12f4101700 2024-04-21T23:11:36.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.757+0000 7f12f37fe640 1 --2- 172.21.15.44:0/2466993875 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12f4101700 0x7f12f4198040 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f12e400e9f0 tx=0x7f12e400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:36.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.757+0000 7f12f0ff9640 1 -- 172.21.15.44:0/2466993875 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e400cd80 con 0x7f12f4101700 2024-04-21T23:11:36.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.757+0000 7f12f9c59640 1 -- 172.21.15.44:0/2466993875 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12f419d320 con 0x7f12f4101700 2024-04-21T23:11:36.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.757+0000 7f12f0ff9640 1 -- 172.21.15.44:0/2466993875 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12e4010910 con 0x7f12f4101700 2024-04-21T23:11:36.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.757+0000 7f12f0ff9640 1 -- 172.21.15.44:0/2466993875 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e40188f0 con 0x7f12f4101700 2024-04-21T23:11:36.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.757+0000 7f12f9c59640 1 -- 172.21.15.44:0/2466993875 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12f41a4bb0 con 0x7f12f4101700 2024-04-21T23:11:36.775 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.760+0000 7f12fc83e740 1 Processor -- start 2024-04-21T23:11:36.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.760+0000 7f12fc83e740 1 -- start start 2024-04-21T23:11:36.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.760+0000 7f12f0ff9640 1 -- 172.21.15.44:0/2466993875 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f12e4010ac0 con 0x7f12f4101700 2024-04-21T23:11:36.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.760+0000 7f12fc83e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500acd0 0x55e43500b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.760+0000 7f12fc83e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e43500b5f0 0x55e4350183f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.760+0000 7f12fc83e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e435018930 0x55e43501ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12fc83e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e434f6f530 con 0x55e43500acd0 2024-04-21T23:11:36.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12fc83e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55e434f17830 con 0x55e43500b5f0 2024-04-21T23:11:36.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12fc83e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e434f6b4f0 con 0x55e435018930 2024-04-21T23:11:36.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12f3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500acd0 0x55e43500b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12f3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500acd0 0x55e43500b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38650/0 (socket says 172.21.15.44:38650) 2024-04-21T23:11:36.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12f3fff640 1 -- 172.21.15.44:0/1353673330 learned_addr learned my addr 172.21.15.44:0/1353673330 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:36.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12f37fe640 1 --2- 172.21.15.44:0/1353673330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e43500b5f0 0x55e4350183f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12f2ffd640 1 --2- 172.21.15.44:0/1353673330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e435018930 0x55e43501ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12f3fff640 1 -- 172.21.15.44:0/1353673330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e435018930 msgr2=0x55e43501ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12f3fff640 1 --2- 172.21.15.44:0/1353673330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e435018930 0x55e43501ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12f3fff640 1 -- 172.21.15.44:0/1353673330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e43500b5f0 msgr2=0x55e4350183f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12f3fff640 1 --2- 172.21.15.44:0/1353673330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e43500b5f0 0x55e4350183f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12f3fff640 1 -- 172.21.15.44:0/1353673330 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e434f3b2c0 con 0x55e43500acd0 2024-04-21T23:11:36.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12f2ffd640 1 --2- 172.21.15.44:0/1353673330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e435018930 0x55e43501ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:11:36.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12f0ff9640 1 --2- 172.21.15.44:0/2466993875 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f12dc0a2d00 0x7f12dc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12f3fff640 1 --2- 172.21.15.44:0/1353673330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500acd0 0x55e43500b0b0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f12ec009fd0 tx=0x7f12ec0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=342f1ffe32ff33e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:36.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12f37fe640 1 --2- 172.21.15.44:0/2466993875 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f12dc0a2d00 0x7f12dc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12f0ff9640 1 -- 172.21.15.44:0/2466993875 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f12e40cfb50 con 0x7f12f4101700 2024-04-21T23:11:36.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12d9ffb640 1 -- 172.21.15.44:0/1353673330 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ec004d20 con 0x55e43500acd0 2024-04-21T23:11:36.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12d9ffb640 1 -- 172.21.15.44:0/1353673330 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12ec004050 con 0x55e43500acd0 2024-04-21T23:11:36.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.761+0000 7f12d9ffb640 1 -- 172.21.15.44:0/1353673330 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ec005260 con 0x55e43500acd0 2024-04-21T23:11:36.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.762+0000 7f12fc83e740 1 -- 172.21.15.44:0/1353673330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500acd0 msgr2=0x55e43500b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.762+0000 7f12f37fe640 1 --2- 172.21.15.44:0/2466993875 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f12dc0a2d00 0x7f12dc0a51c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f12e40ccfb0 tx=0x7f12e40d36a0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:36.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.762+0000 7f12fc83e740 1 --2- 172.21.15.44:0/1353673330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500acd0 0x55e43500b0b0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f12ec009fd0 tx=0x7f12ec0118d0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.762+0000 7f12fc83e740 1 -- 172.21.15.44:0/1353673330 shutdown_connections 2024-04-21T23:11:36.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.762+0000 7f12fc83e740 1 --2- 172.21.15.44:0/1353673330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e435018930 0x55e43501ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.762+0000 7f12fc83e740 1 --2- 172.21.15.44:0/1353673330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e43500b5f0 0x55e4350183f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.762+0000 7f12fc83e740 1 --2- 172.21.15.44:0/1353673330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500acd0 0x55e43500b0b0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.762+0000 7f12fc83e740 1 -- 172.21.15.44:0/1353673330 >> 172.21.15.44:0/1353673330 conn(0x55e435005750 msgr2=0x55e435005b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.762+0000 7f12fc83e740 1 -- 172.21.15.44:0/1353673330 shutdown_connections 2024-04-21T23:11:36.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.762+0000 7f12fc83e740 1 -- 172.21.15.44:0/1353673330 wait complete. 2024-04-21T23:11:36.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.762+0000 7f12fc83e740 1 -- start start 2024-04-21T23:11:36.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.763+0000 7f12fc83e740 1 Processor -- start 2024-04-21T23:11:36.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.763+0000 7f12fc83e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500acd0 0x55e4350181c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.763+0000 7f12fc83e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e43500b5f0 0x55e435015550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.763+0000 7f12fc83e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e435018930 0x55e435015a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.763+0000 7f12fc83e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e43501b770 con 0x55e43500acd0 2024-04-21T23:11:36.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.763+0000 7f12fc83e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55e434f2af10 con 0x55e435018930 2024-04-21T23:11:36.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.763+0000 7f12fc83e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e43501b8f0 con 0x55e43500b5f0 2024-04-21T23:11:36.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.763+0000 7f12fc83e740 10 client.0 _ll_register_callbacks cb 0x55e434c82be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:36.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.763+0000 7f12f3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500acd0 0x55e4350181c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.763+0000 7f12f3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500acd0 0x55e4350181c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38662/0 (socket says 172.21.15.44:38662) 2024-04-21T23:11:36.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.763+0000 7f12f3fff640 1 -- 172.21.15.44:0/178239060 learned_addr learned my addr 172.21.15.44:0/178239060 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:36.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.763+0000 7f12f2ffd640 1 --2- 172.21.15.44:0/178239060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e43500b5f0 0x55e435015550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.763+0000 7f12f3fff640 1 --2- 172.21.15.44:0/178239060 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e435018930 0x55e435015a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.763+0000 7f12f3fff640 1 -- 172.21.15.44:0/178239060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e43500b5f0 msgr2=0x55e435015550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.763+0000 7f12f3fff640 1 --2- 172.21.15.44:0/178239060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e43500b5f0 0x55e435015550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.763+0000 7f12f3fff640 1 -- 172.21.15.44:0/178239060 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e435018930 msgr2=0x55e435015a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.763+0000 7f12f3fff640 1 --2- 172.21.15.44:0/178239060 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e435018930 0x55e435015a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.763+0000 7f12f3fff640 1 -- 172.21.15.44:0/178239060 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e434efcc80 con 0x55e43500acd0 2024-04-21T23:11:36.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.763+0000 7f12f3fff640 1 --2- 172.21.15.44:0/178239060 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500acd0 0x55e4350181c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f12ec011dd0 tx=0x7f12ec009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=237aad0207feb045 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:36.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.763+0000 7f12d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:11:36.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.764+0000 7f12d2ffd640 1 -- 172.21.15.44:0/178239060 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ec004690 con 0x55e43500acd0 2024-04-21T23:11:36.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.764+0000 7f12d2ffd640 1 -- 172.21.15.44:0/178239060 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f12ec005040 con 0x55e43500acd0 2024-04-21T23:11:36.781 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.764+0000 7f12fc83e740 1 -- 172.21.15.44:0/178239060 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12ec004050 con 0x55e43500acd0 2024-04-21T23:11:36.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.764+0000 7f12d2ffd640 1 -- 172.21.15.44:0/178239060 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f12ec035050 con 0x55e43500acd0 2024-04-21T23:11:36.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.764+0000 7f12fc83e740 10 client.16287 fetch_fsmap learned FSMap version 25 2024-04-21T23:11:36.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.764+0000 7f12fc83e740 1 -- 172.21.15.44:0/178239060 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e434c833f0 con 0x55e43500acd0 2024-04-21T23:11:36.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.764+0000 7f12d2ffd640 1 -- 172.21.15.44:0/178239060 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12ec004050 con 0x55e43500acd0 2024-04-21T23:11:36.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.764+0000 7f12d2ffd640 1 -- 172.21.15.44:0/178239060 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f12ec00fa30 con 0x55e43500acd0 2024-04-21T23:11:36.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.764+0000 7f12fc83e740 10 client.16287 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:11:36.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.764+0000 7f12fc83e740 10 client.16287 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:36.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.764+0000 7f12fc83e740 20 client.16287 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:36.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.764+0000 7f12fc83e740 1 --2- 172.21.15.44:0/178239060 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e435028890 0x55e43502ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.764+0000 7f12fc83e740 4 client.16287 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:11:36.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.764+0000 7f12fc83e740 1 -- 172.21.15.44:0/178239060 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e434f7ca70 con 0x55e435028890 2024-04-21T23:11:36.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.764+0000 7f12f2ffd640 1 --2- 172.21.15.44:0/178239060 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e435028890 0x55e43502ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.765+0000 7f12f2ffd640 1 --2- 172.21.15.44:0/178239060 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e435028890 0x55e43502ac80 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a49b605f973d7024 server_cookie=9b951d79c7325e66 in_seq=0 out_seq=0 2024-04-21T23:11:36.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.765+0000 7f12d2ffd640 10 client.16287 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:36.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.766+0000 7f12d2ffd640 1 -- 172.21.15.44:0/178239060 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e434f7ca70 con 0x55e435028890 2024-04-21T23:11:36.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.766+0000 7f12d2ffd640 10 client.16287 handle_command_reply: tid=0 2024-04-21T23:11:36.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.766+0000 7f12fc83e740 1 client.16287 shutdown 2024-04-21T23:11:36.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.766+0000 7f12fc83e740 1 -- 172.21.15.44:0/178239060 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500acd0 msgr2=0x55e4350181c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.766+0000 7f12fc83e740 1 --2- 172.21.15.44:0/178239060 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500acd0 0x55e4350181c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f12ec011dd0 tx=0x7f12ec009ca0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.766+0000 7f12fc83e740 1 -- 172.21.15.44:0/178239060 shutdown_connections 2024-04-21T23:11:36.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.766+0000 7f12fc83e740 1 --2- 172.21.15.44:0/178239060 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e435028890 0x55e43502ac80 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.766+0000 7f12fc83e740 1 --2- 172.21.15.44:0/178239060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e43500b5f0 0x55e435015550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.766+0000 7f12fc83e740 1 --2- 172.21.15.44:0/178239060 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e435018930 0x55e435015a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.766+0000 7f12d2ffd640 0 client.16287 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:36.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.766+0000 7f12fc83e740 1 --2- 172.21.15.44:0/178239060 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500acd0 0x55e4350181c0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.766+0000 7f12fc83e740 1 -- 172.21.15.44:0/178239060 >> 172.21.15.44:0/178239060 conn(0x55e435005750 msgr2=0x55e4350074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.766+0000 7f12fc83e740 1 -- 172.21.15.44:0/178239060 shutdown_connections 2024-04-21T23:11:36.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.766+0000 7f12fc83e740 1 -- 172.21.15.44:0/178239060 wait complete. 2024-04-21T23:11:36.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.766+0000 7f12fc83e740 20 client.16287 trim_cache size 0 max 16384 2024-04-21T23:11:36.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.779+0000 7f12fc83e740 1 Processor -- start 2024-04-21T23:11:36.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.779+0000 7f12fc83e740 1 -- start start 2024-04-21T23:11:36.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.779+0000 7f12fc83e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e43500acd0 0x55e4350246a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.779+0000 7f12fc83e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e43500b5f0 0x55e435004a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.779+0000 7f12fc83e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e435018930 0x55e435017c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.779+0000 7f12fc83e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f12f41aaa80 con 0x55e435018930 2024-04-21T23:11:36.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.779+0000 7f12fc83e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f12f41ad300 con 0x55e43500b5f0 2024-04-21T23:11:36.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.779+0000 7f12fc83e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f12f41ab0f0 con 0x55e43500acd0 2024-04-21T23:11:36.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.780+0000 7f12f3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e435018930 0x55e435017c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.780+0000 7f12f2ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e43500b5f0 0x55e435004a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.780+0000 7f12f3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e435018930 0x55e435017c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38676/0 (socket says 172.21.15.44:38676) 2024-04-21T23:11:36.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.780+0000 7f12f3fff640 1 -- 172.21.15.44:0/3920033674 learned_addr learned my addr 172.21.15.44:0/3920033674 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:36.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.780+0000 7f12f3fff640 1 --2- 172.21.15.44:0/3920033674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e43500acd0 0x55e4350246a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.780+0000 7f12f2ffd640 1 -- 172.21.15.44:0/3920033674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e43500acd0 msgr2=0x55e4350246a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.780+0000 7f12f2ffd640 1 --2- 172.21.15.44:0/3920033674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e43500acd0 0x55e4350246a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.780+0000 7f12f2ffd640 1 -- 172.21.15.44:0/3920033674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e435018930 msgr2=0x55e435017c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.780+0000 7f12f2ffd640 1 --2- 172.21.15.44:0/3920033674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e435018930 0x55e435017c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.780+0000 7f12f2ffd640 1 -- 172.21.15.44:0/3920033674 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e434d4d7d0 con 0x55e43500b5f0 2024-04-21T23:11:36.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.780+0000 7f12f3fff640 1 --2- 172.21.15.44:0/3920033674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e435018930 0x55e435017c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:36.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.780+0000 7f12f2ffd640 1 --2- 172.21.15.44:0/3920033674 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e43500b5f0 0x55e435004a90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f12e8009230 tx=0x7f12e8005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1d7d66edb218cb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:36.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.780+0000 7f12d97fa640 1 -- 172.21.15.44:0/3920033674 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e804c070 con 0x55e43500b5f0 2024-04-21T23:11:36.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.780+0000 7f12d97fa640 1 -- 172.21.15.44:0/3920033674 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12e802cd60 con 0x55e43500b5f0 2024-04-21T23:11:36.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.780+0000 7f12d97fa640 1 -- 172.21.15.44:0/3920033674 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e80046c0 con 0x55e43500b5f0 2024-04-21T23:11:36.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.781+0000 7f12fc83e740 1 -- 172.21.15.44:0/3920033674 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e43500b5f0 msgr2=0x55e435004a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.781+0000 7f12fc83e740 1 --2- 172.21.15.44:0/3920033674 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e43500b5f0 0x55e435004a90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f12e8009230 tx=0x7f12e8005fd0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.781+0000 7f12fc83e740 1 -- 172.21.15.44:0/3920033674 shutdown_connections 2024-04-21T23:11:36.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.781+0000 7f12fc83e740 1 --2- 172.21.15.44:0/3920033674 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e435018930 0x55e435017c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.781+0000 7f12fc83e740 1 --2- 172.21.15.44:0/3920033674 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e43500b5f0 0x55e435004a90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.781+0000 7f12fc83e740 1 --2- 172.21.15.44:0/3920033674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e43500acd0 0x55e4350246a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.781+0000 7f12fc83e740 1 -- 172.21.15.44:0/3920033674 >> 172.21.15.44:0/3920033674 conn(0x55e435005750 msgr2=0x55e435006fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.781+0000 7f12fc83e740 1 -- 172.21.15.44:0/3920033674 shutdown_connections 2024-04-21T23:11:36.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.781+0000 7f12fc83e740 1 -- 172.21.15.44:0/3920033674 wait complete. 2024-04-21T23:11:36.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.781+0000 7f12fc83e740 1 -- start start 2024-04-21T23:11:36.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.781+0000 7f12fc83e740 1 Processor -- start 2024-04-21T23:11:36.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.782+0000 7f12fc83e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e43500acd0 0x55e435020e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.782+0000 7f12fc83e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500b5f0 0x55e43501c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.782+0000 7f12fc83e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e435018930 0x55e43501c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.782+0000 7f12fc83e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f12f41ab7c0 con 0x55e43500b5f0 2024-04-21T23:11:36.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.782+0000 7f12fc83e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f12f41acc30 con 0x55e43500acd0 2024-04-21T23:11:36.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.782+0000 7f12fc83e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f12f41abe90 con 0x55e435018930 2024-04-21T23:11:36.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.782+0000 7f12fc83e740 10 client.0 _ll_register_callbacks cb 0x7f12e40d3570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:36.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.782+0000 7f12f2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500b5f0 0x55e43501c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.782+0000 7f12f3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e43500acd0 0x55e435020e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.782+0000 7f12f2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500b5f0 0x55e43501c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38686/0 (socket says 172.21.15.44:38686) 2024-04-21T23:11:36.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.782+0000 7f12f2ffd640 1 -- 172.21.15.44:0/2653706877 learned_addr learned my addr 172.21.15.44:0/2653706877 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:36.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.782+0000 7f12f3fff640 1 --2- 172.21.15.44:0/2653706877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e435018930 0x55e43501c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.782+0000 7f12f2ffd640 1 -- 172.21.15.44:0/2653706877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e435018930 msgr2=0x55e43501c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.782+0000 7f12f2ffd640 1 --2- 172.21.15.44:0/2653706877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e435018930 0x55e43501c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.782+0000 7f12f2ffd640 1 -- 172.21.15.44:0/2653706877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e43500acd0 msgr2=0x55e435020e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:36.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.782+0000 7f12f2ffd640 1 --2- 172.21.15.44:0/2653706877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e43500acd0 0x55e435020e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.782+0000 7f12f2ffd640 1 -- 172.21.15.44:0/2653706877 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e43501ce20 con 0x55e43500b5f0 2024-04-21T23:11:36.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.782+0000 7f12f2ffd640 1 --2- 172.21.15.44:0/2653706877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500b5f0 0x55e43501c1f0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f12e802c170 tx=0x7f12e8048440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee034d567da509a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:36.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.782+0000 7f12d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:11:36.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.782+0000 7f12d2ffd640 1 -- 172.21.15.44:0/2653706877 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e8055070 con 0x55e43500b5f0 2024-04-21T23:11:36.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.783+0000 7f12d2ffd640 1 -- 172.21.15.44:0/2653706877 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f12e802ca80 con 0x55e43500b5f0 2024-04-21T23:11:36.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.783+0000 7f12fc83e740 1 -- 172.21.15.44:0/2653706877 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12e802cd60 con 0x55e43500b5f0 2024-04-21T23:11:36.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.783+0000 7f12d2ffd640 1 -- 172.21.15.44:0/2653706877 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f12e8061050 con 0x55e43500b5f0 2024-04-21T23:11:36.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.783+0000 7f12fc83e740 10 client.16293 fetch_fsmap learned FSMap version 25 2024-04-21T23:11:36.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.783+0000 7f12fc83e740 1 -- 172.21.15.44:0/2653706877 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e43508eb30 con 0x55e43500b5f0 2024-04-21T23:11:36.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.783+0000 7f12d2ffd640 1 -- 172.21.15.44:0/2653706877 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12e802cd60 con 0x55e43500b5f0 2024-04-21T23:11:36.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.783+0000 7f12d2ffd640 1 -- 172.21.15.44:0/2653706877 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f12e8004f50 con 0x55e43500b5f0 2024-04-21T23:11:36.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.783+0000 7f12fc83e740 10 client.16293 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:11:36.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.783+0000 7f12fc83e740 10 client.16293 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:36.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.783+0000 7f12fc83e740 20 client.16293 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:36.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.783+0000 7f12fc83e740 1 --2- 172.21.15.44:0/2653706877 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e43508f230 0x55e435091650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:36.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.783+0000 7f12fc83e740 4 client.16293 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:11:36.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.783+0000 7f12fc83e740 1 -- 172.21.15.44:0/2653706877 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e434f76dc0 con 0x55e43508f230 2024-04-21T23:11:36.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.783+0000 7f12f3fff640 1 --2- 172.21.15.44:0/2653706877 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e43508f230 0x55e435091650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:36.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.784+0000 7f12f3fff640 1 --2- 172.21.15.44:0/2653706877 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e43508f230 0x55e435091650 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4778083614e75184 server_cookie=6b7be8c579f01016 in_seq=0 out_seq=0 2024-04-21T23:11:36.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.784+0000 7f12d2ffd640 10 client.16293 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:36.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.784+0000 7f12d2ffd640 1 -- 172.21.15.44:0/2653706877 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e434f76dc0 con 0x55e43508f230 2024-04-21T23:11:36.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.784+0000 7f12d2ffd640 10 client.16293 handle_command_reply: tid=0 2024-04-21T23:11:36.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.784+0000 7f12fc83e740 1 client.16293 shutdown 2024-04-21T23:11:36.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.784+0000 7f12fc83e740 1 -- 172.21.15.44:0/2653706877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500b5f0 msgr2=0x55e43501c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.784+0000 7f12fc83e740 1 --2- 172.21.15.44:0/2653706877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500b5f0 0x55e43501c1f0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f12e802c170 tx=0x7f12e8048440 comp rx=0 tx=0).stop 2024-04-21T23:11:36.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.784+0000 7f12fc83e740 1 -- 172.21.15.44:0/2653706877 shutdown_connections 2024-04-21T23:11:36.795 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.784+0000 7f12fc83e740 1 --2- 172.21.15.44:0/2653706877 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e43508f230 0x55e435091650 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.784+0000 7f12fc83e740 1 --2- 172.21.15.44:0/2653706877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e435018930 0x55e43501c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.784+0000 7f12d2ffd640 0 client.16293 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:36.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.784+0000 7f12fc83e740 1 --2- 172.21.15.44:0/2653706877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e43500b5f0 0x55e43501c1f0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.784+0000 7f12fc83e740 1 --2- 172.21.15.44:0/2653706877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e43500acd0 0x55e435020e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.784+0000 7f12fc83e740 1 -- 172.21.15.44:0/2653706877 >> 172.21.15.44:0/2653706877 conn(0x55e435005750 msgr2=0x55e435023690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.784+0000 7f12fc83e740 1 -- 172.21.15.44:0/2653706877 shutdown_connections 2024-04-21T23:11:36.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.784+0000 7f12fc83e740 1 -- 172.21.15.44:0/2653706877 wait complete. 2024-04-21T23:11:36.796 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.785+0000 7f12fc83e740 20 client.16293 trim_cache size 0 max 16384 2024-04-21T23:11:36.796 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-21T23:11:36.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.785+0000 7f12f9c59640 1 -- 172.21.15.44:0/2466993875 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f12dc0a2d00 msgr2=0x7f12dc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.785+0000 7f12f9c59640 1 --2- 172.21.15.44:0/2466993875 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f12dc0a2d00 0x7f12dc0a51c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f12e40ccfb0 tx=0x7f12e40d36a0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.785+0000 7f12f9c59640 1 -- 172.21.15.44:0/2466993875 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12f4101700 msgr2=0x7f12f4198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:36.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.785+0000 7f12f9c59640 1 --2- 172.21.15.44:0/2466993875 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12f4101700 0x7f12f4198040 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f12e400e9f0 tx=0x7f12e400eeb0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.785+0000 7f12f9c59640 1 -- 172.21.15.44:0/2466993875 shutdown_connections 2024-04-21T23:11:36.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.785+0000 7f12f9c59640 1 --2- 172.21.15.44:0/2466993875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12f41020c0 0x7f12f4198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.785+0000 7f12f9c59640 1 --2- 172.21.15.44:0/2466993875 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f12dc0a2d00 0x7f12dc0a51c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.785+0000 7f12f9c59640 1 --2- 172.21.15.44:0/2466993875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f12f4107710 0x7f12f419c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.785+0000 7f12f9c59640 1 --2- 172.21.15.44:0/2466993875 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f12f4101700 0x7f12f4198040 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:36.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.785+0000 7f12f9c59640 1 -- 172.21.15.44:0/2466993875 >> 172.21.15.44:0/2466993875 conn(0x7f12f40fd1f0 msgr2=0x7f12f40fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:36.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.785+0000 7f12f9c59640 1 -- 172.21.15.44:0/2466993875 shutdown_connections 2024-04-21T23:11:36.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:36.785+0000 7f12f9c59640 1 -- 172.21.15.44:0/2466993875 wait complete. 2024-04-21T23:11:37.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:37 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2220455827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:11:37.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 230976 14.53 MB/sec warmup 87 sec latency 13.528 ms 2024-04-21T23:11:37.713 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:37 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2220455827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:11:37.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:37 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2220455827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:11:37.800 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:11:37.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:11:37.800 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:11:37.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.910+0000 7fa173edf640 1 -- 172.21.15.44:0/2637035632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c107890 msgr2=0x7fa16c107c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:37.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.910+0000 7fa173edf640 1 --2- 172.21.15.44:0/2637035632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c107890 0x7fa16c107c70 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fa168009900 tx=0x7fa16802bc80 comp rx=0 tx=0).stop 2024-04-21T23:11:37.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.910+0000 7fa173edf640 1 -- 172.21.15.44:0/2637035632 shutdown_connections 2024-04-21T23:11:37.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.910+0000 7fa173edf640 1 --2- 172.21.15.44:0/2637035632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa16c100000 0x7fa16c100480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.910+0000 7fa173edf640 1 --2- 172.21.15.44:0/2637035632 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa16c1009c0 0x7fa16c10e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.910+0000 7fa173edf640 1 --2- 172.21.15.44:0/2637035632 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c107890 0x7fa16c107c70 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.910+0000 7fa173edf640 1 -- 172.21.15.44:0/2637035632 >> 172.21.15.44:0/2637035632 conn(0x7fa16c076930 msgr2=0x7fa16c076d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:37.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.910+0000 7fa173edf640 1 -- 172.21.15.44:0/2637035632 shutdown_connections 2024-04-21T23:11:37.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.910+0000 7fa173edf640 1 -- 172.21.15.44:0/2637035632 wait complete. 2024-04-21T23:11:37.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.911+0000 7fa173edf640 1 Processor -- start 2024-04-21T23:11:37.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.911+0000 7fa173edf640 1 -- start start 2024-04-21T23:11:37.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.911+0000 7fa173edf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c100000 0x7fa16c1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:37.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.911+0000 7fa173edf640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa16c1009c0 0x7fa16c198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:37.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.911+0000 7fa173edf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa16c107890 0x7fa16c19c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:37.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.911+0000 7fa173edf640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c111b70 con 0x7fa16c100000 2024-04-21T23:11:37.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.911+0000 7fa173edf640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c1119f0 con 0x7fa16c1009c0 2024-04-21T23:11:37.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.911+0000 7fa173edf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c111cf0 con 0x7fa16c107890 2024-04-21T23:11:37.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.911+0000 7fa171c7b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c100000 0x7fa16c1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:37.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.911+0000 7fa171c7b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c100000 0x7fa16c1980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38696/0 (socket says 172.21.15.44:38696) 2024-04-21T23:11:37.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.911+0000 7fa171c7b640 1 -- 172.21.15.44:0/718809569 learned_addr learned my addr 172.21.15.44:0/718809569 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:37.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.911+0000 7fa17147a640 1 --2- 172.21.15.44:0/718809569 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa16c1009c0 0x7fa16c198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:37.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.911+0000 7fa17247c640 1 --2- 172.21.15.44:0/718809569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa16c107890 0x7fa16c19c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:37.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.912+0000 7fa171c7b640 1 -- 172.21.15.44:0/718809569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa16c107890 msgr2=0x7fa16c19c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:37.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.912+0000 7fa171c7b640 1 --2- 172.21.15.44:0/718809569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa16c107890 0x7fa16c19c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.912+0000 7fa171c7b640 1 -- 172.21.15.44:0/718809569 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa16c1009c0 msgr2=0x7fa16c198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:37.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.912+0000 7fa171c7b640 1 --2- 172.21.15.44:0/718809569 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa16c1009c0 0x7fa16c198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.912+0000 7fa171c7b640 1 -- 172.21.15.44:0/718809569 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16c19d0c0 con 0x7fa16c100000 2024-04-21T23:11:37.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.912+0000 7fa171c7b640 1 --2- 172.21.15.44:0/718809569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c100000 0x7fa16c1980d0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa168000c00 tx=0x7fa1680043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:37.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.912+0000 7fa162ffd640 1 -- 172.21.15.44:0/718809569 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa168041070 con 0x7fa16c100000 2024-04-21T23:11:37.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.912+0000 7fa162ffd640 1 -- 172.21.15.44:0/718809569 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa16802ccb0 con 0x7fa16c100000 2024-04-21T23:11:37.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.912+0000 7fa162ffd640 1 -- 172.21.15.44:0/718809569 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa168003de0 con 0x7fa16c100000 2024-04-21T23:11:37.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.912+0000 7fa173edf640 1 -- 172.21.15.44:0/718809569 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa16c19d350 con 0x7fa16c100000 2024-04-21T23:11:37.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.912+0000 7fa173edf640 1 -- 172.21.15.44:0/718809569 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa16c1a4c40 con 0x7fa16c100000 2024-04-21T23:11:37.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.915+0000 7fa162ffd640 1 -- 172.21.15.44:0/718809569 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa168044070 con 0x7fa16c100000 2024-04-21T23:11:37.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.915+0000 7fa176ba6740 1 Processor -- start 2024-04-21T23:11:37.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.915+0000 7fa176ba6740 1 -- start start 2024-04-21T23:11:37.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.915+0000 7fa176ba6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dd720 0x55bcb47dc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:37.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.915+0000 7fa176ba6740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47dca40 0x55bcb47dcec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:37.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa176ba6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47e5db0 0x55bcb47e81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:37.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa176ba6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55bcb4740170 con 0x55bcb47dd720 2024-04-21T23:11:37.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa176ba6740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55bcb46e87c0 con 0x55bcb47dca40 2024-04-21T23:11:37.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa176ba6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55bcb473c130 con 0x55bcb47e5db0 2024-04-21T23:11:37.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa17247c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dd720 0x55bcb47dc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:37.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa17147a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47e5db0 0x55bcb47e81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:37.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa171c7b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47dca40 0x55bcb47dcec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:37.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa17147a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47e5db0 0x55bcb47e81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:59268/0 (socket says 172.21.15.44:59268) 2024-04-21T23:11:37.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa17247c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dd720 0x55bcb47dc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38704/0 (socket says 172.21.15.44:38704) 2024-04-21T23:11:37.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa17147a640 1 -- 172.21.15.44:0/1042496504 learned_addr learned my addr 172.21.15.44:0/1042496504 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:37.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa162ffd640 1 --2- 172.21.15.44:0/718809569 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fa1580a2d80 0x7fa1580a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:37.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa17147a640 1 -- 172.21.15.44:0/1042496504 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47dca40 msgr2=0x55bcb47dcec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:37.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa17147a640 1 --2- 172.21.15.44:0/1042496504 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47dca40 0x55bcb47dcec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa17147a640 1 -- 172.21.15.44:0/1042496504 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dd720 msgr2=0x55bcb47dc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:37.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa17147a640 1 --2- 172.21.15.44:0/1042496504 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dd720 0x55bcb47dc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa17147a640 1 -- 172.21.15.44:0/1042496504 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcb46f9e60 con 0x55bcb47e5db0 2024-04-21T23:11:37.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa17247c640 1 --2- 172.21.15.44:0/1042496504 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dd720 0x55bcb47dc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:37.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa171c7b640 1 --2- 172.21.15.44:0/1042496504 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47dca40 0x55bcb47dcec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:37.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa162ffd640 1 -- 172.21.15.44:0/718809569 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fa1680f0f50 con 0x7fa16c100000 2024-04-21T23:11:37.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa17247c640 1 --2- 172.21.15.44:0/718809569 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fa1580a2d80 0x7fa1580a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:37.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.916+0000 7fa17147a640 1 --2- 172.21.15.44:0/1042496504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47e5db0 0x55bcb47e81a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa15c00bcb0 tx=0x7fa15c0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69689f7e01dd48ca server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:37.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.917+0000 7fa157fff640 1 -- 172.21.15.44:0/1042496504 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15c004030 con 0x55bcb47e5db0 2024-04-21T23:11:37.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.917+0000 7fa157fff640 1 -- 172.21.15.44:0/1042496504 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa15c0041d0 con 0x55bcb47e5db0 2024-04-21T23:11:37.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.917+0000 7fa17247c640 1 --2- 172.21.15.44:0/718809569 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fa1580a2d80 0x7fa1580a5240 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa164009fd0 tx=0x7fa164015040 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:37.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.917+0000 7fa157fff640 1 -- 172.21.15.44:0/1042496504 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15c0117b0 con 0x55bcb47e5db0 2024-04-21T23:11:37.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.917+0000 7fa176ba6740 1 -- 172.21.15.44:0/1042496504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47e5db0 msgr2=0x55bcb47e81a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:37.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.917+0000 7fa176ba6740 1 --2- 172.21.15.44:0/1042496504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47e5db0 0x55bcb47e81a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa15c00bcb0 tx=0x7fa15c0107a0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.917+0000 7fa176ba6740 1 -- 172.21.15.44:0/1042496504 shutdown_connections 2024-04-21T23:11:37.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.917+0000 7fa176ba6740 1 --2- 172.21.15.44:0/1042496504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47e5db0 0x55bcb47e81a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.917+0000 7fa176ba6740 1 --2- 172.21.15.44:0/1042496504 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47dca40 0x55bcb47dcec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.917+0000 7fa176ba6740 1 --2- 172.21.15.44:0/1042496504 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dd720 0x55bcb47dc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.917+0000 7fa176ba6740 1 -- 172.21.15.44:0/1042496504 >> 172.21.15.44:0/1042496504 conn(0x55bcb47a34f0 msgr2=0x55bcb47d7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:37.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.918+0000 7fa176ba6740 1 -- 172.21.15.44:0/1042496504 shutdown_connections 2024-04-21T23:11:37.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.918+0000 7fa176ba6740 1 -- 172.21.15.44:0/1042496504 wait complete. 2024-04-21T23:11:37.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.918+0000 7fa176ba6740 1 -- start start 2024-04-21T23:11:37.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.918+0000 7fa176ba6740 1 Processor -- start 2024-04-21T23:11:37.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.918+0000 7fa176ba6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dca40 0x55bcb47e2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:37.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.918+0000 7fa176ba6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47dd720 0x55bcb47e3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:37.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.918+0000 7fa176ba6740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47e5db0 0x55bcb47e3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:37.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.918+0000 7fa176ba6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55bcb47e8c20 con 0x55bcb47dca40 2024-04-21T23:11:37.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.918+0000 7fa176ba6740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55bcb46fbd40 con 0x55bcb47e5db0 2024-04-21T23:11:37.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.918+0000 7fa176ba6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55bcb47e8da0 con 0x55bcb47dd720 2024-04-21T23:11:37.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.918+0000 7fa176ba6740 10 client.0 _ll_register_callbacks cb 0x55bcb4737820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:37.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.918+0000 7fa171c7b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dca40 0x55bcb47e2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:37.926 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.919+0000 7fa171c7b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dca40 0x55bcb47e2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38710/0 (socket says 172.21.15.44:38710) 2024-04-21T23:11:37.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.919+0000 7fa17147a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47dd720 0x55bcb47e3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:37.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.919+0000 7fa171c7b640 1 -- 172.21.15.44:0/1128072699 learned_addr learned my addr 172.21.15.44:0/1128072699 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:37.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.919+0000 7fa17247c640 1 --2- 172.21.15.44:0/1128072699 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47e5db0 0x55bcb47e3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:37.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.919+0000 7fa171c7b640 1 -- 172.21.15.44:0/1128072699 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47dd720 msgr2=0x55bcb47e3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:37.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.919+0000 7fa171c7b640 1 --2- 172.21.15.44:0/1128072699 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47dd720 0x55bcb47e3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.919+0000 7fa171c7b640 1 -- 172.21.15.44:0/1128072699 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47e5db0 msgr2=0x55bcb47e3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:37.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.919+0000 7fa171c7b640 1 --2- 172.21.15.44:0/1128072699 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47e5db0 0x55bcb47e3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.919+0000 7fa171c7b640 1 -- 172.21.15.44:0/1128072699 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcb451c7e0 con 0x55bcb47dca40 2024-04-21T23:11:37.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.919+0000 7fa171c7b640 1 --2- 172.21.15.44:0/1128072699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dca40 0x55bcb47e2d00 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa1680ee470 tx=0x7fa168005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33dab54867fd38f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:37.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.919+0000 7fa1557fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:11:37.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.919+0000 7fa1557fa640 1 -- 172.21.15.44:0/1128072699 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16804b070 con 0x55bcb47dca40 2024-04-21T23:11:37.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.919+0000 7fa1557fa640 1 -- 172.21.15.44:0/1128072699 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fa1680035f0 con 0x55bcb47dca40 2024-04-21T23:11:37.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.919+0000 7fa176ba6740 1 -- 172.21.15.44:0/1128072699 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa15c0041d0 con 0x55bcb47dca40 2024-04-21T23:11:37.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.919+0000 7fa1557fa640 1 -- 172.21.15.44:0/1128072699 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa1680bd060 con 0x55bcb47dca40 2024-04-21T23:11:37.927 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.919+0000 7fa176ba6740 10 client.16305 fetch_fsmap learned FSMap version 25 2024-04-21T23:11:37.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.919+0000 7fa176ba6740 1 -- 172.21.15.44:0/1128072699 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bcb457cf70 con 0x55bcb47dca40 2024-04-21T23:11:37.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.919+0000 7fa1557fa640 1 -- 172.21.15.44:0/1128072699 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa15c0041d0 con 0x55bcb47dca40 2024-04-21T23:11:37.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.920+0000 7fa1557fa640 1 -- 172.21.15.44:0/1128072699 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fa1680b9900 con 0x55bcb47dca40 2024-04-21T23:11:37.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.920+0000 7fa176ba6740 10 client.16305 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:11:37.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.920+0000 7fa176ba6740 10 client.16305 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:37.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.920+0000 7fa176ba6740 20 client.16305 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:37.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.920+0000 7fa176ba6740 1 --2- 172.21.15.44:0/1128072699 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55bcb47f5cf0 0x55bcb47f8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:37.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.920+0000 7fa176ba6740 4 client.16305 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:11:37.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.920+0000 7fa176ba6740 1 -- 172.21.15.44:0/1128072699 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bcb4714450 con 0x55bcb47f5cf0 2024-04-21T23:11:37.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.920+0000 7fa17147a640 1 --2- 172.21.15.44:0/1128072699 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55bcb47f5cf0 0x55bcb47f8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:37.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.920+0000 7fa17147a640 1 --2- 172.21.15.44:0/1128072699 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55bcb47f5cf0 0x55bcb47f8120 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3a79153edd167e0 server_cookie=d49ddc633d6c09d8 in_seq=0 out_seq=0 2024-04-21T23:11:37.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.920+0000 7fa1557fa640 10 client.16305 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:37.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.921+0000 7fa1557fa640 1 -- 172.21.15.44:0/1128072699 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bcb4714450 con 0x55bcb47f5cf0 2024-04-21T23:11:37.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.921+0000 7fa1557fa640 10 client.16305 handle_command_reply: tid=0 2024-04-21T23:11:37.928 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.921+0000 7fa176ba6740 1 client.16305 shutdown 2024-04-21T23:11:37.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.921+0000 7fa176ba6740 1 -- 172.21.15.44:0/1128072699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dca40 msgr2=0x55bcb47e2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:37.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.921+0000 7fa176ba6740 1 --2- 172.21.15.44:0/1128072699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dca40 0x55bcb47e2d00 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa1680ee470 tx=0x7fa168005c90 comp rx=0 tx=0).stop 2024-04-21T23:11:37.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.921+0000 7fa176ba6740 1 -- 172.21.15.44:0/1128072699 shutdown_connections 2024-04-21T23:11:37.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.921+0000 7fa176ba6740 1 --2- 172.21.15.44:0/1128072699 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55bcb47f5cf0 0x55bcb47f8120 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.921+0000 7fa176ba6740 1 --2- 172.21.15.44:0/1128072699 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47dd720 0x55bcb47e3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.921+0000 7fa176ba6740 1 --2- 172.21.15.44:0/1128072699 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47e5db0 0x55bcb47e3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.921+0000 7fa1557fa640 0 client.16305 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:37.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.921+0000 7fa176ba6740 1 --2- 172.21.15.44:0/1128072699 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dca40 0x55bcb47e2d00 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.921+0000 7fa176ba6740 1 -- 172.21.15.44:0/1128072699 >> 172.21.15.44:0/1128072699 conn(0x55bcb47a34f0 msgr2=0x55bcb47d91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:37.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.921+0000 7fa176ba6740 1 -- 172.21.15.44:0/1128072699 shutdown_connections 2024-04-21T23:11:37.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.921+0000 7fa176ba6740 1 -- 172.21.15.44:0/1128072699 wait complete. 2024-04-21T23:11:37.929 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.922+0000 7fa176ba6740 20 client.16305 trim_cache size 0 max 16384 2024-04-21T23:11:37.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.941+0000 7fa176ba6740 1 Processor -- start 2024-04-21T23:11:37.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.941+0000 7fa176ba6740 1 -- start start 2024-04-21T23:11:37.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.941+0000 7fa176ba6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47dca40 0x55bcb47e5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:37.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.941+0000 7fa176ba6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dd720 0x55bcb47f1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:37.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.941+0000 7fa176ba6740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47e5db0 0x55bcb47fda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:37.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.941+0000 7fa176ba6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c1aa9a0 con 0x55bcb47dd720 2024-04-21T23:11:37.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.941+0000 7fa176ba6740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c1ad220 con 0x55bcb47e5db0 2024-04-21T23:11:37.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.941+0000 7fa176ba6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c1ab010 con 0x55bcb47dca40 2024-04-21T23:11:37.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.941+0000 7fa17147a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dd720 0x55bcb47f1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:37.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.941+0000 7fa17147a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dd720 0x55bcb47f1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38726/0 (socket says 172.21.15.44:38726) 2024-04-21T23:11:37.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.941+0000 7fa17147a640 1 -- 172.21.15.44:0/3454960228 learned_addr learned my addr 172.21.15.44:0/3454960228 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:37.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.941+0000 7fa17247c640 1 --2- 172.21.15.44:0/3454960228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47e5db0 0x55bcb47fda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:37.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.941+0000 7fa171c7b640 1 --2- 172.21.15.44:0/3454960228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47dca40 0x55bcb47e5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:37.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.941+0000 7fa17147a640 1 -- 172.21.15.44:0/3454960228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47dca40 msgr2=0x55bcb47e5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:37.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.941+0000 7fa17147a640 1 --2- 172.21.15.44:0/3454960228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47dca40 0x55bcb47e5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.941+0000 7fa17147a640 1 -- 172.21.15.44:0/3454960228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47e5db0 msgr2=0x55bcb47fda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:37.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.941+0000 7fa17147a640 1 --2- 172.21.15.44:0/3454960228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47e5db0 0x55bcb47fda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.941+0000 7fa17147a640 1 -- 172.21.15.44:0/3454960228 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcb4414170 con 0x55bcb47dd720 2024-04-21T23:11:37.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.942+0000 7fa17147a640 1 --2- 172.21.15.44:0/3454960228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dd720 0x55bcb47f1b20 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa15c010ca0 tx=0x7fa15c00b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df1d54ec00f2c591 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:37.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.942+0000 7fa1577fe640 1 -- 172.21.15.44:0/3454960228 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15c02e070 con 0x55bcb47dd720 2024-04-21T23:11:37.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.942+0000 7fa1577fe640 1 -- 172.21.15.44:0/3454960228 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa15c005520 con 0x55bcb47dd720 2024-04-21T23:11:37.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.942+0000 7fa1577fe640 1 -- 172.21.15.44:0/3454960228 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15c003d60 con 0x55bcb47dd720 2024-04-21T23:11:37.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.942+0000 7fa176ba6740 1 -- 172.21.15.44:0/3454960228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dd720 msgr2=0x55bcb47f1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:37.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.942+0000 7fa176ba6740 1 --2- 172.21.15.44:0/3454960228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dd720 0x55bcb47f1b20 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa15c010ca0 tx=0x7fa15c00b390 comp rx=0 tx=0).stop 2024-04-21T23:11:37.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa176ba6740 1 -- 172.21.15.44:0/3454960228 shutdown_connections 2024-04-21T23:11:37.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa176ba6740 1 --2- 172.21.15.44:0/3454960228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47e5db0 0x55bcb47fda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa176ba6740 1 --2- 172.21.15.44:0/3454960228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dd720 0x55bcb47f1b20 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa176ba6740 1 --2- 172.21.15.44:0/3454960228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47dca40 0x55bcb47e5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa176ba6740 1 -- 172.21.15.44:0/3454960228 >> 172.21.15.44:0/3454960228 conn(0x55bcb47a34f0 msgr2=0x55bcb47d64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:37.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa176ba6740 1 -- 172.21.15.44:0/3454960228 shutdown_connections 2024-04-21T23:11:37.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa176ba6740 1 -- 172.21.15.44:0/3454960228 wait complete. 2024-04-21T23:11:37.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa176ba6740 1 -- start start 2024-04-21T23:11:37.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa176ba6740 1 Processor -- start 2024-04-21T23:11:37.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa176ba6740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dca40 0x55bcb47ee220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:37.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa176ba6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47dd720 0x55bcb47eb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:37.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa176ba6740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47e5db0 0x55bcb47eb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:37.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa176ba6740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c1ab6e0 con 0x55bcb47dca40 2024-04-21T23:11:37.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa176ba6740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c1acb50 con 0x55bcb47e5db0 2024-04-21T23:11:37.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa176ba6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c1abdb0 con 0x55bcb47dd720 2024-04-21T23:11:37.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa176ba6740 10 client.0 _ll_register_callbacks cb 0x7fa168009290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:37.947 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa171c7b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dca40 0x55bcb47ee220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:37.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa171c7b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dca40 0x55bcb47ee220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38728/0 (socket says 172.21.15.44:38728) 2024-04-21T23:11:37.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa17247c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47e5db0 0x55bcb47eb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:37.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa171c7b640 1 -- 172.21.15.44:0/1012874647 learned_addr learned my addr 172.21.15.44:0/1012874647 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:37.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.943+0000 7fa17147a640 1 --2- 172.21.15.44:0/1012874647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47dd720 0x55bcb47eb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:37.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.944+0000 7fa171c7b640 1 -- 172.21.15.44:0/1012874647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47dd720 msgr2=0x55bcb47eb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:37.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.944+0000 7fa171c7b640 1 --2- 172.21.15.44:0/1012874647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47dd720 0x55bcb47eb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.944+0000 7fa171c7b640 1 -- 172.21.15.44:0/1012874647 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47e5db0 msgr2=0x55bcb47eb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:37.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.944+0000 7fa171c7b640 1 --2- 172.21.15.44:0/1012874647 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47e5db0 0x55bcb47eb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.944+0000 7fa171c7b640 1 -- 172.21.15.44:0/1012874647 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcb47eba80 con 0x55bcb47dca40 2024-04-21T23:11:37.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.944+0000 7fa171c7b640 1 --2- 172.21.15.44:0/1012874647 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dca40 0x55bcb47ee220 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fa1680ee900 tx=0x7fa1680ee500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1651d6f554a81c03 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:37.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.944+0000 7fa1557fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:11:37.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.944+0000 7fa1557fa640 1 -- 172.21.15.44:0/1012874647 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16804b040 con 0x55bcb47dca40 2024-04-21T23:11:37.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.944+0000 7fa1557fa640 1 -- 172.21.15.44:0/1012874647 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fa1680c59c0 con 0x55bcb47dca40 2024-04-21T23:11:37.948 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.944+0000 7fa176ba6740 1 -- 172.21.15.44:0/1012874647 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa15c005520 con 0x55bcb47dca40 2024-04-21T23:11:37.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.944+0000 7fa1557fa640 1 -- 172.21.15.44:0/1012874647 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa1680bcbf0 con 0x55bcb47dca40 2024-04-21T23:11:37.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.944+0000 7fa176ba6740 10 client.16311 fetch_fsmap learned FSMap version 25 2024-04-21T23:11:37.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.944+0000 7fa176ba6740 1 -- 172.21.15.44:0/1012874647 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bcb485d060 con 0x55bcb47dca40 2024-04-21T23:11:37.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.944+0000 7fa1557fa640 1 -- 172.21.15.44:0/1012874647 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa15c005520 con 0x55bcb47dca40 2024-04-21T23:11:37.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.944+0000 7fa1557fa640 1 -- 172.21.15.44:0/1012874647 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fa1680c10f0 con 0x55bcb47dca40 2024-04-21T23:11:37.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.945+0000 7fa176ba6740 10 client.16311 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:11:37.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.945+0000 7fa176ba6740 10 client.16311 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:37.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.945+0000 7fa176ba6740 20 client.16311 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:37.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.945+0000 7fa176ba6740 1 --2- 172.21.15.44:0/1012874647 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55bcb485d620 0x55bcb485fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:37.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.945+0000 7fa176ba6740 4 client.16311 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:11:37.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.945+0000 7fa176ba6740 1 -- 172.21.15.44:0/1012874647 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55bcb474d6b0 con 0x55bcb485d620 2024-04-21T23:11:37.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.945+0000 7fa17147a640 1 --2- 172.21.15.44:0/1012874647 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55bcb485d620 0x55bcb485fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:37.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.945+0000 7fa17147a640 1 --2- 172.21.15.44:0/1012874647 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55bcb485d620 0x55bcb485fa40 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13bff396ef23048f server_cookie=694108aeb43f8226 in_seq=0 out_seq=0 2024-04-21T23:11:37.949 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.945+0000 7fa1557fa640 10 client.16311 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:37.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.946+0000 7fa1557fa640 1 -- 172.21.15.44:0/1012874647 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55bcb474d6b0 con 0x55bcb485d620 2024-04-21T23:11:37.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.946+0000 7fa1557fa640 10 client.16311 handle_command_reply: tid=0 2024-04-21T23:11:37.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.947+0000 7fa176ba6740 1 client.16311 shutdown 2024-04-21T23:11:37.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.947+0000 7fa176ba6740 1 -- 172.21.15.44:0/1012874647 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dca40 msgr2=0x55bcb47ee220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:37.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.947+0000 7fa176ba6740 1 --2- 172.21.15.44:0/1012874647 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dca40 0x55bcb47ee220 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fa1680ee900 tx=0x7fa1680ee500 comp rx=0 tx=0).stop 2024-04-21T23:11:37.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.947+0000 7fa176ba6740 1 -- 172.21.15.44:0/1012874647 shutdown_connections 2024-04-21T23:11:37.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.947+0000 7fa176ba6740 1 --2- 172.21.15.44:0/1012874647 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55bcb485d620 0x55bcb485fa40 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.947+0000 7fa176ba6740 1 --2- 172.21.15.44:0/1012874647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcb47dd720 0x55bcb47eb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.947+0000 7fa176ba6740 1 --2- 172.21.15.44:0/1012874647 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bcb47e5db0 0x55bcb47eb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.947+0000 7fa1557fa640 0 client.16311 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:37.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.947+0000 7fa176ba6740 1 --2- 172.21.15.44:0/1012874647 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bcb47dca40 0x55bcb47ee220 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.947+0000 7fa176ba6740 1 -- 172.21.15.44:0/1012874647 >> 172.21.15.44:0/1012874647 conn(0x55bcb47a34f0 msgr2=0x55bcb47e2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:37.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.947+0000 7fa176ba6740 1 -- 172.21.15.44:0/1012874647 shutdown_connections 2024-04-21T23:11:37.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.947+0000 7fa176ba6740 1 -- 172.21.15.44:0/1012874647 wait complete. 2024-04-21T23:11:37.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.947+0000 7fa176ba6740 20 client.16311 trim_cache size 0 max 16384 2024-04-21T23:11:37.952 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:11:37.952 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-21T23:11:37.952 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "c68fc7a5-41e6-47fa-97b1-1ac823093063", 2024-04-21T23:11:37.952 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-21T23:11:37.952 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:11:37.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.947+0000 7fa173edf640 1 -- 172.21.15.44:0/718809569 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fa1580a2d80 msgr2=0x7fa1580a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:37.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.947+0000 7fa173edf640 1 --2- 172.21.15.44:0/718809569 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fa1580a2d80 0x7fa1580a5240 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa164009fd0 tx=0x7fa164015040 comp rx=0 tx=0).stop 2024-04-21T23:11:37.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.947+0000 7fa173edf640 1 -- 172.21.15.44:0/718809569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c100000 msgr2=0x7fa16c1980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:37.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.947+0000 7fa173edf640 1 --2- 172.21.15.44:0/718809569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c100000 0x7fa16c1980d0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa168000c00 tx=0x7fa1680043b0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.948+0000 7fa173edf640 1 -- 172.21.15.44:0/718809569 shutdown_connections 2024-04-21T23:11:37.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.948+0000 7fa173edf640 1 --2- 172.21.15.44:0/718809569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa16c107890 0x7fa16c19c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.948+0000 7fa173edf640 1 --2- 172.21.15.44:0/718809569 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fa1580a2d80 0x7fa1580a5240 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.948+0000 7fa173edf640 1 --2- 172.21.15.44:0/718809569 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa16c1009c0 0x7fa16c198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.948+0000 7fa173edf640 1 --2- 172.21.15.44:0/718809569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa16c100000 0x7fa16c1980d0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:37.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.948+0000 7fa173edf640 1 -- 172.21.15.44:0/718809569 >> 172.21.15.44:0/718809569 conn(0x7fa16c076930 msgr2=0x7fa16c0feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:37.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.948+0000 7fa173edf640 1 -- 172.21.15.44:0/718809569 shutdown_connections 2024-04-21T23:11:37.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:37.948+0000 7fa173edf640 1 -- 172.21.15.44:0/718809569 wait complete. 2024-04-21T23:11:37.957 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:11:37.957 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:11:38.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.074+0000 7f89dd35b640 1 -- 172.21.15.44:0/2000587952 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89d8101710 msgr2=0x7f89d8101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:38.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.074+0000 7f89dd35b640 1 --2- 172.21.15.44:0/2000587952 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89d8101710 0x7f89d8101b90 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f89cc009930 tx=0x7f89cc02bc30 comp rx=0 tx=0).stop 2024-04-21T23:11:38.076 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.074+0000 7f89dd35b640 1 -- 172.21.15.44:0/2000587952 shutdown_connections 2024-04-21T23:11:38.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.074+0000 7f89dd35b640 1 --2- 172.21.15.44:0/2000587952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d81020d0 0x7f89d810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.074+0000 7f89dd35b640 1 --2- 172.21.15.44:0/2000587952 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89d8101710 0x7f89d8101b90 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.074+0000 7f89dd35b640 1 --2- 172.21.15.44:0/2000587952 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89d8107720 0x7f89d8107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.074+0000 7f89dd35b640 1 -- 172.21.15.44:0/2000587952 >> 172.21.15.44:0/2000587952 conn(0x7f89d80fd200 msgr2=0x7f89d80ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:38.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.074+0000 7f89dd35b640 1 -- 172.21.15.44:0/2000587952 shutdown_connections 2024-04-21T23:11:38.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.074+0000 7f89dd35b640 1 -- 172.21.15.44:0/2000587952 wait complete. 2024-04-21T23:11:38.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.074+0000 7f89dd35b640 1 Processor -- start 2024-04-21T23:11:38.077 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.075+0000 7f89dd35b640 1 -- start start 2024-04-21T23:11:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.075+0000 7f89dd35b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89d8101710 0x7f89d8198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.075+0000 7f89dd35b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89d81020d0 0x7f89d81985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.075+0000 7f89dd35b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d8107720 0x7f89d819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.075+0000 7f89dd35b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f89d8111af0 con 0x7f89d81020d0 2024-04-21T23:11:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.075+0000 7f89dd35b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f89d8111970 con 0x7f89d8101710 2024-04-21T23:11:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.075+0000 7f89dd35b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f89d8111c70 con 0x7f89d8107720 2024-04-21T23:11:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.075+0000 7f89d67fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89d81020d0 0x7f89d81985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.075+0000 7f89d77fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d8107720 0x7f89d819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:38.078 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.075+0000 7f89d67fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89d81020d0 0x7f89d81985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38746/0 (socket says 172.21.15.44:38746) 2024-04-21T23:11:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.075+0000 7f89d67fc640 1 -- 172.21.15.44:0/409108483 learned_addr learned my addr 172.21.15.44:0/409108483 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.075+0000 7f89d6ffd640 1 --2- 172.21.15.44:0/409108483 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89d8101710 0x7f89d8198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.075+0000 7f89d67fc640 1 -- 172.21.15.44:0/409108483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d8107720 msgr2=0x7f89d819c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.075+0000 7f89d67fc640 1 --2- 172.21.15.44:0/409108483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d8107720 0x7f89d819c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.075+0000 7f89d67fc640 1 -- 172.21.15.44:0/409108483 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89d8101710 msgr2=0x7f89d8198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.075+0000 7f89d67fc640 1 --2- 172.21.15.44:0/409108483 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89d8101710 0x7f89d8198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.075+0000 7f89d67fc640 1 -- 172.21.15.44:0/409108483 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d819d030 con 0x7f89d81020d0 2024-04-21T23:11:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.075+0000 7f89d77fe640 1 --2- 172.21.15.44:0/409108483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d8107720 0x7f89d819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:38.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.075+0000 7f89d6ffd640 1 --2- 172.21.15.44:0/409108483 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89d8101710 0x7f89d8198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:38.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.075+0000 7f89d67fc640 1 --2- 172.21.15.44:0/409108483 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89d81020d0 0x7f89d81985d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f89cc009230 tx=0x7f89cc02cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:38.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.076+0000 7f89bffff640 1 -- 172.21.15.44:0/409108483 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89cc041070 con 0x7f89d81020d0 2024-04-21T23:11:38.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.076+0000 7f89bffff640 1 -- 172.21.15.44:0/409108483 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89cc004450 con 0x7f89d81020d0 2024-04-21T23:11:38.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.076+0000 7f89dd35b640 1 -- 172.21.15.44:0/409108483 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89d819d2c0 con 0x7f89d81020d0 2024-04-21T23:11:38.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.076+0000 7f89bffff640 1 -- 172.21.15.44:0/409108483 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89cc003e50 con 0x7f89d81020d0 2024-04-21T23:11:38.080 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.076+0000 7f89dd35b640 1 -- 172.21.15.44:0/409108483 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89d81a4bb0 con 0x7f89d81020d0 2024-04-21T23:11:38.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.078+0000 7f89bffff640 1 -- 172.21.15.44:0/409108483 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f89cc044070 con 0x7f89d81020d0 2024-04-21T23:11:38.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.078+0000 7f89e0026740 1 Processor -- start 2024-04-21T23:11:38.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89e0026740 1 -- start start 2024-04-21T23:11:38.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89e0026740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75d5f30 0x55bbd75d6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:38.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89e0026740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 0x55bbd75deef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:38.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89e0026740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75df430 0x55bbd75e1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:38.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89e0026740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55bbd753a560 con 0x55bbd75d6850 2024-04-21T23:11:38.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89e0026740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55bbd74e2730 con 0x55bbd75df430 2024-04-21T23:11:38.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89e0026740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55bbd7536520 con 0x55bbd75d5f30 2024-04-21T23:11:38.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89d6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 0x55bbd75deef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:38.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89d77fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75d5f30 0x55bbd75d6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:38.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89bffff640 1 --2- 172.21.15.44:0/409108483 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f89c00a2f20 0x7f89c00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:38.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89d6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 0x55bbd75deef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38760/0 (socket says 172.21.15.44:38760) 2024-04-21T23:11:38.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89d77fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75d5f30 0x55bbd75d6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:59326/0 (socket says 172.21.15.44:59326) 2024-04-21T23:11:38.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89d6ffd640 1 -- 172.21.15.44:0/3730797990 learned_addr learned my addr 172.21.15.44:0/3730797990 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:38.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89d67fc640 1 --2- 172.21.15.44:0/3730797990 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75df430 0x55bbd75e1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:38.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89d77fe640 1 --2- 172.21.15.44:0/409108483 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f89c00a2f20 0x7f89c00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:38.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89d6ffd640 1 -- 172.21.15.44:0/3730797990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75d5f30 msgr2=0x55bbd75d6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:38.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89d6ffd640 1 --2- 172.21.15.44:0/3730797990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75d5f30 0x55bbd75d6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89d6ffd640 1 -- 172.21.15.44:0/3730797990 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75df430 msgr2=0x55bbd75e1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:38.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89d6ffd640 1 --2- 172.21.15.44:0/3730797990 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75df430 0x55bbd75e1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89d6ffd640 1 -- 172.21.15.44:0/3730797990 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbd73187d0 con 0x55bbd75d6850 2024-04-21T23:11:38.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89d67fc640 1 --2- 172.21.15.44:0/3730797990 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75df430 0x55bbd75e1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:11:38.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89d77fe640 1 --2- 172.21.15.44:0/3730797990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75d5f30 0x55bbd75d6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:38.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.079+0000 7f89bffff640 1 -- 172.21.15.44:0/409108483 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f89cc0f0f40 con 0x7f89d81020d0 2024-04-21T23:11:38.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.080+0000 7f89d6ffd640 1 --2- 172.21.15.44:0/3730797990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 0x55bbd75deef0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f89c80114f0 tx=0x7f89c80119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4b3102c5d0c38e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:38.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.080+0000 7f89d77fe640 1 --2- 172.21.15.44:0/409108483 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f89c00a2f20 0x7f89c00a53e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f89d00049d0 tx=0x7f89d0004ab0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:38.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.080+0000 7f89bd7fa640 1 -- 172.21.15.44:0/3730797990 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c801e070 con 0x55bbd75d6850 2024-04-21T23:11:38.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.080+0000 7f89bd7fa640 1 -- 172.21.15.44:0/3730797990 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89c800faa0 con 0x55bbd75d6850 2024-04-21T23:11:38.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.080+0000 7f89bd7fa640 1 -- 172.21.15.44:0/3730797990 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c8019b50 con 0x55bbd75d6850 2024-04-21T23:11:38.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.080+0000 7f89e0026740 1 -- 172.21.15.44:0/3730797990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 msgr2=0x55bbd75deef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:38.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.080+0000 7f89e0026740 1 --2- 172.21.15.44:0/3730797990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 0x55bbd75deef0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f89c80114f0 tx=0x7f89c80119b0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.081+0000 7f89e0026740 1 -- 172.21.15.44:0/3730797990 shutdown_connections 2024-04-21T23:11:38.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.081+0000 7f89e0026740 1 --2- 172.21.15.44:0/3730797990 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75df430 0x55bbd75e1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.081+0000 7f89e0026740 1 --2- 172.21.15.44:0/3730797990 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 0x55bbd75deef0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.081+0000 7f89e0026740 1 --2- 172.21.15.44:0/3730797990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75d5f30 0x55bbd75d6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.081+0000 7f89e0026740 1 -- 172.21.15.44:0/3730797990 >> 172.21.15.44:0/3730797990 conn(0x55bbd75d0660 msgr2=0x55bbd75d0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:38.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.081+0000 7f89e0026740 1 -- 172.21.15.44:0/3730797990 shutdown_connections 2024-04-21T23:11:38.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.081+0000 7f89e0026740 1 -- 172.21.15.44:0/3730797990 wait complete. 2024-04-21T23:11:38.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.081+0000 7f89e0026740 1 -- start start 2024-04-21T23:11:38.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.081+0000 7f89e0026740 1 Processor -- start 2024-04-21T23:11:38.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.081+0000 7f89e0026740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75d5f30 0x55bbd75dc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:38.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.081+0000 7f89e0026740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 0x55bbd75dc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:38.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.081+0000 7f89e0026740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75df430 0x55bbd75dcef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:38.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.081+0000 7f89e0026740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55bbd75e2220 con 0x55bbd75d6850 2024-04-21T23:11:38.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.081+0000 7f89e0026740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55bbd74f5e00 con 0x55bbd75d5f30 2024-04-21T23:11:38.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.081+0000 7f89e0026740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55bbd75e23a0 con 0x55bbd75df430 2024-04-21T23:11:38.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.081+0000 7f89e0026740 10 client.0 _ll_register_callbacks cb 0x55bbd7530eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:38.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.082+0000 7f89d6ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75d5f30 0x55bbd75dc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:38.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.082+0000 7f89d67fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 0x55bbd75dc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:38.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.082+0000 7f89d6ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75d5f30 0x55bbd75dc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:33582/0 (socket says 172.21.15.44:33582) 2024-04-21T23:11:38.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.082+0000 7f89d67fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 0x55bbd75dc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38774/0 (socket says 172.21.15.44:38774) 2024-04-21T23:11:38.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.082+0000 7f89d6ffd640 1 -- 172.21.15.44:0/1647810829 learned_addr learned my addr 172.21.15.44:0/1647810829 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:38.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.082+0000 7f89d77fe640 1 --2- 172.21.15.44:0/1647810829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75df430 0x55bbd75dcef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.082+0000 7f89d6ffd640 1 -- 172.21.15.44:0/1647810829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75df430 msgr2=0x55bbd75dcef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.082+0000 7f89d6ffd640 1 --2- 172.21.15.44:0/1647810829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75df430 0x55bbd75dcef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.082+0000 7f89d6ffd640 1 -- 172.21.15.44:0/1647810829 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 msgr2=0x55bbd75dc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.082+0000 7f89d6ffd640 1 --2- 172.21.15.44:0/1647810829 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 0x55bbd75dc9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.082+0000 7f89d6ffd640 1 -- 172.21.15.44:0/1647810829 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbd724e3f0 con 0x55bbd75d5f30 2024-04-21T23:11:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.082+0000 7f89d67fc640 1 --2- 172.21.15.44:0/1647810829 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 0x55bbd75dc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.082+0000 7f89d6ffd640 1 --2- 172.21.15.44:0/1647810829 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75d5f30 0x55bbd75dc470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f89c8000c00 tx=0x7f89c8005710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be61cba9137f6bf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.082+0000 7f89baff5640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:11:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.082+0000 7f89baff5640 1 -- 172.21.15.44:0/1647810829 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c801e040 con 0x55bbd75d5f30 2024-04-21T23:11:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.082+0000 7f89e0026740 1 -- 172.21.15.44:0/1647810829 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89c800faa0 con 0x55bbd75d5f30 2024-04-21T23:11:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.082+0000 7f89baff5640 1 -- 172.21.15.44:0/1647810829 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f89c8016040 con 0x55bbd75d5f30 2024-04-21T23:11:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.083+0000 7f89baff5640 1 -- 172.21.15.44:0/1647810829 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f89c800f640 con 0x55bbd75d5f30 2024-04-21T23:11:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.083+0000 7f89e0026740 10 client.25892 fetch_fsmap learned FSMap version 25 2024-04-21T23:11:38.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.083+0000 7f89e0026740 1 -- 172.21.15.44:0/1647810829 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bbd720e0f0 con 0x55bbd75d5f30 2024-04-21T23:11:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.083+0000 7f89baff5640 1 -- 172.21.15.44:0/1647810829 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89c800faa0 con 0x55bbd75d5f30 2024-04-21T23:11:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.083+0000 7f89baff5640 1 -- 172.21.15.44:0/1647810829 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f89c8021070 con 0x55bbd75d5f30 2024-04-21T23:11:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.083+0000 7f89e0026740 10 client.25892 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:11:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.083+0000 7f89e0026740 10 client.25892 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.083+0000 7f89e0026740 20 client.25892 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.083+0000 7f89e0026740 1 --2- 172.21.15.44:0/1647810829 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55bbd75ef660 0x55bbd75f1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.083+0000 7f89e0026740 4 client.25892 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:11:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.083+0000 7f89e0026740 1 -- 172.21.15.44:0/1647810829 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bbd7547aa0 con 0x55bbd75ef660 2024-04-21T23:11:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.083+0000 7f89d67fc640 1 --2- 172.21.15.44:0/1647810829 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55bbd75ef660 0x55bbd75f1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.084+0000 7f89d67fc640 1 --2- 172.21.15.44:0/1647810829 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55bbd75ef660 0x55bbd75f1a50 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a031785f9ee85b9d server_cookie=16315f4aae094519 in_seq=0 out_seq=0 2024-04-21T23:11:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.084+0000 7f89baff5640 10 client.25892 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:38.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.085+0000 7f89baff5640 1 -- 172.21.15.44:0/1647810829 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bbd7547aa0 con 0x55bbd75ef660 2024-04-21T23:11:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.085+0000 7f89baff5640 10 client.25892 handle_command_reply: tid=0 2024-04-21T23:11:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.085+0000 7f89e0026740 1 client.25892 shutdown 2024-04-21T23:11:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.085+0000 7f89e0026740 1 -- 172.21.15.44:0/1647810829 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75d5f30 msgr2=0x55bbd75dc470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.085+0000 7f89e0026740 1 --2- 172.21.15.44:0/1647810829 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75d5f30 0x55bbd75dc470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f89c8000c00 tx=0x7f89c8005710 comp rx=0 tx=0).stop 2024-04-21T23:11:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.085+0000 7f89e0026740 1 -- 172.21.15.44:0/1647810829 shutdown_connections 2024-04-21T23:11:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.085+0000 7f89e0026740 1 --2- 172.21.15.44:0/1647810829 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55bbd75ef660 0x55bbd75f1a50 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.085+0000 7f89e0026740 1 --2- 172.21.15.44:0/1647810829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75df430 0x55bbd75dcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.085+0000 7f89baff5640 0 client.25892 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.085+0000 7f89e0026740 1 --2- 172.21.15.44:0/1647810829 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 0x55bbd75dc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.085+0000 7f89e0026740 1 --2- 172.21.15.44:0/1647810829 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75d5f30 0x55bbd75dc470 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.085+0000 7f89e0026740 1 -- 172.21.15.44:0/1647810829 >> 172.21.15.44:0/1647810829 conn(0x55bbd75d0660 msgr2=0x55bbd75d23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.085+0000 7f89e0026740 1 -- 172.21.15.44:0/1647810829 shutdown_connections 2024-04-21T23:11:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.085+0000 7f89e0026740 1 -- 172.21.15.44:0/1647810829 wait complete. 2024-04-21T23:11:38.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.085+0000 7f89e0026740 20 client.25892 trim_cache size 0 max 16384 2024-04-21T23:11:38.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.104+0000 7f89e0026740 1 Processor -- start 2024-04-21T23:11:38.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.104+0000 7f89e0026740 1 -- start start 2024-04-21T23:11:38.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.104+0000 7f89e0026740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75d5f30 0x55bbd75deba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:38.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.104+0000 7f89e0026740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 0x55bbd75f1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:38.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.104+0000 7f89e0026740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75df430 0x55bbd75eb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:38.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.104+0000 7f89e0026740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f89d81aab00 con 0x55bbd75d6850 2024-04-21T23:11:38.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.104+0000 7f89e0026740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f89d81ad380 con 0x55bbd75d5f30 2024-04-21T23:11:38.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.104+0000 7f89e0026740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f89d81ab170 con 0x55bbd75df430 2024-04-21T23:11:38.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.104+0000 7f89d67fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 0x55bbd75f1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:38.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.104+0000 7f89d6ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75d5f30 0x55bbd75deba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:38.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.104+0000 7f89d67fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 0x55bbd75f1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38784/0 (socket says 172.21.15.44:38784) 2024-04-21T23:11:38.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.104+0000 7f89d77fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75df430 0x55bbd75eb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:38.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.104+0000 7f89d67fc640 1 -- 172.21.15.44:0/1732926062 learned_addr learned my addr 172.21.15.44:0/1732926062 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:38.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.104+0000 7f89d67fc640 1 -- 172.21.15.44:0/1732926062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75df430 msgr2=0x55bbd75eb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:38.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.104+0000 7f89d67fc640 1 --2- 172.21.15.44:0/1732926062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75df430 0x55bbd75eb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.104+0000 7f89d67fc640 1 -- 172.21.15.44:0/1732926062 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75d5f30 msgr2=0x55bbd75deba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:38.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.104+0000 7f89d67fc640 1 --2- 172.21.15.44:0/1732926062 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75d5f30 0x55bbd75deba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.105+0000 7f89d67fc640 1 -- 172.21.15.44:0/1732926062 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbd75d7200 con 0x55bbd75d6850 2024-04-21T23:11:38.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.105+0000 7f89d67fc640 1 --2- 172.21.15.44:0/1732926062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 0x55bbd75f1a50 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f89cc0b9c50 tx=0x7f89cc0902f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7912541be0586887 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:38.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.105+0000 7f89bcff9640 1 -- 172.21.15.44:0/1732926062 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89cc04b040 con 0x55bbd75d6850 2024-04-21T23:11:38.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.105+0000 7f89bcff9640 1 -- 172.21.15.44:0/1732926062 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89cc004c80 con 0x55bbd75d6850 2024-04-21T23:11:38.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.105+0000 7f89bcff9640 1 -- 172.21.15.44:0/1732926062 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89cc0bb580 con 0x55bbd75d6850 2024-04-21T23:11:38.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.105+0000 7f89e0026740 1 -- 172.21.15.44:0/1732926062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 msgr2=0x55bbd75f1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:38.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89e0026740 1 --2- 172.21.15.44:0/1732926062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 0x55bbd75f1a50 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f89cc0b9c50 tx=0x7f89cc0902f0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89e0026740 1 -- 172.21.15.44:0/1732926062 shutdown_connections 2024-04-21T23:11:38.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89e0026740 1 --2- 172.21.15.44:0/1732926062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75df430 0x55bbd75eb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89e0026740 1 --2- 172.21.15.44:0/1732926062 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d6850 0x55bbd75f1a50 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89e0026740 1 --2- 172.21.15.44:0/1732926062 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75d5f30 0x55bbd75deba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89e0026740 1 -- 172.21.15.44:0/1732926062 >> 172.21.15.44:0/1732926062 conn(0x55bbd75d0660 msgr2=0x55bbd75dc430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89e0026740 1 -- 172.21.15.44:0/1732926062 shutdown_connections 2024-04-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89e0026740 1 -- 172.21.15.44:0/1732926062 wait complete. 2024-04-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89e0026740 1 -- start start 2024-04-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89e0026740 1 Processor -- start 2024-04-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89e0026740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d5f30 0x55bbd75e75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89e0026740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75d6850 0x55bbd75e7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89e0026740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75df430 0x55bbd75d8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89e0026740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f89d81ab840 con 0x55bbd75d5f30 2024-04-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89e0026740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f89d81accb0 con 0x55bbd75df430 2024-04-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89e0026740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f89d81abf10 con 0x55bbd75d6850 2024-04-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89e0026740 10 client.0 _ll_register_callbacks cb 0x7f89c801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89d6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d5f30 0x55bbd75e75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89d6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d5f30 0x55bbd75e75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38796/0 (socket says 172.21.15.44:38796) 2024-04-21T23:11:38.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89d6ffd640 1 -- 172.21.15.44:0/199689717 learned_addr learned my addr 172.21.15.44:0/199689717 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:38.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89d77fe640 1 --2- 172.21.15.44:0/199689717 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75df430 0x55bbd75d8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:38.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.106+0000 7f89d67fc640 1 --2- 172.21.15.44:0/199689717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75d6850 0x55bbd75e7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:38.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.107+0000 7f89d6ffd640 1 -- 172.21.15.44:0/199689717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75d6850 msgr2=0x55bbd75e7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:38.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.107+0000 7f89d6ffd640 1 --2- 172.21.15.44:0/199689717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75d6850 0x55bbd75e7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.107+0000 7f89d6ffd640 1 -- 172.21.15.44:0/199689717 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75df430 msgr2=0x55bbd75d8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:38.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.107+0000 7f89d6ffd640 1 --2- 172.21.15.44:0/199689717 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75df430 0x55bbd75d8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.107+0000 7f89d6ffd640 1 -- 172.21.15.44:0/199689717 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbd75d8dc0 con 0x55bbd75d5f30 2024-04-21T23:11:38.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.107+0000 7f89d67fc640 1 --2- 172.21.15.44:0/199689717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75d6850 0x55bbd75e7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:11:38.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.107+0000 7f89d6ffd640 1 --2- 172.21.15.44:0/199689717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d5f30 0x55bbd75e75f0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f89c8001eb0 tx=0x7f89c8011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=680111b31c5f69ab server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:38.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.107+0000 7f89baff5640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:11:38.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.107+0000 7f89baff5640 1 -- 172.21.15.44:0/199689717 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c801e040 con 0x55bbd75d5f30 2024-04-21T23:11:38.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.107+0000 7f89baff5640 1 -- 172.21.15.44:0/199689717 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f89c800f9f0 con 0x55bbd75d5f30 2024-04-21T23:11:38.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.107+0000 7f89e0026740 1 -- 172.21.15.44:0/199689717 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89cc004c80 con 0x55bbd75d5f30 2024-04-21T23:11:38.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.107+0000 7f89baff5640 1 -- 172.21.15.44:0/199689717 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f89c803b050 con 0x55bbd75d5f30 2024-04-21T23:11:38.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.107+0000 7f89e0026740 10 client.16329 fetch_fsmap learned FSMap version 25 2024-04-21T23:11:38.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.107+0000 7f89e0026740 1 -- 172.21.15.44:0/199689717 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bbd76553e0 con 0x55bbd75d5f30 2024-04-21T23:11:38.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.107+0000 7f89baff5640 1 -- 172.21.15.44:0/199689717 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89cc004c80 con 0x55bbd75d5f30 2024-04-21T23:11:38.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.108+0000 7f89baff5640 1 -- 172.21.15.44:0/199689717 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f89c8021040 con 0x55bbd75d5f30 2024-04-21T23:11:38.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.108+0000 7f89e0026740 10 client.16329 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:11:38.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.108+0000 7f89e0026740 10 client.16329 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:38.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.108+0000 7f89e0026740 20 client.16329 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:38.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.108+0000 7f89e0026740 1 --2- 172.21.15.44:0/199689717 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55bbd75ef660 0x55bbd7657ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:38.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.108+0000 7f89e0026740 4 client.16329 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:11:38.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.108+0000 7f89e0026740 1 -- 172.21.15.44:0/199689717 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bbd7541df0 con 0x55bbd75ef660 2024-04-21T23:11:38.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.108+0000 7f89d67fc640 1 --2- 172.21.15.44:0/199689717 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55bbd75ef660 0x55bbd7657ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:38.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.108+0000 7f89baff5640 10 client.16329 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:38.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.108+0000 7f89d67fc640 1 --2- 172.21.15.44:0/199689717 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55bbd75ef660 0x55bbd7657ae0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b2d2f067a2048e8 server_cookie=bf090fd7dbbced95 in_seq=0 out_seq=0 2024-04-21T23:11:38.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.108+0000 7f89baff5640 1 -- 172.21.15.44:0/199689717 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55bbd7541df0 con 0x55bbd75ef660 2024-04-21T23:11:38.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.108+0000 7f89baff5640 10 client.16329 handle_command_reply: tid=0 2024-04-21T23:11:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.108+0000 7f89e0026740 1 client.16329 shutdown 2024-04-21T23:11:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.108+0000 7f89e0026740 1 -- 172.21.15.44:0/199689717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d5f30 msgr2=0x55bbd75e75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.108+0000 7f89e0026740 1 --2- 172.21.15.44:0/199689717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d5f30 0x55bbd75e75f0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f89c8001eb0 tx=0x7f89c8011440 comp rx=0 tx=0).stop 2024-04-21T23:11:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.108+0000 7f89e0026740 1 -- 172.21.15.44:0/199689717 shutdown_connections 2024-04-21T23:11:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.108+0000 7f89e0026740 1 --2- 172.21.15.44:0/199689717 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55bbd75ef660 0x55bbd7657ae0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.108+0000 7f89e0026740 1 --2- 172.21.15.44:0/199689717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bbd75d6850 0x55bbd75e7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.109+0000 7f89e0026740 1 --2- 172.21.15.44:0/199689717 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55bbd75df430 0x55bbd75d8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.109+0000 7f89baff5640 0 client.16329 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.109+0000 7f89e0026740 1 --2- 172.21.15.44:0/199689717 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55bbd75d5f30 0x55bbd75e75f0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.109+0000 7f89e0026740 1 -- 172.21.15.44:0/199689717 >> 172.21.15.44:0/199689717 conn(0x55bbd75d0660 msgr2=0x55bbd75dad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.109+0000 7f89e0026740 1 -- 172.21.15.44:0/199689717 shutdown_connections 2024-04-21T23:11:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.109+0000 7f89e0026740 1 -- 172.21.15.44:0/199689717 wait complete. 2024-04-21T23:11:38.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.109+0000 7f89e0026740 20 client.16329 trim_cache size 0 max 16384 2024-04-21T23:11:38.117 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:11:38.117 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:11:38.117 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-21T23:11:38.117 INFO:teuthology.orchestra.run.smithi044.stdout: "c68fc7a5-41e6-47fa-97b1-1ac823093063": { 2024-04-21T23:11:38.117 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-21T23:11:38.117 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "c68fc7a5-41e6-47fa-97b1-1ac823093063", 2024-04-21T23:11:38.117 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-04-21T23:11:38.117 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:11:38.117 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:11:38.117 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:11:38.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.109+0000 7f89dd35b640 1 -- 172.21.15.44:0/409108483 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f89c00a2f20 msgr2=0x7f89c00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:38.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.109+0000 7f89dd35b640 1 --2- 172.21.15.44:0/409108483 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f89c00a2f20 0x7f89c00a53e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f89d00049d0 tx=0x7f89d0004ab0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.109+0000 7f89dd35b640 1 -- 172.21.15.44:0/409108483 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89d81020d0 msgr2=0x7f89d81985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:38.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.109+0000 7f89dd35b640 1 --2- 172.21.15.44:0/409108483 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89d81020d0 0x7f89d81985d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f89cc009230 tx=0x7f89cc02cb30 comp rx=0 tx=0).stop 2024-04-21T23:11:38.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.109+0000 7f89dd35b640 1 -- 172.21.15.44:0/409108483 shutdown_connections 2024-04-21T23:11:38.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.109+0000 7f89dd35b640 1 --2- 172.21.15.44:0/409108483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89d8107720 0x7f89d819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.109+0000 7f89dd35b640 1 --2- 172.21.15.44:0/409108483 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f89c00a2f20 0x7f89c00a53e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.109+0000 7f89dd35b640 1 --2- 172.21.15.44:0/409108483 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f89d81020d0 0x7f89d81985d0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.109+0000 7f89dd35b640 1 --2- 172.21.15.44:0/409108483 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89d8101710 0x7f89d8198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:38.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.109+0000 7f89dd35b640 1 -- 172.21.15.44:0/409108483 >> 172.21.15.44:0/409108483 conn(0x7f89d80fd200 msgr2=0x7f89d80fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:38.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.109+0000 7f89dd35b640 1 -- 172.21.15.44:0/409108483 shutdown_connections 2024-04-21T23:11:38.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:11:38.109+0000 7f89dd35b640 1 -- 172.21.15.44:0/409108483 wait complete. 2024-04-21T23:11:38.120 INFO:tasks.cephfs.filesystem:scrub status for tag:c68fc7a5-41e6-47fa-97b1-1ac823093063 - {'path': '/', 'tag': 'c68fc7a5-41e6-47fa-97b1-1ac823093063', 'options': 'recursive,force'} 2024-04-21T23:11:38.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:38 smithi044 ceph-mon[27934]: pgmap v88: 129 pgs: 129 active+clean; 49 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 506 op/s 2024-04-21T23:11:38.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:38 smithi044 ceph-mon[27934]: scrub queued for path: / 2024-04-21T23:11:38.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:38 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:11:38.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:38 smithi044 ceph-mon[27934]: scrub summary: active paths [/] 2024-04-21T23:11:38.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:38 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:11:38.615 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:38 smithi069 ceph-mon[31816]: pgmap v88: 129 pgs: 129 active+clean; 49 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 506 op/s 2024-04-21T23:11:38.615 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:38 smithi069 ceph-mon[31816]: scrub queued for path: / 2024-04-21T23:11:38.615 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:38 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:11:38.615 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:38 smithi069 ceph-mon[31816]: scrub summary: active paths [/] 2024-04-21T23:11:38.615 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:38 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:11:38.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 233815 14.51 MB/sec warmup 88 sec latency 22.543 ms 2024-04-21T23:11:38.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:38 smithi055 ceph-mon[31117]: pgmap v88: 129 pgs: 129 active+clean; 49 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 506 op/s 2024-04-21T23:11:38.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:38 smithi055 ceph-mon[31117]: scrub queued for path: / 2024-04-21T23:11:38.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:38 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:11:38.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:38 smithi055 ceph-mon[31117]: scrub summary: active paths [/] 2024-04-21T23:11:38.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:38 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:11:39.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 235557 14.47 MB/sec warmup 89 sec latency 78.635 ms 2024-04-21T23:11:40.515 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:40 smithi055 ceph-mon[31117]: pgmap v89: 129 pgs: 129 active+clean; 52 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 548 op/s 2024-04-21T23:11:40.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:40 smithi044 ceph-mon[27934]: pgmap v89: 129 pgs: 129 active+clean; 52 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 548 op/s 2024-04-21T23:11:40.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 238290 14.47 MB/sec warmup 90 sec latency 33.192 ms 2024-04-21T23:11:40.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:40 smithi069 ceph-mon[31816]: pgmap v89: 129 pgs: 129 active+clean; 52 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.6 MiB/s wr, 548 op/s 2024-04-21T23:11:41.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 241397 14.51 MB/sec warmup 91 sec latency 19.616 ms 2024-04-21T23:11:42.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:42 smithi044 ceph-mon[27934]: pgmap v90: 129 pgs: 129 active+clean; 52 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 397 op/s 2024-04-21T23:11:42.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:42 smithi044 ceph-mon[27934]: scrub summary: idle 2024-04-21T23:11:42.694 INFO:tasks.workunit.client.0.smithi044.stdout: 1 243737 14.45 MB/sec warmup 92 sec latency 14.736 ms 2024-04-21T23:11:42.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:42 smithi055 ceph-mon[31117]: pgmap v90: 129 pgs: 129 active+clean; 52 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 397 op/s 2024-04-21T23:11:42.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:42 smithi055 ceph-mon[31117]: scrub summary: idle 2024-04-21T23:11:42.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:42 smithi069 ceph-mon[31816]: pgmap v90: 129 pgs: 129 active+clean; 52 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 397 op/s 2024-04-21T23:11:42.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:42 smithi069 ceph-mon[31816]: scrub summary: idle 2024-04-21T23:11:43.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:43 smithi044 ceph-mon[27934]: pgmap v91: 129 pgs: 129 active+clean; 45 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.7 MiB/s wr, 554 op/s 2024-04-21T23:11:43.695 INFO:tasks.workunit.client.0.smithi044.stdout: 1 245971 14.44 MB/sec warmup 93 sec latency 15.051 ms 2024-04-21T23:11:43.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:43 smithi069 ceph-mon[31816]: pgmap v91: 129 pgs: 129 active+clean; 45 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.7 MiB/s wr, 554 op/s 2024-04-21T23:11:43.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:43 smithi055 ceph-mon[31117]: pgmap v91: 129 pgs: 129 active+clean; 45 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.7 MiB/s wr, 554 op/s 2024-04-21T23:11:44.695 INFO:tasks.workunit.client.0.smithi044.stdout: 1 248693 14.46 MB/sec warmup 94 sec latency 112.620 ms 2024-04-21T23:11:45.695 INFO:tasks.workunit.client.0.smithi044.stdout: 1 251538 14.45 MB/sec warmup 95 sec latency 19.121 ms 2024-04-21T23:11:45.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:45 smithi069 ceph-mon[31816]: pgmap v92: 129 pgs: 129 active+clean; 45 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 427 op/s 2024-04-21T23:11:45.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:45 smithi055 ceph-mon[31117]: pgmap v92: 129 pgs: 129 active+clean; 45 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 427 op/s 2024-04-21T23:11:46.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:45 smithi044 ceph-mon[27934]: pgmap v92: 129 pgs: 129 active+clean; 45 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 427 op/s 2024-04-21T23:11:46.695 INFO:tasks.workunit.client.0.smithi044.stdout: 1 254226 14.43 MB/sec warmup 96 sec latency 27.453 ms 2024-04-21T23:11:47.695 INFO:tasks.workunit.client.0.smithi044.stdout: 1 256210 14.41 MB/sec warmup 97 sec latency 68.660 ms 2024-04-21T23:11:47.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:47 smithi069 ceph-mon[31816]: pgmap v93: 129 pgs: 129 active+clean; 47 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 472 op/s 2024-04-21T23:11:47.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:47 smithi055 ceph-mon[31117]: pgmap v93: 129 pgs: 129 active+clean; 47 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 472 op/s 2024-04-21T23:11:48.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:47 smithi044 ceph-mon[27934]: pgmap v93: 129 pgs: 129 active+clean; 47 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 472 op/s 2024-04-21T23:11:48.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 259293 14.46 MB/sec warmup 98 sec latency 22.820 ms 2024-04-21T23:11:49.695 INFO:tasks.workunit.client.0.smithi044.stdout: 1 261436 14.39 MB/sec warmup 99 sec latency 97.547 ms 2024-04-21T23:11:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.831+0000 7f109f913640 1 -- 172.21.15.44:0/1636918633 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1098100620 msgr2=0x7f1098100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:49.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.831+0000 7f109f913640 1 --2- 172.21.15.44:0/1636918633 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1098100620 0x7f1098100aa0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f108800dc90 tx=0x7f108802fbe0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.831+0000 7f109f913640 1 -- 172.21.15.44:0/1636918633 shutdown_connections 2024-04-21T23:11:49.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.831+0000 7f109f913640 1 --2- 172.21.15.44:0/1636918633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1098100fe0 0x7f109810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.831+0000 7f109f913640 1 --2- 172.21.15.44:0/1636918633 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1098100620 0x7f1098100aa0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.831+0000 7f109f913640 1 --2- 172.21.15.44:0/1636918633 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1098106630 0x7f1098106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.831+0000 7f109f913640 1 -- 172.21.15.44:0/1636918633 >> 172.21.15.44:0/1636918633 conn(0x7f1098077760 msgr2=0x7f1098077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:49.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.833+0000 7f109f913640 1 -- 172.21.15.44:0/1636918633 shutdown_connections 2024-04-21T23:11:49.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.833+0000 7f109f913640 1 -- 172.21.15.44:0/1636918633 wait complete. 2024-04-21T23:11:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.834+0000 7f109f913640 1 Processor -- start 2024-04-21T23:11:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.835+0000 7f109f913640 1 -- start start 2024-04-21T23:11:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.836+0000 7f109f913640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1098100620 0x7f109819cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.836+0000 7f109f913640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1098100fe0 0x7f1098195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.836+0000 7f109f913640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1098106630 0x7f1098196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.836+0000 7f109f913640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f109810fa40 con 0x7f1098106630 2024-04-21T23:11:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.836+0000 7f109f913640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f109810f8c0 con 0x7f1098100fe0 2024-04-21T23:11:49.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.836+0000 7f109f913640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f109810fbc0 con 0x7f1098100620 2024-04-21T23:11:49.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.836+0000 7f109ceae640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1098100fe0 0x7f1098195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:49.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.836+0000 7f109ceae640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1098100fe0 0x7f1098195de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:60860/0 (socket says 172.21.15.44:60860) 2024-04-21T23:11:49.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.836+0000 7f109ceae640 1 -- 172.21.15.44:0/3093288542 learned_addr learned my addr 172.21.15.44:0/3093288542 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:49.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.837+0000 7f109deb0640 1 --2- 172.21.15.44:0/3093288542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1098106630 0x7f1098196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:49.836 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.837+0000 7f109d6af640 1 --2- 172.21.15.44:0/3093288542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1098100620 0x7f109819cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:49.837 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.837+0000 7f109ceae640 1 -- 172.21.15.44:0/3093288542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1098100620 msgr2=0x7f109819cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:49.837 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.837+0000 7f109ceae640 1 --2- 172.21.15.44:0/3093288542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1098100620 0x7f109819cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.837 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.837+0000 7f109ceae640 1 -- 172.21.15.44:0/3093288542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1098106630 msgr2=0x7f1098196320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:49.837 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.837+0000 7f109ceae640 1 --2- 172.21.15.44:0/3093288542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1098106630 0x7f1098196320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.837 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.837+0000 7f109ceae640 1 -- 172.21.15.44:0/3093288542 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1098196910 con 0x7f1098100fe0 2024-04-21T23:11:49.837 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.838+0000 7f109ceae640 1 --2- 172.21.15.44:0/3093288542 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1098100fe0 0x7f1098195de0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f10880301f0 tx=0x7f1088038400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.839+0000 7f108e7fc640 1 -- 172.21.15.44:0/3093288542 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1088004020 con 0x7f1098100fe0 2024-04-21T23:11:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.839+0000 7f108e7fc640 1 -- 172.21.15.44:0/3093288542 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10880041c0 con 0x7f1098100fe0 2024-04-21T23:11:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.839+0000 7f108e7fc640 1 -- 172.21.15.44:0/3093288542 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10880389c0 con 0x7f1098100fe0 2024-04-21T23:11:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.839+0000 7f109f913640 1 -- 172.21.15.44:0/3093288542 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1098196ba0 con 0x7f1098100fe0 2024-04-21T23:11:49.838 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.839+0000 7f109f913640 1 -- 172.21.15.44:0/3093288542 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f109819aa10 con 0x7f1098100fe0 2024-04-21T23:11:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.844+0000 7f108e7fc640 1 -- 172.21.15.44:0/3093288542 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1088038b60 con 0x7f1098100fe0 2024-04-21T23:11:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.844+0000 7f10a25dd740 1 Processor -- start 2024-04-21T23:11:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.845+0000 7f10a25dd740 1 -- start start 2024-04-21T23:11:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.845+0000 7f10a25dd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562601e4cb20 0x562601e4cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.845+0000 7f10a25dd740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e4d440 0x562601e55cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.845+0000 7f10a25dd740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e56210 0x562601e58600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.845+0000 7f10a25dd740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562601db0b70 con 0x562601e4d440 2024-04-21T23:11:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.845+0000 7f10a25dd740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x562601d59470 con 0x562601e56210 2024-04-21T23:11:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.845+0000 7f10a25dd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562601dacb30 con 0x562601e4cb20 2024-04-21T23:11:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.845+0000 7f109deb0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562601e4cb20 0x562601e4cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.845+0000 7f109d6af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e4d440 0x562601e55cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.846+0000 7f109d6af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e4d440 0x562601e55cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46922/0 (socket says 172.21.15.44:46922) 2024-04-21T23:11:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.846+0000 7f109d6af640 1 -- 172.21.15.44:0/4127669299 learned_addr learned my addr 172.21.15.44:0/4127669299 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:49.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.846+0000 7f109ceae640 1 --2- 172.21.15.44:0/4127669299 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e56210 0x562601e58600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:49.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.846+0000 7f109d6af640 1 -- 172.21.15.44:0/4127669299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562601e4cb20 msgr2=0x562601e4cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:49.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.846+0000 7f109d6af640 1 --2- 172.21.15.44:0/4127669299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562601e4cb20 0x562601e4cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.846+0000 7f109d6af640 1 -- 172.21.15.44:0/4127669299 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e56210 msgr2=0x562601e58600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:49.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.847+0000 7f109d6af640 1 --2- 172.21.15.44:0/4127669299 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e56210 0x562601e58600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.847+0000 7f109d6af640 1 -- 172.21.15.44:0/4127669299 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562601ac5210 con 0x562601e4d440 2024-04-21T23:11:49.846 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.847+0000 7f109ceae640 1 --2- 172.21.15.44:0/4127669299 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e56210 0x562601e58600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:11:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.847+0000 7f108e7fc640 1 --2- 172.21.15.44:0/3093288542 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f10840a2e80 0x7f10840a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.847+0000 7f109d6af640 1 --2- 172.21.15.44:0/4127669299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e4d440 0x562601e55cd0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f1094009a40 tx=0x7f1094009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea75aecffe7cbac2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.847+0000 7f108e7fc640 1 -- 172.21.15.44:0/3093288542 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f10880f11e0 con 0x7f1098100fe0 2024-04-21T23:11:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.847+0000 7f109deb0640 1 --2- 172.21.15.44:0/3093288542 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f10840a2e80 0x7f10840a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.847+0000 7f10837fe640 1 -- 172.21.15.44:0/4127669299 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094017070 con 0x562601e4d440 2024-04-21T23:11:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.847+0000 7f10837fe640 1 -- 172.21.15.44:0/4127669299 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1094013470 con 0x562601e4d440 2024-04-21T23:11:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.847+0000 7f10837fe640 1 -- 172.21.15.44:0/4127669299 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094012630 con 0x562601e4d440 2024-04-21T23:11:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.848+0000 7f109deb0640 1 --2- 172.21.15.44:0/3093288542 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f10840a2e80 0x7f10840a5340 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f109000b4a0 tx=0x7f109000d040 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.848+0000 7f10a25dd740 1 -- 172.21.15.44:0/4127669299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e4d440 msgr2=0x562601e55cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.848+0000 7f10a25dd740 1 --2- 172.21.15.44:0/4127669299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e4d440 0x562601e55cd0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f1094009a40 tx=0x7f1094009f00 comp rx=0 tx=0).stop 2024-04-21T23:11:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.848+0000 7f10a25dd740 1 -- 172.21.15.44:0/4127669299 shutdown_connections 2024-04-21T23:11:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.848+0000 7f10a25dd740 1 --2- 172.21.15.44:0/4127669299 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e56210 0x562601e58600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.848+0000 7f10a25dd740 1 --2- 172.21.15.44:0/4127669299 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e4d440 0x562601e55cd0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.848+0000 7f10a25dd740 1 --2- 172.21.15.44:0/4127669299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562601e4cb20 0x562601e4cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.848+0000 7f10a25dd740 1 -- 172.21.15.44:0/4127669299 >> 172.21.15.44:0/4127669299 conn(0x562601e47440 msgr2=0x562601e47820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.848+0000 7f10a25dd740 1 -- 172.21.15.44:0/4127669299 shutdown_connections 2024-04-21T23:11:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.848+0000 7f10a25dd740 1 -- 172.21.15.44:0/4127669299 wait complete. 2024-04-21T23:11:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.848+0000 7f10a25dd740 1 -- start start 2024-04-21T23:11:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.849+0000 7f10a25dd740 1 Processor -- start 2024-04-21T23:11:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.849+0000 7f10a25dd740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e4cb20 0x562601e53240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.849+0000 7f10a25dd740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e4d440 0x562601e53780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.849+0000 7f10a25dd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562601e56210 0x562601e53cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.849+0000 7f10a25dd740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562601e58e40 con 0x562601e4cb20 2024-04-21T23:11:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.849+0000 7f10a25dd740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x562601d6ca10 con 0x562601e4d440 2024-04-21T23:11:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.849+0000 7f10a25dd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562601e58fc0 con 0x562601e56210 2024-04-21T23:11:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.849+0000 7f10a25dd740 10 client.0 _ll_register_callbacks cb 0x562601da7b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.849+0000 7f109d6af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e4cb20 0x562601e53240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.849+0000 7f109ceae640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e4d440 0x562601e53780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.849+0000 7f109d6af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e4cb20 0x562601e53240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46932/0 (socket says 172.21.15.44:46932) 2024-04-21T23:11:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.849+0000 7f109d6af640 1 -- 172.21.15.44:0/539796032 learned_addr learned my addr 172.21.15.44:0/539796032 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.849+0000 7f109deb0640 1 --2- 172.21.15.44:0/539796032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562601e56210 0x562601e53cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.849+0000 7f109d6af640 1 -- 172.21.15.44:0/539796032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562601e56210 msgr2=0x562601e53cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.849+0000 7f109d6af640 1 --2- 172.21.15.44:0/539796032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562601e56210 0x562601e53cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.849+0000 7f109d6af640 1 -- 172.21.15.44:0/539796032 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e4d440 msgr2=0x562601e53780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.849+0000 7f109d6af640 1 --2- 172.21.15.44:0/539796032 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e4d440 0x562601e53780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.849+0000 7f109d6af640 1 -- 172.21.15.44:0/539796032 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562601b91620 con 0x562601e4cb20 2024-04-21T23:11:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.850+0000 7f109d6af640 1 --2- 172.21.15.44:0/539796032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e4cb20 0x562601e53240 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1094013f60 tx=0x7f109400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410f13a38569a118 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.850+0000 7f1080ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:11:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.850+0000 7f1080ff9640 1 -- 172.21.15.44:0/539796032 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094017050 con 0x562601e4cb20 2024-04-21T23:11:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.850+0000 7f10a25dd740 1 -- 172.21.15.44:0/539796032 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1094013470 con 0x562601e4cb20 2024-04-21T23:11:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.850+0000 7f1080ff9640 1 -- 172.21.15.44:0/539796032 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f109401c450 con 0x562601e4cb20 2024-04-21T23:11:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.850+0000 7f1080ff9640 1 -- 172.21.15.44:0/539796032 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f109402f050 con 0x562601e4cb20 2024-04-21T23:11:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.850+0000 7f10a25dd740 10 client.16341 fetch_fsmap learned FSMap version 25 2024-04-21T23:11:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.850+0000 7f10a25dd740 1 -- 172.21.15.44:0/539796032 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562601e650a0 con 0x562601e4cb20 2024-04-21T23:11:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.850+0000 7f1080ff9640 1 -- 172.21.15.44:0/539796032 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1094013470 con 0x562601e4cb20 2024-04-21T23:11:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.850+0000 7f1080ff9640 1 -- 172.21.15.44:0/539796032 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f10940266e0 con 0x562601e4cb20 2024-04-21T23:11:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.850+0000 7f10a25dd740 10 client.16341 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:11:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.850+0000 7f10a25dd740 10 client.16341 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.850+0000 7f10a25dd740 20 client.16341 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.850+0000 7f10a25dd740 1 --2- 172.21.15.44:0/539796032 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562601e664b0 0x562601e688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.850+0000 7f10a25dd740 4 client.16341 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:11:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.851+0000 7f10a25dd740 1 -- 172.21.15.44:0/539796032 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562601dbe0b0 con 0x562601e664b0 2024-04-21T23:11:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.851+0000 7f109ceae640 1 --2- 172.21.15.44:0/539796032 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562601e664b0 0x562601e688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.851+0000 7f1080ff9640 10 client.16341 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.851+0000 7f109ceae640 1 --2- 172.21.15.44:0/539796032 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562601e664b0 0x562601e688a0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26275f73ad3a5c52 server_cookie=2bf32e4da4fdbb48 in_seq=0 out_seq=0 2024-04-21T23:11:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.852+0000 7f1080ff9640 1 -- 172.21.15.44:0/539796032 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562601dbe0b0 con 0x562601e664b0 2024-04-21T23:11:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.852+0000 7f1080ff9640 10 client.16341 handle_command_reply: tid=0 2024-04-21T23:11:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.852+0000 7f10a25dd740 1 client.16341 shutdown 2024-04-21T23:11:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.852+0000 7f10a25dd740 1 -- 172.21.15.44:0/539796032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e4cb20 msgr2=0x562601e53240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.852+0000 7f10a25dd740 1 --2- 172.21.15.44:0/539796032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e4cb20 0x562601e53240 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1094013f60 tx=0x7f109400ee60 comp rx=0 tx=0).stop 2024-04-21T23:11:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.852+0000 7f10a25dd740 1 -- 172.21.15.44:0/539796032 shutdown_connections 2024-04-21T23:11:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.852+0000 7f10a25dd740 1 --2- 172.21.15.44:0/539796032 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562601e664b0 0x562601e688a0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.852+0000 7f10a25dd740 1 --2- 172.21.15.44:0/539796032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562601e56210 0x562601e53cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.852+0000 7f10a25dd740 1 --2- 172.21.15.44:0/539796032 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e4d440 0x562601e53780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.852+0000 7f10a25dd740 1 --2- 172.21.15.44:0/539796032 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e4cb20 0x562601e53240 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.852+0000 7f10a25dd740 1 -- 172.21.15.44:0/539796032 >> 172.21.15.44:0/539796032 conn(0x562601e47440 msgr2=0x562601e49190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.852+0000 7f1080ff9640 0 client.16341 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.852+0000 7f10a25dd740 1 -- 172.21.15.44:0/539796032 shutdown_connections 2024-04-21T23:11:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.852+0000 7f10a25dd740 1 -- 172.21.15.44:0/539796032 wait complete. 2024-04-21T23:11:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.852+0000 7f10a25dd740 20 client.16341 trim_cache size 0 max 16384 2024-04-21T23:11:49.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.867+0000 7f10a25dd740 1 Processor -- start 2024-04-21T23:11:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.867+0000 7f10a25dd740 1 -- start start 2024-04-21T23:11:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.867+0000 7f10a25dd740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e47440 0x562601e55970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.868+0000 7f10a25dd740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e4cb20 0x562601e61d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.868+0000 7f10a25dd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562601e4d440 0x562601e4d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.868+0000 7f10a25dd740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f10981bb970 con 0x562601e47440 2024-04-21T23:11:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.868+0000 7f10a25dd740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f10981be1f0 con 0x562601e4cb20 2024-04-21T23:11:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.868+0000 7f10a25dd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f10981bbfe0 con 0x562601e4d440 2024-04-21T23:11:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.868+0000 7f109d6af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e47440 0x562601e55970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.868+0000 7f109d6af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e47440 0x562601e55970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46942/0 (socket says 172.21.15.44:46942) 2024-04-21T23:11:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.868+0000 7f109d6af640 1 -- 172.21.15.44:0/1838442029 learned_addr learned my addr 172.21.15.44:0/1838442029 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.868+0000 7f109ceae640 1 --2- 172.21.15.44:0/1838442029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e4cb20 0x562601e61d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.868+0000 7f109deb0640 1 --2- 172.21.15.44:0/1838442029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562601e4d440 0x562601e4d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.868+0000 7f109d6af640 1 -- 172.21.15.44:0/1838442029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562601e4d440 msgr2=0x562601e4d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.868+0000 7f109d6af640 1 --2- 172.21.15.44:0/1838442029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562601e4d440 0x562601e4d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.868+0000 7f109d6af640 1 -- 172.21.15.44:0/1838442029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e4cb20 msgr2=0x562601e61d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.868+0000 7f109d6af640 1 --2- 172.21.15.44:0/1838442029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e4cb20 0x562601e61d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.868+0000 7f109d6af640 1 -- 172.21.15.44:0/1838442029 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562601e51790 con 0x562601e47440 2024-04-21T23:11:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.868+0000 7f109d6af640 1 --2- 172.21.15.44:0/1838442029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e47440 0x562601e55970 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f109400ee30 tx=0x7f10940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cdc5af4ba3e197d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.868+0000 7f1082ffd640 1 -- 172.21.15.44:0/1838442029 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094017040 con 0x562601e47440 2024-04-21T23:11:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.868+0000 7f1082ffd640 1 -- 172.21.15.44:0/1838442029 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1094013c90 con 0x562601e47440 2024-04-21T23:11:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.868+0000 7f1082ffd640 1 -- 172.21.15.44:0/1838442029 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109401c9a0 con 0x562601e47440 2024-04-21T23:11:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.869+0000 7f10a25dd740 1 -- 172.21.15.44:0/1838442029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e47440 msgr2=0x562601e55970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.869+0000 7f10a25dd740 1 --2- 172.21.15.44:0/1838442029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e47440 0x562601e55970 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f109400ee30 tx=0x7f10940046c0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.869+0000 7f10a25dd740 1 -- 172.21.15.44:0/1838442029 shutdown_connections 2024-04-21T23:11:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.869+0000 7f10a25dd740 1 --2- 172.21.15.44:0/1838442029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562601e4d440 0x562601e4d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.869+0000 7f10a25dd740 1 --2- 172.21.15.44:0/1838442029 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e4cb20 0x562601e61d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.869+0000 7f10a25dd740 1 --2- 172.21.15.44:0/1838442029 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e47440 0x562601e55970 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.869+0000 7f10a25dd740 1 -- 172.21.15.44:0/1838442029 >> 172.21.15.44:0/1838442029 conn(0x562601e4ec10 msgr2=0x562601e52270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.869+0000 7f10a25dd740 1 -- 172.21.15.44:0/1838442029 shutdown_connections 2024-04-21T23:11:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.869+0000 7f10a25dd740 1 -- 172.21.15.44:0/1838442029 wait complete. 2024-04-21T23:11:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.869+0000 7f10a25dd740 1 -- start start 2024-04-21T23:11:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.869+0000 7f10a25dd740 1 Processor -- start 2024-04-21T23:11:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.870+0000 7f10a25dd740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e47440 0x562601e5e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.870+0000 7f10a25dd740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e4cb20 0x562601e5e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.870+0000 7f10a25dd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562601e4d440 0x562601e597a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.870+0000 7f10a25dd740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f10981bc6b0 con 0x562601e47440 2024-04-21T23:11:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.870+0000 7f10a25dd740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f10981bdb20 con 0x562601e4cb20 2024-04-21T23:11:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.870+0000 7f10a25dd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f10981bcd80 con 0x562601e4d440 2024-04-21T23:11:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.870+0000 7f109d6af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e47440 0x562601e5e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.870+0000 7f109d6af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e47440 0x562601e5e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:46952/0 (socket says 172.21.15.44:46952) 2024-04-21T23:11:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.870+0000 7f109d6af640 1 -- 172.21.15.44:0/2369495662 learned_addr learned my addr 172.21.15.44:0/2369495662 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:11:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.870+0000 7f109ceae640 1 --2- 172.21.15.44:0/2369495662 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e4cb20 0x562601e5e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.870+0000 7f10a25dd740 10 client.0 _ll_register_callbacks cb 0x7f10900099b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:11:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.870+0000 7f109deb0640 1 --2- 172.21.15.44:0/2369495662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562601e4d440 0x562601e597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.870+0000 7f109d6af640 1 -- 172.21.15.44:0/2369495662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562601e4d440 msgr2=0x562601e597a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.870+0000 7f109d6af640 1 --2- 172.21.15.44:0/2369495662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562601e4d440 0x562601e597a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.870+0000 7f109d6af640 1 -- 172.21.15.44:0/2369495662 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e4cb20 msgr2=0x562601e5e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:11:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.870+0000 7f109d6af640 1 --2- 172.21.15.44:0/2369495662 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e4cb20 0x562601e5e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.870+0000 7f109d6af640 1 -- 172.21.15.44:0/2369495662 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562601e59fa0 con 0x562601e47440 2024-04-21T23:11:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.870+0000 7f109d6af640 1 --2- 172.21.15.44:0/2369495662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e47440 0x562601e5e200 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f109400e660 tx=0x7f1094013c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=372fb9b1dfd27f5c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:11:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.870+0000 7f1080ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:11:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.871+0000 7f1080ff9640 1 -- 172.21.15.44:0/2369495662 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094040060 con 0x562601e47440 2024-04-21T23:11:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.871+0000 7f10a25dd740 1 -- 172.21.15.44:0/2369495662 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1094013c90 con 0x562601e47440 2024-04-21T23:11:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.871+0000 7f1080ff9640 1 -- 172.21.15.44:0/2369495662 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f1094005ba0 con 0x562601e47440 2024-04-21T23:11:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.871+0000 7f1080ff9640 1 -- 172.21.15.44:0/2369495662 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1094017070 con 0x562601e47440 2024-04-21T23:11:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.871+0000 7f10a25dd740 10 client.16347 fetch_fsmap learned FSMap version 25 2024-04-21T23:11:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.871+0000 7f10a25dd740 1 -- 172.21.15.44:0/2369495662 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562601ecc360 con 0x562601e47440 2024-04-21T23:11:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.871+0000 7f1080ff9640 1 -- 172.21.15.44:0/2369495662 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1094013c90 con 0x562601e47440 2024-04-21T23:11:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.872+0000 7f1080ff9640 1 -- 172.21.15.44:0/2369495662 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f109401c6a0 con 0x562601e47440 2024-04-21T23:11:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.872+0000 7f10a25dd740 10 client.16347 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:11:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.872+0000 7f10a25dd740 10 client.16347 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:11:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.872+0000 7f10a25dd740 20 client.16347 populate_metadata read hostname 'smithi044' 2024-04-21T23:11:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.872+0000 7f10a25dd740 1 --2- 172.21.15.44:0/2369495662 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562601e56210 0x562601eceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:11:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.872+0000 7f10a25dd740 4 client.16347 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:11:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.872+0000 7f10a25dd740 1 -- 172.21.15.44:0/2369495662 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562601db8400 con 0x562601e56210 2024-04-21T23:11:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.872+0000 7f109ceae640 1 --2- 172.21.15.44:0/2369495662 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562601e56210 0x562601eceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:11:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.872+0000 7f1080ff9640 10 client.16347 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:49.872+0000 7f109ceae640 1 --2- 172.21.15.44:0/2369495662 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562601e56210 0x562601eceaa0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f54cff0d4eecd3b server_cookie=8f91f246af2c7ab0 in_seq=0 out_seq=0 2024-04-21T23:11:49.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:49 smithi069 ceph-mon[31816]: pgmap v94: 129 pgs: 129 active+clean; 50 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.2 MiB/s wr, 473 op/s 2024-04-21T23:11:49.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:49 smithi055 ceph-mon[31117]: pgmap v94: 129 pgs: 129 active+clean; 50 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.2 MiB/s wr, 473 op/s 2024-04-21T23:11:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.119+0000 7f1080ff9640 1 -- 172.21.15.44:0/2369495662 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562601db8400 con 0x562601e56210 2024-04-21T23:11:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.119+0000 7f1080ff9640 10 client.16347 handle_command_reply: tid=0 2024-04-21T23:11:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.119+0000 7f10a25dd740 1 client.16347 shutdown 2024-04-21T23:11:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.119+0000 7f10a25dd740 1 -- 172.21.15.44:0/2369495662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e47440 msgr2=0x562601e5e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.119+0000 7f10a25dd740 1 --2- 172.21.15.44:0/2369495662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e47440 0x562601e5e200 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f109400e660 tx=0x7f1094013c20 comp rx=0 tx=0).stop 2024-04-21T23:11:50.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f10a25dd740 1 -- 172.21.15.44:0/2369495662 shutdown_connections 2024-04-21T23:11:50.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f10a25dd740 1 --2- 172.21.15.44:0/2369495662 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562601e56210 0x562601eceaa0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:50.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f10a25dd740 1 --2- 172.21.15.44:0/2369495662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562601e4d440 0x562601e597a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:50.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f10a25dd740 1 --2- 172.21.15.44:0/2369495662 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562601e4cb20 0x562601e5e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:50.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f10a25dd740 1 --2- 172.21.15.44:0/2369495662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562601e47440 0x562601e5e200 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:50.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f1080ff9640 0 client.16347 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:11:50.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f10a25dd740 1 -- 172.21.15.44:0/2369495662 >> 172.21.15.44:0/2369495662 conn(0x562601e4ec10 msgr2=0x562601e519e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:50.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f10a25dd740 1 -- 172.21.15.44:0/2369495662 shutdown_connections 2024-04-21T23:11:50.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f10a25dd740 1 -- 172.21.15.44:0/2369495662 wait complete. 2024-04-21T23:11:50.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f10a25dd740 20 client.16347 trim_cache size 0 max 16384 2024-04-21T23:11:50.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:11:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:11:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:11:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:11:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f109f913640 1 -- 172.21.15.44:0/3093288542 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f10840a2e80 msgr2=0x7f10840a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f109f913640 1 --2- 172.21.15.44:0/3093288542 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f10840a2e80 0x7f10840a5340 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f109000b4a0 tx=0x7f109000d040 comp rx=0 tx=0).stop 2024-04-21T23:11:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f109f913640 1 -- 172.21.15.44:0/3093288542 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1098100fe0 msgr2=0x7f1098195de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:11:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f109f913640 1 --2- 172.21.15.44:0/3093288542 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1098100fe0 0x7f1098195de0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f10880301f0 tx=0x7f1088038400 comp rx=0 tx=0).stop 2024-04-21T23:11:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f109f913640 1 -- 172.21.15.44:0/3093288542 shutdown_connections 2024-04-21T23:11:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f109f913640 1 --2- 172.21.15.44:0/3093288542 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f10840a2e80 0x7f10840a5340 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f109f913640 1 --2- 172.21.15.44:0/3093288542 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1098106630 0x7f1098196320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f109f913640 1 --2- 172.21.15.44:0/3093288542 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1098100fe0 0x7f1098195de0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f109f913640 1 --2- 172.21.15.44:0/3093288542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1098100620 0x7f109819cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:11:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f109f913640 1 -- 172.21.15.44:0/3093288542 >> 172.21.15.44:0/3093288542 conn(0x7f1098077760 msgr2=0x7f1098103230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:11:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f109f913640 1 -- 172.21.15.44:0/3093288542 shutdown_connections 2024-04-21T23:11:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:11:50.120+0000 7f109f913640 1 -- 172.21.15.44:0/3093288542 wait complete. 2024-04-21T23:11:50.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:49 smithi044 ceph-mon[27934]: pgmap v94: 129 pgs: 129 active+clean; 50 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.2 MiB/s wr, 473 op/s 2024-04-21T23:11:50.695 INFO:tasks.workunit.client.0.smithi044.stdout: 1 263048 14.36 MB/sec warmup 100 sec latency 93.410 ms 2024-04-21T23:11:51.695 INFO:tasks.workunit.client.0.smithi044.stdout: 1 266146 14.39 MB/sec warmup 101 sec latency 15.028 ms 2024-04-21T23:11:51.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:51 smithi055 ceph-mon[31117]: pgmap v95: 129 pgs: 129 active+clean; 50 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 280 op/s 2024-04-21T23:11:51.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:51 smithi069 ceph-mon[31816]: pgmap v95: 129 pgs: 129 active+clean; 50 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 280 op/s 2024-04-21T23:11:52.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:51 smithi044 ceph-mon[27934]: pgmap v95: 129 pgs: 129 active+clean; 50 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 280 op/s 2024-04-21T23:11:52.695 INFO:tasks.workunit.client.0.smithi044.stdout: 1 268016 14.32 MB/sec warmup 102 sec latency 60.015 ms 2024-04-21T23:11:53.695 INFO:tasks.workunit.client.0.smithi044.stdout: 1 270547 14.32 MB/sec warmup 103 sec latency 16.516 ms 2024-04-21T23:11:53.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:53 smithi055 ceph-mon[31117]: pgmap v96: 129 pgs: 129 active+clean; 43 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.9 MiB/s wr, 557 op/s 2024-04-21T23:11:53.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:53 smithi069 ceph-mon[31816]: pgmap v96: 129 pgs: 129 active+clean; 43 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.9 MiB/s wr, 557 op/s 2024-04-21T23:11:54.184 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:53 smithi044 ceph-mon[27934]: pgmap v96: 129 pgs: 129 active+clean; 43 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.9 MiB/s wr, 557 op/s 2024-04-21T23:11:54.695 INFO:tasks.workunit.client.0.smithi044.stdout: 1 273452 14.36 MB/sec warmup 104 sec latency 104.352 ms 2024-04-21T23:11:55.695 INFO:tasks.workunit.client.0.smithi044.stdout: 1 276123 14.34 MB/sec warmup 105 sec latency 27.113 ms 2024-04-21T23:11:55.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:55 smithi055 ceph-mon[31117]: pgmap v97: 129 pgs: 129 active+clean; 43 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 400 op/s 2024-04-21T23:11:55.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:55 smithi069 ceph-mon[31816]: pgmap v97: 129 pgs: 129 active+clean; 43 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 400 op/s 2024-04-21T23:11:56.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:55 smithi044 ceph-mon[27934]: pgmap v97: 129 pgs: 129 active+clean; 43 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 400 op/s 2024-04-21T23:11:56.695 INFO:tasks.workunit.client.0.smithi044.stdout: 1 278342 14.32 MB/sec warmup 106 sec latency 73.414 ms 2024-04-21T23:11:57.695 INFO:tasks.workunit.client.0.smithi044.stdout: 1 281034 14.32 MB/sec warmup 107 sec latency 30.355 ms 2024-04-21T23:11:57.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:57 smithi069 ceph-mon[31816]: pgmap v98: 129 pgs: 129 active+clean; 47 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.1 MiB/s wr, 488 op/s 2024-04-21T23:11:57.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:57 smithi055 ceph-mon[31117]: pgmap v98: 129 pgs: 129 active+clean; 47 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.1 MiB/s wr, 488 op/s 2024-04-21T23:11:58.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:57 smithi044 ceph-mon[27934]: pgmap v98: 129 pgs: 129 active+clean; 47 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.1 MiB/s wr, 488 op/s 2024-04-21T23:11:58.695 INFO:tasks.workunit.client.0.smithi044.stdout: 1 284155 14.36 MB/sec warmup 108 sec latency 26.787 ms 2024-04-21T23:11:58.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:58 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:11:58.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:58 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:11:59.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:58 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:11:59.695 INFO:tasks.workunit.client.0.smithi044.stdout: 1 286684 14.34 MB/sec warmup 109 sec latency 107.040 ms 2024-04-21T23:12:00.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:59 smithi044 ceph-mon[27934]: pgmap v99: 129 pgs: 129 active+clean; 47 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.1 MiB/s wr, 482 op/s 2024-04-21T23:12:00.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:59 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:00.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:59 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:00.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:11:59 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:12:00.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:59 smithi069 ceph-mon[31816]: pgmap v99: 129 pgs: 129 active+clean; 47 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.1 MiB/s wr, 482 op/s 2024-04-21T23:12:00.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:59 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:00.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:59 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:00.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:11:59 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:12:00.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:59 smithi055 ceph-mon[31117]: pgmap v99: 129 pgs: 129 active+clean; 47 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.1 MiB/s wr, 482 op/s 2024-04-21T23:12:00.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:59 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:12:00.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:59 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:12:00.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:11:59 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:12:00.695 INFO:tasks.workunit.client.0.smithi044.stdout: 1 288554 14.29 MB/sec warmup 110 sec latency 31.042 ms 2024-04-21T23:12:01.695 INFO:tasks.workunit.client.0.smithi044.stdout: 1 289621 14.20 MB/sec warmup 111 sec latency 75.570 ms 2024-04-21T23:12:02.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:01 smithi044 ceph-mon[27934]: pgmap v100: 129 pgs: 129 active+clean; 47 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 403 op/s 2024-04-21T23:12:02.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:01 smithi055 ceph-mon[31117]: pgmap v100: 129 pgs: 129 active+clean; 47 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 403 op/s 2024-04-21T23:12:02.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:01 smithi069 ceph-mon[31816]: pgmap v100: 129 pgs: 129 active+clean; 47 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 403 op/s 2024-04-21T23:12:02.695 INFO:tasks.workunit.client.0.smithi044.stdout: 1 291542 14.19 MB/sec warmup 112 sec latency 86.091 ms 2024-04-21T23:12:03.212 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.210+0000 7f1d5ac96640 1 -- 172.21.15.44:0/3180360667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d541032d0 msgr2=0x7f1d5410b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:03.212 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.210+0000 7f1d5ac96640 1 --2- 172.21.15.44:0/3180360667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d541032d0 0x7f1d5410b7b0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009900 tx=0x7f1d4002bcd0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.212 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.210+0000 7f1d5ac96640 1 -- 172.21.15.44:0/3180360667 shutdown_connections 2024-04-21T23:12:03.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.210+0000 7f1d5ac96640 1 --2- 172.21.15.44:0/3180360667 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d5410bd80 0x7f1d5410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.210+0000 7f1d5ac96640 1 --2- 172.21.15.44:0/3180360667 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d541032d0 0x7f1d5410b7b0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.210+0000 7f1d5ac96640 1 --2- 172.21.15.44:0/3180360667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d541029b0 0x7f1d54102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.210+0000 7f1d5ac96640 1 -- 172.21.15.44:0/3180360667 >> 172.21.15.44:0/3180360667 conn(0x7f1d54077760 msgr2=0x7f1d54077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:03.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.210+0000 7f1d5ac96640 1 -- 172.21.15.44:0/3180360667 shutdown_connections 2024-04-21T23:12:03.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.210+0000 7f1d5ac96640 1 -- 172.21.15.44:0/3180360667 wait complete. 2024-04-21T23:12:03.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.211+0000 7f1d5ac96640 1 Processor -- start 2024-04-21T23:12:03.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.211+0000 7f1d5ac96640 1 -- start start 2024-04-21T23:12:03.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.211+0000 7f1d5ac96640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d541029b0 0x7f1d54198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:03.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.211+0000 7f1d5ac96640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d541032d0 0x7f1d54198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:03.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.211+0000 7f1d5ac96640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d5410bd80 0x7f1d5419c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:03.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.211+0000 7f1d5ac96640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1d54111b10 con 0x7f1d541032d0 2024-04-21T23:12:03.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.211+0000 7f1d5ac96640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1d54111990 con 0x7f1d541029b0 2024-04-21T23:12:03.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.211+0000 7f1d5ac96640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1d54111c90 con 0x7f1d5410bd80 2024-04-21T23:12:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.211+0000 7f1d53fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d541032d0 0x7f1d54198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.211+0000 7f1d53fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d541032d0 0x7f1d54198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43152/0 (socket says 172.21.15.44:43152) 2024-04-21T23:12:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.211+0000 7f1d53fff640 1 -- 172.21.15.44:0/468430214 learned_addr learned my addr 172.21.15.44:0/468430214 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.211+0000 7f1d58a32640 1 --2- 172.21.15.44:0/468430214 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d541029b0 0x7f1d54198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.211+0000 7f1d59233640 1 --2- 172.21.15.44:0/468430214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d5410bd80 0x7f1d5419c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.212+0000 7f1d53fff640 1 -- 172.21.15.44:0/468430214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d5410bd80 msgr2=0x7f1d5419c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.212+0000 7f1d53fff640 1 --2- 172.21.15.44:0/468430214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d5410bd80 0x7f1d5419c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.212+0000 7f1d53fff640 1 -- 172.21.15.44:0/468430214 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d541029b0 msgr2=0x7f1d54198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.212+0000 7f1d53fff640 1 --2- 172.21.15.44:0/468430214 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d541029b0 0x7f1d54198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.212+0000 7f1d53fff640 1 -- 172.21.15.44:0/468430214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d5419d020 con 0x7f1d541032d0 2024-04-21T23:12:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.212+0000 7f1d53fff640 1 --2- 172.21.15.44:0/468430214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d541032d0 0x7f1d54198570 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1d40005940 tx=0x7f1d4002cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.212+0000 7f1d51ffb640 1 -- 172.21.15.44:0/468430214 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d40041070 con 0x7f1d541032d0 2024-04-21T23:12:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.212+0000 7f1d51ffb640 1 -- 172.21.15.44:0/468430214 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d40004450 con 0x7f1d541032d0 2024-04-21T23:12:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.212+0000 7f1d5ac96640 1 -- 172.21.15.44:0/468430214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d5419d2b0 con 0x7f1d541032d0 2024-04-21T23:12:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.212+0000 7f1d51ffb640 1 -- 172.21.15.44:0/468430214 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d40003d80 con 0x7f1d541032d0 2024-04-21T23:12:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.212+0000 7f1d5ac96640 1 -- 172.21.15.44:0/468430214 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d541a4ba0 con 0x7f1d541032d0 2024-04-21T23:12:03.222 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.214+0000 7f1d51ffb640 1 -- 172.21.15.44:0/468430214 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1d40044070 con 0x7f1d541032d0 2024-04-21T23:12:03.222 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d5d884740 1 Processor -- start 2024-04-21T23:12:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d5d884740 1 -- start start 2024-04-21T23:12:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d5d884740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565091474f30 0x565091475310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d5d884740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091475850 0x56509147e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d5d884740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56509147e620 0x565091480a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d5d884740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5650913d4b70 con 0x565091475850 2024-04-21T23:12:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d5d884740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x56509137d470 con 0x56509147e620 2024-04-21T23:12:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d5d884740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5650913d0b30 con 0x565091474f30 2024-04-21T23:12:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d51ffb640 1 --2- 172.21.15.44:0/468430214 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1d440a2cf0 0x7f1d440a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d58a32640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091475850 0x56509147e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d58a32640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091475850 0x56509147e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43160/0 (socket says 172.21.15.44:43160) 2024-04-21T23:12:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d58a32640 1 -- 172.21.15.44:0/1690450211 learned_addr learned my addr 172.21.15.44:0/1690450211 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d59233640 1 --2- 172.21.15.44:0/1690450211 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565091474f30 0x565091475310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d53fff640 1 --2- 172.21.15.44:0/1690450211 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56509147e620 0x565091480a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d59233640 1 --2- 172.21.15.44:0/468430214 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1d440a2cf0 0x7f1d440a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d51ffb640 1 -- 172.21.15.44:0/468430214 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f1d400f1230 con 0x7f1d541032d0 2024-04-21T23:12:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d58a32640 1 -- 172.21.15.44:0/1690450211 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565091474f30 msgr2=0x565091475310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:03.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d58a32640 1 --2- 172.21.15.44:0/1690450211 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565091474f30 0x565091475310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d58a32640 1 -- 172.21.15.44:0/1690450211 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56509147e620 msgr2=0x565091480a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:03.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d58a32640 1 --2- 172.21.15.44:0/1690450211 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56509147e620 0x565091480a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d58a32640 1 -- 172.21.15.44:0/1690450211 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650910e9210 con 0x565091475850 2024-04-21T23:12:03.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.215+0000 7f1d53fff640 1 --2- 172.21.15.44:0/1690450211 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56509147e620 0x565091480a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:12:03.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.216+0000 7f1d58a32640 1 --2- 172.21.15.44:0/1690450211 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091475850 0x56509147e0e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f1d48009d20 tx=0x7f1d48011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e097edf62ab3639f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:03.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.216+0000 7f1d59233640 1 --2- 172.21.15.44:0/468430214 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1d440a2cf0 0x7f1d440a51b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1d5406a040 tx=0x7f1d4c009500 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:03.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.216+0000 7f1d3effd640 1 -- 172.21.15.44:0/1690450211 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4801e070 con 0x565091475850 2024-04-21T23:12:03.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.216+0000 7f1d3effd640 1 -- 172.21.15.44:0/1690450211 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d48019b10 con 0x565091475850 2024-04-21T23:12:03.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.216+0000 7f1d3effd640 1 -- 172.21.15.44:0/1690450211 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4800ebf0 con 0x565091475850 2024-04-21T23:12:03.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.216+0000 7f1d5d884740 1 -- 172.21.15.44:0/1690450211 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091475850 msgr2=0x56509147e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:03.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.216+0000 7f1d5d884740 1 --2- 172.21.15.44:0/1690450211 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091475850 0x56509147e0e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f1d48009d20 tx=0x7f1d48011810 comp rx=0 tx=0).stop 2024-04-21T23:12:03.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.217+0000 7f1d5d884740 1 -- 172.21.15.44:0/1690450211 shutdown_connections 2024-04-21T23:12:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.217+0000 7f1d5d884740 1 --2- 172.21.15.44:0/1690450211 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56509147e620 0x565091480a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.217+0000 7f1d5d884740 1 --2- 172.21.15.44:0/1690450211 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091475850 0x56509147e0e0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.217+0000 7f1d5d884740 1 --2- 172.21.15.44:0/1690450211 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565091474f30 0x565091475310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.217+0000 7f1d5d884740 1 -- 172.21.15.44:0/1690450211 >> 172.21.15.44:0/1690450211 conn(0x56509146f850 msgr2=0x56509146fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.217+0000 7f1d5d884740 1 -- 172.21.15.44:0/1690450211 shutdown_connections 2024-04-21T23:12:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.217+0000 7f1d5d884740 1 -- 172.21.15.44:0/1690450211 wait complete. 2024-04-21T23:12:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.217+0000 7f1d5d884740 1 -- start start 2024-04-21T23:12:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.217+0000 7f1d5d884740 1 Processor -- start 2024-04-21T23:12:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.217+0000 7f1d5d884740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091474f30 0x56509147b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.217+0000 7f1d5d884740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x565091475850 0x56509147bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.217+0000 7f1d5d884740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56509147e620 0x56509147c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.217+0000 7f1d5d884740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x565091481250 con 0x565091474f30 2024-04-21T23:12:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.217+0000 7f1d5d884740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x565091390a10 con 0x565091475850 2024-04-21T23:12:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.217+0000 7f1d5d884740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5650914813d0 con 0x56509147e620 2024-04-21T23:12:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.217+0000 7f1d5d884740 10 client.0 _ll_register_callbacks cb 0x5650913cbb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.217+0000 7f1d58a32640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091474f30 0x56509147b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.217+0000 7f1d58a32640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091474f30 0x56509147b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43168/0 (socket says 172.21.15.44:43168) 2024-04-21T23:12:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.217+0000 7f1d58a32640 1 -- 172.21.15.44:0/1152917405 learned_addr learned my addr 172.21.15.44:0/1152917405 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.217+0000 7f1d53fff640 1 --2- 172.21.15.44:0/1152917405 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x565091475850 0x56509147bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:03.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.218+0000 7f1d59233640 1 --2- 172.21.15.44:0/1152917405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56509147e620 0x56509147c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:03.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.218+0000 7f1d53fff640 1 -- 172.21.15.44:0/1152917405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56509147e620 msgr2=0x56509147c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:03.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.218+0000 7f1d53fff640 1 --2- 172.21.15.44:0/1152917405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56509147e620 0x56509147c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.218+0000 7f1d53fff640 1 -- 172.21.15.44:0/1152917405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091474f30 msgr2=0x56509147b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:03.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.218+0000 7f1d53fff640 1 --2- 172.21.15.44:0/1152917405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091474f30 0x56509147b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.218+0000 7f1d53fff640 1 -- 172.21.15.44:0/1152917405 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650911b5620 con 0x565091475850 2024-04-21T23:12:03.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.218+0000 7f1d58a32640 1 --2- 172.21.15.44:0/1152917405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091474f30 0x56509147b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:03.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.218+0000 7f1d53fff640 1 --2- 172.21.15.44:0/1152917405 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x565091475850 0x56509147bc30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1d400ee690 tx=0x7f1d4008c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c832b63bd1ec3bf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:03.229 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.218+0000 7f1d33fff640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:12:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.218+0000 7f1d33fff640 1 -- 172.21.15.44:0/1152917405 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4004b070 con 0x565091475850 2024-04-21T23:12:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.218+0000 7f1d33fff640 1 -- 172.21.15.44:0/1152917405 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f1d400c5c40 con 0x565091475850 2024-04-21T23:12:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.218+0000 7f1d5d884740 1 -- 172.21.15.44:0/1152917405 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d48019b10 con 0x565091475850 2024-04-21T23:12:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.218+0000 7f1d33fff640 1 -- 172.21.15.44:0/1152917405 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1d400bd2e0 con 0x565091475850 2024-04-21T23:12:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.218+0000 7f1d5d884740 10 client.25925 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.218+0000 7f1d5d884740 1 -- 172.21.15.44:0/1152917405 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56509148d4b0 con 0x565091475850 2024-04-21T23:12:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.219+0000 7f1d33fff640 1 -- 172.21.15.44:0/1152917405 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d48019b10 con 0x565091475850 2024-04-21T23:12:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.219+0000 7f1d33fff640 1 -- 172.21.15.44:0/1152917405 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f1d4008f020 con 0x565091475850 2024-04-21T23:12:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.219+0000 7f1d5d884740 10 client.25925 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.219+0000 7f1d5d884740 10 client.25925 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.219+0000 7f1d5d884740 20 client.25925 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.219+0000 7f1d5d884740 1 --2- 172.21.15.44:0/1152917405 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56509148e910 0x565091490d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.219+0000 7f1d5d884740 4 client.25925 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:12:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.219+0000 7f1d5d884740 1 -- 172.21.15.44:0/1152917405 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650913e20b0 con 0x56509148e910 2024-04-21T23:12:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.219+0000 7f1d58a32640 1 --2- 172.21.15.44:0/1152917405 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56509148e910 0x565091490d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.219+0000 7f1d58a32640 1 --2- 172.21.15.44:0/1152917405 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56509148e910 0x565091490d60 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a58d6c423949c815 server_cookie=55bc8db44505b238 in_seq=0 out_seq=0 2024-04-21T23:12:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.219+0000 7f1d33fff640 10 client.25925 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.220+0000 7f1d33fff640 1 -- 172.21.15.44:0/1152917405 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5650913e20b0 con 0x56509148e910 2024-04-21T23:12:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.220+0000 7f1d33fff640 10 client.25925 handle_command_reply: tid=0 2024-04-21T23:12:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.220+0000 7f1d5d884740 1 client.25925 shutdown 2024-04-21T23:12:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.220+0000 7f1d5d884740 1 -- 172.21.15.44:0/1152917405 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x565091475850 msgr2=0x56509147bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.220+0000 7f1d5d884740 1 --2- 172.21.15.44:0/1152917405 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x565091475850 0x56509147bc30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1d400ee690 tx=0x7f1d4008c440 comp rx=0 tx=0).stop 2024-04-21T23:12:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.221+0000 7f1d5d884740 1 -- 172.21.15.44:0/1152917405 shutdown_connections 2024-04-21T23:12:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.221+0000 7f1d5d884740 1 --2- 172.21.15.44:0/1152917405 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56509148e910 0x565091490d60 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.221+0000 7f1d5d884740 1 --2- 172.21.15.44:0/1152917405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56509147e620 0x56509147c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.221+0000 7f1d5d884740 1 --2- 172.21.15.44:0/1152917405 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x565091475850 0x56509147bc30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.221+0000 7f1d33fff640 0 client.25925 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.221+0000 7f1d5d884740 1 --2- 172.21.15.44:0/1152917405 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091474f30 0x56509147b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.221+0000 7f1d5d884740 1 -- 172.21.15.44:0/1152917405 >> 172.21.15.44:0/1152917405 conn(0x56509146f850 msgr2=0x5650914715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.221+0000 7f1d5d884740 1 -- 172.21.15.44:0/1152917405 shutdown_connections 2024-04-21T23:12:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.221+0000 7f1d5d884740 1 -- 172.21.15.44:0/1152917405 wait complete. 2024-04-21T23:12:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.221+0000 7f1d5d884740 20 client.25925 trim_cache size 0 max 16384 2024-04-21T23:12:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.236+0000 7f1d5d884740 1 Processor -- start 2024-04-21T23:12:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.236+0000 7f1d5d884740 1 -- start start 2024-04-21T23:12:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.236+0000 7f1d5d884740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091474f30 0x56509147de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.236+0000 7f1d5d884740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x565091475850 0x56509148a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.236+0000 7f1d5d884740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56509147e620 0x565091496670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.236+0000 7f1d5d884740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1d541aaa00 con 0x565091474f30 2024-04-21T23:12:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.236+0000 7f1d5d884740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1d541ad280 con 0x565091475850 2024-04-21T23:12:03.241 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.236+0000 7f1d5d884740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1d541ab070 con 0x56509147e620 2024-04-21T23:12:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.236+0000 7f1d58a32640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091474f30 0x56509147de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.236+0000 7f1d58a32640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091474f30 0x56509147de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43172/0 (socket says 172.21.15.44:43172) 2024-04-21T23:12:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.236+0000 7f1d58a32640 1 -- 172.21.15.44:0/1613054502 learned_addr learned my addr 172.21.15.44:0/1613054502 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.236+0000 7f1d53fff640 1 --2- 172.21.15.44:0/1613054502 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x565091475850 0x56509148a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.236+0000 7f1d59233640 1 --2- 172.21.15.44:0/1613054502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56509147e620 0x565091496670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.236+0000 7f1d58a32640 1 -- 172.21.15.44:0/1613054502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56509147e620 msgr2=0x565091496670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.236+0000 7f1d58a32640 1 --2- 172.21.15.44:0/1613054502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56509147e620 0x565091496670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.236+0000 7f1d58a32640 1 -- 172.21.15.44:0/1613054502 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x565091475850 msgr2=0x56509148a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.236+0000 7f1d58a32640 1 --2- 172.21.15.44:0/1613054502 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x565091475850 0x56509148a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.236+0000 7f1d58a32640 1 -- 172.21.15.44:0/1613054502 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565091477290 con 0x565091474f30 2024-04-21T23:12:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.236+0000 7f1d53fff640 1 --2- 172.21.15.44:0/1613054502 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x565091475850 0x56509148a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:12:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.236+0000 7f1d59233640 1 --2- 172.21.15.44:0/1613054502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56509147e620 0x565091496670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:12:03.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.237+0000 7f1d58a32640 1 --2- 172.21.15.44:0/1613054502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091474f30 0x56509147de20 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f1d48009d20 tx=0x7f1d48005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a93fff56ac400f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.237+0000 7f1d3e7fc640 1 -- 172.21.15.44:0/1613054502 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4801e040 con 0x565091474f30 2024-04-21T23:12:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.237+0000 7f1d3e7fc640 1 -- 172.21.15.44:0/1613054502 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d4801f040 con 0x565091474f30 2024-04-21T23:12:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.237+0000 7f1d3e7fc640 1 -- 172.21.15.44:0/1613054502 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4802b180 con 0x565091474f30 2024-04-21T23:12:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.237+0000 7f1d5d884740 1 -- 172.21.15.44:0/1613054502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091474f30 msgr2=0x56509147de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.237+0000 7f1d5d884740 1 --2- 172.21.15.44:0/1613054502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091474f30 0x56509147de20 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f1d48009d20 tx=0x7f1d48005f90 comp rx=0 tx=0).stop 2024-04-21T23:12:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.237+0000 7f1d5d884740 1 -- 172.21.15.44:0/1613054502 shutdown_connections 2024-04-21T23:12:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.237+0000 7f1d5d884740 1 --2- 172.21.15.44:0/1613054502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56509147e620 0x565091496670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.237+0000 7f1d5d884740 1 --2- 172.21.15.44:0/1613054502 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x565091475850 0x56509148a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.237+0000 7f1d5d884740 1 --2- 172.21.15.44:0/1613054502 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x565091474f30 0x56509147de20 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.237+0000 7f1d5d884740 1 -- 172.21.15.44:0/1613054502 >> 172.21.15.44:0/1613054502 conn(0x56509146f850 msgr2=0x565091466220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.237+0000 7f1d5d884740 1 -- 172.21.15.44:0/1613054502 shutdown_connections 2024-04-21T23:12:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.237+0000 7f1d5d884740 1 -- 172.21.15.44:0/1613054502 wait complete. 2024-04-21T23:12:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.238+0000 7f1d5d884740 1 -- start start 2024-04-21T23:12:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.238+0000 7f1d5d884740 1 Processor -- start 2024-04-21T23:12:03.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.238+0000 7f1d5d884740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565091474f30 0x5650914ed250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.238+0000 7f1d5d884740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x565091475850 0x5650914ed790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.238+0000 7f1d5d884740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56509147e620 0x5650914819f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.238+0000 7f1d5d884740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1d541ab740 con 0x56509147e620 2024-04-21T23:12:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.238+0000 7f1d5d884740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1d541acbb0 con 0x565091475850 2024-04-21T23:12:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.238+0000 7f1d5d884740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1d541abe10 con 0x565091474f30 2024-04-21T23:12:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.238+0000 7f1d5d884740 10 client.0 _ll_register_callbacks cb 0x7f1d400b9040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.238+0000 7f1d59233640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56509147e620 0x5650914819f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.238+0000 7f1d58a32640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565091474f30 0x5650914ed250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.238+0000 7f1d59233640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56509147e620 0x5650914819f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43184/0 (socket says 172.21.15.44:43184) 2024-04-21T23:12:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.238+0000 7f1d59233640 1 -- 172.21.15.44:0/2556897599 learned_addr learned my addr 172.21.15.44:0/2556897599 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.238+0000 7f1d53fff640 1 --2- 172.21.15.44:0/2556897599 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x565091475850 0x5650914ed790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.239+0000 7f1d59233640 1 -- 172.21.15.44:0/2556897599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565091474f30 msgr2=0x5650914ed250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.239+0000 7f1d59233640 1 --2- 172.21.15.44:0/2556897599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565091474f30 0x5650914ed250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.239+0000 7f1d59233640 1 -- 172.21.15.44:0/2556897599 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x565091475850 msgr2=0x5650914ed790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.239+0000 7f1d59233640 1 --2- 172.21.15.44:0/2556897599 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x565091475850 0x5650914ed790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.239+0000 7f1d59233640 1 -- 172.21.15.44:0/2556897599 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650914820e0 con 0x56509147e620 2024-04-21T23:12:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.239+0000 7f1d58a32640 1 --2- 172.21.15.44:0/2556897599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565091474f30 0x5650914ed250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.239+0000 7f1d59233640 1 --2- 172.21.15.44:0/2556897599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56509147e620 0x5650914819f0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c00fe50 tx=0x7f1d4c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf1f1c3faa75bbbc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.239+0000 7f1d33fff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:12:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.239+0000 7f1d33fff640 1 -- 172.21.15.44:0/2556897599 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c004d20 con 0x56509147e620 2024-04-21T23:12:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.239+0000 7f1d33fff640 1 -- 172.21.15.44:0/2556897599 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f1d4c004250 con 0x56509147e620 2024-04-21T23:12:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.239+0000 7f1d5d884740 1 -- 172.21.15.44:0/2556897599 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d4801f040 con 0x56509147e620 2024-04-21T23:12:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.239+0000 7f1d33fff640 1 -- 172.21.15.44:0/2556897599 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1d4c035050 con 0x56509147e620 2024-04-21T23:12:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.239+0000 7f1d5d884740 10 client.16365 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.239+0000 7f1d5d884740 1 -- 172.21.15.44:0/2556897599 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5650914f4a00 con 0x56509147e620 2024-04-21T23:12:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.240+0000 7f1d33fff640 1 -- 172.21.15.44:0/2556897599 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d4801f040 con 0x56509147e620 2024-04-21T23:12:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.240+0000 7f1d33fff640 1 -- 172.21.15.44:0/2556897599 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f1d4c01d070 con 0x56509147e620 2024-04-21T23:12:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.240+0000 7f1d5d884740 10 client.16365 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.240+0000 7f1d5d884740 10 client.16365 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.240+0000 7f1d5d884740 20 client.16365 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.240+0000 7f1d5d884740 1 --2- 172.21.15.44:0/2556897599 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5650914f5200 0x5650914f7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.240+0000 7f1d5d884740 4 client.16365 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:12:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.240+0000 7f1d5d884740 1 -- 172.21.15.44:0/2556897599 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5650913dc400 con 0x5650914f5200 2024-04-21T23:12:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.240+0000 7f1d58a32640 1 --2- 172.21.15.44:0/2556897599 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5650914f5200 0x5650914f7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.241+0000 7f1d58a32640 1 --2- 172.21.15.44:0/2556897599 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5650914f5200 0x5650914f7620 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6dbaaee2bf762dc5 server_cookie=5a48af524abb488e in_seq=0 out_seq=0 2024-04-21T23:12:03.246 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.241+0000 7f1d33fff640 10 client.16365 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:03.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.509+0000 7f1d33fff640 1 -- 172.21.15.44:0/2556897599 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5650913dc400 con 0x5650914f5200 2024-04-21T23:12:03.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.509+0000 7f1d33fff640 10 client.16365 handle_command_reply: tid=0 2024-04-21T23:12:03.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.509+0000 7f1d5d884740 1 client.16365 shutdown 2024-04-21T23:12:03.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.509+0000 7f1d5d884740 1 -- 172.21.15.44:0/2556897599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56509147e620 msgr2=0x5650914819f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:03.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.509+0000 7f1d5d884740 1 --2- 172.21.15.44:0/2556897599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56509147e620 0x5650914819f0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c00fe50 tx=0x7f1d4c004ab0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.509+0000 7f1d5d884740 1 -- 172.21.15.44:0/2556897599 shutdown_connections 2024-04-21T23:12:03.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.509+0000 7f1d5d884740 1 --2- 172.21.15.44:0/2556897599 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5650914f5200 0x5650914f7620 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.509+0000 7f1d5d884740 1 --2- 172.21.15.44:0/2556897599 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56509147e620 0x5650914819f0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.509+0000 7f1d5d884740 1 --2- 172.21.15.44:0/2556897599 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x565091475850 0x5650914ed790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.509+0000 7f1d5d884740 1 --2- 172.21.15.44:0/2556897599 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565091474f30 0x5650914ed250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.509+0000 7f1d5d884740 1 -- 172.21.15.44:0/2556897599 >> 172.21.15.44:0/2556897599 conn(0x56509146f850 msgr2=0x565091479f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:03.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.509+0000 7f1d33fff640 0 client.16365 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:03.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.510+0000 7f1d5d884740 1 -- 172.21.15.44:0/2556897599 shutdown_connections 2024-04-21T23:12:03.510 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.510+0000 7f1d5d884740 1 -- 172.21.15.44:0/2556897599 wait complete. 2024-04-21T23:12:03.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.510+0000 7f1d5d884740 20 client.16365 trim_cache size 0 max 16384 2024-04-21T23:12:03.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:12:03.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:12:03.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:12:03.511 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:12:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.510+0000 7f1d5ac96640 1 -- 172.21.15.44:0/468430214 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1d440a2cf0 msgr2=0x7f1d440a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.510+0000 7f1d5ac96640 1 --2- 172.21.15.44:0/468430214 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1d440a2cf0 0x7f1d440a51b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1d5406a040 tx=0x7f1d4c009500 comp rx=0 tx=0).stop 2024-04-21T23:12:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.510+0000 7f1d5ac96640 1 -- 172.21.15.44:0/468430214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d541032d0 msgr2=0x7f1d54198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.510+0000 7f1d5ac96640 1 --2- 172.21.15.44:0/468430214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d541032d0 0x7f1d54198570 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1d40005940 tx=0x7f1d4002cbd0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.510+0000 7f1d5ac96640 1 -- 172.21.15.44:0/468430214 shutdown_connections 2024-04-21T23:12:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.510+0000 7f1d5ac96640 1 --2- 172.21.15.44:0/468430214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d5410bd80 0x7f1d5419c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.510+0000 7f1d5ac96640 1 --2- 172.21.15.44:0/468430214 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1d440a2cf0 0x7f1d440a51b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.512 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.510+0000 7f1d5ac96640 1 --2- 172.21.15.44:0/468430214 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1d541032d0 0x7f1d54198570 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.510+0000 7f1d5ac96640 1 --2- 172.21.15.44:0/468430214 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1d541029b0 0x7f1d54198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.510+0000 7f1d5ac96640 1 -- 172.21.15.44:0/468430214 >> 172.21.15.44:0/468430214 conn(0x7f1d54077760 msgr2=0x7f1d540fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.510+0000 7f1d5ac96640 1 -- 172.21.15.44:0/468430214 shutdown_connections 2024-04-21T23:12:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:03.511+0000 7f1d5ac96640 1 -- 172.21.15.44:0/468430214 wait complete. 2024-04-21T23:12:03.695 INFO:tasks.workunit.client.0.smithi044.stdout: 1 294225 14.19 MB/sec warmup 113 sec latency 141.798 ms 2024-04-21T23:12:04.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:03 smithi044 ceph-mon[27934]: pgmap v101: 129 pgs: 129 active+clean; 51 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.6 MiB/s wr, 548 op/s 2024-04-21T23:12:04.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:03 smithi055 ceph-mon[31117]: pgmap v101: 129 pgs: 129 active+clean; 51 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.6 MiB/s wr, 548 op/s 2024-04-21T23:12:04.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:03 smithi069 ceph-mon[31816]: pgmap v101: 129 pgs: 129 active+clean; 51 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.6 MiB/s wr, 548 op/s 2024-04-21T23:12:04.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 296315 14.15 MB/sec warmup 114 sec latency 53.706 ms 2024-04-21T23:12:05.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 298708 14.15 MB/sec warmup 115 sec latency 65.344 ms 2024-04-21T23:12:06.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:05 smithi044 ceph-mon[27934]: pgmap v102: 129 pgs: 129 active+clean; 51 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 271 op/s 2024-04-21T23:12:06.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:05 smithi055 ceph-mon[31117]: pgmap v102: 129 pgs: 129 active+clean; 51 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 271 op/s 2024-04-21T23:12:06.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:05 smithi069 ceph-mon[31816]: pgmap v102: 129 pgs: 129 active+clean; 51 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 271 op/s 2024-04-21T23:12:06.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 301760 14.19 MB/sec warmup 116 sec latency 24.639 ms 2024-04-21T23:12:07.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 304696 14.18 MB/sec warmup 117 sec latency 15.694 ms 2024-04-21T23:12:08.016 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:07 smithi069 ceph-mon[31816]: pgmap v103: 129 pgs: 129 active+clean; 47 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.8 MiB/s wr, 429 op/s 2024-04-21T23:12:08.120 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:12:08.121 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:12:08.143 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:07 smithi044 ceph-mon[27934]: pgmap v103: 129 pgs: 129 active+clean; 47 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.8 MiB/s wr, 429 op/s 2024-04-21T23:12:08.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:07 smithi055 ceph-mon[31117]: pgmap v103: 129 pgs: 129 active+clean; 47 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.8 MiB/s wr, 429 op/s 2024-04-21T23:12:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.228+0000 7f84e722e640 1 -- 172.21.15.44:0/2079183706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e0100610 msgr2=0x7f84e0100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.228+0000 7f84e722e640 1 --2- 172.21.15.44:0/2079183706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e0100610 0x7f84e0100a90 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f84cc009930 tx=0x7f84cc02bc30 comp rx=0 tx=0).stop 2024-04-21T23:12:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.228+0000 7f84e722e640 1 -- 172.21.15.44:0/2079183706 shutdown_connections 2024-04-21T23:12:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.228+0000 7f84e722e640 1 --2- 172.21.15.44:0/2079183706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84e0100fd0 0x7f84e010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.228+0000 7f84e722e640 1 --2- 172.21.15.44:0/2079183706 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e0100610 0x7f84e0100a90 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.228+0000 7f84e722e640 1 --2- 172.21.15.44:0/2079183706 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84e01066b0 0x7f84e0106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.228+0000 7f84e722e640 1 -- 172.21.15.44:0/2079183706 >> 172.21.15.44:0/2079183706 conn(0x7f84e00777f0 msgr2=0x7f84e0077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.228+0000 7f84e722e640 1 -- 172.21.15.44:0/2079183706 shutdown_connections 2024-04-21T23:12:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.228+0000 7f84e722e640 1 -- 172.21.15.44:0/2079183706 wait complete. 2024-04-21T23:12:08.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.229+0000 7f84e722e640 1 Processor -- start 2024-04-21T23:12:08.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.229+0000 7f84e722e640 1 -- start start 2024-04-21T23:12:08.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.229+0000 7f84e722e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e0100610 0x7f84e0196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.229+0000 7f84e722e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84e0100fd0 0x7f84e01973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.229+0000 7f84e722e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84e01066b0 0x7f84e019b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.229+0000 7f84e722e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f84e01109f0 con 0x7f84e0100610 2024-04-21T23:12:08.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.229+0000 7f84e722e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f84e0110870 con 0x7f84e0100fd0 2024-04-21T23:12:08.231 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.229+0000 7f84e722e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f84e0110b70 con 0x7f84e01066b0 2024-04-21T23:12:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.229+0000 7f84e4fca640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e0100610 0x7f84e0196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.229+0000 7f84e4fca640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e0100610 0x7f84e0196e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43206/0 (socket says 172.21.15.44:43206) 2024-04-21T23:12:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.229+0000 7f84e4fca640 1 -- 172.21.15.44:0/1697249691 learned_addr learned my addr 172.21.15.44:0/1697249691 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.229+0000 7f84dffff640 1 --2- 172.21.15.44:0/1697249691 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84e0100fd0 0x7f84e01973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.229+0000 7f84e57cb640 1 --2- 172.21.15.44:0/1697249691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84e01066b0 0x7f84e019b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.229+0000 7f84e4fca640 1 -- 172.21.15.44:0/1697249691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84e01066b0 msgr2=0x7f84e019b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.229+0000 7f84e4fca640 1 --2- 172.21.15.44:0/1697249691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84e01066b0 0x7f84e019b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.229+0000 7f84e4fca640 1 -- 172.21.15.44:0/1697249691 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84e0100fd0 msgr2=0x7f84e01973d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.229+0000 7f84e4fca640 1 --2- 172.21.15.44:0/1697249691 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84e0100fd0 0x7f84e01973d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.230+0000 7f84e4fca640 1 -- 172.21.15.44:0/1697249691 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e019be80 con 0x7f84e0100610 2024-04-21T23:12:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.230+0000 7f84e4fca640 1 --2- 172.21.15.44:0/1697249691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e0100610 0x7f84e0196e90 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f84d400ead0 tx=0x7f84d400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.230+0000 7f84ddffb640 1 -- 172.21.15.44:0/1697249691 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d400bbd0 con 0x7f84e0100610 2024-04-21T23:12:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.230+0000 7f84e722e640 1 -- 172.21.15.44:0/1697249691 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84e019c170 con 0x7f84e0100610 2024-04-21T23:12:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.230+0000 7f84ddffb640 1 -- 172.21.15.44:0/1697249691 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84d400bd70 con 0x7f84e0100610 2024-04-21T23:12:08.232 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.230+0000 7f84ddffb640 1 -- 172.21.15.44:0/1697249691 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d4017410 con 0x7f84e0100610 2024-04-21T23:12:08.233 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.230+0000 7f84e722e640 1 -- 172.21.15.44:0/1697249691 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84e0068a40 con 0x7f84e0100610 2024-04-21T23:12:08.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.232+0000 7f84ddffb640 1 -- 172.21.15.44:0/1697249691 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f84d401e020 con 0x7f84e0100610 2024-04-21T23:12:08.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.233+0000 7f84e9e12740 1 Processor -- start 2024-04-21T23:12:08.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.233+0000 7f84e9e12740 1 -- start start 2024-04-21T23:12:08.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.233+0000 7f84ddffb640 1 --2- 172.21.15.44:0/1697249691 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f84d00a2d00 0x7f84d00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.233+0000 7f84e9e12740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eebdf30 0x563a8eebe310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.233+0000 7f84e9e12740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eebe850 0x563a8eec6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.233+0000 7f84e9e12740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eec7430 0x563a8eec9820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.233+0000 7f84e9e12740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563a8ee22560 con 0x563a8eec7430 2024-04-21T23:12:08.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.233+0000 7f84e9e12740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x563a8edca730 con 0x563a8eebdf30 2024-04-21T23:12:08.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.233+0000 7f84e9e12740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563a8ee1e520 con 0x563a8eebe850 2024-04-21T23:12:08.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.233+0000 7f84e4fca640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eebdf30 0x563a8eebe310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.233+0000 7f84e57cb640 1 --2- 172.21.15.44:0/1697249691 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f84d00a2d00 0x7f84d00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.233+0000 7f84dffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eebe850 0x563a8eec6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.233+0000 7f84e4fca640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eebdf30 0x563a8eebe310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:40004/0 (socket says 172.21.15.44:40004) 2024-04-21T23:12:08.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.233+0000 7f84e57cb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eec7430 0x563a8eec9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.233+0000 7f84e4fca640 1 -- 172.21.15.44:0/1178335407 learned_addr learned my addr 172.21.15.44:0/1178335407 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:08.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.233+0000 7f84ddffb640 1 -- 172.21.15.44:0/1697249691 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f84d40d0f50 con 0x7f84e0100610 2024-04-21T23:12:08.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.233+0000 7f84e4fca640 1 -- 172.21.15.44:0/1178335407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eebe850 msgr2=0x563a8eec6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:08.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.234+0000 7f84e4fca640 1 --2- 172.21.15.44:0/1178335407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eebe850 0x563a8eec6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.234+0000 7f84e4fca640 1 -- 172.21.15.44:0/1178335407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eec7430 msgr2=0x563a8eec9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:08.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.234+0000 7f84e4fca640 1 --2- 172.21.15.44:0/1178335407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eec7430 0x563a8eec9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.234+0000 7f84e4fca640 1 -- 172.21.15.44:0/1178335407 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a8ec007d0 con 0x563a8eebdf30 2024-04-21T23:12:08.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.234+0000 7f84e4fca640 1 --2- 172.21.15.44:0/1178335407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eebdf30 0x563a8eebe310 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f84d400b690 tx=0x7f84d4022aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e05ef621738b58d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:08.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.234+0000 7f84e57cb640 1 --2- 172.21.15.44:0/1697249691 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f84d00a2d00 0x7f84d00a51c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x563a8eeb97f0 tx=0x7f84d8009540 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:08.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.234+0000 7f84caffd640 1 -- 172.21.15.44:0/1178335407 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d40a2bf0 con 0x563a8eebdf30 2024-04-21T23:12:08.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.234+0000 7f84caffd640 1 -- 172.21.15.44:0/1178335407 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84d4014d00 con 0x563a8eebdf30 2024-04-21T23:12:08.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.234+0000 7f84caffd640 1 -- 172.21.15.44:0/1178335407 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d40a25f0 con 0x563a8eebdf30 2024-04-21T23:12:08.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.235+0000 7f84e9e12740 1 -- 172.21.15.44:0/1178335407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eebdf30 msgr2=0x563a8eebe310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.235+0000 7f84e9e12740 1 --2- 172.21.15.44:0/1178335407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eebdf30 0x563a8eebe310 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f84d400b690 tx=0x7f84d4022aa0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.235+0000 7f84e9e12740 1 -- 172.21.15.44:0/1178335407 shutdown_connections 2024-04-21T23:12:08.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.235+0000 7f84e9e12740 1 --2- 172.21.15.44:0/1178335407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eebe850 0x563a8eec6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.235+0000 7f84e9e12740 1 --2- 172.21.15.44:0/1178335407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eec7430 0x563a8eec9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.235+0000 7f84e9e12740 1 --2- 172.21.15.44:0/1178335407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eebdf30 0x563a8eebe310 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.235+0000 7f84e9e12740 1 -- 172.21.15.44:0/1178335407 >> 172.21.15.44:0/1178335407 conn(0x563a8eeb8660 msgr2=0x563a8eeb8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:08.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.235+0000 7f84e9e12740 1 -- 172.21.15.44:0/1178335407 shutdown_connections 2024-04-21T23:12:08.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.235+0000 7f84e9e12740 1 -- 172.21.15.44:0/1178335407 wait complete. 2024-04-21T23:12:08.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.235+0000 7f84e9e12740 1 -- start start 2024-04-21T23:12:08.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.235+0000 7f84e9e12740 1 Processor -- start 2024-04-21T23:12:08.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84e9e12740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eebdf30 0x563a8eec44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84e9e12740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eebe850 0x563a8eec49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84e9e12740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eec7430 0x563a8eec4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84e9e12740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563a8eeca1c0 con 0x563a8eebe850 2024-04-21T23:12:08.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84e9e12740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x563a8eddde00 con 0x563a8eebdf30 2024-04-21T23:12:08.242 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84e9e12740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563a8eeca340 con 0x563a8eec7430 2024-04-21T23:12:08.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84e9e12740 10 client.0 _ll_register_callbacks cb 0x563a8ee18eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:08.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84e4fca640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eebdf30 0x563a8eec44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eebe850 0x563a8eec49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84e4fca640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eebdf30 0x563a8eec44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:40006/0 (socket says 172.21.15.44:40006) 2024-04-21T23:12:08.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84e4fca640 1 -- 172.21.15.44:0/176145204 learned_addr learned my addr 172.21.15.44:0/176145204 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:08.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eebe850 0x563a8eec49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43234/0 (socket says 172.21.15.44:43234) 2024-04-21T23:12:08.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84e57cb640 1 --2- 172.21.15.44:0/176145204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eec7430 0x563a8eec4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84e4fca640 1 -- 172.21.15.44:0/176145204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eec7430 msgr2=0x563a8eec4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:08.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84e4fca640 1 --2- 172.21.15.44:0/176145204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eec7430 0x563a8eec4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84e4fca640 1 -- 172.21.15.44:0/176145204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eebe850 msgr2=0x563a8eec49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:08.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84e4fca640 1 --2- 172.21.15.44:0/176145204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eebe850 0x563a8eec49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84e4fca640 1 -- 172.21.15.44:0/176145204 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a8eb363f0 con 0x563a8eebdf30 2024-04-21T23:12:08.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84e57cb640 1 --2- 172.21.15.44:0/176145204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eec7430 0x563a8eec4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:12:08.243 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84e4fca640 1 --2- 172.21.15.44:0/176145204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eebdf30 0x563a8eec44a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f84d4099a80 tx=0x7f84d40035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7f95dab09070274 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:08.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84bffff640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:12:08.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84bffff640 1 -- 172.21.15.44:0/176145204 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d40b5020 con 0x563a8eebdf30 2024-04-21T23:12:08.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84e9e12740 1 -- 172.21.15.44:0/176145204 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563a8ee2faa0 con 0x563a8eebdf30 2024-04-21T23:12:08.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.236+0000 7f84bffff640 1 -- 172.21.15.44:0/176145204 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f84d40a2880 con 0x563a8eebdf30 2024-04-21T23:12:08.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.237+0000 7f84bffff640 1 -- 172.21.15.44:0/176145204 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f84d40a25f0 con 0x563a8eebdf30 2024-04-21T23:12:08.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.237+0000 7f84e9e12740 10 client.25943 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:08.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.237+0000 7f84e9e12740 1 -- 172.21.15.44:0/176145204 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563a8eaf60f0 con 0x563a8eebdf30 2024-04-21T23:12:08.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.237+0000 7f84bffff640 1 -- 172.21.15.44:0/176145204 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563a8ee2faa0 con 0x563a8eebdf30 2024-04-21T23:12:08.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.237+0000 7f84bffff640 1 -- 172.21.15.44:0/176145204 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f84d40ce560 con 0x563a8eebdf30 2024-04-21T23:12:08.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.237+0000 7f84e9e12740 10 client.25943 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:08.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.237+0000 7f84e9e12740 10 client.25943 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:08.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.237+0000 7f84e9e12740 20 client.25943 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:08.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.237+0000 7f84e9e12740 1 --2- 172.21.15.44:0/176145204 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563a8eed7630 0x563a8eed9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.237+0000 7f84e9e12740 4 client.25943 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:12:08.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.237+0000 7f84e9e12740 1 -- 172.21.15.44:0/176145204 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a8ee29df0 con 0x563a8eed7630 2024-04-21T23:12:08.244 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.237+0000 7f84dffff640 1 --2- 172.21.15.44:0/176145204 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563a8eed7630 0x563a8eed9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.238+0000 7f84dffff640 1 --2- 172.21.15.44:0/176145204 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563a8eed7630 0x563a8eed9a20 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=372f9ff5d29fdb04 server_cookie=a708f5f0de73e18b in_seq=0 out_seq=0 2024-04-21T23:12:08.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.238+0000 7f84bffff640 10 client.25943 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:08.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.239+0000 7f84bffff640 1 -- 172.21.15.44:0/176145204 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563a8ee29df0 con 0x563a8eed7630 2024-04-21T23:12:08.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.239+0000 7f84bffff640 10 client.25943 handle_command_reply: tid=0 2024-04-21T23:12:08.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.239+0000 7f84e9e12740 1 client.25943 shutdown 2024-04-21T23:12:08.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.239+0000 7f84e9e12740 1 -- 172.21.15.44:0/176145204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eebdf30 msgr2=0x563a8eec44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.239+0000 7f84e9e12740 1 --2- 172.21.15.44:0/176145204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eebdf30 0x563a8eec44a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f84d4099a80 tx=0x7f84d40035f0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.239+0000 7f84e9e12740 1 -- 172.21.15.44:0/176145204 shutdown_connections 2024-04-21T23:12:08.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.239+0000 7f84e9e12740 1 --2- 172.21.15.44:0/176145204 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563a8eed7630 0x563a8eed9a20 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.239+0000 7f84e9e12740 1 --2- 172.21.15.44:0/176145204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eec7430 0x563a8eec4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.239+0000 7f84e9e12740 1 --2- 172.21.15.44:0/176145204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eebe850 0x563a8eec49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.239+0000 7f84e9e12740 1 --2- 172.21.15.44:0/176145204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eebdf30 0x563a8eec44a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.239+0000 7f84bffff640 0 client.25943 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:08.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.239+0000 7f84e9e12740 1 -- 172.21.15.44:0/176145204 >> 172.21.15.44:0/176145204 conn(0x563a8eeb8660 msgr2=0x563a8eeba3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:08.245 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.239+0000 7f84e9e12740 1 -- 172.21.15.44:0/176145204 shutdown_connections 2024-04-21T23:12:08.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.239+0000 7f84e9e12740 1 -- 172.21.15.44:0/176145204 wait complete. 2024-04-21T23:12:08.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.239+0000 7f84e9e12740 20 client.25943 trim_cache size 0 max 16384 2024-04-21T23:12:08.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.262+0000 7f84e9e12740 1 Processor -- start 2024-04-21T23:12:08.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.262+0000 7f84e9e12740 1 -- start start 2024-04-21T23:12:08.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.263+0000 7f84e9e12740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eebdf30 0x563a8eec6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.263+0000 7f84e9e12740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eebe850 0x563a8eed3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.263+0000 7f84e9e12740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eec7430 0x563a8eeb7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.263+0000 7f84e9e12740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f84e01aa9d0 con 0x563a8eebe850 2024-04-21T23:12:08.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.263+0000 7f84e9e12740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f84e01ad250 con 0x563a8eec7430 2024-04-21T23:12:08.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.263+0000 7f84e9e12740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f84e01ab040 con 0x563a8eebdf30 2024-04-21T23:12:08.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.263+0000 7f84e4fca640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eebdf30 0x563a8eec6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.263+0000 7f84e57cb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eec7430 0x563a8eeb7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.263+0000 7f84e4fca640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eebdf30 0x563a8eec6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:38544/0 (socket says 172.21.15.44:38544) 2024-04-21T23:12:08.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.263+0000 7f84e4fca640 1 -- 172.21.15.44:0/1811409435 learned_addr learned my addr 172.21.15.44:0/1811409435 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:08.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.263+0000 7f84dffff640 1 --2- 172.21.15.44:0/1811409435 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eebe850 0x563a8eed3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.263+0000 7f84e4fca640 1 -- 172.21.15.44:0/1811409435 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eec7430 msgr2=0x563a8eeb7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:08.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.263+0000 7f84e4fca640 1 --2- 172.21.15.44:0/1811409435 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eec7430 0x563a8eeb7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.263+0000 7f84e4fca640 1 -- 172.21.15.44:0/1811409435 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eebe850 msgr2=0x563a8eed3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:08.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.263+0000 7f84e4fca640 1 --2- 172.21.15.44:0/1811409435 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eebe850 0x563a8eed3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.263+0000 7f84e4fca640 1 -- 172.21.15.44:0/1811409435 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a8eec9820 con 0x563a8eebdf30 2024-04-21T23:12:08.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.263+0000 7f84dffff640 1 --2- 172.21.15.44:0/1811409435 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eebe850 0x563a8eed3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:12:08.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.264+0000 7f84e4fca640 1 --2- 172.21.15.44:0/1811409435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eebdf30 0x563a8eec6bd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f84d4099510 tx=0x7f84d4003980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec42490a368d411 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:08.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.264+0000 7f84ca7fc640 1 -- 172.21.15.44:0/1811409435 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d40b5070 con 0x563a8eebdf30 2024-04-21T23:12:08.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.264+0000 7f84ca7fc640 1 -- 172.21.15.44:0/1811409435 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84d40a5d20 con 0x563a8eebdf30 2024-04-21T23:12:08.265 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.264+0000 7f84ca7fc640 1 -- 172.21.15.44:0/1811409435 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d40a27e0 con 0x563a8eebdf30 2024-04-21T23:12:08.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.265+0000 7f84e9e12740 1 -- 172.21.15.44:0/1811409435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eebdf30 msgr2=0x563a8eec6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.265+0000 7f84e9e12740 1 --2- 172.21.15.44:0/1811409435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eebdf30 0x563a8eec6bd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f84d4099510 tx=0x7f84d4003980 comp rx=0 tx=0).stop 2024-04-21T23:12:08.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.265+0000 7f84e9e12740 1 -- 172.21.15.44:0/1811409435 shutdown_connections 2024-04-21T23:12:08.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.265+0000 7f84e9e12740 1 --2- 172.21.15.44:0/1811409435 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eec7430 0x563a8eeb7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.265+0000 7f84e9e12740 1 --2- 172.21.15.44:0/1811409435 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eebe850 0x563a8eed3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.265+0000 7f84e9e12740 1 --2- 172.21.15.44:0/1811409435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eebdf30 0x563a8eec6bd0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.265+0000 7f84e9e12740 1 -- 172.21.15.44:0/1811409435 >> 172.21.15.44:0/1811409435 conn(0x563a8eeb8660 msgr2=0x563a8eeb6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:08.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.265+0000 7f84e9e12740 1 -- 172.21.15.44:0/1811409435 shutdown_connections 2024-04-21T23:12:08.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.265+0000 7f84e9e12740 1 -- 172.21.15.44:0/1811409435 wait complete. 2024-04-21T23:12:08.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.265+0000 7f84e9e12740 1 -- start start 2024-04-21T23:12:08.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.265+0000 7f84e9e12740 1 Processor -- start 2024-04-21T23:12:08.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.266+0000 7f84e9e12740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eebdf30 0x563a8eecf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.266+0000 7f84e9e12740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eebe850 0x563a8eecab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.266+0000 7f84e9e12740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eec7430 0x563a8eecb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.266+0000 7f84e9e12740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f84e01ab710 con 0x563a8eebdf30 2024-04-21T23:12:08.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.266+0000 7f84e9e12740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f84e01acb80 con 0x563a8eec7430 2024-04-21T23:12:08.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.266+0000 7f84e9e12740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f84e01abde0 con 0x563a8eebe850 2024-04-21T23:12:08.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.266+0000 7f84e4fca640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eebdf30 0x563a8eecf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.266+0000 7f84e4fca640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eebdf30 0x563a8eecf7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43258/0 (socket says 172.21.15.44:43258) 2024-04-21T23:12:08.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.266+0000 7f84e4fca640 1 -- 172.21.15.44:0/3110929759 learned_addr learned my addr 172.21.15.44:0/3110929759 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:08.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.266+0000 7f84e9e12740 10 client.0 _ll_register_callbacks cb 0x7f84e006b000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:08.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.266+0000 7f84dffff640 1 --2- 172.21.15.44:0/3110929759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eebe850 0x563a8eecab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.266+0000 7f84e57cb640 1 --2- 172.21.15.44:0/3110929759 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eec7430 0x563a8eecb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.266+0000 7f84e57cb640 1 -- 172.21.15.44:0/3110929759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eebe850 msgr2=0x563a8eecab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:08.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.266+0000 7f84e57cb640 1 --2- 172.21.15.44:0/3110929759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eebe850 0x563a8eecab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.266+0000 7f84e57cb640 1 -- 172.21.15.44:0/3110929759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eebdf30 msgr2=0x563a8eecf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.266+0000 7f84e57cb640 1 --2- 172.21.15.44:0/3110929759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eebdf30 0x563a8eecf7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.266+0000 7f84e57cb640 1 -- 172.21.15.44:0/3110929759 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a8eecb6b0 con 0x563a8eec7430 2024-04-21T23:12:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.266+0000 7f84e4fca640 1 --2- 172.21.15.44:0/3110929759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eebdf30 0x563a8eecf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.266+0000 7f84dffff640 1 --2- 172.21.15.44:0/3110929759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eebe850 0x563a8eecab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.267+0000 7f84e57cb640 1 --2- 172.21.15.44:0/3110929759 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eec7430 0x563a8eecb070 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f84d8012410 tx=0x7f84d8004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d1a9720e507eed2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.267+0000 7f84bffff640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:12:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.267+0000 7f84bffff640 1 -- 172.21.15.44:0/3110929759 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d8004b20 con 0x563a8eec7430 2024-04-21T23:12:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.267+0000 7f84bffff640 1 -- 172.21.15.44:0/3110929759 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f84d801b070 con 0x563a8eec7430 2024-04-21T23:12:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.267+0000 7f84e9e12740 1 -- 172.21.15.44:0/3110929759 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84d40a5d20 con 0x563a8eec7430 2024-04-21T23:12:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.267+0000 7f84bffff640 1 -- 172.21.15.44:0/3110929759 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f84d8004cc0 con 0x563a8eec7430 2024-04-21T23:12:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.267+0000 7f84e9e12740 10 client.25949 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.267+0000 7f84e9e12740 1 -- 172.21.15.44:0/3110929759 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563a8ef41ac0 con 0x563a8eec7430 2024-04-21T23:12:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.268+0000 7f84bffff640 1 -- 172.21.15.44:0/3110929759 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84d40a5d20 con 0x563a8eec7430 2024-04-21T23:12:08.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.268+0000 7f84bffff640 1 -- 172.21.15.44:0/3110929759 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f84d801e050 con 0x563a8eec7430 2024-04-21T23:12:08.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.268+0000 7f84e9e12740 10 client.25949 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:08.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.268+0000 7f84e9e12740 10 client.25949 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:08.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.268+0000 7f84e9e12740 20 client.25949 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:08.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.268+0000 7f84e9e12740 1 --2- 172.21.15.44:0/3110929759 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563a8ef422c0 0x563a8ef446e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.268+0000 7f84e9e12740 4 client.25949 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:12:08.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.268+0000 7f84e9e12740 1 -- 172.21.15.44:0/3110929759 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563a8eda75a0 con 0x563a8ef422c0 2024-04-21T23:12:08.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.268+0000 7f84e4fca640 1 --2- 172.21.15.44:0/3110929759 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563a8ef422c0 0x563a8ef446e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.268+0000 7f84bffff640 10 client.25949 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:08.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.269+0000 7f84e4fca640 1 --2- 172.21.15.44:0/3110929759 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563a8ef422c0 0x563a8ef446e0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81a0957ba4ebb8b5 server_cookie=d928d13328dd42a1 in_seq=0 out_seq=0 2024-04-21T23:12:08.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.269+0000 7f84bffff640 1 -- 172.21.15.44:0/3110929759 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563a8eda75a0 con 0x563a8ef422c0 2024-04-21T23:12:08.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.269+0000 7f84bffff640 10 client.25949 handle_command_reply: tid=0 2024-04-21T23:12:08.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.269+0000 7f84e9e12740 1 client.25949 shutdown 2024-04-21T23:12:08.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.269+0000 7f84e9e12740 1 -- 172.21.15.44:0/3110929759 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eec7430 msgr2=0x563a8eecb070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.269+0000 7f84e9e12740 1 --2- 172.21.15.44:0/3110929759 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eec7430 0x563a8eecb070 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f84d8012410 tx=0x7f84d8004ab0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.269+0000 7f84e9e12740 1 -- 172.21.15.44:0/3110929759 shutdown_connections 2024-04-21T23:12:08.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.269+0000 7f84e9e12740 1 --2- 172.21.15.44:0/3110929759 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563a8ef422c0 0x563a8ef446e0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.269+0000 7f84e9e12740 1 --2- 172.21.15.44:0/3110929759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563a8eebe850 0x563a8eecab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.269+0000 7f84e9e12740 1 --2- 172.21.15.44:0/3110929759 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563a8eec7430 0x563a8eecb070 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.269+0000 7f84e9e12740 1 --2- 172.21.15.44:0/3110929759 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563a8eebdf30 0x563a8eecf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.269+0000 7f84bffff640 0 client.25949 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:08.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.269+0000 7f84e9e12740 1 -- 172.21.15.44:0/3110929759 >> 172.21.15.44:0/3110929759 conn(0x563a8eeb8660 msgr2=0x563a8eeb76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:08.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.269+0000 7f84e9e12740 1 -- 172.21.15.44:0/3110929759 shutdown_connections 2024-04-21T23:12:08.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.269+0000 7f84e9e12740 1 -- 172.21.15.44:0/3110929759 wait complete. 2024-04-21T23:12:08.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.269+0000 7f84e9e12740 20 client.25949 trim_cache size 0 max 16384 2024-04-21T23:12:08.274 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:12:08.275 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-21T23:12:08.275 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-21T23:12:08.275 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:12:08.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.270+0000 7f84e722e640 1 -- 172.21.15.44:0/1697249691 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f84d00a2d00 msgr2=0x7f84d00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.270+0000 7f84e722e640 1 --2- 172.21.15.44:0/1697249691 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f84d00a2d00 0x7f84d00a51c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x563a8eeb97f0 tx=0x7f84d8009540 comp rx=0 tx=0).stop 2024-04-21T23:12:08.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.270+0000 7f84e722e640 1 -- 172.21.15.44:0/1697249691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e0100610 msgr2=0x7f84e0196e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.270+0000 7f84e722e640 1 --2- 172.21.15.44:0/1697249691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e0100610 0x7f84e0196e90 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f84d400ead0 tx=0x7f84d400ef90 comp rx=0 tx=0).stop 2024-04-21T23:12:08.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.270+0000 7f84e722e640 1 -- 172.21.15.44:0/1697249691 shutdown_connections 2024-04-21T23:12:08.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.270+0000 7f84e722e640 1 --2- 172.21.15.44:0/1697249691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84e01066b0 0x7f84e019b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.270+0000 7f84e722e640 1 --2- 172.21.15.44:0/1697249691 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f84d00a2d00 0x7f84d00a51c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.270+0000 7f84e722e640 1 --2- 172.21.15.44:0/1697249691 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f84e0100fd0 0x7f84e01973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.270+0000 7f84e722e640 1 --2- 172.21.15.44:0/1697249691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f84e0100610 0x7f84e0196e90 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.270+0000 7f84e722e640 1 -- 172.21.15.44:0/1697249691 >> 172.21.15.44:0/1697249691 conn(0x7f84e00777f0 msgr2=0x7f84e00ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:08.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.270+0000 7f84e722e640 1 -- 172.21.15.44:0/1697249691 shutdown_connections 2024-04-21T23:12:08.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.270+0000 7f84e722e640 1 -- 172.21.15.44:0/1697249691 wait complete. 2024-04-21T23:12:08.280 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:12:08.280 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:12:08.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.428+0000 7f2bf1d65640 1 -- 172.21.15.44:0/2304835388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2bec107810 msgr2=0x7f2bec107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.428+0000 7f2bf1d65640 1 --2- 172.21.15.44:0/2304835388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2bec107810 0x7f2bec107bf0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc009900 tx=0x7f2bdc02bcd0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.428+0000 7f2bf1d65640 1 -- 172.21.15.44:0/2304835388 shutdown_connections 2024-04-21T23:12:08.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.428+0000 7f2bf1d65640 1 --2- 172.21.15.44:0/2304835388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2bec0693e0 0x7f2bec10e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.428+0000 7f2bf1d65640 1 --2- 172.21.15.44:0/2304835388 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bec068a40 0x7f2bec068ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.428+0000 7f2bf1d65640 1 --2- 172.21.15.44:0/2304835388 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2bec107810 0x7f2bec107bf0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.428+0000 7f2bf1d65640 1 -- 172.21.15.44:0/2304835388 >> 172.21.15.44:0/2304835388 conn(0x7f2bec077430 msgr2=0x7f2bec077840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:08.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.429+0000 7f2bf1d65640 1 -- 172.21.15.44:0/2304835388 shutdown_connections 2024-04-21T23:12:08.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.429+0000 7f2bf1d65640 1 -- 172.21.15.44:0/2304835388 wait complete. 2024-04-21T23:12:08.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.429+0000 7f2bf1d65640 1 Processor -- start 2024-04-21T23:12:08.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.429+0000 7f2bf1d65640 1 -- start start 2024-04-21T23:12:08.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.429+0000 7f2bf1d65640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bec068a40 0x7f2bec197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.429+0000 7f2bf1d65640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2bec0693e0 0x7f2bec1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.429+0000 7f2bf1d65640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2bec107810 0x7f2bec19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.429+0000 7f2bf1d65640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2bec111b50 con 0x7f2bec0693e0 2024-04-21T23:12:08.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.429+0000 7f2bf1d65640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2bec1119d0 con 0x7f2bec068a40 2024-04-21T23:12:08.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.429+0000 7f2bf1d65640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2bec111cd0 con 0x7f2bec107810 2024-04-21T23:12:08.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.430+0000 7f2beaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2bec0693e0 0x7f2bec1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.430+0000 7f2beb7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bec068a40 0x7f2bec197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.430+0000 7f2bebfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2bec107810 0x7f2bec19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.430+0000 7f2beb7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bec068a40 0x7f2bec197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:40034/0 (socket says 172.21.15.44:40034) 2024-04-21T23:12:08.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.430+0000 7f2beb7fe640 1 -- 172.21.15.44:0/3531742756 learned_addr learned my addr 172.21.15.44:0/3531742756 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:08.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.430+0000 7f2bebfff640 1 -- 172.21.15.44:0/3531742756 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bec068a40 msgr2=0x7f2bec197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.430+0000 7f2bebfff640 1 --2- 172.21.15.44:0/3531742756 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bec068a40 0x7f2bec197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.430+0000 7f2bebfff640 1 -- 172.21.15.44:0/3531742756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2bec0693e0 msgr2=0x7f2bec1984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.430+0000 7f2bebfff640 1 --2- 172.21.15.44:0/3531742756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2bec0693e0 0x7f2bec1984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.430+0000 7f2bebfff640 1 -- 172.21.15.44:0/3531742756 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bec19cf60 con 0x7f2bec107810 2024-04-21T23:12:08.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.430+0000 7f2beaffd640 1 --2- 172.21.15.44:0/3531742756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2bec0693e0 0x7f2bec1984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:08.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.430+0000 7f2beb7fe640 1 --2- 172.21.15.44:0/3531742756 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bec068a40 0x7f2bec197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:08.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.430+0000 7f2bebfff640 1 --2- 172.21.15.44:0/3531742756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2bec107810 0x7f2bec19c830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2be400e940 tx=0x7f2be400ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:08.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.430+0000 7f2be8ff9640 1 -- 172.21.15.44:0/3531742756 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be4004d20 con 0x7f2bec107810 2024-04-21T23:12:08.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.430+0000 7f2be8ff9640 1 -- 172.21.15.44:0/3531742756 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be4005020 con 0x7f2bec107810 2024-04-21T23:12:08.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.430+0000 7f2bf1d65640 1 -- 172.21.15.44:0/3531742756 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bec19d250 con 0x7f2bec107810 2024-04-21T23:12:08.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.430+0000 7f2be8ff9640 1 -- 172.21.15.44:0/3531742756 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be40116b0 con 0x7f2bec107810 2024-04-21T23:12:08.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.430+0000 7f2bf1d65640 1 -- 172.21.15.44:0/3531742756 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bec1a4ae0 con 0x7f2bec107810 2024-04-21T23:12:08.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.431+0000 7f2bf1d65640 1 -- 172.21.15.44:0/3531742756 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bec0685f0 con 0x7f2bec107810 2024-04-21T23:12:08.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.432+0000 7f2be8ff9640 1 -- 172.21.15.44:0/3531742756 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f2be40054d0 con 0x7f2bec107810 2024-04-21T23:12:08.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.433+0000 7f2be8ff9640 1 --2- 172.21.15.44:0/3531742756 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f2bd40a2d00 0x7f2bd40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.433+0000 7f2beb7fe640 1 --2- 172.21.15.44:0/3531742756 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f2bd40a2d00 0x7f2bd40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.433+0000 7f2be8ff9640 1 -- 172.21.15.44:0/3531742756 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f2be40cfcc0 con 0x7f2bec107810 2024-04-21T23:12:08.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.434+0000 7f2beb7fe640 1 --2- 172.21.15.44:0/3531742756 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f2bd40a2d00 0x7f2bd40a51c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc0096c0 tx=0x7f2bdc0058d0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:08.436 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.437+0000 7f2be8ff9640 1 -- 172.21.15.44:0/3531742756 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f2be4016030 con 0x7f2bec107810 2024-04-21T23:12:08.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.666+0000 7f2bf1d65640 1 -- 172.21.15.44:0/3531742756 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2bec02d170 con 0x7f2bec107810 2024-04-21T23:12:08.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.666+0000 7f2be8ff9640 1 -- 172.21.15.44:0/3531742756 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11791 (secure 0 0 0) 0x7f2be409be90 con 0x7f2bec107810 2024-04-21T23:12:08.666 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:12:08.666 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:11:24.214378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25843":{"gid":25843,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3162569119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3162569119},{"type":"v1","addr":"172.21.15.69:6839","nonce":3162569119}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:12:08.667 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-04-21T23:12:08.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.669+0000 7f2bf1d65640 1 -- 172.21.15.44:0/3531742756 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f2bd40a2d00 msgr2=0x7f2bd40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.669+0000 7f2bf1d65640 1 --2- 172.21.15.44:0/3531742756 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f2bd40a2d00 0x7f2bd40a51c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc0096c0 tx=0x7f2bdc0058d0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.669+0000 7f2bf1d65640 1 -- 172.21.15.44:0/3531742756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2bec107810 msgr2=0x7f2bec19c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.669+0000 7f2bf1d65640 1 --2- 172.21.15.44:0/3531742756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2bec107810 0x7f2bec19c830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2be400e940 tx=0x7f2be400ee00 comp rx=0 tx=0).stop 2024-04-21T23:12:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.669+0000 7f2bf1d65640 1 -- 172.21.15.44:0/3531742756 shutdown_connections 2024-04-21T23:12:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.669+0000 7f2bf1d65640 1 --2- 172.21.15.44:0/3531742756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2bec107810 0x7f2bec19c830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.669+0000 7f2bf1d65640 1 --2- 172.21.15.44:0/3531742756 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f2bd40a2d00 0x7f2bd40a51c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.669+0000 7f2bf1d65640 1 --2- 172.21.15.44:0/3531742756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2bec0693e0 0x7f2bec1984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.669+0000 7f2bf1d65640 1 --2- 172.21.15.44:0/3531742756 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2bec068a40 0x7f2bec197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.669+0000 7f2bf1d65640 1 -- 172.21.15.44:0/3531742756 >> 172.21.15.44:0/3531742756 conn(0x7f2bec077430 msgr2=0x7f2bec0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.669+0000 7f2bf1d65640 1 -- 172.21.15.44:0/3531742756 shutdown_connections 2024-04-21T23:12:08.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.669+0000 7f2bf1d65640 1 -- 172.21.15.44:0/3531742756 wait complete. 2024-04-21T23:12:08.680 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:12:08.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 307394 14.17 MB/sec warmup 118 sec latency 16.729 ms 2024-04-21T23:12:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.798+0000 7f31197cb640 1 -- 172.21.15.44:0/182724959 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3114100610 msgr2=0x7f3114100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.798+0000 7f31197cb640 1 --2- 172.21.15.44:0/182724959 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3114100610 0x7f3114100a90 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f3108009930 tx=0x7f310802bc30 comp rx=0 tx=0).stop 2024-04-21T23:12:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.798+0000 7f31197cb640 1 -- 172.21.15.44:0/182724959 shutdown_connections 2024-04-21T23:12:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.798+0000 7f31197cb640 1 --2- 172.21.15.44:0/182724959 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3114100fd0 0x7f311410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.798+0000 7f31197cb640 1 --2- 172.21.15.44:0/182724959 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3114100610 0x7f3114100a90 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.798+0000 7f31197cb640 1 --2- 172.21.15.44:0/182724959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31141066b0 0x7f3114106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.798+0000 7f31197cb640 1 -- 172.21.15.44:0/182724959 >> 172.21.15.44:0/182724959 conn(0x7f31140777f0 msgr2=0x7f3114077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:08.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.798+0000 7f31197cb640 1 -- 172.21.15.44:0/182724959 shutdown_connections 2024-04-21T23:12:08.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.798+0000 7f31197cb640 1 -- 172.21.15.44:0/182724959 wait complete. 2024-04-21T23:12:08.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.799+0000 7f31197cb640 1 Processor -- start 2024-04-21T23:12:08.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.799+0000 7f31197cb640 1 -- start start 2024-04-21T23:12:08.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.799+0000 7f31197cb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3114100610 0x7f3114196eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.799+0000 7f31197cb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3114100fd0 0x7f31141973f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.799+0000 7f31197cb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31141066b0 0x7f311419b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.799+0000 7f31197cb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f31141109f0 con 0x7f3114100610 2024-04-21T23:12:08.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.799+0000 7f31197cb640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3114110870 con 0x7f3114100fd0 2024-04-21T23:12:08.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.799+0000 7f31197cb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3114110b70 con 0x7f31141066b0 2024-04-21T23:12:08.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.799+0000 7f3112ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3114100610 0x7f3114196eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.799+0000 7f3112ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3114100610 0x7f3114196eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43298/0 (socket says 172.21.15.44:43298) 2024-04-21T23:12:08.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.799+0000 7f31137fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31141066b0 0x7f311419b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.799+0000 7f3112ffd640 1 -- 172.21.15.44:0/1637808298 learned_addr learned my addr 172.21.15.44:0/1637808298 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:08.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.799+0000 7f31127fc640 1 --2- 172.21.15.44:0/1637808298 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3114100fd0 0x7f31141973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.800+0000 7f3112ffd640 1 -- 172.21.15.44:0/1637808298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31141066b0 msgr2=0x7f311419b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.800+0000 7f3112ffd640 1 --2- 172.21.15.44:0/1637808298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31141066b0 0x7f311419b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.800+0000 7f3112ffd640 1 -- 172.21.15.44:0/1637808298 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3114100fd0 msgr2=0x7f31141973f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.800+0000 7f3112ffd640 1 --2- 172.21.15.44:0/1637808298 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3114100fd0 0x7f31141973f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.800+0000 7f3112ffd640 1 -- 172.21.15.44:0/1637808298 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f311419bea0 con 0x7f3114100610 2024-04-21T23:12:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.800+0000 7f31137fe640 1 --2- 172.21.15.44:0/1637808298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31141066b0 0x7f311419b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.800+0000 7f3112ffd640 1 --2- 172.21.15.44:0/1637808298 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3114100610 0x7f3114196eb0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f310400e9c0 tx=0x7f310400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.800+0000 7f30fbfff640 1 -- 172.21.15.44:0/1637808298 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310400cd80 con 0x7f3114100610 2024-04-21T23:12:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.800+0000 7f31197cb640 1 -- 172.21.15.44:0/1637808298 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f311419c190 con 0x7f3114100610 2024-04-21T23:12:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.800+0000 7f30fbfff640 1 -- 172.21.15.44:0/1637808298 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31040035f0 con 0x7f3114100610 2024-04-21T23:12:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.800+0000 7f30fbfff640 1 -- 172.21.15.44:0/1637808298 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3104010690 con 0x7f3114100610 2024-04-21T23:12:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.800+0000 7f31197cb640 1 -- 172.21.15.44:0/1637808298 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31141a3a20 con 0x7f3114100610 2024-04-21T23:12:08.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.802+0000 7f30fbfff640 1 -- 172.21.15.44:0/1637808298 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f31040037a0 con 0x7f3114100610 2024-04-21T23:12:08.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f311c3b0740 1 Processor -- start 2024-04-21T23:12:08.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f311c3b0740 1 -- start start 2024-04-21T23:12:08.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f30fbfff640 1 --2- 172.21.15.44:0/1637808298 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f30fc0a2d00 0x7f30fc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f311c3b0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db27f9a00 0x558db27f9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f311c3b0740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27fa3b0 0x558db28051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f311c3b0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db2805730 0x558db2807b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f311c3b0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558db275e530 con 0x558db27f9a00 2024-04-21T23:12:08.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f311c3b0740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x558db2706830 con 0x558db27fa3b0 2024-04-21T23:12:08.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f311c3b0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558db275a4f0 con 0x558db2805730 2024-04-21T23:12:08.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f3112ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db27f9a00 0x558db27f9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f31137fe640 1 --2- 172.21.15.44:0/1637808298 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f30fc0a2d00 0x7f30fc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f31127fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27fa3b0 0x558db28051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f31127fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27fa3b0 0x558db28051f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:40064/0 (socket says 172.21.15.44:40064) 2024-04-21T23:12:08.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f31127fc640 1 -- 172.21.15.44:0/1245791212 learned_addr learned my addr 172.21.15.44:0/1245791212 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:08.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f31137fe640 1 --2- 172.21.15.44:0/1245791212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db2805730 0x558db2807b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f30fbfff640 1 -- 172.21.15.44:0/1637808298 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f31040cfb10 con 0x7f3114100610 2024-04-21T23:12:08.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f31127fc640 1 -- 172.21.15.44:0/1245791212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db2805730 msgr2=0x558db2807b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:08.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f31127fc640 1 --2- 172.21.15.44:0/1245791212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db2805730 0x558db2807b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f31127fc640 1 -- 172.21.15.44:0/1245791212 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db27f9a00 msgr2=0x558db27f9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:08.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f31127fc640 1 --2- 172.21.15.44:0/1245791212 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db27f9a00 0x558db27f9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f31127fc640 1 -- 172.21.15.44:0/1245791212 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558db272a2c0 con 0x558db27fa3b0 2024-04-21T23:12:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f3112ffd640 1 --2- 172.21.15.44:0/1245791212 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db27f9a00 0x558db27f9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f31137fe640 1 --2- 172.21.15.44:0/1245791212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db2805730 0x558db2807b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.803+0000 7f31137fe640 1 --2- 172.21.15.44:0/1637808298 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f30fc0a2d00 0x7f30fc0a51c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f310c0098e0 tx=0x7f310c012650 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.804+0000 7f31127fc640 1 --2- 172.21.15.44:0/1245791212 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27fa3b0 0x558db28051f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f310802cfd0 tx=0x7f3108005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d49d660a00db5dd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.804+0000 7f30f97fa640 1 -- 172.21.15.44:0/1245791212 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3108042070 con 0x558db27fa3b0 2024-04-21T23:12:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.804+0000 7f30f97fa640 1 -- 172.21.15.44:0/1245791212 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3108004030 con 0x558db27fa3b0 2024-04-21T23:12:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.804+0000 7f30f97fa640 1 -- 172.21.15.44:0/1245791212 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3108004c90 con 0x558db27fa3b0 2024-04-21T23:12:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.804+0000 7f311c3b0740 1 -- 172.21.15.44:0/1245791212 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27fa3b0 msgr2=0x558db28051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.804+0000 7f311c3b0740 1 --2- 172.21.15.44:0/1245791212 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27fa3b0 0x558db28051f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f310802cfd0 tx=0x7f3108005d50 comp rx=0 tx=0).stop 2024-04-21T23:12:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f311c3b0740 1 -- 172.21.15.44:0/1245791212 shutdown_connections 2024-04-21T23:12:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f311c3b0740 1 --2- 172.21.15.44:0/1245791212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db2805730 0x558db2807b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f311c3b0740 1 --2- 172.21.15.44:0/1245791212 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27fa3b0 0x558db28051f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f311c3b0740 1 --2- 172.21.15.44:0/1245791212 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db27f9a00 0x558db27f9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f311c3b0740 1 -- 172.21.15.44:0/1245791212 >> 172.21.15.44:0/1245791212 conn(0x558db27f4750 msgr2=0x558db27f4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f311c3b0740 1 -- 172.21.15.44:0/1245791212 shutdown_connections 2024-04-21T23:12:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f311c3b0740 1 -- 172.21.15.44:0/1245791212 wait complete. 2024-04-21T23:12:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f311c3b0740 1 -- start start 2024-04-21T23:12:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f311c3b0740 1 Processor -- start 2024-04-21T23:12:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f311c3b0740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27f9a00 0x558db2805080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f311c3b0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db27fa3b0 0x558db2802410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f311c3b0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db2805730 0x558db2802970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f311c3b0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558db2808490 con 0x558db27fa3b0 2024-04-21T23:12:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f311c3b0740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x558db2719f10 con 0x558db27f9a00 2024-04-21T23:12:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f311c3b0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558db2808610 con 0x558db2805730 2024-04-21T23:12:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f311c3b0740 10 client.0 _ll_register_callbacks cb 0x558db2471be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f31127fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db27fa3b0 0x558db2802410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f31127fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db27fa3b0 0x558db2802410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43310/0 (socket says 172.21.15.44:43310) 2024-04-21T23:12:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f31127fc640 1 -- 172.21.15.44:0/1601511383 learned_addr learned my addr 172.21.15.44:0/1601511383 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f31137fe640 1 --2- 172.21.15.44:0/1601511383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db2805730 0x558db2802970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.805+0000 7f3112ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27f9a00 0x558db2805080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.806+0000 7f31127fc640 1 -- 172.21.15.44:0/1601511383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db2805730 msgr2=0x558db2802970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:08.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.806+0000 7f31127fc640 1 --2- 172.21.15.44:0/1601511383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db2805730 0x558db2802970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.806+0000 7f31127fc640 1 -- 172.21.15.44:0/1601511383 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27f9a00 msgr2=0x558db2805080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:08.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.806+0000 7f31127fc640 1 --2- 172.21.15.44:0/1601511383 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27f9a00 0x558db2805080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.806+0000 7f31127fc640 1 -- 172.21.15.44:0/1601511383 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558db26ebc80 con 0x558db27fa3b0 2024-04-21T23:12:08.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.806+0000 7f31137fe640 1 --2- 172.21.15.44:0/1601511383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db2805730 0x558db2802970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:08.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.806+0000 7f31127fc640 1 --2- 172.21.15.44:0/1601511383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db27fa3b0 0x558db2802410 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f3108005ab0 tx=0x7f3108004be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7885df41582721a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:08.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.806+0000 7f30f6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:12:08.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.806+0000 7f30f6ff5640 1 -- 172.21.15.44:0/1601511383 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3108042040 con 0x558db27fa3b0 2024-04-21T23:12:08.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.806+0000 7f30f6ff5640 1 -- 172.21.15.44:0/1601511383 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f310802ca90 con 0x558db27fa3b0 2024-04-21T23:12:08.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.806+0000 7f311c3b0740 1 -- 172.21.15.44:0/1601511383 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3108004030 con 0x558db27fa3b0 2024-04-21T23:12:08.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.806+0000 7f30f6ff5640 1 -- 172.21.15.44:0/1601511383 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3108058050 con 0x558db27fa3b0 2024-04-21T23:12:08.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.806+0000 7f311c3b0740 10 client.16401 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:08.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.806+0000 7f311c3b0740 1 -- 172.21.15.44:0/1601511383 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558db24723f0 con 0x558db27fa3b0 2024-04-21T23:12:08.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.806+0000 7f30f6ff5640 1 -- 172.21.15.44:0/1601511383 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3108004030 con 0x558db27fa3b0 2024-04-21T23:12:08.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.807+0000 7f30f6ff5640 1 -- 172.21.15.44:0/1601511383 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f3108003710 con 0x558db27fa3b0 2024-04-21T23:12:08.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.807+0000 7f311c3b0740 10 client.16401 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:08.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.807+0000 7f311c3b0740 10 client.16401 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:08.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.807+0000 7f311c3b0740 20 client.16401 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:08.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.807+0000 7f311c3b0740 1 --2- 172.21.15.44:0/1601511383 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558db28155f0 0x558db28179e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.807+0000 7f311c3b0740 4 client.16401 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:12:08.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.807+0000 7f311c3b0740 1 -- 172.21.15.44:0/1601511383 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558db276ba70 con 0x558db28155f0 2024-04-21T23:12:08.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.807+0000 7f3112ffd640 1 --2- 172.21.15.44:0/1601511383 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558db28155f0 0x558db28179e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.807+0000 7f3112ffd640 1 --2- 172.21.15.44:0/1601511383 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558db28155f0 0x558db28179e0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32ca9456d8781d8c server_cookie=f5cc2a4586a996da in_seq=0 out_seq=0 2024-04-21T23:12:08.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.807+0000 7f30f6ff5640 10 client.16401 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:08.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.808+0000 7f30f6ff5640 1 -- 172.21.15.44:0/1601511383 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558db276ba70 con 0x558db28155f0 2024-04-21T23:12:08.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.808+0000 7f30f6ff5640 10 client.16401 handle_command_reply: tid=0 2024-04-21T23:12:08.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.808+0000 7f311c3b0740 1 client.16401 shutdown 2024-04-21T23:12:08.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.809+0000 7f311c3b0740 1 -- 172.21.15.44:0/1601511383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db27fa3b0 msgr2=0x558db2802410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.809+0000 7f311c3b0740 1 --2- 172.21.15.44:0/1601511383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db27fa3b0 0x558db2802410 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f3108005ab0 tx=0x7f3108004be0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.809+0000 7f311c3b0740 1 -- 172.21.15.44:0/1601511383 shutdown_connections 2024-04-21T23:12:08.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.809+0000 7f311c3b0740 1 --2- 172.21.15.44:0/1601511383 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558db28155f0 0x558db28179e0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.809+0000 7f311c3b0740 1 --2- 172.21.15.44:0/1601511383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db2805730 0x558db2802970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.809+0000 7f311c3b0740 1 --2- 172.21.15.44:0/1601511383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db27fa3b0 0x558db2802410 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.809+0000 7f311c3b0740 1 --2- 172.21.15.44:0/1601511383 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27f9a00 0x558db2805080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.809+0000 7f30f6ff5640 0 client.16401 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:08.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.809+0000 7f311c3b0740 1 -- 172.21.15.44:0/1601511383 >> 172.21.15.44:0/1601511383 conn(0x558db27f4750 msgr2=0x558db27f64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:08.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.809+0000 7f311c3b0740 1 -- 172.21.15.44:0/1601511383 shutdown_connections 2024-04-21T23:12:08.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.809+0000 7f311c3b0740 1 -- 172.21.15.44:0/1601511383 wait complete. 2024-04-21T23:12:08.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.809+0000 7f311c3b0740 20 client.16401 trim_cache size 0 max 16384 2024-04-21T23:12:08.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.822+0000 7f311c3b0740 1 Processor -- start 2024-04-21T23:12:08.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.822+0000 7f311c3b0740 1 -- start start 2024-04-21T23:12:08.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.823+0000 7f311c3b0740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27f9a00 0x558db27f62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.823+0000 7f311c3b0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db27fa3b0 0x558db2804b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.823+0000 7f311c3b0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db2805730 0x558db2805080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.823+0000 7f311c3b0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f31141aa8d0 con 0x558db2805730 2024-04-21T23:12:08.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.823+0000 7f311c3b0740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f31141ad150 con 0x558db27f9a00 2024-04-21T23:12:08.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.823+0000 7f311c3b0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f31141aaf40 con 0x558db27fa3b0 2024-04-21T23:12:08.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.823+0000 7f31137fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db2805730 0x558db2805080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.823+0000 7f31137fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db2805730 0x558db2805080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43322/0 (socket says 172.21.15.44:43322) 2024-04-21T23:12:08.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.823+0000 7f31137fe640 1 -- 172.21.15.44:0/2184197735 learned_addr learned my addr 172.21.15.44:0/2184197735 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:08.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.823+0000 7f3112ffd640 1 --2- 172.21.15.44:0/2184197735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27f9a00 0x558db27f62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.823+0000 7f31127fc640 1 --2- 172.21.15.44:0/2184197735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db27fa3b0 0x558db2804b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.823+0000 7f31137fe640 1 -- 172.21.15.44:0/2184197735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db27fa3b0 msgr2=0x558db2804b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:08.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.823+0000 7f31137fe640 1 --2- 172.21.15.44:0/2184197735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db27fa3b0 0x558db2804b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.823+0000 7f31137fe640 1 -- 172.21.15.44:0/2184197735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27f9a00 msgr2=0x558db27f62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:08.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.823+0000 7f31137fe640 1 --2- 172.21.15.44:0/2184197735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27f9a00 0x558db27f62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.823+0000 7f31137fe640 1 -- 172.21.15.44:0/2184197735 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558db253c7d0 con 0x558db2805730 2024-04-21T23:12:08.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.823+0000 7f31137fe640 1 --2- 172.21.15.44:0/2184197735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db2805730 0x558db2805080 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f310c012aa0 tx=0x7f310c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a7d3cbebe4aafa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:08.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.823+0000 7f30f8ff9640 1 -- 172.21.15.44:0/2184197735 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c004040 con 0x558db2805730 2024-04-21T23:12:08.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.823+0000 7f30f8ff9640 1 -- 172.21.15.44:0/2184197735 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f310c0046c0 con 0x558db2805730 2024-04-21T23:12:08.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.823+0000 7f30f8ff9640 1 -- 172.21.15.44:0/2184197735 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c02cde0 con 0x558db2805730 2024-04-21T23:12:08.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.824+0000 7f311c3b0740 1 -- 172.21.15.44:0/2184197735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db2805730 msgr2=0x558db2805080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.824+0000 7f311c3b0740 1 --2- 172.21.15.44:0/2184197735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db2805730 0x558db2805080 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f310c012aa0 tx=0x7f310c004ab0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.824+0000 7f311c3b0740 1 -- 172.21.15.44:0/2184197735 shutdown_connections 2024-04-21T23:12:08.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.824+0000 7f311c3b0740 1 --2- 172.21.15.44:0/2184197735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db27fa3b0 0x558db2804b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.824+0000 7f311c3b0740 1 --2- 172.21.15.44:0/2184197735 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db2805730 0x558db2805080 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.824+0000 7f311c3b0740 1 --2- 172.21.15.44:0/2184197735 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27f9a00 0x558db27f62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.824+0000 7f311c3b0740 1 -- 172.21.15.44:0/2184197735 >> 172.21.15.44:0/2184197735 conn(0x558db27f4750 msgr2=0x558db27f5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:08.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.824+0000 7f311c3b0740 1 -- 172.21.15.44:0/2184197735 shutdown_connections 2024-04-21T23:12:08.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.824+0000 7f311c3b0740 1 -- 172.21.15.44:0/2184197735 wait complete. 2024-04-21T23:12:08.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.824+0000 7f311c3b0740 1 -- start start 2024-04-21T23:12:08.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.824+0000 7f311c3b0740 1 Processor -- start 2024-04-21T23:12:08.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f311c3b0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db27f9a00 0x558db281ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f311c3b0740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27fa3b0 0x558db28180c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f311c3b0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db2805730 0x558db2818600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f311c3b0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f31141ab610 con 0x558db27f9a00 2024-04-21T23:12:08.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f311c3b0740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f31141aca80 con 0x558db27fa3b0 2024-04-21T23:12:08.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f311c3b0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f31141abce0 con 0x558db2805730 2024-04-21T23:12:08.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f311c3b0740 10 client.0 _ll_register_callbacks cb 0x7f3104003e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:08.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f3112ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db27f9a00 0x558db281ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f3112ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db27f9a00 0x558db281ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43330/0 (socket says 172.21.15.44:43330) 2024-04-21T23:12:08.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f31127fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27fa3b0 0x558db28180c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f3112ffd640 1 -- 172.21.15.44:0/2154139877 learned_addr learned my addr 172.21.15.44:0/2154139877 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:08.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f31137fe640 1 --2- 172.21.15.44:0/2154139877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db2805730 0x558db2818600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f3112ffd640 1 -- 172.21.15.44:0/2154139877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db2805730 msgr2=0x558db2818600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:08.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f3112ffd640 1 --2- 172.21.15.44:0/2154139877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db2805730 0x558db2818600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f3112ffd640 1 -- 172.21.15.44:0/2154139877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27fa3b0 msgr2=0x558db28180c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:08.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f3112ffd640 1 --2- 172.21.15.44:0/2154139877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27fa3b0 0x558db28180c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f3112ffd640 1 -- 172.21.15.44:0/2154139877 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558db2818cf0 con 0x558db27f9a00 2024-04-21T23:12:08.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f31137fe640 1 --2- 172.21.15.44:0/2154139877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db2805730 0x558db2818600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:12:08.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f31127fc640 1 --2- 172.21.15.44:0/2154139877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27fa3b0 0x558db28180c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:08.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f3112ffd640 1 --2- 172.21.15.44:0/2154139877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db27f9a00 0x558db281ace0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f310409b930 tx=0x7f3104066d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af1dbd225664144a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:08.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f30f6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:12:08.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.825+0000 7f30f6ff5640 1 -- 172.21.15.44:0/2154139877 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3104086040 con 0x558db27f9a00 2024-04-21T23:12:08.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.826+0000 7f30f6ff5640 1 -- 172.21.15.44:0/2154139877 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f31040982a0 con 0x558db27f9a00 2024-04-21T23:12:08.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.826+0000 7f311c3b0740 1 -- 172.21.15.44:0/2154139877 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f310c0046c0 con 0x558db27f9a00 2024-04-21T23:12:08.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.826+0000 7f30f6ff5640 1 -- 172.21.15.44:0/2154139877 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f310409c5c0 con 0x558db27f9a00 2024-04-21T23:12:08.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.826+0000 7f311c3b0740 10 client.16407 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:08.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.826+0000 7f311c3b0740 1 -- 172.21.15.44:0/2154139877 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558db287b730 con 0x558db27f9a00 2024-04-21T23:12:08.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.826+0000 7f30f6ff5640 1 -- 172.21.15.44:0/2154139877 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f310c0046c0 con 0x558db27f9a00 2024-04-21T23:12:08.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.826+0000 7f30f6ff5640 1 -- 172.21.15.44:0/2154139877 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f31040cd0e0 con 0x558db27f9a00 2024-04-21T23:12:08.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.826+0000 7f311c3b0740 10 client.16407 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:08.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.826+0000 7f311c3b0740 10 client.16407 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:08.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.826+0000 7f311c3b0740 20 client.16407 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:08.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.826+0000 7f311c3b0740 1 --2- 172.21.15.44:0/2154139877 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558db287be80 0x558db287e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:08.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.826+0000 7f311c3b0740 4 client.16407 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:12:08.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.826+0000 7f311c3b0740 1 -- 172.21.15.44:0/2154139877 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558db2765dc0 con 0x558db287be80 2024-04-21T23:12:08.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.826+0000 7f31127fc640 1 --2- 172.21.15.44:0/2154139877 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558db287be80 0x558db287e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:08.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.827+0000 7f30f6ff5640 10 client.16407 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:08.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.827+0000 7f31127fc640 1 --2- 172.21.15.44:0/2154139877 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558db287be80 0x558db287e270 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddb56abfec148f68 server_cookie=5bd113198736f6a6 in_seq=0 out_seq=0 2024-04-21T23:12:08.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.827+0000 7f30f6ff5640 1 -- 172.21.15.44:0/2154139877 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558db2765dc0 con 0x558db287be80 2024-04-21T23:12:08.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.827+0000 7f30f6ff5640 10 client.16407 handle_command_reply: tid=0 2024-04-21T23:12:08.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.827+0000 7f311c3b0740 1 client.16407 shutdown 2024-04-21T23:12:08.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.827+0000 7f311c3b0740 1 -- 172.21.15.44:0/2154139877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db27f9a00 msgr2=0x558db281ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.827+0000 7f311c3b0740 1 --2- 172.21.15.44:0/2154139877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db27f9a00 0x558db281ace0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f310409b930 tx=0x7f3104066d00 comp rx=0 tx=0).stop 2024-04-21T23:12:08.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.827+0000 7f311c3b0740 1 -- 172.21.15.44:0/2154139877 shutdown_connections 2024-04-21T23:12:08.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.827+0000 7f311c3b0740 1 --2- 172.21.15.44:0/2154139877 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558db287be80 0x558db287e270 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.827+0000 7f311c3b0740 1 --2- 172.21.15.44:0/2154139877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558db2805730 0x558db2818600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.827+0000 7f311c3b0740 1 --2- 172.21.15.44:0/2154139877 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558db27fa3b0 0x558db28180c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.827+0000 7f311c3b0740 1 --2- 172.21.15.44:0/2154139877 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558db27f9a00 0x558db281ace0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.827+0000 7f30f6ff5640 0 client.16407 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:08.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.827+0000 7f311c3b0740 1 -- 172.21.15.44:0/2154139877 >> 172.21.15.44:0/2154139877 conn(0x558db27f4750 msgr2=0x558db2811160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:08.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.827+0000 7f311c3b0740 1 -- 172.21.15.44:0/2154139877 shutdown_connections 2024-04-21T23:12:08.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.827+0000 7f311c3b0740 1 -- 172.21.15.44:0/2154139877 wait complete. 2024-04-21T23:12:08.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.827+0000 7f311c3b0740 20 client.16407 trim_cache size 0 max 16384 2024-04-21T23:12:08.834 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-21T23:12:08.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.828+0000 7f31197cb640 1 -- 172.21.15.44:0/1637808298 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f30fc0a2d00 msgr2=0x7f30fc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.828+0000 7f31197cb640 1 --2- 172.21.15.44:0/1637808298 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f30fc0a2d00 0x7f30fc0a51c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f310c0098e0 tx=0x7f310c012650 comp rx=0 tx=0).stop 2024-04-21T23:12:08.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.828+0000 7f31197cb640 1 -- 172.21.15.44:0/1637808298 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3114100610 msgr2=0x7f3114196eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:08.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.828+0000 7f31197cb640 1 --2- 172.21.15.44:0/1637808298 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3114100610 0x7f3114196eb0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f310400e9c0 tx=0x7f310400ee80 comp rx=0 tx=0).stop 2024-04-21T23:12:08.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.828+0000 7f31197cb640 1 -- 172.21.15.44:0/1637808298 shutdown_connections 2024-04-21T23:12:08.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.828+0000 7f31197cb640 1 --2- 172.21.15.44:0/1637808298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31141066b0 0x7f311419b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.828+0000 7f31197cb640 1 --2- 172.21.15.44:0/1637808298 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f30fc0a2d00 0x7f30fc0a51c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.828+0000 7f31197cb640 1 --2- 172.21.15.44:0/1637808298 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3114100fd0 0x7f31141973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.828+0000 7f31197cb640 1 --2- 172.21.15.44:0/1637808298 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3114100610 0x7f3114196eb0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:08.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.828+0000 7f31197cb640 1 -- 172.21.15.44:0/1637808298 >> 172.21.15.44:0/1637808298 conn(0x7f31140777f0 msgr2=0x7f31140ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:08.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.828+0000 7f31197cb640 1 -- 172.21.15.44:0/1637808298 shutdown_connections 2024-04-21T23:12:08.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:08.828+0000 7f31197cb640 1 -- 172.21.15.44:0/1637808298 wait complete. 2024-04-21T23:12:08.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:08 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3531742756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:12:09.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:08 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3531742756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:12:09.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:08 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3531742756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:12:09.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 309592 14.16 MB/sec warmup 119 sec latency 16.023 ms 2024-04-21T23:12:09.839 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:12:09.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:12:09.839 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:12:09.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.969+0000 7fd8feefb640 1 -- 172.21.15.44:0/1707725455 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8f81032d0 msgr2=0x7fd8f810b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:09.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.969+0000 7fd8feefb640 1 --2- 172.21.15.44:0/1707725455 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8f81032d0 0x7fd8f810b7b0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009900 tx=0x7fd8ec02bcd0 comp rx=0 tx=0).stop 2024-04-21T23:12:09.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.969+0000 7fd8feefb640 1 -- 172.21.15.44:0/1707725455 shutdown_connections 2024-04-21T23:12:09.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.969+0000 7fd8feefb640 1 --2- 172.21.15.44:0/1707725455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8f810bcf0 0x7fd8f810e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:09.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.969+0000 7fd8feefb640 1 --2- 172.21.15.44:0/1707725455 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8f81032d0 0x7fd8f810b7b0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:09.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.969+0000 7fd8feefb640 1 --2- 172.21.15.44:0/1707725455 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8f81029b0 0x7fd8f8102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:09.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.969+0000 7fd8feefb640 1 -- 172.21.15.44:0/1707725455 >> 172.21.15.44:0/1707725455 conn(0x7fd8f80777f0 msgr2=0x7fd8f8077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:09.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.969+0000 7fd8feefb640 1 -- 172.21.15.44:0/1707725455 shutdown_connections 2024-04-21T23:12:09.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.969+0000 7fd8feefb640 1 -- 172.21.15.44:0/1707725455 wait complete. 2024-04-21T23:12:09.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.969+0000 7fd8feefb640 1 Processor -- start 2024-04-21T23:12:09.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.969+0000 7fd8feefb640 1 -- start start 2024-04-21T23:12:09.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.969+0000 7fd8feefb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8f81029b0 0x7fd8f8100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:09.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.969+0000 7fd8feefb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8f81032d0 0x7fd8f8100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:09.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.969+0000 7fd8feefb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8f810bcf0 0x7fd8f8101080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:09.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.969+0000 7fd8feefb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f8111af0 con 0x7fd8f81029b0 2024-04-21T23:12:09.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.969+0000 7fd8feefb640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f8111970 con 0x7fd8f81032d0 2024-04-21T23:12:09.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.969+0000 7fd8feefb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f8111c70 con 0x7fd8f810bcf0 2024-04-21T23:12:09.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.970+0000 7fd8fcc97640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8f81029b0 0x7fd8f8100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:09.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.970+0000 7fd8fcc97640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8f81029b0 0x7fd8f8100600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43344/0 (socket says 172.21.15.44:43344) 2024-04-21T23:12:09.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.970+0000 7fd8fcc97640 1 -- 172.21.15.44:0/2865336277 learned_addr learned my addr 172.21.15.44:0/2865336277 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:09.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.970+0000 7fd8f7fff640 1 --2- 172.21.15.44:0/2865336277 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8f81032d0 0x7fd8f8100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:09.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.970+0000 7fd8fd498640 1 --2- 172.21.15.44:0/2865336277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8f810bcf0 0x7fd8f8101080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:09.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.970+0000 7fd8fcc97640 1 -- 172.21.15.44:0/2865336277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8f810bcf0 msgr2=0x7fd8f8101080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:09.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.970+0000 7fd8fcc97640 1 --2- 172.21.15.44:0/2865336277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8f810bcf0 0x7fd8f8101080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:09.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.970+0000 7fd8fcc97640 1 -- 172.21.15.44:0/2865336277 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8f81032d0 msgr2=0x7fd8f8100b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:09.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.970+0000 7fd8fcc97640 1 --2- 172.21.15.44:0/2865336277 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8f81032d0 0x7fd8f8100b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:09.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.970+0000 7fd8fcc97640 1 -- 172.21.15.44:0/2865336277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f81a65b0 con 0x7fd8f81029b0 2024-04-21T23:12:09.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.970+0000 7fd8fcc97640 1 --2- 172.21.15.44:0/2865336277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8f81029b0 0x7fd8f8100600 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800e9f0 tx=0x7fd8e800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:09.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.970+0000 7fd8f5ffb640 1 -- 172.21.15.44:0/2865336277 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e800cd80 con 0x7fd8f81029b0 2024-04-21T23:12:09.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.970+0000 7fd8feefb640 1 -- 172.21.15.44:0/2865336277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8f81a68a0 con 0x7fd8f81029b0 2024-04-21T23:12:09.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.970+0000 7fd8f5ffb640 1 -- 172.21.15.44:0/2865336277 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8e8010910 con 0x7fd8f81029b0 2024-04-21T23:12:09.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.970+0000 7fd8f5ffb640 1 -- 172.21.15.44:0/2865336277 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e80188f0 con 0x7fd8f81029b0 2024-04-21T23:12:09.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.970+0000 7fd8feefb640 1 -- 172.21.15.44:0/2865336277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8f81a6dc0 con 0x7fd8f81029b0 2024-04-21T23:12:09.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.973+0000 7fd8f5ffb640 1 -- 172.21.15.44:0/2865336277 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd8e8010ac0 con 0x7fd8f81029b0 2024-04-21T23:12:09.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.973+0000 7fd901bc2740 1 Processor -- start 2024-04-21T23:12:09.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.973+0000 7fd901bc2740 1 -- start start 2024-04-21T23:12:09.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.973+0000 7fd901bc2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53650720 0x556a5364f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:09.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.973+0000 7fd901bc2740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a5364fa40 0x556a5364fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:09.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.973+0000 7fd901bc2740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a53658db0 0x556a5365b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:09.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.973+0000 7fd901bc2740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x556a535b3170 con 0x556a53658db0 2024-04-21T23:12:09.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.973+0000 7fd901bc2740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x556a5355b7c0 con 0x556a5364fa40 2024-04-21T23:12:09.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.973+0000 7fd901bc2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556a535af130 con 0x556a53650720 2024-04-21T23:12:09.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.973+0000 7fd8f7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a53658db0 0x556a5365b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:09.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.973+0000 7fd8fcc97640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a5364fa40 0x556a5364fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:09.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.973+0000 7fd8f7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a53658db0 0x556a5365b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43358/0 (socket says 172.21.15.44:43358) 2024-04-21T23:12:09.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.973+0000 7fd8f7fff640 1 -- 172.21.15.44:0/4142681039 learned_addr learned my addr 172.21.15.44:0/4142681039 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:09.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.973+0000 7fd8fd498640 1 --2- 172.21.15.44:0/4142681039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53650720 0x556a5364f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:09.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.974+0000 7fd8f5ffb640 1 --2- 172.21.15.44:0/2865336277 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd8e00a2d00 0x7fd8e00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:09.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.974+0000 7fd8fd498640 1 --2- 172.21.15.44:0/2865336277 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd8e00a2d00 0x7fd8e00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:09.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.974+0000 7fd8f7fff640 1 -- 172.21.15.44:0/4142681039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53650720 msgr2=0x556a5364f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:09.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.974+0000 7fd8f7fff640 1 --2- 172.21.15.44:0/4142681039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53650720 0x556a5364f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:09.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.974+0000 7fd8f7fff640 1 -- 172.21.15.44:0/4142681039 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a5364fa40 msgr2=0x556a5364fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:09.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.974+0000 7fd8f7fff640 1 --2- 172.21.15.44:0/4142681039 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a5364fa40 0x556a5364fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:09.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.974+0000 7fd8f7fff640 1 -- 172.21.15.44:0/4142681039 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a5356ce60 con 0x556a53658db0 2024-04-21T23:12:09.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.974+0000 7fd8f5ffb640 1 -- 172.21.15.44:0/2865336277 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fd8e80cfb80 con 0x7fd8f81029b0 2024-04-21T23:12:09.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.974+0000 7fd8f7fff640 1 --2- 172.21.15.44:0/4142681039 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a53658db0 0x556a5365b1a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009230 tx=0x7fd8ec005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff9de045ad94ce9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:09.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.974+0000 7fd8fd498640 1 --2- 172.21.15.44:0/2865336277 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd8e00a2d00 0x7fd8e00a51c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0009fd0 tx=0x7fd8f0004ab0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:09.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.974+0000 7fd8deffd640 1 -- 172.21.15.44:0/4142681039 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec042070 con 0x556a53658db0 2024-04-21T23:12:09.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.974+0000 7fd8deffd640 1 -- 172.21.15.44:0/4142681039 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8ec02c730 con 0x556a53658db0 2024-04-21T23:12:09.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.974+0000 7fd8deffd640 1 -- 172.21.15.44:0/4142681039 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec004f10 con 0x556a53658db0 2024-04-21T23:12:09.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.975+0000 7fd901bc2740 1 -- 172.21.15.44:0/4142681039 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a53658db0 msgr2=0x556a5365b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:09.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.975+0000 7fd901bc2740 1 --2- 172.21.15.44:0/4142681039 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a53658db0 0x556a5365b1a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009230 tx=0x7fd8ec005bf0 comp rx=0 tx=0).stop 2024-04-21T23:12:09.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.975+0000 7fd901bc2740 1 -- 172.21.15.44:0/4142681039 shutdown_connections 2024-04-21T23:12:09.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.975+0000 7fd901bc2740 1 --2- 172.21.15.44:0/4142681039 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a53658db0 0x556a5365b1a0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:09.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.975+0000 7fd901bc2740 1 --2- 172.21.15.44:0/4142681039 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a5364fa40 0x556a5364fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:09.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.975+0000 7fd901bc2740 1 --2- 172.21.15.44:0/4142681039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53650720 0x556a5364f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:09.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.975+0000 7fd901bc2740 1 -- 172.21.15.44:0/4142681039 >> 172.21.15.44:0/4142681039 conn(0x556a536164f0 msgr2=0x556a5364a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:09.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.975+0000 7fd901bc2740 1 -- 172.21.15.44:0/4142681039 shutdown_connections 2024-04-21T23:12:09.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.975+0000 7fd901bc2740 1 -- 172.21.15.44:0/4142681039 wait complete. 2024-04-21T23:12:09.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.975+0000 7fd901bc2740 1 -- start start 2024-04-21T23:12:09.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.975+0000 7fd901bc2740 1 Processor -- start 2024-04-21T23:12:09.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd901bc2740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 0x556a53655d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:09.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd901bc2740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a53650720 0x556a53656250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:09.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd901bc2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53658db0 0x556a53656790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:09.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd901bc2740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x556a5365bbc0 con 0x556a5364fa40 2024-04-21T23:12:09.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd901bc2740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x556a5356ed40 con 0x556a53650720 2024-04-21T23:12:09.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd901bc2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556a5365bd40 con 0x556a53658db0 2024-04-21T23:12:09.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd901bc2740 10 client.0 _ll_register_callbacks cb 0x556a535aa820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:09.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd8fcc97640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 0x556a53655d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:09.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd8fd498640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53658db0 0x556a53656790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:09.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd8fcc97640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 0x556a53655d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43372/0 (socket says 172.21.15.44:43372) 2024-04-21T23:12:09.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd8fcc97640 1 -- 172.21.15.44:0/30597856 learned_addr learned my addr 172.21.15.44:0/30597856 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:09.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd8f7fff640 1 --2- 172.21.15.44:0/30597856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a53650720 0x556a53656250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:09.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd8fcc97640 1 -- 172.21.15.44:0/30597856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53658db0 msgr2=0x556a53656790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:09.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd8fcc97640 1 --2- 172.21.15.44:0/30597856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53658db0 0x556a53656790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:09.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd8fcc97640 1 -- 172.21.15.44:0/30597856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a53650720 msgr2=0x556a53656250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:09.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd8fcc97640 1 --2- 172.21.15.44:0/30597856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a53650720 0x556a53656250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:09.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd8fcc97640 1 -- 172.21.15.44:0/30597856 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a5338f7e0 con 0x556a5364fa40 2024-04-21T23:12:09.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd8fd498640 1 --2- 172.21.15.44:0/30597856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53658db0 0x556a53656790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:09.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd8f7fff640 1 --2- 172.21.15.44:0/30597856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a53650720 0x556a53656250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:09.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd8fcc97640 1 --2- 172.21.15.44:0/30597856 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 0x556a53655d10 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800c6a0 tx=0x7fd8e809b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e53848f80ff41c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:09.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd8d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:12:09.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd8d7fff640 1 -- 172.21.15.44:0/30597856 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e80cd3c0 con 0x556a5364fa40 2024-04-21T23:12:09.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd8d7fff640 1 -- 172.21.15.44:0/30597856 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fd8e809b200 con 0x556a5364fa40 2024-04-21T23:12:09.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.976+0000 7fd901bc2740 1 -- 172.21.15.44:0/30597856 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556a53587450 con 0x556a5364fa40 2024-04-21T23:12:09.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.977+0000 7fd8d7fff640 1 -- 172.21.15.44:0/30597856 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd8e80a00b0 con 0x556a5364fa40 2024-04-21T23:12:09.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.977+0000 7fd901bc2740 10 client.16419 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:09.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.977+0000 7fd901bc2740 1 -- 172.21.15.44:0/30597856 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556a533eff70 con 0x556a5364fa40 2024-04-21T23:12:09.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.977+0000 7fd8d7fff640 1 -- 172.21.15.44:0/30597856 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556a53587450 con 0x556a5364fa40 2024-04-21T23:12:09.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.977+0000 7fd8d7fff640 1 -- 172.21.15.44:0/30597856 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fd8e8086050 con 0x556a5364fa40 2024-04-21T23:12:09.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.977+0000 7fd901bc2740 10 client.16419 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:09.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.977+0000 7fd901bc2740 10 client.16419 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:09.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.977+0000 7fd901bc2740 20 client.16419 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:09.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.977+0000 7fd901bc2740 1 --2- 172.21.15.44:0/30597856 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556a53668d50 0x556a5366b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:09.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.977+0000 7fd901bc2740 4 client.16419 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:12:09.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.977+0000 7fd901bc2740 1 -- 172.21.15.44:0/30597856 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a535c06b0 con 0x556a53668d50 2024-04-21T23:12:09.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.977+0000 7fd8f7fff640 1 --2- 172.21.15.44:0/30597856 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556a53668d50 0x556a5366b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:09.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.978+0000 7fd8d7fff640 10 client.16419 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:09.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.978+0000 7fd8f7fff640 1 --2- 172.21.15.44:0/30597856 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556a53668d50 0x556a5366b1e0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e84a40d924aa4f2 server_cookie=b18bb89dd1e20ed in_seq=0 out_seq=0 2024-04-21T23:12:09.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.978+0000 7fd8d7fff640 1 -- 172.21.15.44:0/30597856 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556a535c06b0 con 0x556a53668d50 2024-04-21T23:12:09.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.978+0000 7fd8d7fff640 10 client.16419 handle_command_reply: tid=0 2024-04-21T23:12:09.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.978+0000 7fd901bc2740 1 client.16419 shutdown 2024-04-21T23:12:09.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.979+0000 7fd901bc2740 1 -- 172.21.15.44:0/30597856 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 msgr2=0x556a53655d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:09.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.979+0000 7fd901bc2740 1 --2- 172.21.15.44:0/30597856 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 0x556a53655d10 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800c6a0 tx=0x7fd8e809b8f0 comp rx=0 tx=0).stop 2024-04-21T23:12:09.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.979+0000 7fd901bc2740 1 -- 172.21.15.44:0/30597856 shutdown_connections 2024-04-21T23:12:09.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.979+0000 7fd901bc2740 1 --2- 172.21.15.44:0/30597856 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556a53668d50 0x556a5366b1e0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:09.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.979+0000 7fd901bc2740 1 --2- 172.21.15.44:0/30597856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53658db0 0x556a53656790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:09.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.979+0000 7fd8d7fff640 0 client.16419 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:09.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.979+0000 7fd901bc2740 1 --2- 172.21.15.44:0/30597856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a53650720 0x556a53656250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:09.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.979+0000 7fd901bc2740 1 --2- 172.21.15.44:0/30597856 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 0x556a53655d10 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:09.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.979+0000 7fd901bc2740 1 -- 172.21.15.44:0/30597856 >> 172.21.15.44:0/30597856 conn(0x556a536164f0 msgr2=0x556a5364c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:09.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.979+0000 7fd901bc2740 1 -- 172.21.15.44:0/30597856 shutdown_connections 2024-04-21T23:12:09.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.979+0000 7fd901bc2740 1 -- 172.21.15.44:0/30597856 wait complete. 2024-04-21T23:12:09.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.979+0000 7fd901bc2740 20 client.16419 trim_cache size 0 max 16384 2024-04-21T23:12:10.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.998+0000 7fd901bc2740 1 Processor -- start 2024-04-21T23:12:10.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.999+0000 7fd901bc2740 1 -- start start 2024-04-21T23:12:10.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.999+0000 7fd901bc2740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 0x556a53664ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.999+0000 7fd901bc2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53650720 0x556a53649370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.999+0000 7fd901bc2740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a53658db0 0x556a53670ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.999+0000 7fd901bc2740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f81aee20 con 0x556a5364fa40 2024-04-21T23:12:10.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.999+0000 7fd901bc2740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f81b16a0 con 0x556a53658db0 2024-04-21T23:12:10.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.999+0000 7fd901bc2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f81af490 con 0x556a53650720 2024-04-21T23:12:10.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.999+0000 7fd8fcc97640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 0x556a53664ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.999+0000 7fd8fcc97640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 0x556a53664ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43378/0 (socket says 172.21.15.44:43378) 2024-04-21T23:12:10.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.999+0000 7fd8fcc97640 1 -- 172.21.15.44:0/4207421874 learned_addr learned my addr 172.21.15.44:0/4207421874 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:10.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.999+0000 7fd8fd498640 1 --2- 172.21.15.44:0/4207421874 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a53658db0 0x556a53670ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.999+0000 7fd8f7fff640 1 --2- 172.21.15.44:0/4207421874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53650720 0x556a53649370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.999+0000 7fd8fcc97640 1 -- 172.21.15.44:0/4207421874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53650720 msgr2=0x556a53649370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:10.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.999+0000 7fd8fcc97640 1 --2- 172.21.15.44:0/4207421874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53650720 0x556a53649370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.999+0000 7fd8fcc97640 1 -- 172.21.15.44:0/4207421874 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a53658db0 msgr2=0x556a53670ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:10.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.999+0000 7fd8fcc97640 1 --2- 172.21.15.44:0/4207421874 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a53658db0 0x556a53670ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.999+0000 7fd8fcc97640 1 -- 172.21.15.44:0/4207421874 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a53287170 con 0x556a5364fa40 2024-04-21T23:12:10.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.999+0000 7fd8f7fff640 1 --2- 172.21.15.44:0/4207421874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53650720 0x556a53649370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:10.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.999+0000 7fd8fcc97640 1 --2- 172.21.15.44:0/4207421874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 0x556a53664ac0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8098780 tx=0x7fd8e809bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca6aa107133e8e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:10.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:09.999+0000 7fd8de7fc640 1 -- 172.21.15.44:0/4207421874 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e80a9070 con 0x556a5364fa40 2024-04-21T23:12:10.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.000+0000 7fd8de7fc640 1 -- 172.21.15.44:0/4207421874 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8e80a4970 con 0x556a5364fa40 2024-04-21T23:12:10.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.000+0000 7fd8de7fc640 1 -- 172.21.15.44:0/4207421874 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e80a4e20 con 0x556a5364fa40 2024-04-21T23:12:10.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.000+0000 7fd901bc2740 1 -- 172.21.15.44:0/4207421874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 msgr2=0x556a53664ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:10.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.000+0000 7fd901bc2740 1 --2- 172.21.15.44:0/4207421874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 0x556a53664ac0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8098780 tx=0x7fd8e809bf30 comp rx=0 tx=0).stop 2024-04-21T23:12:10.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.000+0000 7fd901bc2740 1 -- 172.21.15.44:0/4207421874 shutdown_connections 2024-04-21T23:12:10.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.000+0000 7fd901bc2740 1 --2- 172.21.15.44:0/4207421874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53650720 0x556a53649370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.000+0000 7fd901bc2740 1 --2- 172.21.15.44:0/4207421874 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a53658db0 0x556a53670ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.000+0000 7fd901bc2740 1 --2- 172.21.15.44:0/4207421874 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 0x556a53664ac0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.000+0000 7fd901bc2740 1 -- 172.21.15.44:0/4207421874 >> 172.21.15.44:0/4207421874 conn(0x556a536164f0 msgr2=0x556a53648c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:10.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.001+0000 7fd901bc2740 1 -- 172.21.15.44:0/4207421874 shutdown_connections 2024-04-21T23:12:10.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.001+0000 7fd901bc2740 1 -- 172.21.15.44:0/4207421874 wait complete. 2024-04-21T23:12:10.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.001+0000 7fd901bc2740 1 -- start start 2024-04-21T23:12:10.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.001+0000 7fd901bc2740 1 Processor -- start 2024-04-21T23:12:10.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.001+0000 7fd901bc2740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 0x556a53653020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.001+0000 7fd901bc2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53650720 0x556a53653560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.001+0000 7fd901bc2740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a53658db0 0x556a536ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.001+0000 7fd901bc2740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f81afb60 con 0x556a5364fa40 2024-04-21T23:12:10.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.001+0000 7fd901bc2740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f81b0fd0 con 0x556a53658db0 2024-04-21T23:12:10.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.001+0000 7fd901bc2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f81b0230 con 0x556a53650720 2024-04-21T23:12:10.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.002+0000 7fd901bc2740 10 client.0 _ll_register_callbacks cb 0x7fd8e800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:10.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.002+0000 7fd8fcc97640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 0x556a53653020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.002+0000 7fd8fcc97640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 0x556a53653020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43386/0 (socket says 172.21.15.44:43386) 2024-04-21T23:12:10.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.002+0000 7fd8fcc97640 1 -- 172.21.15.44:0/3060085263 learned_addr learned my addr 172.21.15.44:0/3060085263 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:10.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.002+0000 7fd8f7fff640 1 --2- 172.21.15.44:0/3060085263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53650720 0x556a53653560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.002+0000 7fd8fd498640 1 --2- 172.21.15.44:0/3060085263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a53658db0 0x556a536ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.002+0000 7fd8fcc97640 1 -- 172.21.15.44:0/3060085263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53650720 msgr2=0x556a53653560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:10.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.002+0000 7fd8fcc97640 1 --2- 172.21.15.44:0/3060085263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53650720 0x556a53653560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.002+0000 7fd8fcc97640 1 -- 172.21.15.44:0/3060085263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a53658db0 msgr2=0x556a536ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:10.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.002+0000 7fd8fcc97640 1 --2- 172.21.15.44:0/3060085263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a53658db0 0x556a536ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.002+0000 7fd8fcc97640 1 -- 172.21.15.44:0/3060085263 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a536ce880 con 0x556a5364fa40 2024-04-21T23:12:10.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.002+0000 7fd8fcc97640 1 --2- 172.21.15.44:0/3060085263 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 0x556a53653020 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8098210 tx=0x7fd8e801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea3279e406faaa45 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:10.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.002+0000 7fd8d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:12:10.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.002+0000 7fd8d7fff640 1 -- 172.21.15.44:0/3060085263 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e80a9020 con 0x556a5364fa40 2024-04-21T23:12:10.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.002+0000 7fd8d7fff640 1 -- 172.21.15.44:0/3060085263 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fd8e8097ba0 con 0x556a5364fa40 2024-04-21T23:12:10.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.002+0000 7fd901bc2740 1 -- 172.21.15.44:0/3060085263 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8e80a4970 con 0x556a5364fa40 2024-04-21T23:12:10.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.002+0000 7fd8d7fff640 1 -- 172.21.15.44:0/3060085263 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd8e80d3030 con 0x556a5364fa40 2024-04-21T23:12:10.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.002+0000 7fd901bc2740 10 client.16425 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:10.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.002+0000 7fd901bc2740 1 -- 172.21.15.44:0/3060085263 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556a536ce880 con 0x556a5364fa40 2024-04-21T23:12:10.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.003+0000 7fd8d7fff640 1 -- 172.21.15.44:0/3060085263 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8e80a4970 con 0x556a5364fa40 2024-04-21T23:12:10.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.003+0000 7fd8d7fff640 1 -- 172.21.15.44:0/3060085263 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fd8e8086040 con 0x556a5364fa40 2024-04-21T23:12:10.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.003+0000 7fd901bc2740 10 client.16425 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:10.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.003+0000 7fd901bc2740 10 client.16425 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:10.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.003+0000 7fd901bc2740 20 client.16425 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:10.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.003+0000 7fd901bc2740 1 --2- 172.21.15.44:0/3060085263 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556a536cfe00 0x556a536d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.003+0000 7fd901bc2740 4 client.16425 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:12:10.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.003+0000 7fd901bc2740 1 -- 172.21.15.44:0/3060085263 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556a535baa00 con 0x556a536cfe00 2024-04-21T23:12:10.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.003+0000 7fd8f7fff640 1 --2- 172.21.15.44:0/3060085263 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556a536cfe00 0x556a536d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.003+0000 7fd8f7fff640 1 --2- 172.21.15.44:0/3060085263 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556a536cfe00 0x556a536d2220 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cef0f34d1e43c4e server_cookie=3e1f33fe6ca10616 in_seq=0 out_seq=0 2024-04-21T23:12:10.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.003+0000 7fd8d7fff640 10 client.16425 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:10.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.004+0000 7fd8d7fff640 1 -- 172.21.15.44:0/3060085263 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556a535baa00 con 0x556a536cfe00 2024-04-21T23:12:10.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.004+0000 7fd8d7fff640 10 client.16425 handle_command_reply: tid=0 2024-04-21T23:12:10.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.004+0000 7fd901bc2740 1 client.16425 shutdown 2024-04-21T23:12:10.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.004+0000 7fd901bc2740 1 -- 172.21.15.44:0/3060085263 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 msgr2=0x556a53653020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:10.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.004+0000 7fd901bc2740 1 --2- 172.21.15.44:0/3060085263 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 0x556a53653020 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8098210 tx=0x7fd8e801f000 comp rx=0 tx=0).stop 2024-04-21T23:12:10.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.005+0000 7fd901bc2740 1 -- 172.21.15.44:0/3060085263 shutdown_connections 2024-04-21T23:12:10.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.005+0000 7fd901bc2740 1 --2- 172.21.15.44:0/3060085263 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556a536cfe00 0x556a536d2220 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.005+0000 7fd901bc2740 1 --2- 172.21.15.44:0/3060085263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a53650720 0x556a53653560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.005+0000 7fd8d7fff640 0 client.16425 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:10.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.005+0000 7fd901bc2740 1 --2- 172.21.15.44:0/3060085263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a53658db0 0x556a536ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.005+0000 7fd901bc2740 1 --2- 172.21.15.44:0/3060085263 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a5364fa40 0x556a53653020 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.005+0000 7fd901bc2740 1 -- 172.21.15.44:0/3060085263 >> 172.21.15.44:0/3060085263 conn(0x556a536164f0 msgr2=0x556a536494f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:10.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.005+0000 7fd901bc2740 1 -- 172.21.15.44:0/3060085263 shutdown_connections 2024-04-21T23:12:10.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.005+0000 7fd901bc2740 1 -- 172.21.15.44:0/3060085263 wait complete. 2024-04-21T23:12:10.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.005+0000 7fd901bc2740 20 client.16425 trim_cache size 0 max 16384 2024-04-21T23:12:10.010 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:12:10.011 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-21T23:12:10.011 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "ca1725ba-969c-45f5-94d2-6b268722f64b", 2024-04-21T23:12:10.011 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-21T23:12:10.011 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:12:10.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.005+0000 7fd8feefb640 1 -- 172.21.15.44:0/2865336277 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd8e00a2d00 msgr2=0x7fd8e00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:10.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.005+0000 7fd8feefb640 1 --2- 172.21.15.44:0/2865336277 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd8e00a2d00 0x7fd8e00a51c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0009fd0 tx=0x7fd8f0004ab0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.005+0000 7fd8feefb640 1 -- 172.21.15.44:0/2865336277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8f81029b0 msgr2=0x7fd8f8100600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:10.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.005+0000 7fd8feefb640 1 --2- 172.21.15.44:0/2865336277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8f81029b0 0x7fd8f8100600 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800e9f0 tx=0x7fd8e800eeb0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.006+0000 7fd8feefb640 1 -- 172.21.15.44:0/2865336277 shutdown_connections 2024-04-21T23:12:10.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.006+0000 7fd8feefb640 1 --2- 172.21.15.44:0/2865336277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8f810bcf0 0x7fd8f8101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.006+0000 7fd8feefb640 1 --2- 172.21.15.44:0/2865336277 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd8e00a2d00 0x7fd8e00a51c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.006+0000 7fd8feefb640 1 --2- 172.21.15.44:0/2865336277 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8f81032d0 0x7fd8f8100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.006+0000 7fd8feefb640 1 --2- 172.21.15.44:0/2865336277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd8f81029b0 0x7fd8f8100600 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.006+0000 7fd8feefb640 1 -- 172.21.15.44:0/2865336277 >> 172.21.15.44:0/2865336277 conn(0x7fd8f80777f0 msgr2=0x7fd8f80feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:10.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.006+0000 7fd8feefb640 1 -- 172.21.15.44:0/2865336277 shutdown_connections 2024-04-21T23:12:10.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.006+0000 7fd8feefb640 1 -- 172.21.15.44:0/2865336277 wait complete. 2024-04-21T23:12:10.015 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:09 smithi044 ceph-mon[27934]: pgmap v104: 129 pgs: 129 active+clean; 45 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 482 op/s 2024-04-21T23:12:10.016 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:12:10.016 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:12:10.114 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:09 smithi055 ceph-mon[31117]: pgmap v104: 129 pgs: 129 active+clean; 45 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 482 op/s 2024-04-21T23:12:10.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.124+0000 7fb6c5888640 1 -- 172.21.15.44:0/2415018562 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6c0103210 msgr2=0x7fb6c0103670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:10.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.124+0000 7fb6c5888640 1 --2- 172.21.15.44:0/2415018562 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6c0103210 0x7fb6c0103670 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8009900 tx=0x7fb6a802bd10 comp rx=0 tx=0).stop 2024-04-21T23:12:10.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.124+0000 7fb6c5888640 1 -- 172.21.15.44:0/2415018562 shutdown_connections 2024-04-21T23:12:10.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.124+0000 7fb6c5888640 1 --2- 172.21.15.44:0/2415018562 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6c0103bb0 0x7fb6c010e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.124+0000 7fb6c5888640 1 --2- 172.21.15.44:0/2415018562 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6c0103210 0x7fb6c0103670 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.124+0000 7fb6c5888640 1 --2- 172.21.15.44:0/2415018562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6c0101ad0 0x7fb6c0101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.124+0000 7fb6c5888640 1 -- 172.21.15.44:0/2415018562 >> 172.21.15.44:0/2415018562 conn(0x7fb6c00fd200 msgr2=0x7fb6c00ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:10.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.124+0000 7fb6c5888640 1 -- 172.21.15.44:0/2415018562 shutdown_connections 2024-04-21T23:12:10.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.124+0000 7fb6c5888640 1 -- 172.21.15.44:0/2415018562 wait complete. 2024-04-21T23:12:10.124 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.124+0000 7fb6c5888640 1 Processor -- start 2024-04-21T23:12:10.125 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.125+0000 7fb6c5888640 1 -- start start 2024-04-21T23:12:10.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.125+0000 7fb6c5888640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6c0101ad0 0x7fb6c01980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.125+0000 7fb6c5888640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6c0103210 0x7fb6c0198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.125+0000 7fb6c5888640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6c0103bb0 0x7fb6c019c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.125+0000 7fb6c5888640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c0111c20 con 0x7fb6c0103210 2024-04-21T23:12:10.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.125+0000 7fb6c5888640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c0111aa0 con 0x7fb6c0101ad0 2024-04-21T23:12:10.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.125+0000 7fb6c5888640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c0111da0 con 0x7fb6c0103bb0 2024-04-21T23:12:10.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.125+0000 7fb6be7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6c0103210 0x7fb6c0198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.125+0000 7fb6be7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6c0103210 0x7fb6c0198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43408/0 (socket says 172.21.15.44:43408) 2024-04-21T23:12:10.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.125+0000 7fb6be7fc640 1 -- 172.21.15.44:0/2342781417 learned_addr learned my addr 172.21.15.44:0/2342781417 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:10.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.125+0000 7fb6bf7fe640 1 --2- 172.21.15.44:0/2342781417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6c0103bb0 0x7fb6c019c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.125+0000 7fb6beffd640 1 --2- 172.21.15.44:0/2342781417 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6c0101ad0 0x7fb6c01980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.125+0000 7fb6be7fc640 1 -- 172.21.15.44:0/2342781417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6c0103bb0 msgr2=0x7fb6c019c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:10.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.125+0000 7fb6be7fc640 1 --2- 172.21.15.44:0/2342781417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6c0103bb0 0x7fb6c019c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.125+0000 7fb6be7fc640 1 -- 172.21.15.44:0/2342781417 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6c0101ad0 msgr2=0x7fb6c01980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:10.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.125+0000 7fb6be7fc640 1 --2- 172.21.15.44:0/2342781417 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6c0101ad0 0x7fb6c01980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.125+0000 7fb6be7fc640 1 -- 172.21.15.44:0/2342781417 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6c019d080 con 0x7fb6c0103210 2024-04-21T23:12:10.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.125+0000 7fb6bf7fe640 1 --2- 172.21.15.44:0/2342781417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6c0103bb0 0x7fb6c019c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:12:10.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.125+0000 7fb6be7fc640 1 --2- 172.21.15.44:0/2342781417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6c0103210 0x7fb6c0198620 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8006010 tx=0x7fb6a8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:10.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.126+0000 7fb6c4886640 1 -- 172.21.15.44:0/2342781417 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8042020 con 0x7fb6c0103210 2024-04-21T23:12:10.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.126+0000 7fb6c4886640 1 -- 172.21.15.44:0/2342781417 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6a802ce10 con 0x7fb6c0103210 2024-04-21T23:12:10.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.126+0000 7fb6c5888640 1 -- 172.21.15.44:0/2342781417 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6c019d310 con 0x7fb6c0103210 2024-04-21T23:12:10.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.126+0000 7fb6c4886640 1 -- 172.21.15.44:0/2342781417 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8003ab0 con 0x7fb6c0103210 2024-04-21T23:12:10.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.126+0000 7fb6c5888640 1 -- 172.21.15.44:0/2342781417 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6c01a4c00 con 0x7fb6c0103210 2024-04-21T23:12:10.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.128+0000 7fb6c4886640 1 -- 172.21.15.44:0/2342781417 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb6a8045020 con 0x7fb6c0103210 2024-04-21T23:12:10.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.128+0000 7fb6c847c740 1 Processor -- start 2024-04-21T23:12:10.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.128+0000 7fb6c847c740 1 -- start start 2024-04-21T23:12:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.128+0000 7fb6c4886640 1 --2- 172.21.15.44:0/2342781417 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb6ac0a2d00 0x7fb6ac0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.128+0000 7fb6c847c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145508c60 0x556145509040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.128+0000 7fb6c847c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 0x556145516300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.128+0000 7fb6c847c740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556145516840 0x556145518c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.128+0000 7fb6c847c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55614546d560 con 0x556145509580 2024-04-21T23:12:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.128+0000 7fb6c847c740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x556145415730 con 0x556145516840 2024-04-21T23:12:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.129+0000 7fb6c847c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556145469520 con 0x556145508c60 2024-04-21T23:12:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.129+0000 7fb6bf7fe640 1 --2- 172.21.15.44:0/2342781417 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb6ac0a2d00 0x7fb6ac0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.129+0000 7fb6c4886640 1 -- 172.21.15.44:0/2342781417 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fb6a80f1640 con 0x7fb6c0103210 2024-04-21T23:12:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.129+0000 7fb6be7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 0x556145516300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.129+0000 7fb6beffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145508c60 0x556145509040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.129+0000 7fb6be7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 0x556145516300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43418/0 (socket says 172.21.15.44:43418) 2024-04-21T23:12:10.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.129+0000 7fb6be7fc640 1 -- 172.21.15.44:0/2030158252 learned_addr learned my addr 172.21.15.44:0/2030158252 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.129+0000 7fb6bf7fe640 1 --2- 172.21.15.44:0/2030158252 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556145516840 0x556145518c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.129+0000 7fb6beffd640 1 -- 172.21.15.44:0/2030158252 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556145516840 msgr2=0x556145518c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.129+0000 7fb6beffd640 1 --2- 172.21.15.44:0/2030158252 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556145516840 0x556145518c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.129+0000 7fb6beffd640 1 -- 172.21.15.44:0/2030158252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 msgr2=0x556145516300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.129+0000 7fb6beffd640 1 --2- 172.21.15.44:0/2030158252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 0x556145516300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.129+0000 7fb6beffd640 1 -- 172.21.15.44:0/2030158252 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55614524b7d0 con 0x556145508c60 2024-04-21T23:12:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.129+0000 7fb6be7fc640 1 --2- 172.21.15.44:0/2030158252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 0x556145516300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.129+0000 7fb6bf7fe640 1 --2- 172.21.15.44:0/2342781417 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb6ac0a2d00 0x7fb6ac0a51c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0005fd0 tx=0x7fb6b0009eb0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.129+0000 7fb6beffd640 1 --2- 172.21.15.44:0/2030158252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145508c60 0x556145509040 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8009b70 tx=0x7fb6b80117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21eef89098467160 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.129+0000 7fb6b5ffb640 1 -- 172.21.15.44:0/2030158252 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8009e60 con 0x556145508c60 2024-04-21T23:12:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.129+0000 7fb6b5ffb640 1 -- 172.21.15.44:0/2030158252 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6b8012930 con 0x556145508c60 2024-04-21T23:12:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.129+0000 7fb6b5ffb640 1 -- 172.21.15.44:0/2030158252 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b800fb90 con 0x556145508c60 2024-04-21T23:12:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.130+0000 7fb6c847c740 1 -- 172.21.15.44:0/2030158252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145508c60 msgr2=0x556145509040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.130+0000 7fb6c847c740 1 --2- 172.21.15.44:0/2030158252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145508c60 0x556145509040 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8009b70 tx=0x7fb6b80117b0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.130+0000 7fb6c847c740 1 -- 172.21.15.44:0/2030158252 shutdown_connections 2024-04-21T23:12:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.130+0000 7fb6c847c740 1 --2- 172.21.15.44:0/2030158252 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556145516840 0x556145518c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.130+0000 7fb6c847c740 1 --2- 172.21.15.44:0/2030158252 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 0x556145516300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.130+0000 7fb6c847c740 1 --2- 172.21.15.44:0/2030158252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145508c60 0x556145509040 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.130+0000 7fb6c847c740 1 -- 172.21.15.44:0/2030158252 >> 172.21.15.44:0/2030158252 conn(0x556145503660 msgr2=0x556145503a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.130+0000 7fb6c847c740 1 -- 172.21.15.44:0/2030158252 shutdown_connections 2024-04-21T23:12:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.130+0000 7fb6c847c740 1 -- 172.21.15.44:0/2030158252 wait complete. 2024-04-21T23:12:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.130+0000 7fb6c847c740 1 -- start start 2024-04-21T23:12:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.131+0000 7fb6c847c740 1 Processor -- start 2024-04-21T23:12:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.131+0000 7fb6c847c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145508c60 0x556145513870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.131+0000 7fb6c847c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 0x556145513db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.131+0000 7fb6c847c740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556145516840 0x5561455142f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.131+0000 7fb6c847c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x556145519600 con 0x556145509580 2024-04-21T23:12:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.131+0000 7fb6c847c740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x556145428e00 con 0x556145516840 2024-04-21T23:12:10.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.131+0000 7fb6c847c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556145519780 con 0x556145508c60 2024-04-21T23:12:10.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.131+0000 7fb6c847c740 10 client.0 _ll_register_callbacks cb 0x556145463eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:10.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.131+0000 7fb6be7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 0x556145513db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.131+0000 7fb6be7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 0x556145513db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43430/0 (socket says 172.21.15.44:43430) 2024-04-21T23:12:10.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.131+0000 7fb6be7fc640 1 -- 172.21.15.44:0/3432011337 learned_addr learned my addr 172.21.15.44:0/3432011337 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:10.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.131+0000 7fb6beffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145508c60 0x556145513870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.131+0000 7fb6bf7fe640 1 --2- 172.21.15.44:0/3432011337 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556145516840 0x5561455142f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.131+0000 7fb6be7fc640 1 -- 172.21.15.44:0/3432011337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145508c60 msgr2=0x556145513870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:10.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.131+0000 7fb6be7fc640 1 --2- 172.21.15.44:0/3432011337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145508c60 0x556145513870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.131+0000 7fb6be7fc640 1 -- 172.21.15.44:0/3432011337 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556145516840 msgr2=0x5561455142f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:10.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.131+0000 7fb6be7fc640 1 --2- 172.21.15.44:0/3432011337 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556145516840 0x5561455142f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.131+0000 7fb6be7fc640 1 -- 172.21.15.44:0/3432011337 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561451813f0 con 0x556145509580 2024-04-21T23:12:10.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.131+0000 7fb6be7fc640 1 --2- 172.21.15.44:0/3432011337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 0x556145513db0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8000c00 tx=0x7fb6a80bd410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db7887199dfccd21 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:10.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.132+0000 7fb69effd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:12:10.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.132+0000 7fb69effd640 1 -- 172.21.15.44:0/3432011337 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a80bdfd0 con 0x556145509580 2024-04-21T23:12:10.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.132+0000 7fb6c847c740 1 -- 172.21.15.44:0/3432011337 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6b8012930 con 0x556145509580 2024-04-21T23:12:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.132+0000 7fb69effd640 1 -- 172.21.15.44:0/3432011337 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fb6a80b9f40 con 0x556145509580 2024-04-21T23:12:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.132+0000 7fb69effd640 1 -- 172.21.15.44:0/3432011337 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb6a80442c0 con 0x556145509580 2024-04-21T23:12:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.132+0000 7fb6c847c740 10 client.16437 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.132+0000 7fb6c847c740 1 -- 172.21.15.44:0/3432011337 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5561451410f0 con 0x556145509580 2024-04-21T23:12:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.132+0000 7fb69effd640 1 -- 172.21.15.44:0/3432011337 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6b8012930 con 0x556145509580 2024-04-21T23:12:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.132+0000 7fb69effd640 1 -- 172.21.15.44:0/3432011337 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fb6a8005320 con 0x556145509580 2024-04-21T23:12:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.132+0000 7fb6c847c740 10 client.16437 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.132+0000 7fb6c847c740 10 client.16437 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.132+0000 7fb6c847c740 20 client.16437 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.132+0000 7fb6c847c740 1 --2- 172.21.15.44:0/3432011337 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556145526b70 0x556145528f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.132+0000 7fb6c847c740 4 client.16437 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:12:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.132+0000 7fb6c847c740 1 -- 172.21.15.44:0/3432011337 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55614547aaa0 con 0x556145526b70 2024-04-21T23:12:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.132+0000 7fb6beffd640 1 --2- 172.21.15.44:0/3432011337 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556145526b70 0x556145528f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.133+0000 7fb69effd640 10 client.16437 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:10.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.133+0000 7fb6beffd640 1 --2- 172.21.15.44:0/3432011337 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556145526b70 0x556145528f90 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f030b025d83a257f server_cookie=9f7c912ad84e2b9 in_seq=0 out_seq=0 2024-04-21T23:12:10.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.135+0000 7fb69effd640 1 -- 172.21.15.44:0/3432011337 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55614547aaa0 con 0x556145526b70 2024-04-21T23:12:10.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.135+0000 7fb69effd640 10 client.16437 handle_command_reply: tid=0 2024-04-21T23:12:10.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.135+0000 7fb6c847c740 1 client.16437 shutdown 2024-04-21T23:12:10.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.135+0000 7fb6c847c740 1 -- 172.21.15.44:0/3432011337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 msgr2=0x556145513db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:10.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.135+0000 7fb6c847c740 1 --2- 172.21.15.44:0/3432011337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 0x556145513db0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8000c00 tx=0x7fb6a80bd410 comp rx=0 tx=0).stop 2024-04-21T23:12:10.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.135+0000 7fb6c847c740 1 -- 172.21.15.44:0/3432011337 shutdown_connections 2024-04-21T23:12:10.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.135+0000 7fb6c847c740 1 --2- 172.21.15.44:0/3432011337 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556145526b70 0x556145528f90 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.135+0000 7fb6c847c740 1 --2- 172.21.15.44:0/3432011337 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556145516840 0x5561455142f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.135+0000 7fb6c847c740 1 --2- 172.21.15.44:0/3432011337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 0x556145513db0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.135+0000 7fb6c847c740 1 --2- 172.21.15.44:0/3432011337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145508c60 0x556145513870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.135+0000 7fb69effd640 0 client.16437 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:10.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.135+0000 7fb6c847c740 1 -- 172.21.15.44:0/3432011337 >> 172.21.15.44:0/3432011337 conn(0x556145503660 msgr2=0x5561455053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:10.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.135+0000 7fb6c847c740 1 -- 172.21.15.44:0/3432011337 shutdown_connections 2024-04-21T23:12:10.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.135+0000 7fb6c847c740 1 -- 172.21.15.44:0/3432011337 wait complete. 2024-04-21T23:12:10.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.136+0000 7fb6c847c740 20 client.16437 trim_cache size 0 max 16384 2024-04-21T23:12:10.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.156+0000 7fb6c847c740 1 Processor -- start 2024-04-21T23:12:10.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.156+0000 7fb6c847c740 1 -- start start 2024-04-21T23:12:10.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.156+0000 7fb6c847c740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556145508c60 0x556145522500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.156+0000 7fb6c847c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 0x5561454fe440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.156+0000 7fb6c847c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145516840 0x5561455156a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.156+0000 7fb6c847c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01aa7b0 con 0x556145509580 2024-04-21T23:12:10.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.156+0000 7fb6c847c740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01ad030 con 0x556145508c60 2024-04-21T23:12:10.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.156+0000 7fb6c847c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01aae20 con 0x556145516840 2024-04-21T23:12:10.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.156+0000 7fb6be7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 0x5561454fe440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.156+0000 7fb6be7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 0x5561454fe440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43434/0 (socket says 172.21.15.44:43434) 2024-04-21T23:12:10.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.156+0000 7fb6be7fc640 1 -- 172.21.15.44:0/1153756696 learned_addr learned my addr 172.21.15.44:0/1153756696 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:10.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.156+0000 7fb6bf7fe640 1 --2- 172.21.15.44:0/1153756696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145516840 0x5561455156a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.156+0000 7fb6beffd640 1 --2- 172.21.15.44:0/1153756696 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556145508c60 0x556145522500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.156+0000 7fb6be7fc640 1 -- 172.21.15.44:0/1153756696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145516840 msgr2=0x5561455156a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:10.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.156+0000 7fb6be7fc640 1 --2- 172.21.15.44:0/1153756696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145516840 0x5561455156a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.156+0000 7fb6be7fc640 1 -- 172.21.15.44:0/1153756696 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556145508c60 msgr2=0x556145522500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:10.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.156+0000 7fb6be7fc640 1 --2- 172.21.15.44:0/1153756696 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556145508c60 0x556145522500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.156+0000 7fb6be7fc640 1 -- 172.21.15.44:0/1153756696 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556145512820 con 0x556145509580 2024-04-21T23:12:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.157+0000 7fb6be7fc640 1 --2- 172.21.15.44:0/1153756696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 0x5561454fe440 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb6a80eeff0 tx=0x7fb6a80bd540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8021487011e7215 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.157+0000 7fb6b57fa640 1 -- 172.21.15.44:0/1153756696 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8087070 con 0x556145509580 2024-04-21T23:12:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.157+0000 7fb6b57fa640 1 -- 172.21.15.44:0/1153756696 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6a80bdfd0 con 0x556145509580 2024-04-21T23:12:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.157+0000 7fb6b57fa640 1 -- 172.21.15.44:0/1153756696 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a804dea0 con 0x556145509580 2024-04-21T23:12:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.157+0000 7fb6c847c740 1 -- 172.21.15.44:0/1153756696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 msgr2=0x5561454fe440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.157+0000 7fb6c847c740 1 --2- 172.21.15.44:0/1153756696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 0x5561454fe440 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb6a80eeff0 tx=0x7fb6a80bd540 comp rx=0 tx=0).stop 2024-04-21T23:12:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.157+0000 7fb6c847c740 1 -- 172.21.15.44:0/1153756696 shutdown_connections 2024-04-21T23:12:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.157+0000 7fb6c847c740 1 --2- 172.21.15.44:0/1153756696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145516840 0x5561455156a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.157+0000 7fb6c847c740 1 --2- 172.21.15.44:0/1153756696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145509580 0x5561454fe440 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.157+0000 7fb6c847c740 1 --2- 172.21.15.44:0/1153756696 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556145508c60 0x556145522500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.157+0000 7fb6c847c740 1 -- 172.21.15.44:0/1153756696 >> 172.21.15.44:0/1153756696 conn(0x556145503660 msgr2=0x556145514f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.158+0000 7fb6c847c740 1 -- 172.21.15.44:0/1153756696 shutdown_connections 2024-04-21T23:12:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.158+0000 7fb6c847c740 1 -- 172.21.15.44:0/1153756696 wait complete. 2024-04-21T23:12:10.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.158+0000 7fb6c847c740 1 -- start start 2024-04-21T23:12:10.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.158+0000 7fb6c847c740 1 Processor -- start 2024-04-21T23:12:10.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.158+0000 7fb6c847c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145508c60 0x55614551a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.158+0000 7fb6c847c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145509580 0x55614551a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.158+0000 7fb6c847c740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556145516840 0x55614551ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.158+0000 7fb6c847c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01ab4f0 con 0x556145508c60 2024-04-21T23:12:10.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.158+0000 7fb6c847c740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01ac960 con 0x556145516840 2024-04-21T23:12:10.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.158+0000 7fb6c847c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c01abbc0 con 0x556145509580 2024-04-21T23:12:10.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.159+0000 7fb6beffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145508c60 0x55614551a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.159+0000 7fb6c847c740 10 client.0 _ll_register_callbacks cb 0x7fb6a80bd6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:10.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.159+0000 7fb6beffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145508c60 0x55614551a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43446/0 (socket says 172.21.15.44:43446) 2024-04-21T23:12:10.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.159+0000 7fb6beffd640 1 -- 172.21.15.44:0/3230988719 learned_addr learned my addr 172.21.15.44:0/3230988719 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:10.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.159+0000 7fb6be7fc640 1 --2- 172.21.15.44:0/3230988719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145509580 0x55614551a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.159+0000 7fb6bf7fe640 1 --2- 172.21.15.44:0/3230988719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556145516840 0x55614551ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.159+0000 7fb6beffd640 1 -- 172.21.15.44:0/3230988719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145509580 msgr2=0x55614551a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:10.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.159+0000 7fb6beffd640 1 --2- 172.21.15.44:0/3230988719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145509580 0x55614551a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.159+0000 7fb6beffd640 1 -- 172.21.15.44:0/3230988719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556145516840 msgr2=0x55614551ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:10.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.159+0000 7fb6beffd640 1 --2- 172.21.15.44:0/3230988719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556145516840 0x55614551ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.159+0000 7fb6beffd640 1 -- 172.21.15.44:0/3230988719 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55614558c110 con 0x556145508c60 2024-04-21T23:12:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.159+0000 7fb6beffd640 1 --2- 172.21.15.44:0/3230988719 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145508c60 0x55614551a3b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8011cb0 tx=0x7fb6b800f6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8689a35ad262dd34 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.159+0000 7fb69effd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:12:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.159+0000 7fb69effd640 1 -- 172.21.15.44:0/3230988719 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b802f060 con 0x556145508c60 2024-04-21T23:12:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.159+0000 7fb69effd640 1 -- 172.21.15.44:0/3230988719 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fb6b8009530 con 0x556145508c60 2024-04-21T23:12:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.159+0000 7fb6c847c740 1 -- 172.21.15.44:0/3230988719 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6a80bdfd0 con 0x556145508c60 2024-04-21T23:12:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.159+0000 7fb69effd640 1 -- 172.21.15.44:0/3230988719 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb6b8036050 con 0x556145508c60 2024-04-21T23:12:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.159+0000 7fb6c847c740 10 client.16443 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.159+0000 7fb6c847c740 1 -- 172.21.15.44:0/3230988719 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55614558d380 con 0x556145508c60 2024-04-21T23:12:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.160+0000 7fb69effd640 1 -- 172.21.15.44:0/3230988719 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6a80bdfd0 con 0x556145508c60 2024-04-21T23:12:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.160+0000 7fb69effd640 1 -- 172.21.15.44:0/3230988719 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fb6b800fd10 con 0x556145508c60 2024-04-21T23:12:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.160+0000 7fb6c847c740 10 client.16443 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.160+0000 7fb6c847c740 10 client.16443 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.160+0000 7fb6c847c740 20 client.16443 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:10.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.160+0000 7fb6c847c740 1 --2- 172.21.15.44:0/3230988719 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55614558d970 0x55614558fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.160+0000 7fb6c847c740 4 client.16443 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:12:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.160+0000 7fb6c847c740 1 -- 172.21.15.44:0/3230988719 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556145474df0 con 0x55614558d970 2024-04-21T23:12:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.160+0000 7fb6be7fc640 1 --2- 172.21.15.44:0/3230988719 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55614558d970 0x55614558fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.160+0000 7fb69effd640 10 client.16443 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.160+0000 7fb6be7fc640 1 --2- 172.21.15.44:0/3230988719 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55614558d970 0x55614558fd90 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0a93947893cd728 server_cookie=523f599e312d1ca2 in_seq=0 out_seq=0 2024-04-21T23:12:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.161+0000 7fb69effd640 1 -- 172.21.15.44:0/3230988719 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556145474df0 con 0x55614558d970 2024-04-21T23:12:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.161+0000 7fb69effd640 10 client.16443 handle_command_reply: tid=0 2024-04-21T23:12:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.161+0000 7fb6c847c740 1 client.16443 shutdown 2024-04-21T23:12:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.161+0000 7fb6c847c740 1 -- 172.21.15.44:0/3230988719 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145508c60 msgr2=0x55614551a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.161+0000 7fb6c847c740 1 --2- 172.21.15.44:0/3230988719 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145508c60 0x55614551a3b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8011cb0 tx=0x7fb6b800f6f0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.161+0000 7fb6c847c740 1 -- 172.21.15.44:0/3230988719 shutdown_connections 2024-04-21T23:12:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.161+0000 7fb6c847c740 1 --2- 172.21.15.44:0/3230988719 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55614558d970 0x55614558fd90 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.161+0000 7fb6c847c740 1 --2- 172.21.15.44:0/3230988719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556145509580 0x55614551a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.161+0000 7fb69effd640 0 client.16443 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:10.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.161+0000 7fb6c847c740 1 --2- 172.21.15.44:0/3230988719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556145516840 0x55614551ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.161+0000 7fb6c847c740 1 --2- 172.21.15.44:0/3230988719 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556145508c60 0x55614551a3b0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.161+0000 7fb6c847c740 1 -- 172.21.15.44:0/3230988719 >> 172.21.15.44:0/3230988719 conn(0x556145503660 msgr2=0x5561455140c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:10.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.161+0000 7fb6c847c740 1 -- 172.21.15.44:0/3230988719 shutdown_connections 2024-04-21T23:12:10.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.161+0000 7fb6c847c740 1 -- 172.21.15.44:0/3230988719 wait complete. 2024-04-21T23:12:10.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.161+0000 7fb6c847c740 20 client.16443 trim_cache size 0 max 16384 2024-04-21T23:12:10.167 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:12:10.167 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:12:10.167 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-21T23:12:10.167 INFO:teuthology.orchestra.run.smithi044.stdout: "ca1725ba-969c-45f5-94d2-6b268722f64b": { 2024-04-21T23:12:10.167 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-21T23:12:10.168 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "ca1725ba-969c-45f5-94d2-6b268722f64b", 2024-04-21T23:12:10.168 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-04-21T23:12:10.168 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:12:10.168 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:12:10.168 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:12:10.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.162+0000 7fb6c5888640 1 -- 172.21.15.44:0/2342781417 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb6ac0a2d00 msgr2=0x7fb6ac0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:10.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.162+0000 7fb6c5888640 1 --2- 172.21.15.44:0/2342781417 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb6ac0a2d00 0x7fb6ac0a51c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0005fd0 tx=0x7fb6b0009eb0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.162+0000 7fb6c5888640 1 -- 172.21.15.44:0/2342781417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6c0103210 msgr2=0x7fb6c0198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:10.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.162+0000 7fb6c5888640 1 --2- 172.21.15.44:0/2342781417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6c0103210 0x7fb6c0198620 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8006010 tx=0x7fb6a8004270 comp rx=0 tx=0).stop 2024-04-21T23:12:10.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.162+0000 7fb6c5888640 1 -- 172.21.15.44:0/2342781417 shutdown_connections 2024-04-21T23:12:10.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.162+0000 7fb6c5888640 1 --2- 172.21.15.44:0/2342781417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6c0103bb0 0x7fb6c019c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.162+0000 7fb6c5888640 1 --2- 172.21.15.44:0/2342781417 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb6ac0a2d00 0x7fb6ac0a51c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.162+0000 7fb6c5888640 1 --2- 172.21.15.44:0/2342781417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6c0103210 0x7fb6c0198620 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.162+0000 7fb6c5888640 1 --2- 172.21.15.44:0/2342781417 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6c0101ad0 0x7fb6c01980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:10.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.162+0000 7fb6c5888640 1 -- 172.21.15.44:0/2342781417 >> 172.21.15.44:0/2342781417 conn(0x7fb6c00fd200 msgr2=0x7fb6c00ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:10.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.162+0000 7fb6c5888640 1 -- 172.21.15.44:0/2342781417 shutdown_connections 2024-04-21T23:12:10.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:10.162+0000 7fb6c5888640 1 -- 172.21.15.44:0/2342781417 wait complete. 2024-04-21T23:12:10.172 INFO:tasks.cephfs.filesystem:scrub status for tag:ca1725ba-969c-45f5-94d2-6b268722f64b - {'path': '/', 'tag': 'ca1725ba-969c-45f5-94d2-6b268722f64b', 'options': 'recursive,force'} 2024-04-21T23:12:10.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:09 smithi069 ceph-mon[31816]: pgmap v104: 129 pgs: 129 active+clean; 45 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 482 op/s 2024-04-21T23:12:11.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:10 smithi044 ceph-mon[27934]: scrub queued for path: / 2024-04-21T23:12:11.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:10 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:12:11.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:10 smithi044 ceph-mon[27934]: scrub summary: active paths [/] 2024-04-21T23:12:11.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:10 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:12:11.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:10 smithi069 ceph-mon[31816]: scrub queued for path: / 2024-04-21T23:12:11.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:10 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:12:11.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:10 smithi069 ceph-mon[31816]: scrub summary: active paths [/] 2024-04-21T23:12:11.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:10 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:12:11.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:10 smithi055 ceph-mon[31117]: scrub queued for path: / 2024-04-21T23:12:11.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:10 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:12:11.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:10 smithi055 ceph-mon[31117]: scrub summary: active paths [/] 2024-04-21T23:12:11.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:10 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:12:11.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 314874 11.29 MB/sec execute 1 sec latency 82.564 ms 2024-04-21T23:12:12.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:11 smithi044 ceph-mon[27934]: pgmap v105: 129 pgs: 129 active+clean; 45 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 443 op/s 2024-04-21T23:12:12.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:11 smithi055 ceph-mon[31117]: pgmap v105: 129 pgs: 129 active+clean; 45 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 443 op/s 2024-04-21T23:12:12.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:11 smithi069 ceph-mon[31816]: pgmap v105: 129 pgs: 129 active+clean; 45 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.3 MiB/s wr, 443 op/s 2024-04-21T23:12:12.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 316859 10.74 MB/sec execute 2 sec latency 35.186 ms 2024-04-21T23:12:13.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:12 smithi044 ceph-mon[27934]: scrub summary: idle 2024-04-21T23:12:13.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:12 smithi055 ceph-mon[31117]: scrub summary: idle 2024-04-21T23:12:13.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:12 smithi069 ceph-mon[31816]: scrub summary: idle 2024-04-21T23:12:13.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 319813 12.85 MB/sec execute 3 sec latency 13.766 ms 2024-04-21T23:12:14.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:13 smithi044 ceph-mon[27934]: pgmap v106: 129 pgs: 129 active+clean; 52 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.9 MiB/s wr, 577 op/s 2024-04-21T23:12:14.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:13 smithi055 ceph-mon[31117]: pgmap v106: 129 pgs: 129 active+clean; 52 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.9 MiB/s wr, 577 op/s 2024-04-21T23:12:14.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:13 smithi069 ceph-mon[31816]: pgmap v106: 129 pgs: 129 active+clean; 52 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.9 MiB/s wr, 577 op/s 2024-04-21T23:12:14.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 322803 14.00 MB/sec execute 4 sec latency 27.545 ms 2024-04-21T23:12:15.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 324499 12.59 MB/sec execute 5 sec latency 60.009 ms 2024-04-21T23:12:16.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:15 smithi044 ceph-mon[27934]: pgmap v107: 129 pgs: 129 active+clean; 52 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 432 op/s 2024-04-21T23:12:16.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:15 smithi069 ceph-mon[31816]: pgmap v107: 129 pgs: 129 active+clean; 52 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 432 op/s 2024-04-21T23:12:16.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:15 smithi055 ceph-mon[31117]: pgmap v107: 129 pgs: 129 active+clean; 52 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 432 op/s 2024-04-21T23:12:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.601+0000 7fb27f335640 1 -- 172.21.15.44:0/371091875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb278101730 msgr2=0x7fb278101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.601+0000 7fb27f335640 1 --2- 172.21.15.44:0/371091875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb278101730 0x7fb278101bb0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb268009930 tx=0x7fb26802bc30 comp rx=0 tx=0).stop 2024-04-21T23:12:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.601+0000 7fb27f335640 1 -- 172.21.15.44:0/371091875 shutdown_connections 2024-04-21T23:12:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.601+0000 7fb27f335640 1 --2- 172.21.15.44:0/371091875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2781020f0 0x7fb27810e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.601+0000 7fb27f335640 1 --2- 172.21.15.44:0/371091875 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb278101730 0x7fb278101bb0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.601+0000 7fb27f335640 1 --2- 172.21.15.44:0/371091875 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb278107740 0x7fb278107b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.601+0000 7fb27f335640 1 -- 172.21.15.44:0/371091875 >> 172.21.15.44:0/371091875 conn(0x7fb2780fd200 msgr2=0x7fb2780ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.601+0000 7fb27f335640 1 -- 172.21.15.44:0/371091875 shutdown_connections 2024-04-21T23:12:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.601+0000 7fb27f335640 1 -- 172.21.15.44:0/371091875 wait complete. 2024-04-21T23:12:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.602+0000 7fb27f335640 1 Processor -- start 2024-04-21T23:12:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.602+0000 7fb27f335640 1 -- start start 2024-04-21T23:12:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.602+0000 7fb27f335640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb278101730 0x7fb278073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.602+0000 7fb27f335640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2781020f0 0x7fb278073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.602+0000 7fb27f335640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb278107740 0x7fb2780744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.602+0000 7fb27f335640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb278111b50 con 0x7fb278107740 2024-04-21T23:12:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.602+0000 7fb27f335640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb2781119d0 con 0x7fb278101730 2024-04-21T23:12:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.602+0000 7fb27f335640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb278111cd0 con 0x7fb2781020f0 2024-04-21T23:12:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.602+0000 7fb27d8d2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb278107740 0x7fb2780744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.602+0000 7fb27d8d2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb278107740 0x7fb2780744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56520/0 (socket says 172.21.15.44:56520) 2024-04-21T23:12:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.602+0000 7fb27d8d2640 1 -- 172.21.15.44:0/590443951 learned_addr learned my addr 172.21.15.44:0/590443951 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.602+0000 7fb27c8d0640 1 --2- 172.21.15.44:0/590443951 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2781020f0 0x7fb278073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.602+0000 7fb27d0d1640 1 --2- 172.21.15.44:0/590443951 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb278101730 0x7fb278073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.603+0000 7fb27d8d2640 1 -- 172.21.15.44:0/590443951 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2781020f0 msgr2=0x7fb278073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.603+0000 7fb27d8d2640 1 --2- 172.21.15.44:0/590443951 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2781020f0 0x7fb278073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.603+0000 7fb27d8d2640 1 -- 172.21.15.44:0/590443951 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb278101730 msgr2=0x7fb278073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.603+0000 7fb27d8d2640 1 --2- 172.21.15.44:0/590443951 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb278101730 0x7fb278073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.603+0000 7fb27d8d2640 1 -- 172.21.15.44:0/590443951 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2781a6530 con 0x7fb278107740 2024-04-21T23:12:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.603+0000 7fb27d0d1640 1 --2- 172.21.15.44:0/590443951 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb278101730 0x7fb278073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:16.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.603+0000 7fb27d8d2640 1 --2- 172.21.15.44:0/590443951 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb278107740 0x7fb2780744c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb27000eaa0 tx=0x7fb27000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:16.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.603+0000 7fb26e7fc640 1 -- 172.21.15.44:0/590443951 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27000cdb0 con 0x7fb278107740 2024-04-21T23:12:16.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.603+0000 7fb26e7fc640 1 -- 172.21.15.44:0/590443951 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb270005020 con 0x7fb278107740 2024-04-21T23:12:16.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.603+0000 7fb27f335640 1 -- 172.21.15.44:0/590443951 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2781a6820 con 0x7fb278107740 2024-04-21T23:12:16.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.603+0000 7fb26e7fc640 1 -- 172.21.15.44:0/590443951 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2700116b0 con 0x7fb278107740 2024-04-21T23:12:16.606 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.603+0000 7fb27f335640 1 -- 172.21.15.44:0/590443951 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2781a6d90 con 0x7fb278107740 2024-04-21T23:12:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.605+0000 7fb26e7fc640 1 -- 172.21.15.44:0/590443951 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb270005560 con 0x7fb278107740 2024-04-21T23:12:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb281ff4740 1 Processor -- start 2024-04-21T23:12:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb281ff4740 1 -- start start 2024-04-21T23:12:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb26e7fc640 1 --2- 172.21.15.44:0/590443951 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb2640a2e20 0x7fb2640a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb281ff4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 0x5644e620ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb281ff4740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e6210440 0x5644e6218cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb281ff4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644e6219210 0x5644e621b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb281ff4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5644e6173b70 con 0x5644e620fb20 2024-04-21T23:12:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb281ff4740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5644e611c470 con 0x5644e6210440 2024-04-21T23:12:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb281ff4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5644e616fb30 con 0x5644e6219210 2024-04-21T23:12:16.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb27c8d0640 1 --2- 172.21.15.44:0/590443951 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb2640a2e20 0x7fb2640a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb27d0d1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 0x5644e620ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb26e7fc640 1 -- 172.21.15.44:0/590443951 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fb2700cf940 con 0x7fb278107740 2024-04-21T23:12:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb27d0d1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 0x5644e620ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56524/0 (socket says 172.21.15.44:56524) 2024-04-21T23:12:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb27d0d1640 1 -- 172.21.15.44:0/2759201857 learned_addr learned my addr 172.21.15.44:0/2759201857 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb27c8d0640 1 --2- 172.21.15.44:0/2759201857 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e6210440 0x5644e6218cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb27d8d2640 1 --2- 172.21.15.44:0/2759201857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644e6219210 0x5644e621b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb27d0d1640 1 -- 172.21.15.44:0/2759201857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644e6219210 msgr2=0x5644e621b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb27d0d1640 1 --2- 172.21.15.44:0/2759201857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644e6219210 0x5644e621b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb27d0d1640 1 -- 172.21.15.44:0/2759201857 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e6210440 msgr2=0x5644e6218cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb27d0d1640 1 --2- 172.21.15.44:0/2759201857 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e6210440 0x5644e6218cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.606+0000 7fb27d0d1640 1 -- 172.21.15.44:0/2759201857 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644e5e88210 con 0x5644e620fb20 2024-04-21T23:12:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.607+0000 7fb27c8d0640 1 --2- 172.21.15.44:0/590443951 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb2640a2e20 0x7fb2640a52e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb268005ab0 tx=0x7fb2680095d0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:16.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.607+0000 7fb27d0d1640 1 --2- 172.21.15.44:0/2759201857 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 0x5644e620ff00 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb2740114f0 tx=0x7fb2740119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf23b5708661e3f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.607+0000 7fb2637fe640 1 -- 172.21.15.44:0/2759201857 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27401e070 con 0x5644e620fb20 2024-04-21T23:12:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.607+0000 7fb2637fe640 1 -- 172.21.15.44:0/2759201857 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb274011250 con 0x5644e620fb20 2024-04-21T23:12:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.607+0000 7fb2637fe640 1 -- 172.21.15.44:0/2759201857 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2740199b0 con 0x5644e620fb20 2024-04-21T23:12:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.608+0000 7fb281ff4740 1 -- 172.21.15.44:0/2759201857 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 msgr2=0x5644e620ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.608+0000 7fb281ff4740 1 --2- 172.21.15.44:0/2759201857 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 0x5644e620ff00 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb2740114f0 tx=0x7fb2740119b0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.608+0000 7fb281ff4740 1 -- 172.21.15.44:0/2759201857 shutdown_connections 2024-04-21T23:12:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.608+0000 7fb281ff4740 1 --2- 172.21.15.44:0/2759201857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644e6219210 0x5644e621b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.608+0000 7fb281ff4740 1 --2- 172.21.15.44:0/2759201857 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e6210440 0x5644e6218cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.608+0000 7fb281ff4740 1 --2- 172.21.15.44:0/2759201857 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 0x5644e620ff00 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.608+0000 7fb281ff4740 1 -- 172.21.15.44:0/2759201857 >> 172.21.15.44:0/2759201857 conn(0x5644e620a440 msgr2=0x5644e620a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.608+0000 7fb281ff4740 1 -- 172.21.15.44:0/2759201857 shutdown_connections 2024-04-21T23:12:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.608+0000 7fb281ff4740 1 -- 172.21.15.44:0/2759201857 wait complete. 2024-04-21T23:12:16.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.608+0000 7fb281ff4740 1 -- start start 2024-04-21T23:12:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.608+0000 7fb281ff4740 1 Processor -- start 2024-04-21T23:12:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.608+0000 7fb281ff4740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e620fb20 0x5644e6216220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.608+0000 7fb281ff4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e6210440 0x5644e6216760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.608+0000 7fb281ff4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644e6219210 0x5644e6216ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.608+0000 7fb281ff4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5644e621be40 con 0x5644e6210440 2024-04-21T23:12:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.608+0000 7fb281ff4740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5644e612fa10 con 0x5644e620fb20 2024-04-21T23:12:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.608+0000 7fb281ff4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5644e621bfc0 con 0x5644e6219210 2024-04-21T23:12:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.608+0000 7fb281ff4740 10 client.0 _ll_register_callbacks cb 0x5644e616ab60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.609+0000 7fb27c8d0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e6210440 0x5644e6216760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.609+0000 7fb27d0d1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e620fb20 0x5644e6216220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.609+0000 7fb27d0d1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e620fb20 0x5644e6216220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:43182/0 (socket says 172.21.15.44:43182) 2024-04-21T23:12:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.609+0000 7fb27c8d0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e6210440 0x5644e6216760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56530/0 (socket says 172.21.15.44:56530) 2024-04-21T23:12:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.609+0000 7fb27d0d1640 1 -- 172.21.15.44:0/2442113484 learned_addr learned my addr 172.21.15.44:0/2442113484 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:16.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.609+0000 7fb27d8d2640 1 --2- 172.21.15.44:0/2442113484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644e6219210 0x5644e6216ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.609+0000 7fb27d0d1640 1 -- 172.21.15.44:0/2442113484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644e6219210 msgr2=0x5644e6216ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.609+0000 7fb27d0d1640 1 --2- 172.21.15.44:0/2442113484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644e6219210 0x5644e6216ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.609+0000 7fb27d0d1640 1 -- 172.21.15.44:0/2442113484 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e6210440 msgr2=0x5644e6216760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.609+0000 7fb27d0d1640 1 --2- 172.21.15.44:0/2442113484 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e6210440 0x5644e6216760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.609+0000 7fb27d0d1640 1 -- 172.21.15.44:0/2442113484 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644e5f54620 con 0x5644e620fb20 2024-04-21T23:12:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.609+0000 7fb27c8d0640 1 --2- 172.21.15.44:0/2442113484 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e6210440 0x5644e6216760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.609+0000 7fb27d0d1640 1 --2- 172.21.15.44:0/2442113484 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e620fb20 0x5644e6216220 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb274005640 tx=0x7fb274005670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=801b7c23187a1a11 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.609+0000 7fb260ff9640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:12:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.609+0000 7fb260ff9640 1 -- 172.21.15.44:0/2442113484 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27401e040 con 0x5644e620fb20 2024-04-21T23:12:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.609+0000 7fb281ff4740 1 -- 172.21.15.44:0/2442113484 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb274011250 con 0x5644e620fb20 2024-04-21T23:12:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.609+0000 7fb260ff9640 1 -- 172.21.15.44:0/2442113484 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fb2740196f0 con 0x5644e620fb20 2024-04-21T23:12:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.609+0000 7fb260ff9640 1 -- 172.21.15.44:0/2442113484 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb274030050 con 0x5644e620fb20 2024-04-21T23:12:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.610+0000 7fb281ff4740 10 client.26021 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:16.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.610+0000 7fb281ff4740 1 -- 172.21.15.44:0/2442113484 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5644e6228080 con 0x5644e620fb20 2024-04-21T23:12:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.610+0000 7fb260ff9640 1 -- 172.21.15.44:0/2442113484 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb274011250 con 0x5644e620fb20 2024-04-21T23:12:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.610+0000 7fb260ff9640 1 -- 172.21.15.44:0/2442113484 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fb274021070 con 0x5644e620fb20 2024-04-21T23:12:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.610+0000 7fb281ff4740 10 client.26021 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.610+0000 7fb281ff4740 10 client.26021 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.610+0000 7fb281ff4740 20 client.26021 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.610+0000 7fb281ff4740 1 --2- 172.21.15.44:0/2442113484 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5644e6229430 0x5644e622b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.610+0000 7fb281ff4740 4 client.26021 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:12:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.610+0000 7fb281ff4740 1 -- 172.21.15.44:0/2442113484 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5644e61810b0 con 0x5644e6229430 2024-04-21T23:12:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.610+0000 7fb27c8d0640 1 --2- 172.21.15.44:0/2442113484 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5644e6229430 0x5644e622b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.611+0000 7fb27c8d0640 1 --2- 172.21.15.44:0/2442113484 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5644e6229430 0x5644e622b820 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de06896586361c12 server_cookie=8d28e971bd4aeba5 in_seq=0 out_seq=0 2024-04-21T23:12:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.611+0000 7fb260ff9640 10 client.26021 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.612+0000 7fb260ff9640 1 -- 172.21.15.44:0/2442113484 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5644e61810b0 con 0x5644e6229430 2024-04-21T23:12:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.612+0000 7fb260ff9640 10 client.26021 handle_command_reply: tid=0 2024-04-21T23:12:16.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.612+0000 7fb281ff4740 1 client.26021 shutdown 2024-04-21T23:12:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.612+0000 7fb281ff4740 1 -- 172.21.15.44:0/2442113484 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e620fb20 msgr2=0x5644e6216220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.612+0000 7fb281ff4740 1 --2- 172.21.15.44:0/2442113484 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e620fb20 0x5644e6216220 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb274005640 tx=0x7fb274005670 comp rx=0 tx=0).stop 2024-04-21T23:12:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.612+0000 7fb281ff4740 1 -- 172.21.15.44:0/2442113484 shutdown_connections 2024-04-21T23:12:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.612+0000 7fb281ff4740 1 --2- 172.21.15.44:0/2442113484 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5644e6229430 0x5644e622b820 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.612+0000 7fb281ff4740 1 --2- 172.21.15.44:0/2442113484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644e6219210 0x5644e6216ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.612+0000 7fb281ff4740 1 --2- 172.21.15.44:0/2442113484 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e6210440 0x5644e6216760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.612+0000 7fb281ff4740 1 --2- 172.21.15.44:0/2442113484 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e620fb20 0x5644e6216220 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.612+0000 7fb260ff9640 0 client.26021 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.612+0000 7fb281ff4740 1 -- 172.21.15.44:0/2442113484 >> 172.21.15.44:0/2442113484 conn(0x5644e620a440 msgr2=0x5644e620c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.613+0000 7fb281ff4740 1 -- 172.21.15.44:0/2442113484 shutdown_connections 2024-04-21T23:12:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.613+0000 7fb281ff4740 1 -- 172.21.15.44:0/2442113484 wait complete. 2024-04-21T23:12:16.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.613+0000 7fb281ff4740 20 client.26021 trim_cache size 0 max 16384 2024-04-21T23:12:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.628+0000 7fb281ff4740 1 Processor -- start 2024-04-21T23:12:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.628+0000 7fb281ff4740 1 -- start start 2024-04-21T23:12:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.628+0000 7fb281ff4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 0x5644e6218950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.628+0000 7fb281ff4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644e6210440 0x5644e6224d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.628+0000 7fb281ff4740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e6219210 0x5644e6230fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.628+0000 7fb281ff4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb2781aeec0 con 0x5644e620fb20 2024-04-21T23:12:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.628+0000 7fb281ff4740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb2781b1740 con 0x5644e6219210 2024-04-21T23:12:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.628+0000 7fb281ff4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb2781af530 con 0x5644e6210440 2024-04-21T23:12:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.628+0000 7fb27d0d1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 0x5644e6218950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.628+0000 7fb27d0d1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 0x5644e6218950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56542/0 (socket says 172.21.15.44:56542) 2024-04-21T23:12:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.628+0000 7fb27d0d1640 1 -- 172.21.15.44:0/1391224003 learned_addr learned my addr 172.21.15.44:0/1391224003 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.628+0000 7fb27d8d2640 1 --2- 172.21.15.44:0/1391224003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e6219210 0x5644e6230fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.629+0000 7fb27c8d0640 1 --2- 172.21.15.44:0/1391224003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644e6210440 0x5644e6224d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.629+0000 7fb27d0d1640 1 -- 172.21.15.44:0/1391224003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644e6210440 msgr2=0x5644e6224d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.629+0000 7fb27d0d1640 1 --2- 172.21.15.44:0/1391224003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644e6210440 0x5644e6224d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.629+0000 7fb27d0d1640 1 -- 172.21.15.44:0/1391224003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e6219210 msgr2=0x5644e6230fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.629+0000 7fb27d0d1640 1 --2- 172.21.15.44:0/1391224003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e6219210 0x5644e6230fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.629+0000 7fb27d0d1640 1 -- 172.21.15.44:0/1391224003 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644e6210270 con 0x5644e620fb20 2024-04-21T23:12:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.629+0000 7fb27d0d1640 1 --2- 172.21.15.44:0/1391224003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 0x5644e6218950 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb2740194a0 tx=0x7fb2740056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e70abb42c6f0e7d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.629+0000 7fb262ffd640 1 -- 172.21.15.44:0/1391224003 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27401e030 con 0x5644e620fb20 2024-04-21T23:12:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.629+0000 7fb262ffd640 1 -- 172.21.15.44:0/1391224003 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb274027070 con 0x5644e620fb20 2024-04-21T23:12:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.629+0000 7fb262ffd640 1 -- 172.21.15.44:0/1391224003 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb274019900 con 0x5644e620fb20 2024-04-21T23:12:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.630+0000 7fb281ff4740 1 -- 172.21.15.44:0/1391224003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 msgr2=0x5644e6218950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.630+0000 7fb281ff4740 1 --2- 172.21.15.44:0/1391224003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 0x5644e6218950 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb2740194a0 tx=0x7fb2740056e0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.630+0000 7fb281ff4740 1 -- 172.21.15.44:0/1391224003 shutdown_connections 2024-04-21T23:12:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.630+0000 7fb281ff4740 1 --2- 172.21.15.44:0/1391224003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644e6210440 0x5644e6224d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.630+0000 7fb281ff4740 1 --2- 172.21.15.44:0/1391224003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e6219210 0x5644e6230fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.630+0000 7fb281ff4740 1 --2- 172.21.15.44:0/1391224003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 0x5644e6218950 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.630+0000 7fb281ff4740 1 -- 172.21.15.44:0/1391224003 >> 172.21.15.44:0/1391224003 conn(0x5644e620a440 msgr2=0x5644e620a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.630+0000 7fb281ff4740 1 -- 172.21.15.44:0/1391224003 shutdown_connections 2024-04-21T23:12:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.630+0000 7fb281ff4740 1 -- 172.21.15.44:0/1391224003 wait complete. 2024-04-21T23:12:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.630+0000 7fb281ff4740 1 -- start start 2024-04-21T23:12:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.630+0000 7fb281ff4740 1 Processor -- start 2024-04-21T23:12:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb281ff4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 0x5644e6221210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb281ff4740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e6210440 0x5644e6221750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb281ff4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644e6219210 0x5644e6212670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb281ff4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb2781afc00 con 0x5644e620fb20 2024-04-21T23:12:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb281ff4740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb2781b1070 con 0x5644e6210440 2024-04-21T23:12:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb281ff4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb2781b02d0 con 0x5644e6219210 2024-04-21T23:12:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb281ff4740 10 client.0 _ll_register_callbacks cb 0x7fb27401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb27d0d1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 0x5644e6221210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb27d0d1640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 0x5644e6221210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:56558/0 (socket says 172.21.15.44:56558) 2024-04-21T23:12:16.635 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb27d0d1640 1 -- 172.21.15.44:0/3695004033 learned_addr learned my addr 172.21.15.44:0/3695004033 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb27c8d0640 1 --2- 172.21.15.44:0/3695004033 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e6210440 0x5644e6221750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb27d8d2640 1 --2- 172.21.15.44:0/3695004033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644e6219210 0x5644e6212670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb27d0d1640 1 -- 172.21.15.44:0/3695004033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644e6219210 msgr2=0x5644e6212670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb27d0d1640 1 --2- 172.21.15.44:0/3695004033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644e6219210 0x5644e6212670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb27d0d1640 1 -- 172.21.15.44:0/3695004033 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e6210440 msgr2=0x5644e6221750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb27d0d1640 1 --2- 172.21.15.44:0/3695004033 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e6210440 0x5644e6221750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb27d0d1640 1 -- 172.21.15.44:0/3695004033 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644e6212e70 con 0x5644e620fb20 2024-04-21T23:12:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb27d0d1640 1 --2- 172.21.15.44:0/3695004033 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 0x5644e6221210 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fb274009530 tx=0x7fb274009d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da24589927db991f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb260ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:12:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb260ff9640 1 -- 172.21.15.44:0/3695004033 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27401e030 con 0x5644e620fb20 2024-04-21T23:12:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb260ff9640 1 -- 172.21.15.44:0/3695004033 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fb27400fa50 con 0x5644e620fb20 2024-04-21T23:12:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.631+0000 7fb281ff4740 1 -- 172.21.15.44:0/3695004033 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb274027070 con 0x5644e620fb20 2024-04-21T23:12:16.636 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.632+0000 7fb260ff9640 1 -- 172.21.15.44:0/3695004033 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb274041050 con 0x5644e620fb20 2024-04-21T23:12:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.632+0000 7fb281ff4740 10 client.16461 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.632+0000 7fb281ff4740 1 -- 172.21.15.44:0/3695004033 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5644e628f330 con 0x5644e620fb20 2024-04-21T23:12:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.632+0000 7fb260ff9640 1 -- 172.21.15.44:0/3695004033 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb274027070 con 0x5644e620fb20 2024-04-21T23:12:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.632+0000 7fb260ff9640 1 -- 172.21.15.44:0/3695004033 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fb274021040 con 0x5644e620fb20 2024-04-21T23:12:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.632+0000 7fb281ff4740 10 client.16461 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.632+0000 7fb281ff4740 10 client.16461 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.632+0000 7fb281ff4740 20 client.16461 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.632+0000 7fb281ff4740 1 --2- 172.21.15.44:0/3695004033 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5644e628f930 0x5644e6291d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.632+0000 7fb281ff4740 4 client.16461 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:12:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.632+0000 7fb281ff4740 1 -- 172.21.15.44:0/3695004033 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5644e617b400 con 0x5644e628f930 2024-04-21T23:12:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.632+0000 7fb27c8d0640 1 --2- 172.21.15.44:0/3695004033 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5644e628f930 0x5644e6291d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.633+0000 7fb260ff9640 10 client.16461 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:16.637 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.633+0000 7fb27c8d0640 1 --2- 172.21.15.44:0/3695004033 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5644e628f930 0x5644e6291d20 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf4f437ffc776823 server_cookie=d1d4be5b04eaff07 in_seq=0 out_seq=0 2024-04-21T23:12:16.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 327014 12.83 MB/sec execute 6 sec latency 31.824 ms 2024-04-21T23:12:16.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.945+0000 7fb260ff9640 1 -- 172.21.15.44:0/3695004033 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5644e617b400 con 0x5644e628f930 2024-04-21T23:12:16.945 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.945+0000 7fb260ff9640 10 client.16461 handle_command_reply: tid=0 2024-04-21T23:12:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.945+0000 7fb281ff4740 1 client.16461 shutdown 2024-04-21T23:12:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.945+0000 7fb281ff4740 1 -- 172.21.15.44:0/3695004033 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 msgr2=0x5644e6221210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.945+0000 7fb281ff4740 1 --2- 172.21.15.44:0/3695004033 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 0x5644e6221210 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fb274009530 tx=0x7fb274009d80 comp rx=0 tx=0).stop 2024-04-21T23:12:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.945+0000 7fb281ff4740 1 -- 172.21.15.44:0/3695004033 shutdown_connections 2024-04-21T23:12:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.945+0000 7fb281ff4740 1 --2- 172.21.15.44:0/3695004033 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5644e628f930 0x5644e6291d20 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.945+0000 7fb281ff4740 1 --2- 172.21.15.44:0/3695004033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5644e6219210 0x5644e6212670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.945+0000 7fb281ff4740 1 --2- 172.21.15.44:0/3695004033 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5644e6210440 0x5644e6221750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.945+0000 7fb260ff9640 0 client.16461 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.945+0000 7fb281ff4740 1 --2- 172.21.15.44:0/3695004033 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5644e620fb20 0x5644e6221210 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.945+0000 7fb281ff4740 1 -- 172.21.15.44:0/3695004033 >> 172.21.15.44:0/3695004033 conn(0x5644e620a440 msgr2=0x5644e6208c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.946+0000 7fb281ff4740 1 -- 172.21.15.44:0/3695004033 shutdown_connections 2024-04-21T23:12:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.946+0000 7fb281ff4740 1 -- 172.21.15.44:0/3695004033 wait complete. 2024-04-21T23:12:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.946+0000 7fb281ff4740 20 client.16461 trim_cache size 0 max 16384 2024-04-21T23:12:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:12:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:12:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:12:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:12:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.946+0000 7fb27f335640 1 -- 172.21.15.44:0/590443951 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb2640a2e20 msgr2=0x7fb2640a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.946+0000 7fb27f335640 1 --2- 172.21.15.44:0/590443951 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb2640a2e20 0x7fb2640a52e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb268005ab0 tx=0x7fb2680095d0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.946+0000 7fb27f335640 1 -- 172.21.15.44:0/590443951 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb278107740 msgr2=0x7fb2780744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.946+0000 7fb27f335640 1 --2- 172.21.15.44:0/590443951 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb278107740 0x7fb2780744c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb27000eaa0 tx=0x7fb27000ef60 comp rx=0 tx=0).stop 2024-04-21T23:12:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.946+0000 7fb27f335640 1 -- 172.21.15.44:0/590443951 shutdown_connections 2024-04-21T23:12:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.946+0000 7fb27f335640 1 --2- 172.21.15.44:0/590443951 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2781020f0 0x7fb278073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.946+0000 7fb27f335640 1 --2- 172.21.15.44:0/590443951 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb2640a2e20 0x7fb2640a52e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.946+0000 7fb27f335640 1 --2- 172.21.15.44:0/590443951 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb278107740 0x7fb2780744c0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.946+0000 7fb27f335640 1 --2- 172.21.15.44:0/590443951 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb278101730 0x7fb278073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.946+0000 7fb27f335640 1 -- 172.21.15.44:0/590443951 >> 172.21.15.44:0/590443951 conn(0x7fb2780fd200 msgr2=0x7fb2780fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.947+0000 7fb27f335640 1 -- 172.21.15.44:0/590443951 shutdown_connections 2024-04-21T23:12:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:16.947+0000 7fb27f335640 1 -- 172.21.15.44:0/590443951 wait complete. 2024-04-21T23:12:17.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 329359 12.77 MB/sec execute 7 sec latency 134.722 ms 2024-04-21T23:12:18.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:17 smithi044 ceph-mon[27934]: pgmap v108: 129 pgs: 129 active+clean; 58 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.5 MiB/s wr, 531 op/s 2024-04-21T23:12:18.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:17 smithi055 ceph-mon[31117]: pgmap v108: 129 pgs: 129 active+clean; 58 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.5 MiB/s wr, 531 op/s 2024-04-21T23:12:18.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:17 smithi069 ceph-mon[31816]: pgmap v108: 129 pgs: 129 active+clean; 58 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.5 MiB/s wr, 531 op/s 2024-04-21T23:12:18.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 331711 12.66 MB/sec execute 8 sec latency 28.293 ms 2024-04-21T23:12:19.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 334213 12.83 MB/sec execute 9 sec latency 15.232 ms 2024-04-21T23:12:20.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:19 smithi044 ceph-mon[27934]: pgmap v109: 129 pgs: 129 active+clean; 58 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.2 MiB/s wr, 404 op/s 2024-04-21T23:12:20.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:19 smithi055 ceph-mon[31117]: pgmap v109: 129 pgs: 129 active+clean; 58 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.2 MiB/s wr, 404 op/s 2024-04-21T23:12:20.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:19 smithi069 ceph-mon[31816]: pgmap v109: 129 pgs: 129 active+clean; 58 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.2 MiB/s wr, 404 op/s 2024-04-21T23:12:20.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 337061 13.27 MB/sec execute 10 sec latency 115.920 ms 2024-04-21T23:12:21.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 338870 12.69 MB/sec execute 11 sec latency 33.588 ms 2024-04-21T23:12:22.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:21 smithi044 ceph-mon[27934]: pgmap v110: 129 pgs: 129 active+clean; 58 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 263 op/s 2024-04-21T23:12:22.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:21 smithi055 ceph-mon[31117]: pgmap v110: 129 pgs: 129 active+clean; 58 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 263 op/s 2024-04-21T23:12:22.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:21 smithi069 ceph-mon[31816]: pgmap v110: 129 pgs: 129 active+clean; 58 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 263 op/s 2024-04-21T23:12:22.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 341309 12.81 MB/sec execute 12 sec latency 14.456 ms 2024-04-21T23:12:23.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 344440 13.26 MB/sec execute 13 sec latency 17.175 ms 2024-04-21T23:12:24.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:23 smithi044 ceph-mon[27934]: pgmap v111: 129 pgs: 129 active+clean; 47 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.9 MiB/s wr, 563 op/s 2024-04-21T23:12:24.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:23 smithi055 ceph-mon[31117]: pgmap v111: 129 pgs: 129 active+clean; 47 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.9 MiB/s wr, 563 op/s 2024-04-21T23:12:24.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:23 smithi069 ceph-mon[31816]: pgmap v111: 129 pgs: 129 active+clean; 47 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.9 MiB/s wr, 563 op/s 2024-04-21T23:12:24.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 346967 13.16 MB/sec execute 14 sec latency 81.122 ms 2024-04-21T23:12:25.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 348790 12.90 MB/sec execute 15 sec latency 75.075 ms 2024-04-21T23:12:26.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:25 smithi044 ceph-mon[27934]: pgmap v112: 129 pgs: 129 active+clean; 47 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 430 op/s 2024-04-21T23:12:26.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:25 smithi055 ceph-mon[31117]: pgmap v112: 129 pgs: 129 active+clean; 47 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 430 op/s 2024-04-21T23:12:26.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:25 smithi069 ceph-mon[31816]: pgmap v112: 129 pgs: 129 active+clean; 47 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.2 MiB/s wr, 430 op/s 2024-04-21T23:12:26.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 351550 13.16 MB/sec execute 16 sec latency 26.184 ms 2024-04-21T23:12:27.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 354684 13.41 MB/sec execute 17 sec latency 33.320 ms 2024-04-21T23:12:28.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:27 smithi044 ceph-mon[27934]: pgmap v113: 129 pgs: 129 active+clean; 52 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 516 op/s 2024-04-21T23:12:28.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:27 smithi055 ceph-mon[31117]: pgmap v113: 129 pgs: 129 active+clean; 52 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 516 op/s 2024-04-21T23:12:28.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:27 smithi069 ceph-mon[31816]: pgmap v113: 129 pgs: 129 active+clean; 52 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 516 op/s 2024-04-21T23:12:28.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 357107 13.16 MB/sec execute 18 sec latency 24.459 ms 2024-04-21T23:12:29.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 359123 13.13 MB/sec execute 19 sec latency 98.191 ms 2024-04-21T23:12:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.039+0000 7f722fcf4640 1 -- 172.21.15.44:0/414389184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72281009d0 msgr2=0x7f722810e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:30.041 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.039+0000 7f722fcf4640 1 --2- 172.21.15.44:0/414389184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72281009d0 0x7f722810e640 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7220009ca0 tx=0x7f722002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.039+0000 7f722fcf4640 1 -- 172.21.15.44:0/414389184 shutdown_connections 2024-04-21T23:12:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.039+0000 7f722fcf4640 1 --2- 172.21.15.44:0/414389184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72281009d0 0x7f722810e640 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.039+0000 7f722fcf4640 1 --2- 172.21.15.44:0/414389184 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7228100010 0x7f7228100490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.039+0000 7f722fcf4640 1 --2- 172.21.15.44:0/414389184 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72281067b0 0x7f7228106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.039+0000 7f722fcf4640 1 -- 172.21.15.44:0/414389184 >> 172.21.15.44:0/414389184 conn(0x7f7228075990 msgr2=0x7f7228075da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.039+0000 7f722fcf4640 1 -- 172.21.15.44:0/414389184 shutdown_connections 2024-04-21T23:12:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.039+0000 7f722fcf4640 1 -- 172.21.15.44:0/414389184 wait complete. 2024-04-21T23:12:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.040+0000 7f722fcf4640 1 Processor -- start 2024-04-21T23:12:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.040+0000 7f722fcf4640 1 -- start start 2024-04-21T23:12:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.040+0000 7f722fcf4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7228100010 0x7f7228198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:30.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.040+0000 7f722fcf4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72281009d0 0x7f7228198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.040+0000 7f722fcf4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72281067b0 0x7f722819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.040+0000 7f722fcf4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7228111be0 con 0x7f72281009d0 2024-04-21T23:12:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.040+0000 7f722fcf4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7228111a60 con 0x7f7228100010 2024-04-21T23:12:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.040+0000 7f722fcf4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7228111d60 con 0x7f72281067b0 2024-04-21T23:12:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.040+0000 7f722d28f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72281009d0 0x7f7228198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.040+0000 7f722d28f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72281009d0 0x7f7228198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51434/0 (socket says 172.21.15.44:51434) 2024-04-21T23:12:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.040+0000 7f722d28f640 1 -- 172.21.15.44:0/1519620848 learned_addr learned my addr 172.21.15.44:0/1519620848 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.040+0000 7f722da90640 1 --2- 172.21.15.44:0/1519620848 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7228100010 0x7f7228198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.040+0000 7f722e291640 1 --2- 172.21.15.44:0/1519620848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72281067b0 0x7f722819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.041+0000 7f722d28f640 1 -- 172.21.15.44:0/1519620848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72281067b0 msgr2=0x7f722819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.041+0000 7f722d28f640 1 --2- 172.21.15.44:0/1519620848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72281067b0 0x7f722819c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.041+0000 7f722d28f640 1 -- 172.21.15.44:0/1519620848 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7228100010 msgr2=0x7f7228198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:30.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.041+0000 7f722d28f640 1 --2- 172.21.15.44:0/1519620848 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7228100010 0x7f7228198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.041+0000 7f722d28f640 1 -- 172.21.15.44:0/1519620848 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f722819cfc0 con 0x7f72281009d0 2024-04-21T23:12:30.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.041+0000 7f722da90640 1 --2- 172.21.15.44:0/1519620848 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7228100010 0x7f7228198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:30.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.041+0000 7f722d28f640 1 --2- 172.21.15.44:0/1519620848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72281009d0 0x7f7228198560 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f721800e9f0 tx=0x7f721800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:30.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.041+0000 7f721effd640 1 -- 172.21.15.44:0/1519620848 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f721800ccc0 con 0x7f72281009d0 2024-04-21T23:12:30.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.041+0000 7f722fcf4640 1 -- 172.21.15.44:0/1519620848 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f722819d2b0 con 0x7f72281009d0 2024-04-21T23:12:30.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.041+0000 7f721effd640 1 -- 172.21.15.44:0/1519620848 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7218010910 con 0x7f72281009d0 2024-04-21T23:12:30.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.041+0000 7f721effd640 1 -- 172.21.15.44:0/1519620848 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72180188a0 con 0x7f72281009d0 2024-04-21T23:12:30.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.041+0000 7f722fcf4640 1 -- 172.21.15.44:0/1519620848 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72281a4b40 con 0x7f72281009d0 2024-04-21T23:12:30.050 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.043+0000 7f721effd640 1 -- 172.21.15.44:0/1519620848 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7218010ac0 con 0x7f72281009d0 2024-04-21T23:12:30.050 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.044+0000 7f72329bb740 1 Processor -- start 2024-04-21T23:12:30.050 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.044+0000 7f72329bb740 1 -- start start 2024-04-21T23:12:30.050 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.044+0000 7f721effd640 1 --2- 172.21.15.44:0/1519620848 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f72140a2cf0 0x7f72140a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.044+0000 7f72329bb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c75640 0x557131c75a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.044+0000 7f72329bb740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c75f60 0x557131c830e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.044+0000 7f72329bb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c83620 0x557131c85a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.044+0000 7f72329bb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557131bd9b70 con 0x557131c83620 2024-04-21T23:12:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.044+0000 7f72329bb740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x557131b82470 con 0x557131c75f60 2024-04-21T23:12:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.044+0000 7f72329bb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557131bd5b30 con 0x557131c75640 2024-04-21T23:12:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.044+0000 7f721effd640 1 -- 172.21.15.44:0/1519620848 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f72180cfbe0 con 0x7f72281009d0 2024-04-21T23:12:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.044+0000 7f722e291640 1 --2- 172.21.15.44:0/1519620848 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f72140a2cf0 0x7f72140a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.044+0000 7f722e291640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c83620 0x557131c85a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.044+0000 7f722d28f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c75f60 0x557131c830e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.044+0000 7f722e291640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c83620 0x557131c85a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51444/0 (socket says 172.21.15.44:51444) 2024-04-21T23:12:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.044+0000 7f722e291640 1 -- 172.21.15.44:0/1862736938 learned_addr learned my addr 172.21.15.44:0/1862736938 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.044+0000 7f722da90640 1 --2- 172.21.15.44:0/1862736938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c75640 0x557131c75a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.045+0000 7f722e291640 1 -- 172.21.15.44:0/1862736938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c75640 msgr2=0x557131c75a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.045+0000 7f722e291640 1 --2- 172.21.15.44:0/1862736938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c75640 0x557131c75a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.045+0000 7f722e291640 1 -- 172.21.15.44:0/1862736938 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c75f60 msgr2=0x557131c830e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.045+0000 7f722e291640 1 --2- 172.21.15.44:0/1862736938 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c75f60 0x557131c830e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.045+0000 7f722e291640 1 -- 172.21.15.44:0/1862736938 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571318ee210 con 0x557131c83620 2024-04-21T23:12:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.045+0000 7f722d28f640 1 --2- 172.21.15.44:0/1862736938 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c75f60 0x557131c830e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.045+0000 7f722e291640 1 --2- 172.21.15.44:0/1862736938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c83620 0x557131c85a10 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f7220009890 tx=0x7f7220032920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff928ede5f7d001b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.045+0000 7f722e291640 1 --2- 172.21.15.44:0/1519620848 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f72140a2cf0 0x7f72140a51b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7220037040 tx=0x7f722002e000 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.045+0000 7f7213fff640 1 -- 172.21.15.44:0/1862736938 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7220039040 con 0x557131c83620 2024-04-21T23:12:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.045+0000 7f7213fff640 1 -- 172.21.15.44:0/1862736938 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7220051070 con 0x557131c83620 2024-04-21T23:12:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.045+0000 7f7213fff640 1 -- 172.21.15.44:0/1862736938 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72200346a0 con 0x557131c83620 2024-04-21T23:12:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.046+0000 7f72329bb740 1 -- 172.21.15.44:0/1862736938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c83620 msgr2=0x557131c85a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.046+0000 7f72329bb740 1 --2- 172.21.15.44:0/1862736938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c83620 0x557131c85a10 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f7220009890 tx=0x7f7220032920 comp rx=0 tx=0).stop 2024-04-21T23:12:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.046+0000 7f72329bb740 1 -- 172.21.15.44:0/1862736938 shutdown_connections 2024-04-21T23:12:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.046+0000 7f72329bb740 1 --2- 172.21.15.44:0/1862736938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c83620 0x557131c85a10 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.046+0000 7f72329bb740 1 --2- 172.21.15.44:0/1862736938 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c75f60 0x557131c830e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.046+0000 7f72329bb740 1 --2- 172.21.15.44:0/1862736938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c75640 0x557131c75a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.046+0000 7f72329bb740 1 -- 172.21.15.44:0/1862736938 >> 172.21.15.44:0/1862736938 conn(0x557131c70440 msgr2=0x557131c70820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.046+0000 7f72329bb740 1 -- 172.21.15.44:0/1862736938 shutdown_connections 2024-04-21T23:12:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.046+0000 7f72329bb740 1 -- 172.21.15.44:0/1862736938 wait complete. 2024-04-21T23:12:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.046+0000 7f72329bb740 1 -- start start 2024-04-21T23:12:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.046+0000 7f72329bb740 1 Processor -- start 2024-04-21T23:12:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.047+0000 7f72329bb740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c75640 0x557131c806c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.047+0000 7f72329bb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c75f60 0x557131c80c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.047+0000 7f72329bb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c83620 0x557131c81140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.047+0000 7f72329bb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557131c86180 con 0x557131c75f60 2024-04-21T23:12:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.047+0000 7f72329bb740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x557131b95a10 con 0x557131c75640 2024-04-21T23:12:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.047+0000 7f72329bb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557131c86300 con 0x557131c83620 2024-04-21T23:12:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.047+0000 7f72329bb740 10 client.0 _ll_register_callbacks cb 0x557131bd0b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.047+0000 7f722da90640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c75640 0x557131c806c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.047+0000 7f722e291640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c83620 0x557131c81140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.047+0000 7f722e291640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c83620 0x557131c81140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:55254/0 (socket says 172.21.15.44:55254) 2024-04-21T23:12:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.047+0000 7f722e291640 1 -- 172.21.15.44:0/2008463434 learned_addr learned my addr 172.21.15.44:0/2008463434 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.047+0000 7f722d28f640 1 --2- 172.21.15.44:0/2008463434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c75f60 0x557131c80c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.048+0000 7f722e291640 1 -- 172.21.15.44:0/2008463434 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c75640 msgr2=0x557131c806c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.048+0000 7f722e291640 1 --2- 172.21.15.44:0/2008463434 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c75640 0x557131c806c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.048+0000 7f722e291640 1 -- 172.21.15.44:0/2008463434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c75f60 msgr2=0x557131c80c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.048+0000 7f722e291640 1 --2- 172.21.15.44:0/2008463434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c75f60 0x557131c80c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.048+0000 7f722e291640 1 -- 172.21.15.44:0/2008463434 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571319ba620 con 0x557131c83620 2024-04-21T23:12:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.048+0000 7f722da90640 1 --2- 172.21.15.44:0/2008463434 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c75640 0x557131c806c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.048+0000 7f722e291640 1 --2- 172.21.15.44:0/2008463434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c83620 0x557131c81140 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f722002c130 tx=0x7f7220004460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91e3925d1fb04712 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:30.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.048+0000 7f72117fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:12:30.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.048+0000 7f72117fa640 1 -- 172.21.15.44:0/2008463434 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7220039030 con 0x557131c83620 2024-04-21T23:12:30.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.048+0000 7f72117fa640 1 -- 172.21.15.44:0/2008463434 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f7220034440 con 0x557131c83620 2024-04-21T23:12:30.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.048+0000 7f72329bb740 1 -- 172.21.15.44:0/2008463434 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7220051070 con 0x557131c83620 2024-04-21T23:12:30.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.048+0000 7f72117fa640 1 -- 172.21.15.44:0/2008463434 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7220064050 con 0x557131c83620 2024-04-21T23:12:30.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.048+0000 7f72329bb740 10 client.26074 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:30.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.048+0000 7f72329bb740 1 -- 172.21.15.44:0/2008463434 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557131c86180 con 0x557131c83620 2024-04-21T23:12:30.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.049+0000 7f72117fa640 1 -- 172.21.15.44:0/2008463434 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7220051070 con 0x557131c83620 2024-04-21T23:12:30.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.049+0000 7f72117fa640 1 -- 172.21.15.44:0/2008463434 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f722005f030 con 0x557131c83620 2024-04-21T23:12:30.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.049+0000 7f72329bb740 10 client.26074 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:30.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.049+0000 7f72329bb740 10 client.26074 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:30.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.049+0000 7f72329bb740 20 client.26074 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:30.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.049+0000 7f72329bb740 1 --2- 172.21.15.44:0/2008463434 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557131c93690 0x557131c95a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:30.055 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.049+0000 7f72329bb740 4 client.26074 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:12:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.049+0000 7f72329bb740 1 -- 172.21.15.44:0/2008463434 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557131be70b0 con 0x557131c93690 2024-04-21T23:12:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.049+0000 7f722da90640 1 --2- 172.21.15.44:0/2008463434 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557131c93690 0x557131c95a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.050+0000 7f722da90640 1 --2- 172.21.15.44:0/2008463434 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557131c93690 0x557131c95a80 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b138c2b3461d909 server_cookie=8249d223efe139f2 in_seq=0 out_seq=0 2024-04-21T23:12:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.050+0000 7f72117fa640 10 client.26074 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.050+0000 7f72117fa640 1 -- 172.21.15.44:0/2008463434 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557131be70b0 con 0x557131c93690 2024-04-21T23:12:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.050+0000 7f72117fa640 10 client.26074 handle_command_reply: tid=0 2024-04-21T23:12:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.050+0000 7f72329bb740 1 client.26074 shutdown 2024-04-21T23:12:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.050+0000 7f72329bb740 1 -- 172.21.15.44:0/2008463434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c83620 msgr2=0x557131c81140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.050+0000 7f72329bb740 1 --2- 172.21.15.44:0/2008463434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c83620 0x557131c81140 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f722002c130 tx=0x7f7220004460 comp rx=0 tx=0).stop 2024-04-21T23:12:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.051+0000 7f72329bb740 1 -- 172.21.15.44:0/2008463434 shutdown_connections 2024-04-21T23:12:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.051+0000 7f72329bb740 1 --2- 172.21.15.44:0/2008463434 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557131c93690 0x557131c95a80 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.051+0000 7f72329bb740 1 --2- 172.21.15.44:0/2008463434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c83620 0x557131c81140 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.051+0000 7f72329bb740 1 --2- 172.21.15.44:0/2008463434 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c75f60 0x557131c80c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.051+0000 7f72329bb740 1 --2- 172.21.15.44:0/2008463434 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c75640 0x557131c806c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.056 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.051+0000 7f72117fa640 0 client.26074 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.051+0000 7f72329bb740 1 -- 172.21.15.44:0/2008463434 >> 172.21.15.44:0/2008463434 conn(0x557131c70440 msgr2=0x557131c72190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.051+0000 7f72329bb740 1 -- 172.21.15.44:0/2008463434 shutdown_connections 2024-04-21T23:12:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.051+0000 7f72329bb740 1 -- 172.21.15.44:0/2008463434 wait complete. 2024-04-21T23:12:30.057 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.051+0000 7f72329bb740 20 client.26074 trim_cache size 0 max 16384 2024-04-21T23:12:30.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.066+0000 7f72329bb740 1 Processor -- start 2024-04-21T23:12:30.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.066+0000 7f72329bb740 1 -- start start 2024-04-21T23:12:30.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.067+0000 7f72329bb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c75640 0x557131c82df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:30.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.067+0000 7f72329bb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c75f60 0x557131c8f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:30.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.067+0000 7f72329bb740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c83620 0x557131c7f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:30.067 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.067+0000 7f72329bb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f72281aa850 con 0x557131c75f60 2024-04-21T23:12:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.067+0000 7f72329bb740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f72281ad0d0 con 0x557131c83620 2024-04-21T23:12:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.067+0000 7f72329bb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72281aaec0 con 0x557131c75640 2024-04-21T23:12:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.067+0000 7f722d28f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c75f60 0x557131c8f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.067+0000 7f722d28f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c75f60 0x557131c8f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51472/0 (socket says 172.21.15.44:51472) 2024-04-21T23:12:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.067+0000 7f722d28f640 1 -- 172.21.15.44:0/4249731078 learned_addr learned my addr 172.21.15.44:0/4249731078 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.067+0000 7f722da90640 1 --2- 172.21.15.44:0/4249731078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c75640 0x557131c82df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.067+0000 7f722e291640 1 --2- 172.21.15.44:0/4249731078 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c83620 0x557131c7f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.067+0000 7f722d28f640 1 -- 172.21.15.44:0/4249731078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c75640 msgr2=0x557131c82df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.067+0000 7f722d28f640 1 --2- 172.21.15.44:0/4249731078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c75640 0x557131c82df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.067+0000 7f722d28f640 1 -- 172.21.15.44:0/4249731078 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c83620 msgr2=0x557131c7f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.067+0000 7f722d28f640 1 --2- 172.21.15.44:0/4249731078 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c83620 0x557131c7f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.067+0000 7f722d28f640 1 -- 172.21.15.44:0/4249731078 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557131c75470 con 0x557131c75f60 2024-04-21T23:12:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.067+0000 7f722d28f640 1 --2- 172.21.15.44:0/4249731078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c75f60 0x557131c8f080 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f72180981c0 tx=0x7f721800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=561ba2688819caa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.067+0000 7f72137fe640 1 -- 172.21.15.44:0/4249731078 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f721809bd60 con 0x557131c75f60 2024-04-21T23:12:30.069 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.068+0000 7f72137fe640 1 -- 172.21.15.44:0/4249731078 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72180a0110 con 0x557131c75f60 2024-04-21T23:12:30.069 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.068+0000 7f72137fe640 1 -- 172.21.15.44:0/4249731078 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72180cd3a0 con 0x557131c75f60 2024-04-21T23:12:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.068+0000 7f72329bb740 1 -- 172.21.15.44:0/4249731078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c75f60 msgr2=0x557131c8f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.068+0000 7f72329bb740 1 --2- 172.21.15.44:0/4249731078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c75f60 0x557131c8f080 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f72180981c0 tx=0x7f721800ce60 comp rx=0 tx=0).stop 2024-04-21T23:12:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.068+0000 7f72329bb740 1 -- 172.21.15.44:0/4249731078 shutdown_connections 2024-04-21T23:12:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.068+0000 7f72329bb740 1 --2- 172.21.15.44:0/4249731078 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c83620 0x557131c7f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.068+0000 7f72329bb740 1 --2- 172.21.15.44:0/4249731078 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c75f60 0x557131c8f080 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.068+0000 7f72329bb740 1 --2- 172.21.15.44:0/4249731078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c75640 0x557131c82df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.068+0000 7f72329bb740 1 -- 172.21.15.44:0/4249731078 >> 172.21.15.44:0/4249731078 conn(0x557131c70440 msgr2=0x557131c6ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.068+0000 7f72329bb740 1 -- 172.21.15.44:0/4249731078 shutdown_connections 2024-04-21T23:12:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.068+0000 7f72329bb740 1 -- 172.21.15.44:0/4249731078 wait complete. 2024-04-21T23:12:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.069+0000 7f72329bb740 1 -- start start 2024-04-21T23:12:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.069+0000 7f72329bb740 1 Processor -- start 2024-04-21T23:12:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.069+0000 7f72329bb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c75640 0x557131c8b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.069+0000 7f72329bb740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c75f60 0x557131c86d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.069+0000 7f72329bb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c83620 0x557131c87250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.069+0000 7f72329bb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f72281ab590 con 0x557131c83620 2024-04-21T23:12:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.069+0000 7f72329bb740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f72281aca00 con 0x557131c75f60 2024-04-21T23:12:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.069+0000 7f72329bb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72281abc60 con 0x557131c75640 2024-04-21T23:12:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.069+0000 7f72329bb740 10 client.0 _ll_register_callbacks cb 0x7f72180978e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.069+0000 7f722e291640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c83620 0x557131c87250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.069+0000 7f722da90640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c75640 0x557131c8b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.069+0000 7f722e291640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c83620 0x557131c87250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:51488/0 (socket says 172.21.15.44:51488) 2024-04-21T23:12:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.069+0000 7f722d28f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c75f60 0x557131c86d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.069+0000 7f722e291640 1 -- 172.21.15.44:0/2984128909 learned_addr learned my addr 172.21.15.44:0/2984128909 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.070+0000 7f722da90640 1 -- 172.21.15.44:0/2984128909 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c75f60 msgr2=0x557131c86d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.070+0000 7f722da90640 1 --2- 172.21.15.44:0/2984128909 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c75f60 0x557131c86d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.070+0000 7f722da90640 1 -- 172.21.15.44:0/2984128909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c83620 msgr2=0x557131c87250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.070+0000 7f722da90640 1 --2- 172.21.15.44:0/2984128909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c83620 0x557131c87250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.070+0000 7f722da90640 1 -- 172.21.15.44:0/2984128909 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557131c87a80 con 0x557131c75640 2024-04-21T23:12:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.070+0000 7f722d28f640 1 --2- 172.21.15.44:0/2984128909 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c75f60 0x557131c86d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.070+0000 7f722e291640 1 --2- 172.21.15.44:0/2984128909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c83620 0x557131c87250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.070+0000 7f722da90640 1 --2- 172.21.15.44:0/2984128909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c75640 0x557131c8b9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x557131c81800 tx=0x7f722401fec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c90b7c54fe19bbb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.070+0000 7f72117fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:12:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.070+0000 7f72117fa640 1 -- 172.21.15.44:0/2984128909 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72240059c0 con 0x557131c75640 2024-04-21T23:12:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.070+0000 7f72117fa640 1 -- 172.21.15.44:0/2984128909 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f7224005b60 con 0x557131c75640 2024-04-21T23:12:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.070+0000 7f72329bb740 1 -- 172.21.15.44:0/2984128909 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72180a0110 con 0x557131c75640 2024-04-21T23:12:30.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.070+0000 7f72117fa640 1 -- 172.21.15.44:0/2984128909 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f722402f050 con 0x557131c75640 2024-04-21T23:12:30.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.070+0000 7f72329bb740 10 client.26080 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:30.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.070+0000 7f72329bb740 1 -- 172.21.15.44:0/2984128909 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557131cf9b30 con 0x557131c75640 2024-04-21T23:12:30.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.070+0000 7f72117fa640 1 -- 172.21.15.44:0/2984128909 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72180a0110 con 0x557131c75640 2024-04-21T23:12:30.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.071+0000 7f72117fa640 1 -- 172.21.15.44:0/2984128909 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f72240063a0 con 0x557131c75640 2024-04-21T23:12:30.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.071+0000 7f72329bb740 10 client.26080 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:30.075 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.071+0000 7f72329bb740 10 client.26080 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.071+0000 7f72329bb740 20 client.26080 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.071+0000 7f72329bb740 1 --2- 172.21.15.44:0/2984128909 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557131cfa200 0x557131cfc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.071+0000 7f72329bb740 4 client.26080 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:12:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.071+0000 7f72329bb740 1 -- 172.21.15.44:0/2984128909 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557131be1400 con 0x557131cfa200 2024-04-21T23:12:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.071+0000 7f722d28f640 1 --2- 172.21.15.44:0/2984128909 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557131cfa200 0x557131cfc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.071+0000 7f722d28f640 1 --2- 172.21.15.44:0/2984128909 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557131cfa200 0x557131cfc5f0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=153f3c3400165a7a server_cookie=4618aa2562336c31 in_seq=0 out_seq=0 2024-04-21T23:12:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.071+0000 7f72117fa640 10 client.26080 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:30.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:29 smithi044 ceph-mon[27934]: pgmap v114: 129 pgs: 129 active+clean; 53 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 453 op/s 2024-04-21T23:12:30.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:29 smithi055 ceph-mon[31117]: pgmap v114: 129 pgs: 129 active+clean; 53 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 453 op/s 2024-04-21T23:12:30.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:29 smithi069 ceph-mon[31816]: pgmap v114: 129 pgs: 129 active+clean; 53 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.6 MiB/s wr, 453 op/s 2024-04-21T23:12:30.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.448+0000 7f72117fa640 1 -- 172.21.15.44:0/2984128909 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557131be1400 con 0x557131cfa200 2024-04-21T23:12:30.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.448+0000 7f72117fa640 10 client.26080 handle_command_reply: tid=0 2024-04-21T23:12:30.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.448+0000 7f72329bb740 1 client.26080 shutdown 2024-04-21T23:12:30.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.448+0000 7f72329bb740 1 -- 172.21.15.44:0/2984128909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c75640 msgr2=0x557131c8b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:30.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.448+0000 7f72329bb740 1 --2- 172.21.15.44:0/2984128909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c75640 0x557131c8b9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x557131c81800 tx=0x7f722401fec0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.448 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.448+0000 7f72329bb740 1 -- 172.21.15.44:0/2984128909 shutdown_connections 2024-04-21T23:12:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.448+0000 7f72329bb740 1 --2- 172.21.15.44:0/2984128909 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557131cfa200 0x557131cfc5f0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.448+0000 7f72329bb740 1 --2- 172.21.15.44:0/2984128909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557131c83620 0x557131c87250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.448+0000 7f72117fa640 0 client.26080 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.448+0000 7f72329bb740 1 --2- 172.21.15.44:0/2984128909 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557131c75f60 0x557131c86d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.448+0000 7f72329bb740 1 --2- 172.21.15.44:0/2984128909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557131c75640 0x557131c8b9a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.448+0000 7f72329bb740 1 -- 172.21.15.44:0/2984128909 >> 172.21.15.44:0/2984128909 conn(0x557131c70440 msgr2=0x557131c80660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.448+0000 7f72329bb740 1 -- 172.21.15.44:0/2984128909 shutdown_connections 2024-04-21T23:12:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.448+0000 7f72329bb740 1 -- 172.21.15.44:0/2984128909 wait complete. 2024-04-21T23:12:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.448+0000 7f72329bb740 20 client.26080 trim_cache size 0 max 16384 2024-04-21T23:12:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:12:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:12:30.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:12:30.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:12:30.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.449+0000 7f722fcf4640 1 -- 172.21.15.44:0/1519620848 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f72140a2cf0 msgr2=0x7f72140a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:30.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.449+0000 7f722fcf4640 1 --2- 172.21.15.44:0/1519620848 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f72140a2cf0 0x7f72140a51b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7220037040 tx=0x7f722002e000 comp rx=0 tx=0).stop 2024-04-21T23:12:30.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.449+0000 7f722fcf4640 1 -- 172.21.15.44:0/1519620848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72281009d0 msgr2=0x7f7228198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:30.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.449+0000 7f722fcf4640 1 --2- 172.21.15.44:0/1519620848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72281009d0 0x7f7228198560 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f721800e9f0 tx=0x7f721800edf0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.449+0000 7f722fcf4640 1 -- 172.21.15.44:0/1519620848 shutdown_connections 2024-04-21T23:12:30.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.449+0000 7f722fcf4640 1 --2- 172.21.15.44:0/1519620848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72281067b0 0x7f722819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.449+0000 7f722fcf4640 1 --2- 172.21.15.44:0/1519620848 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f72140a2cf0 0x7f72140a51b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.449+0000 7f722fcf4640 1 --2- 172.21.15.44:0/1519620848 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72281009d0 0x7f7228198560 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.449+0000 7f722fcf4640 1 --2- 172.21.15.44:0/1519620848 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7228100010 0x7f7228198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:30.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.449+0000 7f722fcf4640 1 -- 172.21.15.44:0/1519620848 >> 172.21.15.44:0/1519620848 conn(0x7f7228075990 msgr2=0x7f72280fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:30.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.449+0000 7f722fcf4640 1 -- 172.21.15.44:0/1519620848 shutdown_connections 2024-04-21T23:12:30.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:30.449+0000 7f722fcf4640 1 -- 172.21.15.44:0/1519620848 wait complete. 2024-04-21T23:12:30.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 361271 13.07 MB/sec execute 20 sec latency 145.696 ms 2024-04-21T23:12:31.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 363617 13.02 MB/sec execute 21 sec latency 30.807 ms 2024-04-21T23:12:32.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:31 smithi044 ceph-mon[27934]: pgmap v115: 129 pgs: 129 active+clean; 53 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 422 op/s 2024-04-21T23:12:32.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:31 smithi055 ceph-mon[31117]: pgmap v115: 129 pgs: 129 active+clean; 53 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 422 op/s 2024-04-21T23:12:32.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:31 smithi069 ceph-mon[31816]: pgmap v115: 129 pgs: 129 active+clean; 53 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 422 op/s 2024-04-21T23:12:32.696 INFO:tasks.workunit.client.0.smithi044.stdout: 1 366098 13.07 MB/sec execute 22 sec latency 15.277 ms 2024-04-21T23:12:33.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 369289 13.31 MB/sec execute 23 sec latency 17.099 ms 2024-04-21T23:12:34.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:33 smithi044 ceph-mon[27934]: pgmap v116: 129 pgs: 129 active+clean; 42 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.9 MiB/s wr, 701 op/s 2024-04-21T23:12:34.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:33 smithi055 ceph-mon[31117]: pgmap v116: 129 pgs: 129 active+clean; 42 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.9 MiB/s wr, 701 op/s 2024-04-21T23:12:34.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:33 smithi069 ceph-mon[31816]: pgmap v116: 129 pgs: 129 active+clean; 42 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.9 MiB/s wr, 701 op/s 2024-04-21T23:12:34.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 372234 13.36 MB/sec execute 24 sec latency 61.707 ms 2024-04-21T23:12:35.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 374040 13.22 MB/sec execute 25 sec latency 79.272 ms 2024-04-21T23:12:36.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:35 smithi044 ceph-mon[27934]: pgmap v117: 129 pgs: 129 active+clean; 42 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 401 op/s 2024-04-21T23:12:36.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:35 smithi055 ceph-mon[31117]: pgmap v117: 129 pgs: 129 active+clean; 42 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 401 op/s 2024-04-21T23:12:36.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:35 smithi069 ceph-mon[31816]: pgmap v117: 129 pgs: 129 active+clean; 42 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 401 op/s 2024-04-21T23:12:36.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 376639 13.25 MB/sec execute 26 sec latency 28.893 ms 2024-04-21T23:12:37.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 379783 13.46 MB/sec execute 27 sec latency 25.068 ms 2024-04-21T23:12:38.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:37 smithi044 ceph-mon[27934]: pgmap v118: 129 pgs: 129 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.2 MiB/s wr, 491 op/s 2024-04-21T23:12:38.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:37 smithi055 ceph-mon[31117]: pgmap v118: 129 pgs: 129 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.2 MiB/s wr, 491 op/s 2024-04-21T23:12:38.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:37 smithi069 ceph-mon[31816]: pgmap v118: 129 pgs: 129 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.2 MiB/s wr, 491 op/s 2024-04-21T23:12:38.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 382529 13.44 MB/sec execute 28 sec latency 15.871 ms 2024-04-21T23:12:39.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 384663 13.38 MB/sec execute 29 sec latency 27.422 ms 2024-04-21T23:12:40.173 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:12:40.174 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:12:40.184 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:39 smithi044 ceph-mon[27934]: pgmap v119: 129 pgs: 129 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.9 MiB/s wr, 432 op/s 2024-04-21T23:12:40.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:39 smithi055 ceph-mon[31117]: pgmap v119: 129 pgs: 129 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.9 MiB/s wr, 432 op/s 2024-04-21T23:12:40.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:39 smithi069 ceph-mon[31816]: pgmap v119: 129 pgs: 129 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.9 MiB/s wr, 432 op/s 2024-04-21T23:12:40.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.282+0000 7fd2bcf65640 1 -- 172.21.15.44:0/2970364158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2b81009e0 msgr2=0x7fd2b8100dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.282+0000 7fd2bcf65640 1 --2- 172.21.15.44:0/2970364158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2b81009e0 0x7fd2b8100dc0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8009900 tx=0x7fd2a802bd10 comp rx=0 tx=0).stop 2024-04-21T23:12:40.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.282+0000 7fd2bcf65640 1 -- 172.21.15.44:0/2970364158 shutdown_connections 2024-04-21T23:12:40.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.282+0000 7fd2bcf65640 1 --2- 172.21.15.44:0/2970364158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b8102120 0x7fd2b8102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.282+0000 7fd2bcf65640 1 --2- 172.21.15.44:0/2970364158 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2b8102ac0 0x7fd2b810d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.282+0000 7fd2bcf65640 1 --2- 172.21.15.44:0/2970364158 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2b81009e0 0x7fd2b8100dc0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.282+0000 7fd2bcf65640 1 -- 172.21.15.44:0/2970364158 >> 172.21.15.44:0/2970364158 conn(0x7fd2b8077760 msgr2=0x7fd2b8077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:40.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.282+0000 7fd2bcf65640 1 -- 172.21.15.44:0/2970364158 shutdown_connections 2024-04-21T23:12:40.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.282+0000 7fd2bcf65640 1 -- 172.21.15.44:0/2970364158 wait complete. 2024-04-21T23:12:40.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.283+0000 7fd2bcf65640 1 Processor -- start 2024-04-21T23:12:40.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.283+0000 7fd2bcf65640 1 -- start start 2024-04-21T23:12:40.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.283+0000 7fd2bcf65640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b81009e0 0x7fd2b819a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.283+0000 7fd2bcf65640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2b8102120 0x7fd2b819a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.283+0000 7fd2bcf65640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2b8102ac0 0x7fd2b819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.283+0000 7fd2bcf65640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b8111c20 con 0x7fd2b8102120 2024-04-21T23:12:40.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.283+0000 7fd2bcf65640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b8111aa0 con 0x7fd2b8102ac0 2024-04-21T23:12:40.287 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.283+0000 7fd2bcf65640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b8111da0 con 0x7fd2b81009e0 2024-04-21T23:12:40.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.283+0000 7fd2affff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2b8102120 0x7fd2b819a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.283+0000 7fd2b659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b81009e0 0x7fd2b819a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.283+0000 7fd2b659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b81009e0 0x7fd2b819a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:33120/0 (socket says 172.21.15.44:33120) 2024-04-21T23:12:40.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.283+0000 7fd2affff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2b8102120 0x7fd2b819a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50126/0 (socket says 172.21.15.44:50126) 2024-04-21T23:12:40.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.283+0000 7fd2b659c640 1 -- 172.21.15.44:0/2284265637 learned_addr learned my addr 172.21.15.44:0/2284265637 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:40.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.283+0000 7fd2b6d9d640 1 --2- 172.21.15.44:0/2284265637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2b8102ac0 0x7fd2b819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.284+0000 7fd2b659c640 1 -- 172.21.15.44:0/2284265637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2b8102ac0 msgr2=0x7fd2b819eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.284+0000 7fd2b659c640 1 --2- 172.21.15.44:0/2284265637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2b8102ac0 0x7fd2b819eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.284+0000 7fd2b659c640 1 -- 172.21.15.44:0/2284265637 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2b8102120 msgr2=0x7fd2b819a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.284+0000 7fd2b659c640 1 --2- 172.21.15.44:0/2284265637 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2b8102120 0x7fd2b819a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.284+0000 7fd2b659c640 1 -- 172.21.15.44:0/2284265637 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b819f2b0 con 0x7fd2b81009e0 2024-04-21T23:12:40.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.284+0000 7fd2affff640 1 --2- 172.21.15.44:0/2284265637 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2b8102120 0x7fd2b819a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:40.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.284+0000 7fd2b659c640 1 --2- 172.21.15.44:0/2284265637 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b81009e0 0x7fd2b819a2c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8006010 tx=0x7fd2a80042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:40.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.284+0000 7fd2af7fe640 1 -- 172.21.15.44:0/2284265637 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8042020 con 0x7fd2b81009e0 2024-04-21T23:12:40.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.284+0000 7fd2af7fe640 1 -- 172.21.15.44:0/2284265637 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2a80054e0 con 0x7fd2b81009e0 2024-04-21T23:12:40.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.284+0000 7fd2bcf65640 1 -- 172.21.15.44:0/2284265637 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2b819f540 con 0x7fd2b81009e0 2024-04-21T23:12:40.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.284+0000 7fd2af7fe640 1 -- 172.21.15.44:0/2284265637 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8003ad0 con 0x7fd2b81009e0 2024-04-21T23:12:40.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.284+0000 7fd2bcf65640 1 -- 172.21.15.44:0/2284265637 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2b81a6e30 con 0x7fd2b81009e0 2024-04-21T23:12:40.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.286+0000 7fd2af7fe640 1 -- 172.21.15.44:0/2284265637 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd2a8045020 con 0x7fd2b81009e0 2024-04-21T23:12:40.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.286+0000 7fd2bfc2f740 1 Processor -- start 2024-04-21T23:12:40.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.287+0000 7fd2bfc2f740 1 -- start start 2024-04-21T23:12:40.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.287+0000 7fd2bfc2f740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303807f30 0x55e303808310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.287+0000 7fd2bfc2f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303808850 0x55e303810ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.287+0000 7fd2bfc2f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303811430 0x55e303813820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.287+0000 7fd2bfc2f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e30376c560 con 0x55e303811430 2024-04-21T23:12:40.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.287+0000 7fd2bfc2f740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55e303714730 con 0x55e303807f30 2024-04-21T23:12:40.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.287+0000 7fd2bfc2f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e303768520 con 0x55e303808850 2024-04-21T23:12:40.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.287+0000 7fd2b6d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303807f30 0x55e303808310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.287+0000 7fd2affff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303811430 0x55e303813820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.287+0000 7fd2b659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303808850 0x55e303810ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.287+0000 7fd2b6d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303807f30 0x55e303808310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:60692/0 (socket says 172.21.15.44:60692) 2024-04-21T23:12:40.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.287+0000 7fd2b6d9d640 1 -- 172.21.15.44:0/3441512282 learned_addr learned my addr 172.21.15.44:0/3441512282 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:40.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.287+0000 7fd2af7fe640 1 --2- 172.21.15.44:0/2284265637 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd2a40a2d00 0x7fd2a40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.287+0000 7fd2b6d9d640 1 -- 172.21.15.44:0/3441512282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303808850 msgr2=0x55e303810ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:40.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.287+0000 7fd2b6d9d640 1 --2- 172.21.15.44:0/3441512282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303808850 0x55e303810ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.287+0000 7fd2b6d9d640 1 -- 172.21.15.44:0/3441512282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303811430 msgr2=0x55e303813820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:40.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.287+0000 7fd2b6d9d640 1 --2- 172.21.15.44:0/3441512282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303811430 0x55e303813820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.287+0000 7fd2b6d9d640 1 -- 172.21.15.44:0/3441512282 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e30354a7d0 con 0x55e303807f30 2024-04-21T23:12:40.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.287+0000 7fd2af7fe640 1 -- 172.21.15.44:0/2284265637 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fd2a80f1370 con 0x7fd2b81009e0 2024-04-21T23:12:40.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.288+0000 7fd2b6d9d640 1 --2- 172.21.15.44:0/2284265637 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd2a40a2d00 0x7fd2a40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.288+0000 7fd2b6d9d640 1 --2- 172.21.15.44:0/3441512282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303807f30 0x55e303808310 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0011590 tx=0x7fd2b0011a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e826f13908b5a5be server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:40.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.288+0000 7fd2acff9640 1 -- 172.21.15.44:0/3441512282 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b0017050 con 0x55e303807f30 2024-04-21T23:12:40.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.288+0000 7fd2acff9640 1 -- 172.21.15.44:0/3441512282 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2b000fd20 con 0x55e303807f30 2024-04-21T23:12:40.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.288+0000 7fd2acff9640 1 -- 172.21.15.44:0/3441512282 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b001ba90 con 0x55e303807f30 2024-04-21T23:12:40.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.288+0000 7fd2b6d9d640 1 --2- 172.21.15.44:0/2284265637 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd2a40a2d00 0x7fd2a40a51c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0022000 tx=0x7fd2b0023000 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:40.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.289+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/3441512282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303807f30 msgr2=0x55e303808310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.289+0000 7fd2bfc2f740 1 --2- 172.21.15.44:0/3441512282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303807f30 0x55e303808310 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0011590 tx=0x7fd2b0011a50 comp rx=0 tx=0).stop 2024-04-21T23:12:40.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.289+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/3441512282 shutdown_connections 2024-04-21T23:12:40.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.289+0000 7fd2bfc2f740 1 --2- 172.21.15.44:0/3441512282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303808850 0x55e303810ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.289+0000 7fd2bfc2f740 1 --2- 172.21.15.44:0/3441512282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303811430 0x55e303813820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.289+0000 7fd2bfc2f740 1 --2- 172.21.15.44:0/3441512282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303807f30 0x55e303808310 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.289+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/3441512282 >> 172.21.15.44:0/3441512282 conn(0x55e303802660 msgr2=0x55e303802a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:40.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.289+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/3441512282 shutdown_connections 2024-04-21T23:12:40.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.289+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/3441512282 wait complete. 2024-04-21T23:12:40.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.289+0000 7fd2bfc2f740 1 -- start start 2024-04-21T23:12:40.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.289+0000 7fd2bfc2f740 1 Processor -- start 2024-04-21T23:12:40.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.289+0000 7fd2bfc2f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303807f30 0x55e30380e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.289+0000 7fd2bfc2f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303808850 0x55e30380ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2bfc2f740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303811430 0x55e30380ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2bfc2f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e303814220 con 0x55e303807f30 2024-04-21T23:12:40.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2bfc2f740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55e303727e00 con 0x55e303811430 2024-04-21T23:12:40.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2bfc2f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e3038143a0 con 0x55e303808850 2024-04-21T23:12:40.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2bfc2f740 10 client.0 _ll_register_callbacks cb 0x55e303762eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:40.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2b659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303807f30 0x55e30380e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2affff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303808850 0x55e30380ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2b659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303807f30 0x55e30380e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50148/0 (socket says 172.21.15.44:50148) 2024-04-21T23:12:40.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2b659c640 1 -- 172.21.15.44:0/788374561 learned_addr learned my addr 172.21.15.44:0/788374561 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:40.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2affff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303808850 0x55e30380ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:33144/0 (socket says 172.21.15.44:33144) 2024-04-21T23:12:40.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2b6d9d640 1 --2- 172.21.15.44:0/788374561 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303811430 0x55e30380ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2b659c640 1 -- 172.21.15.44:0/788374561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303808850 msgr2=0x55e30380ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:40.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2b659c640 1 --2- 172.21.15.44:0/788374561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303808850 0x55e30380ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2b659c640 1 -- 172.21.15.44:0/788374561 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303811430 msgr2=0x55e30380ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:40.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2b659c640 1 --2- 172.21.15.44:0/788374561 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303811430 0x55e30380ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2b659c640 1 -- 172.21.15.44:0/788374561 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3034803f0 con 0x55e303807f30 2024-04-21T23:12:40.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2affff640 1 --2- 172.21.15.44:0/788374561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303808850 0x55e30380ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:40.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2b6d9d640 1 --2- 172.21.15.44:0/788374561 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303811430 0x55e30380ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:40.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2b659c640 1 --2- 172.21.15.44:0/788374561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303807f30 0x55e30380e4f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8005940 tx=0x7fd2a80045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73a364e889dfe391 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:40.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd29dffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:12:40.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd29dffb640 1 -- 172.21.15.44:0/788374561 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a80f1170 con 0x55e303807f30 2024-04-21T23:12:40.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd29dffb640 1 -- 172.21.15.44:0/788374561 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fd2a80f4c70 con 0x55e303807f30 2024-04-21T23:12:40.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/788374561 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2b000fd20 con 0x55e303807f30 2024-04-21T23:12:40.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd29dffb640 1 -- 172.21.15.44:0/788374561 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd2a8005690 con 0x55e303807f30 2024-04-21T23:12:40.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2bfc2f740 10 client.16488 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:40.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.290+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/788374561 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e3034400f0 con 0x55e303807f30 2024-04-21T23:12:40.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.291+0000 7fd29dffb640 1 -- 172.21.15.44:0/788374561 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2b000fd20 con 0x55e303807f30 2024-04-21T23:12:40.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.291+0000 7fd29dffb640 1 -- 172.21.15.44:0/788374561 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fd2a8003c70 con 0x55e303807f30 2024-04-21T23:12:40.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.291+0000 7fd2bfc2f740 10 client.16488 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:40.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.291+0000 7fd2bfc2f740 10 client.16488 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:40.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.291+0000 7fd2bfc2f740 20 client.16488 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:40.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.291+0000 7fd2bfc2f740 1 --2- 172.21.15.44:0/788374561 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e303821790 0x55e303823bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.291+0000 7fd2bfc2f740 4 client.16488 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:12:40.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.291+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/788374561 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e303779aa0 con 0x55e303821790 2024-04-21T23:12:40.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.291+0000 7fd2affff640 1 --2- 172.21.15.44:0/788374561 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e303821790 0x55e303823bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.291+0000 7fd29dffb640 10 client.16488 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:40.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.291+0000 7fd2affff640 1 --2- 172.21.15.44:0/788374561 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e303821790 0x55e303823bc0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e0f79ee1bdfb186 server_cookie=3412849687340fb7 in_seq=0 out_seq=0 2024-04-21T23:12:40.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.293+0000 7fd29dffb640 1 -- 172.21.15.44:0/788374561 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e303779aa0 con 0x55e303821790 2024-04-21T23:12:40.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.293+0000 7fd29dffb640 10 client.16488 handle_command_reply: tid=0 2024-04-21T23:12:40.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.293+0000 7fd2bfc2f740 1 client.16488 shutdown 2024-04-21T23:12:40.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.293+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/788374561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303807f30 msgr2=0x55e30380e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.293+0000 7fd2bfc2f740 1 --2- 172.21.15.44:0/788374561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303807f30 0x55e30380e4f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8005940 tx=0x7fd2a80045a0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.293+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/788374561 shutdown_connections 2024-04-21T23:12:40.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.293+0000 7fd2bfc2f740 1 --2- 172.21.15.44:0/788374561 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e303821790 0x55e303823bc0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.293+0000 7fd2bfc2f740 1 --2- 172.21.15.44:0/788374561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303808850 0x55e30380ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.293+0000 7fd29dffb640 0 client.16488 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:40.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.293+0000 7fd2bfc2f740 1 --2- 172.21.15.44:0/788374561 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303811430 0x55e30380ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.293+0000 7fd2bfc2f740 1 --2- 172.21.15.44:0/788374561 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303807f30 0x55e30380e4f0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.293+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/788374561 >> 172.21.15.44:0/788374561 conn(0x55e303802660 msgr2=0x55e3038043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:40.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.293+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/788374561 shutdown_connections 2024-04-21T23:12:40.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.293+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/788374561 wait complete. 2024-04-21T23:12:40.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.293+0000 7fd2bfc2f740 20 client.16488 trim_cache size 0 max 16384 2024-04-21T23:12:40.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.311+0000 7fd2bfc2f740 1 Processor -- start 2024-04-21T23:12:40.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.311+0000 7fd2bfc2f740 1 -- start start 2024-04-21T23:12:40.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.311+0000 7fd2bfc2f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303807f30 0x55e303810c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.311+0000 7fd2bfc2f740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303808850 0x55e30381d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.311+0000 7fd2bfc2f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303811430 0x55e303801560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.311+0000 7fd2bfc2f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b81aed10 con 0x55e303811430 2024-04-21T23:12:40.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.311+0000 7fd2bfc2f740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b81b1590 con 0x55e303808850 2024-04-21T23:12:40.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.311+0000 7fd2bfc2f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b81af380 con 0x55e303807f30 2024-04-21T23:12:40.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.312+0000 7fd2b6d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303811430 0x55e303801560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.312+0000 7fd2affff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303808850 0x55e30381d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.312+0000 7fd2b659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303807f30 0x55e303810c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.312+0000 7fd2b6d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303811430 0x55e303801560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:50160/0 (socket says 172.21.15.44:50160) 2024-04-21T23:12:40.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.312+0000 7fd2b6d9d640 1 -- 172.21.15.44:0/1280541004 learned_addr learned my addr 172.21.15.44:0/1280541004 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:40.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.312+0000 7fd2b6d9d640 1 -- 172.21.15.44:0/1280541004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303807f30 msgr2=0x55e303810c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:40.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.312+0000 7fd2b6d9d640 1 --2- 172.21.15.44:0/1280541004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303807f30 0x55e303810c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.312+0000 7fd2b6d9d640 1 -- 172.21.15.44:0/1280541004 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303808850 msgr2=0x55e30381d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:40.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.312+0000 7fd2b6d9d640 1 --2- 172.21.15.44:0/1280541004 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303808850 0x55e30381d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.312+0000 7fd2b6d9d640 1 -- 172.21.15.44:0/1280541004 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e303807250 con 0x55e303811430 2024-04-21T23:12:40.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.312+0000 7fd2b659c640 1 --2- 172.21.15.44:0/1280541004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303807f30 0x55e303810c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:40.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.312+0000 7fd2affff640 1 --2- 172.21.15.44:0/1280541004 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303808850 0x55e30381d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:40.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.312+0000 7fd2b6d9d640 1 --2- 172.21.15.44:0/1280541004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303811430 0x55e303801560 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0011f50 tx=0x7fd2b00096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81433ce5c59b5af9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:40.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.312+0000 7fd29ffff640 1 -- 172.21.15.44:0/1280541004 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b0036070 con 0x55e303811430 2024-04-21T23:12:40.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.312+0000 7fd29ffff640 1 -- 172.21.15.44:0/1280541004 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2b000f640 con 0x55e303811430 2024-04-21T23:12:40.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.312+0000 7fd29ffff640 1 -- 172.21.15.44:0/1280541004 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b001b430 con 0x55e303811430 2024-04-21T23:12:40.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.313+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/1280541004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303811430 msgr2=0x55e303801560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.313+0000 7fd2bfc2f740 1 --2- 172.21.15.44:0/1280541004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303811430 0x55e303801560 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0011f50 tx=0x7fd2b00096c0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.313+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/1280541004 shutdown_connections 2024-04-21T23:12:40.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.313+0000 7fd2bfc2f740 1 --2- 172.21.15.44:0/1280541004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303811430 0x55e303801560 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.313+0000 7fd2bfc2f740 1 --2- 172.21.15.44:0/1280541004 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303808850 0x55e30381d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.313+0000 7fd2bfc2f740 1 --2- 172.21.15.44:0/1280541004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303807f30 0x55e303810c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.313+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/1280541004 >> 172.21.15.44:0/1280541004 conn(0x55e303802660 msgr2=0x55e303800e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:40.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.313+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/1280541004 shutdown_connections 2024-04-21T23:12:40.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.313+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/1280541004 wait complete. 2024-04-21T23:12:40.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.313+0000 7fd2bfc2f740 1 -- start start 2024-04-21T23:12:40.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.314+0000 7fd2bfc2f740 1 Processor -- start 2024-04-21T23:12:40.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.314+0000 7fd2bfc2f740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303807f30 0x55e30380b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.314+0000 7fd2bfc2f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303808850 0x55e30380b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.314+0000 7fd2bfc2f740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303811430 0x55e30380bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.314+0000 7fd2bfc2f740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b81afa50 con 0x55e303811430 2024-04-21T23:12:40.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.314+0000 7fd2bfc2f740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b81b0ec0 con 0x55e303807f30 2024-04-21T23:12:40.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.314+0000 7fd2bfc2f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b81b0120 con 0x55e303808850 2024-04-21T23:12:40.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.314+0000 7fd2bfc2f740 10 client.0 _ll_register_callbacks cb 0x7fd2a80f4b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:40.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.314+0000 7fd2b659c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303807f30 0x55e30380b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.314+0000 7fd2b659c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303807f30 0x55e30380b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:60724/0 (socket says 172.21.15.44:60724) 2024-04-21T23:12:40.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.314+0000 7fd2b659c640 1 -- 172.21.15.44:0/2862374144 learned_addr learned my addr 172.21.15.44:0/2862374144 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:40.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.314+0000 7fd2affff640 1 --2- 172.21.15.44:0/2862374144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303808850 0x55e30380b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.314+0000 7fd2b6d9d640 1 --2- 172.21.15.44:0/2862374144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303811430 0x55e30380bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.314+0000 7fd2b659c640 1 -- 172.21.15.44:0/2862374144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303808850 msgr2=0x55e30380b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:40.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.314+0000 7fd2b659c640 1 --2- 172.21.15.44:0/2862374144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303808850 0x55e30380b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.314+0000 7fd2b659c640 1 -- 172.21.15.44:0/2862374144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303811430 msgr2=0x55e30380bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:40.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.314+0000 7fd2b659c640 1 --2- 172.21.15.44:0/2862374144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303811430 0x55e30380bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.314+0000 7fd2b659c640 1 -- 172.21.15.44:0/2862374144 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e303886db0 con 0x55e303807f30 2024-04-21T23:12:40.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.314+0000 7fd2b659c640 1 --2- 172.21.15.44:0/2862374144 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303807f30 0x55e30380b260 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8000c00 tx=0x7fd2a802cb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91ea1b8b79c6ba0a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:40.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.314+0000 7fd29dffb640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:12:40.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.315+0000 7fd29dffb640 1 -- 172.21.15.44:0/2862374144 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a80f1170 con 0x55e303807f30 2024-04-21T23:12:40.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.315+0000 7fd29dffb640 1 -- 172.21.15.44:0/2862374144 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fd2a80ee8b0 con 0x55e303807f30 2024-04-21T23:12:40.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.315+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/2862374144 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2b000f640 con 0x55e303807f30 2024-04-21T23:12:40.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.315+0000 7fd29dffb640 1 -- 172.21.15.44:0/2862374144 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd2a80eeb30 con 0x55e303807f30 2024-04-21T23:12:40.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.315+0000 7fd2bfc2f740 10 client.26063 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:40.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.315+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/2862374144 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e3038880a0 con 0x55e303807f30 2024-04-21T23:12:40.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.315+0000 7fd29dffb640 1 -- 172.21.15.44:0/2862374144 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2b000f640 con 0x55e303807f30 2024-04-21T23:12:40.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.315+0000 7fd29dffb640 1 -- 172.21.15.44:0/2862374144 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fd2a80f4bc0 con 0x55e303807f30 2024-04-21T23:12:40.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.315+0000 7fd2bfc2f740 10 client.26063 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:40.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.315+0000 7fd2bfc2f740 10 client.26063 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:40.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.315+0000 7fd2bfc2f740 20 client.26063 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:40.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.315+0000 7fd2bfc2f740 1 --2- 172.21.15.44:0/2862374144 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e303888670 0x55e30388aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.315+0000 7fd2bfc2f740 4 client.26063 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:12:40.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.315+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/2862374144 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e303773df0 con 0x55e303888670 2024-04-21T23:12:40.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.315+0000 7fd2affff640 1 --2- 172.21.15.44:0/2862374144 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e303888670 0x55e30388aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.316+0000 7fd2affff640 1 --2- 172.21.15.44:0/2862374144 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e303888670 0x55e30388aa90 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c11f6e10686b5735 server_cookie=546d4aa554cba18a in_seq=0 out_seq=0 2024-04-21T23:12:40.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.316+0000 7fd29dffb640 10 client.26063 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:40.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.318+0000 7fd29dffb640 1 -- 172.21.15.44:0/2862374144 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e303773df0 con 0x55e303888670 2024-04-21T23:12:40.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.318+0000 7fd29dffb640 10 client.26063 handle_command_reply: tid=0 2024-04-21T23:12:40.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.318+0000 7fd2bfc2f740 1 client.26063 shutdown 2024-04-21T23:12:40.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.318+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/2862374144 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303807f30 msgr2=0x55e30380b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.318+0000 7fd2bfc2f740 1 --2- 172.21.15.44:0/2862374144 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303807f30 0x55e30380b260 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8000c00 tx=0x7fd2a802cb40 comp rx=0 tx=0).stop 2024-04-21T23:12:40.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.318+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/2862374144 shutdown_connections 2024-04-21T23:12:40.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.318+0000 7fd2bfc2f740 1 --2- 172.21.15.44:0/2862374144 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e303888670 0x55e30388aa90 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.318+0000 7fd2bfc2f740 1 --2- 172.21.15.44:0/2862374144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e303808850 0x55e30380b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.318+0000 7fd2bfc2f740 1 --2- 172.21.15.44:0/2862374144 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e303811430 0x55e30380bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.318+0000 7fd29dffb640 0 client.26063 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:40.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.318+0000 7fd2bfc2f740 1 --2- 172.21.15.44:0/2862374144 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e303807f30 0x55e30380b260 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.318+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/2862374144 >> 172.21.15.44:0/2862374144 conn(0x55e303802660 msgr2=0x55e30380ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:40.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.318+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/2862374144 shutdown_connections 2024-04-21T23:12:40.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.318+0000 7fd2bfc2f740 1 -- 172.21.15.44:0/2862374144 wait complete. 2024-04-21T23:12:40.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.318+0000 7fd2bfc2f740 20 client.26063 trim_cache size 0 max 16384 2024-04-21T23:12:40.322 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:12:40.322 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-21T23:12:40.322 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-21T23:12:40.323 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:12:40.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.319+0000 7fd2bcf65640 1 -- 172.21.15.44:0/2284265637 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd2a40a2d00 msgr2=0x7fd2a40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.319+0000 7fd2bcf65640 1 --2- 172.21.15.44:0/2284265637 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd2a40a2d00 0x7fd2a40a51c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0022000 tx=0x7fd2b0023000 comp rx=0 tx=0).stop 2024-04-21T23:12:40.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.319+0000 7fd2bcf65640 1 -- 172.21.15.44:0/2284265637 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b81009e0 msgr2=0x7fd2b819a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.319+0000 7fd2bcf65640 1 --2- 172.21.15.44:0/2284265637 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b81009e0 0x7fd2b819a2c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8006010 tx=0x7fd2a80042f0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.319+0000 7fd2bcf65640 1 -- 172.21.15.44:0/2284265637 shutdown_connections 2024-04-21T23:12:40.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.319+0000 7fd2bcf65640 1 --2- 172.21.15.44:0/2284265637 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd2a40a2d00 0x7fd2a40a51c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.319+0000 7fd2bcf65640 1 --2- 172.21.15.44:0/2284265637 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2b8102ac0 0x7fd2b819eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.319+0000 7fd2bcf65640 1 --2- 172.21.15.44:0/2284265637 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd2b8102120 0x7fd2b819a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.319+0000 7fd2bcf65640 1 --2- 172.21.15.44:0/2284265637 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b81009e0 0x7fd2b819a2c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.319+0000 7fd2bcf65640 1 -- 172.21.15.44:0/2284265637 >> 172.21.15.44:0/2284265637 conn(0x7fd2b8077760 msgr2=0x7fd2b8105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:40.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.319+0000 7fd2bcf65640 1 -- 172.21.15.44:0/2284265637 shutdown_connections 2024-04-21T23:12:40.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.319+0000 7fd2bcf65640 1 -- 172.21.15.44:0/2284265637 wait complete. 2024-04-21T23:12:40.329 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:12:40.329 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:12:40.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.477+0000 7fbddef25640 1 -- 172.21.15.44:0/3045706339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdd8107800 msgr2=0x7fbdd8107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.477+0000 7fbddef25640 1 --2- 172.21.15.44:0/3045706339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdd8107800 0x7fbdd8107be0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4009900 tx=0x7fbdd402bc30 comp rx=0 tx=0).stop 2024-04-21T23:12:40.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.477+0000 7fbddef25640 1 -- 172.21.15.44:0/3045706339 shutdown_connections 2024-04-21T23:12:40.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.477+0000 7fbddef25640 1 --2- 172.21.15.44:0/3045706339 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbdd80693e0 0x7fbdd810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.477+0000 7fbddef25640 1 --2- 172.21.15.44:0/3045706339 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbdd8068a40 0x7fbdd8068ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.477+0000 7fbddef25640 1 --2- 172.21.15.44:0/3045706339 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdd8107800 0x7fbdd8107be0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.477+0000 7fbddef25640 1 -- 172.21.15.44:0/3045706339 >> 172.21.15.44:0/3045706339 conn(0x7fbdd8077430 msgr2=0x7fbdd8077840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:40.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.477+0000 7fbddef25640 1 -- 172.21.15.44:0/3045706339 shutdown_connections 2024-04-21T23:12:40.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.477+0000 7fbddef25640 1 -- 172.21.15.44:0/3045706339 wait complete. 2024-04-21T23:12:40.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.478+0000 7fbddef25640 1 Processor -- start 2024-04-21T23:12:40.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.478+0000 7fbddef25640 1 -- start start 2024-04-21T23:12:40.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.478+0000 7fbddef25640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbdd8068a40 0x7fbdd8197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.478+0000 7fbddef25640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdd80693e0 0x7fbdd8198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.478+0000 7fbddef25640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbdd8107800 0x7fbdd819c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.478+0000 7fbddccc1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbdd8068a40 0x7fbdd8197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.478+0000 7fbddccc1640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbdd8068a40 0x7fbdd8197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:60754/0 (socket says 172.21.15.44:60754) 2024-04-21T23:12:40.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.478+0000 7fbddccc1640 1 -- 172.21.15.44:0/172527668 learned_addr learned my addr 172.21.15.44:0/172527668 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:40.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.478+0000 7fbddef25640 1 -- 172.21.15.44:0/172527668 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd8111b20 con 0x7fbdd80693e0 2024-04-21T23:12:40.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.478+0000 7fbddef25640 1 -- 172.21.15.44:0/172527668 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd81119a0 con 0x7fbdd8068a40 2024-04-21T23:12:40.478 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.478+0000 7fbddef25640 1 -- 172.21.15.44:0/172527668 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd8111ca0 con 0x7fbdd8107800 2024-04-21T23:12:40.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.479+0000 7fbddccc1640 1 -- 172.21.15.44:0/172527668 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbdd8107800 msgr2=0x7fbdd819c8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:12:40.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.479+0000 7fbddccc1640 1 --2- 172.21.15.44:0/172527668 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbdd8107800 0x7fbdd819c8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.479+0000 7fbddccc1640 1 -- 172.21.15.44:0/172527668 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdd80693e0 msgr2=0x7fbdd8198520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:12:40.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.479+0000 7fbddccc1640 1 --2- 172.21.15.44:0/172527668 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdd80693e0 0x7fbdd8198520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.479+0000 7fbddccc1640 1 -- 172.21.15.44:0/172527668 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdd819cfd0 con 0x7fbdd8068a40 2024-04-21T23:12:40.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.482+0000 7fbddccc1640 1 --2- 172.21.15.44:0/172527668 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbdd8068a40 0x7fbdd8197fe0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4006010 tx=0x7fbdd402cb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:40.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.483+0000 7fbdcdffb640 1 -- 172.21.15.44:0/172527668 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd4004450 con 0x7fbdd8068a40 2024-04-21T23:12:40.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.483+0000 7fbdcdffb640 1 -- 172.21.15.44:0/172527668 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdd40054e0 con 0x7fbdd8068a40 2024-04-21T23:12:40.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.483+0000 7fbddef25640 1 -- 172.21.15.44:0/172527668 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdd819d260 con 0x7fbdd8068a40 2024-04-21T23:12:40.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.484+0000 7fbddef25640 1 -- 172.21.15.44:0/172527668 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdd8100890 con 0x7fbdd8068a40 2024-04-21T23:12:40.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.484+0000 7fbdcdffb640 1 -- 172.21.15.44:0/172527668 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd40035f0 con 0x7fbdd8068a40 2024-04-21T23:12:40.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.485+0000 7fbddef25640 1 -- 172.21.15.44:0/172527668 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdd8102e90 con 0x7fbdd8068a40 2024-04-21T23:12:40.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.493+0000 7fbdcdffb640 1 -- 172.21.15.44:0/172527668 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fbdd404d060 con 0x7fbdd8068a40 2024-04-21T23:12:40.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.495+0000 7fbdcdffb640 1 --2- 172.21.15.44:0/172527668 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbdc00a2ed0 0x7fbdc00a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.495+0000 7fbdcdffb640 1 -- 172.21.15.44:0/172527668 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fbdd40f4f70 con 0x7fbdd8068a40 2024-04-21T23:12:40.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.495+0000 7fbdcffff640 1 --2- 172.21.15.44:0/172527668 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbdc00a2ed0 0x7fbdc00a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.494 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.496+0000 7fbdcffff640 1 --2- 172.21.15.44:0/172527668 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbdc00a2ed0 0x7fbdc00a5390 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8106df0 tx=0x7fbdc8005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:40.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.503+0000 7fbdcdffb640 1 -- 172.21.15.44:0/172527668 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fbdd40c1920 con 0x7fbdd8068a40 2024-04-21T23:12:40.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 387213 13.40 MB/sec execute 30 sec latency 66.413 ms 2024-04-21T23:12:40.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.777+0000 7fbddef25640 1 -- 172.21.15.44:0/172527668 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbdd80685f0 con 0x7fbdd8068a40 2024-04-21T23:12:40.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.778+0000 7fbdcdffb640 1 -- 172.21.15.44:0/172527668 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11791 (secure 0 0 0) 0x7fbdd80685f0 con 0x7fbdd8068a40 2024-04-21T23:12:40.777 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:12:40.777 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:11:24.214378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25843":{"gid":25843,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3162569119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3162569119},{"type":"v1","addr":"172.21.15.69:6839","nonce":3162569119}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:12:40.777 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-04-21T23:12:40.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.780+0000 7fbddef25640 1 -- 172.21.15.44:0/172527668 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbdc00a2ed0 msgr2=0x7fbdc00a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.780+0000 7fbddef25640 1 --2- 172.21.15.44:0/172527668 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbdc00a2ed0 0x7fbdc00a5390 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8106df0 tx=0x7fbdc8005f90 comp rx=0 tx=0).stop 2024-04-21T23:12:40.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.780+0000 7fbddef25640 1 -- 172.21.15.44:0/172527668 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbdd8068a40 msgr2=0x7fbdd8197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.780+0000 7fbddef25640 1 --2- 172.21.15.44:0/172527668 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbdd8068a40 0x7fbdd8197fe0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4006010 tx=0x7fbdd402cb30 comp rx=0 tx=0).stop 2024-04-21T23:12:40.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.780+0000 7fbddef25640 1 -- 172.21.15.44:0/172527668 shutdown_connections 2024-04-21T23:12:40.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.780+0000 7fbddef25640 1 --2- 172.21.15.44:0/172527668 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbdd8107800 0x7fbdd819c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.780+0000 7fbddef25640 1 --2- 172.21.15.44:0/172527668 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbdc00a2ed0 0x7fbdc00a5390 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.780+0000 7fbddef25640 1 --2- 172.21.15.44:0/172527668 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbdd80693e0 0x7fbdd8198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.780+0000 7fbddef25640 1 --2- 172.21.15.44:0/172527668 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbdd8068a40 0x7fbdd8197fe0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.780+0000 7fbddef25640 1 -- 172.21.15.44:0/172527668 >> 172.21.15.44:0/172527668 conn(0x7fbdd8077430 msgr2=0x7fbdd80ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:40.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.781+0000 7fbddef25640 1 -- 172.21.15.44:0/172527668 shutdown_connections 2024-04-21T23:12:40.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.781+0000 7fbddef25640 1 -- 172.21.15.44:0/172527668 wait complete. 2024-04-21T23:12:40.790 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:12:40.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.908+0000 7f9fb17bf640 1 -- 172.21.15.44:0/1140955995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fac1032d0 msgr2=0x7f9fac10b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.908+0000 7f9fb17bf640 1 --2- 172.21.15.44:0/1140955995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fac1032d0 0x7f9fac10b7b0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f9f94009900 tx=0x7f9f9402bcd0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.908+0000 7f9fb17bf640 1 -- 172.21.15.44:0/1140955995 shutdown_connections 2024-04-21T23:12:40.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.908+0000 7f9fb17bf640 1 --2- 172.21.15.44:0/1140955995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fac10bd80 0x7f9fac10e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.908+0000 7f9fb17bf640 1 --2- 172.21.15.44:0/1140955995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fac1032d0 0x7f9fac10b7b0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.908+0000 7f9fb17bf640 1 --2- 172.21.15.44:0/1140955995 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fac1029b0 0x7f9fac102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.908+0000 7f9fb17bf640 1 -- 172.21.15.44:0/1140955995 >> 172.21.15.44:0/1140955995 conn(0x7f9fac077730 msgr2=0x7f9fac077b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:40.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.908+0000 7f9fb17bf640 1 -- 172.21.15.44:0/1140955995 shutdown_connections 2024-04-21T23:12:40.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.908+0000 7f9fb17bf640 1 -- 172.21.15.44:0/1140955995 wait complete. 2024-04-21T23:12:40.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.909+0000 7f9fb17bf640 1 Processor -- start 2024-04-21T23:12:40.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.909+0000 7f9fb17bf640 1 -- start start 2024-04-21T23:12:40.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.909+0000 7f9fb17bf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fac1029b0 0x7f9fac198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.909+0000 7f9fb17bf640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fac1032d0 0x7f9fac198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.909+0000 7f9fb17bf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fac10bd80 0x7f9fac19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.909+0000 7f9fb17bf640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9fac111b10 con 0x7f9fac10bd80 2024-04-21T23:12:40.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.909+0000 7f9fb17bf640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9fac111990 con 0x7f9fac1032d0 2024-04-21T23:12:40.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.909+0000 7f9fb17bf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9fac111c90 con 0x7f9fac1029b0 2024-04-21T23:12:40.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.909+0000 7f9fab7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fac10bd80 0x7f9fac19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.909+0000 7f9fa27fc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fac1032d0 0x7f9fac198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.909+0000 7f9fab7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fac10bd80 0x7f9fac19c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39882/0 (socket says 172.21.15.44:39882) 2024-04-21T23:12:40.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.909+0000 7f9faaffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fac1029b0 0x7f9fac198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.909+0000 7f9fab7fe640 1 -- 172.21.15.44:0/4177958982 learned_addr learned my addr 172.21.15.44:0/4177958982 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:40.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.910+0000 7f9fab7fe640 1 -- 172.21.15.44:0/4177958982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fac1029b0 msgr2=0x7f9fac198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.910+0000 7f9fab7fe640 1 --2- 172.21.15.44:0/4177958982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fac1029b0 0x7f9fac198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.910 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.910+0000 7f9fab7fe640 1 -- 172.21.15.44:0/4177958982 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fac1032d0 msgr2=0x7f9fac198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.910+0000 7f9fab7fe640 1 --2- 172.21.15.44:0/4177958982 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fac1032d0 0x7f9fac198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.910+0000 7f9fab7fe640 1 -- 172.21.15.44:0/4177958982 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fac19cff0 con 0x7f9fac10bd80 2024-04-21T23:12:40.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.910+0000 7f9fa27fc640 1 --2- 172.21.15.44:0/4177958982 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fac1032d0 0x7f9fac198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:40.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.910+0000 7f9fab7fe640 1 --2- 172.21.15.44:0/4177958982 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fac10bd80 0x7f9fac19c8c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9fa400eb40 tx=0x7f9fa400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:40.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.910+0000 7f9fa8ff9640 1 -- 172.21.15.44:0/4177958982 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4004b20 con 0x7f9fac10bd80 2024-04-21T23:12:40.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.910+0000 7f9fa8ff9640 1 -- 172.21.15.44:0/4177958982 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fa4004cc0 con 0x7f9fac10bd80 2024-04-21T23:12:40.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.910+0000 7f9fb17bf640 1 -- 172.21.15.44:0/4177958982 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fac19d2e0 con 0x7f9fac10bd80 2024-04-21T23:12:40.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.910+0000 7f9fa8ff9640 1 -- 172.21.15.44:0/4177958982 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4010850 con 0x7f9fac10bd80 2024-04-21T23:12:40.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.910+0000 7f9fb17bf640 1 -- 172.21.15.44:0/4177958982 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fac1a4b70 con 0x7f9fac10bd80 2024-04-21T23:12:40.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.913+0000 7f9fb43a2740 1 Processor -- start 2024-04-21T23:12:40.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.913+0000 7f9fa8ff9640 1 -- 172.21.15.44:0/4177958982 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f9fa4005020 con 0x7f9fac10bd80 2024-04-21T23:12:40.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.913+0000 7f9fb43a2740 1 -- start start 2024-04-21T23:12:40.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9fb43a2740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0abbd50 0x55d9d0abc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9fb43a2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abc670 0x55d9d0ac93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9fb43a2740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0ac9930 0x55d9d0acbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9fb43a2740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d9d0a20530 con 0x55d9d0ac9930 2024-04-21T23:12:40.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9fb43a2740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55d9d09c8830 con 0x55d9d0abbd50 2024-04-21T23:12:40.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9fb43a2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d9d0a1c4f0 con 0x55d9d0abc670 2024-04-21T23:12:40.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9fa27fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0ac9930 0x55d9d0acbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9faaffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abc670 0x55d9d0ac93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9faaffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abc670 0x55d9d0ac93f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:50212/0 (socket says 172.21.15.44:50212) 2024-04-21T23:12:40.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9faaffd640 1 -- 172.21.15.44:0/3539601720 learned_addr learned my addr 172.21.15.44:0/3539601720 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:40.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9fa27fc640 1 --2- 172.21.15.44:0/3539601720 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0abbd50 0x55d9d0abc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9fa8ff9640 1 --2- 172.21.15.44:0/4177958982 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9f980a2d00 0x7f9f980a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9faaffd640 1 -- 172.21.15.44:0/3539601720 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0abbd50 msgr2=0x55d9d0abc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:40.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9faaffd640 1 --2- 172.21.15.44:0/3539601720 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0abbd50 0x55d9d0abc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9faaffd640 1 -- 172.21.15.44:0/3539601720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0ac9930 msgr2=0x55d9d0acbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:40.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9faaffd640 1 --2- 172.21.15.44:0/3539601720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0ac9930 0x55d9d0acbd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9faaffd640 1 -- 172.21.15.44:0/3539601720 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9d09ec2c0 con 0x55d9d0abc670 2024-04-21T23:12:40.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9fa27fc640 1 --2- 172.21.15.44:0/3539601720 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0abbd50 0x55d9d0abc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:12:40.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9fa8ff9640 1 -- 172.21.15.44:0/4177958982 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f9fa40cfed0 con 0x7f9fac10bd80 2024-04-21T23:12:40.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9fab7fe640 1 --2- 172.21.15.44:0/4177958982 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9f980a2d00 0x7f9f980a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.914+0000 7f9faaffd640 1 --2- 172.21.15.44:0/3539601720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abc670 0x55d9d0ac93f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c009cf0 tx=0x7f9f9c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=294d4ebe7fa4ec3b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:40.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.915+0000 7f9fa17fa640 1 -- 172.21.15.44:0/3539601720 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c004230 con 0x55d9d0abc670 2024-04-21T23:12:40.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.915+0000 7f9fa17fa640 1 -- 172.21.15.44:0/3539601720 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f9c0043d0 con 0x55d9d0abc670 2024-04-21T23:12:40.916 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.915+0000 7f9fa17fa640 1 -- 172.21.15.44:0/3539601720 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c019900 con 0x55d9d0abc670 2024-04-21T23:12:40.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.915+0000 7f9fb43a2740 1 -- 172.21.15.44:0/3539601720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abc670 msgr2=0x55d9d0ac93f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.915+0000 7f9fb43a2740 1 --2- 172.21.15.44:0/3539601720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abc670 0x55d9d0ac93f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c009cf0 tx=0x7f9f9c011810 comp rx=0 tx=0).stop 2024-04-21T23:12:40.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.915+0000 7f9fab7fe640 1 --2- 172.21.15.44:0/4177958982 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9f980a2d00 0x7f9f980a51c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9fa400eb10 tx=0x7f9fa4004a40 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:40.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.915+0000 7f9fb43a2740 1 -- 172.21.15.44:0/3539601720 shutdown_connections 2024-04-21T23:12:40.920 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.915+0000 7f9fb43a2740 1 --2- 172.21.15.44:0/3539601720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abc670 0x55d9d0ac93f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.915+0000 7f9fb43a2740 1 --2- 172.21.15.44:0/3539601720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0ac9930 0x55d9d0acbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.915+0000 7f9fb43a2740 1 --2- 172.21.15.44:0/3539601720 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0abbd50 0x55d9d0abc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.915+0000 7f9fb43a2740 1 -- 172.21.15.44:0/3539601720 >> 172.21.15.44:0/3539601720 conn(0x55d9d0ab6750 msgr2=0x55d9d0ab6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.915+0000 7f9fb43a2740 1 -- 172.21.15.44:0/3539601720 shutdown_connections 2024-04-21T23:12:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.915+0000 7f9fb43a2740 1 -- 172.21.15.44:0/3539601720 wait complete. 2024-04-21T23:12:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.916+0000 7f9fb43a2740 1 -- start start 2024-04-21T23:12:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.916+0000 7f9fb43a2740 1 Processor -- start 2024-04-21T23:12:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.916+0000 7f9fb43a2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abbd50 0x55d9d0ac9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.916+0000 7f9fb43a2740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0abc670 0x55d9d0ac65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.916+0000 7f9fb43a2740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0ac9930 0x55d9d0ac6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.916+0000 7f9fb43a2740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d9d0acc6f0 con 0x55d9d0abc670 2024-04-21T23:12:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.916+0000 7f9fb43a2740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55d9d09dbf10 con 0x55d9d0ac9930 2024-04-21T23:12:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.916+0000 7f9fb43a2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d9d0acc870 con 0x55d9d0abbd50 2024-04-21T23:12:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.916+0000 7f9fb43a2740 10 client.0 _ll_register_callbacks cb 0x55d9d0733be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:40.921 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.916+0000 7f9faaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0abc670 0x55d9d0ac65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.916+0000 7f9fa27fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abbd50 0x55d9d0ac9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.916+0000 7f9fa27fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abbd50 0x55d9d0ac9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:50220/0 (socket says 172.21.15.44:50220) 2024-04-21T23:12:40.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.916+0000 7f9fa27fc640 1 -- 172.21.15.44:0/2276917864 learned_addr learned my addr 172.21.15.44:0/2276917864 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:40.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.916+0000 7f9fa27fc640 1 --2- 172.21.15.44:0/2276917864 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0ac9930 0x55d9d0ac6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.916+0000 7f9fa27fc640 1 -- 172.21.15.44:0/2276917864 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0ac9930 msgr2=0x55d9d0ac6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:40.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.916+0000 7f9fa27fc640 1 --2- 172.21.15.44:0/2276917864 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0ac9930 0x55d9d0ac6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.916+0000 7f9fa27fc640 1 -- 172.21.15.44:0/2276917864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0abc670 msgr2=0x55d9d0ac65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:40.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.916+0000 7f9fa27fc640 1 --2- 172.21.15.44:0/2276917864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0abc670 0x55d9d0ac65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.916+0000 7f9fa27fc640 1 -- 172.21.15.44:0/2276917864 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9d09adc80 con 0x55d9d0abbd50 2024-04-21T23:12:40.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.917+0000 7f9faaffd640 1 --2- 172.21.15.44:0/2276917864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0abc670 0x55d9d0ac65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:40.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.917+0000 7f9fa27fc640 1 --2- 172.21.15.44:0/2276917864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abbd50 0x55d9d0ac9240 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9f94009230 tx=0x7f9f9402cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=435b2deaa0db90f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:40.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.917+0000 7f9f927fc640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:12:40.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.917+0000 7f9f927fc640 1 -- 172.21.15.44:0/2276917864 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f94004260 con 0x55d9d0abbd50 2024-04-21T23:12:40.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.917+0000 7f9f927fc640 1 -- 172.21.15.44:0/2276917864 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f9f94004400 con 0x55d9d0abbd50 2024-04-21T23:12:40.922 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.917+0000 7f9fb43a2740 1 -- 172.21.15.44:0/2276917864 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f9c0043d0 con 0x55d9d0abbd50 2024-04-21T23:12:40.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.917+0000 7f9f927fc640 1 -- 172.21.15.44:0/2276917864 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9f94005160 con 0x55d9d0abbd50 2024-04-21T23:12:40.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.917+0000 7f9fb43a2740 10 client.26113 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:40.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.917+0000 7f9fb43a2740 1 -- 172.21.15.44:0/2276917864 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d9d07343f0 con 0x55d9d0abbd50 2024-04-21T23:12:40.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.917+0000 7f9f927fc640 1 -- 172.21.15.44:0/2276917864 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f9c0043d0 con 0x55d9d0abbd50 2024-04-21T23:12:40.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.917+0000 7f9f927fc640 1 -- 172.21.15.44:0/2276917864 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f9f94003b50 con 0x55d9d0abbd50 2024-04-21T23:12:40.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.917+0000 7f9fb43a2740 10 client.26113 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:40.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.917+0000 7f9fb43a2740 10 client.26113 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:40.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.918+0000 7f9fb43a2740 20 client.26113 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:40.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.918+0000 7f9fb43a2740 1 --2- 172.21.15.44:0/2276917864 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d9d0ad97d0 0x55d9d0adbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.918+0000 7f9fb43a2740 4 client.26113 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:12:40.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.918+0000 7f9fb43a2740 1 -- 172.21.15.44:0/2276917864 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9d0a2da70 con 0x55d9d0ad97d0 2024-04-21T23:12:40.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.918+0000 7f9faaffd640 1 --2- 172.21.15.44:0/2276917864 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d9d0ad97d0 0x55d9d0adbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.918+0000 7f9faaffd640 1 --2- 172.21.15.44:0/2276917864 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d9d0ad97d0 0x55d9d0adbc00 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14d035f18fca834f server_cookie=ec54703505d752df in_seq=0 out_seq=0 2024-04-21T23:12:40.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.918+0000 7f9f927fc640 10 client.26113 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:40.923 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.919+0000 7f9f927fc640 1 -- 172.21.15.44:0/2276917864 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d9d0a2da70 con 0x55d9d0ad97d0 2024-04-21T23:12:40.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.919+0000 7f9f927fc640 10 client.26113 handle_command_reply: tid=0 2024-04-21T23:12:40.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.919+0000 7f9fb43a2740 1 client.26113 shutdown 2024-04-21T23:12:40.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.919+0000 7f9fb43a2740 1 -- 172.21.15.44:0/2276917864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abbd50 msgr2=0x55d9d0ac9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.919+0000 7f9fb43a2740 1 --2- 172.21.15.44:0/2276917864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abbd50 0x55d9d0ac9240 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9f94009230 tx=0x7f9f9402cf90 comp rx=0 tx=0).stop 2024-04-21T23:12:40.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.919+0000 7f9fb43a2740 1 -- 172.21.15.44:0/2276917864 shutdown_connections 2024-04-21T23:12:40.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.919+0000 7f9fb43a2740 1 --2- 172.21.15.44:0/2276917864 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d9d0ad97d0 0x55d9d0adbc00 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.919+0000 7f9fb43a2740 1 --2- 172.21.15.44:0/2276917864 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0ac9930 0x55d9d0ac6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.919+0000 7f9fb43a2740 1 --2- 172.21.15.44:0/2276917864 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0abc670 0x55d9d0ac65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.919+0000 7f9f927fc640 0 client.26113 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:40.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.919+0000 7f9fb43a2740 1 --2- 172.21.15.44:0/2276917864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abbd50 0x55d9d0ac9240 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.924 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.919+0000 7f9fb43a2740 1 -- 172.21.15.44:0/2276917864 >> 172.21.15.44:0/2276917864 conn(0x55d9d0ab6750 msgr2=0x55d9d0ab84a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:40.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.924+0000 7f9fb43a2740 1 -- 172.21.15.44:0/2276917864 shutdown_connections 2024-04-21T23:12:40.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.924+0000 7f9fb43a2740 1 -- 172.21.15.44:0/2276917864 wait complete. 2024-04-21T23:12:40.925 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.925+0000 7f9fb43a2740 20 client.26113 trim_cache size 0 max 16384 2024-04-21T23:12:40.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.955+0000 7f9fb43a2740 1 Processor -- start 2024-04-21T23:12:40.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.956+0000 7f9fb43a2740 1 -- start start 2024-04-21T23:12:40.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.957+0000 7f9fb43a2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abbd50 0x55d9d0ad55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.957+0000 7f9fb43a2740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0abc670 0x55d9d0ac1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.957+0000 7f9fb43a2740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0ac9930 0x55d9d0ac8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.957+0000 7f9fb43a2740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9fac1aa9a0 con 0x55d9d0ac9930 2024-04-21T23:12:40.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.957+0000 7f9fb43a2740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9fac1ad220 con 0x55d9d0abc670 2024-04-21T23:12:40.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.957+0000 7f9fb43a2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9fac1ab010 con 0x55d9d0abbd50 2024-04-21T23:12:40.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.959+0000 7f9fa27fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0ac9930 0x55d9d0ac8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.959+0000 7f9fa27fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0ac9930 0x55d9d0ac8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39920/0 (socket says 172.21.15.44:39920) 2024-04-21T23:12:40.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.959+0000 7f9fa27fc640 1 -- 172.21.15.44:0/3566696194 learned_addr learned my addr 172.21.15.44:0/3566696194 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:40.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.959+0000 7f9fa27fc640 1 --2- 172.21.15.44:0/3566696194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abbd50 0x55d9d0ad55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.961+0000 7f9faaffd640 1 --2- 172.21.15.44:0/3566696194 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0abc670 0x55d9d0ac1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.961+0000 7f9fa27fc640 1 -- 172.21.15.44:0/3566696194 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0abc670 msgr2=0x55d9d0ac1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:40.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.961+0000 7f9fa27fc640 1 --2- 172.21.15.44:0/3566696194 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0abc670 0x55d9d0ac1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.961+0000 7f9fa27fc640 1 -- 172.21.15.44:0/3566696194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0ac9930 msgr2=0x55d9d0ac8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:40.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.961+0000 7f9fa27fc640 1 --2- 172.21.15.44:0/3566696194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0ac9930 0x55d9d0ac8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.961 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.961+0000 7f9fa27fc640 1 -- 172.21.15.44:0/3566696194 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9d07fe7d0 con 0x55d9d0abbd50 2024-04-21T23:12:40.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.963+0000 7f9fa27fc640 1 --2- 172.21.15.44:0/3566696194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abbd50 0x55d9d0ad55f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9f94005ab0 tx=0x7f9f94000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=320f7bf764ea32b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:40.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.964+0000 7f9fa0ff9640 1 -- 172.21.15.44:0/3566696194 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f94058070 con 0x55d9d0abbd50 2024-04-21T23:12:40.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.965+0000 7f9fa0ff9640 1 -- 172.21.15.44:0/3566696194 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f94059040 con 0x55d9d0abbd50 2024-04-21T23:12:40.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.965+0000 7f9fa0ff9640 1 -- 172.21.15.44:0/3566696194 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f94034960 con 0x55d9d0abbd50 2024-04-21T23:12:40.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.966+0000 7f9fb43a2740 1 -- 172.21.15.44:0/3566696194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abbd50 msgr2=0x55d9d0ad55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.965 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.966+0000 7f9fb43a2740 1 --2- 172.21.15.44:0/3566696194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abbd50 0x55d9d0ad55f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9f94005ab0 tx=0x7f9f94000bc0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.967+0000 7f9fb43a2740 1 -- 172.21.15.44:0/3566696194 shutdown_connections 2024-04-21T23:12:40.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.967+0000 7f9fb43a2740 1 --2- 172.21.15.44:0/3566696194 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0ac9930 0x55d9d0ac8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.967+0000 7f9fb43a2740 1 --2- 172.21.15.44:0/3566696194 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0abc670 0x55d9d0ac1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.967+0000 7f9fb43a2740 1 --2- 172.21.15.44:0/3566696194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abbd50 0x55d9d0ad55f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.967+0000 7f9fb43a2740 1 -- 172.21.15.44:0/3566696194 >> 172.21.15.44:0/3566696194 conn(0x55d9d0ab6750 msgr2=0x55d9d0ac1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:40.966 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.968+0000 7f9fb43a2740 1 -- 172.21.15.44:0/3566696194 shutdown_connections 2024-04-21T23:12:40.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.968+0000 7f9fb43a2740 1 -- 172.21.15.44:0/3566696194 wait complete. 2024-04-21T23:12:40.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.969+0000 7f9fb43a2740 1 -- start start 2024-04-21T23:12:40.967 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.969+0000 7f9fb43a2740 1 Processor -- start 2024-04-21T23:12:40.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.969+0000 7f9fb43a2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abbd50 0x55d9d0ac2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.969+0000 7f9fb43a2740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0abc670 0x55d9d0ac3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.969+0000 7f9fb43a2740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0ac9930 0x55d9d0ac3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.970+0000 7f9fb43a2740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9fac1ab6e0 con 0x55d9d0ac9930 2024-04-21T23:12:40.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.970+0000 7f9fb43a2740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9fac1acb50 con 0x55d9d0abc670 2024-04-21T23:12:40.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.970+0000 7f9fb43a2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9fac1abdb0 con 0x55d9d0abbd50 2024-04-21T23:12:40.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.970+0000 7f9fb43a2740 10 client.0 _ll_register_callbacks cb 0x7f9f94009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:40.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.970+0000 7f9fa27fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abbd50 0x55d9d0ac2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.970+0000 7f9fa27fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abbd50 0x55d9d0ac2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:50248/0 (socket says 172.21.15.44:50248) 2024-04-21T23:12:40.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.970+0000 7f9fa27fc640 1 -- 172.21.15.44:0/1657653101 learned_addr learned my addr 172.21.15.44:0/1657653101 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:40.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.970+0000 7f9fa27fc640 1 --2- 172.21.15.44:0/1657653101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0ac9930 0x55d9d0ac3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.970+0000 7f9faaffd640 1 --2- 172.21.15.44:0/1657653101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0abc670 0x55d9d0ac3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.970+0000 7f9fa27fc640 1 -- 172.21.15.44:0/1657653101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0abc670 msgr2=0x55d9d0ac3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:40.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.970+0000 7f9fa27fc640 1 --2- 172.21.15.44:0/1657653101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0abc670 0x55d9d0ac3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.970+0000 7f9fa27fc640 1 -- 172.21.15.44:0/1657653101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0ac9930 msgr2=0x55d9d0ac3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:40.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.970+0000 7f9fa27fc640 1 --2- 172.21.15.44:0/1657653101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0ac9930 0x55d9d0ac3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.970+0000 7f9fa27fc640 1 -- 172.21.15.44:0/1657653101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9d0ad0af0 con 0x55d9d0abbd50 2024-04-21T23:12:40.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.972+0000 7f9fa27fc640 1 --2- 172.21.15.44:0/1657653101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abbd50 0x55d9d0ac2bf0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9f94004020 tx=0x7f9f9402cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5daeb5117b646bc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:40.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.972+0000 7f9f927fc640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:12:40.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.972+0000 7f9f927fc640 1 -- 172.21.15.44:0/1657653101 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f94058040 con 0x55d9d0abbd50 2024-04-21T23:12:40.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.972+0000 7f9fb43a2740 1 -- 172.21.15.44:0/1657653101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f94059040 con 0x55d9d0abbd50 2024-04-21T23:12:40.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.972+0000 7f9f927fc640 1 -- 172.21.15.44:0/1657653101 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f9f94005300 con 0x55d9d0abbd50 2024-04-21T23:12:40.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.972+0000 7f9f927fc640 1 -- 172.21.15.44:0/1657653101 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9f94005500 con 0x55d9d0abbd50 2024-04-21T23:12:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.974+0000 7f9fb43a2740 10 client.26119 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:40.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.974+0000 7f9fb43a2740 1 -- 172.21.15.44:0/1657653101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d9d0ad1d80 con 0x55d9d0abbd50 2024-04-21T23:12:40.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.977+0000 7f9f927fc640 1 -- 172.21.15.44:0/1657653101 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f94059040 con 0x55d9d0abbd50 2024-04-21T23:12:40.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.977+0000 7f9f927fc640 1 -- 172.21.15.44:0/1657653101 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f9f9405c020 con 0x55d9d0abbd50 2024-04-21T23:12:40.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.977+0000 7f9fb43a2740 10 client.26119 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:40.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.977+0000 7f9fb43a2740 10 client.26119 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:40.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.977+0000 7f9fb43a2740 20 client.26119 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:40.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.977+0000 7f9fb43a2740 1 --2- 172.21.15.44:0/1657653101 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d9d0ad97d0 0x55d9d0b46510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:40.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.977+0000 7f9fb43a2740 4 client.26119 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:12:40.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.977+0000 7f9fb43a2740 1 -- 172.21.15.44:0/1657653101 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d9d0a27dc0 con 0x55d9d0ad97d0 2024-04-21T23:12:40.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.979+0000 7f9faaffd640 1 --2- 172.21.15.44:0/1657653101 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d9d0ad97d0 0x55d9d0b46510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:40.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.980+0000 7f9faaffd640 1 --2- 172.21.15.44:0/1657653101 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d9d0ad97d0 0x55d9d0b46510 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14162a7aa6254d2a server_cookie=2235f5f78077666f in_seq=0 out_seq=0 2024-04-21T23:12:40.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.980+0000 7f9f927fc640 10 client.26119 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:40.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.985+0000 7f9f927fc640 1 -- 172.21.15.44:0/1657653101 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d9d0a27dc0 con 0x55d9d0ad97d0 2024-04-21T23:12:40.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.986+0000 7f9f927fc640 10 client.26119 handle_command_reply: tid=0 2024-04-21T23:12:40.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.986+0000 7f9fb43a2740 1 client.26119 shutdown 2024-04-21T23:12:40.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.986+0000 7f9fb43a2740 1 -- 172.21.15.44:0/1657653101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abbd50 msgr2=0x55d9d0ac2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.986+0000 7f9fb43a2740 1 --2- 172.21.15.44:0/1657653101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abbd50 0x55d9d0ac2bf0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9f94004020 tx=0x7f9f9402cf90 comp rx=0 tx=0).stop 2024-04-21T23:12:40.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.986+0000 7f9fb43a2740 1 -- 172.21.15.44:0/1657653101 shutdown_connections 2024-04-21T23:12:40.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.986+0000 7f9fb43a2740 1 --2- 172.21.15.44:0/1657653101 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d9d0ad97d0 0x55d9d0b46510 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.986+0000 7f9fb43a2740 1 --2- 172.21.15.44:0/1657653101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d9d0ac9930 0x55d9d0ac3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.986+0000 7f9fb43a2740 1 --2- 172.21.15.44:0/1657653101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d9d0abc670 0x55d9d0ac3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.986+0000 7f9fb43a2740 1 --2- 172.21.15.44:0/1657653101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d9d0abbd50 0x55d9d0ac2bf0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.986+0000 7f9fb43a2740 1 -- 172.21.15.44:0/1657653101 >> 172.21.15.44:0/1657653101 conn(0x55d9d0ab6750 msgr2=0x55d9d0ac1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:40.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.986+0000 7f9f927fc640 0 client.26119 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:40.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.987+0000 7f9fb43a2740 1 -- 172.21.15.44:0/1657653101 shutdown_connections 2024-04-21T23:12:40.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.987+0000 7f9fb43a2740 1 -- 172.21.15.44:0/1657653101 wait complete. 2024-04-21T23:12:40.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.987+0000 7f9fb43a2740 20 client.26119 trim_cache size 0 max 16384 2024-04-21T23:12:40.987 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-21T23:12:40.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.989+0000 7f9fb17bf640 1 -- 172.21.15.44:0/4177958982 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9f980a2d00 msgr2=0x7f9f980a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.989+0000 7f9fb17bf640 1 --2- 172.21.15.44:0/4177958982 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9f980a2d00 0x7f9f980a51c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9fa400eb10 tx=0x7f9fa4004a40 comp rx=0 tx=0).stop 2024-04-21T23:12:40.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.989+0000 7f9fb17bf640 1 -- 172.21.15.44:0/4177958982 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fac10bd80 msgr2=0x7f9fac19c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:40.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.989+0000 7f9fb17bf640 1 --2- 172.21.15.44:0/4177958982 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fac10bd80 0x7f9fac19c8c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9fa400eb40 tx=0x7f9fa400c2d0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.989+0000 7f9fb17bf640 1 -- 172.21.15.44:0/4177958982 shutdown_connections 2024-04-21T23:12:40.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.989+0000 7f9fb17bf640 1 --2- 172.21.15.44:0/4177958982 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9f980a2d00 0x7f9f980a51c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.989+0000 7f9fb17bf640 1 --2- 172.21.15.44:0/4177958982 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fac10bd80 0x7f9fac19c8c0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.989+0000 7f9fb17bf640 1 --2- 172.21.15.44:0/4177958982 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fac1032d0 0x7f9fac198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.989+0000 7f9fb17bf640 1 --2- 172.21.15.44:0/4177958982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fac1029b0 0x7f9fac198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:40.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.989+0000 7f9fb17bf640 1 -- 172.21.15.44:0/4177958982 >> 172.21.15.44:0/4177958982 conn(0x7f9fac077730 msgr2=0x7f9fac0fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:40.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.990+0000 7f9fb17bf640 1 -- 172.21.15.44:0/4177958982 shutdown_connections 2024-04-21T23:12:40.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:40.990+0000 7f9fb17bf640 1 -- 172.21.15.44:0/4177958982 wait complete. 2024-04-21T23:12:41.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:40 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/172527668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:12:41.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:40 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/172527668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:12:41.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:40 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/172527668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:12:41.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 389897 13.40 MB/sec execute 31 sec latency 26.563 ms 2024-04-21T23:12:42.007 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:12:42.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:12:42.008 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:12:42.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.116+0000 7f09abdc9640 1 -- 172.21.15.44:0/723112785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09a4103360 msgr2=0x7f09a410b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:42.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.116+0000 7f09abdc9640 1 --2- 172.21.15.44:0/723112785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09a4103360 0x7f09a410b7b0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009900 tx=0x7f09a002bc30 comp rx=0 tx=0).stop 2024-04-21T23:12:42.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.116+0000 7f09abdc9640 1 -- 172.21.15.44:0/723112785 shutdown_connections 2024-04-21T23:12:42.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.116+0000 7f09abdc9640 1 --2- 172.21.15.44:0/723112785 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a410bec0 0x7f09a410e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.116+0000 7f09abdc9640 1 --2- 172.21.15.44:0/723112785 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09a4103360 0x7f09a410b7b0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.116+0000 7f09abdc9640 1 --2- 172.21.15.44:0/723112785 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09a41029b0 0x7f09a4102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.116+0000 7f09abdc9640 1 -- 172.21.15.44:0/723112785 >> 172.21.15.44:0/723112785 conn(0x7f09a4077760 msgr2=0x7f09a4077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:42.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.117+0000 7f09abdc9640 1 -- 172.21.15.44:0/723112785 shutdown_connections 2024-04-21T23:12:42.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.117+0000 7f09abdc9640 1 -- 172.21.15.44:0/723112785 wait complete. 2024-04-21T23:12:42.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.117+0000 7f09abdc9640 1 Processor -- start 2024-04-21T23:12:42.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.117+0000 7f09abdc9640 1 -- start start 2024-04-21T23:12:42.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.117+0000 7f09abdc9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09a41029b0 0x7f09a41981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.117+0000 7f09abdc9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09a410bec0 0x7f09a4198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.117+0000 7f09abdc9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a419cab0 0x7f09a419cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.117+0000 7f09abdc9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f09a4111c70 con 0x7f09a41029b0 2024-04-21T23:12:42.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.117+0000 7f09abdc9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f09a4111af0 con 0x7f09a410bec0 2024-04-21T23:12:42.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.117+0000 7f09abdc9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f09a4111df0 con 0x7f09a419cab0 2024-04-21T23:12:42.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.118+0000 7f09a9b65640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09a41029b0 0x7f09a41981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.118+0000 7f09a9b65640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09a41029b0 0x7f09a41981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39946/0 (socket says 172.21.15.44:39946) 2024-04-21T23:12:42.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.118+0000 7f09a9b65640 1 -- 172.21.15.44:0/820472217 learned_addr learned my addr 172.21.15.44:0/820472217 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:42.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.118+0000 7f09aa366640 1 --2- 172.21.15.44:0/820472217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a419cab0 0x7f09a419cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.118+0000 7f09a9364640 1 --2- 172.21.15.44:0/820472217 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09a410bec0 0x7f09a4198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.118+0000 7f09a9b65640 1 -- 172.21.15.44:0/820472217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a419cab0 msgr2=0x7f09a419cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:42.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.118+0000 7f09a9b65640 1 --2- 172.21.15.44:0/820472217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a419cab0 0x7f09a419cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.118+0000 7f09a9b65640 1 -- 172.21.15.44:0/820472217 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09a410bec0 msgr2=0x7f09a4198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:42.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.118+0000 7f09a9b65640 1 --2- 172.21.15.44:0/820472217 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09a410bec0 0x7f09a4198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.118+0000 7f09a9b65640 1 -- 172.21.15.44:0/820472217 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a419d5a0 con 0x7f09a41029b0 2024-04-21T23:12:42.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.118+0000 7f09a9b65640 1 --2- 172.21.15.44:0/820472217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09a41029b0 0x7f09a41981f0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f099800ea60 tx=0x7f099800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:42.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.118+0000 7f0996ffd640 1 -- 172.21.15.44:0/820472217 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099800cde0 con 0x7f09a41029b0 2024-04-21T23:12:42.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.118+0000 7f09abdc9640 1 -- 172.21.15.44:0/820472217 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a419d890 con 0x7f09a41029b0 2024-04-21T23:12:42.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.118+0000 7f0996ffd640 1 -- 172.21.15.44:0/820472217 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09980035f0 con 0x7f09a41029b0 2024-04-21T23:12:42.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.118+0000 7f0996ffd640 1 -- 172.21.15.44:0/820472217 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998010670 con 0x7f09a41029b0 2024-04-21T23:12:42.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.118+0000 7f09abdc9640 1 -- 172.21.15.44:0/820472217 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a41a5050 con 0x7f09a41029b0 2024-04-21T23:12:42.126 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.121+0000 7f0996ffd640 1 -- 172.21.15.44:0/820472217 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f09980037a0 con 0x7f09a41029b0 2024-04-21T23:12:42.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.121+0000 7f09ae9b1740 1 Processor -- start 2024-04-21T23:12:42.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.122+0000 7f09ae9b1740 1 -- start start 2024-04-21T23:12:42.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.122+0000 7f09ae9b1740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313488e720 0x56313488d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.122+0000 7f09ae9b1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488da40 0x56313488dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.122+0000 7f09ae9b1740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313489b1c0 0x56313489d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.122+0000 7f09ae9b1740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5631347f1170 con 0x56313489b1c0 2024-04-21T23:12:42.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.122+0000 7f09ae9b1740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5631347997c0 con 0x56313488e720 2024-04-21T23:12:42.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.122+0000 7f09ae9b1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5631347ed130 con 0x56313488da40 2024-04-21T23:12:42.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.122+0000 7f0996ffd640 1 --2- 172.21.15.44:0/820472217 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f09900a2d00 0x7f09900a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.122+0000 7f09a9364640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313489b1c0 0x56313489d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.122+0000 7f09a9364640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313489b1c0 0x56313489d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39956/0 (socket says 172.21.15.44:39956) 2024-04-21T23:12:42.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.122+0000 7f09a9364640 1 -- 172.21.15.44:0/2442076901 learned_addr learned my addr 172.21.15.44:0/2442076901 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:42.127 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.122+0000 7f09aa366640 1 --2- 172.21.15.44:0/2442076901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313488e720 0x56313488d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.122+0000 7f09a9b65640 1 --2- 172.21.15.44:0/2442076901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488da40 0x56313488dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.122+0000 7f09aa366640 1 --2- 172.21.15.44:0/820472217 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f09900a2d00 0x7f09900a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.122+0000 7f0996ffd640 1 -- 172.21.15.44:0/820472217 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f09980cfaa0 con 0x7f09a41029b0 2024-04-21T23:12:42.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.122+0000 7f09a9364640 1 -- 172.21.15.44:0/2442076901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488da40 msgr2=0x56313488dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:42.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.122+0000 7f09a9364640 1 --2- 172.21.15.44:0/2442076901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488da40 0x56313488dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.123+0000 7f09a9364640 1 -- 172.21.15.44:0/2442076901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313488e720 msgr2=0x56313488d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:42.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.123+0000 7f09a9364640 1 --2- 172.21.15.44:0/2442076901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313488e720 0x56313488d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.123+0000 7f09a9364640 1 -- 172.21.15.44:0/2442076901 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631347aae60 con 0x56313489b1c0 2024-04-21T23:12:42.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.123+0000 7f09aa366640 1 --2- 172.21.15.44:0/2442076901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313488e720 0x56313488d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:42.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.123+0000 7f09aa366640 1 --2- 172.21.15.44:0/820472217 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f09900a2d00 0x7f09900a51c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f099c005fd0 tx=0x7f099c009500 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:42.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.123+0000 7f09a9364640 1 --2- 172.21.15.44:0/2442076901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313489b1c0 0x56313489d5b0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f09a002cfd0 tx=0x7f09a0005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96776d7a9b8ec17 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:42.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.123+0000 7f098ffff640 1 -- 172.21.15.44:0/2442076901 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0042070 con 0x56313489b1c0 2024-04-21T23:12:42.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.123+0000 7f098ffff640 1 -- 172.21.15.44:0/2442076901 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09a0004030 con 0x56313489b1c0 2024-04-21T23:12:42.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.123+0000 7f098ffff640 1 -- 172.21.15.44:0/2442076901 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0004f20 con 0x56313489b1c0 2024-04-21T23:12:42.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.124+0000 7f09ae9b1740 1 -- 172.21.15.44:0/2442076901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313489b1c0 msgr2=0x56313489d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:42.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.124+0000 7f09ae9b1740 1 --2- 172.21.15.44:0/2442076901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313489b1c0 0x56313489d5b0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f09a002cfd0 tx=0x7f09a0005be0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.124+0000 7f09ae9b1740 1 -- 172.21.15.44:0/2442076901 shutdown_connections 2024-04-21T23:12:42.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.124+0000 7f09ae9b1740 1 --2- 172.21.15.44:0/2442076901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488da40 0x56313488dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.124+0000 7f09ae9b1740 1 --2- 172.21.15.44:0/2442076901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313489b1c0 0x56313489d5b0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.124+0000 7f09ae9b1740 1 --2- 172.21.15.44:0/2442076901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313488e720 0x56313488d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.124+0000 7f09ae9b1740 1 -- 172.21.15.44:0/2442076901 >> 172.21.15.44:0/2442076901 conn(0x5631348544f0 msgr2=0x563134888490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:42.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.124+0000 7f09ae9b1740 1 -- 172.21.15.44:0/2442076901 shutdown_connections 2024-04-21T23:12:42.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.124+0000 7f09ae9b1740 1 -- 172.21.15.44:0/2442076901 wait complete. 2024-04-21T23:12:42.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.124+0000 7f09ae9b1740 1 -- start start 2024-04-21T23:12:42.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.124+0000 7f09ae9b1740 1 Processor -- start 2024-04-21T23:12:42.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f09ae9b1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488da40 0x563134898160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f09ae9b1740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313488e720 0x5631348986a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f09ae9b1740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313489b1c0 0x563134898be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f09ae9b1740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56313489dfd0 con 0x56313489b1c0 2024-04-21T23:12:42.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f09ae9b1740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5631347acd40 con 0x56313488e720 2024-04-21T23:12:42.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f09ae9b1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56313489e150 con 0x56313488da40 2024-04-21T23:12:42.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f09ae9b1740 10 client.0 _ll_register_callbacks cb 0x5631347e8820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:42.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f09aa366640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313489b1c0 0x563134898be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f09a9b65640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488da40 0x563134898160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f09a9b65640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488da40 0x563134898160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:50274/0 (socket says 172.21.15.44:50274) 2024-04-21T23:12:42.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f09a9b65640 1 -- 172.21.15.44:0/153590891 learned_addr learned my addr 172.21.15.44:0/153590891 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:42.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f09aa366640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313489b1c0 0x563134898be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39964/0 (socket says 172.21.15.44:39964) 2024-04-21T23:12:42.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f09a9364640 1 --2- 172.21.15.44:0/153590891 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313488e720 0x5631348986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f09a9b65640 1 -- 172.21.15.44:0/153590891 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313488e720 msgr2=0x5631348986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:42.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f09a9b65640 1 --2- 172.21.15.44:0/153590891 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313488e720 0x5631348986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f09a9b65640 1 -- 172.21.15.44:0/153590891 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313489b1c0 msgr2=0x563134898be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:42.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f09a9b65640 1 --2- 172.21.15.44:0/153590891 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313489b1c0 0x563134898be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f09a9b65640 1 -- 172.21.15.44:0/153590891 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631345cd7e0 con 0x56313488da40 2024-04-21T23:12:42.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f09a9b65640 1 --2- 172.21.15.44:0/153590891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488da40 0x563134898160 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f099800c760 tx=0x7f0998066cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7822a8f676b2987 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:42.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f098d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:12:42.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f098d7fa640 1 -- 172.21.15.44:0/153590891 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099809ffd0 con 0x56313488da40 2024-04-21T23:12:42.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f098d7fa640 1 -- 172.21.15.44:0/153590891 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f0998098290 con 0x56313488da40 2024-04-21T23:12:42.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.125+0000 7f09ae9b1740 1 -- 172.21.15.44:0/153590891 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09a0004030 con 0x56313488da40 2024-04-21T23:12:42.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.126+0000 7f098d7fa640 1 -- 172.21.15.44:0/153590891 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f09980a4890 con 0x56313488da40 2024-04-21T23:12:42.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.126+0000 7f09ae9b1740 10 client.26131 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:42.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.126+0000 7f09ae9b1740 1 -- 172.21.15.44:0/153590891 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56313462df70 con 0x56313488da40 2024-04-21T23:12:42.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.126+0000 7f098d7fa640 1 -- 172.21.15.44:0/153590891 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09a0004030 con 0x56313488da40 2024-04-21T23:12:42.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.126+0000 7f098d7fa640 1 -- 172.21.15.44:0/153590891 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f099809b070 con 0x56313488da40 2024-04-21T23:12:42.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.126+0000 7f09ae9b1740 10 client.26131 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:42.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.126+0000 7f09ae9b1740 10 client.26131 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:42.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.126+0000 7f09ae9b1740 20 client.26131 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:42.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.126+0000 7f09ae9b1740 1 --2- 172.21.15.44:0/153590891 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5631348ab2c0 0x5631348ad6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.126+0000 7f09ae9b1740 4 client.26131 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:12:42.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.126+0000 7f09ae9b1740 1 -- 172.21.15.44:0/153590891 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631347c5450 con 0x5631348ab2c0 2024-04-21T23:12:42.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.126+0000 7f09a9364640 1 --2- 172.21.15.44:0/153590891 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5631348ab2c0 0x5631348ad6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.127+0000 7f098d7fa640 10 client.26131 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:42.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.127+0000 7f09a9364640 1 --2- 172.21.15.44:0/153590891 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5631348ab2c0 0x5631348ad6f0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f27eba6ef230212 server_cookie=5f7d8574559499b9 in_seq=0 out_seq=0 2024-04-21T23:12:42.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.127+0000 7f098d7fa640 1 -- 172.21.15.44:0/153590891 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5631347c5450 con 0x5631348ab2c0 2024-04-21T23:12:42.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.127+0000 7f098d7fa640 10 client.26131 handle_command_reply: tid=0 2024-04-21T23:12:42.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.127+0000 7f09ae9b1740 1 client.26131 shutdown 2024-04-21T23:12:42.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.127+0000 7f09ae9b1740 1 -- 172.21.15.44:0/153590891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488da40 msgr2=0x563134898160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:42.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.127+0000 7f09ae9b1740 1 --2- 172.21.15.44:0/153590891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488da40 0x563134898160 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f099800c760 tx=0x7f0998066cf0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.128+0000 7f09ae9b1740 1 -- 172.21.15.44:0/153590891 shutdown_connections 2024-04-21T23:12:42.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.128+0000 7f09ae9b1740 1 --2- 172.21.15.44:0/153590891 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5631348ab2c0 0x5631348ad6f0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.128+0000 7f09ae9b1740 1 --2- 172.21.15.44:0/153590891 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313489b1c0 0x563134898be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.128+0000 7f09ae9b1740 1 --2- 172.21.15.44:0/153590891 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313488e720 0x5631348986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.128+0000 7f098d7fa640 0 client.26131 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:42.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.128+0000 7f09ae9b1740 1 --2- 172.21.15.44:0/153590891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488da40 0x563134898160 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.128+0000 7f09ae9b1740 1 -- 172.21.15.44:0/153590891 >> 172.21.15.44:0/153590891 conn(0x5631348544f0 msgr2=0x56313488a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:42.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.128+0000 7f09ae9b1740 1 -- 172.21.15.44:0/153590891 shutdown_connections 2024-04-21T23:12:42.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.128+0000 7f09ae9b1740 1 -- 172.21.15.44:0/153590891 wait complete. 2024-04-21T23:12:42.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.128+0000 7f09ae9b1740 20 client.26131 trim_cache size 0 max 16384 2024-04-21T23:12:42.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.146+0000 7f09ae9b1740 1 Processor -- start 2024-04-21T23:12:42.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.146+0000 7f09ae9b1740 1 -- start start 2024-04-21T23:12:42.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f09ae9b1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488da40 0x56313489a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f09ae9b1740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313488e720 0x5631348a6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f09ae9b1740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313489b1c0 0x5631348b2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f09ae9b1740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f09a41aa920 con 0x56313489b1c0 2024-04-21T23:12:42.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f09ae9b1740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f09a41ad1a0 con 0x56313488e720 2024-04-21T23:12:42.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f09ae9b1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f09a41aaf90 con 0x56313488da40 2024-04-21T23:12:42.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f09aa366640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313489b1c0 0x5631348b2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f09aa366640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313489b1c0 0x5631348b2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39970/0 (socket says 172.21.15.44:39970) 2024-04-21T23:12:42.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f09aa366640 1 -- 172.21.15.44:0/1889419107 learned_addr learned my addr 172.21.15.44:0/1889419107 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:42.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f09a9b65640 1 --2- 172.21.15.44:0/1889419107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488da40 0x56313489a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f09a9364640 1 --2- 172.21.15.44:0/1889419107 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313488e720 0x5631348a6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f09aa366640 1 -- 172.21.15.44:0/1889419107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488da40 msgr2=0x56313489a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:42.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f09aa366640 1 --2- 172.21.15.44:0/1889419107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488da40 0x56313489a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f09aa366640 1 -- 172.21.15.44:0/1889419107 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313488e720 msgr2=0x5631348a6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:42.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f09aa366640 1 --2- 172.21.15.44:0/1889419107 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313488e720 0x5631348a6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f09aa366640 1 -- 172.21.15.44:0/1889419107 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631344c5170 con 0x56313489b1c0 2024-04-21T23:12:42.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f09a9b65640 1 --2- 172.21.15.44:0/1889419107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488da40 0x56313489a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:42.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f09aa366640 1 --2- 172.21.15.44:0/1889419107 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313489b1c0 0x5631348b2f90 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f099c00fde0 tx=0x7f099c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f548b6fd1cd0d5a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:42.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f098f7fe640 1 -- 172.21.15.44:0/1889419107 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099c004b20 con 0x56313489b1c0 2024-04-21T23:12:42.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f098f7fe640 1 -- 172.21.15.44:0/1889419107 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f099c004cc0 con 0x56313489b1c0 2024-04-21T23:12:42.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.147+0000 7f098f7fe640 1 -- 172.21.15.44:0/1889419107 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099c0055f0 con 0x56313489b1c0 2024-04-21T23:12:42.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.148+0000 7f09ae9b1740 1 -- 172.21.15.44:0/1889419107 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313489b1c0 msgr2=0x5631348b2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:42.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.148+0000 7f09ae9b1740 1 --2- 172.21.15.44:0/1889419107 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313489b1c0 0x5631348b2f90 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f099c00fde0 tx=0x7f099c016000 comp rx=0 tx=0).stop 2024-04-21T23:12:42.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.148+0000 7f09ae9b1740 1 -- 172.21.15.44:0/1889419107 shutdown_connections 2024-04-21T23:12:42.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.148+0000 7f09ae9b1740 1 --2- 172.21.15.44:0/1889419107 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313489b1c0 0x5631348b2f90 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.148+0000 7f09ae9b1740 1 --2- 172.21.15.44:0/1889419107 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313488e720 0x5631348a6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.148+0000 7f09ae9b1740 1 --2- 172.21.15.44:0/1889419107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488da40 0x56313489a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.148+0000 7f09ae9b1740 1 -- 172.21.15.44:0/1889419107 >> 172.21.15.44:0/1889419107 conn(0x5631348544f0 msgr2=0x56313488a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:42.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.148+0000 7f09ae9b1740 1 -- 172.21.15.44:0/1889419107 shutdown_connections 2024-04-21T23:12:42.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.148+0000 7f09ae9b1740 1 -- 172.21.15.44:0/1889419107 wait complete. 2024-04-21T23:12:42.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.148+0000 7f09ae9b1740 1 -- start start 2024-04-21T23:12:42.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.148+0000 7f09ae9b1740 1 Processor -- start 2024-04-21T23:12:42.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.149+0000 7f09ae9b1740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313488da40 0x5631348a33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.149+0000 7f09ae9b1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488e720 0x5631348a3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.149+0000 7f09ae9b1740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313489b1c0 0x56313489c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.149+0000 7f09ae9b1740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f09a41ab660 con 0x56313488da40 2024-04-21T23:12:42.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.149+0000 7f09ae9b1740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f09a41acad0 con 0x56313489b1c0 2024-04-21T23:12:42.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.149+0000 7f09ae9b1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f09a41abd30 con 0x56313488e720 2024-04-21T23:12:42.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.149+0000 7f09ae9b1740 10 client.0 _ll_register_callbacks cb 0x7f099800e9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:42.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.149+0000 7f09a9b65640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313488da40 0x5631348a33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.149+0000 7f09a9b65640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313488da40 0x5631348a33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39978/0 (socket says 172.21.15.44:39978) 2024-04-21T23:12:42.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.149+0000 7f09a9b65640 1 -- 172.21.15.44:0/1181403212 learned_addr learned my addr 172.21.15.44:0/1181403212 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:42.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.149+0000 7f09a9364640 1 --2- 172.21.15.44:0/1181403212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488e720 0x5631348a3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.149+0000 7f09aa366640 1 --2- 172.21.15.44:0/1181403212 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313489b1c0 0x56313489c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.149+0000 7f09a9b65640 1 -- 172.21.15.44:0/1181403212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488e720 msgr2=0x5631348a3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:42.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.149+0000 7f09a9b65640 1 --2- 172.21.15.44:0/1181403212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488e720 0x5631348a3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.149+0000 7f09a9b65640 1 -- 172.21.15.44:0/1181403212 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313489b1c0 msgr2=0x56313489c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:42.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.149+0000 7f09a9b65640 1 --2- 172.21.15.44:0/1181403212 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313489b1c0 0x56313489c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.149+0000 7f09a9b65640 1 -- 172.21.15.44:0/1181403212 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56313489c8a0 con 0x56313488da40 2024-04-21T23:12:42.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.149+0000 7f09a9364640 1 --2- 172.21.15.44:0/1181403212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488e720 0x5631348a3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:42.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.149+0000 7f09a9b65640 1 --2- 172.21.15.44:0/1181403212 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313488da40 0x5631348a33e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f09980cd450 tx=0x7f0998003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed3ccda2b0d15df4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:42.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.149+0000 7f098d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:12:42.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.150+0000 7f098d7fa640 1 -- 172.21.15.44:0/1181403212 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099809ffd0 con 0x56313488da40 2024-04-21T23:12:42.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.150+0000 7f098d7fa640 1 -- 172.21.15.44:0/1181403212 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f0998098290 con 0x56313488da40 2024-04-21T23:12:42.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.150+0000 7f09ae9b1740 1 -- 172.21.15.44:0/1181403212 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f099c004cc0 con 0x56313488da40 2024-04-21T23:12:42.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.150+0000 7f098d7fa640 1 -- 172.21.15.44:0/1181403212 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f09980a4400 con 0x56313488da40 2024-04-21T23:12:42.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.150+0000 7f09ae9b1740 10 client.16533 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:42.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.150+0000 7f09ae9b1740 1 -- 172.21.15.44:0/1181403212 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563134912040 con 0x56313488da40 2024-04-21T23:12:42.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.150+0000 7f098d7fa640 1 -- 172.21.15.44:0/1181403212 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f099c004cc0 con 0x56313488da40 2024-04-21T23:12:42.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.150+0000 7f098d7fa640 1 -- 172.21.15.44:0/1181403212 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f0998097970 con 0x56313488da40 2024-04-21T23:12:42.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.150+0000 7f09ae9b1740 10 client.16533 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:42.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.150+0000 7f09ae9b1740 10 client.16533 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:42.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.150+0000 7f09ae9b1740 20 client.16533 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:42.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.150+0000 7f09ae9b1740 1 --2- 172.21.15.44:0/1181403212 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5631349125a0 0x5631349149c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.150+0000 7f09ae9b1740 4 client.16533 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:12:42.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.150+0000 7f09ae9b1740 1 -- 172.21.15.44:0/1181403212 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5631347fe6b0 con 0x5631349125a0 2024-04-21T23:12:42.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.150+0000 7f09a9364640 1 --2- 172.21.15.44:0/1181403212 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5631349125a0 0x5631349149c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.151+0000 7f098d7fa640 10 client.16533 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:42.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.151+0000 7f09a9364640 1 --2- 172.21.15.44:0/1181403212 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5631349125a0 0x5631349149c0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61abae0115ac7bb8 server_cookie=a6ee69ee41222137 in_seq=0 out_seq=0 2024-04-21T23:12:42.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.153+0000 7f098d7fa640 1 -- 172.21.15.44:0/1181403212 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5631347fe6b0 con 0x5631349125a0 2024-04-21T23:12:42.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.153+0000 7f098d7fa640 10 client.16533 handle_command_reply: tid=0 2024-04-21T23:12:42.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.153+0000 7f09ae9b1740 1 client.16533 shutdown 2024-04-21T23:12:42.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.153+0000 7f09ae9b1740 1 -- 172.21.15.44:0/1181403212 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313488da40 msgr2=0x5631348a33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:42.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.153+0000 7f09ae9b1740 1 --2- 172.21.15.44:0/1181403212 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313488da40 0x5631348a33e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f09980cd450 tx=0x7f0998003d60 comp rx=0 tx=0).stop 2024-04-21T23:12:42.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09ae9b1740 1 -- 172.21.15.44:0/1181403212 shutdown_connections 2024-04-21T23:12:42.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09ae9b1740 1 --2- 172.21.15.44:0/1181403212 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5631349125a0 0x5631349149c0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09ae9b1740 1 --2- 172.21.15.44:0/1181403212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56313488e720 0x5631348a3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f098d7fa640 0 client.16533 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:42.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09ae9b1740 1 --2- 172.21.15.44:0/1181403212 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56313489b1c0 0x56313489c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09ae9b1740 1 --2- 172.21.15.44:0/1181403212 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56313488da40 0x5631348a33e0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09ae9b1740 1 -- 172.21.15.44:0/1181403212 >> 172.21.15.44:0/1181403212 conn(0x5631348544f0 msgr2=0x563134886c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:42.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09ae9b1740 1 -- 172.21.15.44:0/1181403212 shutdown_connections 2024-04-21T23:12:42.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09ae9b1740 1 -- 172.21.15.44:0/1181403212 wait complete. 2024-04-21T23:12:42.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09ae9b1740 20 client.16533 trim_cache size 0 max 16384 2024-04-21T23:12:42.157 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:12:42.157 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-21T23:12:42.157 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "35aa3258-4ae0-4998-ac83-4497b35a5988", 2024-04-21T23:12:42.157 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-21T23:12:42.157 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:12:42.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09abdc9640 1 -- 172.21.15.44:0/820472217 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f09900a2d00 msgr2=0x7f09900a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:42.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09abdc9640 1 --2- 172.21.15.44:0/820472217 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f09900a2d00 0x7f09900a51c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f099c005fd0 tx=0x7f099c009500 comp rx=0 tx=0).stop 2024-04-21T23:12:42.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09abdc9640 1 -- 172.21.15.44:0/820472217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09a41029b0 msgr2=0x7f09a41981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:42.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09abdc9640 1 --2- 172.21.15.44:0/820472217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09a41029b0 0x7f09a41981f0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f099800ea60 tx=0x7f099800ef20 comp rx=0 tx=0).stop 2024-04-21T23:12:42.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09abdc9640 1 -- 172.21.15.44:0/820472217 shutdown_connections 2024-04-21T23:12:42.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09abdc9640 1 --2- 172.21.15.44:0/820472217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a419cab0 0x7f09a419cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09abdc9640 1 --2- 172.21.15.44:0/820472217 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f09900a2d00 0x7f09900a51c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09abdc9640 1 --2- 172.21.15.44:0/820472217 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09a410bec0 0x7f09a4198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09abdc9640 1 --2- 172.21.15.44:0/820472217 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f09a41029b0 0x7f09a41981f0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09abdc9640 1 -- 172.21.15.44:0/820472217 >> 172.21.15.44:0/820472217 conn(0x7f09a4077760 msgr2=0x7f09a40fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:42.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09abdc9640 1 -- 172.21.15.44:0/820472217 shutdown_connections 2024-04-21T23:12:42.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.154+0000 7f09abdc9640 1 -- 172.21.15.44:0/820472217 wait complete. 2024-04-21T23:12:42.162 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:41 smithi044 ceph-mon[27934]: pgmap v120: 129 pgs: 129 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 396 op/s 2024-04-21T23:12:42.164 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:12:42.164 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:12:42.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:41 smithi055 ceph-mon[31117]: pgmap v120: 129 pgs: 129 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 396 op/s 2024-04-21T23:12:42.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:41 smithi069 ceph-mon[31816]: pgmap v120: 129 pgs: 129 active+clean; 48 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 396 op/s 2024-04-21T23:12:42.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.280+0000 7f5ef16c9640 1 -- 172.21.15.44:0/1858959506 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5eec101d10 msgr2=0x7f5eec1052b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:42.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.280+0000 7f5ef16c9640 1 --2- 172.21.15.44:0/1858959506 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5eec101d10 0x7f5eec1052b0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4009900 tx=0x7f5ed402bc30 comp rx=0 tx=0).stop 2024-04-21T23:12:42.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.280+0000 7f5ef16c9640 1 -- 172.21.15.44:0/1858959506 shutdown_connections 2024-04-21T23:12:42.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.280+0000 7f5ef16c9640 1 --2- 172.21.15.44:0/1858959506 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5eec105990 0x7f5eec10d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.280+0000 7f5ef16c9640 1 --2- 172.21.15.44:0/1858959506 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5eec101d10 0x7f5eec1052b0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.280+0000 7f5ef16c9640 1 --2- 172.21.15.44:0/1858959506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eec101360 0x7f5eec101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.280+0000 7f5ef16c9640 1 -- 172.21.15.44:0/1858959506 >> 172.21.15.44:0/1858959506 conn(0x7f5eec075930 msgr2=0x7f5eec075d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:42.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.280+0000 7f5ef16c9640 1 -- 172.21.15.44:0/1858959506 shutdown_connections 2024-04-21T23:12:42.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.280+0000 7f5ef16c9640 1 -- 172.21.15.44:0/1858959506 wait complete. 2024-04-21T23:12:42.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.280+0000 7f5ef16c9640 1 Processor -- start 2024-04-21T23:12:42.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.280+0000 7f5ef16c9640 1 -- start start 2024-04-21T23:12:42.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5ef16c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eec101360 0x7f5eec198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5ef16c9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5eec105990 0x7f5eec198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5ef16c9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5eec19c8c0 0x7f5eec19cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5ef16c9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec110a00 con 0x7f5eec105990 2024-04-21T23:12:42.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5ef16c9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec110880 con 0x7f5eec19c8c0 2024-04-21T23:12:42.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5ef16c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec110b80 con 0x7f5eec101360 2024-04-21T23:12:42.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5ee3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5eec105990 0x7f5eec198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5eeaffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eec101360 0x7f5eec198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5ee3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5eec105990 0x7f5eec198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39996/0 (socket says 172.21.15.44:39996) 2024-04-21T23:12:42.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5ee3fff640 1 -- 172.21.15.44:0/2457945306 learned_addr learned my addr 172.21.15.44:0/2457945306 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:42.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5eeb7fe640 1 --2- 172.21.15.44:0/2457945306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5eec19c8c0 0x7f5eec19cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5eeaffd640 1 -- 172.21.15.44:0/2457945306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5eec19c8c0 msgr2=0x7f5eec19cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:42.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5eeaffd640 1 --2- 172.21.15.44:0/2457945306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5eec19c8c0 0x7f5eec19cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5eeaffd640 1 -- 172.21.15.44:0/2457945306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5eec105990 msgr2=0x7f5eec198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:42.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5eeaffd640 1 --2- 172.21.15.44:0/2457945306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5eec105990 0x7f5eec198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5eeaffd640 1 -- 172.21.15.44:0/2457945306 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eec19d3b0 con 0x7f5eec101360 2024-04-21T23:12:42.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5eeaffd640 1 --2- 172.21.15.44:0/2457945306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eec101360 0x7f5eec198000 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5edc00e9c0 tx=0x7f5edc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:42.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5ee8ff9640 1 -- 172.21.15.44:0/2457945306 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5edc00ccc0 con 0x7f5eec101360 2024-04-21T23:12:42.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5ef16c9640 1 -- 172.21.15.44:0/2457945306 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5eec19d6a0 con 0x7f5eec101360 2024-04-21T23:12:42.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5ee8ff9640 1 -- 172.21.15.44:0/2457945306 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5edc0035f0 con 0x7f5eec101360 2024-04-21T23:12:42.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5ee8ff9640 1 -- 172.21.15.44:0/2457945306 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5edc00ce60 con 0x7f5eec101360 2024-04-21T23:12:42.284 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.281+0000 7f5ef16c9640 1 -- 172.21.15.44:0/2457945306 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5eec1a4e90 con 0x7f5eec101360 2024-04-21T23:12:42.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.284+0000 7f5ee8ff9640 1 -- 172.21.15.44:0/2457945306 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f5edc0037a0 con 0x7f5eec101360 2024-04-21T23:12:42.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.285+0000 7f5ef438d740 1 Processor -- start 2024-04-21T23:12:42.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.285+0000 7f5ef438d740 1 -- start start 2024-04-21T23:12:42.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.285+0000 7f5ee8ff9640 1 --2- 172.21.15.44:0/2457945306 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5ed80a2d00 0x7f5ed80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.285+0000 7f5ef438d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 0x562ee003bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.285+0000 7f5ef438d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562ee003c500 0x562ee0049300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.285+0000 7f5ef438d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562ee0049840 0x562ee004bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.290 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.285+0000 7f5ef438d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562edffa0560 con 0x562ee003bbe0 2024-04-21T23:12:42.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.285+0000 7f5ef438d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x562edff48730 con 0x562ee0049840 2024-04-21T23:12:42.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.285+0000 7f5ef438d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562edff9c520 con 0x562ee003c500 2024-04-21T23:12:42.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.285+0000 7f5eeaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 0x562ee003bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.285+0000 7f5eeb7fe640 1 --2- 172.21.15.44:0/2457945306 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5ed80a2d00 0x7f5ed80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.285+0000 7f5ee3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562ee003c500 0x562ee0049300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.285+0000 7f5eeaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 0x562ee003bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40008/0 (socket says 172.21.15.44:40008) 2024-04-21T23:12:42.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.285+0000 7f5eeaffd640 1 -- 172.21.15.44:0/79613408 learned_addr learned my addr 172.21.15.44:0/79613408 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:42.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.285+0000 7f5eeb7fe640 1 --2- 172.21.15.44:0/79613408 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562ee0049840 0x562ee004bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.285+0000 7f5ee8ff9640 1 -- 172.21.15.44:0/2457945306 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f5edc0cfd50 con 0x7f5eec101360 2024-04-21T23:12:42.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.286+0000 7f5eeaffd640 1 -- 172.21.15.44:0/79613408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562ee003c500 msgr2=0x562ee0049300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:42.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.286+0000 7f5eeaffd640 1 --2- 172.21.15.44:0/79613408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562ee003c500 0x562ee0049300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.286+0000 7f5eeaffd640 1 -- 172.21.15.44:0/79613408 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562ee0049840 msgr2=0x562ee004bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:42.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.286+0000 7f5eeaffd640 1 --2- 172.21.15.44:0/79613408 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562ee0049840 0x562ee004bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.286+0000 7f5eeaffd640 1 -- 172.21.15.44:0/79613408 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562edfd7e7d0 con 0x562ee003bbe0 2024-04-21T23:12:42.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.286+0000 7f5eeb7fe640 1 --2- 172.21.15.44:0/2457945306 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5ed80a2d00 0x7f5ed80a51c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4005fd0 tx=0x7f5ee4009f10 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:42.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.286+0000 7f5eeaffd640 1 --2- 172.21.15.44:0/79613408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 0x562ee003bfc0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f5edc098840 tx=0x7f5edc098240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b05a80df4a27de3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:42.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.286+0000 7f5ee17fa640 1 -- 172.21.15.44:0/79613408 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5edc0ae9b0 con 0x562ee003bbe0 2024-04-21T23:12:42.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.286+0000 7f5ee17fa640 1 -- 172.21.15.44:0/79613408 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5edc0b0110 con 0x562ee003bbe0 2024-04-21T23:12:42.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.286+0000 7f5ee17fa640 1 -- 172.21.15.44:0/79613408 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5edc003b10 con 0x562ee003bbe0 2024-04-21T23:12:42.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.287+0000 7f5ef438d740 1 -- 172.21.15.44:0/79613408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 msgr2=0x562ee003bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:42.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.287+0000 7f5ef438d740 1 --2- 172.21.15.44:0/79613408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 0x562ee003bfc0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f5edc098840 tx=0x7f5edc098240 comp rx=0 tx=0).stop 2024-04-21T23:12:42.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.287+0000 7f5ef438d740 1 -- 172.21.15.44:0/79613408 shutdown_connections 2024-04-21T23:12:42.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.287+0000 7f5ef438d740 1 --2- 172.21.15.44:0/79613408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562ee003c500 0x562ee0049300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.287+0000 7f5ef438d740 1 --2- 172.21.15.44:0/79613408 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562ee0049840 0x562ee004bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.287+0000 7f5ef438d740 1 --2- 172.21.15.44:0/79613408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 0x562ee003bfc0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.287+0000 7f5ef438d740 1 -- 172.21.15.44:0/79613408 >> 172.21.15.44:0/79613408 conn(0x562ee0036660 msgr2=0x562ee0036a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:42.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.287+0000 7f5ef438d740 1 -- 172.21.15.44:0/79613408 shutdown_connections 2024-04-21T23:12:42.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.287+0000 7f5ef438d740 1 -- 172.21.15.44:0/79613408 wait complete. 2024-04-21T23:12:42.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.287+0000 7f5ef438d740 1 -- start start 2024-04-21T23:12:42.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.287+0000 7f5ef438d740 1 Processor -- start 2024-04-21T23:12:42.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5ef438d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 0x562ee0046910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5ef438d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562ee003c500 0x562ee0046e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5ef438d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562ee0049840 0x562ee0047390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5ef438d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562ee004c620 con 0x562ee003bbe0 2024-04-21T23:12:42.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5ef438d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x562edff5be00 con 0x562ee0049840 2024-04-21T23:12:42.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5ef438d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562ee004c7a0 con 0x562ee003c500 2024-04-21T23:12:42.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5ef438d740 10 client.0 _ll_register_callbacks cb 0x562edff96eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:42.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5eeaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 0x562ee0046910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5eeaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 0x562ee0046910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40010/0 (socket says 172.21.15.44:40010) 2024-04-21T23:12:42.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5eeaffd640 1 -- 172.21.15.44:0/2249015215 learned_addr learned my addr 172.21.15.44:0/2249015215 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:42.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5ee3fff640 1 --2- 172.21.15.44:0/2249015215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562ee003c500 0x562ee0046e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5eeb7fe640 1 --2- 172.21.15.44:0/2249015215 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562ee0049840 0x562ee0047390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5eeaffd640 1 -- 172.21.15.44:0/2249015215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562ee003c500 msgr2=0x562ee0046e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:42.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5eeaffd640 1 --2- 172.21.15.44:0/2249015215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562ee003c500 0x562ee0046e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5eeaffd640 1 -- 172.21.15.44:0/2249015215 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562ee0049840 msgr2=0x562ee0047390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:42.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5eeaffd640 1 --2- 172.21.15.44:0/2249015215 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562ee0049840 0x562ee0047390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5eeaffd640 1 -- 172.21.15.44:0/2249015215 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562edfcb43f0 con 0x562ee003bbe0 2024-04-21T23:12:42.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5eeaffd640 1 --2- 172.21.15.44:0/2249015215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 0x562ee0046910 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f5edc0a45b0 tx=0x7f5edc0a4b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ece77a06ee606bc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:42.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5ed27fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:12:42.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5ed27fc640 1 -- 172.21.15.44:0/2249015215 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5edc09c570 con 0x562ee003bbe0 2024-04-21T23:12:42.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5ed27fc640 1 -- 172.21.15.44:0/2249015215 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f5edc003b10 con 0x562ee003bbe0 2024-04-21T23:12:42.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.288+0000 7f5ef438d740 1 -- 172.21.15.44:0/2249015215 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562edffadaa0 con 0x562ee003bbe0 2024-04-21T23:12:42.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.289+0000 7f5ed27fc640 1 -- 172.21.15.44:0/2249015215 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5edc0a0220 con 0x562ee003bbe0 2024-04-21T23:12:42.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.289+0000 7f5ef438d740 10 client.16545 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:42.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.289+0000 7f5ef438d740 1 -- 172.21.15.44:0/2249015215 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562edfc740f0 con 0x562ee003bbe0 2024-04-21T23:12:42.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.289+0000 7f5ed27fc640 1 -- 172.21.15.44:0/2249015215 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x562edffadaa0 con 0x562ee003bbe0 2024-04-21T23:12:42.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.289+0000 7f5ed27fc640 1 -- 172.21.15.44:0/2249015215 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f5edc0b2bd0 con 0x562ee003bbe0 2024-04-21T23:12:42.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.289+0000 7f5ef438d740 10 client.16545 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:42.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.289+0000 7f5ef438d740 10 client.16545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:42.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.289+0000 7f5ef438d740 20 client.16545 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:42.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.289+0000 7f5ef438d740 1 --2- 172.21.15.44:0/2249015215 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562ee0059b40 0x562ee005bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.289+0000 7f5ef438d740 4 client.16545 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:12:42.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.289+0000 7f5ef438d740 1 -- 172.21.15.44:0/2249015215 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562edffa7df0 con 0x562ee0059b40 2024-04-21T23:12:42.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.289+0000 7f5ee3fff640 1 --2- 172.21.15.44:0/2249015215 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562ee0059b40 0x562ee005bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.290+0000 7f5ed27fc640 10 client.16545 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:42.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.290+0000 7f5ee3fff640 1 --2- 172.21.15.44:0/2249015215 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562ee0059b40 0x562ee005bf30 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77ccd394717be98e server_cookie=e966bfd80be2bdfc in_seq=0 out_seq=0 2024-04-21T23:12:42.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.295+0000 7f5ed27fc640 1 -- 172.21.15.44:0/2249015215 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562edffa7df0 con 0x562ee0059b40 2024-04-21T23:12:42.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.295+0000 7f5ed27fc640 10 client.16545 handle_command_reply: tid=0 2024-04-21T23:12:42.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.295+0000 7f5ef438d740 1 client.16545 shutdown 2024-04-21T23:12:42.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.295+0000 7f5ef438d740 1 -- 172.21.15.44:0/2249015215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 msgr2=0x562ee0046910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:42.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.295+0000 7f5ef438d740 1 --2- 172.21.15.44:0/2249015215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 0x562ee0046910 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f5edc0a45b0 tx=0x7f5edc0a4b50 comp rx=0 tx=0).stop 2024-04-21T23:12:42.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.295+0000 7f5ef438d740 1 -- 172.21.15.44:0/2249015215 shutdown_connections 2024-04-21T23:12:42.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.295+0000 7f5ef438d740 1 --2- 172.21.15.44:0/2249015215 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562ee0059b40 0x562ee005bf30 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.295+0000 7f5ef438d740 1 --2- 172.21.15.44:0/2249015215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562ee003c500 0x562ee0046e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.295+0000 7f5ed27fc640 0 client.16545 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:42.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.295+0000 7f5ef438d740 1 --2- 172.21.15.44:0/2249015215 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562ee0049840 0x562ee0047390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.295+0000 7f5ef438d740 1 --2- 172.21.15.44:0/2249015215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 0x562ee0046910 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.295+0000 7f5ef438d740 1 -- 172.21.15.44:0/2249015215 >> 172.21.15.44:0/2249015215 conn(0x562ee0036660 msgr2=0x562ee00383b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:42.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.295+0000 7f5ef438d740 1 -- 172.21.15.44:0/2249015215 shutdown_connections 2024-04-21T23:12:42.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.295+0000 7f5ef438d740 1 -- 172.21.15.44:0/2249015215 wait complete. 2024-04-21T23:12:42.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.295+0000 7f5ef438d740 20 client.16545 trim_cache size 0 max 16384 2024-04-21T23:12:42.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.313+0000 7f5ef438d740 1 Processor -- start 2024-04-21T23:12:42.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.313+0000 7f5ef438d740 1 -- start start 2024-04-21T23:12:42.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.314+0000 7f5ef438d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 0x562ee0049040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.314+0000 7f5ef438d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562ee003c500 0x562ee0055550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.314+0000 7f5ef438d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562ee0049840 0x562ee0036a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.314+0000 7f5ef438d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec1aa950 con 0x562ee003bbe0 2024-04-21T23:12:42.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.314+0000 7f5ef438d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec1ad1d0 con 0x562ee0049840 2024-04-21T23:12:42.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.314+0000 7f5ef438d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec1aafc0 con 0x562ee003c500 2024-04-21T23:12:42.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.314+0000 7f5eeaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 0x562ee0049040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.314+0000 7f5eeaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 0x562ee0049040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40018/0 (socket says 172.21.15.44:40018) 2024-04-21T23:12:42.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.314+0000 7f5eeaffd640 1 -- 172.21.15.44:0/1755626076 learned_addr learned my addr 172.21.15.44:0/1755626076 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:42.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.314+0000 7f5eeb7fe640 1 --2- 172.21.15.44:0/1755626076 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562ee0049840 0x562ee0036a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.314+0000 7f5ee3fff640 1 --2- 172.21.15.44:0/1755626076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562ee003c500 0x562ee0055550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.314+0000 7f5eeaffd640 1 -- 172.21.15.44:0/1755626076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562ee003c500 msgr2=0x562ee0055550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:42.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.314+0000 7f5eeaffd640 1 --2- 172.21.15.44:0/1755626076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562ee003c500 0x562ee0055550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.314+0000 7f5eeaffd640 1 -- 172.21.15.44:0/1755626076 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562ee0049840 msgr2=0x562ee0036a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:42.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.314+0000 7f5eeaffd640 1 --2- 172.21.15.44:0/1755626076 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562ee0049840 0x562ee0036a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.314+0000 7f5eeaffd640 1 -- 172.21.15.44:0/1755626076 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ee005c2a0 con 0x562ee003bbe0 2024-04-21T23:12:42.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.314+0000 7f5eeaffd640 1 --2- 172.21.15.44:0/1755626076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 0x562ee0049040 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f5edc09be00 tx=0x7f5edc098240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=503084ed020d0c5c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:42.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.314+0000 7f5ee0ff9640 1 -- 172.21.15.44:0/1755626076 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5edc09c570 con 0x562ee003bbe0 2024-04-21T23:12:42.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.315+0000 7f5ee0ff9640 1 -- 172.21.15.44:0/1755626076 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5edc09be70 con 0x562ee003bbe0 2024-04-21T23:12:42.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.315+0000 7f5ee0ff9640 1 -- 172.21.15.44:0/1755626076 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5edc097cd0 con 0x562ee003bbe0 2024-04-21T23:12:42.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.315+0000 7f5ef438d740 1 -- 172.21.15.44:0/1755626076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 msgr2=0x562ee0049040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:42.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.315+0000 7f5ef438d740 1 --2- 172.21.15.44:0/1755626076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 0x562ee0049040 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f5edc09be00 tx=0x7f5edc098240 comp rx=0 tx=0).stop 2024-04-21T23:12:42.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.315+0000 7f5ef438d740 1 -- 172.21.15.44:0/1755626076 shutdown_connections 2024-04-21T23:12:42.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.315+0000 7f5ef438d740 1 --2- 172.21.15.44:0/1755626076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562ee003c500 0x562ee0055550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.315+0000 7f5ef438d740 1 --2- 172.21.15.44:0/1755626076 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562ee0049840 0x562ee0036a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.315+0000 7f5ef438d740 1 --2- 172.21.15.44:0/1755626076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003bbe0 0x562ee0049040 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.315+0000 7f5ef438d740 1 -- 172.21.15.44:0/1755626076 >> 172.21.15.44:0/1755626076 conn(0x562ee0036660 msgr2=0x562ee0038340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:42.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.315+0000 7f5ef438d740 1 -- 172.21.15.44:0/1755626076 shutdown_connections 2024-04-21T23:12:42.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.315+0000 7f5ef438d740 1 -- 172.21.15.44:0/1755626076 wait complete. 2024-04-21T23:12:42.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5ef438d740 1 -- start start 2024-04-21T23:12:42.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5ef438d740 1 Processor -- start 2024-04-21T23:12:42.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5ef438d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562ee003bbe0 0x562ee0051ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5ef438d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003c500 0x562ee004d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5ef438d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562ee0049840 0x562ee004d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5ef438d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec1ab690 con 0x562ee003c500 2024-04-21T23:12:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5ef438d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec1acb00 con 0x562ee0049840 2024-04-21T23:12:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5ef438d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec1abd60 con 0x562ee003bbe0 2024-04-21T23:12:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5ef438d740 10 client.0 _ll_register_callbacks cb 0x7f5edc0aef50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5ee3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003c500 0x562ee004d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5ee3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003c500 0x562ee004d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40030/0 (socket says 172.21.15.44:40030) 2024-04-21T23:12:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5eeb7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562ee0049840 0x562ee004d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5ee3fff640 1 -- 172.21.15.44:0/2644641972 learned_addr learned my addr 172.21.15.44:0/2644641972 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5eeaffd640 1 --2- 172.21.15.44:0/2644641972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562ee003bbe0 0x562ee0051ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5ee3fff640 1 -- 172.21.15.44:0/2644641972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562ee003bbe0 msgr2=0x562ee0051ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5ee3fff640 1 --2- 172.21.15.44:0/2644641972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562ee003bbe0 0x562ee0051ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5ee3fff640 1 -- 172.21.15.44:0/2644641972 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562ee0049840 msgr2=0x562ee004d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5ee3fff640 1 --2- 172.21.15.44:0/2644641972 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562ee0049840 0x562ee004d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5ee3fff640 1 -- 172.21.15.44:0/2644641972 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ee004dfa0 con 0x562ee003c500 2024-04-21T23:12:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5ee3fff640 1 --2- 172.21.15.44:0/2644641972 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003c500 0x562ee004d210 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f5ed402cfd0 tx=0x7f5ed4004530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992a52bfee969427 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.316+0000 7f5ed27fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:12:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.317+0000 7f5ed27fc640 1 -- 172.21.15.44:0/2644641972 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed402cc90 con 0x562ee003c500 2024-04-21T23:12:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.317+0000 7f5ed27fc640 1 -- 172.21.15.44:0/2644641972 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f5ed404a410 con 0x562ee003c500 2024-04-21T23:12:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.317+0000 7f5ef438d740 1 -- 172.21.15.44:0/2644641972 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5edc09be70 con 0x562ee003c500 2024-04-21T23:12:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.317+0000 7f5ed27fc640 1 -- 172.21.15.44:0/2644641972 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5ed405b050 con 0x562ee003c500 2024-04-21T23:12:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.317+0000 7f5ef438d740 10 client.16551 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.317+0000 7f5ef438d740 1 -- 172.21.15.44:0/2644641972 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562ee00bfe40 con 0x562ee003c500 2024-04-21T23:12:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.317+0000 7f5ed27fc640 1 -- 172.21.15.44:0/2644641972 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5edc09be70 con 0x562ee003c500 2024-04-21T23:12:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.317+0000 7f5ed27fc640 1 -- 172.21.15.44:0/2644641972 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f5ed4005080 con 0x562ee003c500 2024-04-21T23:12:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.317+0000 7f5ef438d740 10 client.16551 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.317+0000 7f5ef438d740 10 client.16551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.317+0000 7f5ef438d740 20 client.16551 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:42.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.317+0000 7f5ef438d740 1 --2- 172.21.15.44:0/2644641972 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562ee00c0590 0x562ee00c29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:42.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.317+0000 7f5ef438d740 4 client.16551 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:12:42.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.317+0000 7f5ef438d740 1 -- 172.21.15.44:0/2644641972 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562edff255a0 con 0x562ee00c0590 2024-04-21T23:12:42.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.317+0000 7f5eeaffd640 1 --2- 172.21.15.44:0/2644641972 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562ee00c0590 0x562ee00c29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:42.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.318+0000 7f5eeaffd640 1 --2- 172.21.15.44:0/2644641972 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562ee00c0590 0x562ee00c29b0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a444921e929d603 server_cookie=88b3f0b898345699 in_seq=0 out_seq=0 2024-04-21T23:12:42.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.318+0000 7f5ed27fc640 10 client.16551 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:42.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ed27fc640 1 -- 172.21.15.44:0/2644641972 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562edff255a0 con 0x562ee00c0590 2024-04-21T23:12:42.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ed27fc640 10 client.16551 handle_command_reply: tid=0 2024-04-21T23:12:42.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef438d740 1 client.16551 shutdown 2024-04-21T23:12:42.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef438d740 1 -- 172.21.15.44:0/2644641972 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003c500 msgr2=0x562ee004d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:42.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef438d740 1 --2- 172.21.15.44:0/2644641972 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003c500 0x562ee004d210 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f5ed402cfd0 tx=0x7f5ed4004530 comp rx=0 tx=0).stop 2024-04-21T23:12:42.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef438d740 1 -- 172.21.15.44:0/2644641972 shutdown_connections 2024-04-21T23:12:42.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef438d740 1 --2- 172.21.15.44:0/2644641972 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562ee00c0590 0x562ee00c29b0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef438d740 1 --2- 172.21.15.44:0/2644641972 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562ee0049840 0x562ee004d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef438d740 1 --2- 172.21.15.44:0/2644641972 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562ee003c500 0x562ee004d210 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ed27fc640 0 client.16551 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:42.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef438d740 1 --2- 172.21.15.44:0/2644641972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562ee003bbe0 0x562ee0051ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef438d740 1 -- 172.21.15.44:0/2644641972 >> 172.21.15.44:0/2644641972 conn(0x562ee0036660 msgr2=0x562ee0037ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:42.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef438d740 1 -- 172.21.15.44:0/2644641972 shutdown_connections 2024-04-21T23:12:42.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef438d740 1 -- 172.21.15.44:0/2644641972 wait complete. 2024-04-21T23:12:42.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef438d740 20 client.16551 trim_cache size 0 max 16384 2024-04-21T23:12:42.324 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:12:42.324 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:12:42.324 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-21T23:12:42.324 INFO:teuthology.orchestra.run.smithi044.stdout: "35aa3258-4ae0-4998-ac83-4497b35a5988": { 2024-04-21T23:12:42.325 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-21T23:12:42.325 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "35aa3258-4ae0-4998-ac83-4497b35a5988", 2024-04-21T23:12:42.325 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-04-21T23:12:42.325 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:12:42.325 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:12:42.325 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:12:42.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef16c9640 1 -- 172.21.15.44:0/2457945306 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5ed80a2d00 msgr2=0x7f5ed80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:42.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef16c9640 1 --2- 172.21.15.44:0/2457945306 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5ed80a2d00 0x7f5ed80a51c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4005fd0 tx=0x7f5ee4009f10 comp rx=0 tx=0).stop 2024-04-21T23:12:42.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef16c9640 1 -- 172.21.15.44:0/2457945306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eec101360 msgr2=0x7f5eec198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:42.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef16c9640 1 --2- 172.21.15.44:0/2457945306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eec101360 0x7f5eec198000 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5edc00e9c0 tx=0x7f5edc00ee80 comp rx=0 tx=0).stop 2024-04-21T23:12:42.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef16c9640 1 -- 172.21.15.44:0/2457945306 shutdown_connections 2024-04-21T23:12:42.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef16c9640 1 --2- 172.21.15.44:0/2457945306 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5ed80a2d00 0x7f5ed80a51c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef16c9640 1 --2- 172.21.15.44:0/2457945306 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5eec19c8c0 0x7f5eec19cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef16c9640 1 --2- 172.21.15.44:0/2457945306 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5eec105990 0x7f5eec198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef16c9640 1 --2- 172.21.15.44:0/2457945306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eec101360 0x7f5eec198000 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:42.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.319+0000 7f5ef16c9640 1 -- 172.21.15.44:0/2457945306 >> 172.21.15.44:0/2457945306 conn(0x7f5eec075930 msgr2=0x7f5eec104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:42.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.320+0000 7f5ef16c9640 1 -- 172.21.15.44:0/2457945306 shutdown_connections 2024-04-21T23:12:42.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:12:42.320+0000 7f5ef16c9640 1 -- 172.21.15.44:0/2457945306 wait complete. 2024-04-21T23:12:42.329 INFO:tasks.cephfs.filesystem:scrub status for tag:35aa3258-4ae0-4998-ac83-4497b35a5988 - {'path': '/', 'tag': '35aa3258-4ae0-4998-ac83-4497b35a5988', 'options': 'recursive,force'} 2024-04-21T23:12:42.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 392311 13.34 MB/sec execute 32 sec latency 19.302 ms 2024-04-21T23:12:43.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:42 smithi044 ceph-mon[27934]: scrub queued for path: / 2024-04-21T23:12:43.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:42 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:12:43.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:42 smithi044 ceph-mon[27934]: scrub summary: active paths [/] 2024-04-21T23:12:43.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:42 smithi055 ceph-mon[31117]: scrub queued for path: / 2024-04-21T23:12:43.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:42 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:12:43.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:42 smithi055 ceph-mon[31117]: scrub summary: active paths [/] 2024-04-21T23:12:43.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:42 smithi069 ceph-mon[31816]: scrub queued for path: / 2024-04-21T23:12:43.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:42 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:12:43.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:42 smithi069 ceph-mon[31816]: scrub summary: active paths [/] 2024-04-21T23:12:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.540+0000 7f8dac147640 1 -- 172.21.15.44:0/2392522744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8da4100640 msgr2=0x7f8da4100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.540+0000 7f8dac147640 1 --2- 172.21.15.44:0/2392522744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8da4100640 0x7f8da4100ac0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8d94009890 tx=0x7f8d9402bb70 comp rx=0 tx=0).stop 2024-04-21T23:12:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.540+0000 7f8dac147640 1 -- 172.21.15.44:0/2392522744 shutdown_connections 2024-04-21T23:12:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.540+0000 7f8dac147640 1 --2- 172.21.15.44:0/2392522744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8da4100640 0x7f8da4100ac0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.540+0000 7f8dac147640 1 --2- 172.21.15.44:0/2392522744 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8da4101000 0x7f8da410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.540+0000 7f8dac147640 1 --2- 172.21.15.44:0/2392522744 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8da4106650 0x7f8da4106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.540+0000 7f8dac147640 1 -- 172.21.15.44:0/2392522744 >> 172.21.15.44:0/2392522744 conn(0x7f8da40777c0 msgr2=0x7f8da4077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.540+0000 7f8dac147640 1 -- 172.21.15.44:0/2392522744 shutdown_connections 2024-04-21T23:12:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.540+0000 7f8dac147640 1 -- 172.21.15.44:0/2392522744 wait complete. 2024-04-21T23:12:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.540+0000 7f8dac147640 1 Processor -- start 2024-04-21T23:12:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.540+0000 7f8dac147640 1 -- start start 2024-04-21T23:12:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.541+0000 7f8dac147640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8da4100640 0x7f8da4198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.541+0000 7f8dac147640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8da4101000 0x7f8da41985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.541+0000 7f8dac147640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8da4106650 0x7f8da419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.541+0000 7f8dac147640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8da4111b60 con 0x7f8da4106650 2024-04-21T23:12:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.541+0000 7f8dac147640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8da41119e0 con 0x7f8da4101000 2024-04-21T23:12:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.541+0000 7f8dac147640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8da4111ce0 con 0x7f8da4100640 2024-04-21T23:12:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.541+0000 7f8daa6e4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8da4106650 0x7f8da419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.541+0000 7f8da9ee3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8da4100640 0x7f8da4198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.541+0000 7f8daa6e4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8da4106650 0x7f8da419c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40058/0 (socket says 172.21.15.44:40058) 2024-04-21T23:12:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.541+0000 7f8daa6e4640 1 -- 172.21.15.44:0/801990040 learned_addr learned my addr 172.21.15.44:0/801990040 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.541+0000 7f8da96e2640 1 --2- 172.21.15.44:0/801990040 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8da4101000 0x7f8da41985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.541+0000 7f8daa6e4640 1 -- 172.21.15.44:0/801990040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8da4100640 msgr2=0x7f8da4198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.541+0000 7f8daa6e4640 1 --2- 172.21.15.44:0/801990040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8da4100640 0x7f8da4198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.541+0000 7f8daa6e4640 1 -- 172.21.15.44:0/801990040 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8da4101000 msgr2=0x7f8da41985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.541+0000 7f8daa6e4640 1 --2- 172.21.15.44:0/801990040 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8da4101000 0x7f8da41985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.541+0000 7f8daa6e4640 1 -- 172.21.15.44:0/801990040 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da419d050 con 0x7f8da4106650 2024-04-21T23:12:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.541+0000 7f8daa6e4640 1 --2- 172.21.15.44:0/801990040 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8da4106650 0x7f8da419c920 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c00efc0 tx=0x7f8d9c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.541+0000 7f8d9affd640 1 -- 172.21.15.44:0/801990040 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c003970 con 0x7f8da4106650 2024-04-21T23:12:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.541+0000 7f8dac147640 1 -- 172.21.15.44:0/801990040 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8da419d340 con 0x7f8da4106650 2024-04-21T23:12:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.542+0000 7f8d9affd640 1 -- 172.21.15.44:0/801990040 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d9c004020 con 0x7f8da4106650 2024-04-21T23:12:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.542+0000 7f8d9affd640 1 -- 172.21.15.44:0/801990040 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c00b9f0 con 0x7f8da4106650 2024-04-21T23:12:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.542+0000 7f8dac147640 1 -- 172.21.15.44:0/801990040 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8da41a4bd0 con 0x7f8da4106650 2024-04-21T23:12:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.544+0000 7f8daee05740 1 Processor -- start 2024-04-21T23:12:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.544+0000 7f8d9affd640 1 -- 172.21.15.44:0/801990040 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8d9c01d020 con 0x7f8da4106650 2024-04-21T23:12:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.544+0000 7f8daee05740 1 -- start start 2024-04-21T23:12:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.545+0000 7f8daee05740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421dab20 0x5651421daf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.545+0000 7f8daee05740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421db440 0x5651421e3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.545+0000 7f8daee05740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651421e4210 0x5651421e6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.545+0000 7f8daee05740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56514213eb70 con 0x5651421db440 2024-04-21T23:12:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.545+0000 7f8daee05740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5651420e7470 con 0x5651421dab20 2024-04-21T23:12:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.545+0000 7f8daee05740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56514213ab30 con 0x5651421e4210 2024-04-21T23:12:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.545+0000 7f8da9ee3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421db440 0x5651421e3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.545+0000 7f8da9ee3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421db440 0x5651421e3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40072/0 (socket says 172.21.15.44:40072) 2024-04-21T23:12:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.545+0000 7f8da9ee3640 1 -- 172.21.15.44:0/265242616 learned_addr learned my addr 172.21.15.44:0/265242616 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.545+0000 7f8da96e2640 1 --2- 172.21.15.44:0/265242616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651421e4210 0x5651421e6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.545+0000 7f8da96e2640 1 --2- 172.21.15.44:0/265242616 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421dab20 0x5651421daf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.545+0000 7f8d9affd640 1 --2- 172.21.15.44:0/801990040 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f8d900a2cf0 0x7f8d900a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.545+0000 7f8da9ee3640 1 -- 172.21.15.44:0/265242616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651421e4210 msgr2=0x5651421e6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.545+0000 7f8da9ee3640 1 --2- 172.21.15.44:0/265242616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651421e4210 0x5651421e6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.545+0000 7f8da9ee3640 1 -- 172.21.15.44:0/265242616 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421dab20 msgr2=0x5651421daf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.545+0000 7f8da9ee3640 1 --2- 172.21.15.44:0/265242616 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421dab20 0x5651421daf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.545+0000 7f8da9ee3640 1 -- 172.21.15.44:0/265242616 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565141e53210 con 0x5651421db440 2024-04-21T23:12:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.545+0000 7f8daa6e4640 1 --2- 172.21.15.44:0/801990040 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f8d900a2cf0 0x7f8d900a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.545+0000 7f8d9affd640 1 -- 172.21.15.44:0/801990040 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f8d9c0cf660 con 0x7f8da4106650 2024-04-21T23:12:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.546+0000 7f8da9ee3640 1 --2- 172.21.15.44:0/265242616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421db440 0x5651421e3cd0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009ca0 tx=0x7f8da0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e54c95be78d0bd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.546+0000 7f8d8ffff640 1 -- 172.21.15.44:0/265242616 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da001e070 con 0x5651421db440 2024-04-21T23:12:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.546+0000 7f8daa6e4640 1 --2- 172.21.15.44:0/801990040 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f8d900a2cf0 0x7f8d900a51b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c097c40 tx=0x7f8d9c01aa20 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.546+0000 7f8d8ffff640 1 -- 172.21.15.44:0/265242616 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8da0009e50 con 0x5651421db440 2024-04-21T23:12:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.546+0000 7f8d8ffff640 1 -- 172.21.15.44:0/265242616 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0018850 con 0x5651421db440 2024-04-21T23:12:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.547+0000 7f8daee05740 1 -- 172.21.15.44:0/265242616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421db440 msgr2=0x5651421e3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.547+0000 7f8daee05740 1 --2- 172.21.15.44:0/265242616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421db440 0x5651421e3cd0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009ca0 tx=0x7f8da0011810 comp rx=0 tx=0).stop 2024-04-21T23:12:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.547+0000 7f8daee05740 1 -- 172.21.15.44:0/265242616 shutdown_connections 2024-04-21T23:12:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.547+0000 7f8daee05740 1 --2- 172.21.15.44:0/265242616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651421e4210 0x5651421e6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.547+0000 7f8daee05740 1 --2- 172.21.15.44:0/265242616 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421db440 0x5651421e3cd0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.547+0000 7f8daee05740 1 --2- 172.21.15.44:0/265242616 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421dab20 0x5651421daf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.547+0000 7f8daee05740 1 -- 172.21.15.44:0/265242616 >> 172.21.15.44:0/265242616 conn(0x5651421d5440 msgr2=0x5651421d5820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.547+0000 7f8daee05740 1 -- 172.21.15.44:0/265242616 shutdown_connections 2024-04-21T23:12:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.547+0000 7f8daee05740 1 -- 172.21.15.44:0/265242616 wait complete. 2024-04-21T23:12:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.547+0000 7f8daee05740 1 -- start start 2024-04-21T23:12:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.547+0000 7f8daee05740 1 Processor -- start 2024-04-21T23:12:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8daee05740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421dab20 0x5651421e1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8daee05740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421db440 0x5651421e1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8daee05740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651421e4210 0x5651421e1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8daee05740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5651421e6e40 con 0x5651421dab20 2024-04-21T23:12:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8daee05740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5651420faa10 con 0x5651421db440 2024-04-21T23:12:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8daee05740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5651421e6fc0 con 0x5651421e4210 2024-04-21T23:12:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8daee05740 10 client.0 _ll_register_callbacks cb 0x565142135b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8da96e2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421dab20 0x5651421e1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8da9ee3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421db440 0x5651421e1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8da96e2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421dab20 0x5651421e1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40074/0 (socket says 172.21.15.44:40074) 2024-04-21T23:12:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8da96e2640 1 -- 172.21.15.44:0/1073020887 learned_addr learned my addr 172.21.15.44:0/1073020887 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8da96e2640 1 --2- 172.21.15.44:0/1073020887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651421e4210 0x5651421e1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8da96e2640 1 -- 172.21.15.44:0/1073020887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651421e4210 msgr2=0x5651421e1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8da96e2640 1 --2- 172.21.15.44:0/1073020887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651421e4210 0x5651421e1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8da96e2640 1 -- 172.21.15.44:0/1073020887 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421db440 msgr2=0x5651421e1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8da96e2640 1 --2- 172.21.15.44:0/1073020887 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421db440 0x5651421e1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8da96e2640 1 -- 172.21.15.44:0/1073020887 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565141f1f620 con 0x5651421dab20 2024-04-21T23:12:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8da9ee3640 1 --2- 172.21.15.44:0/1073020887 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421db440 0x5651421e1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8da96e2640 1 --2- 172.21.15.44:0/1073020887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421dab20 0x5651421e1250 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f8d94005e00 tx=0x7f8d94005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72bf511d91d148f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8d8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:12:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8d8d7fa640 1 -- 172.21.15.44:0/1073020887 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d94039040 con 0x5651421dab20 2024-04-21T23:12:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8d8d7fa640 1 -- 172.21.15.44:0/1073020887 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f8d9402c510 con 0x5651421dab20 2024-04-21T23:12:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.548+0000 7f8daee05740 1 -- 172.21.15.44:0/1073020887 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8da0009e50 con 0x5651421dab20 2024-04-21T23:12:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.549+0000 7f8d8d7fa640 1 -- 172.21.15.44:0/1073020887 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8d94056050 con 0x5651421dab20 2024-04-21T23:12:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.549+0000 7f8daee05740 10 client.16563 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.549+0000 7f8daee05740 1 -- 172.21.15.44:0/1073020887 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5651421f30c0 con 0x5651421dab20 2024-04-21T23:12:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.549+0000 7f8d8d7fa640 1 -- 172.21.15.44:0/1073020887 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8da0009e50 con 0x5651421dab20 2024-04-21T23:12:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.549+0000 7f8d8d7fa640 1 -- 172.21.15.44:0/1073020887 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f8d9403f040 con 0x5651421dab20 2024-04-21T23:12:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.549+0000 7f8daee05740 10 client.16563 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.549+0000 7f8daee05740 10 client.16563 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.549+0000 7f8daee05740 20 client.16563 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.549+0000 7f8daee05740 1 --2- 172.21.15.44:0/1073020887 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5651421f4530 0x5651421f6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.549+0000 7f8daee05740 4 client.16563 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:12:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.549+0000 7f8daee05740 1 -- 172.21.15.44:0/1073020887 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56514214c0b0 con 0x5651421f4530 2024-04-21T23:12:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.549+0000 7f8da9ee3640 1 --2- 172.21.15.44:0/1073020887 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5651421f4530 0x5651421f6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.550+0000 7f8da9ee3640 1 --2- 172.21.15.44:0/1073020887 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5651421f4530 0x5651421f6920 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=978a65331fccc0ba server_cookie=3f47dc541086fa7 in_seq=0 out_seq=0 2024-04-21T23:12:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.550+0000 7f8d8d7fa640 10 client.16563 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.550+0000 7f8d8d7fa640 1 -- 172.21.15.44:0/1073020887 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56514214c0b0 con 0x5651421f4530 2024-04-21T23:12:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.550+0000 7f8d8d7fa640 10 client.16563 handle_command_reply: tid=0 2024-04-21T23:12:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.550+0000 7f8daee05740 1 client.16563 shutdown 2024-04-21T23:12:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.550+0000 7f8daee05740 1 -- 172.21.15.44:0/1073020887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421dab20 msgr2=0x5651421e1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.550+0000 7f8daee05740 1 --2- 172.21.15.44:0/1073020887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421dab20 0x5651421e1250 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f8d94005e00 tx=0x7f8d94005f90 comp rx=0 tx=0).stop 2024-04-21T23:12:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.551+0000 7f8daee05740 1 -- 172.21.15.44:0/1073020887 shutdown_connections 2024-04-21T23:12:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.551+0000 7f8daee05740 1 --2- 172.21.15.44:0/1073020887 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5651421f4530 0x5651421f6920 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.551+0000 7f8daee05740 1 --2- 172.21.15.44:0/1073020887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651421e4210 0x5651421e1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.551+0000 7f8daee05740 1 --2- 172.21.15.44:0/1073020887 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421db440 0x5651421e1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.551+0000 7f8d8d7fa640 0 client.16563 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.551+0000 7f8daee05740 1 --2- 172.21.15.44:0/1073020887 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421dab20 0x5651421e1250 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.551+0000 7f8daee05740 1 -- 172.21.15.44:0/1073020887 >> 172.21.15.44:0/1073020887 conn(0x5651421d5440 msgr2=0x5651421d7190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.551+0000 7f8daee05740 1 -- 172.21.15.44:0/1073020887 shutdown_connections 2024-04-21T23:12:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.551+0000 7f8daee05740 1 -- 172.21.15.44:0/1073020887 wait complete. 2024-04-21T23:12:43.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.551+0000 7f8daee05740 20 client.16563 trim_cache size 0 max 16384 2024-04-21T23:12:43.570 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.567+0000 7f8daee05740 1 Processor -- start 2024-04-21T23:12:43.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.567+0000 7f8daee05740 1 -- start start 2024-04-21T23:12:43.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.567+0000 7f8daee05740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421dab20 0x5651421e3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:43.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.567+0000 7f8daee05740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651421db440 0x5651421efd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:43.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.567+0000 7f8daee05740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421e4210 0x5651421fc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:43.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.567+0000 7f8daee05740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8da41acc10 con 0x5651421e4210 2024-04-21T23:12:43.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.567+0000 7f8daee05740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8da41af490 con 0x5651421dab20 2024-04-21T23:12:43.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.567+0000 7f8daee05740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8da41ad280 con 0x5651421db440 2024-04-21T23:12:43.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.567+0000 7f8da96e2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421e4210 0x5651421fc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:43.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.567+0000 7f8da96e2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421e4210 0x5651421fc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40082/0 (socket says 172.21.15.44:40082) 2024-04-21T23:12:43.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.567+0000 7f8da96e2640 1 -- 172.21.15.44:0/3290856989 learned_addr learned my addr 172.21.15.44:0/3290856989 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:43.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.567+0000 7f8da96e2640 1 --2- 172.21.15.44:0/3290856989 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421dab20 0x5651421e3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:43.571 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.568+0000 7f8da9ee3640 1 --2- 172.21.15.44:0/3290856989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651421db440 0x5651421efd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:43.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.568+0000 7f8da96e2640 1 -- 172.21.15.44:0/3290856989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651421db440 msgr2=0x5651421efd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:43.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.568+0000 7f8da96e2640 1 --2- 172.21.15.44:0/3290856989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651421db440 0x5651421efd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.568+0000 7f8da96e2640 1 -- 172.21.15.44:0/3290856989 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421dab20 msgr2=0x5651421e3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:43.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.568+0000 7f8da96e2640 1 --2- 172.21.15.44:0/3290856989 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421dab20 0x5651421e3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.568+0000 7f8da96e2640 1 -- 172.21.15.44:0/3290856989 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651421f6c90 con 0x5651421e4210 2024-04-21T23:12:43.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.568+0000 7f8da96e2640 1 --2- 172.21.15.44:0/3290856989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421e4210 0x5651421fc1f0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f8d94001eb0 tx=0x7f8d9402bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e84a9ba753442d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:43.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.568+0000 7f8d8f7fe640 1 -- 172.21.15.44:0/3290856989 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d94039040 con 0x5651421e4210 2024-04-21T23:12:43.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.568+0000 7f8d8f7fe640 1 -- 172.21.15.44:0/3290856989 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d94005550 con 0x5651421e4210 2024-04-21T23:12:43.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.568+0000 7f8d8f7fe640 1 -- 172.21.15.44:0/3290856989 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d940328a0 con 0x5651421e4210 2024-04-21T23:12:43.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.569+0000 7f8daee05740 1 -- 172.21.15.44:0/3290856989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421e4210 msgr2=0x5651421fc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:43.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.569+0000 7f8daee05740 1 --2- 172.21.15.44:0/3290856989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421e4210 0x5651421fc1f0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f8d94001eb0 tx=0x7f8d9402bb70 comp rx=0 tx=0).stop 2024-04-21T23:12:43.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.569+0000 7f8daee05740 1 -- 172.21.15.44:0/3290856989 shutdown_connections 2024-04-21T23:12:43.572 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.569+0000 7f8daee05740 1 --2- 172.21.15.44:0/3290856989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651421db440 0x5651421efd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.569+0000 7f8daee05740 1 --2- 172.21.15.44:0/3290856989 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421e4210 0x5651421fc1f0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.569+0000 7f8daee05740 1 --2- 172.21.15.44:0/3290856989 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421dab20 0x5651421e3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.569+0000 7f8daee05740 1 -- 172.21.15.44:0/3290856989 >> 172.21.15.44:0/3290856989 conn(0x5651421d5440 msgr2=0x5651421d5820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:43.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.569+0000 7f8daee05740 1 -- 172.21.15.44:0/3290856989 shutdown_connections 2024-04-21T23:12:43.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.569+0000 7f8daee05740 1 -- 172.21.15.44:0/3290856989 wait complete. 2024-04-21T23:12:43.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.569+0000 7f8daee05740 1 -- start start 2024-04-21T23:12:43.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.569+0000 7f8daee05740 1 Processor -- start 2024-04-21T23:12:43.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8daee05740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421dab20 0x5651421ec540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:43.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8daee05740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421db440 0x5651421e78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:43.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8daee05740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651421e4210 0x5651421e7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:43.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8daee05740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8da41ad950 con 0x5651421dab20 2024-04-21T23:12:43.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8daee05740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8da41aedc0 con 0x5651421db440 2024-04-21T23:12:43.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8daee05740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8da41ae020 con 0x5651421e4210 2024-04-21T23:12:43.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8daee05740 10 client.0 _ll_register_callbacks cb 0x7f8d940096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:43.573 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8da96e2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421dab20 0x5651421ec540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:43.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8da96e2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421dab20 0x5651421ec540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:40092/0 (socket says 172.21.15.44:40092) 2024-04-21T23:12:43.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8da96e2640 1 -- 172.21.15.44:0/4064429361 learned_addr learned my addr 172.21.15.44:0/4064429361 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:43.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8da9ee3640 1 --2- 172.21.15.44:0/4064429361 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421db440 0x5651421e78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:43.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8da96e2640 1 --2- 172.21.15.44:0/4064429361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651421e4210 0x5651421e7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:43.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8da96e2640 1 -- 172.21.15.44:0/4064429361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651421e4210 msgr2=0x5651421e7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:43.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8da96e2640 1 --2- 172.21.15.44:0/4064429361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651421e4210 0x5651421e7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8da96e2640 1 -- 172.21.15.44:0/4064429361 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421db440 msgr2=0x5651421e78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:43.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8da96e2640 1 --2- 172.21.15.44:0/4064429361 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421db440 0x5651421e78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8da96e2640 1 -- 172.21.15.44:0/4064429361 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651421e85f0 con 0x5651421dab20 2024-04-21T23:12:43.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8da9ee3640 1 --2- 172.21.15.44:0/4064429361 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421db440 0x5651421e78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:12:43.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8da96e2640 1 --2- 172.21.15.44:0/4064429361 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421dab20 0x5651421ec540 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f8d94034990 tx=0x7f8d94034890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a43a3afb450bd78 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:43.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8d8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:12:43.574 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8d8d7fa640 1 -- 172.21.15.44:0/4064429361 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d94062020 con 0x5651421dab20 2024-04-21T23:12:43.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8d8d7fa640 1 -- 172.21.15.44:0/4064429361 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f8d9402c510 con 0x5651421dab20 2024-04-21T23:12:43.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8daee05740 1 -- 172.21.15.44:0/4064429361 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d94005550 con 0x5651421dab20 2024-04-21T23:12:43.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8d8d7fa640 1 -- 172.21.15.44:0/4064429361 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8d94039070 con 0x5651421dab20 2024-04-21T23:12:43.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8daee05740 10 client.16569 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:43.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.570+0000 7f8daee05740 1 -- 172.21.15.44:0/4064429361 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56514225a9e0 con 0x5651421dab20 2024-04-21T23:12:43.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.571+0000 7f8d8d7fa640 1 -- 172.21.15.44:0/4064429361 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d94005550 con 0x5651421dab20 2024-04-21T23:12:43.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.571+0000 7f8d8d7fa640 1 -- 172.21.15.44:0/4064429361 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f8d9403f070 con 0x5651421dab20 2024-04-21T23:12:43.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.571+0000 7f8daee05740 10 client.16569 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:43.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.571+0000 7f8daee05740 10 client.16569 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:43.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.571+0000 7f8daee05740 20 client.16569 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:43.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.571+0000 7f8daee05740 1 --2- 172.21.15.44:0/4064429361 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56514225b040 0x56514225d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:43.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.571+0000 7f8daee05740 4 client.16569 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:12:43.575 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.571+0000 7f8daee05740 1 -- 172.21.15.44:0/4064429361 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5651421d0900 con 0x56514225b040 2024-04-21T23:12:43.576 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.571+0000 7f8da9ee3640 1 --2- 172.21.15.44:0/4064429361 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56514225b040 0x56514225d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:43.576 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.571+0000 7f8d8d7fa640 10 client.16569 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:43.576 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.571+0000 7f8da9ee3640 1 --2- 172.21.15.44:0/4064429361 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56514225b040 0x56514225d460 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acc729965bdcc76c server_cookie=59ead99dab402481 in_seq=0 out_seq=0 2024-04-21T23:12:43.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 394501 13.35 MB/sec execute 33 sec latency 91.162 ms 2024-04-21T23:12:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.865+0000 7f8d8d7fa640 1 -- 172.21.15.44:0/4064429361 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5651421d0900 con 0x56514225b040 2024-04-21T23:12:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.865+0000 7f8d8d7fa640 10 client.16569 handle_command_reply: tid=0 2024-04-21T23:12:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.865+0000 7f8daee05740 1 client.16569 shutdown 2024-04-21T23:12:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.865+0000 7f8daee05740 1 -- 172.21.15.44:0/4064429361 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421dab20 msgr2=0x5651421ec540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.865+0000 7f8daee05740 1 --2- 172.21.15.44:0/4064429361 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421dab20 0x5651421ec540 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f8d94034990 tx=0x7f8d94034890 comp rx=0 tx=0).stop 2024-04-21T23:12:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.865+0000 7f8daee05740 1 -- 172.21.15.44:0/4064429361 shutdown_connections 2024-04-21T23:12:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.865+0000 7f8daee05740 1 --2- 172.21.15.44:0/4064429361 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56514225b040 0x56514225d460 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.865+0000 7f8daee05740 1 --2- 172.21.15.44:0/4064429361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651421e4210 0x5651421e7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.865+0000 7f8d8d7fa640 0 client.16569 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.865+0000 7f8daee05740 1 --2- 172.21.15.44:0/4064429361 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5651421db440 0x5651421e78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.865+0000 7f8daee05740 1 --2- 172.21.15.44:0/4064429361 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5651421dab20 0x5651421ec540 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.865+0000 7f8daee05740 1 -- 172.21.15.44:0/4064429361 >> 172.21.15.44:0/4064429361 conn(0x5651421d5440 msgr2=0x5651421e1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.866+0000 7f8daee05740 1 -- 172.21.15.44:0/4064429361 shutdown_connections 2024-04-21T23:12:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.866+0000 7f8daee05740 1 -- 172.21.15.44:0/4064429361 wait complete. 2024-04-21T23:12:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.866+0000 7f8daee05740 20 client.16569 trim_cache size 0 max 16384 2024-04-21T23:12:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:12:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:12:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:12:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:12:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.867+0000 7f8dac147640 1 -- 172.21.15.44:0/801990040 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f8d900a2cf0 msgr2=0x7f8d900a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.867+0000 7f8dac147640 1 --2- 172.21.15.44:0/801990040 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f8d900a2cf0 0x7f8d900a51b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c097c40 tx=0x7f8d9c01aa20 comp rx=0 tx=0).stop 2024-04-21T23:12:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.867+0000 7f8dac147640 1 -- 172.21.15.44:0/801990040 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8da4106650 msgr2=0x7f8da419c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.867+0000 7f8dac147640 1 --2- 172.21.15.44:0/801990040 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8da4106650 0x7f8da419c920 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c00efc0 tx=0x7f8d9c00c800 comp rx=0 tx=0).stop 2024-04-21T23:12:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.867+0000 7f8dac147640 1 -- 172.21.15.44:0/801990040 shutdown_connections 2024-04-21T23:12:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.867+0000 7f8dac147640 1 --2- 172.21.15.44:0/801990040 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f8d900a2cf0 0x7f8d900a51b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.867+0000 7f8dac147640 1 --2- 172.21.15.44:0/801990040 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8da4106650 0x7f8da419c920 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.867+0000 7f8dac147640 1 --2- 172.21.15.44:0/801990040 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8da4101000 0x7f8da41985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.867+0000 7f8dac147640 1 --2- 172.21.15.44:0/801990040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8da4100640 0x7f8da4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.867+0000 7f8dac147640 1 -- 172.21.15.44:0/801990040 >> 172.21.15.44:0/801990040 conn(0x7f8da40777c0 msgr2=0x7f8da4102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.867+0000 7f8dac147640 1 -- 172.21.15.44:0/801990040 shutdown_connections 2024-04-21T23:12:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:43.867+0000 7f8dac147640 1 -- 172.21.15.44:0/801990040 wait complete. 2024-04-21T23:12:44.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:43 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:12:44.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:43 smithi044 ceph-mon[27934]: pgmap v121: 129 pgs: 129 active+clean; 54 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.9 MiB/s wr, 528 op/s 2024-04-21T23:12:44.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:43 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:12:44.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:43 smithi069 ceph-mon[31816]: pgmap v121: 129 pgs: 129 active+clean; 54 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.9 MiB/s wr, 528 op/s 2024-04-21T23:12:44.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:43 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:12:44.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:43 smithi055 ceph-mon[31117]: pgmap v121: 129 pgs: 129 active+clean; 54 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.9 MiB/s wr, 528 op/s 2024-04-21T23:12:44.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 397398 13.47 MB/sec execute 34 sec latency 150.235 ms 2024-04-21T23:12:45.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 399656 13.32 MB/sec execute 35 sec latency 96.799 ms 2024-04-21T23:12:46.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:45 smithi044 ceph-mon[27934]: pgmap v122: 129 pgs: 129 active+clean; 54 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 249 op/s 2024-04-21T23:12:46.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:45 smithi055 ceph-mon[31117]: pgmap v122: 129 pgs: 129 active+clean; 54 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 249 op/s 2024-04-21T23:12:46.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:45 smithi069 ceph-mon[31816]: pgmap v122: 129 pgs: 129 active+clean; 54 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 249 op/s 2024-04-21T23:12:46.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 401552 13.30 MB/sec execute 36 sec latency 114.249 ms 2024-04-21T23:12:47.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 404645 13.44 MB/sec execute 37 sec latency 21.714 ms 2024-04-21T23:12:48.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:47 smithi044 ceph-mon[27934]: pgmap v123: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.1 MiB/s wr, 425 op/s 2024-04-21T23:12:48.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:47 smithi044 ceph-mon[27934]: scrub summary: idle 2024-04-21T23:12:48.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:47 smithi055 ceph-mon[31117]: pgmap v123: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.1 MiB/s wr, 425 op/s 2024-04-21T23:12:48.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:47 smithi055 ceph-mon[31117]: scrub summary: idle 2024-04-21T23:12:48.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:47 smithi069 ceph-mon[31816]: pgmap v123: 129 pgs: 129 active+clean; 44 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.1 MiB/s wr, 425 op/s 2024-04-21T23:12:48.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:47 smithi069 ceph-mon[31816]: scrub summary: idle 2024-04-21T23:12:48.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 407573 13.44 MB/sec execute 38 sec latency 20.840 ms 2024-04-21T23:12:49.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 410317 13.43 MB/sec execute 39 sec latency 18.322 ms 2024-04-21T23:12:50.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:49 smithi044 ceph-mon[27934]: pgmap v124: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 461 op/s 2024-04-21T23:12:50.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:49 smithi055 ceph-mon[31117]: pgmap v124: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 461 op/s 2024-04-21T23:12:50.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:49 smithi069 ceph-mon[31816]: pgmap v124: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 461 op/s 2024-04-21T23:12:50.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 412679 13.45 MB/sec execute 40 sec latency 18.856 ms 2024-04-21T23:12:51.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 415266 13.53 MB/sec execute 41 sec latency 49.125 ms 2024-04-21T23:12:52.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:51 smithi044 ceph-mon[27934]: pgmap v125: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 434 op/s 2024-04-21T23:12:52.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:51 smithi069 ceph-mon[31816]: pgmap v125: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 434 op/s 2024-04-21T23:12:52.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:51 smithi055 ceph-mon[31117]: pgmap v125: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 434 op/s 2024-04-21T23:12:52.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 418027 13.51 MB/sec execute 42 sec latency 15.677 ms 2024-04-21T23:12:53.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 420177 13.47 MB/sec execute 43 sec latency 29.281 ms 2024-04-21T23:12:54.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:53 smithi044 ceph-mon[27934]: pgmap v126: 129 pgs: 129 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.0 MiB/s wr, 568 op/s 2024-04-21T23:12:54.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:53 smithi055 ceph-mon[31117]: pgmap v126: 129 pgs: 129 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.0 MiB/s wr, 568 op/s 2024-04-21T23:12:54.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:53 smithi069 ceph-mon[31816]: pgmap v126: 129 pgs: 129 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 5.0 MiB/s wr, 568 op/s 2024-04-21T23:12:54.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 422873 13.49 MB/sec execute 44 sec latency 18.730 ms 2024-04-21T23:12:55.697 INFO:tasks.workunit.client.0.smithi044.stdout: 1 425997 13.60 MB/sec execute 45 sec latency 20.217 ms 2024-04-21T23:12:56.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:55 smithi044 ceph-mon[27934]: pgmap v127: 129 pgs: 129 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 436 op/s 2024-04-21T23:12:56.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:55 smithi069 ceph-mon[31816]: pgmap v127: 129 pgs: 129 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 436 op/s 2024-04-21T23:12:56.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:55 smithi055 ceph-mon[31117]: pgmap v127: 129 pgs: 129 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 436 op/s 2024-04-21T23:12:56.698 INFO:tasks.workunit.client.0.smithi044.stdout: 1 428172 13.48 MB/sec execute 46 sec latency 63.748 ms 2024-04-21T23:12:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.962+0000 7ff5e6519640 1 -- 172.21.15.44:0/3061948245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5e0107740 msgr2=0x7ff5e0107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.962+0000 7ff5e6519640 1 --2- 172.21.15.44:0/3061948245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5e0107740 0x7ff5e0107b20 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0009930 tx=0x7ff5d002bc30 comp rx=0 tx=0).stop 2024-04-21T23:12:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.962+0000 7ff5e6519640 1 -- 172.21.15.44:0/3061948245 shutdown_connections 2024-04-21T23:12:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.962+0000 7ff5e6519640 1 --2- 172.21.15.44:0/3061948245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5e01020f0 0x7ff5e010e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.962+0000 7ff5e6519640 1 --2- 172.21.15.44:0/3061948245 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5e0101730 0x7ff5e0101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.962+0000 7ff5e6519640 1 --2- 172.21.15.44:0/3061948245 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5e0107740 0x7ff5e0107b20 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.962+0000 7ff5e6519640 1 -- 172.21.15.44:0/3061948245 >> 172.21.15.44:0/3061948245 conn(0x7ff5e00fd200 msgr2=0x7ff5e00ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.962+0000 7ff5e6519640 1 -- 172.21.15.44:0/3061948245 shutdown_connections 2024-04-21T23:12:56.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.962+0000 7ff5e6519640 1 -- 172.21.15.44:0/3061948245 wait complete. 2024-04-21T23:12:56.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.962+0000 7ff5e6519640 1 Processor -- start 2024-04-21T23:12:56.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.963+0000 7ff5e6519640 1 -- start start 2024-04-21T23:12:56.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.963+0000 7ff5e6519640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5e0101730 0x7ff5e0198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:56.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.963+0000 7ff5e6519640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5e01020f0 0x7ff5e01985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.963+0000 7ff5e6519640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5e0107740 0x7ff5e019c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.964+0000 7ff5e6519640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e0111b10 con 0x7ff5e0101730 2024-04-21T23:12:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.964+0000 7ff5e6519640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e0111990 con 0x7ff5e01020f0 2024-04-21T23:12:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.964+0000 7ff5e6519640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e0111c90 con 0x7ff5e0107740 2024-04-21T23:12:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.964+0000 7ff5df7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5e01020f0 0x7ff5e01985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.964+0000 7ff5df7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5e01020f0 0x7ff5e01985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:34694/0 (socket says 172.21.15.44:34694) 2024-04-21T23:12:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.964+0000 7ff5df7fe640 1 -- 172.21.15.44:0/1972472940 learned_addr learned my addr 172.21.15.44:0/1972472940 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.965+0000 7ff5e4ab6640 1 --2- 172.21.15.44:0/1972472940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5e0107740 0x7ff5e019c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.965+0000 7ff5df7fe640 1 -- 172.21.15.44:0/1972472940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5e0107740 msgr2=0x7ff5e019c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.965+0000 7ff5df7fe640 1 --2- 172.21.15.44:0/1972472940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5e0107740 0x7ff5e019c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.965+0000 7ff5df7fe640 1 -- 172.21.15.44:0/1972472940 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5e0101730 msgr2=0x7ff5e0198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.965+0000 7ff5df7fe640 1 --2- 172.21.15.44:0/1972472940 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5e0101730 0x7ff5e0198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.965+0000 7ff5df7fe640 1 -- 172.21.15.44:0/1972472940 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e019d060 con 0x7ff5e01020f0 2024-04-21T23:12:56.966 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.966+0000 7ff5df7fe640 1 --2- 172.21.15.44:0/1972472940 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5e01020f0 0x7ff5e01985b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009f20 tx=0x7ff5d400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:56.966 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.967+0000 7ff5dd7fa640 1 -- 172.21.15.44:0/1972472940 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4020070 con 0x7ff5e01020f0 2024-04-21T23:12:56.966 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.967+0000 7ff5dd7fa640 1 -- 172.21.15.44:0/1972472940 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5d400cd80 con 0x7ff5e01020f0 2024-04-21T23:12:56.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.967+0000 7ff5e6519640 1 -- 172.21.15.44:0/1972472940 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5e019d350 con 0x7ff5e01020f0 2024-04-21T23:12:56.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.967+0000 7ff5e6519640 1 -- 172.21.15.44:0/1972472940 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5e0073970 con 0x7ff5e01020f0 2024-04-21T23:12:56.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.967+0000 7ff5dd7fa640 1 -- 172.21.15.44:0/1972472940 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d401bbb0 con 0x7ff5e01020f0 2024-04-21T23:12:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.971+0000 7ff5e91d7740 1 Processor -- start 2024-04-21T23:12:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.972+0000 7ff5e91d7740 1 -- start start 2024-04-21T23:12:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.972+0000 7ff5e91d7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c8102b20 0x55f3c8102f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.972+0000 7ff5e91d7740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c8103440 0x55f3c810bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.972+0000 7ff5e91d7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c810c210 0x55f3c810e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.972+0000 7ff5e91d7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f3c8066b70 con 0x55f3c8102b20 2024-04-21T23:12:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.972+0000 7ff5e91d7740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55f3c800f470 con 0x55f3c8103440 2024-04-21T23:12:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.972+0000 7ff5e91d7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f3c8062b30 con 0x55f3c810c210 2024-04-21T23:12:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.972+0000 7ff5e4ab6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c8102b20 0x55f3c8102f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.972+0000 7ff5e4ab6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c8102b20 0x55f3c8102f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52116/0 (socket says 172.21.15.44:52116) 2024-04-21T23:12:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.972+0000 7ff5e4ab6640 1 -- 172.21.15.44:0/4177891053 learned_addr learned my addr 172.21.15.44:0/4177891053 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.973+0000 7ff5e4ab6640 1 -- 172.21.15.44:0/4177891053 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c810c210 msgr2=0x55f3c810e600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T23:12:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.973+0000 7ff5e4ab6640 1 --2- 172.21.15.44:0/4177891053 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c810c210 0x55f3c810e600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.973+0000 7ff5e4ab6640 1 -- 172.21.15.44:0/4177891053 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c8103440 msgr2=0x55f3c810bcd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T23:12:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.973+0000 7ff5e4ab6640 1 --2- 172.21.15.44:0/4177891053 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c8103440 0x55f3c810bcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.973+0000 7ff5e4ab6640 1 -- 172.21.15.44:0/4177891053 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3c7d7b210 con 0x55f3c8102b20 2024-04-21T23:12:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.973+0000 7ff5e4ab6640 1 --2- 172.21.15.44:0/4177891053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c8102b20 0x55f3c8102f00 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8009f20 tx=0x7ff5d8010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377623e0fc833e3a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.976+0000 7ff5dd7fa640 1 -- 172.21.15.44:0/1972472940 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff5d4023070 con 0x7ff5e01020f0 2024-04-21T23:12:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.977+0000 7ff5dd7fa640 1 --2- 172.21.15.44:0/1972472940 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff5c80a2df0 0x7ff5c80a52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.977+0000 7ff5dd7fa640 1 -- 172.21.15.44:0/1972472940 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7ff5d40d3790 con 0x7ff5e01020f0 2024-04-21T23:12:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.978+0000 7ff5c67fc640 1 -- 172.21.15.44:0/4177891053 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8011700 con 0x55f3c8102b20 2024-04-21T23:12:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.979+0000 7ff5c67fc640 1 -- 172.21.15.44:0/4177891053 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5d8011d80 con 0x55f3c8102b20 2024-04-21T23:12:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.979+0000 7ff5c67fc640 1 -- 172.21.15.44:0/4177891053 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d801a380 con 0x55f3c8102b20 2024-04-21T23:12:56.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.980+0000 7ff5e91d7740 1 -- 172.21.15.44:0/4177891053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c8102b20 msgr2=0x55f3c8102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:56.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.980+0000 7ff5e91d7740 1 --2- 172.21.15.44:0/4177891053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c8102b20 0x55f3c8102f00 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8009f20 tx=0x7ff5d8010860 comp rx=0 tx=0).stop 2024-04-21T23:12:56.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.981+0000 7ff5dffff640 1 --2- 172.21.15.44:0/1972472940 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff5c80a2df0 0x7ff5c80a52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:56.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.981+0000 7ff5e91d7740 1 -- 172.21.15.44:0/4177891053 shutdown_connections 2024-04-21T23:12:56.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.981+0000 7ff5e91d7740 1 --2- 172.21.15.44:0/4177891053 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c810c210 0x55f3c810e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:56.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.981+0000 7ff5e91d7740 1 --2- 172.21.15.44:0/4177891053 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c8103440 0x55f3c810bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:56.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.981+0000 7ff5e91d7740 1 --2- 172.21.15.44:0/4177891053 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c8102b20 0x55f3c8102f00 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:56.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.981+0000 7ff5e91d7740 1 -- 172.21.15.44:0/4177891053 >> 172.21.15.44:0/4177891053 conn(0x55f3c80fd440 msgr2=0x55f3c80fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:56.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.982+0000 7ff5dffff640 1 --2- 172.21.15.44:0/1972472940 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff5c80a2df0 0x7ff5c80a52b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00096c0 tx=0x7ff5d0000bc0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:56.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.982+0000 7ff5e91d7740 1 -- 172.21.15.44:0/4177891053 shutdown_connections 2024-04-21T23:12:56.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.984+0000 7ff5e91d7740 1 -- 172.21.15.44:0/4177891053 wait complete. 2024-04-21T23:12:56.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.984+0000 7ff5e91d7740 1 -- start start 2024-04-21T23:12:56.984 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.985+0000 7ff5e91d7740 1 Processor -- start 2024-04-21T23:12:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.989+0000 7ff5e91d7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c8102b20 0x55f3c8109240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.989+0000 7ff5e91d7740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c8103440 0x55f3c8109780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.989+0000 7ff5e91d7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c810c210 0x55f3c8109cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.989+0000 7ff5e91d7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f3c811fe50 con 0x55f3c810c210 2024-04-21T23:12:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.989+0000 7ff5e91d7740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55f3c8022a10 con 0x55f3c8103440 2024-04-21T23:12:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.989+0000 7ff5e91d7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f3c811ffd0 con 0x55f3c8102b20 2024-04-21T23:12:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.989+0000 7ff5e91d7740 10 client.0 _ll_register_callbacks cb 0x55f3c805db60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.990+0000 7ff5dffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c8102b20 0x55f3c8109240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.991+0000 7ff5dffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c8102b20 0x55f3c8109240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:52044/0 (socket says 172.21.15.44:52044) 2024-04-21T23:12:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.991+0000 7ff5dffff640 1 -- 172.21.15.44:0/280349221 learned_addr learned my addr 172.21.15.44:0/280349221 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.991+0000 7ff5df7fe640 1 --2- 172.21.15.44:0/280349221 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c8103440 0x55f3c8109780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.991+0000 7ff5e4ab6640 1 --2- 172.21.15.44:0/280349221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c810c210 0x55f3c8109cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.992+0000 7ff5df7fe640 1 -- 172.21.15.44:0/280349221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c8102b20 msgr2=0x55f3c8109240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.992+0000 7ff5df7fe640 1 --2- 172.21.15.44:0/280349221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c8102b20 0x55f3c8109240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.992+0000 7ff5df7fe640 1 -- 172.21.15.44:0/280349221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c810c210 msgr2=0x55f3c8109cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.992+0000 7ff5df7fe640 1 --2- 172.21.15.44:0/280349221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c810c210 0x55f3c8109cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.992+0000 7ff5df7fe640 1 -- 172.21.15.44:0/280349221 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3c7e47620 con 0x55f3c8103440 2024-04-21T23:12:56.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.992+0000 7ff5df7fe640 1 --2- 172.21.15.44:0/280349221 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c8103440 0x55f3c8109780 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff5d40d0bf0 tx=0x7ff5d401b440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=712a443fd72636d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:56.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.992+0000 7ff5bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:12:56.992 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.993+0000 7ff5dffff640 1 --2- 172.21.15.44:0/280349221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c8102b20 0x55f3c8109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:12:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.994+0000 7ff5bf7fe640 1 -- 172.21.15.44:0/280349221 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d40d0f70 con 0x55f3c8103440 2024-04-21T23:12:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.994+0000 7ff5bf7fe640 1 -- 172.21.15.44:0/280349221 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7ff5d40a81a0 con 0x55f3c8103440 2024-04-21T23:12:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.994+0000 7ff5e91d7740 1 -- 172.21.15.44:0/280349221 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5d8011d80 con 0x55f3c8103440 2024-04-21T23:12:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.997+0000 7ff5bf7fe640 1 -- 172.21.15.44:0/280349221 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff5d40d3570 con 0x55f3c8103440 2024-04-21T23:12:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.997+0000 7ff5e91d7740 10 client.26150 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.997+0000 7ff5e91d7740 1 -- 172.21.15.44:0/280349221 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f3c8123110 con 0x55f3c8103440 2024-04-21T23:12:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.997+0000 7ff5bf7fe640 1 -- 172.21.15.44:0/280349221 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5d8011d80 con 0x55f3c8103440 2024-04-21T23:12:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.998+0000 7ff5bf7fe640 1 -- 172.21.15.44:0/280349221 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7ff5d409ee10 con 0x55f3c8103440 2024-04-21T23:12:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.998+0000 7ff5e91d7740 10 client.26150 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.998+0000 7ff5e91d7740 10 client.26150 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.998+0000 7ff5e91d7740 20 client.26150 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.998+0000 7ff5e91d7740 1 --2- 172.21.15.44:0/280349221 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f3c81245d0 0x55f3c8126a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.998+0000 7ff5e91d7740 4 client.26150 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:12:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.998+0000 7ff5e91d7740 1 -- 172.21.15.44:0/280349221 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3c80740b0 con 0x55f3c81245d0 2024-04-21T23:12:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:56.999+0000 7ff5dffff640 1 --2- 172.21.15.44:0/280349221 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f3c81245d0 0x55f3c8126a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:57.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.000+0000 7ff5dffff640 1 --2- 172.21.15.44:0/280349221 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f3c81245d0 0x55f3c8126a30 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b2752d358c17f53 server_cookie=579b2304b5c50ee8 in_seq=0 out_seq=0 2024-04-21T23:12:57.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.001+0000 7ff5bf7fe640 10 client.26150 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.004+0000 7ff5bf7fe640 1 -- 172.21.15.44:0/280349221 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f3c80740b0 con 0x55f3c81245d0 2024-04-21T23:12:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.004+0000 7ff5bf7fe640 10 client.26150 handle_command_reply: tid=0 2024-04-21T23:12:57.003 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.004+0000 7ff5e91d7740 1 client.26150 shutdown 2024-04-21T23:12:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.004+0000 7ff5e91d7740 1 -- 172.21.15.44:0/280349221 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c8103440 msgr2=0x55f3c8109780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.004+0000 7ff5e91d7740 1 --2- 172.21.15.44:0/280349221 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c8103440 0x55f3c8109780 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff5d40d0bf0 tx=0x7ff5d401b440 comp rx=0 tx=0).stop 2024-04-21T23:12:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.005+0000 7ff5e91d7740 1 -- 172.21.15.44:0/280349221 shutdown_connections 2024-04-21T23:12:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.005+0000 7ff5e91d7740 1 --2- 172.21.15.44:0/280349221 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f3c81245d0 0x55f3c8126a30 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.005+0000 7ff5e91d7740 1 --2- 172.21.15.44:0/280349221 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c810c210 0x55f3c8109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.005+0000 7ff5e91d7740 1 --2- 172.21.15.44:0/280349221 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c8103440 0x55f3c8109780 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.005+0000 7ff5e91d7740 1 --2- 172.21.15.44:0/280349221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c8102b20 0x55f3c8109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.005+0000 7ff5bf7fe640 0 client.26150 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.005+0000 7ff5e91d7740 1 -- 172.21.15.44:0/280349221 >> 172.21.15.44:0/280349221 conn(0x55f3c80fd440 msgr2=0x55f3c80ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.005+0000 7ff5e91d7740 1 -- 172.21.15.44:0/280349221 shutdown_connections 2024-04-21T23:12:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.005+0000 7ff5e91d7740 1 -- 172.21.15.44:0/280349221 wait complete. 2024-04-21T23:12:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.005+0000 7ff5e91d7740 20 client.26150 trim_cache size 0 max 16384 2024-04-21T23:12:57.034 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.035+0000 7ff5e91d7740 1 Processor -- start 2024-04-21T23:12:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.036+0000 7ff5e91d7740 1 -- start start 2024-04-21T23:12:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.036+0000 7ff5e91d7740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c8102b20 0x55f3c810b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.036+0000 7ff5e91d7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c8103440 0x55f3c8147b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.036+0000 7ff5e91d7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c810c210 0x55f3c8148080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:57.035 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.036+0000 7ff5e91d7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff5cc00df50 con 0x55f3c8103440 2024-04-21T23:12:57.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.036+0000 7ff5e91d7740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff5cc0107d0 con 0x55f3c8102b20 2024-04-21T23:12:57.036 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.036+0000 7ff5e91d7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff5cc00e5c0 con 0x55f3c810c210 2024-04-21T23:12:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.037+0000 7ff5e4ab6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c810c210 0x55f3c8148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.037+0000 7ff5e4ab6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c810c210 0x55f3c8148080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:52060/0 (socket says 172.21.15.44:52060) 2024-04-21T23:12:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.037+0000 7ff5e4ab6640 1 -- 172.21.15.44:0/4090480373 learned_addr learned my addr 172.21.15.44:0/4090480373 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.037+0000 7ff5df7fe640 1 --2- 172.21.15.44:0/4090480373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c8103440 0x55f3c8147b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.037+0000 7ff5dffff640 1 --2- 172.21.15.44:0/4090480373 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c8102b20 0x55f3c810b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:57.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.038+0000 7ff5dffff640 1 -- 172.21.15.44:0/4090480373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c810c210 msgr2=0x55f3c8148080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:57.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.038+0000 7ff5dffff640 1 --2- 172.21.15.44:0/4090480373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c810c210 0x55f3c8148080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.038+0000 7ff5dffff640 1 -- 172.21.15.44:0/4090480373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c8103440 msgr2=0x55f3c8147b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:57.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.038+0000 7ff5dffff640 1 --2- 172.21.15.44:0/4090480373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c8103440 0x55f3c8147b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.038+0000 7ff5dffff640 1 -- 172.21.15.44:0/4090480373 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3c8102470 con 0x55f3c8102b20 2024-04-21T23:12:57.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.038+0000 7ff5dffff640 1 --2- 172.21.15.44:0/4090480373 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c8102b20 0x55f3c810b970 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x55f3c8127200 tx=0x7ff5d003b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7ee172f80b2cc16 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:57.038 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.039+0000 7ff5c5ffb640 1 -- 172.21.15.44:0/4090480373 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0045040 con 0x55f3c8102b20 2024-04-21T23:12:57.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.039+0000 7ff5c5ffb640 1 -- 172.21.15.44:0/4090480373 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5d00501d0 con 0x55f3c8102b20 2024-04-21T23:12:57.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.039+0000 7ff5c5ffb640 1 -- 172.21.15.44:0/4090480373 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0003850 con 0x55f3c8102b20 2024-04-21T23:12:57.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.040+0000 7ff5e91d7740 1 -- 172.21.15.44:0/4090480373 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c8102b20 msgr2=0x55f3c810b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:57.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.040+0000 7ff5e91d7740 1 --2- 172.21.15.44:0/4090480373 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c8102b20 0x55f3c810b970 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x55f3c8127200 tx=0x7ff5d003b000 comp rx=0 tx=0).stop 2024-04-21T23:12:57.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.040+0000 7ff5e91d7740 1 -- 172.21.15.44:0/4090480373 shutdown_connections 2024-04-21T23:12:57.039 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.040+0000 7ff5e91d7740 1 --2- 172.21.15.44:0/4090480373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c810c210 0x55f3c8148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.040+0000 7ff5e91d7740 1 --2- 172.21.15.44:0/4090480373 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c8103440 0x55f3c8147b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.040+0000 7ff5e91d7740 1 --2- 172.21.15.44:0/4090480373 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c8102b20 0x55f3c810b970 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.040+0000 7ff5e91d7740 1 -- 172.21.15.44:0/4090480373 >> 172.21.15.44:0/4090480373 conn(0x55f3c80fd440 msgr2=0x55f3c8107b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.040+0000 7ff5e91d7740 1 -- 172.21.15.44:0/4090480373 shutdown_connections 2024-04-21T23:12:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.040+0000 7ff5e91d7740 1 -- 172.21.15.44:0/4090480373 wait complete. 2024-04-21T23:12:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.041+0000 7ff5e91d7740 1 -- start start 2024-04-21T23:12:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.041+0000 7ff5e91d7740 1 Processor -- start 2024-04-21T23:12:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.042+0000 7ff5e91d7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c8102b20 0x55f3c819d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.042+0000 7ff5e91d7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c8103440 0x55f3c819dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.042+0000 7ff5e91d7740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c810c210 0x55f3c8128460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.042+0000 7ff5e91d7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff5cc00ec90 con 0x55f3c8102b20 2024-04-21T23:12:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.042+0000 7ff5e91d7740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff5cc010100 con 0x55f3c810c210 2024-04-21T23:12:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.042+0000 7ff5e91d7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff5cc00f360 con 0x55f3c8103440 2024-04-21T23:12:57.041 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.042+0000 7ff5e91d7740 10 client.0 _ll_register_callbacks cb 0x7ff5d4009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:12:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.042+0000 7ff5dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c8102b20 0x55f3c819d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.042+0000 7ff5df7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c8103440 0x55f3c819dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.042+0000 7ff5dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c8102b20 0x55f3c819d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:52138/0 (socket says 172.21.15.44:52138) 2024-04-21T23:12:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.042+0000 7ff5dffff640 1 -- 172.21.15.44:0/1610122283 learned_addr learned my addr 172.21.15.44:0/1610122283 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:12:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.043+0000 7ff5e4ab6640 1 --2- 172.21.15.44:0/1610122283 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c810c210 0x55f3c8128460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.043+0000 7ff5dffff640 1 -- 172.21.15.44:0/1610122283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c8103440 msgr2=0x55f3c819dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.043+0000 7ff5dffff640 1 --2- 172.21.15.44:0/1610122283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c8103440 0x55f3c819dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.043+0000 7ff5dffff640 1 -- 172.21.15.44:0/1610122283 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c810c210 msgr2=0x55f3c8128460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:12:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.043+0000 7ff5dffff640 1 --2- 172.21.15.44:0/1610122283 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c810c210 0x55f3c8128460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.043+0000 7ff5dffff640 1 -- 172.21.15.44:0/1610122283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3c8128b50 con 0x55f3c8102b20 2024-04-21T23:12:57.043 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.043+0000 7ff5dffff640 1 --2- 172.21.15.44:0/1610122283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c8102b20 0x55f3c819d6b0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff5d002c170 tx=0x7ff5d00042c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=873ee9d549d9ecd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:12:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.043+0000 7ff5bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:12:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.044+0000 7ff5bf7fe640 1 -- 172.21.15.44:0/1610122283 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0045070 con 0x55f3c8102b20 2024-04-21T23:12:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.044+0000 7ff5bf7fe640 1 -- 172.21.15.44:0/1610122283 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7ff5d0005030 con 0x55f3c8102b20 2024-04-21T23:12:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.044+0000 7ff5e91d7740 1 -- 172.21.15.44:0/1610122283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5d00501d0 con 0x55f3c8102b20 2024-04-21T23:12:57.044 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.044+0000 7ff5bf7fe640 1 -- 172.21.15.44:0/1610122283 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff5d0060050 con 0x55f3c8102b20 2024-04-21T23:12:57.045 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.045+0000 7ff5e91d7740 10 client.16584 fetch_fsmap learned FSMap version 25 2024-04-21T23:12:57.045 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.045+0000 7ff5e91d7740 1 -- 172.21.15.44:0/1610122283 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f3c81a4e60 con 0x55f3c8102b20 2024-04-21T23:12:57.045 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.046+0000 7ff5bf7fe640 1 -- 172.21.15.44:0/1610122283 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5d00501d0 con 0x55f3c8102b20 2024-04-21T23:12:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.047+0000 7ff5bf7fe640 1 -- 172.21.15.44:0/1610122283 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7ff5d0004960 con 0x55f3c8102b20 2024-04-21T23:12:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.048+0000 7ff5e91d7740 10 client.16584 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:12:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.048+0000 7ff5e91d7740 10 client.16584 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:12:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.048+0000 7ff5e91d7740 20 client.16584 populate_metadata read hostname 'smithi044' 2024-04-21T23:12:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.048+0000 7ff5e91d7740 1 --2- 172.21.15.44:0/1610122283 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f3c81a54d0 0x55f3c81a78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:12:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.048+0000 7ff5e91d7740 4 client.16584 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:12:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.048+0000 7ff5e91d7740 1 -- 172.21.15.44:0/1610122283 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f3c806e400 con 0x55f3c81a54d0 2024-04-21T23:12:57.048 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.048+0000 7ff5df7fe640 1 --2- 172.21.15.44:0/1610122283 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f3c81a54d0 0x55f3c81a78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:12:57.050 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.050+0000 7ff5df7fe640 1 --2- 172.21.15.44:0/1610122283 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f3c81a54d0 0x55f3c81a78f0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3d2bc9a11713fa server_cookie=aa3c7548a580bc44 in_seq=0 out_seq=0 2024-04-21T23:12:57.050 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.050+0000 7ff5bf7fe640 10 client.16584 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.354+0000 7ff5bf7fe640 1 -- 172.21.15.44:0/1610122283 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f3c806e400 con 0x55f3c81a54d0 2024-04-21T23:12:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.354+0000 7ff5bf7fe640 10 client.16584 handle_command_reply: tid=0 2024-04-21T23:12:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.354+0000 7ff5e91d7740 1 client.16584 shutdown 2024-04-21T23:12:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.355+0000 7ff5e91d7740 1 -- 172.21.15.44:0/1610122283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c8102b20 msgr2=0x55f3c819d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.355+0000 7ff5e91d7740 1 --2- 172.21.15.44:0/1610122283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c8102b20 0x55f3c819d6b0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff5d002c170 tx=0x7ff5d00042c0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.355+0000 7ff5e91d7740 1 -- 172.21.15.44:0/1610122283 shutdown_connections 2024-04-21T23:12:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.355+0000 7ff5e91d7740 1 --2- 172.21.15.44:0/1610122283 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f3c81a54d0 0x55f3c81a78f0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.355+0000 7ff5e91d7740 1 --2- 172.21.15.44:0/1610122283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f3c8103440 0x55f3c819dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.355+0000 7ff5e91d7740 1 --2- 172.21.15.44:0/1610122283 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f3c810c210 0x55f3c8128460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.355+0000 7ff5e91d7740 1 --2- 172.21.15.44:0/1610122283 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f3c8102b20 0x55f3c819d6b0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.355+0000 7ff5e91d7740 1 -- 172.21.15.44:0/1610122283 >> 172.21.15.44:0/1610122283 conn(0x55f3c80fd440 msgr2=0x55f3c8130d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.355+0000 7ff5bf7fe640 0 client.16584 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:12:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.355+0000 7ff5e91d7740 1 -- 172.21.15.44:0/1610122283 shutdown_connections 2024-04-21T23:12:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.355+0000 7ff5e91d7740 1 -- 172.21.15.44:0/1610122283 wait complete. 2024-04-21T23:12:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.355+0000 7ff5e91d7740 20 client.16584 trim_cache size 0 max 16384 2024-04-21T23:12:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:12:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:12:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:12:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:12:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.356+0000 7ff5e6519640 1 -- 172.21.15.44:0/1972472940 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff5c80a2df0 msgr2=0x7ff5c80a52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.356+0000 7ff5e6519640 1 --2- 172.21.15.44:0/1972472940 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff5c80a2df0 0x7ff5c80a52b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00096c0 tx=0x7ff5d0000bc0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.356+0000 7ff5e6519640 1 -- 172.21.15.44:0/1972472940 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5e01020f0 msgr2=0x7ff5e01985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:12:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.356+0000 7ff5e6519640 1 --2- 172.21.15.44:0/1972472940 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5e01020f0 0x7ff5e01985b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009f20 tx=0x7ff5d400ee80 comp rx=0 tx=0).stop 2024-04-21T23:12:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.356+0000 7ff5e6519640 1 -- 172.21.15.44:0/1972472940 shutdown_connections 2024-04-21T23:12:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.356+0000 7ff5e6519640 1 --2- 172.21.15.44:0/1972472940 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5e0107740 0x7ff5e019c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.356+0000 7ff5e6519640 1 --2- 172.21.15.44:0/1972472940 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff5c80a2df0 0x7ff5c80a52b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.356+0000 7ff5e6519640 1 --2- 172.21.15.44:0/1972472940 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff5e01020f0 0x7ff5e01985b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.356+0000 7ff5e6519640 1 --2- 172.21.15.44:0/1972472940 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff5e0101730 0x7ff5e0198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:12:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.356+0000 7ff5e6519640 1 -- 172.21.15.44:0/1972472940 >> 172.21.15.44:0/1972472940 conn(0x7ff5e00fd200 msgr2=0x7ff5e00fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:12:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.356+0000 7ff5e6519640 1 -- 172.21.15.44:0/1972472940 shutdown_connections 2024-04-21T23:12:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:12:57.356+0000 7ff5e6519640 1 -- 172.21.15.44:0/1972472940 wait complete. 2024-04-21T23:12:57.698 INFO:tasks.workunit.client.0.smithi044.stdout: 1 429889 13.44 MB/sec execute 47 sec latency 104.648 ms 2024-04-21T23:12:58.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:57 smithi044 ceph-mon[27934]: pgmap v128: 129 pgs: 129 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 479 op/s 2024-04-21T23:12:58.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:57 smithi055 ceph-mon[31117]: pgmap v128: 129 pgs: 129 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 479 op/s 2024-04-21T23:12:58.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:57 smithi069 ceph-mon[31816]: pgmap v128: 129 pgs: 129 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 479 op/s 2024-04-21T23:12:58.698 INFO:tasks.workunit.client.0.smithi044.stdout: 1 432924 13.52 MB/sec execute 48 sec latency 40.902 ms 2024-04-21T23:12:59.698 INFO:tasks.workunit.client.0.smithi044.stdout: 1 435256 13.43 MB/sec execute 49 sec latency 27.837 ms 2024-04-21T23:13:00.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:59 smithi044 ceph-mon[27934]: pgmap v129: 129 pgs: 129 active+clean; 69 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 358 op/s 2024-04-21T23:13:00.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:59 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:13:00.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:59 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:00.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:59 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:13:00.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:12:59 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:13:00.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:59 smithi055 ceph-mon[31117]: pgmap v129: 129 pgs: 129 active+clean; 69 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 358 op/s 2024-04-21T23:13:00.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:59 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:13:00.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:59 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:00.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:59 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:13:00.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:12:59 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:13:00.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:59 smithi069 ceph-mon[31816]: pgmap v129: 129 pgs: 129 active+clean; 69 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 358 op/s 2024-04-21T23:13:00.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:59 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:13:00.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:59 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:13:00.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:59 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:13:00.216 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:12:59 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:13:00.698 INFO:tasks.workunit.client.0.smithi044.stdout: 1 437464 13.43 MB/sec execute 50 sec latency 16.547 ms 2024-04-21T23:13:01.698 INFO:tasks.workunit.client.0.smithi044.stdout: 1 438601 13.26 MB/sec execute 51 sec latency 196.683 ms 2024-04-21T23:13:02.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:01 smithi044 ceph-mon[27934]: pgmap v130: 129 pgs: 129 active+clean; 69 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 232 op/s 2024-04-21T23:13:02.205 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:01 smithi055 ceph-mon[31117]: pgmap v130: 129 pgs: 129 active+clean; 69 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 232 op/s 2024-04-21T23:13:02.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:01 smithi069 ceph-mon[31816]: pgmap v130: 129 pgs: 129 active+clean; 69 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 232 op/s 2024-04-21T23:13:02.698 INFO:tasks.workunit.client.0.smithi044.stdout: 1 440257 13.24 MB/sec execute 52 sec latency 69.357 ms 2024-04-21T23:13:03.698 INFO:tasks.workunit.client.0.smithi044.stdout: 1 443391 13.32 MB/sec execute 53 sec latency 16.052 ms 2024-04-21T23:13:04.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:03 smithi044 ceph-mon[27934]: pgmap v131: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.8 MiB/s wr, 535 op/s 2024-04-21T23:13:04.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:03 smithi069 ceph-mon[31816]: pgmap v131: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.8 MiB/s wr, 535 op/s 2024-04-21T23:13:04.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:03 smithi055 ceph-mon[31117]: pgmap v131: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.8 MiB/s wr, 535 op/s 2024-04-21T23:13:04.698 INFO:tasks.workunit.client.0.smithi044.stdout: 1 445582 13.22 MB/sec execute 54 sec latency 95.381 ms 2024-04-21T23:13:05.698 INFO:tasks.workunit.client.0.smithi044.stdout: 1 447861 13.23 MB/sec execute 55 sec latency 17.520 ms 2024-04-21T23:13:06.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:05 smithi044 ceph-mon[27934]: pgmap v132: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 401 op/s 2024-04-21T23:13:06.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:05 smithi069 ceph-mon[31816]: pgmap v132: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 401 op/s 2024-04-21T23:13:06.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:05 smithi055 ceph-mon[31117]: pgmap v132: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 401 op/s 2024-04-21T23:13:06.698 INFO:tasks.workunit.client.0.smithi044.stdout: 1 450775 13.32 MB/sec execute 56 sec latency 77.762 ms 2024-04-21T23:13:07.698 INFO:tasks.workunit.client.0.smithi044.stdout: 1 453373 13.29 MB/sec execute 57 sec latency 20.031 ms 2024-04-21T23:13:08.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:07 smithi044 ceph-mon[27934]: pgmap v133: 129 pgs: 129 active+clean; 63 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 496 op/s 2024-04-21T23:13:08.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:07 smithi055 ceph-mon[31117]: pgmap v133: 129 pgs: 129 active+clean; 63 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 496 op/s 2024-04-21T23:13:08.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:07 smithi069 ceph-mon[31816]: pgmap v133: 129 pgs: 129 active+clean; 63 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.2 MiB/s wr, 496 op/s 2024-04-21T23:13:08.698 INFO:tasks.workunit.client.0.smithi044.stdout: 1 454961 13.21 MB/sec execute 58 sec latency 58.735 ms 2024-04-21T23:13:09.698 INFO:tasks.workunit.client.0.smithi044.stdout: 1 458093 13.29 MB/sec execute 59 sec latency 28.638 ms 2024-04-21T23:13:10.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:09 smithi044 ceph-mon[27934]: pgmap v134: 129 pgs: 129 active+clean; 65 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.1 MiB/s wr, 486 op/s 2024-04-21T23:13:10.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:09 smithi069 ceph-mon[31816]: pgmap v134: 129 pgs: 129 active+clean; 65 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.1 MiB/s wr, 486 op/s 2024-04-21T23:13:10.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:09 smithi055 ceph-mon[31117]: pgmap v134: 129 pgs: 129 active+clean; 65 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.1 MiB/s wr, 486 op/s 2024-04-21T23:13:10.458 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.456+0000 7fb00f27d640 1 -- 172.21.15.44:0/3014003407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb008101750 msgr2=0x7fb008101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:10.458 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.456+0000 7fb00f27d640 1 --2- 172.21.15.44:0/3014003407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb008101750 0x7fb008101bd0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fb004009cd0 tx=0x7fb00402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.458 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.456+0000 7fb00f27d640 1 -- 172.21.15.44:0/3014003407 shutdown_connections 2024-04-21T23:13:10.458 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.456+0000 7fb00f27d640 1 --2- 172.21.15.44:0/3014003407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb008102110 0x7fb00810e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.458 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.456+0000 7fb00f27d640 1 --2- 172.21.15.44:0/3014003407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb008101750 0x7fb008101bd0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.458 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.456+0000 7fb00f27d640 1 --2- 172.21.15.44:0/3014003407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb008107760 0x7fb008107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.458 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.456+0000 7fb00f27d640 1 -- 172.21.15.44:0/3014003407 >> 172.21.15.44:0/3014003407 conn(0x7fb0080fd220 msgr2=0x7fb0080ff640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.456+0000 7fb00f27d640 1 -- 172.21.15.44:0/3014003407 shutdown_connections 2024-04-21T23:13:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.456+0000 7fb00f27d640 1 -- 172.21.15.44:0/3014003407 wait complete. 2024-04-21T23:13:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.457+0000 7fb00f27d640 1 Processor -- start 2024-04-21T23:13:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.457+0000 7fb00f27d640 1 -- start start 2024-04-21T23:13:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.457+0000 7fb00f27d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb008101750 0x7fb0081980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.457+0000 7fb00f27d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb008102110 0x7fb008198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.457+0000 7fb00f27d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb008107760 0x7fb00819c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.457+0000 7fb00f27d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb008111b70 con 0x7fb008102110 2024-04-21T23:13:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.457+0000 7fb00f27d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb0081119f0 con 0x7fb008107760 2024-04-21T23:13:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.457+0000 7fb00f27d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb008111cf0 con 0x7fb008101750 2024-04-21T23:13:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.457+0000 7fb00c818640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb008102110 0x7fb008198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.457+0000 7fb00c818640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb008102110 0x7fb008198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53634/0 (socket says 172.21.15.44:53634) 2024-04-21T23:13:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.457+0000 7fb00d019640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb008101750 0x7fb0081980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.457+0000 7fb00c818640 1 -- 172.21.15.44:0/1905053462 learned_addr learned my addr 172.21.15.44:0/1905053462 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.457+0000 7fb00d81a640 1 --2- 172.21.15.44:0/1905053462 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb008107760 0x7fb00819c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.457+0000 7fb00c818640 1 -- 172.21.15.44:0/1905053462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb008101750 msgr2=0x7fb0081980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.457+0000 7fb00c818640 1 --2- 172.21.15.44:0/1905053462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb008101750 0x7fb0081980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.457+0000 7fb00c818640 1 -- 172.21.15.44:0/1905053462 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb008107760 msgr2=0x7fb00819c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.458+0000 7fb00c818640 1 --2- 172.21.15.44:0/1905053462 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb008107760 0x7fb00819c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.458+0000 7fb00c818640 1 -- 172.21.15.44:0/1905053462 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb00819d090 con 0x7fb008102110 2024-04-21T23:13:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.458+0000 7fb00c818640 1 --2- 172.21.15.44:0/1905053462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb008102110 0x7fb008198630 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fb004004af0 tx=0x7fb004032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.458+0000 7faffa7fc640 1 -- 172.21.15.44:0/1905053462 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb004041020 con 0x7fb008102110 2024-04-21T23:13:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.458+0000 7fb00f27d640 1 -- 172.21.15.44:0/1905053462 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb00819d320 con 0x7fb008102110 2024-04-21T23:13:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.458+0000 7faffa7fc640 1 -- 172.21.15.44:0/1905053462 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb004034cd0 con 0x7fb008102110 2024-04-21T23:13:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.458+0000 7fb00f27d640 1 -- 172.21.15.44:0/1905053462 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0081a4c10 con 0x7fb008102110 2024-04-21T23:13:10.460 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.458+0000 7faffa7fc640 1 -- 172.21.15.44:0/1905053462 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb004039040 con 0x7fb008102110 2024-04-21T23:13:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.460+0000 7faffa7fc640 1 -- 172.21.15.44:0/1905053462 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb004044020 con 0x7fb008102110 2024-04-21T23:13:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7fb011e66740 1 Processor -- start 2024-04-21T23:13:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7fb011e66740 1 -- start start 2024-04-21T23:13:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7fb011e66740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebbb20 0x5610bdebbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7fb011e66740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdebc440 0x5610bdec4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7fb011e66740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdec5210 0x5610bdec7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7fb011e66740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5610bde1fb70 con 0x5610bdebbb20 2024-04-21T23:13:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7fb011e66740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5610bddc8470 con 0x5610bdec5210 2024-04-21T23:13:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7fb011e66740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5610bde1bb30 con 0x5610bdebc440 2024-04-21T23:13:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7faffa7fc640 1 --2- 172.21.15.44:0/1905053462 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faff40a2cf0 0x7faff40a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7fb00d81a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebbb20 0x5610bdebbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7fb00d81a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebbb20 0x5610bdebbf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53644/0 (socket says 172.21.15.44:53644) 2024-04-21T23:13:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7fb00d019640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdebc440 0x5610bdec4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7fb00d81a640 1 -- 172.21.15.44:0/1593010895 learned_addr learned my addr 172.21.15.44:0/1593010895 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7fb00c818640 1 --2- 172.21.15.44:0/1593010895 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdec5210 0x5610bdec7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7fb00d81a640 1 --2- 172.21.15.44:0/1905053462 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faff40a2cf0 0x7faff40a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7fb00d019640 1 -- 172.21.15.44:0/1593010895 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdec5210 msgr2=0x5610bdec7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7faffa7fc640 1 -- 172.21.15.44:0/1905053462 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fb0040f13e0 con 0x7fb008102110 2024-04-21T23:13:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7fb00d019640 1 --2- 172.21.15.44:0/1593010895 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdec5210 0x5610bdec7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7fb00d019640 1 -- 172.21.15.44:0/1593010895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebbb20 msgr2=0x5610bdebbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7fb00d019640 1 --2- 172.21.15.44:0/1593010895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebbb20 0x5610bdebbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7fb00d019640 1 -- 172.21.15.44:0/1593010895 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610bdb34210 con 0x5610bdebc440 2024-04-21T23:13:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.461+0000 7fb00d81a640 1 --2- 172.21.15.44:0/1593010895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebbb20 0x5610bdebbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.462+0000 7fb00d019640 1 --2- 172.21.15.44:0/1593010895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdebc440 0x5610bdec4cd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb000009c20 tx=0x7fb000011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=222d464f0c11664 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.462+0000 7fb00d81a640 1 --2- 172.21.15.44:0/1905053462 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faff40a2cf0 0x7faff40a51b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7faffc0129f0 tx=0x7faffc00b2e0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.462+0000 7faff37fe640 1 -- 172.21.15.44:0/1593010895 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb000004050 con 0x5610bdebc440 2024-04-21T23:13:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.462+0000 7faff37fe640 1 -- 172.21.15.44:0/1593010895 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb000005500 con 0x5610bdebc440 2024-04-21T23:13:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.462+0000 7faff37fe640 1 -- 172.21.15.44:0/1593010895 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb000004d60 con 0x5610bdebc440 2024-04-21T23:13:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.463+0000 7fb011e66740 1 -- 172.21.15.44:0/1593010895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdebc440 msgr2=0x5610bdec4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.463+0000 7fb011e66740 1 --2- 172.21.15.44:0/1593010895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdebc440 0x5610bdec4cd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb000009c20 tx=0x7fb000011810 comp rx=0 tx=0).stop 2024-04-21T23:13:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.463+0000 7fb011e66740 1 -- 172.21.15.44:0/1593010895 shutdown_connections 2024-04-21T23:13:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.463+0000 7fb011e66740 1 --2- 172.21.15.44:0/1593010895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdebc440 0x5610bdec4cd0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.463+0000 7fb011e66740 1 --2- 172.21.15.44:0/1593010895 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdec5210 0x5610bdec7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.463+0000 7fb011e66740 1 --2- 172.21.15.44:0/1593010895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebbb20 0x5610bdebbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.463+0000 7fb011e66740 1 -- 172.21.15.44:0/1593010895 >> 172.21.15.44:0/1593010895 conn(0x5610bdeb6440 msgr2=0x5610bdeb6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.463+0000 7fb011e66740 1 -- 172.21.15.44:0/1593010895 shutdown_connections 2024-04-21T23:13:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.463+0000 7fb011e66740 1 -- 172.21.15.44:0/1593010895 wait complete. 2024-04-21T23:13:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.463+0000 7fb011e66740 1 -- start start 2024-04-21T23:13:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.463+0000 7fb011e66740 1 Processor -- start 2024-04-21T23:13:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7fb011e66740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdebbb20 0x5610bdec2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7fb011e66740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 0x5610bdec2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7fb011e66740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdec5210 0x5610bdec2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7fb011e66740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5610bdec7e40 con 0x5610bdebc440 2024-04-21T23:13:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7fb011e66740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5610bdddba10 con 0x5610bdebbb20 2024-04-21T23:13:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7fb011e66740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5610bdec7fc0 con 0x5610bdec5210 2024-04-21T23:13:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7fb011e66740 10 client.0 _ll_register_callbacks cb 0x5610bde16b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7fb00c818640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 0x5610bdec2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7fb00d019640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdebbb20 0x5610bdec2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7fb00c818640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 0x5610bdec2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53654/0 (socket says 172.21.15.44:53654) 2024-04-21T23:13:10.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7fb00d81a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdec5210 0x5610bdec2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:10.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7fb00c818640 1 -- 172.21.15.44:0/92430528 learned_addr learned my addr 172.21.15.44:0/92430528 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:10.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7fb00d019640 1 -- 172.21.15.44:0/92430528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdec5210 msgr2=0x5610bdec2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:10.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7fb00d019640 1 --2- 172.21.15.44:0/92430528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdec5210 0x5610bdec2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7fb00d019640 1 -- 172.21.15.44:0/92430528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 msgr2=0x5610bdec2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:10.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7fb00d019640 1 --2- 172.21.15.44:0/92430528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 0x5610bdec2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7fb00d019640 1 -- 172.21.15.44:0/92430528 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610bdc00620 con 0x5610bdebbb20 2024-04-21T23:13:10.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7fb00c818640 1 --2- 172.21.15.44:0/92430528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 0x5610bdec2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:10.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7fb00d81a640 1 --2- 172.21.15.44:0/92430528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdec5210 0x5610bdec2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:10.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7fb00d019640 1 --2- 172.21.15.44:0/92430528 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdebbb20 0x5610bdec2150 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb000000c00 tx=0x7fb000004480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85d282606793248f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:10.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.464+0000 7faff0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:13:10.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.465+0000 7faff0ff9640 1 -- 172.21.15.44:0/92430528 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0000037f0 con 0x5610bdebbb20 2024-04-21T23:13:10.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.465+0000 7faff0ff9640 1 -- 172.21.15.44:0/92430528 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fb000003990 con 0x5610bdebbb20 2024-04-21T23:13:10.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.465+0000 7fb011e66740 1 -- 172.21.15.44:0/92430528 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb000005500 con 0x5610bdebbb20 2024-04-21T23:13:10.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.465+0000 7faff0ff9640 1 -- 172.21.15.44:0/92430528 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb000033050 con 0x5610bdebbb20 2024-04-21T23:13:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.465+0000 7fb011e66740 10 client.26168 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.465+0000 7fb011e66740 1 -- 172.21.15.44:0/92430528 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5610bded3fc0 con 0x5610bdebbb20 2024-04-21T23:13:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.465+0000 7faff0ff9640 1 -- 172.21.15.44:0/92430528 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb000005500 con 0x5610bdebbb20 2024-04-21T23:13:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.465+0000 7faff0ff9640 1 -- 172.21.15.44:0/92430528 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fb00002e440 con 0x5610bdebbb20 2024-04-21T23:13:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.465+0000 7fb011e66740 10 client.26168 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.465+0000 7fb011e66740 10 client.26168 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.465+0000 7fb011e66740 20 client.26168 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.465+0000 7fb011e66740 1 --2- 172.21.15.44:0/92430528 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5610bded53c0 0x5610bded77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.465+0000 7fb011e66740 4 client.26168 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:13:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.465+0000 7fb011e66740 1 -- 172.21.15.44:0/92430528 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5610bde2d0b0 con 0x5610bded53c0 2024-04-21T23:13:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.465+0000 7fb00c818640 1 --2- 172.21.15.44:0/92430528 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5610bded53c0 0x5610bded77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.466+0000 7faff0ff9640 10 client.26168 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.466+0000 7fb00c818640 1 --2- 172.21.15.44:0/92430528 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5610bded53c0 0x5610bded77b0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67057bc095eaf910 server_cookie=b65ff41ed1ab3639 in_seq=0 out_seq=0 2024-04-21T23:13:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.466+0000 7faff0ff9640 1 -- 172.21.15.44:0/92430528 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5610bde2d0b0 con 0x5610bded53c0 2024-04-21T23:13:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.466+0000 7faff0ff9640 10 client.26168 handle_command_reply: tid=0 2024-04-21T23:13:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.466+0000 7fb011e66740 1 client.26168 shutdown 2024-04-21T23:13:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.467+0000 7fb011e66740 1 -- 172.21.15.44:0/92430528 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdebbb20 msgr2=0x5610bdec2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.467+0000 7fb011e66740 1 --2- 172.21.15.44:0/92430528 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdebbb20 0x5610bdec2150 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb000000c00 tx=0x7fb000004480 comp rx=0 tx=0).stop 2024-04-21T23:13:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.467+0000 7fb011e66740 1 -- 172.21.15.44:0/92430528 shutdown_connections 2024-04-21T23:13:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.467+0000 7fb011e66740 1 --2- 172.21.15.44:0/92430528 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5610bded53c0 0x5610bded77b0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.467+0000 7fb011e66740 1 --2- 172.21.15.44:0/92430528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdec5210 0x5610bdec2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.467+0000 7faff0ff9640 0 client.26168 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.467+0000 7fb011e66740 1 --2- 172.21.15.44:0/92430528 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 0x5610bdec2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.467+0000 7fb011e66740 1 --2- 172.21.15.44:0/92430528 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdebbb20 0x5610bdec2150 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.467+0000 7fb011e66740 1 -- 172.21.15.44:0/92430528 >> 172.21.15.44:0/92430528 conn(0x5610bdeb6440 msgr2=0x5610bdeb8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.467+0000 7fb011e66740 1 -- 172.21.15.44:0/92430528 shutdown_connections 2024-04-21T23:13:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.467+0000 7fb011e66740 1 -- 172.21.15.44:0/92430528 wait complete. 2024-04-21T23:13:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.467+0000 7fb011e66740 20 client.26168 trim_cache size 0 max 16384 2024-04-21T23:13:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.484+0000 7fb011e66740 1 Processor -- start 2024-04-21T23:13:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.484+0000 7fb011e66740 1 -- start start 2024-04-21T23:13:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.484+0000 7fb011e66740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdeb6440 0x5610bdec4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.484+0000 7fb011e66740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdebbb20 0x5610bded0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.484+0000 7fb011e66740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 0x5610bdeb1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.484+0000 7fb011e66740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb0081acc80 con 0x5610bdebc440 2024-04-21T23:13:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.484+0000 7fb011e66740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb0081af500 con 0x5610bdeb6440 2024-04-21T23:13:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.484+0000 7fb011e66740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb0081ad2f0 con 0x5610bdebbb20 2024-04-21T23:13:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.484+0000 7fb00d81a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 0x5610bdeb1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.484+0000 7fb00d019640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdeb6440 0x5610bdec4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.484+0000 7fb00d81a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 0x5610bdeb1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53662/0 (socket says 172.21.15.44:53662) 2024-04-21T23:13:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.484+0000 7fb00d81a640 1 -- 172.21.15.44:0/2291854374 learned_addr learned my addr 172.21.15.44:0/2291854374 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.484+0000 7fb00c818640 1 --2- 172.21.15.44:0/2291854374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdebbb20 0x5610bded0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.484+0000 7fb00d81a640 1 -- 172.21.15.44:0/2291854374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdebbb20 msgr2=0x5610bded0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.484+0000 7fb00d81a640 1 --2- 172.21.15.44:0/2291854374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdebbb20 0x5610bded0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.484+0000 7fb00d81a640 1 -- 172.21.15.44:0/2291854374 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdeb6440 msgr2=0x5610bdec4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.484+0000 7fb00d81a640 1 --2- 172.21.15.44:0/2291854374 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdeb6440 0x5610bdec4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.484+0000 7fb00d81a640 1 -- 172.21.15.44:0/2291854374 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610bdebc270 con 0x5610bdebc440 2024-04-21T23:13:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.484+0000 7fb00d019640 1 --2- 172.21.15.44:0/2291854374 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdeb6440 0x5610bdec4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.485+0000 7fb00d81a640 1 --2- 172.21.15.44:0/2291854374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 0x5610bdeb1450 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7faffc004170 tx=0x7faffc00b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc8e1a8558a95d67 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.485+0000 7faff2ffd640 1 -- 172.21.15.44:0/2291854374 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faffc009c80 con 0x5610bdebc440 2024-04-21T23:13:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.485+0000 7faff2ffd640 1 -- 172.21.15.44:0/2291854374 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faffc009e20 con 0x5610bdebc440 2024-04-21T23:13:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.485+0000 7faff2ffd640 1 -- 172.21.15.44:0/2291854374 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faffc004d20 con 0x5610bdebc440 2024-04-21T23:13:10.490 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.485+0000 7fb011e66740 1 -- 172.21.15.44:0/2291854374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 msgr2=0x5610bdeb1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:10.490 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.485+0000 7fb011e66740 1 --2- 172.21.15.44:0/2291854374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 0x5610bdeb1450 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7faffc004170 tx=0x7faffc00b760 comp rx=0 tx=0).stop 2024-04-21T23:13:10.490 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.486+0000 7fb011e66740 1 -- 172.21.15.44:0/2291854374 shutdown_connections 2024-04-21T23:13:10.490 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.486+0000 7fb011e66740 1 --2- 172.21.15.44:0/2291854374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdebbb20 0x5610bded0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.490 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.486+0000 7fb011e66740 1 --2- 172.21.15.44:0/2291854374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 0x5610bdeb1450 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.490 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.486+0000 7fb011e66740 1 --2- 172.21.15.44:0/2291854374 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdeb6440 0x5610bdec4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.490 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.486+0000 7fb011e66740 1 -- 172.21.15.44:0/2291854374 >> 172.21.15.44:0/2291854374 conn(0x5610bdebb4e0 msgr2=0x5610bdeb7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:10.490 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.486+0000 7fb011e66740 1 -- 172.21.15.44:0/2291854374 shutdown_connections 2024-04-21T23:13:10.491 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.486+0000 7fb011e66740 1 -- 172.21.15.44:0/2291854374 wait complete. 2024-04-21T23:13:10.491 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.486+0000 7fb011e66740 1 -- start start 2024-04-21T23:13:10.491 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.486+0000 7fb011e66740 1 Processor -- start 2024-04-21T23:13:10.491 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.486+0000 7fb011e66740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdeb6440 0x5610bdecd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:10.491 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.486+0000 7fb011e66740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdebbb20 0x5610bdecd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:10.491 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.486+0000 7fb011e66740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 0x5610bdebe770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:10.491 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.486+0000 7fb011e66740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb0081ad9c0 con 0x5610bdebc440 2024-04-21T23:13:10.491 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.486+0000 7fb011e66740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb0081aee30 con 0x5610bdebbb20 2024-04-21T23:13:10.491 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.486+0000 7fb011e66740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb0081ae090 con 0x5610bdeb6440 2024-04-21T23:13:10.491 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.487+0000 7fb011e66740 10 client.0 _ll_register_callbacks cb 0x7faffc00bca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:10.491 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.487+0000 7fb00d81a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 0x5610bdebe770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:10.491 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.487+0000 7fb00c818640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdebbb20 0x5610bdecd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:10.491 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.487+0000 7fb00d019640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdeb6440 0x5610bdecd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:10.491 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.487+0000 7fb00d81a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 0x5610bdebe770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53670/0 (socket says 172.21.15.44:53670) 2024-04-21T23:13:10.492 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.487+0000 7fb00d81a640 1 -- 172.21.15.44:0/2912602772 learned_addr learned my addr 172.21.15.44:0/2912602772 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:10.492 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.487+0000 7fb00d019640 1 -- 172.21.15.44:0/2912602772 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdebbb20 msgr2=0x5610bdecd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:10.492 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.487+0000 7fb00d019640 1 --2- 172.21.15.44:0/2912602772 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdebbb20 0x5610bdecd750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.492 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.487+0000 7fb00d019640 1 -- 172.21.15.44:0/2912602772 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 msgr2=0x5610bdebe770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:10.492 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.487+0000 7fb00d019640 1 --2- 172.21.15.44:0/2912602772 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 0x5610bdebe770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.492 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.487+0000 7fb00d019640 1 -- 172.21.15.44:0/2912602772 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610bdebefa0 con 0x5610bdeb6440 2024-04-21T23:13:10.492 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.487+0000 7fb00c818640 1 --2- 172.21.15.44:0/2912602772 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdebbb20 0x5610bdecd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:10.492 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.487+0000 7fb00d81a640 1 --2- 172.21.15.44:0/2912602772 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 0x5610bdebe770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:10.492 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.487+0000 7fb00d019640 1 --2- 172.21.15.44:0/2912602772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdeb6440 0x5610bdecd210 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb00002ea70 tx=0x7fb000009ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65c7b4b29d3343be server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:10.492 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.487+0000 7faff0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:13:10.492 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.487+0000 7faff0ff9640 1 -- 172.21.15.44:0/2912602772 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00002ed00 con 0x5610bdeb6440 2024-04-21T23:13:10.492 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.487+0000 7faff0ff9640 1 -- 172.21.15.44:0/2912602772 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fb00002e440 con 0x5610bdeb6440 2024-04-21T23:13:10.492 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.487+0000 7fb011e66740 1 -- 172.21.15.44:0/2912602772 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faffc009e20 con 0x5610bdeb6440 2024-04-21T23:13:10.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.487+0000 7faff0ff9640 1 -- 172.21.15.44:0/2912602772 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb00002e640 con 0x5610bdeb6440 2024-04-21T23:13:10.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.488+0000 7fb011e66740 10 client.26206 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:10.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.488+0000 7fb011e66740 1 -- 172.21.15.44:0/2912602772 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5610bdf3f650 con 0x5610bdeb6440 2024-04-21T23:13:10.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.488+0000 7faff0ff9640 1 -- 172.21.15.44:0/2912602772 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faffc009e20 con 0x5610bdeb6440 2024-04-21T23:13:10.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.488+0000 7faff0ff9640 1 -- 172.21.15.44:0/2912602772 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fb000039600 con 0x5610bdeb6440 2024-04-21T23:13:10.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.488+0000 7fb011e66740 10 client.26206 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:10.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.488+0000 7fb011e66740 10 client.26206 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:10.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.488+0000 7fb011e66740 20 client.26206 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:10.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.488+0000 7fb011e66740 1 --2- 172.21.15.44:0/2912602772 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5610bdec5210 0x5610bdf41cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:10.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.488+0000 7fb011e66740 4 client.26206 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:13:10.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.488+0000 7fb011e66740 1 -- 172.21.15.44:0/2912602772 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5610bde27400 con 0x5610bdec5210 2024-04-21T23:13:10.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.488+0000 7fb00c818640 1 --2- 172.21.15.44:0/2912602772 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5610bdec5210 0x5610bdf41cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:10.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.488+0000 7faff0ff9640 10 client.26206 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:10.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.488+0000 7fb00c818640 1 --2- 172.21.15.44:0/2912602772 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5610bdec5210 0x5610bdf41cf0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba3c9a9dd8a657e5 server_cookie=3e3020399a67c1db in_seq=0 out_seq=0 2024-04-21T23:13:10.698 INFO:tasks.workunit.client.0.smithi044.stdout: 1 459443 13.46 MB/sec execute 60 sec latency 102.231 ms 2024-04-21T23:13:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.768+0000 7faff0ff9640 1 -- 172.21.15.44:0/2912602772 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5610bde27400 con 0x5610bdec5210 2024-04-21T23:13:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.768+0000 7faff0ff9640 10 client.26206 handle_command_reply: tid=0 2024-04-21T23:13:10.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.768+0000 7fb011e66740 1 client.26206 shutdown 2024-04-21T23:13:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.768+0000 7fb011e66740 1 -- 172.21.15.44:0/2912602772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdeb6440 msgr2=0x5610bdecd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.768+0000 7fb011e66740 1 --2- 172.21.15.44:0/2912602772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdeb6440 0x5610bdecd210 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb00002ea70 tx=0x7fb000009ae0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.768+0000 7fb011e66740 1 -- 172.21.15.44:0/2912602772 shutdown_connections 2024-04-21T23:13:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.768+0000 7fb011e66740 1 --2- 172.21.15.44:0/2912602772 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5610bdec5210 0x5610bdf41cf0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.768+0000 7fb011e66740 1 --2- 172.21.15.44:0/2912602772 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5610bdebc440 0x5610bdebe770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.768+0000 7fb011e66740 1 --2- 172.21.15.44:0/2912602772 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5610bdebbb20 0x5610bdecd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.768+0000 7faff0ff9640 0 client.26206 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.768+0000 7fb011e66740 1 --2- 172.21.15.44:0/2912602772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5610bdeb6440 0x5610bdecd210 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.768+0000 7fb011e66740 1 -- 172.21.15.44:0/2912602772 >> 172.21.15.44:0/2912602772 conn(0x5610bdebb4e0 msgr2=0x5610bdeb7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.768+0000 7fb011e66740 1 -- 172.21.15.44:0/2912602772 shutdown_connections 2024-04-21T23:13:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.768+0000 7fb011e66740 1 -- 172.21.15.44:0/2912602772 wait complete. 2024-04-21T23:13:10.769 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.768+0000 7fb011e66740 20 client.26206 trim_cache size 0 max 16384 2024-04-21T23:13:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:13:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:13:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:13:10.770 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:13:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.769+0000 7fb00f27d640 1 -- 172.21.15.44:0/1905053462 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faff40a2cf0 msgr2=0x7faff40a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.769+0000 7fb00f27d640 1 --2- 172.21.15.44:0/1905053462 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faff40a2cf0 0x7faff40a51b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7faffc0129f0 tx=0x7faffc00b2e0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.769+0000 7fb00f27d640 1 -- 172.21.15.44:0/1905053462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb008102110 msgr2=0x7fb008198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.769+0000 7fb00f27d640 1 --2- 172.21.15.44:0/1905053462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb008102110 0x7fb008198630 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fb004004af0 tx=0x7fb004032ec0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.769+0000 7fb00f27d640 1 -- 172.21.15.44:0/1905053462 shutdown_connections 2024-04-21T23:13:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.769+0000 7fb00f27d640 1 --2- 172.21.15.44:0/1905053462 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faff40a2cf0 0x7faff40a51b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.769+0000 7fb00f27d640 1 --2- 172.21.15.44:0/1905053462 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb008107760 0x7fb00819c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.769+0000 7fb00f27d640 1 --2- 172.21.15.44:0/1905053462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb008102110 0x7fb008198630 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.769+0000 7fb00f27d640 1 --2- 172.21.15.44:0/1905053462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb008101750 0x7fb0081980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.769+0000 7fb00f27d640 1 -- 172.21.15.44:0/1905053462 >> 172.21.15.44:0/1905053462 conn(0x7fb0080fd220 msgr2=0x7fb0080fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:10.771 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.769+0000 7fb00f27d640 1 -- 172.21.15.44:0/1905053462 shutdown_connections 2024-04-21T23:13:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:10.769+0000 7fb00f27d640 1 -- 172.21.15.44:0/1905053462 wait complete. 2024-04-21T23:13:11.698 INFO:tasks.workunit.client.0.smithi044.stdout: 1 462229 13.51 MB/sec execute 61 sec latency 127.804 ms 2024-04-21T23:13:12.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:11 smithi044 ceph-mon[27934]: pgmap v135: 129 pgs: 129 active+clean; 65 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 431 op/s 2024-04-21T23:13:12.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:11 smithi055 ceph-mon[31117]: pgmap v135: 129 pgs: 129 active+clean; 65 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 431 op/s 2024-04-21T23:13:12.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:11 smithi069 ceph-mon[31816]: pgmap v135: 129 pgs: 129 active+clean; 65 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 431 op/s 2024-04-21T23:13:12.329 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:13:12.330 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:13:12.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.443+0000 7fefe8840640 1 -- 172.21.15.44:0/2731214077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fefe41066b0 msgr2=0x7fefe4106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:12.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.443+0000 7fefe8840640 1 --2- 172.21.15.44:0/2731214077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fefe41066b0 0x7fefe4106a90 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fefd4009930 tx=0x7fefd402bc30 comp rx=0 tx=0).stop 2024-04-21T23:13:12.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.443+0000 7fefe8840640 1 -- 172.21.15.44:0/2731214077 shutdown_connections 2024-04-21T23:13:12.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.443+0000 7fefe8840640 1 --2- 172.21.15.44:0/2731214077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefe4100610 0x7fefe4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.443+0000 7fefe8840640 1 --2- 172.21.15.44:0/2731214077 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefe4100fd0 0x7fefe410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.443+0000 7fefe8840640 1 --2- 172.21.15.44:0/2731214077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fefe41066b0 0x7fefe4106a90 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.443+0000 7fefe8840640 1 -- 172.21.15.44:0/2731214077 >> 172.21.15.44:0/2731214077 conn(0x7fefe40777f0 msgr2=0x7fefe4077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:12.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.443+0000 7fefe8840640 1 -- 172.21.15.44:0/2731214077 shutdown_connections 2024-04-21T23:13:12.443 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.443+0000 7fefe8840640 1 -- 172.21.15.44:0/2731214077 wait complete. 2024-04-21T23:13:12.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.444+0000 7fefe8840640 1 Processor -- start 2024-04-21T23:13:12.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.444+0000 7fefe8840640 1 -- start start 2024-04-21T23:13:12.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.444+0000 7fefe8840640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefe4100610 0x7fefe4198f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.445 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.444+0000 7fefe8840640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fefe4100fd0 0x7fefe4199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.444+0000 7fefe8840640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefe41066b0 0x7fefe41a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.444+0000 7fefe8840640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41109f0 con 0x7fefe4100fd0 2024-04-21T23:13:12.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.444+0000 7fefe8840640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fefe4110870 con 0x7fefe4100610 2024-04-21T23:13:12.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.444+0000 7fefe8840640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fefe4110b70 con 0x7fefe41066b0 2024-04-21T23:13:12.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.444+0000 7fefe259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefe4100610 0x7fefe4198f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.444+0000 7fefe259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefe4100610 0x7fefe4198f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:43714/0 (socket says 172.21.15.44:43714) 2024-04-21T23:13:12.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.444+0000 7fefe259c640 1 -- 172.21.15.44:0/646640524 learned_addr learned my addr 172.21.15.44:0/646640524 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:12.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.444+0000 7fefe2d9d640 1 --2- 172.21.15.44:0/646640524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefe41066b0 0x7fefe41a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.445+0000 7fefe1d9b640 1 --2- 172.21.15.44:0/646640524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fefe4100fd0 0x7fefe4199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.445+0000 7fefe259c640 1 -- 172.21.15.44:0/646640524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefe41066b0 msgr2=0x7fefe41a3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:12.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.445+0000 7fefe259c640 1 --2- 172.21.15.44:0/646640524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefe41066b0 0x7fefe41a3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.445+0000 7fefe259c640 1 -- 172.21.15.44:0/646640524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fefe4100fd0 msgr2=0x7fefe4199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:12.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.445+0000 7fefe259c640 1 --2- 172.21.15.44:0/646640524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fefe4100fd0 0x7fefe4199480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.445+0000 7fefe259c640 1 -- 172.21.15.44:0/646640524 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe41a4500 con 0x7fefe4100610 2024-04-21T23:13:12.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.445+0000 7fefe259c640 1 --2- 172.21.15.44:0/646640524 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefe4100610 0x7fefe4198f40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fefd4009230 tx=0x7fefd402cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:12.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.445+0000 7fefd37fe640 1 -- 172.21.15.44:0/646640524 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd4041070 con 0x7fefe4100610 2024-04-21T23:13:12.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.445+0000 7fefd37fe640 1 -- 172.21.15.44:0/646640524 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefd4004450 con 0x7fefe4100610 2024-04-21T23:13:12.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.445+0000 7fefe8840640 1 -- 172.21.15.44:0/646640524 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefe41a46d0 con 0x7fefe4100610 2024-04-21T23:13:12.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.445+0000 7fefd37fe640 1 -- 172.21.15.44:0/646640524 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd4003e50 con 0x7fefe4100610 2024-04-21T23:13:12.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.445+0000 7fefe8840640 1 -- 172.21.15.44:0/646640524 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefe41a4b90 con 0x7fefe4100610 2024-04-21T23:13:12.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.448+0000 7fefd37fe640 1 -- 172.21.15.44:0/646640524 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fefd4044070 con 0x7fefe4100610 2024-04-21T23:13:12.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.449+0000 7fefd37fe640 1 --2- 172.21.15.44:0/646640524 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fefcc0a2f20 0x7fefcc0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.449+0000 7fefeb42d740 1 Processor -- start 2024-04-21T23:13:12.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.449+0000 7fefd37fe640 1 -- 172.21.15.44:0/646640524 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fefd40f1240 con 0x7fefe4100610 2024-04-21T23:13:12.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.450+0000 7fefe1d9b640 1 --2- 172.21.15.44:0/646640524 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fefcc0a2f20 0x7fefcc0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.450+0000 7fefeb42d740 1 -- start start 2024-04-21T23:13:12.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.450+0000 7fefeb42d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a024463f30 0x55a024464310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.450+0000 7fefeb42d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 0x55a02446cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.450+0000 7fefeb42d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a02446d430 0x55a02446f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.450+0000 7fefeb42d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a0243c8560 con 0x55a024463f30 2024-04-21T23:13:12.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.450+0000 7fefeb42d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55a024370730 con 0x55a02446d430 2024-04-21T23:13:12.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.450+0000 7fefeb42d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a0243c4520 con 0x55a024464850 2024-04-21T23:13:12.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.450+0000 7fefe259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a024463f30 0x55a024464310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.450+0000 7fefe259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a024463f30 0x55a024464310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44322/0 (socket says 172.21.15.44:44322) 2024-04-21T23:13:12.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.450+0000 7fefe259c640 1 -- 172.21.15.44:0/3001662590 learned_addr learned my addr 172.21.15.44:0/3001662590 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:12.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.450+0000 7fefe1d9b640 1 --2- 172.21.15.44:0/646640524 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fefcc0a2f20 0x7fefcc0a53e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fefe4069590 tx=0x7fefd800b590 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:12.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.450+0000 7fefe2d9d640 1 --2- 172.21.15.44:0/3001662590 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a02446d430 0x55a02446f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.450+0000 7fefe1d9b640 1 --2- 172.21.15.44:0/3001662590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 0x55a02446cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.451+0000 7fefe2d9d640 1 -- 172.21.15.44:0/3001662590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 msgr2=0x55a02446cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:12.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.451+0000 7fefe2d9d640 1 --2- 172.21.15.44:0/3001662590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 0x55a02446cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.451+0000 7fefe2d9d640 1 -- 172.21.15.44:0/3001662590 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a024463f30 msgr2=0x55a024464310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:12.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.451+0000 7fefe2d9d640 1 --2- 172.21.15.44:0/3001662590 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a024463f30 0x55a024464310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.451+0000 7fefe2d9d640 1 -- 172.21.15.44:0/3001662590 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0241a67d0 con 0x55a02446d430 2024-04-21T23:13:12.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.451+0000 7fefe1d9b640 1 --2- 172.21.15.44:0/3001662590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 0x55a02446cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:12.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.451+0000 7fefe259c640 1 --2- 172.21.15.44:0/3001662590 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a024463f30 0x55a024464310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:12.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.451+0000 7fefe2d9d640 1 --2- 172.21.15.44:0/3001662590 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a02446d430 0x55a02446f820 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fefdc009ef0 tx=0x7fefdc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=964b849216ad6de9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:12.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.451+0000 7fefd0ff9640 1 -- 172.21.15.44:0/3001662590 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc004d20 con 0x55a02446d430 2024-04-21T23:13:12.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.451+0000 7fefd0ff9640 1 -- 172.21.15.44:0/3001662590 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefdc004050 con 0x55a02446d430 2024-04-21T23:13:12.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.451+0000 7fefd0ff9640 1 -- 172.21.15.44:0/3001662590 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc005260 con 0x55a02446d430 2024-04-21T23:13:12.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.452+0000 7fefeb42d740 1 -- 172.21.15.44:0/3001662590 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a02446d430 msgr2=0x55a02446f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:12.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.452+0000 7fefeb42d740 1 --2- 172.21.15.44:0/3001662590 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a02446d430 0x55a02446f820 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fefdc009ef0 tx=0x7fefdc011810 comp rx=0 tx=0).stop 2024-04-21T23:13:12.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.452+0000 7fefeb42d740 1 -- 172.21.15.44:0/3001662590 shutdown_connections 2024-04-21T23:13:12.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.452+0000 7fefeb42d740 1 --2- 172.21.15.44:0/3001662590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 0x55a02446cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.452+0000 7fefeb42d740 1 --2- 172.21.15.44:0/3001662590 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a02446d430 0x55a02446f820 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.452+0000 7fefeb42d740 1 --2- 172.21.15.44:0/3001662590 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a024463f30 0x55a024464310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.452+0000 7fefeb42d740 1 -- 172.21.15.44:0/3001662590 >> 172.21.15.44:0/3001662590 conn(0x55a02445e660 msgr2=0x55a02445ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:12.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.452+0000 7fefeb42d740 1 -- 172.21.15.44:0/3001662590 shutdown_connections 2024-04-21T23:13:12.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.452+0000 7fefeb42d740 1 -- 172.21.15.44:0/3001662590 wait complete. 2024-04-21T23:13:12.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.452+0000 7fefeb42d740 1 -- start start 2024-04-21T23:13:12.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.452+0000 7fefeb42d740 1 Processor -- start 2024-04-21T23:13:12.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefeb42d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a024463f30 0x55a02446a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefeb42d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 0x55a02446a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefeb42d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a02446d430 0x55a02446af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefeb42d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a024470220 con 0x55a02446d430 2024-04-21T23:13:12.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefeb42d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55a024383e00 con 0x55a024463f30 2024-04-21T23:13:12.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefeb42d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a0244703a0 con 0x55a024464850 2024-04-21T23:13:12.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefeb42d740 10 client.0 _ll_register_callbacks cb 0x55a0243beeb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:12.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefe2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a02446d430 0x55a02446af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefe2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a02446d430 0x55a02446af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44330/0 (socket says 172.21.15.44:44330) 2024-04-21T23:13:12.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefe2d9d640 1 -- 172.21.15.44:0/795565507 learned_addr learned my addr 172.21.15.44:0/795565507 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:12.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefe259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a024463f30 0x55a02446a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefe1d9b640 1 --2- 172.21.15.44:0/795565507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 0x55a02446a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefe2d9d640 1 -- 172.21.15.44:0/795565507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 msgr2=0x55a02446a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:12.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefe2d9d640 1 --2- 172.21.15.44:0/795565507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 0x55a02446a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefe2d9d640 1 -- 172.21.15.44:0/795565507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a024463f30 msgr2=0x55a02446a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:12.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefe2d9d640 1 --2- 172.21.15.44:0/795565507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a024463f30 0x55a02446a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefe2d9d640 1 -- 172.21.15.44:0/795565507 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0240dc3f0 con 0x55a02446d430 2024-04-21T23:13:12.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefe259c640 1 --2- 172.21.15.44:0/795565507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a024463f30 0x55a02446a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:12.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefe2d9d640 1 --2- 172.21.15.44:0/795565507 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a02446d430 0x55a02446af30 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fefdc011d10 tx=0x7fefdc004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f59f9bd5acff427 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:12.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefc9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:13:12.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefc9ffb640 1 -- 172.21.15.44:0/795565507 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc028070 con 0x55a02446d430 2024-04-21T23:13:12.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefc9ffb640 1 -- 172.21.15.44:0/795565507 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fefdc005040 con 0x55a02446d430 2024-04-21T23:13:12.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.453+0000 7fefeb42d740 1 -- 172.21.15.44:0/795565507 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefdc004050 con 0x55a02446d430 2024-04-21T23:13:12.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.454+0000 7fefc9ffb640 1 -- 172.21.15.44:0/795565507 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fefdc034050 con 0x55a02446d430 2024-04-21T23:13:12.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.454+0000 7fefeb42d740 10 client.16614 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:12.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.454+0000 7fefeb42d740 1 -- 172.21.15.44:0/795565507 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a02409c0f0 con 0x55a02446d430 2024-04-21T23:13:12.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.454+0000 7fefc9ffb640 1 -- 172.21.15.44:0/795565507 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefdc004050 con 0x55a02446d430 2024-04-21T23:13:12.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.454+0000 7fefc9ffb640 1 -- 172.21.15.44:0/795565507 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fefdc024440 con 0x55a02446d430 2024-04-21T23:13:12.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.454+0000 7fefeb42d740 10 client.16614 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:12.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.454+0000 7fefeb42d740 10 client.16614 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:12.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.454+0000 7fefeb42d740 20 client.16614 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:12.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.454+0000 7fefeb42d740 1 --2- 172.21.15.44:0/795565507 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a02447d780 0x55a02447fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.454+0000 7fefeb42d740 4 client.16614 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:13:12.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.454+0000 7fefeb42d740 1 -- 172.21.15.44:0/795565507 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0243d5aa0 con 0x55a02447d780 2024-04-21T23:13:12.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.454+0000 7fefe259c640 1 --2- 172.21.15.44:0/795565507 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a02447d780 0x55a02447fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.454+0000 7fefe259c640 1 --2- 172.21.15.44:0/795565507 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a02447d780 0x55a02447fb70 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f607298ac862f89 server_cookie=a59d4af2dcd478aa in_seq=0 out_seq=0 2024-04-21T23:13:12.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.454+0000 7fefc9ffb640 10 client.16614 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:12.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.455+0000 7fefc9ffb640 1 -- 172.21.15.44:0/795565507 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a0243d5aa0 con 0x55a02447d780 2024-04-21T23:13:12.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.455+0000 7fefc9ffb640 10 client.16614 handle_command_reply: tid=0 2024-04-21T23:13:12.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.455+0000 7fefeb42d740 1 client.16614 shutdown 2024-04-21T23:13:12.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.456+0000 7fefeb42d740 1 -- 172.21.15.44:0/795565507 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a02446d430 msgr2=0x55a02446af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:12.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.456+0000 7fefeb42d740 1 --2- 172.21.15.44:0/795565507 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a02446d430 0x55a02446af30 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fefdc011d10 tx=0x7fefdc004430 comp rx=0 tx=0).stop 2024-04-21T23:13:12.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.456+0000 7fefeb42d740 1 -- 172.21.15.44:0/795565507 shutdown_connections 2024-04-21T23:13:12.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.456+0000 7fefeb42d740 1 --2- 172.21.15.44:0/795565507 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a02447d780 0x55a02447fb70 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.456+0000 7fefeb42d740 1 --2- 172.21.15.44:0/795565507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 0x55a02446a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.456+0000 7fefeb42d740 1 --2- 172.21.15.44:0/795565507 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a02446d430 0x55a02446af30 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.456+0000 7fefc9ffb640 0 client.16614 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:12.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.456+0000 7fefeb42d740 1 --2- 172.21.15.44:0/795565507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a024463f30 0x55a02446a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.456+0000 7fefeb42d740 1 -- 172.21.15.44:0/795565507 >> 172.21.15.44:0/795565507 conn(0x55a02445e660 msgr2=0x55a0244603b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:12.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.456+0000 7fefeb42d740 1 -- 172.21.15.44:0/795565507 shutdown_connections 2024-04-21T23:13:12.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.456+0000 7fefeb42d740 1 -- 172.21.15.44:0/795565507 wait complete. 2024-04-21T23:13:12.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.456+0000 7fefeb42d740 20 client.16614 trim_cache size 0 max 16384 2024-04-21T23:13:12.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.475+0000 7fefeb42d740 1 Processor -- start 2024-04-21T23:13:12.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.475+0000 7fefeb42d740 1 -- start start 2024-04-21T23:13:12.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.475+0000 7fefeb42d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a024463f30 0x55a02446cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.475+0000 7fefeb42d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 0x55a024479110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.475+0000 7fefeb42d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a02446d430 0x55a0244657b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.475+0000 7fefeb42d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41accd0 con 0x55a024463f30 2024-04-21T23:13:12.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.475+0000 7fefeb42d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41af550 con 0x55a02446d430 2024-04-21T23:13:12.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.475+0000 7fefeb42d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41ad340 con 0x55a024464850 2024-04-21T23:13:12.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.475+0000 7fefe259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a024463f30 0x55a02446cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.475+0000 7fefe259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a024463f30 0x55a02446cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44342/0 (socket says 172.21.15.44:44342) 2024-04-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.475+0000 7fefe259c640 1 -- 172.21.15.44:0/3672965901 learned_addr learned my addr 172.21.15.44:0/3672965901 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.475+0000 7fefe2d9d640 1 --2- 172.21.15.44:0/3672965901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a02446d430 0x55a0244657b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.475+0000 7fefe1d9b640 1 --2- 172.21.15.44:0/3672965901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 0x55a024479110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.476+0000 7fefe259c640 1 -- 172.21.15.44:0/3672965901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 msgr2=0x55a024479110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.476+0000 7fefe259c640 1 --2- 172.21.15.44:0/3672965901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 0x55a024479110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.476+0000 7fefe259c640 1 -- 172.21.15.44:0/3672965901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a02446d430 msgr2=0x55a0244657b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.476+0000 7fefe259c640 1 --2- 172.21.15.44:0/3672965901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a02446d430 0x55a0244657b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.476+0000 7fefe259c640 1 -- 172.21.15.44:0/3672965901 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a02447fee0 con 0x55a024463f30 2024-04-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.476+0000 7fefe1d9b640 1 --2- 172.21.15.44:0/3672965901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 0x55a024479110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.476+0000 7fefe2d9d640 1 --2- 172.21.15.44:0/3672965901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a02446d430 0x55a0244657b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.476+0000 7fefe259c640 1 --2- 172.21.15.44:0/3672965901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a024463f30 0x55a02446cbe0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fefd40bd1d0 tx=0x7fefd4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708153483301a675 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.476+0000 7fefcbfff640 1 -- 172.21.15.44:0/3672965901 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd404b040 con 0x55a024463f30 2024-04-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.476+0000 7fefcbfff640 1 -- 172.21.15.44:0/3672965901 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefd40bdbd0 con 0x55a024463f30 2024-04-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.476+0000 7fefcbfff640 1 -- 172.21.15.44:0/3672965901 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd40ee870 con 0x55a024463f30 2024-04-21T23:13:12.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.477+0000 7fefeb42d740 1 -- 172.21.15.44:0/3672965901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a024463f30 msgr2=0x55a02446cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:12.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.477+0000 7fefeb42d740 1 --2- 172.21.15.44:0/3672965901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a024463f30 0x55a02446cbe0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fefd40bd1d0 tx=0x7fefd4009260 comp rx=0 tx=0).stop 2024-04-21T23:13:12.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.477+0000 7fefeb42d740 1 -- 172.21.15.44:0/3672965901 shutdown_connections 2024-04-21T23:13:12.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.477+0000 7fefeb42d740 1 --2- 172.21.15.44:0/3672965901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 0x55a024479110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.477+0000 7fefeb42d740 1 --2- 172.21.15.44:0/3672965901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a02446d430 0x55a0244657b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.477+0000 7fefeb42d740 1 --2- 172.21.15.44:0/3672965901 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a024463f30 0x55a02446cbe0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.477+0000 7fefeb42d740 1 -- 172.21.15.44:0/3672965901 >> 172.21.15.44:0/3672965901 conn(0x55a02445e660 msgr2=0x55a02445ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:12.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.477+0000 7fefeb42d740 1 -- 172.21.15.44:0/3672965901 shutdown_connections 2024-04-21T23:13:12.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.477+0000 7fefeb42d740 1 -- 172.21.15.44:0/3672965901 wait complete. 2024-04-21T23:13:12.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.477+0000 7fefeb42d740 1 -- start start 2024-04-21T23:13:12.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.478+0000 7fefeb42d740 1 Processor -- start 2024-04-21T23:13:12.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.478+0000 7fefeb42d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a024463f30 0x55a0244758f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.478+0000 7fefeb42d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 0x55a024466d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.478+0000 7fefeb42d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a02446d430 0x55a0244672a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.478+0000 7fefeb42d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41ada10 con 0x55a02446d430 2024-04-21T23:13:12.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.478+0000 7fefeb42d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41aee80 con 0x55a024463f30 2024-04-21T23:13:12.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.478+0000 7fefeb42d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41ae0e0 con 0x55a024464850 2024-04-21T23:13:12.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.478+0000 7fefeb42d740 10 client.0 _ll_register_callbacks cb 0x7fefd40b90d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:12.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.478+0000 7fefe2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a02446d430 0x55a0244672a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.478+0000 7fefe2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a02446d430 0x55a0244672a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44348/0 (socket says 172.21.15.44:44348) 2024-04-21T23:13:12.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.478+0000 7fefe259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a024463f30 0x55a0244758f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.478+0000 7fefe2d9d640 1 -- 172.21.15.44:0/86555180 learned_addr learned my addr 172.21.15.44:0/86555180 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:12.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.478+0000 7fefe1d9b640 1 --2- 172.21.15.44:0/86555180 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 0x55a024466d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.478+0000 7fefe2d9d640 1 -- 172.21.15.44:0/86555180 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 msgr2=0x55a024466d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:12.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.478+0000 7fefe2d9d640 1 --2- 172.21.15.44:0/86555180 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 0x55a024466d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.478+0000 7fefe2d9d640 1 -- 172.21.15.44:0/86555180 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a024463f30 msgr2=0x55a0244758f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:12.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.478+0000 7fefe2d9d640 1 --2- 172.21.15.44:0/86555180 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a024463f30 0x55a0244758f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.478+0000 7fefe2d9d640 1 -- 172.21.15.44:0/86555180 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a024464680 con 0x55a02446d430 2024-04-21T23:13:12.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.478+0000 7fefe2d9d640 1 --2- 172.21.15.44:0/86555180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a02446d430 0x55a0244672a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fefdc011850 tx=0x7fefdc005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12904e3535bd575 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:12.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.478+0000 7fefc9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:13:12.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.479+0000 7fefc9ffb640 1 -- 172.21.15.44:0/86555180 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc024e60 con 0x55a02446d430 2024-04-21T23:13:12.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.479+0000 7fefc9ffb640 1 -- 172.21.15.44:0/86555180 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fefdc004710 con 0x55a02446d430 2024-04-21T23:13:12.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.479+0000 7fefeb42d740 1 -- 172.21.15.44:0/86555180 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefd40bdbd0 con 0x55a02446d430 2024-04-21T23:13:12.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.479+0000 7fefc9ffb640 1 -- 172.21.15.44:0/86555180 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fefdc041050 con 0x55a02446d430 2024-04-21T23:13:12.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.479+0000 7fefeb42d740 10 client.16620 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:12.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.479+0000 7fefeb42d740 1 -- 172.21.15.44:0/86555180 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a0244e7d70 con 0x55a02446d430 2024-04-21T23:13:12.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.479+0000 7fefc9ffb640 1 -- 172.21.15.44:0/86555180 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefd40bdbd0 con 0x55a02446d430 2024-04-21T23:13:12.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.479+0000 7fefc9ffb640 1 -- 172.21.15.44:0/86555180 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fefdc02d9b0 con 0x55a02446d430 2024-04-21T23:13:12.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.479+0000 7fefeb42d740 10 client.16620 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:12.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.479+0000 7fefeb42d740 10 client.16620 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:12.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.479+0000 7fefeb42d740 20 client.16620 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:12.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.479+0000 7fefeb42d740 1 --2- 172.21.15.44:0/86555180 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a0244e8460 0x55a0244ea880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.479+0000 7fefeb42d740 4 client.16620 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.479+0000 7fefeb42d740 1 -- 172.21.15.44:0/86555180 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a0243cfdf0 con 0x55a0244e8460 2024-04-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.479+0000 7fefe259c640 1 --2- 172.21.15.44:0/86555180 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a0244e8460 0x55a0244ea880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.480+0000 7fefc9ffb640 10 client.16620 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.480+0000 7fefe259c640 1 --2- 172.21.15.44:0/86555180 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a0244e8460 0x55a0244ea880 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3857d3c6bac2c99e server_cookie=f5e0aa1f0a02d84e in_seq=0 out_seq=0 2024-04-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.480+0000 7fefc9ffb640 1 -- 172.21.15.44:0/86555180 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a0243cfdf0 con 0x55a0244e8460 2024-04-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.480+0000 7fefc9ffb640 10 client.16620 handle_command_reply: tid=0 2024-04-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.480+0000 7fefeb42d740 1 client.16620 shutdown 2024-04-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.480+0000 7fefeb42d740 1 -- 172.21.15.44:0/86555180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a02446d430 msgr2=0x55a0244672a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.480+0000 7fefeb42d740 1 --2- 172.21.15.44:0/86555180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a02446d430 0x55a0244672a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fefdc011850 tx=0x7fefdc005fa0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.480+0000 7fefeb42d740 1 -- 172.21.15.44:0/86555180 shutdown_connections 2024-04-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.480+0000 7fefeb42d740 1 --2- 172.21.15.44:0/86555180 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a0244e8460 0x55a0244ea880 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.480+0000 7fefeb42d740 1 --2- 172.21.15.44:0/86555180 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a024464850 0x55a024466d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.480+0000 7fefeb42d740 1 --2- 172.21.15.44:0/86555180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a02446d430 0x55a0244672a0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.480+0000 7fefeb42d740 1 --2- 172.21.15.44:0/86555180 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a024463f30 0x55a0244758f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.480+0000 7fefeb42d740 1 -- 172.21.15.44:0/86555180 >> 172.21.15.44:0/86555180 conn(0x55a02445e660 msgr2=0x55a02446ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:12.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.480+0000 7fefc9ffb640 0 client.16620 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:12.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.480+0000 7fefeb42d740 1 -- 172.21.15.44:0/86555180 shutdown_connections 2024-04-21T23:13:12.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.480+0000 7fefeb42d740 1 -- 172.21.15.44:0/86555180 wait complete. 2024-04-21T23:13:12.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.481+0000 7fefeb42d740 20 client.16620 trim_cache size 0 max 16384 2024-04-21T23:13:12.486 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:13:12.486 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-21T23:13:12.486 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-21T23:13:12.486 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:13:12.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.481+0000 7fefe8840640 1 -- 172.21.15.44:0/646640524 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fefcc0a2f20 msgr2=0x7fefcc0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:12.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.481+0000 7fefe8840640 1 --2- 172.21.15.44:0/646640524 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fefcc0a2f20 0x7fefcc0a53e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fefe4069590 tx=0x7fefd800b590 comp rx=0 tx=0).stop 2024-04-21T23:13:12.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.481+0000 7fefe8840640 1 -- 172.21.15.44:0/646640524 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefe4100610 msgr2=0x7fefe4198f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:12.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.481+0000 7fefe8840640 1 --2- 172.21.15.44:0/646640524 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefe4100610 0x7fefe4198f40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fefd4009230 tx=0x7fefd402cbf0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.481+0000 7fefe8840640 1 -- 172.21.15.44:0/646640524 shutdown_connections 2024-04-21T23:13:12.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.481+0000 7fefe8840640 1 --2- 172.21.15.44:0/646640524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefe41066b0 0x7fefe41a3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.481+0000 7fefe8840640 1 --2- 172.21.15.44:0/646640524 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fefcc0a2f20 0x7fefcc0a53e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.481+0000 7fefe8840640 1 --2- 172.21.15.44:0/646640524 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fefe4100fd0 0x7fefe4199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.481+0000 7fefe8840640 1 --2- 172.21.15.44:0/646640524 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefe4100610 0x7fefe4198f40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.481+0000 7fefe8840640 1 -- 172.21.15.44:0/646640524 >> 172.21.15.44:0/646640524 conn(0x7fefe40777f0 msgr2=0x7fefe40ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:12.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.481+0000 7fefe8840640 1 -- 172.21.15.44:0/646640524 shutdown_connections 2024-04-21T23:13:12.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.481+0000 7fefe8840640 1 -- 172.21.15.44:0/646640524 wait complete. 2024-04-21T23:13:12.492 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:13:12.492 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:13:12.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.604+0000 7fc0cc1ba640 1 -- 172.21.15.44:0/2806443477 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0c41066c0 msgr2=0x7fc0c4106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:12.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.604+0000 7fc0cc1ba640 1 --2- 172.21.15.44:0/2806443477 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0c41066c0 0x7fc0c4106aa0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0009900 tx=0x7fc0c002bc30 comp rx=0 tx=0).stop 2024-04-21T23:13:12.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.604+0000 7fc0cc1ba640 1 -- 172.21.15.44:0/2806443477 shutdown_connections 2024-04-21T23:13:12.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.604+0000 7fc0cc1ba640 1 --2- 172.21.15.44:0/2806443477 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0c4100890 0x7fc0c4100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.604+0000 7fc0cc1ba640 1 --2- 172.21.15.44:0/2806443477 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0c4101230 0x7fc0c410d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.605 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.604+0000 7fc0cc1ba640 1 --2- 172.21.15.44:0/2806443477 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0c41066c0 0x7fc0c4106aa0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.604+0000 7fc0cc1ba640 1 -- 172.21.15.44:0/2806443477 >> 172.21.15.44:0/2806443477 conn(0x7fc0c4077430 msgr2=0x7fc0c4077840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:12.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.604+0000 7fc0cc1ba640 1 -- 172.21.15.44:0/2806443477 shutdown_connections 2024-04-21T23:13:12.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.604+0000 7fc0cc1ba640 1 -- 172.21.15.44:0/2806443477 wait complete. 2024-04-21T23:13:12.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.604+0000 7fc0cc1ba640 1 Processor -- start 2024-04-21T23:13:12.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.604+0000 7fc0cc1ba640 1 -- start start 2024-04-21T23:13:12.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.605+0000 7fc0cc1ba640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0c4100890 0x7fc0c419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.605+0000 7fc0cc1ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0c4101230 0x7fc0c419a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.605+0000 7fc0cc1ba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0c41066c0 0x7fc0c419eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.606 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.605+0000 7fc0cc1ba640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c4111b60 con 0x7fc0c41066c0 2024-04-21T23:13:12.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.605+0000 7fc0cc1ba640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c41119e0 con 0x7fc0c4100890 2024-04-21T23:13:12.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.605+0000 7fc0cc1ba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c4111ce0 con 0x7fc0c4101230 2024-04-21T23:13:12.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.605+0000 7fc0c9f56640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0c4100890 0x7fc0c419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.605+0000 7fc0c9f56640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0c4100890 0x7fc0c419a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:43760/0 (socket says 172.21.15.44:43760) 2024-04-21T23:13:12.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.605+0000 7fc0c9f56640 1 -- 172.21.15.44:0/321099662 learned_addr learned my addr 172.21.15.44:0/321099662 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:12.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.605+0000 7fc0c9755640 1 --2- 172.21.15.44:0/321099662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0c4101230 0x7fc0c419a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.605+0000 7fc0ca757640 1 --2- 172.21.15.44:0/321099662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0c41066c0 0x7fc0c419eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.605+0000 7fc0c9f56640 1 -- 172.21.15.44:0/321099662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0c4101230 msgr2=0x7fc0c419a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:12.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.605+0000 7fc0c9f56640 1 --2- 172.21.15.44:0/321099662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0c4101230 0x7fc0c419a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.605+0000 7fc0c9f56640 1 -- 172.21.15.44:0/321099662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0c41066c0 msgr2=0x7fc0c419eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:12.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.605+0000 7fc0c9f56640 1 --2- 172.21.15.44:0/321099662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0c41066c0 0x7fc0c419eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.605+0000 7fc0c9f56640 1 -- 172.21.15.44:0/321099662 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c419f270 con 0x7fc0c4100890 2024-04-21T23:13:12.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.606+0000 7fc0c9f56640 1 --2- 172.21.15.44:0/321099662 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0c4100890 0x7fc0c419a280 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0006010 tx=0x7fc0c002cb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:12.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.606+0000 7fc0baffd640 1 -- 172.21.15.44:0/321099662 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c0041070 con 0x7fc0c4100890 2024-04-21T23:13:12.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.606+0000 7fc0baffd640 1 -- 172.21.15.44:0/321099662 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0c0004450 con 0x7fc0c4100890 2024-04-21T23:13:12.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.606+0000 7fc0cc1ba640 1 -- 172.21.15.44:0/321099662 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c419f500 con 0x7fc0c4100890 2024-04-21T23:13:12.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.606+0000 7fc0baffd640 1 -- 172.21.15.44:0/321099662 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c0003e00 con 0x7fc0c4100890 2024-04-21T23:13:12.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.606+0000 7fc0cc1ba640 1 -- 172.21.15.44:0/321099662 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c41a6df0 con 0x7fc0c4100890 2024-04-21T23:13:12.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.607+0000 7fc0cc1ba640 1 -- 172.21.15.44:0/321099662 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0c40685f0 con 0x7fc0c4100890 2024-04-21T23:13:12.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.608+0000 7fc0baffd640 1 -- 172.21.15.44:0/321099662 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc0c0044070 con 0x7fc0c4100890 2024-04-21T23:13:12.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.609+0000 7fc0baffd640 1 --2- 172.21.15.44:0/321099662 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc0b00a2d80 0x7fc0b00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.609+0000 7fc0c9755640 1 --2- 172.21.15.44:0/321099662 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc0b00a2d80 0x7fc0b00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.609+0000 7fc0baffd640 1 -- 172.21.15.44:0/321099662 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fc0c00f1170 con 0x7fc0c4100890 2024-04-21T23:13:12.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.609+0000 7fc0c9755640 1 --2- 172.21.15.44:0/321099662 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc0b00a2d80 0x7fc0b00a5240 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4077e10 tx=0x7fc0b400a2d0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:12.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.613+0000 7fc0baffd640 1 -- 172.21.15.44:0/321099662 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fc0c00bd4b0 con 0x7fc0c4100890 2024-04-21T23:13:12.698 INFO:tasks.workunit.client.0.smithi044.stdout: 1 464825 13.50 MB/sec execute 62 sec latency 32.359 ms 2024-04-21T23:13:12.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.859+0000 7fc0cc1ba640 1 -- 172.21.15.44:0/321099662 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0c4103940 con 0x7fc0c4100890 2024-04-21T23:13:12.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.860+0000 7fc0baffd640 1 -- 172.21.15.44:0/321099662 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11791 (secure 0 0 0) 0x7fc0c00c17b0 con 0x7fc0c4100890 2024-04-21T23:13:12.859 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:13:12.860 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:11:24.214378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25843":{"gid":25843,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3162569119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3162569119},{"type":"v1","addr":"172.21.15.69:6839","nonce":3162569119}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:13:12.860 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-04-21T23:13:12.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.862+0000 7fc0cc1ba640 1 -- 172.21.15.44:0/321099662 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc0b00a2d80 msgr2=0x7fc0b00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:12.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.862+0000 7fc0cc1ba640 1 --2- 172.21.15.44:0/321099662 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc0b00a2d80 0x7fc0b00a5240 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4077e10 tx=0x7fc0b400a2d0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.862+0000 7fc0cc1ba640 1 -- 172.21.15.44:0/321099662 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0c4100890 msgr2=0x7fc0c419a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:12.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.862+0000 7fc0cc1ba640 1 --2- 172.21.15.44:0/321099662 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0c4100890 0x7fc0c419a280 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0006010 tx=0x7fc0c002cb30 comp rx=0 tx=0).stop 2024-04-21T23:13:12.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.862+0000 7fc0cc1ba640 1 -- 172.21.15.44:0/321099662 shutdown_connections 2024-04-21T23:13:12.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.862+0000 7fc0cc1ba640 1 --2- 172.21.15.44:0/321099662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0c4101230 0x7fc0c419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.862+0000 7fc0cc1ba640 1 --2- 172.21.15.44:0/321099662 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc0b00a2d80 0x7fc0b00a5240 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.862+0000 7fc0cc1ba640 1 --2- 172.21.15.44:0/321099662 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc0c41066c0 0x7fc0c419eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.862+0000 7fc0cc1ba640 1 --2- 172.21.15.44:0/321099662 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc0c4100890 0x7fc0c419a280 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.862+0000 7fc0cc1ba640 1 -- 172.21.15.44:0/321099662 >> 172.21.15.44:0/321099662 conn(0x7fc0c4077430 msgr2=0x7fc0c410bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:12.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.862+0000 7fc0cc1ba640 1 -- 172.21.15.44:0/321099662 shutdown_connections 2024-04-21T23:13:12.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.862+0000 7fc0cc1ba640 1 -- 172.21.15.44:0/321099662 wait complete. 2024-04-21T23:13:12.874 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:13:12.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.984+0000 7fcfe066d640 1 -- 172.21.15.44:0/239948839 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfd8103360 msgr2=0x7fcfd810b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:12.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.984+0000 7fcfe066d640 1 --2- 172.21.15.44:0/239948839 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfd8103360 0x7fcfd810b7b0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009900 tx=0x7fcfc802bc30 comp rx=0 tx=0).stop 2024-04-21T23:13:12.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.985+0000 7fcfe066d640 1 -- 172.21.15.44:0/239948839 shutdown_connections 2024-04-21T23:13:12.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.985+0000 7fcfe066d640 1 --2- 172.21.15.44:0/239948839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfd810bec0 0x7fcfd810e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.985+0000 7fcfe066d640 1 --2- 172.21.15.44:0/239948839 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfd8103360 0x7fcfd810b7b0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.985+0000 7fcfe066d640 1 --2- 172.21.15.44:0/239948839 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfd81029b0 0x7fcfd8102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.985+0000 7fcfe066d640 1 -- 172.21.15.44:0/239948839 >> 172.21.15.44:0/239948839 conn(0x7fcfd8077760 msgr2=0x7fcfd8077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:12.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.985+0000 7fcfe066d640 1 -- 172.21.15.44:0/239948839 shutdown_connections 2024-04-21T23:13:12.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.985+0000 7fcfe066d640 1 -- 172.21.15.44:0/239948839 wait complete. 2024-04-21T23:13:12.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.985+0000 7fcfe066d640 1 Processor -- start 2024-04-21T23:13:12.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.985+0000 7fcfe066d640 1 -- start start 2024-04-21T23:13:12.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.985+0000 7fcfe066d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfd81029b0 0x7fcfd81982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.985+0000 7fcfe066d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfd810bec0 0x7fcfd8198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.985+0000 7fcfe066d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfd819cb30 0x7fcfd819cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.986+0000 7fcfe066d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd8111d00 con 0x7fcfd81029b0 2024-04-21T23:13:12.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.986+0000 7fcfe066d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd8111b80 con 0x7fcfd810bec0 2024-04-21T23:13:12.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.986+0000 7fcfe066d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd8111e80 con 0x7fcfd819cb30 2024-04-21T23:13:12.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.986+0000 7fcfde409640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfd81029b0 0x7fcfd81982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.986+0000 7fcfde409640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfd81029b0 0x7fcfd81982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44382/0 (socket says 172.21.15.44:44382) 2024-04-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.986+0000 7fcfde409640 1 -- 172.21.15.44:0/530140567 learned_addr learned my addr 172.21.15.44:0/530140567 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.986+0000 7fcfdec0a640 1 --2- 172.21.15.44:0/530140567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfd819cb30 0x7fcfd819cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.986+0000 7fcfddc08640 1 --2- 172.21.15.44:0/530140567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfd810bec0 0x7fcfd8198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.986+0000 7fcfde409640 1 -- 172.21.15.44:0/530140567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfd819cb30 msgr2=0x7fcfd819cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.986+0000 7fcfde409640 1 --2- 172.21.15.44:0/530140567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfd819cb30 0x7fcfd819cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.986+0000 7fcfde409640 1 -- 172.21.15.44:0/530140567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfd810bec0 msgr2=0x7fcfd8198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.986+0000 7fcfde409640 1 --2- 172.21.15.44:0/530140567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfd810bec0 0x7fcfd8198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.986+0000 7fcfde409640 1 -- 172.21.15.44:0/530140567 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd819d620 con 0x7fcfd81029b0 2024-04-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.986+0000 7fcfddc08640 1 --2- 172.21.15.44:0/530140567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfd810bec0 0x7fcfd8198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.986+0000 7fcfde409640 1 --2- 172.21.15.44:0/530140567 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfd81029b0 0x7fcfd81982c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400e9f0 tx=0x7fcfd400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.986+0000 7fcfcf7fe640 1 -- 172.21.15.44:0/530140567 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd400cd80 con 0x7fcfd81029b0 2024-04-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.986+0000 7fcfe066d640 1 -- 172.21.15.44:0/530140567 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd819d910 con 0x7fcfd81029b0 2024-04-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.986+0000 7fcfcf7fe640 1 -- 172.21.15.44:0/530140567 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfd4010910 con 0x7fcfd81029b0 2024-04-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.986+0000 7fcfcf7fe640 1 -- 172.21.15.44:0/530140567 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd40188f0 con 0x7fcfd81029b0 2024-04-21T23:13:12.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.986+0000 7fcfe066d640 1 -- 172.21.15.44:0/530140567 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd81a5120 con 0x7fcfd81029b0 2024-04-21T23:13:12.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.989+0000 7fcfcf7fe640 1 -- 172.21.15.44:0/530140567 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fcfd4010ac0 con 0x7fcfd81029b0 2024-04-21T23:13:12.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.989+0000 7fcfe324d740 1 Processor -- start 2024-04-21T23:13:12.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.989+0000 7fcfe324d740 1 -- start start 2024-04-21T23:13:12.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.989+0000 7fcfe324d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 0x55a842736de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.989+0000 7fcfe324d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a8427373b0 0x55a8427443f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.989+0000 7fcfe324d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a842744930 0x55a842746d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.989+0000 7fcfe324d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a84269b530 con 0x55a842744930 2024-04-21T23:13:12.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.989+0000 7fcfe324d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55a842643830 con 0x55a8427373b0 2024-04-21T23:13:12.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.989+0000 7fcfe324d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a8426974f0 con 0x55a842736a00 2024-04-21T23:13:12.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.990+0000 7fcfdec0a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 0x55a842736de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.990+0000 7fcfdec0a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 0x55a842736de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:36904/0 (socket says 172.21.15.44:36904) 2024-04-21T23:13:12.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.990+0000 7fcfdec0a640 1 -- 172.21.15.44:0/3857046777 learned_addr learned my addr 172.21.15.44:0/3857046777 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:12.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.990+0000 7fcfddc08640 1 --2- 172.21.15.44:0/3857046777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a842744930 0x55a842746d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.990+0000 7fcfde409640 1 --2- 172.21.15.44:0/3857046777 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a8427373b0 0x55a8427443f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.990+0000 7fcfdec0a640 1 -- 172.21.15.44:0/3857046777 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a8427373b0 msgr2=0x55a8427443f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:12.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.990+0000 7fcfdec0a640 1 --2- 172.21.15.44:0/3857046777 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a8427373b0 0x55a8427443f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.990+0000 7fcfdec0a640 1 -- 172.21.15.44:0/3857046777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a842744930 msgr2=0x55a842746d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:12.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.990+0000 7fcfdec0a640 1 --2- 172.21.15.44:0/3857046777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a842744930 0x55a842746d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.990+0000 7fcfdec0a640 1 -- 172.21.15.44:0/3857046777 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8426672c0 con 0x55a842736a00 2024-04-21T23:13:12.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.990+0000 7fcfcf7fe640 1 --2- 172.21.15.44:0/530140567 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcfc40a2d00 0x7fcfc40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.990+0000 7fcfde409640 1 --2- 172.21.15.44:0/3857046777 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a8427373b0 0x55a8427443f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:13:12.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.990+0000 7fcfcf7fe640 1 -- 172.21.15.44:0/530140567 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fcfd40cfbb0 con 0x7fcfd81029b0 2024-04-21T23:13:12.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.990+0000 7fcfde409640 1 --2- 172.21.15.44:0/530140567 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcfc40a2d00 0x7fcfc40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.990+0000 7fcfdec0a640 1 --2- 172.21.15.44:0/3857046777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 0x55a842736de0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0009f70 tx=0x7fcfd0010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a03042670013fbd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:12.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.991+0000 7fcfccbfe640 1 -- 172.21.15.44:0/3857046777 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0016070 con 0x55a842736a00 2024-04-21T23:13:12.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.991+0000 7fcfccbfe640 1 -- 172.21.15.44:0/3857046777 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfd0004d20 con 0x55a842736a00 2024-04-21T23:13:12.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.991+0000 7fcfccbfe640 1 -- 172.21.15.44:0/3857046777 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0012950 con 0x55a842736a00 2024-04-21T23:13:12.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.991+0000 7fcfe324d740 1 -- 172.21.15.44:0/3857046777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 msgr2=0x55a842736de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:12.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.991+0000 7fcfe324d740 1 --2- 172.21.15.44:0/3857046777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 0x55a842736de0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0009f70 tx=0x7fcfd0010800 comp rx=0 tx=0).stop 2024-04-21T23:13:12.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.991+0000 7fcfde409640 1 --2- 172.21.15.44:0/530140567 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcfc40a2d00 0x7fcfc40a51c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40cd010 tx=0x7fcfd401f000 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:12.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.991+0000 7fcfe324d740 1 -- 172.21.15.44:0/3857046777 shutdown_connections 2024-04-21T23:13:12.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.991+0000 7fcfe324d740 1 --2- 172.21.15.44:0/3857046777 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a842744930 0x55a842746d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.991+0000 7fcfe324d740 1 --2- 172.21.15.44:0/3857046777 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a8427373b0 0x55a8427443f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.991+0000 7fcfe324d740 1 --2- 172.21.15.44:0/3857046777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 0x55a842736de0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.991+0000 7fcfe324d740 1 -- 172.21.15.44:0/3857046777 >> 172.21.15.44:0/3857046777 conn(0x55a842731750 msgr2=0x55a842731b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:12.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.992+0000 7fcfe324d740 1 -- 172.21.15.44:0/3857046777 shutdown_connections 2024-04-21T23:13:12.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.992+0000 7fcfe324d740 1 -- 172.21.15.44:0/3857046777 wait complete. 2024-04-21T23:13:12.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.992+0000 7fcfe324d740 1 -- start start 2024-04-21T23:13:12.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.992+0000 7fcfe324d740 1 Processor -- start 2024-04-21T23:13:12.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.992+0000 7fcfe324d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 0x55a8427442a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.992+0000 7fcfe324d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 0x55a842741630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.992+0000 7fcfe324d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a842744930 0x55a842741b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.992+0000 7fcfe324d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a8427476f0 con 0x55a8427373b0 2024-04-21T23:13:12.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.992+0000 7fcfe324d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55a842656f10 con 0x55a842744930 2024-04-21T23:13:12.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.992+0000 7fcfe324d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a842747870 con 0x55a842736a00 2024-04-21T23:13:12.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.992+0000 7fcfe324d740 10 client.0 _ll_register_callbacks cb 0x55a8423aebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:12.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.992+0000 7fcfddc08640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 0x55a842741630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.992+0000 7fcfdec0a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 0x55a8427442a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.992+0000 7fcfddc08640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 0x55a842741630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44398/0 (socket says 172.21.15.44:44398) 2024-04-21T23:13:12.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.992+0000 7fcfddc08640 1 -- 172.21.15.44:0/2839659808 learned_addr learned my addr 172.21.15.44:0/2839659808 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:12.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.992+0000 7fcfdec0a640 1 --2- 172.21.15.44:0/2839659808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a842744930 0x55a842741b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.993+0000 7fcfddc08640 1 -- 172.21.15.44:0/2839659808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 msgr2=0x55a8427442a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:12.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.993+0000 7fcfddc08640 1 --2- 172.21.15.44:0/2839659808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 0x55a8427442a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.993+0000 7fcfddc08640 1 -- 172.21.15.44:0/2839659808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a842744930 msgr2=0x55a842741b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:12.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.993+0000 7fcfddc08640 1 --2- 172.21.15.44:0/2839659808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a842744930 0x55a842741b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:12.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.993+0000 7fcfddc08640 1 -- 172.21.15.44:0/2839659808 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a842628c80 con 0x55a8427373b0 2024-04-21T23:13:12.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.993+0000 7fcfddc08640 1 --2- 172.21.15.44:0/2839659808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 0x55a842741630 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8000c00 tx=0x7fcfc8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab97fd5460cb579 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:12.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.993+0000 7fcfc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:13:12.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.993+0000 7fcfc1ffb640 1 -- 172.21.15.44:0/2839659808 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc802c880 con 0x55a8427373b0 2024-04-21T23:13:12.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.993+0000 7fcfc1ffb640 1 -- 172.21.15.44:0/2839659808 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fcfc802ca20 con 0x55a8427373b0 2024-04-21T23:13:12.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.993+0000 7fcfe324d740 1 -- 172.21.15.44:0/2839659808 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfd0004d20 con 0x55a8427373b0 2024-04-21T23:13:12.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.993+0000 7fcfc1ffb640 1 -- 172.21.15.44:0/2839659808 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fcfc8053050 con 0x55a8427373b0 2024-04-21T23:13:12.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.993+0000 7fcfe324d740 10 client.16638 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:12.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.993+0000 7fcfe324d740 1 -- 172.21.15.44:0/2839659808 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a8423af3f0 con 0x55a8427373b0 2024-04-21T23:13:12.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.993+0000 7fcfc1ffb640 1 -- 172.21.15.44:0/2839659808 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfd0004d20 con 0x55a8427373b0 2024-04-21T23:13:12.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.994+0000 7fcfc1ffb640 1 -- 172.21.15.44:0/2839659808 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fcfc80036d0 con 0x55a8427373b0 2024-04-21T23:13:12.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.994+0000 7fcfe324d740 10 client.16638 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:12.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.994+0000 7fcfe324d740 10 client.16638 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:12.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.994+0000 7fcfe324d740 20 client.16638 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:12.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.994+0000 7fcfe324d740 1 --2- 172.21.15.44:0/2839659808 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a8427549e0 0x55a842756df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:12.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.994+0000 7fcfe324d740 4 client.16638 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:13:12.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.994+0000 7fcfe324d740 1 -- 172.21.15.44:0/2839659808 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8426a8a70 con 0x55a8427549e0 2024-04-21T23:13:12.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.994+0000 7fcfdec0a640 1 --2- 172.21.15.44:0/2839659808 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a8427549e0 0x55a842756df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:12.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.994+0000 7fcfc1ffb640 10 client.16638 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:12.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.994+0000 7fcfdec0a640 1 --2- 172.21.15.44:0/2839659808 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a8427549e0 0x55a842756df0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9e214958be5d349 server_cookie=27a191c6224d212f in_seq=0 out_seq=0 2024-04-21T23:13:13.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.995+0000 7fcfc1ffb640 1 -- 172.21.15.44:0/2839659808 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a8426a8a70 con 0x55a8427549e0 2024-04-21T23:13:13.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.995+0000 7fcfc1ffb640 10 client.16638 handle_command_reply: tid=0 2024-04-21T23:13:13.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.995+0000 7fcfe324d740 1 client.16638 shutdown 2024-04-21T23:13:13.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.995+0000 7fcfe324d740 1 -- 172.21.15.44:0/2839659808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 msgr2=0x55a842741630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:13.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.995+0000 7fcfe324d740 1 --2- 172.21.15.44:0/2839659808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 0x55a842741630 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8000c00 tx=0x7fcfc8004040 comp rx=0 tx=0).stop 2024-04-21T23:13:13.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.995+0000 7fcfe324d740 1 -- 172.21.15.44:0/2839659808 shutdown_connections 2024-04-21T23:13:13.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.995+0000 7fcfe324d740 1 --2- 172.21.15.44:0/2839659808 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a8427549e0 0x55a842756df0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.995+0000 7fcfe324d740 1 --2- 172.21.15.44:0/2839659808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a842744930 0x55a842741b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.995+0000 7fcfe324d740 1 --2- 172.21.15.44:0/2839659808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 0x55a842741630 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.995+0000 7fcfc1ffb640 0 client.16638 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:13.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.995+0000 7fcfe324d740 1 --2- 172.21.15.44:0/2839659808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 0x55a8427442a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.995+0000 7fcfe324d740 1 -- 172.21.15.44:0/2839659808 >> 172.21.15.44:0/2839659808 conn(0x55a842731750 msgr2=0x55a8427334a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:13.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.996+0000 7fcfe324d740 1 -- 172.21.15.44:0/2839659808 shutdown_connections 2024-04-21T23:13:13.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.996+0000 7fcfe324d740 1 -- 172.21.15.44:0/2839659808 wait complete. 2024-04-21T23:13:13.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:12.996+0000 7fcfe324d740 20 client.16638 trim_cache size 0 max 16384 2024-04-21T23:13:13.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.012+0000 7fcfe324d740 1 Processor -- start 2024-04-21T23:13:13.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.012+0000 7fcfe324d740 1 -- start start 2024-04-21T23:13:13.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.012+0000 7fcfe324d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 0x55a8427505f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:13.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.012+0000 7fcfe324d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 0x55a84273caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:13.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.012+0000 7fcfe324d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a842744930 0x55a842743d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:13.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.012+0000 7fcfe324d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81aa9c0 con 0x55a8427373b0 2024-04-21T23:13:13.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.012+0000 7fcfe324d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81ad240 con 0x55a842744930 2024-04-21T23:13:13.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.012+0000 7fcfe324d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81ab030 con 0x55a842736a00 2024-04-21T23:13:13.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.012+0000 7fcfddc08640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 0x55a84273caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:13.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.012+0000 7fcfddc08640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 0x55a84273caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44404/0 (socket says 172.21.15.44:44404) 2024-04-21T23:13:13.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.012+0000 7fcfdec0a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 0x55a8427505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:13.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.012+0000 7fcfddc08640 1 -- 172.21.15.44:0/1367765727 learned_addr learned my addr 172.21.15.44:0/1367765727 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:13.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.012+0000 7fcfddc08640 1 -- 172.21.15.44:0/1367765727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 msgr2=0x55a8427505f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:13.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.012+0000 7fcfddc08640 1 --2- 172.21.15.44:0/1367765727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 0x55a8427505f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.012+0000 7fcfddc08640 1 -- 172.21.15.44:0/1367765727 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a842744930 msgr2=0x55a842743d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:13.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.012+0000 7fcfddc08640 1 --2- 172.21.15.44:0/1367765727 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a842744930 0x55a842743d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.012+0000 7fcfddc08640 1 -- 172.21.15.44:0/1367765727 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8424797d0 con 0x55a8427373b0 2024-04-21T23:13:13.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.013+0000 7fcfddc08640 1 --2- 172.21.15.44:0/1367765727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 0x55a84273caa0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fcfc802c170 tx=0x7fcfc8009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3050198f743c86d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:13.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.013+0000 7fcfc3fff640 1 -- 172.21.15.44:0/1367765727 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8000e60 con 0x55a8427373b0 2024-04-21T23:13:13.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.013+0000 7fcfc3fff640 1 -- 172.21.15.44:0/1367765727 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfc8003e90 con 0x55a8427373b0 2024-04-21T23:13:13.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.013+0000 7fcfc3fff640 1 -- 172.21.15.44:0/1367765727 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8005150 con 0x55a8427373b0 2024-04-21T23:13:13.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.014+0000 7fcfe324d740 1 -- 172.21.15.44:0/1367765727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 msgr2=0x55a84273caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:13.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.014+0000 7fcfe324d740 1 --2- 172.21.15.44:0/1367765727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 0x55a84273caa0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fcfc802c170 tx=0x7fcfc8009600 comp rx=0 tx=0).stop 2024-04-21T23:13:13.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.014+0000 7fcfe324d740 1 -- 172.21.15.44:0/1367765727 shutdown_connections 2024-04-21T23:13:13.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.014+0000 7fcfe324d740 1 --2- 172.21.15.44:0/1367765727 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a842744930 0x55a842743d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.014+0000 7fcfe324d740 1 --2- 172.21.15.44:0/1367765727 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 0x55a84273caa0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.014+0000 7fcfe324d740 1 --2- 172.21.15.44:0/1367765727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 0x55a8427505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.014+0000 7fcfe324d740 1 -- 172.21.15.44:0/1367765727 >> 172.21.15.44:0/1367765727 conn(0x55a842731750 msgr2=0x55a84273c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:13.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.014+0000 7fcfe324d740 1 -- 172.21.15.44:0/1367765727 shutdown_connections 2024-04-21T23:13:13.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.014+0000 7fcfe324d740 1 -- 172.21.15.44:0/1367765727 wait complete. 2024-04-21T23:13:13.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfe324d740 1 -- start start 2024-04-21T23:13:13.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfe324d740 1 Processor -- start 2024-04-21T23:13:13.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfe324d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 0x55a8427b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:13.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfe324d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 0x55a8427b37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:13.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfe324d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a842744930 0x55a842747eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:13.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfe324d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81ab700 con 0x55a8427373b0 2024-04-21T23:13:13.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfe324d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81acb70 con 0x55a842744930 2024-04-21T23:13:13.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfe324d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81abdd0 con 0x55a842736a00 2024-04-21T23:13:13.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfe324d740 10 client.0 _ll_register_callbacks cb 0x7fcfc8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:13.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfddc08640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 0x55a8427b37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:13.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfddc08640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 0x55a8427b37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44406/0 (socket says 172.21.15.44:44406) 2024-04-21T23:13:13.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfddc08640 1 -- 172.21.15.44:0/3372285694 learned_addr learned my addr 172.21.15.44:0/3372285694 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:13.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfdec0a640 1 --2- 172.21.15.44:0/3372285694 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 0x55a8427b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:13.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfdec0a640 1 --2- 172.21.15.44:0/3372285694 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a842744930 0x55a842747eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:13.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfddc08640 1 -- 172.21.15.44:0/3372285694 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 msgr2=0x55a8427b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:13.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfddc08640 1 --2- 172.21.15.44:0/3372285694 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 0x55a8427b32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfddc08640 1 -- 172.21.15.44:0/3372285694 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a842744930 msgr2=0x55a842747eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:13.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfddc08640 1 --2- 172.21.15.44:0/3372285694 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a842744930 0x55a842747eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfddc08640 1 -- 172.21.15.44:0/3372285694 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8427485a0 con 0x55a8427373b0 2024-04-21T23:13:13.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfdec0a640 1 --2- 172.21.15.44:0/3372285694 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a842744930 0x55a842747eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:13:13.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfddc08640 1 --2- 172.21.15.44:0/3372285694 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 0x55a8427b37e0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8000c00 tx=0x7fcfc8005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e65021c994779f20 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:13.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.015+0000 7fcfc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:13:13.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.016+0000 7fcfc1ffb640 1 -- 172.21.15.44:0/3372285694 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc805b070 con 0x55a8427373b0 2024-04-21T23:13:13.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.016+0000 7fcfc1ffb640 1 -- 172.21.15.44:0/3372285694 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fcfc802ca90 con 0x55a8427373b0 2024-04-21T23:13:13.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.016+0000 7fcfe324d740 1 -- 172.21.15.44:0/3372285694 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfc8003e90 con 0x55a8427373b0 2024-04-21T23:13:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.016+0000 7fcfc1ffb640 1 -- 172.21.15.44:0/3372285694 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fcfc8004040 con 0x55a8427373b0 2024-04-21T23:13:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.016+0000 7fcfe324d740 10 client.16644 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.016+0000 7fcfe324d740 1 -- 172.21.15.44:0/3372285694 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a8427baa40 con 0x55a8427373b0 2024-04-21T23:13:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.016+0000 7fcfc1ffb640 1 -- 172.21.15.44:0/3372285694 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfc8003e90 con 0x55a8427373b0 2024-04-21T23:13:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.016+0000 7fcfc1ffb640 1 -- 172.21.15.44:0/3372285694 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fcfc804a580 con 0x55a8427373b0 2024-04-21T23:13:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.016+0000 7fcfe324d740 10 client.16644 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.016+0000 7fcfe324d740 10 client.16644 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.016+0000 7fcfe324d740 20 client.16644 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.016+0000 7fcfe324d740 1 --2- 172.21.15.44:0/3372285694 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a8427bb140 0x55a8427bd530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.016+0000 7fcfe324d740 4 client.16644 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:13:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.016+0000 7fcfe324d740 1 -- 172.21.15.44:0/3372285694 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a8426a2dc0 con 0x55a8427bb140 2024-04-21T23:13:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.016+0000 7fcfdec0a640 1 --2- 172.21.15.44:0/3372285694 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a8427bb140 0x55a8427bd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.017+0000 7fcfdec0a640 1 --2- 172.21.15.44:0/3372285694 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a8427bb140 0x55a8427bd530 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96beb5812a04221b server_cookie=77758227299398b3 in_seq=0 out_seq=0 2024-04-21T23:13:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.017+0000 7fcfc1ffb640 10 client.16644 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.017+0000 7fcfc1ffb640 1 -- 172.21.15.44:0/3372285694 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a8426a2dc0 con 0x55a8427bb140 2024-04-21T23:13:13.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.017+0000 7fcfc1ffb640 10 client.16644 handle_command_reply: tid=0 2024-04-21T23:13:13.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.017+0000 7fcfe324d740 1 client.16644 shutdown 2024-04-21T23:13:13.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.017+0000 7fcfe324d740 1 -- 172.21.15.44:0/3372285694 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 msgr2=0x55a8427b37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:13.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.017+0000 7fcfe324d740 1 --2- 172.21.15.44:0/3372285694 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 0x55a8427b37e0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8000c00 tx=0x7fcfc8005dc0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.017+0000 7fcfe324d740 1 -- 172.21.15.44:0/3372285694 shutdown_connections 2024-04-21T23:13:13.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.017+0000 7fcfe324d740 1 --2- 172.21.15.44:0/3372285694 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a8427bb140 0x55a8427bd530 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.017+0000 7fcfe324d740 1 --2- 172.21.15.44:0/3372285694 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a842744930 0x55a842747eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.017+0000 7fcfe324d740 1 --2- 172.21.15.44:0/3372285694 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a8427373b0 0x55a8427b37e0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.017+0000 7fcfe324d740 1 --2- 172.21.15.44:0/3372285694 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a842736a00 0x55a8427b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.017+0000 7fcfe324d740 1 -- 172.21.15.44:0/3372285694 >> 172.21.15.44:0/3372285694 conn(0x55a842731750 msgr2=0x55a8427333c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:13.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.017+0000 7fcfc1ffb640 0 client.16644 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:13.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.017+0000 7fcfe324d740 1 -- 172.21.15.44:0/3372285694 shutdown_connections 2024-04-21T23:13:13.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.017+0000 7fcfe324d740 1 -- 172.21.15.44:0/3372285694 wait complete. 2024-04-21T23:13:13.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.018+0000 7fcfe324d740 20 client.16644 trim_cache size 0 max 16384 2024-04-21T23:13:13.023 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-21T23:13:13.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.018+0000 7fcfe066d640 1 -- 172.21.15.44:0/530140567 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcfc40a2d00 msgr2=0x7fcfc40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:13.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.018+0000 7fcfe066d640 1 --2- 172.21.15.44:0/530140567 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcfc40a2d00 0x7fcfc40a51c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40cd010 tx=0x7fcfd401f000 comp rx=0 tx=0).stop 2024-04-21T23:13:13.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.018+0000 7fcfe066d640 1 -- 172.21.15.44:0/530140567 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfd81029b0 msgr2=0x7fcfd81982c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:13.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.018+0000 7fcfe066d640 1 --2- 172.21.15.44:0/530140567 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfd81029b0 0x7fcfd81982c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400e9f0 tx=0x7fcfd400eeb0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.018+0000 7fcfe066d640 1 -- 172.21.15.44:0/530140567 shutdown_connections 2024-04-21T23:13:13.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.018+0000 7fcfe066d640 1 --2- 172.21.15.44:0/530140567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfd819cb30 0x7fcfd819cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.018+0000 7fcfe066d640 1 --2- 172.21.15.44:0/530140567 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcfc40a2d00 0x7fcfc40a51c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.018+0000 7fcfe066d640 1 --2- 172.21.15.44:0/530140567 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcfd810bec0 0x7fcfd8198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.018+0000 7fcfe066d640 1 --2- 172.21.15.44:0/530140567 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcfd81029b0 0x7fcfd81982c0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:13.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.018+0000 7fcfe066d640 1 -- 172.21.15.44:0/530140567 >> 172.21.15.44:0/530140567 conn(0x7fcfd8077760 msgr2=0x7fcfd80feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:13.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.018+0000 7fcfe066d640 1 -- 172.21.15.44:0/530140567 shutdown_connections 2024-04-21T23:13:13.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:13.018+0000 7fcfe066d640 1 -- 172.21.15.44:0/530140567 wait complete. 2024-04-21T23:13:13.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:12 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/321099662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:13:13.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:12 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/321099662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:13:13.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:12 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/321099662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:13:13.698 INFO:tasks.workunit.client.0.smithi044.stdout: 1 467114 13.48 MB/sec execute 63 sec latency 28.318 ms 2024-04-21T23:13:14.033 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:13:14.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:13:14.033 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:13:14.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.145+0000 7f6cb8bd2640 1 -- 172.21.15.44:0/1657736055 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cb41029b0 msgr2=0x7f6cb4102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:14.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.145+0000 7f6cb8bd2640 1 --2- 172.21.15.44:0/1657736055 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cb41029b0 0x7f6cb4102d90 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4009900 tx=0x7f6ca402bcd0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.145+0000 7f6cb8bd2640 1 -- 172.21.15.44:0/1657736055 shutdown_connections 2024-04-21T23:13:14.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.145+0000 7f6cb8bd2640 1 --2- 172.21.15.44:0/1657736055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cb41032d0 0x7f6cb410b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.145+0000 7f6cb8bd2640 1 --2- 172.21.15.44:0/1657736055 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6cb410bcf0 0x7f6cb410e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.145+0000 7f6cb8bd2640 1 --2- 172.21.15.44:0/1657736055 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cb41029b0 0x7f6cb4102d90 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.145+0000 7f6cb8bd2640 1 -- 172.21.15.44:0/1657736055 >> 172.21.15.44:0/1657736055 conn(0x7f6cb40777f0 msgr2=0x7f6cb4077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:14.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.145+0000 7f6cb8bd2640 1 -- 172.21.15.44:0/1657736055 shutdown_connections 2024-04-21T23:13:14.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.145+0000 7f6cb8bd2640 1 -- 172.21.15.44:0/1657736055 wait complete. 2024-04-21T23:13:14.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.145+0000 7f6cb8bd2640 1 Processor -- start 2024-04-21T23:13:14.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.146+0000 7f6cb8bd2640 1 -- start start 2024-04-21T23:13:14.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.146+0000 7f6cb8bd2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cb41029b0 0x7f6cb4100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.146+0000 7f6cb8bd2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6cb41032d0 0x7f6cb4100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.146+0000 7f6cb8bd2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cb410bcf0 0x7f6cb4101080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.146+0000 7f6cb8bd2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb4111af0 con 0x7f6cb410bcf0 2024-04-21T23:13:14.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.146+0000 7f6cb8bd2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb4111970 con 0x7f6cb41032d0 2024-04-21T23:13:14.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.146+0000 7f6cb8bd2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb4111c70 con 0x7f6cb41029b0 2024-04-21T23:13:14.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.146+0000 7f6cb2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cb410bcf0 0x7f6cb4101080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.146+0000 7f6cabfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6cb41032d0 0x7f6cb4100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.146+0000 7f6cb2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cb410bcf0 0x7f6cb4101080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44432/0 (socket says 172.21.15.44:44432) 2024-04-21T23:13:14.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.146+0000 7f6cb2d9d640 1 -- 172.21.15.44:0/3475742001 learned_addr learned my addr 172.21.15.44:0/3475742001 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:14.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.146+0000 7f6cb259c640 1 --2- 172.21.15.44:0/3475742001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cb41029b0 0x7f6cb4100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.146+0000 7f6cb2d9d640 1 -- 172.21.15.44:0/3475742001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cb41029b0 msgr2=0x7f6cb4100600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:14.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.146+0000 7f6cb2d9d640 1 --2- 172.21.15.44:0/3475742001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cb41029b0 0x7f6cb4100600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.146+0000 7f6cb2d9d640 1 -- 172.21.15.44:0/3475742001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6cb41032d0 msgr2=0x7f6cb4100b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:14.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.146+0000 7f6cb2d9d640 1 --2- 172.21.15.44:0/3475742001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6cb41032d0 0x7f6cb4100b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.146+0000 7f6cb2d9d640 1 -- 172.21.15.44:0/3475742001 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cb41a65d0 con 0x7f6cb410bcf0 2024-04-21T23:13:14.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.146+0000 7f6cabfff640 1 --2- 172.21.15.44:0/3475742001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6cb41032d0 0x7f6cb4100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:14.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.147+0000 7f6cb2d9d640 1 --2- 172.21.15.44:0/3475742001 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cb410bcf0 0x7f6cb4101080 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f6cac00cab0 tx=0x7f6cac00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:14.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.147+0000 7f6cab7fe640 1 -- 172.21.15.44:0/3475742001 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cac00adc0 con 0x7f6cb410bcf0 2024-04-21T23:13:14.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.147+0000 7f6cab7fe640 1 -- 172.21.15.44:0/3475742001 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cac00f930 con 0x7f6cb410bcf0 2024-04-21T23:13:14.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.147+0000 7f6cb8bd2640 1 -- 172.21.15.44:0/3475742001 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cb41a68c0 con 0x7f6cb410bcf0 2024-04-21T23:13:14.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.147+0000 7f6cab7fe640 1 -- 172.21.15.44:0/3475742001 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cac018ae0 con 0x7f6cb410bcf0 2024-04-21T23:13:14.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.147+0000 7f6cb8bd2640 1 -- 172.21.15.44:0/3475742001 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cb41a6dc0 con 0x7f6cb410bcf0 2024-04-21T23:13:14.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.150+0000 7f6cab7fe640 1 -- 172.21.15.44:0/3475742001 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6cac00fae0 con 0x7f6cb410bcf0 2024-04-21T23:13:14.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.150+0000 7f6cbb7b7740 1 Processor -- start 2024-04-21T23:13:14.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.150+0000 7f6cbb7b7740 1 -- start start 2024-04-21T23:13:14.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.150+0000 7f6cbb7b7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f98f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.150+0000 7f6cbb7b7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f98fa40 0x556a2f98fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.150+0000 7f6cbb7b7740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f998db0 0x556a2f99b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.150+0000 7f6cbb7b7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x556a2f8f3170 con 0x556a2f990720 2024-04-21T23:13:14.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.150+0000 7f6cbb7b7740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x556a2f89b7c0 con 0x556a2f998db0 2024-04-21T23:13:14.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.150+0000 7f6cbb7b7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556a2f8ef130 con 0x556a2f98fa40 2024-04-21T23:13:14.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.150+0000 7f6cabfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f98f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.150+0000 7f6cb259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f98fa40 0x556a2f98fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.150+0000 7f6cabfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f98f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44434/0 (socket says 172.21.15.44:44434) 2024-04-21T23:13:14.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.150+0000 7f6cabfff640 1 -- 172.21.15.44:0/3647703161 learned_addr learned my addr 172.21.15.44:0/3647703161 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:14.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.150+0000 7f6cb259c640 1 --2- 172.21.15.44:0/3647703161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f98fa40 0x556a2f98fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:36976/0 (socket says 172.21.15.44:36976) 2024-04-21T23:13:14.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.150+0000 7f6cab7fe640 1 --2- 172.21.15.44:0/3475742001 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6ca00a2d00 0x7f6ca00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.151+0000 7f6cabfff640 1 --2- 172.21.15.44:0/3647703161 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f998db0 0x556a2f99b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.151+0000 7f6cb259c640 1 -- 172.21.15.44:0/3647703161 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f998db0 msgr2=0x556a2f99b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:14.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.151+0000 7f6cb259c640 1 --2- 172.21.15.44:0/3647703161 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f998db0 0x556a2f99b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.151+0000 7f6cb2d9d640 1 --2- 172.21.15.44:0/3475742001 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6ca00a2d00 0x7f6ca00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.151+0000 7f6cb259c640 1 -- 172.21.15.44:0/3647703161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 msgr2=0x556a2f98f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:14.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.151+0000 7f6cb259c640 1 --2- 172.21.15.44:0/3647703161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f98f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.151+0000 7f6cb259c640 1 -- 172.21.15.44:0/3647703161 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a2f8ace60 con 0x556a2f98fa40 2024-04-21T23:13:14.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.151+0000 7f6cab7fe640 1 -- 172.21.15.44:0/3475742001 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f6cac0cfdd0 con 0x7f6cb410bcf0 2024-04-21T23:13:14.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.151+0000 7f6cb259c640 1 --2- 172.21.15.44:0/3647703161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f98fa40 0x556a2f98fec0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4000c00 tx=0x7f6ca4005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e8927b7c2fa6d3a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:14.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.151+0000 7f6cb2d9d640 1 --2- 172.21.15.44:0/3475742001 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6ca00a2d00 0x7f6ca00a51c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6cac098920 tx=0x7f6cac0981f0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:14.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.151+0000 7f6ca8ff9640 1 -- 172.21.15.44:0/3647703161 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4042070 con 0x556a2f98fa40 2024-04-21T23:13:14.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.151+0000 7f6ca8ff9640 1 -- 172.21.15.44:0/3647703161 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ca402cb20 con 0x556a2f98fa40 2024-04-21T23:13:14.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.151+0000 7f6ca8ff9640 1 -- 172.21.15.44:0/3647703161 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4004f10 con 0x556a2f98fa40 2024-04-21T23:13:14.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.152+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/3647703161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f98fa40 msgr2=0x556a2f98fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:14.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.152+0000 7f6cbb7b7740 1 --2- 172.21.15.44:0/3647703161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f98fa40 0x556a2f98fec0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4000c00 tx=0x7f6ca4005d50 comp rx=0 tx=0).stop 2024-04-21T23:13:14.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.152+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/3647703161 shutdown_connections 2024-04-21T23:13:14.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.152+0000 7f6cbb7b7740 1 --2- 172.21.15.44:0/3647703161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f98fa40 0x556a2f98fec0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.152+0000 7f6cbb7b7740 1 --2- 172.21.15.44:0/3647703161 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f998db0 0x556a2f99b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.152+0000 7f6cbb7b7740 1 --2- 172.21.15.44:0/3647703161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f98f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.152+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/3647703161 >> 172.21.15.44:0/3647703161 conn(0x556a2f9564f0 msgr2=0x556a2f98a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:14.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.152+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/3647703161 shutdown_connections 2024-04-21T23:13:14.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.152+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/3647703161 wait complete. 2024-04-21T23:13:14.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.152+0000 7f6cbb7b7740 1 -- start start 2024-04-21T23:13:14.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.152+0000 7f6cbb7b7740 1 Processor -- start 2024-04-21T23:13:14.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6cbb7b7740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f98fa40 0x556a2f995e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6cbb7b7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f996350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6cbb7b7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f998db0 0x556a2f996890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6cbb7b7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x556a2f99bbc0 con 0x556a2f990720 2024-04-21T23:13:14.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6cbb7b7740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x556a2f8aed40 con 0x556a2f98fa40 2024-04-21T23:13:14.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6cbb7b7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556a2f99bd40 con 0x556a2f998db0 2024-04-21T23:13:14.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6cbb7b7740 10 client.0 _ll_register_callbacks cb 0x556a2f8ea820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:14.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6cb259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f996350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6cb259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f996350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44438/0 (socket says 172.21.15.44:44438) 2024-04-21T23:13:14.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6cb259c640 1 -- 172.21.15.44:0/321179824 learned_addr learned my addr 172.21.15.44:0/321179824 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:14.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6cabfff640 1 --2- 172.21.15.44:0/321179824 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f98fa40 0x556a2f995e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6cabfff640 1 --2- 172.21.15.44:0/321179824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f998db0 0x556a2f996890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6cb259c640 1 -- 172.21.15.44:0/321179824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f998db0 msgr2=0x556a2f996890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:14.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6cb259c640 1 --2- 172.21.15.44:0/321179824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f998db0 0x556a2f996890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6cb259c640 1 -- 172.21.15.44:0/321179824 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f98fa40 msgr2=0x556a2f995e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:14.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6cb259c640 1 --2- 172.21.15.44:0/321179824 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f98fa40 0x556a2f995e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6cb259c640 1 -- 172.21.15.44:0/321179824 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a2f6cf7e0 con 0x556a2f990720 2024-04-21T23:13:14.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6cabfff640 1 --2- 172.21.15.44:0/321179824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f998db0 0x556a2f996890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:13:14.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6cb259c640 1 --2- 172.21.15.44:0/321179824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f996350 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f6ca402bd10 tx=0x7f6ca4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25360b7de981343e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:14.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6c99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:13:14.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6c99ffb640 1 -- 172.21.15.44:0/321179824 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca404c070 con 0x556a2f990720 2024-04-21T23:13:14.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6c99ffb640 1 -- 172.21.15.44:0/321179824 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f6ca4005320 con 0x556a2f990720 2024-04-21T23:13:14.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.153+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/321179824 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ca402cb20 con 0x556a2f990720 2024-04-21T23:13:14.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.154+0000 7f6c99ffb640 1 -- 172.21.15.44:0/321179824 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6ca4058050 con 0x556a2f990720 2024-04-21T23:13:14.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.154+0000 7f6cbb7b7740 10 client.16656 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:14.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.154+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/321179824 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556a2f72ff70 con 0x556a2f990720 2024-04-21T23:13:14.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.154+0000 7f6c99ffb640 1 -- 172.21.15.44:0/321179824 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ca402cb20 con 0x556a2f990720 2024-04-21T23:13:14.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.154+0000 7f6c99ffb640 1 -- 172.21.15.44:0/321179824 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f6ca4034440 con 0x556a2f990720 2024-04-21T23:13:14.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.154+0000 7f6cbb7b7740 10 client.16656 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:14.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.154+0000 7f6cbb7b7740 10 client.16656 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:14.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.154+0000 7f6cbb7b7740 20 client.16656 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:14.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.154+0000 7f6cbb7b7740 1 --2- 172.21.15.44:0/321179824 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556a2f9a8d50 0x556a2f9ab140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.154+0000 7f6cbb7b7740 4 client.16656 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:13:14.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.154+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/321179824 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a2f8c7450 con 0x556a2f9a8d50 2024-04-21T23:13:14.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.154+0000 7f6cabfff640 1 --2- 172.21.15.44:0/321179824 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556a2f9a8d50 0x556a2f9ab140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.155+0000 7f6cabfff640 1 --2- 172.21.15.44:0/321179824 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556a2f9a8d50 0x556a2f9ab140 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=704e72c51b69b763 server_cookie=5c591a87ca56a8a3 in_seq=0 out_seq=0 2024-04-21T23:13:14.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.155+0000 7f6c99ffb640 10 client.16656 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:14.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.156+0000 7f6c99ffb640 1 -- 172.21.15.44:0/321179824 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556a2f8c7450 con 0x556a2f9a8d50 2024-04-21T23:13:14.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.156+0000 7f6c99ffb640 10 client.16656 handle_command_reply: tid=0 2024-04-21T23:13:14.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.156+0000 7f6cbb7b7740 1 client.16656 shutdown 2024-04-21T23:13:14.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.156+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/321179824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 msgr2=0x556a2f996350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:14.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.156+0000 7f6cbb7b7740 1 --2- 172.21.15.44:0/321179824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f996350 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f6ca402bd10 tx=0x7f6ca4004740 comp rx=0 tx=0).stop 2024-04-21T23:13:14.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.156+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/321179824 shutdown_connections 2024-04-21T23:13:14.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.156+0000 7f6cbb7b7740 1 --2- 172.21.15.44:0/321179824 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556a2f9a8d50 0x556a2f9ab140 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.156+0000 7f6cbb7b7740 1 --2- 172.21.15.44:0/321179824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f998db0 0x556a2f996890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.156+0000 7f6c99ffb640 0 client.16656 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:14.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.156+0000 7f6cbb7b7740 1 --2- 172.21.15.44:0/321179824 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f996350 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.156+0000 7f6cbb7b7740 1 --2- 172.21.15.44:0/321179824 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f98fa40 0x556a2f995e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.156+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/321179824 >> 172.21.15.44:0/321179824 conn(0x556a2f9564f0 msgr2=0x556a2f98c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:14.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.156+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/321179824 shutdown_connections 2024-04-21T23:13:14.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.157+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/321179824 wait complete. 2024-04-21T23:13:14.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.157+0000 7f6cbb7b7740 20 client.16656 trim_cache size 0 max 16384 2024-04-21T23:13:14.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.177+0000 7f6cbb7b7740 1 Processor -- start 2024-04-21T23:13:14.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.177+0000 7f6cbb7b7740 1 -- start start 2024-04-21T23:13:14.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.177+0000 7f6cbb7b7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f98fa40 0x556a2f998540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.177+0000 7f6cbb7b7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f9a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.177+0000 7f6cbb7b7740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f998db0 0x556a2f9ae160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.177+0000 7f6cbb7b7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb41aed20 con 0x556a2f990720 2024-04-21T23:13:14.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.177+0000 7f6cbb7b7740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb41b15a0 con 0x556a2f998db0 2024-04-21T23:13:14.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.177+0000 7f6cbb7b7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb41af390 con 0x556a2f98fa40 2024-04-21T23:13:14.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.177+0000 7f6cb259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f9a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.177+0000 7f6cb259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f9a46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44454/0 (socket says 172.21.15.44:44454) 2024-04-21T23:13:14.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.177+0000 7f6cb259c640 1 -- 172.21.15.44:0/1279075986 learned_addr learned my addr 172.21.15.44:0/1279075986 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:14.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.177+0000 7f6cabfff640 1 --2- 172.21.15.44:0/1279075986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f98fa40 0x556a2f998540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.178+0000 7f6cabfff640 1 --2- 172.21.15.44:0/1279075986 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f998db0 0x556a2f9ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.178+0000 7f6cb259c640 1 -- 172.21.15.44:0/1279075986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f98fa40 msgr2=0x556a2f998540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:14.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.178+0000 7f6cb259c640 1 --2- 172.21.15.44:0/1279075986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f98fa40 0x556a2f998540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.178+0000 7f6cb259c640 1 -- 172.21.15.44:0/1279075986 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f998db0 msgr2=0x556a2f9ae160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:14.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.178+0000 7f6cb259c640 1 --2- 172.21.15.44:0/1279075986 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f998db0 0x556a2f9ae160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.178+0000 7f6cb259c640 1 -- 172.21.15.44:0/1279075986 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a2f5c7170 con 0x556a2f990720 2024-04-21T23:13:14.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.178+0000 7f6cb259c640 1 --2- 172.21.15.44:0/1279075986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f9a46e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4006010 tx=0x7f6ca4005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1f8472b7e6f5f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:14.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.178+0000 7f6c9bfff640 1 -- 172.21.15.44:0/1279075986 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca404c070 con 0x556a2f990720 2024-04-21T23:13:14.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.178+0000 7f6c9bfff640 1 -- 172.21.15.44:0/1279075986 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ca4005850 con 0x556a2f990720 2024-04-21T23:13:14.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.178+0000 7f6c9bfff640 1 -- 172.21.15.44:0/1279075986 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4042040 con 0x556a2f990720 2024-04-21T23:13:14.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.179+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/1279075986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 msgr2=0x556a2f9a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:14.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.179+0000 7f6cbb7b7740 1 --2- 172.21.15.44:0/1279075986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f9a46e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4006010 tx=0x7f6ca4005e40 comp rx=0 tx=0).stop 2024-04-21T23:13:14.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.179+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/1279075986 shutdown_connections 2024-04-21T23:13:14.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.179+0000 7f6cbb7b7740 1 --2- 172.21.15.44:0/1279075986 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f998db0 0x556a2f9ae160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.179+0000 7f6cbb7b7740 1 --2- 172.21.15.44:0/1279075986 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f9a46e0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.179+0000 7f6cbb7b7740 1 --2- 172.21.15.44:0/1279075986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f98fa40 0x556a2f998540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.179+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/1279075986 >> 172.21.15.44:0/1279075986 conn(0x556a2f9564f0 msgr2=0x556a2f9a3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:14.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.179+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/1279075986 shutdown_connections 2024-04-21T23:13:14.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.179+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/1279075986 wait complete. 2024-04-21T23:13:14.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.179+0000 7f6cbb7b7740 1 -- start start 2024-04-21T23:13:14.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.179+0000 7f6cbb7b7740 1 Processor -- start 2024-04-21T23:13:14.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.180+0000 7f6cbb7b7740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f98fa40 0x556a2f9ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.180+0000 7f6cbb7b7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f9ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.180+0000 7f6cbb7b7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f998db0 0x556a2f9ab2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.180+0000 7f6cbb7b7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb41afa60 con 0x556a2f990720 2024-04-21T23:13:14.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.180+0000 7f6cbb7b7740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb41b0ed0 con 0x556a2f98fa40 2024-04-21T23:13:14.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.180+0000 7f6cbb7b7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb41b0130 con 0x556a2f998db0 2024-04-21T23:13:14.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.180+0000 7f6cbb7b7740 10 client.0 _ll_register_callbacks cb 0x7f6c9c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:14.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.180+0000 7f6cabfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f98fa40 0x556a2f9ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.180+0000 7f6cabfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f98fa40 0x556a2f9ae170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:43854/0 (socket says 172.21.15.44:43854) 2024-04-21T23:13:14.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.180+0000 7f6cb259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f9ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.180+0000 7f6cabfff640 1 -- 172.21.15.44:0/373219916 learned_addr learned my addr 172.21.15.44:0/373219916 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:14.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.180+0000 7f6cabfff640 1 --2- 172.21.15.44:0/373219916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f998db0 0x556a2f9ab2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.180+0000 7f6cabfff640 1 -- 172.21.15.44:0/373219916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f998db0 msgr2=0x556a2f9ab2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:14.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.180+0000 7f6cabfff640 1 --2- 172.21.15.44:0/373219916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f998db0 0x556a2f9ab2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.180+0000 7f6cabfff640 1 -- 172.21.15.44:0/373219916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 msgr2=0x556a2f9ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:14.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.180+0000 7f6cabfff640 1 --2- 172.21.15.44:0/373219916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f9ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.180+0000 7f6cabfff640 1 -- 172.21.15.44:0/373219916 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a2f9abb20 con 0x556a2f98fa40 2024-04-21T23:13:14.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.180+0000 7f6cabfff640 1 --2- 172.21.15.44:0/373219916 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f98fa40 0x556a2f9ae170 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x556a2f99be00 tx=0x7f6ca402c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c29500a47e7e7ab server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:14.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.180+0000 7f6c99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:13:14.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.181+0000 7f6c99ffb640 1 -- 172.21.15.44:0/373219916 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c028030 con 0x556a2f98fa40 2024-04-21T23:13:14.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.181+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/373219916 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ca4005850 con 0x556a2f98fa40 2024-04-21T23:13:14.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.181+0000 7f6c99ffb640 1 -- 172.21.15.44:0/373219916 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f6c9c01c040 con 0x556a2f98fa40 2024-04-21T23:13:14.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.181+0000 7f6c99ffb640 1 -- 172.21.15.44:0/373219916 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6c9c0036b0 con 0x556a2f98fa40 2024-04-21T23:13:14.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.181+0000 7f6cbb7b7740 10 client.26228 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:14.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.181+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/373219916 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556a2fa0fc00 con 0x556a2f98fa40 2024-04-21T23:13:14.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.181+0000 7f6c99ffb640 1 -- 172.21.15.44:0/373219916 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ca4005850 con 0x556a2f98fa40 2024-04-21T23:13:14.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.181+0000 7f6c99ffb640 1 -- 172.21.15.44:0/373219916 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f6c9c0245a0 con 0x556a2f98fa40 2024-04-21T23:13:14.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.181+0000 7f6cbb7b7740 10 client.26228 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:14.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.181+0000 7f6cbb7b7740 10 client.26228 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:14.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.181+0000 7f6cbb7b7740 20 client.26228 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:14.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.181+0000 7f6cbb7b7740 1 --2- 172.21.15.44:0/373219916 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556a2fa10300 0x556a2fa12720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.181+0000 7f6cbb7b7740 4 client.26228 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:13:14.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.181+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/373219916 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556a2f9006b0 con 0x556a2fa10300 2024-04-21T23:13:14.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.181+0000 7f6cb259c640 1 --2- 172.21.15.44:0/373219916 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556a2fa10300 0x556a2fa12720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.182+0000 7f6cb259c640 1 --2- 172.21.15.44:0/373219916 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556a2fa10300 0x556a2fa12720 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c73d72cc89853179 server_cookie=4d554ae90000c8d in_seq=0 out_seq=0 2024-04-21T23:13:14.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.182+0000 7f6c99ffb640 10 client.26228 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:14.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.183+0000 7f6c99ffb640 1 -- 172.21.15.44:0/373219916 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556a2f9006b0 con 0x556a2fa10300 2024-04-21T23:13:14.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.183+0000 7f6c99ffb640 10 client.26228 handle_command_reply: tid=0 2024-04-21T23:13:14.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.183+0000 7f6cbb7b7740 1 client.26228 shutdown 2024-04-21T23:13:14.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.183+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/373219916 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f98fa40 msgr2=0x556a2f9ae170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:14.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.183+0000 7f6cbb7b7740 1 --2- 172.21.15.44:0/373219916 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f98fa40 0x556a2f9ae170 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x556a2f99be00 tx=0x7f6ca402c920 comp rx=0 tx=0).stop 2024-04-21T23:13:14.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.183+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/373219916 shutdown_connections 2024-04-21T23:13:14.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.183+0000 7f6cbb7b7740 1 --2- 172.21.15.44:0/373219916 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556a2fa10300 0x556a2fa12720 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.183+0000 7f6cbb7b7740 1 --2- 172.21.15.44:0/373219916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a2f998db0 0x556a2f9ab2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.183+0000 7f6cbb7b7740 1 --2- 172.21.15.44:0/373219916 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556a2f990720 0x556a2f9ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.183+0000 7f6cbb7b7740 1 --2- 172.21.15.44:0/373219916 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556a2f98fa40 0x556a2f9ae170 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.183+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/373219916 >> 172.21.15.44:0/373219916 conn(0x556a2f9564f0 msgr2=0x556a2f9a4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:14.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.183+0000 7f6c99ffb640 0 client.26228 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:14.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.183+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/373219916 shutdown_connections 2024-04-21T23:13:14.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.183+0000 7f6cbb7b7740 1 -- 172.21.15.44:0/373219916 wait complete. 2024-04-21T23:13:14.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.183+0000 7f6cbb7b7740 20 client.26228 trim_cache size 0 max 16384 2024-04-21T23:13:14.189 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:13:14.189 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-21T23:13:14.189 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "370bb19c-895f-4047-812f-845bf1847d16", 2024-04-21T23:13:14.189 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-21T23:13:14.189 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:13:14.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.184+0000 7f6cb8bd2640 1 -- 172.21.15.44:0/3475742001 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6ca00a2d00 msgr2=0x7f6ca00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:14.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.184+0000 7f6cb8bd2640 1 --2- 172.21.15.44:0/3475742001 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6ca00a2d00 0x7f6ca00a51c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6cac098920 tx=0x7f6cac0981f0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.184+0000 7f6cb8bd2640 1 -- 172.21.15.44:0/3475742001 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cb410bcf0 msgr2=0x7f6cb4101080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:14.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.184+0000 7f6cb8bd2640 1 --2- 172.21.15.44:0/3475742001 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cb410bcf0 0x7f6cb4101080 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f6cac00cab0 tx=0x7f6cac00cf70 comp rx=0 tx=0).stop 2024-04-21T23:13:14.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.184+0000 7f6cb8bd2640 1 -- 172.21.15.44:0/3475742001 shutdown_connections 2024-04-21T23:13:14.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.184+0000 7f6cb8bd2640 1 --2- 172.21.15.44:0/3475742001 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6ca00a2d00 0x7f6ca00a51c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.184+0000 7f6cb8bd2640 1 --2- 172.21.15.44:0/3475742001 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6cb410bcf0 0x7f6cb4101080 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.184+0000 7f6cb8bd2640 1 --2- 172.21.15.44:0/3475742001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6cb41032d0 0x7f6cb4100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.184+0000 7f6cb8bd2640 1 --2- 172.21.15.44:0/3475742001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cb41029b0 0x7f6cb4100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.184+0000 7f6cb8bd2640 1 -- 172.21.15.44:0/3475742001 >> 172.21.15.44:0/3475742001 conn(0x7f6cb40777f0 msgr2=0x7f6cb40fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:14.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.184+0000 7f6cb8bd2640 1 -- 172.21.15.44:0/3475742001 shutdown_connections 2024-04-21T23:13:14.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.184+0000 7f6cb8bd2640 1 -- 172.21.15.44:0/3475742001 wait complete. 2024-04-21T23:13:14.191 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:13 smithi044 ceph-mon[27934]: pgmap v136: 129 pgs: 129 active+clean; 56 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.0 MiB/s wr, 713 op/s 2024-04-21T23:13:14.195 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:13:14.195 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:13:14.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:13 smithi069 ceph-mon[31816]: pgmap v136: 129 pgs: 129 active+clean; 56 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.0 MiB/s wr, 713 op/s 2024-04-21T23:13:14.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:13 smithi055 ceph-mon[31117]: pgmap v136: 129 pgs: 129 active+clean; 56 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.0 MiB/s wr, 713 op/s 2024-04-21T23:13:14.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.309+0000 7f1c7524e640 1 -- 172.21.15.44:0/591258873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c701029b0 msgr2=0x7f1c70102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:14.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.309+0000 7f1c7524e640 1 --2- 172.21.15.44:0/591258873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c701029b0 0x7f1c70102d90 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f1c60009900 tx=0x7f1c6002bcd0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.309+0000 7f1c7524e640 1 -- 172.21.15.44:0/591258873 shutdown_connections 2024-04-21T23:13:14.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.309+0000 7f1c7524e640 1 --2- 172.21.15.44:0/591258873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c7010bcf0 0x7f1c7010e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.309+0000 7f1c7524e640 1 --2- 172.21.15.44:0/591258873 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c701032d0 0x7f1c7010b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.309+0000 7f1c7524e640 1 --2- 172.21.15.44:0/591258873 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c701029b0 0x7f1c70102d90 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.309+0000 7f1c7524e640 1 -- 172.21.15.44:0/591258873 >> 172.21.15.44:0/591258873 conn(0x7f1c700777f0 msgr2=0x7f1c70077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:14.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.309+0000 7f1c7524e640 1 -- 172.21.15.44:0/591258873 shutdown_connections 2024-04-21T23:13:14.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.309+0000 7f1c7524e640 1 -- 172.21.15.44:0/591258873 wait complete. 2024-04-21T23:13:14.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.310+0000 7f1c7524e640 1 Processor -- start 2024-04-21T23:13:14.310 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.310+0000 7f1c7524e640 1 -- start start 2024-04-21T23:13:14.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.310+0000 7f1c7524e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c701029b0 0x7f1c70100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.310+0000 7f1c7524e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c701032d0 0x7f1c70100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.310+0000 7f1c7524e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c7010bcf0 0x7f1c70101080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.312 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.310+0000 7f1c7524e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1c70111af0 con 0x7f1c7010bcf0 2024-04-21T23:13:14.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.310+0000 7f1c7524e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1c70111970 con 0x7f1c701032d0 2024-04-21T23:13:14.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.310+0000 7f1c7524e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1c70111c70 con 0x7f1c701029b0 2024-04-21T23:13:14.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.310+0000 7f1c6ed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c701029b0 0x7f1c70100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.310+0000 7f1c6f59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c7010bcf0 0x7f1c70101080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.310+0000 7f1c6e59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c701032d0 0x7f1c70100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.310+0000 7f1c6ed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c701029b0 0x7f1c70100600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:37020/0 (socket says 172.21.15.44:37020) 2024-04-21T23:13:14.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.310+0000 7f1c6ed9d640 1 -- 172.21.15.44:0/3174478868 learned_addr learned my addr 172.21.15.44:0/3174478868 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:14.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.311+0000 7f1c6e59c640 1 -- 172.21.15.44:0/3174478868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c701029b0 msgr2=0x7f1c70100600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:14.313 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.311+0000 7f1c6e59c640 1 --2- 172.21.15.44:0/3174478868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c701029b0 0x7f1c70100600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.311+0000 7f1c6e59c640 1 -- 172.21.15.44:0/3174478868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c7010bcf0 msgr2=0x7f1c70101080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:14.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.311+0000 7f1c6e59c640 1 --2- 172.21.15.44:0/3174478868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c7010bcf0 0x7f1c70101080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.311+0000 7f1c6e59c640 1 -- 172.21.15.44:0/3174478868 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c701a6470 con 0x7f1c701032d0 2024-04-21T23:13:14.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.311+0000 7f1c6e59c640 1 --2- 172.21.15.44:0/3174478868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c701032d0 0x7f1c70100b40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1c6400e9f0 tx=0x7f1c6400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:14.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.311+0000 7f1c57fff640 1 -- 172.21.15.44:0/3174478868 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6400ccc0 con 0x7f1c701032d0 2024-04-21T23:13:14.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.311+0000 7f1c7524e640 1 -- 172.21.15.44:0/3174478868 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c701a6760 con 0x7f1c701032d0 2024-04-21T23:13:14.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.311+0000 7f1c57fff640 1 -- 172.21.15.44:0/3174478868 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c64010910 con 0x7f1c701032d0 2024-04-21T23:13:14.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.311+0000 7f1c57fff640 1 -- 172.21.15.44:0/3174478868 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c64018960 con 0x7f1c701032d0 2024-04-21T23:13:14.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.311+0000 7f1c7524e640 1 -- 172.21.15.44:0/3174478868 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c701a6cb0 con 0x7f1c701032d0 2024-04-21T23:13:14.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.314+0000 7f1c57fff640 1 -- 172.21.15.44:0/3174478868 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1c64010ac0 con 0x7f1c701032d0 2024-04-21T23:13:14.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.314+0000 7f1c77e38740 1 Processor -- start 2024-04-21T23:13:14.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.314+0000 7f1c77e38740 1 -- start start 2024-04-21T23:13:14.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c77e38740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c3f30 0x55fdff1c4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c77e38740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1c4850 0x55fdff1ccef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c77e38740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1cd430 0x55fdff1cf820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.321 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c77e38740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55fdff128560 con 0x55fdff1c3f30 2024-04-21T23:13:14.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c77e38740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55fdff0d0730 con 0x55fdff1cd430 2024-04-21T23:13:14.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c77e38740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fdff124520 con 0x55fdff1c4850 2024-04-21T23:13:14.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c6f59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c3f30 0x55fdff1c4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c6f59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c3f30 0x55fdff1c4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44502/0 (socket says 172.21.15.44:44502) 2024-04-21T23:13:14.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c6f59e640 1 -- 172.21.15.44:0/1373086854 learned_addr learned my addr 172.21.15.44:0/1373086854 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:14.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c6e59c640 1 --2- 172.21.15.44:0/1373086854 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1cd430 0x55fdff1cf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c6ed9d640 1 --2- 172.21.15.44:0/1373086854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1c4850 0x55fdff1ccef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c57fff640 1 --2- 172.21.15.44:0/3174478868 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1c580a2d30 0x7f1c580a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c6f59e640 1 -- 172.21.15.44:0/1373086854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1c4850 msgr2=0x55fdff1ccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:14.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c6f59e640 1 --2- 172.21.15.44:0/1373086854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1c4850 0x55fdff1ccef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c6f59e640 1 -- 172.21.15.44:0/1373086854 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1cd430 msgr2=0x55fdff1cf820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:14.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c6f59e640 1 --2- 172.21.15.44:0/1373086854 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1cd430 0x55fdff1cf820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c6f59e640 1 -- 172.21.15.44:0/1373086854 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdfef067d0 con 0x55fdff1c3f30 2024-04-21T23:13:14.322 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c57fff640 1 -- 172.21.15.44:0/3174478868 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f1c640cfb50 con 0x7f1c701032d0 2024-04-21T23:13:14.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c6f59e640 1 --2- 172.21.15.44:0/3174478868 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1c580a2d30 0x7f1c580a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c6f59e640 1 --2- 172.21.15.44:0/1373086854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c3f30 0x55fdff1c4310 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f1c68009f60 tx=0x7f1c68009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e181bb9f12865269 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:14.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.315+0000 7f1c557fa640 1 -- 172.21.15.44:0/1373086854 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c68016070 con 0x55fdff1c3f30 2024-04-21T23:13:14.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.316+0000 7f1c557fa640 1 -- 172.21.15.44:0/1373086854 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c6801c4c0 con 0x55fdff1c3f30 2024-04-21T23:13:14.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.316+0000 7f1c557fa640 1 -- 172.21.15.44:0/1373086854 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6801b700 con 0x55fdff1c3f30 2024-04-21T23:13:14.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.316+0000 7f1c77e38740 1 -- 172.21.15.44:0/1373086854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c3f30 msgr2=0x55fdff1c4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:14.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.316+0000 7f1c6f59e640 1 --2- 172.21.15.44:0/3174478868 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1c580a2d30 0x7f1c580a51f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1c6801c000 tx=0x7f1c6801bed0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:14.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.316+0000 7f1c77e38740 1 --2- 172.21.15.44:0/1373086854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c3f30 0x55fdff1c4310 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f1c68009f60 tx=0x7f1c68009f90 comp rx=0 tx=0).stop 2024-04-21T23:13:14.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.316+0000 7f1c77e38740 1 -- 172.21.15.44:0/1373086854 shutdown_connections 2024-04-21T23:13:14.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.316+0000 7f1c77e38740 1 --2- 172.21.15.44:0/1373086854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1c4850 0x55fdff1ccef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.316+0000 7f1c77e38740 1 --2- 172.21.15.44:0/1373086854 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1cd430 0x55fdff1cf820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.316+0000 7f1c77e38740 1 --2- 172.21.15.44:0/1373086854 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c3f30 0x55fdff1c4310 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.316+0000 7f1c77e38740 1 -- 172.21.15.44:0/1373086854 >> 172.21.15.44:0/1373086854 conn(0x55fdff1be660 msgr2=0x55fdff1bea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:14.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.316+0000 7f1c77e38740 1 -- 172.21.15.44:0/1373086854 shutdown_connections 2024-04-21T23:13:14.323 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.316+0000 7f1c77e38740 1 -- 172.21.15.44:0/1373086854 wait complete. 2024-04-21T23:13:14.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.317+0000 7f1c77e38740 1 -- start start 2024-04-21T23:13:14.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.317+0000 7f1c77e38740 1 Processor -- start 2024-04-21T23:13:14.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.317+0000 7f1c77e38740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1c3f30 0x55fdff1ca420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.317+0000 7f1c77e38740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c4850 0x55fdff1ca960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.317+0000 7f1c77e38740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1cd430 0x55fdff1caea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.317+0000 7f1c77e38740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55fdff1d0220 con 0x55fdff1c4850 2024-04-21T23:13:14.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.317+0000 7f1c77e38740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55fdff0e3e00 con 0x55fdff1cd430 2024-04-21T23:13:14.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.317+0000 7f1c77e38740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fdff1d03a0 con 0x55fdff1c3f30 2024-04-21T23:13:14.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.317+0000 7f1c77e38740 10 client.0 _ll_register_callbacks cb 0x55fdff11eeb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:14.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.317+0000 7f1c6e59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c4850 0x55fdff1ca960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.317+0000 7f1c6ed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1c3f30 0x55fdff1ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.317+0000 7f1c6e59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c4850 0x55fdff1ca960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44506/0 (socket says 172.21.15.44:44506) 2024-04-21T23:13:14.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.317+0000 7f1c6e59c640 1 -- 172.21.15.44:0/300561902 learned_addr learned my addr 172.21.15.44:0/300561902 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:14.324 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.317+0000 7f1c6f59e640 1 --2- 172.21.15.44:0/300561902 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1cd430 0x55fdff1caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.317+0000 7f1c6ed9d640 1 -- 172.21.15.44:0/300561902 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1cd430 msgr2=0x55fdff1caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:14.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.317+0000 7f1c6ed9d640 1 --2- 172.21.15.44:0/300561902 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1cd430 0x55fdff1caea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.317+0000 7f1c6ed9d640 1 -- 172.21.15.44:0/300561902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c4850 msgr2=0x55fdff1ca960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:14.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.317+0000 7f1c6ed9d640 1 --2- 172.21.15.44:0/300561902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c4850 0x55fdff1ca960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.317+0000 7f1c6ed9d640 1 -- 172.21.15.44:0/300561902 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdfee3c3f0 con 0x55fdff1c3f30 2024-04-21T23:13:14.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.317+0000 7f1c6e59c640 1 --2- 172.21.15.44:0/300561902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c4850 0x55fdff1ca960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:14.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.318+0000 7f1c6ed9d640 1 --2- 172.21.15.44:0/300561902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1c3f30 0x55fdff1ca420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1c6002efd0 tx=0x7f1c6002ced0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcf8418b295023da server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:14.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.318+0000 7f1c52ff5640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:13:14.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.318+0000 7f1c52ff5640 1 -- 172.21.15.44:0/300561902 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c60004450 con 0x55fdff1c3f30 2024-04-21T23:13:14.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.318+0000 7f1c52ff5640 1 -- 172.21.15.44:0/300561902 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f1c600046c0 con 0x55fdff1c3f30 2024-04-21T23:13:14.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.318+0000 7f1c77e38740 1 -- 172.21.15.44:0/300561902 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c6801c4c0 con 0x55fdff1c3f30 2024-04-21T23:13:14.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.318+0000 7f1c52ff5640 1 -- 172.21.15.44:0/300561902 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1c600048c0 con 0x55fdff1c3f30 2024-04-21T23:13:14.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.318+0000 7f1c77e38740 10 client.26275 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:14.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.318+0000 7f1c77e38740 1 -- 172.21.15.44:0/300561902 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55fdfedfc0f0 con 0x55fdff1c3f30 2024-04-21T23:13:14.325 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.318+0000 7f1c52ff5640 1 -- 172.21.15.44:0/300561902 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c6801c4c0 con 0x55fdff1c3f30 2024-04-21T23:13:14.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.318+0000 7f1c52ff5640 1 -- 172.21.15.44:0/300561902 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f1c60046070 con 0x55fdff1c3f30 2024-04-21T23:13:14.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.318+0000 7f1c77e38740 10 client.26275 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:14.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.318+0000 7f1c77e38740 10 client.26275 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:14.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.318+0000 7f1c77e38740 20 client.26275 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:14.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.319+0000 7f1c77e38740 1 --2- 172.21.15.44:0/300561902 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55fdff1dd6b0 0x55fdff1dfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.319+0000 7f1c77e38740 4 client.26275 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:13:14.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.319+0000 7f1c77e38740 1 -- 172.21.15.44:0/300561902 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fdff135aa0 con 0x55fdff1dd6b0 2024-04-21T23:13:14.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.319+0000 7f1c6e59c640 1 --2- 172.21.15.44:0/300561902 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55fdff1dd6b0 0x55fdff1dfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.319+0000 7f1c6e59c640 1 --2- 172.21.15.44:0/300561902 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55fdff1dd6b0 0x55fdff1dfac0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4691ae4cf276962 server_cookie=9a7a4b5b3bec04f3 in_seq=0 out_seq=0 2024-04-21T23:13:14.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.319+0000 7f1c52ff5640 10 client.26275 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:14.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.320+0000 7f1c52ff5640 1 -- 172.21.15.44:0/300561902 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fdff135aa0 con 0x55fdff1dd6b0 2024-04-21T23:13:14.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.320+0000 7f1c52ff5640 10 client.26275 handle_command_reply: tid=0 2024-04-21T23:13:14.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.320+0000 7f1c77e38740 1 client.26275 shutdown 2024-04-21T23:13:14.326 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.320+0000 7f1c77e38740 1 -- 172.21.15.44:0/300561902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1c3f30 msgr2=0x55fdff1ca420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:14.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.320+0000 7f1c77e38740 1 --2- 172.21.15.44:0/300561902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1c3f30 0x55fdff1ca420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1c6002efd0 tx=0x7f1c6002ced0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.320+0000 7f1c77e38740 1 -- 172.21.15.44:0/300561902 shutdown_connections 2024-04-21T23:13:14.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.320+0000 7f1c77e38740 1 --2- 172.21.15.44:0/300561902 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55fdff1dd6b0 0x55fdff1dfac0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.320+0000 7f1c77e38740 1 --2- 172.21.15.44:0/300561902 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1cd430 0x55fdff1caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.320+0000 7f1c77e38740 1 --2- 172.21.15.44:0/300561902 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c4850 0x55fdff1ca960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.320+0000 7f1c77e38740 1 --2- 172.21.15.44:0/300561902 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1c3f30 0x55fdff1ca420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.320+0000 7f1c77e38740 1 -- 172.21.15.44:0/300561902 >> 172.21.15.44:0/300561902 conn(0x55fdff1be660 msgr2=0x55fdff1c03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:14.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.320+0000 7f1c52ff5640 0 client.26275 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:14.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.320+0000 7f1c77e38740 1 -- 172.21.15.44:0/300561902 shutdown_connections 2024-04-21T23:13:14.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.320+0000 7f1c77e38740 1 -- 172.21.15.44:0/300561902 wait complete. 2024-04-21T23:13:14.327 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.320+0000 7f1c77e38740 20 client.26275 trim_cache size 0 max 16384 2024-04-21T23:13:14.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.339+0000 7f1c77e38740 1 Processor -- start 2024-04-21T23:13:14.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.339+0000 7f1c77e38740 1 -- start start 2024-04-21T23:13:14.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.339+0000 7f1c77e38740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1c3f30 0x55fdff1d90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.339+0000 7f1c77e38740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c4850 0x55fdff1b9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.339+0000 7f1c77e38740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1cd430 0x55fdff1bd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.339+0000 7f1c77e38740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1c701aed70 con 0x55fdff1c4850 2024-04-21T23:13:14.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.339+0000 7f1c77e38740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1c701b15f0 con 0x55fdff1c3f30 2024-04-21T23:13:14.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.339+0000 7f1c77e38740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1c701af3e0 con 0x55fdff1cd430 2024-04-21T23:13:14.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.339+0000 7f1c6e59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c4850 0x55fdff1b9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.339+0000 7f1c6e59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c4850 0x55fdff1b9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44514/0 (socket says 172.21.15.44:44514) 2024-04-21T23:13:14.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.339+0000 7f1c6e59c640 1 -- 172.21.15.44:0/4020550607 learned_addr learned my addr 172.21.15.44:0/4020550607 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:14.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.339+0000 7f1c6ed9d640 1 --2- 172.21.15.44:0/4020550607 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1c3f30 0x55fdff1d90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.339+0000 7f1c6f59e640 1 --2- 172.21.15.44:0/4020550607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1cd430 0x55fdff1bd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.339+0000 7f1c6e59c640 1 -- 172.21.15.44:0/4020550607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1cd430 msgr2=0x55fdff1bd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:14.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.339+0000 7f1c6e59c640 1 --2- 172.21.15.44:0/4020550607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1cd430 0x55fdff1bd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.340+0000 7f1c6e59c640 1 -- 172.21.15.44:0/4020550607 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1c3f30 msgr2=0x55fdff1d90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:14.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.340+0000 7f1c6e59c640 1 --2- 172.21.15.44:0/4020550607 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1c3f30 0x55fdff1d90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.340+0000 7f1c6e59c640 1 -- 172.21.15.44:0/4020550607 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdff1c93d0 con 0x55fdff1c4850 2024-04-21T23:13:14.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.340+0000 7f1c6f59e640 1 --2- 172.21.15.44:0/4020550607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1cd430 0x55fdff1bd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:13:14.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.340+0000 7f1c6ed9d640 1 --2- 172.21.15.44:0/4020550607 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1c3f30 0x55fdff1d90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:14.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.340+0000 7f1c6e59c640 1 --2- 172.21.15.44:0/4020550607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c4850 0x55fdff1b9440 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f1c640981e0 tx=0x7f1c640d36a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a679ce72f3dd3e32 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:14.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.340+0000 7f1c54ff9640 1 -- 172.21.15.44:0/4020550607 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6409b6e0 con 0x55fdff1c4850 2024-04-21T23:13:14.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.340+0000 7f1c54ff9640 1 -- 172.21.15.44:0/4020550607 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c6409bd80 con 0x55fdff1c4850 2024-04-21T23:13:14.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.340+0000 7f1c54ff9640 1 -- 172.21.15.44:0/4020550607 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6409b430 con 0x55fdff1c4850 2024-04-21T23:13:14.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c77e38740 1 -- 172.21.15.44:0/4020550607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c4850 msgr2=0x55fdff1b9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:14.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c77e38740 1 --2- 172.21.15.44:0/4020550607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c4850 0x55fdff1b9440 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f1c640981e0 tx=0x7f1c640d36a0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c77e38740 1 -- 172.21.15.44:0/4020550607 shutdown_connections 2024-04-21T23:13:14.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c77e38740 1 --2- 172.21.15.44:0/4020550607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1cd430 0x55fdff1bd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c77e38740 1 --2- 172.21.15.44:0/4020550607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1c4850 0x55fdff1b9440 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c77e38740 1 --2- 172.21.15.44:0/4020550607 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1c3f30 0x55fdff1d90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c77e38740 1 -- 172.21.15.44:0/4020550607 >> 172.21.15.44:0/4020550607 conn(0x55fdff1be660 msgr2=0x55fdff1bce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:14.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c77e38740 1 -- 172.21.15.44:0/4020550607 shutdown_connections 2024-04-21T23:13:14.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c77e38740 1 -- 172.21.15.44:0/4020550607 wait complete. 2024-04-21T23:13:14.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c77e38740 1 -- start start 2024-04-21T23:13:14.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c77e38740 1 Processor -- start 2024-04-21T23:13:14.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c77e38740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1c3f30 0x55fdff1d57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c77e38740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1c4850 0x55fdff1d0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c77e38740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1cd430 0x55fdff1d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c77e38740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1c701afab0 con 0x55fdff1cd430 2024-04-21T23:13:14.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c77e38740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1c701b0f20 con 0x55fdff1c3f30 2024-04-21T23:13:14.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c77e38740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1c701b0180 con 0x55fdff1c4850 2024-04-21T23:13:14.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c77e38740 10 client.0 _ll_register_callbacks cb 0x7f1c60009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:14.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c6ed9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1c3f30 0x55fdff1d57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c6f59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1cd430 0x55fdff1d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c6ed9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1c3f30 0x55fdff1d57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:43916/0 (socket says 172.21.15.44:43916) 2024-04-21T23:13:14.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c6f59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1cd430 0x55fdff1d10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44518/0 (socket says 172.21.15.44:44518) 2024-04-21T23:13:14.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.341+0000 7f1c6ed9d640 1 -- 172.21.15.44:0/521697387 learned_addr learned my addr 172.21.15.44:0/521697387 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:14.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.342+0000 7f1c6e59c640 1 --2- 172.21.15.44:0/521697387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1c4850 0x55fdff1d0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.342+0000 7f1c6ed9d640 1 -- 172.21.15.44:0/521697387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1c4850 msgr2=0x55fdff1d0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:14.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.342+0000 7f1c6ed9d640 1 --2- 172.21.15.44:0/521697387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1c4850 0x55fdff1d0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.342+0000 7f1c6ed9d640 1 -- 172.21.15.44:0/521697387 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1cd430 msgr2=0x55fdff1d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:14.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.342+0000 7f1c6ed9d640 1 --2- 172.21.15.44:0/521697387 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1cd430 0x55fdff1d10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.342+0000 7f1c6ed9d640 1 -- 172.21.15.44:0/521697387 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdff1d1730 con 0x55fdff1c3f30 2024-04-21T23:13:14.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.342+0000 7f1c6f59e640 1 --2- 172.21.15.44:0/521697387 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1cd430 0x55fdff1d10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:14.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.342+0000 7f1c6ed9d640 1 --2- 172.21.15.44:0/521697387 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1c3f30 0x55fdff1d57f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1c60009850 tx=0x7f1c60003d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bca1863a2e638b17 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:14.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.342+0000 7f1c52ff5640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:13:14.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.342+0000 7f1c52ff5640 1 -- 172.21.15.44:0/521697387 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c60005d50 con 0x55fdff1c3f30 2024-04-21T23:13:14.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.342+0000 7f1c52ff5640 1 -- 172.21.15.44:0/521697387 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f1c60004460 con 0x55fdff1c3f30 2024-04-21T23:13:14.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.342+0000 7f1c77e38740 1 -- 172.21.15.44:0/521697387 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c6409bd80 con 0x55fdff1c3f30 2024-04-21T23:13:14.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.342+0000 7f1c52ff5640 1 -- 172.21.15.44:0/521697387 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1c60003e80 con 0x55fdff1c3f30 2024-04-21T23:13:14.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.342+0000 7f1c77e38740 10 client.26246 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:14.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.342+0000 7f1c77e38740 1 -- 172.21.15.44:0/521697387 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55fdff243a10 con 0x55fdff1c3f30 2024-04-21T23:13:14.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.343+0000 7f1c52ff5640 1 -- 172.21.15.44:0/521697387 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c6409bd80 con 0x55fdff1c3f30 2024-04-21T23:13:14.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.343+0000 7f1c52ff5640 1 -- 172.21.15.44:0/521697387 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f1c60055030 con 0x55fdff1c3f30 2024-04-21T23:13:14.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.343+0000 7f1c77e38740 10 client.26246 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:14.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.343+0000 7f1c77e38740 10 client.26246 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:14.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.343+0000 7f1c77e38740 20 client.26246 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:14.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.343+0000 7f1c77e38740 1 --2- 172.21.15.44:0/521697387 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55fdff244190 0x55fdff2465b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:14.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.343+0000 7f1c77e38740 4 client.26246 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:13:14.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.343+0000 7f1c77e38740 1 -- 172.21.15.44:0/521697387 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fdff12fdf0 con 0x55fdff244190 2024-04-21T23:13:14.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.343+0000 7f1c6e59c640 1 --2- 172.21.15.44:0/521697387 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55fdff244190 0x55fdff2465b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:14.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.343+0000 7f1c6e59c640 1 --2- 172.21.15.44:0/521697387 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55fdff244190 0x55fdff2465b0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48383261220783b7 server_cookie=d1f01b7a8d7b08be in_seq=0 out_seq=0 2024-04-21T23:13:14.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.343+0000 7f1c52ff5640 10 client.26246 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:14.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.345+0000 7f1c52ff5640 1 -- 172.21.15.44:0/521697387 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55fdff12fdf0 con 0x55fdff244190 2024-04-21T23:13:14.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.345+0000 7f1c52ff5640 10 client.26246 handle_command_reply: tid=0 2024-04-21T23:13:14.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.345+0000 7f1c77e38740 1 client.26246 shutdown 2024-04-21T23:13:14.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c77e38740 1 -- 172.21.15.44:0/521697387 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1c3f30 msgr2=0x55fdff1d57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:14.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c77e38740 1 --2- 172.21.15.44:0/521697387 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1c3f30 0x55fdff1d57f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1c60009850 tx=0x7f1c60003d60 comp rx=0 tx=0).stop 2024-04-21T23:13:14.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c77e38740 1 -- 172.21.15.44:0/521697387 shutdown_connections 2024-04-21T23:13:14.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c77e38740 1 --2- 172.21.15.44:0/521697387 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55fdff244190 0x55fdff2465b0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c77e38740 1 --2- 172.21.15.44:0/521697387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fdff1c4850 0x55fdff1d0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c52ff5640 0 client.26246 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:14.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c77e38740 1 --2- 172.21.15.44:0/521697387 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fdff1cd430 0x55fdff1d10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c77e38740 1 --2- 172.21.15.44:0/521697387 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fdff1c3f30 0x55fdff1d57f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c77e38740 1 -- 172.21.15.44:0/521697387 >> 172.21.15.44:0/521697387 conn(0x55fdff1be660 msgr2=0x55fdff1bd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:14.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c77e38740 1 -- 172.21.15.44:0/521697387 shutdown_connections 2024-04-21T23:13:14.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c77e38740 1 -- 172.21.15.44:0/521697387 wait complete. 2024-04-21T23:13:14.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c77e38740 20 client.26246 trim_cache size 0 max 16384 2024-04-21T23:13:14.351 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:13:14.351 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:13:14.351 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-21T23:13:14.351 INFO:teuthology.orchestra.run.smithi044.stdout: "370bb19c-895f-4047-812f-845bf1847d16": { 2024-04-21T23:13:14.351 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-21T23:13:14.351 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "370bb19c-895f-4047-812f-845bf1847d16", 2024-04-21T23:13:14.351 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-04-21T23:13:14.351 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:13:14.351 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:13:14.351 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:13:14.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c7524e640 1 -- 172.21.15.44:0/3174478868 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1c580a2d30 msgr2=0x7f1c580a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:14.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c7524e640 1 --2- 172.21.15.44:0/3174478868 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1c580a2d30 0x7f1c580a51f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1c6801c000 tx=0x7f1c6801bed0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c7524e640 1 -- 172.21.15.44:0/3174478868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c701032d0 msgr2=0x7f1c70100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:14.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c7524e640 1 --2- 172.21.15.44:0/3174478868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c701032d0 0x7f1c70100b40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1c6400e9f0 tx=0x7f1c6400eeb0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c7524e640 1 -- 172.21.15.44:0/3174478868 shutdown_connections 2024-04-21T23:13:14.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c7524e640 1 --2- 172.21.15.44:0/3174478868 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1c580a2d30 0x7f1c580a51f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c7524e640 1 --2- 172.21.15.44:0/3174478868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1c7010bcf0 0x7f1c70101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c7524e640 1 --2- 172.21.15.44:0/3174478868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c701032d0 0x7f1c70100b40 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c7524e640 1 --2- 172.21.15.44:0/3174478868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1c701029b0 0x7f1c70100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:14.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c7524e640 1 -- 172.21.15.44:0/3174478868 >> 172.21.15.44:0/3174478868 conn(0x7f1c700777f0 msgr2=0x7f1c700fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:14.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c7524e640 1 -- 172.21.15.44:0/3174478868 shutdown_connections 2024-04-21T23:13:14.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:14.346+0000 7f1c7524e640 1 -- 172.21.15.44:0/3174478868 wait complete. 2024-04-21T23:13:14.358 INFO:tasks.cephfs.filesystem:scrub status for tag:370bb19c-895f-4047-812f-845bf1847d16 - {'path': '/', 'tag': '370bb19c-895f-4047-812f-845bf1847d16', 'options': 'recursive,force'} 2024-04-21T23:13:14.698 INFO:tasks.workunit.client.0.smithi044.stdout: 1 469800 13.49 MB/sec execute 64 sec latency 76.052 ms 2024-04-21T23:13:15.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 473014 13.57 MB/sec execute 65 sec latency 8.547 ms 2024-04-21T23:13:16.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:15 smithi044 ceph-mon[27934]: scrub queued for path: / 2024-04-21T23:13:16.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:15 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:13:16.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:15 smithi044 ceph-mon[27934]: scrub summary: active paths [/] 2024-04-21T23:13:16.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:15 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:13:16.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:15 smithi044 ceph-mon[27934]: pgmap v137: 129 pgs: 129 active+clean; 56 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.5 MiB/s wr, 410 op/s 2024-04-21T23:13:16.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:15 smithi055 ceph-mon[31117]: scrub queued for path: / 2024-04-21T23:13:16.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:15 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:13:16.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:15 smithi055 ceph-mon[31117]: scrub summary: active paths [/] 2024-04-21T23:13:16.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:15 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:13:16.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:15 smithi055 ceph-mon[31117]: pgmap v137: 129 pgs: 129 active+clean; 56 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.5 MiB/s wr, 410 op/s 2024-04-21T23:13:16.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:15 smithi069 ceph-mon[31816]: scrub queued for path: / 2024-04-21T23:13:16.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:15 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:13:16.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:15 smithi069 ceph-mon[31816]: scrub summary: active paths [/] 2024-04-21T23:13:16.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:15 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:13:16.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:15 smithi069 ceph-mon[31816]: pgmap v137: 129 pgs: 129 active+clean; 56 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.5 MiB/s wr, 410 op/s 2024-04-21T23:13:16.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 475518 13.55 MB/sec execute 66 sec latency 41.498 ms 2024-04-21T23:13:17.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 477557 13.53 MB/sec execute 67 sec latency 27.887 ms 2024-04-21T23:13:18.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:17 smithi044 ceph-mon[27934]: pgmap v138: 129 pgs: 129 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.8 MiB/s wr, 530 op/s 2024-04-21T23:13:18.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:17 smithi044 ceph-mon[27934]: scrub summary: idle 2024-04-21T23:13:18.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:17 smithi055 ceph-mon[31117]: pgmap v138: 129 pgs: 129 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.8 MiB/s wr, 530 op/s 2024-04-21T23:13:18.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:17 smithi055 ceph-mon[31117]: scrub summary: idle 2024-04-21T23:13:18.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:17 smithi069 ceph-mon[31816]: pgmap v138: 129 pgs: 129 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.8 MiB/s wr, 530 op/s 2024-04-21T23:13:18.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:17 smithi069 ceph-mon[31816]: scrub summary: idle 2024-04-21T23:13:18.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 480352 13.54 MB/sec execute 68 sec latency 26.777 ms 2024-04-21T23:13:19.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 483527 13.61 MB/sec execute 69 sec latency 7.671 ms 2024-04-21T23:13:20.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:19 smithi044 ceph-mon[27934]: pgmap v139: 129 pgs: 129 active+clean; 63 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 454 op/s 2024-04-21T23:13:20.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:19 smithi055 ceph-mon[31117]: pgmap v139: 129 pgs: 129 active+clean; 63 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 454 op/s 2024-04-21T23:13:20.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:19 smithi069 ceph-mon[31816]: pgmap v139: 129 pgs: 129 active+clean; 63 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 454 op/s 2024-04-21T23:13:20.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 486713 13.67 MB/sec execute 70 sec latency 16.496 ms 2024-04-21T23:13:21.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 488352 13.57 MB/sec execute 71 sec latency 55.655 ms 2024-04-21T23:13:22.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:21 smithi044 ceph-mon[27934]: pgmap v140: 129 pgs: 129 active+clean; 63 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 420 op/s 2024-04-21T23:13:22.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:21 smithi055 ceph-mon[31117]: pgmap v140: 129 pgs: 129 active+clean; 63 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 420 op/s 2024-04-21T23:13:22.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:21 smithi069 ceph-mon[31816]: pgmap v140: 129 pgs: 129 active+clean; 63 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 420 op/s 2024-04-21T23:13:22.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 490928 13.58 MB/sec execute 72 sec latency 37.134 ms 2024-04-21T23:13:23.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 494080 13.65 MB/sec execute 73 sec latency 15.071 ms 2024-04-21T23:13:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.859+0000 7fe235c88640 1 -- 172.21.15.44:0/2144224111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2301020f0 msgr2=0x7fe23010e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.859+0000 7fe235c88640 1 --2- 172.21.15.44:0/2144224111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2301020f0 0x7fe23010e600 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fe228009cd0 tx=0x7fe22802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.859+0000 7fe235c88640 1 -- 172.21.15.44:0/2144224111 shutdown_connections 2024-04-21T23:13:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.859+0000 7fe235c88640 1 --2- 172.21.15.44:0/2144224111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe230101730 0x7fe230101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.859+0000 7fe235c88640 1 --2- 172.21.15.44:0/2144224111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2301020f0 0x7fe23010e600 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.859+0000 7fe235c88640 1 --2- 172.21.15.44:0/2144224111 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe230107740 0x7fe230107b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.859+0000 7fe235c88640 1 -- 172.21.15.44:0/2144224111 >> 172.21.15.44:0/2144224111 conn(0x7fe2300fd200 msgr2=0x7fe2300ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:23.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.859+0000 7fe235c88640 1 -- 172.21.15.44:0/2144224111 shutdown_connections 2024-04-21T23:13:23.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.859+0000 7fe235c88640 1 -- 172.21.15.44:0/2144224111 wait complete. 2024-04-21T23:13:23.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.860+0000 7fe235c88640 1 Processor -- start 2024-04-21T23:13:23.860 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.860+0000 7fe235c88640 1 -- start start 2024-04-21T23:13:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.860+0000 7fe235c88640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe230101730 0x7fe230198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.860+0000 7fe235c88640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2301020f0 0x7fe2301985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.860+0000 7fe235c88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe230107740 0x7fe23019c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.860+0000 7fe235c88640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe230111b50 con 0x7fe2301020f0 2024-04-21T23:13:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.860+0000 7fe235c88640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe2301119d0 con 0x7fe230101730 2024-04-21T23:13:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.860+0000 7fe235c88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe230111cd0 con 0x7fe230107740 2024-04-21T23:13:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.860+0000 7fe22effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2301020f0 0x7fe2301985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.860+0000 7fe22effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2301020f0 0x7fe2301985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53304/0 (socket says 172.21.15.44:53304) 2024-04-21T23:13:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.860+0000 7fe22effd640 1 -- 172.21.15.44:0/4259958429 learned_addr learned my addr 172.21.15.44:0/4259958429 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.860+0000 7fe22f7fe640 1 --2- 172.21.15.44:0/4259958429 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe230101730 0x7fe230198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.860+0000 7fe22ffff640 1 --2- 172.21.15.44:0/4259958429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe230107740 0x7fe23019c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.860+0000 7fe22effd640 1 -- 172.21.15.44:0/4259958429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe230107740 msgr2=0x7fe23019c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.860+0000 7fe22effd640 1 --2- 172.21.15.44:0/4259958429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe230107740 0x7fe23019c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.860+0000 7fe22effd640 1 -- 172.21.15.44:0/4259958429 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe230101730 msgr2=0x7fe230198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.860+0000 7fe22effd640 1 --2- 172.21.15.44:0/4259958429 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe230101730 0x7fe230198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.860+0000 7fe22effd640 1 -- 172.21.15.44:0/4259958429 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe23019d080 con 0x7fe2301020f0 2024-04-21T23:13:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.860+0000 7fe22f7fe640 1 --2- 172.21.15.44:0/4259958429 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe230101730 0x7fe230198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.861+0000 7fe22effd640 1 --2- 172.21.15.44:0/4259958429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2301020f0 0x7fe2301985d0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fe22400e9f0 tx=0x7fe22400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.861+0000 7fe22cff9640 1 -- 172.21.15.44:0/4259958429 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe22400cd80 con 0x7fe2301020f0 2024-04-21T23:13:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.861+0000 7fe235c88640 1 -- 172.21.15.44:0/4259958429 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe23019d370 con 0x7fe2301020f0 2024-04-21T23:13:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.861+0000 7fe22cff9640 1 -- 172.21.15.44:0/4259958429 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe224005500 con 0x7fe2301020f0 2024-04-21T23:13:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.861+0000 7fe22cff9640 1 -- 172.21.15.44:0/4259958429 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe224010690 con 0x7fe2301020f0 2024-04-21T23:13:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.861+0000 7fe235c88640 1 -- 172.21.15.44:0/4259958429 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2301a4c00 con 0x7fe2301020f0 2024-04-21T23:13:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.863+0000 7fe22cff9640 1 -- 172.21.15.44:0/4259958429 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe2240046c0 con 0x7fe2301020f0 2024-04-21T23:13:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.863+0000 7fe238875740 1 Processor -- start 2024-04-21T23:13:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe238875740 1 -- start start 2024-04-21T23:13:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe22cff9640 1 --2- 172.21.15.44:0/4259958429 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe2180a2cf0 0x7fe2180a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe238875740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f266b20 0x56371f266f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe238875740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f267440 0x56371f26fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe238875740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f270210 0x56371f272600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe238875740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56371f1cab70 con 0x56371f267440 2024-04-21T23:13:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe238875740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x56371f173470 con 0x56371f266b20 2024-04-21T23:13:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe238875740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56371f1c6b30 con 0x56371f270210 2024-04-21T23:13:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe22effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f267440 0x56371f26fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe22ffff640 1 --2- 172.21.15.44:0/4259958429 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe2180a2cf0 0x7fe2180a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe22effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f267440 0x56371f26fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53318/0 (socket says 172.21.15.44:53318) 2024-04-21T23:13:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe22effd640 1 -- 172.21.15.44:0/1306013059 learned_addr learned my addr 172.21.15.44:0/1306013059 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe22f7fe640 1 --2- 172.21.15.44:0/1306013059 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f266b20 0x56371f266f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe22ffff640 1 --2- 172.21.15.44:0/1306013059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f270210 0x56371f272600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe22cff9640 1 -- 172.21.15.44:0/4259958429 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fe2240cfc70 con 0x7fe2301020f0 2024-04-21T23:13:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe22effd640 1 -- 172.21.15.44:0/1306013059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f270210 msgr2=0x56371f272600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe22effd640 1 --2- 172.21.15.44:0/1306013059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f270210 0x56371f272600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe22effd640 1 -- 172.21.15.44:0/1306013059 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f266b20 msgr2=0x56371f266f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe22effd640 1 --2- 172.21.15.44:0/1306013059 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f266b20 0x56371f266f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe22effd640 1 -- 172.21.15.44:0/1306013059 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56371eedf210 con 0x56371f267440 2024-04-21T23:13:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe22f7fe640 1 --2- 172.21.15.44:0/1306013059 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f266b20 0x56371f266f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe22ffff640 1 --2- 172.21.15.44:0/4259958429 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe2180a2cf0 0x7fe2180a51b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe228005cc0 tx=0x7fe228005f80 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe22effd640 1 --2- 172.21.15.44:0/1306013059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f267440 0x56371f26fcd0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fe2240a44d0 tx=0x7fe22409bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47fbe1d8336220df server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.864+0000 7fe215ffb640 1 -- 172.21.15.44:0/1306013059 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2240aec90 con 0x56371f267440 2024-04-21T23:13:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.865+0000 7fe215ffb640 1 -- 172.21.15.44:0/1306013059 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe22409c490 con 0x56371f267440 2024-04-21T23:13:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.865+0000 7fe215ffb640 1 -- 172.21.15.44:0/1306013059 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2240b1680 con 0x56371f267440 2024-04-21T23:13:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.865+0000 7fe238875740 1 -- 172.21.15.44:0/1306013059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f267440 msgr2=0x56371f26fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.865+0000 7fe238875740 1 --2- 172.21.15.44:0/1306013059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f267440 0x56371f26fcd0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fe2240a44d0 tx=0x7fe22409bc20 comp rx=0 tx=0).stop 2024-04-21T23:13:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.865+0000 7fe238875740 1 -- 172.21.15.44:0/1306013059 shutdown_connections 2024-04-21T23:13:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.865+0000 7fe238875740 1 --2- 172.21.15.44:0/1306013059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f270210 0x56371f272600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.865+0000 7fe238875740 1 --2- 172.21.15.44:0/1306013059 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f267440 0x56371f26fcd0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.865+0000 7fe238875740 1 --2- 172.21.15.44:0/1306013059 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f266b20 0x56371f266f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.865+0000 7fe238875740 1 -- 172.21.15.44:0/1306013059 >> 172.21.15.44:0/1306013059 conn(0x56371f261440 msgr2=0x56371f261820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.865+0000 7fe238875740 1 -- 172.21.15.44:0/1306013059 shutdown_connections 2024-04-21T23:13:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.865+0000 7fe238875740 1 -- 172.21.15.44:0/1306013059 wait complete. 2024-04-21T23:13:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.866+0000 7fe238875740 1 -- start start 2024-04-21T23:13:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.866+0000 7fe238875740 1 Processor -- start 2024-04-21T23:13:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.866+0000 7fe238875740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f266b20 0x56371f26d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.866+0000 7fe238875740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f267440 0x56371f26d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.866+0000 7fe238875740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f270210 0x56371f26dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.866+0000 7fe238875740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56371f272de0 con 0x56371f267440 2024-04-21T23:13:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.866+0000 7fe238875740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x56371f186a10 con 0x56371f270210 2024-04-21T23:13:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.866+0000 7fe238875740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56371f272f60 con 0x56371f266b20 2024-04-21T23:13:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.866+0000 7fe238875740 10 client.0 _ll_register_callbacks cb 0x56371f1c1b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.866+0000 7fe22effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f267440 0x56371f26d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.866+0000 7fe22f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f266b20 0x56371f26d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.866+0000 7fe22ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f270210 0x56371f26dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.866+0000 7fe22f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f266b20 0x56371f26d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:50288/0 (socket says 172.21.15.44:50288) 2024-04-21T23:13:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.866+0000 7fe22f7fe640 1 -- 172.21.15.44:0/1413347965 learned_addr learned my addr 172.21.15.44:0/1413347965 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe22effd640 1 -- 172.21.15.44:0/1413347965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f266b20 msgr2=0x56371f26d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe22effd640 1 --2- 172.21.15.44:0/1413347965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f266b20 0x56371f26d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe22effd640 1 -- 172.21.15.44:0/1413347965 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f270210 msgr2=0x56371f26dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe22effd640 1 --2- 172.21.15.44:0/1413347965 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f270210 0x56371f26dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe22effd640 1 -- 172.21.15.44:0/1413347965 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56371efab620 con 0x56371f267440 2024-04-21T23:13:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe22f7fe640 1 --2- 172.21.15.44:0/1413347965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f266b20 0x56371f26d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe22effd640 1 --2- 172.21.15.44:0/1413347965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f267440 0x56371f26d7d0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x56371f2679d0 tx=0x7fe2240ccf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adfb0a8ac0f8ab2f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe20effd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:13:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe20effd640 1 -- 172.21.15.44:0/1413347965 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2240a8070 con 0x56371f267440 2024-04-21T23:13:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe20effd640 1 -- 172.21.15.44:0/1413347965 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fe2240cd240 con 0x56371f267440 2024-04-21T23:13:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe238875740 1 -- 172.21.15.44:0/1413347965 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe22409c490 con 0x56371f267440 2024-04-21T23:13:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe20effd640 1 -- 172.21.15.44:0/1413347965 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe2240aee70 con 0x56371f267440 2024-04-21T23:13:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe238875740 10 client.16692 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe238875740 1 -- 172.21.15.44:0/1413347965 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56371f27f090 con 0x56371f267440 2024-04-21T23:13:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe20effd640 1 -- 172.21.15.44:0/1413347965 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe22409c490 con 0x56371f267440 2024-04-21T23:13:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe20effd640 1 -- 172.21.15.44:0/1413347965 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fe2240a4570 con 0x56371f267440 2024-04-21T23:13:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe238875740 10 client.16692 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe238875740 10 client.16692 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe238875740 20 client.16692 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe238875740 1 --2- 172.21.15.44:0/1413347965 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56371f2803e0 0x56371f2827d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe238875740 4 client.16692 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:13:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.867+0000 7fe238875740 1 -- 172.21.15.44:0/1413347965 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56371f1d80b0 con 0x56371f2803e0 2024-04-21T23:13:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.868+0000 7fe22f7fe640 1 --2- 172.21.15.44:0/1413347965 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56371f2803e0 0x56371f2827d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.868+0000 7fe22f7fe640 1 --2- 172.21.15.44:0/1413347965 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56371f2803e0 0x56371f2827d0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11323d982e0ae93a server_cookie=5dc6ce7757203bd5 in_seq=0 out_seq=0 2024-04-21T23:13:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.868+0000 7fe20effd640 10 client.16692 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.869+0000 7fe20effd640 1 -- 172.21.15.44:0/1413347965 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56371f1d80b0 con 0x56371f2803e0 2024-04-21T23:13:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.869+0000 7fe20effd640 10 client.16692 handle_command_reply: tid=0 2024-04-21T23:13:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.869+0000 7fe238875740 1 client.16692 shutdown 2024-04-21T23:13:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.869+0000 7fe238875740 1 -- 172.21.15.44:0/1413347965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f267440 msgr2=0x56371f26d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.869+0000 7fe238875740 1 --2- 172.21.15.44:0/1413347965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f267440 0x56371f26d7d0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x56371f2679d0 tx=0x7fe2240ccf30 comp rx=0 tx=0).stop 2024-04-21T23:13:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.869+0000 7fe238875740 1 -- 172.21.15.44:0/1413347965 shutdown_connections 2024-04-21T23:13:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.869+0000 7fe238875740 1 --2- 172.21.15.44:0/1413347965 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56371f2803e0 0x56371f2827d0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.869+0000 7fe238875740 1 --2- 172.21.15.44:0/1413347965 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f270210 0x56371f26dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.869+0000 7fe238875740 1 --2- 172.21.15.44:0/1413347965 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f267440 0x56371f26d7d0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.869+0000 7fe20effd640 0 client.16692 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.869+0000 7fe238875740 1 --2- 172.21.15.44:0/1413347965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f266b20 0x56371f26d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.869+0000 7fe238875740 1 -- 172.21.15.44:0/1413347965 >> 172.21.15.44:0/1413347965 conn(0x56371f261440 msgr2=0x56371f263190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.869+0000 7fe238875740 1 -- 172.21.15.44:0/1413347965 shutdown_connections 2024-04-21T23:13:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.869+0000 7fe238875740 1 -- 172.21.15.44:0/1413347965 wait complete. 2024-04-21T23:13:23.879 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.869+0000 7fe238875740 20 client.16692 trim_cache size 0 max 16384 2024-04-21T23:13:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.885+0000 7fe238875740 1 Processor -- start 2024-04-21T23:13:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.885+0000 7fe238875740 1 -- start start 2024-04-21T23:13:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.885+0000 7fe238875740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f266b20 0x56371f26bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.885+0000 7fe238875740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f267440 0x56371f26a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.885+0000 7fe238875740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f270210 0x56371f26aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.885+0000 7fe238875740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe2301accd0 con 0x56371f267440 2024-04-21T23:13:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.885+0000 7fe238875740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe2301af550 con 0x56371f270210 2024-04-21T23:13:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.885+0000 7fe238875740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe2301ad340 con 0x56371f266b20 2024-04-21T23:13:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.885+0000 7fe22effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f267440 0x56371f26a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.885+0000 7fe22effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f267440 0x56371f26a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53340/0 (socket says 172.21.15.44:53340) 2024-04-21T23:13:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.885+0000 7fe22f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f266b20 0x56371f26bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.885+0000 7fe22effd640 1 -- 172.21.15.44:0/1432525451 learned_addr learned my addr 172.21.15.44:0/1432525451 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.885+0000 7fe22ffff640 1 --2- 172.21.15.44:0/1432525451 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f270210 0x56371f26aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.885+0000 7fe22effd640 1 -- 172.21.15.44:0/1432525451 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f266b20 msgr2=0x56371f26bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.885+0000 7fe22effd640 1 --2- 172.21.15.44:0/1432525451 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f266b20 0x56371f26bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.885+0000 7fe22effd640 1 -- 172.21.15.44:0/1432525451 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f270210 msgr2=0x56371f26aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.885+0000 7fe22effd640 1 --2- 172.21.15.44:0/1432525451 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f270210 0x56371f26aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.885+0000 7fe22effd640 1 -- 172.21.15.44:0/1432525451 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56371f272970 con 0x56371f267440 2024-04-21T23:13:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.885+0000 7fe22f7fe640 1 --2- 172.21.15.44:0/1432525451 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f266b20 0x56371f26bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.885+0000 7fe22effd640 1 --2- 172.21.15.44:0/1432525451 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f267440 0x56371f26a510 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe224098710 tx=0x7fe224098110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b907f7ac100d66b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.886+0000 7fe2157fa640 1 -- 172.21.15.44:0/1432525451 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2240a8070 con 0x56371f267440 2024-04-21T23:13:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.886+0000 7fe2157fa640 1 -- 172.21.15.44:0/1432525451 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2240b1a80 con 0x56371f267440 2024-04-21T23:13:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.886+0000 7fe2157fa640 1 -- 172.21.15.44:0/1432525451 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2240ae8d0 con 0x56371f267440 2024-04-21T23:13:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.886+0000 7fe238875740 1 -- 172.21.15.44:0/1432525451 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f267440 msgr2=0x56371f26a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.886+0000 7fe238875740 1 --2- 172.21.15.44:0/1432525451 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f267440 0x56371f26a510 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe224098710 tx=0x7fe224098110 comp rx=0 tx=0).stop 2024-04-21T23:13:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.886+0000 7fe238875740 1 -- 172.21.15.44:0/1432525451 shutdown_connections 2024-04-21T23:13:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.886+0000 7fe238875740 1 --2- 172.21.15.44:0/1432525451 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f270210 0x56371f26aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.886+0000 7fe238875740 1 --2- 172.21.15.44:0/1432525451 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f267440 0x56371f26a510 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.886+0000 7fe238875740 1 --2- 172.21.15.44:0/1432525451 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f266b20 0x56371f26bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.886+0000 7fe238875740 1 -- 172.21.15.44:0/1432525451 >> 172.21.15.44:0/1432525451 conn(0x56371f261440 msgr2=0x56371f261820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.886+0000 7fe238875740 1 -- 172.21.15.44:0/1432525451 shutdown_connections 2024-04-21T23:13:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.886+0000 7fe238875740 1 -- 172.21.15.44:0/1432525451 wait complete. 2024-04-21T23:13:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.887+0000 7fe238875740 1 -- start start 2024-04-21T23:13:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.887+0000 7fe238875740 1 Processor -- start 2024-04-21T23:13:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.887+0000 7fe238875740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f266b20 0x56371f285500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.887+0000 7fe238875740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f267440 0x56371f285a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.887+0000 7fe238875740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f270210 0x56371f282be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.887+0000 7fe238875740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe2301ada10 con 0x56371f270210 2024-04-21T23:13:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.887+0000 7fe238875740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe2301aee80 con 0x56371f266b20 2024-04-21T23:13:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.887+0000 7fe238875740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe2301ae0e0 con 0x56371f267440 2024-04-21T23:13:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.887+0000 7fe238875740 10 client.0 _ll_register_callbacks cb 0x7fe220009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.887+0000 7fe22ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f270210 0x56371f282be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.887+0000 7fe22f7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f266b20 0x56371f285500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.887+0000 7fe22ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f270210 0x56371f282be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53348/0 (socket says 172.21.15.44:53348) 2024-04-21T23:13:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.887+0000 7fe22ffff640 1 -- 172.21.15.44:0/2118186197 learned_addr learned my addr 172.21.15.44:0/2118186197 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.887+0000 7fe22effd640 1 --2- 172.21.15.44:0/2118186197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f267440 0x56371f285a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.888+0000 7fe22f7fe640 1 -- 172.21.15.44:0/2118186197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f267440 msgr2=0x56371f285a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.888+0000 7fe22f7fe640 1 --2- 172.21.15.44:0/2118186197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f267440 0x56371f285a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.888+0000 7fe22f7fe640 1 -- 172.21.15.44:0/2118186197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f270210 msgr2=0x56371f282be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.888+0000 7fe22f7fe640 1 --2- 172.21.15.44:0/2118186197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f270210 0x56371f282be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.888+0000 7fe22f7fe640 1 -- 172.21.15.44:0/2118186197 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56371f2832d0 con 0x56371f266b20 2024-04-21T23:13:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.888+0000 7fe22ffff640 1 --2- 172.21.15.44:0/2118186197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f270210 0x56371f282be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.888+0000 7fe22f7fe640 1 --2- 172.21.15.44:0/2118186197 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f266b20 0x56371f285500 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe22409a040 tx=0x7fe220013ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c9a633ea56c0755 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.888+0000 7fe20effd640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:13:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.888+0000 7fe20effd640 1 -- 172.21.15.44:0/2118186197 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe220005020 con 0x56371f266b20 2024-04-21T23:13:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.888+0000 7fe238875740 1 -- 172.21.15.44:0/2118186197 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2240b1a80 con 0x56371f266b20 2024-04-21T23:13:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.888+0000 7fe20effd640 1 -- 172.21.15.44:0/2118186197 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fe2200051c0 con 0x56371f266b20 2024-04-21T23:13:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.888+0000 7fe20effd640 1 -- 172.21.15.44:0/2118186197 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe2200053e0 con 0x56371f266b20 2024-04-21T23:13:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.888+0000 7fe238875740 10 client.26264 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.888+0000 7fe238875740 1 -- 172.21.15.44:0/2118186197 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56371f2ea4c0 con 0x56371f266b20 2024-04-21T23:13:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.888+0000 7fe20effd640 1 -- 172.21.15.44:0/2118186197 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2240b1a80 con 0x56371f266b20 2024-04-21T23:13:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.888+0000 7fe20effd640 1 -- 172.21.15.44:0/2118186197 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fe22001f2b0 con 0x56371f266b20 2024-04-21T23:13:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.888+0000 7fe238875740 10 client.26264 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.888+0000 7fe238875740 10 client.26264 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.888+0000 7fe238875740 20 client.26264 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.889+0000 7fe238875740 1 --2- 172.21.15.44:0/2118186197 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56371f2eabe0 0x56371f2ed000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.889+0000 7fe238875740 4 client.26264 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:13:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.889+0000 7fe238875740 1 -- 172.21.15.44:0/2118186197 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56371f1d2400 con 0x56371f2eabe0 2024-04-21T23:13:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.889+0000 7fe22effd640 1 --2- 172.21.15.44:0/2118186197 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56371f2eabe0 0x56371f2ed000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.889+0000 7fe20effd640 10 client.26264 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:23.889+0000 7fe22effd640 1 --2- 172.21.15.44:0/2118186197 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56371f2eabe0 0x56371f2ed000 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba961bd8bfa86e4 server_cookie=7ff4d2b2c4984c3d in_seq=0 out_seq=0 2024-04-21T23:13:24.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:23 smithi044 ceph-mon[27934]: pgmap v141: 129 pgs: 129 active+clean; 69 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.2 MiB/s wr, 551 op/s 2024-04-21T23:13:24.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:23 smithi055 ceph-mon[31117]: pgmap v141: 129 pgs: 129 active+clean; 69 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.2 MiB/s wr, 551 op/s 2024-04-21T23:13:24.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:23 smithi069 ceph-mon[31816]: pgmap v141: 129 pgs: 129 active+clean; 69 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.2 MiB/s wr, 551 op/s 2024-04-21T23:13:24.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.265+0000 7fe20effd640 1 -- 172.21.15.44:0/2118186197 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56371f1d2400 con 0x56371f2eabe0 2024-04-21T23:13:24.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.265+0000 7fe20effd640 10 client.26264 handle_command_reply: tid=0 2024-04-21T23:13:24.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.265+0000 7fe238875740 1 client.26264 shutdown 2024-04-21T23:13:24.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.265+0000 7fe238875740 1 -- 172.21.15.44:0/2118186197 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f266b20 msgr2=0x56371f285500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:24.265 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.265+0000 7fe238875740 1 --2- 172.21.15.44:0/2118186197 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f266b20 0x56371f285500 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe22409a040 tx=0x7fe220013ed0 comp rx=0 tx=0).stop 2024-04-21T23:13:24.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.265+0000 7fe238875740 1 -- 172.21.15.44:0/2118186197 shutdown_connections 2024-04-21T23:13:24.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.265+0000 7fe238875740 1 --2- 172.21.15.44:0/2118186197 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56371f2eabe0 0x56371f2ed000 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:24.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.265+0000 7fe238875740 1 --2- 172.21.15.44:0/2118186197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56371f267440 0x56371f285a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:24.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.265+0000 7fe238875740 1 --2- 172.21.15.44:0/2118186197 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56371f270210 0x56371f282be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:24.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.265+0000 7fe238875740 1 --2- 172.21.15.44:0/2118186197 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56371f266b20 0x56371f285500 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:24.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.265+0000 7fe20effd640 0 client.26264 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:24.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.265+0000 7fe238875740 1 -- 172.21.15.44:0/2118186197 >> 172.21.15.44:0/2118186197 conn(0x56371f261440 msgr2=0x56371f26bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:24.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.266+0000 7fe238875740 1 -- 172.21.15.44:0/2118186197 shutdown_connections 2024-04-21T23:13:24.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.266+0000 7fe238875740 1 -- 172.21.15.44:0/2118186197 wait complete. 2024-04-21T23:13:24.266 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.266+0000 7fe238875740 20 client.26264 trim_cache size 0 max 16384 2024-04-21T23:13:24.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:13:24.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:13:24.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:13:24.267 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:13:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.266+0000 7fe235c88640 1 -- 172.21.15.44:0/4259958429 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe2180a2cf0 msgr2=0x7fe2180a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.266+0000 7fe235c88640 1 --2- 172.21.15.44:0/4259958429 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe2180a2cf0 0x7fe2180a51b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe228005cc0 tx=0x7fe228005f80 comp rx=0 tx=0).stop 2024-04-21T23:13:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.266+0000 7fe235c88640 1 -- 172.21.15.44:0/4259958429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2301020f0 msgr2=0x7fe2301985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.267+0000 7fe235c88640 1 --2- 172.21.15.44:0/4259958429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2301020f0 0x7fe2301985d0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fe22400e9f0 tx=0x7fe22400edf0 comp rx=0 tx=0).stop 2024-04-21T23:13:24.268 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.267+0000 7fe235c88640 1 -- 172.21.15.44:0/4259958429 shutdown_connections 2024-04-21T23:13:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.267+0000 7fe235c88640 1 --2- 172.21.15.44:0/4259958429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe230107740 0x7fe23019c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.267+0000 7fe235c88640 1 --2- 172.21.15.44:0/4259958429 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe2180a2cf0 0x7fe2180a51b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.267+0000 7fe235c88640 1 --2- 172.21.15.44:0/4259958429 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2301020f0 0x7fe2301985d0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.267+0000 7fe235c88640 1 --2- 172.21.15.44:0/4259958429 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe230101730 0x7fe230198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.267+0000 7fe235c88640 1 -- 172.21.15.44:0/4259958429 >> 172.21.15.44:0/4259958429 conn(0x7fe2300fd200 msgr2=0x7fe2300fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.267+0000 7fe235c88640 1 -- 172.21.15.44:0/4259958429 shutdown_connections 2024-04-21T23:13:24.269 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:24.267+0000 7fe235c88640 1 -- 172.21.15.44:0/4259958429 wait complete. 2024-04-21T23:13:24.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 495305 13.54 MB/sec execute 74 sec latency 228.503 ms 2024-04-21T23:13:25.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 498005 13.55 MB/sec execute 75 sec latency 36.074 ms 2024-04-21T23:13:26.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:25 smithi044 ceph-mon[27934]: pgmap v142: 129 pgs: 129 active+clean; 69 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 269 op/s 2024-04-21T23:13:26.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:25 smithi055 ceph-mon[31117]: pgmap v142: 129 pgs: 129 active+clean; 69 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 269 op/s 2024-04-21T23:13:26.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:25 smithi069 ceph-mon[31816]: pgmap v142: 129 pgs: 129 active+clean; 69 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 269 op/s 2024-04-21T23:13:26.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 501033 13.60 MB/sec execute 76 sec latency 107.302 ms 2024-04-21T23:13:27.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 503081 13.52 MB/sec execute 77 sec latency 58.611 ms 2024-04-21T23:13:28.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:27 smithi044 ceph-mon[27934]: pgmap v143: 129 pgs: 129 active+clean; 55 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 521 op/s 2024-04-21T23:13:28.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:27 smithi069 ceph-mon[31816]: pgmap v143: 129 pgs: 129 active+clean; 55 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 521 op/s 2024-04-21T23:13:28.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:27 smithi055 ceph-mon[31117]: pgmap v143: 129 pgs: 129 active+clean; 55 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 521 op/s 2024-04-21T23:13:28.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 505671 13.54 MB/sec execute 78 sec latency 14.365 ms 2024-04-21T23:13:29.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 508615 13.59 MB/sec execute 79 sec latency 19.236 ms 2024-04-21T23:13:30.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:29 smithi044 ceph-mon[27934]: pgmap v144: 129 pgs: 129 active+clean; 56 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 3.6 MiB/s wr, 471 op/s 2024-04-21T23:13:30.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:29 smithi055 ceph-mon[31117]: pgmap v144: 129 pgs: 129 active+clean; 56 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 3.6 MiB/s wr, 471 op/s 2024-04-21T23:13:30.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:29 smithi069 ceph-mon[31816]: pgmap v144: 129 pgs: 129 active+clean; 56 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 3.6 MiB/s wr, 471 op/s 2024-04-21T23:13:30.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 511794 13.65 MB/sec execute 80 sec latency 36.055 ms 2024-04-21T23:13:31.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 514477 13.63 MB/sec execute 81 sec latency 24.120 ms 2024-04-21T23:13:32.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:31 smithi044 ceph-mon[27934]: pgmap v145: 129 pgs: 129 active+clean; 56 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 3.2 MiB/s wr, 453 op/s 2024-04-21T23:13:32.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:31 smithi069 ceph-mon[31816]: pgmap v145: 129 pgs: 129 active+clean; 56 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 3.2 MiB/s wr, 453 op/s 2024-04-21T23:13:32.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:31 smithi055 ceph-mon[31117]: pgmap v145: 129 pgs: 129 active+clean; 56 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 3.2 MiB/s wr, 453 op/s 2024-04-21T23:13:32.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 516038 13.56 MB/sec execute 82 sec latency 49.664 ms 2024-04-21T23:13:33.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 519133 13.62 MB/sec execute 83 sec latency 8.347 ms 2024-04-21T23:13:34.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:33 smithi044 ceph-mon[27934]: pgmap v146: 129 pgs: 129 active+clean; 55 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 565 op/s 2024-04-21T23:13:34.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:33 smithi055 ceph-mon[31117]: pgmap v146: 129 pgs: 129 active+clean; 55 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 565 op/s 2024-04-21T23:13:34.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:33 smithi069 ceph-mon[31816]: pgmap v146: 129 pgs: 129 active+clean; 55 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 565 op/s 2024-04-21T23:13:34.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 522176 13.67 MB/sec execute 84 sec latency 62.986 ms 2024-04-21T23:13:35.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 524644 13.61 MB/sec execute 85 sec latency 47.737 ms 2024-04-21T23:13:36.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:35 smithi044 ceph-mon[27934]: pgmap v147: 129 pgs: 129 active+clean; 55 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 2.4 MiB/s wr, 435 op/s 2024-04-21T23:13:36.213 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:35 smithi055 ceph-mon[31117]: pgmap v147: 129 pgs: 129 active+clean; 55 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 2.4 MiB/s wr, 435 op/s 2024-04-21T23:13:36.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:35 smithi069 ceph-mon[31816]: pgmap v147: 129 pgs: 129 active+clean; 55 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 2.4 MiB/s wr, 435 op/s 2024-04-21T23:13:36.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 526982 13.62 MB/sec execute 86 sec latency 14.708 ms 2024-04-21T23:13:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.371+0000 7f25a62ae640 1 -- 172.21.15.44:0/941264055 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25a01032d0 msgr2=0x7f25a010b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.371+0000 7f25a62ae640 1 --2- 172.21.15.44:0/941264055 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25a01032d0 0x7f25a010b7b0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f2594009900 tx=0x7f259402bcd0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.371+0000 7f25a62ae640 1 -- 172.21.15.44:0/941264055 shutdown_connections 2024-04-21T23:13:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.371+0000 7f25a62ae640 1 --2- 172.21.15.44:0/941264055 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25a010bd80 0x7f25a010e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.371+0000 7f25a62ae640 1 --2- 172.21.15.44:0/941264055 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25a01032d0 0x7f25a010b7b0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.371+0000 7f25a62ae640 1 --2- 172.21.15.44:0/941264055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25a01029b0 0x7f25a0102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.371+0000 7f25a62ae640 1 -- 172.21.15.44:0/941264055 >> 172.21.15.44:0/941264055 conn(0x7f25a0077760 msgr2=0x7f25a0077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.371+0000 7f25a62ae640 1 -- 172.21.15.44:0/941264055 shutdown_connections 2024-04-21T23:13:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.371+0000 7f25a62ae640 1 -- 172.21.15.44:0/941264055 wait complete. 2024-04-21T23:13:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.372+0000 7f25a62ae640 1 Processor -- start 2024-04-21T23:13:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.372+0000 7f25a62ae640 1 -- start start 2024-04-21T23:13:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.372+0000 7f25a62ae640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25a01029b0 0x7f25a0198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.372+0000 7f25a62ae640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25a01032d0 0x7f25a01985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.372+0000 7f25a62ae640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25a010bd80 0x7f25a019c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.372+0000 7f25a62ae640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f25a0111b60 con 0x7f25a01029b0 2024-04-21T23:13:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.372+0000 7f25a62ae640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f25a01119e0 con 0x7f25a010bd80 2024-04-21T23:13:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.372+0000 7f25a62ae640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f25a0111ce0 con 0x7f25a01032d0 2024-04-21T23:13:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.372+0000 7f259ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25a01029b0 0x7f25a0198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.372+0000 7f259ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25a01029b0 0x7f25a0198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38634/0 (socket says 172.21.15.44:38634) 2024-04-21T23:13:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.372+0000 7f259ffff640 1 -- 172.21.15.44:0/2132076001 learned_addr learned my addr 172.21.15.44:0/2132076001 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.372+0000 7f25a484b640 1 --2- 172.21.15.44:0/2132076001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25a010bd80 0x7f25a019c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.372+0000 7f259f7fe640 1 --2- 172.21.15.44:0/2132076001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25a01032d0 0x7f25a01985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.373+0000 7f259ffff640 1 -- 172.21.15.44:0/2132076001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25a01032d0 msgr2=0x7f25a01985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.373+0000 7f259ffff640 1 --2- 172.21.15.44:0/2132076001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25a01032d0 0x7f25a01985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.373+0000 7f259ffff640 1 -- 172.21.15.44:0/2132076001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25a010bd80 msgr2=0x7f25a019c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:37.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.373+0000 7f259ffff640 1 --2- 172.21.15.44:0/2132076001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25a010bd80 0x7f25a019c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.373+0000 7f259ffff640 1 -- 172.21.15.44:0/2132076001 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a019d020 con 0x7f25a01029b0 2024-04-21T23:13:37.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.373+0000 7f259ffff640 1 --2- 172.21.15.44:0/2132076001 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25a01029b0 0x7f25a0198080 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f259000e9f0 tx=0x7f259000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:37.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.373+0000 7f259d7fa640 1 -- 172.21.15.44:0/2132076001 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259000cd80 con 0x7f25a01029b0 2024-04-21T23:13:37.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.373+0000 7f25a62ae640 1 -- 172.21.15.44:0/2132076001 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a019d310 con 0x7f25a01029b0 2024-04-21T23:13:37.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.373+0000 7f259d7fa640 1 -- 172.21.15.44:0/2132076001 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2590005500 con 0x7f25a01029b0 2024-04-21T23:13:37.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.373+0000 7f259d7fa640 1 -- 172.21.15.44:0/2132076001 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2590010690 con 0x7f25a01029b0 2024-04-21T23:13:37.378 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.373+0000 7f25a62ae640 1 -- 172.21.15.44:0/2132076001 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a01a4ba0 con 0x7f25a01029b0 2024-04-21T23:13:37.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.375+0000 7f259d7fa640 1 -- 172.21.15.44:0/2132076001 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f25900046c0 con 0x7f25a01029b0 2024-04-21T23:13:37.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.376+0000 7f25a8f72740 1 Processor -- start 2024-04-21T23:13:37.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.376+0000 7f25a8f72740 1 -- start start 2024-04-21T23:13:37.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.376+0000 7f259d7fa640 1 --2- 172.21.15.44:0/2132076001 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f25880a2cf0 0x7f25880a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:37.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.376+0000 7f25a8f72740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 0x5589788f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:37.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.376+0000 7f25a8f72740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f6ee0 0x5589789040e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:37.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.376+0000 7f25a8f72740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558978904620 0x558978906a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:37.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.376+0000 7f25a8f72740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55897885ab70 con 0x5589788f65c0 2024-04-21T23:13:37.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.376+0000 7f25a8f72740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x558978803470 con 0x558978904620 2024-04-21T23:13:37.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.376+0000 7f25a8f72740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558978856b30 con 0x5589788f6ee0 2024-04-21T23:13:37.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.376+0000 7f259d7fa640 1 -- 172.21.15.44:0/2132076001 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f25900cfc10 con 0x7f25a01029b0 2024-04-21T23:13:37.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.377+0000 7f259ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 0x5589788f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:37.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.377+0000 7f25a484b640 1 --2- 172.21.15.44:0/2132076001 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f25880a2cf0 0x7f25880a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:37.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.377+0000 7f259ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 0x5589788f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38642/0 (socket says 172.21.15.44:38642) 2024-04-21T23:13:37.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.377+0000 7f259ffff640 1 -- 172.21.15.44:0/302157048 learned_addr learned my addr 172.21.15.44:0/302157048 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:37.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.377+0000 7f259f7fe640 1 --2- 172.21.15.44:0/302157048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f6ee0 0x5589789040e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:37.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.377+0000 7f25a484b640 1 --2- 172.21.15.44:0/302157048 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558978904620 0x558978906a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:37.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.377+0000 7f259ffff640 1 -- 172.21.15.44:0/302157048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f6ee0 msgr2=0x5589789040e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:37.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.377+0000 7f259ffff640 1 --2- 172.21.15.44:0/302157048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f6ee0 0x5589789040e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.377+0000 7f259ffff640 1 -- 172.21.15.44:0/302157048 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558978904620 msgr2=0x558978906a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:37.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.377+0000 7f259ffff640 1 --2- 172.21.15.44:0/302157048 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558978904620 0x558978906a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.377+0000 7f259ffff640 1 -- 172.21.15.44:0/302157048 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55897856f210 con 0x5589788f65c0 2024-04-21T23:13:37.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.377+0000 7f259f7fe640 1 --2- 172.21.15.44:0/302157048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f6ee0 0x5589789040e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:37.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.377+0000 7f259ffff640 1 --2- 172.21.15.44:0/302157048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 0x5589788f69a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f2590098760 tx=0x7f259009b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40afd04b2a69dfa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:37.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.377+0000 7f25a484b640 1 --2- 172.21.15.44:0/2132076001 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f25880a2cf0 0x7f25880a51b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f25a006a040 tx=0x7f2598013ae0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:37.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.377+0000 7f25867fc640 1 -- 172.21.15.44:0/302157048 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259009bd90 con 0x5589788f65c0 2024-04-21T23:13:37.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.377+0000 7f25867fc640 1 -- 172.21.15.44:0/302157048 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25900a0140 con 0x5589788f65c0 2024-04-21T23:13:37.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.377+0000 7f25867fc640 1 -- 172.21.15.44:0/302157048 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2590005190 con 0x5589788f65c0 2024-04-21T23:13:37.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.378+0000 7f25a8f72740 1 -- 172.21.15.44:0/302157048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 msgr2=0x5589788f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:37.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.378+0000 7f25a8f72740 1 --2- 172.21.15.44:0/302157048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 0x5589788f69a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f2590098760 tx=0x7f259009b990 comp rx=0 tx=0).stop 2024-04-21T23:13:37.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.378+0000 7f25a8f72740 1 -- 172.21.15.44:0/302157048 shutdown_connections 2024-04-21T23:13:37.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.378+0000 7f25a8f72740 1 --2- 172.21.15.44:0/302157048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f6ee0 0x5589789040e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.378+0000 7f25a8f72740 1 --2- 172.21.15.44:0/302157048 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558978904620 0x558978906a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.378+0000 7f25a8f72740 1 --2- 172.21.15.44:0/302157048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 0x5589788f69a0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.378+0000 7f25a8f72740 1 -- 172.21.15.44:0/302157048 >> 172.21.15.44:0/302157048 conn(0x5589788f1440 msgr2=0x5589788f1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:37.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.378+0000 7f25a8f72740 1 -- 172.21.15.44:0/302157048 shutdown_connections 2024-04-21T23:13:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.378+0000 7f25a8f72740 1 -- 172.21.15.44:0/302157048 wait complete. 2024-04-21T23:13:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.379+0000 7f25a8f72740 1 -- start start 2024-04-21T23:13:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.379+0000 7f25a8f72740 1 Processor -- start 2024-04-21T23:13:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.379+0000 7f25a8f72740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 0x558978901680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.379+0000 7f25a8f72740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f6ee0 0x558978901bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.379+0000 7f25a8f72740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558978904620 0x558978902100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.379+0000 7f25a8f72740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558978907250 con 0x5589788f65c0 2024-04-21T23:13:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.379+0000 7f25a8f72740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x558978816a10 con 0x558978904620 2024-04-21T23:13:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.379+0000 7f25a8f72740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5589789073d0 con 0x5589788f6ee0 2024-04-21T23:13:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.379+0000 7f25a8f72740 10 client.0 _ll_register_callbacks cb 0x558978851b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.379+0000 7f259ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 0x558978901680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.379+0000 7f259ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 0x558978901680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38650/0 (socket says 172.21.15.44:38650) 2024-04-21T23:13:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.379+0000 7f259ffff640 1 -- 172.21.15.44:0/206865981 learned_addr learned my addr 172.21.15.44:0/206865981 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:37.389 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.379+0000 7f25a484b640 1 --2- 172.21.15.44:0/206865981 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558978904620 0x558978902100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.380+0000 7f259f7fe640 1 --2- 172.21.15.44:0/206865981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f6ee0 0x558978901bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.380+0000 7f259ffff640 1 -- 172.21.15.44:0/206865981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f6ee0 msgr2=0x558978901bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.380+0000 7f259ffff640 1 --2- 172.21.15.44:0/206865981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f6ee0 0x558978901bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.380+0000 7f259ffff640 1 -- 172.21.15.44:0/206865981 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558978904620 msgr2=0x558978902100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.380+0000 7f259ffff640 1 --2- 172.21.15.44:0/206865981 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558978904620 0x558978902100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.380+0000 7f259ffff640 1 -- 172.21.15.44:0/206865981 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55897863b620 con 0x5589788f65c0 2024-04-21T23:13:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.380+0000 7f259ffff640 1 --2- 172.21.15.44:0/206865981 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 0x558978901680 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f25900cd070 tx=0x7f259009a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1017a5c740ab5454 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.380+0000 7f257f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:13:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.380+0000 7f257f7fe640 1 -- 172.21.15.44:0/206865981 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25900a4770 con 0x5589788f65c0 2024-04-21T23:13:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.380+0000 7f257f7fe640 1 -- 172.21.15.44:0/206865981 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f259009b1e0 con 0x5589788f65c0 2024-04-21T23:13:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.380+0000 7f25a8f72740 1 -- 172.21.15.44:0/206865981 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25900a0140 con 0x5589788f65c0 2024-04-21T23:13:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.380+0000 7f257f7fe640 1 -- 172.21.15.44:0/206865981 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f25900ac050 con 0x5589788f65c0 2024-04-21T23:13:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.380+0000 7f25a8f72740 10 client.16710 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.380+0000 7f25a8f72740 1 -- 172.21.15.44:0/206865981 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558978913560 con 0x5589788f65c0 2024-04-21T23:13:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.380+0000 7f257f7fe640 1 -- 172.21.15.44:0/206865981 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25900a0140 con 0x5589788f65c0 2024-04-21T23:13:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.381+0000 7f257f7fe640 1 -- 172.21.15.44:0/206865981 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f259009ac50 con 0x5589788f65c0 2024-04-21T23:13:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.381+0000 7f25a8f72740 10 client.16710 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.381+0000 7f25a8f72740 10 client.16710 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.381+0000 7f25a8f72740 20 client.16710 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.381+0000 7f25a8f72740 1 --2- 172.21.15.44:0/206865981 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5589789148b0 0x558978916ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.381+0000 7f25a8f72740 4 client.16710 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:13:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.381+0000 7f25a8f72740 1 -- 172.21.15.44:0/206865981 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589788680b0 con 0x5589789148b0 2024-04-21T23:13:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.381+0000 7f259f7fe640 1 --2- 172.21.15.44:0/206865981 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5589789148b0 0x558978916ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.381+0000 7f257f7fe640 10 client.16710 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.381+0000 7f259f7fe640 1 --2- 172.21.15.44:0/206865981 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5589789148b0 0x558978916ca0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fce406efc99386d server_cookie=bdc01c60281230d in_seq=0 out_seq=0 2024-04-21T23:13:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.382+0000 7f257f7fe640 1 -- 172.21.15.44:0/206865981 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5589788680b0 con 0x5589789148b0 2024-04-21T23:13:37.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.382+0000 7f257f7fe640 10 client.16710 handle_command_reply: tid=0 2024-04-21T23:13:37.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.382+0000 7f25a8f72740 1 client.16710 shutdown 2024-04-21T23:13:37.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.383+0000 7f25a8f72740 1 -- 172.21.15.44:0/206865981 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 msgr2=0x558978901680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:37.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.383+0000 7f25a8f72740 1 --2- 172.21.15.44:0/206865981 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 0x558978901680 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f25900cd070 tx=0x7f259009a2a0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.383+0000 7f25a8f72740 1 -- 172.21.15.44:0/206865981 shutdown_connections 2024-04-21T23:13:37.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.383+0000 7f25a8f72740 1 --2- 172.21.15.44:0/206865981 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5589789148b0 0x558978916ca0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.383+0000 7f25a8f72740 1 --2- 172.21.15.44:0/206865981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f6ee0 0x558978901bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.383+0000 7f25a8f72740 1 --2- 172.21.15.44:0/206865981 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558978904620 0x558978902100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.383+0000 7f25a8f72740 1 --2- 172.21.15.44:0/206865981 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 0x558978901680 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.383+0000 7f25a8f72740 1 -- 172.21.15.44:0/206865981 >> 172.21.15.44:0/206865981 conn(0x5589788f1440 msgr2=0x5589788f3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:37.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.383+0000 7f257f7fe640 0 client.16710 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:37.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.383+0000 7f25a8f72740 1 -- 172.21.15.44:0/206865981 shutdown_connections 2024-04-21T23:13:37.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.383+0000 7f25a8f72740 1 -- 172.21.15.44:0/206865981 wait complete. 2024-04-21T23:13:37.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.383+0000 7f25a8f72740 20 client.16710 trim_cache size 0 max 16384 2024-04-21T23:13:37.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.399+0000 7f25a8f72740 1 Processor -- start 2024-04-21T23:13:37.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.399+0000 7f25a8f72740 1 -- start start 2024-04-21T23:13:37.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.400+0000 7f25a8f72740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f65c0 0x558978903db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:37.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.400+0000 7f25a8f72740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5589788f6ee0 0x558978901f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:37.400 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.400+0000 7f25a8f72740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558978904620 0x5589788fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.400+0000 7f25a8f72740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f25a01aaa00 con 0x558978904620 2024-04-21T23:13:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.400+0000 7f25a8f72740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f25a01ad280 con 0x5589788f6ee0 2024-04-21T23:13:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.400+0000 7f25a8f72740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f25a01ab070 con 0x5589788f65c0 2024-04-21T23:13:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.400+0000 7f25a484b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558978904620 0x5589788fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.400+0000 7f259f7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5589788f6ee0 0x558978901f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.400+0000 7f259f7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5589788f6ee0 0x558978901f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:57436/0 (socket says 172.21.15.44:57436) 2024-04-21T23:13:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.400+0000 7f259f7fe640 1 -- 172.21.15.44:0/2008883693 learned_addr learned my addr 172.21.15.44:0/2008883693 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.400+0000 7f259ffff640 1 --2- 172.21.15.44:0/2008883693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f65c0 0x558978903db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.400+0000 7f259f7fe640 1 -- 172.21.15.44:0/2008883693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f65c0 msgr2=0x558978903db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.400+0000 7f259f7fe640 1 --2- 172.21.15.44:0/2008883693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f65c0 0x558978903db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.400+0000 7f259f7fe640 1 -- 172.21.15.44:0/2008883693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558978904620 msgr2=0x5589788fe920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.400+0000 7f259f7fe640 1 --2- 172.21.15.44:0/2008883693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558978904620 0x5589788fe920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.400+0000 7f259f7fe640 1 -- 172.21.15.44:0/2008883693 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589788f3660 con 0x5589788f6ee0 2024-04-21T23:13:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.400+0000 7f259f7fe640 1 --2- 172.21.15.44:0/2008883693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5589788f6ee0 0x558978901f20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f25940096c0 tx=0x7f2594009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44a7f5e0cb60416a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:37.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.401+0000 7f2585ffb640 1 -- 172.21.15.44:0/2008883693 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2594048bc0 con 0x5589788f6ee0 2024-04-21T23:13:37.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.401+0000 7f2585ffb640 1 -- 172.21.15.44:0/2008883693 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f259404d070 con 0x5589788f6ee0 2024-04-21T23:13:37.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.401+0000 7f2585ffb640 1 -- 172.21.15.44:0/2008883693 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25940046c0 con 0x5589788f6ee0 2024-04-21T23:13:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.401+0000 7f25a8f72740 1 -- 172.21.15.44:0/2008883693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5589788f6ee0 msgr2=0x558978901f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.401+0000 7f25a8f72740 1 --2- 172.21.15.44:0/2008883693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5589788f6ee0 0x558978901f20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f25940096c0 tx=0x7f2594009260 comp rx=0 tx=0).stop 2024-04-21T23:13:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.401+0000 7f25a8f72740 1 -- 172.21.15.44:0/2008883693 shutdown_connections 2024-04-21T23:13:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.401+0000 7f25a8f72740 1 --2- 172.21.15.44:0/2008883693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558978904620 0x5589788fe920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.401+0000 7f25a8f72740 1 --2- 172.21.15.44:0/2008883693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5589788f6ee0 0x558978901f20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.401+0000 7f25a8f72740 1 --2- 172.21.15.44:0/2008883693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f65c0 0x558978903db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.401+0000 7f25a8f72740 1 -- 172.21.15.44:0/2008883693 >> 172.21.15.44:0/2008883693 conn(0x5589788f1440 msgr2=0x5589788fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.401+0000 7f25a8f72740 1 -- 172.21.15.44:0/2008883693 shutdown_connections 2024-04-21T23:13:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.401+0000 7f25a8f72740 1 -- 172.21.15.44:0/2008883693 wait complete. 2024-04-21T23:13:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.402+0000 7f25a8f72740 1 -- start start 2024-04-21T23:13:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.402+0000 7f25a8f72740 1 Processor -- start 2024-04-21T23:13:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.402+0000 7f25a8f72740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 0x558978916ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.402+0000 7f25a8f72740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f6ee0 0x558978917420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.402+0000 7f25a8f72740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558978904620 0x558978972a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.402+0000 7f25a8f72740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f25a01ab740 con 0x5589788f65c0 2024-04-21T23:13:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.402+0000 7f25a8f72740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f25a01acbb0 con 0x558978904620 2024-04-21T23:13:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.402+0000 7f25a8f72740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f25a01abe10 con 0x5589788f6ee0 2024-04-21T23:13:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.402+0000 7f25a8f72740 10 client.0 _ll_register_callbacks cb 0x7f2590005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.402+0000 7f259ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 0x558978916ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.402+0000 7f259f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f6ee0 0x558978917420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.402+0000 7f259ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 0x558978916ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38676/0 (socket says 172.21.15.44:38676) 2024-04-21T23:13:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.402+0000 7f259ffff640 1 -- 172.21.15.44:0/2890529943 learned_addr learned my addr 172.21.15.44:0/2890529943 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.402+0000 7f25a484b640 1 --2- 172.21.15.44:0/2890529943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558978904620 0x558978972a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.403+0000 7f259f7fe640 1 -- 172.21.15.44:0/2890529943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558978904620 msgr2=0x558978972a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.403+0000 7f259f7fe640 1 --2- 172.21.15.44:0/2890529943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558978904620 0x558978972a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.403+0000 7f259f7fe640 1 -- 172.21.15.44:0/2890529943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 msgr2=0x558978916ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.403+0000 7f259f7fe640 1 --2- 172.21.15.44:0/2890529943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 0x558978916ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.403+0000 7f259f7fe640 1 -- 172.21.15.44:0/2890529943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589789730c0 con 0x5589788f6ee0 2024-04-21T23:13:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.403+0000 7f259ffff640 1 --2- 172.21.15.44:0/2890529943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 0x558978916ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.403+0000 7f259f7fe640 1 --2- 172.21.15.44:0/2890529943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f6ee0 0x558978917420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f259402c210 tx=0x7f2594004330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42f264f463bb55cc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.403+0000 7f257f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:13:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.403+0000 7f257f7fe640 1 -- 172.21.15.44:0/2890529943 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259402c920 con 0x5589788f6ee0 2024-04-21T23:13:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.403+0000 7f25a8f72740 1 -- 172.21.15.44:0/2890529943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f259404d070 con 0x5589788f6ee0 2024-04-21T23:13:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.403+0000 7f257f7fe640 1 -- 172.21.15.44:0/2890529943 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f2594046040 con 0x5589788f6ee0 2024-04-21T23:13:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.403+0000 7f257f7fe640 1 -- 172.21.15.44:0/2890529943 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f259405d050 con 0x5589788f6ee0 2024-04-21T23:13:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.403+0000 7f25a8f72740 10 client.26314 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.403+0000 7f25a8f72740 1 -- 172.21.15.44:0/2890529943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55897897a590 con 0x5589788f6ee0 2024-04-21T23:13:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.403+0000 7f257f7fe640 1 -- 172.21.15.44:0/2890529943 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f259404d070 con 0x5589788f6ee0 2024-04-21T23:13:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.404+0000 7f257f7fe640 1 -- 172.21.15.44:0/2890529943 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f2594005000 con 0x5589788f6ee0 2024-04-21T23:13:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.404+0000 7f25a8f72740 10 client.26314 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.404+0000 7f25a8f72740 10 client.26314 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.404+0000 7f25a8f72740 20 client.26314 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.404+0000 7f25a8f72740 1 --2- 172.21.15.44:0/2890529943 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55897897ac20 0x55897897d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:37.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.404+0000 7f25a8f72740 4 client.26314 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:13:37.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.404+0000 7f25a8f72740 1 -- 172.21.15.44:0/2890529943 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558978862400 con 0x55897897ac20 2024-04-21T23:13:37.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.404+0000 7f259ffff640 1 --2- 172.21.15.44:0/2890529943 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55897897ac20 0x55897897d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:37.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.404+0000 7f257f7fe640 10 client.26314 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:37.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.404+0000 7f259ffff640 1 --2- 172.21.15.44:0/2890529943 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55897897ac20 0x55897897d040 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5d5136ef88d28f9 server_cookie=dc8a651e91c38aac in_seq=0 out_seq=0 2024-04-21T23:13:37.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.693+0000 7f257f7fe640 1 -- 172.21.15.44:0/2890529943 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558978862400 con 0x55897897ac20 2024-04-21T23:13:37.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.693+0000 7f257f7fe640 10 client.26314 handle_command_reply: tid=0 2024-04-21T23:13:37.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.693+0000 7f25a8f72740 1 client.26314 shutdown 2024-04-21T23:13:37.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.693+0000 7f25a8f72740 1 -- 172.21.15.44:0/2890529943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f6ee0 msgr2=0x558978917420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:37.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.693+0000 7f25a8f72740 1 --2- 172.21.15.44:0/2890529943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f6ee0 0x558978917420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f259402c210 tx=0x7f2594004330 comp rx=0 tx=0).stop 2024-04-21T23:13:37.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.693+0000 7f25a8f72740 1 -- 172.21.15.44:0/2890529943 shutdown_connections 2024-04-21T23:13:37.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.693+0000 7f25a8f72740 1 --2- 172.21.15.44:0/2890529943 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55897897ac20 0x55897897d040 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.693+0000 7f25a8f72740 1 --2- 172.21.15.44:0/2890529943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5589788f6ee0 0x558978917420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.693+0000 7f25a8f72740 1 --2- 172.21.15.44:0/2890529943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558978904620 0x558978972a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.693+0000 7f257f7fe640 0 client.26314 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:37.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.693+0000 7f25a8f72740 1 --2- 172.21.15.44:0/2890529943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5589788f65c0 0x558978916ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.693+0000 7f25a8f72740 1 -- 172.21.15.44:0/2890529943 >> 172.21.15.44:0/2890529943 conn(0x5589788f1440 msgr2=0x5589788f2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:37.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.693+0000 7f25a8f72740 1 -- 172.21.15.44:0/2890529943 shutdown_connections 2024-04-21T23:13:37.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.693+0000 7f25a8f72740 1 -- 172.21.15.44:0/2890529943 wait complete. 2024-04-21T23:13:37.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.693+0000 7f25a8f72740 20 client.26314 trim_cache size 0 max 16384 2024-04-21T23:13:37.693 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:13:37.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:13:37.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:13:37.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:13:37.694 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.694+0000 7f25a62ae640 1 -- 172.21.15.44:0/2132076001 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f25880a2cf0 msgr2=0x7f25880a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.694+0000 7f25a62ae640 1 --2- 172.21.15.44:0/2132076001 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f25880a2cf0 0x7f25880a51b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f25a006a040 tx=0x7f2598013ae0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.694+0000 7f25a62ae640 1 -- 172.21.15.44:0/2132076001 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25a01029b0 msgr2=0x7f25a0198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.694+0000 7f25a62ae640 1 --2- 172.21.15.44:0/2132076001 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25a01029b0 0x7f25a0198080 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f259000e9f0 tx=0x7f259000edf0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.694+0000 7f25a62ae640 1 -- 172.21.15.44:0/2132076001 shutdown_connections 2024-04-21T23:13:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.694+0000 7f25a62ae640 1 --2- 172.21.15.44:0/2132076001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25a01032d0 0x7f25a01985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.694+0000 7f25a62ae640 1 --2- 172.21.15.44:0/2132076001 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f25880a2cf0 0x7f25880a51b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.694+0000 7f25a62ae640 1 --2- 172.21.15.44:0/2132076001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f25a010bd80 0x7f25a019c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.694+0000 7f25a62ae640 1 --2- 172.21.15.44:0/2132076001 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f25a01029b0 0x7f25a0198080 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.694+0000 7f25a62ae640 1 -- 172.21.15.44:0/2132076001 >> 172.21.15.44:0/2132076001 conn(0x7f25a0077760 msgr2=0x7f25a00fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.694+0000 7f25a62ae640 1 -- 172.21.15.44:0/2132076001 shutdown_connections 2024-04-21T23:13:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:37.694+0000 7f25a62ae640 1 -- 172.21.15.44:0/2132076001 wait complete. 2024-04-21T23:13:37.699 INFO:tasks.workunit.client.0.smithi044.stdout: 1 528664 13.57 MB/sec execute 87 sec latency 198.972 ms 2024-04-21T23:13:38.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:37 smithi044 ceph-mon[27934]: pgmap v148: 129 pgs: 129 active+clean; 62 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.8 MiB/s wr, 538 op/s 2024-04-21T23:13:38.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:37 smithi055 ceph-mon[31117]: pgmap v148: 129 pgs: 129 active+clean; 62 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.8 MiB/s wr, 538 op/s 2024-04-21T23:13:38.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:37 smithi069 ceph-mon[31816]: pgmap v148: 129 pgs: 129 active+clean; 62 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.8 MiB/s wr, 538 op/s 2024-04-21T23:13:38.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 531011 13.56 MB/sec execute 88 sec latency 15.655 ms 2024-04-21T23:13:39.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 533877 13.57 MB/sec execute 89 sec latency 15.523 ms 2024-04-21T23:13:40.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:39 smithi044 ceph-mon[27934]: pgmap v149: 129 pgs: 129 active+clean; 61 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 298 op/s 2024-04-21T23:13:40.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:39 smithi055 ceph-mon[31117]: pgmap v149: 129 pgs: 129 active+clean; 61 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 298 op/s 2024-04-21T23:13:40.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:39 smithi069 ceph-mon[31816]: pgmap v149: 129 pgs: 129 active+clean; 61 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 298 op/s 2024-04-21T23:13:40.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 536897 13.62 MB/sec execute 90 sec latency 27.558 ms 2024-04-21T23:13:41.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 540085 13.67 MB/sec execute 91 sec latency 15.576 ms 2024-04-21T23:13:42.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:41 smithi044 ceph-mon[27934]: pgmap v150: 129 pgs: 129 active+clean; 61 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 227 op/s 2024-04-21T23:13:42.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:41 smithi055 ceph-mon[31117]: pgmap v150: 129 pgs: 129 active+clean; 61 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 227 op/s 2024-04-21T23:13:42.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:41 smithi069 ceph-mon[31816]: pgmap v150: 129 pgs: 129 active+clean; 61 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 227 op/s 2024-04-21T23:13:42.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 541389 13.59 MB/sec execute 92 sec latency 64.733 ms 2024-04-21T23:13:43.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 544187 13.60 MB/sec execute 93 sec latency 29.576 ms 2024-04-21T23:13:44.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:43 smithi044 ceph-mon[27934]: pgmap v151: 129 pgs: 129 active+clean; 51 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 500 op/s 2024-04-21T23:13:44.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:43 smithi069 ceph-mon[31816]: pgmap v151: 129 pgs: 129 active+clean; 51 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 500 op/s 2024-04-21T23:13:44.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:43 smithi055 ceph-mon[31117]: pgmap v151: 129 pgs: 129 active+clean; 51 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 500 op/s 2024-04-21T23:13:44.359 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:13:44.359 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:13:44.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.473+0000 7f9d2c4f4640 1 -- 172.21.15.44:0/3715712652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d24100fd0 msgr2=0x7f9d2410d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:44.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.473+0000 7f9d2c4f4640 1 --2- 172.21.15.44:0/3715712652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d24100fd0 0x7f9d2410d4e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c009cd0 tx=0x7f9d1c02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.474+0000 7f9d2c4f4640 1 -- 172.21.15.44:0/3715712652 shutdown_connections 2024-04-21T23:13:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.474+0000 7f9d2c4f4640 1 --2- 172.21.15.44:0/3715712652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d24100610 0x7f9d24100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.474+0000 7f9d2c4f4640 1 --2- 172.21.15.44:0/3715712652 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d24100fd0 0x7f9d2410d4e0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.474+0000 7f9d2c4f4640 1 --2- 172.21.15.44:0/3715712652 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d241066b0 0x7f9d24106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.474+0000 7f9d2c4f4640 1 -- 172.21.15.44:0/3715712652 >> 172.21.15.44:0/3715712652 conn(0x7f9d240777f0 msgr2=0x7f9d24077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.474+0000 7f9d2c4f4640 1 -- 172.21.15.44:0/3715712652 shutdown_connections 2024-04-21T23:13:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.474+0000 7f9d2c4f4640 1 -- 172.21.15.44:0/3715712652 wait complete. 2024-04-21T23:13:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.474+0000 7f9d2c4f4640 1 Processor -- start 2024-04-21T23:13:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.474+0000 7f9d2c4f4640 1 -- start start 2024-04-21T23:13:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.474+0000 7f9d2c4f4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d24100610 0x7f9d2419a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.474+0000 7f9d2c4f4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d24100fd0 0x7f9d2419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.474+0000 7f9d2c4f4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d241066b0 0x7f9d2419e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.474+0000 7f9d2c4f4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9d241109f0 con 0x7f9d241066b0 2024-04-21T23:13:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.474+0000 7f9d2c4f4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9d24110870 con 0x7f9d24100610 2024-04-21T23:13:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.474+0000 7f9d2c4f4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9d24110b70 con 0x7f9d24100fd0 2024-04-21T23:13:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.475+0000 7f9d2aa91640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d241066b0 0x7f9d2419e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.475+0000 7f9d2a290640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d24100610 0x7f9d2419a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.475+0000 7f9d2a290640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d24100610 0x7f9d2419a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:43366/0 (socket says 172.21.15.44:43366) 2024-04-21T23:13:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.475+0000 7f9d2a290640 1 -- 172.21.15.44:0/2715582284 learned_addr learned my addr 172.21.15.44:0/2715582284 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.475+0000 7f9d2aa91640 1 --2- 172.21.15.44:0/2715582284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d241066b0 0x7f9d2419e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55048/0 (socket says 172.21.15.44:55048) 2024-04-21T23:13:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.475+0000 7f9d29a8f640 1 --2- 172.21.15.44:0/2715582284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d24100fd0 0x7f9d2419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.475+0000 7f9d2a290640 1 -- 172.21.15.44:0/2715582284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d24100fd0 msgr2=0x7f9d2419a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.475+0000 7f9d2a290640 1 --2- 172.21.15.44:0/2715582284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d24100fd0 0x7f9d2419a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.475+0000 7f9d2a290640 1 -- 172.21.15.44:0/2715582284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d241066b0 msgr2=0x7f9d2419e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.475+0000 7f9d2a290640 1 --2- 172.21.15.44:0/2715582284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d241066b0 0x7f9d2419e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.475+0000 7f9d2a290640 1 -- 172.21.15.44:0/2715582284 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d2419f0f0 con 0x7f9d24100610 2024-04-21T23:13:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.475+0000 7f9d2aa91640 1 --2- 172.21.15.44:0/2715582284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d241066b0 0x7f9d2419e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:44.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.475+0000 7f9d2a290640 1 --2- 172.21.15.44:0/2715582284 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d24100610 0x7f9d2419a100 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000e9c0 tx=0x7f9d2000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:44.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.475+0000 7f9d1b7fe640 1 -- 172.21.15.44:0/2715582284 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d2000cd80 con 0x7f9d24100610 2024-04-21T23:13:44.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.475+0000 7f9d2c4f4640 1 -- 172.21.15.44:0/2715582284 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d2419f3e0 con 0x7f9d24100610 2024-04-21T23:13:44.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.475+0000 7f9d1b7fe640 1 -- 172.21.15.44:0/2715582284 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d200035f0 con 0x7f9d24100610 2024-04-21T23:13:44.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.475+0000 7f9d1b7fe640 1 -- 172.21.15.44:0/2715582284 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d20010690 con 0x7f9d24100610 2024-04-21T23:13:44.477 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.475+0000 7f9d2c4f4640 1 -- 172.21.15.44:0/2715582284 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d241a6c70 con 0x7f9d24100610 2024-04-21T23:13:44.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.478+0000 7f9d1b7fe640 1 -- 172.21.15.44:0/2715582284 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f9d200037a0 con 0x7f9d24100610 2024-04-21T23:13:44.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.478+0000 7f9d2f1ba740 1 Processor -- start 2024-04-21T23:13:44.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.478+0000 7f9d2f1ba740 1 -- start start 2024-04-21T23:13:44.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.478+0000 7f9d2f1ba740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edbf30 0x55d871edc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.478+0000 7f9d2f1ba740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871edc850 0x55d871ee4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.478+0000 7f9d2f1ba740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871ee5430 0x55d871ee7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.478+0000 7f9d2f1ba740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d871e40560 con 0x55d871ee5430 2024-04-21T23:13:44.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.478+0000 7f9d2f1ba740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55d871de8730 con 0x55d871edc850 2024-04-21T23:13:44.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.478+0000 7f9d2f1ba740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d871e3c520 con 0x55d871edbf30 2024-04-21T23:13:44.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.479+0000 7f9d2aa91640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edbf30 0x55d871edc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.479+0000 7f9d2a290640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871edc850 0x55d871ee4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.479+0000 7f9d29a8f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871ee5430 0x55d871ee7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.479+0000 7f9d2aa91640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edbf30 0x55d871edc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:37110/0 (socket says 172.21.15.44:37110) 2024-04-21T23:13:44.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.479+0000 7f9d2aa91640 1 -- 172.21.15.44:0/2262371005 learned_addr learned my addr 172.21.15.44:0/2262371005 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:44.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.479+0000 7f9d1b7fe640 1 --2- 172.21.15.44:0/2715582284 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9d100a2d00 0x7f9d100a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.479+0000 7f9d2a290640 1 -- 172.21.15.44:0/2262371005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edbf30 msgr2=0x55d871edc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:44.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.479+0000 7f9d2a290640 1 --2- 172.21.15.44:0/2262371005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edbf30 0x55d871edc310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.479+0000 7f9d2a290640 1 -- 172.21.15.44:0/2262371005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871ee5430 msgr2=0x55d871ee7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:44.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.479+0000 7f9d2a290640 1 --2- 172.21.15.44:0/2262371005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871ee5430 0x55d871ee7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.479+0000 7f9d2a290640 1 -- 172.21.15.44:0/2262371005 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d871c1e7d0 con 0x55d871edc850 2024-04-21T23:13:44.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.479+0000 7f9d2aa91640 1 --2- 172.21.15.44:0/2262371005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edbf30 0x55d871edc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:44.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.479+0000 7f9d29a8f640 1 --2- 172.21.15.44:0/2262371005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871ee5430 0x55d871ee7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:44.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.479+0000 7f9d2aa91640 1 --2- 172.21.15.44:0/2715582284 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9d100a2d00 0x7f9d100a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.479+0000 7f9d1b7fe640 1 -- 172.21.15.44:0/2715582284 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f9d200cfbb0 con 0x7f9d24100610 2024-04-21T23:13:44.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.479+0000 7f9d2a290640 1 --2- 172.21.15.44:0/2262371005 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871edc850 0x55d871ee4ef0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9d200ccf60 tx=0x7f9d20020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c36854e3df9c4d30 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:44.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.479+0000 7f9d2aa91640 1 --2- 172.21.15.44:0/2715582284 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9d100a2d00 0x7f9d100a51c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c009a60 tx=0x7f9d1c02e040 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:44.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.480+0000 7f9d18bfe640 1 -- 172.21.15.44:0/2262371005 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d2009bb00 con 0x55d871edc850 2024-04-21T23:13:44.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.480+0000 7f9d18bfe640 1 -- 172.21.15.44:0/2262371005 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d2009c430 con 0x55d871edc850 2024-04-21T23:13:44.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.480+0000 7f9d18bfe640 1 -- 172.21.15.44:0/2262371005 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d2009b690 con 0x55d871edc850 2024-04-21T23:13:44.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.480+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/2262371005 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871edc850 msgr2=0x55d871ee4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:44.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.480+0000 7f9d2f1ba740 1 --2- 172.21.15.44:0/2262371005 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871edc850 0x55d871ee4ef0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9d200ccf60 tx=0x7f9d20020000 comp rx=0 tx=0).stop 2024-04-21T23:13:44.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.480+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/2262371005 shutdown_connections 2024-04-21T23:13:44.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.480+0000 7f9d2f1ba740 1 --2- 172.21.15.44:0/2262371005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871ee5430 0x55d871ee7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.480+0000 7f9d2f1ba740 1 --2- 172.21.15.44:0/2262371005 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871edc850 0x55d871ee4ef0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.480+0000 7f9d2f1ba740 1 --2- 172.21.15.44:0/2262371005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edbf30 0x55d871edc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.480+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/2262371005 >> 172.21.15.44:0/2262371005 conn(0x55d871ed6660 msgr2=0x55d871ed6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:44.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.480+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/2262371005 shutdown_connections 2024-04-21T23:13:44.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.480+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/2262371005 wait complete. 2024-04-21T23:13:44.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.481+0000 7f9d2f1ba740 1 -- start start 2024-04-21T23:13:44.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.481+0000 7f9d2f1ba740 1 Processor -- start 2024-04-21T23:13:44.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.481+0000 7f9d2f1ba740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871edbf30 0x55d871ee24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.481+0000 7f9d2f1ba740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edc850 0x55d871ee29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.481+0000 7f9d2f1ba740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871ee5430 0x55d871ee2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.481+0000 7f9d2f1ba740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d871ee81c0 con 0x55d871ee5430 2024-04-21T23:13:44.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.481+0000 7f9d2f1ba740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55d871dfbe00 con 0x55d871edbf30 2024-04-21T23:13:44.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.481+0000 7f9d2f1ba740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d871ee8340 con 0x55d871edc850 2024-04-21T23:13:44.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.481+0000 7f9d2aa91640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871ee5430 0x55d871ee2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.481+0000 7f9d2a290640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871edbf30 0x55d871ee24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.481+0000 7f9d2f1ba740 10 client.0 _ll_register_callbacks cb 0x55d871e36eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:44.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.481+0000 7f9d2a290640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871edbf30 0x55d871ee24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:43392/0 (socket says 172.21.15.44:43392) 2024-04-21T23:13:44.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.481+0000 7f9d2aa91640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871ee5430 0x55d871ee2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55064/0 (socket says 172.21.15.44:55064) 2024-04-21T23:13:44.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.481+0000 7f9d2a290640 1 -- 172.21.15.44:0/1816916316 learned_addr learned my addr 172.21.15.44:0/1816916316 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:44.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.481+0000 7f9d29a8f640 1 --2- 172.21.15.44:0/1816916316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edc850 0x55d871ee29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.481+0000 7f9d2aa91640 1 -- 172.21.15.44:0/1816916316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edc850 msgr2=0x55d871ee29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:44.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.481+0000 7f9d2aa91640 1 --2- 172.21.15.44:0/1816916316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edc850 0x55d871ee29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.481+0000 7f9d2aa91640 1 -- 172.21.15.44:0/1816916316 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871edbf30 msgr2=0x55d871ee24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:44.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.481+0000 7f9d2aa91640 1 --2- 172.21.15.44:0/1816916316 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871edbf30 0x55d871ee24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.482+0000 7f9d2aa91640 1 -- 172.21.15.44:0/1816916316 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d871b543f0 con 0x55d871ee5430 2024-04-21T23:13:44.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.482+0000 7f9d2a290640 1 --2- 172.21.15.44:0/1816916316 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871edbf30 0x55d871ee24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:44.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.482+0000 7f9d2aa91640 1 --2- 172.21.15.44:0/1816916316 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871ee5430 0x55d871ee2f20 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c006010 tx=0x7f9d1c005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d0cf594476bc2f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:44.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.482+0000 7f9d0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:13:44.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.482+0000 7f9d0dffb640 1 -- 172.21.15.44:0/1816916316 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c052060 con 0x55d871ee5430 2024-04-21T23:13:44.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.482+0000 7f9d0dffb640 1 -- 172.21.15.44:0/1816916316 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f9d1c003d80 con 0x55d871ee5430 2024-04-21T23:13:44.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.482+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/1816916316 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d2009c430 con 0x55d871ee5430 2024-04-21T23:13:44.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.482+0000 7f9d0dffb640 1 -- 172.21.15.44:0/1816916316 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9d1c046070 con 0x55d871ee5430 2024-04-21T23:13:44.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.482+0000 7f9d2f1ba740 10 client.16728 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.482+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/1816916316 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d871b140f0 con 0x55d871ee5430 2024-04-21T23:13:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.482+0000 7f9d0dffb640 1 -- 172.21.15.44:0/1816916316 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d2009c430 con 0x55d871ee5430 2024-04-21T23:13:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.482+0000 7f9d0dffb640 1 -- 172.21.15.44:0/1816916316 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f9d1c03f040 con 0x55d871ee5430 2024-04-21T23:13:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.482+0000 7f9d2f1ba740 10 client.16728 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.482+0000 7f9d2f1ba740 10 client.16728 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.482+0000 7f9d2f1ba740 20 client.16728 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.483+0000 7f9d2f1ba740 1 --2- 172.21.15.44:0/1816916316 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d871ef57a0 0x55d871ef7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.483+0000 7f9d2f1ba740 4 client.16728 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:13:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.483+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/1816916316 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d871e4daa0 con 0x55d871ef57a0 2024-04-21T23:13:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.483+0000 7f9d2a290640 1 --2- 172.21.15.44:0/1816916316 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d871ef57a0 0x55d871ef7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.483+0000 7f9d0dffb640 10 client.16728 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.483+0000 7f9d2a290640 1 --2- 172.21.15.44:0/1816916316 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d871ef57a0 0x55d871ef7bd0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a022ac20081cd2b3 server_cookie=a0a42f07f38920a2 in_seq=0 out_seq=0 2024-04-21T23:13:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.484+0000 7f9d0dffb640 1 -- 172.21.15.44:0/1816916316 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d871e4daa0 con 0x55d871ef57a0 2024-04-21T23:13:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.484+0000 7f9d0dffb640 10 client.16728 handle_command_reply: tid=0 2024-04-21T23:13:44.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.484+0000 7f9d2f1ba740 1 client.16728 shutdown 2024-04-21T23:13:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.484+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/1816916316 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871ee5430 msgr2=0x55d871ee2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.484+0000 7f9d2f1ba740 1 --2- 172.21.15.44:0/1816916316 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871ee5430 0x55d871ee2f20 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c006010 tx=0x7f9d1c005c50 comp rx=0 tx=0).stop 2024-04-21T23:13:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.484+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/1816916316 shutdown_connections 2024-04-21T23:13:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.484+0000 7f9d2f1ba740 1 --2- 172.21.15.44:0/1816916316 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d871ef57a0 0x55d871ef7bd0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.484+0000 7f9d2f1ba740 1 --2- 172.21.15.44:0/1816916316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edc850 0x55d871ee29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.484+0000 7f9d0dffb640 0 client.16728 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.484+0000 7f9d2f1ba740 1 --2- 172.21.15.44:0/1816916316 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871ee5430 0x55d871ee2f20 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.484+0000 7f9d2f1ba740 1 --2- 172.21.15.44:0/1816916316 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871edbf30 0x55d871ee24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.484+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/1816916316 >> 172.21.15.44:0/1816916316 conn(0x55d871ed6660 msgr2=0x55d871ed83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.484+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/1816916316 shutdown_connections 2024-04-21T23:13:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.484+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/1816916316 wait complete. 2024-04-21T23:13:44.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.484+0000 7f9d2f1ba740 20 client.16728 trim_cache size 0 max 16384 2024-04-21T23:13:44.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.502+0000 7f9d2f1ba740 1 Processor -- start 2024-04-21T23:13:44.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.502+0000 7f9d2f1ba740 1 -- start start 2024-04-21T23:13:44.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.502+0000 7f9d2f1ba740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871edbf30 0x55d871ee4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.502+0000 7f9d2f1ba740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edc850 0x55d871ef1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.502+0000 7f9d2f1ba740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871ee5430 0x55d871ed6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.502+0000 7f9d2f1ba740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9d241aedf0 con 0x55d871edbf30 2024-04-21T23:13:44.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.502+0000 7f9d2f1ba740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9d241b1670 con 0x55d871ee5430 2024-04-21T23:13:44.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.502+0000 7f9d2f1ba740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9d241af460 con 0x55d871edc850 2024-04-21T23:13:44.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.502+0000 7f9d2a290640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871edbf30 0x55d871ee4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.502+0000 7f9d2a290640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871edbf30 0x55d871ee4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55078/0 (socket says 172.21.15.44:55078) 2024-04-21T23:13:44.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.502+0000 7f9d2a290640 1 -- 172.21.15.44:0/39756309 learned_addr learned my addr 172.21.15.44:0/39756309 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:44.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.502+0000 7f9d29a8f640 1 --2- 172.21.15.44:0/39756309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edc850 0x55d871ef1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.502+0000 7f9d2aa91640 1 --2- 172.21.15.44:0/39756309 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871ee5430 0x55d871ed6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.502+0000 7f9d2a290640 1 -- 172.21.15.44:0/39756309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edc850 msgr2=0x55d871ef1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:44.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.502+0000 7f9d2a290640 1 --2- 172.21.15.44:0/39756309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edc850 0x55d871ef1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.502+0000 7f9d2a290640 1 -- 172.21.15.44:0/39756309 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871ee5430 msgr2=0x55d871ed6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:44.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.502+0000 7f9d2a290640 1 --2- 172.21.15.44:0/39756309 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871ee5430 0x55d871ed6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.502+0000 7f9d2a290640 1 -- 172.21.15.44:0/39756309 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d871ede0a0 con 0x55d871edbf30 2024-04-21T23:13:44.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.503+0000 7f9d2aa91640 1 --2- 172.21.15.44:0/39756309 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871ee5430 0x55d871ed6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:13:44.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.503+0000 7f9d2a290640 1 --2- 172.21.15.44:0/39756309 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871edbf30 0x55d871ee4bd0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f9d200058c0 tx=0x7f9d20000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd954d15b39e609c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:44.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.503+0000 7f9d0ffff640 1 -- 172.21.15.44:0/39756309 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d2009b870 con 0x55d871edbf30 2024-04-21T23:13:44.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.503+0000 7f9d0ffff640 1 -- 172.21.15.44:0/39756309 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d2009c430 con 0x55d871edbf30 2024-04-21T23:13:44.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.503+0000 7f9d0ffff640 1 -- 172.21.15.44:0/39756309 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d200affc0 con 0x55d871edbf30 2024-04-21T23:13:44.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.503+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/39756309 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871edbf30 msgr2=0x55d871ee4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:44.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2f1ba740 1 --2- 172.21.15.44:0/39756309 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871edbf30 0x55d871ee4bd0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f9d200058c0 tx=0x7f9d20000bc0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/39756309 shutdown_connections 2024-04-21T23:13:44.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2f1ba740 1 --2- 172.21.15.44:0/39756309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edc850 0x55d871ef1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2f1ba740 1 --2- 172.21.15.44:0/39756309 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871ee5430 0x55d871ed6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2f1ba740 1 --2- 172.21.15.44:0/39756309 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871edbf30 0x55d871ee4bd0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/39756309 >> 172.21.15.44:0/39756309 conn(0x55d871ed6660 msgr2=0x55d871ed4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:44.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/39756309 shutdown_connections 2024-04-21T23:13:44.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/39756309 wait complete. 2024-04-21T23:13:44.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2f1ba740 1 -- start start 2024-04-21T23:13:44.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2f1ba740 1 Processor -- start 2024-04-21T23:13:44.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2f1ba740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edbf30 0x55d871eedae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2f1ba740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871edc850 0x55d871edefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2f1ba740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871ee5430 0x55d871edf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2f1ba740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9d241afb30 con 0x55d871ee5430 2024-04-21T23:13:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2f1ba740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9d241b0fa0 con 0x55d871edc850 2024-04-21T23:13:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2f1ba740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9d241b0200 con 0x55d871edbf30 2024-04-21T23:13:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2f1ba740 10 client.0 _ll_register_callbacks cb 0x7f9d20005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2aa91640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871ee5430 0x55d871edf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2a290640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edbf30 0x55d871eedae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2aa91640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871ee5430 0x55d871edf6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55082/0 (socket says 172.21.15.44:55082) 2024-04-21T23:13:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d2aa91640 1 -- 172.21.15.44:0/4065623969 learned_addr learned my addr 172.21.15.44:0/4065623969 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.504+0000 7f9d29a8f640 1 --2- 172.21.15.44:0/4065623969 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871edc850 0x55d871edefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.505+0000 7f9d2a290640 1 -- 172.21.15.44:0/4065623969 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871edc850 msgr2=0x55d871edefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.505+0000 7f9d2a290640 1 --2- 172.21.15.44:0/4065623969 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871edc850 0x55d871edefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.505+0000 7f9d2a290640 1 -- 172.21.15.44:0/4065623969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871ee5430 msgr2=0x55d871edf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.505+0000 7f9d2a290640 1 --2- 172.21.15.44:0/4065623969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871ee5430 0x55d871edf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.505+0000 7f9d2a290640 1 -- 172.21.15.44:0/4065623969 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d871f5ad10 con 0x55d871edbf30 2024-04-21T23:13:44.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.505+0000 7f9d2aa91640 1 --2- 172.21.15.44:0/4065623969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871ee5430 0x55d871edf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:44.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.505+0000 7f9d2a290640 1 --2- 172.21.15.44:0/4065623969 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edbf30 0x55d871eedae0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9d2009ba30 tx=0x7f9d200d3260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4878e5c650880e2b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:44.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.505+0000 7f9d0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:13:44.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.505+0000 7f9d0dffb640 1 -- 172.21.15.44:0/4065623969 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d2009bb00 con 0x55d871edbf30 2024-04-21T23:13:44.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.505+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/4065623969 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d2009c430 con 0x55d871edbf30 2024-04-21T23:13:44.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.505+0000 7f9d0dffb640 1 -- 172.21.15.44:0/4065623969 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f9d2009a030 con 0x55d871edbf30 2024-04-21T23:13:44.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.505+0000 7f9d0dffb640 1 -- 172.21.15.44:0/4065623969 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9d200a00e0 con 0x55d871edbf30 2024-04-21T23:13:44.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.505+0000 7f9d2f1ba740 10 client.26332 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:44.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.505+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/4065623969 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d871f5be40 con 0x55d871edbf30 2024-04-21T23:13:44.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.505+0000 7f9d0dffb640 1 -- 172.21.15.44:0/4065623969 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d2009c430 con 0x55d871edbf30 2024-04-21T23:13:44.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.506+0000 7f9d0dffb640 1 -- 172.21.15.44:0/4065623969 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f9d200d6070 con 0x55d871edbf30 2024-04-21T23:13:44.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.506+0000 7f9d2f1ba740 10 client.26332 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:44.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.506+0000 7f9d2f1ba740 10 client.26332 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:44.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.506+0000 7f9d2f1ba740 20 client.26332 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:44.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.506+0000 7f9d2f1ba740 1 --2- 172.21.15.44:0/4065623969 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d871f5c070 0x55d871f5e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.506+0000 7f9d2f1ba740 4 client.26332 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:13:44.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.506+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/4065623969 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d871e47df0 con 0x55d871f5c070 2024-04-21T23:13:44.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.506+0000 7f9d29a8f640 1 --2- 172.21.15.44:0/4065623969 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d871f5c070 0x55d871f5e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.506+0000 7f9d0dffb640 10 client.26332 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:44.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.506+0000 7f9d29a8f640 1 --2- 172.21.15.44:0/4065623969 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d871f5c070 0x55d871f5e460 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9d42786a791025d server_cookie=6561c1ae545fb688 in_seq=0 out_seq=0 2024-04-21T23:13:44.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.507+0000 7f9d0dffb640 1 -- 172.21.15.44:0/4065623969 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d871e47df0 con 0x55d871f5c070 2024-04-21T23:13:44.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.507+0000 7f9d0dffb640 10 client.26332 handle_command_reply: tid=0 2024-04-21T23:13:44.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.507+0000 7f9d2f1ba740 1 client.26332 shutdown 2024-04-21T23:13:44.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.507+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/4065623969 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edbf30 msgr2=0x55d871eedae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:44.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.507+0000 7f9d2f1ba740 1 --2- 172.21.15.44:0/4065623969 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edbf30 0x55d871eedae0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9d2009ba30 tx=0x7f9d200d3260 comp rx=0 tx=0).stop 2024-04-21T23:13:44.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.507+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/4065623969 shutdown_connections 2024-04-21T23:13:44.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.507+0000 7f9d2f1ba740 1 --2- 172.21.15.44:0/4065623969 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d871f5c070 0x55d871f5e460 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.507+0000 7f9d2f1ba740 1 --2- 172.21.15.44:0/4065623969 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d871ee5430 0x55d871edf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.507+0000 7f9d0dffb640 0 client.26332 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:44.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.507+0000 7f9d2f1ba740 1 --2- 172.21.15.44:0/4065623969 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d871edc850 0x55d871edefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.507+0000 7f9d2f1ba740 1 --2- 172.21.15.44:0/4065623969 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d871edbf30 0x55d871eedae0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.507+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/4065623969 >> 172.21.15.44:0/4065623969 conn(0x55d871ed6660 msgr2=0x55d871ee2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:44.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.507+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/4065623969 shutdown_connections 2024-04-21T23:13:44.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.507+0000 7f9d2f1ba740 1 -- 172.21.15.44:0/4065623969 wait complete. 2024-04-21T23:13:44.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.507+0000 7f9d2f1ba740 20 client.26332 trim_cache size 0 max 16384 2024-04-21T23:13:44.512 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:13:44.512 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-21T23:13:44.512 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-21T23:13:44.512 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:13:44.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.508+0000 7f9d2c4f4640 1 -- 172.21.15.44:0/2715582284 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9d100a2d00 msgr2=0x7f9d100a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:44.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.508+0000 7f9d2c4f4640 1 --2- 172.21.15.44:0/2715582284 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9d100a2d00 0x7f9d100a51c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c009a60 tx=0x7f9d1c02e040 comp rx=0 tx=0).stop 2024-04-21T23:13:44.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.508+0000 7f9d2c4f4640 1 -- 172.21.15.44:0/2715582284 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d24100610 msgr2=0x7f9d2419a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:44.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.508+0000 7f9d2c4f4640 1 --2- 172.21.15.44:0/2715582284 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d24100610 0x7f9d2419a100 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9d2000e9c0 tx=0x7f9d2000ee80 comp rx=0 tx=0).stop 2024-04-21T23:13:44.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.508+0000 7f9d2c4f4640 1 -- 172.21.15.44:0/2715582284 shutdown_connections 2024-04-21T23:13:44.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.508+0000 7f9d2c4f4640 1 --2- 172.21.15.44:0/2715582284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d24100fd0 0x7f9d2419a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.508+0000 7f9d2c4f4640 1 --2- 172.21.15.44:0/2715582284 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9d100a2d00 0x7f9d100a51c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.508+0000 7f9d2c4f4640 1 --2- 172.21.15.44:0/2715582284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9d241066b0 0x7f9d2419e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.508+0000 7f9d2c4f4640 1 --2- 172.21.15.44:0/2715582284 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9d24100610 0x7f9d2419a100 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.508+0000 7f9d2c4f4640 1 -- 172.21.15.44:0/2715582284 >> 172.21.15.44:0/2715582284 conn(0x7f9d240777f0 msgr2=0x7f9d24068a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:44.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.508+0000 7f9d2c4f4640 1 -- 172.21.15.44:0/2715582284 shutdown_connections 2024-04-21T23:13:44.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.508+0000 7f9d2c4f4640 1 -- 172.21.15.44:0/2715582284 wait complete. 2024-04-21T23:13:44.517 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:13:44.518 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:13:44.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.624+0000 7f3c38f1e640 1 -- 172.21.15.44:0/1057098451 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c34100620 msgr2=0x7f3c34100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:44.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.624+0000 7f3c38f1e640 1 --2- 172.21.15.44:0/1057098451 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c34100620 0x7f3c34100aa0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c009860 tx=0x7f3c1c02bb70 comp rx=0 tx=0).stop 2024-04-21T23:13:44.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.624+0000 7f3c38f1e640 1 -- 172.21.15.44:0/1057098451 shutdown_connections 2024-04-21T23:13:44.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.624+0000 7f3c38f1e640 1 --2- 172.21.15.44:0/1057098451 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c34100fe0 0x7f3c3410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.624+0000 7f3c38f1e640 1 --2- 172.21.15.44:0/1057098451 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c34100620 0x7f3c34100aa0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.624+0000 7f3c38f1e640 1 --2- 172.21.15.44:0/1057098451 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c34106630 0x7f3c34106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.624+0000 7f3c38f1e640 1 -- 172.21.15.44:0/1057098451 >> 172.21.15.44:0/1057098451 conn(0x7f3c34077820 msgr2=0x7f3c34077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:44.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.625+0000 7f3c38f1e640 1 -- 172.21.15.44:0/1057098451 shutdown_connections 2024-04-21T23:13:44.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.625+0000 7f3c38f1e640 1 -- 172.21.15.44:0/1057098451 wait complete. 2024-04-21T23:13:44.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.625+0000 7f3c38f1e640 1 Processor -- start 2024-04-21T23:13:44.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.625+0000 7f3c38f1e640 1 -- start start 2024-04-21T23:13:44.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c38f1e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c34100620 0x7f3c34195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c38f1e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c34100fe0 0x7f3c341962d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c38f1e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c34106630 0x7f3c3419a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c38f1e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3410f930 con 0x7f3c34100620 2024-04-21T23:13:44.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c38f1e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3410f7b0 con 0x7f3c34106630 2024-04-21T23:13:44.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c38f1e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3410fab0 con 0x7f3c34100fe0 2024-04-21T23:13:44.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c3259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c34100620 0x7f3c34195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c3259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c34100620 0x7f3c34195d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55092/0 (socket says 172.21.15.44:55092) 2024-04-21T23:13:44.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c3259c640 1 -- 172.21.15.44:0/174485280 learned_addr learned my addr 172.21.15.44:0/174485280 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:44.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c2bfff640 1 --2- 172.21.15.44:0/174485280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c34100fe0 0x7f3c341962d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c32d9d640 1 --2- 172.21.15.44:0/174485280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c34106630 0x7f3c3419a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c3259c640 1 -- 172.21.15.44:0/174485280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c34100fe0 msgr2=0x7f3c341962d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:44.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c3259c640 1 --2- 172.21.15.44:0/174485280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c34100fe0 0x7f3c341962d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c3259c640 1 -- 172.21.15.44:0/174485280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c34106630 msgr2=0x7f3c3419a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:44.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c3259c640 1 --2- 172.21.15.44:0/174485280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c34106630 0x7f3c3419a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c3259c640 1 -- 172.21.15.44:0/174485280 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c3419ad80 con 0x7f3c34100620 2024-04-21T23:13:44.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c3259c640 1 --2- 172.21.15.44:0/174485280 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c34100620 0x7f3c34195d90 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3c24004650 tx=0x7f3c24003da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:44.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c2b7fe640 1 -- 172.21.15.44:0/174485280 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c24009b60 con 0x7f3c34100620 2024-04-21T23:13:44.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c38f1e640 1 -- 172.21.15.44:0/174485280 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c3419b070 con 0x7f3c34100620 2024-04-21T23:13:44.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c2b7fe640 1 -- 172.21.15.44:0/174485280 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c24009d00 con 0x7f3c34100620 2024-04-21T23:13:44.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.626+0000 7f3c2b7fe640 1 -- 172.21.15.44:0/174485280 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c24004e60 con 0x7f3c34100620 2024-04-21T23:13:44.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.627+0000 7f3c38f1e640 1 -- 172.21.15.44:0/174485280 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c341a2900 con 0x7f3c34100620 2024-04-21T23:13:44.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.628+0000 7f3c38f1e640 1 -- 172.21.15.44:0/174485280 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c34101d60 con 0x7f3c34100620 2024-04-21T23:13:44.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.629+0000 7f3c2b7fe640 1 -- 172.21.15.44:0/174485280 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3c2401f020 con 0x7f3c34100620 2024-04-21T23:13:44.628 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.629+0000 7f3c2b7fe640 1 --2- 172.21.15.44:0/174485280 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3c200a2d00 0x7f3c200a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.630+0000 7f3c2bfff640 1 --2- 172.21.15.44:0/174485280 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3c200a2d00 0x7f3c200a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.630+0000 7f3c2b7fe640 1 -- 172.21.15.44:0/174485280 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f3c240d3120 con 0x7f3c34100620 2024-04-21T23:13:44.629 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.630+0000 7f3c2bfff640 1 --2- 172.21.15.44:0/174485280 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3c200a2d00 0x7f3c200a51c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c02c180 tx=0x7f3c1c0058d0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:44.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.634+0000 7f3c2b7fe640 1 -- 172.21.15.44:0/174485280 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f3c2409f940 con 0x7f3c34100620 2024-04-21T23:13:44.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 547282 13.65 MB/sec execute 94 sec latency 33.643 ms 2024-04-21T23:13:44.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.866+0000 7f3c38f1e640 1 -- 172.21.15.44:0/174485280 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c34068a40 con 0x7f3c34100620 2024-04-21T23:13:44.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.867+0000 7f3c2b7fe640 1 -- 172.21.15.44:0/174485280 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11791 (secure 0 0 0) 0x7f3c2409f240 con 0x7f3c34100620 2024-04-21T23:13:44.865 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:13:44.865 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:11:24.214378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25843":{"gid":25843,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3162569119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3162569119},{"type":"v1","addr":"172.21.15.69:6839","nonce":3162569119}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:13:44.866 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-04-21T23:13:44.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.869+0000 7f3c38f1e640 1 -- 172.21.15.44:0/174485280 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3c200a2d00 msgr2=0x7f3c200a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:44.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.869+0000 7f3c38f1e640 1 --2- 172.21.15.44:0/174485280 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3c200a2d00 0x7f3c200a51c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c02c180 tx=0x7f3c1c0058d0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.869+0000 7f3c38f1e640 1 -- 172.21.15.44:0/174485280 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c34100620 msgr2=0x7f3c34195d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:44.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.869+0000 7f3c38f1e640 1 --2- 172.21.15.44:0/174485280 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c34100620 0x7f3c34195d90 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3c24004650 tx=0x7f3c24003da0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.869+0000 7f3c38f1e640 1 -- 172.21.15.44:0/174485280 shutdown_connections 2024-04-21T23:13:44.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.869+0000 7f3c38f1e640 1 --2- 172.21.15.44:0/174485280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c34100fe0 0x7f3c341962d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.869+0000 7f3c38f1e640 1 --2- 172.21.15.44:0/174485280 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3c200a2d00 0x7f3c200a51c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.869+0000 7f3c38f1e640 1 --2- 172.21.15.44:0/174485280 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c34106630 0x7f3c3419a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.869+0000 7f3c38f1e640 1 --2- 172.21.15.44:0/174485280 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c34100620 0x7f3c34195d90 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.869+0000 7f3c38f1e640 1 -- 172.21.15.44:0/174485280 >> 172.21.15.44:0/174485280 conn(0x7f3c34077820 msgr2=0x7f3c34102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:44.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.869+0000 7f3c38f1e640 1 -- 172.21.15.44:0/174485280 shutdown_connections 2024-04-21T23:13:44.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.869+0000 7f3c38f1e640 1 -- 172.21.15.44:0/174485280 wait complete. 2024-04-21T23:13:44.879 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:13:44.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.987+0000 7ff6f015c640 1 -- 172.21.15.44:0/2919739557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e8103bb0 msgr2=0x7ff6e810e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:44.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.987+0000 7ff6f015c640 1 --2- 172.21.15.44:0/2919739557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e8103bb0 0x7ff6e810e6e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8009ca0 tx=0x7ff6d802c1d0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.987+0000 7ff6f015c640 1 -- 172.21.15.44:0/2919739557 shutdown_connections 2024-04-21T23:13:44.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.987+0000 7ff6f015c640 1 --2- 172.21.15.44:0/2919739557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e8103210 0x7ff6e8103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.987+0000 7ff6f015c640 1 --2- 172.21.15.44:0/2919739557 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e8103bb0 0x7ff6e810e6e0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.987+0000 7ff6f015c640 1 --2- 172.21.15.44:0/2919739557 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e8101ad0 0x7ff6e8101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.987+0000 7ff6f015c640 1 -- 172.21.15.44:0/2919739557 >> 172.21.15.44:0/2919739557 conn(0x7ff6e80fd200 msgr2=0x7ff6e80ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:44.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.987+0000 7ff6f015c640 1 -- 172.21.15.44:0/2919739557 shutdown_connections 2024-04-21T23:13:44.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.987+0000 7ff6f015c640 1 -- 172.21.15.44:0/2919739557 wait complete. 2024-04-21T23:13:44.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.988+0000 7ff6f015c640 1 Processor -- start 2024-04-21T23:13:44.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.988+0000 7ff6f015c640 1 -- start start 2024-04-21T23:13:44.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.988+0000 7ff6f015c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e8101ad0 0x7ff6e81980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.988+0000 7ff6f015c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e8103210 0x7ff6e81985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.988+0000 7ff6f015c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e8103bb0 0x7ff6e819c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.988+0000 7ff6f015c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e8111c20 con 0x7ff6e8103bb0 2024-04-21T23:13:44.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.988+0000 7ff6f015c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e8111aa0 con 0x7ff6e8101ad0 2024-04-21T23:13:44.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.988+0000 7ff6f015c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e8111da0 con 0x7ff6e8103210 2024-04-21T23:13:44.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.988+0000 7ff6ee6f9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e8103bb0 0x7ff6e819c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.988+0000 7ff6ee6f9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e8103bb0 0x7ff6e819c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55108/0 (socket says 172.21.15.44:55108) 2024-04-21T23:13:44.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.988+0000 7ff6ee6f9640 1 -- 172.21.15.44:0/2488797518 learned_addr learned my addr 172.21.15.44:0/2488797518 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:44.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.988+0000 7ff6edef8640 1 --2- 172.21.15.44:0/2488797518 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e8101ad0 0x7ff6e81980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.988+0000 7ff6ed6f7640 1 --2- 172.21.15.44:0/2488797518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e8103210 0x7ff6e81985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.988+0000 7ff6ee6f9640 1 -- 172.21.15.44:0/2488797518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e8103210 msgr2=0x7ff6e81985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:44.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.988+0000 7ff6ee6f9640 1 --2- 172.21.15.44:0/2488797518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e8103210 0x7ff6e81985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.988+0000 7ff6ee6f9640 1 -- 172.21.15.44:0/2488797518 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e8101ad0 msgr2=0x7ff6e81980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:44.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.988+0000 7ff6ee6f9640 1 --2- 172.21.15.44:0/2488797518 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e8101ad0 0x7ff6e81980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.988+0000 7ff6ee6f9640 1 -- 172.21.15.44:0/2488797518 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6e819d090 con 0x7ff6e8103bb0 2024-04-21T23:13:44.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.988+0000 7ff6edef8640 1 --2- 172.21.15.44:0/2488797518 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e8101ad0 0x7ff6e81980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:44.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.988+0000 7ff6ee6f9640 1 --2- 172.21.15.44:0/2488797518 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e8103bb0 0x7ff6e819c960 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8037040 tx=0x7ff6d8034790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:44.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.989+0000 7ff6e6ffd640 1 -- 172.21.15.44:0/2488797518 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8039050 con 0x7ff6e8103bb0 2024-04-21T23:13:44.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.989+0000 7ff6f015c640 1 -- 172.21.15.44:0/2488797518 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6e819d320 con 0x7ff6e8103bb0 2024-04-21T23:13:44.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.989+0000 7ff6e6ffd640 1 -- 172.21.15.44:0/2488797518 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6d8045070 con 0x7ff6e8103bb0 2024-04-21T23:13:44.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.989+0000 7ff6e6ffd640 1 -- 172.21.15.44:0/2488797518 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8032ac0 con 0x7ff6e8103bb0 2024-04-21T23:13:44.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.989+0000 7ff6f015c640 1 -- 172.21.15.44:0/2488797518 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6e81a4c10 con 0x7ff6e8103bb0 2024-04-21T23:13:44.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.991+0000 7ff6e6ffd640 1 -- 172.21.15.44:0/2488797518 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff6d80045b0 con 0x7ff6e8103bb0 2024-04-21T23:13:44.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.991+0000 7ff6f2e19740 1 Processor -- start 2024-04-21T23:13:44.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.991+0000 7ff6f2e19740 1 -- start start 2024-04-21T23:13:44.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6f2e19740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f883a00 0x55953f883de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6f2e19740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f8843b0 0x55953f8913f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6f2e19740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f891930 0x55953f893d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6f2e19740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55953f7e8530 con 0x55953f891930 2024-04-21T23:13:44.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6f2e19740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55953f790830 con 0x55953f883a00 2024-04-21T23:13:44.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6f2e19740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55953f7e44f0 con 0x55953f8843b0 2024-04-21T23:13:44.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6e6ffd640 1 --2- 172.21.15.44:0/2488797518 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff6d00a2d00 0x7ff6d00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6ed6f7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f883a00 0x55953f883de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6ed6f7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f883a00 0x55953f883de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:43472/0 (socket says 172.21.15.44:43472) 2024-04-21T23:13:44.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6ed6f7640 1 -- 172.21.15.44:0/461126490 learned_addr learned my addr 172.21.15.44:0/461126490 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:44.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6ed6f7640 1 --2- 172.21.15.44:0/461126490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f891930 0x55953f893d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6edef8640 1 --2- 172.21.15.44:0/461126490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f8843b0 0x55953f8913f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6ee6f9640 1 --2- 172.21.15.44:0/2488797518 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff6d00a2d00 0x7ff6d00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6e6ffd640 1 -- 172.21.15.44:0/2488797518 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7ff6d80f1f80 con 0x7ff6e8103bb0 2024-04-21T23:13:44.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6ed6f7640 1 -- 172.21.15.44:0/461126490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f8843b0 msgr2=0x55953f8913f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:44.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6ed6f7640 1 --2- 172.21.15.44:0/461126490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f8843b0 0x55953f8913f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6ed6f7640 1 -- 172.21.15.44:0/461126490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f891930 msgr2=0x55953f893d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:44.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6ed6f7640 1 --2- 172.21.15.44:0/461126490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f891930 0x55953f893d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6ed6f7640 1 -- 172.21.15.44:0/461126490 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55953f7b42c0 con 0x55953f883a00 2024-04-21T23:13:44.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6ed6f7640 1 --2- 172.21.15.44:0/461126490 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f883a00 0x55953f883de0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc014930 tx=0x7ff6dc00c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d60cb35a5f7cfcf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:44.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6ee6f9640 1 --2- 172.21.15.44:0/2488797518 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff6d00a2d00 0x7ff6d00a51c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80ef4f0 tx=0x7ff6d802c1d0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:44.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.992+0000 7ff6cffff640 1 -- 172.21.15.44:0/461126490 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc005020 con 0x55953f883a00 2024-04-21T23:13:44.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.993+0000 7ff6cffff640 1 -- 172.21.15.44:0/461126490 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6dc0047a0 con 0x55953f883a00 2024-04-21T23:13:44.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.993+0000 7ff6cffff640 1 -- 172.21.15.44:0/461126490 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc00f9c0 con 0x55953f883a00 2024-04-21T23:13:44.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.993+0000 7ff6f2e19740 1 -- 172.21.15.44:0/461126490 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f883a00 msgr2=0x55953f883de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:44.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.993+0000 7ff6f2e19740 1 --2- 172.21.15.44:0/461126490 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f883a00 0x55953f883de0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc014930 tx=0x7ff6dc00c430 comp rx=0 tx=0).stop 2024-04-21T23:13:44.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.993+0000 7ff6f2e19740 1 -- 172.21.15.44:0/461126490 shutdown_connections 2024-04-21T23:13:44.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.993+0000 7ff6f2e19740 1 --2- 172.21.15.44:0/461126490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f8843b0 0x55953f8913f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.993+0000 7ff6f2e19740 1 --2- 172.21.15.44:0/461126490 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f891930 0x55953f893d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.993+0000 7ff6f2e19740 1 --2- 172.21.15.44:0/461126490 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f883a00 0x55953f883de0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.993+0000 7ff6f2e19740 1 -- 172.21.15.44:0/461126490 >> 172.21.15.44:0/461126490 conn(0x55953f87e750 msgr2=0x55953f87eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:44.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.993+0000 7ff6f2e19740 1 -- 172.21.15.44:0/461126490 shutdown_connections 2024-04-21T23:13:44.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.993+0000 7ff6f2e19740 1 -- 172.21.15.44:0/461126490 wait complete. 2024-04-21T23:13:44.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.994+0000 7ff6f2e19740 1 -- start start 2024-04-21T23:13:44.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.994+0000 7ff6f2e19740 1 Processor -- start 2024-04-21T23:13:44.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.994+0000 7ff6f2e19740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f883a00 0x55953f891210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.994+0000 7ff6f2e19740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f8843b0 0x55953f88e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.994+0000 7ff6f2e19740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f891930 0x55953f88eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:44.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.994+0000 7ff6f2e19740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55953f894690 con 0x55953f883a00 2024-04-21T23:13:44.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.994+0000 7ff6f2e19740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55953f7a3f10 con 0x55953f8843b0 2024-04-21T23:13:44.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.994+0000 7ff6f2e19740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55953f894810 con 0x55953f891930 2024-04-21T23:13:44.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.994+0000 7ff6f2e19740 10 client.0 _ll_register_callbacks cb 0x55953f4fbbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:44.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.994+0000 7ff6ed6f7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f883a00 0x55953f891210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.994+0000 7ff6ed6f7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f883a00 0x55953f891210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55136/0 (socket says 172.21.15.44:55136) 2024-04-21T23:13:44.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.994+0000 7ff6ed6f7640 1 -- 172.21.15.44:0/630306741 learned_addr learned my addr 172.21.15.44:0/630306741 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:44.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.994+0000 7ff6edef8640 1 --2- 172.21.15.44:0/630306741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f8843b0 0x55953f88e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.994+0000 7ff6ed6f7640 1 --2- 172.21.15.44:0/630306741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f891930 0x55953f88eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:44.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.995+0000 7ff6edef8640 1 -- 172.21.15.44:0/630306741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f891930 msgr2=0x55953f88eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:44.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.995+0000 7ff6edef8640 1 --2- 172.21.15.44:0/630306741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f891930 0x55953f88eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.995+0000 7ff6edef8640 1 -- 172.21.15.44:0/630306741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f883a00 msgr2=0x55953f891210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:44.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.995+0000 7ff6edef8640 1 --2- 172.21.15.44:0/630306741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f883a00 0x55953f891210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:44.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.995+0000 7ff6edef8640 1 -- 172.21.15.44:0/630306741 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55953f775c80 con 0x55953f8843b0 2024-04-21T23:13:44.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.995+0000 7ff6ed6f7640 1 --2- 172.21.15.44:0/630306741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f891930 0x55953f88eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:13:44.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.995+0000 7ff6edef8640 1 --2- 172.21.15.44:0/630306741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f8843b0 0x55953f88e5a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0012d00 tx=0x7ff6e000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf3391a0a8704cdb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:44.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.995+0000 7ff6cd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:13:44.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.995+0000 7ff6cd7fa640 1 -- 172.21.15.44:0/630306741 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0009d90 con 0x55953f8843b0 2024-04-21T23:13:44.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.995+0000 7ff6f2e19740 1 -- 172.21.15.44:0/630306741 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6dc0047a0 con 0x55953f8843b0 2024-04-21T23:13:45.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.995+0000 7ff6cd7fa640 1 -- 172.21.15.44:0/630306741 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7ff6e0012110 con 0x55953f8843b0 2024-04-21T23:13:45.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.995+0000 7ff6cd7fa640 1 -- 172.21.15.44:0/630306741 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff6e0025050 con 0x55953f8843b0 2024-04-21T23:13:45.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.995+0000 7ff6f2e19740 10 client.26318 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:45.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.995+0000 7ff6f2e19740 1 -- 172.21.15.44:0/630306741 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55953f4fc3f0 con 0x55953f8843b0 2024-04-21T23:13:45.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.995+0000 7ff6cd7fa640 1 -- 172.21.15.44:0/630306741 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6dc0047a0 con 0x55953f8843b0 2024-04-21T23:13:45.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.996+0000 7ff6cd7fa640 1 -- 172.21.15.44:0/630306741 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7ff6e0010430 con 0x55953f8843b0 2024-04-21T23:13:45.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.996+0000 7ff6f2e19740 10 client.26318 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:45.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.996+0000 7ff6f2e19740 10 client.26318 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:45.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.996+0000 7ff6f2e19740 20 client.26318 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:45.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.996+0000 7ff6f2e19740 1 --2- 172.21.15.44:0/630306741 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55953f8a18d0 0x55953f8a3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:45.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.996+0000 7ff6f2e19740 4 client.26318 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:13:45.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.996+0000 7ff6f2e19740 1 -- 172.21.15.44:0/630306741 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55953f7f5a70 con 0x55953f8a18d0 2024-04-21T23:13:45.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.996+0000 7ff6ed6f7640 1 --2- 172.21.15.44:0/630306741 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55953f8a18d0 0x55953f8a3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:45.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.996+0000 7ff6ed6f7640 1 --2- 172.21.15.44:0/630306741 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55953f8a18d0 0x55953f8a3d40 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93bc330482846fe2 server_cookie=4fc61088e9b83b07 in_seq=0 out_seq=0 2024-04-21T23:13:45.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:44.996+0000 7ff6cd7fa640 10 client.26318 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:45.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.000+0000 7ff6cd7fa640 1 -- 172.21.15.44:0/630306741 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55953f7f5a70 con 0x55953f8a18d0 2024-04-21T23:13:45.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.000+0000 7ff6cd7fa640 10 client.26318 handle_command_reply: tid=0 2024-04-21T23:13:45.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.000+0000 7ff6f2e19740 1 client.26318 shutdown 2024-04-21T23:13:45.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.000+0000 7ff6f2e19740 1 -- 172.21.15.44:0/630306741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f8843b0 msgr2=0x55953f88e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:45.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.000+0000 7ff6f2e19740 1 --2- 172.21.15.44:0/630306741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f8843b0 0x55953f88e5a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0012d00 tx=0x7ff6e000f790 comp rx=0 tx=0).stop 2024-04-21T23:13:45.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.000+0000 7ff6f2e19740 1 -- 172.21.15.44:0/630306741 shutdown_connections 2024-04-21T23:13:45.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.000+0000 7ff6f2e19740 1 --2- 172.21.15.44:0/630306741 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55953f8a18d0 0x55953f8a3d40 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:45.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.000+0000 7ff6f2e19740 1 --2- 172.21.15.44:0/630306741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f891930 0x55953f88eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:45.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.000+0000 7ff6f2e19740 1 --2- 172.21.15.44:0/630306741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f8843b0 0x55953f88e5a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:45.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.000+0000 7ff6cd7fa640 0 client.26318 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:45.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.000+0000 7ff6f2e19740 1 --2- 172.21.15.44:0/630306741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f883a00 0x55953f891210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:45.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.000+0000 7ff6f2e19740 1 -- 172.21.15.44:0/630306741 >> 172.21.15.44:0/630306741 conn(0x55953f87e750 msgr2=0x55953f8804a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:45.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.000+0000 7ff6f2e19740 1 -- 172.21.15.44:0/630306741 shutdown_connections 2024-04-21T23:13:45.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.000+0000 7ff6f2e19740 1 -- 172.21.15.44:0/630306741 wait complete. 2024-04-21T23:13:45.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.000+0000 7ff6f2e19740 20 client.26318 trim_cache size 0 max 16384 2024-04-21T23:13:45.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.014+0000 7ff6f2e19740 1 Processor -- start 2024-04-21T23:13:45.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.014+0000 7ff6f2e19740 1 -- start start 2024-04-21T23:13:45.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.014+0000 7ff6f2e19740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f883a00 0x55953f89d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:45.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.014+0000 7ff6f2e19740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f8843b0 0x55953f890cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:45.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.014+0000 7ff6f2e19740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f891930 0x55953f891210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:45.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.014+0000 7ff6f2e19740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e81aa920 con 0x55953f883a00 2024-04-21T23:13:45.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.014+0000 7ff6f2e19740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e81ad1a0 con 0x55953f891930 2024-04-21T23:13:45.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.014+0000 7ff6f2e19740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e81aaf90 con 0x55953f8843b0 2024-04-21T23:13:45.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.014+0000 7ff6ed6f7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f883a00 0x55953f89d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:45.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.014+0000 7ff6ed6f7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f883a00 0x55953f89d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55150/0 (socket says 172.21.15.44:55150) 2024-04-21T23:13:45.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.014+0000 7ff6ed6f7640 1 -- 172.21.15.44:0/3920809307 learned_addr learned my addr 172.21.15.44:0/3920809307 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:45.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.014+0000 7ff6edef8640 1 --2- 172.21.15.44:0/3920809307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f8843b0 0x55953f890cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:45.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.014+0000 7ff6ed6f7640 1 --2- 172.21.15.44:0/3920809307 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f891930 0x55953f891210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:45.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.014+0000 7ff6ed6f7640 1 -- 172.21.15.44:0/3920809307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f8843b0 msgr2=0x55953f890cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:45.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.014+0000 7ff6ed6f7640 1 --2- 172.21.15.44:0/3920809307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f8843b0 0x55953f890cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:45.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.014+0000 7ff6ed6f7640 1 -- 172.21.15.44:0/3920809307 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f891930 msgr2=0x55953f891210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:45.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.014+0000 7ff6ed6f7640 1 --2- 172.21.15.44:0/3920809307 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f891930 0x55953f891210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:45.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.014+0000 7ff6ed6f7640 1 -- 172.21.15.44:0/3920809307 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55953f5c67d0 con 0x55953f883a00 2024-04-21T23:13:45.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.015+0000 7ff6ed6f7640 1 --2- 172.21.15.44:0/3920809307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f883a00 0x55953f89d5c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc004fa0 tx=0x7ff6dc021040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e0f3fe8be1e7de7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:45.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.015+0000 7ff6cf7fe640 1 -- 172.21.15.44:0/3920809307 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc02ab50 con 0x55953f883a00 2024-04-21T23:13:45.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.015+0000 7ff6cf7fe640 1 -- 172.21.15.44:0/3920809307 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6dc00cba0 con 0x55953f883a00 2024-04-21T23:13:45.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.015+0000 7ff6cf7fe640 1 -- 172.21.15.44:0/3920809307 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc005380 con 0x55953f883a00 2024-04-21T23:13:45.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.015+0000 7ff6f2e19740 1 -- 172.21.15.44:0/3920809307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f883a00 msgr2=0x55953f89d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:45.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.015+0000 7ff6f2e19740 1 --2- 172.21.15.44:0/3920809307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f883a00 0x55953f89d5c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc004fa0 tx=0x7ff6dc021040 comp rx=0 tx=0).stop 2024-04-21T23:13:45.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.015+0000 7ff6f2e19740 1 -- 172.21.15.44:0/3920809307 shutdown_connections 2024-04-21T23:13:45.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.015+0000 7ff6f2e19740 1 --2- 172.21.15.44:0/3920809307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f8843b0 0x55953f890cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:45.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.015+0000 7ff6f2e19740 1 --2- 172.21.15.44:0/3920809307 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f891930 0x55953f891210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:45.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.015+0000 7ff6f2e19740 1 --2- 172.21.15.44:0/3920809307 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f883a00 0x55953f89d5c0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:45.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.015+0000 7ff6f2e19740 1 -- 172.21.15.44:0/3920809307 >> 172.21.15.44:0/3920809307 conn(0x55953f87e750 msgr2=0x55953f889c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:45.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.015+0000 7ff6f2e19740 1 -- 172.21.15.44:0/3920809307 shutdown_connections 2024-04-21T23:13:45.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.016+0000 7ff6f2e19740 1 -- 172.21.15.44:0/3920809307 wait complete. 2024-04-21T23:13:45.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.016+0000 7ff6f2e19740 1 -- start start 2024-04-21T23:13:45.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.016+0000 7ff6f2e19740 1 Processor -- start 2024-04-21T23:13:45.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.016+0000 7ff6f2e19740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f883a00 0x55953f88b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:45.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.016+0000 7ff6f2e19740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f8843b0 0x55953f88b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:45.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.016+0000 7ff6f2e19740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f891930 0x55953f88bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:45.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.016+0000 7ff6f2e19740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e81ab660 con 0x55953f8843b0 2024-04-21T23:13:45.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.016+0000 7ff6f2e19740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e81acad0 con 0x55953f891930 2024-04-21T23:13:45.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.016+0000 7ff6f2e19740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e81abd30 con 0x55953f883a00 2024-04-21T23:13:45.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.016+0000 7ff6f2e19740 10 client.0 _ll_register_callbacks cb 0x7ff6e0009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:45.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.016+0000 7ff6edef8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f8843b0 0x55953f88b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:45.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.016+0000 7ff6edef8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f8843b0 0x55953f88b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55162/0 (socket says 172.21.15.44:55162) 2024-04-21T23:13:45.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.016+0000 7ff6edef8640 1 -- 172.21.15.44:0/3465514795 learned_addr learned my addr 172.21.15.44:0/3465514795 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:45.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.016+0000 7ff6ed6f7640 1 --2- 172.21.15.44:0/3465514795 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f891930 0x55953f88bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:45.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6ed6f7640 1 --2- 172.21.15.44:0/3465514795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f883a00 0x55953f88b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:45.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6edef8640 1 -- 172.21.15.44:0/3465514795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f883a00 msgr2=0x55953f88b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:45.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6edef8640 1 --2- 172.21.15.44:0/3465514795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f883a00 0x55953f88b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:45.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6edef8640 1 -- 172.21.15.44:0/3465514795 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f891930 msgr2=0x55953f88bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:45.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6edef8640 1 --2- 172.21.15.44:0/3465514795 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f891930 0x55953f88bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:45.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6edef8640 1 -- 172.21.15.44:0/3465514795 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55953f906ed0 con 0x55953f8843b0 2024-04-21T23:13:45.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6ed6f7640 1 --2- 172.21.15.44:0/3465514795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f883a00 0x55953f88b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:13:45.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6edef8640 1 --2- 172.21.15.44:0/3465514795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f8843b0 0x55953f88b7f0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x55953f88de00 tx=0x7ff6e0005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=799f0867894a3d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:45.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6cd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:13:45.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6cd7fa640 1 -- 172.21.15.44:0/3465514795 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e002d070 con 0x55953f8843b0 2024-04-21T23:13:45.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6cd7fa640 1 -- 172.21.15.44:0/3465514795 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7ff6e0012250 con 0x55953f8843b0 2024-04-21T23:13:45.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6f2e19740 1 -- 172.21.15.44:0/3465514795 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6dc00cba0 con 0x55953f8843b0 2024-04-21T23:13:45.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6cd7fa640 1 -- 172.21.15.44:0/3465514795 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff6e0039050 con 0x55953f8843b0 2024-04-21T23:13:45.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6f2e19740 10 client.16758 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:45.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6f2e19740 1 -- 172.21.15.44:0/3465514795 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55953f908200 con 0x55953f8843b0 2024-04-21T23:13:45.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6cd7fa640 1 -- 172.21.15.44:0/3465514795 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6dc00cba0 con 0x55953f8843b0 2024-04-21T23:13:45.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6cd7fa640 1 -- 172.21.15.44:0/3465514795 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7ff6e0003710 con 0x55953f8843b0 2024-04-21T23:13:45.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6f2e19740 10 client.16758 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:45.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6f2e19740 10 client.16758 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:45.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6f2e19740 20 client.16758 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:45.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6f2e19740 1 --2- 172.21.15.44:0/3465514795 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55953f9089a0 0x55953f90adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:45.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.017+0000 7ff6f2e19740 4 client.16758 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:13:45.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.018+0000 7ff6f2e19740 1 -- 172.21.15.44:0/3465514795 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55953f7efdc0 con 0x55953f9089a0 2024-04-21T23:13:45.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.018+0000 7ff6ed6f7640 1 --2- 172.21.15.44:0/3465514795 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55953f9089a0 0x55953f90adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:45.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.018+0000 7ff6ed6f7640 1 --2- 172.21.15.44:0/3465514795 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55953f9089a0 0x55953f90adc0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a783c94edc387f0 server_cookie=21a56f1fe1afe36a in_seq=0 out_seq=0 2024-04-21T23:13:45.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.018+0000 7ff6cd7fa640 10 client.16758 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:45.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.020+0000 7ff6cd7fa640 1 -- 172.21.15.44:0/3465514795 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55953f7efdc0 con 0x55953f9089a0 2024-04-21T23:13:45.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.020+0000 7ff6cd7fa640 10 client.16758 handle_command_reply: tid=0 2024-04-21T23:13:45.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.020+0000 7ff6f2e19740 1 client.16758 shutdown 2024-04-21T23:13:45.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.020+0000 7ff6f2e19740 1 -- 172.21.15.44:0/3465514795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f8843b0 msgr2=0x55953f88b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:45.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.020+0000 7ff6f2e19740 1 --2- 172.21.15.44:0/3465514795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f8843b0 0x55953f88b7f0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x55953f88de00 tx=0x7ff6e0005c50 comp rx=0 tx=0).stop 2024-04-21T23:13:45.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.020+0000 7ff6f2e19740 1 -- 172.21.15.44:0/3465514795 shutdown_connections 2024-04-21T23:13:45.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.020+0000 7ff6f2e19740 1 --2- 172.21.15.44:0/3465514795 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55953f9089a0 0x55953f90adc0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:45.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.020+0000 7ff6f2e19740 1 --2- 172.21.15.44:0/3465514795 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953f891930 0x55953f88bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:45.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.020+0000 7ff6f2e19740 1 --2- 172.21.15.44:0/3465514795 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953f8843b0 0x55953f88b7f0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:45.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.020+0000 7ff6cd7fa640 0 client.16758 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:45.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.020+0000 7ff6f2e19740 1 --2- 172.21.15.44:0/3465514795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953f883a00 0x55953f88b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:45.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.020+0000 7ff6f2e19740 1 -- 172.21.15.44:0/3465514795 >> 172.21.15.44:0/3465514795 conn(0x55953f87e750 msgr2=0x55953f87fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:45.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.020+0000 7ff6f2e19740 1 -- 172.21.15.44:0/3465514795 shutdown_connections 2024-04-21T23:13:45.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.020+0000 7ff6f2e19740 1 -- 172.21.15.44:0/3465514795 wait complete. 2024-04-21T23:13:45.024 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.020+0000 7ff6f2e19740 20 client.16758 trim_cache size 0 max 16384 2024-04-21T23:13:45.024 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-21T23:13:45.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.021+0000 7ff6f015c640 1 -- 172.21.15.44:0/2488797518 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff6d00a2d00 msgr2=0x7ff6d00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:45.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.021+0000 7ff6f015c640 1 --2- 172.21.15.44:0/2488797518 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff6d00a2d00 0x7ff6d00a51c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80ef4f0 tx=0x7ff6d802c1d0 comp rx=0 tx=0).stop 2024-04-21T23:13:45.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.021+0000 7ff6f015c640 1 -- 172.21.15.44:0/2488797518 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e8103bb0 msgr2=0x7ff6e819c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:45.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.021+0000 7ff6f015c640 1 --2- 172.21.15.44:0/2488797518 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e8103bb0 0x7ff6e819c960 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8037040 tx=0x7ff6d8034790 comp rx=0 tx=0).stop 2024-04-21T23:13:45.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.021+0000 7ff6f015c640 1 -- 172.21.15.44:0/2488797518 shutdown_connections 2024-04-21T23:13:45.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.021+0000 7ff6f015c640 1 --2- 172.21.15.44:0/2488797518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e8103210 0x7ff6e81985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:45.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.021+0000 7ff6f015c640 1 --2- 172.21.15.44:0/2488797518 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff6d00a2d00 0x7ff6d00a51c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:45.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.021+0000 7ff6f015c640 1 --2- 172.21.15.44:0/2488797518 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff6e8103bb0 0x7ff6e819c960 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:45.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.021+0000 7ff6f015c640 1 --2- 172.21.15.44:0/2488797518 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff6e8101ad0 0x7ff6e81980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:45.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.021+0000 7ff6f015c640 1 -- 172.21.15.44:0/2488797518 >> 172.21.15.44:0/2488797518 conn(0x7ff6e80fd200 msgr2=0x7ff6e81075e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:45.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.021+0000 7ff6f015c640 1 -- 172.21.15.44:0/2488797518 shutdown_connections 2024-04-21T23:13:45.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:45.021+0000 7ff6f015c640 1 -- 172.21.15.44:0/2488797518 wait complete. 2024-04-21T23:13:45.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:44 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/174485280' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:13:45.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:44 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/174485280' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:13:45.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:44 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/174485280' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:13:45.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 549677 13.60 MB/sec execute 95 sec latency 14.680 ms 2024-04-21T23:13:46.034 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:13:46.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:13:46.035 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:13:46.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.153+0000 7f92b159a640 1 -- 172.21.15.44:0/3555978554 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a402cc90 con 0x7f92b41066b0 2024-04-21T23:13:46.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.153+0000 7f92b8896640 1 -- 172.21.15.44:0/3555978554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92b41066b0 msgr2=0x7f92b4106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:46.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.153+0000 7f92b8896640 1 --2- 172.21.15.44:0/3555978554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92b41066b0 0x7f92b4106a90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f92a4009930 tx=0x7f92a402bc40 comp rx=0 tx=0).stop 2024-04-21T23:13:46.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.154+0000 7f92b8896640 1 -- 172.21.15.44:0/3555978554 shutdown_connections 2024-04-21T23:13:46.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.154+0000 7f92b8896640 1 --2- 172.21.15.44:0/3555978554 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b4100fd0 0x7f92b410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.154+0000 7f92b8896640 1 --2- 172.21.15.44:0/3555978554 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92b4100610 0x7f92b4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.154+0000 7f92b8896640 1 --2- 172.21.15.44:0/3555978554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92b41066b0 0x7f92b4106a90 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.154+0000 7f92b8896640 1 -- 172.21.15.44:0/3555978554 >> 172.21.15.44:0/3555978554 conn(0x7f92b40777f0 msgr2=0x7f92b4077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:46.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.155+0000 7f92b8896640 1 -- 172.21.15.44:0/3555978554 shutdown_connections 2024-04-21T23:13:46.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.155+0000 7f92b8896640 1 -- 172.21.15.44:0/3555978554 wait complete. 2024-04-21T23:13:46.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.156+0000 7f92b8896640 1 Processor -- start 2024-04-21T23:13:46.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.156+0000 7f92b8896640 1 -- start start 2024-04-21T23:13:46.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.157+0000 7f92b8896640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92b4100610 0x7f92b4072f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.157+0000 7f92b8896640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b4100fd0 0x7f92b406bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.157+0000 7f92b8896640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92b41066b0 0x7f92b406c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.157+0000 7f92b8896640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f92b410fa10 con 0x7f92b4100fd0 2024-04-21T23:13:46.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.157+0000 7f92b8896640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f92b410f890 con 0x7f92b4100610 2024-04-21T23:13:46.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.157+0000 7f92b8896640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f92b410fb90 con 0x7f92b41066b0 2024-04-21T23:13:46.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.157+0000 7f92b2d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92b41066b0 0x7f92b406c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.157+0000 7f92b2d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92b41066b0 0x7f92b406c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:37218/0 (socket says 172.21.15.44:37218) 2024-04-21T23:13:46.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.157+0000 7f92b2d9d640 1 -- 172.21.15.44:0/3751616132 learned_addr learned my addr 172.21.15.44:0/3751616132 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:46.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.158+0000 7f92b1d9b640 1 --2- 172.21.15.44:0/3751616132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b4100fd0 0x7f92b406bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.159+0000 7f92b1d9b640 1 -- 172.21.15.44:0/3751616132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92b41066b0 msgr2=0x7f92b406c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:46.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.159+0000 7f92b1d9b640 1 --2- 172.21.15.44:0/3751616132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92b41066b0 0x7f92b406c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.159+0000 7f92b1d9b640 1 -- 172.21.15.44:0/3751616132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92b4100610 msgr2=0x7f92b4072f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:46.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.159+0000 7f92b1d9b640 1 --2- 172.21.15.44:0/3751616132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92b4100610 0x7f92b4072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.159+0000 7f92b1d9b640 1 -- 172.21.15.44:0/3751616132 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92b406cac0 con 0x7f92b4100fd0 2024-04-21T23:13:46.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.159+0000 7f92b1d9b640 1 --2- 172.21.15.44:0/3751616132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b4100fd0 0x7f92b406bfc0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f92a800b3e0 tx=0x7f92a800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:46.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.159+0000 7f92a37fe640 1 -- 172.21.15.44:0/3751616132 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a8015020 con 0x7f92b4100fd0 2024-04-21T23:13:46.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.160+0000 7f92a37fe640 1 -- 172.21.15.44:0/3751616132 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92a8011d10 con 0x7f92b4100fd0 2024-04-21T23:13:46.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.160+0000 7f92a37fe640 1 -- 172.21.15.44:0/3751616132 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a800f2e0 con 0x7f92b4100fd0 2024-04-21T23:13:46.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.160+0000 7f92b8896640 1 -- 172.21.15.44:0/3751616132 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92b406cdb0 con 0x7f92b4100fd0 2024-04-21T23:13:46.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.160+0000 7f92b8896640 1 -- 172.21.15.44:0/3751616132 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92b4070d90 con 0x7f92b4100fd0 2024-04-21T23:13:46.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.164+0000 7f92a37fe640 1 -- 172.21.15.44:0/3751616132 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f92a8018020 con 0x7f92b4100fd0 2024-04-21T23:13:46.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.166+0000 7f92a37fe640 1 --2- 172.21.15.44:0/3751616132 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f929c0a50e0 0x7f929c0a75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.166+0000 7f92a37fe640 1 -- 172.21.15.44:0/3751616132 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f92a80cfe90 con 0x7f92b4100fd0 2024-04-21T23:13:46.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.166+0000 7f92b259c640 1 --2- 172.21.15.44:0/3751616132 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f929c0a50e0 0x7f929c0a75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.166+0000 7f92bb480740 1 Processor -- start 2024-04-21T23:13:46.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.167+0000 7f92bb480740 1 -- start start 2024-04-21T23:13:46.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.167+0000 7f92bb480740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0de3420 0x5640f0de3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.167+0000 7f92bb480740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0de3d40 0x5640f0de41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.167+0000 7f92bb480740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0df5900 0x5640f0df7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.167+0000 7f92bb480740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5640f0d47170 con 0x5640f0df5900 2024-04-21T23:13:46.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.167+0000 7f92bb480740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5640f0cef7c0 con 0x5640f0de3d40 2024-04-21T23:13:46.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.167+0000 7f92bb480740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5640f0d43130 con 0x5640f0de3420 2024-04-21T23:13:46.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.167+0000 7f92b2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0df5900 0x5640f0df7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.167+0000 7f92b2d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0df5900 0x5640f0df7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55196/0 (socket says 172.21.15.44:55196) 2024-04-21T23:13:46.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.167+0000 7f92b2d9d640 1 -- 172.21.15.44:0/1525933910 learned_addr learned my addr 172.21.15.44:0/1525933910 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:46.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.168+0000 7f92b1d9b640 1 --2- 172.21.15.44:0/1525933910 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0de3d40 0x5640f0de41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.168+0000 7f92b259c640 1 --2- 172.21.15.44:0/1525933910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0de3420 0x5640f0de3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.168+0000 7f92b259c640 1 --2- 172.21.15.44:0/3751616132 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f929c0a50e0 0x7f929c0a75a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f92a402efd0 tx=0x7f92a4000bc0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:46.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.168+0000 7f92b2d9d640 1 -- 172.21.15.44:0/1525933910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0de3420 msgr2=0x5640f0de3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:46.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.168+0000 7f92b2d9d640 1 --2- 172.21.15.44:0/1525933910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0de3420 0x5640f0de3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.168+0000 7f92b2d9d640 1 -- 172.21.15.44:0/1525933910 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0de3d40 msgr2=0x5640f0de41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:46.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.168+0000 7f92b2d9d640 1 --2- 172.21.15.44:0/1525933910 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0de3d40 0x5640f0de41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.168+0000 7f92b2d9d640 1 -- 172.21.15.44:0/1525933910 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640f0d00e60 con 0x5640f0df5900 2024-04-21T23:13:46.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.169+0000 7f92b2d9d640 1 --2- 172.21.15.44:0/1525933910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0df5900 0x5640f0df7cf0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00e1c0 tx=0x7f92ac009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f19264b7be65b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:46.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.170+0000 7f92a0ff9640 1 -- 172.21.15.44:0/1525933910 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ac018020 con 0x5640f0df5900 2024-04-21T23:13:46.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.170+0000 7f92a0ff9640 1 -- 172.21.15.44:0/1525933910 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92ac014b70 con 0x5640f0df5900 2024-04-21T23:13:46.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.170+0000 7f92a0ff9640 1 -- 172.21.15.44:0/1525933910 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ac013c70 con 0x5640f0df5900 2024-04-21T23:13:46.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.171+0000 7f92bb480740 1 -- 172.21.15.44:0/1525933910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0df5900 msgr2=0x5640f0df7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:46.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.171+0000 7f92bb480740 1 --2- 172.21.15.44:0/1525933910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0df5900 0x5640f0df7cf0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00e1c0 tx=0x7f92ac009660 comp rx=0 tx=0).stop 2024-04-21T23:13:46.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.171+0000 7f92bb480740 1 -- 172.21.15.44:0/1525933910 shutdown_connections 2024-04-21T23:13:46.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.172+0000 7f92bb480740 1 --2- 172.21.15.44:0/1525933910 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0df5900 0x5640f0df7cf0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.172+0000 7f92bb480740 1 --2- 172.21.15.44:0/1525933910 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0de3d40 0x5640f0de41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.172+0000 7f92bb480740 1 --2- 172.21.15.44:0/1525933910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0de3420 0x5640f0de3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.172+0000 7f92bb480740 1 -- 172.21.15.44:0/1525933910 >> 172.21.15.44:0/1525933910 conn(0x5640f0daa4f0 msgr2=0x5640f0dde490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:46.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.172+0000 7f92bb480740 1 -- 172.21.15.44:0/1525933910 shutdown_connections 2024-04-21T23:13:46.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.172+0000 7f92bb480740 1 -- 172.21.15.44:0/1525933910 wait complete. 2024-04-21T23:13:46.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.172+0000 7f92bb480740 1 -- start start 2024-04-21T23:13:46.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.172+0000 7f92bb480740 1 Processor -- start 2024-04-21T23:13:46.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.173+0000 7f92bb480740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0de3420 0x5640f0def420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.173+0000 7f92bb480740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0de3d40 0x5640f0def960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.173+0000 7f92bb480740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0df5900 0x5640f0df5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.173+0000 7f92b259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0de3420 0x5640f0def420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.173+0000 7f92b259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0de3420 0x5640f0def420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:43530/0 (socket says 172.21.15.44:43530) 2024-04-21T23:13:46.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.173+0000 7f92b259c640 1 -- 172.21.15.44:0/2674708796 learned_addr learned my addr 172.21.15.44:0/2674708796 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:46.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.173+0000 7f92bb480740 1 -- 172.21.15.44:0/2674708796 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5640f0de9360 con 0x5640f0de3d40 2024-04-21T23:13:46.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.173+0000 7f92bb480740 1 -- 172.21.15.44:0/2674708796 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5640f0d02d40 con 0x5640f0de3420 2024-04-21T23:13:46.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.173+0000 7f92bb480740 1 -- 172.21.15.44:0/2674708796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5640f0de94e0 con 0x5640f0df5900 2024-04-21T23:13:46.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.173+0000 7f92bb480740 10 client.0 _ll_register_callbacks cb 0x5640f0d3e820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:46.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.173+0000 7f92b2d9d640 1 --2- 172.21.15.44:0/2674708796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0df5900 0x5640f0df5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.174+0000 7f92b1d9b640 1 --2- 172.21.15.44:0/2674708796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0de3d40 0x5640f0def960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.174+0000 7f92b2d9d640 1 -- 172.21.15.44:0/2674708796 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0de3420 msgr2=0x5640f0def420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:46.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.174+0000 7f92b2d9d640 1 --2- 172.21.15.44:0/2674708796 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0de3420 0x5640f0def420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.174+0000 7f92b2d9d640 1 -- 172.21.15.44:0/2674708796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0de3d40 msgr2=0x5640f0def960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:46.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.174+0000 7f92b2d9d640 1 --2- 172.21.15.44:0/2674708796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0de3d40 0x5640f0def960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.174+0000 7f92b2d9d640 1 -- 172.21.15.44:0/2674708796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640f0b237e0 con 0x5640f0df5900 2024-04-21T23:13:46.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.174+0000 7f92b2d9d640 1 --2- 172.21.15.44:0/2674708796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0df5900 0x5640f0df5140 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00eb10 tx=0x7f92ac0148c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a121fb4d776dbdba server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:46.174 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.175+0000 7f9299ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:13:46.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.176+0000 7f9299ffb640 1 -- 172.21.15.44:0/2674708796 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ac018040 con 0x5640f0df5900 2024-04-21T23:13:46.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.177+0000 7f92bb480740 1 -- 172.21.15.44:0/2674708796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92ac014b70 con 0x5640f0df5900 2024-04-21T23:13:46.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.177+0000 7f9299ffb640 1 -- 172.21.15.44:0/2674708796 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f92ac013c00 con 0x5640f0df5900 2024-04-21T23:13:46.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.178+0000 7f9299ffb640 1 -- 172.21.15.44:0/2674708796 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f92ac032050 con 0x5640f0df5900 2024-04-21T23:13:46.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.179+0000 7f92bb480740 10 client.26362 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:46.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.179+0000 7f92bb480740 1 -- 172.21.15.44:0/2674708796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5640f0b83f70 con 0x5640f0df5900 2024-04-21T23:13:46.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.179+0000 7f9299ffb640 1 -- 172.21.15.44:0/2674708796 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92ac014b70 con 0x5640f0df5900 2024-04-21T23:13:46.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.180+0000 7f9299ffb640 1 -- 172.21.15.44:0/2674708796 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f92ac01d430 con 0x5640f0df5900 2024-04-21T23:13:46.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.180+0000 7f92bb480740 10 client.26362 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:46.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.180+0000 7f92bb480740 10 client.26362 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:46.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.180+0000 7f92bb480740 20 client.26362 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:46.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.180+0000 7f92bb480740 1 --2- 172.21.15.44:0/2674708796 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5640f0e0e280 0x5640f0e10700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.180+0000 7f92bb480740 4 client.26362 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:13:46.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.180+0000 7f92bb480740 1 -- 172.21.15.44:0/2674708796 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640f0d1b450 con 0x5640f0e0e280 2024-04-21T23:13:46.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.181+0000 7f92b259c640 1 --2- 172.21.15.44:0/2674708796 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5640f0e0e280 0x5640f0e10700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.181+0000 7f92b259c640 1 --2- 172.21.15.44:0/2674708796 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5640f0e0e280 0x5640f0e10700 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5897f59c5e9d7636 server_cookie=658ce3d5f3520e45 in_seq=0 out_seq=0 2024-04-21T23:13:46.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.181+0000 7f9299ffb640 10 client.26362 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:46.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.183+0000 7f9299ffb640 1 -- 172.21.15.44:0/2674708796 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5640f0d1b450 con 0x5640f0e0e280 2024-04-21T23:13:46.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.183+0000 7f9299ffb640 10 client.26362 handle_command_reply: tid=0 2024-04-21T23:13:46.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.183+0000 7f92bb480740 1 client.26362 shutdown 2024-04-21T23:13:46.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.183+0000 7f92bb480740 1 -- 172.21.15.44:0/2674708796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0df5900 msgr2=0x5640f0df5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:46.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.183+0000 7f92bb480740 1 --2- 172.21.15.44:0/2674708796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0df5900 0x5640f0df5140 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00eb10 tx=0x7f92ac0148c0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.183+0000 7f92bb480740 1 -- 172.21.15.44:0/2674708796 shutdown_connections 2024-04-21T23:13:46.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.183+0000 7f92bb480740 1 --2- 172.21.15.44:0/2674708796 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5640f0e0e280 0x5640f0e10700 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.183+0000 7f92bb480740 1 --2- 172.21.15.44:0/2674708796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0df5900 0x5640f0df5140 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.183+0000 7f92bb480740 1 --2- 172.21.15.44:0/2674708796 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0de3d40 0x5640f0def960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.183+0000 7f92bb480740 1 --2- 172.21.15.44:0/2674708796 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0de3420 0x5640f0def420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.183+0000 7f92bb480740 1 -- 172.21.15.44:0/2674708796 >> 172.21.15.44:0/2674708796 conn(0x5640f0daa4f0 msgr2=0x5640f0de01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:46.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.183+0000 7f9299ffb640 0 client.26362 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:46.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.184+0000 7f92bb480740 1 -- 172.21.15.44:0/2674708796 shutdown_connections 2024-04-21T23:13:46.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.184+0000 7f92bb480740 1 -- 172.21.15.44:0/2674708796 wait complete. 2024-04-21T23:13:46.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.184+0000 7f92bb480740 20 client.26362 trim_cache size 0 max 16384 2024-04-21T23:13:46.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:45 smithi044 ceph-mon[27934]: pgmap v152: 129 pgs: 129 active+clean; 51 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.2 MiB/s wr, 388 op/s 2024-04-21T23:13:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.203+0000 7f92bb480740 1 Processor -- start 2024-04-21T23:13:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.203+0000 7f92bb480740 1 -- start start 2024-04-21T23:13:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.203+0000 7f92bb480740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0de3420 0x5640f0ddd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.203+0000 7f92bb480740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0de3d40 0x5640f0e16010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.203+0000 7f92bb480740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0df5900 0x5640f0e16550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.203+0000 7f92bb480740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f929400df50 con 0x5640f0de3420 2024-04-21T23:13:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.203+0000 7f92bb480740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f92940107d0 con 0x5640f0df5900 2024-04-21T23:13:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.203+0000 7f92bb480740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f929400e5c0 con 0x5640f0de3d40 2024-04-21T23:13:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.203+0000 7f92b259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0de3420 0x5640f0ddd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.207 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.203+0000 7f92b259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0de3420 0x5640f0ddd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55216/0 (socket says 172.21.15.44:55216) 2024-04-21T23:13:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.203+0000 7f92b259c640 1 -- 172.21.15.44:0/2722603092 learned_addr learned my addr 172.21.15.44:0/2722603092 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.204+0000 7f92b2d9d640 1 --2- 172.21.15.44:0/2722603092 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0df5900 0x5640f0e16550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.204+0000 7f92b1d9b640 1 --2- 172.21.15.44:0/2722603092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0de3d40 0x5640f0e16010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.204+0000 7f92b259c640 1 -- 172.21.15.44:0/2722603092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0de3d40 msgr2=0x5640f0e16010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.204+0000 7f92b259c640 1 --2- 172.21.15.44:0/2722603092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0de3d40 0x5640f0e16010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.204+0000 7f92b259c640 1 -- 172.21.15.44:0/2722603092 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0df5900 msgr2=0x5640f0e16550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.204+0000 7f92b259c640 1 --2- 172.21.15.44:0/2722603092 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0df5900 0x5640f0e16550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.204+0000 7f92b259c640 1 -- 172.21.15.44:0/2722603092 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640f0df8060 con 0x5640f0de3420 2024-04-21T23:13:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.204+0000 7f92b259c640 1 --2- 172.21.15.44:0/2722603092 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0de3420 0x5640f0ddd370 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f92a402c830 tx=0x7f92a403c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8fdd914ff79ddba server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.204+0000 7f929bfff640 1 -- 172.21.15.44:0/2722603092 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a405a070 con 0x5640f0de3420 2024-04-21T23:13:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.204+0000 7f929bfff640 1 -- 172.21.15.44:0/2722603092 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92a4043040 con 0x5640f0de3420 2024-04-21T23:13:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.204+0000 7f929bfff640 1 -- 172.21.15.44:0/2722603092 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a4004ca0 con 0x5640f0de3420 2024-04-21T23:13:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.205+0000 7f92bb480740 1 -- 172.21.15.44:0/2722603092 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0de3420 msgr2=0x5640f0ddd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.205+0000 7f92bb480740 1 --2- 172.21.15.44:0/2722603092 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0de3420 0x5640f0ddd370 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f92a402c830 tx=0x7f92a403c000 comp rx=0 tx=0).stop 2024-04-21T23:13:46.208 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.205+0000 7f92bb480740 1 -- 172.21.15.44:0/2722603092 shutdown_connections 2024-04-21T23:13:46.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.205+0000 7f92bb480740 1 --2- 172.21.15.44:0/2722603092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0de3d40 0x5640f0e16010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.205+0000 7f92bb480740 1 --2- 172.21.15.44:0/2722603092 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0df5900 0x5640f0e16550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.205+0000 7f92bb480740 1 --2- 172.21.15.44:0/2722603092 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0de3420 0x5640f0ddd370 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.205+0000 7f92bb480740 1 -- 172.21.15.44:0/2722603092 >> 172.21.15.44:0/2722603092 conn(0x5640f0daa4f0 msgr2=0x5640f0ddcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:46.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.205+0000 7f92bb480740 1 -- 172.21.15.44:0/2722603092 shutdown_connections 2024-04-21T23:13:46.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.205+0000 7f92bb480740 1 -- 172.21.15.44:0/2722603092 wait complete. 2024-04-21T23:13:46.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.205+0000 7f92bb480740 1 -- start start 2024-04-21T23:13:46.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.205+0000 7f92bb480740 1 Processor -- start 2024-04-21T23:13:46.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92bb480740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0de3420 0x5640f0e212a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92bb480740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0de3d40 0x5640f0e217e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92bb480740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0df5900 0x5640f0e1f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92bb480740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f929400ec90 con 0x5640f0de3d40 2024-04-21T23:13:46.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92bb480740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9294010100 con 0x5640f0df5900 2024-04-21T23:13:46.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92bb480740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f929400f360 con 0x5640f0de3420 2024-04-21T23:13:46.209 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92bb480740 10 client.0 _ll_register_callbacks cb 0x7f92ac009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:46.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92b1d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0de3d40 0x5640f0e217e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92b1d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0de3d40 0x5640f0e217e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55222/0 (socket says 172.21.15.44:55222) 2024-04-21T23:13:46.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92b1d9b640 1 -- 172.21.15.44:0/2637529005 learned_addr learned my addr 172.21.15.44:0/2637529005 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:46.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92b259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0de3420 0x5640f0e212a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92b2d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0df5900 0x5640f0e1f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92b1d9b640 1 -- 172.21.15.44:0/2637529005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0de3420 msgr2=0x5640f0e212a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:46.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92b1d9b640 1 --2- 172.21.15.44:0/2637529005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0de3420 0x5640f0e212a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92b1d9b640 1 -- 172.21.15.44:0/2637529005 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0df5900 msgr2=0x5640f0e1f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:46.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92b1d9b640 1 --2- 172.21.15.44:0/2637529005 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0df5900 0x5640f0e1f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92b1d9b640 1 -- 172.21.15.44:0/2637529005 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640f0a1b170 con 0x5640f0de3d40 2024-04-21T23:13:46.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92b2d9d640 1 --2- 172.21.15.44:0/2637529005 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0df5900 0x5640f0e1f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:46.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92b1d9b640 1 --2- 172.21.15.44:0/2637529005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0de3d40 0x5640f0e217e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f92a8098520 tx=0x7f92a809c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=268b3993e2b5c9f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f9299ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:13:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f9299ffb640 1 -- 172.21.15.44:0/2637529005 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a80b4070 con 0x5640f0de3d40 2024-04-21T23:13:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f9299ffb640 1 -- 172.21.15.44:0/2637529005 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f92a8098730 con 0x5640f0de3d40 2024-04-21T23:13:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92bb480740 1 -- 172.21.15.44:0/2637529005 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92a4043040 con 0x5640f0de3d40 2024-04-21T23:13:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f9299ffb640 1 -- 172.21.15.44:0/2637529005 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f92a80a4d30 con 0x5640f0de3d40 2024-04-21T23:13:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92bb480740 10 client.16773 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.206+0000 7f92bb480740 1 -- 172.21.15.44:0/2637529005 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5640f0e74cc0 con 0x5640f0de3d40 2024-04-21T23:13:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.207+0000 7f9299ffb640 1 -- 172.21.15.44:0/2637529005 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92a4043040 con 0x5640f0de3d40 2024-04-21T23:13:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.207+0000 7f9299ffb640 1 -- 172.21.15.44:0/2637529005 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f92a809b510 con 0x5640f0de3d40 2024-04-21T23:13:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.207+0000 7f92bb480740 10 client.16773 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.207+0000 7f92bb480740 10 client.16773 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.207+0000 7f92bb480740 20 client.16773 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.207+0000 7f92bb480740 1 --2- 172.21.15.44:0/2637529005 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5640f0e75470 0x5640f0e77890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.207+0000 7f92bb480740 4 client.16773 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:13:46.211 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.207+0000 7f92bb480740 1 -- 172.21.15.44:0/2637529005 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5640f0d546b0 con 0x5640f0e75470 2024-04-21T23:13:46.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.207+0000 7f92b259c640 1 --2- 172.21.15.44:0/2637529005 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5640f0e75470 0x5640f0e77890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.207+0000 7f92b259c640 1 --2- 172.21.15.44:0/2637529005 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5640f0e75470 0x5640f0e77890 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbd00b37dfe447bd server_cookie=5a017bf3c6dd1374 in_seq=0 out_seq=0 2024-04-21T23:13:46.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.207+0000 7f9299ffb640 10 client.16773 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:46.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.210+0000 7f9299ffb640 1 -- 172.21.15.44:0/2637529005 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5640f0d546b0 con 0x5640f0e75470 2024-04-21T23:13:46.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.210+0000 7f9299ffb640 10 client.16773 handle_command_reply: tid=0 2024-04-21T23:13:46.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.210+0000 7f92bb480740 1 client.16773 shutdown 2024-04-21T23:13:46.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.210+0000 7f92bb480740 1 -- 172.21.15.44:0/2637529005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0de3d40 msgr2=0x5640f0e217e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:46.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.210+0000 7f92bb480740 1 --2- 172.21.15.44:0/2637529005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0de3d40 0x5640f0e217e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f92a8098520 tx=0x7f92a809c880 comp rx=0 tx=0).stop 2024-04-21T23:13:46.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.210+0000 7f92bb480740 1 -- 172.21.15.44:0/2637529005 shutdown_connections 2024-04-21T23:13:46.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.210+0000 7f92bb480740 1 --2- 172.21.15.44:0/2637529005 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5640f0e75470 0x5640f0e77890 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.210+0000 7f92bb480740 1 --2- 172.21.15.44:0/2637529005 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5640f0df5900 0x5640f0e1f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.210+0000 7f92bb480740 1 --2- 172.21.15.44:0/2637529005 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5640f0de3d40 0x5640f0e217e0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.210+0000 7f92bb480740 1 --2- 172.21.15.44:0/2637529005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640f0de3420 0x5640f0e212a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.210+0000 7f92bb480740 1 -- 172.21.15.44:0/2637529005 >> 172.21.15.44:0/2637529005 conn(0x5640f0daa4f0 msgr2=0x5640f0ddd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:46.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.210+0000 7f9299ffb640 0 client.16773 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:46.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.210+0000 7f92bb480740 1 -- 172.21.15.44:0/2637529005 shutdown_connections 2024-04-21T23:13:46.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.210+0000 7f92bb480740 1 -- 172.21.15.44:0/2637529005 wait complete. 2024-04-21T23:13:46.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.211+0000 7f92bb480740 20 client.16773 trim_cache size 0 max 16384 2024-04-21T23:13:46.214 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:13:46.214 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-21T23:13:46.214 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "858d88c8-b695-4788-b26a-0161ed2c6938", 2024-04-21T23:13:46.215 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-21T23:13:46.215 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:13:46.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.211+0000 7f92b8896640 1 -- 172.21.15.44:0/3751616132 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f929c0a50e0 msgr2=0x7f929c0a75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:46.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.211+0000 7f92b8896640 1 --2- 172.21.15.44:0/3751616132 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f929c0a50e0 0x7f929c0a75a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f92a402efd0 tx=0x7f92a4000bc0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.211+0000 7f92b8896640 1 -- 172.21.15.44:0/3751616132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b4100fd0 msgr2=0x7f92b406bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:46.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.211+0000 7f92b8896640 1 --2- 172.21.15.44:0/3751616132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b4100fd0 0x7f92b406bfc0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f92a800b3e0 tx=0x7f92a800b8a0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.211+0000 7f92b8896640 1 -- 172.21.15.44:0/3751616132 shutdown_connections 2024-04-21T23:13:46.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.211+0000 7f92b8896640 1 --2- 172.21.15.44:0/3751616132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92b41066b0 0x7f92b406c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.211+0000 7f92b8896640 1 --2- 172.21.15.44:0/3751616132 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f929c0a50e0 0x7f929c0a75a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.211+0000 7f92b8896640 1 --2- 172.21.15.44:0/3751616132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f92b4100fd0 0x7f92b406bfc0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.211+0000 7f92b8896640 1 --2- 172.21.15.44:0/3751616132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92b4100610 0x7f92b4072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.211+0000 7f92b8896640 1 -- 172.21.15.44:0/3751616132 >> 172.21.15.44:0/3751616132 conn(0x7f92b40777f0 msgr2=0x7f92b40ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:46.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.211+0000 7f92b8896640 1 -- 172.21.15.44:0/3751616132 shutdown_connections 2024-04-21T23:13:46.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.211+0000 7f92b8896640 1 -- 172.21.15.44:0/3751616132 wait complete. 2024-04-21T23:13:46.217 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:45 smithi069 ceph-mon[31816]: pgmap v152: 129 pgs: 129 active+clean; 51 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.2 MiB/s wr, 388 op/s 2024-04-21T23:13:46.217 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:45 smithi055 ceph-mon[31117]: pgmap v152: 129 pgs: 129 active+clean; 51 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.2 MiB/s wr, 388 op/s 2024-04-21T23:13:46.221 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:13:46.221 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:13:46.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.332+0000 7f777a2ed640 1 -- 172.21.15.44:0/1952926540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f777410bcf0 msgr2=0x7f777410e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:46.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.332+0000 7f777a2ed640 1 --2- 172.21.15.44:0/1952926540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f777410bcf0 0x7f777410e190 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f776c009ca0 tx=0x7f776c02c180 comp rx=0 tx=0).stop 2024-04-21T23:13:46.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.332+0000 7f777a2ed640 1 -- 172.21.15.44:0/1952926540 shutdown_connections 2024-04-21T23:13:46.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.332+0000 7f777a2ed640 1 --2- 172.21.15.44:0/1952926540 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f777410bcf0 0x7f777410e190 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.332+0000 7f777a2ed640 1 --2- 172.21.15.44:0/1952926540 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77741032d0 0x7f777410b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.332+0000 7f777a2ed640 1 --2- 172.21.15.44:0/1952926540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77741029b0 0x7f7774102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.332+0000 7f777a2ed640 1 -- 172.21.15.44:0/1952926540 >> 172.21.15.44:0/1952926540 conn(0x7f77740777f0 msgr2=0x7f7774077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:46.333 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.332+0000 7f777a2ed640 1 -- 172.21.15.44:0/1952926540 shutdown_connections 2024-04-21T23:13:46.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.332+0000 7f777a2ed640 1 -- 172.21.15.44:0/1952926540 wait complete. 2024-04-21T23:13:46.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.333+0000 7f777a2ed640 1 Processor -- start 2024-04-21T23:13:46.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.333+0000 7f777a2ed640 1 -- start start 2024-04-21T23:13:46.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.333+0000 7f777a2ed640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77741029b0 0x7f7774198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.333+0000 7f777a2ed640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77741032d0 0x7f7774198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.333+0000 7f777a2ed640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f777410bcf0 0x7f777419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.333+0000 7f777a2ed640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7774111af0 con 0x7f777410bcf0 2024-04-21T23:13:46.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.333+0000 7f777a2ed640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7774111970 con 0x7f77741029b0 2024-04-21T23:13:46.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.333+0000 7f777a2ed640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7774111c70 con 0x7f77741032d0 2024-04-21T23:13:46.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.333+0000 7f777888a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f777410bcf0 0x7f777419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.333+0000 7f777888a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f777410bcf0 0x7f777419c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55238/0 (socket says 172.21.15.44:55238) 2024-04-21T23:13:46.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.333+0000 7f777888a640 1 -- 172.21.15.44:0/2605145665 learned_addr learned my addr 172.21.15.44:0/2605145665 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:46.334 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.333+0000 7f7773fff640 1 --2- 172.21.15.44:0/2605145665 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77741029b0 0x7f7774198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.333+0000 7f77737fe640 1 --2- 172.21.15.44:0/2605145665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77741032d0 0x7f7774198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.334+0000 7f777888a640 1 -- 172.21.15.44:0/2605145665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77741032d0 msgr2=0x7f7774198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:46.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.334+0000 7f777888a640 1 --2- 172.21.15.44:0/2605145665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77741032d0 0x7f7774198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.334+0000 7f777888a640 1 -- 172.21.15.44:0/2605145665 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77741029b0 msgr2=0x7f7774198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:46.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.334+0000 7f777888a640 1 --2- 172.21.15.44:0/2605145665 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77741029b0 0x7f7774198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.334+0000 7f777888a640 1 -- 172.21.15.44:0/2605145665 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f777419d000 con 0x7f777410bcf0 2024-04-21T23:13:46.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.334+0000 7f777888a640 1 --2- 172.21.15.44:0/2605145665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f777410bcf0 0x7f777419c8d0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f776c02c790 tx=0x7f776c035e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:46.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.334+0000 7f77717fa640 1 -- 172.21.15.44:0/2605145665 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c039040 con 0x7f777410bcf0 2024-04-21T23:13:46.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.334+0000 7f777a2ed640 1 -- 172.21.15.44:0/2605145665 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f777419d290 con 0x7f777410bcf0 2024-04-21T23:13:46.335 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.334+0000 7f77717fa640 1 -- 172.21.15.44:0/2605145665 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f776c044070 con 0x7f777410bcf0 2024-04-21T23:13:46.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.334+0000 7f77717fa640 1 -- 172.21.15.44:0/2605145665 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c032940 con 0x7f777410bcf0 2024-04-21T23:13:46.336 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.334+0000 7f777a2ed640 1 -- 172.21.15.44:0/2605145665 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77741a4b80 con 0x7f777410bcf0 2024-04-21T23:13:46.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.336+0000 7f77717fa640 1 -- 172.21.15.44:0/2605145665 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f776c032ae0 con 0x7f777410bcf0 2024-04-21T23:13:46.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.337+0000 7f777cfb4740 1 Processor -- start 2024-04-21T23:13:46.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.337+0000 7f777cfb4740 1 -- start start 2024-04-21T23:13:46.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.337+0000 7f777cfb4740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194ef30 0x55b7a194f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.337+0000 7f777cfb4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a194f850 0x55b7a1957ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.337+0000 7f777cfb4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a1958430 0x55b7a195a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.337+0000 7f777cfb4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b7a18b3560 con 0x55b7a1958430 2024-04-21T23:13:46.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.337+0000 7f777cfb4740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55b7a185b730 con 0x55b7a194ef30 2024-04-21T23:13:46.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.337+0000 7f777cfb4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b7a18af520 con 0x55b7a194f850 2024-04-21T23:13:46.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.337+0000 7f77717fa640 1 --2- 172.21.15.44:0/2605145665 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f775c0a2d00 0x7f775c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.337+0000 7f77737fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194ef30 0x55b7a194f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.337+0000 7f77737fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194ef30 0x55b7a194f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:43572/0 (socket says 172.21.15.44:43572) 2024-04-21T23:13:46.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.337+0000 7f77737fe640 1 -- 172.21.15.44:0/3835432972 learned_addr learned my addr 172.21.15.44:0/3835432972 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:46.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.337+0000 7f77737fe640 1 --2- 172.21.15.44:0/3835432972 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a1958430 0x55b7a195a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.337+0000 7f7773fff640 1 --2- 172.21.15.44:0/3835432972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a194f850 0x55b7a1957ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.337+0000 7f777888a640 1 --2- 172.21.15.44:0/2605145665 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f775c0a2d00 0x7f775c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.338+0000 7f77717fa640 1 -- 172.21.15.44:0/2605145665 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f776c0f1900 con 0x7f777410bcf0 2024-04-21T23:13:46.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.338+0000 7f77737fe640 1 -- 172.21.15.44:0/3835432972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a194f850 msgr2=0x55b7a1957ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:46.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.338+0000 7f77737fe640 1 --2- 172.21.15.44:0/3835432972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a194f850 0x55b7a1957ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.338+0000 7f77737fe640 1 -- 172.21.15.44:0/3835432972 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194ef30 msgr2=0x55b7a194f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:46.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.338+0000 7f77737fe640 1 --2- 172.21.15.44:0/3835432972 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194ef30 0x55b7a194f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.338+0000 7f77737fe640 1 -- 172.21.15.44:0/3835432972 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7a16917d0 con 0x55b7a1958430 2024-04-21T23:13:46.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.338+0000 7f7773fff640 1 --2- 172.21.15.44:0/3835432972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a194f850 0x55b7a1957ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:46.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.338+0000 7f77737fe640 1 --2- 172.21.15.44:0/3835432972 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a1958430 0x55b7a195a820 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f7764009fc0 tx=0x7f7764005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf9b980f21cd06c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:46.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.338+0000 7f777888a640 1 --2- 172.21.15.44:0/2605145665 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f775c0a2d00 0x7f775c0a51c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f776c0ba600 tx=0x7f776c0e8680 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:46.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.338+0000 7f775a7fc640 1 -- 172.21.15.44:0/3835432972 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7764004040 con 0x55b7a1958430 2024-04-21T23:13:46.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.338+0000 7f775a7fc640 1 -- 172.21.15.44:0/3835432972 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77640041e0 con 0x55b7a1958430 2024-04-21T23:13:46.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.338+0000 7f775a7fc640 1 -- 172.21.15.44:0/3835432972 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7764010690 con 0x55b7a1958430 2024-04-21T23:13:46.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.339+0000 7f777cfb4740 1 -- 172.21.15.44:0/3835432972 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a1958430 msgr2=0x55b7a195a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:46.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.339+0000 7f777cfb4740 1 --2- 172.21.15.44:0/3835432972 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a1958430 0x55b7a195a820 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f7764009fc0 tx=0x7f7764005f90 comp rx=0 tx=0).stop 2024-04-21T23:13:46.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.339+0000 7f777cfb4740 1 -- 172.21.15.44:0/3835432972 shutdown_connections 2024-04-21T23:13:46.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.339+0000 7f777cfb4740 1 --2- 172.21.15.44:0/3835432972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a194f850 0x55b7a1957ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.339+0000 7f777cfb4740 1 --2- 172.21.15.44:0/3835432972 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a1958430 0x55b7a195a820 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.339+0000 7f777cfb4740 1 --2- 172.21.15.44:0/3835432972 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194ef30 0x55b7a194f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.339+0000 7f777cfb4740 1 -- 172.21.15.44:0/3835432972 >> 172.21.15.44:0/3835432972 conn(0x55b7a1949660 msgr2=0x55b7a1949a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:46.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.339+0000 7f777cfb4740 1 -- 172.21.15.44:0/3835432972 shutdown_connections 2024-04-21T23:13:46.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.339+0000 7f777cfb4740 1 -- 172.21.15.44:0/3835432972 wait complete. 2024-04-21T23:13:46.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.339+0000 7f777cfb4740 1 -- start start 2024-04-21T23:13:46.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.340+0000 7f777cfb4740 1 Processor -- start 2024-04-21T23:13:46.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.340+0000 7f777cfb4740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194ef30 0x55b7a19554c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.340+0000 7f777cfb4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a194f850 0x55b7a1955a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.340+0000 7f777cfb4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a1958430 0x55b7a1955f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.340+0000 7f777cfb4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b7a195b220 con 0x55b7a1958430 2024-04-21T23:13:46.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.340+0000 7f777cfb4740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55b7a186ee00 con 0x55b7a194ef30 2024-04-21T23:13:46.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.340+0000 7f777cfb4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b7a195b3a0 con 0x55b7a194f850 2024-04-21T23:13:46.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.340+0000 7f777cfb4740 10 client.0 _ll_register_callbacks cb 0x55b7a18a9eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:46.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.340+0000 7f77737fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a1958430 0x55b7a1955f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.340+0000 7f77737fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194ef30 0x55b7a19554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.340+0000 7f77737fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194ef30 0x55b7a19554c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:43576/0 (socket says 172.21.15.44:43576) 2024-04-21T23:13:46.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.340+0000 7f77737fe640 1 -- 172.21.15.44:0/1166282417 learned_addr learned my addr 172.21.15.44:0/1166282417 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:46.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.340+0000 7f7773fff640 1 --2- 172.21.15.44:0/1166282417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a194f850 0x55b7a1955a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.340+0000 7f77737fe640 1 -- 172.21.15.44:0/1166282417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a194f850 msgr2=0x55b7a1955a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:46.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.340+0000 7f77737fe640 1 --2- 172.21.15.44:0/1166282417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a194f850 0x55b7a1955a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.340+0000 7f77737fe640 1 -- 172.21.15.44:0/1166282417 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194ef30 msgr2=0x55b7a19554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:46.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.340+0000 7f77737fe640 1 --2- 172.21.15.44:0/1166282417 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194ef30 0x55b7a19554c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.340+0000 7f77737fe640 1 -- 172.21.15.44:0/1166282417 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7a15c73f0 con 0x55b7a1958430 2024-04-21T23:13:46.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.340+0000 7f7773fff640 1 --2- 172.21.15.44:0/1166282417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a194f850 0x55b7a1955a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:13:46.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.340+0000 7f77737fe640 1 --2- 172.21.15.44:0/1166282417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a1958430 0x55b7a1955f40 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f7764003ed0 tx=0x7f7764013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a771749b75b1a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:46.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.341+0000 7f77537fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:13:46.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.341+0000 7f77537fe640 1 -- 172.21.15.44:0/1166282417 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7764022070 con 0x55b7a1958430 2024-04-21T23:13:46.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.341+0000 7f77537fe640 1 -- 172.21.15.44:0/1166282417 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f7764010430 con 0x55b7a1958430 2024-04-21T23:13:46.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.341+0000 7f777cfb4740 1 -- 172.21.15.44:0/1166282417 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77640041e0 con 0x55b7a1958430 2024-04-21T23:13:46.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.341+0000 7f77537fe640 1 -- 172.21.15.44:0/1166282417 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7764010650 con 0x55b7a1958430 2024-04-21T23:13:46.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.341+0000 7f777cfb4740 10 client.16785 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:46.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.341+0000 7f777cfb4740 1 -- 172.21.15.44:0/1166282417 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b7a15870f0 con 0x55b7a1958430 2024-04-21T23:13:46.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.341+0000 7f77537fe640 1 -- 172.21.15.44:0/1166282417 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77640041e0 con 0x55b7a1958430 2024-04-21T23:13:46.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.341+0000 7f77537fe640 1 -- 172.21.15.44:0/1166282417 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f776401e8c0 con 0x55b7a1958430 2024-04-21T23:13:46.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.341+0000 7f777cfb4740 10 client.16785 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:46.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.341+0000 7f777cfb4740 10 client.16785 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:46.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.341+0000 7f777cfb4740 20 client.16785 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:46.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.341+0000 7f777cfb4740 1 --2- 172.21.15.44:0/1166282417 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b7a19686b0 0x55b7a196aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.341+0000 7f777cfb4740 4 client.16785 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:13:46.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.341+0000 7f777cfb4740 1 -- 172.21.15.44:0/1166282417 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7a18c0aa0 con 0x55b7a19686b0 2024-04-21T23:13:46.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.341+0000 7f7773fff640 1 --2- 172.21.15.44:0/1166282417 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b7a19686b0 0x55b7a196aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.342+0000 7f7773fff640 1 --2- 172.21.15.44:0/1166282417 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b7a19686b0 0x55b7a196aaa0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21f2d62913cc9cd server_cookie=ab67a02392e931c5 in_seq=0 out_seq=0 2024-04-21T23:13:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.342+0000 7f77537fe640 10 client.16785 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.343+0000 7f77537fe640 1 -- 172.21.15.44:0/1166282417 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b7a18c0aa0 con 0x55b7a19686b0 2024-04-21T23:13:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.343+0000 7f77537fe640 10 client.16785 handle_command_reply: tid=0 2024-04-21T23:13:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.343+0000 7f777cfb4740 1 client.16785 shutdown 2024-04-21T23:13:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.343+0000 7f777cfb4740 1 -- 172.21.15.44:0/1166282417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a1958430 msgr2=0x55b7a1955f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.343+0000 7f777cfb4740 1 --2- 172.21.15.44:0/1166282417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a1958430 0x55b7a1955f40 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f7764003ed0 tx=0x7f7764013040 comp rx=0 tx=0).stop 2024-04-21T23:13:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.343+0000 7f777cfb4740 1 -- 172.21.15.44:0/1166282417 shutdown_connections 2024-04-21T23:13:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.343+0000 7f777cfb4740 1 --2- 172.21.15.44:0/1166282417 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b7a19686b0 0x55b7a196aaa0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.343+0000 7f777cfb4740 1 --2- 172.21.15.44:0/1166282417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a194f850 0x55b7a1955a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.343+0000 7f777cfb4740 1 --2- 172.21.15.44:0/1166282417 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a1958430 0x55b7a1955f40 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.343+0000 7f77537fe640 0 client.16785 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.343+0000 7f777cfb4740 1 --2- 172.21.15.44:0/1166282417 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194ef30 0x55b7a19554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.343+0000 7f777cfb4740 1 -- 172.21.15.44:0/1166282417 >> 172.21.15.44:0/1166282417 conn(0x55b7a1949660 msgr2=0x55b7a194b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:46.349 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.343+0000 7f777cfb4740 1 -- 172.21.15.44:0/1166282417 shutdown_connections 2024-04-21T23:13:46.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.343+0000 7f777cfb4740 1 -- 172.21.15.44:0/1166282417 wait complete. 2024-04-21T23:13:46.350 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.343+0000 7f777cfb4740 20 client.16785 trim_cache size 0 max 16384 2024-04-21T23:13:46.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.362+0000 7f777cfb4740 1 Processor -- start 2024-04-21T23:13:46.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.362+0000 7f777cfb4740 1 -- start start 2024-04-21T23:13:46.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.362+0000 7f777cfb4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a1949660 0x55b7a1957bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.362+0000 7f777cfb4740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194ef30 0x55b7a1964110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.362+0000 7f777cfb4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a194f850 0x55b7a1948560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.362+0000 7f777cfb4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f77741acc40 con 0x55b7a194f850 2024-04-21T23:13:46.361 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.362+0000 7f777cfb4740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f77741af4c0 con 0x55b7a194ef30 2024-04-21T23:13:46.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.362+0000 7f777cfb4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f77741ad2b0 con 0x55b7a1949660 2024-04-21T23:13:46.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.362+0000 7f77737fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a194f850 0x55b7a1948560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.362+0000 7f7773fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194ef30 0x55b7a1964110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.362+0000 7f77737fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a194f850 0x55b7a1948560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55268/0 (socket says 172.21.15.44:55268) 2024-04-21T23:13:46.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.362+0000 7f77737fe640 1 -- 172.21.15.44:0/1177652711 learned_addr learned my addr 172.21.15.44:0/1177652711 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:46.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.362+0000 7f77737fe640 1 --2- 172.21.15.44:0/1177652711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a1949660 0x55b7a1957bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.362+0000 7f77737fe640 1 -- 172.21.15.44:0/1177652711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a1949660 msgr2=0x55b7a1957bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:46.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.362+0000 7f77737fe640 1 --2- 172.21.15.44:0/1177652711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a1949660 0x55b7a1957bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.362+0000 7f77737fe640 1 -- 172.21.15.44:0/1177652711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194ef30 msgr2=0x55b7a1964110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:46.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.362+0000 7f77737fe640 1 --2- 172.21.15.44:0/1177652711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194ef30 0x55b7a1964110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.362+0000 7f77737fe640 1 -- 172.21.15.44:0/1177652711 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7a194f680 con 0x55b7a194f850 2024-04-21T23:13:46.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.362+0000 7f77737fe640 1 --2- 172.21.15.44:0/1177652711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a194f850 0x55b7a1948560 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f776401ed10 tx=0x7f7764003f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa1416ce91a8243 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:46.362 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.363+0000 7f7759ffb640 1 -- 172.21.15.44:0/1177652711 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7764022040 con 0x55b7a194f850 2024-04-21T23:13:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.363+0000 7f7759ffb640 1 -- 172.21.15.44:0/1177652711 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7764003900 con 0x55b7a194f850 2024-04-21T23:13:46.363 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.363+0000 7f7759ffb640 1 -- 172.21.15.44:0/1177652711 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7764003f70 con 0x55b7a194f850 2024-04-21T23:13:46.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.363+0000 7f777cfb4740 1 -- 172.21.15.44:0/1177652711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a194f850 msgr2=0x55b7a1948560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:46.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.363+0000 7f777cfb4740 1 --2- 172.21.15.44:0/1177652711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a194f850 0x55b7a1948560 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f776401ed10 tx=0x7f7764003f00 comp rx=0 tx=0).stop 2024-04-21T23:13:46.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.363+0000 7f777cfb4740 1 -- 172.21.15.44:0/1177652711 shutdown_connections 2024-04-21T23:13:46.367 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.363+0000 7f777cfb4740 1 --2- 172.21.15.44:0/1177652711 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a194f850 0x55b7a1948560 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.363+0000 7f777cfb4740 1 --2- 172.21.15.44:0/1177652711 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194ef30 0x55b7a1964110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.363+0000 7f777cfb4740 1 --2- 172.21.15.44:0/1177652711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a1949660 0x55b7a1957bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.363+0000 7f777cfb4740 1 -- 172.21.15.44:0/1177652711 >> 172.21.15.44:0/1177652711 conn(0x55b7a194ffa0 msgr2=0x55b7a1947e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:46.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.363+0000 7f777cfb4740 1 -- 172.21.15.44:0/1177652711 shutdown_connections 2024-04-21T23:13:46.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.363+0000 7f777cfb4740 1 -- 172.21.15.44:0/1177652711 wait complete. 2024-04-21T23:13:46.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.363+0000 7f777cfb4740 1 -- start start 2024-04-21T23:13:46.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.364+0000 7f777cfb4740 1 Processor -- start 2024-04-21T23:13:46.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.364+0000 7f777cfb4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a1949660 0x55b7a1960b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.364+0000 7f777cfb4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a194ef30 0x55b7a195bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.364+0000 7f777cfb4740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194f850 0x55b7a195c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.364+0000 7f777cfb4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f77741ad980 con 0x55b7a1949660 2024-04-21T23:13:46.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.364+0000 7f777cfb4740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f77741aedf0 con 0x55b7a194f850 2024-04-21T23:13:46.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.364+0000 7f777cfb4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f77741ae050 con 0x55b7a194ef30 2024-04-21T23:13:46.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.364+0000 7f777cfb4740 10 client.0 _ll_register_callbacks cb 0x7f7764009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:46.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.364+0000 7f77737fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a1949660 0x55b7a1960b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.364+0000 7f77737fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a1949660 0x55b7a1960b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55272/0 (socket says 172.21.15.44:55272) 2024-04-21T23:13:46.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.364+0000 7f77737fe640 1 -- 172.21.15.44:0/1230413922 learned_addr learned my addr 172.21.15.44:0/1230413922 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:46.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.364+0000 7f7773fff640 1 --2- 172.21.15.44:0/1230413922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a194ef30 0x55b7a195bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.364+0000 7f77737fe640 1 --2- 172.21.15.44:0/1230413922 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194f850 0x55b7a195c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.364+0000 7f77737fe640 1 -- 172.21.15.44:0/1230413922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a194ef30 msgr2=0x55b7a195bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:46.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.364+0000 7f77737fe640 1 --2- 172.21.15.44:0/1230413922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a194ef30 0x55b7a195bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.364+0000 7f77737fe640 1 -- 172.21.15.44:0/1230413922 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194f850 msgr2=0x55b7a195c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:46.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.364+0000 7f77737fe640 1 --2- 172.21.15.44:0/1230413922 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194f850 0x55b7a195c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.364+0000 7f77737fe640 1 -- 172.21.15.44:0/1230413922 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7a19cdb70 con 0x55b7a1949660 2024-04-21T23:13:46.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.364+0000 7f7773fff640 1 --2- 172.21.15.44:0/1230413922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a194ef30 0x55b7a195bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:46.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.365+0000 7f77737fe640 1 --2- 172.21.15.44:0/1230413922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a1949660 0x55b7a1960b80 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7764009c50 tx=0x7f7764010450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58a74a6cbb72ac0b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:46.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.365+0000 7f77537fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:13:46.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.365+0000 7f77537fe640 1 -- 172.21.15.44:0/1230413922 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7764022040 con 0x55b7a1949660 2024-04-21T23:13:46.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.365+0000 7f77537fe640 1 -- 172.21.15.44:0/1230413922 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f776401e410 con 0x55b7a1949660 2024-04-21T23:13:46.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.365+0000 7f777cfb4740 1 -- 172.21.15.44:0/1230413922 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7764003900 con 0x55b7a1949660 2024-04-21T23:13:46.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.365+0000 7f77537fe640 1 -- 172.21.15.44:0/1230413922 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f776401e610 con 0x55b7a1949660 2024-04-21T23:13:46.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.365+0000 7f777cfb4740 10 client.16791 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:46.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.365+0000 7f777cfb4740 1 -- 172.21.15.44:0/1230413922 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b7a19ced20 con 0x55b7a1949660 2024-04-21T23:13:46.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.365+0000 7f77537fe640 1 -- 172.21.15.44:0/1230413922 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7764003900 con 0x55b7a1949660 2024-04-21T23:13:46.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.365+0000 7f77537fe640 1 -- 172.21.15.44:0/1230413922 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f776401da10 con 0x55b7a1949660 2024-04-21T23:13:46.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.365+0000 7f777cfb4740 10 client.16791 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:46.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.365+0000 7f777cfb4740 10 client.16791 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:46.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.365+0000 7f777cfb4740 20 client.16791 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:46.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.365+0000 7f777cfb4740 1 --2- 172.21.15.44:0/1230413922 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b7a1958430 0x55b7a19d1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:46.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.365+0000 7f777cfb4740 4 client.16791 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:13:46.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.365+0000 7f777cfb4740 1 -- 172.21.15.44:0/1230413922 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b7a18badf0 con 0x55b7a1958430 2024-04-21T23:13:46.370 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.366+0000 7f7773fff640 1 --2- 172.21.15.44:0/1230413922 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b7a1958430 0x55b7a19d1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:46.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.366+0000 7f77537fe640 10 client.16791 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:46.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.366+0000 7f7773fff640 1 --2- 172.21.15.44:0/1230413922 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b7a1958430 0x55b7a19d1390 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fe8085d755bac80 server_cookie=383179358f5426e7 in_seq=0 out_seq=0 2024-04-21T23:13:46.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.367+0000 7f77537fe640 1 -- 172.21.15.44:0/1230413922 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b7a18badf0 con 0x55b7a1958430 2024-04-21T23:13:46.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.367+0000 7f77537fe640 10 client.16791 handle_command_reply: tid=0 2024-04-21T23:13:46.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.367+0000 7f777cfb4740 1 client.16791 shutdown 2024-04-21T23:13:46.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.367+0000 7f777cfb4740 1 -- 172.21.15.44:0/1230413922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a1949660 msgr2=0x55b7a1960b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:46.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.367+0000 7f777cfb4740 1 --2- 172.21.15.44:0/1230413922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a1949660 0x55b7a1960b80 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7764009c50 tx=0x7f7764010450 comp rx=0 tx=0).stop 2024-04-21T23:13:46.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.367+0000 7f777cfb4740 1 -- 172.21.15.44:0/1230413922 shutdown_connections 2024-04-21T23:13:46.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.367+0000 7f777cfb4740 1 --2- 172.21.15.44:0/1230413922 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b7a1958430 0x55b7a19d1390 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.367+0000 7f777cfb4740 1 --2- 172.21.15.44:0/1230413922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7a194ef30 0x55b7a195bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.371 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.367+0000 7f777cfb4740 1 --2- 172.21.15.44:0/1230413922 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7a194f850 0x55b7a195c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.367+0000 7f777cfb4740 1 --2- 172.21.15.44:0/1230413922 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7a1949660 0x55b7a1960b80 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.367+0000 7f77537fe640 0 client.16791 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:46.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.367+0000 7f777cfb4740 1 -- 172.21.15.44:0/1230413922 >> 172.21.15.44:0/1230413922 conn(0x55b7a194ffa0 msgr2=0x55b7a19486e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:46.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.367+0000 7f777cfb4740 1 -- 172.21.15.44:0/1230413922 shutdown_connections 2024-04-21T23:13:46.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.367+0000 7f777cfb4740 1 -- 172.21.15.44:0/1230413922 wait complete. 2024-04-21T23:13:46.372 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.367+0000 7f777cfb4740 20 client.16791 trim_cache size 0 max 16384 2024-04-21T23:13:46.372 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:13:46.372 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:13:46.372 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-21T23:13:46.372 INFO:teuthology.orchestra.run.smithi044.stdout: "858d88c8-b695-4788-b26a-0161ed2c6938": { 2024-04-21T23:13:46.373 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-21T23:13:46.373 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "858d88c8-b695-4788-b26a-0161ed2c6938", 2024-04-21T23:13:46.373 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-04-21T23:13:46.373 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:13:46.373 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:13:46.373 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:13:46.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.368+0000 7f777a2ed640 1 -- 172.21.15.44:0/2605145665 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f775c0a2d00 msgr2=0x7f775c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:46.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.368+0000 7f777a2ed640 1 --2- 172.21.15.44:0/2605145665 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f775c0a2d00 0x7f775c0a51c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f776c0ba600 tx=0x7f776c0e8680 comp rx=0 tx=0).stop 2024-04-21T23:13:46.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.368+0000 7f777a2ed640 1 -- 172.21.15.44:0/2605145665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f777410bcf0 msgr2=0x7f777419c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:46.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.368+0000 7f777a2ed640 1 --2- 172.21.15.44:0/2605145665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f777410bcf0 0x7f777419c8d0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f776c02c790 tx=0x7f776c035e00 comp rx=0 tx=0).stop 2024-04-21T23:13:46.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.368+0000 7f777a2ed640 1 -- 172.21.15.44:0/2605145665 shutdown_connections 2024-04-21T23:13:46.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.368+0000 7f777a2ed640 1 --2- 172.21.15.44:0/2605145665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77741032d0 0x7f7774198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.368+0000 7f777a2ed640 1 --2- 172.21.15.44:0/2605145665 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f775c0a2d00 0x7f775c0a51c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.368+0000 7f777a2ed640 1 --2- 172.21.15.44:0/2605145665 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f777410bcf0 0x7f777419c8d0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.368+0000 7f777a2ed640 1 --2- 172.21.15.44:0/2605145665 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77741029b0 0x7f7774198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:46.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.368+0000 7f777a2ed640 1 -- 172.21.15.44:0/2605145665 >> 172.21.15.44:0/2605145665 conn(0x7f77740777f0 msgr2=0x7f77740fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:46.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.368+0000 7f777a2ed640 1 -- 172.21.15.44:0/2605145665 shutdown_connections 2024-04-21T23:13:46.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:13:46.368+0000 7f777a2ed640 1 -- 172.21.15.44:0/2605145665 wait complete. 2024-04-21T23:13:46.377 INFO:tasks.cephfs.filesystem:scrub status for tag:858d88c8-b695-4788-b26a-0161ed2c6938 - {'path': '/', 'tag': '858d88c8-b695-4788-b26a-0161ed2c6938', 'options': 'recursive,force'} 2024-04-21T23:13:46.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 551777 13.59 MB/sec execute 96 sec latency 21.762 ms 2024-04-21T23:13:47.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 554403 13.62 MB/sec execute 97 sec latency 105.765 ms 2024-04-21T23:13:48.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:47 smithi044 ceph-mon[27934]: scrub queued for path: / 2024-04-21T23:13:48.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:47 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:13:48.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:47 smithi044 ceph-mon[27934]: scrub summary: active paths [/] 2024-04-21T23:13:48.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:47 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:13:48.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:47 smithi044 ceph-mon[27934]: pgmap v153: 129 pgs: 129 active+clean; 74 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.9 MiB/s wr, 507 op/s 2024-04-21T23:13:48.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:47 smithi044 ceph-mon[27934]: scrub summary: idle 2024-04-21T23:13:48.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:47 smithi055 ceph-mon[31117]: scrub queued for path: / 2024-04-21T23:13:48.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:47 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:13:48.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:47 smithi055 ceph-mon[31117]: scrub summary: active paths [/] 2024-04-21T23:13:48.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:47 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:13:48.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:47 smithi055 ceph-mon[31117]: pgmap v153: 129 pgs: 129 active+clean; 74 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.9 MiB/s wr, 507 op/s 2024-04-21T23:13:48.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:47 smithi055 ceph-mon[31117]: scrub summary: idle 2024-04-21T23:13:48.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:47 smithi069 ceph-mon[31816]: scrub queued for path: / 2024-04-21T23:13:48.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:47 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:13:48.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:47 smithi069 ceph-mon[31816]: scrub summary: active paths [/] 2024-04-21T23:13:48.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:47 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:13:48.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:47 smithi069 ceph-mon[31816]: pgmap v153: 129 pgs: 129 active+clean; 74 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.9 MiB/s wr, 507 op/s 2024-04-21T23:13:48.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:47 smithi069 ceph-mon[31816]: scrub summary: idle 2024-04-21T23:13:48.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 556656 13.57 MB/sec execute 98 sec latency 29.886 ms 2024-04-21T23:13:49.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 558888 13.57 MB/sec execute 99 sec latency 15.935 ms 2024-04-21T23:13:50.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:49 smithi044 ceph-mon[27934]: pgmap v154: 129 pgs: 129 active+clean; 77 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.0 MiB/s wr, 414 op/s 2024-04-21T23:13:50.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:49 smithi069 ceph-mon[31816]: pgmap v154: 129 pgs: 129 active+clean; 77 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.0 MiB/s wr, 414 op/s 2024-04-21T23:13:50.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:49 smithi055 ceph-mon[31117]: pgmap v154: 129 pgs: 129 active+clean; 77 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.0 MiB/s wr, 414 op/s 2024-04-21T23:13:50.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 561691 13.60 MB/sec execute 100 sec latency 31.583 ms 2024-04-21T23:13:50.791 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.792+0000 7fb4063de640 1 -- 172.21.15.44:0/1598283704 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4001029b0 msgr2=0x7fb400102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:50.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.792+0000 7fb4063de640 1 --2- 172.21.15.44:0/1598283704 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4001029b0 0x7fb400102d90 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4009900 tx=0x7fb3f402bcd0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.792+0000 7fb4063de640 1 -- 172.21.15.44:0/1598283704 shutdown_connections 2024-04-21T23:13:50.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.792+0000 7fb4063de640 1 --2- 172.21.15.44:0/1598283704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb40010bd80 0x7fb40010e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.792+0000 7fb4063de640 1 --2- 172.21.15.44:0/1598283704 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb4001032d0 0x7fb40010b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.792+0000 7fb4063de640 1 --2- 172.21.15.44:0/1598283704 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4001029b0 0x7fb400102d90 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.792+0000 7fb4063de640 1 -- 172.21.15.44:0/1598283704 >> 172.21.15.44:0/1598283704 conn(0x7fb400077760 msgr2=0x7fb400077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:50.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.792+0000 7fb4063de640 1 -- 172.21.15.44:0/1598283704 shutdown_connections 2024-04-21T23:13:50.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.792+0000 7fb4063de640 1 -- 172.21.15.44:0/1598283704 wait complete. 2024-04-21T23:13:50.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.793+0000 7fb4063de640 1 Processor -- start 2024-04-21T23:13:50.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.793+0000 7fb4063de640 1 -- start start 2024-04-21T23:13:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.793+0000 7fb4063de640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4001029b0 0x7fb400198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb4063de640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4001032d0 0x7fb400198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb4063de640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb40010bd80 0x7fb40019c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb4063de640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb400111b10 con 0x7fb4001029b0 2024-04-21T23:13:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb4063de640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb400111990 con 0x7fb40010bd80 2024-04-21T23:13:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb4063de640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb400111c90 con 0x7fb4001032d0 2024-04-21T23:13:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb3fffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4001029b0 0x7fb400198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb3fffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4001029b0 0x7fb400198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57968/0 (socket says 172.21.15.44:57968) 2024-04-21T23:13:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb3fffff640 1 -- 172.21.15.44:0/2881846837 learned_addr learned my addr 172.21.15.44:0/2881846837 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb40497b640 1 --2- 172.21.15.44:0/2881846837 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb40010bd80 0x7fb40019c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb3ff7fe640 1 --2- 172.21.15.44:0/2881846837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4001032d0 0x7fb400198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb3fffff640 1 -- 172.21.15.44:0/2881846837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4001032d0 msgr2=0x7fb400198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb3fffff640 1 --2- 172.21.15.44:0/2881846837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4001032d0 0x7fb400198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb3fffff640 1 -- 172.21.15.44:0/2881846837 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb40010bd80 msgr2=0x7fb40019c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb3fffff640 1 --2- 172.21.15.44:0/2881846837 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb40010bd80 0x7fb40019c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb3fffff640 1 -- 172.21.15.44:0/2881846837 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40019cfd0 con 0x7fb4001029b0 2024-04-21T23:13:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb3fffff640 1 --2- 172.21.15.44:0/2881846837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4001029b0 0x7fb400198030 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4005940 tx=0x7fb3f402cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb3fd7fa640 1 -- 172.21.15.44:0/2881846837 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4041070 con 0x7fb4001029b0 2024-04-21T23:13:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb4063de640 1 -- 172.21.15.44:0/2881846837 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb40019d260 con 0x7fb4001029b0 2024-04-21T23:13:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb3fd7fa640 1 -- 172.21.15.44:0/2881846837 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3f4004450 con 0x7fb4001029b0 2024-04-21T23:13:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb3fd7fa640 1 -- 172.21.15.44:0/2881846837 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4003d80 con 0x7fb4001029b0 2024-04-21T23:13:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.794+0000 7fb4063de640 1 -- 172.21.15.44:0/2881846837 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4001a4b50 con 0x7fb4001029b0 2024-04-21T23:13:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.797+0000 7fb3fd7fa640 1 -- 172.21.15.44:0/2881846837 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb3f4044070 con 0x7fb4001029b0 2024-04-21T23:13:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.797+0000 7fb408fc9740 1 Processor -- start 2024-04-21T23:13:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.797+0000 7fb408fc9740 1 -- start start 2024-04-21T23:13:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.797+0000 7fb408fc9740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942b4b20 0x55dd942b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.797+0000 7fb408fc9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b5440 0x55dd942bdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.797+0000 7fb408fc9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942be210 0x55dd942c0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.797+0000 7fb408fc9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55dd94218b70 con 0x55dd942be210 2024-04-21T23:13:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.797+0000 7fb408fc9740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55dd941c1470 con 0x55dd942b4b20 2024-04-21T23:13:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.797+0000 7fb408fc9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55dd94214b30 con 0x55dd942b5440 2024-04-21T23:13:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.797+0000 7fb3fd7fa640 1 --2- 172.21.15.44:0/2881846837 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb3e80a2cf0 0x7fb3e80a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.797+0000 7fb3ff7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942be210 0x55dd942c0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.797+0000 7fb3fffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b5440 0x55dd942bdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.797+0000 7fb40497b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942b4b20 0x55dd942b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.797+0000 7fb3ff7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942be210 0x55dd942c0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57982/0 (socket says 172.21.15.44:57982) 2024-04-21T23:13:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.797+0000 7fb3ff7fe640 1 -- 172.21.15.44:0/2630347838 learned_addr learned my addr 172.21.15.44:0/2630347838 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.798+0000 7fb40497b640 1 --2- 172.21.15.44:0/2881846837 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb3e80a2cf0 0x7fb3e80a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.798+0000 7fb3fd7fa640 1 -- 172.21.15.44:0/2881846837 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fb3f40f11d0 con 0x7fb4001029b0 2024-04-21T23:13:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.798+0000 7fb3ff7fe640 1 -- 172.21.15.44:0/2630347838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b5440 msgr2=0x55dd942bdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.798+0000 7fb3ff7fe640 1 --2- 172.21.15.44:0/2630347838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b5440 0x55dd942bdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.798+0000 7fb3ff7fe640 1 -- 172.21.15.44:0/2630347838 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942b4b20 msgr2=0x55dd942b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.798+0000 7fb3ff7fe640 1 --2- 172.21.15.44:0/2630347838 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942b4b20 0x55dd942b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.798+0000 7fb3ff7fe640 1 -- 172.21.15.44:0/2630347838 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd93f2d210 con 0x55dd942be210 2024-04-21T23:13:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.798+0000 7fb40497b640 1 --2- 172.21.15.44:0/2630347838 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942b4b20 0x55dd942b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:13:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.798+0000 7fb3ff7fe640 1 --2- 172.21.15.44:0/2630347838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942be210 0x55dd942c0600 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0009c90 tx=0x7fb3f0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc545ee89cf87f51 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.798+0000 7fb40497b640 1 --2- 172.21.15.44:0/2881846837 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb3e80a2cf0 0x7fb3e80a51b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb40006a040 tx=0x7fb3f8009500 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.798+0000 7fb3e67fc640 1 -- 172.21.15.44:0/2630347838 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f001e070 con 0x55dd942be210 2024-04-21T23:13:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.798+0000 7fb3e67fc640 1 -- 172.21.15.44:0/2630347838 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3f0019b10 con 0x55dd942be210 2024-04-21T23:13:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.798+0000 7fb3e67fc640 1 -- 172.21.15.44:0/2630347838 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f000ea80 con 0x55dd942be210 2024-04-21T23:13:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.799+0000 7fb408fc9740 1 -- 172.21.15.44:0/2630347838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942be210 msgr2=0x55dd942c0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.799+0000 7fb408fc9740 1 --2- 172.21.15.44:0/2630347838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942be210 0x55dd942c0600 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0009c90 tx=0x7fb3f0011810 comp rx=0 tx=0).stop 2024-04-21T23:13:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.799+0000 7fb408fc9740 1 -- 172.21.15.44:0/2630347838 shutdown_connections 2024-04-21T23:13:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.799+0000 7fb408fc9740 1 --2- 172.21.15.44:0/2630347838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b5440 0x55dd942bdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.799+0000 7fb408fc9740 1 --2- 172.21.15.44:0/2630347838 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942be210 0x55dd942c0600 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.799+0000 7fb408fc9740 1 --2- 172.21.15.44:0/2630347838 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942b4b20 0x55dd942b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.799+0000 7fb408fc9740 1 -- 172.21.15.44:0/2630347838 >> 172.21.15.44:0/2630347838 conn(0x55dd942af440 msgr2=0x55dd942af820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.799+0000 7fb408fc9740 1 -- 172.21.15.44:0/2630347838 shutdown_connections 2024-04-21T23:13:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.799+0000 7fb408fc9740 1 -- 172.21.15.44:0/2630347838 wait complete. 2024-04-21T23:13:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.799+0000 7fb408fc9740 1 -- start start 2024-04-21T23:13:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.800+0000 7fb408fc9740 1 Processor -- start 2024-04-21T23:13:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.800+0000 7fb408fc9740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942b4b20 0x55dd942bb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.800+0000 7fb408fc9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b5440 0x55dd942bb730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.800+0000 7fb408fc9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942be210 0x55dd942bbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.800+0000 7fb408fc9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55dd942c0e40 con 0x55dd942be210 2024-04-21T23:13:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.800+0000 7fb408fc9740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55dd941d4a10 con 0x55dd942b4b20 2024-04-21T23:13:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.800+0000 7fb408fc9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55dd942c0fc0 con 0x55dd942b5440 2024-04-21T23:13:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.800+0000 7fb408fc9740 10 client.0 _ll_register_callbacks cb 0x55dd9420fb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.800+0000 7fb40497b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942be210 0x55dd942bbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.800+0000 7fb3fffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942b4b20 0x55dd942bb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.800+0000 7fb40497b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942be210 0x55dd942bbc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:57994/0 (socket says 172.21.15.44:57994) 2024-04-21T23:13:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.800+0000 7fb40497b640 1 -- 172.21.15.44:0/2851346784 learned_addr learned my addr 172.21.15.44:0/2851346784 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.800+0000 7fb3ff7fe640 1 --2- 172.21.15.44:0/2851346784 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b5440 0x55dd942bb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.800+0000 7fb40497b640 1 -- 172.21.15.44:0/2851346784 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b5440 msgr2=0x55dd942bb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.800+0000 7fb40497b640 1 --2- 172.21.15.44:0/2851346784 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b5440 0x55dd942bb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.800+0000 7fb40497b640 1 -- 172.21.15.44:0/2851346784 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942b4b20 msgr2=0x55dd942bb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.800+0000 7fb40497b640 1 --2- 172.21.15.44:0/2851346784 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942b4b20 0x55dd942bb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.800+0000 7fb40497b640 1 -- 172.21.15.44:0/2851346784 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd93ff9620 con 0x55dd942be210 2024-04-21T23:13:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.800+0000 7fb40497b640 1 --2- 172.21.15.44:0/2851346784 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942be210 0x55dd942bbc70 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8014da0 tx=0x7fb3f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad33f7b7a8cb4065 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.800+0000 7fb3df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:13:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.801+0000 7fb3df7fe640 1 -- 172.21.15.44:0/2851346784 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8004b20 con 0x55dd942be210 2024-04-21T23:13:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.801+0000 7fb3df7fe640 1 -- 172.21.15.44:0/2851346784 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fb3f8004cc0 con 0x55dd942be210 2024-04-21T23:13:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.801+0000 7fb408fc9740 1 -- 172.21.15.44:0/2851346784 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3f0019b10 con 0x55dd942be210 2024-04-21T23:13:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.801+0000 7fb3df7fe640 1 -- 172.21.15.44:0/2851346784 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb3f8036050 con 0x55dd942be210 2024-04-21T23:13:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.801+0000 7fb408fc9740 10 client.16803 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.801+0000 7fb408fc9740 1 -- 172.21.15.44:0/2851346784 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dd942cd050 con 0x55dd942be210 2024-04-21T23:13:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.801+0000 7fb3df7fe640 1 -- 172.21.15.44:0/2851346784 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3f0019b10 con 0x55dd942be210 2024-04-21T23:13:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.801+0000 7fb3df7fe640 1 -- 172.21.15.44:0/2851346784 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fb3f802c030 con 0x55dd942be210 2024-04-21T23:13:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.801+0000 7fb408fc9740 10 client.16803 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.801+0000 7fb408fc9740 10 client.16803 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.801+0000 7fb408fc9740 20 client.16803 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.801+0000 7fb408fc9740 1 --2- 172.21.15.44:0/2851346784 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55dd942ce560 0x55dd942d0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.801+0000 7fb408fc9740 4 client.16803 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:13:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.801+0000 7fb408fc9740 1 -- 172.21.15.44:0/2851346784 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd942260b0 con 0x55dd942ce560 2024-04-21T23:13:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.801+0000 7fb3fffff640 1 --2- 172.21.15.44:0/2851346784 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55dd942ce560 0x55dd942d0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.802+0000 7fb3fffff640 1 --2- 172.21.15.44:0/2851346784 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55dd942ce560 0x55dd942d0990 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61e0d9425a2f0b5 server_cookie=9fcb9ab3e107437c in_seq=0 out_seq=0 2024-04-21T23:13:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.802+0000 7fb3df7fe640 10 client.16803 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.803+0000 7fb3df7fe640 1 -- 172.21.15.44:0/2851346784 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dd942260b0 con 0x55dd942ce560 2024-04-21T23:13:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.803+0000 7fb3df7fe640 10 client.16803 handle_command_reply: tid=0 2024-04-21T23:13:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.803+0000 7fb408fc9740 1 client.16803 shutdown 2024-04-21T23:13:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.803+0000 7fb408fc9740 1 -- 172.21.15.44:0/2851346784 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942be210 msgr2=0x55dd942bbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.803+0000 7fb408fc9740 1 --2- 172.21.15.44:0/2851346784 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942be210 0x55dd942bbc70 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8014da0 tx=0x7fb3f8004a40 comp rx=0 tx=0).stop 2024-04-21T23:13:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.803+0000 7fb408fc9740 1 -- 172.21.15.44:0/2851346784 shutdown_connections 2024-04-21T23:13:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.803+0000 7fb408fc9740 1 --2- 172.21.15.44:0/2851346784 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55dd942ce560 0x55dd942d0990 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.803+0000 7fb408fc9740 1 --2- 172.21.15.44:0/2851346784 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b5440 0x55dd942bb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.803+0000 7fb408fc9740 1 --2- 172.21.15.44:0/2851346784 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942be210 0x55dd942bbc70 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.803+0000 7fb408fc9740 1 --2- 172.21.15.44:0/2851346784 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942b4b20 0x55dd942bb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.803+0000 7fb3df7fe640 0 client.16803 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.803+0000 7fb408fc9740 1 -- 172.21.15.44:0/2851346784 >> 172.21.15.44:0/2851346784 conn(0x55dd942af440 msgr2=0x55dd942b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:50.810 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.803+0000 7fb408fc9740 1 -- 172.21.15.44:0/2851346784 shutdown_connections 2024-04-21T23:13:50.810 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.803+0000 7fb408fc9740 1 -- 172.21.15.44:0/2851346784 wait complete. 2024-04-21T23:13:50.810 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.803+0000 7fb408fc9740 20 client.16803 trim_cache size 0 max 16384 2024-04-21T23:13:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.818+0000 7fb408fc9740 1 Processor -- start 2024-04-21T23:13:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.818+0000 7fb408fc9740 1 -- start start 2024-04-21T23:13:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.819+0000 7fb408fc9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b4b20 0x55dd942bd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.819+0000 7fb408fc9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942b5440 0x55dd942c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.819+0000 7fb408fc9740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942be210 0x55dd942aa220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.819+0000 7fb408fc9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb4001aa9d0 con 0x55dd942b5440 2024-04-21T23:13:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.819+0000 7fb408fc9740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb4001ad250 con 0x55dd942be210 2024-04-21T23:13:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.819+0000 7fb408fc9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb4001ab040 con 0x55dd942b4b20 2024-04-21T23:13:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.819+0000 7fb3ff7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942b5440 0x55dd942c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.819+0000 7fb3ff7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942b5440 0x55dd942c9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58008/0 (socket says 172.21.15.44:58008) 2024-04-21T23:13:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.819+0000 7fb3ff7fe640 1 -- 172.21.15.44:0/3941958155 learned_addr learned my addr 172.21.15.44:0/3941958155 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.819+0000 7fb40497b640 1 --2- 172.21.15.44:0/3941958155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942be210 0x55dd942aa220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.819+0000 7fb3fffff640 1 --2- 172.21.15.44:0/3941958155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b4b20 0x55dd942bd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.819+0000 7fb3ff7fe640 1 -- 172.21.15.44:0/3941958155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b4b20 msgr2=0x55dd942bd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.819+0000 7fb3ff7fe640 1 --2- 172.21.15.44:0/3941958155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b4b20 0x55dd942bd920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.819+0000 7fb3ff7fe640 1 -- 172.21.15.44:0/3941958155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942be210 msgr2=0x55dd942aa220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.819+0000 7fb3ff7fe640 1 --2- 172.21.15.44:0/3941958155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942be210 0x55dd942aa220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.819+0000 7fb3ff7fe640 1 -- 172.21.15.44:0/3941958155 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd942aec80 con 0x55dd942b5440 2024-04-21T23:13:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.819+0000 7fb3ff7fe640 1 --2- 172.21.15.44:0/3941958155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942b5440 0x55dd942c9d00 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0009c90 tx=0x7fb3f000e640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a39d1412acc6495 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.820+0000 7fb3e5ffb640 1 -- 172.21.15.44:0/3941958155 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0028070 con 0x55dd942b5440 2024-04-21T23:13:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.820+0000 7fb3e5ffb640 1 -- 172.21.15.44:0/3941958155 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3f0005520 con 0x55dd942b5440 2024-04-21T23:13:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.820+0000 7fb3e5ffb640 1 -- 172.21.15.44:0/3941958155 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f001e040 con 0x55dd942b5440 2024-04-21T23:13:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.820+0000 7fb408fc9740 1 -- 172.21.15.44:0/3941958155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942b5440 msgr2=0x55dd942c9d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.820+0000 7fb408fc9740 1 --2- 172.21.15.44:0/3941958155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942b5440 0x55dd942c9d00 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0009c90 tx=0x7fb3f000e640 comp rx=0 tx=0).stop 2024-04-21T23:13:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.820+0000 7fb408fc9740 1 -- 172.21.15.44:0/3941958155 shutdown_connections 2024-04-21T23:13:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.820+0000 7fb408fc9740 1 --2- 172.21.15.44:0/3941958155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942be210 0x55dd942aa220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.820+0000 7fb408fc9740 1 --2- 172.21.15.44:0/3941958155 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942b5440 0x55dd942c9d00 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.820+0000 7fb408fc9740 1 --2- 172.21.15.44:0/3941958155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b4b20 0x55dd942bd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.820+0000 7fb408fc9740 1 -- 172.21.15.44:0/3941958155 >> 172.21.15.44:0/3941958155 conn(0x55dd942af440 msgr2=0x55dd942b4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.820+0000 7fb408fc9740 1 -- 172.21.15.44:0/3941958155 shutdown_connections 2024-04-21T23:13:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.820+0000 7fb408fc9740 1 -- 172.21.15.44:0/3941958155 wait complete. 2024-04-21T23:13:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.821+0000 7fb408fc9740 1 -- start start 2024-04-21T23:13:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.821+0000 7fb408fc9740 1 Processor -- start 2024-04-21T23:13:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.821+0000 7fb408fc9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b4b20 0x55dd942c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.821+0000 7fb408fc9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942b5440 0x55dd942b7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.821+0000 7fb408fc9740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942be210 0x55dd942b7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.821+0000 7fb408fc9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb4001ab710 con 0x55dd942b5440 2024-04-21T23:13:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.821+0000 7fb408fc9740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb4001acb80 con 0x55dd942be210 2024-04-21T23:13:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.821+0000 7fb408fc9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb4001abde0 con 0x55dd942b4b20 2024-04-21T23:13:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.821+0000 7fb408fc9740 10 client.0 _ll_register_callbacks cb 0x7fb3f8009d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:13:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.821+0000 7fb3ff7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942b5440 0x55dd942b7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.821+0000 7fb3ff7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942b5440 0x55dd942b7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:58022/0 (socket says 172.21.15.44:58022) 2024-04-21T23:13:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.821+0000 7fb3ff7fe640 1 -- 172.21.15.44:0/1499863947 learned_addr learned my addr 172.21.15.44:0/1499863947 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:13:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.821+0000 7fb40497b640 1 --2- 172.21.15.44:0/1499863947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942be210 0x55dd942b7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.821+0000 7fb3fffff640 1 --2- 172.21.15.44:0/1499863947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b4b20 0x55dd942c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.821+0000 7fb3ff7fe640 1 -- 172.21.15.44:0/1499863947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b4b20 msgr2=0x55dd942c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.821+0000 7fb3ff7fe640 1 --2- 172.21.15.44:0/1499863947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b4b20 0x55dd942c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.821+0000 7fb3ff7fe640 1 -- 172.21.15.44:0/1499863947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942be210 msgr2=0x55dd942b7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:13:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.821+0000 7fb3ff7fe640 1 --2- 172.21.15.44:0/1499863947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942be210 0x55dd942b7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.821+0000 7fb3ff7fe640 1 -- 172.21.15.44:0/1499863947 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd942b8640 con 0x55dd942b5440 2024-04-21T23:13:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.822+0000 7fb3fffff640 1 --2- 172.21.15.44:0/1499863947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b4b20 0x55dd942c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:13:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.822+0000 7fb3ff7fe640 1 --2- 172.21.15.44:0/1499863947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942b5440 0x55dd942b7a60 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000ecd0 tx=0x7fb3f0005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a95acb88ea68ab82 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:13:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.822+0000 7fb3df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:13:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.822+0000 7fb3df7fe640 1 -- 172.21.15.44:0/1499863947 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0033070 con 0x55dd942b5440 2024-04-21T23:13:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.822+0000 7fb3df7fe640 1 -- 172.21.15.44:0/1499863947 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fb3f000ed30 con 0x55dd942b5440 2024-04-21T23:13:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.822+0000 7fb408fc9740 1 -- 172.21.15.44:0/1499863947 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3f0005520 con 0x55dd942b5440 2024-04-21T23:13:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.822+0000 7fb3df7fe640 1 -- 172.21.15.44:0/1499863947 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb3f0028050 con 0x55dd942b5440 2024-04-21T23:13:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.822+0000 7fb408fc9740 10 client.16809 fetch_fsmap learned FSMap version 25 2024-04-21T23:13:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.822+0000 7fb408fc9740 1 -- 172.21.15.44:0/1499863947 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dd94334970 con 0x55dd942b5440 2024-04-21T23:13:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.822+0000 7fb3df7fe640 1 -- 172.21.15.44:0/1499863947 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3f0005520 con 0x55dd942b5440 2024-04-21T23:13:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.822+0000 7fb3df7fe640 1 -- 172.21.15.44:0/1499863947 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fb3f0036050 con 0x55dd942b5440 2024-04-21T23:13:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.822+0000 7fb408fc9740 10 client.16809 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:13:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.822+0000 7fb408fc9740 10 client.16809 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:13:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.822+0000 7fb408fc9740 20 client.16809 populate_metadata read hostname 'smithi044' 2024-04-21T23:13:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.822+0000 7fb408fc9740 1 --2- 172.21.15.44:0/1499863947 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55dd94334dc0 0x55dd943371e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:13:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.822+0000 7fb408fc9740 4 client.16809 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:13:50.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.822+0000 7fb408fc9740 1 -- 172.21.15.44:0/1499863947 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dd94220400 con 0x55dd94334dc0 2024-04-21T23:13:50.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.823+0000 7fb3fffff640 1 --2- 172.21.15.44:0/1499863947 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55dd94334dc0 0x55dd943371e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:13:50.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.823+0000 7fb3df7fe640 10 client.16809 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:50.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:50.823+0000 7fb3fffff640 1 --2- 172.21.15.44:0/1499863947 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55dd94334dc0 0x55dd943371e0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5a6e45f491e5e29 server_cookie=e173f06e7f8fc4d6 in_seq=0 out_seq=0 2024-04-21T23:13:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.129+0000 7fb3df7fe640 1 -- 172.21.15.44:0/1499863947 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dd94220400 con 0x55dd94334dc0 2024-04-21T23:13:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.129+0000 7fb3df7fe640 10 client.16809 handle_command_reply: tid=0 2024-04-21T23:13:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.129+0000 7fb408fc9740 1 client.16809 shutdown 2024-04-21T23:13:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.130+0000 7fb408fc9740 1 -- 172.21.15.44:0/1499863947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942b5440 msgr2=0x55dd942b7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.130+0000 7fb408fc9740 1 --2- 172.21.15.44:0/1499863947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942b5440 0x55dd942b7a60 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000ecd0 tx=0x7fb3f0005d30 comp rx=0 tx=0).stop 2024-04-21T23:13:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.130+0000 7fb408fc9740 1 -- 172.21.15.44:0/1499863947 shutdown_connections 2024-04-21T23:13:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.130+0000 7fb408fc9740 1 --2- 172.21.15.44:0/1499863947 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55dd94334dc0 0x55dd943371e0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.130+0000 7fb408fc9740 1 --2- 172.21.15.44:0/1499863947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55dd942be210 0x55dd942b7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.130+0000 7fb3df7fe640 0 client.16809 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:13:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.130+0000 7fb408fc9740 1 --2- 172.21.15.44:0/1499863947 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55dd942b5440 0x55dd942b7a60 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.130+0000 7fb408fc9740 1 --2- 172.21.15.44:0/1499863947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd942b4b20 0x55dd942c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.130+0000 7fb408fc9740 1 -- 172.21.15.44:0/1499863947 >> 172.21.15.44:0/1499863947 conn(0x55dd942af440 msgr2=0x55dd942b9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.130+0000 7fb408fc9740 1 -- 172.21.15.44:0/1499863947 shutdown_connections 2024-04-21T23:13:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.130+0000 7fb408fc9740 1 -- 172.21.15.44:0/1499863947 wait complete. 2024-04-21T23:13:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.131+0000 7fb408fc9740 20 client.16809 trim_cache size 0 max 16384 2024-04-21T23:13:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:13:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:13:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:13:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:13:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.131+0000 7fb4063de640 1 -- 172.21.15.44:0/2881846837 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb3e80a2cf0 msgr2=0x7fb3e80a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.132+0000 7fb4063de640 1 --2- 172.21.15.44:0/2881846837 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb3e80a2cf0 0x7fb3e80a51b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb40006a040 tx=0x7fb3f8009500 comp rx=0 tx=0).stop 2024-04-21T23:13:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.132+0000 7fb4063de640 1 -- 172.21.15.44:0/2881846837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4001029b0 msgr2=0x7fb400198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:13:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.132+0000 7fb4063de640 1 --2- 172.21.15.44:0/2881846837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4001029b0 0x7fb400198030 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4005940 tx=0x7fb3f402cbd0 comp rx=0 tx=0).stop 2024-04-21T23:13:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.132+0000 7fb4063de640 1 -- 172.21.15.44:0/2881846837 shutdown_connections 2024-04-21T23:13:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.132+0000 7fb4063de640 1 --2- 172.21.15.44:0/2881846837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4001032d0 0x7fb400198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.132+0000 7fb4063de640 1 --2- 172.21.15.44:0/2881846837 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb3e80a2cf0 0x7fb3e80a51b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.132+0000 7fb4063de640 1 --2- 172.21.15.44:0/2881846837 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb40010bd80 0x7fb40019c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.132+0000 7fb4063de640 1 --2- 172.21.15.44:0/2881846837 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb4001029b0 0x7fb400198030 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:13:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.132+0000 7fb4063de640 1 -- 172.21.15.44:0/2881846837 >> 172.21.15.44:0/2881846837 conn(0x7fb400077760 msgr2=0x7fb4000fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:13:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.132+0000 7fb4063de640 1 -- 172.21.15.44:0/2881846837 shutdown_connections 2024-04-21T23:13:51.134 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:13:51.132+0000 7fb4063de640 1 -- 172.21.15.44:0/2881846837 wait complete. 2024-04-21T23:13:51.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 563708 13.54 MB/sec execute 101 sec latency 160.543 ms 2024-04-21T23:13:52.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:51 smithi044 ceph-mon[27934]: pgmap v155: 129 pgs: 129 active+clean; 77 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 402 op/s 2024-04-21T23:13:52.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:51 smithi055 ceph-mon[31117]: pgmap v155: 129 pgs: 129 active+clean; 77 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 402 op/s 2024-04-21T23:13:52.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:51 smithi069 ceph-mon[31816]: pgmap v155: 129 pgs: 129 active+clean; 77 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 402 op/s 2024-04-21T23:13:52.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 565529 13.53 MB/sec execute 102 sec latency 95.765 ms 2024-04-21T23:13:53.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 568481 13.56 MB/sec execute 103 sec latency 29.167 ms 2024-04-21T23:13:54.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:53 smithi044 ceph-mon[27934]: pgmap v156: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.6 MiB/s wr, 685 op/s 2024-04-21T23:13:54.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:53 smithi069 ceph-mon[31816]: pgmap v156: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.6 MiB/s wr, 685 op/s 2024-04-21T23:13:54.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:53 smithi055 ceph-mon[31117]: pgmap v156: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.6 MiB/s wr, 685 op/s 2024-04-21T23:13:54.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 570798 13.51 MB/sec execute 104 sec latency 27.054 ms 2024-04-21T23:13:55.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 573010 13.52 MB/sec execute 105 sec latency 15.258 ms 2024-04-21T23:13:56.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:55 smithi044 ceph-mon[27934]: pgmap v157: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 411 op/s 2024-04-21T23:13:56.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:55 smithi055 ceph-mon[31117]: pgmap v157: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 411 op/s 2024-04-21T23:13:56.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:55 smithi069 ceph-mon[31816]: pgmap v157: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.9 MiB/s wr, 411 op/s 2024-04-21T23:13:56.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 576001 13.55 MB/sec execute 106 sec latency 10.917 ms 2024-04-21T23:13:57.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 578609 13.55 MB/sec execute 107 sec latency 56.141 ms 2024-04-21T23:13:58.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:57 smithi044 ceph-mon[27934]: pgmap v158: 129 pgs: 129 active+clean; 72 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 6.2 MiB/s wr, 535 op/s 2024-04-21T23:13:58.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:57 smithi055 ceph-mon[31117]: pgmap v158: 129 pgs: 129 active+clean; 72 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 6.2 MiB/s wr, 535 op/s 2024-04-21T23:13:58.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:57 smithi069 ceph-mon[31816]: pgmap v158: 129 pgs: 129 active+clean; 72 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 6.2 MiB/s wr, 535 op/s 2024-04-21T23:13:58.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 580471 13.53 MB/sec execute 108 sec latency 65.543 ms 2024-04-21T23:13:59.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 583146 13.53 MB/sec execute 109 sec latency 45.978 ms 2024-04-21T23:14:00.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:59 smithi044 ceph-mon[27934]: pgmap v159: 129 pgs: 129 active+clean; 72 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 425 op/s 2024-04-21T23:14:00.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:13:59 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:14:00.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:59 smithi055 ceph-mon[31117]: pgmap v159: 129 pgs: 129 active+clean; 72 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 425 op/s 2024-04-21T23:14:00.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:13:59 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:14:00.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:59 smithi069 ceph-mon[31816]: pgmap v159: 129 pgs: 129 active+clean; 72 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 425 op/s 2024-04-21T23:14:00.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:13:59 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:14:00.700 INFO:tasks.workunit.client.0.smithi044.stdout: 1 586135 13.57 MB/sec execute 110 sec latency 16.247 ms 2024-04-21T23:14:01.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:00 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:01.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:00 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:14:01.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:00 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:14:01.213 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:00 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:01.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:00 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:14:01.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:00 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:14:01.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:00 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:14:01.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:00 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:14:01.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:00 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:14:01.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 587087 13.47 MB/sec execute 111 sec latency 73.703 ms 2024-04-21T23:14:02.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:01 smithi044 ceph-mon[27934]: pgmap v160: 129 pgs: 129 active+clean; 72 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 415 op/s 2024-04-21T23:14:02.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:01 smithi069 ceph-mon[31816]: pgmap v160: 129 pgs: 129 active+clean; 72 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 415 op/s 2024-04-21T23:14:02.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:01 smithi055 ceph-mon[31117]: pgmap v160: 129 pgs: 129 active+clean; 72 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 415 op/s 2024-04-21T23:14:02.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 587709 13.39 MB/sec execute 112 sec latency 71.885 ms 2024-04-21T23:14:03.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 590223 13.39 MB/sec execute 113 sec latency 20.412 ms 2024-04-21T23:14:04.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:03 smithi044 ceph-mon[27934]: pgmap v161: 129 pgs: 129 active+clean; 78 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.9 MiB/s wr, 550 op/s 2024-04-21T23:14:04.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:03 smithi069 ceph-mon[31816]: pgmap v161: 129 pgs: 129 active+clean; 78 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.9 MiB/s wr, 550 op/s 2024-04-21T23:14:04.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:03 smithi055 ceph-mon[31117]: pgmap v161: 129 pgs: 129 active+clean; 78 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.9 MiB/s wr, 550 op/s 2024-04-21T23:14:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.233+0000 7fab1d4a3640 1 -- 172.21.15.44:0/2929925569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab181029d0 msgr2=0x7fab18102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.233+0000 7fab1d4a3640 1 --2- 172.21.15.44:0/2929925569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab181029d0 0x7fab18102db0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fab10009be0 tx=0x7fab10030100 comp rx=0 tx=0).stop 2024-04-21T23:14:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.234+0000 7fab1d4a3640 1 -- 172.21.15.44:0/2929925569 shutdown_connections 2024-04-21T23:14:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.234+0000 7fab1d4a3640 1 --2- 172.21.15.44:0/2929925569 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab1810bda0 0x7fab1810e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.234+0000 7fab1d4a3640 1 --2- 172.21.15.44:0/2929925569 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab181032f0 0x7fab1810b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.234+0000 7fab1d4a3640 1 --2- 172.21.15.44:0/2929925569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab181029d0 0x7fab18102db0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.234+0000 7fab1d4a3640 1 -- 172.21.15.44:0/2929925569 >> 172.21.15.44:0/2929925569 conn(0x7fab18077790 msgr2=0x7fab18077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.235+0000 7fab1d4a3640 1 -- 172.21.15.44:0/2929925569 shutdown_connections 2024-04-21T23:14:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.235+0000 7fab1d4a3640 1 -- 172.21.15.44:0/2929925569 wait complete. 2024-04-21T23:14:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.236+0000 7fab1d4a3640 1 Processor -- start 2024-04-21T23:14:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.236+0000 7fab1d4a3640 1 -- start start 2024-04-21T23:14:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.236+0000 7fab1d4a3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab181029d0 0x7fab1819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.236+0000 7fab1d4a3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab181032f0 0x7fab1819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.236+0000 7fab1d4a3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab1810bda0 0x7fab181a2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.236+0000 7fab1d4a3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fab1810f990 con 0x7fab181032f0 2024-04-21T23:14:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.237+0000 7fab1d4a3640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fab1810f810 con 0x7fab1810bda0 2024-04-21T23:14:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.237+0000 7fab1d4a3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fab1810fb10 con 0x7fab181029d0 2024-04-21T23:14:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.237+0000 7fab167fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab181032f0 0x7fab1819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.237+0000 7fab177fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab1810bda0 0x7fab181a2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.237+0000 7fab167fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab181032f0 0x7fab1819eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37374/0 (socket says 172.21.15.44:37374) 2024-04-21T23:14:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.237+0000 7fab167fc640 1 -- 172.21.15.44:0/129923657 learned_addr learned my addr 172.21.15.44:0/129923657 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:04.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.237+0000 7fab16ffd640 1 --2- 172.21.15.44:0/129923657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab181029d0 0x7fab1819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.238+0000 7fab167fc640 1 -- 172.21.15.44:0/129923657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab181029d0 msgr2=0x7fab1819e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.238+0000 7fab167fc640 1 --2- 172.21.15.44:0/129923657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab181029d0 0x7fab1819e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.238+0000 7fab167fc640 1 -- 172.21.15.44:0/129923657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab1810bda0 msgr2=0x7fab181a2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.238+0000 7fab167fc640 1 --2- 172.21.15.44:0/129923657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab1810bda0 0x7fab181a2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.238+0000 7fab167fc640 1 -- 172.21.15.44:0/129923657 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab181a35b0 con 0x7fab181032f0 2024-04-21T23:14:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.238+0000 7fab167fc640 1 --2- 172.21.15.44:0/129923657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab181032f0 0x7fab1819eb80 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fab0c00e9c0 tx=0x7fab0c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.238+0000 7faafffff640 1 -- 172.21.15.44:0/129923657 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0c00cde0 con 0x7fab181032f0 2024-04-21T23:14:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.238+0000 7faafffff640 1 -- 172.21.15.44:0/129923657 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab0c0035f0 con 0x7fab181032f0 2024-04-21T23:14:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.238+0000 7faafffff640 1 -- 172.21.15.44:0/129923657 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0c010690 con 0x7fab181032f0 2024-04-21T23:14:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.238+0000 7fab1d4a3640 1 -- 172.21.15.44:0/129923657 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab181a38a0 con 0x7fab181032f0 2024-04-21T23:14:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.239+0000 7fab1d4a3640 1 -- 172.21.15.44:0/129923657 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab1806b160 con 0x7fab181032f0 2024-04-21T23:14:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.247+0000 7faafffff640 1 -- 172.21.15.44:0/129923657 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fab0c0037a0 con 0x7fab181032f0 2024-04-21T23:14:04.248 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.248+0000 7faafffff640 1 --2- 172.21.15.44:0/129923657 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fab000a2e40 0x7fab000a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.249+0000 7fab20085740 1 Processor -- start 2024-04-21T23:14:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.249+0000 7faafffff640 1 -- 172.21.15.44:0/129923657 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fab0c0cfac0 con 0x7fab181032f0 2024-04-21T23:14:04.249 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.249+0000 7fab16ffd640 1 --2- 172.21.15.44:0/129923657 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fab000a2e40 0x7fab000a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:04.251 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.250+0000 7fab20085740 1 -- start start 2024-04-21T23:14:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.250+0000 7fab20085740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8eb20 0x558340a8ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.250+0000 7fab20085740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a8f440 0x558340a97cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.250+0000 7fab20085740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a98210 0x558340a9a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.250+0000 7fab20085740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5583409f2b70 con 0x558340a8eb20 2024-04-21T23:14:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.250+0000 7fab20085740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55834099b470 con 0x558340a98210 2024-04-21T23:14:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.250+0000 7fab20085740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5583409eeb30 con 0x558340a8f440 2024-04-21T23:14:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.250+0000 7fab177fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a98210 0x558340a9a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.250+0000 7fab177fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a98210 0x558340a9a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:42750/0 (socket says 172.21.15.44:42750) 2024-04-21T23:14:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.250+0000 7fab177fe640 1 -- 172.21.15.44:0/1666047472 learned_addr learned my addr 172.21.15.44:0/1666047472 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.251+0000 7fab167fc640 1 --2- 172.21.15.44:0/1666047472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a8f440 0x558340a97cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:04.252 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.252+0000 7fab16ffd640 1 --2- 172.21.15.44:0/1666047472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8eb20 0x558340a8ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.253+0000 7fab167fc640 1 -- 172.21.15.44:0/1666047472 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a98210 msgr2=0x558340a9a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.253+0000 7fab167fc640 1 --2- 172.21.15.44:0/1666047472 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a98210 0x558340a9a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.253+0000 7fab167fc640 1 -- 172.21.15.44:0/1666047472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8eb20 msgr2=0x558340a8ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.253+0000 7fab167fc640 1 --2- 172.21.15.44:0/1666047472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8eb20 0x558340a8ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.253 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.253+0000 7fab167fc640 1 -- 172.21.15.44:0/1666047472 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558340707210 con 0x558340a8f440 2024-04-21T23:14:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.253+0000 7fab16ffd640 1 --2- 172.21.15.44:0/129923657 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fab000a2e40 0x7fab000a5300 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fab10006010 tx=0x7fab1000f600 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.253+0000 7fab167fc640 1 --2- 172.21.15.44:0/1666047472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a8f440 0x558340a97cd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fab0c0cce10 tx=0x7fab0c09b720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=743ade4138c182e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.253+0000 7faafd7fa640 1 -- 172.21.15.44:0/1666047472 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0c09bbe0 con 0x558340a8f440 2024-04-21T23:14:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.253+0000 7faafd7fa640 1 -- 172.21.15.44:0/1666047472 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab0c09ff90 con 0x558340a8f440 2024-04-21T23:14:04.254 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.253+0000 7faafd7fa640 1 -- 172.21.15.44:0/1666047472 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0c09b030 con 0x558340a8f440 2024-04-21T23:14:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.254+0000 7fab20085740 1 -- 172.21.15.44:0/1666047472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a8f440 msgr2=0x558340a97cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.254+0000 7fab20085740 1 --2- 172.21.15.44:0/1666047472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a8f440 0x558340a97cd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fab0c0cce10 tx=0x7fab0c09b720 comp rx=0 tx=0).stop 2024-04-21T23:14:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.254+0000 7fab20085740 1 -- 172.21.15.44:0/1666047472 shutdown_connections 2024-04-21T23:14:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.254+0000 7fab20085740 1 --2- 172.21.15.44:0/1666047472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a8f440 0x558340a97cd0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.254+0000 7fab20085740 1 --2- 172.21.15.44:0/1666047472 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a98210 0x558340a9a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.254+0000 7fab20085740 1 --2- 172.21.15.44:0/1666047472 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8eb20 0x558340a8ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.254+0000 7fab20085740 1 -- 172.21.15.44:0/1666047472 >> 172.21.15.44:0/1666047472 conn(0x558340a89440 msgr2=0x558340a89820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.254+0000 7fab20085740 1 -- 172.21.15.44:0/1666047472 shutdown_connections 2024-04-21T23:14:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.255+0000 7fab20085740 1 -- 172.21.15.44:0/1666047472 wait complete. 2024-04-21T23:14:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.255+0000 7fab20085740 1 -- start start 2024-04-21T23:14:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.255+0000 7fab20085740 1 Processor -- start 2024-04-21T23:14:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.255+0000 7fab20085740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8eb20 0x558340a952d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.255+0000 7fab20085740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a8f440 0x558340a95810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.255+0000 7fab20085740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a98210 0x558340a95d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.255+0000 7fab20085740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558340a9ade0 con 0x558340a8eb20 2024-04-21T23:14:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.255+0000 7fab20085740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5583409aea10 con 0x558340a8f440 2024-04-21T23:14:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.255+0000 7fab20085740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558340a9af60 con 0x558340a98210 2024-04-21T23:14:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.256+0000 7fab20085740 10 client.0 _ll_register_callbacks cb 0x5583409e9b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.256+0000 7fab177fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a98210 0x558340a95d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.256+0000 7fab177fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a98210 0x558340a95d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:49826/0 (socket says 172.21.15.44:49826) 2024-04-21T23:14:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.256+0000 7fab177fe640 1 -- 172.21.15.44:0/680166876 learned_addr learned my addr 172.21.15.44:0/680166876 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.256+0000 7fab167fc640 1 --2- 172.21.15.44:0/680166876 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a8f440 0x558340a95810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.256+0000 7fab16ffd640 1 --2- 172.21.15.44:0/680166876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8eb20 0x558340a952d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.256+0000 7fab177fe640 1 -- 172.21.15.44:0/680166876 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a8f440 msgr2=0x558340a95810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.256+0000 7fab177fe640 1 --2- 172.21.15.44:0/680166876 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a8f440 0x558340a95810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.256+0000 7fab177fe640 1 -- 172.21.15.44:0/680166876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8eb20 msgr2=0x558340a952d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.256+0000 7fab177fe640 1 --2- 172.21.15.44:0/680166876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8eb20 0x558340a952d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.256+0000 7fab177fe640 1 -- 172.21.15.44:0/680166876 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583407d3620 con 0x558340a98210 2024-04-21T23:14:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.256+0000 7fab16ffd640 1 --2- 172.21.15.44:0/680166876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8eb20 0x558340a952d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:14:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.256+0000 7fab177fe640 1 --2- 172.21.15.44:0/680166876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a98210 0x558340a95d50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fab0800b8b0 tx=0x7fab0800bcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2865cb264461d7b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.256+0000 7faafaff5640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:14:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.256+0000 7faafaff5640 1 -- 172.21.15.44:0/680166876 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0801b440 con 0x558340a98210 2024-04-21T23:14:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.256+0000 7fab20085740 1 -- 172.21.15.44:0/680166876 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab0c09ff90 con 0x558340a98210 2024-04-21T23:14:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.256+0000 7faafaff5640 1 -- 172.21.15.44:0/680166876 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fab08011440 con 0x558340a98210 2024-04-21T23:14:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.257+0000 7faafaff5640 1 -- 172.21.15.44:0/680166876 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fab08011640 con 0x558340a98210 2024-04-21T23:14:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.257+0000 7fab20085740 10 client.26413 fetch_fsmap learned FSMap version 25 2024-04-21T23:14:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.257+0000 7fab20085740 1 -- 172.21.15.44:0/680166876 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558340aa7130 con 0x558340a98210 2024-04-21T23:14:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.257+0000 7faafaff5640 1 -- 172.21.15.44:0/680166876 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab0c09ff90 con 0x558340a98210 2024-04-21T23:14:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.257+0000 7faafaff5640 1 -- 172.21.15.44:0/680166876 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fab08019020 con 0x558340a98210 2024-04-21T23:14:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.257+0000 7fab20085740 10 client.26413 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:14:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.257+0000 7fab20085740 10 client.26413 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.257+0000 7fab20085740 20 client.26413 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.257+0000 7fab20085740 1 --2- 172.21.15.44:0/680166876 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558340aa85f0 0x558340aaa9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.257+0000 7fab20085740 4 client.26413 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:14:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.257+0000 7fab20085740 1 -- 172.21.15.44:0/680166876 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558340a000b0 con 0x558340aa85f0 2024-04-21T23:14:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.257+0000 7fab16ffd640 1 --2- 172.21.15.44:0/680166876 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558340aa85f0 0x558340aaa9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.258+0000 7fab16ffd640 1 --2- 172.21.15.44:0/680166876 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558340aa85f0 0x558340aaa9e0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9e87b18b43532b5 server_cookie=8e1b3adb82ffb342 in_seq=0 out_seq=0 2024-04-21T23:14:04.260 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.258+0000 7faafaff5640 10 client.26413 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.259+0000 7faafaff5640 1 -- 172.21.15.44:0/680166876 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558340a000b0 con 0x558340aa85f0 2024-04-21T23:14:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.259+0000 7faafaff5640 10 client.26413 handle_command_reply: tid=0 2024-04-21T23:14:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.259+0000 7fab20085740 1 client.26413 shutdown 2024-04-21T23:14:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.259+0000 7fab20085740 1 -- 172.21.15.44:0/680166876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a98210 msgr2=0x558340a95d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.259+0000 7fab20085740 1 --2- 172.21.15.44:0/680166876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a98210 0x558340a95d50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fab0800b8b0 tx=0x7fab0800bcb0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.259+0000 7fab20085740 1 -- 172.21.15.44:0/680166876 shutdown_connections 2024-04-21T23:14:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.259+0000 7fab20085740 1 --2- 172.21.15.44:0/680166876 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558340aa85f0 0x558340aaa9e0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.259+0000 7fab20085740 1 --2- 172.21.15.44:0/680166876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a98210 0x558340a95d50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.259+0000 7faafaff5640 0 client.26413 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.259+0000 7fab20085740 1 --2- 172.21.15.44:0/680166876 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a8f440 0x558340a95810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.259+0000 7fab20085740 1 --2- 172.21.15.44:0/680166876 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8eb20 0x558340a952d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.259+0000 7fab20085740 1 -- 172.21.15.44:0/680166876 >> 172.21.15.44:0/680166876 conn(0x558340a89440 msgr2=0x558340a8b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.259+0000 7fab20085740 1 -- 172.21.15.44:0/680166876 shutdown_connections 2024-04-21T23:14:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.259+0000 7fab20085740 1 -- 172.21.15.44:0/680166876 wait complete. 2024-04-21T23:14:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.259+0000 7fab20085740 20 client.26413 trim_cache size 0 max 16384 2024-04-21T23:14:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.275+0000 7fab20085740 1 Processor -- start 2024-04-21T23:14:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.275+0000 7fab20085740 1 -- start start 2024-04-21T23:14:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.275+0000 7fab20085740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8eb20 0x558340a97a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.275+0000 7fab20085740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a8f440 0x558340aa3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.275+0000 7fab20085740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a98210 0x558340a959a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.275+0000 7fab20085740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fab180732d0 con 0x558340a8eb20 2024-04-21T23:14:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.275+0000 7fab20085740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fab18197000 con 0x558340a8f440 2024-04-21T23:14:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.275+0000 7fab20085740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fab18194e20 con 0x558340a98210 2024-04-21T23:14:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.275+0000 7fab16ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8eb20 0x558340a97a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.275+0000 7fab16ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8eb20 0x558340a97a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37404/0 (socket says 172.21.15.44:37404) 2024-04-21T23:14:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.276+0000 7fab16ffd640 1 -- 172.21.15.44:0/3572339101 learned_addr learned my addr 172.21.15.44:0/3572339101 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.276+0000 7fab167fc640 1 --2- 172.21.15.44:0/3572339101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a8f440 0x558340aa3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.276+0000 7fab177fe640 1 --2- 172.21.15.44:0/3572339101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a98210 0x558340a959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.276+0000 7fab16ffd640 1 -- 172.21.15.44:0/3572339101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a98210 msgr2=0x558340a959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.276+0000 7fab16ffd640 1 --2- 172.21.15.44:0/3572339101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a98210 0x558340a959a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.276+0000 7fab16ffd640 1 -- 172.21.15.44:0/3572339101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a8f440 msgr2=0x558340aa3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.276+0000 7fab16ffd640 1 --2- 172.21.15.44:0/3572339101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a8f440 0x558340aa3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.276+0000 7fab16ffd640 1 -- 172.21.15.44:0/3572339101 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558340a9a970 con 0x558340a8eb20 2024-04-21T23:14:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.276+0000 7fab167fc640 1 --2- 172.21.15.44:0/3572339101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a8f440 0x558340aa3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.276+0000 7fab16ffd640 1 --2- 172.21.15.44:0/3572339101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8eb20 0x558340a97a00 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fab10004af0 tx=0x7fab10049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25e765ce5dfc81b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.276+0000 7faafcff9640 1 -- 172.21.15.44:0/3572339101 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab10059070 con 0x558340a8eb20 2024-04-21T23:14:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.276+0000 7faafcff9640 1 -- 172.21.15.44:0/3572339101 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab10037d10 con 0x558340a8eb20 2024-04-21T23:14:04.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.276+0000 7faafcff9640 1 -- 172.21.15.44:0/3572339101 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab100532b0 con 0x558340a8eb20 2024-04-21T23:14:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.277+0000 7fab20085740 1 -- 172.21.15.44:0/3572339101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8eb20 msgr2=0x558340a97a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.277+0000 7fab20085740 1 --2- 172.21.15.44:0/3572339101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8eb20 0x558340a97a00 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fab10004af0 tx=0x7fab10049000 comp rx=0 tx=0).stop 2024-04-21T23:14:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.277+0000 7fab20085740 1 -- 172.21.15.44:0/3572339101 shutdown_connections 2024-04-21T23:14:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.277+0000 7fab20085740 1 --2- 172.21.15.44:0/3572339101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a98210 0x558340a959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.277+0000 7fab20085740 1 --2- 172.21.15.44:0/3572339101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a8f440 0x558340aa3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.277+0000 7fab20085740 1 --2- 172.21.15.44:0/3572339101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8eb20 0x558340a97a00 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.277+0000 7fab20085740 1 -- 172.21.15.44:0/3572339101 >> 172.21.15.44:0/3572339101 conn(0x558340a89440 msgr2=0x558340a95290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.277+0000 7fab20085740 1 -- 172.21.15.44:0/3572339101 shutdown_connections 2024-04-21T23:14:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.277+0000 7fab20085740 1 -- 172.21.15.44:0/3572339101 wait complete. 2024-04-21T23:14:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.277+0000 7fab20085740 1 -- start start 2024-04-21T23:14:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.277+0000 7fab20085740 1 Processor -- start 2024-04-21T23:14:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab20085740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a8eb20 0x558340a92230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab20085740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8f440 0x558340a92770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab20085740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a98210 0x558340b0db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab167fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8f440 0x558340a92770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab16ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a8eb20 0x558340a92230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab16ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a8eb20 0x558340a92230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:49838/0 (socket says 172.21.15.44:49838) 2024-04-21T23:14:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab167fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8f440 0x558340a92770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37408/0 (socket says 172.21.15.44:37408) 2024-04-21T23:14:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab16ffd640 1 -- 172.21.15.44:0/4058335433 learned_addr learned my addr 172.21.15.44:0/4058335433 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab20085740 1 -- 172.21.15.44:0/4058335433 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fab181954c0 con 0x558340a8f440 2024-04-21T23:14:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab20085740 1 -- 172.21.15.44:0/4058335433 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fab18196930 con 0x558340a98210 2024-04-21T23:14:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab20085740 1 -- 172.21.15.44:0/4058335433 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fab18195b90 con 0x558340a8eb20 2024-04-21T23:14:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab20085740 10 client.0 _ll_register_callbacks cb 0x7fab08004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab177fe640 1 --2- 172.21.15.44:0/4058335433 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a98210 0x558340b0db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab167fc640 1 -- 172.21.15.44:0/4058335433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a8eb20 msgr2=0x558340a92230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab167fc640 1 --2- 172.21.15.44:0/4058335433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a8eb20 0x558340a92230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab167fc640 1 -- 172.21.15.44:0/4058335433 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a98210 msgr2=0x558340b0db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab167fc640 1 --2- 172.21.15.44:0/4058335433 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a98210 0x558340b0db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab167fc640 1 -- 172.21.15.44:0/4058335433 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558340b0e050 con 0x558340a8f440 2024-04-21T23:14:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab16ffd640 1 --2- 172.21.15.44:0/4058335433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a8eb20 0x558340a92230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab167fc640 1 --2- 172.21.15.44:0/4058335433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8f440 0x558340a92770 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fab0c003c50 tx=0x7fab0c003be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=681969aa986c7dcf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7faafaff5640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:14:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7faafaff5640 1 -- 172.21.15.44:0/4058335433 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0c0ba4d0 con 0x558340a8f440 2024-04-21T23:14:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7faafaff5640 1 -- 172.21.15.44:0/4058335433 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fab0c09b030 con 0x558340a8f440 2024-04-21T23:14:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.278+0000 7fab20085740 1 -- 172.21.15.44:0/4058335433 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab10037d10 con 0x558340a8f440 2024-04-21T23:14:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.279+0000 7faafaff5640 1 -- 172.21.15.44:0/4058335433 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fab0c09ff90 con 0x558340a8f440 2024-04-21T23:14:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.279+0000 7fab20085740 10 client.16827 fetch_fsmap learned FSMap version 25 2024-04-21T23:14:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.279+0000 7fab20085740 1 -- 172.21.15.44:0/4058335433 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558340b0f0c0 con 0x558340a8f440 2024-04-21T23:14:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.279+0000 7faafaff5640 1 -- 172.21.15.44:0/4058335433 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab10037d10 con 0x558340a8f440 2024-04-21T23:14:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.279+0000 7faafaff5640 1 -- 172.21.15.44:0/4058335433 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fab0c0979c0 con 0x558340a8f440 2024-04-21T23:14:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.279+0000 7fab20085740 10 client.16827 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:14:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.279+0000 7fab20085740 10 client.16827 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.279+0000 7fab20085740 20 client.16827 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.279+0000 7fab20085740 1 --2- 172.21.15.44:0/4058335433 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558340b0f410 0x558340b11830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.279+0000 7fab20085740 4 client.16827 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:14:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.279+0000 7fab20085740 1 -- 172.21.15.44:0/4058335433 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5583409fa400 con 0x558340b0f410 2024-04-21T23:14:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.279+0000 7fab16ffd640 1 --2- 172.21.15.44:0/4058335433 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558340b0f410 0x558340b11830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.279+0000 7fab16ffd640 1 --2- 172.21.15.44:0/4058335433 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558340b0f410 0x558340b11830 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f41195b38cb3ece0 server_cookie=e16c11fbf6749e8f in_seq=0 out_seq=0 2024-04-21T23:14:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.279+0000 7faafaff5640 10 client.16827 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:04.546 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.546+0000 7faafaff5640 1 -- 172.21.15.44:0/4058335433 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5583409fa400 con 0x558340b0f410 2024-04-21T23:14:04.546 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.546+0000 7faafaff5640 10 client.16827 handle_command_reply: tid=0 2024-04-21T23:14:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.546+0000 7fab20085740 1 client.16827 shutdown 2024-04-21T23:14:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.546+0000 7fab20085740 1 -- 172.21.15.44:0/4058335433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8f440 msgr2=0x558340a92770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.546+0000 7fab20085740 1 --2- 172.21.15.44:0/4058335433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8f440 0x558340a92770 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fab0c003c50 tx=0x7fab0c003be0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.546+0000 7fab20085740 1 -- 172.21.15.44:0/4058335433 shutdown_connections 2024-04-21T23:14:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.546+0000 7fab20085740 1 --2- 172.21.15.44:0/4058335433 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558340b0f410 0x558340b11830 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.546+0000 7faafaff5640 0 client.16827 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.546+0000 7fab20085740 1 --2- 172.21.15.44:0/4058335433 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558340a98210 0x558340b0db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.546+0000 7fab20085740 1 --2- 172.21.15.44:0/4058335433 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558340a8f440 0x558340a92770 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.546+0000 7fab20085740 1 --2- 172.21.15.44:0/4058335433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558340a8eb20 0x558340a92230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.546+0000 7fab20085740 1 -- 172.21.15.44:0/4058335433 >> 172.21.15.44:0/4058335433 conn(0x558340a89440 msgr2=0x558340a884c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.547+0000 7fab20085740 1 -- 172.21.15.44:0/4058335433 shutdown_connections 2024-04-21T23:14:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.547+0000 7fab20085740 1 -- 172.21.15.44:0/4058335433 wait complete. 2024-04-21T23:14:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.547+0000 7fab20085740 20 client.16827 trim_cache size 0 max 16384 2024-04-21T23:14:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:14:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:14:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:14:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:14:04.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.547+0000 7fab1d4a3640 1 -- 172.21.15.44:0/129923657 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fab000a2e40 msgr2=0x7fab000a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:04.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.547+0000 7fab1d4a3640 1 --2- 172.21.15.44:0/129923657 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fab000a2e40 0x7fab000a5300 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fab10006010 tx=0x7fab1000f600 comp rx=0 tx=0).stop 2024-04-21T23:14:04.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.548+0000 7fab1d4a3640 1 -- 172.21.15.44:0/129923657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab181032f0 msgr2=0x7fab1819eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:04.549 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.548+0000 7fab1d4a3640 1 --2- 172.21.15.44:0/129923657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab181032f0 0x7fab1819eb80 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fab0c00e9c0 tx=0x7fab0c00ee80 comp rx=0 tx=0).stop 2024-04-21T23:14:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.548+0000 7fab1d4a3640 1 -- 172.21.15.44:0/129923657 shutdown_connections 2024-04-21T23:14:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.548+0000 7fab1d4a3640 1 --2- 172.21.15.44:0/129923657 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fab000a2e40 0x7fab000a5300 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.548+0000 7fab1d4a3640 1 --2- 172.21.15.44:0/129923657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab1810bda0 0x7fab181a2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.548+0000 7fab1d4a3640 1 --2- 172.21.15.44:0/129923657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab181032f0 0x7fab1819eb80 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.548+0000 7fab1d4a3640 1 --2- 172.21.15.44:0/129923657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab181029d0 0x7fab1819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.548+0000 7fab1d4a3640 1 -- 172.21.15.44:0/129923657 >> 172.21.15.44:0/129923657 conn(0x7fab18077790 msgr2=0x7fab180fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.548+0000 7fab1d4a3640 1 -- 172.21.15.44:0/129923657 shutdown_connections 2024-04-21T23:14:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:04.548+0000 7fab1d4a3640 1 -- 172.21.15.44:0/129923657 wait complete. 2024-04-21T23:14:04.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 592607 13.38 MB/sec execute 114 sec latency 112.056 ms 2024-04-21T23:14:05.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 594755 13.37 MB/sec execute 115 sec latency 17.619 ms 2024-04-21T23:14:06.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:05 smithi044 ceph-mon[27934]: pgmap v162: 129 pgs: 129 active+clean; 78 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 268 op/s 2024-04-21T23:14:06.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:05 smithi069 ceph-mon[31816]: pgmap v162: 129 pgs: 129 active+clean; 78 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 268 op/s 2024-04-21T23:14:06.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:05 smithi055 ceph-mon[31117]: pgmap v162: 129 pgs: 129 active+clean; 78 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 268 op/s 2024-04-21T23:14:06.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 597378 13.37 MB/sec execute 116 sec latency 16.728 ms 2024-04-21T23:14:07.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 599922 13.37 MB/sec execute 117 sec latency 117.443 ms 2024-04-21T23:14:08.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:07 smithi055 ceph-mon[31117]: pgmap v163: 129 pgs: 129 active+clean; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.8 MiB/s wr, 513 op/s 2024-04-21T23:14:08.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:07 smithi069 ceph-mon[31816]: pgmap v163: 129 pgs: 129 active+clean; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.8 MiB/s wr, 513 op/s 2024-04-21T23:14:08.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:07 smithi044 ceph-mon[27934]: pgmap v163: 129 pgs: 129 active+clean; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.8 MiB/s wr, 513 op/s 2024-04-21T23:14:08.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 601748 13.35 MB/sec execute 118 sec latency 30.237 ms 2024-04-21T23:14:09.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 604526 13.36 MB/sec execute 119 sec latency 18.945 ms 2024-04-21T23:14:10.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:09 smithi055 ceph-mon[31117]: pgmap v164: 129 pgs: 129 active+clean; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.6 MiB/s wr, 410 op/s 2024-04-21T23:14:10.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:09 smithi069 ceph-mon[31816]: pgmap v164: 129 pgs: 129 active+clean; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.6 MiB/s wr, 410 op/s 2024-04-21T23:14:10.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:09 smithi044 ceph-mon[27934]: pgmap v164: 129 pgs: 129 active+clean; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.6 MiB/s wr, 410 op/s 2024-04-21T23:14:10.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 607645 13.40 MB/sec execute 120 sec latency 11.325 ms 2024-04-21T23:14:11.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 610274 13.40 MB/sec execute 121 sec latency 40.819 ms 2024-04-21T23:14:12.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:11 smithi055 ceph-mon[31117]: pgmap v165: 129 pgs: 129 active+clean; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 401 op/s 2024-04-21T23:14:12.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:11 smithi069 ceph-mon[31816]: pgmap v165: 129 pgs: 129 active+clean; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 401 op/s 2024-04-21T23:14:12.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:11 smithi044 ceph-mon[27934]: pgmap v165: 129 pgs: 129 active+clean; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 401 op/s 2024-04-21T23:14:12.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 612145 13.36 MB/sec execute 122 sec latency 32.371 ms 2024-04-21T23:14:13.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 614683 13.39 MB/sec execute 123 sec latency 88.301 ms 2024-04-21T23:14:14.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:13 smithi055 ceph-mon[31117]: pgmap v166: 129 pgs: 129 active+clean; 74 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.9 MiB/s wr, 529 op/s 2024-04-21T23:14:14.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:13 smithi069 ceph-mon[31816]: pgmap v166: 129 pgs: 129 active+clean; 74 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.9 MiB/s wr, 529 op/s 2024-04-21T23:14:14.439 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:13 smithi044 ceph-mon[27934]: pgmap v166: 129 pgs: 129 active+clean; 74 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.9 MiB/s wr, 529 op/s 2024-04-21T23:14:14.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 617044 13.35 MB/sec execute 124 sec latency 19.833 ms 2024-04-21T23:14:15.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 619150 13.34 MB/sec execute 125 sec latency 14.548 ms 2024-04-21T23:14:16.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:15 smithi055 ceph-mon[31117]: pgmap v167: 129 pgs: 129 active+clean; 74 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.4 MiB/s wr, 393 op/s 2024-04-21T23:14:16.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:15 smithi069 ceph-mon[31816]: pgmap v167: 129 pgs: 129 active+clean; 74 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.4 MiB/s wr, 393 op/s 2024-04-21T23:14:16.378 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:14:16.379 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:14:16.400 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:15 smithi044 ceph-mon[27934]: pgmap v167: 129 pgs: 129 active+clean; 74 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.4 MiB/s wr, 393 op/s 2024-04-21T23:14:16.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.487+0000 7fb65ffb8640 1 -- 172.21.15.44:0/3469070789 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb658100fd0 msgr2=0x7fb65810d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.487+0000 7fb65ffb8640 1 --2- 172.21.15.44:0/3469070789 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb658100fd0 0x7fb65810d4e0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fb650009cd0 tx=0x7fb65002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.487+0000 7fb65ffb8640 1 -- 172.21.15.44:0/3469070789 shutdown_connections 2024-04-21T23:14:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.487+0000 7fb65ffb8640 1 --2- 172.21.15.44:0/3469070789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb658100610 0x7fb658100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.487+0000 7fb65ffb8640 1 --2- 172.21.15.44:0/3469070789 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb658100fd0 0x7fb65810d4e0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.487+0000 7fb65ffb8640 1 --2- 172.21.15.44:0/3469070789 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb6581066b0 0x7fb658106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.487+0000 7fb65ffb8640 1 -- 172.21.15.44:0/3469070789 >> 172.21.15.44:0/3469070789 conn(0x7fb6580777f0 msgr2=0x7fb658077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.488+0000 7fb65ffb8640 1 -- 172.21.15.44:0/3469070789 shutdown_connections 2024-04-21T23:14:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.488+0000 7fb65ffb8640 1 -- 172.21.15.44:0/3469070789 wait complete. 2024-04-21T23:14:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.488+0000 7fb65ffb8640 1 Processor -- start 2024-04-21T23:14:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.488+0000 7fb65ffb8640 1 -- start start 2024-04-21T23:14:16.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.488+0000 7fb65ffb8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb658100610 0x7fb658196e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.488+0000 7fb65ffb8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb658100fd0 0x7fb658197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.488+0000 7fb65ffb8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6581066b0 0x7fb65819b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.488+0000 7fb65ffb8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581109f0 con 0x7fb6581066b0 2024-04-21T23:14:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.488+0000 7fb65ffb8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb658110870 con 0x7fb658100610 2024-04-21T23:14:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.488+0000 7fb65ffb8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb658110b70 con 0x7fb658100fd0 2024-04-21T23:14:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.489+0000 7fb65e555640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6581066b0 0x7fb65819b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.489+0000 7fb65e555640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6581066b0 0x7fb65819b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43646/0 (socket says 172.21.15.44:43646) 2024-04-21T23:14:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.489+0000 7fb65dd54640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb658100610 0x7fb658196e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.489+0000 7fb65e555640 1 -- 172.21.15.44:0/395229064 learned_addr learned my addr 172.21.15.44:0/395229064 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.489+0000 7fb65d553640 1 --2- 172.21.15.44:0/395229064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb658100fd0 0x7fb658197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.489+0000 7fb65e555640 1 -- 172.21.15.44:0/395229064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb658100fd0 msgr2=0x7fb658197370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.489+0000 7fb65e555640 1 --2- 172.21.15.44:0/395229064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb658100fd0 0x7fb658197370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.489+0000 7fb65e555640 1 -- 172.21.15.44:0/395229064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb658100610 msgr2=0x7fb658196e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:16.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.489+0000 7fb65e555640 1 --2- 172.21.15.44:0/395229064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb658100610 0x7fb658196e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.489+0000 7fb65e555640 1 -- 172.21.15.44:0/395229064 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb65819bdd0 con 0x7fb6581066b0 2024-04-21T23:14:16.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.489+0000 7fb65d553640 1 --2- 172.21.15.44:0/395229064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb658100fd0 0x7fb658197370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:14:16.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.489+0000 7fb65e555640 1 --2- 172.21.15.44:0/395229064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6581066b0 0x7fb65819b6a0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb650005e30 tx=0x7fb650032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:16.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.489+0000 7fb64effd640 1 -- 172.21.15.44:0/395229064 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb650041020 con 0x7fb6581066b0 2024-04-21T23:14:16.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.489+0000 7fb64effd640 1 -- 172.21.15.44:0/395229064 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb650034cd0 con 0x7fb6581066b0 2024-04-21T23:14:16.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.489+0000 7fb65ffb8640 1 -- 172.21.15.44:0/395229064 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb65819c060 con 0x7fb6581066b0 2024-04-21T23:14:16.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.489+0000 7fb64effd640 1 -- 172.21.15.44:0/395229064 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb650039040 con 0x7fb6581066b0 2024-04-21T23:14:16.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.489+0000 7fb65ffb8640 1 -- 172.21.15.44:0/395229064 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6581a3950 con 0x7fb6581066b0 2024-04-21T23:14:16.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.491+0000 7fb64effd640 1 -- 172.21.15.44:0/395229064 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb650044020 con 0x7fb6581066b0 2024-04-21T23:14:16.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.492+0000 7fb662ba0740 1 Processor -- start 2024-04-21T23:14:16.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.492+0000 7fb662ba0740 1 -- start start 2024-04-21T23:14:16.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.492+0000 7fb662ba0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c4bf0 0x55d2778c4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.492+0000 7fb64effd640 1 --2- 172.21.15.44:0/395229064 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb6440a2d80 0x7fb6440a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.492+0000 7fb662ba0740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c5510 0x55d2778d0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.492+0000 7fb662ba0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 0x55d2778d2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.493+0000 7fb662ba0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d277829560 con 0x55d2778c4bf0 2024-04-21T23:14:16.498 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.493+0000 7fb662ba0740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55d2777d1730 con 0x55d2778c5510 2024-04-21T23:14:16.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.493+0000 7fb662ba0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d277825520 con 0x55d2778d0640 2024-04-21T23:14:16.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.493+0000 7fb65dd54640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c4bf0 0x55d2778c4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.493+0000 7fb65d553640 1 --2- 172.21.15.44:0/395229064 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb6440a2d80 0x7fb6440a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.493+0000 7fb65e555640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 0x55d2778d2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.493+0000 7fb65e555640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 0x55d2778d2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:43084/0 (socket says 172.21.15.44:43084) 2024-04-21T23:14:16.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.493+0000 7fb65e555640 1 -- 172.21.15.44:0/3853446103 learned_addr learned my addr 172.21.15.44:0/3853446103 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:16.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.493+0000 7fb65d553640 1 --2- 172.21.15.44:0/3853446103 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c5510 0x55d2778d0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.493+0000 7fb64effd640 1 -- 172.21.15.44:0/395229064 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fb6500f1450 con 0x7fb6581066b0 2024-04-21T23:14:16.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.493+0000 7fb65d553640 1 -- 172.21.15.44:0/3853446103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 msgr2=0x55d2778d2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:16.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.493+0000 7fb65d553640 1 --2- 172.21.15.44:0/3853446103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 0x55d2778d2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.493+0000 7fb65d553640 1 -- 172.21.15.44:0/3853446103 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c4bf0 msgr2=0x55d2778c4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:16.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.493+0000 7fb65d553640 1 --2- 172.21.15.44:0/3853446103 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c4bf0 0x55d2778c4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.499 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.493+0000 7fb65d553640 1 -- 172.21.15.44:0/3853446103 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2776077d0 con 0x55d2778c5510 2024-04-21T23:14:16.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.493+0000 7fb65e555640 1 --2- 172.21.15.44:0/3853446103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 0x55d2778d2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:16.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.493+0000 7fb65d553640 1 --2- 172.21.15.44:0/395229064 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb6440a2d80 0x7fb6440a5240 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb654005d90 tx=0x7fb65400bb50 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:16.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.493+0000 7fb65d553640 1 --2- 172.21.15.44:0/3853446103 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c5510 0x55d2778d0100 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb658077d10 tx=0x7fb65401a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2d0c3038425ee1f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:16.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.494+0000 7fb643fff640 1 -- 172.21.15.44:0/3853446103 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb654021070 con 0x55d2778c5510 2024-04-21T23:14:16.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.494+0000 7fb643fff640 1 -- 172.21.15.44:0/3853446103 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb654005500 con 0x55d2778c5510 2024-04-21T23:14:16.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.494+0000 7fb643fff640 1 -- 172.21.15.44:0/3853446103 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65400e790 con 0x55d2778c5510 2024-04-21T23:14:16.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.494+0000 7fb662ba0740 1 -- 172.21.15.44:0/3853446103 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c5510 msgr2=0x55d2778d0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:16.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.494+0000 7fb662ba0740 1 --2- 172.21.15.44:0/3853446103 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c5510 0x55d2778d0100 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb658077d10 tx=0x7fb65401a000 comp rx=0 tx=0).stop 2024-04-21T23:14:16.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.494+0000 7fb662ba0740 1 -- 172.21.15.44:0/3853446103 shutdown_connections 2024-04-21T23:14:16.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.494+0000 7fb662ba0740 1 --2- 172.21.15.44:0/3853446103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 0x55d2778d2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.494+0000 7fb662ba0740 1 --2- 172.21.15.44:0/3853446103 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c5510 0x55d2778d0100 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.494+0000 7fb662ba0740 1 --2- 172.21.15.44:0/3853446103 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c4bf0 0x55d2778c4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.494+0000 7fb662ba0740 1 -- 172.21.15.44:0/3853446103 >> 172.21.15.44:0/3853446103 conn(0x55d2778bf660 msgr2=0x55d2778bfa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:16.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.495+0000 7fb662ba0740 1 -- 172.21.15.44:0/3853446103 shutdown_connections 2024-04-21T23:14:16.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.495+0000 7fb662ba0740 1 -- 172.21.15.44:0/3853446103 wait complete. 2024-04-21T23:14:16.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.495+0000 7fb662ba0740 1 -- start start 2024-04-21T23:14:16.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.495+0000 7fb662ba0740 1 Processor -- start 2024-04-21T23:14:16.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.495+0000 7fb662ba0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c4bf0 0x55d2778cf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.495+0000 7fb662ba0740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c5510 0x55d2778cfce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.495+0000 7fb662ba0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 0x55d2778cbc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.495+0000 7fb662ba0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d2778c61b0 con 0x55d2778c4bf0 2024-04-21T23:14:16.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.495+0000 7fb662ba0740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55d2777e4e00 con 0x55d2778c5510 2024-04-21T23:14:16.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.495+0000 7fb662ba0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d2778c6330 con 0x55d2778d0640 2024-04-21T23:14:16.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.495+0000 7fb662ba0740 10 client.0 _ll_register_callbacks cb 0x55d27781feb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:16.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.495+0000 7fb65dd54640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c4bf0 0x55d2778cf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.495+0000 7fb65d553640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c5510 0x55d2778cfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.495+0000 7fb65dd54640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c4bf0 0x55d2778cf7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43664/0 (socket says 172.21.15.44:43664) 2024-04-21T23:14:16.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.495+0000 7fb65d553640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c5510 0x55d2778cfce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:53668/0 (socket says 172.21.15.44:53668) 2024-04-21T23:14:16.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.496+0000 7fb65dd54640 1 -- 172.21.15.44:0/372948383 learned_addr learned my addr 172.21.15.44:0/372948383 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:16.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.496+0000 7fb65e555640 1 --2- 172.21.15.44:0/372948383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 0x55d2778cbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.496+0000 7fb65d553640 1 -- 172.21.15.44:0/372948383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 msgr2=0x55d2778cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:16.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.496+0000 7fb65d553640 1 --2- 172.21.15.44:0/372948383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 0x55d2778cbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.496+0000 7fb65d553640 1 -- 172.21.15.44:0/372948383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c4bf0 msgr2=0x55d2778cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:16.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.496+0000 7fb65d553640 1 --2- 172.21.15.44:0/372948383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c4bf0 0x55d2778cf7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.496+0000 7fb65d553640 1 -- 172.21.15.44:0/372948383 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d27753d3f0 con 0x55d2778c5510 2024-04-21T23:14:16.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.496+0000 7fb65dd54640 1 --2- 172.21.15.44:0/372948383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c4bf0 0x55d2778cf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:16.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.496+0000 7fb65d553640 1 --2- 172.21.15.44:0/372948383 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c5510 0x55d2778cfce0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb65400b300 tx=0x7fb654004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6167f6625b5d40b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:16.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.496+0000 7fb6417fa640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:14:16.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.496+0000 7fb6417fa640 1 -- 172.21.15.44:0/372948383 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb654021040 con 0x55d2778c5510 2024-04-21T23:14:16.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.496+0000 7fb662ba0740 1 -- 172.21.15.44:0/372948383 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb654005500 con 0x55d2778c5510 2024-04-21T23:14:16.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.496+0000 7fb6417fa640 1 -- 172.21.15.44:0/372948383 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fb654005180 con 0x55d2778c5510 2024-04-21T23:14:16.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.496+0000 7fb6417fa640 1 -- 172.21.15.44:0/372948383 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb65400be60 con 0x55d2778c5510 2024-04-21T23:14:16.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.496+0000 7fb662ba0740 10 client.26402 fetch_fsmap learned FSMap version 25 2024-04-21T23:14:16.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.496+0000 7fb662ba0740 1 -- 172.21.15.44:0/372948383 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d2774fd0f0 con 0x55d2778c5510 2024-04-21T23:14:16.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.497+0000 7fb6417fa640 1 -- 172.21.15.44:0/372948383 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb654005500 con 0x55d2778c5510 2024-04-21T23:14:16.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.497+0000 7fb6417fa640 1 -- 172.21.15.44:0/372948383 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fb65400eaf0 con 0x55d2778c5510 2024-04-21T23:14:16.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.497+0000 7fb662ba0740 10 client.26402 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:14:16.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.497+0000 7fb662ba0740 10 client.26402 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:16.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.497+0000 7fb662ba0740 20 client.26402 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:16.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.497+0000 7fb662ba0740 1 --2- 172.21.15.44:0/372948383 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d2778e2bd0 0x55d2778e4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.497+0000 7fb662ba0740 4 client.26402 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:14:16.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.497+0000 7fb662ba0740 1 -- 172.21.15.44:0/372948383 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d277836aa0 con 0x55d2778e2bd0 2024-04-21T23:14:16.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.497+0000 7fb65dd54640 1 --2- 172.21.15.44:0/372948383 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d2778e2bd0 0x55d2778e4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.497+0000 7fb6417fa640 10 client.26402 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:16.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.497+0000 7fb65dd54640 1 --2- 172.21.15.44:0/372948383 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d2778e2bd0 0x55d2778e4fc0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d24311f519baded server_cookie=d9bf753d6a59cc1f in_seq=0 out_seq=0 2024-04-21T23:14:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.499+0000 7fb6417fa640 1 -- 172.21.15.44:0/372948383 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d277836aa0 con 0x55d2778e2bd0 2024-04-21T23:14:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.499+0000 7fb6417fa640 10 client.26402 handle_command_reply: tid=0 2024-04-21T23:14:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.499+0000 7fb662ba0740 1 client.26402 shutdown 2024-04-21T23:14:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.499+0000 7fb662ba0740 1 -- 172.21.15.44:0/372948383 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c5510 msgr2=0x55d2778cfce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.499+0000 7fb662ba0740 1 --2- 172.21.15.44:0/372948383 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c5510 0x55d2778cfce0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb65400b300 tx=0x7fb654004b50 comp rx=0 tx=0).stop 2024-04-21T23:14:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.500+0000 7fb662ba0740 1 -- 172.21.15.44:0/372948383 shutdown_connections 2024-04-21T23:14:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.500+0000 7fb662ba0740 1 --2- 172.21.15.44:0/372948383 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d2778e2bd0 0x55d2778e4fc0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.500+0000 7fb662ba0740 1 --2- 172.21.15.44:0/372948383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 0x55d2778cbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.500+0000 7fb662ba0740 1 --2- 172.21.15.44:0/372948383 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c5510 0x55d2778cfce0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.500+0000 7fb662ba0740 1 --2- 172.21.15.44:0/372948383 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c4bf0 0x55d2778cf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.500+0000 7fb6417fa640 0 client.26402 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:16.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.500+0000 7fb662ba0740 1 -- 172.21.15.44:0/372948383 >> 172.21.15.44:0/372948383 conn(0x55d2778bf660 msgr2=0x55d2778c13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:16.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.500+0000 7fb662ba0740 1 -- 172.21.15.44:0/372948383 shutdown_connections 2024-04-21T23:14:16.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.500+0000 7fb662ba0740 1 -- 172.21.15.44:0/372948383 wait complete. 2024-04-21T23:14:16.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.500+0000 7fb662ba0740 20 client.26402 trim_cache size 0 max 16384 2024-04-21T23:14:16.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.519+0000 7fb662ba0740 1 Processor -- start 2024-04-21T23:14:16.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.519+0000 7fb662ba0740 1 -- start start 2024-04-21T23:14:16.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.519+0000 7fb662ba0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c4bf0 0x55d2778ea730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.523 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.519+0000 7fb662ba0740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c5510 0x55d2778eac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.519+0000 7fb662ba0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 0x55d2778cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.519+0000 7fb662ba0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581aa920 con 0x55d2778c4bf0 2024-04-21T23:14:16.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.519+0000 7fb662ba0740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581ad1a0 con 0x55d2778c5510 2024-04-21T23:14:16.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.519+0000 7fb662ba0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581aaf90 con 0x55d2778d0640 2024-04-21T23:14:16.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.519+0000 7fb65dd54640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c4bf0 0x55d2778ea730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.519+0000 7fb65dd54640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c4bf0 0x55d2778ea730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43678/0 (socket says 172.21.15.44:43678) 2024-04-21T23:14:16.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.519+0000 7fb65dd54640 1 -- 172.21.15.44:0/2683290037 learned_addr learned my addr 172.21.15.44:0/2683290037 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:16.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.519+0000 7fb65e555640 1 --2- 172.21.15.44:0/2683290037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 0x55d2778cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.519+0000 7fb65d553640 1 --2- 172.21.15.44:0/2683290037 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c5510 0x55d2778eac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.520+0000 7fb65dd54640 1 -- 172.21.15.44:0/2683290037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 msgr2=0x55d2778cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:16.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.520+0000 7fb65dd54640 1 --2- 172.21.15.44:0/2683290037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 0x55d2778cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.520+0000 7fb65dd54640 1 -- 172.21.15.44:0/2683290037 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c5510 msgr2=0x55d2778eac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:16.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.520+0000 7fb65dd54640 1 --2- 172.21.15.44:0/2683290037 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c5510 0x55d2778eac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.520+0000 7fb65dd54640 1 -- 172.21.15.44:0/2683290037 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2778c60f0 con 0x55d2778c4bf0 2024-04-21T23:14:16.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.520+0000 7fb65e555640 1 --2- 172.21.15.44:0/2683290037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 0x55d2778cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:16.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.520+0000 7fb65d553640 1 --2- 172.21.15.44:0/2683290037 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c5510 0x55d2778eac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:16.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.520+0000 7fb65dd54640 1 --2- 172.21.15.44:0/2683290037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c4bf0 0x55d2778ea730 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x55d2778e56a0 tx=0x7fb648005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd7ab0730dcb965 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:16.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.520+0000 7fb6437fe640 1 -- 172.21.15.44:0/2683290037 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb648024070 con 0x55d2778c4bf0 2024-04-21T23:14:16.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.520+0000 7fb6437fe640 1 -- 172.21.15.44:0/2683290037 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb648005020 con 0x55d2778c4bf0 2024-04-21T23:14:16.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.520+0000 7fb6437fe640 1 -- 172.21.15.44:0/2683290037 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb648004b60 con 0x55d2778c4bf0 2024-04-21T23:14:16.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.521+0000 7fb662ba0740 1 -- 172.21.15.44:0/2683290037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c4bf0 msgr2=0x55d2778ea730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:16.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.521+0000 7fb662ba0740 1 --2- 172.21.15.44:0/2683290037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c4bf0 0x55d2778ea730 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x55d2778e56a0 tx=0x7fb648005f90 comp rx=0 tx=0).stop 2024-04-21T23:14:16.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.521+0000 7fb662ba0740 1 -- 172.21.15.44:0/2683290037 shutdown_connections 2024-04-21T23:14:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.521+0000 7fb662ba0740 1 --2- 172.21.15.44:0/2683290037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 0x55d2778cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.521+0000 7fb662ba0740 1 --2- 172.21.15.44:0/2683290037 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c5510 0x55d2778eac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.521+0000 7fb662ba0740 1 --2- 172.21.15.44:0/2683290037 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c4bf0 0x55d2778ea730 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.521+0000 7fb662ba0740 1 -- 172.21.15.44:0/2683290037 >> 172.21.15.44:0/2683290037 conn(0x55d2778bf660 msgr2=0x55d2778ca560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.521+0000 7fb662ba0740 1 -- 172.21.15.44:0/2683290037 shutdown_connections 2024-04-21T23:14:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.521+0000 7fb662ba0740 1 -- 172.21.15.44:0/2683290037 wait complete. 2024-04-21T23:14:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.521+0000 7fb662ba0740 1 -- start start 2024-04-21T23:14:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.521+0000 7fb662ba0740 1 Processor -- start 2024-04-21T23:14:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.521+0000 7fb662ba0740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c4bf0 0x55d2778dc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.521+0000 7fb662ba0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c5510 0x55d2778d9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.521+0000 7fb662ba0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 0x55d2778da0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.521+0000 7fb662ba0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581ab660 con 0x55d2778c5510 2024-04-21T23:14:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.521+0000 7fb662ba0740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581acad0 con 0x55d2778c4bf0 2024-04-21T23:14:16.526 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.521+0000 7fb662ba0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581abd30 con 0x55d2778d0640 2024-04-21T23:14:16.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.522+0000 7fb662ba0740 10 client.0 _ll_register_callbacks cb 0x7fb6480097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:16.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.522+0000 7fb65d553640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c5510 0x55d2778d9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.522+0000 7fb65dd54640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c4bf0 0x55d2778dc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.522+0000 7fb65d553640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c5510 0x55d2778d9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43694/0 (socket says 172.21.15.44:43694) 2024-04-21T23:14:16.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.522+0000 7fb65d553640 1 -- 172.21.15.44:0/1992191257 learned_addr learned my addr 172.21.15.44:0/1992191257 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:16.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.522+0000 7fb65e555640 1 --2- 172.21.15.44:0/1992191257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 0x55d2778da0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.522+0000 7fb65dd54640 1 -- 172.21.15.44:0/1992191257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 msgr2=0x55d2778da0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:16.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.522+0000 7fb65dd54640 1 --2- 172.21.15.44:0/1992191257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 0x55d2778da0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.522+0000 7fb65dd54640 1 -- 172.21.15.44:0/1992191257 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c5510 msgr2=0x55d2778d9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:16.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.522+0000 7fb65dd54640 1 --2- 172.21.15.44:0/1992191257 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c5510 0x55d2778d9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.522+0000 7fb65dd54640 1 -- 172.21.15.44:0/1992191257 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2778da8d0 con 0x55d2778c4bf0 2024-04-21T23:14:16.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.522+0000 7fb65d553640 1 --2- 172.21.15.44:0/1992191257 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c5510 0x55d2778d9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:16.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.522+0000 7fb65dd54640 1 --2- 172.21.15.44:0/1992191257 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c4bf0 0x55d2778dc800 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb6480126c0 tx=0x7fb648009dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4cc2b843d4d2f0a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:16.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.522+0000 7fb6417fa640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:14:16.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.522+0000 7fb6417fa640 1 -- 172.21.15.44:0/1992191257 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb64802f070 con 0x55d2778c4bf0 2024-04-21T23:14:16.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.522+0000 7fb662ba0740 1 -- 172.21.15.44:0/1992191257 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb648005020 con 0x55d2778c4bf0 2024-04-21T23:14:16.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.522+0000 7fb6417fa640 1 -- 172.21.15.44:0/1992191257 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fb648010860 con 0x55d2778c4bf0 2024-04-21T23:14:16.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.522+0000 7fb6417fa640 1 -- 172.21.15.44:0/1992191257 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb64803b050 con 0x55d2778c4bf0 2024-04-21T23:14:16.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.522+0000 7fb662ba0740 10 client.26408 fetch_fsmap learned FSMap version 25 2024-04-21T23:14:16.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.522+0000 7fb662ba0740 1 -- 172.21.15.44:0/1992191257 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d277948f00 con 0x55d2778c4bf0 2024-04-21T23:14:16.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.523+0000 7fb6417fa640 1 -- 172.21.15.44:0/1992191257 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb648005020 con 0x55d2778c4bf0 2024-04-21T23:14:16.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.523+0000 7fb6417fa640 1 -- 172.21.15.44:0/1992191257 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fb6480200f0 con 0x55d2778c4bf0 2024-04-21T23:14:16.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.523+0000 7fb662ba0740 10 client.26408 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:14:16.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.523+0000 7fb662ba0740 10 client.26408 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:16.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.523+0000 7fb662ba0740 20 client.26408 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:16.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.523+0000 7fb662ba0740 1 --2- 172.21.15.44:0/1992191257 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d277949570 0x55d27794b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.523+0000 7fb662ba0740 4 client.26408 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:14:16.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.523+0000 7fb662ba0740 1 -- 172.21.15.44:0/1992191257 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d277830df0 con 0x55d277949570 2024-04-21T23:14:16.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.523+0000 7fb65d553640 1 --2- 172.21.15.44:0/1992191257 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d277949570 0x55d27794b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.523+0000 7fb6417fa640 10 client.26408 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:16.529 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.523+0000 7fb65d553640 1 --2- 172.21.15.44:0/1992191257 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d277949570 0x55d27794b960 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e2089d3dbf1c106 server_cookie=dd24a10eab04aeaa in_seq=0 out_seq=0 2024-04-21T23:14:16.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.525+0000 7fb6417fa640 1 -- 172.21.15.44:0/1992191257 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d277830df0 con 0x55d277949570 2024-04-21T23:14:16.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.525+0000 7fb6417fa640 10 client.26408 handle_command_reply: tid=0 2024-04-21T23:14:16.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.525+0000 7fb662ba0740 1 client.26408 shutdown 2024-04-21T23:14:16.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.525+0000 7fb662ba0740 1 -- 172.21.15.44:0/1992191257 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c4bf0 msgr2=0x55d2778dc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:16.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.525+0000 7fb662ba0740 1 --2- 172.21.15.44:0/1992191257 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c4bf0 0x55d2778dc800 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb6480126c0 tx=0x7fb648009dc0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.525+0000 7fb662ba0740 1 -- 172.21.15.44:0/1992191257 shutdown_connections 2024-04-21T23:14:16.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.525+0000 7fb662ba0740 1 --2- 172.21.15.44:0/1992191257 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d277949570 0x55d27794b960 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.525+0000 7fb662ba0740 1 --2- 172.21.15.44:0/1992191257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2778d0640 0x55d2778da0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.525+0000 7fb662ba0740 1 --2- 172.21.15.44:0/1992191257 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d2778c5510 0x55d2778d9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.525+0000 7fb662ba0740 1 --2- 172.21.15.44:0/1992191257 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d2778c4bf0 0x55d2778dc800 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.525+0000 7fb6417fa640 0 client.26408 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:16.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.525+0000 7fb662ba0740 1 -- 172.21.15.44:0/1992191257 >> 172.21.15.44:0/1992191257 conn(0x55d2778bf660 msgr2=0x55d2778cf3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:16.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.525+0000 7fb662ba0740 1 -- 172.21.15.44:0/1992191257 shutdown_connections 2024-04-21T23:14:16.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.525+0000 7fb662ba0740 1 -- 172.21.15.44:0/1992191257 wait complete. 2024-04-21T23:14:16.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.525+0000 7fb662ba0740 20 client.26408 trim_cache size 0 max 16384 2024-04-21T23:14:16.531 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:14:16.531 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-21T23:14:16.531 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-21T23:14:16.531 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:14:16.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.526+0000 7fb65ffb8640 1 -- 172.21.15.44:0/395229064 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb6440a2d80 msgr2=0x7fb6440a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:16.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.526+0000 7fb65ffb8640 1 --2- 172.21.15.44:0/395229064 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb6440a2d80 0x7fb6440a5240 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb654005d90 tx=0x7fb65400bb50 comp rx=0 tx=0).stop 2024-04-21T23:14:16.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.526+0000 7fb65ffb8640 1 -- 172.21.15.44:0/395229064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6581066b0 msgr2=0x7fb65819b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:16.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.526+0000 7fb65ffb8640 1 --2- 172.21.15.44:0/395229064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6581066b0 0x7fb65819b6a0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb650005e30 tx=0x7fb650032ea0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.526+0000 7fb65ffb8640 1 -- 172.21.15.44:0/395229064 shutdown_connections 2024-04-21T23:14:16.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.526+0000 7fb65ffb8640 1 --2- 172.21.15.44:0/395229064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb658100fd0 0x7fb658197370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.526+0000 7fb65ffb8640 1 --2- 172.21.15.44:0/395229064 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb6440a2d80 0x7fb6440a5240 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.526+0000 7fb65ffb8640 1 --2- 172.21.15.44:0/395229064 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb6581066b0 0x7fb65819b6a0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.526+0000 7fb65ffb8640 1 --2- 172.21.15.44:0/395229064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb658100610 0x7fb658196e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.526+0000 7fb65ffb8640 1 -- 172.21.15.44:0/395229064 >> 172.21.15.44:0/395229064 conn(0x7fb6580777f0 msgr2=0x7fb6580ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:16.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.526+0000 7fb65ffb8640 1 -- 172.21.15.44:0/395229064 shutdown_connections 2024-04-21T23:14:16.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.526+0000 7fb65ffb8640 1 -- 172.21.15.44:0/395229064 wait complete. 2024-04-21T23:14:16.537 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:14:16.537 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:14:16.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.685+0000 7fe9135de640 1 -- 172.21.15.44:0/1345903862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe90c101740 msgr2=0x7fe90c101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:16.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.685+0000 7fe9135de640 1 --2- 172.21.15.44:0/1345903862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe90c101740 0x7fe90c101bc0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009930 tx=0x7fe8fc02bc30 comp rx=0 tx=0).stop 2024-04-21T23:14:16.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.685+0000 7fe9135de640 1 -- 172.21.15.44:0/1345903862 shutdown_connections 2024-04-21T23:14:16.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.685+0000 7fe9135de640 1 --2- 172.21.15.44:0/1345903862 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe90c102100 0x7fe90c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.685+0000 7fe9135de640 1 --2- 172.21.15.44:0/1345903862 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe90c101740 0x7fe90c101bc0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.685+0000 7fe9135de640 1 --2- 172.21.15.44:0/1345903862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90c107750 0x7fe90c107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.685+0000 7fe9135de640 1 -- 172.21.15.44:0/1345903862 >> 172.21.15.44:0/1345903862 conn(0x7fe90c0fd1f0 msgr2=0x7fe90c0ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.685+0000 7fe9135de640 1 -- 172.21.15.44:0/1345903862 shutdown_connections 2024-04-21T23:14:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.685+0000 7fe9135de640 1 -- 172.21.15.44:0/1345903862 wait complete. 2024-04-21T23:14:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.686+0000 7fe9135de640 1 Processor -- start 2024-04-21T23:14:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.686+0000 7fe9135de640 1 -- start start 2024-04-21T23:14:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.686+0000 7fe9135de640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe90c101740 0x7fe90c198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.686+0000 7fe9135de640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90c102100 0x7fe90c1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.686+0000 7fe9135de640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe90c107750 0x7fe90c19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.686+0000 7fe9135de640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c111b60 con 0x7fe90c101740 2024-04-21T23:14:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.686+0000 7fe9135de640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c1119e0 con 0x7fe90c107750 2024-04-21T23:14:16.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.686+0000 7fe9135de640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c111ce0 con 0x7fe90c102100 2024-04-21T23:14:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.686+0000 7fe91137a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe90c101740 0x7fe90c198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.686+0000 7fe91137a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe90c101740 0x7fe90c198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43706/0 (socket says 172.21.15.44:43706) 2024-04-21T23:14:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.686+0000 7fe91137a640 1 -- 172.21.15.44:0/3675579384 learned_addr learned my addr 172.21.15.44:0/3675579384 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.686+0000 7fe910b79640 1 --2- 172.21.15.44:0/3675579384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90c102100 0x7fe90c1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.686+0000 7fe911b7b640 1 --2- 172.21.15.44:0/3675579384 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe90c107750 0x7fe90c19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.686+0000 7fe91137a640 1 -- 172.21.15.44:0/3675579384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90c102100 msgr2=0x7fe90c1985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.686+0000 7fe91137a640 1 --2- 172.21.15.44:0/3675579384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90c102100 0x7fe90c1985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.686+0000 7fe91137a640 1 -- 172.21.15.44:0/3675579384 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe90c107750 msgr2=0x7fe90c19c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.686+0000 7fe91137a640 1 --2- 172.21.15.44:0/3675579384 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe90c107750 0x7fe90c19c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.686+0000 7fe91137a640 1 -- 172.21.15.44:0/3675579384 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe90c19d020 con 0x7fe90c101740 2024-04-21T23:14:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.686+0000 7fe910b79640 1 --2- 172.21.15.44:0/3675579384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90c102100 0x7fe90c1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.687+0000 7fe91137a640 1 --2- 172.21.15.44:0/3675579384 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe90c101740 0x7fe90c198080 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fe90800e9f0 tx=0x7fe90800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:16.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.687+0000 7fe9027fc640 1 -- 172.21.15.44:0/3675579384 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90800cd80 con 0x7fe90c101740 2024-04-21T23:14:16.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.687+0000 7fe9027fc640 1 -- 172.21.15.44:0/3675579384 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe908010910 con 0x7fe90c101740 2024-04-21T23:14:16.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.687+0000 7fe9135de640 1 -- 172.21.15.44:0/3675579384 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe90c19d310 con 0x7fe90c101740 2024-04-21T23:14:16.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.687+0000 7fe9027fc640 1 -- 172.21.15.44:0/3675579384 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe908018990 con 0x7fe90c101740 2024-04-21T23:14:16.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.687+0000 7fe9135de640 1 -- 172.21.15.44:0/3675579384 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe90c1a4ba0 con 0x7fe90c101740 2024-04-21T23:14:16.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.688+0000 7fe9135de640 1 -- 172.21.15.44:0/3675579384 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe90c102e80 con 0x7fe90c101740 2024-04-21T23:14:16.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.689+0000 7fe9027fc640 1 -- 172.21.15.44:0/3675579384 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe908010ac0 con 0x7fe90c101740 2024-04-21T23:14:16.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.690+0000 7fe9027fc640 1 --2- 172.21.15.44:0/3675579384 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe8f80a2d00 0x7fe8f80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:16.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.690+0000 7fe910b79640 1 --2- 172.21.15.44:0/3675579384 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe8f80a2d00 0x7fe8f80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:16.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.690+0000 7fe9027fc640 1 -- 172.21.15.44:0/3675579384 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fe9080cfa70 con 0x7fe90c101740 2024-04-21T23:14:16.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.691+0000 7fe910b79640 1 --2- 172.21.15.44:0/3675579384 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe8f80a2d00 0x7fe8f80a51c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc005ab0 tx=0x7fe8fc005fd0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:16.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.695+0000 7fe9027fc640 1 -- 172.21.15.44:0/3675579384 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fe90809bca0 con 0x7fe90c101740 2024-04-21T23:14:16.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 621999 13.37 MB/sec execute 126 sec latency 7.990 ms 2024-04-21T23:14:16.950 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.951+0000 7fe9135de640 1 -- 172.21.15.44:0/3675579384 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe90c0685f0 con 0x7fe90c101740 2024-04-21T23:14:16.951 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.951+0000 7fe9027fc640 1 -- 172.21.15.44:0/3675579384 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11791 (secure 0 0 0) 0x7fe90809ffa0 con 0x7fe90c101740 2024-04-21T23:14:16.951 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:14:16.951 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:11:24.214378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25843":{"gid":25843,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3162569119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3162569119},{"type":"v1","addr":"172.21.15.69:6839","nonce":3162569119}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:14:16.952 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 25 2024-04-21T23:14:16.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.954+0000 7fe9135de640 1 -- 172.21.15.44:0/3675579384 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe8f80a2d00 msgr2=0x7fe8f80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:16.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.954+0000 7fe9135de640 1 --2- 172.21.15.44:0/3675579384 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe8f80a2d00 0x7fe8f80a51c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc005ab0 tx=0x7fe8fc005fd0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.954+0000 7fe9135de640 1 -- 172.21.15.44:0/3675579384 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe90c101740 msgr2=0x7fe90c198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:16.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.954+0000 7fe9135de640 1 --2- 172.21.15.44:0/3675579384 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe90c101740 0x7fe90c198080 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fe90800e9f0 tx=0x7fe90800eeb0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.954+0000 7fe9135de640 1 -- 172.21.15.44:0/3675579384 shutdown_connections 2024-04-21T23:14:16.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.954+0000 7fe9135de640 1 --2- 172.21.15.44:0/3675579384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90c102100 0x7fe90c1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.954+0000 7fe9135de640 1 --2- 172.21.15.44:0/3675579384 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe8f80a2d00 0x7fe8f80a51c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.954+0000 7fe9135de640 1 --2- 172.21.15.44:0/3675579384 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe90c107750 0x7fe90c19c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.954+0000 7fe9135de640 1 --2- 172.21.15.44:0/3675579384 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe90c101740 0x7fe90c198080 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:16.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.954+0000 7fe9135de640 1 -- 172.21.15.44:0/3675579384 >> 172.21.15.44:0/3675579384 conn(0x7fe90c0fd1f0 msgr2=0x7fe90c0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:16.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.954+0000 7fe9135de640 1 -- 172.21.15.44:0/3675579384 shutdown_connections 2024-04-21T23:14:16.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:16.954+0000 7fe9135de640 1 -- 172.21.15.44:0/3675579384 wait complete. 2024-04-21T23:14:16.964 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:14:17.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.079+0000 7f8a25ed4640 1 -- 172.21.15.44:0/4096172111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a20100a30 msgr2=0x7f8a2010e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:17.081 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.079+0000 7f8a25ed4640 1 --2- 172.21.15.44:0/4096172111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a20100a30 0x7f8a2010e710 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8a18009c10 tx=0x7f8a1802bef0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.079+0000 7f8a25ed4640 1 -- 172.21.15.44:0/4096172111 shutdown_connections 2024-04-21T23:14:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.079+0000 7f8a25ed4640 1 --2- 172.21.15.44:0/4096172111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a20100a30 0x7f8a2010e710 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.079+0000 7f8a25ed4640 1 --2- 172.21.15.44:0/4096172111 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a20100070 0x7f8a201004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.079+0000 7f8a25ed4640 1 --2- 172.21.15.44:0/4096172111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a20106870 0x7f8a20106c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.079+0000 7f8a25ed4640 1 -- 172.21.15.44:0/4096172111 >> 172.21.15.44:0/4096172111 conn(0x7f8a200758c0 msgr2=0x7f8a20075cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.079+0000 7f8a25ed4640 1 -- 172.21.15.44:0/4096172111 shutdown_connections 2024-04-21T23:14:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.079+0000 7f8a25ed4640 1 -- 172.21.15.44:0/4096172111 wait complete. 2024-04-21T23:14:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.080+0000 7f8a25ed4640 1 Processor -- start 2024-04-21T23:14:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.080+0000 7f8a25ed4640 1 -- start start 2024-04-21T23:14:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.080+0000 7f8a25ed4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a20100070 0x7f8a201980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.080+0000 7f8a25ed4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a20100a30 0x7f8a201985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.080+0000 7f8a25ed4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a20106870 0x7f8a2019c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.080+0000 7f8a25ed4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8a20111bf0 con 0x7f8a20100a30 2024-04-21T23:14:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.080+0000 7f8a25ed4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8a20111a70 con 0x7f8a20100070 2024-04-21T23:14:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.080+0000 7f8a25ed4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a20111d70 con 0x7f8a20106870 2024-04-21T23:14:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.080+0000 7f8a1effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a20100a30 0x7f8a201985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.080+0000 7f8a1effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a20100a30 0x7f8a201985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43724/0 (socket says 172.21.15.44:43724) 2024-04-21T23:14:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.080+0000 7f8a1effd640 1 -- 172.21.15.44:0/601521935 learned_addr learned my addr 172.21.15.44:0/601521935 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.080+0000 7f8a1f7fe640 1 --2- 172.21.15.44:0/601521935 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a20100070 0x7f8a201980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.080+0000 7f8a1ffff640 1 --2- 172.21.15.44:0/601521935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a20106870 0x7f8a2019c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.081+0000 7f8a1effd640 1 -- 172.21.15.44:0/601521935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a20106870 msgr2=0x7f8a2019c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.081+0000 7f8a1effd640 1 --2- 172.21.15.44:0/601521935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a20106870 0x7f8a2019c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.081+0000 7f8a1effd640 1 -- 172.21.15.44:0/601521935 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a20100070 msgr2=0x7f8a201980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.081+0000 7f8a1effd640 1 --2- 172.21.15.44:0/601521935 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a20100070 0x7f8a201980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.083 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.081+0000 7f8a1effd640 1 -- 172.21.15.44:0/601521935 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a2019d050 con 0x7f8a20100a30 2024-04-21T23:14:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.081+0000 7f8a1f7fe640 1 --2- 172.21.15.44:0/601521935 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a20100070 0x7f8a201980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.081+0000 7f8a1effd640 1 --2- 172.21.15.44:0/601521935 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a20100a30 0x7f8a201985f0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400e9f0 tx=0x7f8a1400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.081+0000 7f8a1cff9640 1 -- 172.21.15.44:0/601521935 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1400cd80 con 0x7f8a20100a30 2024-04-21T23:14:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.081+0000 7f8a25ed4640 1 -- 172.21.15.44:0/601521935 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a2019d340 con 0x7f8a20100a30 2024-04-21T23:14:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.081+0000 7f8a25ed4640 1 -- 172.21.15.44:0/601521935 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a201a4bd0 con 0x7f8a20100a30 2024-04-21T23:14:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.081+0000 7f8a1cff9640 1 -- 172.21.15.44:0/601521935 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a14010910 con 0x7f8a20100a30 2024-04-21T23:14:17.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.081+0000 7f8a1cff9640 1 -- 172.21.15.44:0/601521935 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a140188f0 con 0x7f8a20100a30 2024-04-21T23:14:17.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.084+0000 7f8a1cff9640 1 -- 172.21.15.44:0/601521935 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8a14010ac0 con 0x7f8a20100a30 2024-04-21T23:14:17.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.084+0000 7f8a28b9e740 1 Processor -- start 2024-04-21T23:14:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.084+0000 7f8a28b9e740 1 -- start start 2024-04-21T23:14:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.084+0000 7f8a28b9e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b020 0x55ab1d30b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.084+0000 7f8a1cff9640 1 --2- 172.21.15.44:0/601521935 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f8a080a2d00 0x7f8a080a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.084+0000 7f8a28b9e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d30b940 0x55ab1d313fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.084+0000 7f8a28b9e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d314520 0x55ab1d316910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.084+0000 7f8a28b9e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ab1d26f530 con 0x55ab1d314520 2024-04-21T23:14:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.084+0000 7f8a28b9e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55ab1d217830 con 0x55ab1d30b020 2024-04-21T23:14:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.084+0000 7f8a28b9e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ab1d26b4f0 con 0x55ab1d30b940 2024-04-21T23:14:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.085+0000 7f8a1ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b020 0x55ab1d30b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.085+0000 7f8a1effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d30b940 0x55ab1d313fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.085+0000 7f8a1f7fe640 1 --2- 172.21.15.44:0/601521935 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f8a080a2d00 0x7f8a080a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.085+0000 7f8a1ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b020 0x55ab1d30b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:53748/0 (socket says 172.21.15.44:53748) 2024-04-21T23:14:17.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.085+0000 7f8a1ffff640 1 -- 172.21.15.44:0/1215741077 learned_addr learned my addr 172.21.15.44:0/1215741077 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:17.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.085+0000 7f8a1ffff640 1 --2- 172.21.15.44:0/1215741077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d314520 0x55ab1d316910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.085+0000 7f8a1cff9640 1 -- 172.21.15.44:0/601521935 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f8a140cfbc0 con 0x7f8a20100a30 2024-04-21T23:14:17.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.085+0000 7f8a1ffff640 1 -- 172.21.15.44:0/1215741077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d30b940 msgr2=0x55ab1d313fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:17.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.085+0000 7f8a1ffff640 1 --2- 172.21.15.44:0/1215741077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d30b940 0x55ab1d313fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.085+0000 7f8a1ffff640 1 -- 172.21.15.44:0/1215741077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d314520 msgr2=0x55ab1d316910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:17.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.085+0000 7f8a1ffff640 1 --2- 172.21.15.44:0/1215741077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d314520 0x55ab1d316910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.085+0000 7f8a1ffff640 1 -- 172.21.15.44:0/1215741077 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab1d23b2c0 con 0x55ab1d30b020 2024-04-21T23:14:17.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.085+0000 7f8a1effd640 1 --2- 172.21.15.44:0/1215741077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d30b940 0x55ab1d313fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:17.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.085+0000 7f8a1f7fe640 1 --2- 172.21.15.44:0/601521935 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f8a080a2d00 0x7f8a080a51c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8a100096c0 tx=0x7f8a10009260 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:17.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.085+0000 7f8a1ffff640 1 --2- 172.21.15.44:0/1215741077 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b020 0x55ab1d30b400 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8a18009c10 tx=0x7f8a18009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8921f6db85260628 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:17.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.085+0000 7f8a05ffb640 1 -- 172.21.15.44:0/1215741077 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a18049070 con 0x55ab1d30b020 2024-04-21T23:14:17.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.085+0000 7f8a05ffb640 1 -- 172.21.15.44:0/1215741077 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a1802d950 con 0x55ab1d30b020 2024-04-21T23:14:17.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.086+0000 7f8a05ffb640 1 -- 172.21.15.44:0/1215741077 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a18004030 con 0x55ab1d30b020 2024-04-21T23:14:17.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.086+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1215741077 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b020 msgr2=0x55ab1d30b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:17.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.086+0000 7f8a28b9e740 1 --2- 172.21.15.44:0/1215741077 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b020 0x55ab1d30b400 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8a18009c10 tx=0x7f8a18009500 comp rx=0 tx=0).stop 2024-04-21T23:14:17.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.086+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1215741077 shutdown_connections 2024-04-21T23:14:17.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.086+0000 7f8a28b9e740 1 --2- 172.21.15.44:0/1215741077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d30b940 0x55ab1d313fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.086+0000 7f8a28b9e740 1 --2- 172.21.15.44:0/1215741077 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d314520 0x55ab1d316910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.086+0000 7f8a28b9e740 1 --2- 172.21.15.44:0/1215741077 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b020 0x55ab1d30b400 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.086+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1215741077 >> 172.21.15.44:0/1215741077 conn(0x55ab1d305750 msgr2=0x55ab1d305b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:17.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.086+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1215741077 shutdown_connections 2024-04-21T23:14:17.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.086+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1215741077 wait complete. 2024-04-21T23:14:17.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.087+0000 7f8a28b9e740 1 -- start start 2024-04-21T23:14:17.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.087+0000 7f8a28b9e740 1 Processor -- start 2024-04-21T23:14:17.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.087+0000 7f8a28b9e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d30b020 0x55ab1d313e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.087+0000 7f8a28b9e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b940 0x55ab1d3111e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.087+0000 7f8a28b9e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d314520 0x55ab1d311720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.087+0000 7f8a28b9e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55ab1d3172b0 con 0x55ab1d314520 2024-04-21T23:14:17.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.087+0000 7f8a28b9e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55ab1d22af10 con 0x55ab1d30b940 2024-04-21T23:14:17.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.087+0000 7f8a28b9e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ab1d317430 con 0x55ab1d30b020 2024-04-21T23:14:17.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.087+0000 7f8a28b9e740 10 client.0 _ll_register_callbacks cb 0x55ab1cf82be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:17.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.087+0000 7f8a1ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d314520 0x55ab1d311720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.087+0000 7f8a1f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d30b020 0x55ab1d313e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.087+0000 7f8a1effd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b940 0x55ab1d3111e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.087+0000 7f8a1ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d314520 0x55ab1d311720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43736/0 (socket says 172.21.15.44:43736) 2024-04-21T23:14:17.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.087+0000 7f8a1ffff640 1 -- 172.21.15.44:0/1447646501 learned_addr learned my addr 172.21.15.44:0/1447646501 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:17.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f8a1ffff640 1 -- 172.21.15.44:0/1447646501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d30b020 msgr2=0x55ab1d313e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:17.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f8a1ffff640 1 --2- 172.21.15.44:0/1447646501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d30b020 0x55ab1d313e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f8a1ffff640 1 -- 172.21.15.44:0/1447646501 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b940 msgr2=0x55ab1d3111e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:17.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f8a1ffff640 1 --2- 172.21.15.44:0/1447646501 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b940 0x55ab1d3111e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f8a1ffff640 1 -- 172.21.15.44:0/1447646501 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab1d1fcc80 con 0x55ab1d314520 2024-04-21T23:14:17.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f8a1effd640 1 --2- 172.21.15.44:0/1447646501 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b940 0x55ab1d3111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:17.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f8a1ffff640 1 --2- 172.21.15.44:0/1447646501 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d314520 0x55ab1d311720 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f8a18009be0 tx=0x7f8a18003bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da038a2dd4beeb6a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:17.094 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f89feffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:14:17.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f89feffd640 1 -- 172.21.15.44:0/1447646501 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a18049070 con 0x55ab1d314520 2024-04-21T23:14:17.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f89feffd640 1 -- 172.21.15.44:0/1447646501 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f8a18004d40 con 0x55ab1d314520 2024-04-21T23:14:17.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1447646501 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a1802d950 con 0x55ab1d314520 2024-04-21T23:14:17.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f89feffd640 1 -- 172.21.15.44:0/1447646501 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8a18033480 con 0x55ab1d314520 2024-04-21T23:14:17.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f8a28b9e740 10 client.16863 fetch_fsmap learned FSMap version 25 2024-04-21T23:14:17.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1447646501 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ab1cf833f0 con 0x55ab1d314520 2024-04-21T23:14:17.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f89feffd640 1 -- 172.21.15.44:0/1447646501 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a1802d950 con 0x55ab1d314520 2024-04-21T23:14:17.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f89feffd640 1 -- 172.21.15.44:0/1447646501 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f8a18041470 con 0x55ab1d314520 2024-04-21T23:14:17.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f8a28b9e740 10 client.16863 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:14:17.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f8a28b9e740 10 client.16863 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:17.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f8a28b9e740 20 client.16863 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:17.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f8a28b9e740 1 --2- 172.21.15.44:0/1447646501 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55ab1d3243b0 0x55ab1d3267a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f8a28b9e740 4 client.16863 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:14:17.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.088+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1447646501 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab1d27ca70 con 0x55ab1d3243b0 2024-04-21T23:14:17.095 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.089+0000 7f8a1f7fe640 1 --2- 172.21.15.44:0/1447646501 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55ab1d3243b0 0x55ab1d3267a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.089+0000 7f8a1f7fe640 1 --2- 172.21.15.44:0/1447646501 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55ab1d3243b0 0x55ab1d3267a0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ba8e22d8a8f59bc server_cookie=566472c95fc7a9f6 in_seq=0 out_seq=0 2024-04-21T23:14:17.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.089+0000 7f89feffd640 10 client.16863 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:17.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.090+0000 7f89feffd640 1 -- 172.21.15.44:0/1447646501 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ab1d27ca70 con 0x55ab1d3243b0 2024-04-21T23:14:17.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.090+0000 7f89feffd640 10 client.16863 handle_command_reply: tid=0 2024-04-21T23:14:17.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.090+0000 7f8a28b9e740 1 client.16863 shutdown 2024-04-21T23:14:17.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.090+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1447646501 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d314520 msgr2=0x55ab1d311720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:17.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.090+0000 7f8a28b9e740 1 --2- 172.21.15.44:0/1447646501 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d314520 0x55ab1d311720 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f8a18009be0 tx=0x7f8a18003bd0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.090+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1447646501 shutdown_connections 2024-04-21T23:14:17.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.090+0000 7f8a28b9e740 1 --2- 172.21.15.44:0/1447646501 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55ab1d3243b0 0x55ab1d3267a0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.090+0000 7f8a28b9e740 1 --2- 172.21.15.44:0/1447646501 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d314520 0x55ab1d311720 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.090+0000 7f8a28b9e740 1 --2- 172.21.15.44:0/1447646501 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b940 0x55ab1d3111e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.090+0000 7f89feffd640 0 client.16863 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:17.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.090+0000 7f8a28b9e740 1 --2- 172.21.15.44:0/1447646501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d30b020 0x55ab1d313e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.090+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1447646501 >> 172.21.15.44:0/1447646501 conn(0x55ab1d305750 msgr2=0x55ab1d3074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:17.096 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.090+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1447646501 shutdown_connections 2024-04-21T23:14:17.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.090+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1447646501 wait complete. 2024-04-21T23:14:17.097 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.090+0000 7f8a28b9e740 20 client.16863 trim_cache size 0 max 16384 2024-04-21T23:14:17.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.104+0000 7f8a28b9e740 1 Processor -- start 2024-04-21T23:14:17.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.104+0000 7f8a28b9e740 1 -- start start 2024-04-21T23:14:17.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.105+0000 7f8a28b9e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b020 0x55ab1d30ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.109 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.105+0000 7f8a28b9e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d30b940 0x55ab1d313910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.105+0000 7f8a28b9e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d314520 0x55ab1d313e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.105+0000 7f8a28b9e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8a201acc50 con 0x55ab1d30b940 2024-04-21T23:14:17.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.105+0000 7f8a28b9e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8a201af4d0 con 0x55ab1d30b020 2024-04-21T23:14:17.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.105+0000 7f8a28b9e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a201ad2c0 con 0x55ab1d314520 2024-04-21T23:14:17.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.105+0000 7f8a1effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d30b940 0x55ab1d313910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.105+0000 7f8a1effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d30b940 0x55ab1d313910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43746/0 (socket says 172.21.15.44:43746) 2024-04-21T23:14:17.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.105+0000 7f8a1effd640 1 -- 172.21.15.44:0/502818224 learned_addr learned my addr 172.21.15.44:0/502818224 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:17.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.105+0000 7f8a1f7fe640 1 --2- 172.21.15.44:0/502818224 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b020 0x55ab1d30ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.105+0000 7f8a1effd640 1 -- 172.21.15.44:0/502818224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d314520 msgr2=0x55ab1d313e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:17.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.105+0000 7f8a1effd640 1 --2- 172.21.15.44:0/502818224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d314520 0x55ab1d313e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.110 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.105+0000 7f8a1effd640 1 -- 172.21.15.44:0/502818224 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b020 msgr2=0x55ab1d30ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:17.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.105+0000 7f8a1effd640 1 --2- 172.21.15.44:0/502818224 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b020 0x55ab1d30ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.105+0000 7f8a1effd640 1 -- 172.21.15.44:0/502818224 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab1d04d7d0 con 0x55ab1d30b940 2024-04-21T23:14:17.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.105+0000 7f8a1effd640 1 --2- 172.21.15.44:0/502818224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d30b940 0x55ab1d313910 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f8a1409baf0 tx=0x7f8a1406bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50335779749e1d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:17.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.105+0000 7f8a057fa640 1 -- 172.21.15.44:0/502818224 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a140a00f0 con 0x55ab1d30b940 2024-04-21T23:14:17.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.105+0000 7f8a057fa640 1 -- 172.21.15.44:0/502818224 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a140a49b0 con 0x55ab1d30b940 2024-04-21T23:14:17.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.105+0000 7f8a057fa640 1 -- 172.21.15.44:0/502818224 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1409b4a0 con 0x55ab1d30b940 2024-04-21T23:14:17.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.106+0000 7f8a28b9e740 1 -- 172.21.15.44:0/502818224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d30b940 msgr2=0x55ab1d313910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:17.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.106+0000 7f8a28b9e740 1 --2- 172.21.15.44:0/502818224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d30b940 0x55ab1d313910 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f8a1409baf0 tx=0x7f8a1406bd10 comp rx=0 tx=0).stop 2024-04-21T23:14:17.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.106+0000 7f8a28b9e740 1 -- 172.21.15.44:0/502818224 shutdown_connections 2024-04-21T23:14:17.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.106+0000 7f8a28b9e740 1 --2- 172.21.15.44:0/502818224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d314520 0x55ab1d313e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.106+0000 7f8a28b9e740 1 --2- 172.21.15.44:0/502818224 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d30b940 0x55ab1d313910 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.111 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.106+0000 7f8a28b9e740 1 --2- 172.21.15.44:0/502818224 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b020 0x55ab1d30ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.106+0000 7f8a28b9e740 1 -- 172.21.15.44:0/502818224 >> 172.21.15.44:0/502818224 conn(0x55ab1d305750 msgr2=0x55ab1d307430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:17.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.106+0000 7f8a28b9e740 1 -- 172.21.15.44:0/502818224 shutdown_connections 2024-04-21T23:14:17.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.106+0000 7f8a28b9e740 1 -- 172.21.15.44:0/502818224 wait complete. 2024-04-21T23:14:17.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.107+0000 7f8a28b9e740 1 -- start start 2024-04-21T23:14:17.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.107+0000 7f8a28b9e740 1 Processor -- start 2024-04-21T23:14:17.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.107+0000 7f8a28b9e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b020 0x55ab1d327230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.107+0000 7f8a28b9e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d30b940 0x55ab1d327770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.107+0000 7f8a28b9e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d314520 0x55ab1d327cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.107+0000 7f8a28b9e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8a201ad990 con 0x55ab1d314520 2024-04-21T23:14:17.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.107+0000 7f8a28b9e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8a201aee00 con 0x55ab1d30b020 2024-04-21T23:14:17.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.107+0000 7f8a28b9e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a201ae060 con 0x55ab1d30b940 2024-04-21T23:14:17.112 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.107+0000 7f8a28b9e740 10 client.0 _ll_register_callbacks cb 0x7f8a18009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:17.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.107+0000 7f8a1ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d314520 0x55ab1d327cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.107+0000 7f8a1f7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b020 0x55ab1d327230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.107+0000 7f8a1effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d30b940 0x55ab1d327770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.107+0000 7f8a1ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d314520 0x55ab1d327cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43756/0 (socket says 172.21.15.44:43756) 2024-04-21T23:14:17.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.107+0000 7f8a1f7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b020 0x55ab1d327230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:53776/0 (socket says 172.21.15.44:53776) 2024-04-21T23:14:17.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.107+0000 7f8a1ffff640 1 -- 172.21.15.44:0/1898887562 learned_addr learned my addr 172.21.15.44:0/1898887562 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:17.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.108+0000 7f8a1ffff640 1 -- 172.21.15.44:0/1898887562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d30b940 msgr2=0x55ab1d327770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:17.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.108+0000 7f8a1ffff640 1 --2- 172.21.15.44:0/1898887562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d30b940 0x55ab1d327770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.108+0000 7f8a1ffff640 1 -- 172.21.15.44:0/1898887562 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b020 msgr2=0x55ab1d327230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:17.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.108+0000 7f8a1ffff640 1 --2- 172.21.15.44:0/1898887562 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b020 0x55ab1d327230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.108+0000 7f8a1ffff640 1 -- 172.21.15.44:0/1898887562 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab1d3895c0 con 0x55ab1d314520 2024-04-21T23:14:17.113 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.108+0000 7f8a1effd640 1 --2- 172.21.15.44:0/1898887562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d30b940 0x55ab1d327770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:17.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.108+0000 7f8a1f7fe640 1 --2- 172.21.15.44:0/1898887562 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b020 0x55ab1d327230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:17.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.108+0000 7f8a1ffff640 1 --2- 172.21.15.44:0/1898887562 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d314520 0x55ab1d327cb0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f8a18002230 tx=0x7f8a18009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d761d8d9d443aa6f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:17.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.108+0000 7f89feffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:14:17.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.108+0000 7f89feffd640 1 -- 172.21.15.44:0/1898887562 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a18049040 con 0x55ab1d314520 2024-04-21T23:14:17.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.108+0000 7f89feffd640 1 -- 172.21.15.44:0/1898887562 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f8a18004b40 con 0x55ab1d314520 2024-04-21T23:14:17.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.108+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1898887562 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a140a49b0 con 0x55ab1d314520 2024-04-21T23:14:17.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.108+0000 7f89feffd640 1 -- 172.21.15.44:0/1898887562 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8a18067050 con 0x55ab1d314520 2024-04-21T23:14:17.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.108+0000 7f8a28b9e740 10 client.16869 fetch_fsmap learned FSMap version 25 2024-04-21T23:14:17.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.108+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1898887562 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ab1d38a8a0 con 0x55ab1d314520 2024-04-21T23:14:17.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.108+0000 7f89feffd640 1 -- 172.21.15.44:0/1898887562 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a140a49b0 con 0x55ab1d314520 2024-04-21T23:14:17.114 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.109+0000 7f89feffd640 1 -- 172.21.15.44:0/1898887562 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f8a180337b0 con 0x55ab1d314520 2024-04-21T23:14:17.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.109+0000 7f8a28b9e740 10 client.16869 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:14:17.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.109+0000 7f8a28b9e740 10 client.16869 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:17.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.109+0000 7f8a28b9e740 20 client.16869 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:17.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.109+0000 7f8a28b9e740 1 --2- 172.21.15.44:0/1898887562 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55ab1d38afc0 0x55ab1d38d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.109+0000 7f8a28b9e740 4 client.16869 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:14:17.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.109+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1898887562 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ab1d276dc0 con 0x55ab1d38afc0 2024-04-21T23:14:17.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.109+0000 7f8a1f7fe640 1 --2- 172.21.15.44:0/1898887562 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55ab1d38afc0 0x55ab1d38d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.109+0000 7f8a1f7fe640 1 --2- 172.21.15.44:0/1898887562 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55ab1d38afc0 0x55ab1d38d3e0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d85a65c6cc4d2c4 server_cookie=5905d6db38a3cfdd in_seq=0 out_seq=0 2024-04-21T23:14:17.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.109+0000 7f89feffd640 10 client.16869 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:17.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.111+0000 7f89feffd640 1 -- 172.21.15.44:0/1898887562 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ab1d276dc0 con 0x55ab1d38afc0 2024-04-21T23:14:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.111+0000 7f89feffd640 10 client.16869 handle_command_reply: tid=0 2024-04-21T23:14:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.111+0000 7f8a28b9e740 1 client.16869 shutdown 2024-04-21T23:14:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.111+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1898887562 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d314520 msgr2=0x55ab1d327cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.111+0000 7f8a28b9e740 1 --2- 172.21.15.44:0/1898887562 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d314520 0x55ab1d327cb0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f8a18002230 tx=0x7f8a18009500 comp rx=0 tx=0).stop 2024-04-21T23:14:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.111+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1898887562 shutdown_connections 2024-04-21T23:14:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.111+0000 7f8a28b9e740 1 --2- 172.21.15.44:0/1898887562 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55ab1d38afc0 0x55ab1d38d3e0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.111+0000 7f8a28b9e740 1 --2- 172.21.15.44:0/1898887562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ab1d30b940 0x55ab1d327770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.111+0000 7f8a28b9e740 1 --2- 172.21.15.44:0/1898887562 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55ab1d314520 0x55ab1d327cb0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.111+0000 7f89feffd640 0 client.16869 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.111+0000 7f8a28b9e740 1 --2- 172.21.15.44:0/1898887562 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55ab1d30b020 0x55ab1d327230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.111+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1898887562 >> 172.21.15.44:0/1898887562 conn(0x55ab1d305750 msgr2=0x55ab1d303fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:17.117 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.111+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1898887562 shutdown_connections 2024-04-21T23:14:17.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.111+0000 7f8a28b9e740 1 -- 172.21.15.44:0/1898887562 wait complete. 2024-04-21T23:14:17.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.111+0000 7f8a28b9e740 20 client.16869 trim_cache size 0 max 16384 2024-04-21T23:14:17.118 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-21T23:14:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.112+0000 7f8a25ed4640 1 -- 172.21.15.44:0/601521935 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f8a080a2d00 msgr2=0x7f8a080a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.112+0000 7f8a25ed4640 1 --2- 172.21.15.44:0/601521935 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f8a080a2d00 0x7f8a080a51c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8a100096c0 tx=0x7f8a10009260 comp rx=0 tx=0).stop 2024-04-21T23:14:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.112+0000 7f8a25ed4640 1 -- 172.21.15.44:0/601521935 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a20100a30 msgr2=0x7f8a201985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.112+0000 7f8a25ed4640 1 --2- 172.21.15.44:0/601521935 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a20100a30 0x7f8a201985f0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400e9f0 tx=0x7f8a1400eeb0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.112+0000 7f8a25ed4640 1 -- 172.21.15.44:0/601521935 shutdown_connections 2024-04-21T23:14:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.112+0000 7f8a25ed4640 1 --2- 172.21.15.44:0/601521935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a20106870 0x7f8a2019c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.112+0000 7f8a25ed4640 1 --2- 172.21.15.44:0/601521935 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f8a080a2d00 0x7f8a080a51c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.112+0000 7f8a25ed4640 1 --2- 172.21.15.44:0/601521935 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f8a20100a30 0x7f8a201985f0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.112+0000 7f8a25ed4640 1 --2- 172.21.15.44:0/601521935 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8a20100070 0x7f8a201980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.112+0000 7f8a25ed4640 1 -- 172.21.15.44:0/601521935 >> 172.21.15.44:0/601521935 conn(0x7f8a200758c0 msgr2=0x7f8a201046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.112+0000 7f8a25ed4640 1 -- 172.21.15.44:0/601521935 shutdown_connections 2024-04-21T23:14:17.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:17.112+0000 7f8a25ed4640 1 -- 172.21.15.44:0/601521935 wait complete. 2024-04-21T23:14:17.639 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.639+0000 7fd7f78c9640 1 -- 172.21.15.44:0/3023387446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f01032d0 msgr2=0x7fd7f010b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:17.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.639+0000 7fd7f78c9640 1 --2- 172.21.15.44:0/3023387446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f01032d0 0x7fd7f010b7b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0009900 tx=0x7fd7e002bcd0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.640+0000 7fd7f78c9640 1 -- 172.21.15.44:0/3023387446 shutdown_connections 2024-04-21T23:14:17.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.640+0000 7fd7f78c9640 1 --2- 172.21.15.44:0/3023387446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f01032d0 0x7fd7f010b7b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.640+0000 7fd7f78c9640 1 --2- 172.21.15.44:0/3023387446 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f010bd80 0x7fd7f010e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.640+0000 7fd7f78c9640 1 --2- 172.21.15.44:0/3023387446 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7f01029b0 0x7fd7f0102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.640+0000 7fd7f78c9640 1 -- 172.21.15.44:0/3023387446 >> 172.21.15.44:0/3023387446 conn(0x7fd7f0077760 msgr2=0x7fd7f0077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:17.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.640+0000 7fd7f78c9640 1 -- 172.21.15.44:0/3023387446 shutdown_connections 2024-04-21T23:14:17.640 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.640+0000 7fd7f78c9640 1 -- 172.21.15.44:0/3023387446 wait complete. 2024-04-21T23:14:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.640+0000 7fd7f78c9640 1 Processor -- start 2024-04-21T23:14:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.640+0000 7fd7f78c9640 1 -- start start 2024-04-21T23:14:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.640+0000 7fd7f78c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f01029b0 0x7fd7f019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.640+0000 7fd7f78c9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7f01032d0 0x7fd7f019a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.640+0000 7fd7f78c9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f010bd80 0x7fd7f019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.640+0000 7fd7f78c9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f0111b10 con 0x7fd7f010bd80 2024-04-21T23:14:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.640+0000 7fd7f78c9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f0111990 con 0x7fd7f01032d0 2024-04-21T23:14:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.640+0000 7fd7f78c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f0111c90 con 0x7fd7f01029b0 2024-04-21T23:14:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.641+0000 7fd7f5e66640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f010bd80 0x7fd7f019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.641+0000 7fd7f5e66640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f010bd80 0x7fd7f019eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43764/0 (socket says 172.21.15.44:43764) 2024-04-21T23:14:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.641+0000 7fd7f5e66640 1 -- 172.21.15.44:0/3697143691 learned_addr learned my addr 172.21.15.44:0/3697143691 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.641+0000 7fd7f4e64640 1 --2- 172.21.15.44:0/3697143691 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7f01032d0 0x7fd7f019a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.641+0000 7fd7f5665640 1 --2- 172.21.15.44:0/3697143691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f01029b0 0x7fd7f019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.641+0000 7fd7f5e66640 1 -- 172.21.15.44:0/3697143691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f01029b0 msgr2=0x7fd7f019a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.641+0000 7fd7f5e66640 1 --2- 172.21.15.44:0/3697143691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f01029b0 0x7fd7f019a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.641+0000 7fd7f5e66640 1 -- 172.21.15.44:0/3697143691 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7f01032d0 msgr2=0x7fd7f019a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.641+0000 7fd7f5e66640 1 --2- 172.21.15.44:0/3697143691 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7f01032d0 0x7fd7f019a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.641+0000 7fd7f5e66640 1 -- 172.21.15.44:0/3697143691 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f019f280 con 0x7fd7f010bd80 2024-04-21T23:14:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.641+0000 7fd7f5e66640 1 --2- 172.21.15.44:0/3697143691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f010bd80 0x7fd7f019eb50 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8009c90 tx=0x7fd7e800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.641+0000 7fd7e67fc640 1 -- 172.21.15.44:0/3697143691 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8003b70 con 0x7fd7f010bd80 2024-04-21T23:14:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.641+0000 7fd7f78c9640 1 -- 172.21.15.44:0/3697143691 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7f019f570 con 0x7fd7f010bd80 2024-04-21T23:14:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.641+0000 7fd7e67fc640 1 -- 172.21.15.44:0/3697143691 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7e8003d10 con 0x7fd7f010bd80 2024-04-21T23:14:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.641+0000 7fd7e67fc640 1 -- 172.21.15.44:0/3697143691 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8010960 con 0x7fd7f010bd80 2024-04-21T23:14:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.641+0000 7fd7f78c9640 1 -- 172.21.15.44:0/3697143691 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7f01a6e00 con 0x7fd7f010bd80 2024-04-21T23:14:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.644+0000 7fd7e67fc640 1 -- 172.21.15.44:0/3697143691 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd7e8004020 con 0x7fd7f010bd80 2024-04-21T23:14:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.644+0000 7fd7fa58e740 1 Processor -- start 2024-04-21T23:14:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.644+0000 7fd7fa58e740 1 -- start start 2024-04-21T23:14:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.644+0000 7fd7fa58e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5608185b8b20 0x5608185b8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.644+0000 7fd7fa58e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 0x5608185c1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.644+0000 7fd7fa58e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5608185c2210 0x5608185c4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.644+0000 7fd7fa58e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56081851cb70 con 0x5608185b9440 2024-04-21T23:14:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.644+0000 7fd7fa58e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5608184c5470 con 0x5608185b8b20 2024-04-21T23:14:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.644+0000 7fd7fa58e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560818518b30 con 0x5608185c2210 2024-04-21T23:14:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.644+0000 7fd7f5665640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 0x5608185c1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.644+0000 7fd7f4e64640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5608185b8b20 0x5608185b8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.644+0000 7fd7f5665640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 0x5608185c1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43768/0 (socket says 172.21.15.44:43768) 2024-04-21T23:14:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.644+0000 7fd7f5665640 1 -- 172.21.15.44:0/3454603008 learned_addr learned my addr 172.21.15.44:0/3454603008 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.645+0000 7fd7f4e64640 1 --2- 172.21.15.44:0/3454603008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5608185c2210 0x5608185c4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.645+0000 7fd7e67fc640 1 --2- 172.21.15.44:0/3697143691 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd7dc0a2cf0 0x7fd7dc0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.645+0000 7fd7f5665640 1 -- 172.21.15.44:0/3454603008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5608185c2210 msgr2=0x5608185c4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.645+0000 7fd7f5665640 1 --2- 172.21.15.44:0/3454603008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5608185c2210 0x5608185c4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.645+0000 7fd7f5665640 1 -- 172.21.15.44:0/3454603008 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5608185b8b20 msgr2=0x5608185b8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.645+0000 7fd7f5665640 1 --2- 172.21.15.44:0/3454603008 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5608185b8b20 0x5608185b8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.645+0000 7fd7f5665640 1 -- 172.21.15.44:0/3454603008 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560818231210 con 0x5608185b9440 2024-04-21T23:14:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.645+0000 7fd7f5e66640 1 --2- 172.21.15.44:0/3697143691 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd7dc0a2cf0 0x7fd7dc0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.645+0000 7fd7e67fc640 1 -- 172.21.15.44:0/3697143691 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fd7e80cfd60 con 0x7fd7f010bd80 2024-04-21T23:14:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.645+0000 7fd7f5665640 1 --2- 172.21.15.44:0/3454603008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 0x5608185c1cd0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec009cf0 tx=0x7fd7ec011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8882818a3e056775 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.645+0000 7fd7db7fe640 1 -- 172.21.15.44:0/3454603008 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec004230 con 0x5608185b9440 2024-04-21T23:14:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.645+0000 7fd7db7fe640 1 -- 172.21.15.44:0/3454603008 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7ec0043d0 con 0x5608185b9440 2024-04-21T23:14:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.645+0000 7fd7db7fe640 1 -- 172.21.15.44:0/3454603008 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec019900 con 0x5608185b9440 2024-04-21T23:14:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.646+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3454603008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 msgr2=0x5608185c1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.646+0000 7fd7f5e66640 1 --2- 172.21.15.44:0/3697143691 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd7dc0a2cf0 0x7fd7dc0a51b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd7e80cd1c0 tx=0x7fd7e8010430 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.646+0000 7fd7fa58e740 1 --2- 172.21.15.44:0/3454603008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 0x5608185c1cd0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec009cf0 tx=0x7fd7ec011810 comp rx=0 tx=0).stop 2024-04-21T23:14:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.646+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3454603008 shutdown_connections 2024-04-21T23:14:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.646+0000 7fd7fa58e740 1 --2- 172.21.15.44:0/3454603008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5608185c2210 0x5608185c4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.646+0000 7fd7fa58e740 1 --2- 172.21.15.44:0/3454603008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 0x5608185c1cd0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.646+0000 7fd7fa58e740 1 --2- 172.21.15.44:0/3454603008 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5608185b8b20 0x5608185b8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.646+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3454603008 >> 172.21.15.44:0/3454603008 conn(0x5608185b3440 msgr2=0x5608185b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.646+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3454603008 shutdown_connections 2024-04-21T23:14:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.646+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3454603008 wait complete. 2024-04-21T23:14:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.646+0000 7fd7fa58e740 1 -- start start 2024-04-21T23:14:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.646+0000 7fd7fa58e740 1 Processor -- start 2024-04-21T23:14:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.647+0000 7fd7fa58e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b8b20 0x5608185bf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.647+0000 7fd7fa58e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5608185b9440 0x5608185bf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.647+0000 7fd7fa58e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5608185c2210 0x5608185bfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.647+0000 7fd7fa58e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5608185c4e40 con 0x5608185b8b20 2024-04-21T23:14:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.647+0000 7fd7fa58e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5608184d8a10 con 0x5608185c2210 2024-04-21T23:14:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.647+0000 7fd7fa58e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5608185c4fc0 con 0x5608185b9440 2024-04-21T23:14:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.647+0000 7fd7fa58e740 10 client.0 _ll_register_callbacks cb 0x560818513b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.647+0000 7fd7f4e64640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b8b20 0x5608185bf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.647+0000 7fd7f4e64640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b8b20 0x5608185bf190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43778/0 (socket says 172.21.15.44:43778) 2024-04-21T23:14:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.647+0000 7fd7f4e64640 1 -- 172.21.15.44:0/3396655813 learned_addr learned my addr 172.21.15.44:0/3396655813 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.647+0000 7fd7f5665640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5608185b9440 0x5608185bf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.647+0000 7fd7f4e64640 1 --2- 172.21.15.44:0/3396655813 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5608185c2210 0x5608185bfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.647+0000 7fd7f4e64640 1 -- 172.21.15.44:0/3396655813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5608185b9440 msgr2=0x5608185bf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.647+0000 7fd7f4e64640 1 --2- 172.21.15.44:0/3396655813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5608185b9440 0x5608185bf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.647+0000 7fd7f4e64640 1 -- 172.21.15.44:0/3396655813 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5608185c2210 msgr2=0x5608185bfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.647+0000 7fd7f4e64640 1 --2- 172.21.15.44:0/3396655813 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5608185c2210 0x5608185bfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.647+0000 7fd7f4e64640 1 -- 172.21.15.44:0/3396655813 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608182fd620 con 0x5608185b8b20 2024-04-21T23:14:17.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.647+0000 7fd7f4e64640 1 --2- 172.21.15.44:0/3396655813 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b8b20 0x5608185bf190 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0005ec0 tx=0x7fd7e0005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73baf3621e973f48 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:17.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.647+0000 7fd7d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:14:17.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.648+0000 7fd7d8ff9640 1 -- 172.21.15.44:0/3396655813 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e00046c0 con 0x5608185b8b20 2024-04-21T23:14:17.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.648+0000 7fd7d8ff9640 1 -- 172.21.15.44:0/3396655813 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fd7e0004860 con 0x5608185b8b20 2024-04-21T23:14:17.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.648+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3396655813 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7ec0043d0 con 0x5608185b8b20 2024-04-21T23:14:17.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.648+0000 7fd7d8ff9640 1 -- 172.21.15.44:0/3396655813 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd7e0055050 con 0x5608185b8b20 2024-04-21T23:14:17.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.648+0000 7fd7fa58e740 10 client.16881 fetch_fsmap learned FSMap version 25 2024-04-21T23:14:17.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.648+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3396655813 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5608185d0fe0 con 0x5608185b8b20 2024-04-21T23:14:17.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.648+0000 7fd7d8ff9640 1 -- 172.21.15.44:0/3396655813 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7ec0043d0 con 0x5608185b8b20 2024-04-21T23:14:17.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.648+0000 7fd7d8ff9640 1 -- 172.21.15.44:0/3396655813 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fd7e0003660 con 0x5608185b8b20 2024-04-21T23:14:17.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.648+0000 7fd7fa58e740 10 client.16881 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:14:17.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.648+0000 7fd7fa58e740 10 client.16881 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:17.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.648+0000 7fd7fa58e740 20 client.16881 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:17.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.648+0000 7fd7fa58e740 1 --2- 172.21.15.44:0/3396655813 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5608185d2450 0x5608185d4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.654 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.648+0000 7fd7fa58e740 4 client.16881 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:14:17.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.648+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3396655813 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56081852a0b0 con 0x5608185d2450 2024-04-21T23:14:17.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.648+0000 7fd7f5665640 1 --2- 172.21.15.44:0/3396655813 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5608185d2450 0x5608185d4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.649+0000 7fd7d8ff9640 10 client.16881 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:17.655 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.649+0000 7fd7f5665640 1 --2- 172.21.15.44:0/3396655813 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5608185d2450 0x5608185d4840 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=297c9153bc606a7d server_cookie=8dea93738f451700 in_seq=0 out_seq=0 2024-04-21T23:14:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.651+0000 7fd7d8ff9640 1 -- 172.21.15.44:0/3396655813 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56081852a0b0 con 0x5608185d2450 2024-04-21T23:14:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.651+0000 7fd7d8ff9640 10 client.16881 handle_command_reply: tid=0 2024-04-21T23:14:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.651+0000 7fd7fa58e740 1 client.16881 shutdown 2024-04-21T23:14:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.651+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3396655813 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b8b20 msgr2=0x5608185bf190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.651+0000 7fd7fa58e740 1 --2- 172.21.15.44:0/3396655813 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b8b20 0x5608185bf190 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0005ec0 tx=0x7fd7e0005b50 comp rx=0 tx=0).stop 2024-04-21T23:14:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.651+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3396655813 shutdown_connections 2024-04-21T23:14:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.651+0000 7fd7fa58e740 1 --2- 172.21.15.44:0/3396655813 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5608185d2450 0x5608185d4840 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.651+0000 7fd7fa58e740 1 --2- 172.21.15.44:0/3396655813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5608185b9440 0x5608185bf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.651+0000 7fd7fa58e740 1 --2- 172.21.15.44:0/3396655813 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5608185c2210 0x5608185bfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.651+0000 7fd7d8ff9640 0 client.16881 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.651+0000 7fd7fa58e740 1 --2- 172.21.15.44:0/3396655813 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b8b20 0x5608185bf190 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.651+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3396655813 >> 172.21.15.44:0/3396655813 conn(0x5608185b3440 msgr2=0x5608185b5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.651+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3396655813 shutdown_connections 2024-04-21T23:14:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.651+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3396655813 wait complete. 2024-04-21T23:14:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.651+0000 7fd7fa58e740 20 client.16881 trim_cache size 0 max 16384 2024-04-21T23:14:17.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.667+0000 7fd7fa58e740 1 Processor -- start 2024-04-21T23:14:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.667+0000 7fd7fa58e740 1 -- start start 2024-04-21T23:14:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.667+0000 7fd7fa58e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5608185b3440 0x5608185c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.667+0000 7fd7fa58e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5608185b8b20 0x5608185cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.667+0000 7fd7fa58e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 0x5608185da110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.667+0000 7fd7fa58e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f01aeea0 con 0x5608185b9440 2024-04-21T23:14:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.667+0000 7fd7fa58e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f01b1720 con 0x5608185b3440 2024-04-21T23:14:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.667+0000 7fd7fa58e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f01af510 con 0x5608185b8b20 2024-04-21T23:14:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.667+0000 7fd7f4e64640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 0x5608185da110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.667+0000 7fd7f4e64640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 0x5608185da110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43792/0 (socket says 172.21.15.44:43792) 2024-04-21T23:14:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.667+0000 7fd7f4e64640 1 -- 172.21.15.44:0/1579242802 learned_addr learned my addr 172.21.15.44:0/1579242802 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.667+0000 7fd7f5665640 1 --2- 172.21.15.44:0/1579242802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5608185b8b20 0x5608185cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.667+0000 7fd7f4e64640 1 --2- 172.21.15.44:0/1579242802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5608185b3440 0x5608185c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.667+0000 7fd7f4e64640 1 -- 172.21.15.44:0/1579242802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5608185b8b20 msgr2=0x5608185cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.667+0000 7fd7f4e64640 1 --2- 172.21.15.44:0/1579242802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5608185b8b20 0x5608185cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.667+0000 7fd7f4e64640 1 -- 172.21.15.44:0/1579242802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5608185b3440 msgr2=0x5608185c18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.667+0000 7fd7f4e64640 1 --2- 172.21.15.44:0/1579242802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5608185b3440 0x5608185c18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.667+0000 7fd7f4e64640 1 -- 172.21.15.44:0/1579242802 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608185be160 con 0x5608185b9440 2024-04-21T23:14:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.668+0000 7fd7f4e64640 1 --2- 172.21.15.44:0/1579242802 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 0x5608185da110 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0004010 tx=0x7fd7e002c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6a021f33bad3a00 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.668+0000 7fd7daffd640 1 -- 172.21.15.44:0/1579242802 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e0043040 con 0x5608185b9440 2024-04-21T23:14:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.668+0000 7fd7daffd640 1 -- 172.21.15.44:0/1579242802 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7e0003c00 con 0x5608185b9440 2024-04-21T23:14:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.668+0000 7fd7daffd640 1 -- 172.21.15.44:0/1579242802 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e0005170 con 0x5608185b9440 2024-04-21T23:14:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.668+0000 7fd7fa58e740 1 -- 172.21.15.44:0/1579242802 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 msgr2=0x5608185da110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.668+0000 7fd7fa58e740 1 --2- 172.21.15.44:0/1579242802 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 0x5608185da110 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0004010 tx=0x7fd7e002c670 comp rx=0 tx=0).stop 2024-04-21T23:14:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.668+0000 7fd7fa58e740 1 -- 172.21.15.44:0/1579242802 shutdown_connections 2024-04-21T23:14:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.668+0000 7fd7fa58e740 1 --2- 172.21.15.44:0/1579242802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5608185b8b20 0x5608185cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.668+0000 7fd7fa58e740 1 --2- 172.21.15.44:0/1579242802 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 0x5608185da110 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.668+0000 7fd7fa58e740 1 --2- 172.21.15.44:0/1579242802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5608185b3440 0x5608185c18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.668+0000 7fd7fa58e740 1 -- 172.21.15.44:0/1579242802 >> 172.21.15.44:0/1579242802 conn(0x5608185ae590 msgr2=0x5608185bf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.668+0000 7fd7fa58e740 1 -- 172.21.15.44:0/1579242802 shutdown_connections 2024-04-21T23:14:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.668+0000 7fd7fa58e740 1 -- 172.21.15.44:0/1579242802 wait complete. 2024-04-21T23:14:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.669+0000 7fd7fa58e740 1 -- start start 2024-04-21T23:14:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.669+0000 7fd7fa58e740 1 Processor -- start 2024-04-21T23:14:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.669+0000 7fd7fa58e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5608185b3440 0x5608185ca210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.669+0000 7fd7fa58e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5608185b8b20 0x5608185ca750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.669+0000 7fd7fa58e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 0x5608185c71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.669+0000 7fd7fa58e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f01afbe0 con 0x5608185b9440 2024-04-21T23:14:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.669+0000 7fd7fa58e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f01b1050 con 0x5608185b8b20 2024-04-21T23:14:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.669+0000 7fd7fa58e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f01b02b0 con 0x5608185b3440 2024-04-21T23:14:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.669+0000 7fd7fa58e740 10 client.0 _ll_register_callbacks cb 0x7fd7e0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.669+0000 7fd7f4e64640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 0x5608185c71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.669+0000 7fd7f5665640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5608185b8b20 0x5608185ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.669+0000 7fd7f4e64640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 0x5608185c71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43806/0 (socket says 172.21.15.44:43806) 2024-04-21T23:14:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.669+0000 7fd7f4e64640 1 -- 172.21.15.44:0/3777550170 learned_addr learned my addr 172.21.15.44:0/3777550170 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.669+0000 7fd7f4e64640 1 --2- 172.21.15.44:0/3777550170 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5608185b3440 0x5608185ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.669+0000 7fd7f4e64640 1 -- 172.21.15.44:0/3777550170 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5608185b3440 msgr2=0x5608185ca210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.670+0000 7fd7f4e64640 1 --2- 172.21.15.44:0/3777550170 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5608185b3440 0x5608185ca210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.670+0000 7fd7f4e64640 1 -- 172.21.15.44:0/3777550170 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5608185b8b20 msgr2=0x5608185ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.670+0000 7fd7f4e64640 1 --2- 172.21.15.44:0/3777550170 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5608185b8b20 0x5608185ca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.670+0000 7fd7f4e64640 1 -- 172.21.15.44:0/3777550170 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608185c79e0 con 0x5608185b9440 2024-04-21T23:14:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.670+0000 7fd7f4e64640 1 --2- 172.21.15.44:0/3777550170 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 0x5608185c71e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd7e00059d0 tx=0x7fd7e00037a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87ded0fe4b5cba55 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.670+0000 7fd7d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:14:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.670+0000 7fd7d8ff9640 1 -- 172.21.15.44:0/3777550170 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e0043040 con 0x5608185b9440 2024-04-21T23:14:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.670+0000 7fd7d8ff9640 1 -- 172.21.15.44:0/3777550170 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7fd7e0004ad0 con 0x5608185b9440 2024-04-21T23:14:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.670+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3777550170 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7e0003c00 con 0x5608185b9440 2024-04-21T23:14:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.670+0000 7fd7d8ff9640 1 -- 172.21.15.44:0/3777550170 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd7e0004cd0 con 0x5608185b9440 2024-04-21T23:14:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.670+0000 7fd7fa58e740 10 client.16887 fetch_fsmap learned FSMap version 25 2024-04-21T23:14:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.670+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3777550170 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5608186393a0 con 0x5608185b9440 2024-04-21T23:14:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.670+0000 7fd7d8ff9640 1 -- 172.21.15.44:0/3777550170 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7e0003c00 con 0x5608185b9440 2024-04-21T23:14:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.670+0000 7fd7d8ff9640 1 -- 172.21.15.44:0/3777550170 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7fd7e0034560 con 0x5608185b9440 2024-04-21T23:14:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.670+0000 7fd7fa58e740 10 client.16887 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:14:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.670+0000 7fd7fa58e740 10 client.16887 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.670+0000 7fd7fa58e740 20 client.16887 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.670+0000 7fd7fa58e740 1 --2- 172.21.15.44:0/3777550170 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5608185c2210 0x56081863b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.670+0000 7fd7fa58e740 4 client.16887 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:14:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.670+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3777550170 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560818524400 con 0x5608185c2210 2024-04-21T23:14:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.671+0000 7fd7f5665640 1 --2- 172.21.15.44:0/3777550170 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5608185c2210 0x56081863b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.671+0000 7fd7f5665640 1 --2- 172.21.15.44:0/3777550170 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5608185c2210 0x56081863b9a0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75a2e30e5b47ce9a server_cookie=1c37877a92617927 in_seq=0 out_seq=0 2024-04-21T23:14:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:17.671+0000 7fd7d8ff9640 10 client.16887 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:17.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 624727 13.37 MB/sec execute 127 sec latency 24.785 ms 2024-04-21T23:14:18.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.110+0000 7fd7d8ff9640 1 -- 172.21.15.44:0/3777550170 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560818524400 con 0x5608185c2210 2024-04-21T23:14:18.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.110+0000 7fd7d8ff9640 10 client.16887 handle_command_reply: tid=0 2024-04-21T23:14:18.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.110+0000 7fd7fa58e740 1 client.16887 shutdown 2024-04-21T23:14:18.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.110+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3777550170 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 msgr2=0x5608185c71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:18.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.110+0000 7fd7fa58e740 1 --2- 172.21.15.44:0/3777550170 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 0x5608185c71e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd7e00059d0 tx=0x7fd7e00037a0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.111 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.110+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3777550170 shutdown_connections 2024-04-21T23:14:18.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.110+0000 7fd7fa58e740 1 --2- 172.21.15.44:0/3777550170 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5608185c2210 0x56081863b9a0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.110+0000 7fd7fa58e740 1 --2- 172.21.15.44:0/3777550170 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5608185b9440 0x5608185c71e0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.110+0000 7fd7fa58e740 1 --2- 172.21.15.44:0/3777550170 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5608185b8b20 0x5608185ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.110+0000 7fd7d8ff9640 0 client.16887 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:18.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.110+0000 7fd7fa58e740 1 --2- 172.21.15.44:0/3777550170 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5608185b3440 0x5608185ca210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.110+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3777550170 >> 172.21.15.44:0/3777550170 conn(0x5608185ae590 msgr2=0x5608185b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:18.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.110+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3777550170 shutdown_connections 2024-04-21T23:14:18.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.110+0000 7fd7fa58e740 1 -- 172.21.15.44:0/3777550170 wait complete. 2024-04-21T23:14:18.112 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.110+0000 7fd7fa58e740 20 client.16887 trim_cache size 0 max 16384 2024-04-21T23:14:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:14:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:14:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:14:18.115 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:14:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.112+0000 7fd7f78c9640 1 -- 172.21.15.44:0/3697143691 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd7dc0a2cf0 msgr2=0x7fd7dc0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.112+0000 7fd7f78c9640 1 --2- 172.21.15.44:0/3697143691 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd7dc0a2cf0 0x7fd7dc0a51b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd7e80cd1c0 tx=0x7fd7e8010430 comp rx=0 tx=0).stop 2024-04-21T23:14:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.112+0000 7fd7f78c9640 1 -- 172.21.15.44:0/3697143691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f010bd80 msgr2=0x7fd7f019eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.112+0000 7fd7f78c9640 1 --2- 172.21.15.44:0/3697143691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f010bd80 0x7fd7f019eb50 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8009c90 tx=0x7fd7e800c6a0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.116 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.112+0000 7fd7f78c9640 1 -- 172.21.15.44:0/3697143691 shutdown_connections 2024-04-21T23:14:18.117 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.112+0000 7fd7f78c9640 1 --2- 172.21.15.44:0/3697143691 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd7dc0a2cf0 0x7fd7dc0a51b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.117 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.112+0000 7fd7f78c9640 1 --2- 172.21.15.44:0/3697143691 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd7f010bd80 0x7fd7f019eb50 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.117 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.112+0000 7fd7f78c9640 1 --2- 172.21.15.44:0/3697143691 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd7f01032d0 0x7fd7f019a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.117 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.112+0000 7fd7f78c9640 1 --2- 172.21.15.44:0/3697143691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f01029b0 0x7fd7f019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.117 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.112+0000 7fd7f78c9640 1 -- 172.21.15.44:0/3697143691 >> 172.21.15.44:0/3697143691 conn(0x7fd7f0077760 msgr2=0x7fd7f0107c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:18.117 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.114+0000 7fd7f78c9640 1 -- 172.21.15.44:0/3697143691 shutdown_connections 2024-04-21T23:14:18.117 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:18.114+0000 7fd7f78c9640 1 -- 172.21.15.44:0/3697143691 wait complete. 2024-04-21T23:14:18.124 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:14:18.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:14:18.125 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:14:18.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:17 smithi055 ceph-mon[31117]: pgmap v168: 129 pgs: 129 active+clean; 84 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.2 MiB/s wr, 512 op/s 2024-04-21T23:14:18.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:17 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3675579384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:14:18.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:17 smithi069 ceph-mon[31816]: pgmap v168: 129 pgs: 129 active+clean; 84 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.2 MiB/s wr, 512 op/s 2024-04-21T23:14:18.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:17 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3675579384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:14:18.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.235+0000 7f04e085f640 1 -- 172.21.15.44:0/2336319294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04dc102120 msgr2=0x7f04dc102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:18.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.235+0000 7f04e085f640 1 --2- 172.21.15.44:0/2336319294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04dc102120 0x7f04dc102580 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f04c4009900 tx=0x7f04c402bd10 comp rx=0 tx=0).stop 2024-04-21T23:14:18.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.235+0000 7f04e085f640 1 -- 172.21.15.44:0/2336319294 shutdown_connections 2024-04-21T23:14:18.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.235+0000 7f04e085f640 1 --2- 172.21.15.44:0/2336319294 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04dc102ac0 0x7f04dc10d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.235+0000 7f04e085f640 1 --2- 172.21.15.44:0/2336319294 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04dc102120 0x7f04dc102580 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.235+0000 7f04e085f640 1 --2- 172.21.15.44:0/2336319294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04dc1009e0 0x7f04dc100dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.235+0000 7f04e085f640 1 -- 172.21.15.44:0/2336319294 >> 172.21.15.44:0/2336319294 conn(0x7f04dc077760 msgr2=0x7f04dc077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:18.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.235+0000 7f04e085f640 1 -- 172.21.15.44:0/2336319294 shutdown_connections 2024-04-21T23:14:18.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.235+0000 7f04e085f640 1 -- 172.21.15.44:0/2336319294 wait complete. 2024-04-21T23:14:18.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.236+0000 7f04e085f640 1 Processor -- start 2024-04-21T23:14:18.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.236+0000 7f04e085f640 1 -- start start 2024-04-21T23:14:18.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.236+0000 7f04e085f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04dc1009e0 0x7f04dc1981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.236+0000 7f04e085f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04dc102120 0x7f04dc198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.236+0000 7f04e085f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04dc102ac0 0x7f04dc19caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.236+0000 7f04e085f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f04dc111c20 con 0x7f04dc1009e0 2024-04-21T23:14:18.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.236+0000 7f04e085f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f04dc111aa0 con 0x7f04dc102120 2024-04-21T23:14:18.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.236+0000 7f04e085f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f04dc111da0 con 0x7f04dc102ac0 2024-04-21T23:14:18.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.236+0000 7f04da59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04dc1009e0 0x7f04dc1981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.236+0000 7f04da59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04dc1009e0 0x7f04dc1981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43820/0 (socket says 172.21.15.44:43820) 2024-04-21T23:14:18.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.236+0000 7f04da59c640 1 -- 172.21.15.44:0/1249156914 learned_addr learned my addr 172.21.15.44:0/1249156914 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:18.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.236+0000 7f04d3fff640 1 --2- 172.21.15.44:0/1249156914 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04dc102120 0x7f04dc198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.237+0000 7f04dad9d640 1 --2- 172.21.15.44:0/1249156914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04dc102ac0 0x7f04dc19caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.237+0000 7f04da59c640 1 -- 172.21.15.44:0/1249156914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04dc102ac0 msgr2=0x7f04dc19caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:18.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.237+0000 7f04da59c640 1 --2- 172.21.15.44:0/1249156914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04dc102ac0 0x7f04dc19caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.237+0000 7f04da59c640 1 -- 172.21.15.44:0/1249156914 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04dc102120 msgr2=0x7f04dc198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:18.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.237+0000 7f04da59c640 1 --2- 172.21.15.44:0/1249156914 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04dc102120 0x7f04dc198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.237+0000 7f04da59c640 1 -- 172.21.15.44:0/1249156914 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04dc19d1d0 con 0x7f04dc1009e0 2024-04-21T23:14:18.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.237+0000 7f04dad9d640 1 --2- 172.21.15.44:0/1249156914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04dc102ac0 0x7f04dc19caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:14:18.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.237+0000 7f04da59c640 1 --2- 172.21.15.44:0/1249156914 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04dc1009e0 0x7f04dc1981e0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f04cc00e9f0 tx=0x7f04cc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:18.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.237+0000 7f04d37fe640 1 -- 172.21.15.44:0/1249156914 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04cc00cd80 con 0x7f04dc1009e0 2024-04-21T23:14:18.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.237+0000 7f04d37fe640 1 -- 172.21.15.44:0/1249156914 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04cc005500 con 0x7f04dc1009e0 2024-04-21T23:14:18.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.237+0000 7f04d37fe640 1 -- 172.21.15.44:0/1249156914 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04cc010690 con 0x7f04dc1009e0 2024-04-21T23:14:18.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.237+0000 7f04e085f640 1 -- 172.21.15.44:0/1249156914 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04dc19d4c0 con 0x7f04dc1009e0 2024-04-21T23:14:18.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.237+0000 7f04e085f640 1 -- 172.21.15.44:0/1249156914 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04dc1a4d50 con 0x7f04dc1009e0 2024-04-21T23:14:18.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.239+0000 7f04e3443740 1 Processor -- start 2024-04-21T23:14:18.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.240+0000 7f04e3443740 1 -- start start 2024-04-21T23:14:18.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.240+0000 7f04d37fe640 1 -- 172.21.15.44:0/1249156914 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f04cc0046c0 con 0x7f04dc1009e0 2024-04-21T23:14:18.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.240+0000 7f04e3443740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9d6720 0x5556bc9d5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.240+0000 7f04e3443740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d5a40 0x5556bc9d5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.240+0000 7f04e3443740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9dedb0 0x5556bc9e11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.240+0000 7f04e3443740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5556bc939170 con 0x5556bc9d5a40 2024-04-21T23:14:18.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.240+0000 7f04e3443740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5556bc8e17c0 con 0x5556bc9d6720 2024-04-21T23:14:18.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.240+0000 7f04e3443740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5556bc935130 con 0x5556bc9dedb0 2024-04-21T23:14:18.246 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.240+0000 7f04dad9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9d6720 0x5556bc9d5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.240+0000 7f04da59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d5a40 0x5556bc9d5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.240+0000 7f04dad9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9d6720 0x5556bc9d5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:53842/0 (socket says 172.21.15.44:53842) 2024-04-21T23:14:18.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.240+0000 7f04dad9d640 1 -- 172.21.15.44:0/1508621626 learned_addr learned my addr 172.21.15.44:0/1508621626 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:18.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.240+0000 7f04d3fff640 1 --2- 172.21.15.44:0/1508621626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9dedb0 0x5556bc9e11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.240+0000 7f04dad9d640 1 -- 172.21.15.44:0/1508621626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9dedb0 msgr2=0x5556bc9e11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:18.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.240+0000 7f04dad9d640 1 --2- 172.21.15.44:0/1508621626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9dedb0 0x5556bc9e11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.240+0000 7f04dad9d640 1 -- 172.21.15.44:0/1508621626 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d5a40 msgr2=0x5556bc9d5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:18.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.240+0000 7f04dad9d640 1 --2- 172.21.15.44:0/1508621626 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d5a40 0x5556bc9d5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.240+0000 7f04dad9d640 1 -- 172.21.15.44:0/1508621626 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556bc8f2e60 con 0x5556bc9d6720 2024-04-21T23:14:18.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.241+0000 7f04dad9d640 1 --2- 172.21.15.44:0/1508621626 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9d6720 0x5556bc9d5470 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f04d4009ee0 tx=0x7f04d400f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=255405cec94302c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:18.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.241+0000 7f04d37fe640 1 --2- 172.21.15.44:0/1249156914 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f04c80a2d00 0x7f04c80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.241+0000 7f04d0ff9640 1 -- 172.21.15.44:0/1508621626 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d4011640 con 0x5556bc9d6720 2024-04-21T23:14:18.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.241+0000 7f04d37fe640 1 -- 172.21.15.44:0/1249156914 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f04cc0cfcc0 con 0x7f04dc1009e0 2024-04-21T23:14:18.247 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.241+0000 7f04d0ff9640 1 -- 172.21.15.44:0/1508621626 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04d4011cc0 con 0x5556bc9d6720 2024-04-21T23:14:18.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.241+0000 7f04da59c640 1 --2- 172.21.15.44:0/1249156914 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f04c80a2d00 0x7f04c80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.241+0000 7f04d0ff9640 1 -- 172.21.15.44:0/1508621626 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d401bde0 con 0x5556bc9d6720 2024-04-21T23:14:18.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.241+0000 7f04e3443740 1 -- 172.21.15.44:0/1508621626 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9d6720 msgr2=0x5556bc9d5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:18.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.241+0000 7f04e3443740 1 --2- 172.21.15.44:0/1508621626 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9d6720 0x5556bc9d5470 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f04d4009ee0 tx=0x7f04d400f730 comp rx=0 tx=0).stop 2024-04-21T23:14:18.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.241+0000 7f04da59c640 1 --2- 172.21.15.44:0/1249156914 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f04c80a2d00 0x7f04c80a51c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f04cc0cd120 tx=0x7f04cc0d3810 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:18.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.241+0000 7f04e3443740 1 -- 172.21.15.44:0/1508621626 shutdown_connections 2024-04-21T23:14:18.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.241+0000 7f04e3443740 1 --2- 172.21.15.44:0/1508621626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9dedb0 0x5556bc9e11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.242+0000 7f04e3443740 1 --2- 172.21.15.44:0/1508621626 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d5a40 0x5556bc9d5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.242+0000 7f04e3443740 1 --2- 172.21.15.44:0/1508621626 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9d6720 0x5556bc9d5470 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.242+0000 7f04e3443740 1 -- 172.21.15.44:0/1508621626 >> 172.21.15.44:0/1508621626 conn(0x5556bc99c4f0 msgr2=0x5556bc9d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:18.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.242+0000 7f04e3443740 1 -- 172.21.15.44:0/1508621626 shutdown_connections 2024-04-21T23:14:18.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.242+0000 7f04e3443740 1 -- 172.21.15.44:0/1508621626 wait complete. 2024-04-21T23:14:18.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.242+0000 7f04e3443740 1 -- start start 2024-04-21T23:14:18.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.242+0000 7f04e3443740 1 Processor -- start 2024-04-21T23:14:18.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.242+0000 7f04e3443740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9d5a40 0x5556bc9dbde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.248 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.242+0000 7f04e3443740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d6720 0x5556bc9dc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.242+0000 7f04e3443740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9dedb0 0x5556bc9dc860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.242+0000 7f04e3443740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5556bc9e1c20 con 0x5556bc9d6720 2024-04-21T23:14:18.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.242+0000 7f04e3443740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5556bc8f4d40 con 0x5556bc9dedb0 2024-04-21T23:14:18.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.242+0000 7f04e3443740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5556bc9e1da0 con 0x5556bc9d5a40 2024-04-21T23:14:18.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.242+0000 7f04e3443740 10 client.0 _ll_register_callbacks cb 0x5556bc930820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:18.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.242+0000 7f04dad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9d5a40 0x5556bc9dbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.242+0000 7f04d3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d6720 0x5556bc9dc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.242+0000 7f04dad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9d5a40 0x5556bc9dbde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:43320/0 (socket says 172.21.15.44:43320) 2024-04-21T23:14:18.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.242+0000 7f04dad9d640 1 -- 172.21.15.44:0/2589114532 learned_addr learned my addr 172.21.15.44:0/2589114532 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:18.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.242+0000 7f04dad9d640 1 --2- 172.21.15.44:0/2589114532 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9dedb0 0x5556bc9dc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.243+0000 7f04d3fff640 1 -- 172.21.15.44:0/2589114532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9d5a40 msgr2=0x5556bc9dbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:18.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.243+0000 7f04d3fff640 1 --2- 172.21.15.44:0/2589114532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9d5a40 0x5556bc9dbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.243+0000 7f04d3fff640 1 -- 172.21.15.44:0/2589114532 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9dedb0 msgr2=0x5556bc9dc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:18.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.243+0000 7f04d3fff640 1 --2- 172.21.15.44:0/2589114532 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9dedb0 0x5556bc9dc860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.243+0000 7f04d3fff640 1 -- 172.21.15.44:0/2589114532 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556bc7157e0 con 0x5556bc9d6720 2024-04-21T23:14:18.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.243+0000 7f04d3fff640 1 --2- 172.21.15.44:0/2589114532 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d6720 0x5556bc9dc320 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f04c4000c00 tx=0x7f04c402c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57642c7f85bb7b98 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:18.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.243+0000 7f04c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:14:18.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.243+0000 7f04c1ffb640 1 -- 172.21.15.44:0/2589114532 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c4044070 con 0x5556bc9d6720 2024-04-21T23:14:18.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.243+0000 7f04c1ffb640 1 -- 172.21.15.44:0/2589114532 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f04c402cb40 con 0x5556bc9d6720 2024-04-21T23:14:18.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.243+0000 7f04e3443740 1 -- 172.21.15.44:0/2589114532 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04d4011cc0 con 0x5556bc9d6720 2024-04-21T23:14:18.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.243+0000 7f04c1ffb640 1 -- 172.21.15.44:0/2589114532 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f04c402cd40 con 0x5556bc9d6720 2024-04-21T23:14:18.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.243+0000 7f04e3443740 10 client.16899 fetch_fsmap learned FSMap version 25 2024-04-21T23:14:18.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.243+0000 7f04e3443740 1 -- 172.21.15.44:0/2589114532 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5556bc775f70 con 0x5556bc9d6720 2024-04-21T23:14:18.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.243+0000 7f04c1ffb640 1 -- 172.21.15.44:0/2589114532 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04d4011cc0 con 0x5556bc9d6720 2024-04-21T23:14:18.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.244+0000 7f04c1ffb640 1 -- 172.21.15.44:0/2589114532 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f04c40053c0 con 0x5556bc9d6720 2024-04-21T23:14:18.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.244+0000 7f04e3443740 10 client.16899 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:14:18.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.244+0000 7f04e3443740 10 client.16899 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:18.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.244+0000 7f04e3443740 20 client.16899 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:18.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.244+0000 7f04e3443740 1 --2- 172.21.15.44:0/2589114532 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5556bc9eeef0 0x5556bc9f12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.244+0000 7f04e3443740 4 client.16899 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:14:18.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.244+0000 7f04e3443740 1 -- 172.21.15.44:0/2589114532 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556bc90d450 con 0x5556bc9eeef0 2024-04-21T23:14:18.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.244+0000 7f04dad9d640 1 --2- 172.21.15.44:0/2589114532 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5556bc9eeef0 0x5556bc9f12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.244+0000 7f04dad9d640 1 --2- 172.21.15.44:0/2589114532 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5556bc9eeef0 0x5556bc9f12e0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92fc685a265e9a54 server_cookie=b8b374c4b8a35a98 in_seq=0 out_seq=0 2024-04-21T23:14:18.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.244+0000 7f04c1ffb640 10 client.16899 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:18.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.245+0000 7f04c1ffb640 1 -- 172.21.15.44:0/2589114532 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5556bc90d450 con 0x5556bc9eeef0 2024-04-21T23:14:18.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.245+0000 7f04c1ffb640 10 client.16899 handle_command_reply: tid=0 2024-04-21T23:14:18.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.245+0000 7f04e3443740 1 client.16899 shutdown 2024-04-21T23:14:18.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.245+0000 7f04e3443740 1 -- 172.21.15.44:0/2589114532 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d6720 msgr2=0x5556bc9dc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:18.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.245+0000 7f04e3443740 1 --2- 172.21.15.44:0/2589114532 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d6720 0x5556bc9dc320 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f04c4000c00 tx=0x7f04c402c880 comp rx=0 tx=0).stop 2024-04-21T23:14:18.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.245+0000 7f04e3443740 1 -- 172.21.15.44:0/2589114532 shutdown_connections 2024-04-21T23:14:18.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.245+0000 7f04e3443740 1 --2- 172.21.15.44:0/2589114532 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5556bc9eeef0 0x5556bc9f12e0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.245+0000 7f04e3443740 1 --2- 172.21.15.44:0/2589114532 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9dedb0 0x5556bc9dc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.245+0000 7f04e3443740 1 --2- 172.21.15.44:0/2589114532 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d6720 0x5556bc9dc320 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.245+0000 7f04c1ffb640 0 client.16899 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:18.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.245+0000 7f04e3443740 1 --2- 172.21.15.44:0/2589114532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9d5a40 0x5556bc9dbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.245+0000 7f04e3443740 1 -- 172.21.15.44:0/2589114532 >> 172.21.15.44:0/2589114532 conn(0x5556bc99c4f0 msgr2=0x5556bc9d21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:18.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.245+0000 7f04e3443740 1 -- 172.21.15.44:0/2589114532 shutdown_connections 2024-04-21T23:14:18.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.245+0000 7f04e3443740 1 -- 172.21.15.44:0/2589114532 wait complete. 2024-04-21T23:14:18.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.246+0000 7f04e3443740 20 client.16899 trim_cache size 0 max 16384 2024-04-21T23:14:18.263 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.264+0000 7f04e3443740 1 Processor -- start 2024-04-21T23:14:18.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.264+0000 7f04e3443740 1 -- start start 2024-04-21T23:14:18.264 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.264+0000 7f04e3443740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d5a40 0x5556bc9eab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.265+0000 7f04e3443740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9d6720 0x5556bc9dc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.265+0000 7f04e3443740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9dedb0 0x5556bc9f6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.265+0000 7f04dad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d5a40 0x5556bc9eab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.265+0000 7f04dad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d5a40 0x5556bc9eab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43848/0 (socket says 172.21.15.44:43848) 2024-04-21T23:14:18.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.265+0000 7f04e3443740 1 -- 172.21.15.44:0/1375446149 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f04dc1aeeb0 con 0x5556bc9d5a40 2024-04-21T23:14:18.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.265+0000 7f04dad9d640 1 -- 172.21.15.44:0/1375446149 learned_addr learned my addr 172.21.15.44:0/1375446149 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:18.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.265+0000 7f04e3443740 1 -- 172.21.15.44:0/1375446149 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f04dc1b1730 con 0x5556bc9dedb0 2024-04-21T23:14:18.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.265+0000 7f04e3443740 1 -- 172.21.15.44:0/1375446149 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f04dc1af520 con 0x5556bc9d6720 2024-04-21T23:14:18.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.265+0000 7f04dad9d640 1 --2- 172.21.15.44:0/1375446149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9dedb0 0x5556bc9f6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.265+0000 7f04d3fff640 1 --2- 172.21.15.44:0/1375446149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9d6720 0x5556bc9dc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.265+0000 7f04dad9d640 1 -- 172.21.15.44:0/1375446149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9d6720 msgr2=0x5556bc9dc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:18.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.265+0000 7f04dad9d640 1 --2- 172.21.15.44:0/1375446149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9d6720 0x5556bc9dc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.265+0000 7f04dad9d640 1 -- 172.21.15.44:0/1375446149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9dedb0 msgr2=0x5556bc9f6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:18.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.265+0000 7f04dad9d640 1 --2- 172.21.15.44:0/1375446149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9dedb0 0x5556bc9f6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.265+0000 7f04dad9d640 1 -- 172.21.15.44:0/1375446149 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556bc60d170 con 0x5556bc9d5a40 2024-04-21T23:14:18.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.265+0000 7f04d3fff640 1 --2- 172.21.15.44:0/1375446149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9d6720 0x5556bc9dc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:14:18.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.265+0000 7f04dad9d640 1 --2- 172.21.15.44:0/1375446149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d5a40 0x5556bc9eab20 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f04d4005fd0 tx=0x7f04d400ffb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39bd0e12eeda8904 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:18.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.265+0000 7f04c3fff640 1 -- 172.21.15.44:0/1375446149 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d401bc50 con 0x5556bc9d5a40 2024-04-21T23:14:18.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.265+0000 7f04c3fff640 1 -- 172.21.15.44:0/1375446149 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04d4023040 con 0x5556bc9d5a40 2024-04-21T23:14:18.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.265+0000 7f04c3fff640 1 -- 172.21.15.44:0/1375446149 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d4005650 con 0x5556bc9d5a40 2024-04-21T23:14:18.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.266+0000 7f04e3443740 1 -- 172.21.15.44:0/1375446149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d5a40 msgr2=0x5556bc9eab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:18.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.266+0000 7f04e3443740 1 --2- 172.21.15.44:0/1375446149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d5a40 0x5556bc9eab20 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f04d4005fd0 tx=0x7f04d400ffb0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.266+0000 7f04e3443740 1 -- 172.21.15.44:0/1375446149 shutdown_connections 2024-04-21T23:14:18.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.266+0000 7f04e3443740 1 --2- 172.21.15.44:0/1375446149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9d6720 0x5556bc9dc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.266+0000 7f04e3443740 1 --2- 172.21.15.44:0/1375446149 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9dedb0 0x5556bc9f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.266+0000 7f04e3443740 1 --2- 172.21.15.44:0/1375446149 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d5a40 0x5556bc9eab20 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.266+0000 7f04e3443740 1 -- 172.21.15.44:0/1375446149 >> 172.21.15.44:0/1375446149 conn(0x5556bc99c4f0 msgr2=0x5556bc9d2170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:18.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.266+0000 7f04e3443740 1 -- 172.21.15.44:0/1375446149 shutdown_connections 2024-04-21T23:14:18.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.266+0000 7f04e3443740 1 -- 172.21.15.44:0/1375446149 wait complete. 2024-04-21T23:14:18.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.266+0000 7f04e3443740 1 -- start start 2024-04-21T23:14:18.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.267+0000 7f04e3443740 1 Processor -- start 2024-04-21T23:14:18.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.267+0000 7f04e3443740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d5a40 0x5556bc9e7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.267+0000 7f04e3443740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9d6720 0x5556bc9d9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.267+0000 7f04e3443740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9dedb0 0x5556bc9d9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.267+0000 7f04e3443740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f04dc1afbf0 con 0x5556bc9d5a40 2024-04-21T23:14:18.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.267+0000 7f04e3443740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f04dc1b1060 con 0x5556bc9dedb0 2024-04-21T23:14:18.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.267+0000 7f04e3443740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f04dc1b02c0 con 0x5556bc9d6720 2024-04-21T23:14:18.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.267+0000 7f04e3443740 10 client.0 _ll_register_callbacks cb 0x7f04c4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:18.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.267+0000 7f04dad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d5a40 0x5556bc9e7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.267+0000 7f04dad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d5a40 0x5556bc9e7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43862/0 (socket says 172.21.15.44:43862) 2024-04-21T23:14:18.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.267+0000 7f04dad9d640 1 -- 172.21.15.44:0/1715916896 learned_addr learned my addr 172.21.15.44:0/1715916896 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:18.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.267+0000 7f04d3fff640 1 --2- 172.21.15.44:0/1715916896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9d6720 0x5556bc9d9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.267+0000 7f04dad9d640 1 --2- 172.21.15.44:0/1715916896 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9dedb0 0x5556bc9d9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.267+0000 7f04dad9d640 1 -- 172.21.15.44:0/1715916896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9d6720 msgr2=0x5556bc9d9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:18.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.267+0000 7f04dad9d640 1 --2- 172.21.15.44:0/1715916896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9d6720 0x5556bc9d9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.267+0000 7f04dad9d640 1 -- 172.21.15.44:0/1715916896 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9dedb0 msgr2=0x5556bc9d9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:18.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.267+0000 7f04dad9d640 1 --2- 172.21.15.44:0/1715916896 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9dedb0 0x5556bc9d9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.267+0000 7f04dad9d640 1 -- 172.21.15.44:0/1715916896 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556bc9dfe60 con 0x5556bc9d5a40 2024-04-21T23:14:18.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.267+0000 7f04dad9d640 1 --2- 172.21.15.44:0/1715916896 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d5a40 0x5556bc9e7500 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f04d401b680 tx=0x7f04d4005b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6629020783e61db4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:18.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.267+0000 7f04c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:14:18.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.268+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1715916896 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d4037040 con 0x5556bc9d5a40 2024-04-21T23:14:18.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.268+0000 7f04e3443740 1 -- 172.21.15.44:0/1715916896 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04d4023040 con 0x5556bc9d5a40 2024-04-21T23:14:18.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.268+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1715916896 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f04d4003970 con 0x5556bc9d5a40 2024-04-21T23:14:18.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.268+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1715916896 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f04d4043050 con 0x5556bc9d5a40 2024-04-21T23:14:18.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.268+0000 7f04e3443740 10 client.16905 fetch_fsmap learned FSMap version 25 2024-04-21T23:14:18.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.268+0000 7f04e3443740 1 -- 172.21.15.44:0/1715916896 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5556bca55920 con 0x5556bc9d5a40 2024-04-21T23:14:18.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.268+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1715916896 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04d4023040 con 0x5556bc9d5a40 2024-04-21T23:14:18.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.268+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1715916896 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f04d402e030 con 0x5556bc9d5a40 2024-04-21T23:14:18.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.268+0000 7f04e3443740 10 client.16905 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:14:18.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.268+0000 7f04e3443740 10 client.16905 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:18.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.268+0000 7f04e3443740 20 client.16905 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:18.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.268+0000 7f04e3443740 1 --2- 172.21.15.44:0/1715916896 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5556bca55cf0 0x5556bca580e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.268+0000 7f04e3443740 4 client.16905 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:14:18.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.268+0000 7f04e3443740 1 -- 172.21.15.44:0/1715916896 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5556bc9466b0 con 0x5556bca55cf0 2024-04-21T23:14:18.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.268+0000 7f04d3fff640 1 --2- 172.21.15.44:0/1715916896 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5556bca55cf0 0x5556bca580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.269+0000 7f04c1ffb640 10 client.16905 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:18.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.269+0000 7f04d3fff640 1 --2- 172.21.15.44:0/1715916896 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5556bca55cf0 0x5556bca580e0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c54a3f700602fb4 server_cookie=49d3abf1a82d9aa2 in_seq=0 out_seq=0 2024-04-21T23:14:18.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.273+0000 7f04c1ffb640 1 -- 172.21.15.44:0/1715916896 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5556bc9466b0 con 0x5556bca55cf0 2024-04-21T23:14:18.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.273+0000 7f04c1ffb640 10 client.16905 handle_command_reply: tid=0 2024-04-21T23:14:18.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.273+0000 7f04e3443740 1 client.16905 shutdown 2024-04-21T23:14:18.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e3443740 1 -- 172.21.15.44:0/1715916896 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d5a40 msgr2=0x5556bc9e7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:18.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e3443740 1 --2- 172.21.15.44:0/1715916896 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d5a40 0x5556bc9e7500 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f04d401b680 tx=0x7f04d4005b30 comp rx=0 tx=0).stop 2024-04-21T23:14:18.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e3443740 1 -- 172.21.15.44:0/1715916896 shutdown_connections 2024-04-21T23:14:18.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e3443740 1 --2- 172.21.15.44:0/1715916896 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5556bca55cf0 0x5556bca580e0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e3443740 1 --2- 172.21.15.44:0/1715916896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bc9d6720 0x5556bc9d9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e3443740 1 --2- 172.21.15.44:0/1715916896 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5556bc9dedb0 0x5556bc9d9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e3443740 1 --2- 172.21.15.44:0/1715916896 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5556bc9d5a40 0x5556bc9e7500 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e3443740 1 -- 172.21.15.44:0/1715916896 >> 172.21.15.44:0/1715916896 conn(0x5556bc99c4f0 msgr2=0x5556bc9cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:18.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04c1ffb640 0 client.16905 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:18.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e3443740 1 -- 172.21.15.44:0/1715916896 shutdown_connections 2024-04-21T23:14:18.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e3443740 1 -- 172.21.15.44:0/1715916896 wait complete. 2024-04-21T23:14:18.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e3443740 20 client.16905 trim_cache size 0 max 16384 2024-04-21T23:14:18.276 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:14:18.276 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-21T23:14:18.276 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "6496f011-fa07-4e0c-908a-9c201b0f0d6a", 2024-04-21T23:14:18.277 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-21T23:14:18.277 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:14:18.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e085f640 1 -- 172.21.15.44:0/1249156914 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f04c80a2d00 msgr2=0x7f04c80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:18.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e085f640 1 --2- 172.21.15.44:0/1249156914 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f04c80a2d00 0x7f04c80a51c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f04cc0cd120 tx=0x7f04cc0d3810 comp rx=0 tx=0).stop 2024-04-21T23:14:18.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e085f640 1 -- 172.21.15.44:0/1249156914 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04dc1009e0 msgr2=0x7f04dc1981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:18.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e085f640 1 --2- 172.21.15.44:0/1249156914 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04dc1009e0 0x7f04dc1981e0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f04cc00e9f0 tx=0x7f04cc00edf0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e085f640 1 -- 172.21.15.44:0/1249156914 shutdown_connections 2024-04-21T23:14:18.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e085f640 1 --2- 172.21.15.44:0/1249156914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04dc102ac0 0x7f04dc19caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e085f640 1 --2- 172.21.15.44:0/1249156914 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f04c80a2d00 0x7f04c80a51c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e085f640 1 --2- 172.21.15.44:0/1249156914 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f04dc102120 0x7f04dc198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e085f640 1 --2- 172.21.15.44:0/1249156914 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f04dc1009e0 0x7f04dc1981e0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.274+0000 7f04e085f640 1 -- 172.21.15.44:0/1249156914 >> 172.21.15.44:0/1249156914 conn(0x7f04dc077760 msgr2=0x7f04dc1055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:18.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.275+0000 7f04e085f640 1 -- 172.21.15.44:0/1249156914 shutdown_connections 2024-04-21T23:14:18.278 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.275+0000 7f04e085f640 1 -- 172.21.15.44:0/1249156914 wait complete. 2024-04-21T23:14:18.285 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:17 smithi044 ceph-mon[27934]: pgmap v168: 129 pgs: 129 active+clean; 84 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.2 MiB/s wr, 512 op/s 2024-04-21T23:14:18.285 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:17 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3675579384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:14:18.286 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:14:18.286 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:14:18.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.442+0000 7f35316bc640 1 -- 172.21.15.44:0/2861421180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f352c100e30 msgr2=0x7f352c1052f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:18.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.442+0000 7f35316bc640 1 --2- 172.21.15.44:0/2861421180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f352c100e30 0x7f352c1052f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f3520009890 tx=0x7f352002bc10 comp rx=0 tx=0).stop 2024-04-21T23:14:18.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.443+0000 7f35316bc640 1 -- 172.21.15.44:0/2861421180 shutdown_connections 2024-04-21T23:14:18.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.443+0000 7f35316bc640 1 --2- 172.21.15.44:0/2861421180 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f352c105910 0x7f352c105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.443+0000 7f35316bc640 1 --2- 172.21.15.44:0/2861421180 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f352c100e30 0x7f352c1052f0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.443+0000 7f35316bc640 1 --2- 172.21.15.44:0/2861421180 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f352c100510 0x7f352c1008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.443+0000 7f35316bc640 1 -- 172.21.15.44:0/2861421180 >> 172.21.15.44:0/2861421180 conn(0x7f352c0771f0 msgr2=0x7f352c0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:18.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.444+0000 7f35316bc640 1 -- 172.21.15.44:0/2861421180 shutdown_connections 2024-04-21T23:14:18.444 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.444+0000 7f35316bc640 1 -- 172.21.15.44:0/2861421180 wait complete. 2024-04-21T23:14:18.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.445+0000 7f35316bc640 1 Processor -- start 2024-04-21T23:14:18.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.445+0000 7f35316bc640 1 -- start start 2024-04-21T23:14:18.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.446+0000 7f35316bc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f352c100510 0x7f352c19e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.446+0000 7f35316bc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f352c100e30 0x7f352c19e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.446+0000 7f35316bc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f352c105910 0x7f352c1a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.446+0000 7f35316bc640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f352c117f70 con 0x7f352c100e30 2024-04-21T23:14:18.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.446+0000 7f35316bc640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f352c117df0 con 0x7f352c105910 2024-04-21T23:14:18.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.446+0000 7f35316bc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f352c1180f0 con 0x7f352c100510 2024-04-21T23:14:18.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.446+0000 7f352b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f352c100e30 0x7f352c19e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.446+0000 7f352b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f352c100e30 0x7f352c19e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43878/0 (socket says 172.21.15.44:43878) 2024-04-21T23:14:18.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.446+0000 7f352b7fe640 1 -- 172.21.15.44:0/822005634 learned_addr learned my addr 172.21.15.44:0/822005634 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:18.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.447+0000 7f3530ebb640 1 --2- 172.21.15.44:0/822005634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f352c105910 0x7f352c1a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.447+0000 7f352b7fe640 1 -- 172.21.15.44:0/822005634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f352c100510 msgr2=0x7f352c19e350 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T23:14:18.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.447+0000 7f352b7fe640 1 --2- 172.21.15.44:0/822005634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f352c100510 0x7f352c19e350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.447 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.447+0000 7f352b7fe640 1 -- 172.21.15.44:0/822005634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f352c105910 msgr2=0x7f352c1a2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:18.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.447+0000 7f352b7fe640 1 --2- 172.21.15.44:0/822005634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f352c105910 0x7f352c1a2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.447+0000 7f352b7fe640 1 -- 172.21.15.44:0/822005634 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f352c1a3340 con 0x7f352c100e30 2024-04-21T23:14:18.448 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.447+0000 7f3530ebb640 1 --2- 172.21.15.44:0/822005634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f352c105910 0x7f352c1a2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:18.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.450+0000 7f352b7fe640 1 --2- 172.21.15.44:0/822005634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f352c100e30 0x7f352c19e890 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f3520037040 tx=0x7f3520034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:18.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.451+0000 7f35297fa640 1 -- 172.21.15.44:0/822005634 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3520039040 con 0x7f352c100e30 2024-04-21T23:14:18.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.451+0000 7f35297fa640 1 -- 172.21.15.44:0/822005634 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3520044070 con 0x7f352c100e30 2024-04-21T23:14:18.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.452+0000 7f35316bc640 1 -- 172.21.15.44:0/822005634 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f352c1a35d0 con 0x7f352c100e30 2024-04-21T23:14:18.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.452+0000 7f35316bc640 1 -- 172.21.15.44:0/822005634 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f352c1a3a00 con 0x7f352c100e30 2024-04-21T23:14:18.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.452+0000 7f35297fa640 1 -- 172.21.15.44:0/822005634 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352003f440 con 0x7f352c100e30 2024-04-21T23:14:18.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.456+0000 7f35355e4740 1 Processor -- start 2024-04-21T23:14:18.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.457+0000 7f35355e4740 1 -- start start 2024-04-21T23:14:18.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.457+0000 7f35355e4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c2620e0 0x558c9c2624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.457+0000 7f35355e4740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c9c262a00 0x558c9c26b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.457+0000 7f35355e4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c9c26b5e0 0x558c9c26d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.457+0000 7f35355e4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558c9c1c6560 con 0x558c9c2620e0 2024-04-21T23:14:18.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.457+0000 7f35355e4740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x558c9c16e730 con 0x558c9c262a00 2024-04-21T23:14:18.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.458+0000 7f35355e4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558c9c1c2520 con 0x558c9c26b5e0 2024-04-21T23:14:18.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.458+0000 7f3530ebb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c2620e0 0x558c9c2624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.458+0000 7f3530ebb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c2620e0 0x558c9c2624c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43888/0 (socket says 172.21.15.44:43888) 2024-04-21T23:14:18.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.458+0000 7f3530ebb640 1 -- 172.21.15.44:0/2258404291 learned_addr learned my addr 172.21.15.44:0/2258404291 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:18.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.458+0000 7f3530ebb640 1 -- 172.21.15.44:0/2258404291 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c9c26b5e0 msgr2=0x558c9c26d9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T23:14:18.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.458+0000 7f3530ebb640 1 --2- 172.21.15.44:0/2258404291 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c9c26b5e0 0x558c9c26d9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.458+0000 7f3530ebb640 1 -- 172.21.15.44:0/2258404291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c9c262a00 msgr2=0x558c9c26b0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T23:14:18.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.458+0000 7f3530ebb640 1 --2- 172.21.15.44:0/2258404291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c9c262a00 0x558c9c26b0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.459+0000 7f3530ebb640 1 -- 172.21.15.44:0/2258404291 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c9bfa47d0 con 0x558c9c2620e0 2024-04-21T23:14:18.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.459+0000 7f3530ebb640 1 --2- 172.21.15.44:0/2258404291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c2620e0 0x558c9c2624c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f352400f600 tx=0x7f352400fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2bb1893e7aca1e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:18.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.459+0000 7f35127fc640 1 -- 172.21.15.44:0/2258404291 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3524003b70 con 0x558c9c2620e0 2024-04-21T23:14:18.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.460+0000 7f35127fc640 1 -- 172.21.15.44:0/2258404291 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f352400dd60 con 0x558c9c2620e0 2024-04-21T23:14:18.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.460+0000 7f35127fc640 1 -- 172.21.15.44:0/2258404291 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3524018440 con 0x558c9c2620e0 2024-04-21T23:14:18.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.460+0000 7f35297fa640 1 -- 172.21.15.44:0/822005634 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f35200035f0 con 0x7f352c100e30 2024-04-21T23:14:18.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.461+0000 7f35355e4740 1 -- 172.21.15.44:0/2258404291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c2620e0 msgr2=0x558c9c2624c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:18.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.461+0000 7f35355e4740 1 --2- 172.21.15.44:0/2258404291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c2620e0 0x558c9c2624c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f352400f600 tx=0x7f352400fac0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.461 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.461+0000 7f35355e4740 1 -- 172.21.15.44:0/2258404291 shutdown_connections 2024-04-21T23:14:18.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.461+0000 7f35355e4740 1 --2- 172.21.15.44:0/2258404291 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c9c26b5e0 0x558c9c26d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.461+0000 7f35355e4740 1 --2- 172.21.15.44:0/2258404291 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c9c262a00 0x558c9c26b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.461+0000 7f35355e4740 1 --2- 172.21.15.44:0/2258404291 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c2620e0 0x558c9c2624c0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.461+0000 7f35355e4740 1 -- 172.21.15.44:0/2258404291 >> 172.21.15.44:0/2258404291 conn(0x558c9c25c810 msgr2=0x558c9c25cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:18.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.462+0000 7f35297fa640 1 --2- 172.21.15.44:0/822005634 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f35140a2f00 0x7f35140a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.462 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.462+0000 7f35297fa640 1 -- 172.21.15.44:0/822005634 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f35200f1460 con 0x7f352c100e30 2024-04-21T23:14:18.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.464+0000 7f352bfff640 1 --2- 172.21.15.44:0/822005634 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f35140a2f00 0x7f35140a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.465+0000 7f352bfff640 1 --2- 172.21.15.44:0/822005634 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f35140a2f00 0x7f35140a53c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x558c9c261bb0 tx=0x7f351c009260 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:18.467 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.467+0000 7f35355e4740 1 -- 172.21.15.44:0/2258404291 shutdown_connections 2024-04-21T23:14:18.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.470+0000 7f35355e4740 1 -- 172.21.15.44:0/2258404291 wait complete. 2024-04-21T23:14:18.469 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.470+0000 7f35355e4740 1 -- start start 2024-04-21T23:14:18.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.471+0000 7f35355e4740 1 Processor -- start 2024-04-21T23:14:18.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.471+0000 7f35355e4740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c9c2620e0 0x558c9c268630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.472+0000 7f35355e4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c9c262a00 0x558c9c268b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.472+0000 7f352bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c9c2620e0 0x558c9c268630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.472+0000 7f352bfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c9c2620e0 0x558c9c268630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:53890/0 (socket says 172.21.15.44:53890) 2024-04-21T23:14:18.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.472+0000 7f352bfff640 1 -- 172.21.15.44:0/1325249487 learned_addr learned my addr 172.21.15.44:0/1325249487 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:18.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.472+0000 7f35355e4740 1 --2- 172.21.15.44:0/1325249487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c26b5e0 0x558c9c2690b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.472+0000 7f35355e4740 1 -- 172.21.15.44:0/1325249487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558c9c27f430 con 0x558c9c26b5e0 2024-04-21T23:14:18.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.472+0000 7f3530ebb640 1 --2- 172.21.15.44:0/1325249487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c26b5e0 0x558c9c2690b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.472+0000 7f35355e4740 1 -- 172.21.15.44:0/1325249487 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x558c9c181e00 con 0x558c9c2620e0 2024-04-21T23:14:18.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.472+0000 7f35355e4740 1 -- 172.21.15.44:0/1325249487 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558c9c27f5b0 con 0x558c9c262a00 2024-04-21T23:14:18.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.473+0000 7f35355e4740 10 client.0 _ll_register_callbacks cb 0x558c9c1bceb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:18.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.473+0000 7f3530ebb640 1 -- 172.21.15.44:0/1325249487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c9c262a00 msgr2=0x558c9c268b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:18.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.473+0000 7f352b7fe640 1 --2- 172.21.15.44:0/1325249487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c9c262a00 0x558c9c268b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.473+0000 7f3530ebb640 1 --2- 172.21.15.44:0/1325249487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c9c262a00 0x558c9c268b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.473+0000 7f3530ebb640 1 -- 172.21.15.44:0/1325249487 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c9c2620e0 msgr2=0x558c9c268630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:18.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.473+0000 7f3530ebb640 1 --2- 172.21.15.44:0/1325249487 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c9c2620e0 0x558c9c268630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.473+0000 7f3530ebb640 1 -- 172.21.15.44:0/1325249487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c9beda3f0 con 0x558c9c26b5e0 2024-04-21T23:14:18.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.474+0000 7f352b7fe640 1 --2- 172.21.15.44:0/1325249487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c9c262a00 0x558c9c268b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:14:18.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.474+0000 7f3530ebb640 1 --2- 172.21.15.44:0/1325249487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c26b5e0 0x558c9c2690b0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f352400f600 tx=0x7f352400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee5fc4e3e9735c8e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:18.474 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.474+0000 7f350b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:14:18.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.475+0000 7f350b7fe640 1 -- 172.21.15.44:0/1325249487 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3524026070 con 0x558c9c26b5e0 2024-04-21T23:14:18.475 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.476+0000 7f35355e4740 1 -- 172.21.15.44:0/1325249487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f352400dd60 con 0x558c9c26b5e0 2024-04-21T23:14:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.476+0000 7f350b7fe640 1 -- 172.21.15.44:0/1325249487 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f3524003970 con 0x558c9c26b5e0 2024-04-21T23:14:18.476 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.477+0000 7f350b7fe640 1 -- 172.21.15.44:0/1325249487 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f352402f050 con 0x558c9c26b5e0 2024-04-21T23:14:18.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.479+0000 7f35355e4740 10 client.16917 fetch_fsmap learned FSMap version 25 2024-04-21T23:14:18.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.479+0000 7f35355e4740 1 -- 172.21.15.44:0/1325249487 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558c9be9a0f0 con 0x558c9c26b5e0 2024-04-21T23:14:18.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.480+0000 7f350b7fe640 1 -- 172.21.15.44:0/1325249487 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f352400dd60 con 0x558c9c26b5e0 2024-04-21T23:14:18.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.480+0000 7f350b7fe640 1 -- 172.21.15.44:0/1325249487 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f3524029020 con 0x558c9c26b5e0 2024-04-21T23:14:18.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.481+0000 7f35355e4740 10 client.16917 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:14:18.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.481+0000 7f35355e4740 10 client.16917 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:18.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.481+0000 7f35355e4740 20 client.16917 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:18.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.481+0000 7f35355e4740 1 --2- 172.21.15.44:0/1325249487 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558c9c28c930 0x558c9c28ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.481+0000 7f35355e4740 4 client.16917 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:14:18.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.481+0000 7f35355e4740 1 -- 172.21.15.44:0/1325249487 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c9c1d3aa0 con 0x558c9c28c930 2024-04-21T23:14:18.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.482+0000 7f352bfff640 1 --2- 172.21.15.44:0/1325249487 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558c9c28c930 0x558c9c28ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.484+0000 7f352bfff640 1 --2- 172.21.15.44:0/1325249487 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558c9c28c930 0x558c9c28ed20 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac5506bd5309a93d server_cookie=cc5902599df1dc78 in_seq=0 out_seq=0 2024-04-21T23:14:18.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.484+0000 7f350b7fe640 10 client.16917 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:18.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.486+0000 7f350b7fe640 1 -- 172.21.15.44:0/1325249487 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558c9c1d3aa0 con 0x558c9c28c930 2024-04-21T23:14:18.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.486+0000 7f350b7fe640 10 client.16917 handle_command_reply: tid=0 2024-04-21T23:14:18.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.486+0000 7f35355e4740 1 client.16917 shutdown 2024-04-21T23:14:18.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.486+0000 7f35355e4740 1 -- 172.21.15.44:0/1325249487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c26b5e0 msgr2=0x558c9c2690b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:18.486 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.486+0000 7f35355e4740 1 --2- 172.21.15.44:0/1325249487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c26b5e0 0x558c9c2690b0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f352400f600 tx=0x7f352400dab0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.486+0000 7f35355e4740 1 -- 172.21.15.44:0/1325249487 shutdown_connections 2024-04-21T23:14:18.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.486+0000 7f35355e4740 1 --2- 172.21.15.44:0/1325249487 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558c9c28c930 0x558c9c28ed20 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.486+0000 7f35355e4740 1 --2- 172.21.15.44:0/1325249487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c9c262a00 0x558c9c268b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.486+0000 7f35355e4740 1 --2- 172.21.15.44:0/1325249487 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c26b5e0 0x558c9c2690b0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.486+0000 7f35355e4740 1 --2- 172.21.15.44:0/1325249487 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c9c2620e0 0x558c9c268630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.486+0000 7f35355e4740 1 -- 172.21.15.44:0/1325249487 >> 172.21.15.44:0/1325249487 conn(0x558c9c25c810 msgr2=0x558c9c25e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:18.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.486+0000 7f350b7fe640 0 client.16917 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:18.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.486+0000 7f35355e4740 1 -- 172.21.15.44:0/1325249487 shutdown_connections 2024-04-21T23:14:18.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.486+0000 7f35355e4740 1 -- 172.21.15.44:0/1325249487 wait complete. 2024-04-21T23:14:18.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.486+0000 7f35355e4740 20 client.16917 trim_cache size 0 max 16384 2024-04-21T23:14:18.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.504+0000 7f35355e4740 1 Processor -- start 2024-04-21T23:14:18.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.504+0000 7f35355e4740 1 -- start start 2024-04-21T23:14:18.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.504+0000 7f35355e4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c9c2620e0 0x558c9c28e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.504+0000 7f35355e4740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c9c262a00 0x558c9c288320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.504+0000 7f35355e4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c26b5e0 0x558c9c267de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.504+0000 7f35355e4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f352c04f540 con 0x558c9c26b5e0 2024-04-21T23:14:18.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.504+0000 7f35355e4740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f352c1acea0 con 0x558c9c262a00 2024-04-21T23:14:18.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.504+0000 7f35355e4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f352c10ff60 con 0x558c9c2620e0 2024-04-21T23:14:18.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.505+0000 7f3530ebb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c26b5e0 0x558c9c267de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.505+0000 7f3530ebb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c26b5e0 0x558c9c267de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43920/0 (socket says 172.21.15.44:43920) 2024-04-21T23:14:18.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.505+0000 7f3530ebb640 1 -- 172.21.15.44:0/2318864801 learned_addr learned my addr 172.21.15.44:0/2318864801 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:18.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.505+0000 7f352b7fe640 1 --2- 172.21.15.44:0/2318864801 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c9c262a00 0x558c9c288320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.505+0000 7f3530ebb640 1 -- 172.21.15.44:0/2318864801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c9c2620e0 msgr2=0x558c9c28e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:18.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.505+0000 7f3530ebb640 1 --2- 172.21.15.44:0/2318864801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c9c2620e0 0x558c9c28e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.505+0000 7f3530ebb640 1 -- 172.21.15.44:0/2318864801 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c9c262a00 msgr2=0x558c9c288320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:18.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.505+0000 7f3530ebb640 1 --2- 172.21.15.44:0/2318864801 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c9c262a00 0x558c9c288320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.505+0000 7f3530ebb640 1 -- 172.21.15.44:0/2318864801 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c9c262830 con 0x558c9c26b5e0 2024-04-21T23:14:18.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.505+0000 7f3530ebb640 1 --2- 172.21.15.44:0/2318864801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c26b5e0 0x558c9c267de0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f3524003fd0 tx=0x7f352400a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98a3f768870265d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:18.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.505+0000 7f3511ffb640 1 -- 172.21.15.44:0/2318864801 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3524026040 con 0x558c9c26b5e0 2024-04-21T23:14:18.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.505+0000 7f3511ffb640 1 -- 172.21.15.44:0/2318864801 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3524004c50 con 0x558c9c26b5e0 2024-04-21T23:14:18.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.505+0000 7f3511ffb640 1 -- 172.21.15.44:0/2318864801 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352400ae00 con 0x558c9c26b5e0 2024-04-21T23:14:18.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.506+0000 7f35355e4740 1 -- 172.21.15.44:0/2318864801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c26b5e0 msgr2=0x558c9c267de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:18.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.506+0000 7f35355e4740 1 --2- 172.21.15.44:0/2318864801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c26b5e0 0x558c9c267de0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f3524003fd0 tx=0x7f352400a300 comp rx=0 tx=0).stop 2024-04-21T23:14:18.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.506+0000 7f35355e4740 1 -- 172.21.15.44:0/2318864801 shutdown_connections 2024-04-21T23:14:18.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.506+0000 7f35355e4740 1 --2- 172.21.15.44:0/2318864801 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c26b5e0 0x558c9c267de0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.506+0000 7f35355e4740 1 --2- 172.21.15.44:0/2318864801 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c9c262a00 0x558c9c288320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.506+0000 7f35355e4740 1 --2- 172.21.15.44:0/2318864801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c9c2620e0 0x558c9c28e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.506+0000 7f35355e4740 1 -- 172.21.15.44:0/2318864801 >> 172.21.15.44:0/2318864801 conn(0x558c9c25c810 msgr2=0x558c9c25af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:18.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.506+0000 7f35355e4740 1 -- 172.21.15.44:0/2318864801 shutdown_connections 2024-04-21T23:14:18.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.506+0000 7f35355e4740 1 -- 172.21.15.44:0/2318864801 wait complete. 2024-04-21T23:14:18.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.506+0000 7f35355e4740 1 -- start start 2024-04-21T23:14:18.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.507+0000 7f35355e4740 1 Processor -- start 2024-04-21T23:14:18.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.507+0000 7f35355e4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c2620e0 0x558c9c27f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.507+0000 7f35355e4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c9c262a00 0x558c9c27ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.507+0000 7f35355e4740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c9c26b5e0 0x558c9c2f0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.507+0000 7f35355e4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f352c1ab360 con 0x558c9c2620e0 2024-04-21T23:14:18.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.507+0000 7f352bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c2620e0 0x558c9c27f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.507+0000 7f352bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c2620e0 0x558c9c27f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:43928/0 (socket says 172.21.15.44:43928) 2024-04-21T23:14:18.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.507+0000 7f352bfff640 1 -- 172.21.15.44:0/2236569930 learned_addr learned my addr 172.21.15.44:0/2236569930 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:18.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.507+0000 7f352b7fe640 1 --2- 172.21.15.44:0/2236569930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c9c262a00 0x558c9c27ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.507+0000 7f35355e4740 1 -- 172.21.15.44:0/2236569930 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f352c1ac7d0 con 0x558c9c26b5e0 2024-04-21T23:14:18.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.507+0000 7f35355e4740 1 -- 172.21.15.44:0/2236569930 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f352c1aba30 con 0x558c9c262a00 2024-04-21T23:14:18.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.507+0000 7f3530ebb640 1 --2- 172.21.15.44:0/2236569930 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c9c26b5e0 0x558c9c2f0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.507+0000 7f35355e4740 10 client.0 _ll_register_callbacks cb 0x7f352400f450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:18.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.507+0000 7f352b7fe640 1 -- 172.21.15.44:0/2236569930 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c9c26b5e0 msgr2=0x558c9c2f0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:18.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.507+0000 7f352b7fe640 1 --2- 172.21.15.44:0/2236569930 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c9c26b5e0 0x558c9c2f0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.507+0000 7f352b7fe640 1 -- 172.21.15.44:0/2236569930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c2620e0 msgr2=0x558c9c27f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:18.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.507+0000 7f352b7fe640 1 --2- 172.21.15.44:0/2236569930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c2620e0 0x558c9c27f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.507+0000 7f352b7fe640 1 -- 172.21.15.44:0/2236569930 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c9c2f1250 con 0x558c9c262a00 2024-04-21T23:14:18.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.507+0000 7f352bfff640 1 --2- 172.21.15.44:0/2236569930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c2620e0 0x558c9c27f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:18.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.508+0000 7f352b7fe640 1 --2- 172.21.15.44:0/2236569930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c9c262a00 0x558c9c27ff20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f35200ee8c0 tx=0x7f3520000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11ac76c7dca30198 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:18.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.508+0000 7f350b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:14:18.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.508+0000 7f350b7fe640 1 -- 172.21.15.44:0/2236569930 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35200bd690 con 0x558c9c262a00 2024-04-21T23:14:18.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.508+0000 7f350b7fe640 1 -- 172.21.15.44:0/2236569930 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f35200f1120 con 0x558c9c262a00 2024-04-21T23:14:18.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.508+0000 7f35355e4740 1 -- 172.21.15.44:0/2236569930 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3524004c50 con 0x558c9c262a00 2024-04-21T23:14:18.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.508+0000 7f350b7fe640 1 -- 172.21.15.44:0/2236569930 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f35200dc080 con 0x558c9c262a00 2024-04-21T23:14:18.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.508+0000 7f35355e4740 10 client.26491 fetch_fsmap learned FSMap version 25 2024-04-21T23:14:18.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.508+0000 7f35355e4740 1 -- 172.21.15.44:0/2236569930 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558c9c2f24d0 con 0x558c9c262a00 2024-04-21T23:14:18.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.508+0000 7f350b7fe640 1 -- 172.21.15.44:0/2236569930 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3524004c50 con 0x558c9c262a00 2024-04-21T23:14:18.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.508+0000 7f350b7fe640 1 -- 172.21.15.44:0/2236569930 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f352003c030 con 0x558c9c262a00 2024-04-21T23:14:18.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.508+0000 7f35355e4740 10 client.26491 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:14:18.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.508+0000 7f35355e4740 10 client.26491 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:18.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.508+0000 7f35355e4740 20 client.26491 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:18.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.508+0000 7f35355e4740 1 --2- 172.21.15.44:0/2236569930 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558c9c2f2bd0 0x558c9c2f4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:18.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.508+0000 7f35355e4740 4 client.26491 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:14:18.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.508+0000 7f35355e4740 1 -- 172.21.15.44:0/2236569930 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558c9c1cddf0 con 0x558c9c2f2bd0 2024-04-21T23:14:18.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.509+0000 7f352bfff640 1 --2- 172.21.15.44:0/2236569930 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558c9c2f2bd0 0x558c9c2f4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:18.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.509+0000 7f350b7fe640 10 client.26491 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:18.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.509+0000 7f352bfff640 1 --2- 172.21.15.44:0/2236569930 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558c9c2f2bd0 0x558c9c2f4ff0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4272e22e7ef477b3 server_cookie=fffd15c8816b64 in_seq=0 out_seq=0 2024-04-21T23:14:18.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.510+0000 7f350b7fe640 1 -- 172.21.15.44:0/2236569930 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558c9c1cddf0 con 0x558c9c2f2bd0 2024-04-21T23:14:18.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.510+0000 7f350b7fe640 10 client.26491 handle_command_reply: tid=0 2024-04-21T23:14:18.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.510+0000 7f35355e4740 1 client.26491 shutdown 2024-04-21T23:14:18.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.510+0000 7f35355e4740 1 -- 172.21.15.44:0/2236569930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c9c262a00 msgr2=0x558c9c27ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:18.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.510+0000 7f35355e4740 1 --2- 172.21.15.44:0/2236569930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c9c262a00 0x558c9c27ff20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f35200ee8c0 tx=0x7f3520000bc0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.510+0000 7f35355e4740 1 -- 172.21.15.44:0/2236569930 shutdown_connections 2024-04-21T23:14:18.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.510+0000 7f35355e4740 1 --2- 172.21.15.44:0/2236569930 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558c9c2f2bd0 0x558c9c2f4ff0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.510+0000 7f35355e4740 1 --2- 172.21.15.44:0/2236569930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c9c262a00 0x558c9c27ff20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.510+0000 7f350b7fe640 0 client.26491 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:18.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.510+0000 7f35355e4740 1 --2- 172.21.15.44:0/2236569930 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c9c26b5e0 0x558c9c2f0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.510+0000 7f35355e4740 1 --2- 172.21.15.44:0/2236569930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c9c2620e0 0x558c9c27f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.510+0000 7f35355e4740 1 -- 172.21.15.44:0/2236569930 >> 172.21.15.44:0/2236569930 conn(0x558c9c25c810 msgr2=0x558c9c25b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:18.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.510+0000 7f35355e4740 1 -- 172.21.15.44:0/2236569930 shutdown_connections 2024-04-21T23:14:18.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.511+0000 7f35355e4740 1 -- 172.21.15.44:0/2236569930 wait complete. 2024-04-21T23:14:18.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.511+0000 7f35355e4740 20 client.26491 trim_cache size 0 max 16384 2024-04-21T23:14:18.517 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:14:18.517 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:14:18.517 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-21T23:14:18.517 INFO:teuthology.orchestra.run.smithi044.stdout: "6496f011-fa07-4e0c-908a-9c201b0f0d6a": { 2024-04-21T23:14:18.517 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-21T23:14:18.517 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "6496f011-fa07-4e0c-908a-9c201b0f0d6a", 2024-04-21T23:14:18.518 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-04-21T23:14:18.518 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:14:18.518 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:14:18.518 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:14:18.518 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.511+0000 7f35316bc640 1 -- 172.21.15.44:0/822005634 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f35140a2f00 msgr2=0x7f35140a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:18.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.511+0000 7f35316bc640 1 --2- 172.21.15.44:0/822005634 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f35140a2f00 0x7f35140a53c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x558c9c261bb0 tx=0x7f351c009260 comp rx=0 tx=0).stop 2024-04-21T23:14:18.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.511+0000 7f35316bc640 1 -- 172.21.15.44:0/822005634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f352c100e30 msgr2=0x7f352c19e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:18.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.511+0000 7f35316bc640 1 --2- 172.21.15.44:0/822005634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f352c100e30 0x7f352c19e890 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f3520037040 tx=0x7f3520034670 comp rx=0 tx=0).stop 2024-04-21T23:14:18.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.511+0000 7f35316bc640 1 -- 172.21.15.44:0/822005634 shutdown_connections 2024-04-21T23:14:18.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.511+0000 7f35316bc640 1 --2- 172.21.15.44:0/822005634 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f35140a2f00 0x7f35140a53c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.511+0000 7f35316bc640 1 --2- 172.21.15.44:0/822005634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f352c105910 0x7f352c1a2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.511+0000 7f35316bc640 1 --2- 172.21.15.44:0/822005634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f352c100e30 0x7f352c19e890 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.511+0000 7f35316bc640 1 --2- 172.21.15.44:0/822005634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f352c100510 0x7f352c19e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:18.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.511+0000 7f35316bc640 1 -- 172.21.15.44:0/822005634 >> 172.21.15.44:0/822005634 conn(0x7f352c0771f0 msgr2=0x7f352c0fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:18.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.511+0000 7f35316bc640 1 -- 172.21.15.44:0/822005634 shutdown_connections 2024-04-21T23:14:18.519 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:18.511+0000 7f35316bc640 1 -- 172.21.15.44:0/822005634 wait complete. 2024-04-21T23:14:18.521 INFO:tasks.cephfs.filesystem:scrub status for tag:6496f011-fa07-4e0c-908a-9c201b0f0d6a - {'path': '/', 'tag': '6496f011-fa07-4e0c-908a-9c201b0f0d6a', 'options': 'recursive,force'} 2024-04-21T23:14:18.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 625630 13.31 MB/sec execute 128 sec latency 254.789 ms 2024-04-21T23:14:19.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 628702 13.35 MB/sec execute 129 sec latency 22.129 ms 2024-04-21T23:14:20.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:19 smithi055 ceph-mon[31117]: scrub queued for path: / 2024-04-21T23:14:20.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:19 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:14:20.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:19 smithi055 ceph-mon[31117]: scrub summary: active paths [/] 2024-04-21T23:14:20.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:19 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:14:20.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:19 smithi055 ceph-mon[31117]: pgmap v169: 129 pgs: 129 active+clean; 84 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 274 op/s 2024-04-21T23:14:20.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:19 smithi069 ceph-mon[31816]: scrub queued for path: / 2024-04-21T23:14:20.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:19 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:14:20.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:19 smithi069 ceph-mon[31816]: scrub summary: active paths [/] 2024-04-21T23:14:20.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:19 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:14:20.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:19 smithi069 ceph-mon[31816]: pgmap v169: 129 pgs: 129 active+clean; 84 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 274 op/s 2024-04-21T23:14:20.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:19 smithi044 ceph-mon[27934]: scrub queued for path: / 2024-04-21T23:14:20.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:19 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:14:20.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:19 smithi044 ceph-mon[27934]: scrub summary: active paths [/] 2024-04-21T23:14:20.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:19 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:14:20.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:19 smithi044 ceph-mon[27934]: pgmap v169: 129 pgs: 129 active+clean; 84 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 274 op/s 2024-04-21T23:14:20.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 631199 13.32 MB/sec execute 130 sec latency 27.670 ms 2024-04-21T23:14:21.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 633135 13.31 MB/sec execute 131 sec latency 30.612 ms 2024-04-21T23:14:22.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:21 smithi055 ceph-mon[31117]: pgmap v170: 129 pgs: 129 active+clean; 84 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.6 MiB/s wr, 254 op/s 2024-04-21T23:14:22.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:21 smithi069 ceph-mon[31816]: pgmap v170: 129 pgs: 129 active+clean; 84 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.6 MiB/s wr, 254 op/s 2024-04-21T23:14:22.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:21 smithi044 ceph-mon[27934]: pgmap v170: 129 pgs: 129 active+clean; 84 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.6 MiB/s wr, 254 op/s 2024-04-21T23:14:22.701 INFO:tasks.workunit.client.0.smithi044.stdout: 1 636106 13.35 MB/sec execute 132 sec latency 14.327 ms 2024-04-21T23:14:23.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:22 smithi055 ceph-mon[31117]: scrub summary: idle 2024-04-21T23:14:23.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:22 smithi069 ceph-mon[31816]: scrub summary: idle 2024-04-21T23:14:23.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:22 smithi044 ceph-mon[27934]: scrub summary: idle 2024-04-21T23:14:23.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 638093 13.30 MB/sec execute 133 sec latency 51.401 ms 2024-04-21T23:14:24.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:23 smithi069 ceph-mon[31816]: pgmap v171: 129 pgs: 129 active+clean; 71 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.4 MiB/s wr, 535 op/s 2024-04-21T23:14:24.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:23 smithi055 ceph-mon[31117]: pgmap v171: 129 pgs: 129 active+clean; 71 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.4 MiB/s wr, 535 op/s 2024-04-21T23:14:24.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:23 smithi044 ceph-mon[27934]: pgmap v171: 129 pgs: 129 active+clean; 71 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.4 MiB/s wr, 535 op/s 2024-04-21T23:14:24.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 640315 13.30 MB/sec execute 134 sec latency 16.878 ms 2024-04-21T23:14:25.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 643332 13.33 MB/sec execute 135 sec latency 15.937 ms 2024-04-21T23:14:26.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:25 smithi055 ceph-mon[31117]: pgmap v172: 129 pgs: 129 active+clean; 71 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.7 MiB/s wr, 407 op/s 2024-04-21T23:14:26.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:25 smithi069 ceph-mon[31816]: pgmap v172: 129 pgs: 129 active+clean; 71 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.7 MiB/s wr, 407 op/s 2024-04-21T23:14:26.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:25 smithi044 ceph-mon[27934]: pgmap v172: 129 pgs: 129 active+clean; 71 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.7 MiB/s wr, 407 op/s 2024-04-21T23:14:26.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 646363 13.36 MB/sec execute 136 sec latency 36.019 ms 2024-04-21T23:14:27.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 648611 13.32 MB/sec execute 137 sec latency 25.897 ms 2024-04-21T23:14:28.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:27 smithi069 ceph-mon[31816]: pgmap v173: 129 pgs: 129 active+clean; 78 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.3 MiB/s wr, 521 op/s 2024-04-21T23:14:28.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:27 smithi044 ceph-mon[27934]: pgmap v173: 129 pgs: 129 active+clean; 78 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.3 MiB/s wr, 521 op/s 2024-04-21T23:14:28.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:27 smithi055 ceph-mon[31117]: pgmap v173: 129 pgs: 129 active+clean; 78 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.3 MiB/s wr, 521 op/s 2024-04-21T23:14:28.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 650341 13.32 MB/sec execute 138 sec latency 111.174 ms 2024-04-21T23:14:29.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 653402 13.35 MB/sec execute 139 sec latency 15.215 ms 2024-04-21T23:14:30.434 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:30 smithi044 ceph-mon[27934]: pgmap v174: 129 pgs: 129 active+clean; 77 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 408 op/s 2024-04-21T23:14:30.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:30 smithi055 ceph-mon[31117]: pgmap v174: 129 pgs: 129 active+clean; 77 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 408 op/s 2024-04-21T23:14:30.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:30 smithi069 ceph-mon[31816]: pgmap v174: 129 pgs: 129 active+clean; 77 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 408 op/s 2024-04-21T23:14:30.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 655580 13.31 MB/sec execute 140 sec latency 16.952 ms 2024-04-21T23:14:31.212 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.210+0000 7f934c377640 1 -- 172.21.15.44:0/6485525 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9344100680 msgr2=0x7f9344100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:31.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.210+0000 7f934c377640 1 --2- 172.21.15.44:0/6485525 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9344100680 0x7f9344100b00 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f9338009930 tx=0x7f933802bc30 comp rx=0 tx=0).stop 2024-04-21T23:14:31.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.211+0000 7f934c377640 1 -- 172.21.15.44:0/6485525 shutdown_connections 2024-04-21T23:14:31.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.211+0000 7f934c377640 1 --2- 172.21.15.44:0/6485525 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9344101040 0x7f934410d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.211+0000 7f934c377640 1 --2- 172.21.15.44:0/6485525 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9344100680 0x7f9344100b00 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.211+0000 7f934c377640 1 --2- 172.21.15.44:0/6485525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9344106640 0x7f9344106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.211+0000 7f934c377640 1 -- 172.21.15.44:0/6485525 >> 172.21.15.44:0/6485525 conn(0x7f93440777f0 msgr2=0x7f9344077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:31.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.211+0000 7f934c377640 1 -- 172.21.15.44:0/6485525 shutdown_connections 2024-04-21T23:14:31.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.211+0000 7f934c377640 1 -- 172.21.15.44:0/6485525 wait complete. 2024-04-21T23:14:31.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.211+0000 7f934c377640 1 Processor -- start 2024-04-21T23:14:31.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.211+0000 7f934c377640 1 -- start start 2024-04-21T23:14:31.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.211+0000 7f934c377640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9344100680 0x7f9344198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:31.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.211+0000 7f934c377640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9344101040 0x7f9344198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:31.213 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.211+0000 7f934c377640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9344106640 0x7f934419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:31.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.211+0000 7f934c377640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9344110a10 con 0x7f9344106640 2024-04-21T23:14:31.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.211+0000 7f934c377640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9344110890 con 0x7f9344101040 2024-04-21T23:14:31.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.212+0000 7f934c377640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9344110b90 con 0x7f9344100680 2024-04-21T23:14:31.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.212+0000 7f934a914640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9344106640 0x7f934419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:31.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.212+0000 7f934a914640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9344106640 0x7f934419c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39524/0 (socket says 172.21.15.44:39524) 2024-04-21T23:14:31.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.212+0000 7f934a914640 1 -- 172.21.15.44:0/3117338003 learned_addr learned my addr 172.21.15.44:0/3117338003 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:31.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.212+0000 7f9349912640 1 --2- 172.21.15.44:0/3117338003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9344101040 0x7f9344198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:31.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.212+0000 7f934a113640 1 --2- 172.21.15.44:0/3117338003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9344100680 0x7f9344198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:31.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.212+0000 7f934a914640 1 -- 172.21.15.44:0/3117338003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9344100680 msgr2=0x7f9344198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:31.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.212+0000 7f934a914640 1 --2- 172.21.15.44:0/3117338003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9344100680 0x7f9344198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.212+0000 7f934a914640 1 -- 172.21.15.44:0/3117338003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9344101040 msgr2=0x7f9344198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:31.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.212+0000 7f934a914640 1 --2- 172.21.15.44:0/3117338003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9344101040 0x7f9344198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.212+0000 7f934a914640 1 -- 172.21.15.44:0/3117338003 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f934419d000 con 0x7f9344106640 2024-04-21T23:14:31.214 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.212+0000 7f934a113640 1 --2- 172.21.15.44:0/3117338003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9344100680 0x7f9344198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:14:31.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.212+0000 7f934a914640 1 --2- 172.21.15.44:0/3117338003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9344106640 0x7f934419c8d0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f933c00e970 tx=0x7f933c00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:31.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.212+0000 7f93437fe640 1 -- 172.21.15.44:0/3117338003 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c00ccc0 con 0x7f9344106640 2024-04-21T23:14:31.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.212+0000 7f934c377640 1 -- 172.21.15.44:0/3117338003 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f934419d2f0 con 0x7f9344106640 2024-04-21T23:14:31.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.212+0000 7f93437fe640 1 -- 172.21.15.44:0/3117338003 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f933c00ce60 con 0x7f9344106640 2024-04-21T23:14:31.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.212+0000 7f934c377640 1 -- 172.21.15.44:0/3117338003 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93441a4b80 con 0x7f9344106640 2024-04-21T23:14:31.215 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.212+0000 7f93437fe640 1 -- 172.21.15.44:0/3117338003 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c010430 con 0x7f9344106640 2024-04-21T23:14:31.222 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.215+0000 7f93437fe640 1 -- 172.21.15.44:0/3117338003 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f933c004fe0 con 0x7f9344106640 2024-04-21T23:14:31.222 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.216+0000 7f934f039740 1 Processor -- start 2024-04-21T23:14:31.222 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.216+0000 7f934f039740 1 -- start start 2024-04-21T23:14:31.222 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.216+0000 7f934f039740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba28db20 0x5566ba28df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:31.222 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.216+0000 7f934f039740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba28e440 0x5566ba296cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:31.222 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.216+0000 7f934f039740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba297210 0x5566ba299600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:31.222 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.216+0000 7f934f039740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5566ba1f1b70 con 0x5566ba28e440 2024-04-21T23:14:31.222 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.216+0000 7f934f039740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5566ba19a470 con 0x5566ba28db20 2024-04-21T23:14:31.222 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.216+0000 7f934f039740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5566ba1edb30 con 0x5566ba297210 2024-04-21T23:14:31.222 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.216+0000 7f93437fe640 1 --2- 172.21.15.44:0/3117338003 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f932c0a2cf0 0x7f932c0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:31.222 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.216+0000 7f9349912640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba28db20 0x5566ba28df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:31.222 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.216+0000 7f934a113640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba28e440 0x5566ba296cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.216+0000 7f9349912640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba28db20 0x5566ba28df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:36702/0 (socket says 172.21.15.44:36702) 2024-04-21T23:14:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.216+0000 7f9349912640 1 -- 172.21.15.44:0/1258733533 learned_addr learned my addr 172.21.15.44:0/1258733533 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.216+0000 7f934a914640 1 --2- 172.21.15.44:0/3117338003 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f932c0a2cf0 0x7f932c0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.216+0000 7f9349912640 1 --2- 172.21.15.44:0/1258733533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba297210 0x5566ba299600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.216+0000 7f93437fe640 1 -- 172.21.15.44:0/3117338003 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f933c0cfbd0 con 0x7f9344106640 2024-04-21T23:14:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.216+0000 7f9349912640 1 -- 172.21.15.44:0/1258733533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba297210 msgr2=0x5566ba299600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.216+0000 7f9349912640 1 --2- 172.21.15.44:0/1258733533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba297210 0x5566ba299600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.216+0000 7f9349912640 1 -- 172.21.15.44:0/1258733533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba28e440 msgr2=0x5566ba296cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.216+0000 7f9349912640 1 --2- 172.21.15.44:0/1258733533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba28e440 0x5566ba296cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.217+0000 7f9349912640 1 -- 172.21.15.44:0/1258733533 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566b9f06210 con 0x5566ba28db20 2024-04-21T23:14:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.217+0000 7f9349912640 1 --2- 172.21.15.44:0/1258733533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba28db20 0x5566ba28df00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9338009900 tx=0x7f9338005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c97ed97e27f9b798 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.217+0000 7f934a914640 1 --2- 172.21.15.44:0/3117338003 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f932c0a2cf0 0x7f932c0a51b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f933c09ba50 tx=0x7f933c0911e0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.217+0000 7f9340bfe640 1 -- 172.21.15.44:0/1258733533 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933802cb80 con 0x5566ba28db20 2024-04-21T23:14:31.223 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.217+0000 7f9340bfe640 1 -- 172.21.15.44:0/1258733533 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93380046c0 con 0x5566ba28db20 2024-04-21T23:14:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.217+0000 7f9340bfe640 1 -- 172.21.15.44:0/1258733533 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93380051b0 con 0x5566ba28db20 2024-04-21T23:14:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.218+0000 7f934f039740 1 -- 172.21.15.44:0/1258733533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba28db20 msgr2=0x5566ba28df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.218+0000 7f934f039740 1 --2- 172.21.15.44:0/1258733533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba28db20 0x5566ba28df00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9338009900 tx=0x7f9338005b50 comp rx=0 tx=0).stop 2024-04-21T23:14:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.218+0000 7f934f039740 1 -- 172.21.15.44:0/1258733533 shutdown_connections 2024-04-21T23:14:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.218+0000 7f934f039740 1 --2- 172.21.15.44:0/1258733533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba297210 0x5566ba299600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.218+0000 7f934f039740 1 --2- 172.21.15.44:0/1258733533 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba28e440 0x5566ba296cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.218+0000 7f934f039740 1 --2- 172.21.15.44:0/1258733533 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba28db20 0x5566ba28df00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.218+0000 7f934f039740 1 -- 172.21.15.44:0/1258733533 >> 172.21.15.44:0/1258733533 conn(0x5566ba288440 msgr2=0x5566ba288820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.218+0000 7f934f039740 1 -- 172.21.15.44:0/1258733533 shutdown_connections 2024-04-21T23:14:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.218+0000 7f934f039740 1 -- 172.21.15.44:0/1258733533 wait complete. 2024-04-21T23:14:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.218+0000 7f934f039740 1 -- start start 2024-04-21T23:14:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.218+0000 7f934f039740 1 Processor -- start 2024-04-21T23:14:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.218+0000 7f934f039740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba28db20 0x5566ba294260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:31.224 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.218+0000 7f934f039740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba28e440 0x5566ba2947a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.218+0000 7f934f039740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba297210 0x5566ba294ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.218+0000 7f934f039740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5566ba299d80 con 0x5566ba28e440 2024-04-21T23:14:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.218+0000 7f934f039740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5566ba1ada10 con 0x5566ba297210 2024-04-21T23:14:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.218+0000 7f934f039740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5566ba299f00 con 0x5566ba28db20 2024-04-21T23:14:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.218+0000 7f934f039740 10 client.0 _ll_register_callbacks cb 0x5566ba1e8b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.219+0000 7f934a113640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba28e440 0x5566ba2947a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.219+0000 7f934a113640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba28e440 0x5566ba2947a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:39542/0 (socket says 172.21.15.44:39542) 2024-04-21T23:14:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.219+0000 7f934a113640 1 -- 172.21.15.44:0/2585794234 learned_addr learned my addr 172.21.15.44:0/2585794234 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.219+0000 7f9349912640 1 --2- 172.21.15.44:0/2585794234 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba297210 0x5566ba294ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.219+0000 7f9349912640 1 --2- 172.21.15.44:0/2585794234 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba28db20 0x5566ba294260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.219+0000 7f934a113640 1 -- 172.21.15.44:0/2585794234 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba28db20 msgr2=0x5566ba294260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.219+0000 7f934a113640 1 --2- 172.21.15.44:0/2585794234 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba28db20 0x5566ba294260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.219+0000 7f934a113640 1 -- 172.21.15.44:0/2585794234 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba297210 msgr2=0x5566ba294ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.219+0000 7f934a113640 1 --2- 172.21.15.44:0/2585794234 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba297210 0x5566ba294ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.219+0000 7f934a113640 1 -- 172.21.15.44:0/2585794234 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566b9fd2620 con 0x5566ba28e440 2024-04-21T23:14:31.225 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.219+0000 7f9349912640 1 --2- 172.21.15.44:0/2585794234 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba28db20 0x5566ba294260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:14:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.219+0000 7f934a113640 1 --2- 172.21.15.44:0/2585794234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba28e440 0x5566ba2947a0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f93340116c0 tx=0x7f9334011b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4b6594b8c6a38e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.219+0000 7f9329ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:14:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.219+0000 7f9329ffb640 1 -- 172.21.15.44:0/2585794234 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9334003b70 con 0x5566ba28e440 2024-04-21T23:14:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.219+0000 7f9329ffb640 1 -- 172.21.15.44:0/2585794234 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f9334003d10 con 0x5566ba28e440 2024-04-21T23:14:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.219+0000 7f934f039740 1 -- 172.21.15.44:0/2585794234 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93380046c0 con 0x5566ba28e440 2024-04-21T23:14:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.219+0000 7f9329ffb640 1 -- 172.21.15.44:0/2585794234 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f933400f940 con 0x5566ba28e440 2024-04-21T23:14:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.219+0000 7f934f039740 10 client.16935 fetch_fsmap learned FSMap version 25 2024-04-21T23:14:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.219+0000 7f934f039740 1 -- 172.21.15.44:0/2585794234 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5566ba2a60a0 con 0x5566ba28e440 2024-04-21T23:14:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.220+0000 7f9329ffb640 1 -- 172.21.15.44:0/2585794234 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93380046c0 con 0x5566ba28e440 2024-04-21T23:14:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.220+0000 7f9329ffb640 1 -- 172.21.15.44:0/2585794234 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f933400f040 con 0x5566ba28e440 2024-04-21T23:14:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.220+0000 7f934f039740 10 client.16935 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:14:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.220+0000 7f934f039740 10 client.16935 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.220+0000 7f934f039740 20 client.16935 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.220+0000 7f934f039740 1 --2- 172.21.15.44:0/2585794234 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5566ba2a76d0 0x5566ba2a9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.220+0000 7f934f039740 4 client.16935 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:14:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.220+0000 7f934f039740 1 -- 172.21.15.44:0/2585794234 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566ba1ff0b0 con 0x5566ba2a76d0 2024-04-21T23:14:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.220+0000 7f9349912640 1 --2- 172.21.15.44:0/2585794234 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5566ba2a76d0 0x5566ba2a9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.220+0000 7f9329ffb640 10 client.16935 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.220+0000 7f9349912640 1 --2- 172.21.15.44:0/2585794234 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5566ba2a76d0 0x5566ba2a9ac0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49840643cf5fac04 server_cookie=8283bcc7a3b0f996 in_seq=0 out_seq=0 2024-04-21T23:14:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.221+0000 7f9329ffb640 1 -- 172.21.15.44:0/2585794234 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5566ba1ff0b0 con 0x5566ba2a76d0 2024-04-21T23:14:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.221+0000 7f9329ffb640 10 client.16935 handle_command_reply: tid=0 2024-04-21T23:14:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.221+0000 7f934f039740 1 client.16935 shutdown 2024-04-21T23:14:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.221+0000 7f934f039740 1 -- 172.21.15.44:0/2585794234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba28e440 msgr2=0x5566ba2947a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.221+0000 7f934f039740 1 --2- 172.21.15.44:0/2585794234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba28e440 0x5566ba2947a0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f93340116c0 tx=0x7f9334011b80 comp rx=0 tx=0).stop 2024-04-21T23:14:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.221+0000 7f934f039740 1 -- 172.21.15.44:0/2585794234 shutdown_connections 2024-04-21T23:14:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.221+0000 7f934f039740 1 --2- 172.21.15.44:0/2585794234 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5566ba2a76d0 0x5566ba2a9ac0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.221+0000 7f9329ffb640 0 client.16935 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.221+0000 7f934f039740 1 --2- 172.21.15.44:0/2585794234 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba297210 0x5566ba294ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.221+0000 7f934f039740 1 --2- 172.21.15.44:0/2585794234 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba28e440 0x5566ba2947a0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.221+0000 7f934f039740 1 --2- 172.21.15.44:0/2585794234 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba28db20 0x5566ba294260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.228 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.221+0000 7f934f039740 1 -- 172.21.15.44:0/2585794234 >> 172.21.15.44:0/2585794234 conn(0x5566ba288440 msgr2=0x5566ba28a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:31.228 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.221+0000 7f934f039740 1 -- 172.21.15.44:0/2585794234 shutdown_connections 2024-04-21T23:14:31.228 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.221+0000 7f934f039740 1 -- 172.21.15.44:0/2585794234 wait complete. 2024-04-21T23:14:31.228 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.221+0000 7f934f039740 20 client.16935 trim_cache size 0 max 16384 2024-04-21T23:14:31.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.237+0000 7f934f039740 1 Processor -- start 2024-04-21T23:14:31.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.237+0000 7f934f039740 1 -- start start 2024-04-21T23:14:31.237 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.237+0000 7f934f039740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba28db20 0x5566ba2a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:31.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.237+0000 7f934f039740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba28e440 0x5566ba294b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:31.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.237+0000 7f934f039740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba297210 0x5566ba289ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:31.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.237+0000 7f934f039740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f93441acb90 con 0x5566ba28e440 2024-04-21T23:14:31.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.237+0000 7f934f039740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f93441af410 con 0x5566ba297210 2024-04-21T23:14:31.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.237+0000 7f934f039740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f93441ad200 con 0x5566ba28db20 2024-04-21T23:14:31.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.237+0000 7f9349912640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba28db20 0x5566ba2a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:31.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.237+0000 7f934a113640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba28e440 0x5566ba294b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:31.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.237+0000 7f9349912640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba28db20 0x5566ba2a2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:43804/0 (socket says 172.21.15.44:43804) 2024-04-21T23:14:31.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.237+0000 7f9349912640 1 -- 172.21.15.44:0/4294904166 learned_addr learned my addr 172.21.15.44:0/4294904166 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:31.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.237+0000 7f9349912640 1 --2- 172.21.15.44:0/4294904166 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba297210 0x5566ba289ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:31.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.238+0000 7f9349912640 1 -- 172.21.15.44:0/4294904166 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba297210 msgr2=0x5566ba289ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:31.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.238+0000 7f9349912640 1 --2- 172.21.15.44:0/4294904166 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba297210 0x5566ba289ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.238+0000 7f9349912640 1 -- 172.21.15.44:0/4294904166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba28e440 msgr2=0x5566ba294b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:31.238 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.238+0000 7f9349912640 1 --2- 172.21.15.44:0/4294904166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba28e440 0x5566ba294b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.238+0000 7f9349912640 1 -- 172.21.15.44:0/4294904166 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566ba2a3d90 con 0x5566ba28db20 2024-04-21T23:14:31.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.238+0000 7f934a113640 1 --2- 172.21.15.44:0/4294904166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba28e440 0x5566ba294b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:31.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.238+0000 7f9349912640 1 --2- 172.21.15.44:0/4294904166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba28db20 0x5566ba2a2c40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9338000c00 tx=0x7f9338005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3dc781fe733191bc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:31.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.238+0000 7f932bfff640 1 -- 172.21.15.44:0/4294904166 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9338004c90 con 0x5566ba28db20 2024-04-21T23:14:31.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.238+0000 7f932bfff640 1 -- 172.21.15.44:0/4294904166 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9338054e50 con 0x5566ba28db20 2024-04-21T23:14:31.239 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.238+0000 7f932bfff640 1 -- 172.21.15.44:0/4294904166 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93380052d0 con 0x5566ba28db20 2024-04-21T23:14:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.239+0000 7f934f039740 1 -- 172.21.15.44:0/4294904166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba28db20 msgr2=0x5566ba2a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.239+0000 7f934f039740 1 --2- 172.21.15.44:0/4294904166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba28db20 0x5566ba2a2c40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9338000c00 tx=0x7f9338005b50 comp rx=0 tx=0).stop 2024-04-21T23:14:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.239+0000 7f934f039740 1 -- 172.21.15.44:0/4294904166 shutdown_connections 2024-04-21T23:14:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.239+0000 7f934f039740 1 --2- 172.21.15.44:0/4294904166 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba297210 0x5566ba289ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.239+0000 7f934f039740 1 --2- 172.21.15.44:0/4294904166 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba28e440 0x5566ba294b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.239+0000 7f934f039740 1 --2- 172.21.15.44:0/4294904166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba28db20 0x5566ba2a2c40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.239+0000 7f934f039740 1 -- 172.21.15.44:0/4294904166 >> 172.21.15.44:0/4294904166 conn(0x5566ba288440 msgr2=0x5566ba2897e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.239+0000 7f934f039740 1 -- 172.21.15.44:0/4294904166 shutdown_connections 2024-04-21T23:14:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.239+0000 7f934f039740 1 -- 172.21.15.44:0/4294904166 wait complete. 2024-04-21T23:14:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.239+0000 7f934f039740 1 -- start start 2024-04-21T23:14:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.239+0000 7f934f039740 1 Processor -- start 2024-04-21T23:14:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f934f039740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba28db20 0x5566ba291550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f934f039740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba28e440 0x5566ba291a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f934f039740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba297210 0x5566ba30cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f934f039740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f93441ad8d0 con 0x5566ba297210 2024-04-21T23:14:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f934f039740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f93441aed40 con 0x5566ba28db20 2024-04-21T23:14:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f934f039740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f93441adfa0 con 0x5566ba28e440 2024-04-21T23:14:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f934f039740 10 client.0 _ll_register_callbacks cb 0x7f9334009db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:31.243 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f9349912640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba28db20 0x5566ba291550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:31.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f9349912640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba297210 0x5566ba30cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:31.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f934a113640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba28e440 0x5566ba291a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:31.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f9349912640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba28db20 0x5566ba291550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:36710/0 (socket says 172.21.15.44:36710) 2024-04-21T23:14:31.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f9349912640 1 -- 172.21.15.44:0/2319189161 learned_addr learned my addr 172.21.15.44:0/2319189161 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:31.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f9349912640 1 -- 172.21.15.44:0/2319189161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba28e440 msgr2=0x5566ba291a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:31.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f9349912640 1 --2- 172.21.15.44:0/2319189161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba28e440 0x5566ba291a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f9349912640 1 -- 172.21.15.44:0/2319189161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba297210 msgr2=0x5566ba30cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:31.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f9349912640 1 --2- 172.21.15.44:0/2319189161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba297210 0x5566ba30cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f9349912640 1 -- 172.21.15.44:0/2319189161 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566ba30d060 con 0x5566ba28db20 2024-04-21T23:14:31.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f9349912640 1 --2- 172.21.15.44:0/2319189161 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba28db20 0x5566ba291550 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9338009800 tx=0x7f933802c7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7092d943afa213ba server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:31.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f9329ffb640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:14:31.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f9329ffb640 1 -- 172.21.15.44:0/2319189161 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9338004020 con 0x5566ba28db20 2024-04-21T23:14:31.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f934f039740 1 -- 172.21.15.44:0/2319189161 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9338054e50 con 0x5566ba28db20 2024-04-21T23:14:31.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.240+0000 7f9329ffb640 1 -- 172.21.15.44:0/2319189161 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f93380041c0 con 0x5566ba28db20 2024-04-21T23:14:31.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.241+0000 7f9329ffb640 1 -- 172.21.15.44:0/2319189161 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f93380043c0 con 0x5566ba28db20 2024-04-21T23:14:31.244 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.241+0000 7f934f039740 10 client.26501 fetch_fsmap learned FSMap version 25 2024-04-21T23:14:31.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.241+0000 7f934f039740 1 -- 172.21.15.44:0/2319189161 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5566ba30e270 con 0x5566ba28db20 2024-04-21T23:14:31.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.241+0000 7f9329ffb640 1 -- 172.21.15.44:0/2319189161 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9338054e50 con 0x5566ba28db20 2024-04-21T23:14:31.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.241+0000 7f9329ffb640 1 -- 172.21.15.44:0/2319189161 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f93380534a0 con 0x5566ba28db20 2024-04-21T23:14:31.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.241+0000 7f934f039740 10 client.26501 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:14:31.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.241+0000 7f934f039740 10 client.26501 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:31.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.241+0000 7f934f039740 20 client.26501 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:31.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.241+0000 7f934f039740 1 --2- 172.21.15.44:0/2319189161 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5566ba30e500 0x5566ba3108f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:31.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.241+0000 7f934f039740 4 client.26501 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:14:31.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.241+0000 7f934f039740 1 -- 172.21.15.44:0/2319189161 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5566ba1f9400 con 0x5566ba30e500 2024-04-21T23:14:31.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.241+0000 7f934a113640 1 --2- 172.21.15.44:0/2319189161 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5566ba30e500 0x5566ba3108f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:31.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.242+0000 7f9329ffb640 10 client.26501 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:31.245 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.242+0000 7f934a113640 1 --2- 172.21.15.44:0/2319189161 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5566ba30e500 0x5566ba3108f0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3aeef029cdf7a9e4 server_cookie=1c3e824126b98a0f in_seq=0 out_seq=0 2024-04-21T23:14:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.553+0000 7f9329ffb640 1 -- 172.21.15.44:0/2319189161 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5566ba1f9400 con 0x5566ba30e500 2024-04-21T23:14:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.553+0000 7f9329ffb640 10 client.26501 handle_command_reply: tid=0 2024-04-21T23:14:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.553+0000 7f934f039740 1 client.26501 shutdown 2024-04-21T23:14:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.553+0000 7f934f039740 1 -- 172.21.15.44:0/2319189161 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba28db20 msgr2=0x5566ba291550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.553+0000 7f934f039740 1 --2- 172.21.15.44:0/2319189161 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba28db20 0x5566ba291550 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9338009800 tx=0x7f933802c7a0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.553+0000 7f934f039740 1 -- 172.21.15.44:0/2319189161 shutdown_connections 2024-04-21T23:14:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.553+0000 7f934f039740 1 --2- 172.21.15.44:0/2319189161 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5566ba30e500 0x5566ba3108f0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.553+0000 7f934f039740 1 --2- 172.21.15.44:0/2319189161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5566ba28e440 0x5566ba291a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.553+0000 7f934f039740 1 --2- 172.21.15.44:0/2319189161 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5566ba297210 0x5566ba30cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.553+0000 7f934f039740 1 --2- 172.21.15.44:0/2319189161 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5566ba28db20 0x5566ba291550 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.553+0000 7f934f039740 1 -- 172.21.15.44:0/2319189161 >> 172.21.15.44:0/2319189161 conn(0x5566ba288440 msgr2=0x5566ba2874c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.553+0000 7f9329ffb640 0 client.26501 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:31.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.554+0000 7f934f039740 1 -- 172.21.15.44:0/2319189161 shutdown_connections 2024-04-21T23:14:31.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.554+0000 7f934f039740 1 -- 172.21.15.44:0/2319189161 wait complete. 2024-04-21T23:14:31.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.554+0000 7f934f039740 20 client.26501 trim_cache size 0 max 16384 2024-04-21T23:14:31.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:14:31.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:14:31.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:14:31.555 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:14:31.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.554+0000 7f934c377640 1 -- 172.21.15.44:0/3117338003 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f932c0a2cf0 msgr2=0x7f932c0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:31.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.554+0000 7f934c377640 1 --2- 172.21.15.44:0/3117338003 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f932c0a2cf0 0x7f932c0a51b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f933c09ba50 tx=0x7f933c0911e0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.555+0000 7f934c377640 1 -- 172.21.15.44:0/3117338003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9344106640 msgr2=0x7f934419c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:31.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.555+0000 7f934c377640 1 --2- 172.21.15.44:0/3117338003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9344106640 0x7f934419c8d0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f933c00e970 tx=0x7f933c00ee30 comp rx=0 tx=0).stop 2024-04-21T23:14:31.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.555+0000 7f934c377640 1 -- 172.21.15.44:0/3117338003 shutdown_connections 2024-04-21T23:14:31.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.555+0000 7f934c377640 1 --2- 172.21.15.44:0/3117338003 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f932c0a2cf0 0x7f932c0a51b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.555+0000 7f934c377640 1 --2- 172.21.15.44:0/3117338003 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9344106640 0x7f934419c8d0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.555+0000 7f934c377640 1 --2- 172.21.15.44:0/3117338003 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9344101040 0x7f9344198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.555+0000 7f934c377640 1 --2- 172.21.15.44:0/3117338003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9344100680 0x7f9344198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:31.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.555+0000 7f934c377640 1 -- 172.21.15.44:0/3117338003 >> 172.21.15.44:0/3117338003 conn(0x7f93440777f0 msgr2=0x7f93440ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:31.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.555+0000 7f934c377640 1 -- 172.21.15.44:0/3117338003 shutdown_connections 2024-04-21T23:14:31.556 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:31.555+0000 7f934c377640 1 -- 172.21.15.44:0/3117338003 wait complete. 2024-04-21T23:14:31.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 657420 13.31 MB/sec execute 141 sec latency 132.949 ms 2024-04-21T23:14:32.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:32 smithi044 ceph-mon[27934]: pgmap v175: 129 pgs: 129 active+clean; 77 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 400 op/s 2024-04-21T23:14:32.463 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:32 smithi055 ceph-mon[31117]: pgmap v175: 129 pgs: 129 active+clean; 77 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 400 op/s 2024-04-21T23:14:32.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:32 smithi069 ceph-mon[31816]: pgmap v175: 129 pgs: 129 active+clean; 77 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 400 op/s 2024-04-21T23:14:32.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 660157 13.31 MB/sec execute 142 sec latency 22.186 ms 2024-04-21T23:14:33.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 661633 13.27 MB/sec execute 143 sec latency 51.905 ms 2024-04-21T23:14:34.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:34 smithi044 ceph-mon[27934]: pgmap v176: 129 pgs: 129 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.1 MiB/s wr, 675 op/s 2024-04-21T23:14:34.463 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:34 smithi055 ceph-mon[31117]: pgmap v176: 129 pgs: 129 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.1 MiB/s wr, 675 op/s 2024-04-21T23:14:34.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:34 smithi069 ceph-mon[31816]: pgmap v176: 129 pgs: 129 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.1 MiB/s wr, 675 op/s 2024-04-21T23:14:34.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 664644 13.30 MB/sec execute 144 sec latency 18.583 ms 2024-04-21T23:14:35.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 667575 13.32 MB/sec execute 145 sec latency 71.115 ms 2024-04-21T23:14:36.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:36 smithi044 ceph-mon[27934]: pgmap v177: 129 pgs: 129 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 394 op/s 2024-04-21T23:14:36.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:36 smithi055 ceph-mon[31117]: pgmap v177: 129 pgs: 129 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 394 op/s 2024-04-21T23:14:36.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:36 smithi069 ceph-mon[31816]: pgmap v177: 129 pgs: 129 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 394 op/s 2024-04-21T23:14:36.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 669602 13.28 MB/sec execute 146 sec latency 47.164 ms 2024-04-21T23:14:37.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 672036 13.29 MB/sec execute 147 sec latency 16.404 ms 2024-04-21T23:14:38.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:38 smithi044 ceph-mon[27934]: pgmap v178: 129 pgs: 129 active+clean; 75 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 503 op/s 2024-04-21T23:14:38.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:38 smithi055 ceph-mon[31117]: pgmap v178: 129 pgs: 129 active+clean; 75 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 503 op/s 2024-04-21T23:14:38.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:38 smithi069 ceph-mon[31816]: pgmap v178: 129 pgs: 129 active+clean; 75 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 503 op/s 2024-04-21T23:14:38.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 674963 13.32 MB/sec execute 148 sec latency 83.564 ms 2024-04-21T23:14:39.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 677332 13.29 MB/sec execute 149 sec latency 32.849 ms 2024-04-21T23:14:40.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:40 smithi044 ceph-mon[27934]: pgmap v179: 129 pgs: 129 active+clean; 74 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 397 op/s 2024-04-21T23:14:40.463 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:40 smithi055 ceph-mon[31117]: pgmap v179: 129 pgs: 129 active+clean; 74 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 397 op/s 2024-04-21T23:14:40.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:40 smithi069 ceph-mon[31816]: pgmap v179: 129 pgs: 129 active+clean; 74 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 397 op/s 2024-04-21T23:14:40.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 679561 13.29 MB/sec execute 150 sec latency 32.859 ms 2024-04-21T23:14:41.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 682437 13.31 MB/sec execute 151 sec latency 15.959 ms 2024-04-21T23:14:42.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:42 smithi044 ceph-mon[27934]: pgmap v180: 129 pgs: 129 active+clean; 74 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 390 op/s 2024-04-21T23:14:42.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:42 smithi055 ceph-mon[31117]: pgmap v180: 129 pgs: 129 active+clean; 74 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 390 op/s 2024-04-21T23:14:42.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:42 smithi069 ceph-mon[31816]: pgmap v180: 129 pgs: 129 active+clean; 74 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 390 op/s 2024-04-21T23:14:42.703 INFO:tasks.workunit.client.0.smithi044.stdout: 1 685404 13.34 MB/sec execute 152 sec latency 31.105 ms 2024-04-21T23:14:43.704 INFO:tasks.workunit.client.0.smithi044.stdout: 1 686853 13.30 MB/sec execute 153 sec latency 70.350 ms 2024-04-21T23:14:44.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:44 smithi044 ceph-mon[27934]: pgmap v181: 129 pgs: 129 active+clean; 82 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.6 MiB/s wr, 512 op/s 2024-04-21T23:14:44.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:44 smithi055 ceph-mon[31117]: pgmap v181: 129 pgs: 129 active+clean; 82 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.6 MiB/s wr, 512 op/s 2024-04-21T23:14:44.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:44 smithi069 ceph-mon[31816]: pgmap v181: 129 pgs: 129 active+clean; 82 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.6 MiB/s wr, 512 op/s 2024-04-21T23:14:44.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.645+0000 7f72e622e640 1 -- 172.21.15.44:0/3864195349 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72e010e0e0 msgr2=0x7f72e01104d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:44.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.645+0000 7f72e622e640 1 --2- 172.21.15.44:0/3864195349 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72e010e0e0 0x7f72e01104d0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009ca0 tx=0x7f72d802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.645+0000 7f72e622e640 1 -- 172.21.15.44:0/3864195349 shutdown_connections 2024-04-21T23:14:44.662 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.645+0000 7f72e622e640 1 --2- 172.21.15.44:0/3864195349 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72e0100e50 0x7f72e010dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.645+0000 7f72e622e640 1 --2- 172.21.15.44:0/3864195349 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72e010e0e0 0x7f72e01104d0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.645+0000 7f72e622e640 1 --2- 172.21.15.44:0/3864195349 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72e0100530 0x7f72e0100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.645+0000 7f72e622e640 1 -- 172.21.15.44:0/3864195349 >> 172.21.15.44:0/3864195349 conn(0x7f72e0077760 msgr2=0x7f72e0077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:44.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.645+0000 7f72e622e640 1 -- 172.21.15.44:0/3864195349 shutdown_connections 2024-04-21T23:14:44.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.645+0000 7f72e622e640 1 -- 172.21.15.44:0/3864195349 wait complete. 2024-04-21T23:14:44.663 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.646+0000 7f72e622e640 1 Processor -- start 2024-04-21T23:14:44.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.646+0000 7f72e622e640 1 -- start start 2024-04-21T23:14:44.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.646+0000 7f72e622e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72e0100530 0x7f72e019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:44.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.646+0000 7f72e622e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72e0100e50 0x7f72e019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:44.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.646+0000 7f72e622e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72e010e0e0 0x7f72e019eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:44.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.646+0000 7f72e622e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f72e0113d40 con 0x7f72e0100e50 2024-04-21T23:14:44.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.646+0000 7f72e622e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f72e0113bc0 con 0x7f72e010e0e0 2024-04-21T23:14:44.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.646+0000 7f72e622e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72e0113ec0 con 0x7f72e0100530 2024-04-21T23:14:44.664 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.646+0000 7f72deffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72e0100e50 0x7f72e019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:44.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.646+0000 7f72deffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72e0100e50 0x7f72e019a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42444/0 (socket says 172.21.15.44:42444) 2024-04-21T23:14:44.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.646+0000 7f72deffd640 1 -- 172.21.15.44:0/1667766195 learned_addr learned my addr 172.21.15.44:0/1667766195 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:44.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.646+0000 7f72dffff640 1 --2- 172.21.15.44:0/1667766195 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72e010e0e0 0x7f72e019eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:44.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.646+0000 7f72df7fe640 1 --2- 172.21.15.44:0/1667766195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72e0100530 0x7f72e019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:44.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.647+0000 7f72deffd640 1 -- 172.21.15.44:0/1667766195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72e0100530 msgr2=0x7f72e019a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:44.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.647+0000 7f72deffd640 1 --2- 172.21.15.44:0/1667766195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72e0100530 0x7f72e019a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.647+0000 7f72deffd640 1 -- 172.21.15.44:0/1667766195 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72e010e0e0 msgr2=0x7f72e019eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:44.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.647+0000 7f72deffd640 1 --2- 172.21.15.44:0/1667766195 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72e010e0e0 0x7f72e019eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.665 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.647+0000 7f72deffd640 1 -- 172.21.15.44:0/1667766195 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e019f1d0 con 0x7f72e0100e50 2024-04-21T23:14:44.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.647+0000 7f72deffd640 1 --2- 172.21.15.44:0/1667766195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72e0100e50 0x7f72e019a720 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f72d400d770 tx=0x7f72d400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:44.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.647+0000 7f72dcff9640 1 -- 172.21.15.44:0/1667766195 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d4017070 con 0x7f72e0100e50 2024-04-21T23:14:44.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.647+0000 7f72dcff9640 1 -- 172.21.15.44:0/1667766195 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72d4005500 con 0x7f72e0100e50 2024-04-21T23:14:44.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.647+0000 7f72e622e640 1 -- 172.21.15.44:0/1667766195 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72e019f4c0 con 0x7f72e0100e50 2024-04-21T23:14:44.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.647+0000 7f72dcff9640 1 -- 172.21.15.44:0/1667766195 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d4004e00 con 0x7f72e0100e50 2024-04-21T23:14:44.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.647+0000 7f72e622e640 1 -- 172.21.15.44:0/1667766195 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72e01a6d50 con 0x7f72e0100e50 2024-04-21T23:14:44.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.649+0000 7f72dcff9640 1 -- 172.21.15.44:0/1667766195 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f72d4004fa0 con 0x7f72e0100e50 2024-04-21T23:14:44.666 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.649+0000 7f72e8e14740 1 Processor -- start 2024-04-21T23:14:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72e8e14740 1 -- start start 2024-04-21T23:14:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72e8e14740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454d2b20 0x5598454d2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72e8e14740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d3440 0x5598454dbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72e8e14740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454dc210 0x5598454de600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72e8e14740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x559845436b70 con 0x5598454dc210 2024-04-21T23:14:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72e8e14740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5598453df470 con 0x5598454d2b20 2024-04-21T23:14:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72e8e14740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559845432b30 con 0x5598454d3440 2024-04-21T23:14:44.667 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72dcff9640 1 --2- 172.21.15.44:0/1667766195 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f72c80a2cf0 0x7f72c80a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:44.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72deffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454dc210 0x5598454de600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:44.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72dffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454d2b20 0x5598454d2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:44.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d3440 0x5598454dbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:44.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72deffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454dc210 0x5598454de600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42452/0 (socket says 172.21.15.44:42452) 2024-04-21T23:14:44.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72dffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454d2b20 0x5598454d2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:58926/0 (socket says 172.21.15.44:58926) 2024-04-21T23:14:44.668 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72deffd640 1 -- 172.21.15.44:0/248321269 learned_addr learned my addr 172.21.15.44:0/248321269 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:44.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72dffff640 1 --2- 172.21.15.44:0/1667766195 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f72c80a2cf0 0x7f72c80a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:44.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72dcff9640 1 -- 172.21.15.44:0/1667766195 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f72d40cf170 con 0x7f72e0100e50 2024-04-21T23:14:44.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72df7fe640 1 -- 172.21.15.44:0/248321269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454d2b20 msgr2=0x5598454d2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:44.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72df7fe640 1 --2- 172.21.15.44:0/248321269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454d2b20 0x5598454d2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72df7fe640 1 -- 172.21.15.44:0/248321269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454dc210 msgr2=0x5598454de600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:44.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72df7fe640 1 --2- 172.21.15.44:0/248321269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454dc210 0x5598454de600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72df7fe640 1 -- 172.21.15.44:0/248321269 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55984514b210 con 0x5598454d3440 2024-04-21T23:14:44.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72deffd640 1 --2- 172.21.15.44:0/248321269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454dc210 0x5598454de600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:44.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.650+0000 7f72df7fe640 1 --2- 172.21.15.44:0/248321269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d3440 0x5598454dbcd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009c20 tx=0x7f72d0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75786c1994220c9c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:44.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.651+0000 7f72dffff640 1 --2- 172.21.15.44:0/1667766195 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f72c80a2cf0 0x7f72c80a51b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f72d802c700 tx=0x7f72d8005c50 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:44.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.651+0000 7f72c5ffb640 1 -- 172.21.15.44:0/248321269 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d0004050 con 0x5598454d3440 2024-04-21T23:14:44.669 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.651+0000 7f72c5ffb640 1 -- 172.21.15.44:0/248321269 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72d0005500 con 0x5598454d3440 2024-04-21T23:14:44.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.651+0000 7f72c5ffb640 1 -- 172.21.15.44:0/248321269 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d0004d90 con 0x5598454d3440 2024-04-21T23:14:44.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.651+0000 7f72e8e14740 1 -- 172.21.15.44:0/248321269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d3440 msgr2=0x5598454dbcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:44.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.651+0000 7f72e8e14740 1 --2- 172.21.15.44:0/248321269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d3440 0x5598454dbcd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009c20 tx=0x7f72d0011810 comp rx=0 tx=0).stop 2024-04-21T23:14:44.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.651+0000 7f72e8e14740 1 -- 172.21.15.44:0/248321269 shutdown_connections 2024-04-21T23:14:44.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.651+0000 7f72e8e14740 1 --2- 172.21.15.44:0/248321269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d3440 0x5598454dbcd0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.651+0000 7f72e8e14740 1 --2- 172.21.15.44:0/248321269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454dc210 0x5598454de600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.651+0000 7f72e8e14740 1 --2- 172.21.15.44:0/248321269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454d2b20 0x5598454d2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.651+0000 7f72e8e14740 1 -- 172.21.15.44:0/248321269 >> 172.21.15.44:0/248321269 conn(0x5598454cd440 msgr2=0x5598454cd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:44.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.651+0000 7f72e8e14740 1 -- 172.21.15.44:0/248321269 shutdown_connections 2024-04-21T23:14:44.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.651+0000 7f72e8e14740 1 -- 172.21.15.44:0/248321269 wait complete. 2024-04-21T23:14:44.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.652+0000 7f72e8e14740 1 -- start start 2024-04-21T23:14:44.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.652+0000 7f72e8e14740 1 Processor -- start 2024-04-21T23:14:44.670 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.652+0000 7f72e8e14740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454d2b20 0x5598454d9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.652+0000 7f72e8e14740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d3440 0x5598454d9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.652+0000 7f72e8e14740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454dc210 0x5598454d9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.652+0000 7f72e8e14740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5598454dee40 con 0x5598454d2b20 2024-04-21T23:14:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.652+0000 7f72e8e14740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5598453f2a10 con 0x5598454dc210 2024-04-21T23:14:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.652+0000 7f72e8e14740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5598454defc0 con 0x5598454d3440 2024-04-21T23:14:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.652+0000 7f72e8e14740 10 client.0 _ll_register_callbacks cb 0x55984542db60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.652+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454d2b20 0x5598454d9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.652+0000 7f72deffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d3440 0x5598454d9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.652+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454d2b20 0x5598454d9210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42454/0 (socket says 172.21.15.44:42454) 2024-04-21T23:14:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.652+0000 7f72df7fe640 1 -- 172.21.15.44:0/1487342689 learned_addr learned my addr 172.21.15.44:0/1487342689 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.652+0000 7f72dffff640 1 --2- 172.21.15.44:0/1487342689 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454dc210 0x5598454d9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.653+0000 7f72df7fe640 1 -- 172.21.15.44:0/1487342689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d3440 msgr2=0x5598454d9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.653+0000 7f72df7fe640 1 --2- 172.21.15.44:0/1487342689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d3440 0x5598454d9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.653+0000 7f72df7fe640 1 -- 172.21.15.44:0/1487342689 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454dc210 msgr2=0x5598454d9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:44.671 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.653+0000 7f72df7fe640 1 --2- 172.21.15.44:0/1487342689 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454dc210 0x5598454d9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.653+0000 7f72df7fe640 1 -- 172.21.15.44:0/1487342689 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559845217620 con 0x5598454d2b20 2024-04-21T23:14:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.653+0000 7f72deffd640 1 --2- 172.21.15.44:0/1487342689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d3440 0x5598454d9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.653+0000 7f72df7fe640 1 --2- 172.21.15.44:0/1487342689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454d2b20 0x5598454d9210 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f72d0004dc0 tx=0x7f72d0004df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8484efefd5373d34 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.653+0000 7f72beffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:14:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.653+0000 7f72beffd640 1 -- 172.21.15.44:0/1487342689 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d0029070 con 0x5598454d2b20 2024-04-21T23:14:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.653+0000 7f72beffd640 1 -- 172.21.15.44:0/1487342689 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f72d0005220 con 0x5598454d2b20 2024-04-21T23:14:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.653+0000 7f72e8e14740 1 -- 172.21.15.44:0/1487342689 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72d0005500 con 0x5598454d2b20 2024-04-21T23:14:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.653+0000 7f72beffd640 1 -- 172.21.15.44:0/1487342689 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f72d0035050 con 0x5598454d2b20 2024-04-21T23:14:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.653+0000 7f72e8e14740 10 client.16953 fetch_fsmap learned FSMap version 25 2024-04-21T23:14:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.653+0000 7f72e8e14740 1 -- 172.21.15.44:0/1487342689 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5598454eb0a0 con 0x5598454d2b20 2024-04-21T23:14:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.653+0000 7f72beffd640 1 -- 172.21.15.44:0/1487342689 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72d0005500 con 0x5598454d2b20 2024-04-21T23:14:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.654+0000 7f72beffd640 1 -- 172.21.15.44:0/1487342689 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f72d0023430 con 0x5598454d2b20 2024-04-21T23:14:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.654+0000 7f72e8e14740 10 client.16953 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:14:44.672 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.654+0000 7f72e8e14740 10 client.16953 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.654+0000 7f72e8e14740 20 client.16953 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.654+0000 7f72e8e14740 1 --2- 172.21.15.44:0/1487342689 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5598454ec4b0 0x5598454ee8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.654+0000 7f72e8e14740 4 client.16953 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:14:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.654+0000 7f72e8e14740 1 -- 172.21.15.44:0/1487342689 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598454440b0 con 0x5598454ec4b0 2024-04-21T23:14:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.654+0000 7f72deffd640 1 --2- 172.21.15.44:0/1487342689 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5598454ec4b0 0x5598454ee8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.654+0000 7f72deffd640 1 --2- 172.21.15.44:0/1487342689 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5598454ec4b0 0x5598454ee8a0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98853f5045eef40 server_cookie=23062b24c7dccd46 in_seq=0 out_seq=0 2024-04-21T23:14:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.654+0000 7f72beffd640 10 client.16953 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.656+0000 7f72beffd640 1 -- 172.21.15.44:0/1487342689 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5598454440b0 con 0x5598454ec4b0 2024-04-21T23:14:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.656+0000 7f72beffd640 10 client.16953 handle_command_reply: tid=0 2024-04-21T23:14:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.656+0000 7f72e8e14740 1 client.16953 shutdown 2024-04-21T23:14:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.657+0000 7f72e8e14740 1 -- 172.21.15.44:0/1487342689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454d2b20 msgr2=0x5598454d9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.657+0000 7f72e8e14740 1 --2- 172.21.15.44:0/1487342689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454d2b20 0x5598454d9210 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f72d0004dc0 tx=0x7f72d0004df0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.657+0000 7f72e8e14740 1 -- 172.21.15.44:0/1487342689 shutdown_connections 2024-04-21T23:14:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.657+0000 7f72e8e14740 1 --2- 172.21.15.44:0/1487342689 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5598454ec4b0 0x5598454ee8a0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.673 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.657+0000 7f72e8e14740 1 --2- 172.21.15.44:0/1487342689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d3440 0x5598454d9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.657+0000 7f72e8e14740 1 --2- 172.21.15.44:0/1487342689 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454dc210 0x5598454d9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.657+0000 7f72e8e14740 1 --2- 172.21.15.44:0/1487342689 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454d2b20 0x5598454d9210 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.657+0000 7f72e8e14740 1 -- 172.21.15.44:0/1487342689 >> 172.21.15.44:0/1487342689 conn(0x5598454cd440 msgr2=0x5598454cf190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:44.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.657+0000 7f72beffd640 0 client.16953 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:44.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.657+0000 7f72e8e14740 1 -- 172.21.15.44:0/1487342689 shutdown_connections 2024-04-21T23:14:44.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.657+0000 7f72e8e14740 1 -- 172.21.15.44:0/1487342689 wait complete. 2024-04-21T23:14:44.674 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.657+0000 7f72e8e14740 20 client.16953 trim_cache size 0 max 16384 2024-04-21T23:14:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.672+0000 7f72e8e14740 1 Processor -- start 2024-04-21T23:14:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.672+0000 7f72e8e14740 1 -- start start 2024-04-21T23:14:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.672+0000 7f72e8e14740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d2b20 0x5598454db940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.672+0000 7f72e8e14740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454d3440 0x5598454e7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.672+0000 7f72e8e14740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454dc210 0x5598454d8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.672+0000 7f72e8e14740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f72e01aedd0 con 0x5598454dc210 2024-04-21T23:14:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.672+0000 7f72e8e14740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f72e01b1650 con 0x5598454d3440 2024-04-21T23:14:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.672+0000 7f72e8e14740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72e01af440 con 0x5598454d2b20 2024-04-21T23:14:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.672+0000 7f72dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454dc210 0x5598454d8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.672+0000 7f72dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454dc210 0x5598454d8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42458/0 (socket says 172.21.15.44:42458) 2024-04-21T23:14:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.672+0000 7f72dffff640 1 -- 172.21.15.44:0/4109001476 learned_addr learned my addr 172.21.15.44:0/4109001476 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.672+0000 7f72deffd640 1 --2- 172.21.15.44:0/4109001476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454d3440 0x5598454e7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.672+0000 7f72df7fe640 1 --2- 172.21.15.44:0/4109001476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d2b20 0x5598454db940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.672+0000 7f72dffff640 1 -- 172.21.15.44:0/4109001476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d2b20 msgr2=0x5598454db940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.672+0000 7f72dffff640 1 --2- 172.21.15.44:0/4109001476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d2b20 0x5598454db940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.672+0000 7f72dffff640 1 -- 172.21.15.44:0/4109001476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454d3440 msgr2=0x5598454e7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.672+0000 7f72dffff640 1 --2- 172.21.15.44:0/4109001476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454d3440 0x5598454e7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.672+0000 7f72dffff640 1 -- 172.21.15.44:0/4109001476 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598454d7780 con 0x5598454dc210 2024-04-21T23:14:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.673+0000 7f72dffff640 1 --2- 172.21.15.44:0/4109001476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454dc210 0x5598454d8210 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009430 tx=0x7f72d802e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=222991e4a5acbfe4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.673+0000 7f72c57fa640 1 -- 172.21.15.44:0/4109001476 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d8039040 con 0x5598454dc210 2024-04-21T23:14:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.673+0000 7f72c57fa640 1 -- 172.21.15.44:0/4109001476 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72d8055070 con 0x5598454dc210 2024-04-21T23:14:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.673+0000 7f72c57fa640 1 -- 172.21.15.44:0/4109001476 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d80506c0 con 0x5598454dc210 2024-04-21T23:14:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.673+0000 7f72e8e14740 1 -- 172.21.15.44:0/4109001476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454dc210 msgr2=0x5598454d8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.673+0000 7f72e8e14740 1 --2- 172.21.15.44:0/4109001476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454dc210 0x5598454d8210 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009430 tx=0x7f72d802e040 comp rx=0 tx=0).stop 2024-04-21T23:14:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.673+0000 7f72e8e14740 1 -- 172.21.15.44:0/4109001476 shutdown_connections 2024-04-21T23:14:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.673+0000 7f72e8e14740 1 --2- 172.21.15.44:0/4109001476 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454dc210 0x5598454d8210 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.673+0000 7f72e8e14740 1 --2- 172.21.15.44:0/4109001476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454d3440 0x5598454e7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.673+0000 7f72e8e14740 1 --2- 172.21.15.44:0/4109001476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d2b20 0x5598454db940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.673+0000 7f72e8e14740 1 -- 172.21.15.44:0/4109001476 >> 172.21.15.44:0/4109001476 conn(0x5598454cd440 msgr2=0x5598454d7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.673+0000 7f72e8e14740 1 -- 172.21.15.44:0/4109001476 shutdown_connections 2024-04-21T23:14:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.673+0000 7f72e8e14740 1 -- 172.21.15.44:0/4109001476 wait complete. 2024-04-21T23:14:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.674+0000 7f72e8e14740 1 -- start start 2024-04-21T23:14:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.674+0000 7f72e8e14740 1 Processor -- start 2024-04-21T23:14:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.674+0000 7f72e8e14740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454d2b20 0x5598454e4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.674+0000 7f72e8e14740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d3440 0x5598454d5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.674+0000 7f72e8e14740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454dc210 0x5598454d5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.674+0000 7f72e8e14740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f72e01afb10 con 0x5598454d2b20 2024-04-21T23:14:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.674+0000 7f72e8e14740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f72e01b0f80 con 0x5598454dc210 2024-04-21T23:14:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.674+0000 7f72e8e14740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72e01b01e0 con 0x5598454d3440 2024-04-21T23:14:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.674+0000 7f72e8e14740 10 client.0 _ll_register_callbacks cb 0x7f72d4010f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.674+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454d2b20 0x5598454e4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.674+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454d2b20 0x5598454e4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42464/0 (socket says 172.21.15.44:42464) 2024-04-21T23:14:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.674+0000 7f72df7fe640 1 -- 172.21.15.44:0/1350569330 learned_addr learned my addr 172.21.15.44:0/1350569330 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.674+0000 7f72deffd640 1 --2- 172.21.15.44:0/1350569330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d3440 0x5598454d5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.674+0000 7f72dffff640 1 --2- 172.21.15.44:0/1350569330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454dc210 0x5598454d5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.675+0000 7f72df7fe640 1 -- 172.21.15.44:0/1350569330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d3440 msgr2=0x5598454d5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.675+0000 7f72df7fe640 1 --2- 172.21.15.44:0/1350569330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d3440 0x5598454d5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.675+0000 7f72df7fe640 1 -- 172.21.15.44:0/1350569330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454dc210 msgr2=0x5598454d5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.675+0000 7f72df7fe640 1 --2- 172.21.15.44:0/1350569330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454dc210 0x5598454d5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.675+0000 7f72df7fe640 1 -- 172.21.15.44:0/1350569330 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598454d6610 con 0x5598454d2b20 2024-04-21T23:14:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.675+0000 7f72deffd640 1 --2- 172.21.15.44:0/1350569330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d3440 0x5598454d5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.675+0000 7f72df7fe640 1 --2- 172.21.15.44:0/1350569330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454d2b20 0x5598454e4470 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f72d0023970 tx=0x7f72d0005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48f7c1615f941c21 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.675+0000 7f72beffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:14:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.675+0000 7f72beffd640 1 -- 172.21.15.44:0/1350569330 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d003e060 con 0x5598454d2b20 2024-04-21T23:14:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.675+0000 7f72beffd640 1 -- 172.21.15.44:0/1350569330 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9445+0+0 (secure 0 0 0) 0x7f72d00239f0 con 0x5598454d2b20 2024-04-21T23:14:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.675+0000 7f72e8e14740 1 -- 172.21.15.44:0/1350569330 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72d8055070 con 0x5598454d2b20 2024-04-21T23:14:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.675+0000 7f72beffd640 1 -- 172.21.15.44:0/1350569330 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f72d0045050 con 0x5598454d2b20 2024-04-21T23:14:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.675+0000 7f72e8e14740 10 client.16959 fetch_fsmap learned FSMap version 25 2024-04-21T23:14:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.675+0000 7f72e8e14740 1 -- 172.21.15.44:0/1350569330 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559845552720 con 0x5598454d2b20 2024-04-21T23:14:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.675+0000 7f72beffd640 1 -- 172.21.15.44:0/1350569330 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72d8055070 con 0x5598454d2b20 2024-04-21T23:14:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.676+0000 7f72beffd640 1 -- 172.21.15.44:0/1350569330 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 25) ==== 8330+0+0 (secure 0 0 0) 0x7f72d0005220 con 0x5598454d2b20 2024-04-21T23:14:44.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.676+0000 7f72e8e14740 10 client.16959 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T23:14:44.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.676+0000 7f72e8e14740 10 client.16959 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:44.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.676+0000 7f72e8e14740 20 client.16959 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:44.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.676+0000 7f72e8e14740 1 --2- 172.21.15.44:0/1350569330 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x559845552e40 0x559845555260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:44.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.676+0000 7f72e8e14740 4 client.16959 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:14:44.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.676+0000 7f72e8e14740 1 -- 172.21.15.44:0/1350569330 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55984543e400 con 0x559845552e40 2024-04-21T23:14:44.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.676+0000 7f72deffd640 1 --2- 172.21.15.44:0/1350569330 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x559845552e40 0x559845555260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:44.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.676+0000 7f72beffd640 10 client.16959 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:44.684 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.676+0000 7f72deffd640 1 --2- 172.21.15.44:0/1350569330 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x559845552e40 0x559845555260 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b79a4078819715da server_cookie=177d27425544ba78 in_seq=0 out_seq=0 2024-04-21T23:14:44.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 689336 13.30 MB/sec execute 154 sec latency 26.905 ms 2024-04-21T23:14:44.985 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.986+0000 7f72beffd640 1 -- 172.21.15.44:0/1350569330 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55984543e400 con 0x559845552e40 2024-04-21T23:14:44.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.986+0000 7f72beffd640 10 client.16959 handle_command_reply: tid=0 2024-04-21T23:14:44.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.986+0000 7f72e8e14740 1 client.16959 shutdown 2024-04-21T23:14:44.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.986+0000 7f72e8e14740 1 -- 172.21.15.44:0/1350569330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454d2b20 msgr2=0x5598454e4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:44.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.986+0000 7f72e8e14740 1 --2- 172.21.15.44:0/1350569330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454d2b20 0x5598454e4470 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f72d0023970 tx=0x7f72d0005d30 comp rx=0 tx=0).stop 2024-04-21T23:14:44.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.986+0000 7f72e8e14740 1 -- 172.21.15.44:0/1350569330 shutdown_connections 2024-04-21T23:14:44.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.987+0000 7f72e8e14740 1 --2- 172.21.15.44:0/1350569330 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x559845552e40 0x559845555260 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.987+0000 7f72e8e14740 1 --2- 172.21.15.44:0/1350569330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5598454d3440 0x5598454d5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.987+0000 7f72beffd640 0 client.16959 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:44.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.987+0000 7f72e8e14740 1 --2- 172.21.15.44:0/1350569330 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5598454dc210 0x5598454d5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.987+0000 7f72e8e14740 1 --2- 172.21.15.44:0/1350569330 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5598454d2b20 0x5598454e4470 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.987+0000 7f72e8e14740 1 -- 172.21.15.44:0/1350569330 >> 172.21.15.44:0/1350569330 conn(0x5598454cd440 msgr2=0x5598454d8390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:44.986 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.987+0000 7f72e8e14740 1 -- 172.21.15.44:0/1350569330 shutdown_connections 2024-04-21T23:14:44.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.987+0000 7f72e8e14740 1 -- 172.21.15.44:0/1350569330 wait complete. 2024-04-21T23:14:44.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.987+0000 7f72e8e14740 20 client.16959 trim_cache size 0 max 16384 2024-04-21T23:14:44.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:14:44.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:14:44.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:14:44.987 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:14:44.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.988+0000 7f72e622e640 1 -- 172.21.15.44:0/1667766195 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f72c80a2cf0 msgr2=0x7f72c80a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:44.988 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.988+0000 7f72e622e640 1 --2- 172.21.15.44:0/1667766195 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f72c80a2cf0 0x7f72c80a51b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f72d802c700 tx=0x7f72d8005c50 comp rx=0 tx=0).stop 2024-04-21T23:14:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.988+0000 7f72e622e640 1 -- 172.21.15.44:0/1667766195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72e0100e50 msgr2=0x7f72e019a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.988+0000 7f72e622e640 1 --2- 172.21.15.44:0/1667766195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72e0100e50 0x7f72e019a720 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f72d400d770 tx=0x7f72d400dc30 comp rx=0 tx=0).stop 2024-04-21T23:14:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.988+0000 7f72e622e640 1 -- 172.21.15.44:0/1667766195 shutdown_connections 2024-04-21T23:14:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.988+0000 7f72e622e640 1 --2- 172.21.15.44:0/1667766195 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f72c80a2cf0 0x7f72c80a51b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.988+0000 7f72e622e640 1 --2- 172.21.15.44:0/1667766195 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72e010e0e0 0x7f72e019eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.988+0000 7f72e622e640 1 --2- 172.21.15.44:0/1667766195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f72e0100e50 0x7f72e019a720 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.988+0000 7f72e622e640 1 --2- 172.21.15.44:0/1667766195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72e0100530 0x7f72e019a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.988+0000 7f72e622e640 1 -- 172.21.15.44:0/1667766195 >> 172.21.15.44:0/1667766195 conn(0x7f72e0077760 msgr2=0x7f72e010f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.989+0000 7f72e622e640 1 -- 172.21.15.44:0/1667766195 shutdown_connections 2024-04-21T23:14:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:44.989+0000 7f72e622e640 1 -- 172.21.15.44:0/1667766195 wait complete. 2024-04-21T23:14:45.464 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:14:45 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: ignoring --setuser ceph since I am not root 2024-04-21T23:14:45.464 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:14:45 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: ignoring --setgroup ceph since I am not root 2024-04-21T23:14:45.464 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:14:45 smithi069 ceph-mds[63204]: -- 172.21.15.69:0/303943677 <== mon.1 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560b9897da40 con 0x560b98950800 2024-04-21T23:14:45.464 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:14:45 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: starting mds.i at 2024-04-21T23:14:45.702 INFO:tasks.workunit.client.0.smithi044.stdout: 1 691344 13.26 MB/sec execute 155 sec latency 120.675 ms 2024-04-21T23:14:46.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:46 smithi044 ceph-mon[27934]: pgmap v182: 129 pgs: 129 active+clean; 82 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 237 op/s 2024-04-21T23:14:46.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:46 smithi044 ceph-mon[27934]: daemon mds.i restarted 2024-04-21T23:14:46.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:46 smithi055 ceph-mon[31117]: pgmap v182: 129 pgs: 129 active+clean; 82 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 237 op/s 2024-04-21T23:14:46.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:46 smithi055 ceph-mon[31117]: daemon mds.i restarted 2024-04-21T23:14:46.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:46 smithi069 ceph-mon[31816]: pgmap v182: 129 pgs: 129 active+clean; 82 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 237 op/s 2024-04-21T23:14:46.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:46 smithi069 ceph-mon[31816]: daemon mds.i restarted 2024-04-21T23:14:46.703 INFO:tasks.workunit.client.0.smithi044.stdout: 1 693436 13.26 MB/sec execute 156 sec latency 18.363 ms 2024-04-21T23:14:47.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:47 smithi044 ceph-mon[27934]: osdmap e98: 12 total, 12 up, 12 in 2024-04-21T23:14:47.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:47 smithi044 ceph-mon[27934]: mds.? [v2:172.21.15.69:6838/404328686,v1:172.21.15.69:6839/404328686] up:boot 2024-04-21T23:14:47.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:47 smithi044 ceph-mon[27934]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:14:47.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:47 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:14:47.436 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:47 smithi044 ceph-mon[27934]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:14:47.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:47 smithi055 ceph-mon[31117]: osdmap e98: 12 total, 12 up, 12 in 2024-04-21T23:14:47.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:47 smithi055 ceph-mon[31117]: mds.? [v2:172.21.15.69:6838/404328686,v1:172.21.15.69:6839/404328686] up:boot 2024-04-21T23:14:47.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:47 smithi055 ceph-mon[31117]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:14:47.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:47 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:14:47.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:47 smithi055 ceph-mon[31117]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:14:47.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:47 smithi069 ceph-mon[31816]: osdmap e98: 12 total, 12 up, 12 in 2024-04-21T23:14:47.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:47 smithi069 ceph-mon[31816]: mds.? [v2:172.21.15.69:6838/404328686,v1:172.21.15.69:6839/404328686] up:boot 2024-04-21T23:14:47.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:47 smithi069 ceph-mon[31816]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:14:47.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:47 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:14:47.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:47 smithi069 ceph-mon[31816]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:14:47.703 INFO:tasks.workunit.client.0.smithi044.stdout: 1 696487 13.29 MB/sec execute 157 sec latency 34.527 ms 2024-04-21T23:14:48.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:48 smithi055 ceph-mon[31117]: pgmap v184: 129 pgs: 129 active+clean; 68 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.0 MiB/s wr, 456 op/s 2024-04-21T23:14:48.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:48 smithi069 ceph-mon[31816]: pgmap v184: 129 pgs: 129 active+clean; 68 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.0 MiB/s wr, 456 op/s 2024-04-21T23:14:48.522 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:14:48.522 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:14:48.564 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:48 smithi044 ceph-mon[27934]: pgmap v184: 129 pgs: 129 active+clean; 68 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.0 MiB/s wr, 456 op/s 2024-04-21T23:14:48.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.662+0000 7faadb682640 1 -- 172.21.15.44:0/680539302 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faad41032d0 msgr2=0x7faad410b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:48.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.662+0000 7faadb682640 1 --2- 172.21.15.44:0/680539302 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faad41032d0 0x7faad410b7b0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7faac4009900 tx=0x7faac402bcd0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.662+0000 7faadb682640 1 -- 172.21.15.44:0/680539302 shutdown_connections 2024-04-21T23:14:48.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.662+0000 7faadb682640 1 --2- 172.21.15.44:0/680539302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faad410bcf0 0x7faad410e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.662+0000 7faadb682640 1 --2- 172.21.15.44:0/680539302 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faad41032d0 0x7faad410b7b0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.662+0000 7faadb682640 1 --2- 172.21.15.44:0/680539302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faad41029b0 0x7faad4102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.662+0000 7faadb682640 1 -- 172.21.15.44:0/680539302 >> 172.21.15.44:0/680539302 conn(0x7faad40777f0 msgr2=0x7faad4077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:48.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.662+0000 7faadb682640 1 -- 172.21.15.44:0/680539302 shutdown_connections 2024-04-21T23:14:48.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.662+0000 7faadb682640 1 -- 172.21.15.44:0/680539302 wait complete. 2024-04-21T23:14:48.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.663+0000 7faadb682640 1 Processor -- start 2024-04-21T23:14:48.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.663+0000 7faadb682640 1 -- start start 2024-04-21T23:14:48.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.663+0000 7faadb682640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faad41029b0 0x7faad4100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.663+0000 7faadb682640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faad41032d0 0x7faad4100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.663+0000 7faadb682640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faad410bcf0 0x7faad4101080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.663+0000 7faadb682640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faad4111b40 con 0x7faad41029b0 2024-04-21T23:14:48.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.663+0000 7faadb682640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7faad41119c0 con 0x7faad410bcf0 2024-04-21T23:14:48.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.663+0000 7faadb682640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faad4111cc0 con 0x7faad41032d0 2024-04-21T23:14:48.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.663+0000 7faad941e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faad41029b0 0x7faad4100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.663+0000 7faad941e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faad41029b0 0x7faad4100600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42476/0 (socket says 172.21.15.44:42476) 2024-04-21T23:14:48.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.663+0000 7faad941e640 1 -- 172.21.15.44:0/2191423500 learned_addr learned my addr 172.21.15.44:0/2191423500 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:48.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.663+0000 7faad9c1f640 1 --2- 172.21.15.44:0/2191423500 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faad410bcf0 0x7faad4101080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.664+0000 7faad8c1d640 1 --2- 172.21.15.44:0/2191423500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faad41032d0 0x7faad4100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.664+0000 7faad941e640 1 -- 172.21.15.44:0/2191423500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faad41032d0 msgr2=0x7faad4100b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:48.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.664+0000 7faad941e640 1 --2- 172.21.15.44:0/2191423500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faad41032d0 0x7faad4100b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.664+0000 7faad941e640 1 -- 172.21.15.44:0/2191423500 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faad410bcf0 msgr2=0x7faad4101080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:48.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.664+0000 7faad941e640 1 --2- 172.21.15.44:0/2191423500 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faad410bcf0 0x7faad4101080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.664+0000 7faad941e640 1 -- 172.21.15.44:0/2191423500 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad41a64e0 con 0x7faad41029b0 2024-04-21T23:14:48.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.664+0000 7faad9c1f640 1 --2- 172.21.15.44:0/2191423500 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faad410bcf0 0x7faad4101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:48.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.664+0000 7faad941e640 1 --2- 172.21.15.44:0/2191423500 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faad41029b0 0x7faad4100600 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7faad000ea60 tx=0x7faad000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:48.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.664+0000 7faaca7fc640 1 -- 172.21.15.44:0/2191423500 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad000cde0 con 0x7faad41029b0 2024-04-21T23:14:48.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.664+0000 7faaca7fc640 1 -- 172.21.15.44:0/2191423500 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faad00035f0 con 0x7faad41029b0 2024-04-21T23:14:48.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.664+0000 7faadb682640 1 -- 172.21.15.44:0/2191423500 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faad41a67d0 con 0x7faad41029b0 2024-04-21T23:14:48.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.664+0000 7faaca7fc640 1 -- 172.21.15.44:0/2191423500 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad0010670 con 0x7faad41029b0 2024-04-21T23:14:48.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.664+0000 7faadb682640 1 -- 172.21.15.44:0/2191423500 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faad41a6cf0 con 0x7faad41029b0 2024-04-21T23:14:48.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.667+0000 7faaca7fc640 1 -- 172.21.15.44:0/2191423500 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7faad00037a0 con 0x7faad41029b0 2024-04-21T23:14:48.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.667+0000 7faade260740 1 Processor -- start 2024-04-21T23:14:48.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.667+0000 7faade260740 1 -- start start 2024-04-21T23:14:48.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.667+0000 7faade260740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298ed5f30 0x562298ed6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.667+0000 7faade260740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed6850 0x562298edeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.667+0000 7faade260740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298edf430 0x562298ee1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.667+0000 7faade260740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562298e3a560 con 0x562298ed6850 2024-04-21T23:14:48.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.667+0000 7faade260740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x562298de2730 con 0x562298edf430 2024-04-21T23:14:48.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.667+0000 7faade260740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562298e36520 con 0x562298ed5f30 2024-04-21T23:14:48.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.667+0000 7faaca7fc640 1 --2- 172.21.15.44:0/2191423500 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faac00a2d00 0x7faac00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.667+0000 7faad941e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed6850 0x562298edeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.667+0000 7faad9c1f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298ed5f30 0x562298ed6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.667+0000 7faad9c1f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298ed5f30 0x562298ed6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:42586/0 (socket says 172.21.15.44:42586) 2024-04-21T23:14:48.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.667+0000 7faad9c1f640 1 -- 172.21.15.44:0/2561201750 learned_addr learned my addr 172.21.15.44:0/2561201750 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:48.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.667+0000 7faad941e640 1 --2- 172.21.15.44:0/2561201750 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed6850 0x562298edeef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42488/0 (socket says 172.21.15.44:42488) 2024-04-21T23:14:48.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.667+0000 7faad8c1d640 1 --2- 172.21.15.44:0/2561201750 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298edf430 0x562298ee1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.667+0000 7faad9c1f640 1 --2- 172.21.15.44:0/2191423500 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faac00a2d00 0x7faac00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.668+0000 7faaca7fc640 1 -- 172.21.15.44:0/2191423500 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9445+0+0 (secure 0 0 0) 0x7faad00cfb00 con 0x7faad41029b0 2024-04-21T23:14:48.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.668+0000 7faad9c1f640 1 -- 172.21.15.44:0/2561201750 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298edf430 msgr2=0x562298ee1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:48.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.668+0000 7faad9c1f640 1 --2- 172.21.15.44:0/2561201750 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298edf430 0x562298ee1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.668+0000 7faad9c1f640 1 -- 172.21.15.44:0/2561201750 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed6850 msgr2=0x562298edeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:48.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.668+0000 7faad9c1f640 1 --2- 172.21.15.44:0/2561201750 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed6850 0x562298edeef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.668+0000 7faad9c1f640 1 -- 172.21.15.44:0/2561201750 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562298c187d0 con 0x562298ed5f30 2024-04-21T23:14:48.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.668+0000 7faad9c1f640 1 --2- 172.21.15.44:0/2561201750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298ed5f30 0x562298ed6310 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faacc00f540 tx=0x7faacc00fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef8b63d71265a8ff server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:48.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.668+0000 7faad9c1f640 1 --2- 172.21.15.44:0/2191423500 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faac00a2d00 0x7faac00a51c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faacc009930 tx=0x7faacc01e000 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:48.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.668+0000 7faabf7fe640 1 -- 172.21.15.44:0/2561201750 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc02b070 con 0x562298ed5f30 2024-04-21T23:14:48.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.668+0000 7faabf7fe640 1 -- 172.21.15.44:0/2561201750 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faacc003e50 con 0x562298ed5f30 2024-04-21T23:14:48.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.668+0000 7faabf7fe640 1 -- 172.21.15.44:0/2561201750 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc022070 con 0x562298ed5f30 2024-04-21T23:14:48.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.669+0000 7faade260740 1 -- 172.21.15.44:0/2561201750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298ed5f30 msgr2=0x562298ed6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:48.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.669+0000 7faade260740 1 --2- 172.21.15.44:0/2561201750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298ed5f30 0x562298ed6310 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faacc00f540 tx=0x7faacc00fa00 comp rx=0 tx=0).stop 2024-04-21T23:14:48.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.669+0000 7faade260740 1 -- 172.21.15.44:0/2561201750 shutdown_connections 2024-04-21T23:14:48.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.669+0000 7faade260740 1 --2- 172.21.15.44:0/2561201750 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298edf430 0x562298ee1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.669+0000 7faade260740 1 --2- 172.21.15.44:0/2561201750 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed6850 0x562298edeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.669+0000 7faade260740 1 --2- 172.21.15.44:0/2561201750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298ed5f30 0x562298ed6310 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.669+0000 7faade260740 1 -- 172.21.15.44:0/2561201750 >> 172.21.15.44:0/2561201750 conn(0x562298ed0660 msgr2=0x562298ed0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:48.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.669+0000 7faade260740 1 -- 172.21.15.44:0/2561201750 shutdown_connections 2024-04-21T23:14:48.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.669+0000 7faade260740 1 -- 172.21.15.44:0/2561201750 wait complete. 2024-04-21T23:14:48.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.669+0000 7faade260740 1 -- start start 2024-04-21T23:14:48.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.669+0000 7faade260740 1 Processor -- start 2024-04-21T23:14:48.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.670+0000 7faade260740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed5f30 0x562298edc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.670+0000 7faade260740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298ed6850 0x562298edcab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.670+0000 7faade260740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298edf430 0x562298edcff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.670+0000 7faade260740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x562298ee21c0 con 0x562298ed5f30 2024-04-21T23:14:48.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.670+0000 7faade260740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x562298df5e00 con 0x562298edf430 2024-04-21T23:14:48.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.670+0000 7faade260740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562298ee2340 con 0x562298ed6850 2024-04-21T23:14:48.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.670+0000 7faade260740 10 client.0 _ll_register_callbacks cb 0x562298e30eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:48.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.670+0000 7faad8c1d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298ed6850 0x562298edcab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.670+0000 7faad941e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed5f30 0x562298edc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.670+0000 7faad8c1d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298ed6850 0x562298edcab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:42600/0 (socket says 172.21.15.44:42600) 2024-04-21T23:14:48.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.670+0000 7faad8c1d640 1 -- 172.21.15.44:0/533471400 learned_addr learned my addr 172.21.15.44:0/533471400 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:48.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.670+0000 7faad9c1f640 1 --2- 172.21.15.44:0/533471400 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298edf430 0x562298edcff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.670+0000 7faad8c1d640 1 -- 172.21.15.44:0/533471400 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298edf430 msgr2=0x562298edcff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:48.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.670+0000 7faad8c1d640 1 --2- 172.21.15.44:0/533471400 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298edf430 0x562298edcff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.670+0000 7faad8c1d640 1 -- 172.21.15.44:0/533471400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed5f30 msgr2=0x562298edc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:48.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.670+0000 7faad8c1d640 1 --2- 172.21.15.44:0/533471400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed5f30 0x562298edc570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.670+0000 7faad8c1d640 1 -- 172.21.15.44:0/533471400 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562298b4e3f0 con 0x562298ed6850 2024-04-21T23:14:48.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.670+0000 7faad941e640 1 --2- 172.21.15.44:0/533471400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed5f30 0x562298edc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:48.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.671+0000 7faad8c1d640 1 --2- 172.21.15.44:0/533471400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298ed6850 0x562298edcab0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7faac4006010 tx=0x7faac4004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe2de5c5f5dc9418 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:48.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.671+0000 7faabcff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:14:48.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.671+0000 7faabcff9640 1 -- 172.21.15.44:0/533471400 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac4044070 con 0x562298ed6850 2024-04-21T23:14:48.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.671+0000 7faade260740 1 -- 172.21.15.44:0/533471400 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faacc003e50 con 0x562298ed6850 2024-04-21T23:14:48.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.671+0000 7faabcff9640 1 -- 172.21.15.44:0/533471400 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9445+0+0 (secure 0 0 0) 0x7faac402cb30 con 0x562298ed6850 2024-04-21T23:14:48.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.671+0000 7faabcff9640 1 -- 172.21.15.44:0/533471400 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faac402cd30 con 0x562298ed6850 2024-04-21T23:14:48.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.671+0000 7faade260740 10 client.26542 fetch_fsmap learned FSMap version 27 2024-04-21T23:14:48.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.671+0000 7faade260740 1 -- 172.21.15.44:0/533471400 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562298b0e0f0 con 0x562298ed6850 2024-04-21T23:14:48.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.671+0000 7faabcff9640 1 -- 172.21.15.44:0/533471400 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faacc003e50 con 0x562298ed6850 2024-04-21T23:14:48.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.671+0000 7faabcff9640 1 -- 172.21.15.44:0/533471400 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 27) ==== 8330+0+0 (secure 0 0 0) 0x7faac40035f0 con 0x562298ed6850 2024-04-21T23:14:48.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.671+0000 7faade260740 10 client.26542 fetch_fsmap finished waiting for FSMap version 27 2024-04-21T23:14:48.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.671+0000 7faade260740 10 client.26542 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:48.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.671+0000 7faade260740 20 client.26542 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:48.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.671+0000 7faade260740 1 --2- 172.21.15.44:0/533471400 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562298eef800 0x562298ef1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.671+0000 7faade260740 4 client.26542 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:14:48.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.671+0000 7faade260740 1 -- 172.21.15.44:0/533471400 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562298e47aa0 con 0x562298eef800 2024-04-21T23:14:48.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.672+0000 7faad941e640 1 --2- 172.21.15.44:0/533471400 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562298eef800 0x562298ef1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.672+0000 7faabcff9640 10 client.26542 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:48.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.672+0000 7faad941e640 1 --2- 172.21.15.44:0/533471400 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562298eef800 0x562298ef1bf0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bcf730631d5febf server_cookie=d4ca2d7277b6255b in_seq=0 out_seq=0 2024-04-21T23:14:48.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.674+0000 7faabcff9640 1 -- 172.21.15.44:0/533471400 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562298e47aa0 con 0x562298eef800 2024-04-21T23:14:48.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.674+0000 7faabcff9640 10 client.26542 handle_command_reply: tid=0 2024-04-21T23:14:48.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.674+0000 7faade260740 1 client.26542 shutdown 2024-04-21T23:14:48.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.674+0000 7faade260740 1 -- 172.21.15.44:0/533471400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298ed6850 msgr2=0x562298edcab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:48.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.674+0000 7faade260740 1 --2- 172.21.15.44:0/533471400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298ed6850 0x562298edcab0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7faac4006010 tx=0x7faac4004610 comp rx=0 tx=0).stop 2024-04-21T23:14:48.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.674+0000 7faade260740 1 -- 172.21.15.44:0/533471400 shutdown_connections 2024-04-21T23:14:48.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.674+0000 7faade260740 1 --2- 172.21.15.44:0/533471400 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562298eef800 0x562298ef1bf0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.674+0000 7faade260740 1 --2- 172.21.15.44:0/533471400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298ed6850 0x562298edcab0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.674+0000 7faade260740 1 --2- 172.21.15.44:0/533471400 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298edf430 0x562298edcff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.674+0000 7faade260740 1 --2- 172.21.15.44:0/533471400 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed5f30 0x562298edc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.674+0000 7faabcff9640 0 client.26542 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:48.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.674+0000 7faade260740 1 -- 172.21.15.44:0/533471400 >> 172.21.15.44:0/533471400 conn(0x562298ed0660 msgr2=0x562298ed23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:48.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.674+0000 7faade260740 1 -- 172.21.15.44:0/533471400 shutdown_connections 2024-04-21T23:14:48.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.674+0000 7faade260740 1 -- 172.21.15.44:0/533471400 wait complete. 2024-04-21T23:14:48.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.674+0000 7faade260740 20 client.26542 trim_cache size 0 max 16384 2024-04-21T23:14:48.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.693+0000 7faade260740 1 Processor -- start 2024-04-21T23:14:48.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.693+0000 7faade260740 1 -- start start 2024-04-21T23:14:48.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.693+0000 7faade260740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298ed5f30 0x562298edeca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.693+0000 7faade260740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed6850 0x562298eeb080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.693+0000 7faade260740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298edf430 0x562298edb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.693+0000 7faade260740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faad41aeeb0 con 0x562298ed6850 2024-04-21T23:14:48.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.693+0000 7faade260740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7faad41b1730 con 0x562298edf430 2024-04-21T23:14:48.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.693+0000 7faade260740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faad41af520 con 0x562298ed5f30 2024-04-21T23:14:48.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.693+0000 7faad8c1d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed6850 0x562298eeb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.693+0000 7faad8c1d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed6850 0x562298eeb080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42508/0 (socket says 172.21.15.44:42508) 2024-04-21T23:14:48.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.693+0000 7faad8c1d640 1 -- 172.21.15.44:0/1170815641 learned_addr learned my addr 172.21.15.44:0/1170815641 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:48.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.693+0000 7faad941e640 1 --2- 172.21.15.44:0/1170815641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298ed5f30 0x562298edeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.693+0000 7faad9c1f640 1 --2- 172.21.15.44:0/1170815641 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298edf430 0x562298edb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.693+0000 7faad8c1d640 1 -- 172.21.15.44:0/1170815641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298ed5f30 msgr2=0x562298edeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:48.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.693+0000 7faad8c1d640 1 --2- 172.21.15.44:0/1170815641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298ed5f30 0x562298edeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.693+0000 7faad8c1d640 1 -- 172.21.15.44:0/1170815641 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298edf430 msgr2=0x562298edb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:48.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.693+0000 7faad8c1d640 1 --2- 172.21.15.44:0/1170815641 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298edf430 0x562298edb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.693+0000 7faad8c1d640 1 -- 172.21.15.44:0/1170815641 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562298ef21d0 con 0x562298ed6850 2024-04-21T23:14:48.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.694+0000 7faad8c1d640 1 --2- 172.21.15.44:0/1170815641 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed6850 0x562298eeb080 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7faac40039c0 tx=0x7faac402bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d7ab2de0f2bea7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:48.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.694+0000 7faabeffd640 1 -- 172.21.15.44:0/1170815641 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac405a070 con 0x562298ed6850 2024-04-21T23:14:48.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.694+0000 7faabeffd640 1 -- 172.21.15.44:0/1170815641 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faac40043e0 con 0x562298ed6850 2024-04-21T23:14:48.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.694+0000 7faabeffd640 1 -- 172.21.15.44:0/1170815641 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac4044030 con 0x562298ed6850 2024-04-21T23:14:48.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.695+0000 7faade260740 1 -- 172.21.15.44:0/1170815641 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed6850 msgr2=0x562298eeb080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:48.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.695+0000 7faade260740 1 --2- 172.21.15.44:0/1170815641 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed6850 0x562298eeb080 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7faac40039c0 tx=0x7faac402bcd0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.695+0000 7faade260740 1 -- 172.21.15.44:0/1170815641 shutdown_connections 2024-04-21T23:14:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.695+0000 7faade260740 1 --2- 172.21.15.44:0/1170815641 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298edf430 0x562298edb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.695+0000 7faade260740 1 --2- 172.21.15.44:0/1170815641 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed6850 0x562298eeb080 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.695+0000 7faade260740 1 --2- 172.21.15.44:0/1170815641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298ed5f30 0x562298edeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.695+0000 7faade260740 1 -- 172.21.15.44:0/1170815641 >> 172.21.15.44:0/1170815641 conn(0x562298ed0660 msgr2=0x562298ecf6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.695+0000 7faade260740 1 -- 172.21.15.44:0/1170815641 shutdown_connections 2024-04-21T23:14:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.695+0000 7faade260740 1 -- 172.21.15.44:0/1170815641 wait complete. 2024-04-21T23:14:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.695+0000 7faade260740 1 -- start start 2024-04-21T23:14:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.695+0000 7faade260740 1 Processor -- start 2024-04-21T23:14:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.695+0000 7faade260740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298ed5f30 0x562298ee3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.695+0000 7faade260740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed6850 0x562298ee3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.695+0000 7faade260740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298edf430 0x562298ee3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.695+0000 7faade260740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faad41afbf0 con 0x562298ed6850 2024-04-21T23:14:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.695+0000 7faade260740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7faad41b1060 con 0x562298ed5f30 2024-04-21T23:14:48.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.695+0000 7faade260740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faad41b02c0 con 0x562298edf430 2024-04-21T23:14:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faade260740 10 client.0 _ll_register_callbacks cb 0x7faac4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faad8c1d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed6850 0x562298ee3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faad941e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298ed5f30 0x562298ee3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faad8c1d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed6850 0x562298ee3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42516/0 (socket says 172.21.15.44:42516) 2024-04-21T23:14:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faad8c1d640 1 -- 172.21.15.44:0/2489900473 learned_addr learned my addr 172.21.15.44:0/2489900473 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faad941e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298ed5f30 0x562298ee3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:59010/0 (socket says 172.21.15.44:59010) 2024-04-21T23:14:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faad9c1f640 1 --2- 172.21.15.44:0/2489900473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298edf430 0x562298ee3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faad8c1d640 1 -- 172.21.15.44:0/2489900473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298edf430 msgr2=0x562298ee3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faad8c1d640 1 --2- 172.21.15.44:0/2489900473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298edf430 0x562298ee3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faad8c1d640 1 -- 172.21.15.44:0/2489900473 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298ed5f30 msgr2=0x562298ee3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faad8c1d640 1 --2- 172.21.15.44:0/2489900473 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298ed5f30 0x562298ee3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faad8c1d640 1 -- 172.21.15.44:0/2489900473 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562298f54f10 con 0x562298ed6850 2024-04-21T23:14:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faad941e640 1 --2- 172.21.15.44:0/2489900473 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298ed5f30 0x562298ee3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faad8c1d640 1 --2- 172.21.15.44:0/2489900473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed6850 0x562298ee3760 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7faac4003b30 tx=0x7faac4004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68327d7d74d131e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:48.700 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faabcff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:14:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faabcff9640 1 -- 172.21.15.44:0/2489900473 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac405a020 con 0x562298ed6850 2024-04-21T23:14:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faabcff9640 1 -- 172.21.15.44:0/2489900473 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9445+0+0 (secure 0 0 0) 0x7faac402cb30 con 0x562298ed6850 2024-04-21T23:14:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faade260740 1 -- 172.21.15.44:0/2489900473 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faac40043e0 con 0x562298ed6850 2024-04-21T23:14:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faabcff9640 1 -- 172.21.15.44:0/2489900473 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faac4062050 con 0x562298ed6850 2024-04-21T23:14:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faade260740 10 client.16983 fetch_fsmap learned FSMap version 27 2024-04-21T23:14:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.696+0000 7faade260740 1 -- 172.21.15.44:0/2489900473 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562298f56200 con 0x562298ed6850 2024-04-21T23:14:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.697+0000 7faabcff9640 1 -- 172.21.15.44:0/2489900473 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faac40043e0 con 0x562298ed6850 2024-04-21T23:14:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.697+0000 7faabcff9640 1 -- 172.21.15.44:0/2489900473 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 27) ==== 8330+0+0 (secure 0 0 0) 0x7faac4055510 con 0x562298ed6850 2024-04-21T23:14:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.697+0000 7faade260740 10 client.16983 fetch_fsmap finished waiting for FSMap version 27 2024-04-21T23:14:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.697+0000 7faade260740 10 client.16983 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.697+0000 7faade260740 20 client.16983 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.697+0000 7faade260740 1 --2- 172.21.15.44:0/2489900473 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562298f568b0 0x562298f58ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.697+0000 7faade260740 4 client.16983 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:14:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.697+0000 7faade260740 1 -- 172.21.15.44:0/2489900473 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562298e41df0 con 0x562298f568b0 2024-04-21T23:14:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.697+0000 7faad941e640 1 --2- 172.21.15.44:0/2489900473 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562298f568b0 0x562298f58ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.701 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.697+0000 7faabcff9640 10 client.16983 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:48.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.697+0000 7faad941e640 1 --2- 172.21.15.44:0/2489900473 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562298f568b0 0x562298f58ca0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55bf5842b37b2aec server_cookie=ad3d73b285fa87fd in_seq=0 out_seq=0 2024-04-21T23:14:48.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.697+0000 7faabcff9640 1 -- 172.21.15.44:0/2489900473 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562298e41df0 con 0x562298f568b0 2024-04-21T23:14:48.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.697+0000 7faabcff9640 10 client.16983 handle_command_reply: tid=0 2024-04-21T23:14:48.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.697+0000 7faade260740 1 client.16983 shutdown 2024-04-21T23:14:48.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.698+0000 7faade260740 1 -- 172.21.15.44:0/2489900473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed6850 msgr2=0x562298ee3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:48.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.698+0000 7faade260740 1 --2- 172.21.15.44:0/2489900473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed6850 0x562298ee3760 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7faac4003b30 tx=0x7faac4004950 comp rx=0 tx=0).stop 2024-04-21T23:14:48.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.698+0000 7faade260740 1 -- 172.21.15.44:0/2489900473 shutdown_connections 2024-04-21T23:14:48.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.698+0000 7faade260740 1 --2- 172.21.15.44:0/2489900473 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x562298f568b0 0x562298f58ca0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.698+0000 7faade260740 1 --2- 172.21.15.44:0/2489900473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562298edf430 0x562298ee3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.698+0000 7faade260740 1 --2- 172.21.15.44:0/2489900473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x562298ed6850 0x562298ee3760 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.698+0000 7faade260740 1 --2- 172.21.15.44:0/2489900473 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x562298ed5f30 0x562298ee3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.698+0000 7faade260740 1 -- 172.21.15.44:0/2489900473 >> 172.21.15.44:0/2489900473 conn(0x562298ed0660 msgr2=0x562298ed1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:48.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.698+0000 7faabcff9640 0 client.16983 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:48.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.698+0000 7faade260740 1 -- 172.21.15.44:0/2489900473 shutdown_connections 2024-04-21T23:14:48.702 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.698+0000 7faade260740 1 -- 172.21.15.44:0/2489900473 wait complete. 2024-04-21T23:14:48.703 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.698+0000 7faade260740 20 client.16983 trim_cache size 0 max 16384 2024-04-21T23:14:48.703 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:14:48.703 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-21T23:14:48.703 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-21T23:14:48.703 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:14:48.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.698+0000 7faadb682640 1 -- 172.21.15.44:0/2191423500 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faac00a2d00 msgr2=0x7faac00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:48.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.698+0000 7faadb682640 1 --2- 172.21.15.44:0/2191423500 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faac00a2d00 0x7faac00a51c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faacc009930 tx=0x7faacc01e000 comp rx=0 tx=0).stop 2024-04-21T23:14:48.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.698+0000 7faadb682640 1 -- 172.21.15.44:0/2191423500 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faad41029b0 msgr2=0x7faad4100600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:48.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.698+0000 7faadb682640 1 --2- 172.21.15.44:0/2191423500 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faad41029b0 0x7faad4100600 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7faad000ea60 tx=0x7faad000ef20 comp rx=0 tx=0).stop 2024-04-21T23:14:48.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.699+0000 7faadb682640 1 -- 172.21.15.44:0/2191423500 shutdown_connections 2024-04-21T23:14:48.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.699+0000 7faadb682640 1 --2- 172.21.15.44:0/2191423500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faad41032d0 0x7faad4100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.699+0000 7faadb682640 1 --2- 172.21.15.44:0/2191423500 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faac00a2d00 0x7faac00a51c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.699+0000 7faadb682640 1 --2- 172.21.15.44:0/2191423500 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faad410bcf0 0x7faad4101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.699+0000 7faadb682640 1 --2- 172.21.15.44:0/2191423500 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faad41029b0 0x7faad4100600 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.699+0000 7faadb682640 1 -- 172.21.15.44:0/2191423500 >> 172.21.15.44:0/2191423500 conn(0x7faad40777f0 msgr2=0x7faad40fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:48.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.699+0000 7faadb682640 1 -- 172.21.15.44:0/2191423500 shutdown_connections 2024-04-21T23:14:48.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.699+0000 7faadb682640 1 -- 172.21.15.44:0/2191423500 wait complete. 2024-04-21T23:14:48.705 INFO:tasks.workunit.client.0.smithi044.stdout: 1 698799 13.28 MB/sec execute 158 sec latency 92.979 ms 2024-04-21T23:14:48.708 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:14:48.708 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:14:48.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.860+0000 7fbfa6375640 1 -- 172.21.15.44:0/613430625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbfa0100890 msgr2=0x7fbfa0100cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:48.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.860+0000 7fbfa6375640 1 --2- 172.21.15.44:0/613430625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbfa0100890 0x7fbfa0100cf0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fbf94009900 tx=0x7fbf9402bc30 comp rx=0 tx=0).stop 2024-04-21T23:14:48.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.860+0000 7fbfa6375640 1 -- 172.21.15.44:0/613430625 shutdown_connections 2024-04-21T23:14:48.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.860+0000 7fbfa6375640 1 --2- 172.21.15.44:0/613430625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfa0101230 0x7fbfa010d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.860+0000 7fbfa6375640 1 --2- 172.21.15.44:0/613430625 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbfa0100890 0x7fbfa0100cf0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.860+0000 7fbfa6375640 1 --2- 172.21.15.44:0/613430625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfa01066c0 0x7fbfa0106aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.860+0000 7fbfa6375640 1 -- 172.21.15.44:0/613430625 >> 172.21.15.44:0/613430625 conn(0x7fbfa0077430 msgr2=0x7fbfa0077840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:48.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.860+0000 7fbfa6375640 1 -- 172.21.15.44:0/613430625 shutdown_connections 2024-04-21T23:14:48.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.860+0000 7fbfa6375640 1 -- 172.21.15.44:0/613430625 wait complete. 2024-04-21T23:14:48.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.861+0000 7fbfa6375640 1 Processor -- start 2024-04-21T23:14:48.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.861+0000 7fbfa6375640 1 -- start start 2024-04-21T23:14:48.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.861+0000 7fbfa6375640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfa0100890 0x7fbfa0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.861+0000 7fbfa6375640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfa0101230 0x7fbfa0198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.861+0000 7fbfa6375640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbfa01066c0 0x7fbfa019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.861+0000 7fbfa6375640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa0110a00 con 0x7fbfa01066c0 2024-04-21T23:14:48.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.861+0000 7fbfa6375640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa0110880 con 0x7fbfa0100890 2024-04-21T23:14:48.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.861+0000 7fbfa6375640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa0110b80 con 0x7fbfa0101230 2024-04-21T23:14:48.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.862+0000 7fbf9f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfa0101230 0x7fbfa0198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.862+0000 7fbfa4912640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbfa01066c0 0x7fbfa019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.862+0000 7fbf9ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfa0100890 0x7fbfa0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.862+0000 7fbf9ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfa0100890 0x7fbfa0197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:59030/0 (socket says 172.21.15.44:59030) 2024-04-21T23:14:48.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.862+0000 7fbfa4912640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbfa01066c0 0x7fbfa019c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42534/0 (socket says 172.21.15.44:42534) 2024-04-21T23:14:48.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.862+0000 7fbf9ffff640 1 -- 172.21.15.44:0/3501182639 learned_addr learned my addr 172.21.15.44:0/3501182639 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:48.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.862+0000 7fbf9f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfa0101230 0x7fbfa0198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:42642/0 (socket says 172.21.15.44:42642) 2024-04-21T23:14:48.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.862+0000 7fbf9f7fe640 1 -- 172.21.15.44:0/3501182639 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfa0100890 msgr2=0x7fbfa0197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:48.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.862+0000 7fbf9f7fe640 1 --2- 172.21.15.44:0/3501182639 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfa0100890 0x7fbfa0197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.862+0000 7fbf9f7fe640 1 -- 172.21.15.44:0/3501182639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbfa01066c0 msgr2=0x7fbfa019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:48.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.862+0000 7fbf9f7fe640 1 --2- 172.21.15.44:0/3501182639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbfa01066c0 0x7fbfa019c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:48.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.862+0000 7fbf9f7fe640 1 -- 172.21.15.44:0/3501182639 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa019cfb0 con 0x7fbfa0101230 2024-04-21T23:14:48.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.862+0000 7fbfa4912640 1 --2- 172.21.15.44:0/3501182639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbfa01066c0 0x7fbfa019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:48.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.862+0000 7fbf9ffff640 1 --2- 172.21.15.44:0/3501182639 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfa0100890 0x7fbfa0197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:48.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.862+0000 7fbf9f7fe640 1 --2- 172.21.15.44:0/3501182639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfa0101230 0x7fbfa0198500 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbf94036040 tx=0x7fbf9402cb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:48.864 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.863+0000 7fbf9d7fa640 1 -- 172.21.15.44:0/3501182639 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf94041070 con 0x7fbfa0101230 2024-04-21T23:14:48.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.863+0000 7fbf9d7fa640 1 -- 172.21.15.44:0/3501182639 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf94004450 con 0x7fbfa0101230 2024-04-21T23:14:48.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.863+0000 7fbf9d7fa640 1 -- 172.21.15.44:0/3501182639 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf94003d90 con 0x7fbfa0101230 2024-04-21T23:14:48.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.863+0000 7fbfa6375640 1 -- 172.21.15.44:0/3501182639 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa019d240 con 0x7fbfa0101230 2024-04-21T23:14:48.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.863+0000 7fbfa6375640 1 -- 172.21.15.44:0/3501182639 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa01a4b30 con 0x7fbfa0101230 2024-04-21T23:14:48.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.864+0000 7fbfa6375640 1 -- 172.21.15.44:0/3501182639 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfa0101da0 con 0x7fbfa0101230 2024-04-21T23:14:48.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.865+0000 7fbf9d7fa640 1 -- 172.21.15.44:0/3501182639 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fbf94044070 con 0x7fbfa0101230 2024-04-21T23:14:48.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.865+0000 7fbf9d7fa640 1 --2- 172.21.15.44:0/3501182639 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbf880a2d00 0x7fbf880a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:48.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.866+0000 7fbf9ffff640 1 --2- 172.21.15.44:0/3501182639 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbf880a2d00 0x7fbf880a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:48.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.866+0000 7fbf9d7fa640 1 -- 172.21.15.44:0/3501182639 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9445+0+0 (secure 0 0 0) 0x7fbf940f12e0 con 0x7fbfa0101230 2024-04-21T23:14:48.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.866+0000 7fbf9ffff640 1 --2- 172.21.15.44:0/3501182639 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbf880a2d00 0x7fbf880a51c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbf980117b0 tx=0x7fbf98009500 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:48.868 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:48.869+0000 7fbf9d7fa640 1 -- 172.21.15.44:0/3501182639 <== mon.1 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fbf940bd5c0 con 0x7fbfa0101230 2024-04-21T23:14:49.101 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.102+0000 7fbfa6375640 1 -- 172.21.15.44:0/3501182639 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbfa00685f0 con 0x7fbfa0101230 2024-04-21T23:14:49.102 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.103+0000 7fbf9d7fa640 1 -- 172.21.15.44:0/3501182639 <== mon.1 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11788 (secure 0 0 0) 0x7fbf940c18c0 con 0x7fbfa0101230 2024-04-21T23:14:49.102 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:14:49.102 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:14:46.193301+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26530":{"gid":26530,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/404328686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":404328686},{"type":"v1","addr":"172.21.15.69:6839","nonce":404328686}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:14:49.102 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 27 2024-04-21T23:14:49.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.105+0000 7fbfa6375640 1 -- 172.21.15.44:0/3501182639 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbf880a2d00 msgr2=0x7fbf880a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:49.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.105+0000 7fbfa6375640 1 --2- 172.21.15.44:0/3501182639 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbf880a2d00 0x7fbf880a51c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbf980117b0 tx=0x7fbf98009500 comp rx=0 tx=0).stop 2024-04-21T23:14:49.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.105+0000 7fbfa6375640 1 -- 172.21.15.44:0/3501182639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfa0101230 msgr2=0x7fbfa0198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:49.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.105+0000 7fbfa6375640 1 --2- 172.21.15.44:0/3501182639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfa0101230 0x7fbfa0198500 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbf94036040 tx=0x7fbf9402cb20 comp rx=0 tx=0).stop 2024-04-21T23:14:49.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.106+0000 7fbfa6375640 1 -- 172.21.15.44:0/3501182639 shutdown_connections 2024-04-21T23:14:49.104 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.106+0000 7fbfa6375640 1 --2- 172.21.15.44:0/3501182639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfa0101230 0x7fbfa0198500 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.106+0000 7fbfa6375640 1 --2- 172.21.15.44:0/3501182639 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fbf880a2d00 0x7fbf880a51c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.106+0000 7fbfa6375640 1 --2- 172.21.15.44:0/3501182639 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fbfa01066c0 0x7fbfa019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.106+0000 7fbfa6375640 1 --2- 172.21.15.44:0/3501182639 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbfa0100890 0x7fbfa0197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.106+0000 7fbfa6375640 1 -- 172.21.15.44:0/3501182639 >> 172.21.15.44:0/3501182639 conn(0x7fbfa0077430 msgr2=0x7fbfa00fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:49.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.106+0000 7fbfa6375640 1 -- 172.21.15.44:0/3501182639 shutdown_connections 2024-04-21T23:14:49.105 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.106+0000 7fbfa6375640 1 -- 172.21.15.44:0/3501182639 wait complete. 2024-04-21T23:14:49.116 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:14:49.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.225+0000 7f6d45a8b640 1 -- 172.21.15.44:0/2883550412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d401032d0 msgr2=0x7f6d4010b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:49.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.225+0000 7f6d45a8b640 1 --2- 172.21.15.44:0/2883550412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d401032d0 0x7f6d4010b7b0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f6d34009900 tx=0x7f6d3402bcd0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.226+0000 7f6d45a8b640 1 -- 172.21.15.44:0/2883550412 shutdown_connections 2024-04-21T23:14:49.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.226+0000 7f6d45a8b640 1 --2- 172.21.15.44:0/2883550412 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d4010bcf0 0x7f6d4010e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.226+0000 7f6d45a8b640 1 --2- 172.21.15.44:0/2883550412 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d401032d0 0x7f6d4010b7b0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.226+0000 7f6d45a8b640 1 --2- 172.21.15.44:0/2883550412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d401029b0 0x7f6d40102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.226+0000 7f6d45a8b640 1 -- 172.21.15.44:0/2883550412 >> 172.21.15.44:0/2883550412 conn(0x7f6d400777f0 msgr2=0x7f6d40077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:49.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.226+0000 7f6d45a8b640 1 -- 172.21.15.44:0/2883550412 shutdown_connections 2024-04-21T23:14:49.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.226+0000 7f6d45a8b640 1 -- 172.21.15.44:0/2883550412 wait complete. 2024-04-21T23:14:49.225 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.226+0000 7f6d45a8b640 1 Processor -- start 2024-04-21T23:14:49.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.226+0000 7f6d45a8b640 1 -- start start 2024-04-21T23:14:49.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d45a8b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d401029b0 0x7f6d40198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:49.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d45a8b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d401032d0 0x7f6d401985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:49.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d45a8b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d4010bcf0 0x7f6d4019c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:49.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d45a8b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6d40111af0 con 0x7f6d4010bcf0 2024-04-21T23:14:49.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d45a8b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6d40111970 con 0x7f6d401032d0 2024-04-21T23:14:49.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d45a8b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6d40111c70 con 0x7f6d401029b0 2024-04-21T23:14:49.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d3ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d4010bcf0 0x7f6d4019c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:49.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d3ffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d4010bcf0 0x7f6d4019c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42562/0 (socket says 172.21.15.44:42562) 2024-04-21T23:14:49.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d3effd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d401032d0 0x7f6d401985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:49.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d3f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d401029b0 0x7f6d40198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:49.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d3ffff640 1 -- 172.21.15.44:0/3893865164 learned_addr learned my addr 172.21.15.44:0/3893865164 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:49.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d3f7fe640 1 -- 172.21.15.44:0/3893865164 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d401032d0 msgr2=0x7f6d401985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:49.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d3f7fe640 1 --2- 172.21.15.44:0/3893865164 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d401032d0 0x7f6d401985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d3f7fe640 1 -- 172.21.15.44:0/3893865164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d4010bcf0 msgr2=0x7f6d4019c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:49.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d3f7fe640 1 --2- 172.21.15.44:0/3893865164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d4010bcf0 0x7f6d4019c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d3f7fe640 1 -- 172.21.15.44:0/3893865164 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d4019d080 con 0x7f6d401029b0 2024-04-21T23:14:49.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d3ffff640 1 --2- 172.21.15.44:0/3893865164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d4010bcf0 0x7f6d4019c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:49.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d3f7fe640 1 --2- 172.21.15.44:0/3893865164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d401029b0 0x7f6d40198090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6d3000e9f0 tx=0x7f6d3000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:49.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d3cff9640 1 -- 172.21.15.44:0/3893865164 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3000cd80 con 0x7f6d401029b0 2024-04-21T23:14:49.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d3cff9640 1 -- 172.21.15.44:0/3893865164 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d300035f0 con 0x7f6d401029b0 2024-04-21T23:14:49.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d45a8b640 1 -- 172.21.15.44:0/3893865164 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d4019d370 con 0x7f6d401029b0 2024-04-21T23:14:49.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d3cff9640 1 -- 172.21.15.44:0/3893865164 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d30010690 con 0x7f6d401029b0 2024-04-21T23:14:49.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.227+0000 7f6d45a8b640 1 -- 172.21.15.44:0/3893865164 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d401a4c00 con 0x7f6d401029b0 2024-04-21T23:14:49.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.230+0000 7f6d3cff9640 1 -- 172.21.15.44:0/3893865164 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6d30003b30 con 0x7f6d401029b0 2024-04-21T23:14:49.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.230+0000 7f6d48672740 1 Processor -- start 2024-04-21T23:14:49.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.230+0000 7f6d48672740 1 -- start start 2024-04-21T23:14:49.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.230+0000 7f6d3cff9640 1 --2- 172.21.15.44:0/3893865164 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6d280a2d00 0x7f6d280a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:49.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.230+0000 7f6d48672740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 0x560834fb1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:49.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.230+0000 7f6d48672740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fb9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:49.235 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.230+0000 7f6d48672740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fba520 0x560834fbc910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:49.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.230+0000 7f6d48672740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560834f15530 con 0x560834fb1940 2024-04-21T23:14:49.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.230+0000 7f6d48672740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x560834ebd830 con 0x560834fba520 2024-04-21T23:14:49.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.230+0000 7f6d48672740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560834f114f0 con 0x560834fb1020 2024-04-21T23:14:49.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.230+0000 7f6d3ffff640 1 --2- 172.21.15.44:0/3893865164 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6d280a2d00 0x7f6d280a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:49.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.230+0000 7f6d3cff9640 1 -- 172.21.15.44:0/3893865164 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9445+0+0 (secure 0 0 0) 0x7f6d300cfad0 con 0x7f6d401029b0 2024-04-21T23:14:49.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.230+0000 7f6d3f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 0x560834fb1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:49.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.230+0000 7f6d3effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fb9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:49.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.230+0000 7f6d3f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 0x560834fb1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:42666/0 (socket says 172.21.15.44:42666) 2024-04-21T23:14:49.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.230+0000 7f6d3effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fb9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42566/0 (socket says 172.21.15.44:42566) 2024-04-21T23:14:49.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.230+0000 7f6d3f7fe640 1 -- 172.21.15.44:0/627210146 learned_addr learned my addr 172.21.15.44:0/627210146 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:49.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.231+0000 7f6d3ffff640 1 --2- 172.21.15.44:0/627210146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fba520 0x560834fbc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:49.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.231+0000 7f6d3f7fe640 1 -- 172.21.15.44:0/627210146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fba520 msgr2=0x560834fbc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:49.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.231+0000 7f6d3f7fe640 1 --2- 172.21.15.44:0/627210146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fba520 0x560834fbc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.231+0000 7f6d3f7fe640 1 -- 172.21.15.44:0/627210146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 msgr2=0x560834fb9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:49.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.231+0000 7f6d3f7fe640 1 --2- 172.21.15.44:0/627210146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fb9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.231+0000 7f6d3f7fe640 1 -- 172.21.15.44:0/627210146 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560834ee12c0 con 0x560834fb1020 2024-04-21T23:14:49.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.231+0000 7f6d3f7fe640 1 --2- 172.21.15.44:0/627210146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 0x560834fb1400 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6d300980b0 tx=0x7f6d3009bb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5db55ada63eb7961 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:49.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.231+0000 7f6d3ffff640 1 --2- 172.21.15.44:0/3893865164 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6d280a2d00 0x7f6d280a51c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x560834fac8e0 tx=0x7f6d38011460 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:49.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.231+0000 7f6d25ffb640 1 -- 172.21.15.44:0/627210146 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d300a0000 con 0x560834fb1020 2024-04-21T23:14:49.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.231+0000 7f6d25ffb640 1 -- 172.21.15.44:0/627210146 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d300b4e50 con 0x560834fb1020 2024-04-21T23:14:49.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.231+0000 7f6d25ffb640 1 -- 172.21.15.44:0/627210146 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d300cd100 con 0x560834fb1020 2024-04-21T23:14:49.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.232+0000 7f6d48672740 1 -- 172.21.15.44:0/627210146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 msgr2=0x560834fb1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:49.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.232+0000 7f6d48672740 1 --2- 172.21.15.44:0/627210146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 0x560834fb1400 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6d300980b0 tx=0x7f6d3009bb50 comp rx=0 tx=0).stop 2024-04-21T23:14:49.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.232+0000 7f6d48672740 1 -- 172.21.15.44:0/627210146 shutdown_connections 2024-04-21T23:14:49.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.232+0000 7f6d48672740 1 --2- 172.21.15.44:0/627210146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fba520 0x560834fbc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.232+0000 7f6d48672740 1 --2- 172.21.15.44:0/627210146 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fb9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.232+0000 7f6d48672740 1 --2- 172.21.15.44:0/627210146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 0x560834fb1400 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.232+0000 7f6d48672740 1 -- 172.21.15.44:0/627210146 >> 172.21.15.44:0/627210146 conn(0x560834fab750 msgr2=0x560834fabb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:49.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.232+0000 7f6d48672740 1 -- 172.21.15.44:0/627210146 shutdown_connections 2024-04-21T23:14:49.237 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.232+0000 7f6d48672740 1 -- 172.21.15.44:0/627210146 wait complete. 2024-04-21T23:14:49.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.232+0000 7f6d48672740 1 -- start start 2024-04-21T23:14:49.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.233+0000 7f6d48672740 1 Processor -- start 2024-04-21T23:14:49.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.233+0000 7f6d48672740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 0x560834fb9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:49.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.233+0000 7f6d48672740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fb7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:49.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.233+0000 7f6d48672740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fba520 0x560834fb78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:49.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.233+0000 7f6d48672740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560834fbd310 con 0x560834fb1940 2024-04-21T23:14:49.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.233+0000 7f6d48672740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x560834ed0f10 con 0x560834fba520 2024-04-21T23:14:49.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.233+0000 7f6d48672740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560834fbd490 con 0x560834fb1020 2024-04-21T23:14:49.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.233+0000 7f6d48672740 10 client.0 _ll_register_callbacks cb 0x560834c28be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:49.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.233+0000 7f6d3f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 0x560834fb9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:49.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.233+0000 7f6d3effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fb7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:49.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.233+0000 7f6d3f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 0x560834fb9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:42668/0 (socket says 172.21.15.44:42668) 2024-04-21T23:14:49.238 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.233+0000 7f6d3f7fe640 1 -- 172.21.15.44:0/398375108 learned_addr learned my addr 172.21.15.44:0/398375108 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:49.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.233+0000 7f6d3ffff640 1 --2- 172.21.15.44:0/398375108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fba520 0x560834fb78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:49.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.233+0000 7f6d3f7fe640 1 -- 172.21.15.44:0/398375108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fba520 msgr2=0x560834fb78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:49.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.233+0000 7f6d3f7fe640 1 --2- 172.21.15.44:0/398375108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fba520 0x560834fb78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.233+0000 7f6d3f7fe640 1 -- 172.21.15.44:0/398375108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 msgr2=0x560834fb7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:49.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.233+0000 7f6d3f7fe640 1 --2- 172.21.15.44:0/398375108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fb7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.233+0000 7f6d3f7fe640 1 -- 172.21.15.44:0/398375108 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560834ea2c80 con 0x560834fb1020 2024-04-21T23:14:49.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.233+0000 7f6d3f7fe640 1 --2- 172.21.15.44:0/398375108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 0x560834fb9f00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6d3009bbe0 tx=0x7f6d3009c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8be71ca5b4a4eba server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:49.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.233+0000 7f6d1effd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:14:49.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.234+0000 7f6d1effd640 1 -- 172.21.15.44:0/398375108 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d300982c0 con 0x560834fb1020 2024-04-21T23:14:49.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.234+0000 7f6d1effd640 1 -- 172.21.15.44:0/398375108 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9445+0+0 (secure 0 0 0) 0x7f6d300c7cd0 con 0x560834fb1020 2024-04-21T23:14:49.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.234+0000 7f6d48672740 1 -- 172.21.15.44:0/398375108 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d300b4e50 con 0x560834fb1020 2024-04-21T23:14:49.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.234+0000 7f6d1effd640 1 -- 172.21.15.44:0/398375108 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6d30098460 con 0x560834fb1020 2024-04-21T23:14:49.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.234+0000 7f6d48672740 10 client.26566 fetch_fsmap learned FSMap version 27 2024-04-21T23:14:49.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.234+0000 7f6d48672740 1 -- 172.21.15.44:0/398375108 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560834c293f0 con 0x560834fb1020 2024-04-21T23:14:49.239 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.234+0000 7f6d1effd640 1 -- 172.21.15.44:0/398375108 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d300b4e50 con 0x560834fb1020 2024-04-21T23:14:49.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.234+0000 7f6d1effd640 1 -- 172.21.15.44:0/398375108 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 27) ==== 8330+0+0 (secure 0 0 0) 0x7f6d300baab0 con 0x560834fb1020 2024-04-21T23:14:49.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.234+0000 7f6d48672740 10 client.26566 fetch_fsmap finished waiting for FSMap version 27 2024-04-21T23:14:49.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.234+0000 7f6d48672740 10 client.26566 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:49.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.234+0000 7f6d48672740 20 client.26566 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:49.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.234+0000 7f6d48672740 1 --2- 172.21.15.44:0/398375108 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560834fca480 0x560834fcc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:49.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.234+0000 7f6d48672740 4 client.26566 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:14:49.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.234+0000 7f6d48672740 1 -- 172.21.15.44:0/398375108 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560834f22a70 con 0x560834fca480 2024-04-21T23:14:49.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.235+0000 7f6d3effd640 1 --2- 172.21.15.44:0/398375108 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560834fca480 0x560834fcc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:49.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.235+0000 7f6d3effd640 1 --2- 172.21.15.44:0/398375108 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560834fca480 0x560834fcc870 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9aeb3b6927921e8 server_cookie=2326a789f0e87199 in_seq=0 out_seq=0 2024-04-21T23:14:49.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.235+0000 7f6d1effd640 10 client.26566 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:49.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.236+0000 7f6d1effd640 1 -- 172.21.15.44:0/398375108 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560834f22a70 con 0x560834fca480 2024-04-21T23:14:49.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.236+0000 7f6d1effd640 10 client.26566 handle_command_reply: tid=0 2024-04-21T23:14:49.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.236+0000 7f6d48672740 1 client.26566 shutdown 2024-04-21T23:14:49.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.236+0000 7f6d48672740 1 -- 172.21.15.44:0/398375108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 msgr2=0x560834fb9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:49.240 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.236+0000 7f6d48672740 1 --2- 172.21.15.44:0/398375108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 0x560834fb9f00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6d3009bbe0 tx=0x7f6d3009c490 comp rx=0 tx=0).stop 2024-04-21T23:14:49.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.236+0000 7f6d48672740 1 -- 172.21.15.44:0/398375108 shutdown_connections 2024-04-21T23:14:49.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.236+0000 7f6d48672740 1 --2- 172.21.15.44:0/398375108 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560834fca480 0x560834fcc870 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.236+0000 7f6d48672740 1 --2- 172.21.15.44:0/398375108 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fba520 0x560834fb78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.236+0000 7f6d48672740 1 --2- 172.21.15.44:0/398375108 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fb7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.236+0000 7f6d48672740 1 --2- 172.21.15.44:0/398375108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 0x560834fb9f00 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.236+0000 7f6d1effd640 0 client.26566 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:49.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.236+0000 7f6d48672740 1 -- 172.21.15.44:0/398375108 >> 172.21.15.44:0/398375108 conn(0x560834fab750 msgr2=0x560834fad4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:49.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.236+0000 7f6d48672740 1 -- 172.21.15.44:0/398375108 shutdown_connections 2024-04-21T23:14:49.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.236+0000 7f6d48672740 1 -- 172.21.15.44:0/398375108 wait complete. 2024-04-21T23:14:49.241 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.236+0000 7f6d48672740 20 client.26566 trim_cache size 0 max 16384 2024-04-21T23:14:49.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.250+0000 7f6d48672740 1 Processor -- start 2024-04-21T23:14:49.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.250+0000 7f6d48672740 1 -- start start 2024-04-21T23:14:49.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.250+0000 7f6d48672740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 0x560834fb67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:49.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.250+0000 7f6d48672740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fb99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:49.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.250+0000 7f6d48672740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fba520 0x560834fb9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:49.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.251+0000 7f6d48672740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6d401acc20 con 0x560834fb1940 2024-04-21T23:14:49.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.251+0000 7f6d48672740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6d401af4a0 con 0x560834fba520 2024-04-21T23:14:49.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.251+0000 7f6d48672740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6d401ad290 con 0x560834fb1020 2024-04-21T23:14:49.250 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.251+0000 7f6d3f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 0x560834fb67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.251+0000 7f6d3effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fb99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.251+0000 7f6d3effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fb99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42572/0 (socket says 172.21.15.44:42572) 2024-04-21T23:14:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.251+0000 7f6d3ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fba520 0x560834fb9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.251+0000 7f6d3f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 0x560834fb67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:42678/0 (socket says 172.21.15.44:42678) 2024-04-21T23:14:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.251+0000 7f6d3effd640 1 -- 172.21.15.44:0/2458080320 learned_addr learned my addr 172.21.15.44:0/2458080320 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.251+0000 7f6d3f7fe640 1 -- 172.21.15.44:0/2458080320 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fba520 msgr2=0x560834fb9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.251+0000 7f6d3f7fe640 1 --2- 172.21.15.44:0/2458080320 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fba520 0x560834fb9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.251+0000 7f6d3f7fe640 1 -- 172.21.15.44:0/2458080320 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 msgr2=0x560834fb99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.251+0000 7f6d3f7fe640 1 --2- 172.21.15.44:0/2458080320 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fb99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.251+0000 7f6d3f7fe640 1 -- 172.21.15.44:0/2458080320 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560834cf37d0 con 0x560834fb1020 2024-04-21T23:14:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.251+0000 7f6d3effd640 1 --2- 172.21.15.44:0/2458080320 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fb99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.251+0000 7f6d3f7fe640 1 --2- 172.21.15.44:0/2458080320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 0x560834fb67f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6d3009b950 tx=0x7f6d300a4a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a91834f9441d759d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.251+0000 7f6d257fa640 1 -- 172.21.15.44:0/2458080320 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d300b1020 con 0x560834fb1020 2024-04-21T23:14:49.251 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.251+0000 7f6d257fa640 1 -- 172.21.15.44:0/2458080320 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d300b4e50 con 0x560834fb1020 2024-04-21T23:14:49.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.251+0000 7f6d257fa640 1 -- 172.21.15.44:0/2458080320 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d300a0000 con 0x560834fb1020 2024-04-21T23:14:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.252+0000 7f6d48672740 1 -- 172.21.15.44:0/2458080320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 msgr2=0x560834fb67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.252+0000 7f6d48672740 1 --2- 172.21.15.44:0/2458080320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 0x560834fb67f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6d3009b950 tx=0x7f6d300a4a00 comp rx=0 tx=0).stop 2024-04-21T23:14:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.252+0000 7f6d48672740 1 -- 172.21.15.44:0/2458080320 shutdown_connections 2024-04-21T23:14:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.252+0000 7f6d48672740 1 --2- 172.21.15.44:0/2458080320 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fba520 0x560834fb9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.252+0000 7f6d48672740 1 --2- 172.21.15.44:0/2458080320 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fb99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.252+0000 7f6d48672740 1 --2- 172.21.15.44:0/2458080320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fb1020 0x560834fb67f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.252+0000 7f6d48672740 1 -- 172.21.15.44:0/2458080320 >> 172.21.15.44:0/2458080320 conn(0x560834fab750 msgr2=0x560834faa850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.252+0000 7f6d48672740 1 -- 172.21.15.44:0/2458080320 shutdown_connections 2024-04-21T23:14:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.252+0000 7f6d48672740 1 -- 172.21.15.44:0/2458080320 wait complete. 2024-04-21T23:14:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.252+0000 7f6d48672740 1 -- start start 2024-04-21T23:14:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.253+0000 7f6d48672740 1 Processor -- start 2024-04-21T23:14:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.253+0000 7f6d48672740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fb1020 0x560834fcfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.253+0000 7f6d48672740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fcce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:49.256 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.253+0000 7f6d48672740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fba520 0x560834fcd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:49.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.253+0000 7f6d48672740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6d401ad960 con 0x560834fb1940 2024-04-21T23:14:49.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.253+0000 7f6d48672740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6d401aedd0 con 0x560834fb1020 2024-04-21T23:14:49.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.253+0000 7f6d48672740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6d401ae030 con 0x560834fba520 2024-04-21T23:14:49.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.253+0000 7f6d48672740 10 client.0 _ll_register_callbacks cb 0x7f6d30005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:49.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.253+0000 7f6d3effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fcce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:49.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.253+0000 7f6d3effd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fcce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42574/0 (socket says 172.21.15.44:42574) 2024-04-21T23:14:49.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.253+0000 7f6d3effd640 1 -- 172.21.15.44:0/1853733302 learned_addr learned my addr 172.21.15.44:0/1853733302 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:49.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.253+0000 7f6d3f7fe640 1 --2- 172.21.15.44:0/1853733302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fb1020 0x560834fcfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:49.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.253+0000 7f6d3ffff640 1 --2- 172.21.15.44:0/1853733302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fba520 0x560834fcd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:49.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.253+0000 7f6d3effd640 1 -- 172.21.15.44:0/1853733302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fba520 msgr2=0x560834fcd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:49.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.253+0000 7f6d3effd640 1 --2- 172.21.15.44:0/1853733302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fba520 0x560834fcd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.253+0000 7f6d3effd640 1 -- 172.21.15.44:0/1853733302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fb1020 msgr2=0x560834fcfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:49.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.253+0000 7f6d3effd640 1 --2- 172.21.15.44:0/1853733302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fb1020 0x560834fcfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.253+0000 7f6d3effd640 1 -- 172.21.15.44:0/1853733302 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560834fcd9a0 con 0x560834fb1940 2024-04-21T23:14:49.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.253+0000 7f6d3f7fe640 1 --2- 172.21.15.44:0/1853733302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fb1020 0x560834fcfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:49.257 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.254+0000 7f6d3effd640 1 --2- 172.21.15.44:0/1853733302 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fcce70 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f6d34005ab0 tx=0x7f6d3402c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ae1ee49023eeb63 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:49.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.254+0000 7f6d1effd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:14:49.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.254+0000 7f6d1effd640 1 -- 172.21.15.44:0/1853733302 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3402c920 con 0x560834fb1940 2024-04-21T23:14:49.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.254+0000 7f6d48672740 1 -- 172.21.15.44:0/1853733302 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d300b4e50 con 0x560834fb1940 2024-04-21T23:14:49.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.254+0000 7f6d1effd640 1 -- 172.21.15.44:0/1853733302 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9445+0+0 (secure 0 0 0) 0x7f6d3402cac0 con 0x560834fb1940 2024-04-21T23:14:49.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.254+0000 7f6d1effd640 1 -- 172.21.15.44:0/1853733302 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6d34059050 con 0x560834fb1940 2024-04-21T23:14:49.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.254+0000 7f6d48672740 10 client.17007 fetch_fsmap learned FSMap version 27 2024-04-21T23:14:49.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.254+0000 7f6d48672740 1 -- 172.21.15.44:0/1853733302 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5608350304a0 con 0x560834fb1940 2024-04-21T23:14:49.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.254+0000 7f6d1effd640 1 -- 172.21.15.44:0/1853733302 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d300b4e50 con 0x560834fb1940 2024-04-21T23:14:49.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.254+0000 7f6d1effd640 1 -- 172.21.15.44:0/1853733302 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 27) ==== 8330+0+0 (secure 0 0 0) 0x7f6d34047040 con 0x560834fb1940 2024-04-21T23:14:49.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.254+0000 7f6d48672740 10 client.17007 fetch_fsmap finished waiting for FSMap version 27 2024-04-21T23:14:49.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.254+0000 7f6d48672740 10 client.17007 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:49.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.254+0000 7f6d48672740 20 client.17007 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:49.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.254+0000 7f6d48672740 1 --2- 172.21.15.44:0/1853733302 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560835030c20 0x560835033040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:49.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.254+0000 7f6d48672740 4 client.17007 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:14:49.258 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.254+0000 7f6d48672740 1 -- 172.21.15.44:0/1853733302 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560834f1cdc0 con 0x560835030c20 2024-04-21T23:14:49.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.255+0000 7f6d3f7fe640 1 --2- 172.21.15.44:0/1853733302 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560835030c20 0x560835033040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:49.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.255+0000 7f6d3f7fe640 1 --2- 172.21.15.44:0/1853733302 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560835030c20 0x560835033040 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86fd3163157845a0 server_cookie=52e600c728e36a27 in_seq=0 out_seq=0 2024-04-21T23:14:49.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.255+0000 7f6d1effd640 10 client.17007 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:49.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.256+0000 7f6d1effd640 1 -- 172.21.15.44:0/1853733302 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560834f1cdc0 con 0x560835030c20 2024-04-21T23:14:49.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.256+0000 7f6d1effd640 10 client.17007 handle_command_reply: tid=0 2024-04-21T23:14:49.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.256+0000 7f6d48672740 1 client.17007 shutdown 2024-04-21T23:14:49.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.256+0000 7f6d48672740 1 -- 172.21.15.44:0/1853733302 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 msgr2=0x560834fcce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:49.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.256+0000 7f6d48672740 1 --2- 172.21.15.44:0/1853733302 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fcce70 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f6d34005ab0 tx=0x7f6d3402c880 comp rx=0 tx=0).stop 2024-04-21T23:14:49.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.256+0000 7f6d48672740 1 -- 172.21.15.44:0/1853733302 shutdown_connections 2024-04-21T23:14:49.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.256+0000 7f6d48672740 1 --2- 172.21.15.44:0/1853733302 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560835030c20 0x560835033040 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.256+0000 7f6d48672740 1 --2- 172.21.15.44:0/1853733302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560834fba520 0x560834fcd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.256+0000 7f6d48672740 1 --2- 172.21.15.44:0/1853733302 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560834fb1940 0x560834fcce70 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.256+0000 7f6d48672740 1 --2- 172.21.15.44:0/1853733302 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560834fb1020 0x560834fcfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.256+0000 7f6d48672740 1 -- 172.21.15.44:0/1853733302 >> 172.21.15.44:0/1853733302 conn(0x560834fab750 msgr2=0x560834fc56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:49.259 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.256+0000 7f6d1effd640 0 client.17007 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:49.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.256+0000 7f6d48672740 1 -- 172.21.15.44:0/1853733302 shutdown_connections 2024-04-21T23:14:49.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.256+0000 7f6d48672740 1 -- 172.21.15.44:0/1853733302 wait complete. 2024-04-21T23:14:49.260 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.256+0000 7f6d48672740 20 client.17007 trim_cache size 0 max 16384 2024-04-21T23:14:49.260 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-21T23:14:49.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.257+0000 7f6d45a8b640 1 -- 172.21.15.44:0/3893865164 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6d280a2d00 msgr2=0x7f6d280a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:49.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.257+0000 7f6d45a8b640 1 --2- 172.21.15.44:0/3893865164 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6d280a2d00 0x7f6d280a51c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x560834fac8e0 tx=0x7f6d38011460 comp rx=0 tx=0).stop 2024-04-21T23:14:49.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.257+0000 7f6d45a8b640 1 -- 172.21.15.44:0/3893865164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d401029b0 msgr2=0x7f6d40198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:49.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.257+0000 7f6d45a8b640 1 --2- 172.21.15.44:0/3893865164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d401029b0 0x7f6d40198090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6d3000e9f0 tx=0x7f6d3000eeb0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.257+0000 7f6d45a8b640 1 -- 172.21.15.44:0/3893865164 shutdown_connections 2024-04-21T23:14:49.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.257+0000 7f6d45a8b640 1 --2- 172.21.15.44:0/3893865164 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6d280a2d00 0x7f6d280a51c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.257+0000 7f6d45a8b640 1 --2- 172.21.15.44:0/3893865164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6d4010bcf0 0x7f6d4019c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.257+0000 7f6d45a8b640 1 --2- 172.21.15.44:0/3893865164 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d401032d0 0x7f6d401985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.257+0000 7f6d45a8b640 1 --2- 172.21.15.44:0/3893865164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d401029b0 0x7f6d40198090 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:49.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.257+0000 7f6d45a8b640 1 -- 172.21.15.44:0/3893865164 >> 172.21.15.44:0/3893865164 conn(0x7f6d400777f0 msgr2=0x7f6d400fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:49.261 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.257+0000 7f6d45a8b640 1 -- 172.21.15.44:0/3893865164 shutdown_connections 2024-04-21T23:14:49.262 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:49.257+0000 7f6d45a8b640 1 -- 172.21.15.44:0/3893865164 wait complete. 2024-04-21T23:14:49.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:49 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3501182639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:14:49.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:49 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3501182639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:14:49.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:49 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3501182639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:14:49.703 INFO:tasks.workunit.client.0.smithi044.stdout: 1 700562 13.25 MB/sec execute 159 sec latency 28.846 ms 2024-04-21T23:14:50.267 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:14:50.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:14:50.267 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:14:50.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.381+0000 7fab3ee78640 1 -- 172.21.15.44:0/812732048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab38100fd0 msgr2=0x7fab3810d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:50.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.381+0000 7fab3ee78640 1 --2- 172.21.15.44:0/812732048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab38100fd0 0x7fab3810d4e0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fab30009cd0 tx=0x7fab3002c0f0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.381+0000 7fab3ee78640 1 -- 172.21.15.44:0/812732048 shutdown_connections 2024-04-21T23:14:50.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.381+0000 7fab3ee78640 1 --2- 172.21.15.44:0/812732048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab38100fd0 0x7fab3810d4e0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.381+0000 7fab3ee78640 1 --2- 172.21.15.44:0/812732048 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab38100610 0x7fab38100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.381+0000 7fab3ee78640 1 --2- 172.21.15.44:0/812732048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab381066b0 0x7fab38106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.381+0000 7fab3ee78640 1 -- 172.21.15.44:0/812732048 >> 172.21.15.44:0/812732048 conn(0x7fab380777f0 msgr2=0x7fab38077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.381+0000 7fab3ee78640 1 -- 172.21.15.44:0/812732048 shutdown_connections 2024-04-21T23:14:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.381+0000 7fab3ee78640 1 -- 172.21.15.44:0/812732048 wait complete. 2024-04-21T23:14:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.382+0000 7fab3ee78640 1 Processor -- start 2024-04-21T23:14:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.382+0000 7fab3ee78640 1 -- start start 2024-04-21T23:14:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.382+0000 7fab3ee78640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab38100610 0x7fab3819a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.382+0000 7fab3ee78640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab38100fd0 0x7fab3819a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.382+0000 7fab3ee78640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab381066b0 0x7fab3819ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.382+0000 7fab3ee78640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fab38110a30 con 0x7fab381066b0 2024-04-21T23:14:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.382+0000 7fab3ee78640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fab381108b0 con 0x7fab38100610 2024-04-21T23:14:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.382+0000 7fab3ee78640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fab38110bb0 con 0x7fab38100fd0 2024-04-21T23:14:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.382+0000 7fab3d415640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab381066b0 0x7fab3819ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.382+0000 7fab37fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab38100fd0 0x7fab3819a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.383 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.382+0000 7fab3d415640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab381066b0 0x7fab3819ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42596/0 (socket says 172.21.15.44:42596) 2024-04-21T23:14:50.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.382+0000 7fab3d415640 1 -- 172.21.15.44:0/1022327995 learned_addr learned my addr 172.21.15.44:0/1022327995 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:50.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.382+0000 7fab3cc14640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab38100610 0x7fab3819a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.382+0000 7fab3d415640 1 -- 172.21.15.44:0/1022327995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab38100fd0 msgr2=0x7fab3819a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:50.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.382+0000 7fab3d415640 1 --2- 172.21.15.44:0/1022327995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab38100fd0 0x7fab3819a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.382+0000 7fab3d415640 1 -- 172.21.15.44:0/1022327995 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab38100610 msgr2=0x7fab3819a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:50.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.383+0000 7fab3d415640 1 --2- 172.21.15.44:0/1022327995 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab38100610 0x7fab3819a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.383+0000 7fab3d415640 1 -- 172.21.15.44:0/1022327995 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab3819f1a0 con 0x7fab381066b0 2024-04-21T23:14:50.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.383+0000 7fab37fff640 1 --2- 172.21.15.44:0/1022327995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab38100fd0 0x7fab3819a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:50.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.383+0000 7fab3d415640 1 --2- 172.21.15.44:0/1022327995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab381066b0 0x7fab3819ea70 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fab30005e30 tx=0x7fab30032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:50.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.383+0000 7fab35ffb640 1 -- 172.21.15.44:0/1022327995 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab30041020 con 0x7fab381066b0 2024-04-21T23:14:50.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.383+0000 7fab35ffb640 1 -- 172.21.15.44:0/1022327995 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab30032e80 con 0x7fab381066b0 2024-04-21T23:14:50.384 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.383+0000 7fab35ffb640 1 -- 172.21.15.44:0/1022327995 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab30039040 con 0x7fab381066b0 2024-04-21T23:14:50.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.383+0000 7fab3ee78640 1 -- 172.21.15.44:0/1022327995 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab3819f430 con 0x7fab381066b0 2024-04-21T23:14:50.385 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.383+0000 7fab3ee78640 1 -- 172.21.15.44:0/1022327995 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab381a6d20 con 0x7fab381066b0 2024-04-21T23:14:50.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.386+0000 7fab35ffb640 1 -- 172.21.15.44:0/1022327995 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fab30044020 con 0x7fab381066b0 2024-04-21T23:14:50.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.386+0000 7fab41a59740 1 Processor -- start 2024-04-21T23:14:50.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.386+0000 7fab35ffb640 1 --2- 172.21.15.44:0/1022327995 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fab280a2d00 0x7fab280a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.386+0000 7fab41a59740 1 -- start start 2024-04-21T23:14:50.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.386+0000 7fab37fff640 1 --2- 172.21.15.44:0/1022327995 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fab280a2d00 0x7fab280a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.386+0000 7fab35ffb640 1 -- 172.21.15.44:0/1022327995 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9445+0+0 (secure 0 0 0) 0x7fab300f1180 con 0x7fab381066b0 2024-04-21T23:14:50.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.386+0000 7fab41a59740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da4a720 0x55953da49470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.386+0000 7fab41a59740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da49a40 0x55953da49ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.387+0000 7fab41a59740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da52db0 0x55953da551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.387+0000 7fab41a59740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55953d9ad170 con 0x55953da4a720 2024-04-21T23:14:50.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.387+0000 7fab41a59740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55953d9557c0 con 0x55953da52db0 2024-04-21T23:14:50.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.387+0000 7fab41a59740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55953d9a9130 con 0x55953da49a40 2024-04-21T23:14:50.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.387+0000 7fab37fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da49a40 0x55953da49ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.387+0000 7fab37fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da49a40 0x55953da49ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:42720/0 (socket says 172.21.15.44:42720) 2024-04-21T23:14:50.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.387+0000 7fab37fff640 1 -- 172.21.15.44:0/340209769 learned_addr learned my addr 172.21.15.44:0/340209769 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:50.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.387+0000 7fab3d415640 1 --2- 172.21.15.44:0/340209769 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da52db0 0x55953da551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.387+0000 7fab3cc14640 1 --2- 172.21.15.44:0/340209769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da4a720 0x55953da49470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.387+0000 7fab3d415640 1 -- 172.21.15.44:0/340209769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da49a40 msgr2=0x55953da49ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:50.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.387+0000 7fab3d415640 1 --2- 172.21.15.44:0/340209769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da49a40 0x55953da49ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.387+0000 7fab3d415640 1 -- 172.21.15.44:0/340209769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da4a720 msgr2=0x55953da49470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:50.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.387+0000 7fab3d415640 1 --2- 172.21.15.44:0/340209769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da4a720 0x55953da49470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.387+0000 7fab3d415640 1 -- 172.21.15.44:0/340209769 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55953d966e60 con 0x55953da52db0 2024-04-21T23:14:50.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.387+0000 7fab37fff640 1 --2- 172.21.15.44:0/1022327995 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fab280a2d00 0x7fab280a51c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fab24011950 tx=0x7fab24011d50 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:50.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.387+0000 7fab3d415640 1 --2- 172.21.15.44:0/340209769 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da52db0 0x55953da551a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fab300ee640 tx=0x7fab300bd000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad960f77774d5e4a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:50.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.387+0000 7fab22ffd640 1 -- 172.21.15.44:0/340209769 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab300c17c0 con 0x55953da52db0 2024-04-21T23:14:50.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.387+0000 7fab22ffd640 1 -- 172.21.15.44:0/340209769 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab300c6080 con 0x55953da52db0 2024-04-21T23:14:50.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.387+0000 7fab22ffd640 1 -- 172.21.15.44:0/340209769 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab300c5bf0 con 0x55953da52db0 2024-04-21T23:14:50.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.388+0000 7fab41a59740 1 -- 172.21.15.44:0/340209769 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da52db0 msgr2=0x55953da551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:50.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.388+0000 7fab41a59740 1 --2- 172.21.15.44:0/340209769 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da52db0 0x55953da551a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fab300ee640 tx=0x7fab300bd000 comp rx=0 tx=0).stop 2024-04-21T23:14:50.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.388+0000 7fab41a59740 1 -- 172.21.15.44:0/340209769 shutdown_connections 2024-04-21T23:14:50.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.388+0000 7fab41a59740 1 --2- 172.21.15.44:0/340209769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da49a40 0x55953da49ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.388+0000 7fab41a59740 1 --2- 172.21.15.44:0/340209769 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da52db0 0x55953da551a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.388+0000 7fab41a59740 1 --2- 172.21.15.44:0/340209769 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da4a720 0x55953da49470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.388+0000 7fab41a59740 1 -- 172.21.15.44:0/340209769 >> 172.21.15.44:0/340209769 conn(0x55953da104f0 msgr2=0x55953da44490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:50.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.388+0000 7fab41a59740 1 -- 172.21.15.44:0/340209769 shutdown_connections 2024-04-21T23:14:50.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.388+0000 7fab41a59740 1 -- 172.21.15.44:0/340209769 wait complete. 2024-04-21T23:14:50.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.388+0000 7fab41a59740 1 -- start start 2024-04-21T23:14:50.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.389+0000 7fab41a59740 1 Processor -- start 2024-04-21T23:14:50.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.389+0000 7fab41a59740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da49a40 0x55953da4fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.389+0000 7fab41a59740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da4a720 0x55953da50270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.389+0000 7fab41a59740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da52db0 0x55953da507b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.389+0000 7fab41a59740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55953da559e0 con 0x55953da4a720 2024-04-21T23:14:50.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.389+0000 7fab41a59740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55953d968d40 con 0x55953da52db0 2024-04-21T23:14:50.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.389+0000 7fab41a59740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55953da55b60 con 0x55953da49a40 2024-04-21T23:14:50.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.389+0000 7fab41a59740 10 client.0 _ll_register_callbacks cb 0x55953d9a4820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:50.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.389+0000 7fab37fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da4a720 0x55953da50270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.389+0000 7fab3d415640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da52db0 0x55953da507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.389+0000 7fab3cc14640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da49a40 0x55953da4fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.389+0000 7fab37fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da4a720 0x55953da50270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42616/0 (socket says 172.21.15.44:42616) 2024-04-21T23:14:50.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.389+0000 7fab37fff640 1 -- 172.21.15.44:0/1161076374 learned_addr learned my addr 172.21.15.44:0/1161076374 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:50.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.389+0000 7fab3d415640 1 -- 172.21.15.44:0/1161076374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da49a40 msgr2=0x55953da4fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:50.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.389+0000 7fab3d415640 1 --2- 172.21.15.44:0/1161076374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da49a40 0x55953da4fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.389+0000 7fab3d415640 1 -- 172.21.15.44:0/1161076374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da4a720 msgr2=0x55953da50270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:50.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.389+0000 7fab3d415640 1 --2- 172.21.15.44:0/1161076374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da4a720 0x55953da50270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.389+0000 7fab3d415640 1 -- 172.21.15.44:0/1161076374 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55953d7897e0 con 0x55953da52db0 2024-04-21T23:14:50.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.389+0000 7fab3cc14640 1 --2- 172.21.15.44:0/1161076374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da49a40 0x55953da4fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:50.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.389+0000 7fab37fff640 1 --2- 172.21.15.44:0/1161076374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da4a720 0x55953da50270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:50.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.390+0000 7fab3d415640 1 --2- 172.21.15.44:0/1161076374 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da52db0 0x55953da507b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fab30009ca0 tx=0x7fab300bcf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b526299ae45bc9b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:50.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.390+0000 7fab17fff640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:14:50.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.390+0000 7fab17fff640 1 -- 172.21.15.44:0/1161076374 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab300c17c0 con 0x55953da52db0 2024-04-21T23:14:50.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.390+0000 7fab41a59740 1 -- 172.21.15.44:0/1161076374 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab300c6080 con 0x55953da52db0 2024-04-21T23:14:50.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.390+0000 7fab17fff640 1 -- 172.21.15.44:0/1161076374 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9445+0+0 (secure 0 0 0) 0x7fab300bb560 con 0x55953da52db0 2024-04-21T23:14:50.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.390+0000 7fab17fff640 1 -- 172.21.15.44:0/1161076374 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fab300a3050 con 0x55953da52db0 2024-04-21T23:14:50.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.390+0000 7fab41a59740 10 client.26579 fetch_fsmap learned FSMap version 27 2024-04-21T23:14:50.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.390+0000 7fab41a59740 1 -- 172.21.15.44:0/1161076374 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55953d7e9f70 con 0x55953da52db0 2024-04-21T23:14:50.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.390+0000 7fab17fff640 1 -- 172.21.15.44:0/1161076374 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab300c6080 con 0x55953da52db0 2024-04-21T23:14:50.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.390+0000 7fab17fff640 1 -- 172.21.15.44:0/1161076374 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 27) ==== 8330+0+0 (secure 0 0 0) 0x7fab3003c030 con 0x55953da52db0 2024-04-21T23:14:50.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.390+0000 7fab41a59740 10 client.26579 fetch_fsmap finished waiting for FSMap version 27 2024-04-21T23:14:50.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.390+0000 7fab41a59740 10 client.26579 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:50.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.390+0000 7fab41a59740 20 client.26579 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:50.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.390+0000 7fab41a59740 1 --2- 172.21.15.44:0/1161076374 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55953da629d0 0x55953da64dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.390+0000 7fab41a59740 4 client.26579 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:14:50.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.390+0000 7fab41a59740 1 -- 172.21.15.44:0/1161076374 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55953d981450 con 0x55953da629d0 2024-04-21T23:14:50.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.391+0000 7fab3cc14640 1 --2- 172.21.15.44:0/1161076374 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55953da629d0 0x55953da64dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.391+0000 7fab17fff640 10 client.26579 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:50.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.391+0000 7fab3cc14640 1 --2- 172.21.15.44:0/1161076374 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55953da629d0 0x55953da64dc0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41708110fc924ba3 server_cookie=fbd02bb7508f5dfa in_seq=0 out_seq=0 2024-04-21T23:14:50.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.392+0000 7fab17fff640 1 -- 172.21.15.44:0/1161076374 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55953d981450 con 0x55953da629d0 2024-04-21T23:14:50.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.392+0000 7fab17fff640 10 client.26579 handle_command_reply: tid=0 2024-04-21T23:14:50.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.392+0000 7fab41a59740 1 client.26579 shutdown 2024-04-21T23:14:50.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.392+0000 7fab41a59740 1 -- 172.21.15.44:0/1161076374 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da52db0 msgr2=0x55953da507b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:50.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.392+0000 7fab41a59740 1 --2- 172.21.15.44:0/1161076374 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da52db0 0x55953da507b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fab30009ca0 tx=0x7fab300bcf20 comp rx=0 tx=0).stop 2024-04-21T23:14:50.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.392+0000 7fab41a59740 1 -- 172.21.15.44:0/1161076374 shutdown_connections 2024-04-21T23:14:50.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.392+0000 7fab41a59740 1 --2- 172.21.15.44:0/1161076374 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55953da629d0 0x55953da64dc0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.392+0000 7fab41a59740 1 --2- 172.21.15.44:0/1161076374 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da52db0 0x55953da507b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.392+0000 7fab41a59740 1 --2- 172.21.15.44:0/1161076374 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da4a720 0x55953da50270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.392+0000 7fab17fff640 0 client.26579 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:50.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.392+0000 7fab41a59740 1 --2- 172.21.15.44:0/1161076374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da49a40 0x55953da4fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.392+0000 7fab41a59740 1 -- 172.21.15.44:0/1161076374 >> 172.21.15.44:0/1161076374 conn(0x55953da104f0 msgr2=0x55953da461e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:50.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.392+0000 7fab41a59740 1 -- 172.21.15.44:0/1161076374 shutdown_connections 2024-04-21T23:14:50.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.392+0000 7fab41a59740 1 -- 172.21.15.44:0/1161076374 wait complete. 2024-04-21T23:14:50.398 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.392+0000 7fab41a59740 20 client.26579 trim_cache size 0 max 16384 2024-04-21T23:14:50.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.420+0000 7fab41a59740 1 Processor -- start 2024-04-21T23:14:50.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.421+0000 7fab41a59740 1 -- start start 2024-04-21T23:14:50.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.422+0000 7fab41a59740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da49a40 0x55953da52460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.422+0000 7fab41a59740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da4a720 0x55953da6a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.422+0000 7fab41a59740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da52db0 0x55953da6aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.423+0000 7fab41a59740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fab381aed00 con 0x55953da52db0 2024-04-21T23:14:50.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.423+0000 7fab37fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da4a720 0x55953da6a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.423+0000 7fab37fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da4a720 0x55953da6a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:42742/0 (socket says 172.21.15.44:42742) 2024-04-21T23:14:50.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.423+0000 7fab37fff640 1 -- 172.21.15.44:0/2438852016 learned_addr learned my addr 172.21.15.44:0/2438852016 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:50.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.423+0000 7fab3d415640 1 --2- 172.21.15.44:0/2438852016 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da52db0 0x55953da6aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.423+0000 7fab41a59740 1 -- 172.21.15.44:0/2438852016 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fab381b1580 con 0x55953da49a40 2024-04-21T23:14:50.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.423+0000 7fab41a59740 1 -- 172.21.15.44:0/2438852016 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fab381af370 con 0x55953da4a720 2024-04-21T23:14:50.422 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.423+0000 7fab3cc14640 1 --2- 172.21.15.44:0/2438852016 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da49a40 0x55953da52460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.423+0000 7fab37fff640 1 -- 172.21.15.44:0/2438852016 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da49a40 msgr2=0x55953da52460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:50.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.423+0000 7fab37fff640 1 --2- 172.21.15.44:0/2438852016 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da49a40 0x55953da52460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.423+0000 7fab37fff640 1 -- 172.21.15.44:0/2438852016 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da52db0 msgr2=0x55953da6aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:50.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.423+0000 7fab37fff640 1 --2- 172.21.15.44:0/2438852016 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da52db0 0x55953da6aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.423+0000 7fab37fff640 1 -- 172.21.15.44:0/2438852016 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55953d681170 con 0x55953da4a720 2024-04-21T23:14:50.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.423+0000 7fab3d415640 1 --2- 172.21.15.44:0/2438852016 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da52db0 0x55953da6aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:14:50.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.424+0000 7fab37fff640 1 --2- 172.21.15.44:0/2438852016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da4a720 0x55953da6a5a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fab240040f0 tx=0x7fab2401f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=319118b21394ef98 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:50.423 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.424+0000 7fab227fc640 1 -- 172.21.15.44:0/2438852016 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2402d070 con 0x55953da4a720 2024-04-21T23:14:50.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.424+0000 7fab227fc640 1 -- 172.21.15.44:0/2438852016 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab24020930 con 0x55953da4a720 2024-04-21T23:14:50.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.424+0000 7fab227fc640 1 -- 172.21.15.44:0/2438852016 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2401a040 con 0x55953da4a720 2024-04-21T23:14:50.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.425+0000 7fab41a59740 1 -- 172.21.15.44:0/2438852016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da4a720 msgr2=0x55953da6a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:50.424 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.425+0000 7fab41a59740 1 --2- 172.21.15.44:0/2438852016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da4a720 0x55953da6a5a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fab240040f0 tx=0x7fab2401f000 comp rx=0 tx=0).stop 2024-04-21T23:14:50.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.426+0000 7fab41a59740 1 -- 172.21.15.44:0/2438852016 shutdown_connections 2024-04-21T23:14:50.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.426+0000 7fab41a59740 1 --2- 172.21.15.44:0/2438852016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da4a720 0x55953da6a5a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.426+0000 7fab41a59740 1 --2- 172.21.15.44:0/2438852016 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da52db0 0x55953da6aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.426+0000 7fab41a59740 1 --2- 172.21.15.44:0/2438852016 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da49a40 0x55953da52460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.425 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.426+0000 7fab41a59740 1 -- 172.21.15.44:0/2438852016 >> 172.21.15.44:0/2438852016 conn(0x55953da104f0 msgr2=0x55953da434f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:50.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.427+0000 7fab41a59740 1 -- 172.21.15.44:0/2438852016 shutdown_connections 2024-04-21T23:14:50.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.427+0000 7fab41a59740 1 -- 172.21.15.44:0/2438852016 wait complete. 2024-04-21T23:14:50.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.428+0000 7fab41a59740 1 -- start start 2024-04-21T23:14:50.426 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.428+0000 7fab41a59740 1 Processor -- start 2024-04-21T23:14:50.427 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.428+0000 7fab41a59740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da49a40 0x55953da5afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.429+0000 7fab41a59740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da4a720 0x55953da4bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.429+0000 7fab3cc14640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da49a40 0x55953da5afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.429+0000 7fab3cc14640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da49a40 0x55953da5afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42634/0 (socket says 172.21.15.44:42634) 2024-04-21T23:14:50.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.429+0000 7fab3cc14640 1 -- 172.21.15.44:0/1722398718 learned_addr learned my addr 172.21.15.44:0/1722398718 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:50.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.429+0000 7fab41a59740 1 --2- 172.21.15.44:0/1722398718 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da52db0 0x55953da4c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.429+0000 7fab37fff640 1 --2- 172.21.15.44:0/1722398718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da4a720 0x55953da4bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.429+0000 7fab41a59740 1 -- 172.21.15.44:0/1722398718 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fab381afa40 con 0x55953da49a40 2024-04-21T23:14:50.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.429+0000 7fab41a59740 1 -- 172.21.15.44:0/1722398718 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fab381b0eb0 con 0x55953da52db0 2024-04-21T23:14:50.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.429+0000 7fab41a59740 1 -- 172.21.15.44:0/1722398718 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fab381b0110 con 0x55953da4a720 2024-04-21T23:14:50.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.429+0000 7fab41a59740 10 client.0 _ll_register_callbacks cb 0x7fab2c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:50.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.431+0000 7fab3d415640 1 --2- 172.21.15.44:0/1722398718 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da52db0 0x55953da4c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.432+0000 7fab3d415640 1 -- 172.21.15.44:0/1722398718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da4a720 msgr2=0x55953da4bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:50.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.432+0000 7fab3d415640 1 --2- 172.21.15.44:0/1722398718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da4a720 0x55953da4bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.432+0000 7fab3d415640 1 -- 172.21.15.44:0/1722398718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da49a40 msgr2=0x55953da5afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:50.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.432+0000 7fab3d415640 1 --2- 172.21.15.44:0/1722398718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da49a40 0x55953da5afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.432+0000 7fab3d415640 1 -- 172.21.15.44:0/1722398718 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55953da4ca60 con 0x55953da52db0 2024-04-21T23:14:50.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.432+0000 7fab3d415640 1 --2- 172.21.15.44:0/1722398718 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da52db0 0x55953da4c520 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x55953da6b020 tx=0x7fab300bd130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf6ff1289ec68269 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:50.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.433+0000 7fab17fff640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:14:50.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.433+0000 7fab17fff640 1 -- 172.21.15.44:0/1722398718 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab300b9760 con 0x55953da52db0 2024-04-21T23:14:50.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.433+0000 7fab17fff640 1 -- 172.21.15.44:0/1722398718 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9445+0+0 (secure 0 0 0) 0x7fab300bca80 con 0x55953da52db0 2024-04-21T23:14:50.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.434+0000 7fab3cc14640 1 --2- 172.21.15.44:0/1722398718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da49a40 0x55953da5afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:14:50.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.434+0000 7fab37fff640 1 --2- 172.21.15.44:0/1722398718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da4a720 0x55953da4bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:14:50.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.434+0000 7fab41a59740 1 -- 172.21.15.44:0/1722398718 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab24020930 con 0x55953da52db0 2024-04-21T23:14:50.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.435+0000 7fab17fff640 1 -- 172.21.15.44:0/1722398718 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fab300c6080 con 0x55953da52db0 2024-04-21T23:14:50.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.435+0000 7fab41a59740 10 client.26585 fetch_fsmap learned FSMap version 27 2024-04-21T23:14:50.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.435+0000 7fab41a59740 1 -- 172.21.15.44:0/1722398718 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55953da4ca60 con 0x55953da52db0 2024-04-21T23:14:50.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.441+0000 7fab17fff640 1 -- 172.21.15.44:0/1722398718 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab24020930 con 0x55953da52db0 2024-04-21T23:14:50.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.441+0000 7fab17fff640 1 -- 172.21.15.44:0/1722398718 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 27) ==== 8330+0+0 (secure 0 0 0) 0x7fab300c5d00 con 0x55953da52db0 2024-04-21T23:14:50.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.441+0000 7fab41a59740 10 client.26585 fetch_fsmap finished waiting for FSMap version 27 2024-04-21T23:14:50.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.441+0000 7fab41a59740 10 client.26585 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:50.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.441+0000 7fab41a59740 20 client.26585 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:50.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.441+0000 7fab41a59740 1 --2- 172.21.15.44:0/1722398718 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55953dad2740 0x55953dad4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.441+0000 7fab41a59740 4 client.26585 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:14:50.440 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.441+0000 7fab41a59740 1 -- 172.21.15.44:0/1722398718 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55953d9ba6b0 con 0x55953dad2740 2024-04-21T23:14:50.442 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.443+0000 7fab3cc14640 1 --2- 172.21.15.44:0/1722398718 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55953dad2740 0x55953dad4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.448+0000 7fab3cc14640 1 --2- 172.21.15.44:0/1722398718 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55953dad2740 0x55953dad4b60 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cae650148191040a server_cookie=84fb6eb31a56b1b in_seq=0 out_seq=0 2024-04-21T23:14:50.446 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.448+0000 7fab17fff640 10 client.26585 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:50.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.451+0000 7fab17fff640 1 -- 172.21.15.44:0/1722398718 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55953d9ba6b0 con 0x55953dad2740 2024-04-21T23:14:50.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.451+0000 7fab17fff640 10 client.26585 handle_command_reply: tid=0 2024-04-21T23:14:50.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.451+0000 7fab41a59740 1 client.26585 shutdown 2024-04-21T23:14:50.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.451+0000 7fab41a59740 1 -- 172.21.15.44:0/1722398718 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da52db0 msgr2=0x55953da4c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:50.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.451+0000 7fab41a59740 1 --2- 172.21.15.44:0/1722398718 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da52db0 0x55953da4c520 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x55953da6b020 tx=0x7fab300bd130 comp rx=0 tx=0).stop 2024-04-21T23:14:50.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.451+0000 7fab41a59740 1 -- 172.21.15.44:0/1722398718 shutdown_connections 2024-04-21T23:14:50.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.451+0000 7fab41a59740 1 --2- 172.21.15.44:0/1722398718 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55953dad2740 0x55953dad4b60 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.451+0000 7fab41a59740 1 --2- 172.21.15.44:0/1722398718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55953da4a720 0x55953da4bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.451+0000 7fab41a59740 1 --2- 172.21.15.44:0/1722398718 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55953da52db0 0x55953da4c520 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.451+0000 7fab41a59740 1 --2- 172.21.15.44:0/1722398718 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55953da49a40 0x55953da5afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.451+0000 7fab17fff640 0 client.26585 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:50.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.451+0000 7fab41a59740 1 -- 172.21.15.44:0/1722398718 >> 172.21.15.44:0/1722398718 conn(0x55953da104f0 msgr2=0x55953da458c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:50.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.451+0000 7fab41a59740 1 -- 172.21.15.44:0/1722398718 shutdown_connections 2024-04-21T23:14:50.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.451+0000 7fab41a59740 1 -- 172.21.15.44:0/1722398718 wait complete. 2024-04-21T23:14:50.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.452+0000 7fab41a59740 20 client.26585 trim_cache size 0 max 16384 2024-04-21T23:14:50.451 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:14:50.451 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-21T23:14:50.451 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "e47e9f51-eb2c-41e1-b5e8-95e495faee64", 2024-04-21T23:14:50.451 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-21T23:14:50.451 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:14:50.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.452+0000 7fab3ee78640 1 -- 172.21.15.44:0/1022327995 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fab280a2d00 msgr2=0x7fab280a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:50.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.452+0000 7fab3ee78640 1 --2- 172.21.15.44:0/1022327995 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fab280a2d00 0x7fab280a51c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fab24011950 tx=0x7fab24011d50 comp rx=0 tx=0).stop 2024-04-21T23:14:50.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.452+0000 7fab3ee78640 1 -- 172.21.15.44:0/1022327995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab381066b0 msgr2=0x7fab3819ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:50.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.452+0000 7fab3ee78640 1 --2- 172.21.15.44:0/1022327995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab381066b0 0x7fab3819ea70 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fab30005e30 tx=0x7fab30032c90 comp rx=0 tx=0).stop 2024-04-21T23:14:50.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.452+0000 7fab3ee78640 1 -- 172.21.15.44:0/1022327995 shutdown_connections 2024-04-21T23:14:50.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.452+0000 7fab3ee78640 1 --2- 172.21.15.44:0/1022327995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab38100fd0 0x7fab3819a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.452+0000 7fab3ee78640 1 --2- 172.21.15.44:0/1022327995 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fab280a2d00 0x7fab280a51c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.452+0000 7fab3ee78640 1 --2- 172.21.15.44:0/1022327995 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fab381066b0 0x7fab3819ea70 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.452+0000 7fab3ee78640 1 --2- 172.21.15.44:0/1022327995 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fab38100610 0x7fab3819a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.452+0000 7fab3ee78640 1 -- 172.21.15.44:0/1022327995 >> 172.21.15.44:0/1022327995 conn(0x7fab380777f0 msgr2=0x7fab38068a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:50.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.452+0000 7fab3ee78640 1 -- 172.21.15.44:0/1022327995 shutdown_connections 2024-04-21T23:14:50.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.452+0000 7fab3ee78640 1 -- 172.21.15.44:0/1022327995 wait complete. 2024-04-21T23:14:50.461 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:50 smithi044 ceph-mon[27934]: pgmap v185: 129 pgs: 129 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.1 MiB/s wr, 476 op/s 2024-04-21T23:14:50.462 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:14:50.462 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:14:50.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:50 smithi055 ceph-mon[31117]: pgmap v185: 129 pgs: 129 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.1 MiB/s wr, 476 op/s 2024-04-21T23:14:50.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:50 smithi069 ceph-mon[31816]: pgmap v185: 129 pgs: 129 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.1 MiB/s wr, 476 op/s 2024-04-21T23:14:50.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.574+0000 7f60107e5640 1 -- 172.21.15.44:0/2463426499 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6008100fc0 msgr2=0x7f600810d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:50.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.574+0000 7f60107e5640 1 --2- 172.21.15.44:0/2463426499 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6008100fc0 0x7f600810d4d0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8009cd0 tx=0x7f5ff802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.575+0000 7f60107e5640 1 -- 172.21.15.44:0/2463426499 shutdown_connections 2024-04-21T23:14:50.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.575+0000 7f60107e5640 1 --2- 172.21.15.44:0/2463426499 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6008100fc0 0x7f600810d4d0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.575+0000 7f60107e5640 1 --2- 172.21.15.44:0/2463426499 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6008100600 0x7f6008100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.575+0000 7f60107e5640 1 --2- 172.21.15.44:0/2463426499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60081066a0 0x7f6008106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.575+0000 7f60107e5640 1 -- 172.21.15.44:0/2463426499 >> 172.21.15.44:0/2463426499 conn(0x7f6008077850 msgr2=0x7f6008077c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:50.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.575+0000 7f60107e5640 1 -- 172.21.15.44:0/2463426499 shutdown_connections 2024-04-21T23:14:50.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.575+0000 7f60107e5640 1 -- 172.21.15.44:0/2463426499 wait complete. 2024-04-21T23:14:50.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.575+0000 7f60107e5640 1 Processor -- start 2024-04-21T23:14:50.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.575+0000 7f60107e5640 1 -- start start 2024-04-21T23:14:50.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.575+0000 7f60107e5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6008100600 0x7f60081022a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.575+0000 7f60107e5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6008100fc0 0x7f60081027e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.576+0000 7f60107e5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60081066a0 0x7f6008102d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.576+0000 7f60107e5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f60081109e0 con 0x7f6008100fc0 2024-04-21T23:14:50.578 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.576+0000 7f60107e5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6008110860 con 0x7f6008100600 2024-04-21T23:14:50.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.576+0000 7f60107e5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6008110b60 con 0x7f60081066a0 2024-04-21T23:14:50.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.576+0000 7f600dd80640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6008100fc0 0x7f60081027e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.576+0000 7f600dd80640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6008100fc0 0x7f60081027e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42652/0 (socket says 172.21.15.44:42652) 2024-04-21T23:14:50.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.576+0000 7f600dd80640 1 -- 172.21.15.44:0/4133099593 learned_addr learned my addr 172.21.15.44:0/4133099593 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:50.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.576+0000 7f600e581640 1 --2- 172.21.15.44:0/4133099593 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6008100600 0x7f60081022a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.576+0000 7f600ed82640 1 --2- 172.21.15.44:0/4133099593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60081066a0 0x7f6008102d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.576+0000 7f600dd80640 1 -- 172.21.15.44:0/4133099593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60081066a0 msgr2=0x7f6008102d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:50.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.576+0000 7f600dd80640 1 --2- 172.21.15.44:0/4133099593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60081066a0 0x7f6008102d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.576+0000 7f600dd80640 1 -- 172.21.15.44:0/4133099593 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6008100600 msgr2=0x7f60081022a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:50.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.576+0000 7f600dd80640 1 --2- 172.21.15.44:0/4133099593 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6008100600 0x7f60081022a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.576+0000 7f600dd80640 1 -- 172.21.15.44:0/4133099593 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60081a63e0 con 0x7f6008100fc0 2024-04-21T23:14:50.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.576+0000 7f600dd80640 1 --2- 172.21.15.44:0/4133099593 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6008100fc0 0x7f60081027e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00e9f0 tx=0x7f5ffc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:50.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.576+0000 7f60077fe640 1 -- 172.21.15.44:0/4133099593 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc00cd80 con 0x7f6008100fc0 2024-04-21T23:14:50.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.576+0000 7f60077fe640 1 -- 172.21.15.44:0/4133099593 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ffc010910 con 0x7f6008100fc0 2024-04-21T23:14:50.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.576+0000 7f60107e5640 1 -- 172.21.15.44:0/4133099593 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60081a66d0 con 0x7f6008100fc0 2024-04-21T23:14:50.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.576+0000 7f60077fe640 1 -- 172.21.15.44:0/4133099593 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc018990 con 0x7f6008100fc0 2024-04-21T23:14:50.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.576+0000 7f60107e5640 1 -- 172.21.15.44:0/4133099593 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60081a6c20 con 0x7f6008100fc0 2024-04-21T23:14:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.579+0000 7f60133cd740 1 Processor -- start 2024-04-21T23:14:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.579+0000 7f60077fe640 1 -- 172.21.15.44:0/4133099593 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f5ffc010ac0 con 0x7f6008100fc0 2024-04-21T23:14:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.579+0000 7f60133cd740 1 -- start start 2024-04-21T23:14:50.588 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.579+0000 7f60133cd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7cef30 0x55a43e7cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.579+0000 7f60133cd740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7cf850 0x55a43e7d7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.579+0000 7f60133cd740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7d8430 0x55a43e7da820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.579+0000 7f60133cd740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a43e733560 con 0x55a43e7d8430 2024-04-21T23:14:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.579+0000 7f60133cd740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55a43e6db730 con 0x55a43e7cf850 2024-04-21T23:14:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.579+0000 7f60133cd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a43e72f520 con 0x55a43e7cef30 2024-04-21T23:14:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.579+0000 7f600dd80640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7d8430 0x55a43e7da820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.579+0000 7f600e581640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7cf850 0x55a43e7d7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.589 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.579+0000 7f600dd80640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7d8430 0x55a43e7da820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42662/0 (socket says 172.21.15.44:42662) 2024-04-21T23:14:50.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.579+0000 7f600e581640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7cf850 0x55a43e7d7ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:59184/0 (socket says 172.21.15.44:59184) 2024-04-21T23:14:50.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.579+0000 7f600dd80640 1 -- 172.21.15.44:0/1572217743 learned_addr learned my addr 172.21.15.44:0/1572217743 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:50.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.579+0000 7f600ed82640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7cef30 0x55a43e7cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.580+0000 7f60077fe640 1 --2- 172.21.15.44:0/4133099593 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5ff00a2d00 0x7f5ff00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.580+0000 7f600e581640 1 -- 172.21.15.44:0/1572217743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7cef30 msgr2=0x55a43e7cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:50.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.580+0000 7f600e581640 1 --2- 172.21.15.44:0/1572217743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7cef30 0x55a43e7cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.580+0000 7f600e581640 1 -- 172.21.15.44:0/1572217743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7d8430 msgr2=0x55a43e7da820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:50.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.580+0000 7f600e581640 1 --2- 172.21.15.44:0/1572217743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7d8430 0x55a43e7da820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.580+0000 7f600e581640 1 -- 172.21.15.44:0/1572217743 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a43e5117d0 con 0x55a43e7cf850 2024-04-21T23:14:50.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.580+0000 7f600ed82640 1 --2- 172.21.15.44:0/4133099593 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5ff00a2d00 0x7f5ff00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.580+0000 7f60077fe640 1 -- 172.21.15.44:0/4133099593 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9445+0+0 (secure 0 0 0) 0x7f5ffc0cfbd0 con 0x7f6008100fc0 2024-04-21T23:14:50.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.580+0000 7f600e581640 1 --2- 172.21.15.44:0/1572217743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7cf850 0x55a43e7d7ef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6000009bf0 tx=0x7f6000011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4c6a8ca89cfc45e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:50.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.580+0000 7f6004ff9640 1 -- 172.21.15.44:0/1572217743 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6000009da0 con 0x55a43e7cf850 2024-04-21T23:14:50.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.580+0000 7f6004ff9640 1 -- 172.21.15.44:0/1572217743 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6000005020 con 0x55a43e7cf850 2024-04-21T23:14:50.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.580+0000 7f6004ff9640 1 -- 172.21.15.44:0/1572217743 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6000004a50 con 0x55a43e7cf850 2024-04-21T23:14:50.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.581+0000 7f60133cd740 1 -- 172.21.15.44:0/1572217743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7cf850 msgr2=0x55a43e7d7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:50.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.581+0000 7f600ed82640 1 --2- 172.21.15.44:0/4133099593 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5ff00a2d00 0x7f5ff00a51c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5ff802c700 tx=0x7f5ff80095c0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:50.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.581+0000 7f60133cd740 1 --2- 172.21.15.44:0/1572217743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7cf850 0x55a43e7d7ef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6000009bf0 tx=0x7f6000011810 comp rx=0 tx=0).stop 2024-04-21T23:14:50.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.581+0000 7f60133cd740 1 -- 172.21.15.44:0/1572217743 shutdown_connections 2024-04-21T23:14:50.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.581+0000 7f60133cd740 1 --2- 172.21.15.44:0/1572217743 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7d8430 0x55a43e7da820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.581+0000 7f60133cd740 1 --2- 172.21.15.44:0/1572217743 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7cf850 0x55a43e7d7ef0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.581+0000 7f60133cd740 1 --2- 172.21.15.44:0/1572217743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7cef30 0x55a43e7cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.581+0000 7f60133cd740 1 -- 172.21.15.44:0/1572217743 >> 172.21.15.44:0/1572217743 conn(0x55a43e7c9660 msgr2=0x55a43e7c9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:50.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.581+0000 7f60133cd740 1 -- 172.21.15.44:0/1572217743 shutdown_connections 2024-04-21T23:14:50.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.581+0000 7f60133cd740 1 -- 172.21.15.44:0/1572217743 wait complete. 2024-04-21T23:14:50.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.581+0000 7f60133cd740 1 -- start start 2024-04-21T23:14:50.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.581+0000 7f60133cd740 1 Processor -- start 2024-04-21T23:14:50.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f60133cd740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7cef30 0x55a43e7d5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f60133cd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7cf850 0x55a43e7d59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f60133cd740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7d8430 0x55a43e7d5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f60133cd740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a43e7db220 con 0x55a43e7cef30 2024-04-21T23:14:50.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f60133cd740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55a43e6eee00 con 0x55a43e7d8430 2024-04-21T23:14:50.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f60133cd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a43e7db3a0 con 0x55a43e7cf850 2024-04-21T23:14:50.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f60133cd740 10 client.0 _ll_register_callbacks cb 0x55a43e729eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:50.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f600e581640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7cef30 0x55a43e7d5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f600e581640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7cef30 0x55a43e7d5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:42664/0 (socket says 172.21.15.44:42664) 2024-04-21T23:14:50.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f600e581640 1 -- 172.21.15.44:0/1734200002 learned_addr learned my addr 172.21.15.44:0/1734200002 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:50.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f600ed82640 1 --2- 172.21.15.44:0/1734200002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7d8430 0x55a43e7d5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f600dd80640 1 --2- 172.21.15.44:0/1734200002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7cf850 0x55a43e7d59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f600e581640 1 -- 172.21.15.44:0/1734200002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7cf850 msgr2=0x55a43e7d59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:50.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f600e581640 1 --2- 172.21.15.44:0/1734200002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7cf850 0x55a43e7d59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f600e581640 1 -- 172.21.15.44:0/1734200002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7d8430 msgr2=0x55a43e7d5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:50.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f600e581640 1 --2- 172.21.15.44:0/1734200002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7d8430 0x55a43e7d5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f600e581640 1 -- 172.21.15.44:0/1734200002 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a43e4473f0 con 0x55a43e7cef30 2024-04-21T23:14:50.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f600ed82640 1 --2- 172.21.15.44:0/1734200002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7d8430 0x55a43e7d5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:50.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f600e581640 1 --2- 172.21.15.44:0/1734200002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7cef30 0x55a43e7d5470 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6000005bc0 tx=0x7f60000056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38e793adf38377f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:50.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f5fedffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:14:50.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.582+0000 7f5fedffb640 1 -- 172.21.15.44:0/1734200002 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6000029020 con 0x55a43e7cef30 2024-04-21T23:14:50.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.583+0000 7f5fedffb640 1 -- 172.21.15.44:0/1734200002 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9445+0+0 (secure 0 0 0) 0x7f6000004a20 con 0x55a43e7cef30 2024-04-21T23:14:50.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.583+0000 7f60133cd740 1 -- 172.21.15.44:0/1734200002 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6000005020 con 0x55a43e7cef30 2024-04-21T23:14:50.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.583+0000 7f5fedffb640 1 -- 172.21.15.44:0/1734200002 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6000034050 con 0x55a43e7cef30 2024-04-21T23:14:50.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.583+0000 7f60133cd740 10 client.17034 fetch_fsmap learned FSMap version 27 2024-04-21T23:14:50.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.583+0000 7f60133cd740 1 -- 172.21.15.44:0/1734200002 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a43e4070f0 con 0x55a43e7cef30 2024-04-21T23:14:50.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.583+0000 7f5fedffb640 1 -- 172.21.15.44:0/1734200002 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6000005020 con 0x55a43e7cef30 2024-04-21T23:14:50.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.583+0000 7f5fedffb640 1 -- 172.21.15.44:0/1734200002 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 27) ==== 8330+0+0 (secure 0 0 0) 0x7f6000019450 con 0x55a43e7cef30 2024-04-21T23:14:50.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.583+0000 7f60133cd740 10 client.17034 fetch_fsmap finished waiting for FSMap version 27 2024-04-21T23:14:50.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.583+0000 7f60133cd740 10 client.17034 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:50.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.583+0000 7f60133cd740 20 client.17034 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:50.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.583+0000 7f60133cd740 1 --2- 172.21.15.44:0/1734200002 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a43e7e87d0 0x55a43e7eabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.583+0000 7f60133cd740 4 client.17034 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:14:50.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.583+0000 7f60133cd740 1 -- 172.21.15.44:0/1734200002 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a43e740aa0 con 0x55a43e7e87d0 2024-04-21T23:14:50.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.583+0000 7f600dd80640 1 --2- 172.21.15.44:0/1734200002 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a43e7e87d0 0x55a43e7eabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.584+0000 7f600dd80640 1 --2- 172.21.15.44:0/1734200002 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a43e7e87d0 0x55a43e7eabc0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11e2eac4c9bafff2 server_cookie=33187dff4d8ee211 in_seq=0 out_seq=0 2024-04-21T23:14:50.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.584+0000 7f5fedffb640 10 client.17034 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:50.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.585+0000 7f5fedffb640 1 -- 172.21.15.44:0/1734200002 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a43e740aa0 con 0x55a43e7e87d0 2024-04-21T23:14:50.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.585+0000 7f5fedffb640 10 client.17034 handle_command_reply: tid=0 2024-04-21T23:14:50.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.585+0000 7f60133cd740 1 client.17034 shutdown 2024-04-21T23:14:50.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.585+0000 7f60133cd740 1 -- 172.21.15.44:0/1734200002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7cef30 msgr2=0x55a43e7d5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:50.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.585+0000 7f60133cd740 1 --2- 172.21.15.44:0/1734200002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7cef30 0x55a43e7d5470 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6000005bc0 tx=0x7f60000056a0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.585+0000 7f60133cd740 1 -- 172.21.15.44:0/1734200002 shutdown_connections 2024-04-21T23:14:50.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.585+0000 7f60133cd740 1 --2- 172.21.15.44:0/1734200002 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a43e7e87d0 0x55a43e7eabc0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.585+0000 7f60133cd740 1 --2- 172.21.15.44:0/1734200002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7cf850 0x55a43e7d59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.585+0000 7f60133cd740 1 --2- 172.21.15.44:0/1734200002 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7d8430 0x55a43e7d5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.585+0000 7f60133cd740 1 --2- 172.21.15.44:0/1734200002 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7cef30 0x55a43e7d5470 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.585+0000 7f60133cd740 1 -- 172.21.15.44:0/1734200002 >> 172.21.15.44:0/1734200002 conn(0x55a43e7c9660 msgr2=0x55a43e7cb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:50.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.585+0000 7f5fedffb640 0 client.17034 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:50.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.585+0000 7f60133cd740 1 -- 172.21.15.44:0/1734200002 shutdown_connections 2024-04-21T23:14:50.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.585+0000 7f60133cd740 1 -- 172.21.15.44:0/1734200002 wait complete. 2024-04-21T23:14:50.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.585+0000 7f60133cd740 20 client.17034 trim_cache size 0 max 16384 2024-04-21T23:14:50.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.606+0000 7f60133cd740 1 Processor -- start 2024-04-21T23:14:50.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.606+0000 7f60133cd740 1 -- start start 2024-04-21T23:14:50.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.606+0000 7f60133cd740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7cef30 0x55a43e7d7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.611 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.606+0000 7f60133cd740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7cf850 0x55a43e7e40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.606+0000 7f60133cd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7d8430 0x55a43e7c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.606+0000 7f60133cd740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f60081aee90 con 0x55a43e7cf850 2024-04-21T23:14:50.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.606+0000 7f60133cd740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f60081b1710 con 0x55a43e7cef30 2024-04-21T23:14:50.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.606+0000 7f60133cd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f60081af500 con 0x55a43e7d8430 2024-04-21T23:14:50.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.606+0000 7f600dd80640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7cf850 0x55a43e7e40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.606+0000 7f600dd80640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7cf850 0x55a43e7e40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48364/0 (socket says 172.21.15.44:48364) 2024-04-21T23:14:50.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.606+0000 7f600dd80640 1 -- 172.21.15.44:0/979999175 learned_addr learned my addr 172.21.15.44:0/979999175 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:50.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.606+0000 7f600e581640 1 --2- 172.21.15.44:0/979999175 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7cef30 0x55a43e7d7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.606+0000 7f600dd80640 1 -- 172.21.15.44:0/979999175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7d8430 msgr2=0x55a43e7c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:50.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.606+0000 7f600ed82640 1 --2- 172.21.15.44:0/979999175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7d8430 0x55a43e7c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.606+0000 7f600dd80640 1 --2- 172.21.15.44:0/979999175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7d8430 0x55a43e7c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.606+0000 7f600dd80640 1 -- 172.21.15.44:0/979999175 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7cef30 msgr2=0x55a43e7d7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:50.612 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.606+0000 7f600dd80640 1 --2- 172.21.15.44:0/979999175 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7cef30 0x55a43e7d7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.606+0000 7f600dd80640 1 -- 172.21.15.44:0/979999175 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a43e7d1040 con 0x55a43e7cf850 2024-04-21T23:14:50.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.606+0000 7f600ed82640 1 --2- 172.21.15.44:0/979999175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7d8430 0x55a43e7c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:14:50.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.606+0000 7f600dd80640 1 --2- 172.21.15.44:0/979999175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7cf850 0x55a43e7e40e0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc097a30 tx=0x7f5ffc098120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e135d8882761e875 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:50.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.607+0000 7f5feffff640 1 -- 172.21.15.44:0/979999175 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc0a0100 con 0x55a43e7cf850 2024-04-21T23:14:50.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.607+0000 7f5feffff640 1 -- 172.21.15.44:0/979999175 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ffc0a49c0 con 0x55a43e7cf850 2024-04-21T23:14:50.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.607+0000 7f5feffff640 1 -- 172.21.15.44:0/979999175 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc018b30 con 0x55a43e7cf850 2024-04-21T23:14:50.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.607+0000 7f60133cd740 1 -- 172.21.15.44:0/979999175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7cf850 msgr2=0x55a43e7e40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:50.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.607+0000 7f60133cd740 1 --2- 172.21.15.44:0/979999175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7cf850 0x55a43e7e40e0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc097a30 tx=0x7f5ffc098120 comp rx=0 tx=0).stop 2024-04-21T23:14:50.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.607+0000 7f60133cd740 1 -- 172.21.15.44:0/979999175 shutdown_connections 2024-04-21T23:14:50.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.607+0000 7f60133cd740 1 --2- 172.21.15.44:0/979999175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7d8430 0x55a43e7c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.607+0000 7f60133cd740 1 --2- 172.21.15.44:0/979999175 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7cf850 0x55a43e7e40e0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.607+0000 7f60133cd740 1 --2- 172.21.15.44:0/979999175 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7cef30 0x55a43e7d7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.607+0000 7f60133cd740 1 -- 172.21.15.44:0/979999175 >> 172.21.15.44:0/979999175 conn(0x55a43e7c9660 msgr2=0x55a43e7caab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:50.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.607+0000 7f60133cd740 1 -- 172.21.15.44:0/979999175 shutdown_connections 2024-04-21T23:14:50.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.607+0000 7f60133cd740 1 -- 172.21.15.44:0/979999175 wait complete. 2024-04-21T23:14:50.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.608+0000 7f60133cd740 1 -- start start 2024-04-21T23:14:50.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.608+0000 7f60133cd740 1 Processor -- start 2024-04-21T23:14:50.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.608+0000 7f60133cd740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7cef30 0x55a43e7db5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.608+0000 7f60133cd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7cf850 0x55a43e7dbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.608+0000 7f60133cd740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7d8430 0x55a43e7dc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.608+0000 7f60133cd740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f60081afbd0 con 0x55a43e7d8430 2024-04-21T23:14:50.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.608+0000 7f60133cd740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f60081b1040 con 0x55a43e7cef30 2024-04-21T23:14:50.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.608+0000 7f60133cd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f60081b02a0 con 0x55a43e7cf850 2024-04-21T23:14:50.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.608+0000 7f60133cd740 10 client.0 _ll_register_callbacks cb 0x7f5ffc097ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:50.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.608+0000 7f600ed82640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7d8430 0x55a43e7dc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.608+0000 7f600e581640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7cef30 0x55a43e7db5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.608+0000 7f600ed82640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7d8430 0x55a43e7dc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48366/0 (socket says 172.21.15.44:48366) 2024-04-21T23:14:50.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.608+0000 7f600ed82640 1 -- 172.21.15.44:0/1285730988 learned_addr learned my addr 172.21.15.44:0/1285730988 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:50.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.608+0000 7f600dd80640 1 --2- 172.21.15.44:0/1285730988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7cf850 0x55a43e7dbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.608+0000 7f600ed82640 1 -- 172.21.15.44:0/1285730988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7cf850 msgr2=0x55a43e7dbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:50.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.608+0000 7f600ed82640 1 --2- 172.21.15.44:0/1285730988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7cf850 0x55a43e7dbaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.608+0000 7f600ed82640 1 -- 172.21.15.44:0/1285730988 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7cef30 msgr2=0x55a43e7db5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:50.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.608+0000 7f600ed82640 1 --2- 172.21.15.44:0/1285730988 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7cef30 0x55a43e7db5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.608+0000 7f600ed82640 1 -- 172.21.15.44:0/1285730988 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a43e84cd30 con 0x55a43e7d8430 2024-04-21T23:14:50.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.608+0000 7f600e581640 1 --2- 172.21.15.44:0/1285730988 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7cef30 0x55a43e7db5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:14:50.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.609+0000 7f600ed82640 1 --2- 172.21.15.44:0/1285730988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7d8430 0x55a43e7dc030 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8009cd0 tx=0x7f5ff8005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f7368c105f9697a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:50.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.609+0000 7f5fedffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:14:50.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.609+0000 7f5fedffb640 1 -- 172.21.15.44:0/1285730988 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff8039040 con 0x55a43e7d8430 2024-04-21T23:14:50.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.609+0000 7f5fedffb640 1 -- 172.21.15.44:0/1285730988 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9445+0+0 (secure 0 0 0) 0x7f5ff8004790 con 0x55a43e7d8430 2024-04-21T23:14:50.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.609+0000 7f60133cd740 1 -- 172.21.15.44:0/1285730988 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ffc0a49c0 con 0x55a43e7d8430 2024-04-21T23:14:50.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.609+0000 7f5fedffb640 1 -- 172.21.15.44:0/1285730988 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5ff8032a20 con 0x55a43e7d8430 2024-04-21T23:14:50.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.609+0000 7f60133cd740 10 client.17040 fetch_fsmap learned FSMap version 27 2024-04-21T23:14:50.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.609+0000 7f60133cd740 1 -- 172.21.15.44:0/1285730988 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a43e84df90 con 0x55a43e7d8430 2024-04-21T23:14:50.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.609+0000 7f5fedffb640 1 -- 172.21.15.44:0/1285730988 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ffc0a49c0 con 0x55a43e7d8430 2024-04-21T23:14:50.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.609+0000 7f5fedffb640 1 -- 172.21.15.44:0/1285730988 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 27) ==== 8330+0+0 (secure 0 0 0) 0x7f5ff803f040 con 0x55a43e7d8430 2024-04-21T23:14:50.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.609+0000 7f60133cd740 10 client.17040 fetch_fsmap finished waiting for FSMap version 27 2024-04-21T23:14:50.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.609+0000 7f60133cd740 10 client.17040 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:50.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.609+0000 7f60133cd740 20 client.17040 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:50.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.609+0000 7f60133cd740 1 --2- 172.21.15.44:0/1285730988 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a43e84e740 0x55a43e850b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:50.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.609+0000 7f60133cd740 4 client.17040 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:14:50.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.609+0000 7f60133cd740 1 -- 172.21.15.44:0/1285730988 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a43e73adf0 con 0x55a43e84e740 2024-04-21T23:14:50.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.610+0000 7f600e581640 1 --2- 172.21.15.44:0/1285730988 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a43e84e740 0x55a43e850b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:50.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.610+0000 7f5fedffb640 10 client.17040 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:50.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.610+0000 7f600e581640 1 --2- 172.21.15.44:0/1285730988 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a43e84e740 0x55a43e850b60 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2e33879fb988c8b server_cookie=2d6254ef6fe3cd95 in_seq=0 out_seq=0 2024-04-21T23:14:50.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.610+0000 7f5fedffb640 1 -- 172.21.15.44:0/1285730988 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a43e73adf0 con 0x55a43e84e740 2024-04-21T23:14:50.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.610+0000 7f5fedffb640 10 client.17040 handle_command_reply: tid=0 2024-04-21T23:14:50.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.610+0000 7f60133cd740 1 client.17040 shutdown 2024-04-21T23:14:50.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.610+0000 7f60133cd740 1 -- 172.21.15.44:0/1285730988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7d8430 msgr2=0x55a43e7dc030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:50.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.610+0000 7f60133cd740 1 --2- 172.21.15.44:0/1285730988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7d8430 0x55a43e7dc030 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8009cd0 tx=0x7f5ff8005d80 comp rx=0 tx=0).stop 2024-04-21T23:14:50.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.610+0000 7f60133cd740 1 -- 172.21.15.44:0/1285730988 shutdown_connections 2024-04-21T23:14:50.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.610+0000 7f60133cd740 1 --2- 172.21.15.44:0/1285730988 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a43e84e740 0x55a43e850b60 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.610+0000 7f60133cd740 1 --2- 172.21.15.44:0/1285730988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a43e7cf850 0x55a43e7dbaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.610+0000 7f60133cd740 1 --2- 172.21.15.44:0/1285730988 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a43e7d8430 0x55a43e7dc030 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.610+0000 7f5fedffb640 0 client.17040 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:50.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.610+0000 7f60133cd740 1 --2- 172.21.15.44:0/1285730988 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a43e7cef30 0x55a43e7db5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.610+0000 7f60133cd740 1 -- 172.21.15.44:0/1285730988 >> 172.21.15.44:0/1285730988 conn(0x55a43e7c9660 msgr2=0x55a43e7d5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:50.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.610+0000 7f60133cd740 1 -- 172.21.15.44:0/1285730988 shutdown_connections 2024-04-21T23:14:50.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.611+0000 7f60133cd740 1 -- 172.21.15.44:0/1285730988 wait complete. 2024-04-21T23:14:50.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.611+0000 7f60133cd740 20 client.17040 trim_cache size 0 max 16384 2024-04-21T23:14:50.618 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:14:50.619 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:14:50.619 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-21T23:14:50.619 INFO:teuthology.orchestra.run.smithi044.stdout: "e47e9f51-eb2c-41e1-b5e8-95e495faee64": { 2024-04-21T23:14:50.619 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-21T23:14:50.619 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "e47e9f51-eb2c-41e1-b5e8-95e495faee64", 2024-04-21T23:14:50.619 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-04-21T23:14:50.619 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:14:50.619 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:14:50.619 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:14:50.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.611+0000 7f60107e5640 1 -- 172.21.15.44:0/4133099593 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5ff00a2d00 msgr2=0x7f5ff00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:50.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.611+0000 7f60107e5640 1 --2- 172.21.15.44:0/4133099593 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5ff00a2d00 0x7f5ff00a51c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5ff802c700 tx=0x7f5ff80095c0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.611+0000 7f60107e5640 1 -- 172.21.15.44:0/4133099593 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6008100fc0 msgr2=0x7f60081027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:50.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.611+0000 7f60107e5640 1 --2- 172.21.15.44:0/4133099593 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6008100fc0 0x7f60081027e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00e9f0 tx=0x7f5ffc00eeb0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.611+0000 7f60107e5640 1 -- 172.21.15.44:0/4133099593 shutdown_connections 2024-04-21T23:14:50.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.611+0000 7f60107e5640 1 --2- 172.21.15.44:0/4133099593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60081066a0 0x7f6008102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.611+0000 7f60107e5640 1 --2- 172.21.15.44:0/4133099593 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5ff00a2d00 0x7f5ff00a51c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.611+0000 7f60107e5640 1 --2- 172.21.15.44:0/4133099593 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6008100fc0 0x7f60081027e0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.611+0000 7f60107e5640 1 --2- 172.21.15.44:0/4133099593 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6008100600 0x7f60081022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:50.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.611+0000 7f60107e5640 1 -- 172.21.15.44:0/4133099593 >> 172.21.15.44:0/4133099593 conn(0x7f6008077850 msgr2=0x7f60080ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:50.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.611+0000 7f60107e5640 1 -- 172.21.15.44:0/4133099593 shutdown_connections 2024-04-21T23:14:50.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:14:50.611+0000 7f60107e5640 1 -- 172.21.15.44:0/4133099593 wait complete. 2024-04-21T23:14:50.622 INFO:tasks.cephfs.filesystem:scrub status for tag:e47e9f51-eb2c-41e1-b5e8-95e495faee64 - {'path': '/', 'tag': 'e47e9f51-eb2c-41e1-b5e8-95e495faee64', 'options': 'recursive,force'} 2024-04-21T23:14:50.703 INFO:tasks.workunit.client.0.smithi044.stdout: 1 703579 13.28 MB/sec execute 160 sec latency 25.572 ms 2024-04-21T23:14:51.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:51 smithi055 ceph-mon[31117]: scrub queued for path: / 2024-04-21T23:14:51.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:51 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:14:51.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:51 smithi055 ceph-mon[31117]: scrub summary: active paths [/] 2024-04-21T23:14:51.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:51 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:14:51.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:51 smithi069 ceph-mon[31816]: scrub queued for path: / 2024-04-21T23:14:51.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:51 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:14:51.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:51 smithi069 ceph-mon[31816]: scrub summary: active paths [/] 2024-04-21T23:14:51.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:51 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:14:51.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:51 smithi044 ceph-mon[27934]: scrub queued for path: / 2024-04-21T23:14:51.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:51 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:14:51.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:51 smithi044 ceph-mon[27934]: scrub summary: active paths [/] 2024-04-21T23:14:51.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:51 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:14:51.703 INFO:tasks.workunit.client.0.smithi044.stdout: 1 706338 13.28 MB/sec execute 161 sec latency 53.742 ms 2024-04-21T23:14:52.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:52 smithi069 ceph-mon[31816]: pgmap v186: 129 pgs: 129 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.1 MiB/s wr, 476 op/s 2024-04-21T23:14:52.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:52 smithi069 ceph-mon[31816]: scrub summary: idle 2024-04-21T23:14:52.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:52 smithi055 ceph-mon[31117]: pgmap v186: 129 pgs: 129 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.1 MiB/s wr, 476 op/s 2024-04-21T23:14:52.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:52 smithi055 ceph-mon[31117]: scrub summary: idle 2024-04-21T23:14:52.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:52 smithi044 ceph-mon[27934]: pgmap v186: 129 pgs: 129 active+clean; 68 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.1 MiB/s wr, 476 op/s 2024-04-21T23:14:52.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:52 smithi044 ceph-mon[27934]: scrub summary: idle 2024-04-21T23:14:52.703 INFO:tasks.workunit.client.0.smithi044.stdout: 1 708733 13.27 MB/sec execute 162 sec latency 31.543 ms 2024-04-21T23:14:53.703 INFO:tasks.workunit.client.0.smithi044.stdout: 1 710958 13.27 MB/sec execute 163 sec latency 74.454 ms 2024-04-21T23:14:54.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:54 smithi055 ceph-mon[31117]: pgmap v187: 129 pgs: 129 active+clean; 75 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.0 MiB/s wr, 480 op/s 2024-04-21T23:14:54.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:54 smithi069 ceph-mon[31816]: pgmap v187: 129 pgs: 129 active+clean; 75 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.0 MiB/s wr, 480 op/s 2024-04-21T23:14:54.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:54 smithi044 ceph-mon[27934]: pgmap v187: 129 pgs: 129 active+clean; 75 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.0 MiB/s wr, 480 op/s 2024-04-21T23:14:54.703 INFO:tasks.workunit.client.0.smithi044.stdout: 1 713778 13.30 MB/sec execute 164 sec latency 83.518 ms 2024-04-21T23:14:55.703 INFO:tasks.workunit.client.0.smithi044.stdout: 1 715800 13.26 MB/sec execute 165 sec latency 18.048 ms 2024-04-21T23:14:56.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:56 smithi069 ceph-mon[31816]: pgmap v188: 129 pgs: 129 active+clean; 75 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.0 MiB/s wr, 480 op/s 2024-04-21T23:14:56.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:56 smithi055 ceph-mon[31117]: pgmap v188: 129 pgs: 129 active+clean; 75 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.0 MiB/s wr, 480 op/s 2024-04-21T23:14:56.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:56 smithi044 ceph-mon[27934]: pgmap v188: 129 pgs: 129 active+clean; 75 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.0 MiB/s wr, 480 op/s 2024-04-21T23:14:56.703 INFO:tasks.workunit.client.0.smithi044.stdout: 1 718051 13.26 MB/sec execute 166 sec latency 29.287 ms 2024-04-21T23:14:57.703 INFO:tasks.workunit.client.0.smithi044.stdout: 1 721139 13.30 MB/sec execute 167 sec latency 12.187 ms 2024-04-21T23:14:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.089+0000 7fdcd86b5640 1 -- 172.21.15.44:0/1962432521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcd0100760 msgr2=0x7fdcd010b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.089+0000 7fdcd86b5640 1 --2- 172.21.15.44:0/1962432521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcd0100760 0x7fdcd010b7e0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0009900 tx=0x7fdcc002bcd0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.089+0000 7fdcd86b5640 1 -- 172.21.15.44:0/1962432521 shutdown_connections 2024-04-21T23:14:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.089+0000 7fdcd86b5640 1 --2- 172.21.15.44:0/1962432521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcd010bd20 0x7fdcd010e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.089+0000 7fdcd86b5640 1 --2- 172.21.15.44:0/1962432521 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcd0100760 0x7fdcd010b7e0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.089+0000 7fdcd86b5640 1 --2- 172.21.15.44:0/1962432521 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdcd00ffe40 0x7fdcd0100220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.089+0000 7fdcd86b5640 1 -- 172.21.15.44:0/1962432521 >> 172.21.15.44:0/1962432521 conn(0x7fdcd0075550 msgr2=0x7fdcd0075960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.089+0000 7fdcd86b5640 1 -- 172.21.15.44:0/1962432521 shutdown_connections 2024-04-21T23:14:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.089+0000 7fdcd86b5640 1 -- 172.21.15.44:0/1962432521 wait complete. 2024-04-21T23:14:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.090+0000 7fdcd86b5640 1 Processor -- start 2024-04-21T23:14:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.090+0000 7fdcd86b5640 1 -- start start 2024-04-21T23:14:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.090+0000 7fdcd86b5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcd00ffe40 0x7fdcd019a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.090+0000 7fdcd86b5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcd0100760 0x7fdcd019a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.090+0000 7fdcd86b5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdcd010bd20 0x7fdcd019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.090+0000 7fdcd86b5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd0111af0 con 0x7fdcd0100760 2024-04-21T23:14:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.090+0000 7fdcd86b5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd0111970 con 0x7fdcd010bd20 2024-04-21T23:14:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.090+0000 7fdcd86b5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd0111c70 con 0x7fdcd00ffe40 2024-04-21T23:14:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.090+0000 7fdcd6451640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcd00ffe40 0x7fdcd019a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.090+0000 7fdcd6451640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcd00ffe40 0x7fdcd019a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:60466/0 (socket says 172.21.15.44:60466) 2024-04-21T23:14:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.090+0000 7fdcd6451640 1 -- 172.21.15.44:0/4256605676 learned_addr learned my addr 172.21.15.44:0/4256605676 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.090+0000 7fdcd6c52640 1 --2- 172.21.15.44:0/4256605676 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdcd010bd20 0x7fdcd019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.091+0000 7fdcd5c50640 1 --2- 172.21.15.44:0/4256605676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcd0100760 0x7fdcd019a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.091+0000 7fdcd6451640 1 -- 172.21.15.44:0/4256605676 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdcd010bd20 msgr2=0x7fdcd019ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.091+0000 7fdcd6451640 1 --2- 172.21.15.44:0/4256605676 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdcd010bd20 0x7fdcd019ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.091+0000 7fdcd6451640 1 -- 172.21.15.44:0/4256605676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcd0100760 msgr2=0x7fdcd019a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.091+0000 7fdcd6451640 1 --2- 172.21.15.44:0/4256605676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcd0100760 0x7fdcd019a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.091+0000 7fdcd6451640 1 -- 172.21.15.44:0/4256605676 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcd019f2e0 con 0x7fdcd00ffe40 2024-04-21T23:14:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.091+0000 7fdcd6451640 1 --2- 172.21.15.44:0/4256605676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcd00ffe40 0x7fdcd019a2f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fdccc00e9f0 tx=0x7fdccc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.091+0000 7fdcc77fe640 1 -- 172.21.15.44:0/4256605676 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdccc00cd80 con 0x7fdcd00ffe40 2024-04-21T23:14:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.091+0000 7fdcd86b5640 1 -- 172.21.15.44:0/4256605676 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcd019f5d0 con 0x7fdcd00ffe40 2024-04-21T23:14:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.091+0000 7fdcc77fe640 1 -- 172.21.15.44:0/4256605676 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdccc010910 con 0x7fdcd00ffe40 2024-04-21T23:14:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.091+0000 7fdcc77fe640 1 -- 172.21.15.44:0/4256605676 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdccc018990 con 0x7fdcd00ffe40 2024-04-21T23:14:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.092+0000 7fdcd86b5640 1 -- 172.21.15.44:0/4256605676 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcd01a6e60 con 0x7fdcd00ffe40 2024-04-21T23:14:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.094+0000 7fdcc77fe640 1 -- 172.21.15.44:0/4256605676 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fdccc010ac0 con 0x7fdcd00ffe40 2024-04-21T23:14:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.095+0000 7fdcc77fe640 1 --2- 172.21.15.44:0/4256605676 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fdcbc0a2d70 0x7fdcbc0a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.095+0000 7fdcdb37e740 1 Processor -- start 2024-04-21T23:14:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.095+0000 7fdcd5c50640 1 --2- 172.21.15.44:0/4256605676 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fdcbc0a2d70 0x7fdcbc0a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.095+0000 7fdcc77fe640 1 -- 172.21.15.44:0/4256605676 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9445+0+0 (secure 0 0 0) 0x7fdccc0cfac0 con 0x7fdcd00ffe40 2024-04-21T23:14:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.095+0000 7fdcdb37e740 1 -- start start 2024-04-21T23:14:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.095+0000 7fdcdb37e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 0x558c6f423f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.095+0000 7fdcdb37e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f424440 0x558c6f42ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.096+0000 7fdcdb37e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c6f42d210 0x558c6f42f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.096+0000 7fdcdb37e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558c6f387b70 con 0x558c6f423b20 2024-04-21T23:14:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.096+0000 7fdcdb37e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x558c6f330470 con 0x558c6f42d210 2024-04-21T23:14:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.096+0000 7fdcdb37e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558c6f383b30 con 0x558c6f424440 2024-04-21T23:14:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.096+0000 7fdcd5c50640 1 --2- 172.21.15.44:0/4256605676 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fdcbc0a2d70 0x7fdcbc0a5230 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdcc002efd0 tx=0x7fdcc002cf90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.096+0000 7fdcd6451640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 0x558c6f423f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.096+0000 7fdcd6451640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 0x558c6f423f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48400/0 (socket says 172.21.15.44:48400) 2024-04-21T23:14:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.096+0000 7fdcd6451640 1 -- 172.21.15.44:0/2993191721 learned_addr learned my addr 172.21.15.44:0/2993191721 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.096+0000 7fdcd6c52640 1 --2- 172.21.15.44:0/2993191721 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c6f42d210 0x558c6f42f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.096+0000 7fdcd5c50640 1 --2- 172.21.15.44:0/2993191721 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f424440 0x558c6f42ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.096+0000 7fdcd6451640 1 -- 172.21.15.44:0/2993191721 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f424440 msgr2=0x558c6f42ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.096+0000 7fdcd6451640 1 --2- 172.21.15.44:0/2993191721 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f424440 0x558c6f42ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.096+0000 7fdcd6451640 1 -- 172.21.15.44:0/2993191721 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c6f42d210 msgr2=0x558c6f42f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.096+0000 7fdcd6451640 1 --2- 172.21.15.44:0/2993191721 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c6f42d210 0x558c6f42f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.096+0000 7fdcd6451640 1 -- 172.21.15.44:0/2993191721 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c6f09c210 con 0x558c6f423b20 2024-04-21T23:14:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.096+0000 7fdcd5c50640 1 --2- 172.21.15.44:0/2993191721 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f424440 0x558c6f42ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:14:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.096+0000 7fdcd6451640 1 --2- 172.21.15.44:0/2993191721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 0x558c6f423f00 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0a4320 tx=0x7fdccc09bb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f514b8fa72ec23 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.096+0000 7fdcc4ff9640 1 -- 172.21.15.44:0/2993191721 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdccc099f90 con 0x558c6f423b20 2024-04-21T23:14:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.096+0000 7fdcc4ff9640 1 -- 172.21.15.44:0/2993191721 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdccc0cd090 con 0x558c6f423b20 2024-04-21T23:14:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.096+0000 7fdcc4ff9640 1 -- 172.21.15.44:0/2993191721 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdccc09b340 con 0x558c6f423b20 2024-04-21T23:14:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.097+0000 7fdcdb37e740 1 -- 172.21.15.44:0/2993191721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 msgr2=0x558c6f423f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.097+0000 7fdcdb37e740 1 --2- 172.21.15.44:0/2993191721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 0x558c6f423f00 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0a4320 tx=0x7fdccc09bb90 comp rx=0 tx=0).stop 2024-04-21T23:14:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.097+0000 7fdcdb37e740 1 -- 172.21.15.44:0/2993191721 shutdown_connections 2024-04-21T23:14:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.097+0000 7fdcdb37e740 1 --2- 172.21.15.44:0/2993191721 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f424440 0x558c6f42ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.097+0000 7fdcdb37e740 1 --2- 172.21.15.44:0/2993191721 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c6f42d210 0x558c6f42f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.097+0000 7fdcdb37e740 1 --2- 172.21.15.44:0/2993191721 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 0x558c6f423f00 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.097+0000 7fdcdb37e740 1 -- 172.21.15.44:0/2993191721 >> 172.21.15.44:0/2993191721 conn(0x558c6f41e440 msgr2=0x558c6f41e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.097+0000 7fdcdb37e740 1 -- 172.21.15.44:0/2993191721 shutdown_connections 2024-04-21T23:14:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.097+0000 7fdcdb37e740 1 -- 172.21.15.44:0/2993191721 wait complete. 2024-04-21T23:14:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.098+0000 7fdcdb37e740 1 -- start start 2024-04-21T23:14:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.098+0000 7fdcdb37e740 1 Processor -- start 2024-04-21T23:14:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.098+0000 7fdcdb37e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 0x558c6f42a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.098+0000 7fdcdb37e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c6f424440 0x558c6f42a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.098+0000 7fdcdb37e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f42d210 0x558c6f42ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.098+0000 7fdcdb37e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558c6f42fe40 con 0x558c6f423b20 2024-04-21T23:14:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.098+0000 7fdcdb37e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x558c6f343a10 con 0x558c6f424440 2024-04-21T23:14:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.098+0000 7fdcdb37e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558c6f42ffc0 con 0x558c6f42d210 2024-04-21T23:14:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.098+0000 7fdcd6451640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 0x558c6f42a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.098+0000 7fdcdb37e740 10 client.0 _ll_register_callbacks cb 0x558c6f37eb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.098+0000 7fdcd6451640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 0x558c6f42a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48402/0 (socket says 172.21.15.44:48402) 2024-04-21T23:14:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.098+0000 7fdcd6451640 1 -- 172.21.15.44:0/1382645895 learned_addr learned my addr 172.21.15.44:0/1382645895 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.098+0000 7fdcd5c50640 1 --2- 172.21.15.44:0/1382645895 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c6f424440 0x558c6f42a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.098+0000 7fdcd6c52640 1 --2- 172.21.15.44:0/1382645895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f42d210 0x558c6f42ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.098+0000 7fdcd6451640 1 -- 172.21.15.44:0/1382645895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f42d210 msgr2=0x558c6f42ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.098+0000 7fdcd6451640 1 --2- 172.21.15.44:0/1382645895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f42d210 0x558c6f42ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.098+0000 7fdcd6451640 1 -- 172.21.15.44:0/1382645895 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c6f424440 msgr2=0x558c6f42a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.098+0000 7fdcd6451640 1 --2- 172.21.15.44:0/1382645895 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c6f424440 0x558c6f42a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.098+0000 7fdcd6451640 1 -- 172.21.15.44:0/1382645895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c6f168620 con 0x558c6f423b20 2024-04-21T23:14:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.098+0000 7fdcd6c52640 1 --2- 172.21.15.44:0/1382645895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f42d210 0x558c6f42ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:14:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.099+0000 7fdcd6451640 1 --2- 172.21.15.44:0/1382645895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 0x558c6f42a280 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0cce70 tx=0x7fdccc0a4910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1e4df43aa6b6b63 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.099+0000 7fdcb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:14:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.099+0000 7fdcb9ffb640 1 -- 172.21.15.44:0/1382645895 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdccc0b9070 con 0x558c6f423b20 2024-04-21T23:14:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.099+0000 7fdcb9ffb640 1 -- 172.21.15.44:0/1382645895 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9445+0+0 (secure 0 0 0) 0x7fdccc09b0e0 con 0x558c6f423b20 2024-04-21T23:14:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.099+0000 7fdcdb37e740 1 -- 172.21.15.44:0/1382645895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdccc0cd090 con 0x558c6f423b20 2024-04-21T23:14:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.099+0000 7fdcb9ffb640 1 -- 172.21.15.44:0/1382645895 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdccc0af050 con 0x558c6f423b20 2024-04-21T23:14:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.099+0000 7fdcdb37e740 10 client.17052 fetch_fsmap learned FSMap version 27 2024-04-21T23:14:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.099+0000 7fdcdb37e740 1 -- 172.21.15.44:0/1382645895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558c6f43c0f0 con 0x558c6f423b20 2024-04-21T23:14:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.099+0000 7fdcb9ffb640 1 -- 172.21.15.44:0/1382645895 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdccc0cd090 con 0x558c6f423b20 2024-04-21T23:14:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.099+0000 7fdcb9ffb640 1 -- 172.21.15.44:0/1382645895 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 27) ==== 8330+0+0 (secure 0 0 0) 0x7fdccc0ba420 con 0x558c6f423b20 2024-04-21T23:14:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.099+0000 7fdcdb37e740 10 client.17052 fetch_fsmap finished waiting for FSMap version 27 2024-04-21T23:14:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.099+0000 7fdcdb37e740 10 client.17052 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.099+0000 7fdcdb37e740 20 client.17052 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.099+0000 7fdcdb37e740 1 --2- 172.21.15.44:0/1382645895 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558c6f43d440 0x558c6f43f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.099+0000 7fdcdb37e740 4 client.17052 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:14:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.099+0000 7fdcdb37e740 1 -- 172.21.15.44:0/1382645895 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c6f3950b0 con 0x558c6f43d440 2024-04-21T23:14:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.099+0000 7fdcd5c50640 1 --2- 172.21.15.44:0/1382645895 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558c6f43d440 0x558c6f43f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.100+0000 7fdcd5c50640 1 --2- 172.21.15.44:0/1382645895 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558c6f43d440 0x558c6f43f830 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abe841ed0ae21308 server_cookie=53a1f56d0d6914e4 in_seq=0 out_seq=0 2024-04-21T23:14:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.100+0000 7fdcb9ffb640 10 client.17052 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.101+0000 7fdcb9ffb640 1 -- 172.21.15.44:0/1382645895 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558c6f3950b0 con 0x558c6f43d440 2024-04-21T23:14:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.101+0000 7fdcb9ffb640 10 client.17052 handle_command_reply: tid=0 2024-04-21T23:14:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.101+0000 7fdcdb37e740 1 client.17052 shutdown 2024-04-21T23:14:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.101+0000 7fdcdb37e740 1 -- 172.21.15.44:0/1382645895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 msgr2=0x558c6f42a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.101+0000 7fdcdb37e740 1 --2- 172.21.15.44:0/1382645895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 0x558c6f42a280 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0cce70 tx=0x7fdccc0a4910 comp rx=0 tx=0).stop 2024-04-21T23:14:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.101+0000 7fdcdb37e740 1 -- 172.21.15.44:0/1382645895 shutdown_connections 2024-04-21T23:14:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.101+0000 7fdcdb37e740 1 --2- 172.21.15.44:0/1382645895 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558c6f43d440 0x558c6f43f830 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.101+0000 7fdcdb37e740 1 --2- 172.21.15.44:0/1382645895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f42d210 0x558c6f42ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.101+0000 7fdcdb37e740 1 --2- 172.21.15.44:0/1382645895 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c6f424440 0x558c6f42a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.101+0000 7fdcb9ffb640 0 client.17052 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.101+0000 7fdcdb37e740 1 --2- 172.21.15.44:0/1382645895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 0x558c6f42a280 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.101+0000 7fdcdb37e740 1 -- 172.21.15.44:0/1382645895 >> 172.21.15.44:0/1382645895 conn(0x558c6f41e440 msgr2=0x558c6f420190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.102+0000 7fdcdb37e740 1 -- 172.21.15.44:0/1382645895 shutdown_connections 2024-04-21T23:14:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.102+0000 7fdcdb37e740 1 -- 172.21.15.44:0/1382645895 wait complete. 2024-04-21T23:14:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.102+0000 7fdcdb37e740 20 client.17052 trim_cache size 0 max 16384 2024-04-21T23:14:58.119 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.119+0000 7fdcdb37e740 1 Processor -- start 2024-04-21T23:14:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.119+0000 7fdcdb37e740 1 -- start start 2024-04-21T23:14:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.119+0000 7fdcdb37e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 0x558c6f42c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.119+0000 7fdcdb37e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c6f424440 0x558c6f438d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.119+0000 7fdcdb37e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f42d210 0x558c6f429b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.119+0000 7fdcdb37e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd01aeec0 con 0x558c6f423b20 2024-04-21T23:14:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.119+0000 7fdcdb37e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd01b1740 con 0x558c6f424440 2024-04-21T23:14:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.119+0000 7fdcdb37e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd01af530 con 0x558c6f42d210 2024-04-21T23:14:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.119+0000 7fdcd6451640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 0x558c6f42c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.119+0000 7fdcd6451640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 0x558c6f42c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48414/0 (socket says 172.21.15.44:48414) 2024-04-21T23:14:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.119+0000 7fdcd6451640 1 -- 172.21.15.44:0/7561423 learned_addr learned my addr 172.21.15.44:0/7561423 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.119+0000 7fdcd5c50640 1 --2- 172.21.15.44:0/7561423 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c6f424440 0x558c6f438d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.119+0000 7fdcd6c52640 1 --2- 172.21.15.44:0/7561423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f42d210 0x558c6f429b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.119+0000 7fdcd6451640 1 -- 172.21.15.44:0/7561423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f42d210 msgr2=0x558c6f429b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.119+0000 7fdcd6451640 1 --2- 172.21.15.44:0/7561423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f42d210 0x558c6f429b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.119+0000 7fdcd6451640 1 -- 172.21.15.44:0/7561423 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c6f424440 msgr2=0x558c6f438d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.119+0000 7fdcd6451640 1 --2- 172.21.15.44:0/7561423 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c6f424440 0x558c6f438d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.119+0000 7fdcd6451640 1 -- 172.21.15.44:0/7561423 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c6f424270 con 0x558c6f423b20 2024-04-21T23:14:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.119+0000 7fdcd6c52640 1 --2- 172.21.15.44:0/7561423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f42d210 0x558c6f429b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:14:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.120+0000 7fdcd6451640 1 --2- 172.21.15.44:0/7561423 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 0x558c6f42c9b0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fdccc098660 tx=0x7fdccc09a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=876a4f0a8b2f8efb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.120+0000 7fdcbbfff640 1 -- 172.21.15.44:0/7561423 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdccc0b9070 con 0x558c6f423b20 2024-04-21T23:14:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.120+0000 7fdcbbfff640 1 -- 172.21.15.44:0/7561423 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdccc09ad90 con 0x558c6f423b20 2024-04-21T23:14:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.120+0000 7fdcbbfff640 1 -- 172.21.15.44:0/7561423 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdccc0979e0 con 0x558c6f423b20 2024-04-21T23:14:58.124 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.121+0000 7fdcdb37e740 1 -- 172.21.15.44:0/7561423 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 msgr2=0x558c6f42c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:58.124 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.121+0000 7fdcdb37e740 1 --2- 172.21.15.44:0/7561423 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 0x558c6f42c9b0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fdccc098660 tx=0x7fdccc09a1a0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.124 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.121+0000 7fdcdb37e740 1 -- 172.21.15.44:0/7561423 shutdown_connections 2024-04-21T23:14:58.124 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.121+0000 7fdcdb37e740 1 --2- 172.21.15.44:0/7561423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f42d210 0x558c6f429b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.121+0000 7fdcdb37e740 1 --2- 172.21.15.44:0/7561423 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c6f424440 0x558c6f438d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.121+0000 7fdcdb37e740 1 --2- 172.21.15.44:0/7561423 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f423b20 0x558c6f42c9b0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.121+0000 7fdcdb37e740 1 -- 172.21.15.44:0/7561423 >> 172.21.15.44:0/7561423 conn(0x558c6f41e440 msgr2=0x558c6f420120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:58.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.121+0000 7fdcdb37e740 1 -- 172.21.15.44:0/7561423 shutdown_connections 2024-04-21T23:14:58.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.121+0000 7fdcdb37e740 1 -- 172.21.15.44:0/7561423 wait complete. 2024-04-21T23:14:58.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.121+0000 7fdcdb37e740 1 -- start start 2024-04-21T23:14:58.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.121+0000 7fdcdb37e740 1 Processor -- start 2024-04-21T23:14:58.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.121+0000 7fdcdb37e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f423b20 0x558c6f435270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:58.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.121+0000 7fdcdb37e740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f424440 0x558c6f4305e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:58.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.121+0000 7fdcdb37e740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c6f42d210 0x558c6f430b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:58.125 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.121+0000 7fdcdb37e740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd01afc00 con 0x558c6f424440 2024-04-21T23:14:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.121+0000 7fdcdb37e740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd01b1070 con 0x558c6f42d210 2024-04-21T23:14:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.121+0000 7fdcdb37e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd01b02d0 con 0x558c6f423b20 2024-04-21T23:14:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.121+0000 7fdcdb37e740 10 client.0 _ll_register_callbacks cb 0x7fdcc8009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:14:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.122+0000 7fdcd5c50640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f424440 0x558c6f4305e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.122+0000 7fdcd6451640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f423b20 0x558c6f435270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.122+0000 7fdcd5c50640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f424440 0x558c6f4305e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48418/0 (socket says 172.21.15.44:48418) 2024-04-21T23:14:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.122+0000 7fdcd5c50640 1 -- 172.21.15.44:0/4141499529 learned_addr learned my addr 172.21.15.44:0/4141499529 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:14:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.122+0000 7fdcd6c52640 1 --2- 172.21.15.44:0/4141499529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c6f42d210 0x558c6f430b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.122+0000 7fdcd5c50640 1 -- 172.21.15.44:0/4141499529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f423b20 msgr2=0x558c6f435270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.122+0000 7fdcd5c50640 1 --2- 172.21.15.44:0/4141499529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f423b20 0x558c6f435270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.122+0000 7fdcd5c50640 1 -- 172.21.15.44:0/4141499529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c6f42d210 msgr2=0x558c6f430b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:14:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.122+0000 7fdcd5c50640 1 --2- 172.21.15.44:0/4141499529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c6f42d210 0x558c6f430b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.122+0000 7fdcd5c50640 1 -- 172.21.15.44:0/4141499529 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c6f431060 con 0x558c6f424440 2024-04-21T23:14:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.122+0000 7fdcd5c50640 1 --2- 172.21.15.44:0/4141499529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f424440 0x558c6f4305e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0006010 tx=0x7fdcc002bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e50d580a09ec4dfa server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:14:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.122+0000 7fdcb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:14:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.122+0000 7fdcb9ffb640 1 -- 172.21.15.44:0/4141499529 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0004440 con 0x558c6f424440 2024-04-21T23:14:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.122+0000 7fdcdb37e740 1 -- 172.21.15.44:0/4141499529 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdccc09ad90 con 0x558c6f424440 2024-04-21T23:14:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.122+0000 7fdcb9ffb640 1 -- 172.21.15.44:0/4141499529 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9445+0+0 (secure 0 0 0) 0x7fdcc00563a0 con 0x558c6f424440 2024-04-21T23:14:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.122+0000 7fdcb9ffb640 1 -- 172.21.15.44:0/4141499529 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdcc0068050 con 0x558c6f424440 2024-04-21T23:14:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.122+0000 7fdcdb37e740 10 client.17058 fetch_fsmap learned FSMap version 27 2024-04-21T23:14:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.122+0000 7fdcdb37e740 1 -- 172.21.15.44:0/4141499529 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558c6f4a3500 con 0x558c6f424440 2024-04-21T23:14:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.123+0000 7fdcb9ffb640 1 -- 172.21.15.44:0/4141499529 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdccc09ad90 con 0x558c6f424440 2024-04-21T23:14:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.123+0000 7fdcb9ffb640 1 -- 172.21.15.44:0/4141499529 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 27) ==== 8330+0+0 (secure 0 0 0) 0x7fdcc0004fa0 con 0x558c6f424440 2024-04-21T23:14:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.123+0000 7fdcdb37e740 10 client.17058 fetch_fsmap finished waiting for FSMap version 27 2024-04-21T23:14:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.123+0000 7fdcdb37e740 10 client.17058 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:14:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.123+0000 7fdcdb37e740 20 client.17058 populate_metadata read hostname 'smithi044' 2024-04-21T23:14:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.123+0000 7fdcdb37e740 1 --2- 172.21.15.44:0/4141499529 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558c6f4a3c60 0x558c6f4a6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:14:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.123+0000 7fdcdb37e740 4 client.17058 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:14:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.123+0000 7fdcdb37e740 1 -- 172.21.15.44:0/4141499529 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558c6f38f400 con 0x558c6f4a3c60 2024-04-21T23:14:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.123+0000 7fdcd6451640 1 --2- 172.21.15.44:0/4141499529 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558c6f4a3c60 0x558c6f4a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:14:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.123+0000 7fdcb9ffb640 10 client.17058 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.123+0000 7fdcd6451640 1 --2- 172.21.15.44:0/4141499529 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558c6f4a3c60 0x558c6f4a6080 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=925a30a6f4f9e6f0 server_cookie=46ba19a4acebe632 in_seq=0 out_seq=0 2024-04-21T23:14:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.464+0000 7fdcb9ffb640 1 -- 172.21.15.44:0/4141499529 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558c6f38f400 con 0x558c6f4a3c60 2024-04-21T23:14:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.464+0000 7fdcb9ffb640 10 client.17058 handle_command_reply: tid=0 2024-04-21T23:14:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.464+0000 7fdcdb37e740 1 client.17058 shutdown 2024-04-21T23:14:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.465+0000 7fdcdb37e740 1 -- 172.21.15.44:0/4141499529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f424440 msgr2=0x558c6f4305e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:58.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.465+0000 7fdcdb37e740 1 --2- 172.21.15.44:0/4141499529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f424440 0x558c6f4305e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0006010 tx=0x7fdcc002bcd0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.465+0000 7fdcdb37e740 1 -- 172.21.15.44:0/4141499529 shutdown_connections 2024-04-21T23:14:58.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.465+0000 7fdcdb37e740 1 --2- 172.21.15.44:0/4141499529 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558c6f4a3c60 0x558c6f4a6080 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.465+0000 7fdcdb37e740 1 --2- 172.21.15.44:0/4141499529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558c6f42d210 0x558c6f430b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.465+0000 7fdcb9ffb640 0 client.17058 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:14:58.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.465+0000 7fdcdb37e740 1 --2- 172.21.15.44:0/4141499529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558c6f424440 0x558c6f4305e0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.465+0000 7fdcdb37e740 1 --2- 172.21.15.44:0/4141499529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558c6f423b20 0x558c6f435270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.468 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.465+0000 7fdcdb37e740 1 -- 172.21.15.44:0/4141499529 >> 172.21.15.44:0/4141499529 conn(0x558c6f41e440 msgr2=0x558c6f41cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:58.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.465+0000 7fdcdb37e740 1 -- 172.21.15.44:0/4141499529 shutdown_connections 2024-04-21T23:14:58.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.465+0000 7fdcdb37e740 1 -- 172.21.15.44:0/4141499529 wait complete. 2024-04-21T23:14:58.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.465+0000 7fdcdb37e740 20 client.17058 trim_cache size 0 max 16384 2024-04-21T23:14:58.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:14:58.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:14:58.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:14:58.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:14:58.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.466+0000 7fdcd86b5640 1 -- 172.21.15.44:0/4256605676 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fdcbc0a2d70 msgr2=0x7fdcbc0a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:58.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.466+0000 7fdcd86b5640 1 --2- 172.21.15.44:0/4256605676 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fdcbc0a2d70 0x7fdcbc0a5230 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdcc002efd0 tx=0x7fdcc002cf90 comp rx=0 tx=0).stop 2024-04-21T23:14:58.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.466+0000 7fdcd86b5640 1 -- 172.21.15.44:0/4256605676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcd00ffe40 msgr2=0x7fdcd019a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:14:58.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.466+0000 7fdcd86b5640 1 --2- 172.21.15.44:0/4256605676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcd00ffe40 0x7fdcd019a2f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fdccc00e9f0 tx=0x7fdccc00eeb0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.466+0000 7fdcd86b5640 1 -- 172.21.15.44:0/4256605676 shutdown_connections 2024-04-21T23:14:58.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.466+0000 7fdcd86b5640 1 --2- 172.21.15.44:0/4256605676 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fdcbc0a2d70 0x7fdcbc0a5230 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.466+0000 7fdcd86b5640 1 --2- 172.21.15.44:0/4256605676 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdcd010bd20 0x7fdcd019ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.466+0000 7fdcd86b5640 1 --2- 172.21.15.44:0/4256605676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdcd0100760 0x7fdcd019a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.466+0000 7fdcd86b5640 1 --2- 172.21.15.44:0/4256605676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcd00ffe40 0x7fdcd019a2f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:14:58.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.467+0000 7fdcd86b5640 1 -- 172.21.15.44:0/4256605676 >> 172.21.15.44:0/4256605676 conn(0x7fdcd0075550 msgr2=0x7fdcd00fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:14:58.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.467+0000 7fdcd86b5640 1 -- 172.21.15.44:0/4256605676 shutdown_connections 2024-04-21T23:14:58.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:14:58.467+0000 7fdcd86b5640 1 -- 172.21.15.44:0/4256605676 wait complete. 2024-04-21T23:14:58.684 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:14:58 smithi044 ceph-mon[27934]: pgmap v189: 129 pgs: 129 active+clean; 81 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.5 MiB/s wr, 586 op/s 2024-04-21T23:14:58.703 INFO:tasks.workunit.client.0.smithi044.stdout: 1 723184 13.26 MB/sec execute 168 sec latency 253.599 ms 2024-04-21T23:14:58.713 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:14:58 smithi055 ceph-mon[31117]: pgmap v189: 129 pgs: 129 active+clean; 81 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.5 MiB/s wr, 586 op/s 2024-04-21T23:14:58.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:14:58 smithi069 ceph-mon[31816]: pgmap v189: 129 pgs: 129 active+clean; 81 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.5 MiB/s wr, 586 op/s 2024-04-21T23:14:59.703 INFO:tasks.workunit.client.0.smithi044.stdout: 1 724724 13.25 MB/sec execute 169 sec latency 178.507 ms 2024-04-21T23:15:00.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:00 smithi044 ceph-mon[27934]: pgmap v190: 129 pgs: 129 active+clean; 81 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 270 op/s 2024-04-21T23:15:00.703 INFO:tasks.workunit.client.0.smithi044.stdout: 1 725434 13.18 MB/sec execute 170 sec latency 76.892 ms 2024-04-21T23:15:00.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:00 smithi069 ceph-mon[31816]: pgmap v190: 129 pgs: 129 active+clean; 81 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 270 op/s 2024-04-21T23:15:00.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:00 smithi055 ceph-mon[31117]: pgmap v190: 129 pgs: 129 active+clean; 81 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.3 MiB/s wr, 270 op/s 2024-04-21T23:15:01.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:01 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:15:01.703 INFO:tasks.workunit.client.0.smithi044.stdout: 1 727571 13.17 MB/sec execute 171 sec latency 76.860 ms 2024-04-21T23:15:01.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:01 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:15:01.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:01 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:15:02.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:02 smithi044 ceph-mon[27934]: pgmap v191: 129 pgs: 129 active+clean; 81 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 246 op/s 2024-04-21T23:15:02.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:02 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:02.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:02 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:15:02.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:02 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:15:02.703 INFO:tasks.workunit.client.0.smithi044.stdout: 1 729130 13.14 MB/sec execute 172 sec latency 21.821 ms 2024-04-21T23:15:02.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:02 smithi055 ceph-mon[31117]: pgmap v191: 129 pgs: 129 active+clean; 81 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 246 op/s 2024-04-21T23:15:02.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:02 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:02.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:02 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:15:02.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:02 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:15:02.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:02 smithi069 ceph-mon[31816]: pgmap v191: 129 pgs: 129 active+clean; 81 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 246 op/s 2024-04-21T23:15:02.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:02 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:15:02.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:02 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:15:02.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:02 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:15:03.703 INFO:tasks.workunit.client.0.smithi044.stdout: 1 730342 13.10 MB/sec execute 173 sec latency 48.633 ms 2024-04-21T23:15:03.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:03 smithi044 ceph-mon[27934]: osdmap e99: 12 total, 12 up, 12 in 2024-04-21T23:15:03.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:03 smithi044 ceph-mon[27934]: pgmap v193: 129 pgs: 129 active+clean; 70 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 522 op/s 2024-04-21T23:15:03.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:03 smithi055 ceph-mon[31117]: osdmap e99: 12 total, 12 up, 12 in 2024-04-21T23:15:03.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:03 smithi055 ceph-mon[31117]: pgmap v193: 129 pgs: 129 active+clean; 70 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 522 op/s 2024-04-21T23:15:03.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:03 smithi069 ceph-mon[31816]: osdmap e99: 12 total, 12 up, 12 in 2024-04-21T23:15:03.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:03 smithi069 ceph-mon[31816]: pgmap v193: 129 pgs: 129 active+clean; 70 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 522 op/s 2024-04-21T23:15:04.703 INFO:tasks.workunit.client.0.smithi044.stdout: 1 730944 13.05 MB/sec execute 174 sec latency 80.033 ms 2024-04-21T23:15:05.704 INFO:tasks.workunit.client.0.smithi044.stdout: 1 731855 13.02 MB/sec execute 175 sec latency 38.544 ms 2024-04-21T23:15:06.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:05 smithi044 ceph-mon[27934]: pgmap v194: 129 pgs: 129 active+clean; 70 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 522 op/s 2024-04-21T23:15:06.213 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:05 smithi055 ceph-mon[31117]: pgmap v194: 129 pgs: 129 active+clean; 70 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 522 op/s 2024-04-21T23:15:06.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:05 smithi069 ceph-mon[31816]: pgmap v194: 129 pgs: 129 active+clean; 70 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 522 op/s 2024-04-21T23:15:06.703 INFO:tasks.workunit.client.0.smithi044.stdout: 1 732717 12.95 MB/sec execute 176 sec latency 33.948 ms 2024-04-21T23:15:07.704 INFO:tasks.workunit.client.0.smithi044.stdout: 1 735326 12.98 MB/sec execute 177 sec latency 14.160 ms 2024-04-21T23:15:08.016 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:07 smithi069 ceph-mon[31816]: pgmap v195: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 74 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 477 op/s 2024-04-21T23:15:08.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:07 smithi044 ceph-mon[27934]: pgmap v195: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 74 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 477 op/s 2024-04-21T23:15:08.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:07 smithi055 ceph-mon[31117]: pgmap v195: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 74 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 477 op/s 2024-04-21T23:15:08.704 INFO:tasks.workunit.client.0.smithi044.stdout: 1 737608 12.95 MB/sec execute 178 sec latency 26.814 ms 2024-04-21T23:15:09.704 INFO:tasks.workunit.client.0.smithi044.stdout: 1 739272 12.94 MB/sec execute 179 sec latency 50.455 ms 2024-04-21T23:15:10.015 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:09 smithi055 ceph-mon[31117]: pgmap v196: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 73 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 474 op/s 2024-04-21T23:15:10.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:09 smithi044 ceph-mon[27934]: pgmap v196: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 73 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 474 op/s 2024-04-21T23:15:10.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:09 smithi069 ceph-mon[31816]: pgmap v196: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 73 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 474 op/s 2024-04-21T23:15:10.704 INFO:tasks.workunit.client.0.smithi044.stdout: 1 742288 12.97 MB/sec execute 180 sec latency 23.031 ms 2024-04-21T23:15:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.581+0000 7f124300e640 1 -- 172.21.15.44:0/2842479284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f123c100fe0 msgr2=0x7f123c10d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.581+0000 7f124300e640 1 --2- 172.21.15.44:0/2842479284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f123c100fe0 0x7f123c10d4f0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f1234009cd0 tx=0x7f123402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.581+0000 7f124300e640 1 -- 172.21.15.44:0/2842479284 shutdown_connections 2024-04-21T23:15:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.581+0000 7f124300e640 1 --2- 172.21.15.44:0/2842479284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f123c100620 0x7f123c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.581+0000 7f124300e640 1 --2- 172.21.15.44:0/2842479284 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f123c100fe0 0x7f123c10d4f0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.581+0000 7f124300e640 1 --2- 172.21.15.44:0/2842479284 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f123c106630 0x7f123c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.581+0000 7f124300e640 1 -- 172.21.15.44:0/2842479284 >> 172.21.15.44:0/2842479284 conn(0x7f123c077760 msgr2=0x7f123c077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.581+0000 7f124300e640 1 -- 172.21.15.44:0/2842479284 shutdown_connections 2024-04-21T23:15:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.581+0000 7f124300e640 1 -- 172.21.15.44:0/2842479284 wait complete. 2024-04-21T23:15:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.581+0000 7f124300e640 1 Processor -- start 2024-04-21T23:15:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.581+0000 7f124300e640 1 -- start start 2024-04-21T23:15:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f124300e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f123c100620 0x7f123c198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f124300e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f123c100fe0 0x7f123c199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f124300e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f123c106630 0x7f123c1a3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f124300e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f123c110a40 con 0x7f123c100fe0 2024-04-21T23:15:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f124300e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f123c1108c0 con 0x7f123c106630 2024-04-21T23:15:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f124300e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f123c110bc0 con 0x7f123c100620 2024-04-21T23:15:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f123bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f123c100fe0 0x7f123c199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f123bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f123c100fe0 0x7f123c199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54396/0 (socket says 172.21.15.44:54396) 2024-04-21T23:15:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f123bfff640 1 -- 172.21.15.44:0/1424799858 learned_addr learned my addr 172.21.15.44:0/1424799858 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f1240daa640 1 --2- 172.21.15.44:0/1424799858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f123c100620 0x7f123c198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f12415ab640 1 --2- 172.21.15.44:0/1424799858 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f123c106630 0x7f123c1a3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f123bfff640 1 -- 172.21.15.44:0/1424799858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f123c100620 msgr2=0x7f123c198f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f123bfff640 1 --2- 172.21.15.44:0/1424799858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f123c100620 0x7f123c198f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f123bfff640 1 -- 172.21.15.44:0/1424799858 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f123c106630 msgr2=0x7f123c1a3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f123bfff640 1 --2- 172.21.15.44:0/1424799858 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f123c106630 0x7f123c1a3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f123bfff640 1 -- 172.21.15.44:0/1424799858 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f123c1a44d0 con 0x7f123c100fe0 2024-04-21T23:15:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f12415ab640 1 --2- 172.21.15.44:0/1424799858 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f123c106630 0x7f123c1a3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f123bfff640 1 --2- 172.21.15.44:0/1424799858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f123c100fe0 0x7f123c199450 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f122800e9f0 tx=0x7f122800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f1239ffb640 1 -- 172.21.15.44:0/1424799858 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122800cd80 con 0x7f123c100fe0 2024-04-21T23:15:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f1239ffb640 1 -- 172.21.15.44:0/1424799858 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1228005500 con 0x7f123c100fe0 2024-04-21T23:15:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f124300e640 1 -- 172.21.15.44:0/1424799858 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f123c1a46a0 con 0x7f123c100fe0 2024-04-21T23:15:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.582+0000 7f1239ffb640 1 -- 172.21.15.44:0/1424799858 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1228010690 con 0x7f123c100fe0 2024-04-21T23:15:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.583+0000 7f124300e640 1 -- 172.21.15.44:0/1424799858 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f123c1a4b70 con 0x7f123c100fe0 2024-04-21T23:15:11.591 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.585+0000 7f1239ffb640 1 -- 172.21.15.44:0/1424799858 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f12280046c0 con 0x7f123c100fe0 2024-04-21T23:15:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.585+0000 7f1245bf9740 1 Processor -- start 2024-04-21T23:15:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.585+0000 7f1245bf9740 1 -- start start 2024-04-21T23:15:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.585+0000 7f1239ffb640 1 --2- 172.21.15.44:0/1424799858 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f122c0a2cf0 0x7f122c0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f12415ab640 1 --2- 172.21.15.44:0/1424799858 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f122c0a2cf0 0x7f122c0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f1239ffb640 1 -- 172.21.15.44:0/1424799858 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7f12280cfd20 con 0x7f123c100fe0 2024-04-21T23:15:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f1245bf9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b517b20 0x55584b517f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f1245bf9740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b518440 0x55584b520cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f1245bf9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b521210 0x55584b523600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f1245bf9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55584b47bb70 con 0x55584b521210 2024-04-21T23:15:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f1245bf9740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55584b424470 con 0x55584b518440 2024-04-21T23:15:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f1245bf9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55584b477b30 con 0x55584b517b20 2024-04-21T23:15:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f12415ab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b521210 0x55584b523600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f12415ab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b521210 0x55584b523600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54398/0 (socket says 172.21.15.44:54398) 2024-04-21T23:15:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f12415ab640 1 -- 172.21.15.44:0/4292421343 learned_addr learned my addr 172.21.15.44:0/4292421343 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f123bfff640 1 --2- 172.21.15.44:0/4292421343 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b518440 0x55584b520cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f1240daa640 1 --2- 172.21.15.44:0/4292421343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b517b20 0x55584b517f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f12415ab640 1 -- 172.21.15.44:0/4292421343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b517b20 msgr2=0x55584b517f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f12415ab640 1 --2- 172.21.15.44:0/4292421343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b517b20 0x55584b517f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f12415ab640 1 -- 172.21.15.44:0/4292421343 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b518440 msgr2=0x55584b520cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f12415ab640 1 --2- 172.21.15.44:0/4292421343 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b518440 0x55584b520cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f12415ab640 1 -- 172.21.15.44:0/4292421343 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55584b190210 con 0x55584b521210 2024-04-21T23:15:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f1240daa640 1 --2- 172.21.15.44:0/4292421343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b517b20 0x55584b517f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:15:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f123bfff640 1 --2- 172.21.15.44:0/4292421343 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b518440 0x55584b520cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f12415ab640 1 --2- 172.21.15.44:0/1424799858 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f122c0a2cf0 0x7f122c0a51b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1234009a60 tx=0x7f123402cec0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.586+0000 7f12415ab640 1 --2- 172.21.15.44:0/4292421343 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b521210 0x55584b523600 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f123402c700 tx=0x7f1234047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=485842ba13abd5d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.587+0000 7f1226ffd640 1 -- 172.21.15.44:0/4292421343 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1234039040 con 0x55584b521210 2024-04-21T23:15:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.587+0000 7f1226ffd640 1 -- 172.21.15.44:0/4292421343 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1234034cd0 con 0x55584b521210 2024-04-21T23:15:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.587+0000 7f1226ffd640 1 -- 172.21.15.44:0/4292421343 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1234004330 con 0x55584b521210 2024-04-21T23:15:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.587+0000 7f1245bf9740 1 -- 172.21.15.44:0/4292421343 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b521210 msgr2=0x55584b523600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.587+0000 7f1245bf9740 1 --2- 172.21.15.44:0/4292421343 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b521210 0x55584b523600 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f123402c700 tx=0x7f1234047000 comp rx=0 tx=0).stop 2024-04-21T23:15:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.587+0000 7f1245bf9740 1 -- 172.21.15.44:0/4292421343 shutdown_connections 2024-04-21T23:15:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.587+0000 7f1245bf9740 1 --2- 172.21.15.44:0/4292421343 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b521210 0x55584b523600 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.587+0000 7f1245bf9740 1 --2- 172.21.15.44:0/4292421343 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b518440 0x55584b520cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.587+0000 7f1245bf9740 1 --2- 172.21.15.44:0/4292421343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b517b20 0x55584b517f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.587+0000 7f1245bf9740 1 -- 172.21.15.44:0/4292421343 >> 172.21.15.44:0/4292421343 conn(0x55584b512440 msgr2=0x55584b512820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.587+0000 7f1245bf9740 1 -- 172.21.15.44:0/4292421343 shutdown_connections 2024-04-21T23:15:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.587+0000 7f1245bf9740 1 -- 172.21.15.44:0/4292421343 wait complete. 2024-04-21T23:15:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.588+0000 7f1245bf9740 1 -- start start 2024-04-21T23:15:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.588+0000 7f1245bf9740 1 Processor -- start 2024-04-21T23:15:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.588+0000 7f1245bf9740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b517b20 0x55584b51e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.588+0000 7f1245bf9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b518440 0x55584b51e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.588+0000 7f1245bf9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b521210 0x55584b51ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.588+0000 7f1245bf9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55584b523de0 con 0x55584b518440 2024-04-21T23:15:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.588+0000 7f1245bf9740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55584b437a10 con 0x55584b517b20 2024-04-21T23:15:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.588+0000 7f1245bf9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55584b523f60 con 0x55584b521210 2024-04-21T23:15:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.588+0000 7f1245bf9740 10 client.0 _ll_register_callbacks cb 0x55584b472b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.588+0000 7f123bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b518440 0x55584b51e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.588+0000 7f1240daa640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b517b20 0x55584b51e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.588+0000 7f123bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b518440 0x55584b51e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54408/0 (socket says 172.21.15.44:54408) 2024-04-21T23:15:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.588+0000 7f123bfff640 1 -- 172.21.15.44:0/2125520450 learned_addr learned my addr 172.21.15.44:0/2125520450 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.588+0000 7f12415ab640 1 --2- 172.21.15.44:0/2125520450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b521210 0x55584b51ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.589+0000 7f12415ab640 1 -- 172.21.15.44:0/2125520450 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b517b20 msgr2=0x55584b51e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.589+0000 7f12415ab640 1 --2- 172.21.15.44:0/2125520450 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b517b20 0x55584b51e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.589+0000 7f12415ab640 1 -- 172.21.15.44:0/2125520450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b518440 msgr2=0x55584b51e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.589+0000 7f12415ab640 1 --2- 172.21.15.44:0/2125520450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b518440 0x55584b51e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.589+0000 7f12415ab640 1 -- 172.21.15.44:0/2125520450 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55584b25c620 con 0x55584b521210 2024-04-21T23:15:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.589+0000 7f1240daa640 1 --2- 172.21.15.44:0/2125520450 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b517b20 0x55584b51e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.589+0000 7f12415ab640 1 --2- 172.21.15.44:0/2125520450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b521210 0x55584b51ed30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1234045040 tx=0x7f123402cca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6a35d4ea3fb90d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.589+0000 7f121bfff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:15:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.589+0000 7f121bfff640 1 -- 172.21.15.44:0/2125520450 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1234039040 con 0x55584b521210 2024-04-21T23:15:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.589+0000 7f1245bf9740 1 -- 172.21.15.44:0/2125520450 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1234034cd0 con 0x55584b521210 2024-04-21T23:15:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.589+0000 7f121bfff640 1 -- 172.21.15.44:0/2125520450 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7f12340040b0 con 0x55584b521210 2024-04-21T23:15:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.589+0000 7f121bfff640 1 -- 172.21.15.44:0/2125520450 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1234003d80 con 0x55584b521210 2024-04-21T23:15:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.589+0000 7f1245bf9740 10 client.26626 fetch_fsmap learned FSMap version 27 2024-04-21T23:15:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.589+0000 7f1245bf9740 1 -- 172.21.15.44:0/2125520450 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55584b523de0 con 0x55584b521210 2024-04-21T23:15:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.589+0000 7f121bfff640 1 -- 172.21.15.44:0/2125520450 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1234034cd0 con 0x55584b521210 2024-04-21T23:15:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.590+0000 7f121bfff640 1 -- 172.21.15.44:0/2125520450 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 27) ==== 8330+0+0 (secure 0 0 0) 0x7f123404d030 con 0x55584b521210 2024-04-21T23:15:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.590+0000 7f1245bf9740 10 client.26626 fetch_fsmap finished waiting for FSMap version 27 2024-04-21T23:15:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.590+0000 7f1245bf9740 10 client.26626 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.590+0000 7f1245bf9740 20 client.26626 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.590+0000 7f1245bf9740 1 --2- 172.21.15.44:0/2125520450 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55584b531220 0x55584b533610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.590+0000 7f1245bf9740 4 client.26626 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:15:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.590+0000 7f1245bf9740 1 -- 172.21.15.44:0/2125520450 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55584b4890b0 con 0x55584b531220 2024-04-21T23:15:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.590+0000 7f1240daa640 1 --2- 172.21.15.44:0/2125520450 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55584b531220 0x55584b533610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.590+0000 7f1240daa640 1 --2- 172.21.15.44:0/2125520450 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55584b531220 0x55584b533610 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94489034fec731f7 server_cookie=b8722cecbe010de5 in_seq=0 out_seq=0 2024-04-21T23:15:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.590+0000 7f121bfff640 10 client.26626 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.592+0000 7f121bfff640 1 -- 172.21.15.44:0/2125520450 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55584b4890b0 con 0x55584b531220 2024-04-21T23:15:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.592+0000 7f121bfff640 10 client.26626 handle_command_reply: tid=0 2024-04-21T23:15:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.593+0000 7f1245bf9740 1 client.26626 shutdown 2024-04-21T23:15:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.593+0000 7f1245bf9740 1 -- 172.21.15.44:0/2125520450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b521210 msgr2=0x55584b51ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.593+0000 7f1245bf9740 1 --2- 172.21.15.44:0/2125520450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b521210 0x55584b51ed30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1234045040 tx=0x7f123402cca0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.593+0000 7f1245bf9740 1 -- 172.21.15.44:0/2125520450 shutdown_connections 2024-04-21T23:15:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.593+0000 7f1245bf9740 1 --2- 172.21.15.44:0/2125520450 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55584b531220 0x55584b533610 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.593+0000 7f1245bf9740 1 --2- 172.21.15.44:0/2125520450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b521210 0x55584b51ed30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.593+0000 7f1245bf9740 1 --2- 172.21.15.44:0/2125520450 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b518440 0x55584b51e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.593+0000 7f121bfff640 0 client.26626 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.593+0000 7f1245bf9740 1 --2- 172.21.15.44:0/2125520450 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b517b20 0x55584b51e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.593+0000 7f1245bf9740 1 -- 172.21.15.44:0/2125520450 >> 172.21.15.44:0/2125520450 conn(0x55584b512440 msgr2=0x55584b514190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.593+0000 7f1245bf9740 1 -- 172.21.15.44:0/2125520450 shutdown_connections 2024-04-21T23:15:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.593+0000 7f1245bf9740 1 -- 172.21.15.44:0/2125520450 wait complete. 2024-04-21T23:15:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.593+0000 7f1245bf9740 20 client.26626 trim_cache size 0 max 16384 2024-04-21T23:15:11.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.609+0000 7f1245bf9740 1 Processor -- start 2024-04-21T23:15:11.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.609+0000 7f1245bf9740 1 -- start start 2024-04-21T23:15:11.613 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.609+0000 7f1245bf9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b517b20 0x55584b5209e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.609+0000 7f1245bf9740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b518440 0x55584b52cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.609+0000 7f1245bf9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b521210 0x55584b51eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.609+0000 7f1245bf9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f123c1aefa0 con 0x55584b517b20 2024-04-21T23:15:11.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.609+0000 7f1245bf9740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f123c1b1800 con 0x55584b518440 2024-04-21T23:15:11.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.609+0000 7f1245bf9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f123c1af5f0 con 0x55584b521210 2024-04-21T23:15:11.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.609+0000 7f1240daa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b517b20 0x55584b5209e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.609+0000 7f1240daa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b517b20 0x55584b5209e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54424/0 (socket says 172.21.15.44:54424) 2024-04-21T23:15:11.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.609+0000 7f1240daa640 1 -- 172.21.15.44:0/4172928744 learned_addr learned my addr 172.21.15.44:0/4172928744 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:11.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.609+0000 7f12415ab640 1 --2- 172.21.15.44:0/4172928744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b521210 0x55584b51eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.609+0000 7f123bfff640 1 --2- 172.21.15.44:0/4172928744 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b518440 0x55584b52cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.609+0000 7f1240daa640 1 -- 172.21.15.44:0/4172928744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b521210 msgr2=0x55584b51eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:11.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.609+0000 7f1240daa640 1 --2- 172.21.15.44:0/4172928744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b521210 0x55584b51eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.609+0000 7f1240daa640 1 -- 172.21.15.44:0/4172928744 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b518440 msgr2=0x55584b52cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:11.614 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.609+0000 7f1240daa640 1 --2- 172.21.15.44:0/4172928744 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b518440 0x55584b52cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.609+0000 7f1240daa640 1 -- 172.21.15.44:0/4172928744 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55584b512dc0 con 0x55584b517b20 2024-04-21T23:15:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.610+0000 7f1240daa640 1 --2- 172.21.15.44:0/4172928744 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b517b20 0x55584b5209e0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x55584b533c10 tx=0x7f1230005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f45aada909183fa server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.610+0000 7f12267fc640 1 -- 172.21.15.44:0/4172928744 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1230004250 con 0x55584b517b20 2024-04-21T23:15:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.610+0000 7f12267fc640 1 -- 172.21.15.44:0/4172928744 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12300043f0 con 0x55584b517b20 2024-04-21T23:15:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.610+0000 7f12267fc640 1 -- 172.21.15.44:0/4172928744 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12300223e0 con 0x55584b517b20 2024-04-21T23:15:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.610+0000 7f1245bf9740 1 -- 172.21.15.44:0/4172928744 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b517b20 msgr2=0x55584b5209e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.610+0000 7f1245bf9740 1 --2- 172.21.15.44:0/4172928744 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b517b20 0x55584b5209e0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x55584b533c10 tx=0x7f1230005f90 comp rx=0 tx=0).stop 2024-04-21T23:15:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.611+0000 7f1245bf9740 1 -- 172.21.15.44:0/4172928744 shutdown_connections 2024-04-21T23:15:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.611+0000 7f1245bf9740 1 --2- 172.21.15.44:0/4172928744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b521210 0x55584b51eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.611+0000 7f1245bf9740 1 --2- 172.21.15.44:0/4172928744 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b518440 0x55584b52cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.611+0000 7f1245bf9740 1 --2- 172.21.15.44:0/4172928744 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b517b20 0x55584b5209e0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.611+0000 7f1245bf9740 1 -- 172.21.15.44:0/4172928744 >> 172.21.15.44:0/4172928744 conn(0x55584b512440 msgr2=0x55584b5140b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.611+0000 7f1245bf9740 1 -- 172.21.15.44:0/4172928744 shutdown_connections 2024-04-21T23:15:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.611+0000 7f1245bf9740 1 -- 172.21.15.44:0/4172928744 wait complete. 2024-04-21T23:15:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.611+0000 7f1245bf9740 1 -- start start 2024-04-21T23:15:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.611+0000 7f1245bf9740 1 Processor -- start 2024-04-21T23:15:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.611+0000 7f1245bf9740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b517b20 0x55584b529420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.611+0000 7f1245bf9740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b518440 0x55584b524790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.611+0000 7f1245bf9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b521210 0x55584b524cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.611+0000 7f1245bf9740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f123c1afcc0 con 0x55584b518440 2024-04-21T23:15:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.611+0000 7f1245bf9740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f123c1b1130 con 0x55584b517b20 2024-04-21T23:15:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.611+0000 7f1245bf9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f123c1b0390 con 0x55584b521210 2024-04-21T23:15:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.612+0000 7f1245bf9740 10 client.0 _ll_register_callbacks cb 0x7f12280056b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.612+0000 7f123bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b518440 0x55584b524790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.612+0000 7f1240daa640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b517b20 0x55584b529420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.612+0000 7f123bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b518440 0x55584b524790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54438/0 (socket says 172.21.15.44:54438) 2024-04-21T23:15:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.612+0000 7f123bfff640 1 -- 172.21.15.44:0/2425244455 learned_addr learned my addr 172.21.15.44:0/2425244455 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.612+0000 7f1240daa640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b517b20 0x55584b529420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:57728/0 (socket says 172.21.15.44:57728) 2024-04-21T23:15:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.612+0000 7f12415ab640 1 --2- 172.21.15.44:0/2425244455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b521210 0x55584b524cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.612+0000 7f123bfff640 1 -- 172.21.15.44:0/2425244455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b521210 msgr2=0x55584b524cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.612+0000 7f123bfff640 1 --2- 172.21.15.44:0/2425244455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b521210 0x55584b524cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.612+0000 7f123bfff640 1 -- 172.21.15.44:0/2425244455 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b517b20 msgr2=0x55584b529420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.612+0000 7f123bfff640 1 --2- 172.21.15.44:0/2425244455 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b517b20 0x55584b529420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.612+0000 7f123bfff640 1 -- 172.21.15.44:0/2425244455 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55584b525370 con 0x55584b518440 2024-04-21T23:15:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.612+0000 7f1240daa640 1 --2- 172.21.15.44:0/2425244455 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b517b20 0x55584b529420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.612+0000 7f123bfff640 1 --2- 172.21.15.44:0/2425244455 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b518440 0x55584b524790 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f12280057a0 tx=0x7f122809c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff90a798e5c9970 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.612+0000 7f121bfff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:15:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.612+0000 7f121bfff640 1 -- 172.21.15.44:0/2425244455 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12280a4770 con 0x55584b518440 2024-04-21T23:15:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.612+0000 7f1245bf9740 1 -- 172.21.15.44:0/2425244455 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12300043f0 con 0x55584b518440 2024-04-21T23:15:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.612+0000 7f121bfff640 1 -- 172.21.15.44:0/2425244455 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7f122809b1e0 con 0x55584b518440 2024-04-21T23:15:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.612+0000 7f121bfff640 1 -- 172.21.15.44:0/2425244455 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f12280a4a00 con 0x55584b518440 2024-04-21T23:15:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.612+0000 7f1245bf9740 10 client.17076 fetch_fsmap learned FSMap version 27 2024-04-21T23:15:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.613+0000 7f1245bf9740 1 -- 172.21.15.44:0/2425244455 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55584b597450 con 0x55584b518440 2024-04-21T23:15:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.613+0000 7f121bfff640 1 -- 172.21.15.44:0/2425244455 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12300043f0 con 0x55584b518440 2024-04-21T23:15:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.613+0000 7f121bfff640 1 -- 172.21.15.44:0/2425244455 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 27) ==== 8330+0+0 (secure 0 0 0) 0x7f12280ba020 con 0x55584b518440 2024-04-21T23:15:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.613+0000 7f1245bf9740 10 client.17076 fetch_fsmap finished waiting for FSMap version 27 2024-04-21T23:15:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.613+0000 7f1245bf9740 10 client.17076 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.613+0000 7f1245bf9740 20 client.17076 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.613+0000 7f1245bf9740 1 --2- 172.21.15.44:0/2425244455 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55584b597bd0 0x55584b599ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:11.618 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.613+0000 7f1245bf9740 4 client.17076 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:15:11.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.613+0000 7f1245bf9740 1 -- 172.21.15.44:0/2425244455 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55584b483400 con 0x55584b597bd0 2024-04-21T23:15:11.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.613+0000 7f1240daa640 1 --2- 172.21.15.44:0/2425244455 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55584b597bd0 0x55584b599ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:11.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.613+0000 7f1240daa640 1 --2- 172.21.15.44:0/2425244455 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55584b597bd0 0x55584b599ff0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c665ae9761c125e server_cookie=488065642900cbdf in_seq=0 out_seq=0 2024-04-21T23:15:11.619 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.613+0000 7f121bfff640 10 client.17076 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:11.704 INFO:tasks.workunit.client.0.smithi044.stdout: 1 744498 12.94 MB/sec execute 181 sec latency 37.523 ms 2024-04-21T23:15:11.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.854+0000 7f121bfff640 1 -- 172.21.15.44:0/2425244455 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55584b483400 con 0x55584b597bd0 2024-04-21T23:15:11.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.854+0000 7f121bfff640 10 client.17076 handle_command_reply: tid=0 2024-04-21T23:15:11.855 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.854+0000 7f1245bf9740 1 client.17076 shutdown 2024-04-21T23:15:11.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.855+0000 7f1245bf9740 1 -- 172.21.15.44:0/2425244455 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b518440 msgr2=0x55584b524790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:11.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.855+0000 7f1245bf9740 1 --2- 172.21.15.44:0/2425244455 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b518440 0x55584b524790 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f12280057a0 tx=0x7f122809c550 comp rx=0 tx=0).stop 2024-04-21T23:15:11.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.855+0000 7f1245bf9740 1 -- 172.21.15.44:0/2425244455 shutdown_connections 2024-04-21T23:15:11.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.855+0000 7f1245bf9740 1 --2- 172.21.15.44:0/2425244455 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55584b597bd0 0x55584b599ff0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.855+0000 7f1245bf9740 1 --2- 172.21.15.44:0/2425244455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55584b521210 0x55584b524cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.855+0000 7f121bfff640 0 client.17076 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:11.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.855+0000 7f1245bf9740 1 --2- 172.21.15.44:0/2425244455 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55584b518440 0x55584b524790 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.855+0000 7f1245bf9740 1 --2- 172.21.15.44:0/2425244455 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55584b517b20 0x55584b529420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.855+0000 7f1245bf9740 1 -- 172.21.15.44:0/2425244455 >> 172.21.15.44:0/2425244455 conn(0x55584b512440 msgr2=0x55584b510c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:11.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.855+0000 7f1245bf9740 1 -- 172.21.15.44:0/2425244455 shutdown_connections 2024-04-21T23:15:11.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.855+0000 7f1245bf9740 1 -- 172.21.15.44:0/2425244455 wait complete. 2024-04-21T23:15:11.856 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.855+0000 7f1245bf9740 20 client.17076 trim_cache size 0 max 16384 2024-04-21T23:15:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:15:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:15:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:15:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:15:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.856+0000 7f124300e640 1 -- 172.21.15.44:0/1424799858 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f122c0a2cf0 msgr2=0x7f122c0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.856+0000 7f124300e640 1 --2- 172.21.15.44:0/1424799858 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f122c0a2cf0 0x7f122c0a51b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1234009a60 tx=0x7f123402cec0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.856+0000 7f124300e640 1 -- 172.21.15.44:0/1424799858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f123c100fe0 msgr2=0x7f123c199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.856+0000 7f124300e640 1 --2- 172.21.15.44:0/1424799858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f123c100fe0 0x7f123c199450 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f122800e9f0 tx=0x7f122800edf0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.856+0000 7f124300e640 1 -- 172.21.15.44:0/1424799858 shutdown_connections 2024-04-21T23:15:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.856+0000 7f124300e640 1 --2- 172.21.15.44:0/1424799858 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f122c0a2cf0 0x7f122c0a51b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.856+0000 7f124300e640 1 --2- 172.21.15.44:0/1424799858 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f123c106630 0x7f123c1a3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.856+0000 7f124300e640 1 --2- 172.21.15.44:0/1424799858 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f123c100fe0 0x7f123c199450 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.856+0000 7f124300e640 1 --2- 172.21.15.44:0/1424799858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f123c100620 0x7f123c198f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.856+0000 7f124300e640 1 -- 172.21.15.44:0/1424799858 >> 172.21.15.44:0/1424799858 conn(0x7f123c077760 msgr2=0x7f123c1032f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.856+0000 7f124300e640 1 -- 172.21.15.44:0/1424799858 shutdown_connections 2024-04-21T23:15:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:11.856+0000 7f124300e640 1 -- 172.21.15.44:0/1424799858 wait complete. 2024-04-21T23:15:12.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:11 smithi044 ceph-mon[27934]: pgmap v197: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 73 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 474 op/s 2024-04-21T23:15:12.213 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:11 smithi055 ceph-mon[31117]: pgmap v197: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 73 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 474 op/s 2024-04-21T23:15:12.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:11 smithi069 ceph-mon[31816]: pgmap v197: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 73 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 474 op/s 2024-04-21T23:15:12.704 INFO:tasks.workunit.client.0.smithi044.stdout: 1 746296 12.94 MB/sec execute 182 sec latency 178.480 ms 2024-04-21T23:15:12.714 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:15:12 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: ignoring --setuser ceph since I am not root 2024-04-21T23:15:12.714 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:15:12 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: ignoring --setgroup ceph since I am not root 2024-04-21T23:15:12.714 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:15:12 smithi069 ceph-mds[63204]: -- 172.21.15.69:0/834295275 <== mon.1 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5607ea2e5a40 con 0x5607ea2b8800 2024-04-21T23:15:12.714 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:15:12 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: starting mds.i at 2024-04-21T23:15:13.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:12 smithi044 ceph-mon[27934]: daemon mds.i restarted 2024-04-21T23:15:13.213 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:12 smithi055 ceph-mon[31117]: daemon mds.i restarted 2024-04-21T23:15:13.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:12 smithi069 ceph-mon[31816]: daemon mds.i restarted 2024-04-21T23:15:13.704 INFO:tasks.workunit.client.0.smithi044.stdout: 1 749294 12.96 MB/sec execute 183 sec latency 28.938 ms 2024-04-21T23:15:14.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:13 smithi044 ceph-mon[27934]: pgmap v198: 129 pgs: 129 active+clean; 80 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.5 MiB/s wr, 267 op/s 2024-04-21T23:15:14.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:13 smithi044 ceph-mon[27934]: osdmap e100: 12 total, 12 up, 12 in 2024-04-21T23:15:14.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:13 smithi044 ceph-mon[27934]: mds.? [v2:172.21.15.69:6838/1729482148,v1:172.21.15.69:6839/1729482148] up:boot 2024-04-21T23:15:14.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:13 smithi044 ceph-mon[27934]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:15:14.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:13 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:15:14.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:13 smithi044 ceph-mon[27934]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:15:14.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:13 smithi044 ceph-mon[27934]: osdmap e101: 12 total, 12 up, 12 in 2024-04-21T23:15:14.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:13 smithi055 ceph-mon[31117]: pgmap v198: 129 pgs: 129 active+clean; 80 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.5 MiB/s wr, 267 op/s 2024-04-21T23:15:14.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:13 smithi055 ceph-mon[31117]: osdmap e100: 12 total, 12 up, 12 in 2024-04-21T23:15:14.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:13 smithi055 ceph-mon[31117]: mds.? [v2:172.21.15.69:6838/1729482148,v1:172.21.15.69:6839/1729482148] up:boot 2024-04-21T23:15:14.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:13 smithi055 ceph-mon[31117]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:15:14.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:13 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:15:14.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:13 smithi055 ceph-mon[31117]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:15:14.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:13 smithi055 ceph-mon[31117]: osdmap e101: 12 total, 12 up, 12 in 2024-04-21T23:15:14.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:13 smithi069 ceph-mon[31816]: pgmap v198: 129 pgs: 129 active+clean; 80 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.5 MiB/s wr, 267 op/s 2024-04-21T23:15:14.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:13 smithi069 ceph-mon[31816]: osdmap e100: 12 total, 12 up, 12 in 2024-04-21T23:15:14.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:13 smithi069 ceph-mon[31816]: mds.? [v2:172.21.15.69:6838/1729482148,v1:172.21.15.69:6839/1729482148] up:boot 2024-04-21T23:15:14.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:13 smithi069 ceph-mon[31816]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:15:14.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:13 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:15:14.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:13 smithi069 ceph-mon[31816]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:15:14.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:13 smithi069 ceph-mon[31816]: osdmap e101: 12 total, 12 up, 12 in 2024-04-21T23:15:14.704 INFO:tasks.workunit.client.0.smithi044.stdout: 1 750198 12.90 MB/sec execute 184 sec latency 102.188 ms 2024-04-21T23:15:15.704 INFO:tasks.workunit.client.0.smithi044.stdout: 1 753146 12.93 MB/sec execute 185 sec latency 11.137 ms 2024-04-21T23:15:16.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:15 smithi044 ceph-mon[27934]: pgmap v201: 129 pgs: 129 active+clean; 80 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 226 op/s 2024-04-21T23:15:16.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:15 smithi069 ceph-mon[31816]: pgmap v201: 129 pgs: 129 active+clean; 80 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 226 op/s 2024-04-21T23:15:16.218 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:15 smithi055 ceph-mon[31117]: pgmap v201: 129 pgs: 129 active+clean; 80 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 226 op/s 2024-04-21T23:15:16.704 INFO:tasks.workunit.client.0.smithi044.stdout: 1 755352 12.91 MB/sec execute 186 sec latency 121.101 ms 2024-04-21T23:15:17.704 INFO:tasks.workunit.client.0.smithi044.stdout: 1 757376 12.91 MB/sec execute 187 sec latency 16.320 ms 2024-04-21T23:15:18.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:17 smithi044 ceph-mon[27934]: pgmap v202: 129 pgs: 129 active+clean; 65 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 333 op/s 2024-04-21T23:15:18.213 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:17 smithi055 ceph-mon[31117]: pgmap v202: 129 pgs: 129 active+clean; 65 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 333 op/s 2024-04-21T23:15:18.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:17 smithi069 ceph-mon[31816]: pgmap v202: 129 pgs: 129 active+clean; 65 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.8 MiB/s wr, 333 op/s 2024-04-21T23:15:18.704 INFO:tasks.workunit.client.0.smithi044.stdout: 1 760238 12.93 MB/sec execute 188 sec latency 20.184 ms 2024-04-21T23:15:19.704 INFO:tasks.workunit.client.0.smithi044.stdout: 1 761957 12.89 MB/sec execute 189 sec latency 106.594 ms 2024-04-21T23:15:20.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:19 smithi044 ceph-mon[27934]: pgmap v203: 129 pgs: 129 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.4 MiB/s wr, 520 op/s 2024-04-21T23:15:20.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:19 smithi069 ceph-mon[31816]: pgmap v203: 129 pgs: 129 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.4 MiB/s wr, 520 op/s 2024-04-21T23:15:20.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:19 smithi055 ceph-mon[31117]: pgmap v203: 129 pgs: 129 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.4 MiB/s wr, 520 op/s 2024-04-21T23:15:20.623 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:15:20.623 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:15:20.704 INFO:tasks.workunit.client.0.smithi044.stdout: 1 764215 12.90 MB/sec execute 190 sec latency 15.086 ms 2024-04-21T23:15:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.748+0000 7fad2f1d7640 1 -- 172.21.15.44:0/2590041855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad281029b0 msgr2=0x7fad28102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.748+0000 7fad2f1d7640 1 --2- 172.21.15.44:0/2590041855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad281029b0 0x7fad28102d90 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fad10009900 tx=0x7fad1002bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.748+0000 7fad2f1d7640 1 -- 172.21.15.44:0/2590041855 shutdown_connections 2024-04-21T23:15:20.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.748+0000 7fad2f1d7640 1 --2- 172.21.15.44:0/2590041855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad28103360 0x7fad2810b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.748+0000 7fad2f1d7640 1 --2- 172.21.15.44:0/2590041855 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fad2810bec0 0x7fad2810e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.748+0000 7fad2f1d7640 1 --2- 172.21.15.44:0/2590041855 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad281029b0 0x7fad28102d90 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.748+0000 7fad2f1d7640 1 -- 172.21.15.44:0/2590041855 >> 172.21.15.44:0/2590041855 conn(0x7fad28077760 msgr2=0x7fad28077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.748+0000 7fad2f1d7640 1 -- 172.21.15.44:0/2590041855 shutdown_connections 2024-04-21T23:15:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.748+0000 7fad2f1d7640 1 -- 172.21.15.44:0/2590041855 wait complete. 2024-04-21T23:15:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.749+0000 7fad2f1d7640 1 Processor -- start 2024-04-21T23:15:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.749+0000 7fad2f1d7640 1 -- start start 2024-04-21T23:15:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.749+0000 7fad2f1d7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad281029b0 0x7fad28198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.749+0000 7fad2f1d7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fad2810bec0 0x7fad28198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.749+0000 7fad2f1d7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad2819cb10 0x7fad2819cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.749+0000 7fad2f1d7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fad28111cd0 con 0x7fad281029b0 2024-04-21T23:15:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.749+0000 7fad2f1d7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fad28111b50 con 0x7fad2810bec0 2024-04-21T23:15:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.749+0000 7fad2f1d7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fad28111e50 con 0x7fad2819cb10 2024-04-21T23:15:20.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.750+0000 7fad2cf73640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad281029b0 0x7fad28198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.750+0000 7fad2cf73640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad281029b0 0x7fad28198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53882/0 (socket says 172.21.15.44:53882) 2024-04-21T23:15:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.750+0000 7fad27fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fad2810bec0 0x7fad28198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.750+0000 7fad2cf73640 1 -- 172.21.15.44:0/2172859828 learned_addr learned my addr 172.21.15.44:0/2172859828 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.750+0000 7fad2d774640 1 --2- 172.21.15.44:0/2172859828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad2819cb10 0x7fad2819cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.750+0000 7fad2cf73640 1 -- 172.21.15.44:0/2172859828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad2819cb10 msgr2=0x7fad2819cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.750+0000 7fad2cf73640 1 --2- 172.21.15.44:0/2172859828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad2819cb10 0x7fad2819cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.750+0000 7fad2cf73640 1 -- 172.21.15.44:0/2172859828 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fad2810bec0 msgr2=0x7fad28198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.750+0000 7fad2cf73640 1 --2- 172.21.15.44:0/2172859828 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fad2810bec0 0x7fad28198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.750+0000 7fad2cf73640 1 -- 172.21.15.44:0/2172859828 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2819d600 con 0x7fad281029b0 2024-04-21T23:15:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.750+0000 7fad27fff640 1 --2- 172.21.15.44:0/2172859828 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fad2810bec0 0x7fad28198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.750+0000 7fad2cf73640 1 --2- 172.21.15.44:0/2172859828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad281029b0 0x7fad28198250 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fad10006010 tx=0x7fad1002cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.750+0000 7fad25ffb640 1 -- 172.21.15.44:0/2172859828 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad10041070 con 0x7fad281029b0 2024-04-21T23:15:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.750+0000 7fad25ffb640 1 -- 172.21.15.44:0/2172859828 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad10004450 con 0x7fad281029b0 2024-04-21T23:15:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.750+0000 7fad2f1d7640 1 -- 172.21.15.44:0/2172859828 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad2819d890 con 0x7fad281029b0 2024-04-21T23:15:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.750+0000 7fad25ffb640 1 -- 172.21.15.44:0/2172859828 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad10003e00 con 0x7fad281029b0 2024-04-21T23:15:20.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.751+0000 7fad2f1d7640 1 -- 172.21.15.44:0/2172859828 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad281a50b0 con 0x7fad281029b0 2024-04-21T23:15:20.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.753+0000 7fad25ffb640 1 -- 172.21.15.44:0/2172859828 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fad10044070 con 0x7fad281029b0 2024-04-21T23:15:20.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.754+0000 7fad31dc0740 1 Processor -- start 2024-04-21T23:15:20.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.754+0000 7fad31dc0740 1 -- start start 2024-04-21T23:15:20.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.754+0000 7fad25ffb640 1 --2- 172.21.15.44:0/2172859828 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fad140a2d80 0x7fad140a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.754+0000 7fad25ffb640 1 -- 172.21.15.44:0/2172859828 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7fad100f1160 con 0x7fad281029b0 2024-04-21T23:15:20.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.754+0000 7fad2d774640 1 --2- 172.21.15.44:0/2172859828 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fad140a2d80 0x7fad140a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.754+0000 7fad31dc0740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c97f30 0x55fc09c98310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.754+0000 7fad31dc0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09c98850 0x55fc09ca0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.754+0000 7fad31dc0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09ca1430 0x55fc09ca3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.754+0000 7fad31dc0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55fc09bfc560 con 0x55fc09c98850 2024-04-21T23:15:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.754+0000 7fad31dc0740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55fc09ba4730 con 0x55fc09c97f30 2024-04-21T23:15:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.754+0000 7fad31dc0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fc09bf8520 con 0x55fc09ca1430 2024-04-21T23:15:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.755+0000 7fad2cf73640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c97f30 0x55fc09c98310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.755+0000 7fad27fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09c98850 0x55fc09ca0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.755+0000 7fad2cf73640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c97f30 0x55fc09c98310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:58692/0 (socket says 172.21.15.44:58692) 2024-04-21T23:15:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.755+0000 7fad2cf73640 1 -- 172.21.15.44:0/1107191198 learned_addr learned my addr 172.21.15.44:0/1107191198 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.755+0000 7fad2d774640 1 --2- 172.21.15.44:0/1107191198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09ca1430 0x55fc09ca3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.755+0000 7fad2cf73640 1 -- 172.21.15.44:0/1107191198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09ca1430 msgr2=0x55fc09ca3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.755+0000 7fad2cf73640 1 --2- 172.21.15.44:0/1107191198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09ca1430 0x55fc09ca3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.755+0000 7fad2cf73640 1 -- 172.21.15.44:0/1107191198 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09c98850 msgr2=0x55fc09ca0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.755+0000 7fad2cf73640 1 --2- 172.21.15.44:0/1107191198 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09c98850 0x55fc09ca0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.755+0000 7fad2cf73640 1 -- 172.21.15.44:0/1107191198 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc099da7d0 con 0x55fc09c97f30 2024-04-21T23:15:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.755+0000 7fad2d774640 1 --2- 172.21.15.44:0/1107191198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09ca1430 0x55fc09ca3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:15:20.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.755+0000 7fad27fff640 1 --2- 172.21.15.44:0/1107191198 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09c98850 0x55fc09ca0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.755+0000 7fad2d774640 1 --2- 172.21.15.44:0/2172859828 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fad140a2d80 0x7fad140a5240 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fad20005fd0 tx=0x7fad20009500 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.755+0000 7fad2cf73640 1 --2- 172.21.15.44:0/1107191198 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c97f30 0x55fc09c98310 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fad100c5a80 tx=0x7fad100bcf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c15f0baa8874f5a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.755+0000 7fad1effd640 1 -- 172.21.15.44:0/1107191198 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1004b040 con 0x55fc09c97f30 2024-04-21T23:15:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.755+0000 7fad1effd640 1 -- 172.21.15.44:0/1107191198 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad100bd3f0 con 0x55fc09c97f30 2024-04-21T23:15:20.757 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.755+0000 7fad1effd640 1 -- 172.21.15.44:0/1107191198 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad100b9a60 con 0x55fc09c97f30 2024-04-21T23:15:20.761 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.756+0000 7fad31dc0740 1 -- 172.21.15.44:0/1107191198 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c97f30 msgr2=0x55fc09c98310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:20.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.756+0000 7fad31dc0740 1 --2- 172.21.15.44:0/1107191198 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c97f30 0x55fc09c98310 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fad100c5a80 tx=0x7fad100bcf30 comp rx=0 tx=0).stop 2024-04-21T23:15:20.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.756+0000 7fad31dc0740 1 -- 172.21.15.44:0/1107191198 shutdown_connections 2024-04-21T23:15:20.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.756+0000 7fad31dc0740 1 --2- 172.21.15.44:0/1107191198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09ca1430 0x55fc09ca3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.756+0000 7fad31dc0740 1 --2- 172.21.15.44:0/1107191198 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09c98850 0x55fc09ca0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.756+0000 7fad31dc0740 1 --2- 172.21.15.44:0/1107191198 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c97f30 0x55fc09c98310 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.756+0000 7fad31dc0740 1 -- 172.21.15.44:0/1107191198 >> 172.21.15.44:0/1107191198 conn(0x55fc09c92660 msgr2=0x55fc09c92a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:20.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.756+0000 7fad31dc0740 1 -- 172.21.15.44:0/1107191198 shutdown_connections 2024-04-21T23:15:20.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.756+0000 7fad31dc0740 1 -- 172.21.15.44:0/1107191198 wait complete. 2024-04-21T23:15:20.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.757+0000 7fad31dc0740 1 -- start start 2024-04-21T23:15:20.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.757+0000 7fad31dc0740 1 Processor -- start 2024-04-21T23:15:20.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.757+0000 7fad31dc0740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c97f30 0x55fc09c9e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.757+0000 7fad31dc0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09c98850 0x55fc09c9eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.757+0000 7fad31dc0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09ca1430 0x55fc09c9eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.758+0000 7fad31dc0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55fc09ca3fe0 con 0x55fc09c98850 2024-04-21T23:15:20.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.758+0000 7fad31dc0740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55fc09bb7e00 con 0x55fc09c97f30 2024-04-21T23:15:20.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.758+0000 7fad2cf73640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c97f30 0x55fc09c9e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.758+0000 7fad27fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09c98850 0x55fc09c9eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.758+0000 7fad31dc0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fc09ca4160 con 0x55fc09ca1430 2024-04-21T23:15:20.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.758+0000 7fad2cf73640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c97f30 0x55fc09c9e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:58702/0 (socket says 172.21.15.44:58702) 2024-04-21T23:15:20.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.758+0000 7fad2cf73640 1 -- 172.21.15.44:0/778812642 learned_addr learned my addr 172.21.15.44:0/778812642 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:20.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.758+0000 7fad31dc0740 10 client.0 _ll_register_callbacks cb 0x55fc09bf2eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:20.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.758+0000 7fad2d774640 1 --2- 172.21.15.44:0/778812642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09ca1430 0x55fc09c9eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.758+0000 7fad27fff640 1 -- 172.21.15.44:0/778812642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09ca1430 msgr2=0x55fc09c9eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:20.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.758+0000 7fad27fff640 1 --2- 172.21.15.44:0/778812642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09ca1430 0x55fc09c9eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.758+0000 7fad27fff640 1 -- 172.21.15.44:0/778812642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c97f30 msgr2=0x55fc09c9e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:20.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.758+0000 7fad27fff640 1 --2- 172.21.15.44:0/778812642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c97f30 0x55fc09c9e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.758+0000 7fad27fff640 1 -- 172.21.15.44:0/778812642 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc099103f0 con 0x55fc09c98850 2024-04-21T23:15:20.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.758+0000 7fad2cf73640 1 --2- 172.21.15.44:0/778812642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c97f30 0x55fc09c9e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:20.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.758+0000 7fad27fff640 1 --2- 172.21.15.44:0/778812642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09c98850 0x55fc09c9eab0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fad2810dd40 tx=0x7fad1800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d205e5f1ba6960c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:20.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.758+0000 7fad07fff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:15:20.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.758+0000 7fad07fff640 1 -- 172.21.15.44:0/778812642 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad18004260 con 0x55fc09c98850 2024-04-21T23:15:20.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.758+0000 7fad07fff640 1 -- 172.21.15.44:0/778812642 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7fad18004400 con 0x55fc09c98850 2024-04-21T23:15:20.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.758+0000 7fad31dc0740 1 -- 172.21.15.44:0/778812642 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad100bd3f0 con 0x55fc09c98850 2024-04-21T23:15:20.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.759+0000 7fad07fff640 1 -- 172.21.15.44:0/778812642 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fad18016110 con 0x55fc09c98850 2024-04-21T23:15:20.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.759+0000 7fad31dc0740 10 client.17094 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:20.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.759+0000 7fad31dc0740 1 -- 172.21.15.44:0/778812642 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fc098d00f0 con 0x55fc09c98850 2024-04-21T23:15:20.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.759+0000 7fad07fff640 1 -- 172.21.15.44:0/778812642 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad100bd3f0 con 0x55fc09c98850 2024-04-21T23:15:20.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.759+0000 7fad07fff640 1 -- 172.21.15.44:0/778812642 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7fad18003680 con 0x55fc09c98850 2024-04-21T23:15:20.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.759+0000 7fad31dc0740 10 client.17094 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:20.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.759+0000 7fad31dc0740 10 client.17094 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:20.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.759+0000 7fad31dc0740 20 client.17094 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:20.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.759+0000 7fad31dc0740 1 --2- 172.21.15.44:0/778812642 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55fc09cb1390 0x55fc09cb3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.759+0000 7fad31dc0740 4 client.17094 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:15:20.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.759+0000 7fad31dc0740 1 -- 172.21.15.44:0/778812642 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc09c09aa0 con 0x55fc09cb1390 2024-04-21T23:15:20.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.759+0000 7fad2cf73640 1 --2- 172.21.15.44:0/778812642 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55fc09cb1390 0x55fc09cb3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.760+0000 7fad2cf73640 1 --2- 172.21.15.44:0/778812642 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55fc09cb1390 0x55fc09cb3780 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4a8b99defc437c3 server_cookie=2ba8d0fa01bd0659 in_seq=0 out_seq=0 2024-04-21T23:15:20.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.760+0000 7fad07fff640 10 client.17094 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:20.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.760+0000 7fad07fff640 1 -- 172.21.15.44:0/778812642 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fc09c09aa0 con 0x55fc09cb1390 2024-04-21T23:15:20.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.760+0000 7fad07fff640 10 client.17094 handle_command_reply: tid=0 2024-04-21T23:15:20.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.760+0000 7fad31dc0740 1 client.17094 shutdown 2024-04-21T23:15:20.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.761+0000 7fad31dc0740 1 -- 172.21.15.44:0/778812642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09c98850 msgr2=0x55fc09c9eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:20.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.761+0000 7fad31dc0740 1 --2- 172.21.15.44:0/778812642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09c98850 0x55fc09c9eab0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fad2810dd40 tx=0x7fad1800f790 comp rx=0 tx=0).stop 2024-04-21T23:15:20.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.761+0000 7fad31dc0740 1 -- 172.21.15.44:0/778812642 shutdown_connections 2024-04-21T23:15:20.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.761+0000 7fad31dc0740 1 --2- 172.21.15.44:0/778812642 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55fc09cb1390 0x55fc09cb3780 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.761+0000 7fad31dc0740 1 --2- 172.21.15.44:0/778812642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09ca1430 0x55fc09c9eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.761+0000 7fad31dc0740 1 --2- 172.21.15.44:0/778812642 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09c98850 0x55fc09c9eab0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.761+0000 7fad31dc0740 1 --2- 172.21.15.44:0/778812642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c97f30 0x55fc09c9e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.761+0000 7fad31dc0740 1 -- 172.21.15.44:0/778812642 >> 172.21.15.44:0/778812642 conn(0x55fc09c92660 msgr2=0x55fc09c943b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:20.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.761+0000 7fad07fff640 0 client.17094 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:20.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.761+0000 7fad31dc0740 1 -- 172.21.15.44:0/778812642 shutdown_connections 2024-04-21T23:15:20.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.761+0000 7fad31dc0740 1 -- 172.21.15.44:0/778812642 wait complete. 2024-04-21T23:15:20.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.761+0000 7fad31dc0740 20 client.17094 trim_cache size 0 max 16384 2024-04-21T23:15:20.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.781+0000 7fad31dc0740 1 Processor -- start 2024-04-21T23:15:20.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.782+0000 7fad31dc0740 1 -- start start 2024-04-21T23:15:20.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.782+0000 7fad31dc0740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c97f30 0x55fc09ca0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.782+0000 7fad31dc0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09c98850 0x55fc09cace10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.782+0000 7fad31dc0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09ca1430 0x55fc09c91560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.782+0000 7fad31dc0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fad281aacb0 con 0x55fc09ca1430 2024-04-21T23:15:20.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.782+0000 7fad31dc0740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fad281ad510 con 0x55fc09c97f30 2024-04-21T23:15:20.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.782+0000 7fad31dc0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fad281ab300 con 0x55fc09c98850 2024-04-21T23:15:20.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.782+0000 7fad2d774640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09ca1430 0x55fc09c91560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.782+0000 7fad2d774640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09ca1430 0x55fc09c91560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53914/0 (socket says 172.21.15.44:53914) 2024-04-21T23:15:20.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.782+0000 7fad2d774640 1 -- 172.21.15.44:0/3242589631 learned_addr learned my addr 172.21.15.44:0/3242589631 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:20.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.782+0000 7fad2cf73640 1 --2- 172.21.15.44:0/3242589631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c97f30 0x55fc09ca0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.782+0000 7fad27fff640 1 --2- 172.21.15.44:0/3242589631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09c98850 0x55fc09cace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.782+0000 7fad2d774640 1 -- 172.21.15.44:0/3242589631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09c98850 msgr2=0x55fc09cace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:20.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.782+0000 7fad2d774640 1 --2- 172.21.15.44:0/3242589631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09c98850 0x55fc09cace10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.782+0000 7fad2d774640 1 -- 172.21.15.44:0/3242589631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c97f30 msgr2=0x55fc09ca0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:20.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.782+0000 7fad2d774640 1 --2- 172.21.15.44:0/3242589631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c97f30 0x55fc09ca0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.782+0000 7fad2d774640 1 -- 172.21.15.44:0/3242589631 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc09c98650 con 0x55fc09ca1430 2024-04-21T23:15:20.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.782+0000 7fad2d774640 1 --2- 172.21.15.44:0/3242589631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09ca1430 0x55fc09c91560 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fad20003c40 tx=0x7fad2001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a1ea173997b1f1b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:20.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.783+0000 7fad1e7fc640 1 -- 172.21.15.44:0/3242589631 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20004d20 con 0x55fc09ca1430 2024-04-21T23:15:20.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.783+0000 7fad1e7fc640 1 -- 172.21.15.44:0/3242589631 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad20005500 con 0x55fc09ca1430 2024-04-21T23:15:20.784 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.783+0000 7fad1e7fc640 1 -- 172.21.15.44:0/3242589631 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20027870 con 0x55fc09ca1430 2024-04-21T23:15:20.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.783+0000 7fad31dc0740 1 -- 172.21.15.44:0/3242589631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09ca1430 msgr2=0x55fc09c91560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:20.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.783+0000 7fad31dc0740 1 --2- 172.21.15.44:0/3242589631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09ca1430 0x55fc09c91560 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fad20003c40 tx=0x7fad2001a000 comp rx=0 tx=0).stop 2024-04-21T23:15:20.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.783+0000 7fad31dc0740 1 -- 172.21.15.44:0/3242589631 shutdown_connections 2024-04-21T23:15:20.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.783+0000 7fad31dc0740 1 --2- 172.21.15.44:0/3242589631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09c98850 0x55fc09cace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.783+0000 7fad31dc0740 1 --2- 172.21.15.44:0/3242589631 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09ca1430 0x55fc09c91560 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.783+0000 7fad31dc0740 1 --2- 172.21.15.44:0/3242589631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c97f30 0x55fc09ca0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.783+0000 7fad31dc0740 1 -- 172.21.15.44:0/3242589631 >> 172.21.15.44:0/3242589631 conn(0x55fc09c92660 msgr2=0x55fc09c90e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:20.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.784+0000 7fad31dc0740 1 -- 172.21.15.44:0/3242589631 shutdown_connections 2024-04-21T23:15:20.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.784+0000 7fad31dc0740 1 -- 172.21.15.44:0/3242589631 wait complete. 2024-04-21T23:15:20.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.784+0000 7fad31dc0740 1 -- start start 2024-04-21T23:15:20.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.784+0000 7fad31dc0740 1 Processor -- start 2024-04-21T23:15:20.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.784+0000 7fad31dc0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09c97f30 0x55fc09ca9820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.784+0000 7fad31dc0740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c98850 0x55fc09ca4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.784+0000 7fad31dc0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09ca1430 0x55fc09ca52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.784+0000 7fad31dc0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fad281ab9d0 con 0x55fc09ca1430 2024-04-21T23:15:20.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.784+0000 7fad31dc0740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fad281ace40 con 0x55fc09c98850 2024-04-21T23:15:20.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.784+0000 7fad31dc0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fad281ac0a0 con 0x55fc09c97f30 2024-04-21T23:15:20.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.784+0000 7fad31dc0740 10 client.0 _ll_register_callbacks cb 0x7fad18009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:20.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.784+0000 7fad2cf73640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09c97f30 0x55fc09ca9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.784+0000 7fad27fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c98850 0x55fc09ca4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.784+0000 7fad2d774640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09ca1430 0x55fc09ca52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.784+0000 7fad2cf73640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09c97f30 0x55fc09ca9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:49274/0 (socket says 172.21.15.44:49274) 2024-04-21T23:15:20.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.784+0000 7fad27fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c98850 0x55fc09ca4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:58730/0 (socket says 172.21.15.44:58730) 2024-04-21T23:15:20.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.784+0000 7fad2cf73640 1 -- 172.21.15.44:0/666737549 learned_addr learned my addr 172.21.15.44:0/666737549 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:20.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.785+0000 7fad2d774640 1 -- 172.21.15.44:0/666737549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09c97f30 msgr2=0x55fc09ca9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:20.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.785+0000 7fad2d774640 1 --2- 172.21.15.44:0/666737549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09c97f30 0x55fc09ca9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.785+0000 7fad2d774640 1 -- 172.21.15.44:0/666737549 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c98850 msgr2=0x55fc09ca4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:20.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.785+0000 7fad2d774640 1 --2- 172.21.15.44:0/666737549 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c98850 0x55fc09ca4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.785+0000 7fad2d774640 1 -- 172.21.15.44:0/666737549 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc09d16900 con 0x55fc09ca1430 2024-04-21T23:15:20.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.785+0000 7fad27fff640 1 --2- 172.21.15.44:0/666737549 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c98850 0x55fc09ca4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:20.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.785+0000 7fad2d774640 1 --2- 172.21.15.44:0/666737549 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09ca1430 0x55fc09ca52d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fad20003c10 tx=0x7fad20005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=687593cadc9f0b6d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:20.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.785+0000 7fad07fff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:15:20.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.785+0000 7fad07fff640 1 -- 172.21.15.44:0/666737549 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20030060 con 0x55fc09ca1430 2024-04-21T23:15:20.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.785+0000 7fad07fff640 1 -- 172.21.15.44:0/666737549 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7fad20005000 con 0x55fc09ca1430 2024-04-21T23:15:20.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.785+0000 7fad31dc0740 1 -- 172.21.15.44:0/666737549 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad20005500 con 0x55fc09ca1430 2024-04-21T23:15:20.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.785+0000 7fad07fff640 1 -- 172.21.15.44:0/666737549 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fad20037050 con 0x55fc09ca1430 2024-04-21T23:15:20.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.785+0000 7fad31dc0740 10 client.17100 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:20.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.785+0000 7fad31dc0740 1 -- 172.21.15.44:0/666737549 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fc09d17a40 con 0x55fc09ca1430 2024-04-21T23:15:20.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.785+0000 7fad07fff640 1 -- 172.21.15.44:0/666737549 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad20005500 con 0x55fc09ca1430 2024-04-21T23:15:20.790 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.786+0000 7fad07fff640 1 -- 172.21.15.44:0/666737549 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7fad2001c420 con 0x55fc09ca1430 2024-04-21T23:15:20.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.786+0000 7fad31dc0740 10 client.17100 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:20.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.786+0000 7fad31dc0740 10 client.17100 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:20.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.786+0000 7fad31dc0740 20 client.17100 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:20.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.786+0000 7fad31dc0740 1 --2- 172.21.15.44:0/666737549 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55fc09d18160 0x55fc09d1a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.786+0000 7fad31dc0740 4 client.17100 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:15:20.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.786+0000 7fad31dc0740 1 -- 172.21.15.44:0/666737549 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fc09c03df0 con 0x55fc09d18160 2024-04-21T23:15:20.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.786+0000 7fad2cf73640 1 --2- 172.21.15.44:0/666737549 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55fc09d18160 0x55fc09d1a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.786+0000 7fad07fff640 10 client.17100 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:20.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.786+0000 7fad2cf73640 1 --2- 172.21.15.44:0/666737549 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55fc09d18160 0x55fc09d1a580 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=578f81fc3f0128e7 server_cookie=12ec0bce42466a41 in_seq=0 out_seq=0 2024-04-21T23:15:20.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.787+0000 7fad07fff640 1 -- 172.21.15.44:0/666737549 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55fc09c03df0 con 0x55fc09d18160 2024-04-21T23:15:20.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.787+0000 7fad07fff640 10 client.17100 handle_command_reply: tid=0 2024-04-21T23:15:20.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.787+0000 7fad31dc0740 1 client.17100 shutdown 2024-04-21T23:15:20.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.787+0000 7fad31dc0740 1 -- 172.21.15.44:0/666737549 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09ca1430 msgr2=0x55fc09ca52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:20.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.787+0000 7fad31dc0740 1 --2- 172.21.15.44:0/666737549 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09ca1430 0x55fc09ca52d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fad20003c10 tx=0x7fad20005810 comp rx=0 tx=0).stop 2024-04-21T23:15:20.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.787+0000 7fad31dc0740 1 -- 172.21.15.44:0/666737549 shutdown_connections 2024-04-21T23:15:20.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.787+0000 7fad31dc0740 1 --2- 172.21.15.44:0/666737549 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55fc09d18160 0x55fc09d1a580 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.787+0000 7fad31dc0740 1 --2- 172.21.15.44:0/666737549 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55fc09ca1430 0x55fc09ca52d0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.788+0000 7fad31dc0740 1 --2- 172.21.15.44:0/666737549 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55fc09c98850 0x55fc09ca4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.788+0000 7fad31dc0740 1 --2- 172.21.15.44:0/666737549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc09c97f30 0x55fc09ca9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.788+0000 7fad31dc0740 1 -- 172.21.15.44:0/666737549 >> 172.21.15.44:0/666737549 conn(0x55fc09c92660 msgr2=0x55fc09c9e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:20.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.788+0000 7fad07fff640 0 client.17100 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:20.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.788+0000 7fad31dc0740 1 -- 172.21.15.44:0/666737549 shutdown_connections 2024-04-21T23:15:20.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.788+0000 7fad31dc0740 1 -- 172.21.15.44:0/666737549 wait complete. 2024-04-21T23:15:20.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.788+0000 7fad31dc0740 20 client.17100 trim_cache size 0 max 16384 2024-04-21T23:15:20.792 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:15:20.792 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-21T23:15:20.793 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-21T23:15:20.793 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:15:20.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.788+0000 7fad2f1d7640 1 -- 172.21.15.44:0/2172859828 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fad140a2d80 msgr2=0x7fad140a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:20.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.788+0000 7fad2f1d7640 1 --2- 172.21.15.44:0/2172859828 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fad140a2d80 0x7fad140a5240 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fad20005fd0 tx=0x7fad20009500 comp rx=0 tx=0).stop 2024-04-21T23:15:20.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.788+0000 7fad2f1d7640 1 -- 172.21.15.44:0/2172859828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad281029b0 msgr2=0x7fad28198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:20.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.788+0000 7fad2f1d7640 1 --2- 172.21.15.44:0/2172859828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad281029b0 0x7fad28198250 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fad10006010 tx=0x7fad1002cb30 comp rx=0 tx=0).stop 2024-04-21T23:15:20.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.789+0000 7fad2f1d7640 1 -- 172.21.15.44:0/2172859828 shutdown_connections 2024-04-21T23:15:20.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.789+0000 7fad2f1d7640 1 --2- 172.21.15.44:0/2172859828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fad2819cb10 0x7fad2819cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.789+0000 7fad2f1d7640 1 --2- 172.21.15.44:0/2172859828 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fad140a2d80 0x7fad140a5240 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.789+0000 7fad2f1d7640 1 --2- 172.21.15.44:0/2172859828 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fad2810bec0 0x7fad28198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.789+0000 7fad2f1d7640 1 --2- 172.21.15.44:0/2172859828 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fad281029b0 0x7fad28198250 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.789+0000 7fad2f1d7640 1 -- 172.21.15.44:0/2172859828 >> 172.21.15.44:0/2172859828 conn(0x7fad28077760 msgr2=0x7fad280fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:20.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.789+0000 7fad2f1d7640 1 -- 172.21.15.44:0/2172859828 shutdown_connections 2024-04-21T23:15:20.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.789+0000 7fad2f1d7640 1 -- 172.21.15.44:0/2172859828 wait complete. 2024-04-21T23:15:20.800 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:15:20.801 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:15:20.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.909+0000 7f819e79f640 1 -- 172.21.15.44:0/727173917 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f819810bbf0 msgr2=0x7f819810e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:20.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.909+0000 7f819e79f640 1 --2- 172.21.15.44:0/727173917 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f819810bbf0 0x7f819810e090 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f8190009ca0 tx=0x7f819002c180 comp rx=0 tx=0).stop 2024-04-21T23:15:20.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.909+0000 7f819e79f640 1 -- 172.21.15.44:0/727173917 shutdown_connections 2024-04-21T23:15:20.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.909+0000 7f819e79f640 1 --2- 172.21.15.44:0/727173917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81981030f0 0x7f819810b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.909+0000 7f819e79f640 1 --2- 172.21.15.44:0/727173917 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f819810bbf0 0x7f819810e090 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.909+0000 7f819e79f640 1 --2- 172.21.15.44:0/727173917 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81981027d0 0x7f8198102bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.909+0000 7f819e79f640 1 -- 172.21.15.44:0/727173917 >> 172.21.15.44:0/727173917 conn(0x7f81980fbfd0 msgr2=0x7f81980fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:20.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.909+0000 7f819e79f640 1 -- 172.21.15.44:0/727173917 shutdown_connections 2024-04-21T23:15:20.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.909+0000 7f819e79f640 1 -- 172.21.15.44:0/727173917 wait complete. 2024-04-21T23:15:20.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.910+0000 7f819e79f640 1 Processor -- start 2024-04-21T23:15:20.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.910+0000 7f819e79f640 1 -- start start 2024-04-21T23:15:20.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.910+0000 7f819e79f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81981027d0 0x7f819819a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.910+0000 7f819e79f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81981030f0 0x7f819819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.910+0000 7f819e79f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f819810bbf0 0x7f819819e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.911 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.910+0000 7f819e79f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f81981119f0 con 0x7f81981030f0 2024-04-21T23:15:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.910+0000 7f819e79f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8198111870 con 0x7f81981027d0 2024-04-21T23:15:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.910+0000 7f819e79f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8198111b70 con 0x7f819810bbf0 2024-04-21T23:15:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.911+0000 7f819d79d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81981027d0 0x7f819819a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.911+0000 7f819cf9c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81981030f0 0x7f819819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.911+0000 7f819d79d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81981027d0 0x7f819819a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:58744/0 (socket says 172.21.15.44:58744) 2024-04-21T23:15:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.911+0000 7f819d79d640 1 -- 172.21.15.44:0/2771448195 learned_addr learned my addr 172.21.15.44:0/2771448195 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.911+0000 7f819df9e640 1 --2- 172.21.15.44:0/2771448195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f819810bbf0 0x7f819819e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.911+0000 7f819d79d640 1 -- 172.21.15.44:0/2771448195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f819810bbf0 msgr2=0x7f819819e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.911+0000 7f819d79d640 1 --2- 172.21.15.44:0/2771448195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f819810bbf0 0x7f819819e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.911+0000 7f819d79d640 1 -- 172.21.15.44:0/2771448195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81981030f0 msgr2=0x7f819819a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.911+0000 7f819d79d640 1 --2- 172.21.15.44:0/2771448195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81981030f0 0x7f819819a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.911+0000 7f819d79d640 1 -- 172.21.15.44:0/2771448195 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f819819f0a0 con 0x7f81981027d0 2024-04-21T23:15:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.911+0000 7f819cf9c640 1 --2- 172.21.15.44:0/2771448195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81981030f0 0x7f819819a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.911+0000 7f819d79d640 1 --2- 172.21.15.44:0/2771448195 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81981027d0 0x7f819819a0b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f819400e9f0 tx=0x7f819400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.911+0000 7f818e7fc640 1 -- 172.21.15.44:0/2771448195 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f819400cd80 con 0x7f81981027d0 2024-04-21T23:15:20.912 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.911+0000 7f819e79f640 1 -- 172.21.15.44:0/2771448195 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f819819f390 con 0x7f81981027d0 2024-04-21T23:15:20.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.911+0000 7f818e7fc640 1 -- 172.21.15.44:0/2771448195 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8194010910 con 0x7f81981027d0 2024-04-21T23:15:20.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.911+0000 7f819e79f640 1 -- 172.21.15.44:0/2771448195 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8198103570 con 0x7f81981027d0 2024-04-21T23:15:20.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.911+0000 7f818e7fc640 1 -- 172.21.15.44:0/2771448195 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81940188f0 con 0x7f81981027d0 2024-04-21T23:15:20.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.912+0000 7f819e79f640 1 -- 172.21.15.44:0/2771448195 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f819804f570 con 0x7f81981027d0 2024-04-21T23:15:20.913 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.914+0000 7f818e7fc640 1 -- 172.21.15.44:0/2771448195 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8194010ac0 con 0x7f81981027d0 2024-04-21T23:15:20.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.915+0000 7f818e7fc640 1 --2- 172.21.15.44:0/2771448195 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f81840a2cf0 0x7f81840a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:20.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.915+0000 7f819cf9c640 1 --2- 172.21.15.44:0/2771448195 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f81840a2cf0 0x7f81840a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:20.914 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.915+0000 7f818e7fc640 1 -- 172.21.15.44:0/2771448195 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f81940cffb0 con 0x7f81981027d0 2024-04-21T23:15:20.915 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.915+0000 7f819cf9c640 1 --2- 172.21.15.44:0/2771448195 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f81840a2cf0 0x7f81840a51b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f819810e4a0 tx=0x7f8188005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:20.918 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:20.918+0000 7f818e7fc640 1 -- 172.21.15.44:0/2771448195 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f819409c190 con 0x7f81981027d0 2024-04-21T23:15:21.172 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.173+0000 7f819e79f640 1 -- 172.21.15.44:0/2771448195 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f819819b400 con 0x7f81981027d0 2024-04-21T23:15:21.173 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.174+0000 7f818e7fc640 1 -- 172.21.15.44:0/2771448195 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11791 (secure 0 0 0) 0x7f81940a0490 con 0x7f81981027d0 2024-04-21T23:15:21.173 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:15:21.173 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:15:12.778523+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26645":{"gid":26645,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/1729482148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1729482148},{"type":"v1","addr":"172.21.15.69:6839","nonce":1729482148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:15:21.174 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 29 2024-04-21T23:15:21.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.176+0000 7f819e79f640 1 -- 172.21.15.44:0/2771448195 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f81840a2cf0 msgr2=0x7f81840a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:21.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.176+0000 7f819e79f640 1 --2- 172.21.15.44:0/2771448195 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f81840a2cf0 0x7f81840a51b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f819810e4a0 tx=0x7f8188005f90 comp rx=0 tx=0).stop 2024-04-21T23:15:21.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.176+0000 7f819e79f640 1 -- 172.21.15.44:0/2771448195 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81981027d0 msgr2=0x7f819819a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:21.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.176+0000 7f819e79f640 1 --2- 172.21.15.44:0/2771448195 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81981027d0 0x7f819819a0b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f819400e9f0 tx=0x7f819400eeb0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.176+0000 7f819e79f640 1 -- 172.21.15.44:0/2771448195 shutdown_connections 2024-04-21T23:15:21.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.176+0000 7f819e79f640 1 --2- 172.21.15.44:0/2771448195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f819810bbf0 0x7f819819e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.176+0000 7f819e79f640 1 --2- 172.21.15.44:0/2771448195 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f81840a2cf0 0x7f81840a51b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.176+0000 7f819e79f640 1 --2- 172.21.15.44:0/2771448195 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f81981030f0 0x7f819819a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.176+0000 7f819e79f640 1 --2- 172.21.15.44:0/2771448195 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f81981027d0 0x7f819819a0b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.176+0000 7f819e79f640 1 -- 172.21.15.44:0/2771448195 >> 172.21.15.44:0/2771448195 conn(0x7f81980fbfd0 msgr2=0x7f819810ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:21.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.176+0000 7f819e79f640 1 -- 172.21.15.44:0/2771448195 shutdown_connections 2024-04-21T23:15:21.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.176+0000 7f819e79f640 1 -- 172.21.15.44:0/2771448195 wait complete. 2024-04-21T23:15:21.187 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:15:21.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.303+0000 7f4e48d3e640 1 -- 172.21.15.44:0/878314229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e4410bcf0 msgr2=0x7f4e4410e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:21.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.303+0000 7f4e48d3e640 1 --2- 172.21.15.44:0/878314229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e4410bcf0 0x7f4e4410e190 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c009ca0 tx=0x7f4e3c02c180 comp rx=0 tx=0).stop 2024-04-21T23:15:21.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.303+0000 7f4e48d3e640 1 -- 172.21.15.44:0/878314229 shutdown_connections 2024-04-21T23:15:21.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.303+0000 7f4e48d3e640 1 --2- 172.21.15.44:0/878314229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e441032d0 0x7f4e4410b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.303+0000 7f4e48d3e640 1 --2- 172.21.15.44:0/878314229 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e4410bcf0 0x7f4e4410e190 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.303+0000 7f4e48d3e640 1 --2- 172.21.15.44:0/878314229 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e441029b0 0x7f4e44102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.303+0000 7f4e48d3e640 1 -- 172.21.15.44:0/878314229 >> 172.21.15.44:0/878314229 conn(0x7f4e440777f0 msgr2=0x7f4e44077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:21.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.303+0000 7f4e48d3e640 1 -- 172.21.15.44:0/878314229 shutdown_connections 2024-04-21T23:15:21.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.303+0000 7f4e48d3e640 1 -- 172.21.15.44:0/878314229 wait complete. 2024-04-21T23:15:21.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.304+0000 7f4e48d3e640 1 Processor -- start 2024-04-21T23:15:21.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.304+0000 7f4e48d3e640 1 -- start start 2024-04-21T23:15:21.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.304+0000 7f4e48d3e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e441029b0 0x7f4e44100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.304+0000 7f4e48d3e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e441032d0 0x7f4e44100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.304+0000 7f4e48d3e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e4410bcf0 0x7f4e44101080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.304+0000 7f4e48d3e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4e44111af0 con 0x7f4e441032d0 2024-04-21T23:15:21.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.304+0000 7f4e48d3e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4e44111970 con 0x7f4e441029b0 2024-04-21T23:15:21.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.304+0000 7f4e48d3e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4e44111c70 con 0x7f4e4410bcf0 2024-04-21T23:15:21.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.304+0000 7f4e39d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e441032d0 0x7f4e44100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.304+0000 7f4e4259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e441029b0 0x7f4e44100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.304+0000 7f4e39d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e441032d0 0x7f4e44100b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53950/0 (socket says 172.21.15.44:53950) 2024-04-21T23:15:21.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.304+0000 7f4e42d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e4410bcf0 0x7f4e44101080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.304+0000 7f4e39d9b640 1 -- 172.21.15.44:0/411413410 learned_addr learned my addr 172.21.15.44:0/411413410 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:21.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.305+0000 7f4e4259c640 1 -- 172.21.15.44:0/411413410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e4410bcf0 msgr2=0x7f4e44101080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:21.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.305+0000 7f4e4259c640 1 --2- 172.21.15.44:0/411413410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e4410bcf0 0x7f4e44101080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.305+0000 7f4e4259c640 1 -- 172.21.15.44:0/411413410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e441032d0 msgr2=0x7f4e44100b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:21.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.305+0000 7f4e4259c640 1 --2- 172.21.15.44:0/411413410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e441032d0 0x7f4e44100b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.305+0000 7f4e4259c640 1 -- 172.21.15.44:0/411413410 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e441a6550 con 0x7f4e441029b0 2024-04-21T23:15:21.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.305+0000 7f4e39d9b640 1 --2- 172.21.15.44:0/411413410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e441032d0 0x7f4e44100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:21.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.305+0000 7f4e4259c640 1 --2- 172.21.15.44:0/411413410 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e441029b0 0x7f4e44100600 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400e9f0 tx=0x7f4e3400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:21.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.305+0000 7f4e3bfff640 1 -- 172.21.15.44:0/411413410 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3400ccc0 con 0x7f4e441029b0 2024-04-21T23:15:21.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.305+0000 7f4e48d3e640 1 -- 172.21.15.44:0/411413410 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e441a6840 con 0x7f4e441029b0 2024-04-21T23:15:21.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.305+0000 7f4e3bfff640 1 -- 172.21.15.44:0/411413410 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e34010910 con 0x7f4e441029b0 2024-04-21T23:15:21.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.305+0000 7f4e3bfff640 1 -- 172.21.15.44:0/411413410 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e34018960 con 0x7f4e441029b0 2024-04-21T23:15:21.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.305+0000 7f4e48d3e640 1 -- 172.21.15.44:0/411413410 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e441a6de0 con 0x7f4e441029b0 2024-04-21T23:15:21.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.308+0000 7f4e3bfff640 1 -- 172.21.15.44:0/411413410 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f4e34010ac0 con 0x7f4e441029b0 2024-04-21T23:15:21.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.308+0000 7f4e4ba08740 1 Processor -- start 2024-04-21T23:15:21.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.308+0000 7f4e3bfff640 1 --2- 172.21.15.44:0/411413410 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4e300a2d00 0x7f4e300a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e4ba08740 1 -- start start 2024-04-21T23:15:21.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e42d9d640 1 --2- 172.21.15.44:0/411413410 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4e300a2d00 0x7f4e300a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e3bfff640 1 -- 172.21.15.44:0/411413410 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f4e340cfbf0 con 0x7f4e441029b0 2024-04-21T23:15:21.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e4ba08740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd05020 0x55a53bd05400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e4ba08740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd05940 0x55a53bd0dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e4ba08740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd0e520 0x55a53bd10910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e4ba08740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a53bc69530 con 0x55a53bd05940 2024-04-21T23:15:21.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e4ba08740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55a53bc11830 con 0x55a53bd05020 2024-04-21T23:15:21.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e4ba08740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a53bc654f0 con 0x55a53bd0e520 2024-04-21T23:15:21.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e39d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd05940 0x55a53bd0dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e4259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd05020 0x55a53bd05400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e39d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd05940 0x55a53bd0dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53964/0 (socket says 172.21.15.44:53964) 2024-04-21T23:15:21.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e39d9b640 1 -- 172.21.15.44:0/1935083118 learned_addr learned my addr 172.21.15.44:0/1935083118 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:21.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e42d9d640 1 --2- 172.21.15.44:0/411413410 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4e300a2d00 0x7f4e300a51c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c004af0 tx=0x7f4e3c005c50 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:21.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e42d9d640 1 --2- 172.21.15.44:0/1935083118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd0e520 0x55a53bd10910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e4259c640 1 -- 172.21.15.44:0/1935083118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd0e520 msgr2=0x55a53bd10910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:21.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e4259c640 1 --2- 172.21.15.44:0/1935083118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd0e520 0x55a53bd10910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e4259c640 1 -- 172.21.15.44:0/1935083118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd05940 msgr2=0x55a53bd0dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e4259c640 1 --2- 172.21.15.44:0/1935083118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd05940 0x55a53bd0dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e4259c640 1 -- 172.21.15.44:0/1935083118 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a53bc352c0 con 0x55a53bd05020 2024-04-21T23:15:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.309+0000 7f4e39d9b640 1 --2- 172.21.15.44:0/1935083118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd05940 0x55a53bd0dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.310+0000 7f4e4259c640 1 --2- 172.21.15.44:0/1935083118 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd05020 0x55a53bd05400 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4e340ccf30 tx=0x7f4e3409bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=226fb1f59c5ab816 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.310+0000 7f4e38d99640 1 -- 172.21.15.44:0/1935083118 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e340cd1b0 con 0x55a53bd05020 2024-04-21T23:15:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.310+0000 7f4e38d99640 1 -- 172.21.15.44:0/1935083118 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e34098420 con 0x55a53bd05020 2024-04-21T23:15:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.310+0000 7f4e38d99640 1 -- 172.21.15.44:0/1935083118 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e34018d60 con 0x55a53bd05020 2024-04-21T23:15:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.310+0000 7f4e4ba08740 1 -- 172.21.15.44:0/1935083118 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd05020 msgr2=0x55a53bd05400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.310+0000 7f4e4ba08740 1 --2- 172.21.15.44:0/1935083118 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd05020 0x55a53bd05400 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4e340ccf30 tx=0x7f4e3409bdb0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.311+0000 7f4e4ba08740 1 -- 172.21.15.44:0/1935083118 shutdown_connections 2024-04-21T23:15:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.311+0000 7f4e4ba08740 1 --2- 172.21.15.44:0/1935083118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd0e520 0x55a53bd10910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.311+0000 7f4e4ba08740 1 --2- 172.21.15.44:0/1935083118 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd05940 0x55a53bd0dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.311+0000 7f4e4ba08740 1 --2- 172.21.15.44:0/1935083118 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd05020 0x55a53bd05400 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.311+0000 7f4e4ba08740 1 -- 172.21.15.44:0/1935083118 >> 172.21.15.44:0/1935083118 conn(0x55a53bcff750 msgr2=0x55a53bcffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.311+0000 7f4e4ba08740 1 -- 172.21.15.44:0/1935083118 shutdown_connections 2024-04-21T23:15:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.311+0000 7f4e4ba08740 1 -- 172.21.15.44:0/1935083118 wait complete. 2024-04-21T23:15:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.311+0000 7f4e4ba08740 1 -- start start 2024-04-21T23:15:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.311+0000 7f4e4ba08740 1 Processor -- start 2024-04-21T23:15:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.311+0000 7f4e4ba08740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd05020 0x55a53bd0df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.311+0000 7f4e4ba08740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd05940 0x55a53bd0b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.311+0000 7f4e4ba08740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd0e520 0x55a53bd0ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.311+0000 7f4e4ba08740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55a53bd112b0 con 0x55a53bd0e520 2024-04-21T23:15:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.311+0000 7f4e4ba08740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55a53bc24f10 con 0x55a53bd05020 2024-04-21T23:15:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.311+0000 7f4e4ba08740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a53bd11430 con 0x55a53bd05940 2024-04-21T23:15:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.311+0000 7f4e42d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd0e520 0x55a53bd0ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.311+0000 7f4e4ba08740 10 client.0 _ll_register_callbacks cb 0x55a53b97cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.312+0000 7f4e42d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd0e520 0x55a53bd0ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53978/0 (socket says 172.21.15.44:53978) 2024-04-21T23:15:21.317 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.312+0000 7f4e42d9d640 1 -- 172.21.15.44:0/2828080432 learned_addr learned my addr 172.21.15.44:0/2828080432 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:21.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.312+0000 7f4e39d9b640 1 --2- 172.21.15.44:0/2828080432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd05940 0x55a53bd0b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.312+0000 7f4e4259c640 1 --2- 172.21.15.44:0/2828080432 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd05020 0x55a53bd0df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.312+0000 7f4e42d9d640 1 -- 172.21.15.44:0/2828080432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd05940 msgr2=0x55a53bd0b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:21.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.312+0000 7f4e42d9d640 1 --2- 172.21.15.44:0/2828080432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd05940 0x55a53bd0b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.312+0000 7f4e42d9d640 1 -- 172.21.15.44:0/2828080432 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd05020 msgr2=0x55a53bd0df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:21.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.312+0000 7f4e42d9d640 1 --2- 172.21.15.44:0/2828080432 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd05020 0x55a53bd0df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.312+0000 7f4e42d9d640 1 -- 172.21.15.44:0/2828080432 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a53bbf6c80 con 0x55a53bd0e520 2024-04-21T23:15:21.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.312+0000 7f4e39d9b640 1 --2- 172.21.15.44:0/2828080432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd05940 0x55a53bd0b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:21.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.312+0000 7f4e42d9d640 1 --2- 172.21.15.44:0/2828080432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd0e520 0x55a53bd0ba20 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c037040 tx=0x7f4e3c02c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f1ac8b77274af92 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:21.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.312+0000 7f4e29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:15:21.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.312+0000 7f4e29ffb640 1 -- 172.21.15.44:0/2828080432 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c050070 con 0x55a53bd0e520 2024-04-21T23:15:21.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.312+0000 7f4e29ffb640 1 -- 172.21.15.44:0/2828080432 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f4e3c032870 con 0x55a53bd0e520 2024-04-21T23:15:21.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.312+0000 7f4e4ba08740 1 -- 172.21.15.44:0/2828080432 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e34098420 con 0x55a53bd0e520 2024-04-21T23:15:21.318 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.312+0000 7f4e29ffb640 1 -- 172.21.15.44:0/2828080432 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4e3c05c050 con 0x55a53bd0e520 2024-04-21T23:15:21.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.312+0000 7f4e4ba08740 10 client.17118 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:21.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.312+0000 7f4e4ba08740 1 -- 172.21.15.44:0/2828080432 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a53b97d3f0 con 0x55a53bd0e520 2024-04-21T23:15:21.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.313+0000 7f4e29ffb640 1 -- 172.21.15.44:0/2828080432 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e34098420 con 0x55a53bd0e520 2024-04-21T23:15:21.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.313+0000 7f4e29ffb640 1 -- 172.21.15.44:0/2828080432 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f4e3c034bb0 con 0x55a53bd0e520 2024-04-21T23:15:21.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.313+0000 7f4e4ba08740 10 client.17118 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:21.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.313+0000 7f4e4ba08740 10 client.17118 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:21.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.313+0000 7f4e4ba08740 20 client.17118 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:21.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.313+0000 7f4e4ba08740 1 --2- 172.21.15.44:0/2828080432 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a53bd1e7e0 0x55a53bd20bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.313+0000 7f4e4ba08740 4 client.17118 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:15:21.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.313+0000 7f4e4ba08740 1 -- 172.21.15.44:0/2828080432 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a53bc76a70 con 0x55a53bd1e7e0 2024-04-21T23:15:21.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.313+0000 7f4e4259c640 1 --2- 172.21.15.44:0/2828080432 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a53bd1e7e0 0x55a53bd20bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.313+0000 7f4e4259c640 1 --2- 172.21.15.44:0/2828080432 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a53bd1e7e0 0x55a53bd20bd0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee203892d24759e3 server_cookie=814a46546177a071 in_seq=0 out_seq=0 2024-04-21T23:15:21.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.313+0000 7f4e29ffb640 10 client.17118 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:21.319 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.314+0000 7f4e29ffb640 1 -- 172.21.15.44:0/2828080432 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a53bc76a70 con 0x55a53bd1e7e0 2024-04-21T23:15:21.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.314+0000 7f4e29ffb640 10 client.17118 handle_command_reply: tid=0 2024-04-21T23:15:21.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.314+0000 7f4e4ba08740 1 client.17118 shutdown 2024-04-21T23:15:21.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.314+0000 7f4e4ba08740 1 -- 172.21.15.44:0/2828080432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd0e520 msgr2=0x55a53bd0ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:21.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.314+0000 7f4e4ba08740 1 --2- 172.21.15.44:0/2828080432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd0e520 0x55a53bd0ba20 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c037040 tx=0x7f4e3c02c180 comp rx=0 tx=0).stop 2024-04-21T23:15:21.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.314+0000 7f4e4ba08740 1 -- 172.21.15.44:0/2828080432 shutdown_connections 2024-04-21T23:15:21.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.314+0000 7f4e4ba08740 1 --2- 172.21.15.44:0/2828080432 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a53bd1e7e0 0x55a53bd20bd0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.314+0000 7f4e4ba08740 1 --2- 172.21.15.44:0/2828080432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd05940 0x55a53bd0b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.314+0000 7f4e29ffb640 0 client.17118 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:21.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.314+0000 7f4e4ba08740 1 --2- 172.21.15.44:0/2828080432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd0e520 0x55a53bd0ba20 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.314+0000 7f4e4ba08740 1 --2- 172.21.15.44:0/2828080432 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd05020 0x55a53bd0df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.314+0000 7f4e4ba08740 1 -- 172.21.15.44:0/2828080432 >> 172.21.15.44:0/2828080432 conn(0x55a53bcff750 msgr2=0x55a53bd014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:21.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.314+0000 7f4e4ba08740 1 -- 172.21.15.44:0/2828080432 shutdown_connections 2024-04-21T23:15:21.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.314+0000 7f4e4ba08740 1 -- 172.21.15.44:0/2828080432 wait complete. 2024-04-21T23:15:21.320 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.315+0000 7f4e4ba08740 20 client.17118 trim_cache size 0 max 16384 2024-04-21T23:15:21.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.331+0000 7f4e4ba08740 1 Processor -- start 2024-04-21T23:15:21.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.331+0000 7f4e4ba08740 1 -- start start 2024-04-21T23:15:21.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.331+0000 7f4e4ba08740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd05020 0x55a53bcfe6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.331+0000 7f4e4ba08740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd05940 0x55a53bd26290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.331+0000 7f4e4ba08740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd0e520 0x55a53bd267d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.331+0000 7f4e4ba08740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4e441aef20 con 0x55a53bd05940 2024-04-21T23:15:21.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.331+0000 7f4e4ba08740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4e441b1780 con 0x55a53bd0e520 2024-04-21T23:15:21.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.331+0000 7f4e4ba08740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4e441af570 con 0x55a53bd05020 2024-04-21T23:15:21.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.331+0000 7f4e39d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd05940 0x55a53bd26290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.331+0000 7f4e39d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd05940 0x55a53bd26290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53980/0 (socket says 172.21.15.44:53980) 2024-04-21T23:15:21.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.331+0000 7f4e39d9b640 1 -- 172.21.15.44:0/1807040676 learned_addr learned my addr 172.21.15.44:0/1807040676 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:21.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.331+0000 7f4e42d9d640 1 --2- 172.21.15.44:0/1807040676 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd0e520 0x55a53bd267d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.331+0000 7f4e4259c640 1 --2- 172.21.15.44:0/1807040676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd05020 0x55a53bcfe6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.331+0000 7f4e42d9d640 1 -- 172.21.15.44:0/1807040676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd05020 msgr2=0x55a53bcfe6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:21.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.331+0000 7f4e42d9d640 1 --2- 172.21.15.44:0/1807040676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd05020 0x55a53bcfe6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.331+0000 7f4e42d9d640 1 -- 172.21.15.44:0/1807040676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd05940 msgr2=0x55a53bd26290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:21.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.331+0000 7f4e42d9d640 1 --2- 172.21.15.44:0/1807040676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd05940 0x55a53bd26290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.331+0000 7f4e42d9d640 1 -- 172.21.15.44:0/1807040676 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a53ba477d0 con 0x55a53bd0e520 2024-04-21T23:15:21.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.332+0000 7f4e42d9d640 1 --2- 172.21.15.44:0/1807040676 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd0e520 0x55a53bd267d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c02c6c0 tx=0x7f4e3c003f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfe2a29870bba73 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:21.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.332+0000 7f4e2bfff640 1 -- 172.21.15.44:0/1807040676 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c050070 con 0x55a53bd0e520 2024-04-21T23:15:21.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.332+0000 7f4e2bfff640 1 -- 172.21.15.44:0/1807040676 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e3c044030 con 0x55a53bd0e520 2024-04-21T23:15:21.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.332+0000 7f4e2bfff640 1 -- 172.21.15.44:0/1807040676 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c039040 con 0x55a53bd0e520 2024-04-21T23:15:21.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4ba08740 1 -- 172.21.15.44:0/1807040676 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd0e520 msgr2=0x55a53bd267d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:21.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4ba08740 1 --2- 172.21.15.44:0/1807040676 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd0e520 0x55a53bd267d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c02c6c0 tx=0x7f4e3c003f50 comp rx=0 tx=0).stop 2024-04-21T23:15:21.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4ba08740 1 -- 172.21.15.44:0/1807040676 shutdown_connections 2024-04-21T23:15:21.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4ba08740 1 --2- 172.21.15.44:0/1807040676 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd0e520 0x55a53bd267d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4ba08740 1 --2- 172.21.15.44:0/1807040676 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd05940 0x55a53bd26290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4ba08740 1 --2- 172.21.15.44:0/1807040676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd05020 0x55a53bcfe6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4ba08740 1 -- 172.21.15.44:0/1807040676 >> 172.21.15.44:0/1807040676 conn(0x55a53bcff750 msgr2=0x55a53bcfdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:21.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4ba08740 1 -- 172.21.15.44:0/1807040676 shutdown_connections 2024-04-21T23:15:21.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4ba08740 1 -- 172.21.15.44:0/1807040676 wait complete. 2024-04-21T23:15:21.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4ba08740 1 -- start start 2024-04-21T23:15:21.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4ba08740 1 Processor -- start 2024-04-21T23:15:21.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4ba08740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd05020 0x55a53bd120c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4ba08740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd05940 0x55a53bd12600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4ba08740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd0e520 0x55a53bd12b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4ba08740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4e441afc40 con 0x55a53bd05020 2024-04-21T23:15:21.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4ba08740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4e441b10b0 con 0x55a53bd0e520 2024-04-21T23:15:21.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4ba08740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4e441b0310 con 0x55a53bd05940 2024-04-21T23:15:21.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4ba08740 10 client.0 _ll_register_callbacks cb 0x7f4e34005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:21.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd05020 0x55a53bd120c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd05020 0x55a53bd120c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:53990/0 (socket says 172.21.15.44:53990) 2024-04-21T23:15:21.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e4259c640 1 -- 172.21.15.44:0/1974124578 learned_addr learned my addr 172.21.15.44:0/1974124578 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:21.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e39d9b640 1 --2- 172.21.15.44:0/1974124578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd05940 0x55a53bd12600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.333+0000 7f4e42d9d640 1 --2- 172.21.15.44:0/1974124578 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd0e520 0x55a53bd12b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.334+0000 7f4e4259c640 1 -- 172.21.15.44:0/1974124578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd05940 msgr2=0x55a53bd12600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:21.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.334+0000 7f4e4259c640 1 --2- 172.21.15.44:0/1974124578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd05940 0x55a53bd12600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.334+0000 7f4e4259c640 1 -- 172.21.15.44:0/1974124578 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd0e520 msgr2=0x55a53bd12b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:21.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.334+0000 7f4e4259c640 1 --2- 172.21.15.44:0/1974124578 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd0e520 0x55a53bd12b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.334+0000 7f4e4259c640 1 -- 172.21.15.44:0/1974124578 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a53bd83b30 con 0x55a53bd05020 2024-04-21T23:15:21.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.334+0000 7f4e39d9b640 1 --2- 172.21.15.44:0/1974124578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd05940 0x55a53bd12600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:21.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.334+0000 7f4e42d9d640 1 --2- 172.21.15.44:0/1974124578 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd0e520 0x55a53bd12b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:21.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.334+0000 7f4e4259c640 1 --2- 172.21.15.44:0/1974124578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd05020 0x55a53bd120c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400e9c0 tx=0x7f4e34098180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92565e847f578173 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:21.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.334+0000 7f4e29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:15:21.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.334+0000 7f4e29ffb640 1 -- 172.21.15.44:0/1974124578 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3407abd0 con 0x55a53bd05020 2024-04-21T23:15:21.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.334+0000 7f4e29ffb640 1 -- 172.21.15.44:0/1974124578 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f4e3407ad70 con 0x55a53bd05020 2024-04-21T23:15:21.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.334+0000 7f4e4ba08740 1 -- 172.21.15.44:0/1974124578 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e3c044030 con 0x55a53bd05020 2024-04-21T23:15:21.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.334+0000 7f4e29ffb640 1 -- 172.21.15.44:0/1974124578 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4e34082050 con 0x55a53bd05020 2024-04-21T23:15:21.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.334+0000 7f4e4ba08740 10 client.17124 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:21.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.334+0000 7f4e4ba08740 1 -- 172.21.15.44:0/1974124578 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a53bd84e10 con 0x55a53bd05020 2024-04-21T23:15:21.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.334+0000 7f4e29ffb640 1 -- 172.21.15.44:0/1974124578 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e3c044030 con 0x55a53bd05020 2024-04-21T23:15:21.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.334+0000 7f4e29ffb640 1 -- 172.21.15.44:0/1974124578 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f4e34086050 con 0x55a53bd05020 2024-04-21T23:15:21.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.335+0000 7f4e4ba08740 10 client.17124 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:21.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.335+0000 7f4e4ba08740 10 client.17124 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:21.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.335+0000 7f4e4ba08740 20 client.17124 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:21.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.335+0000 7f4e4ba08740 1 --2- 172.21.15.44:0/1974124578 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a53bd85150 0x55a53bd87540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:21.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.335+0000 7f4e4ba08740 4 client.17124 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:15:21.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.335+0000 7f4e4ba08740 1 -- 172.21.15.44:0/1974124578 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a53bc70dc0 con 0x55a53bd85150 2024-04-21T23:15:21.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.335+0000 7f4e39d9b640 1 --2- 172.21.15.44:0/1974124578 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a53bd85150 0x55a53bd87540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:21.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.335+0000 7f4e29ffb640 10 client.17124 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:21.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.335+0000 7f4e39d9b640 1 --2- 172.21.15.44:0/1974124578 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a53bd85150 0x55a53bd87540 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fe68f4373f5b8db server_cookie=b97625b01172c04d in_seq=0 out_seq=0 2024-04-21T23:15:21.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.336+0000 7f4e29ffb640 1 -- 172.21.15.44:0/1974124578 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a53bc70dc0 con 0x55a53bd85150 2024-04-21T23:15:21.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.336+0000 7f4e29ffb640 10 client.17124 handle_command_reply: tid=0 2024-04-21T23:15:21.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.336+0000 7f4e4ba08740 1 client.17124 shutdown 2024-04-21T23:15:21.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.336+0000 7f4e4ba08740 1 -- 172.21.15.44:0/1974124578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd05020 msgr2=0x55a53bd120c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:21.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.336+0000 7f4e4ba08740 1 --2- 172.21.15.44:0/1974124578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd05020 0x55a53bd120c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400e9c0 tx=0x7f4e34098180 comp rx=0 tx=0).stop 2024-04-21T23:15:21.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.336+0000 7f4e4ba08740 1 -- 172.21.15.44:0/1974124578 shutdown_connections 2024-04-21T23:15:21.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.336+0000 7f4e4ba08740 1 --2- 172.21.15.44:0/1974124578 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55a53bd85150 0x55a53bd87540 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.336+0000 7f4e4ba08740 1 --2- 172.21.15.44:0/1974124578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a53bd05940 0x55a53bd12600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.336+0000 7f4e4ba08740 1 --2- 172.21.15.44:0/1974124578 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55a53bd0e520 0x55a53bd12b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.336+0000 7f4e4ba08740 1 --2- 172.21.15.44:0/1974124578 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55a53bd05020 0x55a53bd120c0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.336+0000 7f4e4ba08740 1 -- 172.21.15.44:0/1974124578 >> 172.21.15.44:0/1974124578 conn(0x55a53bcff750 msgr2=0x55a53bcfdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:21.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.336+0000 7f4e29ffb640 0 client.17124 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:21.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.336+0000 7f4e4ba08740 1 -- 172.21.15.44:0/1974124578 shutdown_connections 2024-04-21T23:15:21.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.336+0000 7f4e4ba08740 1 -- 172.21.15.44:0/1974124578 wait complete. 2024-04-21T23:15:21.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.336+0000 7f4e4ba08740 20 client.17124 trim_cache size 0 max 16384 2024-04-21T23:15:21.346 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-21T23:15:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.336+0000 7f4e48d3e640 1 -- 172.21.15.44:0/411413410 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4e300a2d00 msgr2=0x7f4e300a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.337+0000 7f4e48d3e640 1 --2- 172.21.15.44:0/411413410 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4e300a2d00 0x7f4e300a51c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c004af0 tx=0x7f4e3c005c50 comp rx=0 tx=0).stop 2024-04-21T23:15:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.337+0000 7f4e48d3e640 1 -- 172.21.15.44:0/411413410 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e441029b0 msgr2=0x7f4e44100600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.337+0000 7f4e48d3e640 1 --2- 172.21.15.44:0/411413410 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e441029b0 0x7f4e44100600 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400e9f0 tx=0x7f4e3400eeb0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.337+0000 7f4e48d3e640 1 -- 172.21.15.44:0/411413410 shutdown_connections 2024-04-21T23:15:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.337+0000 7f4e48d3e640 1 --2- 172.21.15.44:0/411413410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e4410bcf0 0x7f4e44101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.337+0000 7f4e48d3e640 1 --2- 172.21.15.44:0/411413410 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4e300a2d00 0x7f4e300a51c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.337+0000 7f4e48d3e640 1 --2- 172.21.15.44:0/411413410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4e441032d0 0x7f4e44100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.337+0000 7f4e48d3e640 1 --2- 172.21.15.44:0/411413410 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4e441029b0 0x7f4e44100600 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:21.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.337+0000 7f4e48d3e640 1 -- 172.21.15.44:0/411413410 >> 172.21.15.44:0/411413410 conn(0x7f4e440777f0 msgr2=0x7f4e440fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:21.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.337+0000 7f4e48d3e640 1 -- 172.21.15.44:0/411413410 shutdown_connections 2024-04-21T23:15:21.348 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:21.337+0000 7f4e48d3e640 1 -- 172.21.15.44:0/411413410 wait complete. 2024-04-21T23:15:21.704 INFO:tasks.workunit.client.0.smithi044.stdout: 1 767177 12.93 MB/sec execute 191 sec latency 37.398 ms 2024-04-21T23:15:22.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:21 smithi044 ceph-mon[27934]: pgmap v204: 129 pgs: 129 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 302 op/s 2024-04-21T23:15:22.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:21 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2771448195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:22.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:21 smithi069 ceph-mon[31816]: pgmap v204: 129 pgs: 129 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 302 op/s 2024-04-21T23:15:22.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:21 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2771448195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:22.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:21 smithi055 ceph-mon[31117]: pgmap v204: 129 pgs: 129 active+clean; 72 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 302 op/s 2024-04-21T23:15:22.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:21 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2771448195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:22.349 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:15:22.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:15:22.350 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:15:22.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.468+0000 7f5daa65f640 1 -- 172.21.15.44:0/3816558072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5da41020c0 msgr2=0x7f5da410e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:22.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.468+0000 7f5daa65f640 1 --2- 172.21.15.44:0/3816558072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5da41020c0 0x7f5da410e5d0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c009cd0 tx=0x7f5d9c02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.468+0000 7f5daa65f640 1 -- 172.21.15.44:0/3816558072 shutdown_connections 2024-04-21T23:15:22.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.468+0000 7f5daa65f640 1 --2- 172.21.15.44:0/3816558072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5da4101700 0x7f5da4101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.470 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.468+0000 7f5daa65f640 1 --2- 172.21.15.44:0/3816558072 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5da41020c0 0x7f5da410e5d0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.468+0000 7f5daa65f640 1 --2- 172.21.15.44:0/3816558072 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5da4107710 0x7f5da4107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.468+0000 7f5daa65f640 1 -- 172.21.15.44:0/3816558072 >> 172.21.15.44:0/3816558072 conn(0x7f5da40fd1f0 msgr2=0x7f5da40ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.468+0000 7f5daa65f640 1 -- 172.21.15.44:0/3816558072 shutdown_connections 2024-04-21T23:15:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.468+0000 7f5daa65f640 1 -- 172.21.15.44:0/3816558072 wait complete. 2024-04-21T23:15:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.468+0000 7f5daa65f640 1 Processor -- start 2024-04-21T23:15:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.468+0000 7f5daa65f640 1 -- start start 2024-04-21T23:15:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.469+0000 7f5daa65f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5da4101700 0x7f5da4073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.469+0000 7f5daa65f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5da41020c0 0x7f5da4073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.469+0000 7f5daa65f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5da4107710 0x7f5da40744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.469+0000 7f5daa65f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5da4111b20 con 0x7f5da4101700 2024-04-21T23:15:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.469+0000 7f5daa65f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5da41119a0 con 0x7f5da41020c0 2024-04-21T23:15:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.469+0000 7f5daa65f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5da4111ca0 con 0x7f5da4107710 2024-04-21T23:15:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.469+0000 7f5da3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5da4101700 0x7f5da4073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.471 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.469+0000 7f5da3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5da4101700 0x7f5da4073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54000/0 (socket says 172.21.15.44:54000) 2024-04-21T23:15:22.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.469+0000 7f5da3fff640 1 -- 172.21.15.44:0/2916188913 learned_addr learned my addr 172.21.15.44:0/2916188913 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:22.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.469+0000 7f5da37fe640 1 --2- 172.21.15.44:0/2916188913 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5da41020c0 0x7f5da4073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.469+0000 7f5da8bfc640 1 --2- 172.21.15.44:0/2916188913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5da4107710 0x7f5da40744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.469+0000 7f5da3fff640 1 -- 172.21.15.44:0/2916188913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5da4107710 msgr2=0x7f5da40744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:22.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.469+0000 7f5da3fff640 1 --2- 172.21.15.44:0/2916188913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5da4107710 0x7f5da40744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.469+0000 7f5da3fff640 1 -- 172.21.15.44:0/2916188913 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5da41020c0 msgr2=0x7f5da4073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:22.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.469+0000 7f5da3fff640 1 --2- 172.21.15.44:0/2916188913 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5da41020c0 0x7f5da4073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.469+0000 7f5da3fff640 1 -- 172.21.15.44:0/2916188913 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5da41a64e0 con 0x7f5da4101700 2024-04-21T23:15:22.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.469+0000 7f5da8bfc640 1 --2- 172.21.15.44:0/2916188913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5da4107710 0x7f5da40744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:15:22.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.469+0000 7f5da3fff640 1 --2- 172.21.15.44:0/2916188913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5da4101700 0x7f5da4073a40 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f5d9400e9f0 tx=0x7f5d9400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:22.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.470+0000 7f5da17fa640 1 -- 172.21.15.44:0/2916188913 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9400cd80 con 0x7f5da4101700 2024-04-21T23:15:22.472 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.470+0000 7f5da17fa640 1 -- 172.21.15.44:0/2916188913 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d94010910 con 0x7f5da4101700 2024-04-21T23:15:22.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.470+0000 7f5da17fa640 1 -- 172.21.15.44:0/2916188913 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d94018990 con 0x7f5da4101700 2024-04-21T23:15:22.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.470+0000 7f5daa65f640 1 -- 172.21.15.44:0/2916188913 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5da41a67d0 con 0x7f5da4101700 2024-04-21T23:15:22.473 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.470+0000 7f5daa65f640 1 -- 172.21.15.44:0/2916188913 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5da41a6cd0 con 0x7f5da4101700 2024-04-21T23:15:22.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.472+0000 7f5da17fa640 1 -- 172.21.15.44:0/2916188913 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f5d94010ac0 con 0x7f5da4101700 2024-04-21T23:15:22.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.473+0000 7f5dad240740 1 Processor -- start 2024-04-21T23:15:22.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.473+0000 7f5dad240740 1 -- start start 2024-04-21T23:15:22.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.473+0000 7f5dad240740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d213a720 0x55b7d2139470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.479 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.473+0000 7f5dad240740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2139a40 0x55b7d2139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.473+0000 7f5dad240740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d2142db0 0x55b7d21451a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.473+0000 7f5dad240740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b7d209d170 con 0x55b7d2139a40 2024-04-21T23:15:22.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.473+0000 7f5dad240740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55b7d20457c0 con 0x55b7d213a720 2024-04-21T23:15:22.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.473+0000 7f5dad240740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b7d2099130 con 0x55b7d2142db0 2024-04-21T23:15:22.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.473+0000 7f5da8bfc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d213a720 0x55b7d2139470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.473+0000 7f5da3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2139a40 0x55b7d2139ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.473+0000 7f5da8bfc640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d213a720 0x55b7d2139470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:58816/0 (socket says 172.21.15.44:58816) 2024-04-21T23:15:22.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.473+0000 7f5da8bfc640 1 -- 172.21.15.44:0/2947870688 learned_addr learned my addr 172.21.15.44:0/2947870688 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:22.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.473+0000 7f5da37fe640 1 --2- 172.21.15.44:0/2947870688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d2142db0 0x55b7d21451a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.473+0000 7f5da17fa640 1 --2- 172.21.15.44:0/2916188913 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5d8c0a2d00 0x7f5d8c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.473+0000 7f5da8bfc640 1 -- 172.21.15.44:0/2947870688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d2142db0 msgr2=0x55b7d21451a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:22.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.473+0000 7f5da8bfc640 1 --2- 172.21.15.44:0/2947870688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d2142db0 0x55b7d21451a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.473+0000 7f5da8bfc640 1 -- 172.21.15.44:0/2947870688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2139a40 msgr2=0x55b7d2139ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:22.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.473+0000 7f5da8bfc640 1 --2- 172.21.15.44:0/2947870688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2139a40 0x55b7d2139ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.480 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.473+0000 7f5da8bfc640 1 -- 172.21.15.44:0/2947870688 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7d2056e60 con 0x55b7d213a720 2024-04-21T23:15:22.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.474+0000 7f5da17fa640 1 -- 172.21.15.44:0/2916188913 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f5d940cfb80 con 0x7f5da4101700 2024-04-21T23:15:22.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.474+0000 7f5da8bfc640 1 --2- 172.21.15.44:0/2916188913 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5d8c0a2d00 0x7f5d8c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.474+0000 7f5da8bfc640 1 --2- 172.21.15.44:0/2947870688 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d213a720 0x55b7d2139470 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c009430 tx=0x7f5d9c009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57ac6e7a1461bdcb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:22.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.474+0000 7f5d8a7fc640 1 -- 172.21.15.44:0/2947870688 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c039050 con 0x55b7d213a720 2024-04-21T23:15:22.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.474+0000 7f5da8bfc640 1 --2- 172.21.15.44:0/2916188913 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5d8c0a2d00 0x7f5d8c0a51c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c006010 tx=0x7f5d9c04b000 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:22.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.474+0000 7f5d8a7fc640 1 -- 172.21.15.44:0/2947870688 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d9c046030 con 0x55b7d213a720 2024-04-21T23:15:22.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.474+0000 7f5d8a7fc640 1 -- 172.21.15.44:0/2947870688 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c042670 con 0x55b7d213a720 2024-04-21T23:15:22.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.475+0000 7f5dad240740 1 -- 172.21.15.44:0/2947870688 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d213a720 msgr2=0x55b7d2139470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:22.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.475+0000 7f5dad240740 1 --2- 172.21.15.44:0/2947870688 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d213a720 0x55b7d2139470 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c009430 tx=0x7f5d9c009800 comp rx=0 tx=0).stop 2024-04-21T23:15:22.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.475+0000 7f5dad240740 1 -- 172.21.15.44:0/2947870688 shutdown_connections 2024-04-21T23:15:22.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.475+0000 7f5dad240740 1 --2- 172.21.15.44:0/2947870688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d2142db0 0x55b7d21451a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.475+0000 7f5dad240740 1 --2- 172.21.15.44:0/2947870688 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2139a40 0x55b7d2139ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.475+0000 7f5dad240740 1 --2- 172.21.15.44:0/2947870688 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d213a720 0x55b7d2139470 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.475+0000 7f5dad240740 1 -- 172.21.15.44:0/2947870688 >> 172.21.15.44:0/2947870688 conn(0x55b7d21004f0 msgr2=0x55b7d2134490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:22.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.475+0000 7f5dad240740 1 -- 172.21.15.44:0/2947870688 shutdown_connections 2024-04-21T23:15:22.481 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.475+0000 7f5dad240740 1 -- 172.21.15.44:0/2947870688 wait complete. 2024-04-21T23:15:22.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.475+0000 7f5dad240740 1 -- start start 2024-04-21T23:15:22.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.475+0000 7f5dad240740 1 Processor -- start 2024-04-21T23:15:22.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.475+0000 7f5dad240740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d2139a40 0x55b7d213fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5dad240740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d213a720 0x55b7d2140360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5dad240740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2142db0 0x55b7d21408a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5dad240740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b7d2145c20 con 0x55b7d2142db0 2024-04-21T23:15:22.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5dad240740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55b7d2058d40 con 0x55b7d213a720 2024-04-21T23:15:22.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5dad240740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b7d2145da0 con 0x55b7d2139a40 2024-04-21T23:15:22.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5dad240740 10 client.0 _ll_register_callbacks cb 0x55b7d2094820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:22.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5da37fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d213a720 0x55b7d2140360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5da8bfc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2142db0 0x55b7d21408a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5da37fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d213a720 0x55b7d2140360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:58828/0 (socket says 172.21.15.44:58828) 2024-04-21T23:15:22.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5da8bfc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2142db0 0x55b7d21408a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54024/0 (socket says 172.21.15.44:54024) 2024-04-21T23:15:22.482 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5da37fe640 1 -- 172.21.15.44:0/1564910650 learned_addr learned my addr 172.21.15.44:0/1564910650 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:22.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5da3fff640 1 --2- 172.21.15.44:0/1564910650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d2139a40 0x55b7d213fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5da8bfc640 1 -- 172.21.15.44:0/1564910650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d2139a40 msgr2=0x55b7d213fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:22.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5da8bfc640 1 --2- 172.21.15.44:0/1564910650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d2139a40 0x55b7d213fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5da8bfc640 1 -- 172.21.15.44:0/1564910650 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d213a720 msgr2=0x55b7d2140360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:22.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5da8bfc640 1 --2- 172.21.15.44:0/1564910650 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d213a720 0x55b7d2140360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5da8bfc640 1 -- 172.21.15.44:0/1564910650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7d1e797e0 con 0x55b7d2142db0 2024-04-21T23:15:22.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5da8bfc640 1 --2- 172.21.15.44:0/1564910650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2142db0 0x55b7d21408a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c009cd0 tx=0x7f5d9c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee652f8ab0a7cdc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:22.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5d837fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:15:22.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5d837fe640 1 -- 172.21.15.44:0/1564910650 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c05a060 con 0x55b7d2142db0 2024-04-21T23:15:22.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5d837fe640 1 -- 172.21.15.44:0/1564910650 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f5d9c003c40 con 0x55b7d2142db0 2024-04-21T23:15:22.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.476+0000 7f5dad240740 1 -- 172.21.15.44:0/1564910650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d9c046030 con 0x55b7d2142db0 2024-04-21T23:15:22.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.477+0000 7f5d837fe640 1 -- 172.21.15.44:0/1564910650 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5d9c061050 con 0x55b7d2142db0 2024-04-21T23:15:22.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.477+0000 7f5dad240740 10 client.17136 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:22.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.477+0000 7f5dad240740 1 -- 172.21.15.44:0/1564910650 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b7d1ed9f70 con 0x55b7d2142db0 2024-04-21T23:15:22.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.477+0000 7f5d837fe640 1 -- 172.21.15.44:0/1564910650 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d9c046030 con 0x55b7d2142db0 2024-04-21T23:15:22.483 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.477+0000 7f5d837fe640 1 -- 172.21.15.44:0/1564910650 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f5d9c034910 con 0x55b7d2142db0 2024-04-21T23:15:22.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.477+0000 7f5dad240740 10 client.17136 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:22.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.477+0000 7f5dad240740 10 client.17136 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:22.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.477+0000 7f5dad240740 20 client.17136 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:22.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.477+0000 7f5dad240740 1 --2- 172.21.15.44:0/1564910650 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b7d2153100 0x55b7d2155520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.477+0000 7f5dad240740 4 client.17136 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:15:22.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.477+0000 7f5dad240740 1 -- 172.21.15.44:0/1564910650 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7d2071450 con 0x55b7d2153100 2024-04-21T23:15:22.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.477+0000 7f5da3fff640 1 --2- 172.21.15.44:0/1564910650 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b7d2153100 0x55b7d2155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.478+0000 7f5d837fe640 10 client.17136 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:22.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.478+0000 7f5da3fff640 1 --2- 172.21.15.44:0/1564910650 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b7d2153100 0x55b7d2155520 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c779a4d4e8a9e88 server_cookie=baafa63258d4aec8 in_seq=0 out_seq=0 2024-04-21T23:15:22.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.478+0000 7f5d837fe640 1 -- 172.21.15.44:0/1564910650 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b7d2071450 con 0x55b7d2153100 2024-04-21T23:15:22.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.478+0000 7f5d837fe640 10 client.17136 handle_command_reply: tid=0 2024-04-21T23:15:22.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.478+0000 7f5dad240740 1 client.17136 shutdown 2024-04-21T23:15:22.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.479+0000 7f5dad240740 1 -- 172.21.15.44:0/1564910650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2142db0 msgr2=0x55b7d21408a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:22.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.479+0000 7f5dad240740 1 --2- 172.21.15.44:0/1564910650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2142db0 0x55b7d21408a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c009cd0 tx=0x7f5d9c0095c0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.479+0000 7f5dad240740 1 -- 172.21.15.44:0/1564910650 shutdown_connections 2024-04-21T23:15:22.484 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.479+0000 7f5dad240740 1 --2- 172.21.15.44:0/1564910650 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b7d2153100 0x55b7d2155520 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.479+0000 7f5dad240740 1 --2- 172.21.15.44:0/1564910650 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2142db0 0x55b7d21408a0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.479+0000 7f5dad240740 1 --2- 172.21.15.44:0/1564910650 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d213a720 0x55b7d2140360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.479+0000 7f5dad240740 1 --2- 172.21.15.44:0/1564910650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d2139a40 0x55b7d213fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.479+0000 7f5dad240740 1 -- 172.21.15.44:0/1564910650 >> 172.21.15.44:0/1564910650 conn(0x55b7d21004f0 msgr2=0x55b7d21361e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:22.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.479+0000 7f5d837fe640 0 client.17136 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:22.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.479+0000 7f5dad240740 1 -- 172.21.15.44:0/1564910650 shutdown_connections 2024-04-21T23:15:22.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.479+0000 7f5dad240740 1 -- 172.21.15.44:0/1564910650 wait complete. 2024-04-21T23:15:22.485 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.479+0000 7f5dad240740 20 client.17136 trim_cache size 0 max 16384 2024-04-21T23:15:22.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.501+0000 7f5dad240740 1 Processor -- start 2024-04-21T23:15:22.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.501+0000 7f5dad240740 1 -- start start 2024-04-21T23:15:22.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.502+0000 7f5dad240740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2139a40 0x55b7d2142550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.502+0000 7f5dad240740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d213a720 0x55b7d21406c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.502+0000 7f5dad240740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d2142db0 0x55b7d2133370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.502+0000 7f5dad240740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5da41aef00 con 0x55b7d2139a40 2024-04-21T23:15:22.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.502+0000 7f5dad240740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5da41b1760 con 0x55b7d2142db0 2024-04-21T23:15:22.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.502+0000 7f5dad240740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5da41af550 con 0x55b7d213a720 2024-04-21T23:15:22.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.502+0000 7f5da3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2139a40 0x55b7d2142550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.502+0000 7f5da3fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2139a40 0x55b7d2142550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54034/0 (socket says 172.21.15.44:54034) 2024-04-21T23:15:22.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.502+0000 7f5da3fff640 1 -- 172.21.15.44:0/1262785525 learned_addr learned my addr 172.21.15.44:0/1262785525 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:22.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.502+0000 7f5da37fe640 1 --2- 172.21.15.44:0/1262785525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d213a720 0x55b7d21406c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.502+0000 7f5da8bfc640 1 --2- 172.21.15.44:0/1262785525 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d2142db0 0x55b7d2133370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.502+0000 7f5da3fff640 1 -- 172.21.15.44:0/1262785525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d213a720 msgr2=0x55b7d21406c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:22.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.502+0000 7f5da3fff640 1 --2- 172.21.15.44:0/1262785525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d213a720 0x55b7d21406c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.502+0000 7f5da3fff640 1 -- 172.21.15.44:0/1262785525 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d2142db0 msgr2=0x55b7d2133370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:22.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.502+0000 7f5da3fff640 1 --2- 172.21.15.44:0/1262785525 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d2142db0 0x55b7d2133370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.502+0000 7f5da3fff640 1 -- 172.21.15.44:0/1262785525 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7d1d71170 con 0x55b7d2139a40 2024-04-21T23:15:22.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.502+0000 7f5da37fe640 1 --2- 172.21.15.44:0/1262785525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d213a720 0x55b7d21406c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:22.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.502+0000 7f5da3fff640 1 --2- 172.21.15.44:0/1262785525 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2139a40 0x55b7d2142550 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f5d940cd4e0 tx=0x7f5d940d2090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e8a9fcc63e51a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:22.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.502+0000 7f5d89ffb640 1 -- 172.21.15.44:0/1262785525 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d940a4920 con 0x55b7d2139a40 2024-04-21T23:15:22.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.503+0000 7f5d89ffb640 1 -- 172.21.15.44:0/1262785525 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d940a0060 con 0x55b7d2139a40 2024-04-21T23:15:22.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.503+0000 7f5d89ffb640 1 -- 172.21.15.44:0/1262785525 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d940cd2f0 con 0x55b7d2139a40 2024-04-21T23:15:22.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.503+0000 7f5dad240740 1 -- 172.21.15.44:0/1262785525 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2139a40 msgr2=0x55b7d2142550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:22.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.503+0000 7f5dad240740 1 --2- 172.21.15.44:0/1262785525 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2139a40 0x55b7d2142550 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f5d940cd4e0 tx=0x7f5d940d2090 comp rx=0 tx=0).stop 2024-04-21T23:15:22.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.503+0000 7f5dad240740 1 -- 172.21.15.44:0/1262785525 shutdown_connections 2024-04-21T23:15:22.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.503+0000 7f5dad240740 1 --2- 172.21.15.44:0/1262785525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d213a720 0x55b7d21406c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.503+0000 7f5dad240740 1 --2- 172.21.15.44:0/1262785525 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d2142db0 0x55b7d2133370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.503+0000 7f5dad240740 1 --2- 172.21.15.44:0/1262785525 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2139a40 0x55b7d2142550 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.503+0000 7f5dad240740 1 -- 172.21.15.44:0/1262785525 >> 172.21.15.44:0/1262785525 conn(0x55b7d21004f0 msgr2=0x55b7d2132c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:22.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.503+0000 7f5dad240740 1 -- 172.21.15.44:0/1262785525 shutdown_connections 2024-04-21T23:15:22.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.503+0000 7f5dad240740 1 -- 172.21.15.44:0/1262785525 wait complete. 2024-04-21T23:15:22.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.504+0000 7f5dad240740 1 -- start start 2024-04-21T23:15:22.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.504+0000 7f5dad240740 1 Processor -- start 2024-04-21T23:15:22.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.504+0000 7f5dad240740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d2139a40 0x55b7d213db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.504+0000 7f5dad240740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d213a720 0x55b7d214b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.504+0000 7f5dad240740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2142db0 0x55b7d2143da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.504+0000 7f5dad240740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5da41afc20 con 0x55b7d2142db0 2024-04-21T23:15:22.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.504+0000 7f5dad240740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5da41b1090 con 0x55b7d2139a40 2024-04-21T23:15:22.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.504+0000 7f5dad240740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5da41b02f0 con 0x55b7d213a720 2024-04-21T23:15:22.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.504+0000 7f5dad240740 10 client.0 _ll_register_callbacks cb 0x7f5d94097f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:22.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.504+0000 7f5da8bfc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2142db0 0x55b7d2143da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.504+0000 7f5da3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d2139a40 0x55b7d213db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.504+0000 7f5da3fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d2139a40 0x55b7d213db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:58840/0 (socket says 172.21.15.44:58840) 2024-04-21T23:15:22.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.504+0000 7f5da3fff640 1 -- 172.21.15.44:0/411483498 learned_addr learned my addr 172.21.15.44:0/411483498 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:22.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.504+0000 7f5da8bfc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2142db0 0x55b7d2143da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54038/0 (socket says 172.21.15.44:54038) 2024-04-21T23:15:22.509 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.504+0000 7f5da37fe640 1 --2- 172.21.15.44:0/411483498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d213a720 0x55b7d214b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.505+0000 7f5da8bfc640 1 -- 172.21.15.44:0/411483498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d213a720 msgr2=0x55b7d214b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:22.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.505+0000 7f5da8bfc640 1 --2- 172.21.15.44:0/411483498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d213a720 0x55b7d214b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.505+0000 7f5da8bfc640 1 -- 172.21.15.44:0/411483498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d2139a40 msgr2=0x55b7d213db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:22.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.505+0000 7f5da8bfc640 1 --2- 172.21.15.44:0/411483498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d2139a40 0x55b7d213db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.505+0000 7f5da8bfc640 1 -- 172.21.15.44:0/411483498 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7d21442e0 con 0x55b7d2142db0 2024-04-21T23:15:22.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.505+0000 7f5da3fff640 1 --2- 172.21.15.44:0/411483498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d2139a40 0x55b7d213db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:22.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.505+0000 7f5da8bfc640 1 --2- 172.21.15.44:0/411483498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2142db0 0x55b7d2143da0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c02c6d0 tx=0x7f5d9c032b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75b997760054c50 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:22.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.505+0000 7f5d837fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:15:22.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.505+0000 7f5d837fe640 1 -- 172.21.15.44:0/411483498 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c039050 con 0x55b7d2142db0 2024-04-21T23:15:22.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.505+0000 7f5dad240740 1 -- 172.21.15.44:0/411483498 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d940a0060 con 0x55b7d2142db0 2024-04-21T23:15:22.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.505+0000 7f5d837fe640 1 -- 172.21.15.44:0/411483498 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f5d9c003c40 con 0x55b7d2142db0 2024-04-21T23:15:22.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.505+0000 7f5d837fe640 1 -- 172.21.15.44:0/411483498 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5d9c05a040 con 0x55b7d2142db0 2024-04-21T23:15:22.510 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.505+0000 7f5dad240740 10 client.17142 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:22.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.505+0000 7f5dad240740 1 -- 172.21.15.44:0/411483498 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b7d21b9cd0 con 0x55b7d2142db0 2024-04-21T23:15:22.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.505+0000 7f5d837fe640 1 -- 172.21.15.44:0/411483498 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d940a0060 con 0x55b7d2142db0 2024-04-21T23:15:22.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.505+0000 7f5d837fe640 1 -- 172.21.15.44:0/411483498 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f5d9c056470 con 0x55b7d2142db0 2024-04-21T23:15:22.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.505+0000 7f5dad240740 10 client.17142 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:22.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.505+0000 7f5dad240740 10 client.17142 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:22.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.506+0000 7f5dad240740 20 client.17142 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:22.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.506+0000 7f5dad240740 1 --2- 172.21.15.44:0/411483498 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b7d21ba330 0x55b7d21bc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.506+0000 7f5da3fff640 1 --2- 172.21.15.44:0/411483498 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b7d21ba330 0x55b7d21bc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.506+0000 7f5dad240740 4 client.17142 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:15:22.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.506+0000 7f5dad240740 1 -- 172.21.15.44:0/411483498 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b7d20aa6b0 con 0x55b7d21ba330 2024-04-21T23:15:22.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.506+0000 7f5d837fe640 10 client.17142 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:22.511 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.506+0000 7f5da3fff640 1 --2- 172.21.15.44:0/411483498 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b7d21ba330 0x55b7d21bc750 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35a0b989bf67fa33 server_cookie=4ae7b05ab6389d17 in_seq=0 out_seq=0 2024-04-21T23:15:22.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.511+0000 7f5d837fe640 1 -- 172.21.15.44:0/411483498 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b7d20aa6b0 con 0x55b7d21ba330 2024-04-21T23:15:22.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.511+0000 7f5d837fe640 10 client.17142 handle_command_reply: tid=0 2024-04-21T23:15:22.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.511+0000 7f5dad240740 1 client.17142 shutdown 2024-04-21T23:15:22.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.511+0000 7f5dad240740 1 -- 172.21.15.44:0/411483498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2142db0 msgr2=0x55b7d2143da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:22.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.511+0000 7f5dad240740 1 --2- 172.21.15.44:0/411483498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2142db0 0x55b7d2143da0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c02c6d0 tx=0x7f5d9c032b60 comp rx=0 tx=0).stop 2024-04-21T23:15:22.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.511+0000 7f5dad240740 1 -- 172.21.15.44:0/411483498 shutdown_connections 2024-04-21T23:15:22.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.511+0000 7f5dad240740 1 --2- 172.21.15.44:0/411483498 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b7d21ba330 0x55b7d21bc750 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.511+0000 7f5dad240740 1 --2- 172.21.15.44:0/411483498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7d213a720 0x55b7d214b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.511+0000 7f5dad240740 1 --2- 172.21.15.44:0/411483498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b7d2142db0 0x55b7d2143da0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.511+0000 7f5d837fe640 0 client.17142 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:22.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.511+0000 7f5dad240740 1 --2- 172.21.15.44:0/411483498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b7d2139a40 0x55b7d213db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.511+0000 7f5dad240740 1 -- 172.21.15.44:0/411483498 >> 172.21.15.44:0/411483498 conn(0x55b7d21004f0 msgr2=0x55b7d21334f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:22.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.511+0000 7f5dad240740 1 -- 172.21.15.44:0/411483498 shutdown_connections 2024-04-21T23:15:22.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.511+0000 7f5dad240740 1 -- 172.21.15.44:0/411483498 wait complete. 2024-04-21T23:15:22.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.511+0000 7f5dad240740 20 client.17142 trim_cache size 0 max 16384 2024-04-21T23:15:22.514 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:15:22.514 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-21T23:15:22.514 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "b3b4f3ab-c649-4729-983c-e24e7312de15", 2024-04-21T23:15:22.514 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-21T23:15:22.514 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:15:22.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.512+0000 7f5daa65f640 1 -- 172.21.15.44:0/2916188913 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5d8c0a2d00 msgr2=0x7f5d8c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:22.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.512+0000 7f5daa65f640 1 --2- 172.21.15.44:0/2916188913 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5d8c0a2d00 0x7f5d8c0a51c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c006010 tx=0x7f5d9c04b000 comp rx=0 tx=0).stop 2024-04-21T23:15:22.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.512+0000 7f5daa65f640 1 -- 172.21.15.44:0/2916188913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5da4101700 msgr2=0x7f5da4073a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:22.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.512+0000 7f5daa65f640 1 --2- 172.21.15.44:0/2916188913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5da4101700 0x7f5da4073a40 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f5d9400e9f0 tx=0x7f5d9400eeb0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.512+0000 7f5daa65f640 1 -- 172.21.15.44:0/2916188913 shutdown_connections 2024-04-21T23:15:22.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.512+0000 7f5daa65f640 1 --2- 172.21.15.44:0/2916188913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5da4107710 0x7f5da40744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.512+0000 7f5daa65f640 1 --2- 172.21.15.44:0/2916188913 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5d8c0a2d00 0x7f5d8c0a51c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.512+0000 7f5daa65f640 1 --2- 172.21.15.44:0/2916188913 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5da41020c0 0x7f5da4073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.512+0000 7f5daa65f640 1 --2- 172.21.15.44:0/2916188913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5da4101700 0x7f5da4073a40 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.512+0000 7f5daa65f640 1 -- 172.21.15.44:0/2916188913 >> 172.21.15.44:0/2916188913 conn(0x7f5da40fd1f0 msgr2=0x7f5da40fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:22.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.512+0000 7f5daa65f640 1 -- 172.21.15.44:0/2916188913 shutdown_connections 2024-04-21T23:15:22.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.512+0000 7f5daa65f640 1 -- 172.21.15.44:0/2916188913 wait complete. 2024-04-21T23:15:22.523 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:15:22.523 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:15:22.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.655+0000 7f0303b9a640 1 -- 172.21.15.44:0/1457772116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02fc100610 msgr2=0x7f02fc100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:22.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.655+0000 7f0303b9a640 1 --2- 172.21.15.44:0/1457772116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02fc100610 0x7f02fc100a90 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f02ec009930 tx=0x7f02ec02bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:22.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.655+0000 7f0303b9a640 1 -- 172.21.15.44:0/1457772116 shutdown_connections 2024-04-21T23:15:22.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.655+0000 7f0303b9a640 1 --2- 172.21.15.44:0/1457772116 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02fc100fd0 0x7f02fc10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.655+0000 7f0303b9a640 1 --2- 172.21.15.44:0/1457772116 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02fc100610 0x7f02fc100a90 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.655+0000 7f0303b9a640 1 --2- 172.21.15.44:0/1457772116 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02fc1066b0 0x7f02fc106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.655+0000 7f0303b9a640 1 -- 172.21.15.44:0/1457772116 >> 172.21.15.44:0/1457772116 conn(0x7f02fc0777f0 msgr2=0x7f02fc077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:22.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.655+0000 7f0303b9a640 1 -- 172.21.15.44:0/1457772116 shutdown_connections 2024-04-21T23:15:22.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.655+0000 7f0303b9a640 1 -- 172.21.15.44:0/1457772116 wait complete. 2024-04-21T23:15:22.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.656+0000 7f0303b9a640 1 Processor -- start 2024-04-21T23:15:22.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.656+0000 7f0303b9a640 1 -- start start 2024-04-21T23:15:22.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.656+0000 7f0303b9a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02fc100610 0x7f02fc1022b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.656+0000 7f0303b9a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02fc100fd0 0x7f02fc1027f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.656+0000 7f0303b9a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02fc1066b0 0x7f02fc102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.656+0000 7f0303b9a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc110a30 con 0x7f02fc100610 2024-04-21T23:15:22.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.656+0000 7f0303b9a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc1108b0 con 0x7f02fc100fd0 2024-04-21T23:15:22.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.656+0000 7f0303b9a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc110bb0 con 0x7f02fc1066b0 2024-04-21T23:15:22.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.656+0000 7f0301936640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02fc100610 0x7f02fc1022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.656+0000 7f0301936640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02fc100610 0x7f02fc1022b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54054/0 (socket says 172.21.15.44:54054) 2024-04-21T23:15:22.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.656+0000 7f0301936640 1 -- 172.21.15.44:0/900892109 learned_addr learned my addr 172.21.15.44:0/900892109 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:22.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.656+0000 7f0301135640 1 --2- 172.21.15.44:0/900892109 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02fc100fd0 0x7f02fc1027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.656+0000 7f0302137640 1 --2- 172.21.15.44:0/900892109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02fc1066b0 0x7f02fc102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.656+0000 7f0301936640 1 -- 172.21.15.44:0/900892109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02fc1066b0 msgr2=0x7f02fc102d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:22.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.656+0000 7f0301936640 1 --2- 172.21.15.44:0/900892109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02fc1066b0 0x7f02fc102d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.656+0000 7f0301936640 1 -- 172.21.15.44:0/900892109 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02fc100fd0 msgr2=0x7f02fc1027f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:22.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.656+0000 7f0301936640 1 --2- 172.21.15.44:0/900892109 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02fc100fd0 0x7f02fc1027f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.656+0000 7f0301936640 1 -- 172.21.15.44:0/900892109 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02fc1a64c0 con 0x7f02fc100610 2024-04-21T23:15:22.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.656+0000 7f0302137640 1 --2- 172.21.15.44:0/900892109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02fc1066b0 0x7f02fc102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:15:22.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.657+0000 7f0301936640 1 --2- 172.21.15.44:0/900892109 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02fc100610 0x7f02fc1022b0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f02f800e9c0 tx=0x7f02f800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:22.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.657+0000 7f02f2ffd640 1 -- 172.21.15.44:0/900892109 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f800cd80 con 0x7f02fc100610 2024-04-21T23:15:22.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.657+0000 7f02f2ffd640 1 -- 172.21.15.44:0/900892109 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02f80035f0 con 0x7f02fc100610 2024-04-21T23:15:22.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.657+0000 7f0303b9a640 1 -- 172.21.15.44:0/900892109 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02fc1a67b0 con 0x7f02fc100610 2024-04-21T23:15:22.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.657+0000 7f02f2ffd640 1 -- 172.21.15.44:0/900892109 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f8010690 con 0x7f02fc100610 2024-04-21T23:15:22.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.657+0000 7f0303b9a640 1 -- 172.21.15.44:0/900892109 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02fc1a6cd0 con 0x7f02fc100610 2024-04-21T23:15:22.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.659+0000 7f02f2ffd640 1 -- 172.21.15.44:0/900892109 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f02f80037a0 con 0x7f02fc100610 2024-04-21T23:15:22.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.660+0000 7f0306784740 1 Processor -- start 2024-04-21T23:15:22.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.660+0000 7f0306784740 1 -- start start 2024-04-21T23:15:22.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.660+0000 7f02f2ffd640 1 --2- 172.21.15.44:0/900892109 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f02e80a2d80 0x7f02e80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.660+0000 7f0306784740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c5657bf30 0x556c5657c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.660+0000 7f0306784740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c5657c850 0x556c56584ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.660+0000 7f0306784740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c56585430 0x556c56587820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.660+0000 7f0306784740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x556c564e0560 con 0x556c5657c850 2024-04-21T23:15:22.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.660+0000 7f0306784740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x556c56488730 con 0x556c56585430 2024-04-21T23:15:22.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.660+0000 7f0306784740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556c564dc520 con 0x556c5657bf30 2024-04-21T23:15:22.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.660+0000 7f0302137640 1 --2- 172.21.15.44:0/900892109 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f02e80a2d80 0x7f02e80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.660+0000 7f0301936640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c5657bf30 0x556c5657c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.660+0000 7f02f2ffd640 1 -- 172.21.15.44:0/900892109 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f02f80cfb90 con 0x7f02fc100610 2024-04-21T23:15:22.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.660+0000 7f0301936640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c5657bf30 0x556c5657c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:49414/0 (socket says 172.21.15.44:49414) 2024-04-21T23:15:22.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.660+0000 7f0301936640 1 -- 172.21.15.44:0/3036437748 learned_addr learned my addr 172.21.15.44:0/3036437748 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:22.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.660+0000 7f0302137640 1 --2- 172.21.15.44:0/3036437748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c56585430 0x556c56587820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.660+0000 7f0301135640 1 --2- 172.21.15.44:0/3036437748 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c5657c850 0x556c56584ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.661+0000 7f0301936640 1 -- 172.21.15.44:0/3036437748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c56585430 msgr2=0x556c56587820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:22.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.661+0000 7f0301936640 1 --2- 172.21.15.44:0/3036437748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c56585430 0x556c56587820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.661+0000 7f0301936640 1 -- 172.21.15.44:0/3036437748 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c5657c850 msgr2=0x556c56584ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:22.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.661+0000 7f0301936640 1 --2- 172.21.15.44:0/3036437748 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c5657c850 0x556c56584ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.661+0000 7f0301936640 1 -- 172.21.15.44:0/3036437748 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c562be7d0 con 0x556c5657bf30 2024-04-21T23:15:22.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.661+0000 7f0302137640 1 --2- 172.21.15.44:0/900892109 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f02e80a2d80 0x7f02e80a5240 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009fd0 tx=0x7f02f4013400 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:22.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.661+0000 7f0301936640 1 --2- 172.21.15.44:0/3036437748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c5657bf30 0x556c5657c310 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f02f8004630 tx=0x7f02f809bb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c50e3dd8bfc704b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:22.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.661+0000 7f02e7fff640 1 -- 172.21.15.44:0/3036437748 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f809c3c0 con 0x556c5657bf30 2024-04-21T23:15:22.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.661+0000 7f02e7fff640 1 -- 172.21.15.44:0/3036437748 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02f809bcc0 con 0x556c5657bf30 2024-04-21T23:15:22.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.661+0000 7f02e7fff640 1 -- 172.21.15.44:0/3036437748 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f8098330 con 0x556c5657bf30 2024-04-21T23:15:22.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.662+0000 7f0306784740 1 -- 172.21.15.44:0/3036437748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c5657bf30 msgr2=0x556c5657c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:22.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.662+0000 7f0306784740 1 --2- 172.21.15.44:0/3036437748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c5657bf30 0x556c5657c310 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f02f8004630 tx=0x7f02f809bb50 comp rx=0 tx=0).stop 2024-04-21T23:15:22.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.662+0000 7f0306784740 1 -- 172.21.15.44:0/3036437748 shutdown_connections 2024-04-21T23:15:22.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.662+0000 7f0306784740 1 --2- 172.21.15.44:0/3036437748 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c56585430 0x556c56587820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.662+0000 7f0306784740 1 --2- 172.21.15.44:0/3036437748 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c5657c850 0x556c56584ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.662+0000 7f0306784740 1 --2- 172.21.15.44:0/3036437748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c5657bf30 0x556c5657c310 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.662+0000 7f0306784740 1 -- 172.21.15.44:0/3036437748 >> 172.21.15.44:0/3036437748 conn(0x556c56576660 msgr2=0x556c56576a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:22.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.662+0000 7f0306784740 1 -- 172.21.15.44:0/3036437748 shutdown_connections 2024-04-21T23:15:22.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.662+0000 7f0306784740 1 -- 172.21.15.44:0/3036437748 wait complete. 2024-04-21T23:15:22.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.663+0000 7f0306784740 1 -- start start 2024-04-21T23:15:22.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.663+0000 7f0306784740 1 Processor -- start 2024-04-21T23:15:22.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.663+0000 7f0306784740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c5657bf30 0x556c56582550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.663+0000 7f0306784740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c5657c850 0x556c56582a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.663+0000 7f0306784740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c56585430 0x556c56582fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.663+0000 7f0306784740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x556c56588220 con 0x556c5657c850 2024-04-21T23:15:22.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.663+0000 7f0306784740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x556c5649be00 con 0x556c56585430 2024-04-21T23:15:22.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.663+0000 7f0306784740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556c565883a0 con 0x556c5657bf30 2024-04-21T23:15:22.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.663+0000 7f0306784740 10 client.0 _ll_register_callbacks cb 0x556c564d6eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:22.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.663+0000 7f0301135640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c5657c850 0x556c56582a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.663+0000 7f0301936640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c5657bf30 0x556c56582550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.663+0000 7f0302137640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c56585430 0x556c56582fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.663+0000 7f0301135640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c5657c850 0x556c56582a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54082/0 (socket says 172.21.15.44:54082) 2024-04-21T23:15:22.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.663+0000 7f0301135640 1 -- 172.21.15.44:0/3464887251 learned_addr learned my addr 172.21.15.44:0/3464887251 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:22.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.664+0000 7f0301936640 1 -- 172.21.15.44:0/3464887251 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c56585430 msgr2=0x556c56582fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:22.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.664+0000 7f0301936640 1 --2- 172.21.15.44:0/3464887251 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c56585430 0x556c56582fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.664+0000 7f0301936640 1 -- 172.21.15.44:0/3464887251 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c5657c850 msgr2=0x556c56582a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:22.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.664+0000 7f0301936640 1 --2- 172.21.15.44:0/3464887251 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c5657c850 0x556c56582a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.664+0000 7f0301936640 1 -- 172.21.15.44:0/3464887251 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c561f43f0 con 0x556c5657bf30 2024-04-21T23:15:22.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.664+0000 7f0302137640 1 --2- 172.21.15.44:0/3464887251 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c56585430 0x556c56582fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:22.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.664+0000 7f0301936640 1 --2- 172.21.15.44:0/3464887251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c5657bf30 0x556c56582550 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f02f80a4af0 tx=0x7f02f809b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c259f0c8345164a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:22.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.664+0000 7f02e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:15:22.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.664+0000 7f02e57fa640 1 -- 172.21.15.44:0/3464887251 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f809c3c0 con 0x556c5657bf30 2024-04-21T23:15:22.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.664+0000 7f02e57fa640 1 -- 172.21.15.44:0/3464887251 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f02f807aae0 con 0x556c5657bf30 2024-04-21T23:15:22.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.664+0000 7f0306784740 1 -- 172.21.15.44:0/3464887251 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02f809bcc0 con 0x556c5657bf30 2024-04-21T23:15:22.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.664+0000 7f02e57fa640 1 -- 172.21.15.44:0/3464887251 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f02f807f050 con 0x556c5657bf30 2024-04-21T23:15:22.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.664+0000 7f0306784740 10 client.26701 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:22.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.664+0000 7f0306784740 1 -- 172.21.15.44:0/3464887251 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556c561b40f0 con 0x556c5657bf30 2024-04-21T23:15:22.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.664+0000 7f02e57fa640 1 -- 172.21.15.44:0/3464887251 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02f809bcc0 con 0x556c5657bf30 2024-04-21T23:15:22.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.664+0000 7f02e57fa640 1 -- 172.21.15.44:0/3464887251 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f02f809b110 con 0x556c5657bf30 2024-04-21T23:15:22.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.664+0000 7f0306784740 10 client.26701 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:22.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.664+0000 7f0306784740 10 client.26701 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:22.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.664+0000 7f0306784740 20 client.26701 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:22.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.664+0000 7f0306784740 1 --2- 172.21.15.44:0/3464887251 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556c56595730 0x556c56597b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.665+0000 7f0306784740 4 client.26701 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:15:22.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.665+0000 7f0306784740 1 -- 172.21.15.44:0/3464887251 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c564edaa0 con 0x556c56595730 2024-04-21T23:15:22.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.665+0000 7f0301135640 1 --2- 172.21.15.44:0/3464887251 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556c56595730 0x556c56597b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.665+0000 7f0301135640 1 --2- 172.21.15.44:0/3464887251 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556c56595730 0x556c56597b20 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db0cfb0af63aa35c server_cookie=239b6511cfa3d361 in_seq=0 out_seq=0 2024-04-21T23:15:22.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.665+0000 7f02e57fa640 10 client.26701 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:22.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.666+0000 7f02e57fa640 1 -- 172.21.15.44:0/3464887251 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556c564edaa0 con 0x556c56595730 2024-04-21T23:15:22.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.666+0000 7f02e57fa640 10 client.26701 handle_command_reply: tid=0 2024-04-21T23:15:22.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.666+0000 7f0306784740 1 client.26701 shutdown 2024-04-21T23:15:22.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.666+0000 7f0306784740 1 -- 172.21.15.44:0/3464887251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c5657bf30 msgr2=0x556c56582550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:22.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.666+0000 7f0306784740 1 --2- 172.21.15.44:0/3464887251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c5657bf30 0x556c56582550 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f02f80a4af0 tx=0x7f02f809b710 comp rx=0 tx=0).stop 2024-04-21T23:15:22.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.666+0000 7f0306784740 1 -- 172.21.15.44:0/3464887251 shutdown_connections 2024-04-21T23:15:22.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.666+0000 7f0306784740 1 --2- 172.21.15.44:0/3464887251 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556c56595730 0x556c56597b20 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.666+0000 7f0306784740 1 --2- 172.21.15.44:0/3464887251 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c56585430 0x556c56582fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.666+0000 7f0306784740 1 --2- 172.21.15.44:0/3464887251 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c5657c850 0x556c56582a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.666+0000 7f02e57fa640 0 client.26701 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:22.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.666+0000 7f0306784740 1 --2- 172.21.15.44:0/3464887251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c5657bf30 0x556c56582550 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.666+0000 7f0306784740 1 -- 172.21.15.44:0/3464887251 >> 172.21.15.44:0/3464887251 conn(0x556c56576660 msgr2=0x556c565783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:22.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.666+0000 7f0306784740 1 -- 172.21.15.44:0/3464887251 shutdown_connections 2024-04-21T23:15:22.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.666+0000 7f0306784740 1 -- 172.21.15.44:0/3464887251 wait complete. 2024-04-21T23:15:22.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.666+0000 7f0306784740 20 client.26701 trim_cache size 0 max 16384 2024-04-21T23:15:22.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0306784740 1 Processor -- start 2024-04-21T23:15:22.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0306784740 1 -- start start 2024-04-21T23:15:22.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0306784740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c5657bf30 0x556c56584c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0306784740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c5657c850 0x556c5657da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0306784740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c56585430 0x556c56581400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0306784740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc1aefd0 con 0x556c56585430 2024-04-21T23:15:22.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0306784740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc1b1830 con 0x556c5657c850 2024-04-21T23:15:22.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0306784740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc1af620 con 0x556c5657bf30 2024-04-21T23:15:22.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0302137640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c56585430 0x556c56581400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0301135640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c5657c850 0x556c5657da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0301936640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c5657bf30 0x556c56584c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0301135640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c5657c850 0x556c5657da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:58890/0 (socket says 172.21.15.44:58890) 2024-04-21T23:15:22.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0301936640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c5657bf30 0x556c56584c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:49432/0 (socket says 172.21.15.44:49432) 2024-04-21T23:15:22.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0302137640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c56585430 0x556c56581400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54084/0 (socket says 172.21.15.44:54084) 2024-04-21T23:15:22.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0301135640 1 -- 172.21.15.44:0/226054318 learned_addr learned my addr 172.21.15.44:0/226054318 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:22.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0301936640 1 -- 172.21.15.44:0/226054318 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c5657c850 msgr2=0x556c5657da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:22.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0301936640 1 --2- 172.21.15.44:0/226054318 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c5657c850 0x556c5657da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0301936640 1 -- 172.21.15.44:0/226054318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c56585430 msgr2=0x556c56581400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:22.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0301936640 1 --2- 172.21.15.44:0/226054318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c56585430 0x556c56581400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0301936640 1 -- 172.21.15.44:0/226054318 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c5657c650 con 0x556c5657bf30 2024-04-21T23:15:22.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.687+0000 7f0302137640 1 --2- 172.21.15.44:0/226054318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c56585430 0x556c56581400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:22.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.686+0000 7f0301135640 1 --2- 172.21.15.44:0/226054318 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c5657c850 0x556c5657da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:22.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.687+0000 7f0301936640 1 --2- 172.21.15.44:0/226054318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c5657bf30 0x556c56584c80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f02f80ccfa0 tx=0x7f02f809a1d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bc1227c9a7bb652 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:22.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.687+0000 7f02e77fe640 1 -- 172.21.15.44:0/226054318 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f80a4930 con 0x556c5657bf30 2024-04-21T23:15:22.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.687+0000 7f02e77fe640 1 -- 172.21.15.44:0/226054318 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02f809c3c0 con 0x556c5657bf30 2024-04-21T23:15:22.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.687+0000 7f02e77fe640 1 -- 172.21.15.44:0/226054318 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f8098330 con 0x556c5657bf30 2024-04-21T23:15:22.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.688+0000 7f0306784740 1 -- 172.21.15.44:0/226054318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c5657bf30 msgr2=0x556c56584c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:22.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.688+0000 7f0306784740 1 --2- 172.21.15.44:0/226054318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c5657bf30 0x556c56584c80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f02f80ccfa0 tx=0x7f02f809a1d0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.688+0000 7f0306784740 1 -- 172.21.15.44:0/226054318 shutdown_connections 2024-04-21T23:15:22.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.688+0000 7f0306784740 1 --2- 172.21.15.44:0/226054318 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c56585430 0x556c56581400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.688+0000 7f0306784740 1 --2- 172.21.15.44:0/226054318 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c5657c850 0x556c5657da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.688+0000 7f0306784740 1 --2- 172.21.15.44:0/226054318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c5657bf30 0x556c56584c80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.688+0000 7f0306784740 1 -- 172.21.15.44:0/226054318 >> 172.21.15.44:0/226054318 conn(0x556c56576660 msgr2=0x556c56580cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:22.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.688+0000 7f0306784740 1 -- 172.21.15.44:0/226054318 shutdown_connections 2024-04-21T23:15:22.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.688+0000 7f0306784740 1 -- 172.21.15.44:0/226054318 wait complete. 2024-04-21T23:15:22.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.688+0000 7f0306784740 1 -- start start 2024-04-21T23:15:22.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.688+0000 7f0306784740 1 Processor -- start 2024-04-21T23:15:22.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.689+0000 7f0306784740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c5657bf30 0x556c5659b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.689+0000 7f0306784740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c5657c850 0x556c565984a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.689+0000 7f0306784740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c56585430 0x556c56598b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.689+0000 7f0306784740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc1afcf0 con 0x556c5657bf30 2024-04-21T23:15:22.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.689+0000 7f0306784740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc1b1160 con 0x556c5657c850 2024-04-21T23:15:22.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.689+0000 7f0306784740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc1b03c0 con 0x556c56585430 2024-04-21T23:15:22.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.689+0000 7f0306784740 10 client.0 _ll_register_callbacks cb 0x7f02fc104340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:22.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.689+0000 7f0301936640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c5657bf30 0x556c5659b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.689+0000 7f0301135640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c5657c850 0x556c565984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.689+0000 7f0301936640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c5657bf30 0x556c5659b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54090/0 (socket says 172.21.15.44:54090) 2024-04-21T23:15:22.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.689+0000 7f0301936640 1 -- 172.21.15.44:0/3955913584 learned_addr learned my addr 172.21.15.44:0/3955913584 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:22.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.689+0000 7f0302137640 1 --2- 172.21.15.44:0/3955913584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c56585430 0x556c56598b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.689+0000 7f0301135640 1 -- 172.21.15.44:0/3955913584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c56585430 msgr2=0x556c56598b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:22.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.689+0000 7f0301135640 1 --2- 172.21.15.44:0/3955913584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c56585430 0x556c56598b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.689+0000 7f0301135640 1 -- 172.21.15.44:0/3955913584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c5657bf30 msgr2=0x556c5659b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:22.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.689+0000 7f0301135640 1 --2- 172.21.15.44:0/3955913584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c5657bf30 0x556c5659b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.689+0000 7f0301135640 1 -- 172.21.15.44:0/3955913584 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c565fa910 con 0x556c5657c850 2024-04-21T23:15:22.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.690+0000 7f0301135640 1 --2- 172.21.15.44:0/3955913584 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c5657c850 0x556c565984a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f02ec005ab0 tx=0x7f02ec02c9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b7a896265bae4e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:22.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.690+0000 7f02e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:15:22.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.690+0000 7f02e57fa640 1 -- 172.21.15.44:0/3955913584 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec04e060 con 0x556c5657c850 2024-04-21T23:15:22.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.690+0000 7f02e57fa640 1 -- 172.21.15.44:0/3955913584 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f02ec004240 con 0x556c5657c850 2024-04-21T23:15:22.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.690+0000 7f0306784740 1 -- 172.21.15.44:0/3955913584 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02f809c3c0 con 0x556c5657c850 2024-04-21T23:15:22.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.690+0000 7f02e57fa640 1 -- 172.21.15.44:0/3955913584 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f02ec055050 con 0x556c5657c850 2024-04-21T23:15:22.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.690+0000 7f0306784740 10 client.26723 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:22.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.690+0000 7f0306784740 1 -- 172.21.15.44:0/3955913584 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556c565fbb20 con 0x556c5657c850 2024-04-21T23:15:22.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.690+0000 7f02e57fa640 1 -- 172.21.15.44:0/3955913584 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02f809c3c0 con 0x556c5657c850 2024-04-21T23:15:22.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.690+0000 7f02e57fa640 1 -- 172.21.15.44:0/3955913584 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f02ec004aa0 con 0x556c5657c850 2024-04-21T23:15:22.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.690+0000 7f0306784740 10 client.26723 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:22.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.690+0000 7f0306784740 10 client.26723 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:22.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.690+0000 7f0306784740 20 client.26723 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:22.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.690+0000 7f0306784740 1 --2- 172.21.15.44:0/3955913584 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556c565fc2d0 0x556c565fe6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:22.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.690+0000 7f0306784740 4 client.26723 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:15:22.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.690+0000 7f0306784740 1 -- 172.21.15.44:0/3955913584 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556c564e7df0 con 0x556c565fc2d0 2024-04-21T23:15:22.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.691+0000 7f0301936640 1 --2- 172.21.15.44:0/3955913584 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556c565fc2d0 0x556c565fe6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:22.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.691+0000 7f0301936640 1 --2- 172.21.15.44:0/3955913584 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556c565fc2d0 0x556c565fe6f0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=825966b7ed69e8ca server_cookie=a1c23eb728407903 in_seq=0 out_seq=0 2024-04-21T23:15:22.695 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.691+0000 7f02e57fa640 10 client.26723 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:22.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.693+0000 7f02e57fa640 1 -- 172.21.15.44:0/3955913584 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556c564e7df0 con 0x556c565fc2d0 2024-04-21T23:15:22.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.693+0000 7f02e57fa640 10 client.26723 handle_command_reply: tid=0 2024-04-21T23:15:22.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.693+0000 7f0306784740 1 client.26723 shutdown 2024-04-21T23:15:22.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.693+0000 7f0306784740 1 -- 172.21.15.44:0/3955913584 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c5657c850 msgr2=0x556c565984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:22.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.693+0000 7f0306784740 1 --2- 172.21.15.44:0/3955913584 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c5657c850 0x556c565984a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f02ec005ab0 tx=0x7f02ec02c9d0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.693+0000 7f0306784740 1 -- 172.21.15.44:0/3955913584 shutdown_connections 2024-04-21T23:15:22.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.693+0000 7f0306784740 1 --2- 172.21.15.44:0/3955913584 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556c565fc2d0 0x556c565fe6f0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.693+0000 7f0306784740 1 --2- 172.21.15.44:0/3955913584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c56585430 0x556c56598b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.693+0000 7f0306784740 1 --2- 172.21.15.44:0/3955913584 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c5657c850 0x556c565984a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.696 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.693+0000 7f0306784740 1 --2- 172.21.15.44:0/3955913584 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c5657bf30 0x556c5659b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.693+0000 7f02e57fa640 0 client.26723 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:22.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.693+0000 7f0306784740 1 -- 172.21.15.44:0/3955913584 >> 172.21.15.44:0/3955913584 conn(0x556c56576660 msgr2=0x556c56581580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:22.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.693+0000 7f0306784740 1 -- 172.21.15.44:0/3955913584 shutdown_connections 2024-04-21T23:15:22.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.693+0000 7f0306784740 1 -- 172.21.15.44:0/3955913584 wait complete. 2024-04-21T23:15:22.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.693+0000 7f0306784740 20 client.26723 trim_cache size 0 max 16384 2024-04-21T23:15:22.697 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:15:22.697 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:15:22.697 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-21T23:15:22.697 INFO:teuthology.orchestra.run.smithi044.stdout: "b3b4f3ab-c649-4729-983c-e24e7312de15": { 2024-04-21T23:15:22.697 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-21T23:15:22.697 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "b3b4f3ab-c649-4729-983c-e24e7312de15", 2024-04-21T23:15:22.698 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-04-21T23:15:22.698 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:15:22.698 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:15:22.698 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:15:22.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.694+0000 7f0303b9a640 1 -- 172.21.15.44:0/900892109 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f02e80a2d80 msgr2=0x7f02e80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:22.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.694+0000 7f0303b9a640 1 --2- 172.21.15.44:0/900892109 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f02e80a2d80 0x7f02e80a5240 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009fd0 tx=0x7f02f4013400 comp rx=0 tx=0).stop 2024-04-21T23:15:22.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.694+0000 7f0303b9a640 1 -- 172.21.15.44:0/900892109 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02fc100610 msgr2=0x7f02fc1022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:22.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.694+0000 7f0303b9a640 1 --2- 172.21.15.44:0/900892109 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02fc100610 0x7f02fc1022b0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f02f800e9c0 tx=0x7f02f800ee80 comp rx=0 tx=0).stop 2024-04-21T23:15:22.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.694+0000 7f0303b9a640 1 -- 172.21.15.44:0/900892109 shutdown_connections 2024-04-21T23:15:22.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.694+0000 7f0303b9a640 1 --2- 172.21.15.44:0/900892109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02fc1066b0 0x7f02fc102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.694+0000 7f0303b9a640 1 --2- 172.21.15.44:0/900892109 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f02e80a2d80 0x7f02e80a5240 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.694+0000 7f0303b9a640 1 --2- 172.21.15.44:0/900892109 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02fc100fd0 0x7f02fc1027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.694+0000 7f0303b9a640 1 --2- 172.21.15.44:0/900892109 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f02fc100610 0x7f02fc1022b0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:22.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.694+0000 7f0303b9a640 1 -- 172.21.15.44:0/900892109 >> 172.21.15.44:0/900892109 conn(0x7f02fc0777f0 msgr2=0x7f02fc068ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:22.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.694+0000 7f0303b9a640 1 -- 172.21.15.44:0/900892109 shutdown_connections 2024-04-21T23:15:22.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:22.694+0000 7f0303b9a640 1 -- 172.21.15.44:0/900892109 wait complete. 2024-04-21T23:15:22.705 INFO:tasks.cephfs.filesystem:scrub status for tag:b3b4f3ab-c649-4729-983c-e24e7312de15 - {'path': '/', 'tag': 'b3b4f3ab-c649-4729-983c-e24e7312de15', 'options': 'recursive,force'} 2024-04-21T23:15:22.705 INFO:tasks.workunit.client.0.smithi044.stdout: 1 769457 12.90 MB/sec execute 192 sec latency 30.972 ms 2024-04-21T23:15:23.705 INFO:tasks.workunit.client.0.smithi044.stdout: 1 771502 12.90 MB/sec execute 193 sec latency 20.700 ms 2024-04-21T23:15:24.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:23 smithi044 ceph-mon[27934]: scrub queued for path: / 2024-04-21T23:15:24.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:23 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:15:24.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:23 smithi044 ceph-mon[27934]: scrub summary: active paths [/] 2024-04-21T23:15:24.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:23 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:15:24.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:23 smithi044 ceph-mon[27934]: pgmap v205: 129 pgs: 129 active+clean; 79 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.1 MiB/s wr, 383 op/s 2024-04-21T23:15:24.213 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:23 smithi055 ceph-mon[31117]: scrub queued for path: / 2024-04-21T23:15:24.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:23 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:15:24.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:23 smithi055 ceph-mon[31117]: scrub summary: active paths [/] 2024-04-21T23:15:24.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:23 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:15:24.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:23 smithi055 ceph-mon[31117]: pgmap v205: 129 pgs: 129 active+clean; 79 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.1 MiB/s wr, 383 op/s 2024-04-21T23:15:24.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:23 smithi069 ceph-mon[31816]: scrub queued for path: / 2024-04-21T23:15:24.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:23 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:15:24.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:23 smithi069 ceph-mon[31816]: scrub summary: active paths [/] 2024-04-21T23:15:24.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:23 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:15:24.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:23 smithi069 ceph-mon[31816]: pgmap v205: 129 pgs: 129 active+clean; 79 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.1 MiB/s wr, 383 op/s 2024-04-21T23:15:24.704 INFO:tasks.workunit.client.0.smithi044.stdout: 1 774155 12.92 MB/sec execute 194 sec latency 94.038 ms 2024-04-21T23:15:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.947+0000 7fe2b6874640 1 -- 172.21.15.44:0/2958269320 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2b0106650 msgr2=0x7fe2b0106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.947+0000 7fe2b6874640 1 --2- 172.21.15.44:0/2958269320 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2b0106650 0x7fe2b0106a30 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0009930 tx=0x7fe2a002bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.947+0000 7fe2b6874640 1 -- 172.21.15.44:0/2958269320 shutdown_connections 2024-04-21T23:15:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.947+0000 7fe2b6874640 1 --2- 172.21.15.44:0/2958269320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2b0100640 0x7fe2b0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.947+0000 7fe2b6874640 1 --2- 172.21.15.44:0/2958269320 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2b0101000 0x7fe2b010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.947+0000 7fe2b6874640 1 --2- 172.21.15.44:0/2958269320 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2b0106650 0x7fe2b0106a30 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.947+0000 7fe2b6874640 1 -- 172.21.15.44:0/2958269320 >> 172.21.15.44:0/2958269320 conn(0x7fe2b00777c0 msgr2=0x7fe2b0077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.947+0000 7fe2b6874640 1 -- 172.21.15.44:0/2958269320 shutdown_connections 2024-04-21T23:15:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.947+0000 7fe2b6874640 1 -- 172.21.15.44:0/2958269320 wait complete. 2024-04-21T23:15:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.948+0000 7fe2b6874640 1 Processor -- start 2024-04-21T23:15:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.948+0000 7fe2b6874640 1 -- start start 2024-04-21T23:15:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.948+0000 7fe2b6874640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2b0100640 0x7fe2b0102c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.948+0000 7fe2b6874640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2b0101000 0x7fe2b01031a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.948+0000 7fe2b6874640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2b0106650 0x7fe2b01036e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.948+0000 7fe2b6874640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b0110a20 con 0x7fe2b0106650 2024-04-21T23:15:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.948+0000 7fe2b6874640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b01108a0 con 0x7fe2b0101000 2024-04-21T23:15:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.948+0000 7fe2b6874640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b0110ba0 con 0x7fe2b0100640 2024-04-21T23:15:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.948+0000 7fe2b4e11640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2b0106650 0x7fe2b01036e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.948+0000 7fe2b4e11640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2b0106650 0x7fe2b01036e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54102/0 (socket says 172.21.15.44:54102) 2024-04-21T23:15:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.948+0000 7fe2b4e11640 1 -- 172.21.15.44:0/835466924 learned_addr learned my addr 172.21.15.44:0/835466924 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.948+0000 7fe2af7fe640 1 --2- 172.21.15.44:0/835466924 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2b0101000 0x7fe2b01031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.948+0000 7fe2affff640 1 --2- 172.21.15.44:0/835466924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2b0100640 0x7fe2b0102c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.949+0000 7fe2b4e11640 1 -- 172.21.15.44:0/835466924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2b0100640 msgr2=0x7fe2b0102c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.949+0000 7fe2b4e11640 1 --2- 172.21.15.44:0/835466924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2b0100640 0x7fe2b0102c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.949+0000 7fe2b4e11640 1 -- 172.21.15.44:0/835466924 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2b0101000 msgr2=0x7fe2b01031a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.949+0000 7fe2b4e11640 1 --2- 172.21.15.44:0/835466924 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2b0101000 0x7fe2b01031a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.949+0000 7fe2b4e11640 1 -- 172.21.15.44:0/835466924 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2b01a65b0 con 0x7fe2b0106650 2024-04-21T23:15:24.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.949+0000 7fe2b4e11640 1 --2- 172.21.15.44:0/835466924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2b0106650 0x7fe2b01036e0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8009fd0 tx=0x7fe2a800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:24.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.949+0000 7fe2ad7fa640 1 -- 172.21.15.44:0/835466924 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a8017070 con 0x7fe2b0106650 2024-04-21T23:15:24.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.949+0000 7fe2ad7fa640 1 -- 172.21.15.44:0/835466924 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2a8003e50 con 0x7fe2b0106650 2024-04-21T23:15:24.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.949+0000 7fe2b6874640 1 -- 172.21.15.44:0/835466924 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2b01a68a0 con 0x7fe2b0106650 2024-04-21T23:15:24.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.949+0000 7fe2ad7fa640 1 -- 172.21.15.44:0/835466924 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a80106c0 con 0x7fe2b0106650 2024-04-21T23:15:24.951 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.949+0000 7fe2b6874640 1 -- 172.21.15.44:0/835466924 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2b01a6da0 con 0x7fe2b0106650 2024-04-21T23:15:24.957 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.951+0000 7fe2ad7fa640 1 -- 172.21.15.44:0/835466924 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe2a8003970 con 0x7fe2b0106650 2024-04-21T23:15:24.957 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.951+0000 7fe2b9456740 1 Processor -- start 2024-04-21T23:15:24.957 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.951+0000 7fe2b9456740 1 -- start start 2024-04-21T23:15:24.957 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2b9456740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662dbb20 0x558d662dbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.957 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2b9456740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662dc440 0x558d662e4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.957 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2b9456740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662e5210 0x558d662e7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2b9456740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558d6623fb70 con 0x558d662e5210 2024-04-21T23:15:24.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2b9456740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x558d661e8470 con 0x558d662dc440 2024-04-21T23:15:24.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2b9456740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558d6623bb30 con 0x558d662dbb20 2024-04-21T23:15:24.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2af7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662e5210 0x558d662e7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2ad7fa640 1 --2- 172.21.15.44:0/835466924 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe2980a2cf0 0x7fe2980a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2affff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662dc440 0x558d662e4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2af7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662e5210 0x558d662e7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54110/0 (socket says 172.21.15.44:54110) 2024-04-21T23:15:24.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2affff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662dc440 0x558d662e4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:58918/0 (socket says 172.21.15.44:58918) 2024-04-21T23:15:24.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2af7fe640 1 -- 172.21.15.44:0/1202389863 learned_addr learned my addr 172.21.15.44:0/1202389863 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:24.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2af7fe640 1 --2- 172.21.15.44:0/1202389863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662dbb20 0x558d662dbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2b4e11640 1 --2- 172.21.15.44:0/835466924 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe2980a2cf0 0x7fe2980a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2affff640 1 -- 172.21.15.44:0/1202389863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662dbb20 msgr2=0x558d662dbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:24.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2affff640 1 --2- 172.21.15.44:0/1202389863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662dbb20 0x558d662dbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.958 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2ad7fa640 1 -- 172.21.15.44:0/835466924 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7fe2a80cf500 con 0x7fe2b0106650 2024-04-21T23:15:24.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2affff640 1 -- 172.21.15.44:0/1202389863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662e5210 msgr2=0x558d662e7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:24.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2affff640 1 --2- 172.21.15.44:0/1202389863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662e5210 0x558d662e7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2affff640 1 -- 172.21.15.44:0/1202389863 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d65f54210 con 0x558d662dc440 2024-04-21T23:15:24.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2af7fe640 1 --2- 172.21.15.44:0/1202389863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662dbb20 0x558d662dbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:24.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.952+0000 7fe2affff640 1 --2- 172.21.15.44:0/1202389863 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662dc440 0x558d662e4cd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0009230 tx=0x7fe2a0005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebe0eb71c782e765 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:24.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.953+0000 7fe2b4e11640 1 --2- 172.21.15.44:0/835466924 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe2980a2cf0 0x7fe2980a51b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe2a809b670 tx=0x7fe2a8086040 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:24.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.953+0000 7fe2967fc640 1 -- 172.21.15.44:0/1202389863 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0042070 con 0x558d662dc440 2024-04-21T23:15:24.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.953+0000 7fe2967fc640 1 -- 172.21.15.44:0/1202389863 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2a002ca80 con 0x558d662dc440 2024-04-21T23:15:24.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.953+0000 7fe2967fc640 1 -- 172.21.15.44:0/1202389863 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0004f50 con 0x558d662dc440 2024-04-21T23:15:24.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.953+0000 7fe2b9456740 1 -- 172.21.15.44:0/1202389863 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662dc440 msgr2=0x558d662e4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:24.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.953+0000 7fe2b9456740 1 --2- 172.21.15.44:0/1202389863 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662dc440 0x558d662e4cd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0009230 tx=0x7fe2a0005c00 comp rx=0 tx=0).stop 2024-04-21T23:15:24.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.953+0000 7fe2b9456740 1 -- 172.21.15.44:0/1202389863 shutdown_connections 2024-04-21T23:15:24.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.953+0000 7fe2b9456740 1 --2- 172.21.15.44:0/1202389863 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662e5210 0x558d662e7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.953+0000 7fe2b9456740 1 --2- 172.21.15.44:0/1202389863 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662dc440 0x558d662e4cd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.959 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.953+0000 7fe2b9456740 1 --2- 172.21.15.44:0/1202389863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662dbb20 0x558d662dbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.953+0000 7fe2b9456740 1 -- 172.21.15.44:0/1202389863 >> 172.21.15.44:0/1202389863 conn(0x558d662d6440 msgr2=0x558d662d6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.953+0000 7fe2b9456740 1 -- 172.21.15.44:0/1202389863 shutdown_connections 2024-04-21T23:15:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.953+0000 7fe2b9456740 1 -- 172.21.15.44:0/1202389863 wait complete. 2024-04-21T23:15:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.954+0000 7fe2b9456740 1 -- start start 2024-04-21T23:15:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.954+0000 7fe2b9456740 1 Processor -- start 2024-04-21T23:15:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.954+0000 7fe2b9456740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662dbb20 0x558d662e2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.954+0000 7fe2b9456740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662dc440 0x558d662e27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.954+0000 7fe2b9456740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662e5210 0x558d662e2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.954+0000 7fe2b9456740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558d662e7de0 con 0x558d662e5210 2024-04-21T23:15:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.954+0000 7fe2b9456740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x558d661fba10 con 0x558d662dc440 2024-04-21T23:15:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.954+0000 7fe2b9456740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558d662e7f60 con 0x558d662dbb20 2024-04-21T23:15:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.954+0000 7fe2b9456740 10 client.0 _ll_register_callbacks cb 0x558d66236b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.954+0000 7fe2affff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662dc440 0x558d662e27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.954+0000 7fe2af7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662e5210 0x558d662e2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.954+0000 7fe2affff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662dc440 0x558d662e27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:58924/0 (socket says 172.21.15.44:58924) 2024-04-21T23:15:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.954+0000 7fe2affff640 1 -- 172.21.15.44:0/3370133919 learned_addr learned my addr 172.21.15.44:0/3370133919 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.954+0000 7fe2af7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662e5210 0x558d662e2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54116/0 (socket says 172.21.15.44:54116) 2024-04-21T23:15:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.954+0000 7fe2af7fe640 1 --2- 172.21.15.44:0/3370133919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662dbb20 0x558d662e2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.955+0000 7fe2af7fe640 1 -- 172.21.15.44:0/3370133919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662dbb20 msgr2=0x558d662e2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.955+0000 7fe2af7fe640 1 --2- 172.21.15.44:0/3370133919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662dbb20 0x558d662e2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.955+0000 7fe2af7fe640 1 -- 172.21.15.44:0/3370133919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662dc440 msgr2=0x558d662e27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.955+0000 7fe2af7fe640 1 --2- 172.21.15.44:0/3370133919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662dc440 0x558d662e27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.955+0000 7fe2af7fe640 1 -- 172.21.15.44:0/3370133919 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d66020620 con 0x558d662e5210 2024-04-21T23:15:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.955+0000 7fe2affff640 1 --2- 172.21.15.44:0/3370133919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662dc440 0x558d662e27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.955+0000 7fe2af7fe640 1 --2- 172.21.15.44:0/3370133919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662e5210 0x558d662e2cf0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x558d662d75d0 tx=0x7fe2a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a50f46ce07a8980c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.955+0000 7fe28f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:15:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.955+0000 7fe28f7fe640 1 -- 172.21.15.44:0/3370133919 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a401b040 con 0x558d662e5210 2024-04-21T23:15:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.955+0000 7fe28f7fe640 1 -- 172.21.15.44:0/3370133919 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7fe2a4004250 con 0x558d662e5210 2024-04-21T23:15:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.955+0000 7fe2b9456740 1 -- 172.21.15.44:0/3370133919 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2a002ca80 con 0x558d662e5210 2024-04-21T23:15:24.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.955+0000 7fe28f7fe640 1 -- 172.21.15.44:0/3370133919 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe2a4010e70 con 0x558d662e5210 2024-04-21T23:15:24.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.955+0000 7fe2b9456740 10 client.17172 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:24.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.955+0000 7fe2b9456740 1 -- 172.21.15.44:0/3370133919 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558d662f40d0 con 0x558d662e5210 2024-04-21T23:15:24.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.955+0000 7fe28f7fe640 1 -- 172.21.15.44:0/3370133919 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2a002ca80 con 0x558d662e5210 2024-04-21T23:15:24.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.956+0000 7fe28f7fe640 1 -- 172.21.15.44:0/3370133919 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7fe2a4022430 con 0x558d662e5210 2024-04-21T23:15:24.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.956+0000 7fe2b9456740 10 client.17172 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:24.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.956+0000 7fe2b9456740 10 client.17172 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:24.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.956+0000 7fe2b9456740 20 client.17172 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:24.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.956+0000 7fe2b9456740 1 --2- 172.21.15.44:0/3370133919 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558d662f56f0 0x558d662f7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.956+0000 7fe2b9456740 4 client.17172 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:15:24.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.956+0000 7fe2b9456740 1 -- 172.21.15.44:0/3370133919 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d6624d0b0 con 0x558d662f56f0 2024-04-21T23:15:24.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.956+0000 7fe2affff640 1 --2- 172.21.15.44:0/3370133919 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558d662f56f0 0x558d662f7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.956+0000 7fe28f7fe640 10 client.17172 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.956+0000 7fe2affff640 1 --2- 172.21.15.44:0/3370133919 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558d662f56f0 0x558d662f7b70 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc0a09ca7dec421b server_cookie=c3748d87a42098a5 in_seq=0 out_seq=0 2024-04-21T23:15:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.957+0000 7fe28f7fe640 1 -- 172.21.15.44:0/3370133919 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558d6624d0b0 con 0x558d662f56f0 2024-04-21T23:15:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.957+0000 7fe28f7fe640 10 client.17172 handle_command_reply: tid=0 2024-04-21T23:15:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.957+0000 7fe2b9456740 1 client.17172 shutdown 2024-04-21T23:15:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.957+0000 7fe2b9456740 1 -- 172.21.15.44:0/3370133919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662e5210 msgr2=0x558d662e2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.957+0000 7fe2b9456740 1 --2- 172.21.15.44:0/3370133919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662e5210 0x558d662e2cf0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x558d662d75d0 tx=0x7fe2a4004730 comp rx=0 tx=0).stop 2024-04-21T23:15:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.957+0000 7fe2b9456740 1 -- 172.21.15.44:0/3370133919 shutdown_connections 2024-04-21T23:15:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.957+0000 7fe2b9456740 1 --2- 172.21.15.44:0/3370133919 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558d662f56f0 0x558d662f7b70 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.957+0000 7fe2b9456740 1 --2- 172.21.15.44:0/3370133919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662e5210 0x558d662e2cf0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.957+0000 7fe2b9456740 1 --2- 172.21.15.44:0/3370133919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662dc440 0x558d662e27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.957+0000 7fe28f7fe640 0 client.17172 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.957+0000 7fe2b9456740 1 --2- 172.21.15.44:0/3370133919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662dbb20 0x558d662e2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.957+0000 7fe2b9456740 1 -- 172.21.15.44:0/3370133919 >> 172.21.15.44:0/3370133919 conn(0x558d662d6440 msgr2=0x558d662d8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.957+0000 7fe2b9456740 1 -- 172.21.15.44:0/3370133919 shutdown_connections 2024-04-21T23:15:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.957+0000 7fe2b9456740 1 -- 172.21.15.44:0/3370133919 wait complete. 2024-04-21T23:15:24.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.957+0000 7fe2b9456740 20 client.17172 trim_cache size 0 max 16384 2024-04-21T23:15:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.972+0000 7fe2b9456740 1 Processor -- start 2024-04-21T23:15:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.972+0000 7fe2b9456740 1 -- start start 2024-04-21T23:15:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.972+0000 7fe2b9456740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662dbb20 0x558d662e49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.975 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.972+0000 7fe2b9456740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662dc440 0x558d662f0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.972+0000 7fe2b9456740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662e5210 0x558d662e1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.972+0000 7fe2b9456740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b01aeeb0 con 0x558d662dc440 2024-04-21T23:15:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.972+0000 7fe2b9456740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b01b1710 con 0x558d662e5210 2024-04-21T23:15:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.972+0000 7fe2b9456740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b01af500 con 0x558d662dbb20 2024-04-21T23:15:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.972+0000 7fe2affff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662dc440 0x558d662f0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.972+0000 7fe2affff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662dc440 0x558d662f0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54126/0 (socket says 172.21.15.44:54126) 2024-04-21T23:15:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.972+0000 7fe2affff640 1 -- 172.21.15.44:0/847711591 learned_addr learned my addr 172.21.15.44:0/847711591 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.972+0000 7fe2af7fe640 1 --2- 172.21.15.44:0/847711591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662e5210 0x558d662e1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.972+0000 7fe2af7fe640 1 --2- 172.21.15.44:0/847711591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662dbb20 0x558d662e49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.972+0000 7fe2affff640 1 -- 172.21.15.44:0/847711591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662dbb20 msgr2=0x558d662e49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.972+0000 7fe2affff640 1 --2- 172.21.15.44:0/847711591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662dbb20 0x558d662e49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.972+0000 7fe2affff640 1 -- 172.21.15.44:0/847711591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662e5210 msgr2=0x558d662e1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.972+0000 7fe2affff640 1 --2- 172.21.15.44:0/847711591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662e5210 0x558d662e1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.972+0000 7fe2affff640 1 -- 172.21.15.44:0/847711591 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d662d07a0 con 0x558d662dc440 2024-04-21T23:15:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.972+0000 7fe2af7fe640 1 --2- 172.21.15.44:0/847711591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662dbb20 0x558d662e49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:15:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.973+0000 7fe2affff640 1 --2- 172.21.15.44:0/847711591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662dc440 0x558d662f0ca0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0009230 tx=0x7fe2a002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10adb49db5a320c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.973+0000 7fe295ffb640 1 -- 172.21.15.44:0/847711591 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0042040 con 0x558d662dc440 2024-04-21T23:15:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.973+0000 7fe295ffb640 1 -- 172.21.15.44:0/847711591 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2a0004040 con 0x558d662dc440 2024-04-21T23:15:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.973+0000 7fe295ffb640 1 -- 172.21.15.44:0/847711591 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0005240 con 0x558d662dc440 2024-04-21T23:15:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.973+0000 7fe2b9456740 1 -- 172.21.15.44:0/847711591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662dc440 msgr2=0x558d662f0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.973+0000 7fe2b9456740 1 --2- 172.21.15.44:0/847711591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662dc440 0x558d662f0ca0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0009230 tx=0x7fe2a002e040 comp rx=0 tx=0).stop 2024-04-21T23:15:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.973+0000 7fe2b9456740 1 -- 172.21.15.44:0/847711591 shutdown_connections 2024-04-21T23:15:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.973+0000 7fe2b9456740 1 --2- 172.21.15.44:0/847711591 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662e5210 0x558d662e1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.973+0000 7fe2b9456740 1 --2- 172.21.15.44:0/847711591 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662dc440 0x558d662f0ca0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.973+0000 7fe2b9456740 1 --2- 172.21.15.44:0/847711591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662dbb20 0x558d662e49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.973+0000 7fe2b9456740 1 -- 172.21.15.44:0/847711591 >> 172.21.15.44:0/847711591 conn(0x558d662d6440 msgr2=0x558d662e2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.973+0000 7fe2b9456740 1 -- 172.21.15.44:0/847711591 shutdown_connections 2024-04-21T23:15:24.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.973+0000 7fe2b9456740 1 -- 172.21.15.44:0/847711591 wait complete. 2024-04-21T23:15:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.974+0000 7fe2b9456740 1 -- start start 2024-04-21T23:15:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.974+0000 7fe2b9456740 1 Processor -- start 2024-04-21T23:15:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.974+0000 7fe2b9456740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662dbb20 0x558d662ed1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.974+0000 7fe2b9456740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662dc440 0x558d662ed6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.974+0000 7fe2b9456740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662e5210 0x558d662de580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.974+0000 7fe2b9456740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b01afbd0 con 0x558d662dc440 2024-04-21T23:15:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.974+0000 7fe2b9456740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b01b1040 con 0x558d662dbb20 2024-04-21T23:15:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.974+0000 7fe2b9456740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b01b02a0 con 0x558d662e5210 2024-04-21T23:15:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.974+0000 7fe2b9456740 10 client.0 _ll_register_callbacks cb 0x7fe2a40129e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.974+0000 7fe2affff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662dc440 0x558d662ed6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.974+0000 7fe2af7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662dbb20 0x558d662ed1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.974+0000 7fe2affff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662dc440 0x558d662ed6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:54140/0 (socket says 172.21.15.44:54140) 2024-04-21T23:15:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.974+0000 7fe2affff640 1 -- 172.21.15.44:0/2818330269 learned_addr learned my addr 172.21.15.44:0/2818330269 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:24.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.974+0000 7fe2af7fe640 1 --2- 172.21.15.44:0/2818330269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662e5210 0x558d662de580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.975+0000 7fe2affff640 1 -- 172.21.15.44:0/2818330269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662e5210 msgr2=0x558d662de580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.975+0000 7fe2affff640 1 --2- 172.21.15.44:0/2818330269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662e5210 0x558d662de580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.975+0000 7fe2affff640 1 -- 172.21.15.44:0/2818330269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662dbb20 msgr2=0x558d662ed1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.975+0000 7fe2affff640 1 --2- 172.21.15.44:0/2818330269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662dbb20 0x558d662ed1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.975+0000 7fe2affff640 1 -- 172.21.15.44:0/2818330269 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d662debf0 con 0x558d662dc440 2024-04-21T23:15:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.975+0000 7fe2affff640 1 --2- 172.21.15.44:0/2818330269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662dc440 0x558d662ed6f0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0005ec0 tx=0x7fe2a0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d85b5029c46e188f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.975+0000 7fe28f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:15:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.975+0000 7fe28f7fe640 1 -- 172.21.15.44:0/2818330269 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a005c060 con 0x558d662dc440 2024-04-21T23:15:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.975+0000 7fe28f7fe640 1 -- 172.21.15.44:0/2818330269 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7fe2a002c880 con 0x558d662dc440 2024-04-21T23:15:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.975+0000 7fe2b9456740 1 -- 172.21.15.44:0/2818330269 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2a0004040 con 0x558d662dc440 2024-04-21T23:15:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.975+0000 7fe28f7fe640 1 -- 172.21.15.44:0/2818330269 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe2a0042070 con 0x558d662dc440 2024-04-21T23:15:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.975+0000 7fe2b9456740 10 client.17178 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.975+0000 7fe2b9456740 1 -- 172.21.15.44:0/2818330269 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558d66363d40 con 0x558d662dc440 2024-04-21T23:15:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.975+0000 7fe28f7fe640 1 -- 172.21.15.44:0/2818330269 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2a0004040 con 0x558d662dc440 2024-04-21T23:15:24.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.975+0000 7fe28f7fe640 1 -- 172.21.15.44:0/2818330269 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7fe2a0003710 con 0x558d662dc440 2024-04-21T23:15:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.976+0000 7fe2b9456740 10 client.17178 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.976+0000 7fe2b9456740 10 client.17178 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.976+0000 7fe2b9456740 20 client.17178 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.976+0000 7fe2b9456740 1 --2- 172.21.15.44:0/2818330269 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558d66363fc0 0x558d663663b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.976+0000 7fe2b9456740 4 client.17178 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:15:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.976+0000 7fe2b9456740 1 -- 172.21.15.44:0/2818330269 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558d66247400 con 0x558d66363fc0 2024-04-21T23:15:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.976+0000 7fe2af7fe640 1 --2- 172.21.15.44:0/2818330269 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558d66363fc0 0x558d663663b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.976+0000 7fe28f7fe640 10 client.17178 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:24.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:24.976+0000 7fe2af7fe640 1 --2- 172.21.15.44:0/2818330269 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558d66363fc0 0x558d663663b0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b675f279192eca53 server_cookie=b005c4cfd56415df in_seq=0 out_seq=0 2024-04-21T23:15:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.336+0000 7fe28f7fe640 1 -- 172.21.15.44:0/2818330269 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558d66247400 con 0x558d66363fc0 2024-04-21T23:15:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.336+0000 7fe28f7fe640 10 client.17178 handle_command_reply: tid=0 2024-04-21T23:15:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.336+0000 7fe2b9456740 1 client.17178 shutdown 2024-04-21T23:15:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.337+0000 7fe2b9456740 1 -- 172.21.15.44:0/2818330269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662dc440 msgr2=0x558d662ed6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.337+0000 7fe2b9456740 1 --2- 172.21.15.44:0/2818330269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662dc440 0x558d662ed6f0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0005ec0 tx=0x7fe2a0005fd0 comp rx=0 tx=0).stop 2024-04-21T23:15:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.337+0000 7fe2b9456740 1 -- 172.21.15.44:0/2818330269 shutdown_connections 2024-04-21T23:15:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.337+0000 7fe2b9456740 1 --2- 172.21.15.44:0/2818330269 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558d66363fc0 0x558d663663b0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.337+0000 7fe2b9456740 1 --2- 172.21.15.44:0/2818330269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558d662e5210 0x558d662de580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.337+0000 7fe28f7fe640 0 client.17178 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.337+0000 7fe2b9456740 1 --2- 172.21.15.44:0/2818330269 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558d662dc440 0x558d662ed6f0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.337+0000 7fe2b9456740 1 --2- 172.21.15.44:0/2818330269 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558d662dbb20 0x558d662ed1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.337+0000 7fe2b9456740 1 -- 172.21.15.44:0/2818330269 >> 172.21.15.44:0/2818330269 conn(0x558d662d6440 msgr2=0x558d662d4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.337+0000 7fe2b9456740 1 -- 172.21.15.44:0/2818330269 shutdown_connections 2024-04-21T23:15:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.337+0000 7fe2b9456740 1 -- 172.21.15.44:0/2818330269 wait complete. 2024-04-21T23:15:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.337+0000 7fe2b9456740 20 client.17178 trim_cache size 0 max 16384 2024-04-21T23:15:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:15:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:15:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:15:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:15:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.338+0000 7fe2b6874640 1 -- 172.21.15.44:0/835466924 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe2980a2cf0 msgr2=0x7fe2980a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.338+0000 7fe2b6874640 1 --2- 172.21.15.44:0/835466924 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe2980a2cf0 0x7fe2980a51b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe2a809b670 tx=0x7fe2a8086040 comp rx=0 tx=0).stop 2024-04-21T23:15:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.338+0000 7fe2b6874640 1 -- 172.21.15.44:0/835466924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2b0106650 msgr2=0x7fe2b01036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.338+0000 7fe2b6874640 1 --2- 172.21.15.44:0/835466924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2b0106650 0x7fe2b01036e0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8009fd0 tx=0x7fe2a800b260 comp rx=0 tx=0).stop 2024-04-21T23:15:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.338+0000 7fe2b6874640 1 -- 172.21.15.44:0/835466924 shutdown_connections 2024-04-21T23:15:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.338+0000 7fe2b6874640 1 --2- 172.21.15.44:0/835466924 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe2980a2cf0 0x7fe2980a51b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:25.340 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.338+0000 7fe2b6874640 1 --2- 172.21.15.44:0/835466924 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe2b0106650 0x7fe2b01036e0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:25.340 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.338+0000 7fe2b6874640 1 --2- 172.21.15.44:0/835466924 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2b0101000 0x7fe2b01031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:25.340 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.338+0000 7fe2b6874640 1 --2- 172.21.15.44:0/835466924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2b0100640 0x7fe2b0102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:25.340 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.338+0000 7fe2b6874640 1 -- 172.21.15.44:0/835466924 >> 172.21.15.44:0/835466924 conn(0x7fe2b00777c0 msgr2=0x7fe2b006a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:25.340 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.338+0000 7fe2b6874640 1 -- 172.21.15.44:0/835466924 shutdown_connections 2024-04-21T23:15:25.340 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:25.338+0000 7fe2b6874640 1 -- 172.21.15.44:0/835466924 wait complete. 2024-04-21T23:15:25.704 INFO:tasks.workunit.client.0.smithi044.stdout: 1 775156 12.87 MB/sec execute 195 sec latency 34.822 ms 2024-04-21T23:15:26.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:25 smithi044 ceph-mon[27934]: pgmap v206: 129 pgs: 129 active+clean; 79 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 330 op/s 2024-04-21T23:15:26.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:25 smithi069 ceph-mon[31816]: pgmap v206: 129 pgs: 129 active+clean; 79 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 330 op/s 2024-04-21T23:15:26.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:25 smithi055 ceph-mon[31117]: pgmap v206: 129 pgs: 129 active+clean; 79 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 330 op/s 2024-04-21T23:15:26.704 INFO:tasks.workunit.client.0.smithi044.stdout: 1 778013 12.89 MB/sec execute 196 sec latency 27.678 ms 2024-04-21T23:15:27.705 INFO:tasks.workunit.client.0.smithi044.stdout: 1 780604 12.89 MB/sec execute 197 sec latency 23.309 ms 2024-04-21T23:15:28.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:27 smithi044 ceph-mon[27934]: pgmap v207: 129 pgs: 129 active+clean; 80 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 451 op/s 2024-04-21T23:15:28.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:27 smithi044 ceph-mon[27934]: scrub summary: idle 2024-04-21T23:15:28.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:27 smithi055 ceph-mon[31117]: pgmap v207: 129 pgs: 129 active+clean; 80 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 451 op/s 2024-04-21T23:15:28.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:27 smithi055 ceph-mon[31117]: scrub summary: idle 2024-04-21T23:15:28.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:27 smithi069 ceph-mon[31816]: pgmap v207: 129 pgs: 129 active+clean; 80 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 451 op/s 2024-04-21T23:15:28.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:27 smithi069 ceph-mon[31816]: scrub summary: idle 2024-04-21T23:15:28.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 782697 12.88 MB/sec execute 198 sec latency 25.389 ms 2024-04-21T23:15:29.705 INFO:tasks.workunit.client.0.smithi044.stdout: 1 785167 12.89 MB/sec execute 199 sec latency 100.771 ms 2024-04-21T23:15:30.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:29 smithi044 ceph-mon[27934]: pgmap v208: 129 pgs: 129 active+clean; 69 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 513 op/s 2024-04-21T23:15:30.213 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:29 smithi055 ceph-mon[31117]: pgmap v208: 129 pgs: 129 active+clean; 69 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 513 op/s 2024-04-21T23:15:30.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:29 smithi069 ceph-mon[31816]: pgmap v208: 129 pgs: 129 active+clean; 69 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 513 op/s 2024-04-21T23:15:30.705 INFO:tasks.workunit.client.0.smithi044.stdout: 1 787672 12.89 MB/sec execute 200 sec latency 52.818 ms 2024-04-21T23:15:31.705 INFO:tasks.workunit.client.0.smithi044.stdout: 1 788822 12.86 MB/sec execute 201 sec latency 440.932 ms 2024-04-21T23:15:32.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:31 smithi044 ceph-mon[27934]: pgmap v209: 129 pgs: 129 active+clean; 69 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 388 op/s 2024-04-21T23:15:32.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:31 smithi055 ceph-mon[31117]: pgmap v209: 129 pgs: 129 active+clean; 69 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 388 op/s 2024-04-21T23:15:32.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:31 smithi069 ceph-mon[31816]: pgmap v209: 129 pgs: 129 active+clean; 69 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 388 op/s 2024-04-21T23:15:32.705 INFO:tasks.workunit.client.0.smithi044.stdout: 1 788822 12.79 MB/sec execute 202 sec latency 1441.071 ms 2024-04-21T23:15:33.705 INFO:tasks.workunit.client.0.smithi044.stdout: 1 788822 12.73 MB/sec execute 203 sec latency 2441.171 ms 2024-04-21T23:15:34.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:33 smithi044 ceph-mon[27934]: pgmap v210: 129 pgs: 129 active+clean; 75 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.9 MiB/s wr, 499 op/s 2024-04-21T23:15:34.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:33 smithi055 ceph-mon[31117]: pgmap v210: 129 pgs: 129 active+clean; 75 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.9 MiB/s wr, 499 op/s 2024-04-21T23:15:34.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:33 smithi069 ceph-mon[31816]: pgmap v210: 129 pgs: 129 active+clean; 75 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.9 MiB/s wr, 499 op/s 2024-04-21T23:15:34.705 INFO:tasks.workunit.client.0.smithi044.stdout: 1 788822 12.67 MB/sec execute 204 sec latency 3441.241 ms 2024-04-21T23:15:35.705 INFO:tasks.workunit.client.0.smithi044.stdout: 1 788822 12.60 MB/sec execute 205 sec latency 4441.301 ms 2024-04-21T23:15:36.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:35 smithi044 ceph-mon[27934]: pgmap v211: 129 pgs: 129 active+clean; 75 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 383 op/s 2024-04-21T23:15:36.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:35 smithi069 ceph-mon[31816]: pgmap v211: 129 pgs: 129 active+clean; 75 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 383 op/s 2024-04-21T23:15:36.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:35 smithi055 ceph-mon[31117]: pgmap v211: 129 pgs: 129 active+clean; 75 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 383 op/s 2024-04-21T23:15:36.705 INFO:tasks.workunit.client.0.smithi044.stdout: 1 788822 12.54 MB/sec execute 206 sec latency 5441.390 ms 2024-04-21T23:15:37.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 788822 12.48 MB/sec execute 207 sec latency 6441.488 ms 2024-04-21T23:15:38.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:37 smithi044 ceph-mon[27934]: pgmap v212: 129 pgs: 129 active+clean; 76 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.5 MiB/s wr, 395 op/s 2024-04-21T23:15:38.213 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:37 smithi055 ceph-mon[31117]: pgmap v212: 129 pgs: 129 active+clean; 76 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.5 MiB/s wr, 395 op/s 2024-04-21T23:15:38.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:37 smithi069 ceph-mon[31816]: pgmap v212: 129 pgs: 129 active+clean; 76 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.5 MiB/s wr, 395 op/s 2024-04-21T23:15:38.439 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.438+0000 7f5b475bf640 1 -- 172.21.15.44:0/538307661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b40107740 msgr2=0x7f5b40107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.438+0000 7f5b475bf640 1 --2- 172.21.15.44:0/538307661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b40107740 0x7f5b40107b20 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c009930 tx=0x7f5b3c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.438+0000 7f5b475bf640 1 -- 172.21.15.44:0/538307661 shutdown_connections 2024-04-21T23:15:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.438+0000 7f5b475bf640 1 --2- 172.21.15.44:0/538307661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b401020f0 0x7f5b4010e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.438+0000 7f5b475bf640 1 --2- 172.21.15.44:0/538307661 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5b40101730 0x7f5b40101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.438+0000 7f5b475bf640 1 --2- 172.21.15.44:0/538307661 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b40107740 0x7f5b40107b20 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.438+0000 7f5b475bf640 1 -- 172.21.15.44:0/538307661 >> 172.21.15.44:0/538307661 conn(0x7f5b400fd200 msgr2=0x7f5b400ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.438+0000 7f5b475bf640 1 -- 172.21.15.44:0/538307661 shutdown_connections 2024-04-21T23:15:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.438+0000 7f5b475bf640 1 -- 172.21.15.44:0/538307661 wait complete. 2024-04-21T23:15:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.438+0000 7f5b475bf640 1 Processor -- start 2024-04-21T23:15:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.438+0000 7f5b475bf640 1 -- start start 2024-04-21T23:15:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.439+0000 7f5b475bf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b40101730 0x7f5b40197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.439+0000 7f5b475bf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b401020f0 0x7f5b40198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.439+0000 7f5b475bf640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5b40107740 0x7f5b4019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.439+0000 7f5b475bf640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5b40111b50 con 0x7f5b40101730 2024-04-21T23:15:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.439+0000 7f5b475bf640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5b401119d0 con 0x7f5b40107740 2024-04-21T23:15:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.439+0000 7f5b475bf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5b40111cd0 con 0x7f5b401020f0 2024-04-21T23:15:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.439+0000 7f5b4535b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b40101730 0x7f5b40197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.439+0000 7f5b4535b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b40101730 0x7f5b40197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34062/0 (socket says 172.21.15.44:34062) 2024-04-21T23:15:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.439+0000 7f5b4535b640 1 -- 172.21.15.44:0/2831030228 learned_addr learned my addr 172.21.15.44:0/2831030228 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.439+0000 7f5b45b5c640 1 --2- 172.21.15.44:0/2831030228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5b40107740 0x7f5b4019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.439+0000 7f5b44b5a640 1 --2- 172.21.15.44:0/2831030228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b401020f0 0x7f5b40198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.439+0000 7f5b4535b640 1 -- 172.21.15.44:0/2831030228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b401020f0 msgr2=0x7f5b40198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.439+0000 7f5b4535b640 1 --2- 172.21.15.44:0/2831030228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b401020f0 0x7f5b40198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.439+0000 7f5b4535b640 1 -- 172.21.15.44:0/2831030228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5b40107740 msgr2=0x7f5b4019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:38.441 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.439+0000 7f5b4535b640 1 --2- 172.21.15.44:0/2831030228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5b40107740 0x7f5b4019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.439+0000 7f5b4535b640 1 -- 172.21.15.44:0/2831030228 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b4019cfc0 con 0x7f5b40101730 2024-04-21T23:15:38.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.439+0000 7f5b45b5c640 1 --2- 172.21.15.44:0/2831030228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5b40107740 0x7f5b4019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:15:38.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.439+0000 7f5b44b5a640 1 --2- 172.21.15.44:0/2831030228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b401020f0 0x7f5b40198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:15:38.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.439+0000 7f5b4535b640 1 --2- 172.21.15.44:0/2831030228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b40101730 0x7f5b40197fd0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c009230 tx=0x7f5b3c02cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:38.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.440+0000 7f5b367fc640 1 -- 172.21.15.44:0/2831030228 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c041070 con 0x7f5b40101730 2024-04-21T23:15:38.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.440+0000 7f5b475bf640 1 -- 172.21.15.44:0/2831030228 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b4019d250 con 0x7f5b40101730 2024-04-21T23:15:38.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.440+0000 7f5b367fc640 1 -- 172.21.15.44:0/2831030228 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b3c004450 con 0x7f5b40101730 2024-04-21T23:15:38.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.440+0000 7f5b367fc640 1 -- 172.21.15.44:0/2831030228 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c034440 con 0x7f5b40101730 2024-04-21T23:15:38.442 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.440+0000 7f5b475bf640 1 -- 172.21.15.44:0/2831030228 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b401a4b40 con 0x7f5b40101730 2024-04-21T23:15:38.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b367fc640 1 -- 172.21.15.44:0/2831030228 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f5b3c044070 con 0x7f5b40101730 2024-04-21T23:15:38.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b4a1a3740 1 Processor -- start 2024-04-21T23:15:38.449 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b4a1a3740 1 -- start start 2024-04-21T23:15:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b4a1a3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbd2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b4a1a3740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbd3440 0x55b0ebbdbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b4a1a3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbdc210 0x55b0ebbde600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b4a1a3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b0ebb36b70 con 0x55b0ebbdc210 2024-04-21T23:15:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b4a1a3740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55b0ebadf470 con 0x55b0ebbd3440 2024-04-21T23:15:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b4a1a3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b0ebb32b30 con 0x55b0ebbd2b20 2024-04-21T23:15:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b367fc640 1 --2- 172.21.15.44:0/2831030228 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5b2c0a2cf0 0x7f5b2c0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b44b5a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbdc210 0x55b0ebbde600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b4535b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbd3440 0x55b0ebbdbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b44b5a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbdc210 0x55b0ebbde600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34076/0 (socket says 172.21.15.44:34076) 2024-04-21T23:15:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b44b5a640 1 -- 172.21.15.44:0/4258729131 learned_addr learned my addr 172.21.15.44:0/4258729131 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b45b5c640 1 --2- 172.21.15.44:0/4258729131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbd2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b45b5c640 1 --2- 172.21.15.44:0/2831030228 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5b2c0a2cf0 0x7f5b2c0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b367fc640 1 -- 172.21.15.44:0/2831030228 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f5b3c0f12b0 con 0x7f5b40101730 2024-04-21T23:15:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b44b5a640 1 -- 172.21.15.44:0/4258729131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbd2b20 msgr2=0x55b0ebbd2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b44b5a640 1 --2- 172.21.15.44:0/4258729131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbd2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b44b5a640 1 -- 172.21.15.44:0/4258729131 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbd3440 msgr2=0x55b0ebbdbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b44b5a640 1 --2- 172.21.15.44:0/4258729131 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbd3440 0x55b0ebbdbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b44b5a640 1 -- 172.21.15.44:0/4258729131 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0eb84b210 con 0x55b0ebbdc210 2024-04-21T23:15:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b45b5c640 1 --2- 172.21.15.44:0/4258729131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbd2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:15:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b44b5a640 1 --2- 172.21.15.44:0/4258729131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbdc210 0x55b0ebbde600 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f5b3000bbe0 tx=0x7f5b3000d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9973c42899144e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b45b5c640 1 --2- 172.21.15.44:0/2831030228 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5b2c0a2cf0 0x7f5b2c0a51b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5b38005c40 tx=0x7f5b38009280 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.444+0000 7f5b2b7fe640 1 -- 172.21.15.44:0/4258729131 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b30017620 con 0x55b0ebbdc210 2024-04-21T23:15:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.445+0000 7f5b2b7fe640 1 -- 172.21.15.44:0/4258729131 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b30017ca0 con 0x55b0ebbdc210 2024-04-21T23:15:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.445+0000 7f5b2b7fe640 1 -- 172.21.15.44:0/4258729131 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3000fdd0 con 0x55b0ebbdc210 2024-04-21T23:15:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.445+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/4258729131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbdc210 msgr2=0x55b0ebbde600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.445+0000 7f5b4a1a3740 1 --2- 172.21.15.44:0/4258729131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbdc210 0x55b0ebbde600 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f5b3000bbe0 tx=0x7f5b3000d710 comp rx=0 tx=0).stop 2024-04-21T23:15:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.445+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/4258729131 shutdown_connections 2024-04-21T23:15:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.445+0000 7f5b4a1a3740 1 --2- 172.21.15.44:0/4258729131 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbdc210 0x55b0ebbde600 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.445+0000 7f5b4a1a3740 1 --2- 172.21.15.44:0/4258729131 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbd3440 0x55b0ebbdbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.445+0000 7f5b4a1a3740 1 --2- 172.21.15.44:0/4258729131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbd2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.445+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/4258729131 >> 172.21.15.44:0/4258729131 conn(0x55b0ebbcd440 msgr2=0x55b0ebbcd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.445+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/4258729131 shutdown_connections 2024-04-21T23:15:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.445+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/4258729131 wait complete. 2024-04-21T23:15:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.446+0000 7f5b4a1a3740 1 -- start start 2024-04-21T23:15:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.446+0000 7f5b4a1a3740 1 Processor -- start 2024-04-21T23:15:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.446+0000 7f5b4a1a3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbd92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.446+0000 7f5b4a1a3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbd3440 0x55b0ebbd97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.446+0000 7f5b4a1a3740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbdc210 0x55b0ebbd9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.446+0000 7f5b4a1a3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b0ebbdee40 con 0x55b0ebbd2b20 2024-04-21T23:15:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.446+0000 7f5b4a1a3740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55b0ebaf2a10 con 0x55b0ebbdc210 2024-04-21T23:15:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.446+0000 7f5b4a1a3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b0ebbdefc0 con 0x55b0ebbd3440 2024-04-21T23:15:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.446+0000 7f5b4a1a3740 10 client.0 _ll_register_callbacks cb 0x55b0ebb2db60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.446+0000 7f5b4535b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbd92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.446+0000 7f5b4535b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbd92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34084/0 (socket says 172.21.15.44:34084) 2024-04-21T23:15:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.446+0000 7f5b4535b640 1 -- 172.21.15.44:0/1940923432 learned_addr learned my addr 172.21.15.44:0/1940923432 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.447+0000 7f5b44b5a640 1 --2- 172.21.15.44:0/1940923432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbd3440 0x55b0ebbd97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.447+0000 7f5b45b5c640 1 --2- 172.21.15.44:0/1940923432 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbdc210 0x55b0ebbd9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.447+0000 7f5b4535b640 1 -- 172.21.15.44:0/1940923432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbd3440 msgr2=0x55b0ebbd97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.447+0000 7f5b4535b640 1 --2- 172.21.15.44:0/1940923432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbd3440 0x55b0ebbd97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.447+0000 7f5b4535b640 1 -- 172.21.15.44:0/1940923432 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbdc210 msgr2=0x55b0ebbd9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.447+0000 7f5b4535b640 1 --2- 172.21.15.44:0/1940923432 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbdc210 0x55b0ebbd9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.447+0000 7f5b4535b640 1 -- 172.21.15.44:0/1940923432 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0eb917620 con 0x55b0ebbd2b20 2024-04-21T23:15:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.447+0000 7f5b4535b640 1 --2- 172.21.15.44:0/1940923432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbd92a0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c0b9950 tx=0x7f5b3c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b2f4f55716e80d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.447+0000 7f5b28ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:15:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.447+0000 7f5b28ff9640 1 -- 172.21.15.44:0/1940923432 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c0bdbf0 con 0x55b0ebbd2b20 2024-04-21T23:15:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.447+0000 7f5b28ff9640 1 -- 172.21.15.44:0/1940923432 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f5b3c0c5cd0 con 0x55b0ebbd2b20 2024-04-21T23:15:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.447+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/1940923432 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b30017ca0 con 0x55b0ebbd2b20 2024-04-21T23:15:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.447+0000 7f5b28ff9640 1 -- 172.21.15.44:0/1940923432 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5b3c0bd4f0 con 0x55b0ebbd2b20 2024-04-21T23:15:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.447+0000 7f5b4a1a3740 10 client.17190 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.447+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/1940923432 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b0ebbeb100 con 0x55b0ebbd2b20 2024-04-21T23:15:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.448+0000 7f5b28ff9640 1 -- 172.21.15.44:0/1940923432 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b30017ca0 con 0x55b0ebbd2b20 2024-04-21T23:15:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.448+0000 7f5b28ff9640 1 -- 172.21.15.44:0/1940923432 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f5b3c0bc9d0 con 0x55b0ebbd2b20 2024-04-21T23:15:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.448+0000 7f5b4a1a3740 10 client.17190 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.448+0000 7f5b4a1a3740 10 client.17190 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.448+0000 7f5b4a1a3740 20 client.17190 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.448+0000 7f5b4a1a3740 1 --2- 172.21.15.44:0/1940923432 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b0ebbec600 0x55b0ebbeea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.448+0000 7f5b4a1a3740 4 client.17190 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:15:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.448+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/1940923432 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0ebb440b0 con 0x55b0ebbec600 2024-04-21T23:15:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.448+0000 7f5b44b5a640 1 --2- 172.21.15.44:0/1940923432 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b0ebbec600 0x55b0ebbeea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.449+0000 7f5b28ff9640 10 client.17190 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.449+0000 7f5b44b5a640 1 --2- 172.21.15.44:0/1940923432 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b0ebbec600 0x55b0ebbeea50 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d2db2229160781 server_cookie=8daf31411adadcaa in_seq=0 out_seq=0 2024-04-21T23:15:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.449+0000 7f5b28ff9640 1 -- 172.21.15.44:0/1940923432 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b0ebb440b0 con 0x55b0ebbec600 2024-04-21T23:15:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.449+0000 7f5b28ff9640 10 client.17190 handle_command_reply: tid=0 2024-04-21T23:15:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.449+0000 7f5b4a1a3740 1 client.17190 shutdown 2024-04-21T23:15:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.449+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/1940923432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd2b20 msgr2=0x55b0ebbd92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.449+0000 7f5b4a1a3740 1 --2- 172.21.15.44:0/1940923432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbd92a0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c0b9950 tx=0x7f5b3c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.450+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/1940923432 shutdown_connections 2024-04-21T23:15:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.450+0000 7f5b4a1a3740 1 --2- 172.21.15.44:0/1940923432 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b0ebbec600 0x55b0ebbeea50 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.450+0000 7f5b4a1a3740 1 --2- 172.21.15.44:0/1940923432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbd3440 0x55b0ebbd97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.450+0000 7f5b4a1a3740 1 --2- 172.21.15.44:0/1940923432 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbdc210 0x55b0ebbd9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.450+0000 7f5b28ff9640 0 client.17190 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.450+0000 7f5b4a1a3740 1 --2- 172.21.15.44:0/1940923432 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbd92a0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.450+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/1940923432 >> 172.21.15.44:0/1940923432 conn(0x55b0ebbcd440 msgr2=0x55b0ebbcf190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.450+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/1940923432 shutdown_connections 2024-04-21T23:15:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.450+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/1940923432 wait complete. 2024-04-21T23:15:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.450+0000 7f5b4a1a3740 20 client.17190 trim_cache size 0 max 16384 2024-04-21T23:15:38.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.465+0000 7f5b4a1a3740 1 Processor -- start 2024-04-21T23:15:38.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.465+0000 7f5b4a1a3740 1 -- start start 2024-04-21T23:15:38.469 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.465+0000 7f5b4a1a3740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbdb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.465+0000 7f5b4a1a3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd3440 0x55b0ebbe7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.465+0000 7f5b4a1a3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbdc210 0x55b0ebbc8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.465+0000 7f5b4a1a3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5b401acd40 con 0x55b0ebbd3440 2024-04-21T23:15:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.465+0000 7f5b4a1a3740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5b401af5a0 con 0x55b0ebbd2b20 2024-04-21T23:15:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.465+0000 7f5b4a1a3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5b401ad390 con 0x55b0ebbdc210 2024-04-21T23:15:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.465+0000 7f5b44b5a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd3440 0x55b0ebbe7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.465+0000 7f5b44b5a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd3440 0x55b0ebbe7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34088/0 (socket says 172.21.15.44:34088) 2024-04-21T23:15:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.465+0000 7f5b44b5a640 1 -- 172.21.15.44:0/1500195266 learned_addr learned my addr 172.21.15.44:0/1500195266 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.466+0000 7f5b4535b640 1 --2- 172.21.15.44:0/1500195266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbdb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.466+0000 7f5b44b5a640 1 -- 172.21.15.44:0/1500195266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbdc210 msgr2=0x55b0ebbc8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.466+0000 7f5b44b5a640 1 --2- 172.21.15.44:0/1500195266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbdc210 0x55b0ebbc8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.466+0000 7f5b44b5a640 1 -- 172.21.15.44:0/1500195266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbd2b20 msgr2=0x55b0ebbdb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:38.470 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.466+0000 7f5b44b5a640 1 --2- 172.21.15.44:0/1500195266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbdb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.466+0000 7f5b44b5a640 1 -- 172.21.15.44:0/1500195266 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0ebbccc40 con 0x55b0ebbd3440 2024-04-21T23:15:38.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.466+0000 7f5b4535b640 1 --2- 172.21.15.44:0/1500195266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbdb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:15:38.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.466+0000 7f5b44b5a640 1 --2- 172.21.15.44:0/1500195266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd3440 0x55b0ebbe7d00 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f5b3000bbe0 tx=0x7f5b3000f930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf15cdc8049e38b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:38.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.466+0000 7f5b2affd640 1 -- 172.21.15.44:0/1500195266 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3000b6d0 con 0x55b0ebbd3440 2024-04-21T23:15:38.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.466+0000 7f5b2affd640 1 -- 172.21.15.44:0/1500195266 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b3002d070 con 0x55b0ebbd3440 2024-04-21T23:15:38.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.466+0000 7f5b2affd640 1 -- 172.21.15.44:0/1500195266 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b300054d0 con 0x55b0ebbd3440 2024-04-21T23:15:38.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.466+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/1500195266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd3440 msgr2=0x55b0ebbe7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:38.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.466+0000 7f5b4a1a3740 1 --2- 172.21.15.44:0/1500195266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd3440 0x55b0ebbe7d00 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f5b3000bbe0 tx=0x7f5b3000f930 comp rx=0 tx=0).stop 2024-04-21T23:15:38.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.467+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/1500195266 shutdown_connections 2024-04-21T23:15:38.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.467+0000 7f5b4a1a3740 1 --2- 172.21.15.44:0/1500195266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbdc210 0x55b0ebbc8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.467+0000 7f5b4a1a3740 1 --2- 172.21.15.44:0/1500195266 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd3440 0x55b0ebbe7d00 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.467+0000 7f5b4a1a3740 1 --2- 172.21.15.44:0/1500195266 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbdb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.471 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.467+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/1500195266 >> 172.21.15.44:0/1500195266 conn(0x55b0ebbcd440 msgr2=0x55b0ebbce890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:38.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.467+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/1500195266 shutdown_connections 2024-04-21T23:15:38.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.467+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/1500195266 wait complete. 2024-04-21T23:15:38.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.467+0000 7f5b4a1a3740 1 -- start start 2024-04-21T23:15:38.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.467+0000 7f5b4a1a3740 1 Processor -- start 2024-04-21T23:15:38.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.467+0000 7f5b4a1a3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbe0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.467+0000 7f5b4a1a3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbd3440 0x55b0ebbe07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.467+0000 7f5b4a1a3740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbdc210 0x55b0ebc51b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.467+0000 7f5b4a1a3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f5b401ada60 con 0x55b0ebbd2b20 2024-04-21T23:15:38.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.467+0000 7f5b4a1a3740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f5b401aeed0 con 0x55b0ebbdc210 2024-04-21T23:15:38.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.467+0000 7f5b4a1a3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5b401ae130 con 0x55b0ebbd3440 2024-04-21T23:15:38.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.467+0000 7f5b4a1a3740 10 client.0 _ll_register_callbacks cb 0x7f5b3c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:38.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.468+0000 7f5b4535b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbe0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.472 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.468+0000 7f5b4535b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbe0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:34098/0 (socket says 172.21.15.44:34098) 2024-04-21T23:15:38.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.468+0000 7f5b4535b640 1 -- 172.21.15.44:0/2264688177 learned_addr learned my addr 172.21.15.44:0/2264688177 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:38.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.468+0000 7f5b45b5c640 1 --2- 172.21.15.44:0/2264688177 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbdc210 0x55b0ebc51b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.468+0000 7f5b44b5a640 1 --2- 172.21.15.44:0/2264688177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbd3440 0x55b0ebbe07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.468+0000 7f5b4535b640 1 -- 172.21.15.44:0/2264688177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbd3440 msgr2=0x55b0ebbe07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:38.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.468+0000 7f5b4535b640 1 --2- 172.21.15.44:0/2264688177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbd3440 0x55b0ebbe07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.468+0000 7f5b4535b640 1 -- 172.21.15.44:0/2264688177 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbdc210 msgr2=0x55b0ebc51b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:38.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.468+0000 7f5b4535b640 1 --2- 172.21.15.44:0/2264688177 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbdc210 0x55b0ebc51b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.468+0000 7f5b4535b640 1 -- 172.21.15.44:0/2264688177 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0ebc52080 con 0x55b0ebbd2b20 2024-04-21T23:15:38.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.468+0000 7f5b44b5a640 1 --2- 172.21.15.44:0/2264688177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbd3440 0x55b0ebbe07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:15:38.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.468+0000 7f5b4535b640 1 --2- 172.21.15.44:0/2264688177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbe0280 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c0eec10 tx=0x7f5b3c0bcb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=276ee9faa7cb7da8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:38.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.468+0000 7f5b28ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:15:38.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.468+0000 7f5b28ff9640 1 -- 172.21.15.44:0/2264688177 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c087060 con 0x55b0ebbd2b20 2024-04-21T23:15:38.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.468+0000 7f5b28ff9640 1 -- 172.21.15.44:0/2264688177 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f5b3c0b9240 con 0x55b0ebbd2b20 2024-04-21T23:15:38.473 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.468+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/2264688177 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b3002d070 con 0x55b0ebbd2b20 2024-04-21T23:15:38.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.468+0000 7f5b28ff9640 1 -- 172.21.15.44:0/2264688177 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5b3c04b070 con 0x55b0ebbd2b20 2024-04-21T23:15:38.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.468+0000 7f5b4a1a3740 10 client.17196 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:38.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.468+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/2264688177 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b0ebc53230 con 0x55b0ebbd2b20 2024-04-21T23:15:38.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.469+0000 7f5b28ff9640 1 -- 172.21.15.44:0/2264688177 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b3002d070 con 0x55b0ebbd2b20 2024-04-21T23:15:38.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.469+0000 7f5b28ff9640 1 -- 172.21.15.44:0/2264688177 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f5b3c0b9440 con 0x55b0ebbd2b20 2024-04-21T23:15:38.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.469+0000 7f5b4a1a3740 10 client.17196 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:38.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.469+0000 7f5b4a1a3740 10 client.17196 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:38.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.469+0000 7f5b4a1a3740 20 client.17196 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:38.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.469+0000 7f5b4a1a3740 1 --2- 172.21.15.44:0/2264688177 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b0ebc537f0 0x55b0ebc55c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:38.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.469+0000 7f5b4a1a3740 4 client.17196 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:15:38.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.469+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/2264688177 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b0ebb3e400 con 0x55b0ebc537f0 2024-04-21T23:15:38.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.469+0000 7f5b44b5a640 1 --2- 172.21.15.44:0/2264688177 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b0ebc537f0 0x55b0ebc55c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:38.474 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.470+0000 7f5b44b5a640 1 --2- 172.21.15.44:0/2264688177 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b0ebc537f0 0x55b0ebc55c10 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49c5b1edfd338b11 server_cookie=a1a0fdcb9a6b324e in_seq=0 out_seq=0 2024-04-21T23:15:38.475 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.470+0000 7f5b28ff9640 10 client.17196 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.689+0000 7f5b28ff9640 1 -- 172.21.15.44:0/2264688177 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b0ebb3e400 con 0x55b0ebc537f0 2024-04-21T23:15:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.689+0000 7f5b28ff9640 10 client.17196 handle_command_reply: tid=0 2024-04-21T23:15:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.689+0000 7f5b4a1a3740 1 client.17196 shutdown 2024-04-21T23:15:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.689+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/2264688177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd2b20 msgr2=0x55b0ebbe0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.689+0000 7f5b4a1a3740 1 --2- 172.21.15.44:0/2264688177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbe0280 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c0eec10 tx=0x7f5b3c0bcb70 comp rx=0 tx=0).stop 2024-04-21T23:15:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.689+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/2264688177 shutdown_connections 2024-04-21T23:15:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.689+0000 7f5b4a1a3740 1 --2- 172.21.15.44:0/2264688177 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b0ebc537f0 0x55b0ebc55c10 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.689+0000 7f5b4a1a3740 1 --2- 172.21.15.44:0/2264688177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b0ebbd3440 0x55b0ebbe07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.689+0000 7f5b4a1a3740 1 --2- 172.21.15.44:0/2264688177 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b0ebbdc210 0x55b0ebc51b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.689+0000 7f5b28ff9640 0 client.17196 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.689+0000 7f5b4a1a3740 1 --2- 172.21.15.44:0/2264688177 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b0ebbd2b20 0x55b0ebbe0280 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.690+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/2264688177 >> 172.21.15.44:0/2264688177 conn(0x55b0ebbcd440 msgr2=0x55b0ebbcc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.690+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/2264688177 shutdown_connections 2024-04-21T23:15:38.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.690+0000 7f5b4a1a3740 1 -- 172.21.15.44:0/2264688177 wait complete. 2024-04-21T23:15:38.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.690+0000 7f5b4a1a3740 20 client.17196 trim_cache size 0 max 16384 2024-04-21T23:15:38.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:15:38.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:15:38.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:15:38.690 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:15:38.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.690+0000 7f5b475bf640 1 -- 172.21.15.44:0/2831030228 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5b2c0a2cf0 msgr2=0x7f5b2c0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:38.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.690+0000 7f5b475bf640 1 --2- 172.21.15.44:0/2831030228 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5b2c0a2cf0 0x7f5b2c0a51b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5b38005c40 tx=0x7f5b38009280 comp rx=0 tx=0).stop 2024-04-21T23:15:38.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.690+0000 7f5b475bf640 1 -- 172.21.15.44:0/2831030228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b40101730 msgr2=0x7f5b40197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:38.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.690+0000 7f5b475bf640 1 --2- 172.21.15.44:0/2831030228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b40101730 0x7f5b40197fd0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c009230 tx=0x7f5b3c02cc70 comp rx=0 tx=0).stop 2024-04-21T23:15:38.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.690+0000 7f5b475bf640 1 -- 172.21.15.44:0/2831030228 shutdown_connections 2024-04-21T23:15:38.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.690+0000 7f5b475bf640 1 --2- 172.21.15.44:0/2831030228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b401020f0 0x7f5b40198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.690+0000 7f5b475bf640 1 --2- 172.21.15.44:0/2831030228 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f5b2c0a2cf0 0x7f5b2c0a51b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.691 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.690+0000 7f5b475bf640 1 --2- 172.21.15.44:0/2831030228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5b40107740 0x7f5b4019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.690+0000 7f5b475bf640 1 --2- 172.21.15.44:0/2831030228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5b40101730 0x7f5b40197fd0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:38.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.690+0000 7f5b475bf640 1 -- 172.21.15.44:0/2831030228 >> 172.21.15.44:0/2831030228 conn(0x7f5b400fd200 msgr2=0x7f5b400fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:38.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.690+0000 7f5b475bf640 1 -- 172.21.15.44:0/2831030228 shutdown_connections 2024-04-21T23:15:38.692 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:38.690+0000 7f5b475bf640 1 -- 172.21.15.44:0/2831030228 wait complete. 2024-04-21T23:15:38.705 INFO:tasks.workunit.client.0.smithi044.stdout: 1 788908 12.42 MB/sec execute 208 sec latency 7319.892 ms 2024-04-21T23:15:39.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 791416 12.43 MB/sec execute 209 sec latency 16.845 ms 2024-04-21T23:15:40.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:39 smithi044 ceph-mon[27934]: pgmap v213: 129 pgs: 129 active+clean; 77 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 272 op/s 2024-04-21T23:15:40.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:39 smithi055 ceph-mon[31117]: pgmap v213: 129 pgs: 129 active+clean; 77 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 272 op/s 2024-04-21T23:15:40.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:39 smithi069 ceph-mon[31816]: pgmap v213: 129 pgs: 129 active+clean; 77 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 272 op/s 2024-04-21T23:15:40.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 792939 12.40 MB/sec execute 210 sec latency 39.548 ms 2024-04-21T23:15:41.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 795596 12.43 MB/sec execute 211 sec latency 33.200 ms 2024-04-21T23:15:42.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:41 smithi044 ceph-mon[27934]: pgmap v214: 129 pgs: 129 active+clean; 77 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 133 op/s 2024-04-21T23:15:42.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:41 smithi055 ceph-mon[31117]: pgmap v214: 129 pgs: 129 active+clean; 77 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 133 op/s 2024-04-21T23:15:42.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:41 smithi069 ceph-mon[31816]: pgmap v214: 129 pgs: 129 active+clean; 77 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 133 op/s 2024-04-21T23:15:42.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 797729 12.40 MB/sec execute 212 sec latency 16.219 ms 2024-04-21T23:15:43.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 799841 12.40 MB/sec execute 213 sec latency 27.122 ms 2024-04-21T23:15:44.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:43 smithi044 ceph-mon[27934]: pgmap v215: 129 pgs: 129 active+clean; 67 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.3 MiB/s wr, 287 op/s 2024-04-21T23:15:44.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:43 smithi055 ceph-mon[31117]: pgmap v215: 129 pgs: 129 active+clean; 67 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.3 MiB/s wr, 287 op/s 2024-04-21T23:15:44.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:43 smithi069 ceph-mon[31816]: pgmap v215: 129 pgs: 129 active+clean; 67 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.3 MiB/s wr, 287 op/s 2024-04-21T23:15:44.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 802632 12.43 MB/sec execute 214 sec latency 18.506 ms 2024-04-21T23:15:45.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 804060 12.40 MB/sec execute 215 sec latency 48.586 ms 2024-04-21T23:15:46.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:45 smithi044 ceph-mon[27934]: pgmap v216: 129 pgs: 129 active+clean; 67 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 1.6 MiB/s wr, 176 op/s 2024-04-21T23:15:46.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:45 smithi055 ceph-mon[31117]: pgmap v216: 129 pgs: 129 active+clean; 67 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 1.6 MiB/s wr, 176 op/s 2024-04-21T23:15:46.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:45 smithi069 ceph-mon[31816]: pgmap v216: 129 pgs: 129 active+clean; 67 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 1.6 MiB/s wr, 176 op/s 2024-04-21T23:15:46.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 806526 12.40 MB/sec execute 216 sec latency 30.302 ms 2024-04-21T23:15:47.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 808885 12.41 MB/sec execute 217 sec latency 17.274 ms 2024-04-21T23:15:48.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:47 smithi044 ceph-mon[27934]: pgmap v217: 129 pgs: 129 active+clean; 70 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 221 op/s 2024-04-21T23:15:48.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:47 smithi055 ceph-mon[31117]: pgmap v217: 129 pgs: 129 active+clean; 70 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 221 op/s 2024-04-21T23:15:48.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:47 smithi069 ceph-mon[31816]: pgmap v217: 129 pgs: 129 active+clean; 70 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 221 op/s 2024-04-21T23:15:48.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 810995 12.40 MB/sec execute 218 sec latency 27.288 ms 2024-04-21T23:15:49.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 813511 12.41 MB/sec execute 219 sec latency 22.034 ms 2024-04-21T23:15:50.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:49 smithi044 ceph-mon[27934]: pgmap v218: 129 pgs: 129 active+clean; 74 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 274 op/s 2024-04-21T23:15:50.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:49 smithi055 ceph-mon[31117]: pgmap v218: 129 pgs: 129 active+clean; 74 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 274 op/s 2024-04-21T23:15:50.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:49 smithi069 ceph-mon[31816]: pgmap v218: 129 pgs: 129 active+clean; 74 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 274 op/s 2024-04-21T23:15:50.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 815135 12.38 MB/sec execute 220 sec latency 109.217 ms 2024-04-21T23:15:51.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 817353 12.39 MB/sec execute 221 sec latency 43.876 ms 2024-04-21T23:15:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.793+0000 7f301e5d2640 1 -- 172.21.15.44:0/67954974 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f301810bd80 msgr2=0x7f301810e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.793+0000 7f301e5d2640 1 --2- 172.21.15.44:0/67954974 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f301810bd80 0x7f301810e1b0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f3010009ca0 tx=0x7f301002c180 comp rx=0 tx=0).stop 2024-04-21T23:15:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.793+0000 7f301e5d2640 1 -- 172.21.15.44:0/67954974 shutdown_connections 2024-04-21T23:15:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.793+0000 7f301e5d2640 1 --2- 172.21.15.44:0/67954974 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f301810bd80 0x7f301810e1b0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.793+0000 7f301e5d2640 1 --2- 172.21.15.44:0/67954974 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30181032d0 0x7f301810b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.793+0000 7f301e5d2640 1 --2- 172.21.15.44:0/67954974 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30181029b0 0x7f3018102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.793+0000 7f301e5d2640 1 -- 172.21.15.44:0/67954974 >> 172.21.15.44:0/67954974 conn(0x7f3018077760 msgr2=0x7f3018077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.793+0000 7f301e5d2640 1 -- 172.21.15.44:0/67954974 shutdown_connections 2024-04-21T23:15:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.793+0000 7f301e5d2640 1 -- 172.21.15.44:0/67954974 wait complete. 2024-04-21T23:15:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.793+0000 7f301e5d2640 1 Processor -- start 2024-04-21T23:15:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.793+0000 7f301e5d2640 1 -- start start 2024-04-21T23:15:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.794+0000 7f301e5d2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30181029b0 0x7f3018100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.794+0000 7f301e5d2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30181032d0 0x7f3018100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.794+0000 7f301e5d2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f301810bd80 0x7f3018101080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.794+0000 7f301e5d2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3018111b10 con 0x7f30181032d0 2024-04-21T23:15:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.794+0000 7f301e5d2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3018111990 con 0x7f30181029b0 2024-04-21T23:15:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.794+0000 7f301e5d2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3018111c90 con 0x7f301810bd80 2024-04-21T23:15:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.794+0000 7f30177fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30181032d0 0x7f3018100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.794+0000 7f30177fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30181032d0 0x7f3018100b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48616/0 (socket says 172.21.15.44:48616) 2024-04-21T23:15:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.794+0000 7f30177fe640 1 -- 172.21.15.44:0/1101260675 learned_addr learned my addr 172.21.15.44:0/1101260675 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.794+0000 7f3017fff640 1 --2- 172.21.15.44:0/1101260675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30181029b0 0x7f3018100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.794+0000 7f301cb6f640 1 --2- 172.21.15.44:0/1101260675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f301810bd80 0x7f3018101080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.794+0000 7f30177fe640 1 -- 172.21.15.44:0/1101260675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f301810bd80 msgr2=0x7f3018101080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.794+0000 7f30177fe640 1 --2- 172.21.15.44:0/1101260675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f301810bd80 0x7f3018101080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.794+0000 7f30177fe640 1 -- 172.21.15.44:0/1101260675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30181029b0 msgr2=0x7f3018100600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.794+0000 7f30177fe640 1 --2- 172.21.15.44:0/1101260675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30181029b0 0x7f3018100600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.794+0000 7f30177fe640 1 -- 172.21.15.44:0/1101260675 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30181a6450 con 0x7f30181032d0 2024-04-21T23:15:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.794+0000 7f3017fff640 1 --2- 172.21.15.44:0/1101260675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30181029b0 0x7f3018100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.795+0000 7f30177fe640 1 --2- 172.21.15.44:0/1101260675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30181032d0 0x7f3018100b40 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f300c00e9f0 tx=0x7f300c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.795+0000 7f30157fa640 1 -- 172.21.15.44:0/1101260675 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c00cd80 con 0x7f30181032d0 2024-04-21T23:15:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.795+0000 7f301e5d2640 1 -- 172.21.15.44:0/1101260675 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30181a6740 con 0x7f30181032d0 2024-04-21T23:15:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.795+0000 7f301e5d2640 1 -- 172.21.15.44:0/1101260675 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30181a6ce0 con 0x7f30181032d0 2024-04-21T23:15:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.795+0000 7f30157fa640 1 -- 172.21.15.44:0/1101260675 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f300c010910 con 0x7f30181032d0 2024-04-21T23:15:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.795+0000 7f30157fa640 1 -- 172.21.15.44:0/1101260675 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c0188f0 con 0x7f30181032d0 2024-04-21T23:15:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.798+0000 7f30157fa640 1 -- 172.21.15.44:0/1101260675 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f300c010ac0 con 0x7f30181032d0 2024-04-21T23:15:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.798+0000 7f30211b7740 1 Processor -- start 2024-04-21T23:15:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.798+0000 7f30211b7740 1 -- start start 2024-04-21T23:15:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.798+0000 7f30211b7740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cebdb20 0x56301cebdf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.798+0000 7f30157fa640 1 --2- 172.21.15.44:0/1101260675 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f30000a2da0 0x7f30000a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.799+0000 7f30211b7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cebe440 0x56301cec6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.799+0000 7f30211b7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cec7210 0x56301cec9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.799+0000 7f30211b7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56301ce21b70 con 0x56301cec7210 2024-04-21T23:15:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.799+0000 7f30211b7740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x56301cdca470 con 0x56301cebdb20 2024-04-21T23:15:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.799+0000 7f30211b7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56301ce1db30 con 0x56301cebe440 2024-04-21T23:15:51.798 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.799+0000 7f301cb6f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cebdb20 0x56301cebdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.799+0000 7f3017fff640 1 --2- 172.21.15.44:0/1101260675 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f30000a2da0 0x7f30000a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.799+0000 7f301cb6f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cebdb20 0x56301cebdf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:53398/0 (socket says 172.21.15.44:53398) 2024-04-21T23:15:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.799+0000 7f301cb6f640 1 -- 172.21.15.44:0/1600562460 learned_addr learned my addr 172.21.15.44:0/1600562460 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.799+0000 7f30157fa640 1 -- 172.21.15.44:0/1101260675 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f300c0cfb00 con 0x7f30181032d0 2024-04-21T23:15:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.799+0000 7f30177fe640 1 --2- 172.21.15.44:0/1600562460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cebe440 0x56301cec6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.799+0000 7f301cb6f640 1 --2- 172.21.15.44:0/1600562460 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cec7210 0x56301cec9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.799+0000 7f30177fe640 1 -- 172.21.15.44:0/1600562460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cebdb20 msgr2=0x56301cebdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.799+0000 7f30177fe640 1 --2- 172.21.15.44:0/1600562460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cebdb20 0x56301cebdf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.799+0000 7f30177fe640 1 -- 172.21.15.44:0/1600562460 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cec7210 msgr2=0x56301cec9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.799+0000 7f30177fe640 1 --2- 172.21.15.44:0/1600562460 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cec7210 0x56301cec9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.799+0000 7f30177fe640 1 -- 172.21.15.44:0/1600562460 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56301cb36210 con 0x56301cebe440 2024-04-21T23:15:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.799+0000 7f3017fff640 1 --2- 172.21.15.44:0/1101260675 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f30000a2da0 0x7f30000a5260 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f30080095f0 tx=0x7f3008005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:51.799 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.799+0000 7f30177fe640 1 --2- 172.21.15.44:0/1600562460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cebe440 0x56301cec6cd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f300c0cce60 tx=0x7f300c09b700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d4309ebc37501e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:51.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.799+0000 7f2ffe7fc640 1 -- 172.21.15.44:0/1600562460 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c09bce0 con 0x56301cebe440 2024-04-21T23:15:51.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.800+0000 7f2ffe7fc640 1 -- 172.21.15.44:0/1600562460 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f300c09ffe0 con 0x56301cebe440 2024-04-21T23:15:51.800 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.800+0000 7f2ffe7fc640 1 -- 172.21.15.44:0/1600562460 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c076510 con 0x56301cebe440 2024-04-21T23:15:51.804 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.800+0000 7f30211b7740 1 -- 172.21.15.44:0/1600562460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cebe440 msgr2=0x56301cec6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.800+0000 7f30211b7740 1 --2- 172.21.15.44:0/1600562460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cebe440 0x56301cec6cd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f300c0cce60 tx=0x7f300c09b700 comp rx=0 tx=0).stop 2024-04-21T23:15:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.800+0000 7f30211b7740 1 -- 172.21.15.44:0/1600562460 shutdown_connections 2024-04-21T23:15:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.800+0000 7f30211b7740 1 --2- 172.21.15.44:0/1600562460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cebe440 0x56301cec6cd0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.800+0000 7f30211b7740 1 --2- 172.21.15.44:0/1600562460 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cec7210 0x56301cec9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.800+0000 7f30211b7740 1 --2- 172.21.15.44:0/1600562460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cebdb20 0x56301cebdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.800+0000 7f30211b7740 1 -- 172.21.15.44:0/1600562460 >> 172.21.15.44:0/1600562460 conn(0x56301ceb8440 msgr2=0x56301ceb8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.800+0000 7f30211b7740 1 -- 172.21.15.44:0/1600562460 shutdown_connections 2024-04-21T23:15:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.800+0000 7f30211b7740 1 -- 172.21.15.44:0/1600562460 wait complete. 2024-04-21T23:15:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.801+0000 7f30211b7740 1 -- start start 2024-04-21T23:15:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.801+0000 7f30211b7740 1 Processor -- start 2024-04-21T23:15:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.801+0000 7f30211b7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebdb20 0x56301cec4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.801+0000 7f30211b7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cebe440 0x56301cec4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.801+0000 7f30211b7740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cec7210 0x56301cec4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.801+0000 7f3017fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebdb20 0x56301cec4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.801+0000 7f30211b7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56301cec9de0 con 0x56301cebdb20 2024-04-21T23:15:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.801+0000 7f30211b7740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x56301cddda10 con 0x56301cec7210 2024-04-21T23:15:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.801+0000 7f30211b7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56301cec9f60 con 0x56301cebe440 2024-04-21T23:15:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.801+0000 7f3017fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebdb20 0x56301cec4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48644/0 (socket says 172.21.15.44:48644) 2024-04-21T23:15:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.801+0000 7f3017fff640 1 -- 172.21.15.44:0/1558046120 learned_addr learned my addr 172.21.15.44:0/1558046120 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.802+0000 7f30211b7740 10 client.0 _ll_register_callbacks cb 0x56301ce18b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.802+0000 7f30177fe640 1 --2- 172.21.15.44:0/1558046120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cebe440 0x56301cec4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.802+0000 7f301cb6f640 1 --2- 172.21.15.44:0/1558046120 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cec7210 0x56301cec4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.802+0000 7f3017fff640 1 -- 172.21.15.44:0/1558046120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cebe440 msgr2=0x56301cec4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.802+0000 7f3017fff640 1 --2- 172.21.15.44:0/1558046120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cebe440 0x56301cec4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.802+0000 7f3017fff640 1 -- 172.21.15.44:0/1558046120 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cec7210 msgr2=0x56301cec4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:51.806 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.802+0000 7f3017fff640 1 --2- 172.21.15.44:0/1558046120 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cec7210 0x56301cec4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.802+0000 7f3017fff640 1 -- 172.21.15.44:0/1558046120 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56301cc02620 con 0x56301cebdb20 2024-04-21T23:15:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.802+0000 7f3017fff640 1 --2- 172.21.15.44:0/1558046120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebdb20 0x56301cec4250 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f300800fd60 tx=0x7f3008004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e68b13aacf47ad66 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.802+0000 7f2ff77fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:15:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.802+0000 7f2ff77fe640 1 -- 172.21.15.44:0/1558046120 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300800eab0 con 0x56301cebdb20 2024-04-21T23:15:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.802+0000 7f30211b7740 1 -- 172.21.15.44:0/1558046120 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f300c09ffe0 con 0x56301cebdb20 2024-04-21T23:15:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.803+0000 7f2ff77fe640 1 -- 172.21.15.44:0/1558046120 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f300800ec50 con 0x56301cebdb20 2024-04-21T23:15:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.803+0000 7f2ff77fe640 1 -- 172.21.15.44:0/1558046120 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f300800ee50 con 0x56301cebdb20 2024-04-21T23:15:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.803+0000 7f30211b7740 10 client.17208 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.803+0000 7f30211b7740 1 -- 172.21.15.44:0/1558046120 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56301ced6110 con 0x56301cebdb20 2024-04-21T23:15:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.803+0000 7f2ff77fe640 1 -- 172.21.15.44:0/1558046120 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f300c09ffe0 con 0x56301cebdb20 2024-04-21T23:15:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.804+0000 7f2ff77fe640 1 -- 172.21.15.44:0/1558046120 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f300801d670 con 0x56301cebdb20 2024-04-21T23:15:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.804+0000 7f30211b7740 10 client.17208 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.804+0000 7f30211b7740 10 client.17208 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:51.807 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.804+0000 7f30211b7740 20 client.17208 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.804+0000 7f30211b7740 1 --2- 172.21.15.44:0/1558046120 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56301ced75d0 0x56301ced9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.804+0000 7f30211b7740 4 client.17208 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:15:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.804+0000 7f30211b7740 1 -- 172.21.15.44:0/1558046120 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56301ce2f0b0 con 0x56301ced75d0 2024-04-21T23:15:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.804+0000 7f30177fe640 1 --2- 172.21.15.44:0/1558046120 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56301ced75d0 0x56301ced9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.805+0000 7f2ff77fe640 10 client.17208 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.805+0000 7f30177fe640 1 --2- 172.21.15.44:0/1558046120 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56301ced75d0 0x56301ced9a60 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52c6611571076c38 server_cookie=a573bba80c7ded77 in_seq=0 out_seq=0 2024-04-21T23:15:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.806+0000 7f2ff77fe640 1 -- 172.21.15.44:0/1558046120 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f300c06dae0 con 0x56301ced75d0 2024-04-21T23:15:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.806+0000 7f2ff77fe640 10 client.17208 handle_command_reply: tid=0 2024-04-21T23:15:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.806+0000 7f30211b7740 1 client.17208 shutdown 2024-04-21T23:15:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.806+0000 7f30211b7740 1 -- 172.21.15.44:0/1558046120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebdb20 msgr2=0x56301cec4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.806+0000 7f30211b7740 1 --2- 172.21.15.44:0/1558046120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebdb20 0x56301cec4250 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f300800fd60 tx=0x7f3008004730 comp rx=0 tx=0).stop 2024-04-21T23:15:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.806+0000 7f30211b7740 1 -- 172.21.15.44:0/1558046120 shutdown_connections 2024-04-21T23:15:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.806+0000 7f30211b7740 1 --2- 172.21.15.44:0/1558046120 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56301ced75d0 0x56301ced9a60 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.806+0000 7f30211b7740 1 --2- 172.21.15.44:0/1558046120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cebe440 0x56301cec4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.808 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.806+0000 7f30211b7740 1 --2- 172.21.15.44:0/1558046120 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cec7210 0x56301cec4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.806+0000 7f30211b7740 1 --2- 172.21.15.44:0/1558046120 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebdb20 0x56301cec4250 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.806+0000 7f2ff77fe640 0 client.17208 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.806+0000 7f30211b7740 1 -- 172.21.15.44:0/1558046120 >> 172.21.15.44:0/1558046120 conn(0x56301ceb8440 msgr2=0x56301ceba190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.806+0000 7f30211b7740 1 -- 172.21.15.44:0/1558046120 shutdown_connections 2024-04-21T23:15:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.806+0000 7f30211b7740 1 -- 172.21.15.44:0/1558046120 wait complete. 2024-04-21T23:15:51.809 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.806+0000 7f30211b7740 20 client.17208 trim_cache size 0 max 16384 2024-04-21T23:15:51.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.823+0000 7f30211b7740 1 Processor -- start 2024-04-21T23:15:51.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.823+0000 7f30211b7740 1 -- start start 2024-04-21T23:15:51.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.823+0000 7f30211b7740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cebdb20 0x56301cec6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:51.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.823+0000 7f30211b7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebe440 0x56301ced2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:51.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.823+0000 7f30211b7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cec7210 0x56301ceb3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:51.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.823+0000 7f30211b7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f30181af010 con 0x56301cebe440 2024-04-21T23:15:51.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.823+0000 7f30211b7740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f30181b1870 con 0x56301cebdb20 2024-04-21T23:15:51.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.823+0000 7f30211b7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f30181af660 con 0x56301cec7210 2024-04-21T23:15:51.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.824+0000 7f30177fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebe440 0x56301ced2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:51.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.824+0000 7f30177fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebe440 0x56301ced2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48654/0 (socket says 172.21.15.44:48654) 2024-04-21T23:15:51.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.824+0000 7f3017fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cebdb20 0x56301cec6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:51.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.824+0000 7f30177fe640 1 -- 172.21.15.44:0/3692157457 learned_addr learned my addr 172.21.15.44:0/3692157457 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:51.824 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.824+0000 7f301cb6f640 1 --2- 172.21.15.44:0/3692157457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cec7210 0x56301ceb3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.824+0000 7f30177fe640 1 -- 172.21.15.44:0/3692157457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cec7210 msgr2=0x56301ceb3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.824+0000 7f30177fe640 1 --2- 172.21.15.44:0/3692157457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cec7210 0x56301ceb3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.824+0000 7f30177fe640 1 -- 172.21.15.44:0/3692157457 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cebdb20 msgr2=0x56301cec6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.824+0000 7f30177fe640 1 --2- 172.21.15.44:0/3692157457 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cebdb20 0x56301cec6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.824+0000 7f30177fe640 1 -- 172.21.15.44:0/3692157457 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56301cec3240 con 0x56301cebe440 2024-04-21T23:15:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.824+0000 7f30177fe640 1 --2- 172.21.15.44:0/3692157457 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebe440 0x56301ced2ca0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f300c09b9e0 tx=0x7f300c0cf040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4456b15e9dd3ed83 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.824+0000 7f2ffdffb640 1 -- 172.21.15.44:0/3692157457 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c09ffe0 con 0x56301cebe440 2024-04-21T23:15:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.824+0000 7f2ffdffb640 1 -- 172.21.15.44:0/3692157457 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56301ce2f0b0 con 0x56301cebe440 2024-04-21T23:15:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.824+0000 7f2ffdffb640 1 -- 172.21.15.44:0/3692157457 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c09b390 con 0x56301cebe440 2024-04-21T23:15:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.825+0000 7f30211b7740 1 -- 172.21.15.44:0/3692157457 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebe440 msgr2=0x56301ced2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.825+0000 7f30211b7740 1 --2- 172.21.15.44:0/3692157457 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebe440 0x56301ced2ca0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f300c09b9e0 tx=0x7f300c0cf040 comp rx=0 tx=0).stop 2024-04-21T23:15:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.825+0000 7f30211b7740 1 -- 172.21.15.44:0/3692157457 shutdown_connections 2024-04-21T23:15:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.825+0000 7f30211b7740 1 --2- 172.21.15.44:0/3692157457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cec7210 0x56301ceb3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.825+0000 7f30211b7740 1 --2- 172.21.15.44:0/3692157457 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebe440 0x56301ced2ca0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.825+0000 7f30211b7740 1 --2- 172.21.15.44:0/3692157457 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cebdb20 0x56301cec6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.825+0000 7f30211b7740 1 -- 172.21.15.44:0/3692157457 >> 172.21.15.44:0/3692157457 conn(0x56301ceb8440 msgr2=0x56301cec4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.825+0000 7f30211b7740 1 -- 172.21.15.44:0/3692157457 shutdown_connections 2024-04-21T23:15:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.825+0000 7f30211b7740 1 -- 172.21.15.44:0/3692157457 wait complete. 2024-04-21T23:15:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.825+0000 7f30211b7740 1 -- start start 2024-04-21T23:15:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.826+0000 7f30211b7740 1 Processor -- start 2024-04-21T23:15:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.826+0000 7f30211b7740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebdb20 0x56301cecf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.826+0000 7f30211b7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cebe440 0x56301cecf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.826+0000 7f30211b7740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cec7210 0x56301cecc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.826+0000 7f30211b7740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f30181afd30 con 0x56301cebdb20 2024-04-21T23:15:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.826+0000 7f30211b7740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f30181b11a0 con 0x56301cec7210 2024-04-21T23:15:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.826+0000 7f30211b7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f30181b0400 con 0x56301cebe440 2024-04-21T23:15:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.826+0000 7f30211b7740 10 client.0 _ll_register_callbacks cb 0x7f30080094b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.826+0000 7f3017fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebdb20 0x56301cecf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.826+0000 7f301cb6f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cec7210 0x56301cecc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.826+0000 7f3017fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebdb20 0x56301cecf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48668/0 (socket says 172.21.15.44:48668) 2024-04-21T23:15:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.826+0000 7f3017fff640 1 -- 172.21.15.44:0/3440359608 learned_addr learned my addr 172.21.15.44:0/3440359608 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.826+0000 7f30177fe640 1 --2- 172.21.15.44:0/3440359608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cebe440 0x56301cecf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.827+0000 7f301cb6f640 1 -- 172.21.15.44:0/3440359608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cebe440 msgr2=0x56301cecf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.827+0000 7f301cb6f640 1 --2- 172.21.15.44:0/3440359608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cebe440 0x56301cecf6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.827+0000 7f301cb6f640 1 -- 172.21.15.44:0/3440359608 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebdb20 msgr2=0x56301cecf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.827+0000 7f301cb6f640 1 --2- 172.21.15.44:0/3440359608 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebdb20 0x56301cecf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.827+0000 7f301cb6f640 1 -- 172.21.15.44:0/3440359608 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56301cecc9f0 con 0x56301cec7210 2024-04-21T23:15:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.827+0000 7f3017fff640 1 --2- 172.21.15.44:0/3440359608 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebdb20 0x56301cecf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.827+0000 7f301cb6f640 1 --2- 172.21.15.44:0/3440359608 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cec7210 0x56301cecc1c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3010032610 tx=0x7f3010032ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=daca05211f091119 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.827+0000 7f2ff77fe640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:15:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.827+0000 7f2ff77fe640 1 -- 172.21.15.44:0/3440359608 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3010039040 con 0x56301cec7210 2024-04-21T23:15:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.827+0000 7f2ff77fe640 1 -- 172.21.15.44:0/3440359608 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f3010034440 con 0x56301cec7210 2024-04-21T23:15:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.827+0000 7f30211b7740 1 -- 172.21.15.44:0/3440359608 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56301ce2f0b0 con 0x56301cec7210 2024-04-21T23:15:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.827+0000 7f2ff77fe640 1 -- 172.21.15.44:0/3440359608 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f301004d640 con 0x56301cec7210 2024-04-21T23:15:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.827+0000 7f30211b7740 10 client.26771 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.827+0000 7f30211b7740 1 -- 172.21.15.44:0/3440359608 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56301cf3e700 con 0x56301cec7210 2024-04-21T23:15:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.827+0000 7f2ff77fe640 1 -- 172.21.15.44:0/3440359608 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56301ce2f0b0 con 0x56301cec7210 2024-04-21T23:15:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.828+0000 7f2ff77fe640 1 -- 172.21.15.44:0/3440359608 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f301004b070 con 0x56301cec7210 2024-04-21T23:15:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.828+0000 7f30211b7740 10 client.26771 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.828+0000 7f30211b7740 10 client.26771 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.828+0000 7f30211b7740 20 client.26771 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.828+0000 7f30211b7740 1 --2- 172.21.15.44:0/3440359608 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56301cf3ee20 0x56301cf41210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.828+0000 7f30211b7740 4 client.26771 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:15:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.828+0000 7f30211b7740 1 -- 172.21.15.44:0/3440359608 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56301ceb3900 con 0x56301cf3ee20 2024-04-21T23:15:51.832 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.828+0000 7f3017fff640 1 --2- 172.21.15.44:0/3440359608 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56301cf3ee20 0x56301cf41210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:51.832 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.828+0000 7f3017fff640 1 --2- 172.21.15.44:0/3440359608 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56301cf3ee20 0x56301cf41210 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21f3f30225deeab4 server_cookie=e629dc68f3e72f94 in_seq=0 out_seq=0 2024-04-21T23:15:51.832 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:51.828+0000 7f2ff77fe640 10 client.26771 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:52.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.151+0000 7f2ff77fe640 1 -- 172.21.15.44:0/3440359608 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56301ceb3900 con 0x56301cf3ee20 2024-04-21T23:15:52.152 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.152+0000 7f2ff77fe640 10 client.26771 handle_command_reply: tid=0 2024-04-21T23:15:52.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.152+0000 7f30211b7740 1 client.26771 shutdown 2024-04-21T23:15:52.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.152+0000 7f30211b7740 1 -- 172.21.15.44:0/3440359608 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cec7210 msgr2=0x56301cecc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.152+0000 7f30211b7740 1 --2- 172.21.15.44:0/3440359608 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cec7210 0x56301cecc1c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3010032610 tx=0x7f3010032ba0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.152+0000 7f30211b7740 1 -- 172.21.15.44:0/3440359608 shutdown_connections 2024-04-21T23:15:52.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.152+0000 7f30211b7740 1 --2- 172.21.15.44:0/3440359608 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56301cf3ee20 0x56301cf41210 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.152+0000 7f30211b7740 1 --2- 172.21.15.44:0/3440359608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56301cebe440 0x56301cecf6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.152+0000 7f2ff77fe640 0 client.26771 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:52.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.152+0000 7f30211b7740 1 --2- 172.21.15.44:0/3440359608 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56301cec7210 0x56301cecc1c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.152+0000 7f30211b7740 1 --2- 172.21.15.44:0/3440359608 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56301cebdb20 0x56301cecf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.152+0000 7f30211b7740 1 -- 172.21.15.44:0/3440359608 >> 172.21.15.44:0/3440359608 conn(0x56301ceb8440 msgr2=0x56301ceb6c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:52.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.153+0000 7f30211b7740 1 -- 172.21.15.44:0/3440359608 shutdown_connections 2024-04-21T23:15:52.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.153+0000 7f30211b7740 1 -- 172.21.15.44:0/3440359608 wait complete. 2024-04-21T23:15:52.153 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.153+0000 7f30211b7740 20 client.26771 trim_cache size 0 max 16384 2024-04-21T23:15:52.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:15:52.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:15:52.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:15:52.154 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:15:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.153+0000 7f301e5d2640 1 -- 172.21.15.44:0/1101260675 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f30000a2da0 msgr2=0x7f30000a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.153+0000 7f301e5d2640 1 --2- 172.21.15.44:0/1101260675 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f30000a2da0 0x7f30000a5260 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f30080095f0 tx=0x7f3008005f90 comp rx=0 tx=0).stop 2024-04-21T23:15:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.154+0000 7f301e5d2640 1 -- 172.21.15.44:0/1101260675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30181032d0 msgr2=0x7f3018100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.154+0000 7f301e5d2640 1 --2- 172.21.15.44:0/1101260675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30181032d0 0x7f3018100b40 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f300c00e9f0 tx=0x7f300c00eeb0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.154+0000 7f301e5d2640 1 -- 172.21.15.44:0/1101260675 shutdown_connections 2024-04-21T23:15:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.154+0000 7f301e5d2640 1 --2- 172.21.15.44:0/1101260675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f301810bd80 0x7f3018101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.154+0000 7f301e5d2640 1 --2- 172.21.15.44:0/1101260675 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f30000a2da0 0x7f30000a5260 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.154+0000 7f301e5d2640 1 --2- 172.21.15.44:0/1101260675 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f30181032d0 0x7f3018100b40 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.154+0000 7f301e5d2640 1 --2- 172.21.15.44:0/1101260675 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30181029b0 0x7f3018100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.154+0000 7f301e5d2640 1 -- 172.21.15.44:0/1101260675 >> 172.21.15.44:0/1101260675 conn(0x7f3018077760 msgr2=0x7f30180fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.154+0000 7f301e5d2640 1 -- 172.21.15.44:0/1101260675 shutdown_connections 2024-04-21T23:15:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:15:52.154+0000 7f301e5d2640 1 -- 172.21.15.44:0/1101260675 wait complete. 2024-04-21T23:15:52.184 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:51 smithi044 ceph-mon[27934]: pgmap v219: 129 pgs: 129 active+clean; 74 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 263 op/s 2024-04-21T23:15:52.213 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:51 smithi055 ceph-mon[31117]: pgmap v219: 129 pgs: 129 active+clean; 74 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 263 op/s 2024-04-21T23:15:52.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:51 smithi069 ceph-mon[31816]: pgmap v219: 129 pgs: 129 active+clean; 74 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 263 op/s 2024-04-21T23:15:52.706 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:15:52.706 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:15:52.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 819048 12.36 MB/sec execute 222 sec latency 138.958 ms 2024-04-21T23:15:52.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.817+0000 7fc238a5f640 1 -- 172.21.15.44:0/2235641467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2341032d0 msgr2=0x7fc23410b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.817+0000 7fc238a5f640 1 --2- 172.21.15.44:0/2235641467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2341032d0 0x7fc23410b7b0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fc21c009900 tx=0x7fc21c02bcd0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.817+0000 7fc238a5f640 1 -- 172.21.15.44:0/2235641467 shutdown_connections 2024-04-21T23:15:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.817+0000 7fc238a5f640 1 --2- 172.21.15.44:0/2235641467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc23410bcf0 0x7fc23410e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.817+0000 7fc238a5f640 1 --2- 172.21.15.44:0/2235641467 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2341032d0 0x7fc23410b7b0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.817+0000 7fc238a5f640 1 --2- 172.21.15.44:0/2235641467 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2341029b0 0x7fc234102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.817+0000 7fc238a5f640 1 -- 172.21.15.44:0/2235641467 >> 172.21.15.44:0/2235641467 conn(0x7fc2340777f0 msgr2=0x7fc234077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.817+0000 7fc238a5f640 1 -- 172.21.15.44:0/2235641467 shutdown_connections 2024-04-21T23:15:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.817+0000 7fc238a5f640 1 -- 172.21.15.44:0/2235641467 wait complete. 2024-04-21T23:15:52.818 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.818+0000 7fc238a5f640 1 Processor -- start 2024-04-21T23:15:52.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.818+0000 7fc238a5f640 1 -- start start 2024-04-21T23:15:52.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.818+0000 7fc238a5f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2341029b0 0x7fc234100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.819+0000 7fc238a5f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2341032d0 0x7fc234100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.819+0000 7fc238a5f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc23410bcf0 0x7fc234101080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.819+0000 7fc238a5f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc234111af0 con 0x7fc2341029b0 2024-04-21T23:15:52.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.819+0000 7fc238a5f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc234111970 con 0x7fc23410bcf0 2024-04-21T23:15:52.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.819+0000 7fc238a5f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc234111c70 con 0x7fc2341032d0 2024-04-21T23:15:52.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.819+0000 7fc23259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2341029b0 0x7fc234100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.819+0000 7fc23259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2341029b0 0x7fc234100600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48684/0 (socket says 172.21.15.44:48684) 2024-04-21T23:15:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.819+0000 7fc23259c640 1 -- 172.21.15.44:0/2333477917 learned_addr learned my addr 172.21.15.44:0/2333477917 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.819+0000 7fc229d9b640 1 --2- 172.21.15.44:0/2333477917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2341032d0 0x7fc234100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.819+0000 7fc232d9d640 1 --2- 172.21.15.44:0/2333477917 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc23410bcf0 0x7fc234101080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.819+0000 7fc23259c640 1 -- 172.21.15.44:0/2333477917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2341032d0 msgr2=0x7fc234100b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.819+0000 7fc23259c640 1 --2- 172.21.15.44:0/2333477917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2341032d0 0x7fc234100b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.819+0000 7fc23259c640 1 -- 172.21.15.44:0/2333477917 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc23410bcf0 msgr2=0x7fc234101080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.819+0000 7fc23259c640 1 --2- 172.21.15.44:0/2333477917 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc23410bcf0 0x7fc234101080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.819+0000 7fc23259c640 1 -- 172.21.15.44:0/2333477917 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2341a6490 con 0x7fc2341029b0 2024-04-21T23:15:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.819+0000 7fc229d9b640 1 --2- 172.21.15.44:0/2333477917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2341032d0 0x7fc234100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.819+0000 7fc232d9d640 1 --2- 172.21.15.44:0/2333477917 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc23410bcf0 0x7fc234101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.819+0000 7fc23259c640 1 --2- 172.21.15.44:0/2333477917 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2341029b0 0x7fc234100600 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fc22400e9f0 tx=0x7fc22400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:52.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.819+0000 7fc22bfff640 1 -- 172.21.15.44:0/2333477917 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc22400cd80 con 0x7fc2341029b0 2024-04-21T23:15:52.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.820+0000 7fc22bfff640 1 -- 172.21.15.44:0/2333477917 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc224010910 con 0x7fc2341029b0 2024-04-21T23:15:52.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.820+0000 7fc22bfff640 1 -- 172.21.15.44:0/2333477917 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc224018990 con 0x7fc2341029b0 2024-04-21T23:15:52.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.820+0000 7fc238a5f640 1 -- 172.21.15.44:0/2333477917 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2341a6780 con 0x7fc2341029b0 2024-04-21T23:15:52.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.820+0000 7fc238a5f640 1 -- 172.21.15.44:0/2333477917 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2341a6c50 con 0x7fc2341029b0 2024-04-21T23:15:52.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.822+0000 7fc22bfff640 1 -- 172.21.15.44:0/2333477917 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc224010ac0 con 0x7fc2341029b0 2024-04-21T23:15:52.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.822+0000 7fc23b642740 1 Processor -- start 2024-04-21T23:15:52.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc23b642740 1 -- start start 2024-04-21T23:15:52.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc23b642740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391dbef30 0x563391dbf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc23b642740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbf850 0x563391dc7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc22bfff640 1 --2- 172.21.15.44:0/2333477917 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc2200a2d30 0x7fc2200a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc23b642740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391dc8430 0x563391dca820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc23b642740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563391d23560 con 0x563391dbef30 2024-04-21T23:15:52.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc23b642740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x563391ccb730 con 0x563391dc8430 2024-04-21T23:15:52.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc23b642740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563391d1f520 con 0x563391dbf850 2024-04-21T23:15:52.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc229d9b640 1 --2- 172.21.15.44:0/2333477917 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc2200a2d30 0x7fc2200a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc232d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391dbef30 0x563391dbf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc23259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbf850 0x563391dc7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc232d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391dbef30 0x563391dbf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48694/0 (socket says 172.21.15.44:48694) 2024-04-21T23:15:52.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc232d9d640 1 -- 172.21.15.44:0/240302943 learned_addr learned my addr 172.21.15.44:0/240302943 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:52.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc232d9d640 1 --2- 172.21.15.44:0/240302943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391dc8430 0x563391dca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc22bfff640 1 -- 172.21.15.44:0/2333477917 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7fc2240cfaf0 con 0x7fc2341029b0 2024-04-21T23:15:52.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc23259c640 1 -- 172.21.15.44:0/240302943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391dc8430 msgr2=0x563391dca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:52.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc23259c640 1 --2- 172.21.15.44:0/240302943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391dc8430 0x563391dca820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc23259c640 1 -- 172.21.15.44:0/240302943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391dbef30 msgr2=0x563391dbf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:52.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc23259c640 1 --2- 172.21.15.44:0/240302943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391dbef30 0x563391dbf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc23259c640 1 -- 172.21.15.44:0/240302943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563391b017d0 con 0x563391dbf850 2024-04-21T23:15:52.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.823+0000 7fc232d9d640 1 --2- 172.21.15.44:0/240302943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391dc8430 0x563391dca820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:15:52.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.824+0000 7fc229d9b640 1 --2- 172.21.15.44:0/2333477917 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc2200a2d30 0x7fc2200a51f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc21c02efd0 tx=0x7fc21c02cf90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:52.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.824+0000 7fc23259c640 1 --2- 172.21.15.44:0/240302943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbf850 0x563391dc7ef0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc2240cce50 tx=0x7fc22409b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc68734729325bbe server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:52.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.824+0000 7fc228d99640 1 -- 172.21.15.44:0/240302943 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc22409bcd0 con 0x563391dbf850 2024-04-21T23:15:52.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.824+0000 7fc228d99640 1 -- 172.21.15.44:0/240302943 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc22409ffd0 con 0x563391dbf850 2024-04-21T23:15:52.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.824+0000 7fc228d99640 1 -- 172.21.15.44:0/240302943 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2240a4660 con 0x563391dbf850 2024-04-21T23:15:52.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.825+0000 7fc23b642740 1 -- 172.21.15.44:0/240302943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbf850 msgr2=0x563391dc7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.825+0000 7fc23b642740 1 --2- 172.21.15.44:0/240302943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbf850 0x563391dc7ef0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc2240cce50 tx=0x7fc22409b8d0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.825+0000 7fc23b642740 1 -- 172.21.15.44:0/240302943 shutdown_connections 2024-04-21T23:15:52.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.825+0000 7fc23b642740 1 --2- 172.21.15.44:0/240302943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbf850 0x563391dc7ef0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.825+0000 7fc23b642740 1 --2- 172.21.15.44:0/240302943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391dc8430 0x563391dca820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.825+0000 7fc23b642740 1 --2- 172.21.15.44:0/240302943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391dbef30 0x563391dbf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.825+0000 7fc23b642740 1 -- 172.21.15.44:0/240302943 >> 172.21.15.44:0/240302943 conn(0x563391db9660 msgr2=0x563391db9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:52.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.825+0000 7fc23b642740 1 -- 172.21.15.44:0/240302943 shutdown_connections 2024-04-21T23:15:52.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.825+0000 7fc23b642740 1 -- 172.21.15.44:0/240302943 wait complete. 2024-04-21T23:15:52.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.825+0000 7fc23b642740 1 -- start start 2024-04-21T23:15:52.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.825+0000 7fc23b642740 1 Processor -- start 2024-04-21T23:15:52.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc23b642740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbef30 0x563391dc5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc23b642740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391dbf850 0x563391dc5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc23b642740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391dc8430 0x563391dc6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc23b642740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563391dcb220 con 0x563391dbf850 2024-04-21T23:15:52.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc23b642740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x563391cdee00 con 0x563391dc8430 2024-04-21T23:15:52.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc23b642740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563391dcb3a0 con 0x563391dbef30 2024-04-21T23:15:52.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc23259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbef30 0x563391dc5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc229d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391dbf850 0x563391dc5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc23b642740 10 client.0 _ll_register_callbacks cb 0x563391d19eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:52.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc23259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbef30 0x563391dc5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:55298/0 (socket says 172.21.15.44:55298) 2024-04-21T23:15:52.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc23259c640 1 -- 172.21.15.44:0/1915347309 learned_addr learned my addr 172.21.15.44:0/1915347309 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:52.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc232d9d640 1 --2- 172.21.15.44:0/1915347309 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391dc8430 0x563391dc6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc23259c640 1 -- 172.21.15.44:0/1915347309 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391dc8430 msgr2=0x563391dc6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:52.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc23259c640 1 --2- 172.21.15.44:0/1915347309 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391dc8430 0x563391dc6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc23259c640 1 -- 172.21.15.44:0/1915347309 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391dbf850 msgr2=0x563391dc5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:52.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc23259c640 1 --2- 172.21.15.44:0/1915347309 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391dbf850 0x563391dc5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc23259c640 1 -- 172.21.15.44:0/1915347309 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563391a373f0 con 0x563391dbef30 2024-04-21T23:15:52.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc229d9b640 1 --2- 172.21.15.44:0/1915347309 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391dbf850 0x563391dc5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:52.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc23259c640 1 --2- 172.21.15.44:0/1915347309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbef30 0x563391dc5580 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc224098130 tx=0x7fc22409b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5d6df95e489df9b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:52.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc219ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:15:52.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc219ffb640 1 -- 172.21.15.44:0/1915347309 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2240cd260 con 0x563391dbef30 2024-04-21T23:15:52.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc219ffb640 1 -- 172.21.15.44:0/1915347309 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7fc224018d40 con 0x563391dbef30 2024-04-21T23:15:52.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.826+0000 7fc23b642740 1 -- 172.21.15.44:0/1915347309 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc22409ffd0 con 0x563391dbef30 2024-04-21T23:15:52.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.827+0000 7fc219ffb640 1 -- 172.21.15.44:0/1915347309 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc224098340 con 0x563391dbef30 2024-04-21T23:15:52.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.827+0000 7fc23b642740 10 client.26764 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:52.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.827+0000 7fc23b642740 1 -- 172.21.15.44:0/1915347309 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5633919f70f0 con 0x563391dbef30 2024-04-21T23:15:52.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.827+0000 7fc219ffb640 1 -- 172.21.15.44:0/1915347309 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc22409ffd0 con 0x563391dbef30 2024-04-21T23:15:52.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.827+0000 7fc219ffb640 1 -- 172.21.15.44:0/1915347309 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7fc22409ab90 con 0x563391dbef30 2024-04-21T23:15:52.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.827+0000 7fc23b642740 10 client.26764 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:52.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.827+0000 7fc23b642740 10 client.26764 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:52.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.827+0000 7fc23b642740 20 client.26764 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:52.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.827+0000 7fc23b642740 1 --2- 172.21.15.44:0/1915347309 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563391dd8770 0x563391ddab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.827+0000 7fc23b642740 4 client.26764 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:15:52.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.827+0000 7fc23b642740 1 -- 172.21.15.44:0/1915347309 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563391d30aa0 con 0x563391dd8770 2024-04-21T23:15:52.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.827+0000 7fc229d9b640 1 --2- 172.21.15.44:0/1915347309 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563391dd8770 0x563391ddab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.828+0000 7fc229d9b640 1 --2- 172.21.15.44:0/1915347309 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563391dd8770 0x563391ddab60 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ce6c4f3717ff55f server_cookie=139b10b8046152f3 in_seq=0 out_seq=0 2024-04-21T23:15:52.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.828+0000 7fc219ffb640 10 client.26764 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:52.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.829+0000 7fc219ffb640 1 -- 172.21.15.44:0/1915347309 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563391d30aa0 con 0x563391dd8770 2024-04-21T23:15:52.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.829+0000 7fc219ffb640 10 client.26764 handle_command_reply: tid=0 2024-04-21T23:15:52.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.829+0000 7fc23b642740 1 client.26764 shutdown 2024-04-21T23:15:52.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.829+0000 7fc23b642740 1 -- 172.21.15.44:0/1915347309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbef30 msgr2=0x563391dc5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.829+0000 7fc23b642740 1 --2- 172.21.15.44:0/1915347309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbef30 0x563391dc5580 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc224098130 tx=0x7fc22409b330 comp rx=0 tx=0).stop 2024-04-21T23:15:52.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.829+0000 7fc23b642740 1 -- 172.21.15.44:0/1915347309 shutdown_connections 2024-04-21T23:15:52.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.829+0000 7fc23b642740 1 --2- 172.21.15.44:0/1915347309 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563391dd8770 0x563391ddab60 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.829+0000 7fc23b642740 1 --2- 172.21.15.44:0/1915347309 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391dc8430 0x563391dc6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.829+0000 7fc219ffb640 0 client.26764 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:52.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.829+0000 7fc23b642740 1 --2- 172.21.15.44:0/1915347309 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391dbf850 0x563391dc5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.829+0000 7fc23b642740 1 --2- 172.21.15.44:0/1915347309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbef30 0x563391dc5580 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.829+0000 7fc23b642740 1 -- 172.21.15.44:0/1915347309 >> 172.21.15.44:0/1915347309 conn(0x563391db9660 msgr2=0x563391dbb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:52.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.830+0000 7fc23b642740 1 -- 172.21.15.44:0/1915347309 shutdown_connections 2024-04-21T23:15:52.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.830+0000 7fc23b642740 1 -- 172.21.15.44:0/1915347309 wait complete. 2024-04-21T23:15:52.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.830+0000 7fc23b642740 20 client.26764 trim_cache size 0 max 16384 2024-04-21T23:15:52.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.849+0000 7fc23b642740 1 Processor -- start 2024-04-21T23:15:52.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.849+0000 7fc23b642740 1 -- start start 2024-04-21T23:15:52.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.849+0000 7fc23b642740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391db9660 0x563391dc7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.849+0000 7fc23b642740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391dbef30 0x563391dddb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.849+0000 7fc23b642740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbf850 0x563391dde0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.849+0000 7fc23b642740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc2341aef40 con 0x563391dbef30 2024-04-21T23:15:52.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.849+0000 7fc23b642740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc2341b17a0 con 0x563391db9660 2024-04-21T23:15:52.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.849+0000 7fc23b642740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc2341af590 con 0x563391dbf850 2024-04-21T23:15:52.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.849+0000 7fc229d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391dbef30 0x563391dddb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.849+0000 7fc23259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391db9660 0x563391dc7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.849+0000 7fc232d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbf850 0x563391dde0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.849+0000 7fc23259c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391db9660 0x563391dc7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:53462/0 (socket says 172.21.15.44:53462) 2024-04-21T23:15:52.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.849+0000 7fc229d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391dbef30 0x563391dddb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48722/0 (socket says 172.21.15.44:48722) 2024-04-21T23:15:52.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.849+0000 7fc23259c640 1 -- 172.21.15.44:0/1668013993 learned_addr learned my addr 172.21.15.44:0/1668013993 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:52.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.850+0000 7fc229d9b640 1 -- 172.21.15.44:0/1668013993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbf850 msgr2=0x563391dde0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:52.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.850+0000 7fc229d9b640 1 --2- 172.21.15.44:0/1668013993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbf850 0x563391dde0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.850+0000 7fc229d9b640 1 -- 172.21.15.44:0/1668013993 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391db9660 msgr2=0x563391dc7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.850+0000 7fc229d9b640 1 --2- 172.21.15.44:0/1668013993 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391db9660 0x563391dc7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.850+0000 7fc229d9b640 1 -- 172.21.15.44:0/1668013993 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563391dc3b10 con 0x563391dbef30 2024-04-21T23:15:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.850+0000 7fc229d9b640 1 --2- 172.21.15.44:0/1668013993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391dbef30 0x563391dddb80 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fc21c000c00 tx=0x7fc21c03f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c29190cabb3434c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.850+0000 7fc21bfff640 1 -- 172.21.15.44:0/1668013993 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21c02cd40 con 0x563391dbef30 2024-04-21T23:15:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.850+0000 7fc21bfff640 1 -- 172.21.15.44:0/1668013993 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc21c0553b0 con 0x563391dbef30 2024-04-21T23:15:52.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.850+0000 7fc21bfff640 1 -- 172.21.15.44:0/1668013993 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21c005390 con 0x563391dbef30 2024-04-21T23:15:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.851+0000 7fc23b642740 1 -- 172.21.15.44:0/1668013993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391dbef30 msgr2=0x563391dddb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.851+0000 7fc23b642740 1 --2- 172.21.15.44:0/1668013993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391dbef30 0x563391dddb80 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fc21c000c00 tx=0x7fc21c03f000 comp rx=0 tx=0).stop 2024-04-21T23:15:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.851+0000 7fc23b642740 1 -- 172.21.15.44:0/1668013993 shutdown_connections 2024-04-21T23:15:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.851+0000 7fc23b642740 1 --2- 172.21.15.44:0/1668013993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbf850 0x563391dde0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.851+0000 7fc23b642740 1 --2- 172.21.15.44:0/1668013993 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391dbef30 0x563391dddb80 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.851+0000 7fc23b642740 1 --2- 172.21.15.44:0/1668013993 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391db9660 0x563391dc7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.851+0000 7fc23b642740 1 -- 172.21.15.44:0/1668013993 >> 172.21.15.44:0/1668013993 conn(0x563391dc00e0 msgr2=0x563391db7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.851+0000 7fc23b642740 1 -- 172.21.15.44:0/1668013993 shutdown_connections 2024-04-21T23:15:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.851+0000 7fc23b642740 1 -- 172.21.15.44:0/1668013993 wait complete. 2024-04-21T23:15:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.851+0000 7fc23b642740 1 -- start start 2024-04-21T23:15:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.851+0000 7fc23b642740 1 Processor -- start 2024-04-21T23:15:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.851+0000 7fc23b642740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391db9660 0x563391ddbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.851+0000 7fc23b642740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391dbef30 0x563391ddc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.851+0000 7fc23b642740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbf850 0x563391e379b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.851+0000 7fc23b642740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc2341afc60 con 0x563391db9660 2024-04-21T23:15:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.851+0000 7fc23b642740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc2341b10d0 con 0x563391dbef30 2024-04-21T23:15:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.851+0000 7fc23b642740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc2341b0330 con 0x563391dbf850 2024-04-21T23:15:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.851+0000 7fc23b642740 10 client.0 _ll_register_callbacks cb 0x7fc22c013640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.852+0000 7fc23259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391db9660 0x563391ddbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.852+0000 7fc23259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391db9660 0x563391ddbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48726/0 (socket says 172.21.15.44:48726) 2024-04-21T23:15:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.852+0000 7fc23259c640 1 -- 172.21.15.44:0/2039931009 learned_addr learned my addr 172.21.15.44:0/2039931009 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.852+0000 7fc229d9b640 1 --2- 172.21.15.44:0/2039931009 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391dbef30 0x563391ddc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.852+0000 7fc232d9d640 1 --2- 172.21.15.44:0/2039931009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbf850 0x563391e379b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.852+0000 7fc23259c640 1 -- 172.21.15.44:0/2039931009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbf850 msgr2=0x563391e379b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.852+0000 7fc23259c640 1 --2- 172.21.15.44:0/2039931009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbf850 0x563391e379b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.852+0000 7fc23259c640 1 -- 172.21.15.44:0/2039931009 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391dbef30 msgr2=0x563391ddc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:52.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.852+0000 7fc23259c640 1 --2- 172.21.15.44:0/2039931009 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391dbef30 0x563391ddc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.852+0000 7fc23259c640 1 -- 172.21.15.44:0/2039931009 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563391e37fa0 con 0x563391db9660 2024-04-21T23:15:52.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.852+0000 7fc23259c640 1 --2- 172.21.15.44:0/2039931009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391db9660 0x563391ddbdf0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fc2240cd450 tx=0x7fc22409b6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c03979e4fdab6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:52.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.852+0000 7fc219ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:15:52.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.852+0000 7fc219ffb640 1 -- 172.21.15.44:0/2039931009 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc224010630 con 0x563391db9660 2024-04-21T23:15:52.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.852+0000 7fc219ffb640 1 -- 172.21.15.44:0/2039931009 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7fc2240a11d0 con 0x563391db9660 2024-04-21T23:15:52.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.852+0000 7fc23b642740 1 -- 172.21.15.44:0/2039931009 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc21c0553b0 con 0x563391db9660 2024-04-21T23:15:52.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.852+0000 7fc219ffb640 1 -- 172.21.15.44:0/2039931009 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc2240cd220 con 0x563391db9660 2024-04-21T23:15:52.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.852+0000 7fc23b642740 10 client.17232 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:52.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.852+0000 7fc23b642740 1 -- 172.21.15.44:0/2039931009 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563391e3f330 con 0x563391db9660 2024-04-21T23:15:52.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.852+0000 7fc219ffb640 1 -- 172.21.15.44:0/2039931009 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc21c0553b0 con 0x563391db9660 2024-04-21T23:15:52.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.853+0000 7fc219ffb640 1 -- 172.21.15.44:0/2039931009 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7fc224086280 con 0x563391db9660 2024-04-21T23:15:52.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.853+0000 7fc23b642740 10 client.17232 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:52.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.853+0000 7fc23b642740 10 client.17232 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:52.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.853+0000 7fc23b642740 20 client.17232 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:52.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.853+0000 7fc23b642740 1 --2- 172.21.15.44:0/2039931009 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563391dc8430 0x563391e418d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.853+0000 7fc23b642740 4 client.17232 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:15:52.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.853+0000 7fc23b642740 1 -- 172.21.15.44:0/2039931009 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563391d2adf0 con 0x563391dc8430 2024-04-21T23:15:52.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.853+0000 7fc229d9b640 1 --2- 172.21.15.44:0/2039931009 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563391dc8430 0x563391e418d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.853+0000 7fc229d9b640 1 --2- 172.21.15.44:0/2039931009 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563391dc8430 0x563391e418d0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95ec911164a048e3 server_cookie=8061111647cf8196 in_seq=0 out_seq=0 2024-04-21T23:15:52.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.853+0000 7fc219ffb640 10 client.17232 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:52.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.853+0000 7fc219ffb640 1 -- 172.21.15.44:0/2039931009 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563391d2adf0 con 0x563391dc8430 2024-04-21T23:15:52.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.853+0000 7fc219ffb640 10 client.17232 handle_command_reply: tid=0 2024-04-21T23:15:52.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.853+0000 7fc23b642740 1 client.17232 shutdown 2024-04-21T23:15:52.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.854+0000 7fc23b642740 1 -- 172.21.15.44:0/2039931009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391db9660 msgr2=0x563391ddbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.854+0000 7fc23b642740 1 --2- 172.21.15.44:0/2039931009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391db9660 0x563391ddbdf0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fc2240cd450 tx=0x7fc22409b6f0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.854+0000 7fc23b642740 1 -- 172.21.15.44:0/2039931009 shutdown_connections 2024-04-21T23:15:52.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.854+0000 7fc23b642740 1 --2- 172.21.15.44:0/2039931009 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563391dc8430 0x563391e418d0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.854+0000 7fc23b642740 1 --2- 172.21.15.44:0/2039931009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563391dbf850 0x563391e379b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.854+0000 7fc219ffb640 0 client.17232 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:52.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.854+0000 7fc23b642740 1 --2- 172.21.15.44:0/2039931009 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563391dbef30 0x563391ddc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.854+0000 7fc23b642740 1 --2- 172.21.15.44:0/2039931009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563391db9660 0x563391ddbdf0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.854+0000 7fc23b642740 1 -- 172.21.15.44:0/2039931009 >> 172.21.15.44:0/2039931009 conn(0x563391dc00e0 msgr2=0x563391dd35f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:52.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.854+0000 7fc23b642740 1 -- 172.21.15.44:0/2039931009 shutdown_connections 2024-04-21T23:15:52.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.854+0000 7fc23b642740 1 -- 172.21.15.44:0/2039931009 wait complete. 2024-04-21T23:15:52.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.854+0000 7fc23b642740 20 client.17232 trim_cache size 0 max 16384 2024-04-21T23:15:52.860 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:15:52.860 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-21T23:15:52.860 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-21T23:15:52.860 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:15:52.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.854+0000 7fc238a5f640 1 -- 172.21.15.44:0/2333477917 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc2200a2d30 msgr2=0x7fc2200a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.854+0000 7fc238a5f640 1 --2- 172.21.15.44:0/2333477917 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc2200a2d30 0x7fc2200a51f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc21c02efd0 tx=0x7fc21c02cf90 comp rx=0 tx=0).stop 2024-04-21T23:15:52.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.854+0000 7fc238a5f640 1 -- 172.21.15.44:0/2333477917 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2341029b0 msgr2=0x7fc234100600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.854+0000 7fc238a5f640 1 --2- 172.21.15.44:0/2333477917 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2341029b0 0x7fc234100600 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fc22400e9f0 tx=0x7fc22400eeb0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.855+0000 7fc238a5f640 1 -- 172.21.15.44:0/2333477917 shutdown_connections 2024-04-21T23:15:52.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.855+0000 7fc238a5f640 1 --2- 172.21.15.44:0/2333477917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2341032d0 0x7fc234100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.855+0000 7fc238a5f640 1 --2- 172.21.15.44:0/2333477917 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc2200a2d30 0x7fc2200a51f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.855+0000 7fc238a5f640 1 --2- 172.21.15.44:0/2333477917 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc23410bcf0 0x7fc234101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.855+0000 7fc238a5f640 1 --2- 172.21.15.44:0/2333477917 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2341029b0 0x7fc234100600 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.855+0000 7fc238a5f640 1 -- 172.21.15.44:0/2333477917 >> 172.21.15.44:0/2333477917 conn(0x7fc2340777f0 msgr2=0x7fc2340fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:52.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.855+0000 7fc238a5f640 1 -- 172.21.15.44:0/2333477917 shutdown_connections 2024-04-21T23:15:52.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.855+0000 7fc238a5f640 1 -- 172.21.15.44:0/2333477917 wait complete. 2024-04-21T23:15:52.865 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:15:52.865 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:15:52.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.973+0000 7fefdae88640 1 -- 172.21.15.44:0/29812363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fefd4100610 msgr2=0x7fefd4100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.973+0000 7fefdae88640 1 --2- 172.21.15.44:0/29812363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fefd4100610 0x7fefd4100a90 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fefc0009930 tx=0x7fefc002bc30 comp rx=0 tx=0).stop 2024-04-21T23:15:52.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.974+0000 7fefdae88640 1 -- 172.21.15.44:0/29812363 shutdown_connections 2024-04-21T23:15:52.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.974+0000 7fefdae88640 1 --2- 172.21.15.44:0/29812363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefd4100fd0 0x7fefd410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.974+0000 7fefdae88640 1 --2- 172.21.15.44:0/29812363 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fefd4100610 0x7fefd4100a90 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.974+0000 7fefdae88640 1 --2- 172.21.15.44:0/29812363 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefd41066b0 0x7fefd4106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.974+0000 7fefdae88640 1 -- 172.21.15.44:0/29812363 >> 172.21.15.44:0/29812363 conn(0x7fefd4077850 msgr2=0x7fefd4077c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:52.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.974+0000 7fefdae88640 1 -- 172.21.15.44:0/29812363 shutdown_connections 2024-04-21T23:15:52.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.974+0000 7fefdae88640 1 -- 172.21.15.44:0/29812363 wait complete. 2024-04-21T23:15:52.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.974+0000 7fefdae88640 1 Processor -- start 2024-04-21T23:15:52.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.974+0000 7fefdae88640 1 -- start start 2024-04-21T23:15:52.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.974+0000 7fefdae88640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefd4100610 0x7fefd419a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.974+0000 7fefdae88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefd4100fd0 0x7fefd419a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.974+0000 7fefdae88640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fefd41066b0 0x7fefd419ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.974+0000 7fefdae88640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fefd4110a30 con 0x7fefd41066b0 2024-04-21T23:15:52.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.974+0000 7fefdae88640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fefd41108b0 con 0x7fefd4100610 2024-04-21T23:15:52.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.974+0000 7fefdae88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fefd4110bb0 con 0x7fefd4100fd0 2024-04-21T23:15:52.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.975+0000 7fefd9425640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fefd41066b0 0x7fefd419ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.975+0000 7fefd8c24640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefd4100610 0x7fefd419a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.975+0000 7fefd9425640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fefd41066b0 0x7fefd419ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48740/0 (socket says 172.21.15.44:48740) 2024-04-21T23:15:52.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.975+0000 7fefd9425640 1 -- 172.21.15.44:0/1227363740 learned_addr learned my addr 172.21.15.44:0/1227363740 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:52.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.975+0000 7fefd8c24640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefd4100610 0x7fefd419a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:53476/0 (socket says 172.21.15.44:53476) 2024-04-21T23:15:52.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.975+0000 7fefd3fff640 1 --2- 172.21.15.44:0/1227363740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefd4100fd0 0x7fefd419a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.975+0000 7fefd9425640 1 -- 172.21.15.44:0/1227363740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefd4100fd0 msgr2=0x7fefd419a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.975+0000 7fefd9425640 1 --2- 172.21.15.44:0/1227363740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefd4100fd0 0x7fefd419a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.975+0000 7fefd9425640 1 -- 172.21.15.44:0/1227363740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefd4100610 msgr2=0x7fefd419a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:52.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.975+0000 7fefd9425640 1 --2- 172.21.15.44:0/1227363740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefd4100610 0x7fefd419a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:52.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.975+0000 7fefd9425640 1 -- 172.21.15.44:0/1227363740 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefd419f1b0 con 0x7fefd41066b0 2024-04-21T23:15:52.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.975+0000 7fefd8c24640 1 --2- 172.21.15.44:0/1227363740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefd4100610 0x7fefd419a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:52.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.975+0000 7fefd9425640 1 --2- 172.21.15.44:0/1227363740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fefd41066b0 0x7fefd419ea80 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00eb10 tx=0x7fefcc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:52.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.975+0000 7fefd1ffb640 1 -- 172.21.15.44:0/1227363740 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefcc004b20 con 0x7fefd41066b0 2024-04-21T23:15:52.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.975+0000 7fefd1ffb640 1 -- 172.21.15.44:0/1227363740 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefcc004cc0 con 0x7fefd41066b0 2024-04-21T23:15:52.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.975+0000 7fefdae88640 1 -- 172.21.15.44:0/1227363740 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefd419f4a0 con 0x7fefd41066b0 2024-04-21T23:15:52.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.975+0000 7fefd1ffb640 1 -- 172.21.15.44:0/1227363740 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefcc010760 con 0x7fefd41066b0 2024-04-21T23:15:52.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.976+0000 7fefdae88640 1 -- 172.21.15.44:0/1227363740 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefd41a6d30 con 0x7fefd41066b0 2024-04-21T23:15:52.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.977+0000 7fefdae88640 1 -- 172.21.15.44:0/1227363740 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefd4101d50 con 0x7fefd41066b0 2024-04-21T23:15:52.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.978+0000 7fefd1ffb640 1 -- 172.21.15.44:0/1227363740 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fefcc005020 con 0x7fefd41066b0 2024-04-21T23:15:52.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.978+0000 7fefd1ffb640 1 --2- 172.21.15.44:0/1227363740 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fefc40a2d00 0x7fefc40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:52.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.979+0000 7fefd1ffb640 1 -- 172.21.15.44:0/1227363740 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7fefcc0cfee0 con 0x7fefd41066b0 2024-04-21T23:15:52.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.979+0000 7fefd8c24640 1 --2- 172.21.15.44:0/1227363740 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fefc40a2d00 0x7fefc40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:52.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.979+0000 7fefd8c24640 1 --2- 172.21.15.44:0/1227363740 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fefc40a2d00 0x7fefc40a51c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fefd4101750 tx=0x7fefc8005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:52.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:52.983+0000 7fefd1ffb640 1 -- 172.21.15.44:0/1227363740 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fefcc09c710 con 0x7fefd41066b0 2024-04-21T23:15:53.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.220+0000 7fefdae88640 1 -- 172.21.15.44:0/1227363740 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fefd419b510 con 0x7fefd41066b0 2024-04-21T23:15:53.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.221+0000 7fefd1ffb640 1 -- 172.21.15.44:0/1227363740 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11791 (secure 0 0 0) 0x7fefcc0bc0d0 con 0x7fefd41066b0 2024-04-21T23:15:53.221 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:15:53.221 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:15:12.778523+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26645":{"gid":26645,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/1729482148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1729482148},{"type":"v1","addr":"172.21.15.69:6839","nonce":1729482148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:15:53.221 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 29 2024-04-21T23:15:53.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.223+0000 7fefdae88640 1 -- 172.21.15.44:0/1227363740 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fefc40a2d00 msgr2=0x7fefc40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.223+0000 7fefdae88640 1 --2- 172.21.15.44:0/1227363740 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fefc40a2d00 0x7fefc40a51c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fefd4101750 tx=0x7fefc8005f90 comp rx=0 tx=0).stop 2024-04-21T23:15:53.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.223+0000 7fefdae88640 1 -- 172.21.15.44:0/1227363740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fefd41066b0 msgr2=0x7fefd419ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.223+0000 7fefdae88640 1 --2- 172.21.15.44:0/1227363740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fefd41066b0 0x7fefd419ea80 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00eb10 tx=0x7fefcc00c2d0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.223+0000 7fefdae88640 1 -- 172.21.15.44:0/1227363740 shutdown_connections 2024-04-21T23:15:53.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.223+0000 7fefdae88640 1 --2- 172.21.15.44:0/1227363740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefd4100fd0 0x7fefd419a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.223+0000 7fefdae88640 1 --2- 172.21.15.44:0/1227363740 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fefc40a2d00 0x7fefc40a51c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.223+0000 7fefdae88640 1 --2- 172.21.15.44:0/1227363740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fefd41066b0 0x7fefd419ea80 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.223+0000 7fefdae88640 1 --2- 172.21.15.44:0/1227363740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fefd4100610 0x7fefd419a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.223+0000 7fefdae88640 1 -- 172.21.15.44:0/1227363740 >> 172.21.15.44:0/1227363740 conn(0x7fefd4077850 msgr2=0x7fefd4069df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:53.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.224+0000 7fefdae88640 1 -- 172.21.15.44:0/1227363740 shutdown_connections 2024-04-21T23:15:53.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.224+0000 7fefdae88640 1 -- 172.21.15.44:0/1227363740 wait complete. 2024-04-21T23:15:53.235 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:15:53.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.343+0000 7f7ad9615640 1 -- 172.21.15.44:0/4010602577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad41032d0 msgr2=0x7f7ad410b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.343+0000 7f7ad9615640 1 --2- 172.21.15.44:0/4010602577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad41032d0 0x7f7ad410b7b0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009900 tx=0x7f7abc02bcd0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.343+0000 7f7ad9615640 1 -- 172.21.15.44:0/4010602577 shutdown_connections 2024-04-21T23:15:53.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.343+0000 7f7ad9615640 1 --2- 172.21.15.44:0/4010602577 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ad410bcf0 0x7f7ad410e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.343+0000 7f7ad9615640 1 --2- 172.21.15.44:0/4010602577 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad41032d0 0x7f7ad410b7b0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.343+0000 7f7ad9615640 1 --2- 172.21.15.44:0/4010602577 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ad41029b0 0x7f7ad4102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.343+0000 7f7ad9615640 1 -- 172.21.15.44:0/4010602577 >> 172.21.15.44:0/4010602577 conn(0x7f7ad40777f0 msgr2=0x7f7ad4077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:53.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.343+0000 7f7ad9615640 1 -- 172.21.15.44:0/4010602577 shutdown_connections 2024-04-21T23:15:53.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.343+0000 7f7ad9615640 1 -- 172.21.15.44:0/4010602577 wait complete. 2024-04-21T23:15:53.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.343+0000 7f7ad9615640 1 Processor -- start 2024-04-21T23:15:53.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.343+0000 7f7ad9615640 1 -- start start 2024-04-21T23:15:53.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7ad9615640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad41029b0 0x7f7ad41980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7ad9615640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ad41032d0 0x7f7ad4198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7ad9615640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ad410bcf0 0x7f7ad419c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7ad9615640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad4111b40 con 0x7f7ad41029b0 2024-04-21T23:15:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7ad9615640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad41119c0 con 0x7f7ad41032d0 2024-04-21T23:15:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7ad9615640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad4111cc0 con 0x7f7ad410bcf0 2024-04-21T23:15:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7ad2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad41029b0 0x7f7ad41980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7ad2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad41029b0 0x7f7ad41980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48762/0 (socket says 172.21.15.44:48762) 2024-04-21T23:15:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7ad2ffd640 1 -- 172.21.15.44:0/217064012 learned_addr learned my addr 172.21.15.44:0/217064012 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7aca7fc640 1 --2- 172.21.15.44:0/217064012 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ad41032d0 0x7f7ad4198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7ad37fe640 1 --2- 172.21.15.44:0/217064012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ad410bcf0 0x7f7ad419c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7ad2ffd640 1 -- 172.21.15.44:0/217064012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ad410bcf0 msgr2=0x7f7ad419c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7ad2ffd640 1 --2- 172.21.15.44:0/217064012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ad410bcf0 0x7f7ad419c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7ad2ffd640 1 -- 172.21.15.44:0/217064012 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ad41032d0 msgr2=0x7f7ad4198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7ad2ffd640 1 --2- 172.21.15.44:0/217064012 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ad41032d0 0x7f7ad4198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7ad2ffd640 1 -- 172.21.15.44:0/217064012 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ad419d060 con 0x7f7ad41029b0 2024-04-21T23:15:53.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7ad37fe640 1 --2- 172.21.15.44:0/217064012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ad410bcf0 0x7f7ad419c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:15:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7aca7fc640 1 --2- 172.21.15.44:0/217064012 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ad41032d0 0x7f7ad4198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7ad2ffd640 1 --2- 172.21.15.44:0/217064012 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad41029b0 0x7f7ad41980c0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400e9f0 tx=0x7f7ac400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7ad0ff9640 1 -- 172.21.15.44:0/217064012 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac400cd80 con 0x7f7ad41029b0 2024-04-21T23:15:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7ad0ff9640 1 -- 172.21.15.44:0/217064012 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ac4005500 con 0x7f7ad41029b0 2024-04-21T23:15:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.344+0000 7f7ad9615640 1 -- 172.21.15.44:0/217064012 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ad419d350 con 0x7f7ad41029b0 2024-04-21T23:15:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.345+0000 7f7ad0ff9640 1 -- 172.21.15.44:0/217064012 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4010690 con 0x7f7ad41029b0 2024-04-21T23:15:53.347 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.345+0000 7f7ad9615640 1 -- 172.21.15.44:0/217064012 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ad41a4be0 con 0x7f7ad41029b0 2024-04-21T23:15:53.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.347+0000 7f7ad0ff9640 1 -- 172.21.15.44:0/217064012 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7ac40046c0 con 0x7f7ad41029b0 2024-04-21T23:15:53.352 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.348+0000 7f7adc203740 1 Processor -- start 2024-04-21T23:15:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.348+0000 7f7adc203740 1 -- start start 2024-04-21T23:15:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.348+0000 7f7adc203740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c020 0x55e22018c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.348+0000 7f7adc203740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e22018c940 0x55e220194fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.348+0000 7f7adc203740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e220195520 0x55e220197910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.348+0000 7f7adc203740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e2200f0530 con 0x55e220195520 2024-04-21T23:15:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.348+0000 7f7adc203740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55e220098830 con 0x55e22018c940 2024-04-21T23:15:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.348+0000 7f7adc203740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e2200ec4f0 con 0x55e22018c020 2024-04-21T23:15:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.348+0000 7f7aca7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e220195520 0x55e220197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.348+0000 7f7ad2ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e22018c940 0x55e220194fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.348+0000 7f7aca7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e220195520 0x55e220197910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48774/0 (socket says 172.21.15.44:48774) 2024-04-21T23:15:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.348+0000 7f7aca7fc640 1 -- 172.21.15.44:0/1607528519 learned_addr learned my addr 172.21.15.44:0/1607528519 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.348+0000 7f7ad0ff9640 1 --2- 172.21.15.44:0/217064012 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7ac00a2d00 0x7f7ac00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.348+0000 7f7ad37fe640 1 --2- 172.21.15.44:0/1607528519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c020 0x55e22018c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.353 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.349+0000 7f7ad37fe640 1 --2- 172.21.15.44:0/217064012 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7ac00a2d00 0x7f7ac00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.349+0000 7f7aca7fc640 1 -- 172.21.15.44:0/1607528519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c020 msgr2=0x55e22018c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:53.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.349+0000 7f7aca7fc640 1 --2- 172.21.15.44:0/1607528519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c020 0x55e22018c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.349+0000 7f7aca7fc640 1 -- 172.21.15.44:0/1607528519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e22018c940 msgr2=0x55e220194fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:53.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.349+0000 7f7aca7fc640 1 --2- 172.21.15.44:0/1607528519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e22018c940 0x55e220194fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.349+0000 7f7aca7fc640 1 -- 172.21.15.44:0/1607528519 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2200bc2c0 con 0x55e220195520 2024-04-21T23:15:53.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.349+0000 7f7ad0ff9640 1 -- 172.21.15.44:0/217064012 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f7ac40cfd10 con 0x7f7ad41029b0 2024-04-21T23:15:53.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.349+0000 7f7aca7fc640 1 --2- 172.21.15.44:0/1607528519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e220195520 0x55e220197910 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009230 tx=0x7f7abc009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c551a031cebab1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:53.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.349+0000 7f7ac97fa640 1 -- 172.21.15.44:0/1607528519 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7abc043070 con 0x55e220195520 2024-04-21T23:15:53.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.349+0000 7f7ad37fe640 1 --2- 172.21.15.44:0/217064012 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7ac00a2d00 0x7f7ac00a51c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7acc005c40 tx=0x7f7acc009500 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:53.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.349+0000 7f7ac97fa640 1 -- 172.21.15.44:0/1607528519 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7abc02c6f0 con 0x55e220195520 2024-04-21T23:15:53.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.349+0000 7f7ac97fa640 1 -- 172.21.15.44:0/1607528519 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7abc004ee0 con 0x55e220195520 2024-04-21T23:15:53.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.350+0000 7f7adc203740 1 -- 172.21.15.44:0/1607528519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e220195520 msgr2=0x55e220197910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.350+0000 7f7adc203740 1 --2- 172.21.15.44:0/1607528519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e220195520 0x55e220197910 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009230 tx=0x7f7abc009560 comp rx=0 tx=0).stop 2024-04-21T23:15:53.354 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.350+0000 7f7adc203740 1 -- 172.21.15.44:0/1607528519 shutdown_connections 2024-04-21T23:15:53.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.350+0000 7f7adc203740 1 --2- 172.21.15.44:0/1607528519 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e220195520 0x55e220197910 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.350+0000 7f7adc203740 1 --2- 172.21.15.44:0/1607528519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e22018c940 0x55e220194fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.350+0000 7f7adc203740 1 --2- 172.21.15.44:0/1607528519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c020 0x55e22018c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.350+0000 7f7adc203740 1 -- 172.21.15.44:0/1607528519 >> 172.21.15.44:0/1607528519 conn(0x55e220186750 msgr2=0x55e220186b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:53.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.350+0000 7f7adc203740 1 -- 172.21.15.44:0/1607528519 shutdown_connections 2024-04-21T23:15:53.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.350+0000 7f7adc203740 1 -- 172.21.15.44:0/1607528519 wait complete. 2024-04-21T23:15:53.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.350+0000 7f7adc203740 1 -- start start 2024-04-21T23:15:53.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.350+0000 7f7adc203740 1 Processor -- start 2024-04-21T23:15:53.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.350+0000 7f7adc203740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c020 0x55e220194ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7adc203740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e22018c940 0x55e220192280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7adc203740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e220195520 0x55e220192890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7adc203740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55e2201982b0 con 0x55e22018c940 2024-04-21T23:15:53.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7adc203740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55e2200abf10 con 0x55e220195520 2024-04-21T23:15:53.355 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7adc203740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e220198430 con 0x55e22018c020 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7adc203740 10 client.0 _ll_register_callbacks cb 0x55e21fe03be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7aca7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e22018c940 0x55e220192280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7aca7fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e22018c940 0x55e220192280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48780/0 (socket says 172.21.15.44:48780) 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7aca7fc640 1 -- 172.21.15.44:0/295058282 learned_addr learned my addr 172.21.15.44:0/295058282 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7ad37fe640 1 --2- 172.21.15.44:0/295058282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e220195520 0x55e220192890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7ad2ffd640 1 --2- 172.21.15.44:0/295058282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c020 0x55e220194ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7aca7fc640 1 -- 172.21.15.44:0/295058282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c020 msgr2=0x55e220194ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7aca7fc640 1 --2- 172.21.15.44:0/295058282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c020 0x55e220194ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7aca7fc640 1 -- 172.21.15.44:0/295058282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e220195520 msgr2=0x55e220192890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7aca7fc640 1 --2- 172.21.15.44:0/295058282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e220195520 0x55e220192890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7aca7fc640 1 -- 172.21.15.44:0/295058282 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e22007dc80 con 0x55e22018c940 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7aca7fc640 1 --2- 172.21.15.44:0/295058282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e22018c940 0x55e220192280 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f7abc02c210 tx=0x7f7abc004da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77aa7dfd55c2e6c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7aba7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7aba7fc640 1 -- 172.21.15.44:0/295058282 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7abc04f070 con 0x55e22018c940 2024-04-21T23:15:53.356 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7aba7fc640 1 -- 172.21.15.44:0/295058282 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f7abc0052f0 con 0x55e22018c940 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7adc203740 1 -- 172.21.15.44:0/295058282 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7abc02c6f0 con 0x55e22018c940 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7aba7fc640 1 -- 172.21.15.44:0/295058282 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7abc043040 con 0x55e22018c940 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7adc203740 10 client.17250 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.351+0000 7f7adc203740 1 -- 172.21.15.44:0/295058282 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e21fe043f0 con 0x55e22018c940 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.352+0000 7f7aba7fc640 1 -- 172.21.15.44:0/295058282 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7abc02c6f0 con 0x55e22018c940 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.352+0000 7f7aba7fc640 1 -- 172.21.15.44:0/295058282 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f7abc052020 con 0x55e22018c940 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.352+0000 7f7adc203740 10 client.17250 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.352+0000 7f7adc203740 10 client.17250 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.352+0000 7f7adc203740 20 client.17250 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.352+0000 7f7adc203740 1 --2- 172.21.15.44:0/295058282 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e2201a5400 0x55e2201a77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.352+0000 7f7adc203740 4 client.17250 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.352+0000 7f7adc203740 1 -- 172.21.15.44:0/295058282 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2200fda70 con 0x55e2201a5400 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.352+0000 7f7ad2ffd640 1 --2- 172.21.15.44:0/295058282 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e2201a5400 0x55e2201a77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.352+0000 7f7aba7fc640 10 client.17250 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:53.357 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.352+0000 7f7ad2ffd640 1 --2- 172.21.15.44:0/295058282 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e2201a5400 0x55e2201a77f0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e51d19839e7b67a9 server_cookie=740025909b458c80 in_seq=0 out_seq=0 2024-04-21T23:15:53.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.354+0000 7f7aba7fc640 1 -- 172.21.15.44:0/295058282 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e2200fda70 con 0x55e2201a5400 2024-04-21T23:15:53.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.354+0000 7f7aba7fc640 10 client.17250 handle_command_reply: tid=0 2024-04-21T23:15:53.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.354+0000 7f7adc203740 1 client.17250 shutdown 2024-04-21T23:15:53.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.354+0000 7f7adc203740 1 -- 172.21.15.44:0/295058282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e22018c940 msgr2=0x55e220192280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.354+0000 7f7adc203740 1 --2- 172.21.15.44:0/295058282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e22018c940 0x55e220192280 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f7abc02c210 tx=0x7f7abc004da0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.354+0000 7f7adc203740 1 -- 172.21.15.44:0/295058282 shutdown_connections 2024-04-21T23:15:53.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.354+0000 7f7adc203740 1 --2- 172.21.15.44:0/295058282 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e2201a5400 0x55e2201a77f0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.354+0000 7f7adc203740 1 --2- 172.21.15.44:0/295058282 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e220195520 0x55e220192890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.354+0000 7f7aba7fc640 0 client.17250 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:53.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.354+0000 7f7adc203740 1 --2- 172.21.15.44:0/295058282 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e22018c940 0x55e220192280 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.359 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.354+0000 7f7adc203740 1 --2- 172.21.15.44:0/295058282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c020 0x55e220194ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.354+0000 7f7adc203740 1 -- 172.21.15.44:0/295058282 >> 172.21.15.44:0/295058282 conn(0x55e220186750 msgr2=0x55e2201884a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:53.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.354+0000 7f7adc203740 1 -- 172.21.15.44:0/295058282 shutdown_connections 2024-04-21T23:15:53.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.354+0000 7f7adc203740 1 -- 172.21.15.44:0/295058282 wait complete. 2024-04-21T23:15:53.360 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.354+0000 7f7adc203740 20 client.17250 trim_cache size 0 max 16384 2024-04-21T23:15:53.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.368+0000 7f7adc203740 1 Processor -- start 2024-04-21T23:15:53.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.368+0000 7f7adc203740 1 -- start start 2024-04-21T23:15:53.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.368+0000 7f7adc203740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e22018c020 0x55e2201a1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.368+0000 7f7adc203740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c940 0x55e2201949b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.368+0000 7f7adc203740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e220195520 0x55e220194ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.368+0000 7f7adc203740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad41acd40 con 0x55e22018c020 2024-04-21T23:15:53.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.368+0000 7f7adc203740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad41af5a0 con 0x55e220195520 2024-04-21T23:15:53.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.368+0000 7f7adc203740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad41ad390 con 0x55e22018c940 2024-04-21T23:15:53.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.368+0000 7f7ad2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e22018c020 0x55e2201a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.368+0000 7f7ad2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e22018c020 0x55e2201a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48786/0 (socket says 172.21.15.44:48786) 2024-04-21T23:15:53.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.368+0000 7f7ad2ffd640 1 -- 172.21.15.44:0/3705939008 learned_addr learned my addr 172.21.15.44:0/3705939008 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:53.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.368+0000 7f7ad37fe640 1 --2- 172.21.15.44:0/3705939008 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e220195520 0x55e220194ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.368+0000 7f7aca7fc640 1 --2- 172.21.15.44:0/3705939008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c940 0x55e2201949b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.368+0000 7f7ad2ffd640 1 -- 172.21.15.44:0/3705939008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c940 msgr2=0x55e2201949b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:53.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.368+0000 7f7ad2ffd640 1 --2- 172.21.15.44:0/3705939008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c940 0x55e2201949b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.368+0000 7f7ad2ffd640 1 -- 172.21.15.44:0/3705939008 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e220195520 msgr2=0x55e220194ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:53.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.368+0000 7f7ad2ffd640 1 --2- 172.21.15.44:0/3705939008 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e220195520 0x55e220194ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.368+0000 7f7ad2ffd640 1 -- 172.21.15.44:0/3705939008 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e21fece7d0 con 0x55e22018c020 2024-04-21T23:15:53.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.368+0000 7f7aca7fc640 1 --2- 172.21.15.44:0/3705939008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c940 0x55e2201949b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:15:53.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.369+0000 7f7ad2ffd640 1 --2- 172.21.15.44:0/3705939008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e22018c020 0x55e2201a1170 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4004c00 tx=0x7f7ac40bc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cddb3f90b2eab4ea server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:53.374 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.369+0000 7f7ac8ff9640 1 -- 172.21.15.44:0/3705939008 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac409e030 con 0x55e22018c020 2024-04-21T23:15:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.369+0000 7f7ac8ff9640 1 -- 172.21.15.44:0/3705939008 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ac409c540 con 0x55e22018c020 2024-04-21T23:15:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.369+0000 7f7ac8ff9640 1 -- 172.21.15.44:0/3705939008 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac40cd2f0 con 0x55e22018c020 2024-04-21T23:15:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.369+0000 7f7adc203740 1 -- 172.21.15.44:0/3705939008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e22018c020 msgr2=0x55e2201a1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.369+0000 7f7adc203740 1 --2- 172.21.15.44:0/3705939008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e22018c020 0x55e2201a1170 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4004c00 tx=0x7f7ac40bc040 comp rx=0 tx=0).stop 2024-04-21T23:15:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.369+0000 7f7adc203740 1 -- 172.21.15.44:0/3705939008 shutdown_connections 2024-04-21T23:15:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.369+0000 7f7adc203740 1 --2- 172.21.15.44:0/3705939008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c940 0x55e2201949b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.369+0000 7f7adc203740 1 --2- 172.21.15.44:0/3705939008 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e220195520 0x55e220194ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.369+0000 7f7adc203740 1 --2- 172.21.15.44:0/3705939008 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e22018c020 0x55e2201a1170 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.369+0000 7f7adc203740 1 -- 172.21.15.44:0/3705939008 >> 172.21.15.44:0/3705939008 conn(0x55e220186750 msgr2=0x55e220185850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.369+0000 7f7adc203740 1 -- 172.21.15.44:0/3705939008 shutdown_connections 2024-04-21T23:15:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.369+0000 7f7adc203740 1 -- 172.21.15.44:0/3705939008 wait complete. 2024-04-21T23:15:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.370+0000 7f7adc203740 1 -- start start 2024-04-21T23:15:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.370+0000 7f7adc203740 1 Processor -- start 2024-04-21T23:15:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.370+0000 7f7adc203740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e22018c020 0x55e220199800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.370+0000 7f7adc203740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c940 0x55e220199d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.370+0000 7f7adc203740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e220195520 0x55e220212e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.370+0000 7f7adc203740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad41ada60 con 0x55e220195520 2024-04-21T23:15:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.370+0000 7f7adc203740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad41aeed0 con 0x55e22018c020 2024-04-21T23:15:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.370+0000 7f7adc203740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad41ae130 con 0x55e22018c940 2024-04-21T23:15:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.370+0000 7f7adc203740 10 client.0 _ll_register_callbacks cb 0x7f7ac4097a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.370+0000 7f7ad37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e220195520 0x55e220212e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.370+0000 7f7ad2ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e22018c020 0x55e220199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.370+0000 7f7ad37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e220195520 0x55e220212e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48802/0 (socket says 172.21.15.44:48802) 2024-04-21T23:15:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.370+0000 7f7ad37fe640 1 -- 172.21.15.44:0/1528693068 learned_addr learned my addr 172.21.15.44:0/1528693068 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.370+0000 7f7aca7fc640 1 --2- 172.21.15.44:0/1528693068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c940 0x55e220199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.370+0000 7f7ad37fe640 1 -- 172.21.15.44:0/1528693068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c940 msgr2=0x55e220199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.370+0000 7f7ad37fe640 1 --2- 172.21.15.44:0/1528693068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c940 0x55e220199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.370+0000 7f7ad37fe640 1 -- 172.21.15.44:0/1528693068 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e22018c020 msgr2=0x55e220199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:53.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.370+0000 7f7ad37fe640 1 --2- 172.21.15.44:0/1528693068 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e22018c020 0x55e220199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.370+0000 7f7ad37fe640 1 -- 172.21.15.44:0/1528693068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2202133b0 con 0x55e220195520 2024-04-21T23:15:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.370+0000 7f7ad2ffd640 1 --2- 172.21.15.44:0/1528693068 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e22018c020 0x55e220199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.371+0000 7f7ad37fe640 1 --2- 172.21.15.44:0/1528693068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e220195520 0x55e220212e40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f7acc00eea0 tx=0x7f7acc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7280af9cf339145 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.371+0000 7f7aba7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:15:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.371+0000 7f7aba7fc640 1 -- 172.21.15.44:0/1528693068 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7acc018070 con 0x55e220195520 2024-04-21T23:15:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.371+0000 7f7aba7fc640 1 -- 172.21.15.44:0/1528693068 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f7acc004b20 con 0x55e220195520 2024-04-21T23:15:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.371+0000 7f7adc203740 1 -- 172.21.15.44:0/1528693068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ac409c540 con 0x55e220195520 2024-04-21T23:15:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.371+0000 7f7aba7fc640 1 -- 172.21.15.44:0/1528693068 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7acc004d20 con 0x55e220195520 2024-04-21T23:15:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.371+0000 7f7adc203740 10 client.17256 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.371+0000 7f7adc203740 1 -- 172.21.15.44:0/1528693068 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e2202146b0 con 0x55e220195520 2024-04-21T23:15:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.371+0000 7f7aba7fc640 1 -- 172.21.15.44:0/1528693068 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ac409c540 con 0x55e220195520 2024-04-21T23:15:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.371+0000 7f7aba7fc640 1 -- 172.21.15.44:0/1528693068 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f7acc028070 con 0x55e220195520 2024-04-21T23:15:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.371+0000 7f7adc203740 10 client.17256 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.371+0000 7f7adc203740 10 client.17256 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:53.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.371+0000 7f7adc203740 20 client.17256 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:53.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.371+0000 7f7adc203740 1 --2- 172.21.15.44:0/1528693068 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e220214ed0 0x55e2202172f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:53.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.371+0000 7f7adc203740 4 client.17256 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:15:53.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.371+0000 7f7adc203740 1 -- 172.21.15.44:0/1528693068 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e2200f7dc0 con 0x55e220214ed0 2024-04-21T23:15:53.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.372+0000 7f7ad2ffd640 1 --2- 172.21.15.44:0/1528693068 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e220214ed0 0x55e2202172f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:53.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.372+0000 7f7ad2ffd640 1 --2- 172.21.15.44:0/1528693068 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e220214ed0 0x55e2202172f0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c02f7d47daeb57e server_cookie=811a2ea0376faa38 in_seq=0 out_seq=0 2024-04-21T23:15:53.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.372+0000 7f7aba7fc640 10 client.17256 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:53.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.373+0000 7f7aba7fc640 1 -- 172.21.15.44:0/1528693068 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e2200f7dc0 con 0x55e220214ed0 2024-04-21T23:15:53.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.373+0000 7f7aba7fc640 10 client.17256 handle_command_reply: tid=0 2024-04-21T23:15:53.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.373+0000 7f7adc203740 1 client.17256 shutdown 2024-04-21T23:15:53.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.373+0000 7f7adc203740 1 -- 172.21.15.44:0/1528693068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e220195520 msgr2=0x55e220212e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.373+0000 7f7adc203740 1 --2- 172.21.15.44:0/1528693068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e220195520 0x55e220212e40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f7acc00eea0 tx=0x7f7acc004a40 comp rx=0 tx=0).stop 2024-04-21T23:15:53.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.373+0000 7f7adc203740 1 -- 172.21.15.44:0/1528693068 shutdown_connections 2024-04-21T23:15:53.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.373+0000 7f7adc203740 1 --2- 172.21.15.44:0/1528693068 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55e220214ed0 0x55e2202172f0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.378 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.373+0000 7f7adc203740 1 --2- 172.21.15.44:0/1528693068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e22018c940 0x55e220199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.373+0000 7f7adc203740 1 --2- 172.21.15.44:0/1528693068 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55e220195520 0x55e220212e40 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.373+0000 7f7aba7fc640 0 client.17256 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:53.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.373+0000 7f7adc203740 1 --2- 172.21.15.44:0/1528693068 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55e22018c020 0x55e220199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.373+0000 7f7adc203740 1 -- 172.21.15.44:0/1528693068 >> 172.21.15.44:0/1528693068 conn(0x55e220186750 msgr2=0x55e22018cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:53.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.373+0000 7f7adc203740 1 -- 172.21.15.44:0/1528693068 shutdown_connections 2024-04-21T23:15:53.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.373+0000 7f7adc203740 1 -- 172.21.15.44:0/1528693068 wait complete. 2024-04-21T23:15:53.379 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.373+0000 7f7adc203740 20 client.17256 trim_cache size 0 max 16384 2024-04-21T23:15:53.379 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-21T23:15:53.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.374+0000 7f7ad9615640 1 -- 172.21.15.44:0/217064012 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7ac00a2d00 msgr2=0x7f7ac00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.374+0000 7f7ad9615640 1 --2- 172.21.15.44:0/217064012 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7ac00a2d00 0x7f7ac00a51c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7acc005c40 tx=0x7f7acc009500 comp rx=0 tx=0).stop 2024-04-21T23:15:53.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.374+0000 7f7ad9615640 1 -- 172.21.15.44:0/217064012 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad41029b0 msgr2=0x7f7ad41980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:53.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.374+0000 7f7ad9615640 1 --2- 172.21.15.44:0/217064012 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad41029b0 0x7f7ad41980c0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400e9f0 tx=0x7f7ac400edf0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.374+0000 7f7ad9615640 1 -- 172.21.15.44:0/217064012 shutdown_connections 2024-04-21T23:15:53.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.374+0000 7f7ad9615640 1 --2- 172.21.15.44:0/217064012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ad410bcf0 0x7f7ad419c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.374+0000 7f7ad9615640 1 --2- 172.21.15.44:0/217064012 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7ac00a2d00 0x7f7ac00a51c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.374+0000 7f7ad9615640 1 --2- 172.21.15.44:0/217064012 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7ad41032d0 0x7f7ad4198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.374+0000 7f7ad9615640 1 --2- 172.21.15.44:0/217064012 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7ad41029b0 0x7f7ad41980c0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:53.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.374+0000 7f7ad9615640 1 -- 172.21.15.44:0/217064012 >> 172.21.15.44:0/217064012 conn(0x7f7ad40777f0 msgr2=0x7f7ad40fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:53.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.374+0000 7f7ad9615640 1 -- 172.21.15.44:0/217064012 shutdown_connections 2024-04-21T23:15:53.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:53.374+0000 7f7ad9615640 1 -- 172.21.15.44:0/217064012 wait complete. 2024-04-21T23:15:53.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 821204 12.37 MB/sec execute 223 sec latency 26.661 ms 2024-04-21T23:15:54.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:53 smithi044 ceph-mon[27934]: pgmap v220: 129 pgs: 129 active+clean; 81 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 383 op/s 2024-04-21T23:15:54.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:53 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1227363740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:54.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:53 smithi069 ceph-mon[31816]: pgmap v220: 129 pgs: 129 active+clean; 81 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 383 op/s 2024-04-21T23:15:54.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:53 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1227363740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:54.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:53 smithi055 ceph-mon[31117]: pgmap v220: 129 pgs: 129 active+clean; 81 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 383 op/s 2024-04-21T23:15:54.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:53 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1227363740' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:15:54.385 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:15:54.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:15:54.385 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:15:54.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.499+0000 7f93d08d4640 1 -- 172.21.15.44:0/3685729074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93cc103bb0 msgr2=0x7f93cc10e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.499+0000 7f93d08d4640 1 --2- 172.21.15.44:0/3685729074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93cc103bb0 0x7f93cc10e6e0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f93c4009ca0 tx=0x7f93c402c1d0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.499+0000 7f93d08d4640 1 -- 172.21.15.44:0/3685729074 shutdown_connections 2024-04-21T23:15:54.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.499+0000 7f93d08d4640 1 --2- 172.21.15.44:0/3685729074 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93cc103bb0 0x7f93cc10e6e0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.499+0000 7f93d08d4640 1 --2- 172.21.15.44:0/3685729074 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93cc103210 0x7f93cc103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.499+0000 7f93d08d4640 1 --2- 172.21.15.44:0/3685729074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93cc101ad0 0x7f93cc101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.499+0000 7f93d08d4640 1 -- 172.21.15.44:0/3685729074 >> 172.21.15.44:0/3685729074 conn(0x7f93cc0fd200 msgr2=0x7f93cc0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:54.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.499+0000 7f93d08d4640 1 -- 172.21.15.44:0/3685729074 shutdown_connections 2024-04-21T23:15:54.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.499+0000 7f93d08d4640 1 -- 172.21.15.44:0/3685729074 wait complete. 2024-04-21T23:15:54.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.500+0000 7f93d08d4640 1 Processor -- start 2024-04-21T23:15:54.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.500+0000 7f93d08d4640 1 -- start start 2024-04-21T23:15:54.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.500+0000 7f93d08d4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93cc101ad0 0x7f93cc198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.500+0000 7f93d08d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93cc103210 0x7f93cc1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.500+0000 7f93d08d4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93cc103bb0 0x7f93cc19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.500+0000 7f93d08d4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f93cc111c20 con 0x7f93cc103bb0 2024-04-21T23:15:54.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.500+0000 7f93d08d4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f93cc111aa0 con 0x7f93cc101ad0 2024-04-21T23:15:54.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.500+0000 7f93d08d4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f93cc111da0 con 0x7f93cc103210 2024-04-21T23:15:54.502 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.501+0000 7f93cad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93cc103bb0 0x7f93cc19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.501+0000 7f93cad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93cc103bb0 0x7f93cc19c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48824/0 (socket says 172.21.15.44:48824) 2024-04-21T23:15:54.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.501+0000 7f93ca59c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93cc101ad0 0x7f93cc198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.501+0000 7f93cad9d640 1 -- 172.21.15.44:0/4090735067 learned_addr learned my addr 172.21.15.44:0/4090735067 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:54.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.501+0000 7f93c9d9b640 1 --2- 172.21.15.44:0/4090735067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93cc103210 0x7f93cc1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.501+0000 7f93cad9d640 1 -- 172.21.15.44:0/4090735067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93cc103210 msgr2=0x7f93cc1985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.501+0000 7f93cad9d640 1 --2- 172.21.15.44:0/4090735067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93cc103210 0x7f93cc1985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.501+0000 7f93cad9d640 1 -- 172.21.15.44:0/4090735067 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93cc101ad0 msgr2=0x7f93cc198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.501+0000 7f93cad9d640 1 --2- 172.21.15.44:0/4090735067 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93cc101ad0 0x7f93cc198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.501+0000 7f93cad9d640 1 -- 172.21.15.44:0/4090735067 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93cc19d020 con 0x7f93cc103bb0 2024-04-21T23:15:54.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.501+0000 7f93c9d9b640 1 --2- 172.21.15.44:0/4090735067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93cc103210 0x7f93cc1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:54.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.501+0000 7f93ca59c640 1 --2- 172.21.15.44:0/4090735067 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93cc101ad0 0x7f93cc198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:54.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.501+0000 7f93cad9d640 1 --2- 172.21.15.44:0/4090735067 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93cc103bb0 0x7f93cc19c8f0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f93c4037040 tx=0x7f93c40346e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:54.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.501+0000 7f93bb7fe640 1 -- 172.21.15.44:0/4090735067 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c4039050 con 0x7f93cc103bb0 2024-04-21T23:15:54.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.501+0000 7f93d08d4640 1 -- 172.21.15.44:0/4090735067 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93cc19d2b0 con 0x7f93cc103bb0 2024-04-21T23:15:54.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.501+0000 7f93bb7fe640 1 -- 172.21.15.44:0/4090735067 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93c4045070 con 0x7f93cc103bb0 2024-04-21T23:15:54.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.501+0000 7f93bb7fe640 1 -- 172.21.15.44:0/4090735067 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c4032a40 con 0x7f93cc103bb0 2024-04-21T23:15:54.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.501+0000 7f93d08d4640 1 -- 172.21.15.44:0/4090735067 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93cc1a4ba0 con 0x7f93cc103bb0 2024-04-21T23:15:54.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.504+0000 7f93bb7fe640 1 -- 172.21.15.44:0/4090735067 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f93c4032be0 con 0x7f93cc103bb0 2024-04-21T23:15:54.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.504+0000 7f93d35a0740 1 Processor -- start 2024-04-21T23:15:54.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.505+0000 7f93d35a0740 1 -- start start 2024-04-21T23:15:54.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.505+0000 7f93d35a0740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98cac720 0x560e98cab470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.505+0000 7f93d35a0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98caba40 0x560e98cabec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.505+0000 7f93d35a0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98cb4db0 0x560e98cb71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.505+0000 7f93d35a0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560e98c0f170 con 0x560e98caba40 2024-04-21T23:15:54.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.505+0000 7f93d35a0740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x560e98bb77c0 con 0x560e98cac720 2024-04-21T23:15:54.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.505+0000 7f93d35a0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560e98c0b130 con 0x560e98cb4db0 2024-04-21T23:15:54.506 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.505+0000 7f93c9d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98cac720 0x560e98cab470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.505+0000 7f93c9d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98cac720 0x560e98cab470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:53570/0 (socket says 172.21.15.44:53570) 2024-04-21T23:15:54.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.505+0000 7f93ca59c640 1 --2- 172.21.15.44:0/2598193779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98caba40 0x560e98cabec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.505+0000 7f93c9d9b640 1 -- 172.21.15.44:0/2598193779 learned_addr learned my addr 172.21.15.44:0/2598193779 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:54.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.505+0000 7f93c9d9b640 1 --2- 172.21.15.44:0/2598193779 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98cb4db0 0x560e98cb71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.506+0000 7f93c9d9b640 1 -- 172.21.15.44:0/2598193779 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98cb4db0 msgr2=0x560e98cb71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:54.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.506+0000 7f93c9d9b640 1 --2- 172.21.15.44:0/2598193779 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98cb4db0 0x560e98cb71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.506+0000 7f93c9d9b640 1 -- 172.21.15.44:0/2598193779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98caba40 msgr2=0x560e98cabec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:54.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.506+0000 7f93bb7fe640 1 --2- 172.21.15.44:0/4090735067 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f93b40a2d00 0x7f93b40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.506+0000 7f93c9d9b640 1 --2- 172.21.15.44:0/2598193779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98caba40 0x560e98cabec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.506+0000 7f93c9d9b640 1 -- 172.21.15.44:0/2598193779 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e98bc8e60 con 0x560e98cac720 2024-04-21T23:15:54.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.506+0000 7f93bb7fe640 1 -- 172.21.15.44:0/4090735067 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f93c40f1f20 con 0x7f93cc103bb0 2024-04-21T23:15:54.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.506+0000 7f93cad9d640 1 --2- 172.21.15.44:0/4090735067 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f93b40a2d00 0x7f93b40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.506+0000 7f93c9d9b640 1 --2- 172.21.15.44:0/2598193779 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98cac720 0x560e98cab470 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f93c0011be0 tx=0x7f93c0009ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fa6eab3f05c52ed server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:54.507 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.506+0000 7f93b8ff9640 1 -- 172.21.15.44:0/2598193779 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c0024070 con 0x560e98cac720 2024-04-21T23:15:54.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.506+0000 7f93b8ff9640 1 -- 172.21.15.44:0/2598193779 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93c000fe50 con 0x560e98cac720 2024-04-21T23:15:54.508 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.506+0000 7f93b8ff9640 1 -- 172.21.15.44:0/2598193779 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c000e910 con 0x560e98cac720 2024-04-21T23:15:54.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.507+0000 7f93cad9d640 1 --2- 172.21.15.44:0/4090735067 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f93b40a2d00 0x7f93b40a51c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f93c40049d0 tx=0x7f93c40f2220 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:54.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.507+0000 7f93d35a0740 1 -- 172.21.15.44:0/2598193779 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98cac720 msgr2=0x560e98cab470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.512 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.507+0000 7f93d35a0740 1 --2- 172.21.15.44:0/2598193779 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98cac720 0x560e98cab470 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f93c0011be0 tx=0x7f93c0009ba0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.507+0000 7f93d35a0740 1 -- 172.21.15.44:0/2598193779 shutdown_connections 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.507+0000 7f93d35a0740 1 --2- 172.21.15.44:0/2598193779 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98cb4db0 0x560e98cb71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.507+0000 7f93d35a0740 1 --2- 172.21.15.44:0/2598193779 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98caba40 0x560e98cabec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.507+0000 7f93d35a0740 1 --2- 172.21.15.44:0/2598193779 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98cac720 0x560e98cab470 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.507+0000 7f93d35a0740 1 -- 172.21.15.44:0/2598193779 >> 172.21.15.44:0/2598193779 conn(0x560e98c724f0 msgr2=0x560e98ca6490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.508+0000 7f93d35a0740 1 -- 172.21.15.44:0/2598193779 shutdown_connections 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.508+0000 7f93d35a0740 1 -- 172.21.15.44:0/2598193779 wait complete. 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.508+0000 7f93d35a0740 1 -- start start 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.508+0000 7f93d35a0740 1 Processor -- start 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.508+0000 7f93d35a0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98caba40 0x560e98cb1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.508+0000 7f93d35a0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98cac720 0x560e98cb2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.508+0000 7f93d35a0740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98cb4db0 0x560e98cb28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.513 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.508+0000 7f93d35a0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x560e98cb7c20 con 0x560e98caba40 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.508+0000 7f93d35a0740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x560e98bcad40 con 0x560e98cb4db0 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.508+0000 7f93d35a0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560e98cb7da0 con 0x560e98cac720 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.509+0000 7f93c9d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98caba40 0x560e98cb1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.509+0000 7f93d35a0740 10 client.0 _ll_register_callbacks cb 0x560e98c06820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.509+0000 7f93c9d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98caba40 0x560e98cb1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48848/0 (socket says 172.21.15.44:48848) 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.509+0000 7f93ca59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98cac720 0x560e98cb2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.509+0000 7f93c9d9b640 1 -- 172.21.15.44:0/4188393559 learned_addr learned my addr 172.21.15.44:0/4188393559 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.509+0000 7f93c9d9b640 1 --2- 172.21.15.44:0/4188393559 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98cb4db0 0x560e98cb28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.509+0000 7f93ca59c640 1 -- 172.21.15.44:0/4188393559 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98cb4db0 msgr2=0x560e98cb28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.509+0000 7f93ca59c640 1 --2- 172.21.15.44:0/4188393559 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98cb4db0 0x560e98cb28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.509+0000 7f93ca59c640 1 -- 172.21.15.44:0/4188393559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98caba40 msgr2=0x560e98cb1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.509+0000 7f93ca59c640 1 --2- 172.21.15.44:0/4188393559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98caba40 0x560e98cb1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.509+0000 7f93ca59c640 1 -- 172.21.15.44:0/4188393559 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e989eb7e0 con 0x560e98cac720 2024-04-21T23:15:54.514 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.509+0000 7f93c9d9b640 1 --2- 172.21.15.44:0/4188393559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98caba40 0x560e98cb1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:54.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.509+0000 7f93ca59c640 1 --2- 172.21.15.44:0/4188393559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98cac720 0x560e98cb2360 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009fd0 tx=0x7f93bc012e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a92aae25bd76420d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:54.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.509+0000 7f93b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:15:54.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.509+0000 7f93b1ffb640 1 -- 172.21.15.44:0/4188393559 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc00f9e0 con 0x560e98cac720 2024-04-21T23:15:54.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.510+0000 7f93d35a0740 1 -- 172.21.15.44:0/4188393559 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93c000fe50 con 0x560e98cac720 2024-04-21T23:15:54.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.510+0000 7f93b1ffb640 1 -- 172.21.15.44:0/4188393559 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f93bc00fb80 con 0x560e98cac720 2024-04-21T23:15:54.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.510+0000 7f93b1ffb640 1 -- 172.21.15.44:0/4188393559 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f93bc025050 con 0x560e98cac720 2024-04-21T23:15:54.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.510+0000 7f93d35a0740 10 client.26806 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:54.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.510+0000 7f93d35a0740 1 -- 172.21.15.44:0/4188393559 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560e98a4bf70 con 0x560e98cac720 2024-04-21T23:15:54.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.510+0000 7f93b1ffb640 1 -- 172.21.15.44:0/4188393559 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93c000fe50 con 0x560e98cac720 2024-04-21T23:15:54.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.510+0000 7f93b1ffb640 1 -- 172.21.15.44:0/4188393559 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f93bc003b50 con 0x560e98cac720 2024-04-21T23:15:54.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.510+0000 7f93d35a0740 10 client.26806 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:54.515 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.510+0000 7f93d35a0740 10 client.26806 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.510+0000 7f93d35a0740 20 client.26806 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.510+0000 7f93d35a0740 1 --2- 172.21.15.44:0/4188393559 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560e98cc5050 0x560e98cc7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.510+0000 7f93d35a0740 4 client.26806 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:15:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.510+0000 7f93d35a0740 1 -- 172.21.15.44:0/4188393559 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e98be3450 con 0x560e98cc5050 2024-04-21T23:15:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.510+0000 7f93c9d9b640 1 --2- 172.21.15.44:0/4188393559 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560e98cc5050 0x560e98cc7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.511+0000 7f93c9d9b640 1 --2- 172.21.15.44:0/4188393559 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560e98cc5050 0x560e98cc7460 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82d171cd7dc95fa2 server_cookie=e6b61d63e1b2280c in_seq=0 out_seq=0 2024-04-21T23:15:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.511+0000 7f93b1ffb640 10 client.26806 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.512+0000 7f93b1ffb640 1 -- 172.21.15.44:0/4188393559 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560e98be3450 con 0x560e98cc5050 2024-04-21T23:15:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.512+0000 7f93b1ffb640 10 client.26806 handle_command_reply: tid=0 2024-04-21T23:15:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.512+0000 7f93d35a0740 1 client.26806 shutdown 2024-04-21T23:15:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.512+0000 7f93d35a0740 1 -- 172.21.15.44:0/4188393559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98cac720 msgr2=0x560e98cb2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.512+0000 7f93d35a0740 1 --2- 172.21.15.44:0/4188393559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98cac720 0x560e98cb2360 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009fd0 tx=0x7f93bc012e70 comp rx=0 tx=0).stop 2024-04-21T23:15:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.512+0000 7f93d35a0740 1 -- 172.21.15.44:0/4188393559 shutdown_connections 2024-04-21T23:15:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.512+0000 7f93d35a0740 1 --2- 172.21.15.44:0/4188393559 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560e98cc5050 0x560e98cc7460 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.512+0000 7f93d35a0740 1 --2- 172.21.15.44:0/4188393559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98cac720 0x560e98cb2360 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.516 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.512+0000 7f93d35a0740 1 --2- 172.21.15.44:0/4188393559 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98cb4db0 0x560e98cb28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.512+0000 7f93b1ffb640 0 client.26806 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:54.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.512+0000 7f93d35a0740 1 --2- 172.21.15.44:0/4188393559 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98caba40 0x560e98cb1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.512+0000 7f93d35a0740 1 -- 172.21.15.44:0/4188393559 >> 172.21.15.44:0/4188393559 conn(0x560e98c724f0 msgr2=0x560e98ca81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:54.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.512+0000 7f93d35a0740 1 -- 172.21.15.44:0/4188393559 shutdown_connections 2024-04-21T23:15:54.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.512+0000 7f93d35a0740 1 -- 172.21.15.44:0/4188393559 wait complete. 2024-04-21T23:15:54.517 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.512+0000 7f93d35a0740 20 client.26806 trim_cache size 0 max 16384 2024-04-21T23:15:54.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.531+0000 7f93d35a0740 1 Processor -- start 2024-04-21T23:15:54.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.531+0000 7f93d35a0740 1 -- start start 2024-04-21T23:15:54.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.531+0000 7f93d35a0740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98caba40 0x560e98cb4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.531+0000 7f93d35a0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98cac720 0x560e98cc0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.531+0000 7f93d35a0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98cb4db0 0x560e98ca5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.531+0000 7f93d35a0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f93cc1aa9a0 con 0x560e98cac720 2024-04-21T23:15:54.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.531+0000 7f93d35a0740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f93cc1ad200 con 0x560e98caba40 2024-04-21T23:15:54.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.531+0000 7f93d35a0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f93cc1aaff0 con 0x560e98cb4db0 2024-04-21T23:15:54.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.531+0000 7f93ca59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98cac720 0x560e98cc0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.531+0000 7f93c9d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98caba40 0x560e98cb4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.531+0000 7f93ca59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98cac720 0x560e98cc0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48862/0 (socket says 172.21.15.44:48862) 2024-04-21T23:15:54.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.531+0000 7f93ca59c640 1 -- 172.21.15.44:0/1251884912 learned_addr learned my addr 172.21.15.44:0/1251884912 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:54.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.531+0000 7f93c9d9b640 1 --2- 172.21.15.44:0/1251884912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98cb4db0 0x560e98ca5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.532+0000 7f93ca59c640 1 -- 172.21.15.44:0/1251884912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98cb4db0 msgr2=0x560e98ca5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:54.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.532+0000 7f93ca59c640 1 --2- 172.21.15.44:0/1251884912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98cb4db0 0x560e98ca5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.532+0000 7f93ca59c640 1 -- 172.21.15.44:0/1251884912 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98caba40 msgr2=0x560e98cb4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:54.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.532+0000 7f93ca59c640 1 --2- 172.21.15.44:0/1251884912 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98caba40 0x560e98cb4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.532+0000 7f93ca59c640 1 -- 172.21.15.44:0/1251884912 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e988e3170 con 0x560e98cac720 2024-04-21T23:15:54.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.532+0000 7f93c9d9b640 1 --2- 172.21.15.44:0/1251884912 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98caba40 0x560e98cb4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:54.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.532+0000 7f93ca59c640 1 --2- 172.21.15.44:0/1251884912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98cac720 0x560e98cc0b80 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f93bc000c00 tx=0x7f93bc010850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85aacca60a1a6ea3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:54.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.532+0000 7f93b3fff640 1 -- 172.21.15.44:0/1251884912 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc00f9e0 con 0x560e98cac720 2024-04-21T23:15:54.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.532+0000 7f93b3fff640 1 -- 172.21.15.44:0/1251884912 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93bc02e070 con 0x560e98cac720 2024-04-21T23:15:54.533 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.532+0000 7f93b3fff640 1 -- 172.21.15.44:0/1251884912 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93bc0051c0 con 0x560e98cac720 2024-04-21T23:15:54.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.533+0000 7f93d35a0740 1 -- 172.21.15.44:0/1251884912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98cac720 msgr2=0x560e98cc0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.533+0000 7f93d35a0740 1 --2- 172.21.15.44:0/1251884912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98cac720 0x560e98cc0b80 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f93bc000c00 tx=0x7f93bc010850 comp rx=0 tx=0).stop 2024-04-21T23:15:54.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.533+0000 7f93d35a0740 1 -- 172.21.15.44:0/1251884912 shutdown_connections 2024-04-21T23:15:54.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.533+0000 7f93d35a0740 1 --2- 172.21.15.44:0/1251884912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98cb4db0 0x560e98ca5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.533+0000 7f93d35a0740 1 --2- 172.21.15.44:0/1251884912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98cac720 0x560e98cc0b80 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.533+0000 7f93d35a0740 1 --2- 172.21.15.44:0/1251884912 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98caba40 0x560e98cb4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.533+0000 7f93d35a0740 1 -- 172.21.15.44:0/1251884912 >> 172.21.15.44:0/1251884912 conn(0x560e98c724f0 msgr2=0x560e98ca4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:54.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.533+0000 7f93d35a0740 1 -- 172.21.15.44:0/1251884912 shutdown_connections 2024-04-21T23:15:54.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.533+0000 7f93d35a0740 1 -- 172.21.15.44:0/1251884912 wait complete. 2024-04-21T23:15:54.538 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.533+0000 7f93d35a0740 1 -- start start 2024-04-21T23:15:54.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.533+0000 7f93d35a0740 1 Processor -- start 2024-04-21T23:15:54.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93d35a0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98caba40 0x560e98cbd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93d35a0740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98cac720 0x560e98caf410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93d35a0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98cb4db0 0x560e98cafa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93d35a0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f93cc1ab6c0 con 0x560e98cb4db0 2024-04-21T23:15:54.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93d35a0740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f93cc1acb30 con 0x560e98cac720 2024-04-21T23:15:54.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93d35a0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f93cc1abd90 con 0x560e98caba40 2024-04-21T23:15:54.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93d35a0740 10 client.0 _ll_register_callbacks cb 0x7f93bc009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:54.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93c9d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98cb4db0 0x560e98cafa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93c9d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98cb4db0 0x560e98cafa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48870/0 (socket says 172.21.15.44:48870) 2024-04-21T23:15:54.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93c9d9b640 1 -- 172.21.15.44:0/2037491657 learned_addr learned my addr 172.21.15.44:0/2037491657 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:54.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93ca59c640 1 --2- 172.21.15.44:0/2037491657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98cac720 0x560e98caf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93c9d9b640 1 --2- 172.21.15.44:0/2037491657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98caba40 0x560e98cbd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93c9d9b640 1 -- 172.21.15.44:0/2037491657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98cac720 msgr2=0x560e98caf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:54.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93c9d9b640 1 --2- 172.21.15.44:0/2037491657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98cac720 0x560e98caf410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93c9d9b640 1 -- 172.21.15.44:0/2037491657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98cb4db0 msgr2=0x560e98cafa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:54.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93c9d9b640 1 --2- 172.21.15.44:0/2037491657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98cb4db0 0x560e98cafa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93c9d9b640 1 -- 172.21.15.44:0/2037491657 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e98cb5d50 con 0x560e98caba40 2024-04-21T23:15:54.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93ca59c640 1 --2- 172.21.15.44:0/2037491657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98cac720 0x560e98caf410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:54.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93c9d9b640 1 --2- 172.21.15.44:0/2037491657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98caba40 0x560e98cbd560 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f93c0009730 tx=0x7f93c00096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75f6127e8499d54f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:54.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:15:54.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93b1ffb640 1 -- 172.21.15.44:0/2037491657 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c0024040 con 0x560e98caba40 2024-04-21T23:15:54.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93d35a0740 1 -- 172.21.15.44:0/2037491657 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93bc02e070 con 0x560e98caba40 2024-04-21T23:15:54.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.534+0000 7f93b1ffb640 1 -- 172.21.15.44:0/2037491657 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f93c000fdf0 con 0x560e98caba40 2024-04-21T23:15:54.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.535+0000 7f93b1ffb640 1 -- 172.21.15.44:0/2037491657 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f93c0044050 con 0x560e98caba40 2024-04-21T23:15:54.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.535+0000 7f93d35a0740 10 client.26812 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:54.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.535+0000 7f93d35a0740 1 -- 172.21.15.44:0/2037491657 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560e98d2bb10 con 0x560e98caba40 2024-04-21T23:15:54.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.535+0000 7f93b1ffb640 1 -- 172.21.15.44:0/2037491657 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93bc02e070 con 0x560e98caba40 2024-04-21T23:15:54.540 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.535+0000 7f93b1ffb640 1 -- 172.21.15.44:0/2037491657 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f93c001d8b0 con 0x560e98caba40 2024-04-21T23:15:54.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.535+0000 7f93d35a0740 10 client.26812 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:54.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.535+0000 7f93d35a0740 10 client.26812 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:54.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.535+0000 7f93d35a0740 20 client.26812 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:54.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.535+0000 7f93d35a0740 1 --2- 172.21.15.44:0/2037491657 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560e98d2bee0 0x560e98d2e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.535+0000 7f93d35a0740 4 client.26812 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:15:54.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.535+0000 7f93d35a0740 1 -- 172.21.15.44:0/2037491657 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560e98c1c6b0 con 0x560e98d2bee0 2024-04-21T23:15:54.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.535+0000 7f93ca59c640 1 --2- 172.21.15.44:0/2037491657 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560e98d2bee0 0x560e98d2e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.536+0000 7f93ca59c640 1 --2- 172.21.15.44:0/2037491657 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560e98d2bee0 0x560e98d2e2d0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2f5c0a3bca1044a server_cookie=97845a680687c106 in_seq=0 out_seq=0 2024-04-21T23:15:54.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.536+0000 7f93b1ffb640 10 client.26812 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:54.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.536+0000 7f93b1ffb640 1 -- 172.21.15.44:0/2037491657 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560e98c1c6b0 con 0x560e98d2bee0 2024-04-21T23:15:54.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.537+0000 7f93b1ffb640 10 client.26812 handle_command_reply: tid=0 2024-04-21T23:15:54.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.537+0000 7f93d35a0740 1 client.26812 shutdown 2024-04-21T23:15:54.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.537+0000 7f93d35a0740 1 -- 172.21.15.44:0/2037491657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98caba40 msgr2=0x560e98cbd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.537+0000 7f93d35a0740 1 --2- 172.21.15.44:0/2037491657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98caba40 0x560e98cbd560 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f93c0009730 tx=0x7f93c00096c0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.537+0000 7f93d35a0740 1 -- 172.21.15.44:0/2037491657 shutdown_connections 2024-04-21T23:15:54.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.537+0000 7f93d35a0740 1 --2- 172.21.15.44:0/2037491657 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x560e98d2bee0 0x560e98d2e2d0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.537+0000 7f93d35a0740 1 --2- 172.21.15.44:0/2037491657 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560e98cb4db0 0x560e98cafa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.537+0000 7f93d35a0740 1 --2- 172.21.15.44:0/2037491657 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560e98cac720 0x560e98caf410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.537+0000 7f93d35a0740 1 --2- 172.21.15.44:0/2037491657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e98caba40 0x560e98cbd560 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.537+0000 7f93b1ffb640 0 client.26812 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:54.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.537+0000 7f93d35a0740 1 -- 172.21.15.44:0/2037491657 >> 172.21.15.44:0/2037491657 conn(0x560e98c724f0 msgr2=0x560e98ca54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:54.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.537+0000 7f93d35a0740 1 -- 172.21.15.44:0/2037491657 shutdown_connections 2024-04-21T23:15:54.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.537+0000 7f93d35a0740 1 -- 172.21.15.44:0/2037491657 wait complete. 2024-04-21T23:15:54.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.537+0000 7f93d35a0740 20 client.26812 trim_cache size 0 max 16384 2024-04-21T23:15:54.542 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:15:54.543 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-21T23:15:54.543 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "02dc7e0b-4971-449b-b0c1-2887bdfae5a8", 2024-04-21T23:15:54.543 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-21T23:15:54.543 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:15:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.538+0000 7f93d08d4640 1 -- 172.21.15.44:0/4090735067 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f93b40a2d00 msgr2=0x7f93b40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.538+0000 7f93d08d4640 1 --2- 172.21.15.44:0/4090735067 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f93b40a2d00 0x7f93b40a51c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f93c40049d0 tx=0x7f93c40f2220 comp rx=0 tx=0).stop 2024-04-21T23:15:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.538+0000 7f93d08d4640 1 -- 172.21.15.44:0/4090735067 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93cc103bb0 msgr2=0x7f93cc19c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.538+0000 7f93d08d4640 1 --2- 172.21.15.44:0/4090735067 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93cc103bb0 0x7f93cc19c8f0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f93c4037040 tx=0x7f93c40346e0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.538+0000 7f93d08d4640 1 -- 172.21.15.44:0/4090735067 shutdown_connections 2024-04-21T23:15:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.538+0000 7f93d08d4640 1 --2- 172.21.15.44:0/4090735067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93cc103210 0x7f93cc1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.538+0000 7f93d08d4640 1 --2- 172.21.15.44:0/4090735067 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f93b40a2d00 0x7f93b40a51c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.538+0000 7f93d08d4640 1 --2- 172.21.15.44:0/4090735067 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f93cc103bb0 0x7f93cc19c8f0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.538+0000 7f93d08d4640 1 --2- 172.21.15.44:0/4090735067 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93cc101ad0 0x7f93cc198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.538+0000 7f93d08d4640 1 -- 172.21.15.44:0/4090735067 >> 172.21.15.44:0/4090735067 conn(0x7f93cc0fd200 msgr2=0x7f93cc107580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.538+0000 7f93d08d4640 1 -- 172.21.15.44:0/4090735067 shutdown_connections 2024-04-21T23:15:54.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.538+0000 7f93d08d4640 1 -- 172.21.15.44:0/4090735067 wait complete. 2024-04-21T23:15:54.548 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:15:54.548 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:15:54.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.667+0000 7f4ec99d7640 1 -- 172.21.15.44:0/24404365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec410bec0 msgr2=0x7f4ec410e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.667+0000 7f4ec99d7640 1 --2- 172.21.15.44:0/24404365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec410bec0 0x7f4ec410e350 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc009ca0 tx=0x7f4ebc02c0f0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.667+0000 7f4ec99d7640 1 -- 172.21.15.44:0/24404365 shutdown_connections 2024-04-21T23:15:54.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.667+0000 7f4ec99d7640 1 --2- 172.21.15.44:0/24404365 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec410bec0 0x7f4ec410e350 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.667+0000 7f4ec99d7640 1 --2- 172.21.15.44:0/24404365 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ec4103360 0x7f4ec410b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.667+0000 7f4ec99d7640 1 --2- 172.21.15.44:0/24404365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec41029b0 0x7f4ec4102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.667+0000 7f4ec99d7640 1 -- 172.21.15.44:0/24404365 >> 172.21.15.44:0/24404365 conn(0x7f4ec4077760 msgr2=0x7f4ec4077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:54.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.667+0000 7f4ec99d7640 1 -- 172.21.15.44:0/24404365 shutdown_connections 2024-04-21T23:15:54.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.667+0000 7f4ec99d7640 1 -- 172.21.15.44:0/24404365 wait complete. 2024-04-21T23:15:54.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.667+0000 7f4ec99d7640 1 Processor -- start 2024-04-21T23:15:54.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.667+0000 7f4ec99d7640 1 -- start start 2024-04-21T23:15:54.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec99d7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec41029b0 0x7f4ec4198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec99d7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ec410bec0 0x7f4ec4198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec99d7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec419cad0 0x7f4ec419cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec99d7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec4111cd0 con 0x7f4ec419cad0 2024-04-21T23:15:54.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec99d7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec4111b50 con 0x7f4ec410bec0 2024-04-21T23:15:54.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec99d7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec4111e50 con 0x7f4ec41029b0 2024-04-21T23:15:54.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec419cad0 0x7f4ec419cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec37fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec419cad0 0x7f4ec419cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48902/0 (socket says 172.21.15.44:48902) 2024-04-21T23:15:54.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec37fe640 1 -- 172.21.15.44:0/1122569101 learned_addr learned my addr 172.21.15.44:0/1122569101 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:54.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ebbfff640 1 --2- 172.21.15.44:0/1122569101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ec410bec0 0x7f4ec4198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec2ffd640 1 --2- 172.21.15.44:0/1122569101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec41029b0 0x7f4ec4198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec37fe640 1 -- 172.21.15.44:0/1122569101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec41029b0 msgr2=0x7f4ec4198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec37fe640 1 --2- 172.21.15.44:0/1122569101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec41029b0 0x7f4ec4198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec37fe640 1 -- 172.21.15.44:0/1122569101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ec410bec0 msgr2=0x7f4ec4198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec37fe640 1 --2- 172.21.15.44:0/1122569101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ec410bec0 0x7f4ec4198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec37fe640 1 -- 172.21.15.44:0/1122569101 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec419d5c0 con 0x7f4ec419cad0 2024-04-21T23:15:54.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec37fe640 1 --2- 172.21.15.44:0/1122569101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec419cad0 0x7f4ec419cf80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc02c700 tx=0x7f4ebc032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:54.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec89d5640 1 -- 172.21.15.44:0/1122569101 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc041020 con 0x7f4ec419cad0 2024-04-21T23:15:54.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec89d5640 1 -- 172.21.15.44:0/1122569101 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ebc034cd0 con 0x7f4ec419cad0 2024-04-21T23:15:54.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec99d7640 1 -- 172.21.15.44:0/1122569101 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ec419d850 con 0x7f4ec419cad0 2024-04-21T23:15:54.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec89d5640 1 -- 172.21.15.44:0/1122569101 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc039040 con 0x7f4ec419cad0 2024-04-21T23:15:54.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.668+0000 7f4ec99d7640 1 -- 172.21.15.44:0/1122569101 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ec41a5010 con 0x7f4ec419cad0 2024-04-21T23:15:54.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.671+0000 7f4ecc5bf740 1 Processor -- start 2024-04-21T23:15:54.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.671+0000 7f4ec89d5640 1 -- 172.21.15.44:0/1122569101 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f4ebc044020 con 0x7f4ec419cad0 2024-04-21T23:15:54.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.671+0000 7f4ecc5bf740 1 -- start start 2024-04-21T23:15:54.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.671+0000 7f4ecc5bf740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae92dc60 0x557dae92e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.671+0000 7f4ecc5bf740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae92e580 0x557dae93b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.671+0000 7f4ecc5bf740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae93b840 0x557dae93dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.671+0000 7f4ecc5bf740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557dae892560 con 0x557dae92e580 2024-04-21T23:15:54.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.671+0000 7f4ecc5bf740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x557dae83a730 con 0x557dae92dc60 2024-04-21T23:15:54.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.671+0000 7f4ecc5bf740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557dae88e520 con 0x557dae93b840 2024-04-21T23:15:54.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.671+0000 7f4ec2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae92e580 0x557dae93b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.671+0000 7f4ebbfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae92dc60 0x557dae92e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.671+0000 7f4ec2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae92e580 0x557dae93b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48916/0 (socket says 172.21.15.44:48916) 2024-04-21T23:15:54.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.671+0000 7f4ec2ffd640 1 -- 172.21.15.44:0/3101652048 learned_addr learned my addr 172.21.15.44:0/3101652048 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:54.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.671+0000 7f4ebbfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae92dc60 0x557dae92e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:53622/0 (socket says 172.21.15.44:53622) 2024-04-21T23:15:54.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.671+0000 7f4ebbfff640 1 --2- 172.21.15.44:0/3101652048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae93b840 0x557dae93dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.672+0000 7f4ec2ffd640 1 -- 172.21.15.44:0/3101652048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae93b840 msgr2=0x557dae93dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:54.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.672+0000 7f4ec2ffd640 1 --2- 172.21.15.44:0/3101652048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae93b840 0x557dae93dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.672+0000 7f4ec2ffd640 1 -- 172.21.15.44:0/3101652048 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae92dc60 msgr2=0x557dae92e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:54.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.672+0000 7f4ec2ffd640 1 --2- 172.21.15.44:0/3101652048 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae92dc60 0x557dae92e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.672+0000 7f4ec2ffd640 1 -- 172.21.15.44:0/3101652048 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557dae6707d0 con 0x557dae92e580 2024-04-21T23:15:54.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.672+0000 7f4ec89d5640 1 --2- 172.21.15.44:0/1122569101 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4eb00a2d00 0x7f4eb00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.672+0000 7f4ec2ffd640 1 --2- 172.21.15.44:0/3101652048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae92e580 0x557dae93b300 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009d20 tx=0x7f4eb4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61c14c63abf8abc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:54.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.672+0000 7f4ebbfff640 1 --2- 172.21.15.44:0/1122569101 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4eb00a2d00 0x7f4eb00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.672+0000 7f4ec89d5640 1 -- 172.21.15.44:0/1122569101 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f4ebc0f1430 con 0x7f4ec419cad0 2024-04-21T23:15:54.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.672+0000 7f4eb9ffb640 1 -- 172.21.15.44:0/3101652048 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb401e070 con 0x557dae92e580 2024-04-21T23:15:54.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.672+0000 7f4eb9ffb640 1 -- 172.21.15.44:0/3101652048 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4eb4009e10 con 0x557dae92e580 2024-04-21T23:15:54.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.672+0000 7f4eb9ffb640 1 -- 172.21.15.44:0/3101652048 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4018910 con 0x557dae92e580 2024-04-21T23:15:54.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.673+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/3101652048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae92e580 msgr2=0x557dae93b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.673+0000 7f4ebbfff640 1 --2- 172.21.15.44:0/1122569101 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4eb00a2d00 0x7f4eb00a51c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x557dae9297f0 tx=0x7f4eac004730 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:54.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.673+0000 7f4ecc5bf740 1 --2- 172.21.15.44:0/3101652048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae92e580 0x557dae93b300 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009d20 tx=0x7f4eb4011810 comp rx=0 tx=0).stop 2024-04-21T23:15:54.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.673+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/3101652048 shutdown_connections 2024-04-21T23:15:54.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.673+0000 7f4ecc5bf740 1 --2- 172.21.15.44:0/3101652048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae93b840 0x557dae93dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.673+0000 7f4ecc5bf740 1 --2- 172.21.15.44:0/3101652048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae92e580 0x557dae93b300 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.673+0000 7f4ecc5bf740 1 --2- 172.21.15.44:0/3101652048 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae92dc60 0x557dae92e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.673+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/3101652048 >> 172.21.15.44:0/3101652048 conn(0x557dae928660 msgr2=0x557dae928a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:54.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.673+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/3101652048 shutdown_connections 2024-04-21T23:15:54.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.673+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/3101652048 wait complete. 2024-04-21T23:15:54.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.673+0000 7f4ecc5bf740 1 -- start start 2024-04-21T23:15:54.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.673+0000 7f4ecc5bf740 1 Processor -- start 2024-04-21T23:15:54.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.674+0000 7f4ecc5bf740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae92dc60 0x557dae9388e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.674+0000 7f4ecc5bf740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae92e580 0x557dae938e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.674+0000 7f4ecc5bf740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae93b840 0x557dae939360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.674+0000 7f4ecc5bf740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x557dae93e600 con 0x557dae92e580 2024-04-21T23:15:54.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.674+0000 7f4ecc5bf740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x557dae84de00 con 0x557dae93b840 2024-04-21T23:15:54.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.674+0000 7f4ecc5bf740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557dae93e780 con 0x557dae92dc60 2024-04-21T23:15:54.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.674+0000 7f4ecc5bf740 10 client.0 _ll_register_callbacks cb 0x557dae888eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:54.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.674+0000 7f4ec2ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae92dc60 0x557dae9388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.674+0000 7f4ebbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae92e580 0x557dae938e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.674+0000 7f4ec2ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae92dc60 0x557dae9388e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:55514/0 (socket says 172.21.15.44:55514) 2024-04-21T23:15:54.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.674+0000 7f4ec2ffd640 1 -- 172.21.15.44:0/357163630 learned_addr learned my addr 172.21.15.44:0/357163630 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:54.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.674+0000 7f4ec37fe640 1 --2- 172.21.15.44:0/357163630 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae93b840 0x557dae939360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.674+0000 7f4ec2ffd640 1 -- 172.21.15.44:0/357163630 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae93b840 msgr2=0x557dae939360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:54.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.674+0000 7f4ec2ffd640 1 --2- 172.21.15.44:0/357163630 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae93b840 0x557dae939360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.674+0000 7f4ec2ffd640 1 -- 172.21.15.44:0/357163630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae92e580 msgr2=0x557dae938e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:54.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.674+0000 7f4ec2ffd640 1 --2- 172.21.15.44:0/357163630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae92e580 0x557dae938e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.674+0000 7f4ec2ffd640 1 -- 172.21.15.44:0/357163630 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557dae5a63f0 con 0x557dae92dc60 2024-04-21T23:15:54.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.674+0000 7f4ebbfff640 1 --2- 172.21.15.44:0/357163630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae92e580 0x557dae938e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:54.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.674+0000 7f4ec2ffd640 1 --2- 172.21.15.44:0/357163630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae92dc60 0x557dae9388e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4005890 tx=0x7f4eb4005730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7142df81f5dea42 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:54.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.674+0000 7f4ea2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:15:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.675+0000 7f4ea2ffd640 1 -- 172.21.15.44:0/357163630 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb401e040 con 0x557dae92dc60 2024-04-21T23:15:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.675+0000 7f4ea2ffd640 1 -- 172.21.15.44:0/357163630 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f4eb4016050 con 0x557dae92dc60 2024-04-21T23:15:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.675+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/357163630 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4eb4009e10 con 0x557dae92dc60 2024-04-21T23:15:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.675+0000 7f4ea2ffd640 1 -- 172.21.15.44:0/357163630 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4eb402f050 con 0x557dae92dc60 2024-04-21T23:15:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.675+0000 7f4ecc5bf740 10 client.26824 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.675+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/357163630 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557dae5660f0 con 0x557dae92dc60 2024-04-21T23:15:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.675+0000 7f4ea2ffd640 1 -- 172.21.15.44:0/357163630 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4eb4009e10 con 0x557dae92dc60 2024-04-21T23:15:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.675+0000 7f4ea2ffd640 1 -- 172.21.15.44:0/357163630 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f4eb4021070 con 0x557dae92dc60 2024-04-21T23:15:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.675+0000 7f4ecc5bf740 10 client.26824 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.675+0000 7f4ecc5bf740 10 client.26824 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.675+0000 7f4ecc5bf740 20 client.26824 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:54.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.675+0000 7f4ecc5bf740 1 --2- 172.21.15.44:0/357163630 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557dae94bb70 0x557dae94df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.675+0000 7f4ecc5bf740 4 client.26824 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:15:54.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.675+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/357163630 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557dae89faa0 con 0x557dae94bb70 2024-04-21T23:15:54.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.676+0000 7f4ebbfff640 1 --2- 172.21.15.44:0/357163630 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557dae94bb70 0x557dae94df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.676+0000 7f4ea2ffd640 10 client.26824 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:54.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.676+0000 7f4ebbfff640 1 --2- 172.21.15.44:0/357163630 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557dae94bb70 0x557dae94df60 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=636189ee23c856fb server_cookie=1460ec4b7cf1542 in_seq=0 out_seq=0 2024-04-21T23:15:54.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.678+0000 7f4ea2ffd640 1 -- 172.21.15.44:0/357163630 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557dae89faa0 con 0x557dae94bb70 2024-04-21T23:15:54.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.678+0000 7f4ea2ffd640 10 client.26824 handle_command_reply: tid=0 2024-04-21T23:15:54.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.678+0000 7f4ecc5bf740 1 client.26824 shutdown 2024-04-21T23:15:54.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.678+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/357163630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae92dc60 msgr2=0x557dae9388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.678+0000 7f4ecc5bf740 1 --2- 172.21.15.44:0/357163630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae92dc60 0x557dae9388e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4005890 tx=0x7f4eb4005730 comp rx=0 tx=0).stop 2024-04-21T23:15:54.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.678+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/357163630 shutdown_connections 2024-04-21T23:15:54.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.678+0000 7f4ecc5bf740 1 --2- 172.21.15.44:0/357163630 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557dae94bb70 0x557dae94df60 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.678+0000 7f4ecc5bf740 1 --2- 172.21.15.44:0/357163630 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae93b840 0x557dae939360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.678+0000 7f4ecc5bf740 1 --2- 172.21.15.44:0/357163630 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae92e580 0x557dae938e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.678+0000 7f4ea2ffd640 0 client.26824 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:54.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.678+0000 7f4ecc5bf740 1 --2- 172.21.15.44:0/357163630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae92dc60 0x557dae9388e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.678+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/357163630 >> 172.21.15.44:0/357163630 conn(0x557dae928660 msgr2=0x557dae92a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:54.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.678+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/357163630 shutdown_connections 2024-04-21T23:15:54.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.678+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/357163630 wait complete. 2024-04-21T23:15:54.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.679+0000 7f4ecc5bf740 20 client.26824 trim_cache size 0 max 16384 2024-04-21T23:15:54.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.699+0000 7f4ecc5bf740 1 Processor -- start 2024-04-21T23:15:54.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.699+0000 7f4ecc5bf740 1 -- start start 2024-04-21T23:15:54.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.699+0000 7f4ecc5bf740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae92dc60 0x557dae93b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.699+0000 7f4ecc5bf740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae92e580 0x557dae937840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.699+0000 7f4ecc5bf740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae93b840 0x557dae947500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.699+0000 7f4ecc5bf740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec41aaa60 con 0x557dae92dc60 2024-04-21T23:15:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.699+0000 7f4ecc5bf740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec41ad2c0 con 0x557dae93b840 2024-04-21T23:15:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.699+0000 7f4ecc5bf740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec41ab0b0 con 0x557dae92e580 2024-04-21T23:15:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.699+0000 7f4ec2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae92dc60 0x557dae93b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.699+0000 7f4ec2ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae92dc60 0x557dae93b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48938/0 (socket says 172.21.15.44:48938) 2024-04-21T23:15:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.699+0000 7f4ec2ffd640 1 -- 172.21.15.44:0/1021603452 learned_addr learned my addr 172.21.15.44:0/1021603452 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.699+0000 7f4ec37fe640 1 --2- 172.21.15.44:0/1021603452 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae93b840 0x557dae947500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.699+0000 7f4ec2ffd640 1 -- 172.21.15.44:0/1021603452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae92e580 msgr2=0x557dae937840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.699+0000 7f4ec2ffd640 1 --2- 172.21.15.44:0/1021603452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae92e580 0x557dae937840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.699+0000 7f4ec2ffd640 1 -- 172.21.15.44:0/1021603452 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae93b840 msgr2=0x557dae947500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:54.705 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.699+0000 7f4ec2ffd640 1 --2- 172.21.15.44:0/1021603452 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae93b840 0x557dae947500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.699+0000 7f4ec2ffd640 1 -- 172.21.15.44:0/1021603452 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557dae93e4e0 con 0x557dae92dc60 2024-04-21T23:15:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.699+0000 7f4ec2ffd640 1 --2- 172.21.15.44:0/1021603452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae92dc60 0x557dae93b010 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4006010 tx=0x7f4eb4009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf69e16f02fa0a44 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.699+0000 7f4eb97fa640 1 -- 172.21.15.44:0/1021603452 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4016070 con 0x557dae92dc60 2024-04-21T23:15:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.700+0000 7f4eb97fa640 1 -- 172.21.15.44:0/1021603452 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4eb4009780 con 0x557dae92dc60 2024-04-21T23:15:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.700+0000 7f4eb97fa640 1 -- 172.21.15.44:0/1021603452 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb401e030 con 0x557dae92dc60 2024-04-21T23:15:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.700+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/1021603452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae92dc60 msgr2=0x557dae93b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.700+0000 7f4ecc5bf740 1 --2- 172.21.15.44:0/1021603452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae92dc60 0x557dae93b010 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4006010 tx=0x7f4eb4009e90 comp rx=0 tx=0).stop 2024-04-21T23:15:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.700+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/1021603452 shutdown_connections 2024-04-21T23:15:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.700+0000 7f4ecc5bf740 1 --2- 172.21.15.44:0/1021603452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae92e580 0x557dae937840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.700+0000 7f4ecc5bf740 1 --2- 172.21.15.44:0/1021603452 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae93b840 0x557dae947500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.700+0000 7f4ecc5bf740 1 --2- 172.21.15.44:0/1021603452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae92dc60 0x557dae93b010 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.706 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.700+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/1021603452 >> 172.21.15.44:0/1021603452 conn(0x557dae928660 msgr2=0x557dae92a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:54.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.700+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/1021603452 shutdown_connections 2024-04-21T23:15:54.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.700+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/1021603452 wait complete. 2024-04-21T23:15:54.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.701+0000 7f4ecc5bf740 1 -- start start 2024-04-21T23:15:54.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.701+0000 7f4ecc5bf740 1 Processor -- start 2024-04-21T23:15:54.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.701+0000 7f4ecc5bf740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae92dc60 0x557dae9355f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.701+0000 7f4ecc5bf740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae92e580 0x557dae935b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.701+0000 7f4ecc5bf740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae93b840 0x557dae936070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.701+0000 7f4ecc5bf740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec41ab780 con 0x557dae93b840 2024-04-21T23:15:54.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.701+0000 7f4ecc5bf740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec41acbf0 con 0x557dae92e580 2024-04-21T23:15:54.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.701+0000 7f4ecc5bf740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec41abe50 con 0x557dae92dc60 2024-04-21T23:15:54.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.701+0000 7f4ecc5bf740 10 client.0 _ll_register_callbacks cb 0x7f4eb401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:15:54.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.701+0000 7f4ec2ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae92dc60 0x557dae9355f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.701+0000 7f4ebbfff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae92e580 0x557dae935b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.701+0000 7f4ec2ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae92dc60 0x557dae9355f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:55528/0 (socket says 172.21.15.44:55528) 2024-04-21T23:15:54.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.701+0000 7f4ec2ffd640 1 -- 172.21.15.44:0/2656871486 learned_addr learned my addr 172.21.15.44:0/2656871486 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:15:54.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.701+0000 7f4ec37fe640 1 --2- 172.21.15.44:0/2656871486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae93b840 0x557dae936070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.702+0000 7f4ec2ffd640 1 -- 172.21.15.44:0/2656871486 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae92e580 msgr2=0x557dae935b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:54.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.702+0000 7f4ec2ffd640 1 --2- 172.21.15.44:0/2656871486 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae92e580 0x557dae935b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.702+0000 7f4ec2ffd640 1 -- 172.21.15.44:0/2656871486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae93b840 msgr2=0x557dae936070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:15:54.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.702+0000 7f4ec2ffd640 1 --2- 172.21.15.44:0/2656871486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae93b840 0x557dae936070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.702+0000 7f4ec2ffd640 1 -- 172.21.15.44:0/2656871486 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557dae9b0f60 con 0x557dae92dc60 2024-04-21T23:15:54.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.702+0000 7f4ebbfff640 1 --2- 172.21.15.44:0/2656871486 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae92e580 0x557dae935b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:54.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.702+0000 7f4ec37fe640 1 --2- 172.21.15.44:0/2656871486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae93b840 0x557dae936070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:15:54.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.702+0000 7f4ec2ffd640 1 --2- 172.21.15.44:0/2656871486 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae92dc60 0x557dae9355f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4018a00 tx=0x7f4eb4018da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f3de5ed348f6e0a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:15:54.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.702+0000 7f4ea2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:15:54.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.702+0000 7f4ea2ffd640 1 -- 172.21.15.44:0/2656871486 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4038060 con 0x557dae92dc60 2024-04-21T23:15:54.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.702+0000 7f4ea2ffd640 1 -- 172.21.15.44:0/2656871486 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9533+0+0 (secure 0 0 0) 0x7f4eb40185c0 con 0x557dae92dc60 2024-04-21T23:15:54.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.702+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/2656871486 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4eb4009780 con 0x557dae92dc60 2024-04-21T23:15:54.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.702+0000 7f4ea2ffd640 1 -- 172.21.15.44:0/2656871486 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4eb403f050 con 0x557dae92dc60 2024-04-21T23:15:54.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.702+0000 7f4ecc5bf740 10 client.26827 fetch_fsmap learned FSMap version 29 2024-04-21T23:15:54.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.702+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/2656871486 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557dae9b2240 con 0x557dae92dc60 2024-04-21T23:15:54.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.703+0000 7f4ea2ffd640 1 -- 172.21.15.44:0/2656871486 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4eb4009780 con 0x557dae92dc60 2024-04-21T23:15:54.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.703+0000 7f4ea2ffd640 1 -- 172.21.15.44:0/2656871486 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f4eb4021070 con 0x557dae92dc60 2024-04-21T23:15:54.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.703+0000 7f4ecc5bf740 10 client.26827 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:15:54.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.703+0000 7f4ecc5bf740 10 client.26827 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:15:54.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.703+0000 7f4ecc5bf740 20 client.26827 populate_metadata read hostname 'smithi044' 2024-04-21T23:15:54.709 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.703+0000 7f4ecc5bf740 1 --2- 172.21.15.44:0/2656871486 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557dae9b2950 0x557dae9b4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:15:54.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.703+0000 7f4ecc5bf740 4 client.26827 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:15:54.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.703+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/2656871486 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557dae899df0 con 0x557dae9b2950 2024-04-21T23:15:54.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.703+0000 7f4ebbfff640 1 --2- 172.21.15.44:0/2656871486 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557dae9b2950 0x557dae9b4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:15:54.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.703+0000 7f4ebbfff640 1 --2- 172.21.15.44:0/2656871486 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557dae9b2950 0x557dae9b4d70 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3103979d01d7441 server_cookie=628509848e553b1a in_seq=0 out_seq=0 2024-04-21T23:15:54.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.703+0000 7f4ea2ffd640 10 client.26827 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:54.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.704+0000 7f4ea2ffd640 1 -- 172.21.15.44:0/2656871486 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x557dae899df0 con 0x557dae9b2950 2024-04-21T23:15:54.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.704+0000 7f4ea2ffd640 10 client.26827 handle_command_reply: tid=0 2024-04-21T23:15:54.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.704+0000 7f4ecc5bf740 1 client.26827 shutdown 2024-04-21T23:15:54.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.704+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/2656871486 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae92dc60 msgr2=0x557dae9355f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.704+0000 7f4ecc5bf740 1 --2- 172.21.15.44:0/2656871486 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae92dc60 0x557dae9355f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4018a00 tx=0x7f4eb4018da0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.704+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/2656871486 shutdown_connections 2024-04-21T23:15:54.710 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.704+0000 7f4ecc5bf740 1 --2- 172.21.15.44:0/2656871486 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x557dae9b2950 0x557dae9b4d70 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.704+0000 7f4ecc5bf740 1 --2- 172.21.15.44:0/2656871486 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x557dae93b840 0x557dae936070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.704+0000 7f4ea2ffd640 0 client.26827 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:15:54.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.704+0000 7f4ecc5bf740 1 --2- 172.21.15.44:0/2656871486 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x557dae92e580 0x557dae935b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.704+0000 7f4ecc5bf740 1 --2- 172.21.15.44:0/2656871486 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557dae92dc60 0x557dae9355f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.704+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/2656871486 >> 172.21.15.44:0/2656871486 conn(0x557dae928660 msgr2=0x557dae938880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:54.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.704+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/2656871486 shutdown_connections 2024-04-21T23:15:54.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.704+0000 7f4ecc5bf740 1 -- 172.21.15.44:0/2656871486 wait complete. 2024-04-21T23:15:54.711 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.704+0000 7f4ecc5bf740 20 client.26827 trim_cache size 0 max 16384 2024-04-21T23:15:54.711 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:15:54.712 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:15:54.712 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-21T23:15:54.712 INFO:teuthology.orchestra.run.smithi044.stdout: "02dc7e0b-4971-449b-b0c1-2887bdfae5a8": { 2024-04-21T23:15:54.712 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-21T23:15:54.712 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "02dc7e0b-4971-449b-b0c1-2887bdfae5a8", 2024-04-21T23:15:54.712 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-04-21T23:15:54.712 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:15:54.712 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:15:54.712 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:15:54.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.705+0000 7f4ec99d7640 1 -- 172.21.15.44:0/1122569101 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4eb00a2d00 msgr2=0x7f4eb00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.705+0000 7f4ec99d7640 1 --2- 172.21.15.44:0/1122569101 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4eb00a2d00 0x7f4eb00a51c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x557dae9297f0 tx=0x7f4eac004730 comp rx=0 tx=0).stop 2024-04-21T23:15:54.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.705+0000 7f4ec99d7640 1 -- 172.21.15.44:0/1122569101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec419cad0 msgr2=0x7f4ec419cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:15:54.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.705+0000 7f4ec99d7640 1 --2- 172.21.15.44:0/1122569101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec419cad0 0x7f4ec419cf80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc02c700 tx=0x7f4ebc032f20 comp rx=0 tx=0).stop 2024-04-21T23:15:54.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.705+0000 7f4ec99d7640 1 -- 172.21.15.44:0/1122569101 shutdown_connections 2024-04-21T23:15:54.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.705+0000 7f4ec99d7640 1 --2- 172.21.15.44:0/1122569101 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4eb00a2d00 0x7f4eb00a51c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.705+0000 7f4ec99d7640 1 --2- 172.21.15.44:0/1122569101 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4ec419cad0 0x7f4ec419cf80 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.705+0000 7f4ec99d7640 1 --2- 172.21.15.44:0/1122569101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4ec410bec0 0x7f4ec4198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.705+0000 7f4ec99d7640 1 --2- 172.21.15.44:0/1122569101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ec41029b0 0x7f4ec4198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:15:54.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.705+0000 7f4ec99d7640 1 -- 172.21.15.44:0/1122569101 >> 172.21.15.44:0/1122569101 conn(0x7f4ec4077760 msgr2=0x7f4ec40fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:15:54.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.705+0000 7f4ec99d7640 1 -- 172.21.15.44:0/1122569101 shutdown_connections 2024-04-21T23:15:54.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:15:54.705+0000 7f4ec99d7640 1 -- 172.21.15.44:0/1122569101 wait complete. 2024-04-21T23:15:54.714 INFO:tasks.workunit.client.0.smithi044.stdout: 1 823892 12.39 MB/sec execute 224 sec latency 18.343 ms 2024-04-21T23:15:54.715 INFO:tasks.cephfs.filesystem:scrub status for tag:02dc7e0b-4971-449b-b0c1-2887bdfae5a8 - {'path': '/', 'tag': '02dc7e0b-4971-449b-b0c1-2887bdfae5a8', 'options': 'recursive,force'} 2024-04-21T23:15:55.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 825243 12.36 MB/sec execute 225 sec latency 77.305 ms 2024-04-21T23:15:56.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:55 smithi044 ceph-mon[27934]: scrub queued for path: / 2024-04-21T23:15:56.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:55 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:15:56.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:55 smithi044 ceph-mon[27934]: scrub summary: active paths [/] 2024-04-21T23:15:56.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:55 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:15:56.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:55 smithi044 ceph-mon[27934]: pgmap v221: 129 pgs: 129 active+clean; 81 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 230 op/s 2024-04-21T23:15:56.213 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:55 smithi055 ceph-mon[31117]: scrub queued for path: / 2024-04-21T23:15:56.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:55 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:15:56.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:55 smithi055 ceph-mon[31117]: scrub summary: active paths [/] 2024-04-21T23:15:56.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:55 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:15:56.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:55 smithi055 ceph-mon[31117]: pgmap v221: 129 pgs: 129 active+clean; 81 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 230 op/s 2024-04-21T23:15:56.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:55 smithi069 ceph-mon[31816]: scrub queued for path: / 2024-04-21T23:15:56.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:55 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:15:56.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:55 smithi069 ceph-mon[31816]: scrub summary: active paths [/] 2024-04-21T23:15:56.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:55 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:15:56.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:55 smithi069 ceph-mon[31816]: pgmap v221: 129 pgs: 129 active+clean; 81 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 230 op/s 2024-04-21T23:15:56.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 827514 12.36 MB/sec execute 226 sec latency 53.426 ms 2024-04-21T23:15:57.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 829054 12.33 MB/sec execute 227 sec latency 31.931 ms 2024-04-21T23:15:58.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:57 smithi044 ceph-mon[27934]: pgmap v222: 129 pgs: 129 active+clean; 79 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.1 MiB/s wr, 380 op/s 2024-04-21T23:15:58.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:57 smithi044 ceph-mon[27934]: scrub summary: idle 2024-04-21T23:15:58.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:57 smithi055 ceph-mon[31117]: pgmap v222: 129 pgs: 129 active+clean; 79 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.1 MiB/s wr, 380 op/s 2024-04-21T23:15:58.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:57 smithi055 ceph-mon[31117]: scrub summary: idle 2024-04-21T23:15:58.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:57 smithi069 ceph-mon[31816]: pgmap v222: 129 pgs: 129 active+clean; 79 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.1 MiB/s wr, 380 op/s 2024-04-21T23:15:58.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:57 smithi069 ceph-mon[31816]: scrub summary: idle 2024-04-21T23:15:58.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 831381 12.34 MB/sec execute 228 sec latency 12.655 ms 2024-04-21T23:15:59.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 833558 12.33 MB/sec execute 229 sec latency 21.517 ms 2024-04-21T23:16:00.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:15:59 smithi044 ceph-mon[27934]: pgmap v223: 129 pgs: 129 active+clean; 70 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.1 MiB/s wr, 449 op/s 2024-04-21T23:16:00.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:15:59 smithi069 ceph-mon[31816]: pgmap v223: 129 pgs: 129 active+clean; 70 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.1 MiB/s wr, 449 op/s 2024-04-21T23:16:00.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:15:59 smithi055 ceph-mon[31117]: pgmap v223: 129 pgs: 129 active+clean; 70 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.1 MiB/s wr, 449 op/s 2024-04-21T23:16:00.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 835002 12.32 MB/sec execute 230 sec latency 69.174 ms 2024-04-21T23:16:01.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 835349 12.27 MB/sec execute 231 sec latency 88.707 ms 2024-04-21T23:16:02.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:01 smithi044 ceph-mon[27934]: pgmap v224: 129 pgs: 129 active+clean; 70 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 384 op/s 2024-04-21T23:16:02.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:01 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:16:02.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:01 smithi055 ceph-mon[31117]: pgmap v224: 129 pgs: 129 active+clean; 70 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 384 op/s 2024-04-21T23:16:02.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:01 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:16:02.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:01 smithi069 ceph-mon[31816]: pgmap v224: 129 pgs: 129 active+clean; 70 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 384 op/s 2024-04-21T23:16:02.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:01 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:16:02.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 836795 12.24 MB/sec execute 232 sec latency 81.843 ms 2024-04-21T23:16:03.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:02 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:16:03.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:02 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:16:03.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:02 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:16:03.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:02 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:16:03.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:02 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:16:03.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:02 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:16:03.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:02 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:16:03.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:02 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:16:03.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:02 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:16:03.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 837913 12.23 MB/sec execute 233 sec latency 115.192 ms 2024-04-21T23:16:04.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:03 smithi044 ceph-mon[27934]: pgmap v225: 129 pgs: 129 active+clean; 76 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.7 MiB/s wr, 511 op/s 2024-04-21T23:16:04.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:03 smithi044 ceph-mon[27934]: osdmap e102: 12 total, 12 up, 12 in 2024-04-21T23:16:04.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:03 smithi055 ceph-mon[31117]: pgmap v225: 129 pgs: 129 active+clean; 76 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.7 MiB/s wr, 511 op/s 2024-04-21T23:16:04.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:03 smithi055 ceph-mon[31117]: osdmap e102: 12 total, 12 up, 12 in 2024-04-21T23:16:04.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:03 smithi069 ceph-mon[31816]: pgmap v225: 129 pgs: 129 active+clean; 76 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.7 MiB/s wr, 511 op/s 2024-04-21T23:16:04.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:03 smithi069 ceph-mon[31816]: osdmap e102: 12 total, 12 up, 12 in 2024-04-21T23:16:04.706 INFO:tasks.workunit.client.0.smithi044.stdout: 1 838352 12.19 MB/sec execute 234 sec latency 24.833 ms 2024-04-21T23:16:05.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.303+0000 7f630ab0f640 1 -- 172.21.15.44:0/1834500025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6304106630 msgr2=0x7f6304106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:05.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.303+0000 7f630ab0f640 1 --2- 172.21.15.44:0/1834500025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6304106630 0x7f6304106a10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009cf0 tx=0x7f62fc02e310 comp rx=0 tx=0).stop 2024-04-21T23:16:05.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.304+0000 7f630ab0f640 1 -- 172.21.15.44:0/1834500025 shutdown_connections 2024-04-21T23:16:05.303 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.304+0000 7f630ab0f640 1 --2- 172.21.15.44:0/1834500025 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6304100fe0 0x7f630410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.304+0000 7f630ab0f640 1 --2- 172.21.15.44:0/1834500025 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6304100620 0x7f6304100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.304+0000 7f630ab0f640 1 --2- 172.21.15.44:0/1834500025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6304106630 0x7f6304106a10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.304 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.304+0000 7f630ab0f640 1 -- 172.21.15.44:0/1834500025 >> 172.21.15.44:0/1834500025 conn(0x7f6304077760 msgr2=0x7f6304077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:05.308 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.305+0000 7f630ab0f640 1 -- 172.21.15.44:0/1834500025 shutdown_connections 2024-04-21T23:16:05.308 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.307+0000 7f630ab0f640 1 -- 172.21.15.44:0/1834500025 wait complete. 2024-04-21T23:16:05.308 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.307+0000 7f630ab0f640 1 Processor -- start 2024-04-21T23:16:05.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.309+0000 7f630ab0f640 1 -- start start 2024-04-21T23:16:05.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.309+0000 7f630ab0f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6304100620 0x7f630410bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.309+0000 7f630ab0f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6304100fe0 0x7f630410c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.309+0000 7f630ab0f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6304106630 0x7f630410ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.309+0000 7f630ab0f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f630410f3a0 con 0x7f6304100fe0 2024-04-21T23:16:05.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.309+0000 7f630ab0f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f630410f220 con 0x7f6304106630 2024-04-21T23:16:05.309 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.309+0000 7f630ab0f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f630410f520 con 0x7f6304100620 2024-04-21T23:16:05.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.309+0000 7f63088ab640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6304100620 0x7f630410bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:05.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.309+0000 7f63088ab640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6304100620 0x7f630410bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:43230/0 (socket says 172.21.15.44:43230) 2024-04-21T23:16:05.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.309+0000 7f63088ab640 1 -- 172.21.15.44:0/3605399348 learned_addr learned my addr 172.21.15.44:0/3605399348 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:05.310 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.309+0000 7f63090ac640 1 --2- 172.21.15.44:0/3605399348 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6304106630 0x7f630410ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:05.311 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.310+0000 7f6303fff640 1 --2- 172.21.15.44:0/3605399348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6304100fe0 0x7f630410c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:05.311 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.310+0000 7f63090ac640 1 -- 172.21.15.44:0/3605399348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6304100620 msgr2=0x7f630410bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:05.311 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.310+0000 7f63090ac640 1 --2- 172.21.15.44:0/3605399348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6304100620 0x7f630410bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.312 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.310+0000 7f63090ac640 1 -- 172.21.15.44:0/3605399348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6304100fe0 msgr2=0x7f630410c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:05.312 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.310+0000 7f63090ac640 1 --2- 172.21.15.44:0/3605399348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6304100fe0 0x7f630410c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.312 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.310+0000 7f63090ac640 1 -- 172.21.15.44:0/3605399348 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f630406b260 con 0x7f6304106630 2024-04-21T23:16:05.312 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.310+0000 7f6303fff640 1 --2- 172.21.15.44:0/3605399348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6304100fe0 0x7f630410c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:05.312 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.311+0000 7f63090ac640 1 --2- 172.21.15.44:0/3605399348 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6304106630 0x7f630410ca70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f62f800d6b0 tx=0x7f62f800db70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:05.312 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.311+0000 7f6301ffb640 1 -- 172.21.15.44:0/3605399348 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f8017070 con 0x7f6304106630 2024-04-21T23:16:05.312 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.311+0000 7f6301ffb640 1 -- 172.21.15.44:0/3605399348 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62f8009820 con 0x7f6304106630 2024-04-21T23:16:05.312 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.311+0000 7f6301ffb640 1 -- 172.21.15.44:0/3605399348 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f8004d00 con 0x7f6304106630 2024-04-21T23:16:05.312 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.311+0000 7f630ab0f640 1 -- 172.21.15.44:0/3605399348 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f630406b550 con 0x7f6304106630 2024-04-21T23:16:05.313 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.312+0000 7f630ab0f640 1 -- 172.21.15.44:0/3605399348 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f630406ba50 con 0x7f6304106630 2024-04-21T23:16:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.316+0000 7f630d7cb740 1 Processor -- start 2024-04-21T23:16:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.318+0000 7f6301ffb640 1 -- 172.21.15.44:0/3605399348 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f62f80099d0 con 0x7f6304106630 2024-04-21T23:16:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.318+0000 7f630d7cb740 1 -- start start 2024-04-21T23:16:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.318+0000 7f630d7cb740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f6fb20 0x563bf2f6ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.318+0000 7f630d7cb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f70440 0x563bf2f78cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.318+0000 7f630d7cb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563bf2f79210 0x563bf2f7b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.318+0000 7f630d7cb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563bf2ed3b70 con 0x563bf2f79210 2024-04-21T23:16:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.318+0000 7f630d7cb740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x563bf2e7c470 con 0x563bf2f6fb20 2024-04-21T23:16:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.318+0000 7f630d7cb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563bf2ecfb30 con 0x563bf2f70440 2024-04-21T23:16:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.319+0000 7f6301ffb640 1 --2- 172.21.15.44:0/3605399348 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f62ec0a2e80 0x7f62ec0a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.319+0000 7f6301ffb640 1 -- 172.21.15.44:0/3605399348 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9621+0+0 (secure 0 0 0) 0x7f62f80cea20 con 0x7f6304106630 2024-04-21T23:16:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.319+0000 7f63088ab640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f70440 0x563bf2f78cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.319+0000 7f63088ab640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f70440 0x563bf2f78cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:43242/0 (socket says 172.21.15.44:43242) 2024-04-21T23:16:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.319+0000 7f63088ab640 1 -- 172.21.15.44:0/1711680428 learned_addr learned my addr 172.21.15.44:0/1711680428 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.319+0000 7f63090ac640 1 --2- 172.21.15.44:0/3605399348 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f62ec0a2e80 0x7f62ec0a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.320+0000 7f63088ab640 1 -- 172.21.15.44:0/1711680428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f6fb20 msgr2=0x563bf2f6ff00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T23:16:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.320+0000 7f63088ab640 1 --2- 172.21.15.44:0/1711680428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f6fb20 0x563bf2f6ff00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.320+0000 7f63088ab640 1 -- 172.21.15.44:0/1711680428 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563bf2f79210 msgr2=0x563bf2f7b600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T23:16:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.320+0000 7f63088ab640 1 --2- 172.21.15.44:0/1711680428 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563bf2f79210 0x563bf2f7b600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.321 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.320+0000 7f63088ab640 1 -- 172.21.15.44:0/1711680428 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bf2be8210 con 0x563bf2f70440 2024-04-21T23:16:05.322 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.322+0000 7f63088ab640 1 --2- 172.21.15.44:0/1711680428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f70440 0x563bf2f78cd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00e500 tx=0x7f62fc034ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df991cd8fc08d792 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:05.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.322+0000 7f62eaffd640 1 -- 172.21.15.44:0/1711680428 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc044020 con 0x563bf2f70440 2024-04-21T23:16:05.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.322+0000 7f62eaffd640 1 -- 172.21.15.44:0/1711680428 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62fc03c040 con 0x563bf2f70440 2024-04-21T23:16:05.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.322+0000 7f62eaffd640 1 -- 172.21.15.44:0/1711680428 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc03b070 con 0x563bf2f70440 2024-04-21T23:16:05.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.323+0000 7f630d7cb740 1 -- 172.21.15.44:0/1711680428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f70440 msgr2=0x563bf2f78cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:05.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.324+0000 7f630d7cb740 1 --2- 172.21.15.44:0/1711680428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f70440 0x563bf2f78cd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00e500 tx=0x7f62fc034ef0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.324+0000 7f630d7cb740 1 -- 172.21.15.44:0/1711680428 shutdown_connections 2024-04-21T23:16:05.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.324+0000 7f630d7cb740 1 --2- 172.21.15.44:0/1711680428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f70440 0x563bf2f78cd0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.324+0000 7f630d7cb740 1 --2- 172.21.15.44:0/1711680428 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563bf2f79210 0x563bf2f7b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.324+0000 7f630d7cb740 1 --2- 172.21.15.44:0/1711680428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f6fb20 0x563bf2f6ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.324+0000 7f630d7cb740 1 -- 172.21.15.44:0/1711680428 >> 172.21.15.44:0/1711680428 conn(0x563bf2f6a440 msgr2=0x563bf2f6a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:05.324 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.324+0000 7f63090ac640 1 --2- 172.21.15.44:0/3605399348 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f62ec0a2e80 0x7f62ec0a5340 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f62f8005d80 tx=0x7f62f8005cd0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.326+0000 7f630d7cb740 1 -- 172.21.15.44:0/1711680428 shutdown_connections 2024-04-21T23:16:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.327+0000 7f630d7cb740 1 -- 172.21.15.44:0/1711680428 wait complete. 2024-04-21T23:16:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.328+0000 7f630d7cb740 1 -- start start 2024-04-21T23:16:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.328+0000 7f630d7cb740 1 Processor -- start 2024-04-21T23:16:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.328+0000 7f630d7cb740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f6fb20 0x563bf2f76290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.328+0000 7f630d7cb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563bf2f70440 0x563bf2f767d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.328+0000 7f630d7cb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f79210 0x563bf2f76d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.329+0000 7f630d7cb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563bf2f7bde0 con 0x563bf2f70440 2024-04-21T23:16:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.329+0000 7f630d7cb740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x563bf2e8fa10 con 0x563bf2f6fb20 2024-04-21T23:16:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.329+0000 7f630d7cb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563bf2f7bf60 con 0x563bf2f79210 2024-04-21T23:16:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.329+0000 7f6303fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f6fb20 0x563bf2f76290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:05.329 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.329+0000 7f6303fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f6fb20 0x563bf2f76290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:36610/0 (socket says 172.21.15.44:36610) 2024-04-21T23:16:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.329+0000 7f6303fff640 1 -- 172.21.15.44:0/1680753279 learned_addr learned my addr 172.21.15.44:0/1680753279 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.329+0000 7f6303fff640 1 --2- 172.21.15.44:0/1680753279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f79210 0x563bf2f76d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.329+0000 7f630d7cb740 10 client.0 _ll_register_callbacks cb 0x563bf2ecab60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.329+0000 7f6303fff640 1 -- 172.21.15.44:0/1680753279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f79210 msgr2=0x563bf2f76d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.329+0000 7f6303fff640 1 --2- 172.21.15.44:0/1680753279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f79210 0x563bf2f76d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.329+0000 7f6303fff640 1 -- 172.21.15.44:0/1680753279 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563bf2f70440 msgr2=0x563bf2f767d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T23:16:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.329+0000 7f6303fff640 1 --2- 172.21.15.44:0/1680753279 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563bf2f70440 0x563bf2f767d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.329+0000 7f6303fff640 1 -- 172.21.15.44:0/1680753279 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bf2cb4620 con 0x563bf2f6fb20 2024-04-21T23:16:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.330+0000 7f6303fff640 1 --2- 172.21.15.44:0/1680753279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f6fb20 0x563bf2f76290 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f62f4015480 tx=0x7f62f4009390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f844d3defc5c1f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.331+0000 7f62e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:16:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.331+0000 7f62e3fff640 1 -- 172.21.15.44:0/1680753279 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f403c020 con 0x563bf2f6fb20 2024-04-21T23:16:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.331+0000 7f630d7cb740 1 -- 172.21.15.44:0/1680753279 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62fc03c040 con 0x563bf2f6fb20 2024-04-21T23:16:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.331+0000 7f62e3fff640 1 -- 172.21.15.44:0/1680753279 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9621+0+0 (secure 0 0 0) 0x7f62f400ca80 con 0x563bf2f6fb20 2024-04-21T23:16:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.331+0000 7f62e3fff640 1 -- 172.21.15.44:0/1680753279 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f62f4047050 con 0x563bf2f6fb20 2024-04-21T23:16:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.332+0000 7f630d7cb740 10 client.26858 fetch_fsmap learned FSMap version 29 2024-04-21T23:16:05.332 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.332+0000 7f630d7cb740 1 -- 172.21.15.44:0/1680753279 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563bf2f87f50 con 0x563bf2f6fb20 2024-04-21T23:16:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.335+0000 7f62e3fff640 1 -- 172.21.15.44:0/1680753279 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62f400cc80 con 0x563bf2f6fb20 2024-04-21T23:16:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.335+0000 7f62e3fff640 1 -- 172.21.15.44:0/1680753279 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f62f4043490 con 0x563bf2f6fb20 2024-04-21T23:16:05.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.335+0000 7f630d7cb740 10 client.26858 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:16:05.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.335+0000 7f630d7cb740 10 client.26858 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:05.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.335+0000 7f630d7cb740 20 client.26858 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:05.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.335+0000 7f630d7cb740 1 --2- 172.21.15.44:0/1680753279 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563bf2f89880 0x563bf2f8bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.335+0000 7f630d7cb740 4 client.26858 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:16:05.335 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.335+0000 7f630d7cb740 1 -- 172.21.15.44:0/1680753279 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f62fc03c040 con 0x563bf2f89880 2024-04-21T23:16:05.341 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.341+0000 7f63088ab640 1 --2- 172.21.15.44:0/1680753279 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563bf2f89880 0x563bf2f8bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:05.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.348+0000 7f63088ab640 1 --2- 172.21.15.44:0/1680753279 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563bf2f89880 0x563bf2f8bd20 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73d450aaa0a84fd7 server_cookie=5f1dedcf4f5954a5 in_seq=0 out_seq=0 2024-04-21T23:16:05.347 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.348+0000 7f62e3fff640 10 client.26858 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:05.349 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.350+0000 7f62e3fff640 1 -- 172.21.15.44:0/1680753279 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f62fc03c040 con 0x563bf2f89880 2024-04-21T23:16:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.350+0000 7f62e3fff640 10 client.26858 handle_command_reply: tid=0 2024-04-21T23:16:05.350 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.350+0000 7f630d7cb740 1 client.26858 shutdown 2024-04-21T23:16:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.351+0000 7f630d7cb740 1 -- 172.21.15.44:0/1680753279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f6fb20 msgr2=0x563bf2f76290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.351+0000 7f630d7cb740 1 --2- 172.21.15.44:0/1680753279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f6fb20 0x563bf2f76290 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f62f4015480 tx=0x7f62f4009390 comp rx=0 tx=0).stop 2024-04-21T23:16:05.351 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.351+0000 7f630d7cb740 1 -- 172.21.15.44:0/1680753279 shutdown_connections 2024-04-21T23:16:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.351+0000 7f630d7cb740 1 --2- 172.21.15.44:0/1680753279 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563bf2f89880 0x563bf2f8bd20 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.351+0000 7f630d7cb740 1 --2- 172.21.15.44:0/1680753279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f79210 0x563bf2f76d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.351+0000 7f630d7cb740 1 --2- 172.21.15.44:0/1680753279 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563bf2f70440 0x563bf2f767d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.351+0000 7f630d7cb740 1 --2- 172.21.15.44:0/1680753279 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f6fb20 0x563bf2f76290 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.351+0000 7f630d7cb740 1 -- 172.21.15.44:0/1680753279 >> 172.21.15.44:0/1680753279 conn(0x563bf2f6a440 msgr2=0x563bf2f6c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.352+0000 7f62e3fff640 0 client.26858 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.353+0000 7f630d7cb740 1 -- 172.21.15.44:0/1680753279 shutdown_connections 2024-04-21T23:16:05.355 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.355+0000 7f630d7cb740 1 -- 172.21.15.44:0/1680753279 wait complete. 2024-04-21T23:16:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.357+0000 7f630d7cb740 20 client.26858 trim_cache size 0 max 16384 2024-04-21T23:16:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.393+0000 7f630d7cb740 1 Processor -- start 2024-04-21T23:16:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.393+0000 7f630d7cb740 1 -- start start 2024-04-21T23:16:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.393+0000 7f630d7cb740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f6fb20 0x563bf2f789c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.394+0000 7f630d7cb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f70440 0x563bf2f84ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.394+0000 7f630d7cb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563bf2f79210 0x563bf2f76b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.394+0000 7f630d7cb740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f630404fd50 con 0x563bf2f79210 2024-04-21T23:16:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.394+0000 7f630d7cb740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f63041b5da0 con 0x563bf2f6fb20 2024-04-21T23:16:05.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.394+0000 7f630d7cb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f63041b3bb0 con 0x563bf2f70440 2024-04-21T23:16:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.396+0000 7f63088ab640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f70440 0x563bf2f84ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.396+0000 7f63088ab640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f70440 0x563bf2f84ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:43270/0 (socket says 172.21.15.44:43270) 2024-04-21T23:16:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.396+0000 7f63088ab640 1 -- 172.21.15.44:0/2174651408 learned_addr learned my addr 172.21.15.44:0/2174651408 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.396+0000 7f6303fff640 1 --2- 172.21.15.44:0/2174651408 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f6fb20 0x563bf2f789c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.396+0000 7f6303fff640 1 --2- 172.21.15.44:0/2174651408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563bf2f79210 0x563bf2f76b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.397+0000 7f63088ab640 1 -- 172.21.15.44:0/2174651408 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f6fb20 msgr2=0x563bf2f789c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.397+0000 7f63088ab640 1 --2- 172.21.15.44:0/2174651408 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f6fb20 0x563bf2f789c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.397+0000 7f63088ab640 1 -- 172.21.15.44:0/2174651408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563bf2f79210 msgr2=0x563bf2f76b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.397+0000 7f63088ab640 1 --2- 172.21.15.44:0/2174651408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563bf2f79210 0x563bf2f76b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.396 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.397+0000 7f63088ab640 1 -- 172.21.15.44:0/2174651408 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bf2f7b970 con 0x563bf2f70440 2024-04-21T23:16:05.397 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.397+0000 7f63088ab640 1 --2- 172.21.15.44:0/2174651408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f70440 0x563bf2f84ca0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009cc0 tx=0x7f62fc004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fae109b62d0905bb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:05.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.398+0000 7f62ea7fc640 1 -- 172.21.15.44:0/2174651408 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc044040 con 0x563bf2f70440 2024-04-21T23:16:05.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.398+0000 7f62ea7fc640 1 -- 172.21.15.44:0/2174651408 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62fc04d030 con 0x563bf2f70440 2024-04-21T23:16:05.398 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.398+0000 7f62ea7fc640 1 -- 172.21.15.44:0/2174651408 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc03b040 con 0x563bf2f70440 2024-04-21T23:16:05.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.401+0000 7f630d7cb740 1 -- 172.21.15.44:0/2174651408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f70440 msgr2=0x563bf2f84ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:05.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.402+0000 7f630d7cb740 1 --2- 172.21.15.44:0/2174651408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f70440 0x563bf2f84ca0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009cc0 tx=0x7f62fc004a80 comp rx=0 tx=0).stop 2024-04-21T23:16:05.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.403+0000 7f630d7cb740 1 -- 172.21.15.44:0/2174651408 shutdown_connections 2024-04-21T23:16:05.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.403+0000 7f630d7cb740 1 --2- 172.21.15.44:0/2174651408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f70440 0x563bf2f84ca0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.403+0000 7f630d7cb740 1 --2- 172.21.15.44:0/2174651408 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563bf2f79210 0x563bf2f76b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.403+0000 7f630d7cb740 1 --2- 172.21.15.44:0/2174651408 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f6fb20 0x563bf2f789c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.403+0000 7f630d7cb740 1 -- 172.21.15.44:0/2174651408 >> 172.21.15.44:0/2174651408 conn(0x563bf2f6a440 msgr2=0x563bf2f74b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.407+0000 7f630d7cb740 1 -- 172.21.15.44:0/2174651408 shutdown_connections 2024-04-21T23:16:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.410+0000 7f630d7cb740 1 -- 172.21.15.44:0/2174651408 wait complete. 2024-04-21T23:16:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.410+0000 7f630d7cb740 1 -- start start 2024-04-21T23:16:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.410+0000 7f630d7cb740 1 Processor -- start 2024-04-21T23:16:05.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.411+0000 7f630d7cb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f6fb20 0x563bf2f81720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.411+0000 7f630d7cb740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f70440 0x563bf2f735f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.411+0000 7f630d7cb740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563bf2f79210 0x563bf2f73ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.412+0000 7f63088ab640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f70440 0x563bf2f735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:05.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.412+0000 7f63088ab640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f70440 0x563bf2f735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:36638/0 (socket says 172.21.15.44:36638) 2024-04-21T23:16:05.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.412+0000 7f63088ab640 1 -- 172.21.15.44:0/2679349725 learned_addr learned my addr 172.21.15.44:0/2679349725 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:05.411 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.412+0000 7f630d7cb740 1 -- 172.21.15.44:0/2679349725 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f63041b4260 con 0x563bf2f79210 2024-04-21T23:16:05.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.412+0000 7f630d7cb740 1 -- 172.21.15.44:0/2679349725 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f63041b56d0 con 0x563bf2f70440 2024-04-21T23:16:05.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.412+0000 7f630d7cb740 1 -- 172.21.15.44:0/2679349725 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f63041b4930 con 0x563bf2f6fb20 2024-04-21T23:16:05.412 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.412+0000 7f630d7cb740 10 client.0 _ll_register_callbacks cb 0x7f62f40098d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:05.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.413+0000 7f63088ab640 1 -- 172.21.15.44:0/2679349725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f6fb20 msgr2=0x563bf2f81720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:05.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.413+0000 7f63088ab640 1 --2- 172.21.15.44:0/2679349725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f6fb20 0x563bf2f81720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.413+0000 7f63088ab640 1 -- 172.21.15.44:0/2679349725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563bf2f79210 msgr2=0x563bf2f73ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:05.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.413+0000 7f63088ab640 1 --2- 172.21.15.44:0/2679349725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563bf2f79210 0x563bf2f73ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.413 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.413+0000 7f63088ab640 1 -- 172.21.15.44:0/2679349725 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bf2f72500 con 0x563bf2f70440 2024-04-21T23:16:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.416+0000 7f63088ab640 1 --2- 172.21.15.44:0/2679349725 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f70440 0x563bf2f735f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f62fc039040 tx=0x7f62fc003970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63705c5a3dbe6e42 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.416+0000 7f62e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:16:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.417+0000 7f62e3fff640 1 -- 172.21.15.44:0/2679349725 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc03b030 con 0x563bf2f70440 2024-04-21T23:16:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.417+0000 7f62e3fff640 1 -- 172.21.15.44:0/2679349725 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9621+0+0 (secure 0 0 0) 0x7f62fc034400 con 0x563bf2f70440 2024-04-21T23:16:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.417+0000 7f630d7cb740 1 -- 172.21.15.44:0/2679349725 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62fc04d030 con 0x563bf2f70440 2024-04-21T23:16:05.417 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.417+0000 7f62e3fff640 1 -- 172.21.15.44:0/2679349725 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f62fc044040 con 0x563bf2f70440 2024-04-21T23:16:05.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.423+0000 7f630d7cb740 10 client.26864 fetch_fsmap learned FSMap version 29 2024-04-21T23:16:05.423 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.424+0000 7f630d7cb740 1 -- 172.21.15.44:0/2679349725 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563bf2fef070 con 0x563bf2f70440 2024-04-21T23:16:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.426+0000 7f62e3fff640 1 -- 172.21.15.44:0/2679349725 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62fc034600 con 0x563bf2f70440 2024-04-21T23:16:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.426+0000 7f62e3fff640 1 -- 172.21.15.44:0/2679349725 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7f62fc0547d0 con 0x563bf2f70440 2024-04-21T23:16:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.426+0000 7f630d7cb740 10 client.26864 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:16:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.426+0000 7f630d7cb740 10 client.26864 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.426+0000 7f630d7cb740 20 client.26864 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.426+0000 7f630d7cb740 1 --2- 172.21.15.44:0/2679349725 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563bf2fef240 0x563bf2ff1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.426+0000 7f630d7cb740 4 client.26864 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:16:05.426 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.426+0000 7f630d7cb740 1 -- 172.21.15.44:0/2679349725 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f62fc04d030 con 0x563bf2fef240 2024-04-21T23:16:05.427 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.427+0000 7f6303fff640 1 --2- 172.21.15.44:0/2679349725 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563bf2fef240 0x563bf2ff1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:05.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.429+0000 7f6303fff640 1 --2- 172.21.15.44:0/2679349725 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563bf2fef240 0x563bf2ff1630 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4dc61adcdb043142 server_cookie=14c8c82dc079be45 in_seq=0 out_seq=0 2024-04-21T23:16:05.429 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.429+0000 7f62e3fff640 10 client.26864 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:05.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 838707 12.14 MB/sec execute 235 sec latency 131.808 ms 2024-04-21T23:16:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.863+0000 7f62e3fff640 1 -- 172.21.15.44:0/2679349725 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f62fc04d030 con 0x563bf2fef240 2024-04-21T23:16:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.863+0000 7f62e3fff640 10 client.26864 handle_command_reply: tid=0 2024-04-21T23:16:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.863+0000 7f630d7cb740 1 client.26864 shutdown 2024-04-21T23:16:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.864+0000 7f630d7cb740 1 -- 172.21.15.44:0/2679349725 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f70440 msgr2=0x563bf2f735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.864+0000 7f630d7cb740 1 --2- 172.21.15.44:0/2679349725 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f70440 0x563bf2f735f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f62fc039040 tx=0x7f62fc003970 comp rx=0 tx=0).stop 2024-04-21T23:16:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.864+0000 7f630d7cb740 1 -- 172.21.15.44:0/2679349725 shutdown_connections 2024-04-21T23:16:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.864+0000 7f630d7cb740 1 --2- 172.21.15.44:0/2679349725 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563bf2fef240 0x563bf2ff1630 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.864+0000 7f62e3fff640 0 client.26864 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.864+0000 7f630d7cb740 1 --2- 172.21.15.44:0/2679349725 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x563bf2f79210 0x563bf2f73ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.864+0000 7f630d7cb740 1 --2- 172.21.15.44:0/2679349725 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x563bf2f70440 0x563bf2f735f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.864+0000 7f630d7cb740 1 --2- 172.21.15.44:0/2679349725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bf2f6fb20 0x563bf2f81720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.864+0000 7f630d7cb740 1 -- 172.21.15.44:0/2679349725 >> 172.21.15.44:0/2679349725 conn(0x563bf2f6a440 msgr2=0x563bf2f6b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.864+0000 7f630d7cb740 1 -- 172.21.15.44:0/2679349725 shutdown_connections 2024-04-21T23:16:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.864+0000 7f630d7cb740 1 -- 172.21.15.44:0/2679349725 wait complete. 2024-04-21T23:16:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.864+0000 7f630d7cb740 20 client.26864 trim_cache size 0 max 16384 2024-04-21T23:16:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:16:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:16:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:16:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:16:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.865+0000 7f630ab0f640 1 -- 172.21.15.44:0/3605399348 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f62ec0a2e80 msgr2=0x7f62ec0a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.865+0000 7f630ab0f640 1 --2- 172.21.15.44:0/3605399348 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f62ec0a2e80 0x7f62ec0a5340 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f62f8005d80 tx=0x7f62f8005cd0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.865+0000 7f630ab0f640 1 -- 172.21.15.44:0/3605399348 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6304106630 msgr2=0x7f630410ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.865+0000 7f630ab0f640 1 --2- 172.21.15.44:0/3605399348 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6304106630 0x7f630410ca70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f62f800d6b0 tx=0x7f62f800db70 comp rx=0 tx=0).stop 2024-04-21T23:16:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.865+0000 7f630ab0f640 1 -- 172.21.15.44:0/3605399348 shutdown_connections 2024-04-21T23:16:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.865+0000 7f630ab0f640 1 --2- 172.21.15.44:0/3605399348 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f62ec0a2e80 0x7f62ec0a5340 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.865+0000 7f630ab0f640 1 --2- 172.21.15.44:0/3605399348 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6304106630 0x7f630410ca70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.865+0000 7f630ab0f640 1 --2- 172.21.15.44:0/3605399348 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6304100fe0 0x7f630410c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.865+0000 7f630ab0f640 1 --2- 172.21.15.44:0/3605399348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6304100620 0x7f630410bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.865+0000 7f630ab0f640 1 -- 172.21.15.44:0/3605399348 >> 172.21.15.44:0/3605399348 conn(0x7f6304077760 msgr2=0x7f6304104130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.866+0000 7f630ab0f640 1 -- 172.21.15.44:0/3605399348 shutdown_connections 2024-04-21T23:16:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:05.866+0000 7f630ab0f640 1 -- 172.21.15.44:0/3605399348 wait complete. 2024-04-21T23:16:06.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:05 smithi044 ceph-mon[27934]: pgmap v227: 129 pgs: 129 active+clean; 76 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.7 MiB/s wr, 469 op/s 2024-04-21T23:16:06.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:05 smithi055 ceph-mon[31117]: pgmap v227: 129 pgs: 129 active+clean; 76 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.7 MiB/s wr, 469 op/s 2024-04-21T23:16:06.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:05 smithi069 ceph-mon[31816]: pgmap v227: 129 pgs: 129 active+clean; 76 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.7 MiB/s wr, 469 op/s 2024-04-21T23:16:06.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 839395 12.10 MB/sec execute 236 sec latency 201.674 ms 2024-04-21T23:16:07.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 840807 12.08 MB/sec execute 237 sec latency 38.727 ms 2024-04-21T23:16:08.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:07 smithi044 ceph-mon[27934]: pgmap v228: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 62 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 385 op/s 2024-04-21T23:16:08.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:07 smithi069 ceph-mon[31816]: pgmap v228: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 62 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 385 op/s 2024-04-21T23:16:08.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:07 smithi055 ceph-mon[31117]: pgmap v228: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 62 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 385 op/s 2024-04-21T23:16:08.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 841883 12.06 MB/sec execute 238 sec latency 51.407 ms 2024-04-21T23:16:09.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 843227 12.03 MB/sec execute 239 sec latency 26.644 ms 2024-04-21T23:16:10.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:09 smithi044 ceph-mon[27934]: pgmap v229: 129 pgs: 12 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 86 active+clean; 64 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 341 op/s 2024-04-21T23:16:10.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:09 smithi069 ceph-mon[31816]: pgmap v229: 129 pgs: 12 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 86 active+clean; 64 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 341 op/s 2024-04-21T23:16:10.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:09 smithi055 ceph-mon[31117]: pgmap v229: 129 pgs: 12 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 86 active+clean; 64 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 341 op/s 2024-04-21T23:16:10.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 845181 12.04 MB/sec execute 240 sec latency 90.698 ms 2024-04-21T23:16:11.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 845709 11.99 MB/sec execute 241 sec latency 210.739 ms 2024-04-21T23:16:12.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:11 smithi044 ceph-mon[27934]: pgmap v230: 129 pgs: 12 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 86 active+clean; 64 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 341 op/s 2024-04-21T23:16:12.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:11 smithi069 ceph-mon[31816]: pgmap v230: 129 pgs: 12 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 86 active+clean; 64 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 341 op/s 2024-04-21T23:16:12.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:11 smithi055 ceph-mon[31117]: pgmap v230: 129 pgs: 12 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 86 active+clean; 64 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.5 MiB/s wr, 341 op/s 2024-04-21T23:16:12.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 846658 11.96 MB/sec execute 242 sec latency 64.638 ms 2024-04-21T23:16:13.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 849001 11.97 MB/sec execute 243 sec latency 21.751 ms 2024-04-21T23:16:14.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:14 smithi044 ceph-mon[27934]: pgmap v231: 129 pgs: 129 active+clean; 111 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.0 MiB/s wr, 330 op/s 2024-04-21T23:16:14.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:14 smithi044 ceph-mon[27934]: osdmap e103: 12 total, 12 up, 12 in 2024-04-21T23:16:14.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:14 smithi069 ceph-mon[31816]: pgmap v231: 129 pgs: 129 active+clean; 111 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.0 MiB/s wr, 330 op/s 2024-04-21T23:16:14.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:14 smithi069 ceph-mon[31816]: osdmap e103: 12 total, 12 up, 12 in 2024-04-21T23:16:14.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:14 smithi055 ceph-mon[31117]: pgmap v231: 129 pgs: 129 active+clean; 111 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.0 MiB/s wr, 330 op/s 2024-04-21T23:16:14.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:14 smithi055 ceph-mon[31117]: osdmap e103: 12 total, 12 up, 12 in 2024-04-21T23:16:14.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 850910 11.95 MB/sec execute 244 sec latency 42.067 ms 2024-04-21T23:16:15.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 852994 11.95 MB/sec execute 245 sec latency 31.296 ms 2024-04-21T23:16:16.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:16 smithi044 ceph-mon[27934]: pgmap v233: 129 pgs: 129 active+clean; 111 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.0 MiB/s wr, 330 op/s 2024-04-21T23:16:16.467 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:16 smithi069 ceph-mon[31816]: pgmap v233: 129 pgs: 129 active+clean; 111 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.0 MiB/s wr, 330 op/s 2024-04-21T23:16:16.467 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:16 smithi055 ceph-mon[31117]: pgmap v233: 129 pgs: 129 active+clean; 111 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.0 MiB/s wr, 330 op/s 2024-04-21T23:16:16.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 855733 11.97 MB/sec execute 246 sec latency 32.070 ms 2024-04-21T23:16:17.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 857429 11.95 MB/sec execute 247 sec latency 15.984 ms 2024-04-21T23:16:18.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:18 smithi044 ceph-mon[27934]: pgmap v234: 129 pgs: 129 active+clean; 91 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.8 MiB/s wr, 319 op/s 2024-04-21T23:16:18.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:18 smithi069 ceph-mon[31816]: pgmap v234: 129 pgs: 129 active+clean; 91 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.8 MiB/s wr, 319 op/s 2024-04-21T23:16:18.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:18 smithi055 ceph-mon[31117]: pgmap v234: 129 pgs: 129 active+clean; 91 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.8 MiB/s wr, 319 op/s 2024-04-21T23:16:18.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 859924 11.96 MB/sec execute 248 sec latency 36.739 ms 2024-04-21T23:16:18.965 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.965+0000 7fa288b9f640 1 -- 172.21.15.44:0/1971344004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa284102740 msgr2=0x7fa284102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:18.965 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.965+0000 7fa288b9f640 1 --2- 172.21.15.44:0/1971344004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa284102740 0x7fa284102b20 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa274009900 tx=0x7fa27402bcd0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.965 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.965+0000 7fa288b9f640 1 -- 172.21.15.44:0/1971344004 shutdown_connections 2024-04-21T23:16:18.965 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.965+0000 7fa288b9f640 1 --2- 172.21.15.44:0/1971344004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa28410bb60 0x7fa28410e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.965 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.965+0000 7fa288b9f640 1 --2- 172.21.15.44:0/1971344004 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa284103060 0x7fa28410b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.965 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.965+0000 7fa288b9f640 1 --2- 172.21.15.44:0/1971344004 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa284102740 0x7fa284102b20 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.965 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.965+0000 7fa288b9f640 1 -- 172.21.15.44:0/1971344004 >> 172.21.15.44:0/1971344004 conn(0x7fa2840fbf60 msgr2=0x7fa2840fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.965+0000 7fa288b9f640 1 -- 172.21.15.44:0/1971344004 shutdown_connections 2024-04-21T23:16:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.965+0000 7fa288b9f640 1 -- 172.21.15.44:0/1971344004 wait complete. 2024-04-21T23:16:18.966 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.966+0000 7fa288b9f640 1 Processor -- start 2024-04-21T23:16:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.966+0000 7fa288b9f640 1 -- start start 2024-04-21T23:16:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.966+0000 7fa288b9f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa284102740 0x7fa284100390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.966+0000 7fa288b9f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa284103060 0x7fa2841008d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.966+0000 7fa288b9f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa28410bb60 0x7fa284100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.966+0000 7fa288b9f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa284111990 con 0x7fa284103060 2024-04-21T23:16:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.966+0000 7fa288b9f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa284111810 con 0x7fa284102740 2024-04-21T23:16:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.966+0000 7fa288b9f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa284111b10 con 0x7fa28410bb60 2024-04-21T23:16:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.966+0000 7fa282ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa284103060 0x7fa2841008d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.966+0000 7fa283fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa28410bb60 0x7fa284100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.966+0000 7fa282ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa284103060 0x7fa2841008d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44820/0 (socket says 172.21.15.44:44820) 2024-04-21T23:16:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.966+0000 7fa282ffd640 1 -- 172.21.15.44:0/22397442 learned_addr learned my addr 172.21.15.44:0/22397442 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.966+0000 7fa2837fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa284102740 0x7fa284100390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.967+0000 7fa282ffd640 1 -- 172.21.15.44:0/22397442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa28410bb60 msgr2=0x7fa284100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.967+0000 7fa282ffd640 1 --2- 172.21.15.44:0/22397442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa28410bb60 0x7fa284100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.967+0000 7fa282ffd640 1 -- 172.21.15.44:0/22397442 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa284102740 msgr2=0x7fa284100390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.967+0000 7fa282ffd640 1 --2- 172.21.15.44:0/22397442 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa284102740 0x7fa284100390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.967+0000 7fa282ffd640 1 -- 172.21.15.44:0/22397442 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2841a6310 con 0x7fa284103060 2024-04-21T23:16:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.967+0000 7fa283fff640 1 --2- 172.21.15.44:0/22397442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa28410bb60 0x7fa284100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.967+0000 7fa282ffd640 1 --2- 172.21.15.44:0/22397442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa284103060 0x7fa2841008d0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fa27800d950 tx=0x7fa27800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.967+0000 7fa280ff9640 1 -- 172.21.15.44:0/22397442 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa278017030 con 0x7fa284103060 2024-04-21T23:16:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.967+0000 7fa288b9f640 1 -- 172.21.15.44:0/22397442 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2841a6600 con 0x7fa284103060 2024-04-21T23:16:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.967+0000 7fa280ff9640 1 -- 172.21.15.44:0/22397442 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa278004020 con 0x7fa284103060 2024-04-21T23:16:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.967+0000 7fa280ff9640 1 -- 172.21.15.44:0/22397442 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2780047a0 con 0x7fa284103060 2024-04-21T23:16:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.967+0000 7fa288b9f640 1 -- 172.21.15.44:0/22397442 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2841a6b50 con 0x7fa284103060 2024-04-21T23:16:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.969+0000 7fa280ff9640 1 -- 172.21.15.44:0/22397442 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa2780041d0 con 0x7fa284103060 2024-04-21T23:16:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.970+0000 7fa28c9ed740 1 Processor -- start 2024-04-21T23:16:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.970+0000 7fa280ff9640 1 --2- 172.21.15.44:0/22397442 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fa26c0a2ce0 0x7fa26c0a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.970+0000 7fa28c9ed740 1 -- start start 2024-04-21T23:16:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.970+0000 7fa28c9ed740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeda766d40 0x55aeda767120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.970+0000 7fa28c9ed740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda767660 0x55aeda76fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.970+0000 7fa283fff640 1 --2- 172.21.15.44:0/22397442 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fa26c0a2ce0 0x7fa26c0a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.970+0000 7fa28c9ed740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda770430 0x55aeda772820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.970+0000 7fa28c9ed740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55aeda6cab70 con 0x55aeda770430 2024-04-21T23:16:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.970+0000 7fa28c9ed740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55aeda673470 con 0x55aeda767660 2024-04-21T23:16:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.970+0000 7fa28c9ed740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55aeda6c6b30 con 0x55aeda766d40 2024-04-21T23:16:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.970+0000 7fa280ff9640 1 -- 172.21.15.44:0/22397442 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9533+0+0 (secure 0 0 0) 0x7fa2780cf110 con 0x7fa284103060 2024-04-21T23:16:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.971+0000 7fa283fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda770430 0x55aeda772820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.971+0000 7fa283fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda770430 0x55aeda772820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44828/0 (socket says 172.21.15.44:44828) 2024-04-21T23:16:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.971+0000 7fa283fff640 1 -- 172.21.15.44:0/2224771167 learned_addr learned my addr 172.21.15.44:0/2224771167 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.971+0000 7fa282ffd640 1 --2- 172.21.15.44:0/2224771167 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda767660 0x55aeda76fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.971+0000 7fa2837fe640 1 --2- 172.21.15.44:0/2224771167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeda766d40 0x55aeda767120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.971+0000 7fa282ffd640 1 -- 172.21.15.44:0/2224771167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeda766d40 msgr2=0x55aeda767120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.971+0000 7fa282ffd640 1 --2- 172.21.15.44:0/2224771167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeda766d40 0x55aeda767120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.971+0000 7fa282ffd640 1 -- 172.21.15.44:0/2224771167 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda770430 msgr2=0x55aeda772820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.971+0000 7fa282ffd640 1 --2- 172.21.15.44:0/2224771167 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda770430 0x55aeda772820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.971+0000 7fa282ffd640 1 -- 172.21.15.44:0/2224771167 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aeda3df210 con 0x55aeda767660 2024-04-21T23:16:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.971+0000 7fa283fff640 1 --2- 172.21.15.44:0/22397442 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fa26c0a2ce0 0x7fa26c0a51a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa27c009d20 tx=0x7fa27c012650 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.971+0000 7fa282ffd640 1 --2- 172.21.15.44:0/2224771167 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda767660 0x55aeda76fef0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa2780cc470 tx=0x7fa27809ae80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc28dc0c8c6d736e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.971+0000 7fa269ffb640 1 -- 172.21.15.44:0/2224771167 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27809f5f0 con 0x55aeda767660 2024-04-21T23:16:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.971+0000 7fa269ffb640 1 -- 172.21.15.44:0/2224771167 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2780a3eb0 con 0x55aeda767660 2024-04-21T23:16:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.971+0000 7fa269ffb640 1 -- 172.21.15.44:0/2224771167 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2780ccbc0 con 0x55aeda767660 2024-04-21T23:16:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.972+0000 7fa28c9ed740 1 -- 172.21.15.44:0/2224771167 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda767660 msgr2=0x55aeda76fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.972+0000 7fa28c9ed740 1 --2- 172.21.15.44:0/2224771167 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda767660 0x55aeda76fef0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa2780cc470 tx=0x7fa27809ae80 comp rx=0 tx=0).stop 2024-04-21T23:16:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.972+0000 7fa28c9ed740 1 -- 172.21.15.44:0/2224771167 shutdown_connections 2024-04-21T23:16:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.972+0000 7fa28c9ed740 1 --2- 172.21.15.44:0/2224771167 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda770430 0x55aeda772820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.972+0000 7fa28c9ed740 1 --2- 172.21.15.44:0/2224771167 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda767660 0x55aeda76fef0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.972+0000 7fa28c9ed740 1 --2- 172.21.15.44:0/2224771167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeda766d40 0x55aeda767120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.972+0000 7fa28c9ed740 1 -- 172.21.15.44:0/2224771167 >> 172.21.15.44:0/2224771167 conn(0x55aeda761660 msgr2=0x55aeda761a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.972+0000 7fa28c9ed740 1 -- 172.21.15.44:0/2224771167 shutdown_connections 2024-04-21T23:16:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.972+0000 7fa28c9ed740 1 -- 172.21.15.44:0/2224771167 wait complete. 2024-04-21T23:16:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.972+0000 7fa28c9ed740 1 -- start start 2024-04-21T23:16:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.973+0000 7fa28c9ed740 1 Processor -- start 2024-04-21T23:16:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.973+0000 7fa28c9ed740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeda766d40 0x55aeda76d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.973+0000 7fa28c9ed740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda767660 0x55aeda76da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.973+0000 7fa28c9ed740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda770430 0x55aeda76dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.973+0000 7fa28c9ed740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55aeda773000 con 0x55aeda767660 2024-04-21T23:16:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.973+0000 7fa28c9ed740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55aeda686a10 con 0x55aeda770430 2024-04-21T23:16:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.973+0000 7fa28c9ed740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55aeda773180 con 0x55aeda766d40 2024-04-21T23:16:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.973+0000 7fa28c9ed740 10 client.0 _ll_register_callbacks cb 0x55aeda6c1b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.973+0000 7fa282ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda767660 0x55aeda76da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.973+0000 7fa283fff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda770430 0x55aeda76dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.973+0000 7fa282ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda767660 0x55aeda76da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44844/0 (socket says 172.21.15.44:44844) 2024-04-21T23:16:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.973+0000 7fa282ffd640 1 -- 172.21.15.44:0/4021765651 learned_addr learned my addr 172.21.15.44:0/4021765651 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.973+0000 7fa2837fe640 1 --2- 172.21.15.44:0/4021765651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeda766d40 0x55aeda76d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.973+0000 7fa283fff640 1 -- 172.21.15.44:0/4021765651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeda766d40 msgr2=0x55aeda76d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.973+0000 7fa283fff640 1 --2- 172.21.15.44:0/4021765651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeda766d40 0x55aeda76d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.973+0000 7fa283fff640 1 -- 172.21.15.44:0/4021765651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda767660 msgr2=0x55aeda76da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.973+0000 7fa283fff640 1 --2- 172.21.15.44:0/4021765651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda767660 0x55aeda76da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.973+0000 7fa283fff640 1 -- 172.21.15.44:0/4021765651 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aeda4ab620 con 0x55aeda770430 2024-04-21T23:16:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.973+0000 7fa282ffd640 1 --2- 172.21.15.44:0/4021765651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda767660 0x55aeda76da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.974+0000 7fa283fff640 1 --2- 172.21.15.44:0/4021765651 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda770430 0x55aeda76dfb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa27c0092f0 tx=0x7fa27c01c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99d7a3e9d13888fd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.974+0000 7fa262ffd640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:16:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.974+0000 7fa262ffd640 1 -- 172.21.15.44:0/4021765651 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27c0207a0 con 0x55aeda770430 2024-04-21T23:16:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.974+0000 7fa28c9ed740 1 -- 172.21.15.44:0/4021765651 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2780a3eb0 con 0x55aeda770430 2024-04-21T23:16:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.974+0000 7fa262ffd640 1 -- 172.21.15.44:0/4021765651 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9533+0+0 (secure 0 0 0) 0x7fa27c020940 con 0x55aeda770430 2024-04-21T23:16:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.974+0000 7fa262ffd640 1 -- 172.21.15.44:0/4021765651 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa27c020b40 con 0x55aeda770430 2024-04-21T23:16:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.974+0000 7fa28c9ed740 10 client.26876 fetch_fsmap learned FSMap version 29 2024-04-21T23:16:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.974+0000 7fa28c9ed740 1 -- 172.21.15.44:0/4021765651 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55aeda77f340 con 0x55aeda770430 2024-04-21T23:16:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.974+0000 7fa262ffd640 1 -- 172.21.15.44:0/4021765651 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2780a3eb0 con 0x55aeda770430 2024-04-21T23:16:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.974+0000 7fa262ffd640 1 -- 172.21.15.44:0/4021765651 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7fa27c01f050 con 0x55aeda770430 2024-04-21T23:16:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.974+0000 7fa28c9ed740 10 client.26876 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:16:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.974+0000 7fa28c9ed740 10 client.26876 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.974+0000 7fa28c9ed740 20 client.26876 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.974+0000 7fa28c9ed740 1 --2- 172.21.15.44:0/4021765651 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55aeda780850 0x55aeda782c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.974+0000 7fa28c9ed740 4 client.26876 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:16:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.974+0000 7fa28c9ed740 1 -- 172.21.15.44:0/4021765651 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aeda6d80b0 con 0x55aeda780850 2024-04-21T23:16:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.975+0000 7fa2837fe640 1 --2- 172.21.15.44:0/4021765651 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55aeda780850 0x55aeda782c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.975+0000 7fa2837fe640 1 --2- 172.21.15.44:0/4021765651 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55aeda780850 0x55aeda782c40 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6dd7df15379035b server_cookie=5da2d6a9e4ec71c9 in_seq=0 out_seq=0 2024-04-21T23:16:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.975+0000 7fa262ffd640 10 client.26876 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.976+0000 7fa262ffd640 1 -- 172.21.15.44:0/4021765651 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aeda6d80b0 con 0x55aeda780850 2024-04-21T23:16:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.976+0000 7fa262ffd640 10 client.26876 handle_command_reply: tid=0 2024-04-21T23:16:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.976+0000 7fa28c9ed740 1 client.26876 shutdown 2024-04-21T23:16:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.976+0000 7fa28c9ed740 1 -- 172.21.15.44:0/4021765651 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda770430 msgr2=0x55aeda76dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.976+0000 7fa28c9ed740 1 --2- 172.21.15.44:0/4021765651 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda770430 0x55aeda76dfb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa27c0092f0 tx=0x7fa27c01c000 comp rx=0 tx=0).stop 2024-04-21T23:16:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.976+0000 7fa28c9ed740 1 -- 172.21.15.44:0/4021765651 shutdown_connections 2024-04-21T23:16:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.976+0000 7fa28c9ed740 1 --2- 172.21.15.44:0/4021765651 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55aeda780850 0x55aeda782c40 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.976+0000 7fa28c9ed740 1 --2- 172.21.15.44:0/4021765651 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda770430 0x55aeda76dfb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.976+0000 7fa262ffd640 0 client.26876 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.976+0000 7fa28c9ed740 1 --2- 172.21.15.44:0/4021765651 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda767660 0x55aeda76da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.976+0000 7fa28c9ed740 1 --2- 172.21.15.44:0/4021765651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeda766d40 0x55aeda76d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.976+0000 7fa28c9ed740 1 -- 172.21.15.44:0/4021765651 >> 172.21.15.44:0/4021765651 conn(0x55aeda761660 msgr2=0x55aeda7633b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.976+0000 7fa28c9ed740 1 -- 172.21.15.44:0/4021765651 shutdown_connections 2024-04-21T23:16:18.982 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.976+0000 7fa28c9ed740 1 -- 172.21.15.44:0/4021765651 wait complete. 2024-04-21T23:16:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.977+0000 7fa28c9ed740 20 client.26876 trim_cache size 0 max 16384 2024-04-21T23:16:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.992+0000 7fa28c9ed740 1 Processor -- start 2024-04-21T23:16:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.992+0000 7fa28c9ed740 1 -- start start 2024-04-21T23:16:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.992+0000 7fa28c9ed740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda766d40 0x55aeda76fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.992+0000 7fa28c9ed740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeda767660 0x55aeda75c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.992+0000 7fa28c9ed740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda770430 0x55aeda77bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.992+0000 7fa28c9ed740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa2841aed90 con 0x55aeda770430 2024-04-21T23:16:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.992+0000 7fa28c9ed740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa2841b15f0 con 0x55aeda766d40 2024-04-21T23:16:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.992+0000 7fa28c9ed740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa2841af3e0 con 0x55aeda767660 2024-04-21T23:16:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.992+0000 7fa283fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda770430 0x55aeda77bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.992+0000 7fa283fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda770430 0x55aeda77bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44860/0 (socket says 172.21.15.44:44860) 2024-04-21T23:16:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.992+0000 7fa283fff640 1 -- 172.21.15.44:0/917163210 learned_addr learned my addr 172.21.15.44:0/917163210 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.992+0000 7fa2837fe640 1 --2- 172.21.15.44:0/917163210 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda766d40 0x55aeda76fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.992+0000 7fa282ffd640 1 --2- 172.21.15.44:0/917163210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeda767660 0x55aeda75c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.992+0000 7fa283fff640 1 -- 172.21.15.44:0/917163210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeda767660 msgr2=0x55aeda75c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.992+0000 7fa283fff640 1 --2- 172.21.15.44:0/917163210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeda767660 0x55aeda75c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.992+0000 7fa283fff640 1 -- 172.21.15.44:0/917163210 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda766d40 msgr2=0x55aeda76fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.992+0000 7fa283fff640 1 --2- 172.21.15.44:0/917163210 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda766d40 0x55aeda76fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.992+0000 7fa283fff640 1 -- 172.21.15.44:0/917163210 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aeda76c4e0 con 0x55aeda770430 2024-04-21T23:16:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.992+0000 7fa283fff640 1 --2- 172.21.15.44:0/917163210 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda770430 0x55aeda77bec0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fa27c010d00 tx=0x7fa27c012550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4550b975583dc884 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.993+0000 7fa2697fa640 1 -- 172.21.15.44:0/917163210 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27c012d90 con 0x55aeda770430 2024-04-21T23:16:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.993+0000 7fa2697fa640 1 -- 172.21.15.44:0/917163210 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa27c03e070 con 0x55aeda770430 2024-04-21T23:16:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.993+0000 7fa2697fa640 1 -- 172.21.15.44:0/917163210 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27c02a650 con 0x55aeda770430 2024-04-21T23:16:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.993+0000 7fa28c9ed740 1 -- 172.21.15.44:0/917163210 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda770430 msgr2=0x55aeda77bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.993+0000 7fa28c9ed740 1 --2- 172.21.15.44:0/917163210 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda770430 0x55aeda77bec0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fa27c010d00 tx=0x7fa27c012550 comp rx=0 tx=0).stop 2024-04-21T23:16:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.993+0000 7fa28c9ed740 1 -- 172.21.15.44:0/917163210 shutdown_connections 2024-04-21T23:16:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.993+0000 7fa28c9ed740 1 --2- 172.21.15.44:0/917163210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeda767660 0x55aeda75c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.993+0000 7fa28c9ed740 1 --2- 172.21.15.44:0/917163210 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda770430 0x55aeda77bec0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.993+0000 7fa28c9ed740 1 --2- 172.21.15.44:0/917163210 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda766d40 0x55aeda76fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.993+0000 7fa28c9ed740 1 -- 172.21.15.44:0/917163210 >> 172.21.15.44:0/917163210 conn(0x55aeda761660 msgr2=0x55aeda760600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.993+0000 7fa28c9ed740 1 -- 172.21.15.44:0/917163210 shutdown_connections 2024-04-21T23:16:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.993+0000 7fa28c9ed740 1 -- 172.21.15.44:0/917163210 wait complete. 2024-04-21T23:16:18.995 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.994+0000 7fa28c9ed740 1 -- start start 2024-04-21T23:16:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.994+0000 7fa28c9ed740 1 Processor -- start 2024-04-21T23:16:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.994+0000 7fa28c9ed740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda766d40 0x55aeda774500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.994+0000 7fa28c9ed740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda767660 0x55aeda774a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.994+0000 7fa28c9ed740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeda770430 0x55aeda7e5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.994+0000 7fa28c9ed740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa2841afab0 con 0x55aeda766d40 2024-04-21T23:16:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.994+0000 7fa28c9ed740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa2841b0f20 con 0x55aeda767660 2024-04-21T23:16:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.994+0000 7fa28c9ed740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa2841b0180 con 0x55aeda770430 2024-04-21T23:16:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.994+0000 7fa28c9ed740 10 client.0 _ll_register_callbacks cb 0x7fa27c0128a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.994+0000 7fa2837fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda766d40 0x55aeda774500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.994+0000 7fa282ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda767660 0x55aeda774a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.994+0000 7fa2837fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda766d40 0x55aeda774500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:44870/0 (socket says 172.21.15.44:44870) 2024-04-21T23:16:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.994+0000 7fa2837fe640 1 -- 172.21.15.44:0/89927165 learned_addr learned my addr 172.21.15.44:0/89927165 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.994+0000 7fa283fff640 1 --2- 172.21.15.44:0/89927165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeda770430 0x55aeda7e5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.995+0000 7fa2837fe640 1 -- 172.21.15.44:0/89927165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeda770430 msgr2=0x55aeda7e5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.995+0000 7fa2837fe640 1 --2- 172.21.15.44:0/89927165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeda770430 0x55aeda7e5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.995+0000 7fa2837fe640 1 -- 172.21.15.44:0/89927165 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda767660 msgr2=0x55aeda774a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.995+0000 7fa2837fe640 1 --2- 172.21.15.44:0/89927165 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda767660 0x55aeda774a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.995+0000 7fa2837fe640 1 -- 172.21.15.44:0/89927165 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aeda772b90 con 0x55aeda766d40 2024-04-21T23:16:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.995+0000 7fa282ffd640 1 --2- 172.21.15.44:0/89927165 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda767660 0x55aeda774a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.995+0000 7fa2837fe640 1 --2- 172.21.15.44:0/89927165 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda766d40 0x55aeda774500 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fa274009230 tx=0x7fa274005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7fda851dc781c28 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.995+0000 7fa262ffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:16:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.995+0000 7fa262ffd640 1 -- 172.21.15.44:0/89927165 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27404d070 con 0x55aeda766d40 2024-04-21T23:16:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.995+0000 7fa262ffd640 1 -- 172.21.15.44:0/89927165 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9533+0+0 (secure 0 0 0) 0x7fa274049dd0 con 0x55aeda766d40 2024-04-21T23:16:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.995+0000 7fa28c9ed740 1 -- 172.21.15.44:0/89927165 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa27c03e070 con 0x55aeda766d40 2024-04-21T23:16:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.995+0000 7fa262ffd640 1 -- 172.21.15.44:0/89927165 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa274059050 con 0x55aeda766d40 2024-04-21T23:16:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.995+0000 7fa28c9ed740 10 client.17316 fetch_fsmap learned FSMap version 29 2024-04-21T23:16:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.995+0000 7fa28c9ed740 1 -- 172.21.15.44:0/89927165 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55aeda7e7430 con 0x55aeda766d40 2024-04-21T23:16:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.995+0000 7fa262ffd640 1 -- 172.21.15.44:0/89927165 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa27c03e070 con 0x55aeda766d40 2024-04-21T23:16:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.996+0000 7fa262ffd640 1 -- 172.21.15.44:0/89927165 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 29) ==== 8330+0+0 (secure 0 0 0) 0x7fa2740491a0 con 0x55aeda766d40 2024-04-21T23:16:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.996+0000 7fa28c9ed740 10 client.17316 fetch_fsmap finished waiting for FSMap version 29 2024-04-21T23:16:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.996+0000 7fa28c9ed740 10 client.17316 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.996+0000 7fa28c9ed740 20 client.17316 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.996+0000 7fa28c9ed740 1 --2- 172.21.15.44:0/89927165 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55aeda7e7bd0 0x55aeda7e9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.996+0000 7fa28c9ed740 4 client.17316 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:16:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.996+0000 7fa28c9ed740 1 -- 172.21.15.44:0/89927165 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55aeda6d2400 con 0x55aeda7e7bd0 2024-04-21T23:16:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.996+0000 7fa282ffd640 1 --2- 172.21.15.44:0/89927165 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55aeda7e7bd0 0x55aeda7e9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.996+0000 7fa282ffd640 1 --2- 172.21.15.44:0/89927165 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55aeda7e7bd0 0x55aeda7e9ff0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=125c901eac8198cd server_cookie=2f8060dc6caed19 in_seq=0 out_seq=0 2024-04-21T23:16:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:18.996+0000 7fa262ffd640 10 client.17316 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:19.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.276+0000 7fa262ffd640 1 -- 172.21.15.44:0/89927165 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55aeda6d2400 con 0x55aeda7e7bd0 2024-04-21T23:16:19.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.276+0000 7fa262ffd640 10 client.17316 handle_command_reply: tid=0 2024-04-21T23:16:19.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.276+0000 7fa28c9ed740 1 client.17316 shutdown 2024-04-21T23:16:19.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.276+0000 7fa28c9ed740 1 -- 172.21.15.44:0/89927165 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda766d40 msgr2=0x55aeda774500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:19.276 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.276+0000 7fa28c9ed740 1 --2- 172.21.15.44:0/89927165 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda766d40 0x55aeda774500 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fa274009230 tx=0x7fa274005a20 comp rx=0 tx=0).stop 2024-04-21T23:16:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.276+0000 7fa28c9ed740 1 -- 172.21.15.44:0/89927165 shutdown_connections 2024-04-21T23:16:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.276+0000 7fa28c9ed740 1 --2- 172.21.15.44:0/89927165 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55aeda7e7bd0 0x55aeda7e9ff0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.276+0000 7fa262ffd640 0 client.17316 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.276+0000 7fa28c9ed740 1 --2- 172.21.15.44:0/89927165 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeda770430 0x55aeda7e5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.276+0000 7fa28c9ed740 1 --2- 172.21.15.44:0/89927165 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55aeda767660 0x55aeda774a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.276+0000 7fa28c9ed740 1 --2- 172.21.15.44:0/89927165 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55aeda766d40 0x55aeda774500 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.276+0000 7fa28c9ed740 1 -- 172.21.15.44:0/89927165 >> 172.21.15.44:0/89927165 conn(0x55aeda761660 msgr2=0x55aeda760600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.277+0000 7fa28c9ed740 1 -- 172.21.15.44:0/89927165 shutdown_connections 2024-04-21T23:16:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.277+0000 7fa28c9ed740 1 -- 172.21.15.44:0/89927165 wait complete. 2024-04-21T23:16:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.277+0000 7fa28c9ed740 20 client.17316 trim_cache size 0 max 16384 2024-04-21T23:16:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:16:19.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:16:19.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:16:19.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:16:19.278 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.277+0000 7fa288b9f640 1 -- 172.21.15.44:0/22397442 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fa26c0a2ce0 msgr2=0x7fa26c0a51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.277+0000 7fa288b9f640 1 --2- 172.21.15.44:0/22397442 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fa26c0a2ce0 0x7fa26c0a51a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa27c009d20 tx=0x7fa27c012650 comp rx=0 tx=0).stop 2024-04-21T23:16:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.278+0000 7fa288b9f640 1 -- 172.21.15.44:0/22397442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa284103060 msgr2=0x7fa2841008d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.278+0000 7fa288b9f640 1 --2- 172.21.15.44:0/22397442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa284103060 0x7fa2841008d0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fa27800d950 tx=0x7fa27800de10 comp rx=0 tx=0).stop 2024-04-21T23:16:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.278+0000 7fa288b9f640 1 -- 172.21.15.44:0/22397442 shutdown_connections 2024-04-21T23:16:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.278+0000 7fa288b9f640 1 --2- 172.21.15.44:0/22397442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa28410bb60 0x7fa284100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.278+0000 7fa288b9f640 1 --2- 172.21.15.44:0/22397442 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fa26c0a2ce0 0x7fa26c0a51a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.278+0000 7fa288b9f640 1 --2- 172.21.15.44:0/22397442 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa284103060 0x7fa2841008d0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.278+0000 7fa288b9f640 1 --2- 172.21.15.44:0/22397442 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa284102740 0x7fa284100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.278+0000 7fa288b9f640 1 -- 172.21.15.44:0/22397442 >> 172.21.15.44:0/22397442 conn(0x7fa2840fbf60 msgr2=0x7fa2840fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.278+0000 7fa288b9f640 1 -- 172.21.15.44:0/22397442 shutdown_connections 2024-04-21T23:16:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:19.278+0000 7fa288b9f640 1 -- 172.21.15.44:0/22397442 wait complete. 2024-04-21T23:16:19.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 861657 11.94 MB/sec execute 249 sec latency 117.426 ms 2024-04-21T23:16:19.714 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:16:19 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: ignoring --setuser ceph since I am not root 2024-04-21T23:16:19.714 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:16:19 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: ignoring --setgroup ceph since I am not root 2024-04-21T23:16:19.714 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:16:19 smithi069 ceph-mds[63204]: -- 172.21.15.69:0/1093327966 <== mon.1 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5648d09cda40 con 0x5648d09a1000 2024-04-21T23:16:19.714 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:16:19 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: starting mds.i at 2024-04-21T23:16:20.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:20 smithi055 ceph-mon[31117]: pgmap v235: 129 pgs: 129 active+clean; 72 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.1 MiB/s wr, 362 op/s 2024-04-21T23:16:20.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:20 smithi055 ceph-mon[31117]: daemon mds.i restarted 2024-04-21T23:16:20.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:20 smithi069 ceph-mon[31816]: pgmap v235: 129 pgs: 129 active+clean; 72 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.1 MiB/s wr, 362 op/s 2024-04-21T23:16:20.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:20 smithi069 ceph-mon[31816]: daemon mds.i restarted 2024-04-21T23:16:20.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:20 smithi044 ceph-mon[27934]: pgmap v235: 129 pgs: 129 active+clean; 72 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.1 MiB/s wr, 362 op/s 2024-04-21T23:16:20.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:20 smithi044 ceph-mon[27934]: daemon mds.i restarted 2024-04-21T23:16:20.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 863659 11.94 MB/sec execute 250 sec latency 19.461 ms 2024-04-21T23:16:21.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:21 smithi055 ceph-mon[31117]: osdmap e104: 12 total, 12 up, 12 in 2024-04-21T23:16:21.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:21 smithi055 ceph-mon[31117]: mds.? [v2:172.21.15.69:6838/3179646768,v1:172.21.15.69:6839/3179646768] up:boot 2024-04-21T23:16:21.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:21 smithi055 ceph-mon[31117]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:16:21.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:21 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:16:21.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:21 smithi055 ceph-mon[31117]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:16:21.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:21 smithi069 ceph-mon[31816]: osdmap e104: 12 total, 12 up, 12 in 2024-04-21T23:16:21.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:21 smithi069 ceph-mon[31816]: mds.? [v2:172.21.15.69:6838/3179646768,v1:172.21.15.69:6839/3179646768] up:boot 2024-04-21T23:16:21.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:21 smithi069 ceph-mon[31816]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:16:21.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:21 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:16:21.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:21 smithi069 ceph-mon[31816]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:16:21.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:21 smithi044 ceph-mon[27934]: osdmap e104: 12 total, 12 up, 12 in 2024-04-21T23:16:21.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:21 smithi044 ceph-mon[27934]: mds.? [v2:172.21.15.69:6838/3179646768,v1:172.21.15.69:6839/3179646768] up:boot 2024-04-21T23:16:21.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:21 smithi044 ceph-mon[27934]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T23:16:21.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:21 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:16:21.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:21 smithi044 ceph-mon[27934]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T23:16:21.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 866257 11.96 MB/sec execute 251 sec latency 36.260 ms 2024-04-21T23:16:22.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:22 smithi055 ceph-mon[31117]: pgmap v237: 129 pgs: 129 active+clean; 72 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 275 op/s 2024-04-21T23:16:22.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:22 smithi069 ceph-mon[31816]: pgmap v237: 129 pgs: 129 active+clean; 72 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 275 op/s 2024-04-21T23:16:22.684 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:22 smithi044 ceph-mon[27934]: pgmap v237: 129 pgs: 129 active+clean; 72 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 275 op/s 2024-04-21T23:16:22.710 INFO:tasks.workunit.client.0.smithi044.stdout: 1 867988 11.94 MB/sec execute 252 sec latency 24.495 ms 2024-04-21T23:16:23.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 870444 11.95 MB/sec execute 253 sec latency 28.437 ms 2024-04-21T23:16:24.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:24 smithi055 ceph-mon[31117]: pgmap v238: 129 pgs: 129 active+clean; 64 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 439 op/s 2024-04-21T23:16:24.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:24 smithi069 ceph-mon[31816]: pgmap v238: 129 pgs: 129 active+clean; 64 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 439 op/s 2024-04-21T23:16:24.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:24 smithi044 ceph-mon[27934]: pgmap v238: 129 pgs: 129 active+clean; 64 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.0 MiB/s wr, 439 op/s 2024-04-21T23:16:24.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 872816 11.95 MB/sec execute 254 sec latency 35.516 ms 2024-04-21T23:16:24.716 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:16:24.717 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:16:24.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.837+0000 7f34e2c7d640 1 -- 172.21.15.44:0/2233617342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34dc073d10 msgr2=0x7f34dc0740f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.837+0000 7f34e2c7d640 1 --2- 172.21.15.44:0/2233617342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34dc073d10 0x7f34dc0740f0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009900 tx=0x7f34d002bcd0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.837+0000 7f34e2c7d640 1 -- 172.21.15.44:0/2233617342 shutdown_connections 2024-04-21T23:16:24.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.837+0000 7f34e2c7d640 1 --2- 172.21.15.44:0/2233617342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34dc074630 0x7f34dc10da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.837+0000 7f34e2c7d640 1 --2- 172.21.15.44:0/2233617342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34dc10e0e0 0x7f34dc1104d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.837+0000 7f34e2c7d640 1 --2- 172.21.15.44:0/2233617342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34dc073d10 0x7f34dc0740f0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.837+0000 7f34e2c7d640 1 -- 172.21.15.44:0/2233617342 >> 172.21.15.44:0/2233617342 conn(0x7f34dc0fd200 msgr2=0x7f34dc0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:24.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.837+0000 7f34e2c7d640 1 -- 172.21.15.44:0/2233617342 shutdown_connections 2024-04-21T23:16:24.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.837+0000 7f34e2c7d640 1 -- 172.21.15.44:0/2233617342 wait complete. 2024-04-21T23:16:24.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34e2c7d640 1 Processor -- start 2024-04-21T23:16:24.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34e2c7d640 1 -- start start 2024-04-21T23:16:24.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34e2c7d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34dc073d10 0x7f34dc19a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34e2c7d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34dc074630 0x7f34dc19a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34e2c7d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34dc10e0e0 0x7f34dc19ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34e2c7d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f34dc113e10 con 0x7f34dc074630 2024-04-21T23:16:24.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34e2c7d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f34dc113c90 con 0x7f34dc10e0e0 2024-04-21T23:16:24.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34e2c7d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f34dc113f90 con 0x7f34dc073d10 2024-04-21T23:16:24.840 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34dbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34dc074630 0x7f34dc19a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34dbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34dc074630 0x7f34dc19a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36932/0 (socket says 172.21.15.44:36932) 2024-04-21T23:16:24.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34dbfff640 1 -- 172.21.15.44:0/60653301 learned_addr learned my addr 172.21.15.44:0/60653301 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:24.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34e121a640 1 --2- 172.21.15.44:0/60653301 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34dc10e0e0 0x7f34dc19ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34e0a19640 1 --2- 172.21.15.44:0/60653301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34dc073d10 0x7f34dc19a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34dbfff640 1 -- 172.21.15.44:0/60653301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34dc073d10 msgr2=0x7f34dc19a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34dbfff640 1 --2- 172.21.15.44:0/60653301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34dc073d10 0x7f34dc19a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34dbfff640 1 -- 172.21.15.44:0/60653301 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34dc10e0e0 msgr2=0x7f34dc19ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34dbfff640 1 --2- 172.21.15.44:0/60653301 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34dc10e0e0 0x7f34dc19ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34dbfff640 1 -- 172.21.15.44:0/60653301 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34dc19f360 con 0x7f34dc074630 2024-04-21T23:16:24.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34e0a19640 1 --2- 172.21.15.44:0/60653301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34dc073d10 0x7f34dc19a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:16:24.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34e121a640 1 --2- 172.21.15.44:0/60653301 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34dc10e0e0 0x7f34dc19ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:24.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.838+0000 7f34dbfff640 1 --2- 172.21.15.44:0/60653301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34dc074630 0x7f34dc19a8b0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f34c800e9c0 tx=0x7f34c800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:24.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.839+0000 7f34d9ffb640 1 -- 172.21.15.44:0/60653301 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c800cd80 con 0x7f34dc074630 2024-04-21T23:16:24.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.839+0000 7f34d9ffb640 1 -- 172.21.15.44:0/60653301 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34c80035f0 con 0x7f34dc074630 2024-04-21T23:16:24.841 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.839+0000 7f34e2c7d640 1 -- 172.21.15.44:0/60653301 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34dc19f650 con 0x7f34dc074630 2024-04-21T23:16:24.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.839+0000 7f34d9ffb640 1 -- 172.21.15.44:0/60653301 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c8010690 con 0x7f34dc074630 2024-04-21T23:16:24.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.839+0000 7f34e2c7d640 1 -- 172.21.15.44:0/60653301 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34dc1a6ee0 con 0x7f34dc074630 2024-04-21T23:16:24.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.841+0000 7f34d9ffb640 1 -- 172.21.15.44:0/60653301 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f34c80037a0 con 0x7f34dc074630 2024-04-21T23:16:24.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.841+0000 7f34e5867740 1 Processor -- start 2024-04-21T23:16:24.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.841+0000 7f34e5867740 1 -- start start 2024-04-21T23:16:24.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.841+0000 7f34d9ffb640 1 --2- 172.21.15.44:0/60653301 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f34cc0a2d00 0x7f34cc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34e5867740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916b9f30 0x5636916ba310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34e5867740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916ba850 0x5636916c2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34d9ffb640 1 -- 172.21.15.44:0/60653301 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7f34c80cfbf0 con 0x7f34dc074630 2024-04-21T23:16:24.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34e5867740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916c3430 0x5636916c5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34e5867740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56369161e560 con 0x5636916b9f30 2024-04-21T23:16:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34e5867740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5636915c6730 con 0x5636916ba850 2024-04-21T23:16:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34e5867740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56369161a520 con 0x5636916c3430 2024-04-21T23:16:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34e121a640 1 --2- 172.21.15.44:0/60653301 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f34cc0a2d00 0x7f34cc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34e0a19640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916b9f30 0x5636916ba310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34e0a19640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916b9f30 0x5636916ba310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36942/0 (socket says 172.21.15.44:36942) 2024-04-21T23:16:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34e0a19640 1 -- 172.21.15.44:0/2261692919 learned_addr learned my addr 172.21.15.44:0/2261692919 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34dbfff640 1 --2- 172.21.15.44:0/2261692919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916ba850 0x5636916c2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34e0a19640 1 -- 172.21.15.44:0/2261692919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916c3430 msgr2=0x5636916c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34e121a640 1 --2- 172.21.15.44:0/2261692919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916c3430 0x5636916c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34e0a19640 1 --2- 172.21.15.44:0/2261692919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916c3430 0x5636916c5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34e0a19640 1 -- 172.21.15.44:0/2261692919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916ba850 msgr2=0x5636916c2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34e0a19640 1 --2- 172.21.15.44:0/2261692919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916ba850 0x5636916c2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34e0a19640 1 -- 172.21.15.44:0/2261692919 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636913fc7d0 con 0x5636916b9f30 2024-04-21T23:16:24.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34dbfff640 1 --2- 172.21.15.44:0/2261692919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916ba850 0x5636916c2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34e121a640 1 --2- 172.21.15.44:0/2261692919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916c3430 0x5636916c5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34e121a640 1 --2- 172.21.15.44:0/60653301 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f34cc0a2d00 0x7f34cc0a51c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f34d4009fd0 tx=0x7f34d4011b20 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34e0a19640 1 --2- 172.21.15.44:0/2261692919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916b9f30 0x5636916ba310 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009900 tx=0x7f34d0005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74ed142b1e1441f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.842+0000 7f34c6ffd640 1 -- 172.21.15.44:0/2261692919 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d0042070 con 0x5636916b9f30 2024-04-21T23:16:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.843+0000 7f34c6ffd640 1 -- 172.21.15.44:0/2261692919 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34d002cb20 con 0x5636916b9f30 2024-04-21T23:16:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.843+0000 7f34c6ffd640 1 -- 172.21.15.44:0/2261692919 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d00046c0 con 0x5636916b9f30 2024-04-21T23:16:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.843+0000 7f34e5867740 1 -- 172.21.15.44:0/2261692919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916b9f30 msgr2=0x5636916ba310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.843+0000 7f34e5867740 1 --2- 172.21.15.44:0/2261692919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916b9f30 0x5636916ba310 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009900 tx=0x7f34d0005dd0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.843+0000 7f34e5867740 1 -- 172.21.15.44:0/2261692919 shutdown_connections 2024-04-21T23:16:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.843+0000 7f34e5867740 1 --2- 172.21.15.44:0/2261692919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916c3430 0x5636916c5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.843+0000 7f34e5867740 1 --2- 172.21.15.44:0/2261692919 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916ba850 0x5636916c2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.843+0000 7f34e5867740 1 --2- 172.21.15.44:0/2261692919 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916b9f30 0x5636916ba310 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.843+0000 7f34e5867740 1 -- 172.21.15.44:0/2261692919 >> 172.21.15.44:0/2261692919 conn(0x5636916b4660 msgr2=0x5636916b4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:24.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.843+0000 7f34e5867740 1 -- 172.21.15.44:0/2261692919 shutdown_connections 2024-04-21T23:16:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.843+0000 7f34e5867740 1 -- 172.21.15.44:0/2261692919 wait complete. 2024-04-21T23:16:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.844+0000 7f34e5867740 1 -- start start 2024-04-21T23:16:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.844+0000 7f34e5867740 1 Processor -- start 2024-04-21T23:16:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.844+0000 7f34e5867740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916b9f30 0x5636916c0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.844+0000 7f34e5867740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916ba850 0x5636916c09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.844+0000 7f34e5867740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916c3430 0x5636916c0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.844+0000 7f34e5867740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5636916c61c0 con 0x5636916b9f30 2024-04-21T23:16:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.844+0000 7f34e5867740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5636915d9e00 con 0x5636916ba850 2024-04-21T23:16:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.844+0000 7f34e5867740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5636916c6340 con 0x5636916c3430 2024-04-21T23:16:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.844+0000 7f34e0a19640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916b9f30 0x5636916c0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.844+0000 7f34e5867740 10 client.0 _ll_register_callbacks cb 0x563691614eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.844+0000 7f34e0a19640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916b9f30 0x5636916c0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36950/0 (socket says 172.21.15.44:36950) 2024-04-21T23:16:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.844+0000 7f34e0a19640 1 -- 172.21.15.44:0/2875022018 learned_addr learned my addr 172.21.15.44:0/2875022018 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.844+0000 7f34dbfff640 1 --2- 172.21.15.44:0/2875022018 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916ba850 0x5636916c09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.844+0000 7f34e121a640 1 --2- 172.21.15.44:0/2875022018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916c3430 0x5636916c0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.844+0000 7f34e0a19640 1 -- 172.21.15.44:0/2875022018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916c3430 msgr2=0x5636916c0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.844+0000 7f34e0a19640 1 --2- 172.21.15.44:0/2875022018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916c3430 0x5636916c0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.844+0000 7f34e0a19640 1 -- 172.21.15.44:0/2875022018 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916ba850 msgr2=0x5636916c09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.844+0000 7f34e0a19640 1 --2- 172.21.15.44:0/2875022018 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916ba850 0x5636916c09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.844+0000 7f34e0a19640 1 -- 172.21.15.44:0/2875022018 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636913323f0 con 0x5636916b9f30 2024-04-21T23:16:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.845+0000 7f34e121a640 1 --2- 172.21.15.44:0/2875022018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916c3430 0x5636916c0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.845+0000 7f34e0a19640 1 --2- 172.21.15.44:0/2875022018 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916b9f30 0x5636916c0470 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f34d0000c00 tx=0x7f34d0004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92fc87ef36913c0c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.845+0000 7f34bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:16:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.845+0000 7f34bbfff640 1 -- 172.21.15.44:0/2875022018 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d004b070 con 0x5636916b9f30 2024-04-21T23:16:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.845+0000 7f34bbfff640 1 -- 172.21.15.44:0/2875022018 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7f34d0005500 con 0x5636916b9f30 2024-04-21T23:16:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.845+0000 7f34e5867740 1 -- 172.21.15.44:0/2875022018 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34d002cb20 con 0x5636916b9f30 2024-04-21T23:16:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.845+0000 7f34bbfff640 1 -- 172.21.15.44:0/2875022018 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f34d0042070 con 0x5636916b9f30 2024-04-21T23:16:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.845+0000 7f34e5867740 10 client.17334 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.845+0000 7f34e5867740 1 -- 172.21.15.44:0/2875022018 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5636912f20f0 con 0x5636916b9f30 2024-04-21T23:16:24.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.845+0000 7f34bbfff640 1 -- 172.21.15.44:0/2875022018 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34d002cb20 con 0x5636916b9f30 2024-04-21T23:16:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.845+0000 7f34bbfff640 1 -- 172.21.15.44:0/2875022018 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f34d00038f0 con 0x5636916b9f30 2024-04-21T23:16:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.845+0000 7f34e5867740 10 client.17334 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.845+0000 7f34e5867740 10 client.17334 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.845+0000 7f34e5867740 20 client.17334 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.845+0000 7f34e5867740 1 --2- 172.21.15.44:0/2875022018 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5636916d3570 0x5636916d5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.845+0000 7f34e5867740 4 client.17334 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:16:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.845+0000 7f34e5867740 1 -- 172.21.15.44:0/2875022018 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56369162baa0 con 0x5636916d3570 2024-04-21T23:16:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.846+0000 7f34dbfff640 1 --2- 172.21.15.44:0/2875022018 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5636916d3570 0x5636916d5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.846+0000 7f34bbfff640 10 client.17334 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.846+0000 7f34dbfff640 1 --2- 172.21.15.44:0/2875022018 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5636916d3570 0x5636916d5960 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebaa1aae2ad6842a server_cookie=fe8ed0fbaa3d3418 in_seq=0 out_seq=0 2024-04-21T23:16:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.847+0000 7f34bbfff640 1 -- 172.21.15.44:0/2875022018 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56369162baa0 con 0x5636916d3570 2024-04-21T23:16:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.847+0000 7f34bbfff640 10 client.17334 handle_command_reply: tid=0 2024-04-21T23:16:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.847+0000 7f34e5867740 1 client.17334 shutdown 2024-04-21T23:16:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.847+0000 7f34e5867740 1 -- 172.21.15.44:0/2875022018 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916b9f30 msgr2=0x5636916c0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.847+0000 7f34e5867740 1 --2- 172.21.15.44:0/2875022018 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916b9f30 0x5636916c0470 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f34d0000c00 tx=0x7f34d0004d30 comp rx=0 tx=0).stop 2024-04-21T23:16:24.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.847+0000 7f34e5867740 1 -- 172.21.15.44:0/2875022018 shutdown_connections 2024-04-21T23:16:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.847+0000 7f34e5867740 1 --2- 172.21.15.44:0/2875022018 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5636916d3570 0x5636916d5960 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.847+0000 7f34e5867740 1 --2- 172.21.15.44:0/2875022018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916c3430 0x5636916c0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.847+0000 7f34e5867740 1 --2- 172.21.15.44:0/2875022018 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916ba850 0x5636916c09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.847+0000 7f34bbfff640 0 client.17334 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.847+0000 7f34e5867740 1 --2- 172.21.15.44:0/2875022018 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916b9f30 0x5636916c0470 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.847+0000 7f34e5867740 1 -- 172.21.15.44:0/2875022018 >> 172.21.15.44:0/2875022018 conn(0x5636916b4660 msgr2=0x5636916b63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.847+0000 7f34e5867740 1 -- 172.21.15.44:0/2875022018 shutdown_connections 2024-04-21T23:16:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.847+0000 7f34e5867740 1 -- 172.21.15.44:0/2875022018 wait complete. 2024-04-21T23:16:24.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.847+0000 7f34e5867740 20 client.17334 trim_cache size 0 max 16384 2024-04-21T23:16:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.864+0000 7f34e5867740 1 Processor -- start 2024-04-21T23:16:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.864+0000 7f34e5867740 1 -- start start 2024-04-21T23:16:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34e5867740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916b9f30 0x5636916c2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34e5867740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916ba850 0x5636916cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34e5867740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916c3430 0x5636916c0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34e5867740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f34dc1aedd0 con 0x5636916c3430 2024-04-21T23:16:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34e5867740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f34dc1b1650 con 0x5636916ba850 2024-04-21T23:16:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34e5867740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f34dc1af440 con 0x5636916b9f30 2024-04-21T23:16:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34e121a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916c3430 0x5636916c0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34e121a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916c3430 0x5636916c0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36952/0 (socket says 172.21.15.44:36952) 2024-04-21T23:16:24.869 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34e121a640 1 -- 172.21.15.44:0/1969341459 learned_addr learned my addr 172.21.15.44:0/1969341459 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:24.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34dbfff640 1 --2- 172.21.15.44:0/1969341459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916ba850 0x5636916cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34e0a19640 1 --2- 172.21.15.44:0/1969341459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916b9f30 0x5636916c2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34e121a640 1 -- 172.21.15.44:0/1969341459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916b9f30 msgr2=0x5636916c2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:24.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34e121a640 1 --2- 172.21.15.44:0/1969341459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916b9f30 0x5636916c2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34e121a640 1 -- 172.21.15.44:0/1969341459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916ba850 msgr2=0x5636916cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:24.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34e121a640 1 --2- 172.21.15.44:0/1969341459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916ba850 0x5636916cf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34e121a640 1 -- 172.21.15.44:0/1969341459 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636916be9c0 con 0x5636916c3430 2024-04-21T23:16:24.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34e0a19640 1 --2- 172.21.15.44:0/1969341459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916b9f30 0x5636916c2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:16:24.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34dbfff640 1 --2- 172.21.15.44:0/1969341459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916ba850 0x5636916cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:16:24.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34e121a640 1 --2- 172.21.15.44:0/1969341459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916c3430 0x5636916c0b40 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f34d400fce0 tx=0x7f34d401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbc4a5ec87c35d5e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:24.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34c67fc640 1 -- 172.21.15.44:0/1969341459 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d4017040 con 0x5636916c3430 2024-04-21T23:16:24.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34c67fc640 1 -- 172.21.15.44:0/1969341459 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34d4019070 con 0x5636916c3430 2024-04-21T23:16:24.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.865+0000 7f34c67fc640 1 -- 172.21.15.44:0/1969341459 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d401d3e0 con 0x5636916c3430 2024-04-21T23:16:24.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.866+0000 7f34e5867740 1 -- 172.21.15.44:0/1969341459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916c3430 msgr2=0x5636916c0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.870 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.866+0000 7f34e5867740 1 --2- 172.21.15.44:0/1969341459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916c3430 0x5636916c0b40 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f34d400fce0 tx=0x7f34d401b000 comp rx=0 tx=0).stop 2024-04-21T23:16:24.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.866+0000 7f34e5867740 1 -- 172.21.15.44:0/1969341459 shutdown_connections 2024-04-21T23:16:24.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.866+0000 7f34e5867740 1 --2- 172.21.15.44:0/1969341459 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916c3430 0x5636916c0b40 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.866+0000 7f34e5867740 1 --2- 172.21.15.44:0/1969341459 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916ba850 0x5636916cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.866+0000 7f34e5867740 1 --2- 172.21.15.44:0/1969341459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916b9f30 0x5636916c2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.866+0000 7f34e5867740 1 -- 172.21.15.44:0/1969341459 >> 172.21.15.44:0/1969341459 conn(0x5636916b4660 msgr2=0x5636916c0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:24.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.866+0000 7f34e5867740 1 -- 172.21.15.44:0/1969341459 shutdown_connections 2024-04-21T23:16:24.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.866+0000 7f34e5867740 1 -- 172.21.15.44:0/1969341459 wait complete. 2024-04-21T23:16:24.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.867+0000 7f34e5867740 1 -- start start 2024-04-21T23:16:24.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.867+0000 7f34e5867740 1 Processor -- start 2024-04-21T23:16:24.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.867+0000 7f34e5867740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916b9f30 0x5636916cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.867+0000 7f34e5867740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916ba850 0x5636916bced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.867+0000 7f34e5867740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916c3430 0x5636916bd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.867+0000 7f34e5867740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f34dc1afb10 con 0x5636916ba850 2024-04-21T23:16:24.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.867+0000 7f34e5867740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f34dc1b0f80 con 0x5636916c3430 2024-04-21T23:16:24.871 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.867+0000 7f34e5867740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f34dc1b01e0 con 0x5636916b9f30 2024-04-21T23:16:24.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.867+0000 7f34e5867740 10 client.0 _ll_register_callbacks cb 0x7f34c8009760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:24.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.867+0000 7f34dbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916ba850 0x5636916bced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.867+0000 7f34e0a19640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916b9f30 0x5636916cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.867+0000 7f34dbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916ba850 0x5636916bced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36966/0 (socket says 172.21.15.44:36966) 2024-04-21T23:16:24.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.867+0000 7f34dbfff640 1 -- 172.21.15.44:0/2852015646 learned_addr learned my addr 172.21.15.44:0/2852015646 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:24.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.867+0000 7f34e121a640 1 --2- 172.21.15.44:0/2852015646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916c3430 0x5636916bd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.867+0000 7f34dbfff640 1 -- 172.21.15.44:0/2852015646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916b9f30 msgr2=0x5636916cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:24.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.867+0000 7f34dbfff640 1 --2- 172.21.15.44:0/2852015646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916b9f30 0x5636916cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.867+0000 7f34dbfff640 1 -- 172.21.15.44:0/2852015646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916c3430 msgr2=0x5636916bd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:24.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.867+0000 7f34dbfff640 1 --2- 172.21.15.44:0/2852015646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916c3430 0x5636916bd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.867+0000 7f34dbfff640 1 -- 172.21.15.44:0/2852015646 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636916bdc60 con 0x5636916ba850 2024-04-21T23:16:24.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.868+0000 7f34dbfff640 1 --2- 172.21.15.44:0/2852015646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916ba850 0x5636916bced0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f34c800c6a0 tx=0x7f34c809bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d708bd7ba196b4b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:24.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.868+0000 7f34bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:16:24.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.868+0000 7f34bbfff640 1 -- 172.21.15.44:0/2852015646 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c80cd220 con 0x5636916ba850 2024-04-21T23:16:24.872 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.868+0000 7f34bbfff640 1 -- 172.21.15.44:0/2852015646 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7f34c80983e0 con 0x5636916ba850 2024-04-21T23:16:24.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.868+0000 7f34e5867740 1 -- 172.21.15.44:0/2852015646 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34d4019070 con 0x5636916ba850 2024-04-21T23:16:24.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.868+0000 7f34bbfff640 1 -- 172.21.15.44:0/2852015646 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f34c80a4550 con 0x5636916ba850 2024-04-21T23:16:24.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.868+0000 7f34e5867740 10 client.17340 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:24.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.868+0000 7f34e5867740 1 -- 172.21.15.44:0/2852015646 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5636917398a0 con 0x5636916ba850 2024-04-21T23:16:24.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.868+0000 7f34bbfff640 1 -- 172.21.15.44:0/2852015646 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34d4019070 con 0x5636916ba850 2024-04-21T23:16:24.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.868+0000 7f34bbfff640 1 -- 172.21.15.44:0/2852015646 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f34c809b1c0 con 0x5636916ba850 2024-04-21T23:16:24.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.868+0000 7f34e5867740 10 client.17340 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:24.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.868+0000 7f34e5867740 10 client.17340 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:24.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.868+0000 7f34e5867740 20 client.17340 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:24.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.868+0000 7f34e5867740 1 --2- 172.21.15.44:0/2852015646 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563691739fc0 0x56369173c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.868+0000 7f34e5867740 4 client.17340 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:16:24.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.868+0000 7f34e5867740 1 -- 172.21.15.44:0/2852015646 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563691625df0 con 0x563691739fc0 2024-04-21T23:16:24.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.868+0000 7f34e0a19640 1 --2- 172.21.15.44:0/2852015646 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563691739fc0 0x56369173c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.869+0000 7f34e0a19640 1 --2- 172.21.15.44:0/2852015646 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563691739fc0 0x56369173c3e0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe23cef2c9440210 server_cookie=f3e710eebe78fe68 in_seq=0 out_seq=0 2024-04-21T23:16:24.873 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.869+0000 7f34bbfff640 10 client.17340 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:24.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.871+0000 7f34bbfff640 1 -- 172.21.15.44:0/2852015646 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563691625df0 con 0x563691739fc0 2024-04-21T23:16:24.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.871+0000 7f34bbfff640 10 client.17340 handle_command_reply: tid=0 2024-04-21T23:16:24.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.871+0000 7f34e5867740 1 client.17340 shutdown 2024-04-21T23:16:24.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.871+0000 7f34e5867740 1 -- 172.21.15.44:0/2852015646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916ba850 msgr2=0x5636916bced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.871+0000 7f34e5867740 1 --2- 172.21.15.44:0/2852015646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916ba850 0x5636916bced0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f34c800c6a0 tx=0x7f34c809bd70 comp rx=0 tx=0).stop 2024-04-21T23:16:24.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.871+0000 7f34e5867740 1 -- 172.21.15.44:0/2852015646 shutdown_connections 2024-04-21T23:16:24.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.871+0000 7f34e5867740 1 --2- 172.21.15.44:0/2852015646 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x563691739fc0 0x56369173c3e0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.871+0000 7f34bbfff640 0 client.17340 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:24.875 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.871+0000 7f34e5867740 1 --2- 172.21.15.44:0/2852015646 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5636916c3430 0x5636916bd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.871+0000 7f34e5867740 1 --2- 172.21.15.44:0/2852015646 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5636916ba850 0x5636916bced0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.871+0000 7f34e5867740 1 --2- 172.21.15.44:0/2852015646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5636916b9f30 0x5636916cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.871+0000 7f34e5867740 1 -- 172.21.15.44:0/2852015646 >> 172.21.15.44:0/2852015646 conn(0x5636916b4660 msgr2=0x5636916b36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:24.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.872+0000 7f34e5867740 1 -- 172.21.15.44:0/2852015646 shutdown_connections 2024-04-21T23:16:24.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.872+0000 7f34e5867740 1 -- 172.21.15.44:0/2852015646 wait complete. 2024-04-21T23:16:24.876 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.872+0000 7f34e5867740 20 client.17340 trim_cache size 0 max 16384 2024-04-21T23:16:24.876 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:16:24.876 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-21T23:16:24.876 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-21T23:16:24.876 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:16:24.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.872+0000 7f34e2c7d640 1 -- 172.21.15.44:0/60653301 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f34cc0a2d00 msgr2=0x7f34cc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.872+0000 7f34e2c7d640 1 --2- 172.21.15.44:0/60653301 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f34cc0a2d00 0x7f34cc0a51c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f34d4009fd0 tx=0x7f34d4011b20 comp rx=0 tx=0).stop 2024-04-21T23:16:24.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.872+0000 7f34e2c7d640 1 -- 172.21.15.44:0/60653301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34dc074630 msgr2=0x7f34dc19a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.877 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.872+0000 7f34e2c7d640 1 --2- 172.21.15.44:0/60653301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34dc074630 0x7f34dc19a8b0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f34c800e9c0 tx=0x7f34c800ee80 comp rx=0 tx=0).stop 2024-04-21T23:16:24.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.872+0000 7f34e2c7d640 1 -- 172.21.15.44:0/60653301 shutdown_connections 2024-04-21T23:16:24.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.872+0000 7f34e2c7d640 1 --2- 172.21.15.44:0/60653301 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f34cc0a2d00 0x7f34cc0a51c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.872+0000 7f34e2c7d640 1 --2- 172.21.15.44:0/60653301 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f34dc10e0e0 0x7f34dc19ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.872+0000 7f34e2c7d640 1 --2- 172.21.15.44:0/60653301 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f34dc074630 0x7f34dc19a8b0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.872+0000 7f34e2c7d640 1 --2- 172.21.15.44:0/60653301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34dc073d10 0x7f34dc19a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.872+0000 7f34e2c7d640 1 -- 172.21.15.44:0/60653301 >> 172.21.15.44:0/60653301 conn(0x7f34dc0fd200 msgr2=0x7f34dc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:24.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.873+0000 7f34e2c7d640 1 -- 172.21.15.44:0/60653301 shutdown_connections 2024-04-21T23:16:24.878 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.873+0000 7f34e2c7d640 1 -- 172.21.15.44:0/60653301 wait complete. 2024-04-21T23:16:24.884 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:16:24.884 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:16:24.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.996+0000 7efde851e640 1 -- 172.21.15.44:0/2066960369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efde01029b0 msgr2=0x7efde0102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.996+0000 7efde851e640 1 --2- 172.21.15.44:0/2066960369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efde01029b0 0x7efde0102d90 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7efddc009900 tx=0x7efddc02bcd0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.996+0000 7efde851e640 1 -- 172.21.15.44:0/2066960369 shutdown_connections 2024-04-21T23:16:24.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.996+0000 7efde851e640 1 --2- 172.21.15.44:0/2066960369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde01032d0 0x7efde010b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.996+0000 7efde851e640 1 --2- 172.21.15.44:0/2066960369 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efde010bd80 0x7efde010e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.996+0000 7efde851e640 1 --2- 172.21.15.44:0/2066960369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efde01029b0 0x7efde0102d90 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.996+0000 7efde851e640 1 -- 172.21.15.44:0/2066960369 >> 172.21.15.44:0/2066960369 conn(0x7efde0077740 msgr2=0x7efde0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:24.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.996+0000 7efde851e640 1 -- 172.21.15.44:0/2066960369 shutdown_connections 2024-04-21T23:16:24.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.996+0000 7efde851e640 1 -- 172.21.15.44:0/2066960369 wait complete. 2024-04-21T23:16:24.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.996+0000 7efde851e640 1 Processor -- start 2024-04-21T23:16:24.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.997+0000 7efde851e640 1 -- start start 2024-04-21T23:16:24.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.997+0000 7efde851e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efde01029b0 0x7efde01980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.997+0000 7efde851e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efde01032d0 0x7efde0198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.997+0000 7efde851e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde010bd80 0x7efde019c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.997+0000 7efde851e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7efde0111b10 con 0x7efde01029b0 2024-04-21T23:16:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.997+0000 7efde851e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7efde0111990 con 0x7efde01032d0 2024-04-21T23:16:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.997+0000 7efde851e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efde0111c90 con 0x7efde010bd80 2024-04-21T23:16:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.997+0000 7efde62ba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efde01029b0 0x7efde01980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.997+0000 7efde62ba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efde01029b0 0x7efde01980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:36982/0 (socket says 172.21.15.44:36982) 2024-04-21T23:16:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.997+0000 7efde62ba640 1 -- 172.21.15.44:0/1376664452 learned_addr learned my addr 172.21.15.44:0/1376664452 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.997+0000 7efde5ab9640 1 --2- 172.21.15.44:0/1376664452 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efde01032d0 0x7efde0198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.997+0000 7efde6abb640 1 --2- 172.21.15.44:0/1376664452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde010bd80 0x7efde019c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.997+0000 7efde62ba640 1 -- 172.21.15.44:0/1376664452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde010bd80 msgr2=0x7efde019c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.997+0000 7efde62ba640 1 --2- 172.21.15.44:0/1376664452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde010bd80 0x7efde019c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.997+0000 7efde62ba640 1 -- 172.21.15.44:0/1376664452 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efde01032d0 msgr2=0x7efde0198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.997+0000 7efde62ba640 1 --2- 172.21.15.44:0/1376664452 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efde01032d0 0x7efde0198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:24.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.997+0000 7efde62ba640 1 -- 172.21.15.44:0/1376664452 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efde019d070 con 0x7efde01029b0 2024-04-21T23:16:25.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.997+0000 7efde62ba640 1 --2- 172.21.15.44:0/1376664452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efde01029b0 0x7efde01980d0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7efddc005940 tx=0x7efddc02cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:25.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.997+0000 7efdd77fe640 1 -- 172.21.15.44:0/1376664452 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efddc041070 con 0x7efde01029b0 2024-04-21T23:16:25.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.998+0000 7efdd77fe640 1 -- 172.21.15.44:0/1376664452 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efddc004450 con 0x7efde01029b0 2024-04-21T23:16:25.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.998+0000 7efdd77fe640 1 -- 172.21.15.44:0/1376664452 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efddc003df0 con 0x7efde01029b0 2024-04-21T23:16:25.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.998+0000 7efde851e640 1 -- 172.21.15.44:0/1376664452 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efde019d300 con 0x7efde01029b0 2024-04-21T23:16:25.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.998+0000 7efde851e640 1 -- 172.21.15.44:0/1376664452 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efde01a4bf0 con 0x7efde01029b0 2024-04-21T23:16:25.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:24.999+0000 7efde851e640 1 -- 172.21.15.44:0/1376664452 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efde006a670 con 0x7efde01029b0 2024-04-21T23:16:25.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.000+0000 7efdd77fe640 1 -- 172.21.15.44:0/1376664452 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7efddc044070 con 0x7efde01029b0 2024-04-21T23:16:25.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.000+0000 7efdd77fe640 1 --2- 172.21.15.44:0/1376664452 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7efdcc0a2d80 0x7efdcc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.001+0000 7efdd77fe640 1 -- 172.21.15.44:0/1376664452 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7efddc0f1200 con 0x7efde01029b0 2024-04-21T23:16:25.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.001+0000 7efde5ab9640 1 --2- 172.21.15.44:0/1376664452 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7efdcc0a2d80 0x7efdcc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.001+0000 7efde5ab9640 1 --2- 172.21.15.44:0/1376664452 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7efdcc0a2d80 0x7efdcc0a5240 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efde01000e0 tx=0x7efdd0005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:25.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.005+0000 7efdd77fe640 1 -- 172.21.15.44:0/1376664452 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7efddc0bdb90 con 0x7efde01029b0 2024-04-21T23:16:25.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.249+0000 7efde851e640 1 -- 172.21.15.44:0/1376664452 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efde0106e70 con 0x7efde01029b0 2024-04-21T23:16:25.249 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.250+0000 7efdd77fe640 1 -- 172.21.15.44:0/1376664452 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11791 (secure 0 0 0) 0x7efddc0bd490 con 0x7efde01029b0 2024-04-21T23:16:25.250 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:16:25.250 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:16:20.187271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26863":{"gid":26863,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3179646768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3179646768},{"type":"v1","addr":"172.21.15.69:6839","nonce":3179646768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:16:25.250 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 31 2024-04-21T23:16:25.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.252+0000 7efde851e640 1 -- 172.21.15.44:0/1376664452 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7efdcc0a2d80 msgr2=0x7efdcc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.252+0000 7efde851e640 1 --2- 172.21.15.44:0/1376664452 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7efdcc0a2d80 0x7efdcc0a5240 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efde01000e0 tx=0x7efdd0005f90 comp rx=0 tx=0).stop 2024-04-21T23:16:25.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.252+0000 7efde851e640 1 -- 172.21.15.44:0/1376664452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efde01029b0 msgr2=0x7efde01980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.252+0000 7efde851e640 1 --2- 172.21.15.44:0/1376664452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efde01029b0 0x7efde01980d0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7efddc005940 tx=0x7efddc02cc40 comp rx=0 tx=0).stop 2024-04-21T23:16:25.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.252+0000 7efde851e640 1 -- 172.21.15.44:0/1376664452 shutdown_connections 2024-04-21T23:16:25.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.252+0000 7efde851e640 1 --2- 172.21.15.44:0/1376664452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde010bd80 0x7efde019c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.252 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.252+0000 7efde851e640 1 --2- 172.21.15.44:0/1376664452 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7efdcc0a2d80 0x7efdcc0a5240 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.252+0000 7efde851e640 1 --2- 172.21.15.44:0/1376664452 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efde01032d0 0x7efde0198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.252+0000 7efde851e640 1 --2- 172.21.15.44:0/1376664452 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efde01029b0 0x7efde01980d0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.252+0000 7efde851e640 1 -- 172.21.15.44:0/1376664452 >> 172.21.15.44:0/1376664452 conn(0x7efde0077740 msgr2=0x7efde00fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:25.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.252+0000 7efde851e640 1 -- 172.21.15.44:0/1376664452 shutdown_connections 2024-04-21T23:16:25.253 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.252+0000 7efde851e640 1 -- 172.21.15.44:0/1376664452 wait complete. 2024-04-21T23:16:25.263 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:16:25.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.373+0000 7f1b2d59f640 1 -- 172.21.15.44:0/2740341075 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b281064e0 msgr2=0x7f1b281068c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.373+0000 7f1b2d59f640 1 --2- 172.21.15.44:0/2740341075 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b281064e0 0x7f1b281068c0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f1b18009930 tx=0x7f1b1802bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:25.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.373+0000 7f1b2d59f640 1 -- 172.21.15.44:0/2740341075 shutdown_connections 2024-04-21T23:16:25.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.373+0000 7f1b2d59f640 1 --2- 172.21.15.44:0/2740341075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b28100e00 0x7f1b2810d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.373+0000 7f1b2d59f640 1 --2- 172.21.15.44:0/2740341075 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b28100440 0x7f1b281008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.373+0000 7f1b2d59f640 1 --2- 172.21.15.44:0/2740341075 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b281064e0 0x7f1b281068c0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.373+0000 7f1b2d59f640 1 -- 172.21.15.44:0/2740341075 >> 172.21.15.44:0/2740341075 conn(0x7f1b280fbf50 msgr2=0x7f1b280fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:25.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.373+0000 7f1b2d59f640 1 -- 172.21.15.44:0/2740341075 shutdown_connections 2024-04-21T23:16:25.373 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.373+0000 7f1b2d59f640 1 -- 172.21.15.44:0/2740341075 wait complete. 2024-04-21T23:16:25.375 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.374+0000 7f1b2d59f640 1 Processor -- start 2024-04-21T23:16:25.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.374+0000 7f1b2d59f640 1 -- start start 2024-04-21T23:16:25.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.374+0000 7f1b2d59f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b28100440 0x7f1b28196d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.374+0000 7f1b2d59f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b28100e00 0x7f1b281972b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.374+0000 7f1b2d59f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b281064e0 0x7f1b2819b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.374+0000 7f1b2d59f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1b28110850 con 0x7f1b28100440 2024-04-21T23:16:25.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.374+0000 7f1b2d59f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1b281106d0 con 0x7f1b28100e00 2024-04-21T23:16:25.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.374+0000 7f1b2d59f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1b281109d0 con 0x7f1b281064e0 2024-04-21T23:16:25.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.375+0000 7f1b27fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b28100440 0x7f1b28196d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.375+0000 7f1b277fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b28100e00 0x7f1b281972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.375+0000 7f1b2cd9e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b281064e0 0x7f1b2819b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.375+0000 7f1b27fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b28100440 0x7f1b28196d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37000/0 (socket says 172.21.15.44:37000) 2024-04-21T23:16:25.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.375+0000 7f1b27fff640 1 -- 172.21.15.44:0/3953206082 learned_addr learned my addr 172.21.15.44:0/3953206082 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:25.376 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.375+0000 7f1b277fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b28100e00 0x7f1b281972b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:58666/0 (socket says 172.21.15.44:58666) 2024-04-21T23:16:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.375+0000 7f1b277fe640 1 -- 172.21.15.44:0/3953206082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b281064e0 msgr2=0x7f1b2819b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.375+0000 7f1b277fe640 1 --2- 172.21.15.44:0/3953206082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b281064e0 0x7f1b2819b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.375+0000 7f1b277fe640 1 -- 172.21.15.44:0/3953206082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b28100440 msgr2=0x7f1b28196d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.375+0000 7f1b277fe640 1 --2- 172.21.15.44:0/3953206082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b28100440 0x7f1b28196d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.375+0000 7f1b277fe640 1 -- 172.21.15.44:0/3953206082 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b2819bd60 con 0x7f1b28100e00 2024-04-21T23:16:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.375+0000 7f1b2cd9e640 1 --2- 172.21.15.44:0/3953206082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b281064e0 0x7f1b2819b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.375+0000 7f1b27fff640 1 --2- 172.21.15.44:0/3953206082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b28100440 0x7f1b28196d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.375+0000 7f1b277fe640 1 --2- 172.21.15.44:0/3953206082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b28100e00 0x7f1b281972b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c010c00 tx=0x7f1b1c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.376+0000 7f1b257fa640 1 -- 172.21.15.44:0/3953206082 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c01d070 con 0x7f1b28100e00 2024-04-21T23:16:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.376+0000 7f1b257fa640 1 -- 172.21.15.44:0/3953206082 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b1c018b10 con 0x7f1b28100e00 2024-04-21T23:16:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.376+0000 7f1b2d59f640 1 -- 172.21.15.44:0/3953206082 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b2819c050 con 0x7f1b28100e00 2024-04-21T23:16:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.376+0000 7f1b257fa640 1 -- 172.21.15.44:0/3953206082 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c016c10 con 0x7f1b28100e00 2024-04-21T23:16:25.377 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.376+0000 7f1b2d59f640 1 -- 172.21.15.44:0/3953206082 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b28069710 con 0x7f1b28100e00 2024-04-21T23:16:25.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.378+0000 7f1b257fa640 1 -- 172.21.15.44:0/3953206082 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1b1c020070 con 0x7f1b28100e00 2024-04-21T23:16:25.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.378+0000 7f1b313ec740 1 Processor -- start 2024-04-21T23:16:25.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.378+0000 7f1b313ec740 1 -- start start 2024-04-21T23:16:25.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.378+0000 7f1b257fa640 1 --2- 172.21.15.44:0/3953206082 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1b100a2d40 0x7f1b100a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.378+0000 7f1b313ec740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4e1d0 0x55621ee4e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.378+0000 7f1b313ec740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55621ee4eaf0 0x55621ee57190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.378+0000 7f1b313ec740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55621ee576d0 0x55621ee59ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.378+0000 7f1b313ec740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55621edb2530 con 0x55621ee4e1d0 2024-04-21T23:16:25.380 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.378+0000 7f1b313ec740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55621ed5a830 con 0x55621ee4eaf0 2024-04-21T23:16:25.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.378+0000 7f1b313ec740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55621edae4f0 con 0x55621ee576d0 2024-04-21T23:16:25.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.379+0000 7f1b27fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4e1d0 0x55621ee4e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.379+0000 7f1b27fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4e1d0 0x55621ee4e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37010/0 (socket says 172.21.15.44:37010) 2024-04-21T23:16:25.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.379+0000 7f1b27fff640 1 -- 172.21.15.44:0/1147734170 learned_addr learned my addr 172.21.15.44:0/1147734170 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:25.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.379+0000 7f1b257fa640 1 -- 172.21.15.44:0/3953206082 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7f1b1c0d15f0 con 0x7f1b28100e00 2024-04-21T23:16:25.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.379+0000 7f1b277fe640 1 --2- 172.21.15.44:0/1147734170 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55621ee4eaf0 0x55621ee57190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.379+0000 7f1b2cd9e640 1 --2- 172.21.15.44:0/3953206082 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1b100a2d40 0x7f1b100a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.379+0000 7f1b27fff640 1 -- 172.21.15.44:0/1147734170 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55621ee576d0 msgr2=0x55621ee59ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T23:16:25.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.379+0000 7f1b27fff640 1 --2- 172.21.15.44:0/1147734170 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55621ee576d0 0x55621ee59ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.379+0000 7f1b27fff640 1 -- 172.21.15.44:0/1147734170 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55621ee4eaf0 msgr2=0x55621ee57190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:25.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.379+0000 7f1b27fff640 1 --2- 172.21.15.44:0/1147734170 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55621ee4eaf0 0x55621ee57190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.379+0000 7f1b27fff640 1 -- 172.21.15.44:0/1147734170 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55621ed7e2c0 con 0x55621ee4e1d0 2024-04-21T23:16:25.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.379+0000 7f1b27fff640 1 --2- 172.21.15.44:0/1147734170 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4e1d0 0x55621ee4e5b0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1b18009630 tx=0x7f1b18009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceb1a249a567ea40 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:25.381 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.380+0000 7f1b2cd9e640 1 --2- 172.21.15.44:0/3953206082 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1b100a2d40 0x7f1b100a5200 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x55621ee49a90 tx=0x7f1b20009500 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:25.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.380+0000 7f1b0e7fc640 1 -- 172.21.15.44:0/1147734170 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b18004040 con 0x55621ee4e1d0 2024-04-21T23:16:25.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.380+0000 7f1b0e7fc640 1 -- 172.21.15.44:0/1147734170 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b180041e0 con 0x55621ee4e1d0 2024-04-21T23:16:25.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.380+0000 7f1b0e7fc640 1 -- 172.21.15.44:0/1147734170 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b180035f0 con 0x55621ee4e1d0 2024-04-21T23:16:25.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.380+0000 7f1b313ec740 1 -- 172.21.15.44:0/1147734170 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4e1d0 msgr2=0x55621ee4e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.380+0000 7f1b313ec740 1 --2- 172.21.15.44:0/1147734170 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4e1d0 0x55621ee4e5b0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1b18009630 tx=0x7f1b18009660 comp rx=0 tx=0).stop 2024-04-21T23:16:25.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.381+0000 7f1b313ec740 1 -- 172.21.15.44:0/1147734170 shutdown_connections 2024-04-21T23:16:25.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.381+0000 7f1b313ec740 1 --2- 172.21.15.44:0/1147734170 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55621ee576d0 0x55621ee59ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.381+0000 7f1b313ec740 1 --2- 172.21.15.44:0/1147734170 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55621ee4eaf0 0x55621ee57190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.381+0000 7f1b313ec740 1 --2- 172.21.15.44:0/1147734170 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4e1d0 0x55621ee4e5b0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.381+0000 7f1b313ec740 1 -- 172.21.15.44:0/1147734170 >> 172.21.15.44:0/1147734170 conn(0x55621ee48900 msgr2=0x55621ee48ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:25.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.381+0000 7f1b313ec740 1 -- 172.21.15.44:0/1147734170 shutdown_connections 2024-04-21T23:16:25.382 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.381+0000 7f1b313ec740 1 -- 172.21.15.44:0/1147734170 wait complete. 2024-04-21T23:16:25.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.381+0000 7f1b313ec740 1 -- start start 2024-04-21T23:16:25.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.382+0000 7f1b313ec740 1 Processor -- start 2024-04-21T23:16:25.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.382+0000 7f1b313ec740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4e1d0 0x55621ee57040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.382+0000 7f1b313ec740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55621ee4eaf0 0x55621ee54440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.382+0000 7f1b27fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4e1d0 0x55621ee57040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.382+0000 7f1b27fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4e1d0 0x55621ee57040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37026/0 (socket says 172.21.15.44:37026) 2024-04-21T23:16:25.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.382+0000 7f1b27fff640 1 -- 172.21.15.44:0/107982081 learned_addr learned my addr 172.21.15.44:0/107982081 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:25.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.382+0000 7f1b313ec740 1 --2- 172.21.15.44:0/107982081 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55621ee576d0 0x55621ee54a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.382+0000 7f1b313ec740 1 -- 172.21.15.44:0/107982081 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55621ee5a460 con 0x55621ee4e1d0 2024-04-21T23:16:25.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.382+0000 7f1b313ec740 1 -- 172.21.15.44:0/107982081 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55621ed6df10 con 0x55621ee576d0 2024-04-21T23:16:25.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.382+0000 7f1b313ec740 1 -- 172.21.15.44:0/107982081 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55621ee5a5e0 con 0x55621ee4eaf0 2024-04-21T23:16:25.386 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.382+0000 7f1b277fe640 1 --2- 172.21.15.44:0/107982081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55621ee4eaf0 0x55621ee54440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.383+0000 7f1b2cd9e640 1 --2- 172.21.15.44:0/107982081 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55621ee576d0 0x55621ee54a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.383+0000 7f1b313ec740 10 client.0 _ll_register_callbacks cb 0x55621eac5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:25.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.383+0000 7f1b27fff640 1 -- 172.21.15.44:0/107982081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55621ee4eaf0 msgr2=0x55621ee54440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:25.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.383+0000 7f1b27fff640 1 --2- 172.21.15.44:0/107982081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55621ee4eaf0 0x55621ee54440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.383+0000 7f1b27fff640 1 -- 172.21.15.44:0/107982081 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55621ee576d0 msgr2=0x55621ee54a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:25.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.383+0000 7f1b27fff640 1 --2- 172.21.15.44:0/107982081 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55621ee576d0 0x55621ee54a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.383+0000 7f1b27fff640 1 -- 172.21.15.44:0/107982081 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55621ed3fc80 con 0x55621ee4e1d0 2024-04-21T23:16:25.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.383+0000 7f1b277fe640 1 --2- 172.21.15.44:0/107982081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55621ee4eaf0 0x55621ee54440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:25.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.383+0000 7f1b27fff640 1 --2- 172.21.15.44:0/107982081 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4e1d0 0x55621ee57040 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f1b18036040 tx=0x7f1b18041e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1016937233fa471b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:25.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.383+0000 7f1b077fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:16:25.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.383+0000 7f1b077fe640 1 -- 172.21.15.44:0/107982081 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1802c880 con 0x55621ee4e1d0 2024-04-21T23:16:25.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.383+0000 7f1b077fe640 1 -- 172.21.15.44:0/107982081 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7f1b1802ca20 con 0x55621ee4e1d0 2024-04-21T23:16:25.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.383+0000 7f1b313ec740 1 -- 172.21.15.44:0/107982081 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b180041e0 con 0x55621ee4e1d0 2024-04-21T23:16:25.387 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.383+0000 7f1b077fe640 1 -- 172.21.15.44:0/107982081 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1b1802cc20 con 0x55621ee4e1d0 2024-04-21T23:16:25.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.384+0000 7f1b313ec740 10 client.17358 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:25.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.384+0000 7f1b313ec740 1 -- 172.21.15.44:0/107982081 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55621eac63f0 con 0x55621ee4e1d0 2024-04-21T23:16:25.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.384+0000 7f1b077fe640 1 -- 172.21.15.44:0/107982081 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b180041e0 con 0x55621ee4e1d0 2024-04-21T23:16:25.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.385+0000 7f1b077fe640 1 -- 172.21.15.44:0/107982081 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f1b180564d0 con 0x55621ee4e1d0 2024-04-21T23:16:25.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.385+0000 7f1b313ec740 10 client.17358 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:25.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.385+0000 7f1b313ec740 10 client.17358 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:25.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.385+0000 7f1b313ec740 20 client.17358 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:25.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.385+0000 7f1b313ec740 1 --2- 172.21.15.44:0/107982081 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55621ee675a0 0x55621ee69990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.385+0000 7f1b313ec740 4 client.17358 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:16:25.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.385+0000 7f1b313ec740 1 -- 172.21.15.44:0/107982081 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55621edbfa70 con 0x55621ee675a0 2024-04-21T23:16:25.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.385+0000 7f1b277fe640 1 --2- 172.21.15.44:0/107982081 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55621ee675a0 0x55621ee69990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.386+0000 7f1b077fe640 10 client.17358 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:25.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.386+0000 7f1b277fe640 1 --2- 172.21.15.44:0/107982081 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55621ee675a0 0x55621ee69990 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b16c9c2fdd578ce server_cookie=5546faf3f8f1cb4 in_seq=0 out_seq=0 2024-04-21T23:16:25.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.386+0000 7f1b077fe640 1 -- 172.21.15.44:0/107982081 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55621edbfa70 con 0x55621ee675a0 2024-04-21T23:16:25.388 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.386+0000 7f1b077fe640 10 client.17358 handle_command_reply: tid=0 2024-04-21T23:16:25.389 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.386+0000 7f1b313ec740 1 client.17358 shutdown 2024-04-21T23:16:25.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.387+0000 7f1b313ec740 1 -- 172.21.15.44:0/107982081 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4e1d0 msgr2=0x55621ee57040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.387+0000 7f1b313ec740 1 --2- 172.21.15.44:0/107982081 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4e1d0 0x55621ee57040 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f1b18036040 tx=0x7f1b18041e70 comp rx=0 tx=0).stop 2024-04-21T23:16:25.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.387+0000 7f1b313ec740 1 -- 172.21.15.44:0/107982081 shutdown_connections 2024-04-21T23:16:25.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.387+0000 7f1b313ec740 1 --2- 172.21.15.44:0/107982081 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55621ee675a0 0x55621ee69990 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.387+0000 7f1b313ec740 1 --2- 172.21.15.44:0/107982081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55621ee4eaf0 0x55621ee54440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.387+0000 7f1b313ec740 1 --2- 172.21.15.44:0/107982081 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55621ee576d0 0x55621ee54a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.387+0000 7f1b313ec740 1 --2- 172.21.15.44:0/107982081 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4e1d0 0x55621ee57040 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.387+0000 7f1b313ec740 1 -- 172.21.15.44:0/107982081 >> 172.21.15.44:0/107982081 conn(0x55621ee48900 msgr2=0x55621ee4a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:25.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.387+0000 7f1b077fe640 0 client.17358 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:25.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.387+0000 7f1b313ec740 1 -- 172.21.15.44:0/107982081 shutdown_connections 2024-04-21T23:16:25.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.387+0000 7f1b313ec740 1 -- 172.21.15.44:0/107982081 wait complete. 2024-04-21T23:16:25.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.387+0000 7f1b313ec740 20 client.17358 trim_cache size 0 max 16384 2024-04-21T23:16:25.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.402+0000 7f1b313ec740 1 Processor -- start 2024-04-21T23:16:25.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.402+0000 7f1b313ec740 1 -- start start 2024-04-21T23:16:25.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.403+0000 7f1b313ec740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55621ee4e1d0 0x55621ee4a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.403+0000 7f1b313ec740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4eaf0 0x55621ee53bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.403+0000 7f1b313ec740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55621ee576d0 0x55621ee56b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.403+0000 7f1b313ec740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1b281aa7b0 con 0x55621ee4eaf0 2024-04-21T23:16:25.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.403+0000 7f1b313ec740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1b281ad030 con 0x55621ee576d0 2024-04-21T23:16:25.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.403+0000 7f1b313ec740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1b281aae20 con 0x55621ee4e1d0 2024-04-21T23:16:25.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.403+0000 7f1b277fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4eaf0 0x55621ee53bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.403+0000 7f1b277fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4eaf0 0x55621ee53bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37030/0 (socket says 172.21.15.44:37030) 2024-04-21T23:16:25.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.403+0000 7f1b277fe640 1 -- 172.21.15.44:0/688377826 learned_addr learned my addr 172.21.15.44:0/688377826 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:25.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.403+0000 7f1b27fff640 1 --2- 172.21.15.44:0/688377826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55621ee4e1d0 0x55621ee4a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.403+0000 7f1b2cd9e640 1 --2- 172.21.15.44:0/688377826 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55621ee576d0 0x55621ee56b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.403+0000 7f1b277fe640 1 -- 172.21.15.44:0/688377826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55621ee4e1d0 msgr2=0x55621ee4a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:25.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.403+0000 7f1b277fe640 1 --2- 172.21.15.44:0/688377826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55621ee4e1d0 0x55621ee4a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.403+0000 7f1b277fe640 1 -- 172.21.15.44:0/688377826 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55621ee576d0 msgr2=0x55621ee56b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:25.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.403+0000 7f1b277fe640 1 --2- 172.21.15.44:0/688377826 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55621ee576d0 0x55621ee56b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.403+0000 7f1b277fe640 1 -- 172.21.15.44:0/688377826 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55621ee54de0 con 0x55621ee4eaf0 2024-04-21T23:16:25.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.403+0000 7f1b277fe640 1 --2- 172.21.15.44:0/688377826 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4eaf0 0x55621ee53bf0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c009770 tx=0x7f1b1c009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4218b35bbf2ac2a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:25.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.403+0000 7f1b0dffb640 1 -- 172.21.15.44:0/688377826 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c09d820 con 0x55621ee4eaf0 2024-04-21T23:16:25.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.404+0000 7f1b0dffb640 1 -- 172.21.15.44:0/688377826 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b1c0a1b20 con 0x55621ee4eaf0 2024-04-21T23:16:25.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.404+0000 7f1b0dffb640 1 -- 172.21.15.44:0/688377826 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c099e90 con 0x55621ee4eaf0 2024-04-21T23:16:25.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.404+0000 7f1b313ec740 1 -- 172.21.15.44:0/688377826 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4eaf0 msgr2=0x55621ee53bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.404+0000 7f1b313ec740 1 --2- 172.21.15.44:0/688377826 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4eaf0 0x55621ee53bf0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c009770 tx=0x7f1b1c009700 comp rx=0 tx=0).stop 2024-04-21T23:16:25.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.404+0000 7f1b313ec740 1 -- 172.21.15.44:0/688377826 shutdown_connections 2024-04-21T23:16:25.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.404+0000 7f1b313ec740 1 --2- 172.21.15.44:0/688377826 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55621ee576d0 0x55621ee56b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.404+0000 7f1b313ec740 1 --2- 172.21.15.44:0/688377826 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee4eaf0 0x55621ee53bf0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.404+0000 7f1b313ec740 1 --2- 172.21.15.44:0/688377826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55621ee4e1d0 0x55621ee4a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.404+0000 7f1b313ec740 1 -- 172.21.15.44:0/688377826 >> 172.21.15.44:0/688377826 conn(0x55621ee48900 msgr2=0x55621ee630c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:25.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.404+0000 7f1b313ec740 1 -- 172.21.15.44:0/688377826 shutdown_connections 2024-04-21T23:16:25.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.404+0000 7f1b313ec740 1 -- 172.21.15.44:0/688377826 wait complete. 2024-04-21T23:16:25.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.405+0000 7f1b313ec740 1 -- start start 2024-04-21T23:16:25.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.405+0000 7f1b313ec740 1 Processor -- start 2024-04-21T23:16:25.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.405+0000 7f1b313ec740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55621ee4e1d0 0x55621ee6c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.405+0000 7f1b313ec740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55621ee4eaf0 0x55621ee6c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.405+0000 7f1b313ec740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee576d0 0x55621ee697c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.405+0000 7f1b313ec740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1b281ab4f0 con 0x55621ee576d0 2024-04-21T23:16:25.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.405+0000 7f1b313ec740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1b281ac960 con 0x55621ee4e1d0 2024-04-21T23:16:25.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.405+0000 7f1b313ec740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1b281abbc0 con 0x55621ee4eaf0 2024-04-21T23:16:25.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.405+0000 7f1b313ec740 10 client.0 _ll_register_callbacks cb 0x7f1b18009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:25.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.405+0000 7f1b2cd9e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee576d0 0x55621ee697c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.405+0000 7f1b277fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55621ee4eaf0 0x55621ee6c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.405+0000 7f1b2cd9e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee576d0 0x55621ee697c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37032/0 (socket says 172.21.15.44:37032) 2024-04-21T23:16:25.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.405+0000 7f1b2cd9e640 1 -- 172.21.15.44:0/967641751 learned_addr learned my addr 172.21.15.44:0/967641751 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:25.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.405+0000 7f1b27fff640 1 --2- 172.21.15.44:0/967641751 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55621ee4e1d0 0x55621ee6c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.406+0000 7f1b2cd9e640 1 -- 172.21.15.44:0/967641751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55621ee4eaf0 msgr2=0x55621ee6c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:25.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.406+0000 7f1b2cd9e640 1 --2- 172.21.15.44:0/967641751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55621ee4eaf0 0x55621ee6c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.406+0000 7f1b2cd9e640 1 -- 172.21.15.44:0/967641751 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55621ee4e1d0 msgr2=0x55621ee6c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:25.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.406+0000 7f1b2cd9e640 1 --2- 172.21.15.44:0/967641751 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55621ee4e1d0 0x55621ee6c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.406+0000 7f1b2cd9e640 1 -- 172.21.15.44:0/967641751 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55621eb907d0 con 0x55621ee576d0 2024-04-21T23:16:25.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.406+0000 7f1b2cd9e640 1 --2- 172.21.15.44:0/967641751 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee576d0 0x55621ee697c0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f1b2000fb90 tx=0x7f1b20004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=868a1c15904d7443 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:25.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.406+0000 7f1b077fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:16:25.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.406+0000 7f1b077fe640 1 -- 172.21.15.44:0/967641751 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2002b060 con 0x55621ee576d0 2024-04-21T23:16:25.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.406+0000 7f1b077fe640 1 -- 172.21.15.44:0/967641751 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7f1b20026600 con 0x55621ee576d0 2024-04-21T23:16:25.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.406+0000 7f1b313ec740 1 -- 172.21.15.44:0/967641751 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b1c0a1b20 con 0x55621ee576d0 2024-04-21T23:16:25.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.406+0000 7f1b077fe640 1 -- 172.21.15.44:0/967641751 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1b20032050 con 0x55621ee576d0 2024-04-21T23:16:25.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.406+0000 7f1b313ec740 10 client.17364 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:25.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.406+0000 7f1b313ec740 1 -- 172.21.15.44:0/967641751 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55621eecd0a0 con 0x55621ee576d0 2024-04-21T23:16:25.411 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.407+0000 7f1b077fe640 1 -- 172.21.15.44:0/967641751 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b1c0a1b20 con 0x55621ee576d0 2024-04-21T23:16:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.407+0000 7f1b077fe640 1 -- 172.21.15.44:0/967641751 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f1b20018040 con 0x55621ee576d0 2024-04-21T23:16:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.407+0000 7f1b313ec740 10 client.17364 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.407+0000 7f1b313ec740 10 client.17364 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.407+0000 7f1b313ec740 20 client.17364 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.407+0000 7f1b313ec740 1 --2- 172.21.15.44:0/967641751 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55621eecd820 0x55621eecfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.407+0000 7f1b313ec740 4 client.17364 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:16:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.407+0000 7f1b313ec740 1 -- 172.21.15.44:0/967641751 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55621edb9dc0 con 0x55621eecd820 2024-04-21T23:16:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.407+0000 7f1b27fff640 1 --2- 172.21.15.44:0/967641751 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55621eecd820 0x55621eecfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.407+0000 7f1b27fff640 1 --2- 172.21.15.44:0/967641751 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55621eecd820 0x55621eecfc40 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d17603766fe60b19 server_cookie=522598e87c1e8791 in_seq=0 out_seq=0 2024-04-21T23:16:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.407+0000 7f1b077fe640 10 client.17364 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.409+0000 7f1b077fe640 1 -- 172.21.15.44:0/967641751 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55621edb9dc0 con 0x55621eecd820 2024-04-21T23:16:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.409+0000 7f1b077fe640 10 client.17364 handle_command_reply: tid=0 2024-04-21T23:16:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.410+0000 7f1b313ec740 1 client.17364 shutdown 2024-04-21T23:16:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.410+0000 7f1b313ec740 1 -- 172.21.15.44:0/967641751 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee576d0 msgr2=0x55621ee697c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.410+0000 7f1b313ec740 1 --2- 172.21.15.44:0/967641751 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee576d0 0x55621ee697c0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f1b2000fb90 tx=0x7f1b20004ab0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.410+0000 7f1b313ec740 1 -- 172.21.15.44:0/967641751 shutdown_connections 2024-04-21T23:16:25.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.410+0000 7f1b313ec740 1 --2- 172.21.15.44:0/967641751 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55621eecd820 0x55621eecfc40 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.410+0000 7f1b313ec740 1 --2- 172.21.15.44:0/967641751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55621ee4eaf0 0x55621ee6c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.410+0000 7f1b313ec740 1 --2- 172.21.15.44:0/967641751 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55621ee576d0 0x55621ee697c0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.410+0000 7f1b077fe640 0 client.17364 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:25.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.410+0000 7f1b313ec740 1 --2- 172.21.15.44:0/967641751 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55621ee4e1d0 0x55621ee6c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.410+0000 7f1b313ec740 1 -- 172.21.15.44:0/967641751 >> 172.21.15.44:0/967641751 conn(0x55621ee48900 msgr2=0x55621ee4f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:25.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.410+0000 7f1b313ec740 1 -- 172.21.15.44:0/967641751 shutdown_connections 2024-04-21T23:16:25.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.410+0000 7f1b313ec740 1 -- 172.21.15.44:0/967641751 wait complete. 2024-04-21T23:16:25.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.410+0000 7f1b313ec740 20 client.17364 trim_cache size 0 max 16384 2024-04-21T23:16:25.413 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-21T23:16:25.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.410+0000 7f1b2d59f640 1 -- 172.21.15.44:0/3953206082 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1b100a2d40 msgr2=0x7f1b100a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.410+0000 7f1b2d59f640 1 --2- 172.21.15.44:0/3953206082 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1b100a2d40 0x7f1b100a5200 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x55621ee49a90 tx=0x7f1b20009500 comp rx=0 tx=0).stop 2024-04-21T23:16:25.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.410+0000 7f1b2d59f640 1 -- 172.21.15.44:0/3953206082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b28100e00 msgr2=0x7f1b281972b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:25.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.410+0000 7f1b2d59f640 1 --2- 172.21.15.44:0/3953206082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b28100e00 0x7f1b281972b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c010c00 tx=0x7f1b1c00c6a0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.411+0000 7f1b2d59f640 1 -- 172.21.15.44:0/3953206082 shutdown_connections 2024-04-21T23:16:25.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.411+0000 7f1b2d59f640 1 --2- 172.21.15.44:0/3953206082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b281064e0 0x7f1b2819b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.411+0000 7f1b2d59f640 1 --2- 172.21.15.44:0/3953206082 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1b100a2d40 0x7f1b100a5200 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.411+0000 7f1b2d59f640 1 --2- 172.21.15.44:0/3953206082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b28100e00 0x7f1b281972b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.411+0000 7f1b2d59f640 1 --2- 172.21.15.44:0/3953206082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b28100440 0x7f1b28196d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:25.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.411+0000 7f1b2d59f640 1 -- 172.21.15.44:0/3953206082 >> 172.21.15.44:0/3953206082 conn(0x7f1b280fbf50 msgr2=0x7f1b280fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:25.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.411+0000 7f1b2d59f640 1 -- 172.21.15.44:0/3953206082 shutdown_connections 2024-04-21T23:16:25.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:25.411+0000 7f1b2d59f640 1 -- 172.21.15.44:0/3953206082 wait complete. 2024-04-21T23:16:25.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 874809 11.95 MB/sec execute 255 sec latency 22.811 ms 2024-04-21T23:16:26.423 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:16:26.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:16:26.424 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:16:26.463 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:26 smithi055 ceph-mon[31117]: pgmap v239: 129 pgs: 129 active+clean; 64 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 418 op/s 2024-04-21T23:16:26.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:26 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1376664452' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:16:26.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:26 smithi069 ceph-mon[31816]: pgmap v239: 129 pgs: 129 active+clean; 64 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 418 op/s 2024-04-21T23:16:26.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:26 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1376664452' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:16:26.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.539+0000 7fe9f21bf640 1 -- 172.21.15.44:0/478512715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9ec10bd10 msgr2=0x7fe9ec10e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:26.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.539+0000 7fe9f21bf640 1 --2- 172.21.15.44:0/478512715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9ec10bd10 0x7fe9ec10e190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4009be0 tx=0x7fe9e402bf80 comp rx=0 tx=0).stop 2024-04-21T23:16:26.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.539+0000 7fe9f21bf640 1 -- 172.21.15.44:0/478512715 shutdown_connections 2024-04-21T23:16:26.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.539+0000 7fe9f21bf640 1 --2- 172.21.15.44:0/478512715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9ec10bd10 0x7fe9ec10e190 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.539+0000 7fe9f21bf640 1 --2- 172.21.15.44:0/478512715 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9ec1032f0 0x7fe9ec10b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.539+0000 7fe9f21bf640 1 --2- 172.21.15.44:0/478512715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9ec1029d0 0x7fe9ec102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.539+0000 7fe9f21bf640 1 -- 172.21.15.44:0/478512715 >> 172.21.15.44:0/478512715 conn(0x7fe9ec0777f0 msgr2=0x7fe9ec077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:26.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.539+0000 7fe9f21bf640 1 -- 172.21.15.44:0/478512715 shutdown_connections 2024-04-21T23:16:26.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.539+0000 7fe9f21bf640 1 -- 172.21.15.44:0/478512715 wait complete. 2024-04-21T23:16:26.539 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.539+0000 7fe9f21bf640 1 Processor -- start 2024-04-21T23:16:26.541 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.540+0000 7fe9f21bf640 1 -- start start 2024-04-21T23:16:26.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.540+0000 7fe9f21bf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9ec1029d0 0x7fe9ec100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.540+0000 7fe9f21bf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9ec1032f0 0x7fe9ec100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.540+0000 7fe9f21bf640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9ec10bd10 0x7fe9ec1010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.540+0000 7fe9f21bf640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec111b40 con 0x7fe9ec1029d0 2024-04-21T23:16:26.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.540+0000 7fe9f21bf640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec1119c0 con 0x7fe9ec10bd10 2024-04-21T23:16:26.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.540+0000 7fe9f21bf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec111cc0 con 0x7fe9ec1032f0 2024-04-21T23:16:26.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.540+0000 7fe9eb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9ec1029d0 0x7fe9ec100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.540+0000 7fe9eb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9ec1029d0 0x7fe9ec100620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37044/0 (socket says 172.21.15.44:37044) 2024-04-21T23:16:26.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.540+0000 7fe9eb7fe640 1 -- 172.21.15.44:0/4037992028 learned_addr learned my addr 172.21.15.44:0/4037992028 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:26.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.540+0000 7fe9eaffd640 1 --2- 172.21.15.44:0/4037992028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9ec1032f0 0x7fe9ec100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.540+0000 7fe9ebfff640 1 --2- 172.21.15.44:0/4037992028 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9ec10bd10 0x7fe9ec1010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.540+0000 7fe9eb7fe640 1 -- 172.21.15.44:0/4037992028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9ec1032f0 msgr2=0x7fe9ec100b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:26.542 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.540+0000 7fe9eb7fe640 1 --2- 172.21.15.44:0/4037992028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9ec1032f0 0x7fe9ec100b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.540+0000 7fe9eb7fe640 1 -- 172.21.15.44:0/4037992028 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9ec10bd10 msgr2=0x7fe9ec1010a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:26.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.540+0000 7fe9eb7fe640 1 --2- 172.21.15.44:0/4037992028 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9ec10bd10 0x7fe9ec1010a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.540+0000 7fe9eb7fe640 1 -- 172.21.15.44:0/4037992028 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9ec1a64c0 con 0x7fe9ec1029d0 2024-04-21T23:16:26.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.540+0000 7fe9eb7fe640 1 --2- 172.21.15.44:0/4037992028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9ec1029d0 0x7fe9ec100620 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc00e9c0 tx=0x7fe9dc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:26.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.541+0000 7fe9e8ff9640 1 -- 172.21.15.44:0/4037992028 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc00cde0 con 0x7fe9ec1029d0 2024-04-21T23:16:26.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.541+0000 7fe9f21bf640 1 -- 172.21.15.44:0/4037992028 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9ec1a67b0 con 0x7fe9ec1029d0 2024-04-21T23:16:26.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.541+0000 7fe9e8ff9640 1 -- 172.21.15.44:0/4037992028 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9dc0035f0 con 0x7fe9ec1029d0 2024-04-21T23:16:26.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.541+0000 7fe9e8ff9640 1 -- 172.21.15.44:0/4037992028 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc010650 con 0x7fe9ec1029d0 2024-04-21T23:16:26.543 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.541+0000 7fe9f21bf640 1 -- 172.21.15.44:0/4037992028 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9ec1a6d50 con 0x7fe9ec1029d0 2024-04-21T23:16:26.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.543+0000 7fe9e8ff9640 1 -- 172.21.15.44:0/4037992028 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe9dc0037a0 con 0x7fe9ec1029d0 2024-04-21T23:16:26.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.543+0000 7fe9f4da4740 1 Processor -- start 2024-04-21T23:16:26.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.543+0000 7fe9f4da4740 1 -- start start 2024-04-21T23:16:26.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9f4da4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0b3720 0x55f59b0b2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9f4da4740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b2a40 0x55f59b0b2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9f4da4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0bbdb0 0x55f59b0be1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9f4da4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f59b016170 con 0x55f59b0bbdb0 2024-04-21T23:16:26.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9f4da4740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55f59afbe7c0 con 0x55f59b0b2a40 2024-04-21T23:16:26.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9f4da4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f59b012130 con 0x55f59b0b3720 2024-04-21T23:16:26.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9eaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0bbdb0 0x55f59b0be1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9ebfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0b3720 0x55f59b0b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9ebfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0b3720 0x55f59b0b2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:58324/0 (socket says 172.21.15.44:58324) 2024-04-21T23:16:26.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9eb7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b2a40 0x55f59b0b2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9eaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0bbdb0 0x55f59b0be1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37056/0 (socket says 172.21.15.44:37056) 2024-04-21T23:16:26.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9ebfff640 1 -- 172.21.15.44:0/1455315124 learned_addr learned my addr 172.21.15.44:0/1455315124 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:26.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9ebfff640 1 -- 172.21.15.44:0/1455315124 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b2a40 msgr2=0x55f59b0b2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:26.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9e8ff9640 1 --2- 172.21.15.44:0/4037992028 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe9d40a2d00 0x7fe9d40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9ebfff640 1 --2- 172.21.15.44:0/1455315124 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b2a40 0x55f59b0b2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9ebfff640 1 -- 172.21.15.44:0/1455315124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0bbdb0 msgr2=0x55f59b0be1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:26.551 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9ebfff640 1 --2- 172.21.15.44:0/1455315124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0bbdb0 0x55f59b0be1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9ebfff640 1 -- 172.21.15.44:0/1455315124 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f59afcfe60 con 0x55f59b0b3720 2024-04-21T23:16:26.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9eb7fe640 1 --2- 172.21.15.44:0/1455315124 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b2a40 0x55f59b0b2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:26.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9e8ff9640 1 -- 172.21.15.44:0/4037992028 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7fe9dc0cfb30 con 0x7fe9ec1029d0 2024-04-21T23:16:26.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9ebfff640 1 --2- 172.21.15.44:0/4037992028 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe9d40a2d00 0x7fe9d40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.544+0000 7fe9ebfff640 1 --2- 172.21.15.44:0/1455315124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0b3720 0x55f59b0b2470 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4009be0 tx=0x7fe9e4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=157951ffca080bae server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:26.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.545+0000 7fe9d1ffb640 1 -- 172.21.15.44:0/1455315124 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4043040 con 0x55f59b0b3720 2024-04-21T23:16:26.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.545+0000 7fe9d1ffb640 1 -- 172.21.15.44:0/1455315124 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9e402cdf0 con 0x55f59b0b3720 2024-04-21T23:16:26.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.545+0000 7fe9d1ffb640 1 -- 172.21.15.44:0/1455315124 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4033520 con 0x55f59b0b3720 2024-04-21T23:16:26.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.545+0000 7fe9f4da4740 1 -- 172.21.15.44:0/1455315124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0b3720 msgr2=0x55f59b0b2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:26.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.545+0000 7fe9ebfff640 1 --2- 172.21.15.44:0/4037992028 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe9d40a2d00 0x7fe9d40a51c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe9e404b000 tx=0x7fe9e404e000 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:26.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.545+0000 7fe9f4da4740 1 --2- 172.21.15.44:0/1455315124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0b3720 0x55f59b0b2470 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4009be0 tx=0x7fe9e4009840 comp rx=0 tx=0).stop 2024-04-21T23:16:26.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.545+0000 7fe9f4da4740 1 -- 172.21.15.44:0/1455315124 shutdown_connections 2024-04-21T23:16:26.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.545+0000 7fe9f4da4740 1 --2- 172.21.15.44:0/1455315124 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0bbdb0 0x55f59b0be1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.545+0000 7fe9f4da4740 1 --2- 172.21.15.44:0/1455315124 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b2a40 0x55f59b0b2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.552 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.545+0000 7fe9f4da4740 1 --2- 172.21.15.44:0/1455315124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0b3720 0x55f59b0b2470 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.545+0000 7fe9f4da4740 1 -- 172.21.15.44:0/1455315124 >> 172.21.15.44:0/1455315124 conn(0x55f59b0794f0 msgr2=0x55f59b0ad490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:26.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.545+0000 7fe9f4da4740 1 -- 172.21.15.44:0/1455315124 shutdown_connections 2024-04-21T23:16:26.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.546+0000 7fe9f4da4740 1 -- 172.21.15.44:0/1455315124 wait complete. 2024-04-21T23:16:26.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.546+0000 7fe9f4da4740 1 -- start start 2024-04-21T23:16:26.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.546+0000 7fe9f4da4740 1 Processor -- start 2024-04-21T23:16:26.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.546+0000 7fe9f4da4740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b2a40 0x55f59b0b8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.546+0000 7fe9f4da4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0b3720 0x55f59b0b92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.546+0000 7fe9f4da4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0bbdb0 0x55f59b0b9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.546+0000 7fe9f4da4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f59b0bec20 con 0x55f59b0b3720 2024-04-21T23:16:26.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.546+0000 7fe9f4da4740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55f59afd1d40 con 0x55f59b0b2a40 2024-04-21T23:16:26.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.546+0000 7fe9f4da4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f59b0beda0 con 0x55f59b0bbdb0 2024-04-21T23:16:26.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.546+0000 7fe9f4da4740 10 client.0 _ll_register_callbacks cb 0x55f59b00d820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:26.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.546+0000 7fe9eaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0b3720 0x55f59b0b92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.546+0000 7fe9ebfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0bbdb0 0x55f59b0b9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.553 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.546+0000 7fe9eaffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0b3720 0x55f59b0b92c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37068/0 (socket says 172.21.15.44:37068) 2024-04-21T23:16:26.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.546+0000 7fe9eb7fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b2a40 0x55f59b0b8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.546+0000 7fe9eaffd640 1 -- 172.21.15.44:0/3552182971 learned_addr learned my addr 172.21.15.44:0/3552182971 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:26.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.547+0000 7fe9ebfff640 1 -- 172.21.15.44:0/3552182971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b2a40 msgr2=0x55f59b0b8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:26.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.547+0000 7fe9ebfff640 1 --2- 172.21.15.44:0/3552182971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b2a40 0x55f59b0b8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.547+0000 7fe9ebfff640 1 -- 172.21.15.44:0/3552182971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0b3720 msgr2=0x55f59b0b92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:26.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.547+0000 7fe9ebfff640 1 --2- 172.21.15.44:0/3552182971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0b3720 0x55f59b0b92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.547+0000 7fe9ebfff640 1 -- 172.21.15.44:0/3552182971 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f59adf27e0 con 0x55f59b0bbdb0 2024-04-21T23:16:26.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.547+0000 7fe9eaffd640 1 --2- 172.21.15.44:0/3552182971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0b3720 0x55f59b0b92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:26.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.547+0000 7fe9ebfff640 1 --2- 172.21.15.44:0/3552182971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0bbdb0 0x55f59b0b9800 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4009be0 tx=0x7fe9e4004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15532db73a707512 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:26.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.547+0000 7fe9caffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:16:26.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.547+0000 7fe9caffd640 1 -- 172.21.15.44:0/3552182971 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4003c50 con 0x55f59b0bbdb0 2024-04-21T23:16:26.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.547+0000 7fe9f4da4740 1 -- 172.21.15.44:0/3552182971 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9e402cdf0 con 0x55f59b0bbdb0 2024-04-21T23:16:26.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.547+0000 7fe9caffd640 1 -- 172.21.15.44:0/3552182971 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7fe9e40041e0 con 0x55f59b0bbdb0 2024-04-21T23:16:26.554 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.547+0000 7fe9caffd640 1 -- 172.21.15.44:0/3552182971 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe9e4065050 con 0x55f59b0bbdb0 2024-04-21T23:16:26.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.547+0000 7fe9f4da4740 10 client.26908 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:26.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.547+0000 7fe9f4da4740 1 -- 172.21.15.44:0/3552182971 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f59ae52f70 con 0x55f59b0bbdb0 2024-04-21T23:16:26.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.547+0000 7fe9caffd640 1 -- 172.21.15.44:0/3552182971 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9e402cdf0 con 0x55f59b0bbdb0 2024-04-21T23:16:26.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.548+0000 7fe9caffd640 1 -- 172.21.15.44:0/3552182971 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7fe9e405c590 con 0x55f59b0bbdb0 2024-04-21T23:16:26.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.548+0000 7fe9f4da4740 10 client.26908 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:26.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.548+0000 7fe9f4da4740 10 client.26908 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:26.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.548+0000 7fe9f4da4740 20 client.26908 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:26.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.548+0000 7fe9f4da4740 1 --2- 172.21.15.44:0/3552182971 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f59b0cbd20 0x55f59b0ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.548+0000 7fe9f4da4740 4 client.26908 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:16:26.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.548+0000 7fe9f4da4740 1 -- 172.21.15.44:0/3552182971 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f59afea450 con 0x55f59b0cbd20 2024-04-21T23:16:26.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.548+0000 7fe9eb7fe640 1 --2- 172.21.15.44:0/3552182971 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f59b0cbd20 0x55f59b0ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.548+0000 7fe9eb7fe640 1 --2- 172.21.15.44:0/3552182971 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f59b0cbd20 0x55f59b0ce110 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac80ae2fb657665d server_cookie=a24fc0203a609cee in_seq=0 out_seq=0 2024-04-21T23:16:26.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.548+0000 7fe9caffd640 10 client.26908 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:26.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.549+0000 7fe9caffd640 1 -- 172.21.15.44:0/3552182971 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f59afea450 con 0x55f59b0cbd20 2024-04-21T23:16:26.555 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.549+0000 7fe9caffd640 10 client.26908 handle_command_reply: tid=0 2024-04-21T23:16:26.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.549+0000 7fe9f4da4740 1 client.26908 shutdown 2024-04-21T23:16:26.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.549+0000 7fe9f4da4740 1 -- 172.21.15.44:0/3552182971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0bbdb0 msgr2=0x55f59b0b9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:26.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.549+0000 7fe9f4da4740 1 --2- 172.21.15.44:0/3552182971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0bbdb0 0x55f59b0b9800 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4009be0 tx=0x7fe9e4004770 comp rx=0 tx=0).stop 2024-04-21T23:16:26.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.549+0000 7fe9f4da4740 1 -- 172.21.15.44:0/3552182971 shutdown_connections 2024-04-21T23:16:26.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.549+0000 7fe9f4da4740 1 --2- 172.21.15.44:0/3552182971 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f59b0cbd20 0x55f59b0ce110 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.549+0000 7fe9f4da4740 1 --2- 172.21.15.44:0/3552182971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0bbdb0 0x55f59b0b9800 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.549+0000 7fe9f4da4740 1 --2- 172.21.15.44:0/3552182971 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0b3720 0x55f59b0b92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.549+0000 7fe9caffd640 0 client.26908 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:26.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.549+0000 7fe9f4da4740 1 --2- 172.21.15.44:0/3552182971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b2a40 0x55f59b0b8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.549+0000 7fe9f4da4740 1 -- 172.21.15.44:0/3552182971 >> 172.21.15.44:0/3552182971 conn(0x55f59b0794f0 msgr2=0x55f59b0af1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:26.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.550+0000 7fe9f4da4740 1 -- 172.21.15.44:0/3552182971 shutdown_connections 2024-04-21T23:16:26.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.550+0000 7fe9f4da4740 1 -- 172.21.15.44:0/3552182971 wait complete. 2024-04-21T23:16:26.556 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.550+0000 7fe9f4da4740 20 client.26908 trim_cache size 0 max 16384 2024-04-21T23:16:26.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.568+0000 7fe9f4da4740 1 Processor -- start 2024-04-21T23:16:26.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.568+0000 7fe9f4da4740 1 -- start start 2024-04-21T23:16:26.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.568+0000 7fe9f4da4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0b2a40 0x55f59b0c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.569+0000 7fe9f4da4740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b3720 0x55f59b0b9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.569+0000 7fe9f4da4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0bbdb0 0x55f59b0d3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.569+0000 7fe9f4da4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec1aee80 con 0x55f59b0bbdb0 2024-04-21T23:16:26.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.569+0000 7fe9f4da4740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec1b1700 con 0x55f59b0b3720 2024-04-21T23:16:26.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.569+0000 7fe9f4da4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec1af4f0 con 0x55f59b0b2a40 2024-04-21T23:16:26.569 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.569+0000 7fe9ebfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0bbdb0 0x55f59b0d3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.569+0000 7fe9eaffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b3720 0x55f59b0b9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.569+0000 7fe9eb7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0b2a40 0x55f59b0c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.569+0000 7fe9eaffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b3720 0x55f59b0b9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:58740/0 (socket says 172.21.15.44:58740) 2024-04-21T23:16:26.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.569+0000 7fe9eaffd640 1 -- 172.21.15.44:0/4217841576 learned_addr learned my addr 172.21.15.44:0/4217841576 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:26.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.569+0000 7fe9eb7fe640 1 -- 172.21.15.44:0/4217841576 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b3720 msgr2=0x55f59b0b9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:26.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.569+0000 7fe9eb7fe640 1 --2- 172.21.15.44:0/4217841576 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b3720 0x55f59b0b9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.569+0000 7fe9eb7fe640 1 -- 172.21.15.44:0/4217841576 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0bbdb0 msgr2=0x55f59b0d3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:26.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.569+0000 7fe9eb7fe640 1 --2- 172.21.15.44:0/4217841576 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0bbdb0 0x55f59b0d3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.569+0000 7fe9eb7fe640 1 -- 172.21.15.44:0/4217841576 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f59acea170 con 0x55f59b0b2a40 2024-04-21T23:16:26.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.569+0000 7fe9ebfff640 1 --2- 172.21.15.44:0/4217841576 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0bbdb0 0x55f59b0d3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:26.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.569+0000 7fe9eb7fe640 1 --2- 172.21.15.44:0/4217841576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0b2a40 0x55f59b0c7b20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc0ccee0 tx=0x7fe9dc0d2040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f119b0dcd6f20054 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:26.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.569+0000 7fe9d17fa640 1 -- 172.21.15.44:0/4217841576 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc09b610 con 0x55f59b0b2a40 2024-04-21T23:16:26.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.569+0000 7fe9d17fa640 1 -- 172.21.15.44:0/4217841576 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9dc09c3b0 con 0x55f59b0b2a40 2024-04-21T23:16:26.570 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.569+0000 7fe9d17fa640 1 -- 172.21.15.44:0/4217841576 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc0cd160 con 0x55f59b0b2a40 2024-04-21T23:16:26.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.570+0000 7fe9f4da4740 1 -- 172.21.15.44:0/4217841576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0b2a40 msgr2=0x55f59b0c7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:26.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.570+0000 7fe9f4da4740 1 --2- 172.21.15.44:0/4217841576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0b2a40 0x55f59b0c7b20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc0ccee0 tx=0x7fe9dc0d2040 comp rx=0 tx=0).stop 2024-04-21T23:16:26.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.570+0000 7fe9f4da4740 1 -- 172.21.15.44:0/4217841576 shutdown_connections 2024-04-21T23:16:26.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.570+0000 7fe9f4da4740 1 --2- 172.21.15.44:0/4217841576 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0bbdb0 0x55f59b0d3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.570+0000 7fe9f4da4740 1 --2- 172.21.15.44:0/4217841576 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b3720 0x55f59b0b9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.570+0000 7fe9f4da4740 1 --2- 172.21.15.44:0/4217841576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0b2a40 0x55f59b0c7b20 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.570+0000 7fe9f4da4740 1 -- 172.21.15.44:0/4217841576 >> 172.21.15.44:0/4217841576 conn(0x55f59b0794f0 msgr2=0x55f59b0ac4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:26.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.570+0000 7fe9f4da4740 1 -- 172.21.15.44:0/4217841576 shutdown_connections 2024-04-21T23:16:26.574 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.570+0000 7fe9f4da4740 1 -- 172.21.15.44:0/4217841576 wait complete. 2024-04-21T23:16:26.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.570+0000 7fe9f4da4740 1 -- start start 2024-04-21T23:16:26.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.571+0000 7fe9f4da4740 1 Processor -- start 2024-04-21T23:16:26.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.571+0000 7fe9f4da4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0b2a40 0x55f59b0c4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.571+0000 7fe9f4da4740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b3720 0x55f59b0c4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.571+0000 7fe9f4da4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0bbdb0 0x55f59b0c1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.571+0000 7fe9f4da4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec1afbc0 con 0x55f59b0b2a40 2024-04-21T23:16:26.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.571+0000 7fe9f4da4740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec1b1030 con 0x55f59b0b3720 2024-04-21T23:16:26.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.571+0000 7fe9f4da4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec1b0290 con 0x55f59b0bbdb0 2024-04-21T23:16:26.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.571+0000 7fe9f4da4740 10 client.0 _ll_register_callbacks cb 0x7fe9e0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:26.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.571+0000 7fe9ebfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0bbdb0 0x55f59b0c1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.571+0000 7fe9eb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0b2a40 0x55f59b0c4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.571+0000 7fe9eaffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b3720 0x55f59b0c4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.571+0000 7fe9ebfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0bbdb0 0x55f59b0c1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:58348/0 (socket says 172.21.15.44:58348) 2024-04-21T23:16:26.575 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.571+0000 7fe9eb7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0b2a40 0x55f59b0c4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37086/0 (socket says 172.21.15.44:37086) 2024-04-21T23:16:26.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.571+0000 7fe9ebfff640 1 -- 172.21.15.44:0/3627976658 learned_addr learned my addr 172.21.15.44:0/3627976658 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:26.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.572+0000 7fe9eaffd640 1 -- 172.21.15.44:0/3627976658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0bbdb0 msgr2=0x55f59b0c1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:26.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.572+0000 7fe9eaffd640 1 --2- 172.21.15.44:0/3627976658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0bbdb0 0x55f59b0c1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.572+0000 7fe9eaffd640 1 -- 172.21.15.44:0/3627976658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0b2a40 msgr2=0x55f59b0c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:26.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.572+0000 7fe9eaffd640 1 --2- 172.21.15.44:0/3627976658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0b2a40 0x55f59b0c4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.572+0000 7fe9eaffd640 1 -- 172.21.15.44:0/3627976658 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f59b0c1990 con 0x55f59b0b3720 2024-04-21T23:16:26.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.572+0000 7fe9ebfff640 1 --2- 172.21.15.44:0/3627976658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0bbdb0 0x55f59b0c1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:26.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.572+0000 7fe9eaffd640 1 --2- 172.21.15.44:0/3627976658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b3720 0x55f59b0c4570 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55f59b0ae6a0 tx=0x7fe9e000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0becda61a671ca7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:26.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.572+0000 7fe9caffd640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:16:26.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.572+0000 7fe9caffd640 1 -- 172.21.15.44:0/3627976658 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0023060 con 0x55f59b0b3720 2024-04-21T23:16:26.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.572+0000 7fe9caffd640 1 -- 172.21.15.44:0/3627976658 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7fe9e0004030 con 0x55f59b0b3720 2024-04-21T23:16:26.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.572+0000 7fe9f4da4740 1 -- 172.21.15.44:0/3627976658 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9dc09c3b0 con 0x55f59b0b3720 2024-04-21T23:16:26.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.572+0000 7fe9caffd640 1 -- 172.21.15.44:0/3627976658 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe9e002a050 con 0x55f59b0b3720 2024-04-21T23:16:26.576 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.572+0000 7fe9f4da4740 10 client.26948 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:26.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.572+0000 7fe9f4da4740 1 -- 172.21.15.44:0/3627976658 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f59b132c30 con 0x55f59b0b3720 2024-04-21T23:16:26.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.572+0000 7fe9caffd640 1 -- 172.21.15.44:0/3627976658 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9dc09c3b0 con 0x55f59b0b3720 2024-04-21T23:16:26.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.573+0000 7fe9caffd640 1 -- 172.21.15.44:0/3627976658 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7fe9e0010430 con 0x55f59b0b3720 2024-04-21T23:16:26.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.573+0000 7fe9f4da4740 10 client.26948 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:26.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.573+0000 7fe9f4da4740 10 client.26948 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:26.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.573+0000 7fe9f4da4740 20 client.26948 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:26.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.573+0000 7fe9f4da4740 1 --2- 172.21.15.44:0/3627976658 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f59b1333b0 0x55f59b1357d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.573+0000 7fe9f4da4740 4 client.26948 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:16:26.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.573+0000 7fe9f4da4740 1 -- 172.21.15.44:0/3627976658 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f59b0236b0 con 0x55f59b1333b0 2024-04-21T23:16:26.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.573+0000 7fe9eb7fe640 1 --2- 172.21.15.44:0/3627976658 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f59b1333b0 0x55f59b1357d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.573+0000 7fe9caffd640 10 client.26948 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:26.577 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.573+0000 7fe9eb7fe640 1 --2- 172.21.15.44:0/3627976658 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f59b1333b0 0x55f59b1357d0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ccc99f005ebb529 server_cookie=7f12c7cf57aebb12 in_seq=0 out_seq=0 2024-04-21T23:16:26.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.575+0000 7fe9caffd640 1 -- 172.21.15.44:0/3627976658 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f59b0236b0 con 0x55f59b1333b0 2024-04-21T23:16:26.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.575+0000 7fe9caffd640 10 client.26948 handle_command_reply: tid=0 2024-04-21T23:16:26.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.575+0000 7fe9f4da4740 1 client.26948 shutdown 2024-04-21T23:16:26.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.575+0000 7fe9f4da4740 1 -- 172.21.15.44:0/3627976658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b3720 msgr2=0x55f59b0c4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:26.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.575+0000 7fe9f4da4740 1 --2- 172.21.15.44:0/3627976658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b3720 0x55f59b0c4570 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55f59b0ae6a0 tx=0x7fe9e000f790 comp rx=0 tx=0).stop 2024-04-21T23:16:26.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.575+0000 7fe9f4da4740 1 -- 172.21.15.44:0/3627976658 shutdown_connections 2024-04-21T23:16:26.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.575+0000 7fe9f4da4740 1 --2- 172.21.15.44:0/3627976658 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f59b1333b0 0x55f59b1357d0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.575+0000 7fe9f4da4740 1 --2- 172.21.15.44:0/3627976658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f59b0bbdb0 0x55f59b0c1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.575+0000 7fe9f4da4740 1 --2- 172.21.15.44:0/3627976658 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f59b0b3720 0x55f59b0c4570 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.575+0000 7fe9f4da4740 1 --2- 172.21.15.44:0/3627976658 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f59b0b2a40 0x55f59b0c4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.575+0000 7fe9caffd640 0 client.26948 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:26.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.576+0000 7fe9f4da4740 1 -- 172.21.15.44:0/3627976658 >> 172.21.15.44:0/3627976658 conn(0x55f59b0794f0 msgr2=0x55f59b0ae780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:26.579 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.576+0000 7fe9f4da4740 1 -- 172.21.15.44:0/3627976658 shutdown_connections 2024-04-21T23:16:26.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.576+0000 7fe9f4da4740 1 -- 172.21.15.44:0/3627976658 wait complete. 2024-04-21T23:16:26.580 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.576+0000 7fe9f4da4740 20 client.26948 trim_cache size 0 max 16384 2024-04-21T23:16:26.580 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:16:26.580 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-21T23:16:26.580 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "034a06f5-da71-4918-9930-6f2ff5d081b3", 2024-04-21T23:16:26.580 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-21T23:16:26.580 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:16:26.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.576+0000 7fe9f21bf640 1 -- 172.21.15.44:0/4037992028 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe9d40a2d00 msgr2=0x7fe9d40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:26.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.576+0000 7fe9f21bf640 1 --2- 172.21.15.44:0/4037992028 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe9d40a2d00 0x7fe9d40a51c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe9e404b000 tx=0x7fe9e404e000 comp rx=0 tx=0).stop 2024-04-21T23:16:26.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.576+0000 7fe9f21bf640 1 -- 172.21.15.44:0/4037992028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9ec1029d0 msgr2=0x7fe9ec100620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:26.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.576+0000 7fe9f21bf640 1 --2- 172.21.15.44:0/4037992028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9ec1029d0 0x7fe9ec100620 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc00e9c0 tx=0x7fe9dc00ee80 comp rx=0 tx=0).stop 2024-04-21T23:16:26.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.576+0000 7fe9f21bf640 1 -- 172.21.15.44:0/4037992028 shutdown_connections 2024-04-21T23:16:26.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.576+0000 7fe9f21bf640 1 --2- 172.21.15.44:0/4037992028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9ec1032f0 0x7fe9ec100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.576+0000 7fe9f21bf640 1 --2- 172.21.15.44:0/4037992028 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fe9d40a2d00 0x7fe9d40a51c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.576+0000 7fe9f21bf640 1 --2- 172.21.15.44:0/4037992028 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9ec10bd10 0x7fe9ec1010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.581 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.576+0000 7fe9f21bf640 1 --2- 172.21.15.44:0/4037992028 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fe9ec1029d0 0x7fe9ec100620 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.576+0000 7fe9f21bf640 1 -- 172.21.15.44:0/4037992028 >> 172.21.15.44:0/4037992028 conn(0x7fe9ec0777f0 msgr2=0x7fe9ec0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:26.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.577+0000 7fe9f21bf640 1 -- 172.21.15.44:0/4037992028 shutdown_connections 2024-04-21T23:16:26.582 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.577+0000 7fe9f21bf640 1 -- 172.21.15.44:0/4037992028 wait complete. 2024-04-21T23:16:26.586 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:26 smithi044 ceph-mon[27934]: pgmap v239: 129 pgs: 129 active+clean; 64 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 418 op/s 2024-04-21T23:16:26.586 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:26 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1376664452' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:16:26.587 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:16:26.588 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:16:26.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.704+0000 7fde7c496640 1 -- 172.21.15.44:0/2461238016 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde74101710 msgr2=0x7fde74101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:26.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.704+0000 7fde7c496640 1 --2- 172.21.15.44:0/2461238016 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde74101710 0x7fde74101b90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fde64009860 tx=0x7fde6402bb70 comp rx=0 tx=0).stop 2024-04-21T23:16:26.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.704+0000 7fde7c496640 1 -- 172.21.15.44:0/2461238016 shutdown_connections 2024-04-21T23:16:26.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.704+0000 7fde7c496640 1 --2- 172.21.15.44:0/2461238016 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde741020d0 0x7fde7410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.704+0000 7fde7c496640 1 --2- 172.21.15.44:0/2461238016 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde74101710 0x7fde74101b90 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.704+0000 7fde7c496640 1 --2- 172.21.15.44:0/2461238016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde74107720 0x7fde74107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.704+0000 7fde7c496640 1 -- 172.21.15.44:0/2461238016 >> 172.21.15.44:0/2461238016 conn(0x7fde740fd200 msgr2=0x7fde740ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:26.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.704+0000 7fde7c496640 1 -- 172.21.15.44:0/2461238016 shutdown_connections 2024-04-21T23:16:26.704 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.704+0000 7fde7c496640 1 -- 172.21.15.44:0/2461238016 wait complete. 2024-04-21T23:16:26.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.705+0000 7fde7c496640 1 Processor -- start 2024-04-21T23:16:26.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.705+0000 7fde7c496640 1 -- start start 2024-04-21T23:16:26.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.705+0000 7fde7c496640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde74101710 0x7fde741a3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.705+0000 7fde7c496640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde741020d0 0x7fde741a3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.706+0000 7fde7a232640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde74101710 0x7fde741a3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.706+0000 7fde79a31640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde741020d0 0x7fde741a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.706+0000 7fde79a31640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde741020d0 0x7fde741a3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37100/0 (socket says 172.21.15.44:37100) 2024-04-21T23:16:26.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.706+0000 7fde79a31640 1 -- 172.21.15.44:0/2439520923 learned_addr learned my addr 172.21.15.44:0/2439520923 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:26.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.706+0000 7fde7c496640 1 --2- 172.21.15.44:0/2439520923 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde74107720 0x7fde7410d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.706+0000 7fde7c496640 1 -- 172.21.15.44:0/2439520923 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fde74111af0 con 0x7fde741020d0 2024-04-21T23:16:26.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.706+0000 7fde7c496640 1 -- 172.21.15.44:0/2439520923 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fde74111970 con 0x7fde74107720 2024-04-21T23:16:26.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.706+0000 7fde7c496640 1 -- 172.21.15.44:0/2439520923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fde74111c70 con 0x7fde74101710 2024-04-21T23:16:26.707 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.706+0000 7fde7aa33640 1 --2- 172.21.15.44:0/2439520923 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde74107720 0x7fde7410d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.706+0000 7fde7a232640 1 -- 172.21.15.44:0/2439520923 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde74107720 msgr2=0x7fde7410d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:26.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.706+0000 7fde7a232640 1 --2- 172.21.15.44:0/2439520923 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde74107720 0x7fde7410d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.707+0000 7fde7a232640 1 -- 172.21.15.44:0/2439520923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde741020d0 msgr2=0x7fde741a3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:26.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.707+0000 7fde7a232640 1 --2- 172.21.15.44:0/2439520923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde741020d0 0x7fde741a3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.707+0000 7fde7a232640 1 -- 172.21.15.44:0/2439520923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde7410e070 con 0x7fde74101710 2024-04-21T23:16:26.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.707+0000 7fde79a31640 1 --2- 172.21.15.44:0/2439520923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde741020d0 0x7fde741a3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:16:26.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.707+0000 7fde7aa33640 1 --2- 172.21.15.44:0/2439520923 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde74107720 0x7fde7410d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:26.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.707+0000 7fde7a232640 1 --2- 172.21.15.44:0/2439520923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde74101710 0x7fde741a3220 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fde7000e9f0 tx=0x7fde7000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:26.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.707+0000 7fde6b7fe640 1 -- 172.21.15.44:0/2439520923 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7000cd80 con 0x7fde74101710 2024-04-21T23:16:26.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.707+0000 7fde6b7fe640 1 -- 172.21.15.44:0/2439520923 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde700035f0 con 0x7fde74101710 2024-04-21T23:16:26.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.707+0000 7fde6b7fe640 1 -- 172.21.15.44:0/2439520923 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde70010690 con 0x7fde74101710 2024-04-21T23:16:26.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.707+0000 7fde7c496640 1 -- 172.21.15.44:0/2439520923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde7410e360 con 0x7fde74101710 2024-04-21T23:16:26.708 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.707+0000 7fde7c496640 1 -- 172.21.15.44:0/2439520923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde7410e680 con 0x7fde74101710 2024-04-21T23:16:26.709 INFO:tasks.workunit.client.0.smithi044.stdout: 1 877239 11.96 MB/sec execute 256 sec latency 23.827 ms 2024-04-21T23:16:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.710+0000 7fde6b7fe640 1 -- 172.21.15.44:0/2439520923 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fde70003b30 con 0x7fde74101710 2024-04-21T23:16:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.710+0000 7fde7f084740 1 Processor -- start 2024-04-21T23:16:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.710+0000 7fde7f084740 1 -- start start 2024-04-21T23:16:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.710+0000 7fde7f084740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f55f30 0x55b570f56310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.711+0000 7fde7f084740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f56850 0x55b570f5eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.711+0000 7fde7f084740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f5f430 0x55b570f61820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.712 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.711+0000 7fde7f084740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b570eba560 con 0x55b570f5f430 2024-04-21T23:16:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.711+0000 7fde7f084740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55b570e62730 con 0x55b570f56850 2024-04-21T23:16:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.711+0000 7fde7f084740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b570eb6520 con 0x55b570f55f30 2024-04-21T23:16:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.711+0000 7fde7a232640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f56850 0x55b570f5eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.711+0000 7fde79a31640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f5f430 0x55b570f61820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.711+0000 7fde7aa33640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f55f30 0x55b570f56310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.711+0000 7fde7a232640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f56850 0x55b570f5eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:58758/0 (socket says 172.21.15.44:58758) 2024-04-21T23:16:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.711+0000 7fde7a232640 1 -- 172.21.15.44:0/2858094261 learned_addr learned my addr 172.21.15.44:0/2858094261 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.711+0000 7fde7a232640 1 -- 172.21.15.44:0/2858094261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f55f30 msgr2=0x55b570f56310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.711+0000 7fde7a232640 1 --2- 172.21.15.44:0/2858094261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f55f30 0x55b570f56310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.711+0000 7fde7a232640 1 -- 172.21.15.44:0/2858094261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f5f430 msgr2=0x55b570f61820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.711+0000 7fde7a232640 1 --2- 172.21.15.44:0/2858094261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f5f430 0x55b570f61820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.711+0000 7fde6b7fe640 1 --2- 172.21.15.44:0/2439520923 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fde600a2d00 0x7fde600a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.711+0000 7fde7a232640 1 -- 172.21.15.44:0/2858094261 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b570c987d0 con 0x55b570f56850 2024-04-21T23:16:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.711+0000 7fde7aa33640 1 --2- 172.21.15.44:0/2439520923 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fde600a2d00 0x7fde600a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.711+0000 7fde6b7fe640 1 -- 172.21.15.44:0/2439520923 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7fde700cfb70 con 0x7fde74101710 2024-04-21T23:16:26.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.711+0000 7fde7a232640 1 --2- 172.21.15.44:0/2858094261 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f56850 0x55b570f5eef0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fde700cce90 tx=0x7fde70098050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6cb1dd7c6fef121 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:26.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.712+0000 7fde68bfe640 1 -- 172.21.15.44:0/2858094261 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7009bac0 con 0x55b570f56850 2024-04-21T23:16:26.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.712+0000 7fde68bfe640 1 -- 172.21.15.44:0/2858094261 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde70009450 con 0x55b570f56850 2024-04-21T23:16:26.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.712+0000 7fde68bfe640 1 -- 172.21.15.44:0/2858094261 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7009b650 con 0x55b570f56850 2024-04-21T23:16:26.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.712+0000 7fde7f084740 1 -- 172.21.15.44:0/2858094261 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f56850 msgr2=0x55b570f5eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:26.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.712+0000 7fde7f084740 1 --2- 172.21.15.44:0/2858094261 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f56850 0x55b570f5eef0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fde700cce90 tx=0x7fde70098050 comp rx=0 tx=0).stop 2024-04-21T23:16:26.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.712+0000 7fde7aa33640 1 --2- 172.21.15.44:0/2439520923 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fde600a2d00 0x7fde600a51c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fde6c005fd0 tx=0x7fde6c0095c0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:26.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.712+0000 7fde7f084740 1 -- 172.21.15.44:0/2858094261 shutdown_connections 2024-04-21T23:16:26.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.712+0000 7fde7f084740 1 --2- 172.21.15.44:0/2858094261 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f5f430 0x55b570f61820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.712+0000 7fde7f084740 1 --2- 172.21.15.44:0/2858094261 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f56850 0x55b570f5eef0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.712+0000 7fde7f084740 1 --2- 172.21.15.44:0/2858094261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f55f30 0x55b570f56310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.712+0000 7fde7f084740 1 -- 172.21.15.44:0/2858094261 >> 172.21.15.44:0/2858094261 conn(0x55b570f50660 msgr2=0x55b570f50a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:26.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.712+0000 7fde7f084740 1 -- 172.21.15.44:0/2858094261 shutdown_connections 2024-04-21T23:16:26.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.713+0000 7fde7f084740 1 -- 172.21.15.44:0/2858094261 wait complete. 2024-04-21T23:16:26.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.713+0000 7fde7f084740 1 -- start start 2024-04-21T23:16:26.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.713+0000 7fde7f084740 1 Processor -- start 2024-04-21T23:16:26.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.713+0000 7fde7f084740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f55f30 0x55b570f5c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.713+0000 7fde7f084740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f56850 0x55b570f5ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.713+0000 7fde7f084740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f5f430 0x55b570f5cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.713+0000 7fde7f084740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b570f62220 con 0x55b570f55f30 2024-04-21T23:16:26.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.713+0000 7fde7f084740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55b570e75e00 con 0x55b570f56850 2024-04-21T23:16:26.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.713+0000 7fde7f084740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b570f623a0 con 0x55b570f5f430 2024-04-21T23:16:26.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.713+0000 7fde7f084740 10 client.0 _ll_register_callbacks cb 0x55b570eb0eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:26.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.713+0000 7fde7a232640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f55f30 0x55b570f5c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.713+0000 7fde7a232640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f55f30 0x55b570f5c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37114/0 (socket says 172.21.15.44:37114) 2024-04-21T23:16:26.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.713+0000 7fde7a232640 1 -- 172.21.15.44:0/2076991818 learned_addr learned my addr 172.21.15.44:0/2076991818 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:26.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.713+0000 7fde79a31640 1 --2- 172.21.15.44:0/2076991818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f56850 0x55b570f5ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.713+0000 7fde7aa33640 1 --2- 172.21.15.44:0/2076991818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f5f430 0x55b570f5cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.714+0000 7fde7a232640 1 -- 172.21.15.44:0/2076991818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f5f430 msgr2=0x55b570f5cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:26.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.714+0000 7fde7a232640 1 --2- 172.21.15.44:0/2076991818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f5f430 0x55b570f5cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.714+0000 7fde7a232640 1 -- 172.21.15.44:0/2076991818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f56850 msgr2=0x55b570f5ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:26.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.714+0000 7fde7a232640 1 --2- 172.21.15.44:0/2076991818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f56850 0x55b570f5ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.714+0000 7fde7a232640 1 -- 172.21.15.44:0/2076991818 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b570bce3f0 con 0x55b570f55f30 2024-04-21T23:16:26.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.714+0000 7fde7a232640 1 --2- 172.21.15.44:0/2076991818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f55f30 0x55b570f5c4f0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fde7009b9f0 tx=0x7fde700cfe70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=714ae45a1281cc78 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:26.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.714+0000 7fde5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:16:26.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.714+0000 7fde5dffb640 1 -- 172.21.15.44:0/2076991818 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7009b830 con 0x55b570f55f30 2024-04-21T23:16:26.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.714+0000 7fde5dffb640 1 -- 172.21.15.44:0/2076991818 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7fde700037a0 con 0x55b570f55f30 2024-04-21T23:16:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.714+0000 7fde7f084740 1 -- 172.21.15.44:0/2076991818 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde70009450 con 0x55b570f55f30 2024-04-21T23:16:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.714+0000 7fde5dffb640 1 -- 172.21.15.44:0/2076991818 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fde7009b650 con 0x55b570f55f30 2024-04-21T23:16:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.714+0000 7fde7f084740 10 client.17394 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.714+0000 7fde7f084740 1 -- 172.21.15.44:0/2076991818 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b570b8e0f0 con 0x55b570f55f30 2024-04-21T23:16:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.714+0000 7fde5dffb640 1 -- 172.21.15.44:0/2076991818 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde70009450 con 0x55b570f55f30 2024-04-21T23:16:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.715+0000 7fde5dffb640 1 -- 172.21.15.44:0/2076991818 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7fde700cd1a0 con 0x55b570f55f30 2024-04-21T23:16:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.715+0000 7fde7f084740 10 client.17394 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.715+0000 7fde7f084740 10 client.17394 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.715+0000 7fde7f084740 20 client.17394 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.715+0000 7fde7f084740 1 --2- 172.21.15.44:0/2076991818 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b570f6f6d0 0x55b570f71ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.715+0000 7fde7f084740 4 client.17394 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:16:26.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.715+0000 7fde7f084740 1 -- 172.21.15.44:0/2076991818 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b570ec7aa0 con 0x55b570f6f6d0 2024-04-21T23:16:26.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.721+0000 7fde79a31640 1 --2- 172.21.15.44:0/2076991818 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b570f6f6d0 0x55b570f71ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.724+0000 7fde79a31640 1 --2- 172.21.15.44:0/2076991818 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b570f6f6d0 0x55b570f71ac0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26ee4dae32a32987 server_cookie=8c4ee74def2e2203 in_seq=0 out_seq=0 2024-04-21T23:16:26.724 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.725+0000 7fde5dffb640 10 client.17394 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.727+0000 7fde5dffb640 1 -- 172.21.15.44:0/2076991818 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b570ec7aa0 con 0x55b570f6f6d0 2024-04-21T23:16:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.727+0000 7fde5dffb640 10 client.17394 handle_command_reply: tid=0 2024-04-21T23:16:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.727+0000 7fde7f084740 1 client.17394 shutdown 2024-04-21T23:16:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.727+0000 7fde7f084740 1 -- 172.21.15.44:0/2076991818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f55f30 msgr2=0x55b570f5c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.727+0000 7fde7f084740 1 --2- 172.21.15.44:0/2076991818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f55f30 0x55b570f5c4f0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fde7009b9f0 tx=0x7fde700cfe70 comp rx=0 tx=0).stop 2024-04-21T23:16:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.727+0000 7fde7f084740 1 -- 172.21.15.44:0/2076991818 shutdown_connections 2024-04-21T23:16:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.727+0000 7fde7f084740 1 --2- 172.21.15.44:0/2076991818 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b570f6f6d0 0x55b570f71ac0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.727+0000 7fde5dffb640 0 client.17394 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.727+0000 7fde7f084740 1 --2- 172.21.15.44:0/2076991818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f5f430 0x55b570f5cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.727+0000 7fde7f084740 1 --2- 172.21.15.44:0/2076991818 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f56850 0x55b570f5ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.727+0000 7fde7f084740 1 --2- 172.21.15.44:0/2076991818 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f55f30 0x55b570f5c4f0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.727+0000 7fde7f084740 1 -- 172.21.15.44:0/2076991818 >> 172.21.15.44:0/2076991818 conn(0x55b570f50660 msgr2=0x55b570f523b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.727+0000 7fde7f084740 1 -- 172.21.15.44:0/2076991818 shutdown_connections 2024-04-21T23:16:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.727+0000 7fde7f084740 1 -- 172.21.15.44:0/2076991818 wait complete. 2024-04-21T23:16:26.727 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.727+0000 7fde7f084740 20 client.17394 trim_cache size 0 max 16384 2024-04-21T23:16:26.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.763+0000 7fde7f084740 1 Processor -- start 2024-04-21T23:16:26.762 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.763+0000 7fde7f084740 1 -- start start 2024-04-21T23:16:26.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.764+0000 7fde7f084740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f55f30 0x55b570f5ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.763 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.764+0000 7fde7f084740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f56850 0x55b570f56cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.764+0000 7fde7a232640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f55f30 0x55b570f5ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.764+0000 7fde7a232640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f55f30 0x55b570f5ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:58386/0 (socket says 172.21.15.44:58386) 2024-04-21T23:16:26.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.764+0000 7fde7a232640 1 -- 172.21.15.44:0/3274223807 learned_addr learned my addr 172.21.15.44:0/3274223807 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:26.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.764+0000 7fde79a31640 1 --2- 172.21.15.44:0/3274223807 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f56850 0x55b570f56cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.764+0000 7fde7f084740 1 --2- 172.21.15.44:0/3274223807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f5f430 0x55b570f521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.765+0000 7fde7f084740 1 -- 172.21.15.44:0/3274223807 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fde741b3340 con 0x55b570f5f430 2024-04-21T23:16:26.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.765+0000 7fde7aa33640 1 --2- 172.21.15.44:0/3274223807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f5f430 0x55b570f521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.765+0000 7fde7f084740 1 -- 172.21.15.44:0/3274223807 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fde741b5bc0 con 0x55b570f56850 2024-04-21T23:16:26.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.765+0000 7fde7f084740 1 -- 172.21.15.44:0/3274223807 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fde741b39b0 con 0x55b570f55f30 2024-04-21T23:16:26.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.765+0000 7fde79a31640 1 -- 172.21.15.44:0/3274223807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f55f30 msgr2=0x55b570f5ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:26.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.765+0000 7fde79a31640 1 --2- 172.21.15.44:0/3274223807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f55f30 0x55b570f5ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.765+0000 7fde79a31640 1 -- 172.21.15.44:0/3274223807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f5f430 msgr2=0x55b570f521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:26.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.765+0000 7fde79a31640 1 --2- 172.21.15.44:0/3274223807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f5f430 0x55b570f521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.765+0000 7fde79a31640 1 -- 172.21.15.44:0/3274223807 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b570f5b4a0 con 0x55b570f56850 2024-04-21T23:16:26.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.766+0000 7fde79a31640 1 --2- 172.21.15.44:0/3274223807 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f56850 0x55b570f56cb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fde6402c180 tx=0x7fde64009490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56f3ed69f5013e6e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:26.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.766+0000 7fde5ffff640 1 -- 172.21.15.44:0/3274223807 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde64049020 con 0x55b570f56850 2024-04-21T23:16:26.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.766+0000 7fde5ffff640 1 -- 172.21.15.44:0/3274223807 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde64034440 con 0x55b570f56850 2024-04-21T23:16:26.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.766+0000 7fde5ffff640 1 -- 172.21.15.44:0/3274223807 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde64039040 con 0x55b570f56850 2024-04-21T23:16:26.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.767+0000 7fde7f084740 1 -- 172.21.15.44:0/3274223807 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f56850 msgr2=0x55b570f56cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:26.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.767+0000 7fde7f084740 1 --2- 172.21.15.44:0/3274223807 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f56850 0x55b570f56cb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fde6402c180 tx=0x7fde64009490 comp rx=0 tx=0).stop 2024-04-21T23:16:26.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.767+0000 7fde7f084740 1 -- 172.21.15.44:0/3274223807 shutdown_connections 2024-04-21T23:16:26.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.767+0000 7fde7f084740 1 --2- 172.21.15.44:0/3274223807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f5f430 0x55b570f521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.767+0000 7fde7f084740 1 --2- 172.21.15.44:0/3274223807 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f56850 0x55b570f56cb0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.767+0000 7fde7f084740 1 --2- 172.21.15.44:0/3274223807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f55f30 0x55b570f5ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.767+0000 7fde7f084740 1 -- 172.21.15.44:0/3274223807 >> 172.21.15.44:0/3274223807 conn(0x55b570f50660 msgr2=0x55b570f51a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:26.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.767+0000 7fde7f084740 1 -- 172.21.15.44:0/3274223807 shutdown_connections 2024-04-21T23:16:26.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.767+0000 7fde7f084740 1 -- 172.21.15.44:0/3274223807 wait complete. 2024-04-21T23:16:26.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.767+0000 7fde7f084740 1 -- start start 2024-04-21T23:16:26.767 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.768+0000 7fde7f084740 1 Processor -- start 2024-04-21T23:16:26.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.768+0000 7fde7f084740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f55f30 0x55b570f74c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.768+0000 7fde7f084740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f56850 0x55b570f720c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.768+0000 7fde7f084740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f5f430 0x55b570f72600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.768+0000 7fde7f084740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fde741b4080 con 0x55b570f56850 2024-04-21T23:16:26.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.768+0000 7fde7f084740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fde741b54f0 con 0x55b570f55f30 2024-04-21T23:16:26.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.768+0000 7fde7f084740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fde741b4750 con 0x55b570f5f430 2024-04-21T23:16:26.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.768+0000 7fde7f084740 10 client.0 _ll_register_callbacks cb 0x7fde64037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:26.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.768+0000 7fde79a31640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f56850 0x55b570f720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.768+0000 7fde79a31640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f56850 0x55b570f720c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:37144/0 (socket says 172.21.15.44:37144) 2024-04-21T23:16:26.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.768+0000 7fde79a31640 1 -- 172.21.15.44:0/4288252462 learned_addr learned my addr 172.21.15.44:0/4288252462 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:26.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.768+0000 7fde7aa33640 1 --2- 172.21.15.44:0/4288252462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f5f430 0x55b570f72600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.768+0000 7fde7a232640 1 --2- 172.21.15.44:0/4288252462 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f55f30 0x55b570f74c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.769+0000 7fde79a31640 1 -- 172.21.15.44:0/4288252462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f5f430 msgr2=0x55b570f72600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:26.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.769+0000 7fde79a31640 1 --2- 172.21.15.44:0/4288252462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f5f430 0x55b570f72600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.769+0000 7fde79a31640 1 -- 172.21.15.44:0/4288252462 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f55f30 msgr2=0x55b570f74c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:26.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.769+0000 7fde79a31640 1 --2- 172.21.15.44:0/4288252462 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f55f30 0x55b570f74c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.769+0000 7fde79a31640 1 -- 172.21.15.44:0/4288252462 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b570f72b40 con 0x55b570f56850 2024-04-21T23:16:26.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.769+0000 7fde7aa33640 1 --2- 172.21.15.44:0/4288252462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f5f430 0x55b570f72600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:26.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.769+0000 7fde79a31640 1 --2- 172.21.15.44:0/4288252462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f56850 0x55b570f720c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fde64037040 tx=0x7fde640324c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a51785b4c250031 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:26.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.769+0000 7fde5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:16:26.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.769+0000 7fde5dffb640 1 -- 172.21.15.44:0/4288252462 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde64049030 con 0x55b570f56850 2024-04-21T23:16:26.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.769+0000 7fde5dffb640 1 -- 172.21.15.44:0/4288252462 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7fde64004040 con 0x55b570f56850 2024-04-21T23:16:26.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.769+0000 7fde7f084740 1 -- 172.21.15.44:0/4288252462 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde64034440 con 0x55b570f56850 2024-04-21T23:16:26.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.769+0000 7fde5dffb640 1 -- 172.21.15.44:0/4288252462 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fde64060050 con 0x55b570f56850 2024-04-21T23:16:26.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.769+0000 7fde7f084740 10 client.17400 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:26.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.770+0000 7fde7f084740 1 -- 172.21.15.44:0/4288252462 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b570fd55b0 con 0x55b570f56850 2024-04-21T23:16:26.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.770+0000 7fde5dffb640 1 -- 172.21.15.44:0/4288252462 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde64034440 con 0x55b570f56850 2024-04-21T23:16:26.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.770+0000 7fde5dffb640 1 -- 172.21.15.44:0/4288252462 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7fde6402c770 con 0x55b570f56850 2024-04-21T23:16:26.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.770+0000 7fde7f084740 10 client.17400 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:26.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.770+0000 7fde7f084740 10 client.17400 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:26.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.770+0000 7fde7f084740 20 client.17400 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:26.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.770+0000 7fde7f084740 1 --2- 172.21.15.44:0/4288252462 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b570f6f6d0 0x55b570fd7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:26.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.770+0000 7fde7f084740 4 client.17400 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:16:26.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.770+0000 7fde7f084740 1 -- 172.21.15.44:0/4288252462 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b570ec1df0 con 0x55b570f6f6d0 2024-04-21T23:16:26.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.770+0000 7fde7a232640 1 --2- 172.21.15.44:0/4288252462 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b570f6f6d0 0x55b570fd7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:26.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.771+0000 7fde5dffb640 10 client.17400 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:26.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.771+0000 7fde7a232640 1 --2- 172.21.15.44:0/4288252462 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b570f6f6d0 0x55b570fd7cc0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca49cf203e569f3c server_cookie=1b13ac4464d736ba in_seq=0 out_seq=0 2024-04-21T23:16:26.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.772+0000 7fde5dffb640 1 -- 172.21.15.44:0/4288252462 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b570ec1df0 con 0x55b570f6f6d0 2024-04-21T23:16:26.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.772+0000 7fde5dffb640 10 client.17400 handle_command_reply: tid=0 2024-04-21T23:16:26.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.772+0000 7fde7f084740 1 client.17400 shutdown 2024-04-21T23:16:26.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.772+0000 7fde7f084740 1 -- 172.21.15.44:0/4288252462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f56850 msgr2=0x55b570f720c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:26.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.772+0000 7fde7f084740 1 --2- 172.21.15.44:0/4288252462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f56850 0x55b570f720c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fde64037040 tx=0x7fde640324c0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.772+0000 7fde7f084740 1 -- 172.21.15.44:0/4288252462 shutdown_connections 2024-04-21T23:16:26.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.772+0000 7fde7f084740 1 --2- 172.21.15.44:0/4288252462 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b570f6f6d0 0x55b570fd7cc0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.772+0000 7fde7f084740 1 --2- 172.21.15.44:0/4288252462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b570f5f430 0x55b570f72600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.772+0000 7fde7f084740 1 --2- 172.21.15.44:0/4288252462 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b570f56850 0x55b570f720c0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.772+0000 7fde5dffb640 0 client.17400 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:26.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.772+0000 7fde7f084740 1 --2- 172.21.15.44:0/4288252462 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b570f55f30 0x55b570f74c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.772+0000 7fde7f084740 1 -- 172.21.15.44:0/4288252462 >> 172.21.15.44:0/4288252462 conn(0x55b570f50660 msgr2=0x55b570f6ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:26.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.772+0000 7fde7f084740 1 -- 172.21.15.44:0/4288252462 shutdown_connections 2024-04-21T23:16:26.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.772+0000 7fde7f084740 1 -- 172.21.15.44:0/4288252462 wait complete. 2024-04-21T23:16:26.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.772+0000 7fde7f084740 20 client.17400 trim_cache size 0 max 16384 2024-04-21T23:16:26.775 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:16:26.775 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:16:26.775 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-21T23:16:26.775 INFO:teuthology.orchestra.run.smithi044.stdout: "034a06f5-da71-4918-9930-6f2ff5d081b3": { 2024-04-21T23:16:26.775 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-21T23:16:26.775 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "034a06f5-da71-4918-9930-6f2ff5d081b3", 2024-04-21T23:16:26.775 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-04-21T23:16:26.775 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:16:26.775 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:16:26.775 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:16:26.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.773+0000 7fde7c496640 1 -- 172.21.15.44:0/2439520923 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fde600a2d00 msgr2=0x7fde600a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:26.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.773+0000 7fde7c496640 1 --2- 172.21.15.44:0/2439520923 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fde600a2d00 0x7fde600a51c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fde6c005fd0 tx=0x7fde6c0095c0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.773+0000 7fde7c496640 1 -- 172.21.15.44:0/2439520923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde74101710 msgr2=0x7fde741a3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:26.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.773+0000 7fde7c496640 1 --2- 172.21.15.44:0/2439520923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde74101710 0x7fde741a3220 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fde7000e9f0 tx=0x7fde7000eeb0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.773+0000 7fde7c496640 1 -- 172.21.15.44:0/2439520923 shutdown_connections 2024-04-21T23:16:26.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.773+0000 7fde7c496640 1 --2- 172.21.15.44:0/2439520923 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fde600a2d00 0x7fde600a51c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.773+0000 7fde7c496640 1 --2- 172.21.15.44:0/2439520923 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fde74107720 0x7fde7410d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.776 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.773+0000 7fde7c496640 1 --2- 172.21.15.44:0/2439520923 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fde741020d0 0x7fde741a3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.773+0000 7fde7c496640 1 --2- 172.21.15.44:0/2439520923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde74101710 0x7fde741a3220 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:26.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.773+0000 7fde7c496640 1 -- 172.21.15.44:0/2439520923 >> 172.21.15.44:0/2439520923 conn(0x7fde740fd200 msgr2=0x7fde740fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:26.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.773+0000 7fde7c496640 1 -- 172.21.15.44:0/2439520923 shutdown_connections 2024-04-21T23:16:26.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:26.773+0000 7fde7c496640 1 -- 172.21.15.44:0/2439520923 wait complete. 2024-04-21T23:16:26.784 INFO:tasks.cephfs.filesystem:scrub status for tag:034a06f5-da71-4918-9930-6f2ff5d081b3 - {'path': '/', 'tag': '034a06f5-da71-4918-9930-6f2ff5d081b3', 'options': 'recursive,force'} 2024-04-21T23:16:27.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:27 smithi055 ceph-mon[31117]: scrub queued for path: / 2024-04-21T23:16:27.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:27 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:16:27.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:27 smithi055 ceph-mon[31117]: scrub summary: active paths [/] 2024-04-21T23:16:27.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:27 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:16:27.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:27 smithi055 ceph-mon[31117]: scrub summary: idle 2024-04-21T23:16:27.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:27 smithi069 ceph-mon[31816]: scrub queued for path: / 2024-04-21T23:16:27.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:27 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:16:27.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:27 smithi069 ceph-mon[31816]: scrub summary: active paths [/] 2024-04-21T23:16:27.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:27 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:16:27.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:27 smithi069 ceph-mon[31816]: scrub summary: idle 2024-04-21T23:16:27.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:27 smithi044 ceph-mon[27934]: scrub queued for path: / 2024-04-21T23:16:27.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:27 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:16:27.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:27 smithi044 ceph-mon[27934]: scrub summary: active paths [/] 2024-04-21T23:16:27.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:27 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:16:27.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:27 smithi044 ceph-mon[27934]: scrub summary: idle 2024-04-21T23:16:27.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 879071 11.94 MB/sec execute 257 sec latency 60.321 ms 2024-04-21T23:16:28.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:28 smithi055 ceph-mon[31117]: pgmap v240: 129 pgs: 129 active+clean; 70 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 389 op/s 2024-04-21T23:16:28.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:28 smithi069 ceph-mon[31816]: pgmap v240: 129 pgs: 129 active+clean; 70 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 389 op/s 2024-04-21T23:16:28.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:28 smithi044 ceph-mon[27934]: pgmap v240: 129 pgs: 129 active+clean; 70 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 389 op/s 2024-04-21T23:16:28.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 881159 11.94 MB/sec execute 258 sec latency 34.600 ms 2024-04-21T23:16:29.708 INFO:tasks.workunit.client.0.smithi044.stdout: 1 883777 11.95 MB/sec execute 259 sec latency 17.526 ms 2024-04-21T23:16:30.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:30 smithi055 ceph-mon[31117]: pgmap v241: 129 pgs: 129 active+clean; 74 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 296 op/s 2024-04-21T23:16:30.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:30 smithi069 ceph-mon[31816]: pgmap v241: 129 pgs: 129 active+clean; 74 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 296 op/s 2024-04-21T23:16:30.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:30 smithi044 ceph-mon[27934]: pgmap v241: 129 pgs: 129 active+clean; 74 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 296 op/s 2024-04-21T23:16:30.707 INFO:tasks.workunit.client.0.smithi044.stdout: 1 885470 11.94 MB/sec execute 260 sec latency 15.934 ms 2024-04-21T23:16:31.708 INFO:tasks.workunit.client.0.smithi044.stdout: 1 887891 11.95 MB/sec execute 261 sec latency 33.923 ms 2024-04-21T23:16:32.372 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.371+0000 7fdb83adf640 1 -- 172.21.15.44:0/2763741871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb7c1032d0 msgr2=0x7fdb7c10b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.371+0000 7fdb83adf640 1 --2- 172.21.15.44:0/2763741871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb7c1032d0 0x7fdb7c10b7b0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fdb70009900 tx=0x7fdb7002bcd0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.371+0000 7fdb83adf640 1 -- 172.21.15.44:0/2763741871 shutdown_connections 2024-04-21T23:16:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.371+0000 7fdb83adf640 1 --2- 172.21.15.44:0/2763741871 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb7c10bd80 0x7fdb7c10e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.371+0000 7fdb83adf640 1 --2- 172.21.15.44:0/2763741871 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb7c1032d0 0x7fdb7c10b7b0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.371+0000 7fdb83adf640 1 --2- 172.21.15.44:0/2763741871 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb7c1029b0 0x7fdb7c102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.371+0000 7fdb83adf640 1 -- 172.21.15.44:0/2763741871 >> 172.21.15.44:0/2763741871 conn(0x7fdb7c077760 msgr2=0x7fdb7c077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.371+0000 7fdb83adf640 1 -- 172.21.15.44:0/2763741871 shutdown_connections 2024-04-21T23:16:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.371+0000 7fdb83adf640 1 -- 172.21.15.44:0/2763741871 wait complete. 2024-04-21T23:16:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.371+0000 7fdb83adf640 1 Processor -- start 2024-04-21T23:16:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.371+0000 7fdb83adf640 1 -- start start 2024-04-21T23:16:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb83adf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb7c1029b0 0x7fdb7c198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb83adf640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb7c1032d0 0x7fdb7c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb83adf640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb7c10bd80 0x7fdb7c19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb83adf640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c111b10 con 0x7fdb7c10bd80 2024-04-21T23:16:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb83adf640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c111990 con 0x7fdb7c1032d0 2024-04-21T23:16:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb83adf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c111c90 con 0x7fdb7c1029b0 2024-04-21T23:16:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb8207c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb7c10bd80 0x7fdb7c19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb8207c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb7c10bd80 0x7fdb7c19c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35130/0 (socket says 172.21.15.44:35130) 2024-04-21T23:16:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb8207c640 1 -- 172.21.15.44:0/788614045 learned_addr learned my addr 172.21.15.44:0/788614045 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb8107a640 1 --2- 172.21.15.44:0/788614045 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb7c1032d0 0x7fdb7c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb8187b640 1 --2- 172.21.15.44:0/788614045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb7c1029b0 0x7fdb7c198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb8207c640 1 -- 172.21.15.44:0/788614045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb7c1029b0 msgr2=0x7fdb7c198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb8207c640 1 --2- 172.21.15.44:0/788614045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb7c1029b0 0x7fdb7c198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb8207c640 1 -- 172.21.15.44:0/788614045 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb7c1032d0 msgr2=0x7fdb7c198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb8207c640 1 --2- 172.21.15.44:0/788614045 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb7c1032d0 0x7fdb7c198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb8207c640 1 -- 172.21.15.44:0/788614045 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7c19d020 con 0x7fdb7c10bd80 2024-04-21T23:16:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb8207c640 1 --2- 172.21.15.44:0/788614045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb7c10bd80 0x7fdb7c19c8f0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fdb7400e970 tx=0x7fdb7400ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb7affd640 1 -- 172.21.15.44:0/788614045 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7400ccc0 con 0x7fdb7c10bd80 2024-04-21T23:16:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb7affd640 1 -- 172.21.15.44:0/788614045 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb74005500 con 0x7fdb7c10bd80 2024-04-21T23:16:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb83adf640 1 -- 172.21.15.44:0/788614045 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb7c19d310 con 0x7fdb7c10bd80 2024-04-21T23:16:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb7affd640 1 -- 172.21.15.44:0/788614045 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb74010690 con 0x7fdb7c10bd80 2024-04-21T23:16:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.372+0000 7fdb83adf640 1 -- 172.21.15.44:0/788614045 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb7c1a4ba0 con 0x7fdb7c10bd80 2024-04-21T23:16:32.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.374+0000 7fdb7affd640 1 -- 172.21.15.44:0/788614045 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fdb740046c0 con 0x7fdb7c10bd80 2024-04-21T23:16:32.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.375+0000 7fdb867a4740 1 Processor -- start 2024-04-21T23:16:32.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb867a4740 1 -- start start 2024-04-21T23:16:32.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb7affd640 1 --2- 172.21.15.44:0/788614045 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fdb640a2cf0 0x7fdb640a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:32.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb867a4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 0x55f122dd1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:32.382 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb867a4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 0x55f122ddacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb867a4740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122ddb210 0x55f122ddd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb867a4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f122d35b70 con 0x55f122dd2440 2024-04-21T23:16:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb867a4740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55f122cde470 con 0x55f122ddb210 2024-04-21T23:16:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb867a4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f122d31b30 con 0x55f122dd1b20 2024-04-21T23:16:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb7affd640 1 -- 172.21.15.44:0/788614045 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7fdb740cfb90 con 0x7fdb7c10bd80 2024-04-21T23:16:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb8107a640 1 --2- 172.21.15.44:0/788614045 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fdb640a2cf0 0x7fdb640a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb8107a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 0x55f122ddacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb8107a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 0x55f122ddacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35140/0 (socket says 172.21.15.44:35140) 2024-04-21T23:16:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb8107a640 1 -- 172.21.15.44:0/2414378492 learned_addr learned my addr 172.21.15.44:0/2414378492 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb8207c640 1 --2- 172.21.15.44:0/2414378492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122ddb210 0x55f122ddd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb8187b640 1 --2- 172.21.15.44:0/2414378492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 0x55f122dd1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb8207c640 1 -- 172.21.15.44:0/2414378492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 msgr2=0x55f122dd1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb8207c640 1 --2- 172.21.15.44:0/2414378492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 0x55f122dd1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb8207c640 1 -- 172.21.15.44:0/2414378492 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 msgr2=0x55f122ddacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb8207c640 1 --2- 172.21.15.44:0/2414378492 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 0x55f122ddacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb8207c640 1 -- 172.21.15.44:0/2414378492 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f122a4a210 con 0x55f122ddb210 2024-04-21T23:16:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb8187b640 1 --2- 172.21.15.44:0/2414378492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 0x55f122dd1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:16:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.376+0000 7fdb8107a640 1 --2- 172.21.15.44:0/2414378492 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 0x55f122ddacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.377+0000 7fdb8107a640 1 --2- 172.21.15.44:0/788614045 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fdb640a2cf0 0x7fdb640a51b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdb70004ec0 tx=0x7fdb70005e60 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.377+0000 7fdb8207c640 1 --2- 172.21.15.44:0/2414378492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122ddb210 0x55f122ddd600 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fdb740ccf20 tx=0x7fdb7409b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cb852a59f954249 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.377+0000 7fdb63fff640 1 -- 172.21.15.44:0/2414378492 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7409bcf0 con 0x55f122ddb210 2024-04-21T23:16:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.377+0000 7fdb63fff640 1 -- 172.21.15.44:0/2414378492 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb740a00a0 con 0x55f122ddb210 2024-04-21T23:16:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.377+0000 7fdb63fff640 1 -- 172.21.15.44:0/2414378492 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb740a4730 con 0x55f122ddb210 2024-04-21T23:16:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.378+0000 7fdb867a4740 1 -- 172.21.15.44:0/2414378492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122ddb210 msgr2=0x55f122ddd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.378+0000 7fdb867a4740 1 --2- 172.21.15.44:0/2414378492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122ddb210 0x55f122ddd600 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fdb740ccf20 tx=0x7fdb7409b8f0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.378+0000 7fdb867a4740 1 -- 172.21.15.44:0/2414378492 shutdown_connections 2024-04-21T23:16:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.378+0000 7fdb867a4740 1 --2- 172.21.15.44:0/2414378492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122ddb210 0x55f122ddd600 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.378+0000 7fdb867a4740 1 --2- 172.21.15.44:0/2414378492 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 0x55f122ddacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.378+0000 7fdb867a4740 1 --2- 172.21.15.44:0/2414378492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 0x55f122dd1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.378+0000 7fdb867a4740 1 -- 172.21.15.44:0/2414378492 >> 172.21.15.44:0/2414378492 conn(0x55f122dcc440 msgr2=0x55f122dcc820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.378+0000 7fdb867a4740 1 -- 172.21.15.44:0/2414378492 shutdown_connections 2024-04-21T23:16:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.378+0000 7fdb867a4740 1 -- 172.21.15.44:0/2414378492 wait complete. 2024-04-21T23:16:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.378+0000 7fdb867a4740 1 -- start start 2024-04-21T23:16:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.378+0000 7fdb867a4740 1 Processor -- start 2024-04-21T23:16:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb867a4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 0x55f122dd82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb867a4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 0x55f122dd87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb867a4740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122ddb210 0x55f122dd8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb867a4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f122ddde40 con 0x55f122dd2440 2024-04-21T23:16:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb867a4740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55f122cf1a10 con 0x55f122ddb210 2024-04-21T23:16:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb867a4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f122dddfc0 con 0x55f122dd1b20 2024-04-21T23:16:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb867a4740 10 client.0 _ll_register_callbacks cb 0x55f122d2cb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb8107a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 0x55f122dd87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb8107a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 0x55f122dd87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35144/0 (socket says 172.21.15.44:35144) 2024-04-21T23:16:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb8107a640 1 -- 172.21.15.44:0/3897224389 learned_addr learned my addr 172.21.15.44:0/3897224389 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb8207c640 1 --2- 172.21.15.44:0/3897224389 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122ddb210 0x55f122dd8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb8187b640 1 --2- 172.21.15.44:0/3897224389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 0x55f122dd82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb8107a640 1 -- 172.21.15.44:0/3897224389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 msgr2=0x55f122dd82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb8107a640 1 --2- 172.21.15.44:0/3897224389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 0x55f122dd82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb8107a640 1 -- 172.21.15.44:0/3897224389 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122ddb210 msgr2=0x55f122dd8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb8107a640 1 --2- 172.21.15.44:0/3897224389 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122ddb210 0x55f122dd8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb8107a640 1 -- 172.21.15.44:0/3897224389 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f122b16620 con 0x55f122dd2440 2024-04-21T23:16:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb8207c640 1 --2- 172.21.15.44:0/3897224389 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122ddb210 0x55f122dd8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb8107a640 1 --2- 172.21.15.44:0/3897224389 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 0x55f122dd87f0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fdb70036040 tx=0x7fdb70005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dded314d06ce1b63 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb617fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:16:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb617fa640 1 -- 172.21.15.44:0/3897224389 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb70005110 con 0x55f122dd2440 2024-04-21T23:16:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb617fa640 1 -- 172.21.15.44:0/3897224389 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7fdb700052b0 con 0x55f122dd2440 2024-04-21T23:16:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.379+0000 7fdb867a4740 1 -- 172.21.15.44:0/3897224389 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb740a00a0 con 0x55f122dd2440 2024-04-21T23:16:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.380+0000 7fdb617fa640 1 -- 172.21.15.44:0/3897224389 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdb7005d050 con 0x55f122dd2440 2024-04-21T23:16:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.380+0000 7fdb867a4740 10 client.17412 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.380+0000 7fdb867a4740 1 -- 172.21.15.44:0/3897224389 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f122dea100 con 0x55f122dd2440 2024-04-21T23:16:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.380+0000 7fdb617fa640 1 -- 172.21.15.44:0/3897224389 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb740a00a0 con 0x55f122dd2440 2024-04-21T23:16:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.380+0000 7fdb617fa640 1 -- 172.21.15.44:0/3897224389 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7fdb7004f450 con 0x55f122dd2440 2024-04-21T23:16:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.380+0000 7fdb867a4740 10 client.17412 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.380+0000 7fdb867a4740 10 client.17412 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.380+0000 7fdb867a4740 20 client.17412 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.380+0000 7fdb867a4740 1 --2- 172.21.15.44:0/3897224389 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f122deb610 0x55f122deda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.380+0000 7fdb867a4740 4 client.17412 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:16:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.380+0000 7fdb867a4740 1 -- 172.21.15.44:0/3897224389 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f122d430b0 con 0x55f122deb610 2024-04-21T23:16:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.380+0000 7fdb8187b640 1 --2- 172.21.15.44:0/3897224389 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f122deb610 0x55f122deda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.381+0000 7fdb8187b640 1 --2- 172.21.15.44:0/3897224389 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f122deb610 0x55f122deda20 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f54b37e6655ffdb server_cookie=533fe611c337207 in_seq=0 out_seq=0 2024-04-21T23:16:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.381+0000 7fdb617fa640 10 client.17412 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:32.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.382+0000 7fdb617fa640 1 -- 172.21.15.44:0/3897224389 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f122d430b0 con 0x55f122deb610 2024-04-21T23:16:32.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.382+0000 7fdb617fa640 10 client.17412 handle_command_reply: tid=0 2024-04-21T23:16:32.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.382+0000 7fdb867a4740 1 client.17412 shutdown 2024-04-21T23:16:32.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.382+0000 7fdb867a4740 1 -- 172.21.15.44:0/3897224389 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 msgr2=0x55f122dd87f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:32.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.382+0000 7fdb867a4740 1 --2- 172.21.15.44:0/3897224389 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 0x55f122dd87f0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fdb70036040 tx=0x7fdb70005fd0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.382+0000 7fdb867a4740 1 -- 172.21.15.44:0/3897224389 shutdown_connections 2024-04-21T23:16:32.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.382+0000 7fdb867a4740 1 --2- 172.21.15.44:0/3897224389 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f122deb610 0x55f122deda20 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.382+0000 7fdb867a4740 1 --2- 172.21.15.44:0/3897224389 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122ddb210 0x55f122dd8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.382+0000 7fdb617fa640 0 client.17412 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:32.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.382+0000 7fdb867a4740 1 --2- 172.21.15.44:0/3897224389 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 0x55f122dd87f0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.382+0000 7fdb867a4740 1 --2- 172.21.15.44:0/3897224389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 0x55f122dd82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.382+0000 7fdb867a4740 1 -- 172.21.15.44:0/3897224389 >> 172.21.15.44:0/3897224389 conn(0x55f122dcc440 msgr2=0x55f122dce190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:32.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.382+0000 7fdb867a4740 1 -- 172.21.15.44:0/3897224389 shutdown_connections 2024-04-21T23:16:32.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.382+0000 7fdb867a4740 1 -- 172.21.15.44:0/3897224389 wait complete. 2024-04-21T23:16:32.388 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.382+0000 7fdb867a4740 20 client.17412 trim_cache size 0 max 16384 2024-04-21T23:16:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.397+0000 7fdb867a4740 1 Processor -- start 2024-04-21T23:16:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.397+0000 7fdb867a4740 1 -- start start 2024-04-21T23:16:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.397+0000 7fdb867a4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 0x55f122dda9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.397+0000 7fdb867a4740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122dd2440 0x55f122de6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.397+0000 7fdb867a4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122ddb210 0x55f122df3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.397+0000 7fdb867a4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c1acb70 con 0x55f122ddb210 2024-04-21T23:16:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.397+0000 7fdb867a4740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c1af3f0 con 0x55f122dd2440 2024-04-21T23:16:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.397+0000 7fdb867a4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c1ad1e0 con 0x55f122dd1b20 2024-04-21T23:16:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.398+0000 7fdb8207c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122ddb210 0x55f122df3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.398+0000 7fdb8207c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122ddb210 0x55f122df3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35150/0 (socket says 172.21.15.44:35150) 2024-04-21T23:16:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.398+0000 7fdb8107a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122dd2440 0x55f122de6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.398+0000 7fdb8207c640 1 -- 172.21.15.44:0/1680635038 learned_addr learned my addr 172.21.15.44:0/1680635038 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.398+0000 7fdb8187b640 1 --2- 172.21.15.44:0/1680635038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 0x55f122dda9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.398+0000 7fdb8207c640 1 -- 172.21.15.44:0/1680635038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 msgr2=0x55f122dda9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.398+0000 7fdb8207c640 1 --2- 172.21.15.44:0/1680635038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 0x55f122dda9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.398+0000 7fdb8207c640 1 -- 172.21.15.44:0/1680635038 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122dd2440 msgr2=0x55f122de6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.398+0000 7fdb8207c640 1 --2- 172.21.15.44:0/1680635038 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122dd2440 0x55f122de6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.398+0000 7fdb8207c640 1 -- 172.21.15.44:0/1680635038 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f122dee0a0 con 0x55f122ddb210 2024-04-21T23:16:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.398+0000 7fdb8107a640 1 --2- 172.21.15.44:0/1680635038 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122dd2440 0x55f122de6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.398+0000 7fdb8187b640 1 --2- 172.21.15.44:0/1680635038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 0x55f122dda9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.398+0000 7fdb8207c640 1 --2- 172.21.15.44:0/1680635038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122ddb210 0x55f122df3330 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fdb740097f0 tx=0x7fdb7409c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc4d830758801c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.398+0000 7fdb637fe640 1 -- 172.21.15.44:0/1680635038 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7407f070 con 0x55f122ddb210 2024-04-21T23:16:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.398+0000 7fdb637fe640 1 -- 172.21.15.44:0/1680635038 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb740a4960 con 0x55f122ddb210 2024-04-21T23:16:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.398+0000 7fdb637fe640 1 -- 172.21.15.44:0/1680635038 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7409b140 con 0x55f122ddb210 2024-04-21T23:16:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.399+0000 7fdb867a4740 1 -- 172.21.15.44:0/1680635038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122ddb210 msgr2=0x55f122df3330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.399+0000 7fdb867a4740 1 --2- 172.21.15.44:0/1680635038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122ddb210 0x55f122df3330 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fdb740097f0 tx=0x7fdb7409c530 comp rx=0 tx=0).stop 2024-04-21T23:16:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.399+0000 7fdb867a4740 1 -- 172.21.15.44:0/1680635038 shutdown_connections 2024-04-21T23:16:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.399+0000 7fdb867a4740 1 --2- 172.21.15.44:0/1680635038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122ddb210 0x55f122df3330 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.399+0000 7fdb867a4740 1 --2- 172.21.15.44:0/1680635038 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122dd2440 0x55f122de6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.399+0000 7fdb867a4740 1 --2- 172.21.15.44:0/1680635038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 0x55f122dda9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.399+0000 7fdb867a4740 1 -- 172.21.15.44:0/1680635038 >> 172.21.15.44:0/1680635038 conn(0x55f122dcc440 msgr2=0x55f122dcb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.399+0000 7fdb867a4740 1 -- 172.21.15.44:0/1680635038 shutdown_connections 2024-04-21T23:16:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.399+0000 7fdb867a4740 1 -- 172.21.15.44:0/1680635038 wait complete. 2024-04-21T23:16:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.399+0000 7fdb867a4740 1 -- start start 2024-04-21T23:16:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.399+0000 7fdb867a4740 1 Processor -- start 2024-04-21T23:16:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb867a4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 0x55f122de2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb867a4740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 0x55f122de3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb867a4740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122ddb210 0x55f122ddf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb867a4740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c1ad8b0 con 0x55f122dd2440 2024-04-21T23:16:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb867a4740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c1aed20 con 0x55f122ddb210 2024-04-21T23:16:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb867a4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c1adf80 con 0x55f122dd1b20 2024-04-21T23:16:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb867a4740 10 client.0 _ll_register_callbacks cb 0x7fdb7002ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb8107a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 0x55f122de3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb8207c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122ddb210 0x55f122ddf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb8107a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 0x55f122de3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:35152/0 (socket says 172.21.15.44:35152) 2024-04-21T23:16:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb8107a640 1 -- 172.21.15.44:0/1866251912 learned_addr learned my addr 172.21.15.44:0/1866251912 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb8207c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122ddb210 0x55f122ddf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:55386/0 (socket says 172.21.15.44:55386) 2024-04-21T23:16:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb8187b640 1 --2- 172.21.15.44:0/1866251912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 0x55f122de2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb8207c640 1 -- 172.21.15.44:0/1866251912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 msgr2=0x55f122de2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb8207c640 1 --2- 172.21.15.44:0/1866251912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 0x55f122de2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb8207c640 1 -- 172.21.15.44:0/1866251912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 msgr2=0x55f122de3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb8207c640 1 --2- 172.21.15.44:0/1866251912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 0x55f122de3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb8207c640 1 -- 172.21.15.44:0/1866251912 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f122ddfb80 con 0x55f122ddb210 2024-04-21T23:16:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb8107a640 1 --2- 172.21.15.44:0/1866251912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 0x55f122de3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb8207c640 1 --2- 172.21.15.44:0/1866251912 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122ddb210 0x55f122ddf3f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fdb7409b9f0 tx=0x7fdb740a4700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67119068c7fae9ac server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.400+0000 7fdb617fa640 10 client.0 ms_handle_connect on v2:172.21.15.55:3300/0 2024-04-21T23:16:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.401+0000 7fdb617fa640 1 -- 172.21.15.44:0/1866251912 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb740a00a0 con 0x55f122ddb210 2024-04-21T23:16:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.401+0000 7fdb617fa640 1 -- 172.21.15.44:0/1866251912 <== mon.2 v2:172.21.15.55:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7fdb7409b350 con 0x55f122ddb210 2024-04-21T23:16:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.401+0000 7fdb867a4740 1 -- 172.21.15.44:0/1866251912 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f122d3d400 con 0x55f122ddb210 2024-04-21T23:16:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.401+0000 7fdb617fa640 1 -- 172.21.15.44:0/1866251912 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdb740a4bb0 con 0x55f122ddb210 2024-04-21T23:16:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.401+0000 7fdb867a4740 10 client.26981 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.401+0000 7fdb867a4740 1 -- 172.21.15.44:0/1866251912 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f122e52570 con 0x55f122ddb210 2024-04-21T23:16:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.401+0000 7fdb617fa640 1 -- 172.21.15.44:0/1866251912 <== mon.2 v2:172.21.15.55:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f122d3d400 con 0x55f122ddb210 2024-04-21T23:16:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.401+0000 7fdb617fa640 1 -- 172.21.15.44:0/1866251912 <== mon.2 v2:172.21.15.55:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7fdb7409adb0 con 0x55f122ddb210 2024-04-21T23:16:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.401+0000 7fdb867a4740 10 client.26981 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.401+0000 7fdb867a4740 10 client.26981 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.401+0000 7fdb867a4740 20 client.26981 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.401+0000 7fdb867a4740 1 --2- 172.21.15.44:0/1866251912 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f122e52790 0x55f122e54b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.401+0000 7fdb867a4740 4 client.26981 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:16:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.401+0000 7fdb867a4740 1 -- 172.21.15.44:0/1866251912 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f122cbb7a0 con 0x55f122e52790 2024-04-21T23:16:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.402+0000 7fdb8187b640 1 --2- 172.21.15.44:0/1866251912 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f122e52790 0x55f122e54b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.402+0000 7fdb8187b640 1 --2- 172.21.15.44:0/1866251912 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f122e52790 0x55f122e54b80 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=badc95fbe4861bb5 server_cookie=c39d18c4ef190f70 in_seq=0 out_seq=0 2024-04-21T23:16:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.402+0000 7fdb617fa640 10 client.26981 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:32.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:32 smithi055 ceph-mon[31117]: pgmap v242: 129 pgs: 129 active+clean; 74 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 281 op/s 2024-04-21T23:16:32.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:32 smithi069 ceph-mon[31816]: pgmap v242: 129 pgs: 129 active+clean; 74 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 281 op/s 2024-04-21T23:16:32.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:32 smithi044 ceph-mon[27934]: pgmap v242: 129 pgs: 129 active+clean; 74 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 281 op/s 2024-04-21T23:16:32.708 INFO:tasks.workunit.client.0.smithi044.stdout: 1 889225 11.93 MB/sec execute 262 sec latency 205.510 ms 2024-04-21T23:16:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.765+0000 7fdb617fa640 1 -- 172.21.15.44:0/1866251912 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f122cbb7a0 con 0x55f122e52790 2024-04-21T23:16:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.765+0000 7fdb617fa640 10 client.26981 handle_command_reply: tid=0 2024-04-21T23:16:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.765+0000 7fdb867a4740 1 client.26981 shutdown 2024-04-21T23:16:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.765+0000 7fdb867a4740 1 -- 172.21.15.44:0/1866251912 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122ddb210 msgr2=0x55f122ddf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.765+0000 7fdb867a4740 1 --2- 172.21.15.44:0/1866251912 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122ddb210 0x55f122ddf3f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fdb7409b9f0 tx=0x7fdb740a4700 comp rx=0 tx=0).stop 2024-04-21T23:16:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.765+0000 7fdb867a4740 1 -- 172.21.15.44:0/1866251912 shutdown_connections 2024-04-21T23:16:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.765+0000 7fdb867a4740 1 --2- 172.21.15.44:0/1866251912 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f122e52790 0x55f122e54b80 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.765 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.765+0000 7fdb867a4740 1 --2- 172.21.15.44:0/1866251912 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f122ddb210 0x55f122ddf3f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.765+0000 7fdb867a4740 1 --2- 172.21.15.44:0/1866251912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f122dd2440 0x55f122de3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.765+0000 7fdb617fa640 0 client.26981 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.765+0000 7fdb867a4740 1 --2- 172.21.15.44:0/1866251912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f122dd1b20 0x55f122de2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.765+0000 7fdb867a4740 1 -- 172.21.15.44:0/1866251912 >> 172.21.15.44:0/1866251912 conn(0x55f122dcc440 msgr2=0x55f122dd8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.765+0000 7fdb867a4740 1 -- 172.21.15.44:0/1866251912 shutdown_connections 2024-04-21T23:16:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.765+0000 7fdb867a4740 1 -- 172.21.15.44:0/1866251912 wait complete. 2024-04-21T23:16:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.765+0000 7fdb867a4740 20 client.26981 trim_cache size 0 max 16384 2024-04-21T23:16:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:16:32.766 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:16:32.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:16:32.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:16:32.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.766+0000 7fdb83adf640 1 -- 172.21.15.44:0/788614045 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fdb640a2cf0 msgr2=0x7fdb640a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:32.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.766+0000 7fdb83adf640 1 --2- 172.21.15.44:0/788614045 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fdb640a2cf0 0x7fdb640a51b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdb70004ec0 tx=0x7fdb70005e60 comp rx=0 tx=0).stop 2024-04-21T23:16:32.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.766+0000 7fdb83adf640 1 -- 172.21.15.44:0/788614045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb7c10bd80 msgr2=0x7fdb7c19c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:32.767 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.766+0000 7fdb83adf640 1 --2- 172.21.15.44:0/788614045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb7c10bd80 0x7fdb7c19c8f0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fdb7400e970 tx=0x7fdb7400ee30 comp rx=0 tx=0).stop 2024-04-21T23:16:32.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.766+0000 7fdb83adf640 1 -- 172.21.15.44:0/788614045 shutdown_connections 2024-04-21T23:16:32.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.766+0000 7fdb83adf640 1 --2- 172.21.15.44:0/788614045 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fdb640a2cf0 0x7fdb640a51b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.766+0000 7fdb83adf640 1 --2- 172.21.15.44:0/788614045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fdb7c10bd80 0x7fdb7c19c8f0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.766+0000 7fdb83adf640 1 --2- 172.21.15.44:0/788614045 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb7c1032d0 0x7fdb7c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.766+0000 7fdb83adf640 1 --2- 172.21.15.44:0/788614045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb7c1029b0 0x7fdb7c198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:32.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.766+0000 7fdb83adf640 1 -- 172.21.15.44:0/788614045 >> 172.21.15.44:0/788614045 conn(0x7fdb7c077760 msgr2=0x7fdb7c0fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:32.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.766+0000 7fdb83adf640 1 -- 172.21.15.44:0/788614045 shutdown_connections 2024-04-21T23:16:32.768 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:32.766+0000 7fdb83adf640 1 -- 172.21.15.44:0/788614045 wait complete. 2024-04-21T23:16:33.708 INFO:tasks.workunit.client.0.smithi044.stdout: 1 891684 11.93 MB/sec execute 263 sec latency 234.570 ms 2024-04-21T23:16:34.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:34 smithi044 ceph-mon[27934]: pgmap v243: 129 pgs: 129 active+clean; 82 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.5 MiB/s wr, 316 op/s 2024-04-21T23:16:34.708 INFO:tasks.workunit.client.0.smithi044.stdout: 1 893886 11.93 MB/sec execute 264 sec latency 51.751 ms 2024-04-21T23:16:34.713 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:34 smithi055 ceph-mon[31117]: pgmap v243: 129 pgs: 129 active+clean; 82 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.5 MiB/s wr, 316 op/s 2024-04-21T23:16:34.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:34 smithi069 ceph-mon[31816]: pgmap v243: 129 pgs: 129 active+clean; 82 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.5 MiB/s wr, 316 op/s 2024-04-21T23:16:35.708 INFO:tasks.workunit.client.0.smithi044.stdout: 1 895864 11.92 MB/sec execute 265 sec latency 14.802 ms 2024-04-21T23:16:36.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:36 smithi044 ceph-mon[27934]: pgmap v244: 129 pgs: 129 active+clean; 82 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 151 op/s 2024-04-21T23:16:36.708 INFO:tasks.workunit.client.0.smithi044.stdout: 1 898463 11.94 MB/sec execute 266 sec latency 21.298 ms 2024-04-21T23:16:36.713 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:36 smithi055 ceph-mon[31117]: pgmap v244: 129 pgs: 129 active+clean; 82 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 151 op/s 2024-04-21T23:16:36.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:36 smithi069 ceph-mon[31816]: pgmap v244: 129 pgs: 129 active+clean; 82 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.7 MiB/s wr, 151 op/s 2024-04-21T23:16:37.708 INFO:tasks.workunit.client.0.smithi044.stdout: 1 900025 11.92 MB/sec execute 267 sec latency 34.563 ms 2024-04-21T23:16:38.516 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:38 smithi069 ceph-mon[31816]: pgmap v245: 129 pgs: 129 active+clean; 77 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-04-21T23:16:38.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:38 smithi044 ceph-mon[27934]: pgmap v245: 129 pgs: 129 active+clean; 77 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-04-21T23:16:38.708 INFO:tasks.workunit.client.0.smithi044.stdout: 1 902415 11.92 MB/sec execute 268 sec latency 45.776 ms 2024-04-21T23:16:38.713 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:38 smithi055 ceph-mon[31117]: pgmap v245: 129 pgs: 129 active+clean; 77 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-04-21T23:16:39.708 INFO:tasks.workunit.client.0.smithi044.stdout: 1 904922 11.93 MB/sec execute 269 sec latency 29.214 ms 2024-04-21T23:16:40.515 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:40 smithi055 ceph-mon[31117]: pgmap v246: 129 pgs: 129 active+clean; 74 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 312 op/s 2024-04-21T23:16:40.684 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:40 smithi044 ceph-mon[27934]: pgmap v246: 129 pgs: 129 active+clean; 74 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 312 op/s 2024-04-21T23:16:40.708 INFO:tasks.workunit.client.0.smithi044.stdout: 1 906725 11.93 MB/sec execute 270 sec latency 22.942 ms 2024-04-21T23:16:40.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:40 smithi069 ceph-mon[31816]: pgmap v246: 129 pgs: 129 active+clean; 74 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 312 op/s 2024-04-21T23:16:41.708 INFO:tasks.workunit.client.0.smithi044.stdout: 1 909087 11.93 MB/sec execute 271 sec latency 31.110 ms 2024-04-21T23:16:42.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:42 smithi044 ceph-mon[27934]: pgmap v247: 129 pgs: 129 active+clean; 74 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 276 op/s 2024-04-21T23:16:42.708 INFO:tasks.workunit.client.0.smithi044.stdout: 1 910930 11.91 MB/sec execute 272 sec latency 18.474 ms 2024-04-21T23:16:42.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:42 smithi069 ceph-mon[31816]: pgmap v247: 129 pgs: 129 active+clean; 74 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 276 op/s 2024-04-21T23:16:42.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:42 smithi055 ceph-mon[31117]: pgmap v247: 129 pgs: 129 active+clean; 74 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 276 op/s 2024-04-21T23:16:43.708 INFO:tasks.workunit.client.0.smithi044.stdout: 1 913121 11.92 MB/sec execute 273 sec latency 18.020 ms 2024-04-21T23:16:44.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:44 smithi044 ceph-mon[27934]: pgmap v248: 129 pgs: 129 active+clean; 82 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 347 op/s 2024-04-21T23:16:44.708 INFO:tasks.workunit.client.0.smithi044.stdout: 1 915731 11.93 MB/sec execute 274 sec latency 35.902 ms 2024-04-21T23:16:44.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:44 smithi069 ceph-mon[31816]: pgmap v248: 129 pgs: 129 active+clean; 82 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 347 op/s 2024-04-21T23:16:44.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:44 smithi055 ceph-mon[31117]: pgmap v248: 129 pgs: 129 active+clean; 82 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 347 op/s 2024-04-21T23:16:45.709 INFO:tasks.workunit.client.0.smithi044.stdout: 1 917141 11.96 MB/sec execute 275 sec latency 60.370 ms 2024-04-21T23:16:45.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.872+0000 7fc2d6e88640 1 -- 172.21.15.44:0/3985394140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2d01006a0 msgr2=0x7fc2d0100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:45.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.872+0000 7fc2d6e88640 1 --2- 172.21.15.44:0/3985394140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2d01006a0 0x7fc2d0100b20 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc009930 tx=0x7fc2bc02bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:45.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.872+0000 7fc2d6e88640 1 -- 172.21.15.44:0/3985394140 shutdown_connections 2024-04-21T23:16:45.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.872+0000 7fc2d6e88640 1 --2- 172.21.15.44:0/3985394140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2d0101060 0x7fc2d010d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.872+0000 7fc2d6e88640 1 --2- 172.21.15.44:0/3985394140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2d01006a0 0x7fc2d0100b20 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.872+0000 7fc2d6e88640 1 --2- 172.21.15.44:0/3985394140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2d0106660 0x7fc2d0106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.873 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.872+0000 7fc2d6e88640 1 -- 172.21.15.44:0/3985394140 >> 172.21.15.44:0/3985394140 conn(0x7fc2d0077790 msgr2=0x7fc2d0077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:45.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.872+0000 7fc2d6e88640 1 -- 172.21.15.44:0/3985394140 shutdown_connections 2024-04-21T23:16:45.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.872+0000 7fc2d6e88640 1 -- 172.21.15.44:0/3985394140 wait complete. 2024-04-21T23:16:45.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.873+0000 7fc2d6e88640 1 Processor -- start 2024-04-21T23:16:45.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.873+0000 7fc2d6e88640 1 -- start start 2024-04-21T23:16:45.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.873+0000 7fc2d6e88640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2d01006a0 0x7fc2d0198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:45.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.873+0000 7fc2d6e88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2d0101060 0x7fc2d0198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:45.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.873+0000 7fc2d6e88640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2d0106660 0x7fc2d019c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:45.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.873+0000 7fc2d6e88640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d0110a70 con 0x7fc2d0106660 2024-04-21T23:16:45.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.873+0000 7fc2d6e88640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d01108f0 con 0x7fc2d01006a0 2024-04-21T23:16:45.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.873+0000 7fc2d6e88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d0110bf0 con 0x7fc2d0101060 2024-04-21T23:16:45.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.873+0000 7fc2d5425640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2d0106660 0x7fc2d019c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:45.874 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.873+0000 7fc2d4c24640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2d01006a0 0x7fc2d0198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:45.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.873+0000 7fc2d5425640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2d0106660 0x7fc2d019c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49348/0 (socket says 172.21.15.44:49348) 2024-04-21T23:16:45.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.873+0000 7fc2d5425640 1 -- 172.21.15.44:0/1921606022 learned_addr learned my addr 172.21.15.44:0/1921606022 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:45.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.873+0000 7fc2cffff640 1 --2- 172.21.15.44:0/1921606022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2d0101060 0x7fc2d0198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:45.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.874+0000 7fc2d4c24640 1 -- 172.21.15.44:0/1921606022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2d0101060 msgr2=0x7fc2d0198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:45.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.874+0000 7fc2d4c24640 1 --2- 172.21.15.44:0/1921606022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2d0101060 0x7fc2d0198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.874+0000 7fc2d4c24640 1 -- 172.21.15.44:0/1921606022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2d0106660 msgr2=0x7fc2d019c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:45.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.874+0000 7fc2d4c24640 1 --2- 172.21.15.44:0/1921606022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2d0106660 0x7fc2d019c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.874+0000 7fc2d4c24640 1 -- 172.21.15.44:0/1921606022 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d019d010 con 0x7fc2d01006a0 2024-04-21T23:16:45.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.874+0000 7fc2d5425640 1 --2- 172.21.15.44:0/1921606022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2d0106660 0x7fc2d019c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:45.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.874+0000 7fc2d4c24640 1 --2- 172.21.15.44:0/1921606022 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2d01006a0 0x7fc2d0198020 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc2c400e9f0 tx=0x7fc2c400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:45.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.874+0000 7fc2cdffb640 1 -- 172.21.15.44:0/1921606022 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c400ccc0 con 0x7fc2d01006a0 2024-04-21T23:16:45.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.874+0000 7fc2cdffb640 1 -- 172.21.15.44:0/1921606022 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2c4010910 con 0x7fc2d01006a0 2024-04-21T23:16:45.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.874+0000 7fc2d6e88640 1 -- 172.21.15.44:0/1921606022 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2d019d300 con 0x7fc2d01006a0 2024-04-21T23:16:45.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.874+0000 7fc2cdffb640 1 -- 172.21.15.44:0/1921606022 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4018960 con 0x7fc2d01006a0 2024-04-21T23:16:45.875 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.874+0000 7fc2d6e88640 1 -- 172.21.15.44:0/1921606022 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2d01a4b90 con 0x7fc2d01006a0 2024-04-21T23:16:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.876+0000 7fc2cdffb640 1 -- 172.21.15.44:0/1921606022 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc2c4010ac0 con 0x7fc2d01006a0 2024-04-21T23:16:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.877+0000 7fc2d9a73740 1 Processor -- start 2024-04-21T23:16:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.877+0000 7fc2d9a73740 1 -- start start 2024-04-21T23:16:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.877+0000 7fc2d9a73740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a482b20 0x56491a482f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.877+0000 7fc2d9a73740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a483440 0x56491a48bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.877+0000 7fc2d9a73740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a48c210 0x56491a48e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.877+0000 7fc2d9a73740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56491a3e6b70 con 0x56491a48c210 2024-04-21T23:16:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.877+0000 7fc2d9a73740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x56491a38f470 con 0x56491a483440 2024-04-21T23:16:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.877+0000 7fc2d9a73740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56491a3e2b30 con 0x56491a482b20 2024-04-21T23:16:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.877+0000 7fc2cdffb640 1 --2- 172.21.15.44:0/1921606022 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc2c00a2cf0 0x7fc2c00a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.877+0000 7fc2d4c24640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a483440 0x56491a48bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.877+0000 7fc2d4c24640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a483440 0x56491a48bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:45020/0 (socket says 172.21.15.44:45020) 2024-04-21T23:16:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.877+0000 7fc2d4c24640 1 -- 172.21.15.44:0/2938536407 learned_addr learned my addr 172.21.15.44:0/2938536407 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.877+0000 7fc2d5425640 1 --2- 172.21.15.44:0/2938536407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a482b20 0x56491a482f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:45.884 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.877+0000 7fc2cffff640 1 --2- 172.21.15.44:0/2938536407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a48c210 0x56491a48e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:45.884 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.877+0000 7fc2d5425640 1 --2- 172.21.15.44:0/1921606022 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc2c00a2cf0 0x7fc2c00a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:45.884 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.877+0000 7fc2cdffb640 1 -- 172.21.15.44:0/1921606022 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7fc2c40cfb00 con 0x7fc2d01006a0 2024-04-21T23:16:45.884 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.878+0000 7fc2d4c24640 1 -- 172.21.15.44:0/2938536407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a482b20 msgr2=0x56491a482f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:45.884 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.878+0000 7fc2d4c24640 1 --2- 172.21.15.44:0/2938536407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a482b20 0x56491a482f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.884 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.878+0000 7fc2d4c24640 1 -- 172.21.15.44:0/2938536407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a48c210 msgr2=0x56491a48e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:45.884 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.878+0000 7fc2d4c24640 1 --2- 172.21.15.44:0/2938536407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a48c210 0x56491a48e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.884 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.878+0000 7fc2d4c24640 1 -- 172.21.15.44:0/2938536407 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56491a0fb210 con 0x56491a483440 2024-04-21T23:16:45.884 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.878+0000 7fc2d5425640 1 --2- 172.21.15.44:0/1921606022 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc2c00a2cf0 0x7fc2c00a51b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc2c80049d0 tx=0x7fc2c8009500 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:45.884 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.878+0000 7fc2d4c24640 1 --2- 172.21.15.44:0/2938536407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a483440 0x56491a48bcd0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc2c400c6a0 tx=0x7fc2c400cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e1cc703ae2a5cc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:45.884 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.878+0000 7fc2baffd640 1 -- 172.21.15.44:0/2938536407 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4026040 con 0x56491a483440 2024-04-21T23:16:45.884 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.878+0000 7fc2baffd640 1 -- 172.21.15.44:0/2938536407 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2c40a0030 con 0x56491a483440 2024-04-21T23:16:45.884 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.878+0000 7fc2baffd640 1 -- 172.21.15.44:0/2938536407 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c40983a0 con 0x56491a483440 2024-04-21T23:16:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.879+0000 7fc2d9a73740 1 -- 172.21.15.44:0/2938536407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a483440 msgr2=0x56491a48bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.879+0000 7fc2d9a73740 1 --2- 172.21.15.44:0/2938536407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a483440 0x56491a48bcd0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc2c400c6a0 tx=0x7fc2c400cf60 comp rx=0 tx=0).stop 2024-04-21T23:16:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.879+0000 7fc2d9a73740 1 -- 172.21.15.44:0/2938536407 shutdown_connections 2024-04-21T23:16:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.879+0000 7fc2d9a73740 1 --2- 172.21.15.44:0/2938536407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a48c210 0x56491a48e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.879+0000 7fc2d9a73740 1 --2- 172.21.15.44:0/2938536407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a483440 0x56491a48bcd0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.879+0000 7fc2d9a73740 1 --2- 172.21.15.44:0/2938536407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a482b20 0x56491a482f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.879+0000 7fc2d9a73740 1 -- 172.21.15.44:0/2938536407 >> 172.21.15.44:0/2938536407 conn(0x56491a47d440 msgr2=0x56491a47d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.879+0000 7fc2d9a73740 1 -- 172.21.15.44:0/2938536407 shutdown_connections 2024-04-21T23:16:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.879+0000 7fc2d9a73740 1 -- 172.21.15.44:0/2938536407 wait complete. 2024-04-21T23:16:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.879+0000 7fc2d9a73740 1 -- start start 2024-04-21T23:16:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.879+0000 7fc2d9a73740 1 Processor -- start 2024-04-21T23:16:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.880+0000 7fc2d9a73740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a482b20 0x56491a4892c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.880+0000 7fc2d9a73740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 0x56491a489800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.880+0000 7fc2d9a73740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a48c210 0x56491a489d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.880+0000 7fc2d9a73740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56491a48ed80 con 0x56491a483440 2024-04-21T23:16:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.880+0000 7fc2d9a73740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x56491a3a2a10 con 0x56491a48c210 2024-04-21T23:16:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.880+0000 7fc2d9a73740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56491a48ef00 con 0x56491a482b20 2024-04-21T23:16:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.880+0000 7fc2d9a73740 10 client.0 _ll_register_callbacks cb 0x56491a3ddb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.880+0000 7fc2cffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 0x56491a489800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.880+0000 7fc2cffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 0x56491a489800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49362/0 (socket says 172.21.15.44:49362) 2024-04-21T23:16:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.880+0000 7fc2cffff640 1 -- 172.21.15.44:0/652476481 learned_addr learned my addr 172.21.15.44:0/652476481 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.880+0000 7fc2d4c24640 1 --2- 172.21.15.44:0/652476481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a482b20 0x56491a4892c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.880+0000 7fc2d5425640 1 --2- 172.21.15.44:0/652476481 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a48c210 0x56491a489d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.880+0000 7fc2cffff640 1 -- 172.21.15.44:0/652476481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a482b20 msgr2=0x56491a4892c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.880+0000 7fc2cffff640 1 --2- 172.21.15.44:0/652476481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a482b20 0x56491a4892c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.880+0000 7fc2cffff640 1 -- 172.21.15.44:0/652476481 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a48c210 msgr2=0x56491a489d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:45.886 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.880+0000 7fc2cffff640 1 --2- 172.21.15.44:0/652476481 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a48c210 0x56491a489d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.880+0000 7fc2cffff640 1 -- 172.21.15.44:0/652476481 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56491a1c7620 con 0x56491a483440 2024-04-21T23:16:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.880+0000 7fc2d5425640 1 --2- 172.21.15.44:0/652476481 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a48c210 0x56491a489d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.880+0000 7fc2cffff640 1 --2- 172.21.15.44:0/652476481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 0x56491a489800 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc02cfd0 tx=0x7fc2bc02cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e9c844250ed99f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.880+0000 7fc2affff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:16:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.881+0000 7fc2affff640 1 -- 172.21.15.44:0/652476481 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc004240 con 0x56491a483440 2024-04-21T23:16:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.881+0000 7fc2affff640 1 -- 172.21.15.44:0/652476481 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7fc2bc0043e0 con 0x56491a483440 2024-04-21T23:16:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.881+0000 7fc2d9a73740 1 -- 172.21.15.44:0/652476481 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2c40a0030 con 0x56491a483440 2024-04-21T23:16:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.881+0000 7fc2affff640 1 -- 172.21.15.44:0/652476481 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc2bc0046c0 con 0x56491a483440 2024-04-21T23:16:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.881+0000 7fc2d9a73740 10 client.17430 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.881+0000 7fc2d9a73740 1 -- 172.21.15.44:0/652476481 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56491a49b070 con 0x56491a483440 2024-04-21T23:16:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.881+0000 7fc2affff640 1 -- 172.21.15.44:0/652476481 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2c40a0030 con 0x56491a483440 2024-04-21T23:16:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.881+0000 7fc2affff640 1 -- 172.21.15.44:0/652476481 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7fc2bc0035f0 con 0x56491a483440 2024-04-21T23:16:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.881+0000 7fc2d9a73740 10 client.17430 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.881+0000 7fc2d9a73740 10 client.17430 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:45.887 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.881+0000 7fc2d9a73740 20 client.17430 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.881+0000 7fc2d9a73740 1 --2- 172.21.15.44:0/652476481 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56491a49c580 0x56491a49e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.881+0000 7fc2d9a73740 4 client.17430 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:16:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.881+0000 7fc2d9a73740 1 -- 172.21.15.44:0/652476481 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56491a3f40b0 con 0x56491a49c580 2024-04-21T23:16:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.882+0000 7fc2d4c24640 1 --2- 172.21.15.44:0/652476481 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56491a49c580 0x56491a49e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.882+0000 7fc2d4c24640 1 --2- 172.21.15.44:0/652476481 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56491a49c580 0x56491a49e9d0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3bb127ece8d6d53 server_cookie=f8aa6a1e95941985 in_seq=0 out_seq=0 2024-04-21T23:16:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.882+0000 7fc2affff640 10 client.17430 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.883+0000 7fc2affff640 1 -- 172.21.15.44:0/652476481 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56491a3f40b0 con 0x56491a49c580 2024-04-21T23:16:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.883+0000 7fc2affff640 10 client.17430 handle_command_reply: tid=0 2024-04-21T23:16:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.883+0000 7fc2d9a73740 1 client.17430 shutdown 2024-04-21T23:16:45.888 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.883+0000 7fc2d9a73740 1 -- 172.21.15.44:0/652476481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 msgr2=0x56491a489800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:45.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.883+0000 7fc2d9a73740 1 --2- 172.21.15.44:0/652476481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 0x56491a489800 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc02cfd0 tx=0x7fc2bc02cda0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.883+0000 7fc2d9a73740 1 -- 172.21.15.44:0/652476481 shutdown_connections 2024-04-21T23:16:45.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.883+0000 7fc2d9a73740 1 --2- 172.21.15.44:0/652476481 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56491a49c580 0x56491a49e9d0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.883+0000 7fc2d9a73740 1 --2- 172.21.15.44:0/652476481 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a48c210 0x56491a489d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.883+0000 7fc2d9a73740 1 --2- 172.21.15.44:0/652476481 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 0x56491a489800 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.883+0000 7fc2affff640 0 client.17430 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:45.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.883+0000 7fc2d9a73740 1 --2- 172.21.15.44:0/652476481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a482b20 0x56491a4892c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.883+0000 7fc2d9a73740 1 -- 172.21.15.44:0/652476481 >> 172.21.15.44:0/652476481 conn(0x56491a47d440 msgr2=0x56491a47f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:45.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.883+0000 7fc2d9a73740 1 -- 172.21.15.44:0/652476481 shutdown_connections 2024-04-21T23:16:45.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.883+0000 7fc2d9a73740 1 -- 172.21.15.44:0/652476481 wait complete. 2024-04-21T23:16:45.889 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.883+0000 7fc2d9a73740 20 client.17430 trim_cache size 0 max 16384 2024-04-21T23:16:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.899+0000 7fc2d9a73740 1 Processor -- start 2024-04-21T23:16:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.899+0000 7fc2d9a73740 1 -- start start 2024-04-21T23:16:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.899+0000 7fc2d9a73740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a482b20 0x56491a48b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.899+0000 7fc2d9a73740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 0x56491a497c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.899+0000 7fc2d9a73740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a48c210 0x56491a487b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.899+0000 7fc2d9a73740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d01acca0 con 0x56491a483440 2024-04-21T23:16:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.899+0000 7fc2d9a73740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d01af520 con 0x56491a482b20 2024-04-21T23:16:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.899+0000 7fc2d9a73740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d01ad310 con 0x56491a48c210 2024-04-21T23:16:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.900+0000 7fc2cffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 0x56491a497c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.900+0000 7fc2d5425640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a48c210 0x56491a487b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.900+0000 7fc2d4c24640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a482b20 0x56491a48b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.900+0000 7fc2cffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 0x56491a497c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49376/0 (socket says 172.21.15.44:49376) 2024-04-21T23:16:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.900+0000 7fc2cffff640 1 -- 172.21.15.44:0/299522073 learned_addr learned my addr 172.21.15.44:0/299522073 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.900+0000 7fc2d5425640 1 -- 172.21.15.44:0/299522073 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a482b20 msgr2=0x56491a48b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.900+0000 7fc2d5425640 1 --2- 172.21.15.44:0/299522073 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a482b20 0x56491a48b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.900+0000 7fc2d5425640 1 -- 172.21.15.44:0/299522073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 msgr2=0x56491a497c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.900+0000 7fc2d5425640 1 --2- 172.21.15.44:0/299522073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 0x56491a497c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.900+0000 7fc2d5425640 1 -- 172.21.15.44:0/299522073 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56491a488b50 con 0x56491a48c210 2024-04-21T23:16:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.900+0000 7fc2cffff640 1 --2- 172.21.15.44:0/299522073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 0x56491a497c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.900+0000 7fc2d4c24640 1 --2- 172.21.15.44:0/299522073 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a482b20 0x56491a48b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.900+0000 7fc2d5425640 1 --2- 172.21.15.44:0/299522073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a48c210 0x56491a487b90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8021820 tx=0x7fc2c8016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4eff7fd79447926c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.900+0000 7fc2ba7fc640 1 -- 172.21.15.44:0/299522073 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c8021e60 con 0x56491a48c210 2024-04-21T23:16:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.900+0000 7fc2ba7fc640 1 -- 172.21.15.44:0/299522073 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2c801f7f0 con 0x56491a48c210 2024-04-21T23:16:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.900+0000 7fc2ba7fc640 1 -- 172.21.15.44:0/299522073 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c8005350 con 0x56491a48c210 2024-04-21T23:16:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.901+0000 7fc2d9a73740 1 -- 172.21.15.44:0/299522073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a48c210 msgr2=0x56491a487b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.901+0000 7fc2d9a73740 1 --2- 172.21.15.44:0/299522073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a48c210 0x56491a487b90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8021820 tx=0x7fc2c8016000 comp rx=0 tx=0).stop 2024-04-21T23:16:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.901+0000 7fc2d9a73740 1 -- 172.21.15.44:0/299522073 shutdown_connections 2024-04-21T23:16:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.901+0000 7fc2d9a73740 1 --2- 172.21.15.44:0/299522073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a48c210 0x56491a487b90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.901+0000 7fc2d9a73740 1 --2- 172.21.15.44:0/299522073 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 0x56491a497c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.901+0000 7fc2d9a73740 1 --2- 172.21.15.44:0/299522073 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a482b20 0x56491a48b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.901+0000 7fc2d9a73740 1 -- 172.21.15.44:0/299522073 >> 172.21.15.44:0/299522073 conn(0x56491a47d440 msgr2=0x56491a483ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.901+0000 7fc2d9a73740 1 -- 172.21.15.44:0/299522073 shutdown_connections 2024-04-21T23:16:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.901+0000 7fc2d9a73740 1 -- 172.21.15.44:0/299522073 wait complete. 2024-04-21T23:16:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.901+0000 7fc2d9a73740 1 -- start start 2024-04-21T23:16:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.901+0000 7fc2d9a73740 1 Processor -- start 2024-04-21T23:16:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.902+0000 7fc2d9a73740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a482b20 0x56491a494070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.902+0000 7fc2d9a73740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 0x56491a4945b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.902+0000 7fc2d9a73740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a48c210 0x56491a48f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.902+0000 7fc2d9a73740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d01ad9e0 con 0x56491a483440 2024-04-21T23:16:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.902+0000 7fc2d9a73740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d01aee50 con 0x56491a482b20 2024-04-21T23:16:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.902+0000 7fc2d9a73740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d01ae0b0 con 0x56491a48c210 2024-04-21T23:16:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.902+0000 7fc2d9a73740 10 client.0 _ll_register_callbacks cb 0x7fc2bc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.902+0000 7fc2cffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 0x56491a4945b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.902+0000 7fc2cffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 0x56491a4945b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49380/0 (socket says 172.21.15.44:49380) 2024-04-21T23:16:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.902+0000 7fc2cffff640 1 -- 172.21.15.44:0/2018757745 learned_addr learned my addr 172.21.15.44:0/2018757745 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.902+0000 7fc2d4c24640 1 --2- 172.21.15.44:0/2018757745 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a482b20 0x56491a494070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.902+0000 7fc2d5425640 1 --2- 172.21.15.44:0/2018757745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a48c210 0x56491a48f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.902+0000 7fc2cffff640 1 -- 172.21.15.44:0/2018757745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a48c210 msgr2=0x56491a48f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.902+0000 7fc2cffff640 1 --2- 172.21.15.44:0/2018757745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a48c210 0x56491a48f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.905 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.902+0000 7fc2cffff640 1 -- 172.21.15.44:0/2018757745 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a482b20 msgr2=0x56491a494070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.902+0000 7fc2cffff640 1 --2- 172.21.15.44:0/2018757745 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a482b20 0x56491a494070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.902+0000 7fc2cffff640 1 -- 172.21.15.44:0/2018757745 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56491a48fd90 con 0x56491a483440 2024-04-21T23:16:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.902+0000 7fc2d4c24640 1 --2- 172.21.15.44:0/2018757745 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a482b20 0x56491a494070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.902+0000 7fc2d5425640 1 --2- 172.21.15.44:0/2018757745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a48c210 0x56491a48f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.903+0000 7fc2cffff640 1 --2- 172.21.15.44:0/2018757745 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 0x56491a4945b0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc003c00 tx=0x7fc2bc005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea1a6d143178860c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.903+0000 7fc2affff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:16:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.903+0000 7fc2affff640 1 -- 172.21.15.44:0/2018757745 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc05b070 con 0x56491a483440 2024-04-21T23:16:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.903+0000 7fc2affff640 1 -- 172.21.15.44:0/2018757745 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7fc2bc005150 con 0x56491a483440 2024-04-21T23:16:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.903+0000 7fc2d9a73740 1 -- 172.21.15.44:0/2018757745 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2c801f7f0 con 0x56491a483440 2024-04-21T23:16:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.903+0000 7fc2affff640 1 -- 172.21.15.44:0/2018757745 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc2bc005d50 con 0x56491a483440 2024-04-21T23:16:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.903+0000 7fc2d9a73740 10 client.17436 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.903+0000 7fc2d9a73740 1 -- 172.21.15.44:0/2018757745 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56491a502720 con 0x56491a483440 2024-04-21T23:16:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.903+0000 7fc2affff640 1 -- 172.21.15.44:0/2018757745 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2c801f7f0 con 0x56491a483440 2024-04-21T23:16:45.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.904+0000 7fc2affff640 1 -- 172.21.15.44:0/2018757745 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7fc2bc003d10 con 0x56491a483440 2024-04-21T23:16:45.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.904+0000 7fc2d9a73740 10 client.17436 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:45.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.904+0000 7fc2d9a73740 10 client.17436 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:45.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.904+0000 7fc2d9a73740 20 client.17436 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:45.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.904+0000 7fc2d9a73740 1 --2- 172.21.15.44:0/2018757745 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56491a502d40 0x56491a505160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:45.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.904+0000 7fc2d9a73740 4 client.17436 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:16:45.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.904+0000 7fc2d9a73740 1 -- 172.21.15.44:0/2018757745 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56491a3ee400 con 0x56491a502d40 2024-04-21T23:16:45.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.904+0000 7fc2d4c24640 1 --2- 172.21.15.44:0/2018757745 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56491a502d40 0x56491a505160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:45.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.904+0000 7fc2affff640 10 client.17436 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:45.907 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:45.904+0000 7fc2d4c24640 1 --2- 172.21.15.44:0/2018757745 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56491a502d40 0x56491a505160 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17240ac5a1ee86fa server_cookie=6364cb8bcf4ce705 in_seq=0 out_seq=0 2024-04-21T23:16:46.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.391+0000 7fc2affff640 1 -- 172.21.15.44:0/2018757745 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56491a3ee400 con 0x56491a502d40 2024-04-21T23:16:46.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.391+0000 7fc2affff640 10 client.17436 handle_command_reply: tid=0 2024-04-21T23:16:46.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.391+0000 7fc2d9a73740 1 client.17436 shutdown 2024-04-21T23:16:46.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.391+0000 7fc2d9a73740 1 -- 172.21.15.44:0/2018757745 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 msgr2=0x56491a4945b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:46.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.391+0000 7fc2d9a73740 1 --2- 172.21.15.44:0/2018757745 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 0x56491a4945b0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc003c00 tx=0x7fc2bc005850 comp rx=0 tx=0).stop 2024-04-21T23:16:46.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.392+0000 7fc2d9a73740 1 -- 172.21.15.44:0/2018757745 shutdown_connections 2024-04-21T23:16:46.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.392+0000 7fc2d9a73740 1 --2- 172.21.15.44:0/2018757745 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56491a502d40 0x56491a505160 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:46.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.392+0000 7fc2d9a73740 1 --2- 172.21.15.44:0/2018757745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56491a48c210 0x56491a48f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:46.391 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.392+0000 7fc2affff640 0 client.17436 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:46.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.392+0000 7fc2d9a73740 1 --2- 172.21.15.44:0/2018757745 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56491a483440 0x56491a4945b0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:46.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.392+0000 7fc2d9a73740 1 --2- 172.21.15.44:0/2018757745 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56491a482b20 0x56491a494070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:46.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.392+0000 7fc2d9a73740 1 -- 172.21.15.44:0/2018757745 >> 172.21.15.44:0/2018757745 conn(0x56491a47d440 msgr2=0x56491a487ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:46.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.392+0000 7fc2d9a73740 1 -- 172.21.15.44:0/2018757745 shutdown_connections 2024-04-21T23:16:46.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.392+0000 7fc2d9a73740 1 -- 172.21.15.44:0/2018757745 wait complete. 2024-04-21T23:16:46.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.392+0000 7fc2d9a73740 20 client.17436 trim_cache size 0 max 16384 2024-04-21T23:16:46.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:16:46.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:16:46.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:16:46.392 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:16:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.393+0000 7fc2d6e88640 1 -- 172.21.15.44:0/1921606022 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc2c00a2cf0 msgr2=0x7fc2c00a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.393+0000 7fc2d6e88640 1 --2- 172.21.15.44:0/1921606022 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc2c00a2cf0 0x7fc2c00a51b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc2c80049d0 tx=0x7fc2c8009500 comp rx=0 tx=0).stop 2024-04-21T23:16:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.393+0000 7fc2d6e88640 1 -- 172.21.15.44:0/1921606022 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2d01006a0 msgr2=0x7fc2d0198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.393+0000 7fc2d6e88640 1 --2- 172.21.15.44:0/1921606022 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2d01006a0 0x7fc2d0198020 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc2c400e9f0 tx=0x7fc2c400eeb0 comp rx=0 tx=0).stop 2024-04-21T23:16:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.393+0000 7fc2d6e88640 1 -- 172.21.15.44:0/1921606022 shutdown_connections 2024-04-21T23:16:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.393+0000 7fc2d6e88640 1 --2- 172.21.15.44:0/1921606022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2d0101060 0x7fc2d0198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.393+0000 7fc2d6e88640 1 --2- 172.21.15.44:0/1921606022 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc2c00a2cf0 0x7fc2c00a51b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.393+0000 7fc2d6e88640 1 --2- 172.21.15.44:0/1921606022 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc2d0106660 0x7fc2d019c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.393+0000 7fc2d6e88640 1 --2- 172.21.15.44:0/1921606022 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc2d01006a0 0x7fc2d0198020 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:46.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.393+0000 7fc2d6e88640 1 -- 172.21.15.44:0/1921606022 >> 172.21.15.44:0/1921606022 conn(0x7fc2d0077790 msgr2=0x7fc2d006a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:46.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.393+0000 7fc2d6e88640 1 -- 172.21.15.44:0/1921606022 shutdown_connections 2024-04-21T23:16:46.394 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:46.393+0000 7fc2d6e88640 1 -- 172.21.15.44:0/1921606022 wait complete. 2024-04-21T23:16:46.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:46 smithi044 ceph-mon[27934]: pgmap v249: 129 pgs: 129 active+clean; 82 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 277 op/s 2024-04-21T23:16:46.708 INFO:tasks.workunit.client.0.smithi044.stdout: 1 918211 11.95 MB/sec execute 276 sec latency 231.932 ms 2024-04-21T23:16:46.713 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:46 smithi055 ceph-mon[31117]: pgmap v249: 129 pgs: 129 active+clean; 82 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 277 op/s 2024-04-21T23:16:46.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:46 smithi069 ceph-mon[31816]: pgmap v249: 129 pgs: 129 active+clean; 82 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 277 op/s 2024-04-21T23:16:47.708 INFO:tasks.workunit.client.0.smithi044.stdout: 1 920647 11.96 MB/sec execute 277 sec latency 48.426 ms 2024-04-21T23:16:48.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:48 smithi044 ceph-mon[27934]: pgmap v250: 129 pgs: 129 active+clean; 78 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 351 op/s 2024-04-21T23:16:48.708 INFO:tasks.workunit.client.0.smithi044.stdout: 1 922837 11.94 MB/sec execute 278 sec latency 18.871 ms 2024-04-21T23:16:48.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:48 smithi069 ceph-mon[31816]: pgmap v250: 129 pgs: 129 active+clean; 78 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 351 op/s 2024-04-21T23:16:48.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:48 smithi055 ceph-mon[31117]: pgmap v250: 129 pgs: 129 active+clean; 78 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 351 op/s 2024-04-21T23:16:49.708 INFO:tasks.workunit.client.0.smithi044.stdout: 1 925064 11.95 MB/sec execute 279 sec latency 24.709 ms 2024-04-21T23:16:50.485 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:50 smithi044 ceph-mon[27934]: pgmap v251: 129 pgs: 129 active+clean; 76 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 398 op/s 2024-04-21T23:16:50.709 INFO:tasks.workunit.client.0.smithi044.stdout: 1 927769 11.97 MB/sec execute 280 sec latency 13.453 ms 2024-04-21T23:16:50.713 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:50 smithi055 ceph-mon[31117]: pgmap v251: 129 pgs: 129 active+clean; 76 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 398 op/s 2024-04-21T23:16:50.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:50 smithi069 ceph-mon[31816]: pgmap v251: 129 pgs: 129 active+clean; 76 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 398 op/s 2024-04-21T23:16:51.709 INFO:tasks.workunit.client.0.smithi044.stdout: 1 929730 11.95 MB/sec execute 281 sec latency 151.325 ms 2024-04-21T23:16:52.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:52 smithi044 ceph-mon[27934]: pgmap v252: 129 pgs: 129 active+clean; 76 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 282 op/s 2024-04-21T23:16:52.708 INFO:tasks.workunit.client.0.smithi044.stdout: 1 931436 11.95 MB/sec execute 282 sec latency 47.529 ms 2024-04-21T23:16:52.713 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:52 smithi055 ceph-mon[31117]: pgmap v252: 129 pgs: 129 active+clean; 76 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 282 op/s 2024-04-21T23:16:52.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:52 smithi069 ceph-mon[31816]: pgmap v252: 129 pgs: 129 active+clean; 76 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 282 op/s 2024-04-21T23:16:53.709 INFO:tasks.workunit.client.0.smithi044.stdout: 1 933783 11.95 MB/sec execute 283 sec latency 15.465 ms 2024-04-21T23:16:54.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:54 smithi044 ceph-mon[27934]: pgmap v253: 129 pgs: 129 active+clean; 86 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.7 MiB/s wr, 387 op/s 2024-04-21T23:16:54.709 INFO:tasks.workunit.client.0.smithi044.stdout: 1 935819 11.95 MB/sec execute 284 sec latency 30.296 ms 2024-04-21T23:16:54.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:54 smithi055 ceph-mon[31117]: pgmap v253: 129 pgs: 129 active+clean; 86 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.7 MiB/s wr, 387 op/s 2024-04-21T23:16:54.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:54 smithi069 ceph-mon[31816]: pgmap v253: 129 pgs: 129 active+clean; 86 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.7 MiB/s wr, 387 op/s 2024-04-21T23:16:55.709 INFO:tasks.workunit.client.0.smithi044.stdout: 1 938415 11.96 MB/sec execute 285 sec latency 14.022 ms 2024-04-21T23:16:56.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:56 smithi044 ceph-mon[27934]: pgmap v254: 129 pgs: 129 active+clean; 86 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 317 op/s 2024-04-21T23:16:56.709 INFO:tasks.workunit.client.0.smithi044.stdout: 1 939906 11.94 MB/sec execute 286 sec latency 126.496 ms 2024-04-21T23:16:56.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:56 smithi069 ceph-mon[31816]: pgmap v254: 129 pgs: 129 active+clean; 86 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 317 op/s 2024-04-21T23:16:56.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:56 smithi055 ceph-mon[31117]: pgmap v254: 129 pgs: 129 active+clean; 86 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 317 op/s 2024-04-21T23:16:56.784 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:16:56.785 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:16:56.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.952+0000 7f3c6cbb8640 1 -- 172.21.15.44:0/3517113960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c68107720 msgr2=0x7f3c68107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:56.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.952+0000 7f3c6cbb8640 1 --2- 172.21.15.44:0/3517113960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c68107720 0x7f3c68107b00 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c009930 tx=0x7f3c5c02bc20 comp rx=0 tx=0).stop 2024-04-21T23:16:56.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.954+0000 7f3c6cbb8640 1 -- 172.21.15.44:0/3517113960 shutdown_connections 2024-04-21T23:16:56.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.954+0000 7f3c6cbb8640 1 --2- 172.21.15.44:0/3517113960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c68101710 0x7f3c68101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:56.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.954+0000 7f3c6cbb8640 1 --2- 172.21.15.44:0/3517113960 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c681020d0 0x7f3c6810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:56.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.954+0000 7f3c6cbb8640 1 --2- 172.21.15.44:0/3517113960 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c68107720 0x7f3c68107b00 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:56.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.954+0000 7f3c6cbb8640 1 -- 172.21.15.44:0/3517113960 >> 172.21.15.44:0/3517113960 conn(0x7f3c680fd200 msgr2=0x7f3c680ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:56.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.954+0000 7f3c6cbb8640 1 -- 172.21.15.44:0/3517113960 shutdown_connections 2024-04-21T23:16:56.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.954+0000 7f3c6cbb8640 1 -- 172.21.15.44:0/3517113960 wait complete. 2024-04-21T23:16:56.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.955+0000 7f3c6cbb8640 1 Processor -- start 2024-04-21T23:16:56.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.956+0000 7f3c6cbb8640 1 -- start start 2024-04-21T23:16:56.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.956+0000 7f3c6cbb8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c68101710 0x7f3c68195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:56.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.956+0000 7f3c6cbb8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c681020d0 0x7f3c681963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:56.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.956+0000 7f3c6cbb8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c68107720 0x7f3c6819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:56.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.956+0000 7f3c6cbb8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6810f980 con 0x7f3c68107720 2024-04-21T23:16:56.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.956+0000 7f3c6cbb8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6810f800 con 0x7f3c681020d0 2024-04-21T23:16:56.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.956+0000 7f3c6cbb8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6810fb00 con 0x7f3c68101710 2024-04-21T23:16:56.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.956+0000 7f3c66d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c68107720 0x7f3c6819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:56.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.956+0000 7f3c66d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c68107720 0x7f3c6819a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59580/0 (socket says 172.21.15.44:59580) 2024-04-21T23:16:56.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.956+0000 7f3c66d9d640 1 -- 172.21.15.44:0/3073561410 learned_addr learned my addr 172.21.15.44:0/3073561410 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:56.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.956+0000 7f3c65d9b640 1 --2- 172.21.15.44:0/3073561410 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c681020d0 0x7f3c681963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:56.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.956+0000 7f3c6659c640 1 --2- 172.21.15.44:0/3073561410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c68101710 0x7f3c68195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:56.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.957+0000 7f3c66d9d640 1 -- 172.21.15.44:0/3073561410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c68101710 msgr2=0x7f3c68195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:56.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.957+0000 7f3c66d9d640 1 --2- 172.21.15.44:0/3073561410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c68101710 0x7f3c68195e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:56.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.957+0000 7f3c66d9d640 1 -- 172.21.15.44:0/3073561410 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c681020d0 msgr2=0x7f3c681963a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:56.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.957+0000 7f3c66d9d640 1 --2- 172.21.15.44:0/3073561410 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c681020d0 0x7f3c681963a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:56.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.957+0000 7f3c66d9d640 1 -- 172.21.15.44:0/3073561410 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c6819ae50 con 0x7f3c68107720 2024-04-21T23:16:56.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.957+0000 7f3c66d9d640 1 --2- 172.21.15.44:0/3073561410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c68107720 0x7f3c6819a720 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f3c6000b970 tx=0x7f3c6000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:56.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.958+0000 7f3c577fe640 1 -- 172.21.15.44:0/3073561410 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c60019070 con 0x7f3c68107720 2024-04-21T23:16:56.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.958+0000 7f3c577fe640 1 -- 172.21.15.44:0/3073561410 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c6000cb40 con 0x7f3c68107720 2024-04-21T23:16:56.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.958+0000 7f3c577fe640 1 -- 172.21.15.44:0/3073561410 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c60007610 con 0x7f3c68107720 2024-04-21T23:16:56.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.958+0000 7f3c6cbb8640 1 -- 172.21.15.44:0/3073561410 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c6819b140 con 0x7f3c68107720 2024-04-21T23:16:56.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.958+0000 7f3c6cbb8640 1 -- 172.21.15.44:0/3073561410 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c681a29d0 con 0x7f3c68107720 2024-04-21T23:16:56.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.963+0000 7f3c577fe640 1 -- 172.21.15.44:0/3073561410 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3c6001c070 con 0x7f3c68107720 2024-04-21T23:16:56.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.966+0000 7f3c6f7a0740 1 Processor -- start 2024-04-21T23:16:56.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.974+0000 7f3c577fe640 1 --2- 172.21.15.44:0/3073561410 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3c500a71d0 0x7f3c500a9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:56.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.974+0000 7f3c577fe640 1 -- 172.21.15.44:0/3073561410 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7f3c600cec30 con 0x7f3c68107720 2024-04-21T23:16:56.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.974+0000 7f3c6f7a0740 1 -- start start 2024-04-21T23:16:56.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.975+0000 7f3c6f7a0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238915be0 0x559238915fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:56.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.975+0000 7f3c6f7a0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559238916500 0x559238927700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:56.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.975+0000 7f3c6f7a0740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559238927c40 0x55923892a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:56.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.975+0000 7f3c6f7a0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55923887a560 con 0x559238916500 2024-04-21T23:16:56.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.975+0000 7f3c6f7a0740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x559238822730 con 0x559238927c40 2024-04-21T23:16:56.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.975+0000 7f3c6f7a0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559238876520 con 0x559238915be0 2024-04-21T23:16:56.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.975+0000 7f3c66d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559238927c40 0x55923892a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:56.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.975+0000 7f3c66d9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559238927c40 0x55923892a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:35296/0 (socket says 172.21.15.44:35296) 2024-04-21T23:16:56.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.975+0000 7f3c66d9d640 1 -- 172.21.15.44:0/1536614756 learned_addr learned my addr 172.21.15.44:0/1536614756 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:56.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.976+0000 7f3c6659c640 1 --2- 172.21.15.44:0/1536614756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238915be0 0x559238915fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:56.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.977+0000 7f3c65d9b640 1 --2- 172.21.15.44:0/3073561410 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3c500a71d0 0x7f3c500a9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:56.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.977+0000 7f3c6659c640 1 -- 172.21.15.44:0/1536614756 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559238927c40 msgr2=0x55923892a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:56.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.977+0000 7f3c6659c640 1 --2- 172.21.15.44:0/1536614756 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559238927c40 0x55923892a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:56.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.977+0000 7f3c6659c640 1 -- 172.21.15.44:0/1536614756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559238916500 msgr2=0x559238927700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:56.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.977+0000 7f3c6659c640 1 --2- 172.21.15.44:0/1536614756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559238916500 0x559238927700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:56.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.977+0000 7f3c6659c640 1 -- 172.21.15.44:0/1536614756 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592386587d0 con 0x559238915be0 2024-04-21T23:16:56.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.977+0000 7f3c6659c640 1 --2- 172.21.15.44:0/1536614756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238915be0 0x559238915fc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c036040 tx=0x7f3c5c02c690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4ec5b64bf19e0f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:56.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.978+0000 7f3c54ff9640 1 -- 172.21.15.44:0/1536614756 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c045070 con 0x559238915be0 2024-04-21T23:16:56.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.978+0000 7f3c54ff9640 1 -- 172.21.15.44:0/1536614756 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c5c02c870 con 0x559238915be0 2024-04-21T23:16:56.978 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.978+0000 7f3c54ff9640 1 -- 172.21.15.44:0/1536614756 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c004e90 con 0x559238915be0 2024-04-21T23:16:56.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.979+0000 7f3c65d9b640 1 --2- 172.21.15.44:0/3073561410 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3c500a71d0 0x7f3c500a9690 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3c680ff7e0 tx=0x7f3c580099c0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:56.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.979+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/1536614756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238915be0 msgr2=0x559238915fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:56.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.979+0000 7f3c6f7a0740 1 --2- 172.21.15.44:0/1536614756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238915be0 0x559238915fc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c036040 tx=0x7f3c5c02c690 comp rx=0 tx=0).stop 2024-04-21T23:16:56.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.980+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/1536614756 shutdown_connections 2024-04-21T23:16:56.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.980+0000 7f3c6f7a0740 1 --2- 172.21.15.44:0/1536614756 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559238927c40 0x55923892a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:56.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.980+0000 7f3c6f7a0740 1 --2- 172.21.15.44:0/1536614756 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559238916500 0x559238927700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:56.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.980+0000 7f3c6f7a0740 1 --2- 172.21.15.44:0/1536614756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238915be0 0x559238915fc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:56.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.980+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/1536614756 >> 172.21.15.44:0/1536614756 conn(0x559238910660 msgr2=0x559238910a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:56.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.982+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/1536614756 shutdown_connections 2024-04-21T23:16:56.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.986+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/1536614756 wait complete. 2024-04-21T23:16:56.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.986+0000 7f3c6f7a0740 1 -- start start 2024-04-21T23:16:56.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.986+0000 7f3c6f7a0740 1 Processor -- start 2024-04-21T23:16:56.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.987+0000 7f3c6f7a0740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559238915be0 0x559238921990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:56.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.987+0000 7f3c6f7a0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238916500 0x5592389276b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:56.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.987+0000 7f3c6f7a0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559238927c40 0x559238922080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:56.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.988+0000 7f3c6f7a0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5592389173c0 con 0x559238927c40 2024-04-21T23:16:56.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.988+0000 7f3c6f7a0740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x559238835e00 con 0x559238915be0 2024-04-21T23:16:56.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.988+0000 7f3c6f7a0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559238917540 con 0x559238916500 2024-04-21T23:16:56.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.988+0000 7f3c6f7a0740 10 client.0 _ll_register_callbacks cb 0x559238870eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:56.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.988+0000 7f3c65d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238916500 0x5592389276b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:56.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.988+0000 7f3c65d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238916500 0x5592389276b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34882/0 (socket says 172.21.15.44:34882) 2024-04-21T23:16:56.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.988+0000 7f3c65d9b640 1 -- 172.21.15.44:0/469327962 learned_addr learned my addr 172.21.15.44:0/469327962 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:56.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.989+0000 7f3c66d9d640 1 --2- 172.21.15.44:0/469327962 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559238927c40 0x559238922080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:56.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.989+0000 7f3c65d9b640 1 -- 172.21.15.44:0/469327962 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559238915be0 msgr2=0x559238921990 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T23:16:56.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.989+0000 7f3c65d9b640 1 --2- 172.21.15.44:0/469327962 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559238915be0 0x559238921990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:56.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.990+0000 7f3c65d9b640 1 -- 172.21.15.44:0/469327962 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559238927c40 msgr2=0x559238922080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:56.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.990+0000 7f3c65d9b640 1 --2- 172.21.15.44:0/469327962 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559238927c40 0x559238922080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:56.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.990+0000 7f3c65d9b640 1 -- 172.21.15.44:0/469327962 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55923858e3f0 con 0x559238916500 2024-04-21T23:16:56.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.991+0000 7f3c65d9b640 1 --2- 172.21.15.44:0/469327962 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238916500 0x5592389276b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3c580096d0 tx=0x7f3c58005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc0b046afe75dc99 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:56.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.991+0000 7f3c4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:16:56.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.991+0000 7f3c4dffb640 1 -- 172.21.15.44:0/469327962 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5800e900 con 0x559238916500 2024-04-21T23:16:56.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.991+0000 7f3c4dffb640 1 -- 172.21.15.44:0/469327962 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7f3c5800eaa0 con 0x559238916500 2024-04-21T23:16:56.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.991+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/469327962 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c5c02c870 con 0x559238916500 2024-04-21T23:16:56.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.991+0000 7f3c4dffb640 1 -- 172.21.15.44:0/469327962 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3c5800eca0 con 0x559238916500 2024-04-21T23:16:56.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.994+0000 7f3c6f7a0740 10 client.26977 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:56.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.994+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/469327962 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55923854e0f0 con 0x559238916500 2024-04-21T23:16:56.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.995+0000 7f3c4dffb640 1 -- 172.21.15.44:0/469327962 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c5800ee40 con 0x559238916500 2024-04-21T23:16:56.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.995+0000 7f3c4dffb640 1 -- 172.21.15.44:0/469327962 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f3c580266e0 con 0x559238916500 2024-04-21T23:16:56.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.995+0000 7f3c6f7a0740 10 client.26977 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:56.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.995+0000 7f3c6f7a0740 10 client.26977 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:56.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.995+0000 7f3c6f7a0740 20 client.26977 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:56.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.995+0000 7f3c6f7a0740 1 --2- 172.21.15.44:0/469327962 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5592389406d0 0x559238942ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:56.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.996+0000 7f3c6f7a0740 4 client.26977 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:16:56.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.996+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/469327962 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3c5c02c870 con 0x5592389406d0 2024-04-21T23:16:56.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.998+0000 7f3c6659c640 1 --2- 172.21.15.44:0/469327962 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5592389406d0 0x559238942ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:56.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:56.999+0000 7f3c6659c640 1 --2- 172.21.15.44:0/469327962 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5592389406d0 0x559238942ae0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea87eac458702d02 server_cookie=69d40751970a86db in_seq=0 out_seq=0 2024-04-21T23:16:56.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.000+0000 7f3c4dffb640 10 client.26977 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:57.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.003+0000 7f3c4dffb640 1 -- 172.21.15.44:0/469327962 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f3c5c02c870 con 0x5592389406d0 2024-04-21T23:16:57.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.003+0000 7f3c4dffb640 10 client.26977 handle_command_reply: tid=0 2024-04-21T23:16:57.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.004+0000 7f3c6f7a0740 1 client.26977 shutdown 2024-04-21T23:16:57.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.005+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/469327962 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238916500 msgr2=0x5592389276b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:57.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.005+0000 7f3c6f7a0740 1 --2- 172.21.15.44:0/469327962 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238916500 0x5592389276b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3c580096d0 tx=0x7f3c58005f90 comp rx=0 tx=0).stop 2024-04-21T23:16:57.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.007+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/469327962 shutdown_connections 2024-04-21T23:16:57.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.007+0000 7f3c6f7a0740 1 --2- 172.21.15.44:0/469327962 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5592389406d0 0x559238942ae0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.007+0000 7f3c6f7a0740 1 --2- 172.21.15.44:0/469327962 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238916500 0x5592389276b0 secure :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3c580096d0 tx=0x7f3c58005f90 comp rx=0 tx=0).stop 2024-04-21T23:16:57.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.007+0000 7f3c6f7a0740 1 --2- 172.21.15.44:0/469327962 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559238927c40 0x559238922080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.007+0000 7f3c6f7a0740 1 --2- 172.21.15.44:0/469327962 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559238915be0 0x559238921990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.007+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/469327962 >> 172.21.15.44:0/469327962 conn(0x559238910660 msgr2=0x5592389123b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:57.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.007+0000 7f3c4dffb640 0 client.26977 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:57.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.007+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/469327962 shutdown_connections 2024-04-21T23:16:57.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.007+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/469327962 wait complete. 2024-04-21T23:16:57.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.008+0000 7f3c6f7a0740 20 client.26977 trim_cache size 0 max 16384 2024-04-21T23:16:57.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.044+0000 7f3c6f7a0740 1 Processor -- start 2024-04-21T23:16:57.044 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.044+0000 7f3c6f7a0740 1 -- start start 2024-04-21T23:16:57.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.045+0000 7f3c6f7a0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559238915be0 0x5592389180f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.045+0000 7f3c6f7a0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238927c40 0x55923890f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.045+0000 7f3c6f7a0740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5592389213b0 0x5592389217f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.045+0000 7f3c6f7a0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3c681aaae0 con 0x559238915be0 2024-04-21T23:16:57.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.045+0000 7f3c6f7a0740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3c681ad360 con 0x5592389213b0 2024-04-21T23:16:57.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.045+0000 7f3c6f7a0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3c681ab150 con 0x559238927c40 2024-04-21T23:16:57.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.045+0000 7f3c6659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559238915be0 0x5592389180f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.050 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.045+0000 7f3c6659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559238915be0 0x5592389180f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59618/0 (socket says 172.21.15.44:59618) 2024-04-21T23:16:57.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.046+0000 7f3c6659c640 1 -- 172.21.15.44:0/3453300763 learned_addr learned my addr 172.21.15.44:0/3453300763 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:57.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.046+0000 7f3c65d9b640 1 --2- 172.21.15.44:0/3453300763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238927c40 0x55923890f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.046+0000 7f3c65d9b640 1 -- 172.21.15.44:0/3453300763 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5592389213b0 msgr2=0x5592389217f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T23:16:57.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.046+0000 7f3c65d9b640 1 --2- 172.21.15.44:0/3453300763 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5592389213b0 0x5592389217f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.046+0000 7f3c65d9b640 1 -- 172.21.15.44:0/3453300763 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559238915be0 msgr2=0x5592389180f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:57.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.046+0000 7f3c65d9b640 1 --2- 172.21.15.44:0/3453300763 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559238915be0 0x5592389180f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.046+0000 7f3c65d9b640 1 -- 172.21.15.44:0/3453300763 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55923891bd40 con 0x559238927c40 2024-04-21T23:16:57.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.046+0000 7f3c65d9b640 1 --2- 172.21.15.44:0/3453300763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238927c40 0x55923890f560 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55923891de20 tx=0x7f3c58004600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa362d286cc782fe server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:57.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.047+0000 7f3c4ffff640 1 -- 172.21.15.44:0/3453300763 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5800e900 con 0x559238927c40 2024-04-21T23:16:57.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.047+0000 7f3c4ffff640 1 -- 172.21.15.44:0/3453300763 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c5800ee40 con 0x559238927c40 2024-04-21T23:16:57.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.047+0000 7f3c4ffff640 1 -- 172.21.15.44:0/3453300763 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5801c9b0 con 0x559238927c40 2024-04-21T23:16:57.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.049+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/3453300763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238927c40 msgr2=0x55923890f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:57.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.049+0000 7f3c6f7a0740 1 --2- 172.21.15.44:0/3453300763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238927c40 0x55923890f560 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55923891de20 tx=0x7f3c58004600 comp rx=0 tx=0).stop 2024-04-21T23:16:57.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.049+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/3453300763 shutdown_connections 2024-04-21T23:16:57.051 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.050+0000 7f3c6f7a0740 1 --2- 172.21.15.44:0/3453300763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238927c40 0x55923890f560 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.050+0000 7f3c6f7a0740 1 --2- 172.21.15.44:0/3453300763 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5592389213b0 0x5592389217f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.050+0000 7f3c6f7a0740 1 --2- 172.21.15.44:0/3453300763 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x559238915be0 0x5592389180f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.050+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/3453300763 >> 172.21.15.44:0/3453300763 conn(0x559238910660 msgr2=0x55923890ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:57.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.050+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/3453300763 shutdown_connections 2024-04-21T23:16:57.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.050+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/3453300763 wait complete. 2024-04-21T23:16:57.052 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.050+0000 7f3c6f7a0740 1 -- start start 2024-04-21T23:16:57.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.051+0000 7f3c6f7a0740 1 Processor -- start 2024-04-21T23:16:57.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.051+0000 7f3c6f7a0740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559238915be0 0x559238924c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.051+0000 7f3c6f7a0740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5592389213b0 0x5592389251c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.051+0000 7f3c6f7a0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238927c40 0x559238925700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.051+0000 7f3c6f7a0740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3c681ab820 con 0x5592389213b0 2024-04-21T23:16:57.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.051+0000 7f3c6f7a0740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3c681acc90 con 0x559238915be0 2024-04-21T23:16:57.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.051+0000 7f3c6f7a0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3c681abef0 con 0x559238927c40 2024-04-21T23:16:57.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.051+0000 7f3c66d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238927c40 0x559238925700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.052+0000 7f3c66d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238927c40 0x559238925700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34894/0 (socket says 172.21.15.44:34894) 2024-04-21T23:16:57.053 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.052+0000 7f3c66d9d640 1 -- 172.21.15.44:0/850732342 learned_addr learned my addr 172.21.15.44:0/850732342 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:57.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.052+0000 7f3c6f7a0740 10 client.0 _ll_register_callbacks cb 0x7f3c58009bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:57.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.052+0000 7f3c6659c640 1 --2- 172.21.15.44:0/850732342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559238915be0 0x559238924c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.052+0000 7f3c66d9d640 1 -- 172.21.15.44:0/850732342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559238915be0 msgr2=0x559238924c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:57.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.052+0000 7f3c66d9d640 1 --2- 172.21.15.44:0/850732342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559238915be0 0x559238924c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.052+0000 7f3c66d9d640 1 -- 172.21.15.44:0/850732342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5592389213b0 msgr2=0x5592389251c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:57.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.052+0000 7f3c66d9d640 1 --2- 172.21.15.44:0/850732342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5592389213b0 0x5592389251c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.052+0000 7f3c66d9d640 1 -- 172.21.15.44:0/850732342 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592389a5d40 con 0x559238927c40 2024-04-21T23:16:57.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.052+0000 7f3c66d9d640 1 --2- 172.21.15.44:0/850732342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238927c40 0x559238925700 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3c600cc5e0 tx=0x7f3c6009abd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a96eac4e9c2cb52d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:57.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.053+0000 7f3c4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:16:57.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.053+0000 7f3c4dffb640 1 -- 172.21.15.44:0/850732342 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c600a36a0 con 0x559238927c40 2024-04-21T23:16:57.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.054+0000 7f3c4dffb640 1 -- 172.21.15.44:0/850732342 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7f3c600a3840 con 0x559238927c40 2024-04-21T23:16:57.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.054+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/850732342 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559238887aa0 con 0x559238927c40 2024-04-21T23:16:57.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.055+0000 7f3c4dffb640 1 -- 172.21.15.44:0/850732342 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3c6009a3c0 con 0x559238927c40 2024-04-21T23:16:57.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.055+0000 7f3c6f7a0740 10 client.26983 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:57.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.055+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/850732342 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5592389a7060 con 0x559238927c40 2024-04-21T23:16:57.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.056+0000 7f3c4dffb640 1 -- 172.21.15.44:0/850732342 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559238887aa0 con 0x559238927c40 2024-04-21T23:16:57.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.056+0000 7f3c4dffb640 1 -- 172.21.15.44:0/850732342 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f3c6007e070 con 0x559238927c40 2024-04-21T23:16:57.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.056+0000 7f3c6f7a0740 10 client.26983 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:57.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.056+0000 7f3c6f7a0740 10 client.26983 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:57.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.056+0000 7f3c6f7a0740 20 client.26983 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:57.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.057+0000 7f3c6f7a0740 1 --2- 172.21.15.44:0/850732342 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5592389a77c0 0x5592389a9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.057+0000 7f3c6f7a0740 4 client.26983 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:16:57.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.057+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/850732342 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559238881df0 con 0x5592389a77c0 2024-04-21T23:16:57.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.059+0000 7f3c6659c640 1 --2- 172.21.15.44:0/850732342 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5592389a77c0 0x5592389a9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.064+0000 7f3c6659c640 1 --2- 172.21.15.44:0/850732342 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5592389a77c0 0x5592389a9be0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4435bc6736f27756 server_cookie=76a987bb0d77af04 in_seq=0 out_seq=0 2024-04-21T23:16:57.064 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.065+0000 7f3c4dffb640 10 client.26983 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:57.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.065+0000 7f3c4dffb640 1 -- 172.21.15.44:0/850732342 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559238881df0 con 0x5592389a77c0 2024-04-21T23:16:57.067 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.066+0000 7f3c4dffb640 10 client.26983 handle_command_reply: tid=0 2024-04-21T23:16:57.068 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.068+0000 7f3c6f7a0740 1 client.26983 shutdown 2024-04-21T23:16:57.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.070+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/850732342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238927c40 msgr2=0x559238925700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:57.069 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.070+0000 7f3c6f7a0740 1 --2- 172.21.15.44:0/850732342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238927c40 0x559238925700 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3c600cc5e0 tx=0x7f3c6009abd0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.072+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/850732342 shutdown_connections 2024-04-21T23:16:57.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.073+0000 7f3c6f7a0740 1 --2- 172.21.15.44:0/850732342 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5592389a77c0 0x5592389a9be0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.073+0000 7f3c6f7a0740 1 --2- 172.21.15.44:0/850732342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559238927c40 0x559238925700 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.073+0000 7f3c6f7a0740 1 --2- 172.21.15.44:0/850732342 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5592389213b0 0x5592389251c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.073+0000 7f3c6f7a0740 1 --2- 172.21.15.44:0/850732342 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x559238915be0 0x559238924c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.072 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.073+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/850732342 >> 172.21.15.44:0/850732342 conn(0x559238910660 msgr2=0x559238920c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:57.074 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.075+0000 7f3c4dffb640 0 client.26983 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:57.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.077+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/850732342 shutdown_connections 2024-04-21T23:16:57.079 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.077+0000 7f3c6f7a0740 1 -- 172.21.15.44:0/850732342 wait complete. 2024-04-21T23:16:57.082 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.081+0000 7f3c6f7a0740 20 client.26983 trim_cache size 0 max 16384 2024-04-21T23:16:57.083 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:16:57.083 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-21T23:16:57.083 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-21T23:16:57.083 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:16:57.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.085+0000 7f3c6cbb8640 1 -- 172.21.15.44:0/3073561410 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3c500a71d0 msgr2=0x7f3c500a9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:57.084 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.085+0000 7f3c6cbb8640 1 --2- 172.21.15.44:0/3073561410 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3c500a71d0 0x7f3c500a9690 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3c680ff7e0 tx=0x7f3c580099c0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.085+0000 7f3c6cbb8640 1 -- 172.21.15.44:0/3073561410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c68107720 msgr2=0x7f3c6819a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:57.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.085+0000 7f3c6cbb8640 1 --2- 172.21.15.44:0/3073561410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c68107720 0x7f3c6819a720 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f3c6000b970 tx=0x7f3c6000be30 comp rx=0 tx=0).stop 2024-04-21T23:16:57.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.086+0000 7f3c6cbb8640 1 -- 172.21.15.44:0/3073561410 shutdown_connections 2024-04-21T23:16:57.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.086+0000 7f3c6cbb8640 1 --2- 172.21.15.44:0/3073561410 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3c500a71d0 0x7f3c500a9690 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.086+0000 7f3c6cbb8640 1 --2- 172.21.15.44:0/3073561410 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3c68107720 0x7f3c6819a720 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.086+0000 7f3c6cbb8640 1 --2- 172.21.15.44:0/3073561410 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3c681020d0 0x7f3c681963a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.085 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.086+0000 7f3c6cbb8640 1 --2- 172.21.15.44:0/3073561410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c68101710 0x7f3c68195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.086+0000 7f3c6cbb8640 1 -- 172.21.15.44:0/3073561410 >> 172.21.15.44:0/3073561410 conn(0x7f3c680fd200 msgr2=0x7f3c680fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:57.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.088+0000 7f3c6cbb8640 1 -- 172.21.15.44:0/3073561410 shutdown_connections 2024-04-21T23:16:57.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.088+0000 7f3c6cbb8640 1 -- 172.21.15.44:0/3073561410 wait complete. 2024-04-21T23:16:57.104 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:16:57.104 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:16:57.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.210+0000 7fb3b414f640 1 -- 172.21.15.44:0/3958468740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3ac107800 msgr2=0x7fb3ac107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:57.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.210+0000 7fb3b414f640 1 --2- 172.21.15.44:0/3958468740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3ac107800 0x7fb3ac107be0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4009900 tx=0x7fb3a402bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:57.212 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.211+0000 7fb3b414f640 1 -- 172.21.15.44:0/3958468740 shutdown_connections 2024-04-21T23:16:57.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.211+0000 7fb3b414f640 1 --2- 172.21.15.44:0/3958468740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3ac0693e0 0x7fb3ac10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.211+0000 7fb3b414f640 1 --2- 172.21.15.44:0/3958468740 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3ac068a40 0x7fb3ac068ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.211+0000 7fb3b414f640 1 --2- 172.21.15.44:0/3958468740 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3ac107800 0x7fb3ac107be0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.211+0000 7fb3b414f640 1 -- 172.21.15.44:0/3958468740 >> 172.21.15.44:0/3958468740 conn(0x7fb3ac077430 msgr2=0x7fb3ac077840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:57.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.211+0000 7fb3b414f640 1 -- 172.21.15.44:0/3958468740 shutdown_connections 2024-04-21T23:16:57.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.211+0000 7fb3b414f640 1 -- 172.21.15.44:0/3958468740 wait complete. 2024-04-21T23:16:57.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.211+0000 7fb3b414f640 1 Processor -- start 2024-04-21T23:16:57.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.211+0000 7fb3b414f640 1 -- start start 2024-04-21T23:16:57.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.211+0000 7fb3b414f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3ac068a40 0x7fb3ac197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.211+0000 7fb3b414f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3ac0693e0 0x7fb3ac198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.211+0000 7fb3b414f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3ac107800 0x7fb3ac19c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.211+0000 7fb3b414f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac111b20 con 0x7fb3ac0693e0 2024-04-21T23:16:57.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.211+0000 7fb3b414f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1119a0 con 0x7fb3ac107800 2024-04-21T23:16:57.213 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.211+0000 7fb3b414f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac111ca0 con 0x7fb3ac068a40 2024-04-21T23:16:57.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.212+0000 7fb3b16ea640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3ac0693e0 0x7fb3ac198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.212+0000 7fb3b16ea640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3ac0693e0 0x7fb3ac198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59648/0 (socket says 172.21.15.44:59648) 2024-04-21T23:16:57.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.212+0000 7fb3b16ea640 1 -- 172.21.15.44:0/3429492865 learned_addr learned my addr 172.21.15.44:0/3429492865 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:57.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.212+0000 7fb3b26ec640 1 --2- 172.21.15.44:0/3429492865 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3ac107800 0x7fb3ac19c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.212+0000 7fb3b1eeb640 1 --2- 172.21.15.44:0/3429492865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3ac068a40 0x7fb3ac197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.212+0000 7fb3b16ea640 1 -- 172.21.15.44:0/3429492865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3ac068a40 msgr2=0x7fb3ac197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:57.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.212+0000 7fb3b16ea640 1 --2- 172.21.15.44:0/3429492865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3ac068a40 0x7fb3ac197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.212+0000 7fb3b16ea640 1 -- 172.21.15.44:0/3429492865 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3ac107800 msgr2=0x7fb3ac19c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:57.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.212+0000 7fb3b16ea640 1 --2- 172.21.15.44:0/3429492865 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3ac107800 0x7fb3ac19c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.212+0000 7fb3b16ea640 1 -- 172.21.15.44:0/3429492865 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3ac19cfd0 con 0x7fb3ac0693e0 2024-04-21T23:16:57.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.212+0000 7fb3b16ea640 1 --2- 172.21.15.44:0/3429492865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3ac0693e0 0x7fb3ac198520 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fb3a000ea60 tx=0x7fb3a000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:57.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.212+0000 7fb3aaffd640 1 -- 172.21.15.44:0/3429492865 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a000cde0 con 0x7fb3ac0693e0 2024-04-21T23:16:57.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.212+0000 7fb3b414f640 1 -- 172.21.15.44:0/3429492865 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3ac19d2c0 con 0x7fb3ac0693e0 2024-04-21T23:16:57.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.212+0000 7fb3aaffd640 1 -- 172.21.15.44:0/3429492865 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3a00035f0 con 0x7fb3ac0693e0 2024-04-21T23:16:57.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.212+0000 7fb3aaffd640 1 -- 172.21.15.44:0/3429492865 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0010670 con 0x7fb3ac0693e0 2024-04-21T23:16:57.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.212+0000 7fb3b414f640 1 -- 172.21.15.44:0/3429492865 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3ac1a4b50 con 0x7fb3ac0693e0 2024-04-21T23:16:57.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.213+0000 7fb3b414f640 1 -- 172.21.15.44:0/3429492865 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3ac0685f0 con 0x7fb3ac0693e0 2024-04-21T23:16:57.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.214+0000 7fb3aaffd640 1 -- 172.21.15.44:0/3429492865 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb3a00037a0 con 0x7fb3ac0693e0 2024-04-21T23:16:57.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.215+0000 7fb3aaffd640 1 --2- 172.21.15.44:0/3429492865 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb3940a2d00 0x7fb3940a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.215+0000 7fb3b1eeb640 1 --2- 172.21.15.44:0/3429492865 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb3940a2d00 0x7fb3940a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.215+0000 7fb3aaffd640 1 -- 172.21.15.44:0/3429492865 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7fb3a00cfa90 con 0x7fb3ac0693e0 2024-04-21T23:16:57.216 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.216+0000 7fb3b1eeb640 1 --2- 172.21.15.44:0/3429492865 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb3940a2d00 0x7fb3940a51c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4006010 tx=0x7fb3a4005dd0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:57.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.220+0000 7fb3aaffd640 1 -- 172.21.15.44:0/3429492865 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb3a009c310 con 0x7fb3ac0693e0 2024-04-21T23:16:57.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.460+0000 7fb3b414f640 1 -- 172.21.15.44:0/3429492865 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3ac1096a0 con 0x7fb3ac0693e0 2024-04-21T23:16:57.460 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.461+0000 7fb3aaffd640 1 -- 172.21.15.44:0/3429492865 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11791 (secure 0 0 0) 0x7fb3a009bc10 con 0x7fb3ac0693e0 2024-04-21T23:16:57.461 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:16:57.461 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:16:20.187271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26863":{"gid":26863,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3179646768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3179646768},{"type":"v1","addr":"172.21.15.69:6839","nonce":3179646768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:16:57.461 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 31 2024-04-21T23:16:57.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.463+0000 7fb3b414f640 1 -- 172.21.15.44:0/3429492865 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb3940a2d00 msgr2=0x7fb3940a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:57.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.463+0000 7fb3b414f640 1 --2- 172.21.15.44:0/3429492865 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb3940a2d00 0x7fb3940a51c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4006010 tx=0x7fb3a4005dd0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.463+0000 7fb3b414f640 1 -- 172.21.15.44:0/3429492865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3ac0693e0 msgr2=0x7fb3ac198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:57.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.463+0000 7fb3b414f640 1 --2- 172.21.15.44:0/3429492865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3ac0693e0 0x7fb3ac198520 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fb3a000ea60 tx=0x7fb3a000ef20 comp rx=0 tx=0).stop 2024-04-21T23:16:57.463 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.463+0000 7fb3b414f640 1 -- 172.21.15.44:0/3429492865 shutdown_connections 2024-04-21T23:16:57.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.463+0000 7fb3b414f640 1 --2- 172.21.15.44:0/3429492865 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb3940a2d00 0x7fb3940a51c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.463+0000 7fb3b414f640 1 --2- 172.21.15.44:0/3429492865 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb3ac107800 0x7fb3ac19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.463+0000 7fb3b414f640 1 --2- 172.21.15.44:0/3429492865 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb3ac0693e0 0x7fb3ac198520 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.463+0000 7fb3b414f640 1 --2- 172.21.15.44:0/3429492865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3ac068a40 0x7fb3ac197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.463+0000 7fb3b414f640 1 -- 172.21.15.44:0/3429492865 >> 172.21.15.44:0/3429492865 conn(0x7fb3ac077430 msgr2=0x7fb3ac0ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:57.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.463+0000 7fb3b414f640 1 -- 172.21.15.44:0/3429492865 shutdown_connections 2024-04-21T23:16:57.464 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.463+0000 7fb3b414f640 1 -- 172.21.15.44:0/3429492865 wait complete. 2024-04-21T23:16:57.474 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:16:57.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.581+0000 7f2b1ed3a640 1 -- 172.21.15.44:0/3138083812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b181066b0 msgr2=0x7f2b18106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:57.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.581+0000 7f2b1ed3a640 1 --2- 172.21.15.44:0/3138083812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b181066b0 0x7f2b18106a90 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c009930 tx=0x7f2b0c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:57.583 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.581+0000 7f2b1ed3a640 1 -- 172.21.15.44:0/3138083812 shutdown_connections 2024-04-21T23:16:57.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.581+0000 7f2b1ed3a640 1 --2- 172.21.15.44:0/3138083812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b18100fd0 0x7f2b1810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.581+0000 7f2b1ed3a640 1 --2- 172.21.15.44:0/3138083812 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b18100610 0x7f2b18100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.581+0000 7f2b1ed3a640 1 --2- 172.21.15.44:0/3138083812 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b181066b0 0x7f2b18106a90 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.581+0000 7f2b1ed3a640 1 -- 172.21.15.44:0/3138083812 >> 172.21.15.44:0/3138083812 conn(0x7f2b180777f0 msgr2=0x7f2b18077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:57.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.581+0000 7f2b1ed3a640 1 -- 172.21.15.44:0/3138083812 shutdown_connections 2024-04-21T23:16:57.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.581+0000 7f2b1ed3a640 1 -- 172.21.15.44:0/3138083812 wait complete. 2024-04-21T23:16:57.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.582+0000 7f2b1ed3a640 1 Processor -- start 2024-04-21T23:16:57.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.582+0000 7f2b1ed3a640 1 -- start start 2024-04-21T23:16:57.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.582+0000 7f2b1ed3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b18100610 0x7f2b18197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.582+0000 7f2b1ed3a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b18100fd0 0x7f2b181984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.582+0000 7f2b1ed3a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b181066b0 0x7f2b1819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.582+0000 7f2b1ed3a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2b181109f0 con 0x7f2b18100fd0 2024-04-21T23:16:57.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.582+0000 7f2b1ed3a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2b18110870 con 0x7f2b181066b0 2024-04-21T23:16:57.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.582+0000 7f2b1ed3a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2b18110b70 con 0x7f2b18100610 2024-04-21T23:16:57.584 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.582+0000 7f2b17fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b18100fd0 0x7f2b181984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.582+0000 7f2b17fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b18100fd0 0x7f2b181984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59654/0 (socket says 172.21.15.44:59654) 2024-04-21T23:16:57.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.582+0000 7f2b17fff640 1 -- 172.21.15.44:0/1638128892 learned_addr learned my addr 172.21.15.44:0/1638128892 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:57.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.582+0000 7f2b1cad6640 1 --2- 172.21.15.44:0/1638128892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b18100610 0x7f2b18197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.582+0000 7f2b1d2d7640 1 --2- 172.21.15.44:0/1638128892 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b181066b0 0x7f2b1819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.583+0000 7f2b1cad6640 1 -- 172.21.15.44:0/1638128892 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b181066b0 msgr2=0x7f2b1819c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:57.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.583+0000 7f2b1cad6640 1 --2- 172.21.15.44:0/1638128892 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b181066b0 0x7f2b1819c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.583+0000 7f2b1cad6640 1 -- 172.21.15.44:0/1638128892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b18100fd0 msgr2=0x7f2b181984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:57.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.583+0000 7f2b1cad6640 1 --2- 172.21.15.44:0/1638128892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b18100fd0 0x7f2b181984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.583+0000 7f2b1cad6640 1 -- 172.21.15.44:0/1638128892 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b1819cf80 con 0x7f2b18100610 2024-04-21T23:16:57.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.583+0000 7f2b17fff640 1 --2- 172.21.15.44:0/1638128892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b18100fd0 0x7f2b181984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:57.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.583+0000 7f2b1cad6640 1 --2- 172.21.15.44:0/1638128892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b18100610 0x7f2b18197f90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c02c170 tx=0x7f2b0c02cb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:57.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.583+0000 7f2b15ffb640 1 -- 172.21.15.44:0/1638128892 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c041070 con 0x7f2b18100610 2024-04-21T23:16:57.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.583+0000 7f2b1ed3a640 1 -- 172.21.15.44:0/1638128892 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b1819d210 con 0x7f2b18100610 2024-04-21T23:16:57.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.583+0000 7f2b15ffb640 1 -- 172.21.15.44:0/1638128892 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b0c004450 con 0x7f2b18100610 2024-04-21T23:16:57.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.583+0000 7f2b15ffb640 1 -- 172.21.15.44:0/1638128892 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c003de0 con 0x7f2b18100610 2024-04-21T23:16:57.585 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.583+0000 7f2b1ed3a640 1 -- 172.21.15.44:0/1638128892 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b181a4b00 con 0x7f2b18100610 2024-04-21T23:16:57.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.585+0000 7f2b15ffb640 1 -- 172.21.15.44:0/1638128892 <== mon.1 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f2b0c044070 con 0x7f2b18100610 2024-04-21T23:16:57.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b21a04740 1 Processor -- start 2024-04-21T23:16:57.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b21a04740 1 -- start start 2024-04-21T23:16:57.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b21a04740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35020 0x5653bdc35400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b21a04740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc35940 0x5653bdc3dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.591 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b21a04740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc3e520 0x5653bdc40910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b21a04740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5653bdb99530 con 0x5653bdc35020 2024-04-21T23:16:57.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b21a04740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5653bdb41830 con 0x5653bdc3e520 2024-04-21T23:16:57.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b21a04740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5653bdb954f0 con 0x5653bdc35940 2024-04-21T23:16:57.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b15ffb640 1 --2- 172.21.15.44:0/1638128892 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f2b080a2d00 0x7f2b080a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b1d2d7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35020 0x5653bdc35400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b1d2d7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35020 0x5653bdc35400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59660/0 (socket says 172.21.15.44:59660) 2024-04-21T23:16:57.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b1d2d7640 1 -- 172.21.15.44:0/3388198903 learned_addr learned my addr 172.21.15.44:0/3388198903 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:57.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b1cad6640 1 --2- 172.21.15.44:0/3388198903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc35940 0x5653bdc3dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b17fff640 1 --2- 172.21.15.44:0/3388198903 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc3e520 0x5653bdc40910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b1d2d7640 1 --2- 172.21.15.44:0/1638128892 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f2b080a2d00 0x7f2b080a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b15ffb640 1 -- 172.21.15.44:0/1638128892 <== mon.1 v2:172.21.15.69:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7f2b0c0f0f10 con 0x7f2b18100610 2024-04-21T23:16:57.592 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b1d2d7640 1 -- 172.21.15.44:0/3388198903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc35940 msgr2=0x5653bdc3dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:57.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b1d2d7640 1 --2- 172.21.15.44:0/3388198903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc35940 0x5653bdc3dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b1d2d7640 1 -- 172.21.15.44:0/3388198903 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc3e520 msgr2=0x5653bdc40910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:57.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b1d2d7640 1 --2- 172.21.15.44:0/3388198903 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc3e520 0x5653bdc40910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.586+0000 7f2b1d2d7640 1 -- 172.21.15.44:0/3388198903 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653bdb652c0 con 0x5653bdc35020 2024-04-21T23:16:57.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.587+0000 7f2b1d2d7640 1 --2- 172.21.15.44:0/3388198903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35020 0x5653bdc35400 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f2b10013b00 tx=0x7f2b1000f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6cb02bd39199290 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:57.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.587+0000 7f2b1d2d7640 1 --2- 172.21.15.44:0/1638128892 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f2b080a2d00 0x7f2b080a51c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000f8c0 tx=0x7f2b1001e000 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:57.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.587+0000 7f2b02ffd640 1 -- 172.21.15.44:0/3388198903 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b10024070 con 0x5653bdc35020 2024-04-21T23:16:57.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.587+0000 7f2b02ffd640 1 -- 172.21.15.44:0/3388198903 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b10010910 con 0x5653bdc35020 2024-04-21T23:16:57.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.587+0000 7f2b02ffd640 1 -- 172.21.15.44:0/3388198903 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b1001e7a0 con 0x5653bdc35020 2024-04-21T23:16:57.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.587+0000 7f2b21a04740 1 -- 172.21.15.44:0/3388198903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35020 msgr2=0x5653bdc35400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:57.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.587+0000 7f2b21a04740 1 --2- 172.21.15.44:0/3388198903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35020 0x5653bdc35400 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f2b10013b00 tx=0x7f2b1000f3c0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.587+0000 7f2b21a04740 1 -- 172.21.15.44:0/3388198903 shutdown_connections 2024-04-21T23:16:57.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.587+0000 7f2b21a04740 1 --2- 172.21.15.44:0/3388198903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc35940 0x5653bdc3dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.587+0000 7f2b21a04740 1 --2- 172.21.15.44:0/3388198903 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc3e520 0x5653bdc40910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.587+0000 7f2b21a04740 1 --2- 172.21.15.44:0/3388198903 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35020 0x5653bdc35400 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.587+0000 7f2b21a04740 1 -- 172.21.15.44:0/3388198903 >> 172.21.15.44:0/3388198903 conn(0x5653bdc2f750 msgr2=0x5653bdc2fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:57.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.588+0000 7f2b21a04740 1 -- 172.21.15.44:0/3388198903 shutdown_connections 2024-04-21T23:16:57.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.588+0000 7f2b21a04740 1 -- 172.21.15.44:0/3388198903 wait complete. 2024-04-21T23:16:57.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.588+0000 7f2b21a04740 1 -- start start 2024-04-21T23:16:57.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.588+0000 7f2b21a04740 1 Processor -- start 2024-04-21T23:16:57.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.588+0000 7f2b21a04740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc35020 0x5653bdc3def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.588+0000 7f2b21a04740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc35940 0x5653bdc3b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.588+0000 7f2b21a04740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc3e520 0x5653bdc3b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.588+0000 7f2b21a04740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5653bdc412b0 con 0x5653bdc3e520 2024-04-21T23:16:57.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.588+0000 7f2b21a04740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5653bdb54f10 con 0x5653bdc35020 2024-04-21T23:16:57.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.588+0000 7f2b21a04740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5653bdc41430 con 0x5653bdc35940 2024-04-21T23:16:57.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.588+0000 7f2b21a04740 10 client.0 _ll_register_callbacks cb 0x5653bd8acbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:57.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.588+0000 7f2b1d2d7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc3e520 0x5653bdc3b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.589+0000 7f2b1cad6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc35020 0x5653bdc3def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.589+0000 7f2b1d2d7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc3e520 0x5653bdc3b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59662/0 (socket says 172.21.15.44:59662) 2024-04-21T23:16:57.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.589+0000 7f2b1d2d7640 1 -- 172.21.15.44:0/490788498 learned_addr learned my addr 172.21.15.44:0/490788498 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:57.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.589+0000 7f2b17fff640 1 --2- 172.21.15.44:0/490788498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc35940 0x5653bdc3b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.589+0000 7f2b1d2d7640 1 -- 172.21.15.44:0/490788498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc35940 msgr2=0x5653bdc3b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:57.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.589+0000 7f2b1d2d7640 1 --2- 172.21.15.44:0/490788498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc35940 0x5653bdc3b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.589+0000 7f2b1d2d7640 1 -- 172.21.15.44:0/490788498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc35020 msgr2=0x5653bdc3def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:57.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.589+0000 7f2b1d2d7640 1 --2- 172.21.15.44:0/490788498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc35020 0x5653bdc3def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.589+0000 7f2b1d2d7640 1 -- 172.21.15.44:0/490788498 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653bdb26c80 con 0x5653bdc3e520 2024-04-21T23:16:57.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.589+0000 7f2b17fff640 1 --2- 172.21.15.44:0/490788498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc35940 0x5653bdc3b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:16:57.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.589+0000 7f2b1cad6640 1 --2- 172.21.15.44:0/490788498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc35020 0x5653bdc3def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:57.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.589+0000 7f2b1d2d7640 1 --2- 172.21.15.44:0/490788498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc3e520 0x5653bdc3b890 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f2b10009560 tx=0x7f2b10010660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=227e98cc40c935a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:57.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.589+0000 7f2af7fff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:16:57.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.589+0000 7f2af7fff640 1 -- 172.21.15.44:0/490788498 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b10024030 con 0x5653bdc3e520 2024-04-21T23:16:57.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.589+0000 7f2af7fff640 1 -- 172.21.15.44:0/490788498 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7f2b100050c0 con 0x5653bdc3e520 2024-04-21T23:16:57.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.589+0000 7f2b21a04740 1 -- 172.21.15.44:0/490788498 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b10010910 con 0x5653bdc3e520 2024-04-21T23:16:57.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.589+0000 7f2af7fff640 1 -- 172.21.15.44:0/490788498 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2b1003b050 con 0x5653bdc3e520 2024-04-21T23:16:57.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.590+0000 7f2b21a04740 10 client.17466 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:57.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.590+0000 7f2b21a04740 1 -- 172.21.15.44:0/490788498 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5653bd8ad3f0 con 0x5653bdc3e520 2024-04-21T23:16:57.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.590+0000 7f2af7fff640 1 -- 172.21.15.44:0/490788498 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b10010910 con 0x5653bdc3e520 2024-04-21T23:16:57.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.590+0000 7f2af7fff640 1 -- 172.21.15.44:0/490788498 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f2b1001e6a0 con 0x5653bdc3e520 2024-04-21T23:16:57.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.590+0000 7f2b21a04740 10 client.17466 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:57.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.590+0000 7f2b21a04740 10 client.17466 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:57.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.590+0000 7f2b21a04740 20 client.17466 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:57.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.590+0000 7f2b21a04740 1 --2- 172.21.15.44:0/490788498 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5653bdc4e510 0x5653bdc50900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.590+0000 7f2b21a04740 4 client.17466 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:16:57.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.590+0000 7f2b21a04740 1 -- 172.21.15.44:0/490788498 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653bdba6a70 con 0x5653bdc4e510 2024-04-21T23:16:57.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.590+0000 7f2b1cad6640 1 --2- 172.21.15.44:0/490788498 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5653bdc4e510 0x5653bdc50900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.591+0000 7f2af7fff640 10 client.17466 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:57.596 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.591+0000 7f2b1cad6640 1 --2- 172.21.15.44:0/490788498 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5653bdc4e510 0x5653bdc50900 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f867a59357b42813 server_cookie=29d0bdc9e778792 in_seq=0 out_seq=0 2024-04-21T23:16:57.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.592+0000 7f2af7fff640 1 -- 172.21.15.44:0/490788498 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5653bdba6a70 con 0x5653bdc4e510 2024-04-21T23:16:57.597 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.592+0000 7f2af7fff640 10 client.17466 handle_command_reply: tid=0 2024-04-21T23:16:57.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.592+0000 7f2b21a04740 1 client.17466 shutdown 2024-04-21T23:16:57.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.593+0000 7f2b21a04740 1 -- 172.21.15.44:0/490788498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc3e520 msgr2=0x5653bdc3b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:57.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.593+0000 7f2b21a04740 1 --2- 172.21.15.44:0/490788498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc3e520 0x5653bdc3b890 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f2b10009560 tx=0x7f2b10010660 comp rx=0 tx=0).stop 2024-04-21T23:16:57.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.593+0000 7f2b21a04740 1 -- 172.21.15.44:0/490788498 shutdown_connections 2024-04-21T23:16:57.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.593+0000 7f2b21a04740 1 --2- 172.21.15.44:0/490788498 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5653bdc4e510 0x5653bdc50900 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.593+0000 7f2b21a04740 1 --2- 172.21.15.44:0/490788498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc35940 0x5653bdc3b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.593+0000 7f2b21a04740 1 --2- 172.21.15.44:0/490788498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc3e520 0x5653bdc3b890 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.593+0000 7f2af7fff640 0 client.17466 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:57.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.593+0000 7f2b21a04740 1 --2- 172.21.15.44:0/490788498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc35020 0x5653bdc3def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.593+0000 7f2b21a04740 1 -- 172.21.15.44:0/490788498 >> 172.21.15.44:0/490788498 conn(0x5653bdc2f750 msgr2=0x5653bdc314a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:57.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.593+0000 7f2b21a04740 1 -- 172.21.15.44:0/490788498 shutdown_connections 2024-04-21T23:16:57.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.593+0000 7f2b21a04740 1 -- 172.21.15.44:0/490788498 wait complete. 2024-04-21T23:16:57.598 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.593+0000 7f2b21a04740 20 client.17466 trim_cache size 0 max 16384 2024-04-21T23:16:57.607 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.606+0000 7f2b21a04740 1 Processor -- start 2024-04-21T23:16:57.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.606+0000 7f2b21a04740 1 -- start start 2024-04-21T23:16:57.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.606+0000 7f2b21a04740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35020 0x5653bdc4a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.606+0000 7f2b21a04740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc35940 0x5653bdc3d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.606+0000 7f2b21a04740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc3e520 0x5653bdc3def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.606+0000 7f2b21a04740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2b181acb00 con 0x5653bdc35020 2024-04-21T23:16:57.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.606+0000 7f2b21a04740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2b181af380 con 0x5653bdc35940 2024-04-21T23:16:57.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.606+0000 7f2b21a04740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2b181ad170 con 0x5653bdc3e520 2024-04-21T23:16:57.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.606+0000 7f2b1cad6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35020 0x5653bdc4a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.607+0000 7f2b1cad6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35020 0x5653bdc4a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59664/0 (socket says 172.21.15.44:59664) 2024-04-21T23:16:57.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.607+0000 7f2b1cad6640 1 -- 172.21.15.44:0/3446396015 learned_addr learned my addr 172.21.15.44:0/3446396015 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:57.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.607+0000 7f2b17fff640 1 --2- 172.21.15.44:0/3446396015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc35940 0x5653bdc3d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.607+0000 7f2b1d2d7640 1 --2- 172.21.15.44:0/3446396015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc3e520 0x5653bdc3def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.607+0000 7f2b1cad6640 1 -- 172.21.15.44:0/3446396015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc3e520 msgr2=0x5653bdc3def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:57.608 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.607+0000 7f2b1cad6640 1 --2- 172.21.15.44:0/3446396015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc3e520 0x5653bdc3def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.607+0000 7f2b1cad6640 1 -- 172.21.15.44:0/3446396015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc35940 msgr2=0x5653bdc3d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:57.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.607+0000 7f2b1cad6640 1 --2- 172.21.15.44:0/3446396015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc35940 0x5653bdc3d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.607+0000 7f2b1cad6640 1 -- 172.21.15.44:0/3446396015 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653bd9777d0 con 0x5653bdc35020 2024-04-21T23:16:57.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.607+0000 7f2b1cad6640 1 --2- 172.21.15.44:0/3446396015 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35020 0x5653bdc4a170 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c02c4e0 tx=0x7f2b0c0f3420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc27cedde260c813 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:57.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.607+0000 7f2b027fc640 1 -- 172.21.15.44:0/3446396015 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c04b040 con 0x5653bdc35020 2024-04-21T23:16:57.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.607+0000 7f2b027fc640 1 -- 172.21.15.44:0/3446396015 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b0c0bd250 con 0x5653bdc35020 2024-04-21T23:16:57.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.607+0000 7f2b027fc640 1 -- 172.21.15.44:0/3446396015 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c0035f0 con 0x5653bdc35020 2024-04-21T23:16:57.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.608+0000 7f2b21a04740 1 -- 172.21.15.44:0/3446396015 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35020 msgr2=0x5653bdc4a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:57.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.608+0000 7f2b21a04740 1 --2- 172.21.15.44:0/3446396015 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35020 0x5653bdc4a170 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c02c4e0 tx=0x7f2b0c0f3420 comp rx=0 tx=0).stop 2024-04-21T23:16:57.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.608+0000 7f2b21a04740 1 -- 172.21.15.44:0/3446396015 shutdown_connections 2024-04-21T23:16:57.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.608+0000 7f2b21a04740 1 --2- 172.21.15.44:0/3446396015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc3e520 0x5653bdc3def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.608+0000 7f2b21a04740 1 --2- 172.21.15.44:0/3446396015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc35940 0x5653bdc3d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.608+0000 7f2b21a04740 1 --2- 172.21.15.44:0/3446396015 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35020 0x5653bdc4a170 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.608+0000 7f2b21a04740 1 -- 172.21.15.44:0/3446396015 >> 172.21.15.44:0/3446396015 conn(0x5653bdc2f750 msgr2=0x5653bdc31360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:57.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.608+0000 7f2b21a04740 1 -- 172.21.15.44:0/3446396015 shutdown_connections 2024-04-21T23:16:57.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.608+0000 7f2b21a04740 1 -- 172.21.15.44:0/3446396015 wait complete. 2024-04-21T23:16:57.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.608+0000 7f2b21a04740 1 -- start start 2024-04-21T23:16:57.610 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.608+0000 7f2b21a04740 1 Processor -- start 2024-04-21T23:16:57.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2b21a04740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc35020 0x5653bdc463f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2b21a04740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35940 0x5653bdc46930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2b21a04740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc3e520 0x5653bdc41a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2b21a04740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2b181ad840 con 0x5653bdc35940 2024-04-21T23:16:57.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2b21a04740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2b181aecb0 con 0x5653bdc3e520 2024-04-21T23:16:57.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2b21a04740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2b181adf10 con 0x5653bdc35020 2024-04-21T23:16:57.613 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2b21a04740 10 client.0 _ll_register_callbacks cb 0x7f2b10022070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:57.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2b17fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35940 0x5653bdc46930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2b17fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35940 0x5653bdc46930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59678/0 (socket says 172.21.15.44:59678) 2024-04-21T23:16:57.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2b17fff640 1 -- 172.21.15.44:0/2129603138 learned_addr learned my addr 172.21.15.44:0/2129603138 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:57.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2b1cad6640 1 --2- 172.21.15.44:0/2129603138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc35020 0x5653bdc463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2b1d2d7640 1 --2- 172.21.15.44:0/2129603138 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc3e520 0x5653bdc41a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2b17fff640 1 -- 172.21.15.44:0/2129603138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc35020 msgr2=0x5653bdc463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:57.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2b17fff640 1 --2- 172.21.15.44:0/2129603138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc35020 0x5653bdc463f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2b17fff640 1 -- 172.21.15.44:0/2129603138 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc3e520 msgr2=0x5653bdc41a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:57.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2b17fff640 1 --2- 172.21.15.44:0/2129603138 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc3e520 0x5653bdc41a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2b17fff640 1 -- 172.21.15.44:0/2129603138 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653bdc42210 con 0x5653bdc35940 2024-04-21T23:16:57.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2b17fff640 1 --2- 172.21.15.44:0/2129603138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35940 0x5653bdc46930 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009620 tx=0x7f2b0400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0c4ec377dd004c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:57.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2af7fff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:16:57.614 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2af7fff640 1 -- 172.21.15.44:0/2129603138 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b04005220 con 0x5653bdc35940 2024-04-21T23:16:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2af7fff640 1 -- 172.21.15.44:0/2129603138 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7f2b040053c0 con 0x5653bdc35940 2024-04-21T23:16:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.609+0000 7f2b21a04740 1 -- 172.21.15.44:0/2129603138 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b0c0bd250 con 0x5653bdc35940 2024-04-21T23:16:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.610+0000 7f2af7fff640 1 -- 172.21.15.44:0/2129603138 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2b04028050 con 0x5653bdc35940 2024-04-21T23:16:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.610+0000 7f2b21a04740 10 client.17472 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.610+0000 7f2b21a04740 1 -- 172.21.15.44:0/2129603138 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5653bdcb4430 con 0x5653bdc35940 2024-04-21T23:16:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.610+0000 7f2af7fff640 1 -- 172.21.15.44:0/2129603138 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b0c0bd250 con 0x5653bdc35940 2024-04-21T23:16:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.610+0000 7f2af7fff640 1 -- 172.21.15.44:0/2129603138 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f2b0401d5d0 con 0x5653bdc35940 2024-04-21T23:16:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.610+0000 7f2b21a04740 10 client.17472 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.610+0000 7f2b21a04740 10 client.17472 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.610+0000 7f2b21a04740 20 client.17472 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.610+0000 7f2b21a04740 1 --2- 172.21.15.44:0/2129603138 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5653bdcb4c60 0x5653bdcb7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.610+0000 7f2b21a04740 4 client.17472 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:16:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.610+0000 7f2b21a04740 1 -- 172.21.15.44:0/2129603138 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5653bdba0dc0 con 0x5653bdcb4c60 2024-04-21T23:16:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.610+0000 7f2b1cad6640 1 --2- 172.21.15.44:0/2129603138 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5653bdcb4c60 0x5653bdcb7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:57.615 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.611+0000 7f2b1cad6640 1 --2- 172.21.15.44:0/2129603138 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5653bdcb4c60 0x5653bdcb7080 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4736275f4c04285 server_cookie=180c1348215c06c7 in_seq=0 out_seq=0 2024-04-21T23:16:57.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.611+0000 7f2af7fff640 10 client.17472 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:57.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.611+0000 7f2af7fff640 1 -- 172.21.15.44:0/2129603138 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5653bdba0dc0 con 0x5653bdcb4c60 2024-04-21T23:16:57.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.611+0000 7f2af7fff640 10 client.17472 handle_command_reply: tid=0 2024-04-21T23:16:57.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.611+0000 7f2b21a04740 1 client.17472 shutdown 2024-04-21T23:16:57.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.611+0000 7f2b21a04740 1 -- 172.21.15.44:0/2129603138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35940 msgr2=0x5653bdc46930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:57.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.611+0000 7f2b21a04740 1 --2- 172.21.15.44:0/2129603138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35940 0x5653bdc46930 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009620 tx=0x7f2b0400f790 comp rx=0 tx=0).stop 2024-04-21T23:16:57.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.611+0000 7f2b21a04740 1 -- 172.21.15.44:0/2129603138 shutdown_connections 2024-04-21T23:16:57.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.611+0000 7f2b21a04740 1 --2- 172.21.15.44:0/2129603138 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5653bdcb4c60 0x5653bdcb7080 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.611+0000 7f2b21a04740 1 --2- 172.21.15.44:0/2129603138 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5653bdc3e520 0x5653bdc41a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.611+0000 7f2b21a04740 1 --2- 172.21.15.44:0/2129603138 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5653bdc35940 0x5653bdc46930 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.611+0000 7f2af7fff640 0 client.17472 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:57.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.611+0000 7f2b21a04740 1 --2- 172.21.15.44:0/2129603138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653bdc35020 0x5653bdc463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.611+0000 7f2b21a04740 1 -- 172.21.15.44:0/2129603138 >> 172.21.15.44:0/2129603138 conn(0x5653bdc2f750 msgr2=0x5653bdc2dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:57.616 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.611+0000 7f2b21a04740 1 -- 172.21.15.44:0/2129603138 shutdown_connections 2024-04-21T23:16:57.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.611+0000 7f2b21a04740 1 -- 172.21.15.44:0/2129603138 wait complete. 2024-04-21T23:16:57.617 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.611+0000 7f2b21a04740 20 client.17472 trim_cache size 0 max 16384 2024-04-21T23:16:57.617 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-21T23:16:57.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.612+0000 7f2b1ed3a640 1 -- 172.21.15.44:0/1638128892 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f2b080a2d00 msgr2=0x7f2b080a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:57.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.612+0000 7f2b1ed3a640 1 --2- 172.21.15.44:0/1638128892 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f2b080a2d00 0x7f2b080a51c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000f8c0 tx=0x7f2b1001e000 comp rx=0 tx=0).stop 2024-04-21T23:16:57.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.612+0000 7f2b1ed3a640 1 -- 172.21.15.44:0/1638128892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b18100610 msgr2=0x7f2b18197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:57.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.612+0000 7f2b1ed3a640 1 --2- 172.21.15.44:0/1638128892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b18100610 0x7f2b18197f90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c02c170 tx=0x7f2b0c02cb20 comp rx=0 tx=0).stop 2024-04-21T23:16:57.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.612+0000 7f2b1ed3a640 1 -- 172.21.15.44:0/1638128892 shutdown_connections 2024-04-21T23:16:57.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.612+0000 7f2b1ed3a640 1 --2- 172.21.15.44:0/1638128892 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f2b080a2d00 0x7f2b080a51c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.612+0000 7f2b1ed3a640 1 --2- 172.21.15.44:0/1638128892 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2b181066b0 0x7f2b1819c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.612+0000 7f2b1ed3a640 1 --2- 172.21.15.44:0/1638128892 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2b18100fd0 0x7f2b181984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.612+0000 7f2b1ed3a640 1 --2- 172.21.15.44:0/1638128892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b18100610 0x7f2b18197f90 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:57.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.612+0000 7f2b1ed3a640 1 -- 172.21.15.44:0/1638128892 >> 172.21.15.44:0/1638128892 conn(0x7f2b180777f0 msgr2=0x7f2b180ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:57.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.612+0000 7f2b1ed3a640 1 -- 172.21.15.44:0/1638128892 shutdown_connections 2024-04-21T23:16:57.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:57.612+0000 7f2b1ed3a640 1 -- 172.21.15.44:0/1638128892 wait complete. 2024-04-21T23:16:57.709 INFO:tasks.workunit.client.0.smithi044.stdout: 1 942009 11.94 MB/sec execute 287 sec latency 43.158 ms 2024-04-21T23:16:58.623 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:16:58.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:16:58.623 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:16:58.645 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:58 smithi044 ceph-mon[27934]: pgmap v255: 129 pgs: 129 active+clean; 89 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.5 MiB/s wr, 357 op/s 2024-04-21T23:16:58.645 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:58 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3429492865' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:16:58.709 INFO:tasks.workunit.client.0.smithi044.stdout: 1 944048 11.93 MB/sec execute 288 sec latency 21.292 ms 2024-04-21T23:16:58.713 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:58 smithi055 ceph-mon[31117]: pgmap v255: 129 pgs: 129 active+clean; 89 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.5 MiB/s wr, 357 op/s 2024-04-21T23:16:58.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:58 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3429492865' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:16:58.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:58 smithi069 ceph-mon[31816]: pgmap v255: 129 pgs: 129 active+clean; 89 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.5 MiB/s wr, 357 op/s 2024-04-21T23:16:58.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:58 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3429492865' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:16:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.739+0000 7fc44d8c2640 1 -- 172.21.15.44:0/987167613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4481029b0 msgr2=0x7fc448102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.739+0000 7fc44d8c2640 1 --2- 172.21.15.44:0/987167613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4481029b0 0x7fc448102d90 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fc440009900 tx=0x7fc44002bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.739+0000 7fc44d8c2640 1 -- 172.21.15.44:0/987167613 shutdown_connections 2024-04-21T23:16:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.739+0000 7fc44d8c2640 1 --2- 172.21.15.44:0/987167613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc448103360 0x7fc44810b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.739+0000 7fc44d8c2640 1 --2- 172.21.15.44:0/987167613 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc44810bec0 0x7fc44810e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.739+0000 7fc44d8c2640 1 --2- 172.21.15.44:0/987167613 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc4481029b0 0x7fc448102d90 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.739+0000 7fc44d8c2640 1 -- 172.21.15.44:0/987167613 >> 172.21.15.44:0/987167613 conn(0x7fc448077760 msgr2=0x7fc448077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.739+0000 7fc44d8c2640 1 -- 172.21.15.44:0/987167613 shutdown_connections 2024-04-21T23:16:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.739+0000 7fc44d8c2640 1 -- 172.21.15.44:0/987167613 wait complete. 2024-04-21T23:16:58.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.739+0000 7fc44d8c2640 1 Processor -- start 2024-04-21T23:16:58.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.740+0000 7fc44d8c2640 1 -- start start 2024-04-21T23:16:58.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.740+0000 7fc44d8c2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4481029b0 0x7fc4481981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.740+0000 7fc44d8c2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc44810bec0 0x7fc448198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.740+0000 7fc44d8c2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc44819ca40 0x7fc44819cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.740+0000 7fc44d8c2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc448111cb0 con 0x7fc44810bec0 2024-04-21T23:16:58.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.740+0000 7fc44d8c2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc448111b30 con 0x7fc4481029b0 2024-04-21T23:16:58.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.740+0000 7fc44d8c2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc448111e30 con 0x7fc44819ca40 2024-04-21T23:16:58.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.740+0000 7fc4467fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc44810bec0 0x7fc448198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.740+0000 7fc446ffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4481029b0 0x7fc4481981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.740+0000 7fc4467fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc44810bec0 0x7fc448198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59710/0 (socket says 172.21.15.44:59710) 2024-04-21T23:16:58.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.740+0000 7fc4467fc640 1 -- 172.21.15.44:0/3801381553 learned_addr learned my addr 172.21.15.44:0/3801381553 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:58.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.740+0000 7fc4477fe640 1 --2- 172.21.15.44:0/3801381553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc44819ca40 0x7fc44819cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.740+0000 7fc4467fc640 1 -- 172.21.15.44:0/3801381553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc44819ca40 msgr2=0x7fc44819cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.740+0000 7fc4467fc640 1 --2- 172.21.15.44:0/3801381553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc44819ca40 0x7fc44819cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.740+0000 7fc4467fc640 1 -- 172.21.15.44:0/3801381553 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4481029b0 msgr2=0x7fc4481981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.740+0000 7fc4467fc640 1 --2- 172.21.15.44:0/3801381553 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4481029b0 0x7fc4481981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.740+0000 7fc4467fc640 1 -- 172.21.15.44:0/3801381553 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc44819d530 con 0x7fc44810bec0 2024-04-21T23:16:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.741+0000 7fc4467fc640 1 --2- 172.21.15.44:0/3801381553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc44810bec0 0x7fc448198710 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fc43000c8c0 tx=0x7fc43000cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.741+0000 7fc44c8c0640 1 -- 172.21.15.44:0/3801381553 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc430007bb0 con 0x7fc44810bec0 2024-04-21T23:16:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.741+0000 7fc44d8c2640 1 -- 172.21.15.44:0/3801381553 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc44819d820 con 0x7fc44810bec0 2024-04-21T23:16:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.741+0000 7fc44c8c0640 1 -- 172.21.15.44:0/3801381553 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc430007d50 con 0x7fc44810bec0 2024-04-21T23:16:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.741+0000 7fc44c8c0640 1 -- 172.21.15.44:0/3801381553 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc430018830 con 0x7fc44810bec0 2024-04-21T23:16:58.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.741+0000 7fc44d8c2640 1 -- 172.21.15.44:0/3801381553 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4481a5010 con 0x7fc44810bec0 2024-04-21T23:16:58.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.743+0000 7fc44c8c0640 1 -- 172.21.15.44:0/3801381553 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc430018a10 con 0x7fc44810bec0 2024-04-21T23:16:58.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.743+0000 7fc450582740 1 Processor -- start 2024-04-21T23:16:58.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc450582740 1 -- start start 2024-04-21T23:16:58.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc44c8c0640 1 --2- 172.21.15.44:0/3801381553 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc4340a2d00 0x7fc4340a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc450582740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c593470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc450582740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c593a40 0x55864c593ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc450582740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c59cdb0 0x55864c59f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc450582740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55864c4f7170 con 0x55864c594720 2024-04-21T23:16:58.749 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc450582740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55864c49f7c0 con 0x55864c593a40 2024-04-21T23:16:58.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc450582740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55864c4f3130 con 0x55864c59cdb0 2024-04-21T23:16:58.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc4477fe640 1 --2- 172.21.15.44:0/3801381553 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc4340a2d00 0x7fc4340a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc44c8c0640 1 -- 172.21.15.44:0/3801381553 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7fc4300cf8f0 con 0x7fc44810bec0 2024-04-21T23:16:58.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc446ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c593470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc446ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c593470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59716/0 (socket says 172.21.15.44:59716) 2024-04-21T23:16:58.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc446ffd640 1 -- 172.21.15.44:0/2138568038 learned_addr learned my addr 172.21.15.44:0/2138568038 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:58.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc4467fc640 1 --2- 172.21.15.44:0/2138568038 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c593a40 0x55864c593ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc4467fc640 1 -- 172.21.15.44:0/2138568038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c59cdb0 msgr2=0x55864c59f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:58.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc4467fc640 1 --2- 172.21.15.44:0/2138568038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c59cdb0 0x55864c59f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc4467fc640 1 -- 172.21.15.44:0/2138568038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 msgr2=0x55864c593470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:58.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc4467fc640 1 --2- 172.21.15.44:0/2138568038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c593470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc4467fc640 1 -- 172.21.15.44:0/2138568038 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55864c4b0e60 con 0x55864c593a40 2024-04-21T23:16:58.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc446ffd640 1 --2- 172.21.15.44:0/2138568038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c593470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:58.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc4477fe640 1 --2- 172.21.15.44:0/3801381553 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc4340a2d00 0x7fc4340a51c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc438005fd0 tx=0x7fc4380107a0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:58.750 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.744+0000 7fc4467fc640 1 --2- 172.21.15.44:0/2138568038 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c593a40 0x55864c593ec0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc4300ccca0 tx=0x7fc43004a240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8092f00d5b525a2f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.745+0000 7fc43dffb640 1 -- 172.21.15.44:0/2138568038 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43009fe20 con 0x55864c593a40 2024-04-21T23:16:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.745+0000 7fc43dffb640 1 -- 172.21.15.44:0/2138568038 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4300a46e0 con 0x55864c593a40 2024-04-21T23:16:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.745+0000 7fc43dffb640 1 -- 172.21.15.44:0/2138568038 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc430053060 con 0x55864c593a40 2024-04-21T23:16:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.745+0000 7fc450582740 1 -- 172.21.15.44:0/2138568038 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c593a40 msgr2=0x55864c593ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.745+0000 7fc450582740 1 --2- 172.21.15.44:0/2138568038 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c593a40 0x55864c593ec0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc4300ccca0 tx=0x7fc43004a240 comp rx=0 tx=0).stop 2024-04-21T23:16:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.746+0000 7fc450582740 1 -- 172.21.15.44:0/2138568038 shutdown_connections 2024-04-21T23:16:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.746+0000 7fc450582740 1 --2- 172.21.15.44:0/2138568038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c59cdb0 0x55864c59f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.746+0000 7fc450582740 1 --2- 172.21.15.44:0/2138568038 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c593a40 0x55864c593ec0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.746+0000 7fc450582740 1 --2- 172.21.15.44:0/2138568038 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c593470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.746+0000 7fc450582740 1 -- 172.21.15.44:0/2138568038 >> 172.21.15.44:0/2138568038 conn(0x55864c55a4f0 msgr2=0x55864c58e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.746+0000 7fc450582740 1 -- 172.21.15.44:0/2138568038 shutdown_connections 2024-04-21T23:16:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.746+0000 7fc450582740 1 -- 172.21.15.44:0/2138568038 wait complete. 2024-04-21T23:16:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.746+0000 7fc450582740 1 -- start start 2024-04-21T23:16:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.746+0000 7fc450582740 1 Processor -- start 2024-04-21T23:16:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.747+0000 7fc450582740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c593a40 0x55864c599df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.751 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.747+0000 7fc450582740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c59a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.747+0000 7fc450582740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c59cdb0 0x55864c59a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.747+0000 7fc450582740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55864c59fc20 con 0x55864c594720 2024-04-21T23:16:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.747+0000 7fc450582740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55864c4b2d40 con 0x55864c59cdb0 2024-04-21T23:16:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.747+0000 7fc450582740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55864c59fda0 con 0x55864c593a40 2024-04-21T23:16:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.747+0000 7fc450582740 10 client.0 _ll_register_callbacks cb 0x55864c4ee820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.747+0000 7fc446ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c593a40 0x55864c599df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.747+0000 7fc446ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c593a40 0x55864c599df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:34998/0 (socket says 172.21.15.44:34998) 2024-04-21T23:16:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.747+0000 7fc4477fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c59cdb0 0x55864c59a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.747+0000 7fc446ffd640 1 -- 172.21.15.44:0/3504079930 learned_addr learned my addr 172.21.15.44:0/3504079930 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.747+0000 7fc4467fc640 1 --2- 172.21.15.44:0/3504079930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c59a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.747+0000 7fc446ffd640 1 -- 172.21.15.44:0/3504079930 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c59cdb0 msgr2=0x55864c59a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.747+0000 7fc446ffd640 1 --2- 172.21.15.44:0/3504079930 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c59cdb0 0x55864c59a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.747+0000 7fc446ffd640 1 -- 172.21.15.44:0/3504079930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 msgr2=0x55864c59a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:58.752 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.747+0000 7fc446ffd640 1 --2- 172.21.15.44:0/3504079930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c59a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.747+0000 7fc446ffd640 1 -- 172.21.15.44:0/3504079930 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55864c2d37e0 con 0x55864c593a40 2024-04-21T23:16:58.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.747+0000 7fc446ffd640 1 --2- 172.21.15.44:0/3504079930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c593a40 0x55864c599df0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc440036040 tx=0x7fc440004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1559514a606af696 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:58.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.747+0000 7fc426ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:16:58.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.747+0000 7fc426ffd640 1 -- 172.21.15.44:0/3504079930 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc440004470 con 0x55864c593a40 2024-04-21T23:16:58.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.748+0000 7fc450582740 1 -- 172.21.15.44:0/3504079930 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4300a46e0 con 0x55864c593a40 2024-04-21T23:16:58.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.748+0000 7fc426ffd640 1 -- 172.21.15.44:0/3504079930 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7fc4400046c0 con 0x55864c593a40 2024-04-21T23:16:58.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.748+0000 7fc426ffd640 1 -- 172.21.15.44:0/3504079930 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc4400048c0 con 0x55864c593a40 2024-04-21T23:16:58.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.748+0000 7fc450582740 10 client.27013 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:58.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.748+0000 7fc450582740 1 -- 172.21.15.44:0/3504079930 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55864c333f70 con 0x55864c593a40 2024-04-21T23:16:58.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.748+0000 7fc426ffd640 1 -- 172.21.15.44:0/3504079930 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4300a46e0 con 0x55864c593a40 2024-04-21T23:16:58.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.748+0000 7fc426ffd640 1 -- 172.21.15.44:0/3504079930 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7fc440003710 con 0x55864c593a40 2024-04-21T23:16:58.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.748+0000 7fc450582740 10 client.27013 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:58.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.748+0000 7fc450582740 10 client.27013 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:58.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.748+0000 7fc450582740 20 client.27013 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:58.753 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.748+0000 7fc450582740 1 --2- 172.21.15.44:0/3504079930 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55864c5acf40 0x55864c5af370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.748+0000 7fc450582740 4 client.27013 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:16:58.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.748+0000 7fc450582740 1 -- 172.21.15.44:0/3504079930 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55864c4cb450 con 0x55864c5acf40 2024-04-21T23:16:58.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.748+0000 7fc4467fc640 1 --2- 172.21.15.44:0/3504079930 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55864c5acf40 0x55864c5af370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.749+0000 7fc426ffd640 10 client.27013 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:58.754 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.749+0000 7fc4467fc640 1 --2- 172.21.15.44:0/3504079930 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55864c5acf40 0x55864c5af370 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddc3a10735f6d6a2 server_cookie=7a8bf5110ade4fdc in_seq=0 out_seq=0 2024-04-21T23:16:58.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.750+0000 7fc426ffd640 1 -- 172.21.15.44:0/3504079930 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55864c4cb450 con 0x55864c5acf40 2024-04-21T23:16:58.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.750+0000 7fc426ffd640 10 client.27013 handle_command_reply: tid=0 2024-04-21T23:16:58.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.750+0000 7fc450582740 1 client.27013 shutdown 2024-04-21T23:16:58.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.750+0000 7fc450582740 1 -- 172.21.15.44:0/3504079930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c593a40 msgr2=0x55864c599df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.750+0000 7fc450582740 1 --2- 172.21.15.44:0/3504079930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c593a40 0x55864c599df0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc440036040 tx=0x7fc440004040 comp rx=0 tx=0).stop 2024-04-21T23:16:58.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.750+0000 7fc450582740 1 -- 172.21.15.44:0/3504079930 shutdown_connections 2024-04-21T23:16:58.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.750+0000 7fc450582740 1 --2- 172.21.15.44:0/3504079930 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55864c5acf40 0x55864c5af370 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.755 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.750+0000 7fc450582740 1 --2- 172.21.15.44:0/3504079930 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c59cdb0 0x55864c59a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.750+0000 7fc450582740 1 --2- 172.21.15.44:0/3504079930 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c59a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.750+0000 7fc450582740 1 --2- 172.21.15.44:0/3504079930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c593a40 0x55864c599df0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.750+0000 7fc426ffd640 0 client.27013 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:58.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.750+0000 7fc450582740 1 -- 172.21.15.44:0/3504079930 >> 172.21.15.44:0/3504079930 conn(0x55864c55a4f0 msgr2=0x55864c5901e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:58.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.750+0000 7fc450582740 1 -- 172.21.15.44:0/3504079930 shutdown_connections 2024-04-21T23:16:58.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.750+0000 7fc450582740 1 -- 172.21.15.44:0/3504079930 wait complete. 2024-04-21T23:16:58.756 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.750+0000 7fc450582740 20 client.27013 trim_cache size 0 max 16384 2024-04-21T23:16:58.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.770+0000 7fc450582740 1 Processor -- start 2024-04-21T23:16:58.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.770+0000 7fc450582740 1 -- start start 2024-04-21T23:16:58.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.770+0000 7fc450582740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c593a40 0x55864c59c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.770+0000 7fc450582740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c5a8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.770+0000 7fc450582740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c59cdb0 0x55864c5b4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.770+0000 7fc450582740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc4481aaa70 con 0x55864c594720 2024-04-21T23:16:58.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.770+0000 7fc450582740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc4481ad2f0 con 0x55864c59cdb0 2024-04-21T23:16:58.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.770+0000 7fc450582740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc4481ab0e0 con 0x55864c593a40 2024-04-21T23:16:58.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.770+0000 7fc4467fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c5a8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.770+0000 7fc4467fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c5a8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59728/0 (socket says 172.21.15.44:59728) 2024-04-21T23:16:58.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.770+0000 7fc4467fc640 1 -- 172.21.15.44:0/939600575 learned_addr learned my addr 172.21.15.44:0/939600575 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:58.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.770+0000 7fc4477fe640 1 --2- 172.21.15.44:0/939600575 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c59cdb0 0x55864c5b4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.770+0000 7fc446ffd640 1 --2- 172.21.15.44:0/939600575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c593a40 0x55864c59c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.771+0000 7fc4467fc640 1 -- 172.21.15.44:0/939600575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c593a40 msgr2=0x55864c59c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:58.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.771+0000 7fc4467fc640 1 --2- 172.21.15.44:0/939600575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c593a40 0x55864c59c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.771+0000 7fc4467fc640 1 -- 172.21.15.44:0/939600575 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c59cdb0 msgr2=0x55864c5b4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:58.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.771+0000 7fc4467fc640 1 --2- 172.21.15.44:0/939600575 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c59cdb0 0x55864c5b4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.771+0000 7fc4467fc640 1 -- 172.21.15.44:0/939600575 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55864c1cb170 con 0x55864c594720 2024-04-21T23:16:58.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.771+0000 7fc4467fc640 1 --2- 172.21.15.44:0/939600575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c5a8b20 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fc430007590 tx=0x7fc430086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cd4223302d3b39e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:58.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.771+0000 7fc43d7fa640 1 -- 172.21.15.44:0/939600575 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc430018e20 con 0x55864c594720 2024-04-21T23:16:58.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.771+0000 7fc43d7fa640 1 -- 172.21.15.44:0/939600575 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc43000fac0 con 0x55864c594720 2024-04-21T23:16:58.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.771+0000 7fc43d7fa640 1 -- 172.21.15.44:0/939600575 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4300a4250 con 0x55864c594720 2024-04-21T23:16:58.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.772+0000 7fc450582740 1 -- 172.21.15.44:0/939600575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 msgr2=0x55864c5a8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.772+0000 7fc450582740 1 --2- 172.21.15.44:0/939600575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c5a8b20 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fc430007590 tx=0x7fc430086040 comp rx=0 tx=0).stop 2024-04-21T23:16:58.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.772+0000 7fc450582740 1 -- 172.21.15.44:0/939600575 shutdown_connections 2024-04-21T23:16:58.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.772+0000 7fc450582740 1 --2- 172.21.15.44:0/939600575 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c59cdb0 0x55864c5b4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.772+0000 7fc450582740 1 --2- 172.21.15.44:0/939600575 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c5a8b20 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.772+0000 7fc450582740 1 --2- 172.21.15.44:0/939600575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c593a40 0x55864c59c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.772+0000 7fc450582740 1 -- 172.21.15.44:0/939600575 >> 172.21.15.44:0/939600575 conn(0x55864c55a4f0 msgr2=0x55864c598f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:58.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.772+0000 7fc450582740 1 -- 172.21.15.44:0/939600575 shutdown_connections 2024-04-21T23:16:58.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.772+0000 7fc450582740 1 -- 172.21.15.44:0/939600575 wait complete. 2024-04-21T23:16:58.774 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.772+0000 7fc450582740 1 -- start start 2024-04-21T23:16:58.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.772+0000 7fc450582740 1 Processor -- start 2024-04-21T23:16:58.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.773+0000 7fc450582740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c593a40 0x55864c5a5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.773+0000 7fc450582740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c5970e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.773+0000 7fc450582740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c59cdb0 0x55864c597680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.773+0000 7fc450582740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fc4481ab7b0 con 0x55864c594720 2024-04-21T23:16:58.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.773+0000 7fc450582740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fc4481acc20 con 0x55864c593a40 2024-04-21T23:16:58.777 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.773+0000 7fc450582740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc4481abe80 con 0x55864c59cdb0 2024-04-21T23:16:58.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.773+0000 7fc450582740 10 client.0 _ll_register_callbacks cb 0x7fc440009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:58.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.773+0000 7fc4467fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c5970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.773+0000 7fc4467fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c5970e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59740/0 (socket says 172.21.15.44:59740) 2024-04-21T23:16:58.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.773+0000 7fc4467fc640 1 -- 172.21.15.44:0/1447149617 learned_addr learned my addr 172.21.15.44:0/1447149617 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:58.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.773+0000 7fc446ffd640 1 --2- 172.21.15.44:0/1447149617 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c593a40 0x55864c5a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.773+0000 7fc4477fe640 1 --2- 172.21.15.44:0/1447149617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c59cdb0 0x55864c597680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.773+0000 7fc4467fc640 1 -- 172.21.15.44:0/1447149617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c59cdb0 msgr2=0x55864c597680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:58.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.773+0000 7fc4467fc640 1 --2- 172.21.15.44:0/1447149617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c59cdb0 0x55864c597680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.773+0000 7fc4467fc640 1 -- 172.21.15.44:0/1447149617 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c593a40 msgr2=0x55864c5a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:58.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.773+0000 7fc4467fc640 1 --2- 172.21.15.44:0/1447149617 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c593a40 0x55864c5a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.773+0000 7fc4467fc640 1 -- 172.21.15.44:0/1447149617 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55864c59d870 con 0x55864c594720 2024-04-21T23:16:58.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.774+0000 7fc4467fc640 1 --2- 172.21.15.44:0/1447149617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c5970e0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fc430097ed0 tx=0x7fc43000fc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9f18437b67b5a0d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:58.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.774+0000 7fc426ffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:16:58.778 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.774+0000 7fc426ffd640 1 -- 172.21.15.44:0/1447149617 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4300a46e0 con 0x55864c594720 2024-04-21T23:16:58.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.774+0000 7fc426ffd640 1 -- 172.21.15.44:0/1447149617 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7fc4300ba2f0 con 0x55864c594720 2024-04-21T23:16:58.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.774+0000 7fc450582740 1 -- 172.21.15.44:0/1447149617 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc43000fac0 con 0x55864c594720 2024-04-21T23:16:58.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.774+0000 7fc426ffd640 1 -- 172.21.15.44:0/1447149617 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc4300ccf20 con 0x55864c594720 2024-04-21T23:16:58.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.774+0000 7fc450582740 10 client.17490 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:58.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.774+0000 7fc450582740 1 -- 172.21.15.44:0/1447149617 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55864c613630 con 0x55864c594720 2024-04-21T23:16:58.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.774+0000 7fc426ffd640 1 -- 172.21.15.44:0/1447149617 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc43000fac0 con 0x55864c594720 2024-04-21T23:16:58.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.774+0000 7fc426ffd640 1 -- 172.21.15.44:0/1447149617 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7fc430052fc0 con 0x55864c594720 2024-04-21T23:16:58.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.774+0000 7fc450582740 10 client.17490 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:58.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.774+0000 7fc450582740 10 client.17490 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:58.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.774+0000 7fc450582740 20 client.17490 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:58.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.774+0000 7fc450582740 1 --2- 172.21.15.44:0/1447149617 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55864c613800 0x55864c615bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.774+0000 7fc450582740 4 client.17490 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:16:58.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.774+0000 7fc450582740 1 -- 172.21.15.44:0/1447149617 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55864c5046b0 con 0x55864c613800 2024-04-21T23:16:58.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.775+0000 7fc446ffd640 1 --2- 172.21.15.44:0/1447149617 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55864c613800 0x55864c615bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.779 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.775+0000 7fc446ffd640 1 --2- 172.21.15.44:0/1447149617 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55864c613800 0x55864c615bf0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54eec2713a85a707 server_cookie=5f48d7db51852049 in_seq=0 out_seq=0 2024-04-21T23:16:58.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.775+0000 7fc426ffd640 10 client.17490 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:58.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.776+0000 7fc426ffd640 1 -- 172.21.15.44:0/1447149617 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55864c5046b0 con 0x55864c613800 2024-04-21T23:16:58.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.776+0000 7fc426ffd640 10 client.17490 handle_command_reply: tid=0 2024-04-21T23:16:58.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.776+0000 7fc450582740 1 client.17490 shutdown 2024-04-21T23:16:58.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.776+0000 7fc450582740 1 -- 172.21.15.44:0/1447149617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 msgr2=0x55864c5970e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.776+0000 7fc450582740 1 --2- 172.21.15.44:0/1447149617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c5970e0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fc430097ed0 tx=0x7fc43000fc70 comp rx=0 tx=0).stop 2024-04-21T23:16:58.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.777+0000 7fc450582740 1 -- 172.21.15.44:0/1447149617 shutdown_connections 2024-04-21T23:16:58.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.777+0000 7fc450582740 1 --2- 172.21.15.44:0/1447149617 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55864c613800 0x55864c615bf0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.777+0000 7fc450582740 1 --2- 172.21.15.44:0/1447149617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55864c59cdb0 0x55864c597680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.777+0000 7fc450582740 1 --2- 172.21.15.44:0/1447149617 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55864c594720 0x55864c5970e0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.777+0000 7fc450582740 1 --2- 172.21.15.44:0/1447149617 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55864c593a40 0x55864c5a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.777+0000 7fc450582740 1 -- 172.21.15.44:0/1447149617 >> 172.21.15.44:0/1447149617 conn(0x55864c55a4f0 msgr2=0x55864c58cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:58.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.777+0000 7fc426ffd640 0 client.17490 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:58.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.777+0000 7fc450582740 1 -- 172.21.15.44:0/1447149617 shutdown_connections 2024-04-21T23:16:58.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.777+0000 7fc450582740 1 -- 172.21.15.44:0/1447149617 wait complete. 2024-04-21T23:16:58.780 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.777+0000 7fc450582740 20 client.17490 trim_cache size 0 max 16384 2024-04-21T23:16:58.781 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:16:58.781 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-21T23:16:58.781 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "96dae338-6355-4f4b-84bb-c3f9c701dae5", 2024-04-21T23:16:58.781 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-21T23:16:58.781 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:16:58.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.777+0000 7fc44d8c2640 1 -- 172.21.15.44:0/3801381553 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc4340a2d00 msgr2=0x7fc4340a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.777+0000 7fc44d8c2640 1 --2- 172.21.15.44:0/3801381553 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc4340a2d00 0x7fc4340a51c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc438005fd0 tx=0x7fc4380107a0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.777+0000 7fc44d8c2640 1 -- 172.21.15.44:0/3801381553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc44810bec0 msgr2=0x7fc448198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.777+0000 7fc44d8c2640 1 --2- 172.21.15.44:0/3801381553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc44810bec0 0x7fc448198710 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fc43000c8c0 tx=0x7fc43000cd80 comp rx=0 tx=0).stop 2024-04-21T23:16:58.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.777+0000 7fc44d8c2640 1 -- 172.21.15.44:0/3801381553 shutdown_connections 2024-04-21T23:16:58.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.777+0000 7fc44d8c2640 1 --2- 172.21.15.44:0/3801381553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc44819ca40 0x7fc44819cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.777+0000 7fc44d8c2640 1 --2- 172.21.15.44:0/3801381553 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fc4340a2d00 0x7fc4340a51c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.777+0000 7fc44d8c2640 1 --2- 172.21.15.44:0/3801381553 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fc44810bec0 0x7fc448198710 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.777+0000 7fc44d8c2640 1 --2- 172.21.15.44:0/3801381553 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4481029b0 0x7fc4481981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.782 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.777+0000 7fc44d8c2640 1 -- 172.21.15.44:0/3801381553 >> 172.21.15.44:0/3801381553 conn(0x7fc448077760 msgr2=0x7fc4480fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:58.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.778+0000 7fc44d8c2640 1 -- 172.21.15.44:0/3801381553 shutdown_connections 2024-04-21T23:16:58.783 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.778+0000 7fc44d8c2640 1 -- 172.21.15.44:0/3801381553 wait complete. 2024-04-21T23:16:58.789 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:16:58.789 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:16:58.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.942+0000 7fccc2d1e640 1 -- 172.21.15.44:0/4059068775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccbc1066b0 msgr2=0x7fccbc106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.942+0000 7fccc2d1e640 1 --2- 172.21.15.44:0/4059068775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccbc1066b0 0x7fccbc106a90 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009930 tx=0x7fccb002bc30 comp rx=0 tx=0).stop 2024-04-21T23:16:58.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.942+0000 7fccc2d1e640 1 -- 172.21.15.44:0/4059068775 shutdown_connections 2024-04-21T23:16:58.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.942+0000 7fccc2d1e640 1 --2- 172.21.15.44:0/4059068775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccbc100610 0x7fccbc100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.942+0000 7fccc2d1e640 1 --2- 172.21.15.44:0/4059068775 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fccbc100fd0 0x7fccbc10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.942+0000 7fccc2d1e640 1 --2- 172.21.15.44:0/4059068775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccbc1066b0 0x7fccbc106a90 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.942+0000 7fccc2d1e640 1 -- 172.21.15.44:0/4059068775 >> 172.21.15.44:0/4059068775 conn(0x7fccbc0777f0 msgr2=0x7fccbc077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:58.942 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.942+0000 7fccc2d1e640 1 -- 172.21.15.44:0/4059068775 shutdown_connections 2024-04-21T23:16:58.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.942+0000 7fccc2d1e640 1 -- 172.21.15.44:0/4059068775 wait complete. 2024-04-21T23:16:58.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.943+0000 7fccc2d1e640 1 Processor -- start 2024-04-21T23:16:58.943 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.943+0000 7fccc2d1e640 1 -- start start 2024-04-21T23:16:58.944 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.943+0000 7fccc2d1e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccbc100610 0x7fccbc197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.943+0000 7fccc2d1e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fccbc100fd0 0x7fccbc1984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.943+0000 7fccc2d1e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccbc1066b0 0x7fccbc19c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.943+0000 7fccc2d1e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc111b80 con 0x7fccbc100610 2024-04-21T23:16:58.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.943+0000 7fccc2d1e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc111a00 con 0x7fccbc100fd0 2024-04-21T23:16:58.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.943+0000 7fccc2d1e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc111d00 con 0x7fccbc1066b0 2024-04-21T23:16:58.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.943+0000 7fccc0aba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccbc100610 0x7fccbc197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.943+0000 7fccc0aba640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccbc100610 0x7fccbc197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59748/0 (socket says 172.21.15.44:59748) 2024-04-21T23:16:58.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.943+0000 7fccc0aba640 1 -- 172.21.15.44:0/550693369 learned_addr learned my addr 172.21.15.44:0/550693369 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:58.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.943+0000 7fccbbfff640 1 --2- 172.21.15.44:0/550693369 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fccbc100fd0 0x7fccbc1984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.943+0000 7fccc12bb640 1 --2- 172.21.15.44:0/550693369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccbc1066b0 0x7fccbc19c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.943+0000 7fccc0aba640 1 -- 172.21.15.44:0/550693369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccbc1066b0 msgr2=0x7fccbc19c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.943+0000 7fccc0aba640 1 --2- 172.21.15.44:0/550693369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccbc1066b0 0x7fccbc19c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.944+0000 7fccc0aba640 1 -- 172.21.15.44:0/550693369 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fccbc100fd0 msgr2=0x7fccbc1984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.945 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.944+0000 7fccc0aba640 1 --2- 172.21.15.44:0/550693369 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fccbc100fd0 0x7fccbc1984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.944+0000 7fccc0aba640 1 -- 172.21.15.44:0/550693369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccbc19cf40 con 0x7fccbc100610 2024-04-21T23:16:58.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.944+0000 7fccbbfff640 1 --2- 172.21.15.44:0/550693369 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fccbc100fd0 0x7fccbc1984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:58.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.944+0000 7fccc0aba640 1 --2- 172.21.15.44:0/550693369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccbc100610 0x7fccbc197fa0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009900 tx=0x7fccb0004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:58.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.944+0000 7fccb9ffb640 1 -- 172.21.15.44:0/550693369 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb0043020 con 0x7fccbc100610 2024-04-21T23:16:58.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.944+0000 7fccb9ffb640 1 -- 172.21.15.44:0/550693369 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccb00054e0 con 0x7fccbc100610 2024-04-21T23:16:58.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.944+0000 7fccc2d1e640 1 -- 172.21.15.44:0/550693369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccbc19d1d0 con 0x7fccbc100610 2024-04-21T23:16:58.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.944+0000 7fccb9ffb640 1 -- 172.21.15.44:0/550693369 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb0003b80 con 0x7fccbc100610 2024-04-21T23:16:58.946 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.944+0000 7fccc2d1e640 1 -- 172.21.15.44:0/550693369 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccbc1a4ac0 con 0x7fccbc100610 2024-04-21T23:16:58.952 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.946+0000 7fccb9ffb640 1 -- 172.21.15.44:0/550693369 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fccb0046020 con 0x7fccbc100610 2024-04-21T23:16:58.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.946+0000 7fccc59e3740 1 Processor -- start 2024-04-21T23:16:58.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.946+0000 7fccc59e3740 1 -- start start 2024-04-21T23:16:58.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.947+0000 7fccc59e3740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c35f30 0x5557f5c36310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.947+0000 7fccc59e3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c36850 0x5557f5c3eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.947+0000 7fccc59e3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c3f430 0x5557f5c41820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.947+0000 7fccc59e3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5557f5b9a560 con 0x5557f5c3f430 2024-04-21T23:16:58.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.947+0000 7fccc59e3740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5557f5b42730 con 0x5557f5c35f30 2024-04-21T23:16:58.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.947+0000 7fccc59e3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5557f5b96520 con 0x5557f5c36850 2024-04-21T23:16:58.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.947+0000 7fccbbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c3f430 0x5557f5c41820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.947+0000 7fccc12bb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c35f30 0x5557f5c36310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.947+0000 7fccc12bb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c35f30 0x5557f5c36310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:35474/0 (socket says 172.21.15.44:35474) 2024-04-21T23:16:58.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.947+0000 7fccc12bb640 1 -- 172.21.15.44:0/2616896337 learned_addr learned my addr 172.21.15.44:0/2616896337 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:58.953 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.947+0000 7fccc0aba640 1 --2- 172.21.15.44:0/2616896337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c36850 0x5557f5c3eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.947+0000 7fccc12bb640 1 -- 172.21.15.44:0/2616896337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c36850 msgr2=0x5557f5c3eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:58.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.947+0000 7fccc12bb640 1 --2- 172.21.15.44:0/2616896337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c36850 0x5557f5c3eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.947+0000 7fccc12bb640 1 -- 172.21.15.44:0/2616896337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c3f430 msgr2=0x5557f5c41820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:58.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.947+0000 7fccc12bb640 1 --2- 172.21.15.44:0/2616896337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c3f430 0x5557f5c41820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.947+0000 7fccc12bb640 1 -- 172.21.15.44:0/2616896337 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557f59787d0 con 0x5557f5c35f30 2024-04-21T23:16:58.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.947+0000 7fccc0aba640 1 --2- 172.21.15.44:0/2616896337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c36850 0x5557f5c3eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:16:58.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.947+0000 7fccb9ffb640 1 --2- 172.21.15.44:0/550693369 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fccac0a2d00 0x7fccac0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.947+0000 7fccc12bb640 1 --2- 172.21.15.44:0/2616896337 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c35f30 0x5557f5c36310 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fccb4009fa0 tx=0x7fccb40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13652e9ae2191f5a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:58.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.947+0000 7fccc0aba640 1 --2- 172.21.15.44:0/550693369 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fccac0a2d00 0x7fccac0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.948+0000 7fcca6ffd640 1 -- 172.21.15.44:0/2616896337 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb4004d20 con 0x5557f5c35f30 2024-04-21T23:16:58.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.948+0000 7fcca6ffd640 1 -- 172.21.15.44:0/2616896337 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccb4004050 con 0x5557f5c35f30 2024-04-21T23:16:58.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.948+0000 7fcca6ffd640 1 -- 172.21.15.44:0/2616896337 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb4005280 con 0x5557f5c35f30 2024-04-21T23:16:58.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.948+0000 7fccb9ffb640 1 -- 172.21.15.44:0/550693369 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7fccb00f1740 con 0x7fccbc100610 2024-04-21T23:16:58.954 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.948+0000 7fccc59e3740 1 -- 172.21.15.44:0/2616896337 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c35f30 msgr2=0x5557f5c36310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.948+0000 7fccc0aba640 1 --2- 172.21.15.44:0/550693369 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fccac0a2d00 0x7fccac0a51c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fccb00eecb0 tx=0x7fccb002c450 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:58.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.948+0000 7fccc59e3740 1 --2- 172.21.15.44:0/2616896337 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c35f30 0x5557f5c36310 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fccb4009fa0 tx=0x7fccb40118d0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.948+0000 7fccc59e3740 1 -- 172.21.15.44:0/2616896337 shutdown_connections 2024-04-21T23:16:58.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.948+0000 7fccc59e3740 1 --2- 172.21.15.44:0/2616896337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c36850 0x5557f5c3eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.948+0000 7fccc59e3740 1 --2- 172.21.15.44:0/2616896337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c3f430 0x5557f5c41820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.948+0000 7fccc59e3740 1 --2- 172.21.15.44:0/2616896337 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c35f30 0x5557f5c36310 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.948+0000 7fccc59e3740 1 -- 172.21.15.44:0/2616896337 >> 172.21.15.44:0/2616896337 conn(0x5557f5c30660 msgr2=0x5557f5c30a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:58.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.949+0000 7fccc59e3740 1 -- 172.21.15.44:0/2616896337 shutdown_connections 2024-04-21T23:16:58.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.949+0000 7fccc59e3740 1 -- 172.21.15.44:0/2616896337 wait complete. 2024-04-21T23:16:58.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.949+0000 7fccc59e3740 1 -- start start 2024-04-21T23:16:58.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.949+0000 7fccc59e3740 1 Processor -- start 2024-04-21T23:16:58.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.949+0000 7fccc59e3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c35f30 0x5557f5c3c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.949+0000 7fccc59e3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c36850 0x5557f5c3c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.955 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.949+0000 7fccc59e3740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c3f430 0x5557f5c3cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.949+0000 7fccc59e3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5557f5c42220 con 0x5557f5c35f30 2024-04-21T23:16:58.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.949+0000 7fccc59e3740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5557f5b55e00 con 0x5557f5c3f430 2024-04-21T23:16:58.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.949+0000 7fccc59e3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5557f5c423a0 con 0x5557f5c36850 2024-04-21T23:16:58.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.950+0000 7fccc59e3740 10 client.0 _ll_register_callbacks cb 0x5557f5b90eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:58.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.950+0000 7fccbbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c36850 0x5557f5c3c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.950+0000 7fccc12bb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c35f30 0x5557f5c3c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.950+0000 7fccbbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c36850 0x5557f5c3c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:35038/0 (socket says 172.21.15.44:35038) 2024-04-21T23:16:58.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.950+0000 7fccc12bb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c35f30 0x5557f5c3c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59772/0 (socket says 172.21.15.44:59772) 2024-04-21T23:16:58.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.950+0000 7fccbbfff640 1 -- 172.21.15.44:0/722102807 learned_addr learned my addr 172.21.15.44:0/722102807 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:58.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.950+0000 7fccc12bb640 1 --2- 172.21.15.44:0/722102807 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c3f430 0x5557f5c3cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.950+0000 7fccbbfff640 1 -- 172.21.15.44:0/722102807 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c3f430 msgr2=0x5557f5c3cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:58.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.950+0000 7fccbbfff640 1 --2- 172.21.15.44:0/722102807 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c3f430 0x5557f5c3cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.950+0000 7fccbbfff640 1 -- 172.21.15.44:0/722102807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c35f30 msgr2=0x5557f5c3c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:58.956 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.950+0000 7fccbbfff640 1 --2- 172.21.15.44:0/722102807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c35f30 0x5557f5c3c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.950+0000 7fccbbfff640 1 -- 172.21.15.44:0/722102807 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557f58ae3f0 con 0x5557f5c36850 2024-04-21T23:16:58.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.950+0000 7fccbbfff640 1 --2- 172.21.15.44:0/722102807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c36850 0x5557f5c3c9c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcca8011770 tx=0x7fcca8011c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c47d011ac3c1c56 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:58.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.950+0000 7fcc9bfff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:16:58.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.950+0000 7fcc9bfff640 1 -- 172.21.15.44:0/722102807 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca8010ba0 con 0x5557f5c36850 2024-04-21T23:16:58.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.950+0000 7fcc9bfff640 1 -- 172.21.15.44:0/722102807 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7fcca8010d40 con 0x5557f5c36850 2024-04-21T23:16:58.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.950+0000 7fccc59e3740 1 -- 172.21.15.44:0/722102807 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fccb4004050 con 0x5557f5c36850 2024-04-21T23:16:58.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.950+0000 7fcc9bfff640 1 -- 172.21.15.44:0/722102807 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcca8025050 con 0x5557f5c36850 2024-04-21T23:16:58.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.950+0000 7fccc59e3740 10 client.27028 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:58.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.951+0000 7fccc59e3740 1 -- 172.21.15.44:0/722102807 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5557f586e0f0 con 0x5557f5c36850 2024-04-21T23:16:58.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.951+0000 7fcc9bfff640 1 -- 172.21.15.44:0/722102807 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccb4004050 con 0x5557f5c36850 2024-04-21T23:16:58.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.951+0000 7fcc9bfff640 1 -- 172.21.15.44:0/722102807 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7fcca800f570 con 0x5557f5c36850 2024-04-21T23:16:58.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.951+0000 7fccc59e3740 10 client.27028 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:58.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.951+0000 7fccc59e3740 10 client.27028 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:58.957 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.951+0000 7fccc59e3740 20 client.27028 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:58.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.951+0000 7fccc59e3740 1 --2- 172.21.15.44:0/722102807 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5557f5c4f740 0x5557f5c51bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.951+0000 7fccc59e3740 4 client.27028 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:16:58.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.951+0000 7fccc59e3740 1 -- 172.21.15.44:0/722102807 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557f5ba7aa0 con 0x5557f5c4f740 2024-04-21T23:16:58.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.951+0000 7fccc12bb640 1 --2- 172.21.15.44:0/722102807 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5557f5c4f740 0x5557f5c51bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.951+0000 7fcc9bfff640 10 client.27028 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:58.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.951+0000 7fccc12bb640 1 --2- 172.21.15.44:0/722102807 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5557f5c4f740 0x5557f5c51bc0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ed113c1858c6dc1 server_cookie=1d1da977184b8290 in_seq=0 out_seq=0 2024-04-21T23:16:58.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.952+0000 7fcc9bfff640 1 -- 172.21.15.44:0/722102807 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5557f5ba7aa0 con 0x5557f5c4f740 2024-04-21T23:16:58.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.952+0000 7fcc9bfff640 10 client.27028 handle_command_reply: tid=0 2024-04-21T23:16:58.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.952+0000 7fccc59e3740 1 client.27028 shutdown 2024-04-21T23:16:58.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.952+0000 7fccc59e3740 1 -- 172.21.15.44:0/722102807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c36850 msgr2=0x5557f5c3c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.952+0000 7fccc59e3740 1 --2- 172.21.15.44:0/722102807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c36850 0x5557f5c3c9c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcca8011770 tx=0x7fcca8011c30 comp rx=0 tx=0).stop 2024-04-21T23:16:58.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.952+0000 7fccc59e3740 1 -- 172.21.15.44:0/722102807 shutdown_connections 2024-04-21T23:16:58.958 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.952+0000 7fccc59e3740 1 --2- 172.21.15.44:0/722102807 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5557f5c4f740 0x5557f5c51bc0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.952+0000 7fccc59e3740 1 --2- 172.21.15.44:0/722102807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c36850 0x5557f5c3c9c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.952+0000 7fccc59e3740 1 --2- 172.21.15.44:0/722102807 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c3f430 0x5557f5c3cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.952+0000 7fcc9bfff640 0 client.27028 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:58.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.952+0000 7fccc59e3740 1 --2- 172.21.15.44:0/722102807 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c35f30 0x5557f5c3c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.952+0000 7fccc59e3740 1 -- 172.21.15.44:0/722102807 >> 172.21.15.44:0/722102807 conn(0x5557f5c30660 msgr2=0x5557f5c323b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:58.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.952+0000 7fccc59e3740 1 -- 172.21.15.44:0/722102807 shutdown_connections 2024-04-21T23:16:58.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.952+0000 7fccc59e3740 1 -- 172.21.15.44:0/722102807 wait complete. 2024-04-21T23:16:58.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.953+0000 7fccc59e3740 20 client.27028 trim_cache size 0 max 16384 2024-04-21T23:16:58.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.978+0000 7fccc59e3740 1 Processor -- start 2024-04-21T23:16:58.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.978+0000 7fccc59e3740 1 -- start start 2024-04-21T23:16:58.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.978+0000 7fccc59e3740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c35f30 0x5557f5c3ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.978+0000 7fccc59e3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c36850 0x5557f5c4b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.978+0000 7fccc59e3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c3f430 0x5557f5c32140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.978+0000 7fccc59e3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc1acaf0 con 0x5557f5c36850 2024-04-21T23:16:58.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.978+0000 7fccc59e3740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc1af370 con 0x5557f5c35f30 2024-04-21T23:16:58.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.978+0000 7fccc59e3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc1ad160 con 0x5557f5c3f430 2024-04-21T23:16:58.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.978+0000 7fccbbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c36850 0x5557f5c4b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.978+0000 7fccc12bb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c35f30 0x5557f5c3ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.978+0000 7fccbbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c36850 0x5557f5c4b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59778/0 (socket says 172.21.15.44:59778) 2024-04-21T23:16:58.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.978+0000 7fccc12bb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c35f30 0x5557f5c3ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:35496/0 (socket says 172.21.15.44:35496) 2024-04-21T23:16:58.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.978+0000 7fccbbfff640 1 -- 172.21.15.44:0/2879185012 learned_addr learned my addr 172.21.15.44:0/2879185012 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:58.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.978+0000 7fccc12bb640 1 --2- 172.21.15.44:0/2879185012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c3f430 0x5557f5c32140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.978+0000 7fccbbfff640 1 -- 172.21.15.44:0/2879185012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c3f430 msgr2=0x5557f5c32140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:58.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.978+0000 7fccbbfff640 1 --2- 172.21.15.44:0/2879185012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c3f430 0x5557f5c32140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.978+0000 7fccbbfff640 1 -- 172.21.15.44:0/2879185012 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c35f30 msgr2=0x5557f5c3ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:58.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.978+0000 7fccbbfff640 1 --2- 172.21.15.44:0/2879185012 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c35f30 0x5557f5c3ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.978+0000 7fccbbfff640 1 -- 172.21.15.44:0/2879185012 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557f5c41820 con 0x5557f5c36850 2024-04-21T23:16:58.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.978+0000 7fccc12bb640 1 --2- 172.21.15.44:0/2879185012 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c35f30 0x5557f5c3ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:58.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.979+0000 7fccbbfff640 1 --2- 172.21.15.44:0/2879185012 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c36850 0x5557f5c4b0e0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fcca800be50 tx=0x7fcca8010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0efacbc6aaf78bc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:58.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.979+0000 7fcca67fc640 1 -- 172.21.15.44:0/2879185012 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca802d070 con 0x5557f5c36850 2024-04-21T23:16:58.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.979+0000 7fcca67fc640 1 -- 172.21.15.44:0/2879185012 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcca8005bb0 con 0x5557f5c36850 2024-04-21T23:16:58.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.979+0000 7fcca67fc640 1 -- 172.21.15.44:0/2879185012 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca8004280 con 0x5557f5c36850 2024-04-21T23:16:58.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.979+0000 7fccc59e3740 1 -- 172.21.15.44:0/2879185012 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c36850 msgr2=0x5557f5c4b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.979+0000 7fccc59e3740 1 --2- 172.21.15.44:0/2879185012 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c36850 0x5557f5c4b0e0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fcca800be50 tx=0x7fcca8010800 comp rx=0 tx=0).stop 2024-04-21T23:16:58.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.979+0000 7fccc59e3740 1 -- 172.21.15.44:0/2879185012 shutdown_connections 2024-04-21T23:16:58.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.979+0000 7fccc59e3740 1 --2- 172.21.15.44:0/2879185012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c3f430 0x5557f5c32140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.979+0000 7fccc59e3740 1 --2- 172.21.15.44:0/2879185012 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c36850 0x5557f5c4b0e0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.979+0000 7fccc59e3740 1 --2- 172.21.15.44:0/2879185012 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c35f30 0x5557f5c3ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.979+0000 7fccc59e3740 1 -- 172.21.15.44:0/2879185012 >> 172.21.15.44:0/2879185012 conn(0x5557f5c30660 msgr2=0x5557f5c31a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:58.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.980+0000 7fccc59e3740 1 -- 172.21.15.44:0/2879185012 shutdown_connections 2024-04-21T23:16:58.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.980+0000 7fccc59e3740 1 -- 172.21.15.44:0/2879185012 wait complete. 2024-04-21T23:16:58.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.980+0000 7fccc59e3740 1 -- start start 2024-04-21T23:16:58.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.980+0000 7fccc59e3740 1 Processor -- start 2024-04-21T23:16:58.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.980+0000 7fccc59e3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c35f30 0x5557f5c39500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.980+0000 7fccc59e3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c36850 0x5557f5c39a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.980+0000 7fccc59e3740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c3f430 0x5557f5cb4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.980+0000 7fccc59e3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc1ad830 con 0x5557f5c36850 2024-04-21T23:16:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.980+0000 7fccc59e3740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc1aeca0 con 0x5557f5c3f430 2024-04-21T23:16:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.980+0000 7fccc59e3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fccbc1adf00 con 0x5557f5c35f30 2024-04-21T23:16:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.980+0000 7fccc59e3740 10 client.0 _ll_register_callbacks cb 0x7fcca800bbd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.980+0000 7fccbbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c36850 0x5557f5c39a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.980+0000 7fccbbfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c36850 0x5557f5c39a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59794/0 (socket says 172.21.15.44:59794) 2024-04-21T23:16:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.980+0000 7fccbbfff640 1 -- 172.21.15.44:0/2352176183 learned_addr learned my addr 172.21.15.44:0/2352176183 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.980+0000 7fccc12bb640 1 --2- 172.21.15.44:0/2352176183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c3f430 0x5557f5cb4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.981+0000 7fccc12bb640 1 --2- 172.21.15.44:0/2352176183 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c35f30 0x5557f5c39500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.981+0000 7fccbbfff640 1 -- 172.21.15.44:0/2352176183 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c35f30 msgr2=0x5557f5c39500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.981+0000 7fccbbfff640 1 --2- 172.21.15.44:0/2352176183 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c35f30 0x5557f5c39500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.981+0000 7fccbbfff640 1 -- 172.21.15.44:0/2352176183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c3f430 msgr2=0x5557f5cb4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.981+0000 7fccbbfff640 1 --2- 172.21.15.44:0/2352176183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c3f430 0x5557f5cb4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.981+0000 7fccbbfff640 1 -- 172.21.15.44:0/2352176183 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557f5cb5240 con 0x5557f5c36850 2024-04-21T23:16:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.981+0000 7fccbbfff640 1 --2- 172.21.15.44:0/2352176183 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c36850 0x5557f5c39a40 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fcca800b620 tx=0x7fcca800fdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27201393394a8cec server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.981+0000 7fcc9bfff640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:16:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.981+0000 7fcc9bfff640 1 -- 172.21.15.44:0/2352176183 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca802d040 con 0x5557f5c36850 2024-04-21T23:16:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.981+0000 7fcc9bfff640 1 -- 172.21.15.44:0/2352176183 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7fcca800f3c0 con 0x5557f5c36850 2024-04-21T23:16:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.981+0000 7fccc59e3740 1 -- 172.21.15.44:0/2352176183 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcca8005bb0 con 0x5557f5c36850 2024-04-21T23:16:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.981+0000 7fcc9bfff640 1 -- 172.21.15.44:0/2352176183 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcca8010e50 con 0x5557f5c36850 2024-04-21T23:16:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.981+0000 7fccc59e3740 10 client.17508 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.981+0000 7fccc59e3740 1 -- 172.21.15.44:0/2352176183 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5557f5cb6390 con 0x5557f5c36850 2024-04-21T23:16:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.982+0000 7fcc9bfff640 1 -- 172.21.15.44:0/2352176183 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcca8005bb0 con 0x5557f5c36850 2024-04-21T23:16:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.982+0000 7fcc9bfff640 1 -- 172.21.15.44:0/2352176183 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7fcca8005350 con 0x5557f5c36850 2024-04-21T23:16:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.982+0000 7fccc59e3740 10 client.17508 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:58.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.982+0000 7fccc59e3740 10 client.17508 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:58.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.982+0000 7fccc59e3740 20 client.17508 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:58.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.982+0000 7fccc59e3740 1 --2- 172.21.15.44:0/2352176183 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5557f5cb6a40 0x5557f5cb8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:58.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.982+0000 7fccc59e3740 4 client.17508 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:16:58.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.982+0000 7fccc59e3740 1 -- 172.21.15.44:0/2352176183 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5557f5ba1df0 con 0x5557f5cb6a40 2024-04-21T23:16:58.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.982+0000 7fccc12bb640 1 --2- 172.21.15.44:0/2352176183 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5557f5cb6a40 0x5557f5cb8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:58.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.983+0000 7fccc12bb640 1 --2- 172.21.15.44:0/2352176183 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5557f5cb6a40 0x5557f5cb8e30 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db94645a6842c362 server_cookie=56c94d8918dee438 in_seq=0 out_seq=0 2024-04-21T23:16:58.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.983+0000 7fcc9bfff640 10 client.17508 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:58.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.984+0000 7fcc9bfff640 1 -- 172.21.15.44:0/2352176183 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5557f5ba1df0 con 0x5557f5cb6a40 2024-04-21T23:16:58.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.984+0000 7fcc9bfff640 10 client.17508 handle_command_reply: tid=0 2024-04-21T23:16:58.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.984+0000 7fccc59e3740 1 client.17508 shutdown 2024-04-21T23:16:58.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc59e3740 1 -- 172.21.15.44:0/2352176183 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c36850 msgr2=0x5557f5c39a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc59e3740 1 --2- 172.21.15.44:0/2352176183 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c36850 0x5557f5c39a40 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fcca800b620 tx=0x7fcca800fdb0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc59e3740 1 -- 172.21.15.44:0/2352176183 shutdown_connections 2024-04-21T23:16:58.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc59e3740 1 --2- 172.21.15.44:0/2352176183 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5557f5cb6a40 0x5557f5cb8e30 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc59e3740 1 --2- 172.21.15.44:0/2352176183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5557f5c3f430 0x5557f5cb4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fcc9bfff640 0 client.17508 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:58.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc59e3740 1 --2- 172.21.15.44:0/2352176183 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5557f5c36850 0x5557f5c39a40 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc59e3740 1 --2- 172.21.15.44:0/2352176183 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5557f5c35f30 0x5557f5c39500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc59e3740 1 -- 172.21.15.44:0/2352176183 >> 172.21.15.44:0/2352176183 conn(0x5557f5c30660 msgr2=0x5557f5c322c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:58.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc59e3740 1 -- 172.21.15.44:0/2352176183 shutdown_connections 2024-04-21T23:16:58.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc59e3740 1 -- 172.21.15.44:0/2352176183 wait complete. 2024-04-21T23:16:58.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc59e3740 20 client.17508 trim_cache size 0 max 16384 2024-04-21T23:16:58.989 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:16:58.989 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:16:58.990 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-21T23:16:58.990 INFO:teuthology.orchestra.run.smithi044.stdout: "96dae338-6355-4f4b-84bb-c3f9c701dae5": { 2024-04-21T23:16:58.990 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-21T23:16:58.990 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "96dae338-6355-4f4b-84bb-c3f9c701dae5", 2024-04-21T23:16:58.990 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-04-21T23:16:58.990 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:16:58.990 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:16:58.990 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:16:58.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc2d1e640 1 -- 172.21.15.44:0/550693369 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fccac0a2d00 msgr2=0x7fccac0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc2d1e640 1 --2- 172.21.15.44:0/550693369 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fccac0a2d00 0x7fccac0a51c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fccb00eecb0 tx=0x7fccb002c450 comp rx=0 tx=0).stop 2024-04-21T23:16:58.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc2d1e640 1 -- 172.21.15.44:0/550693369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccbc100610 msgr2=0x7fccbc197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:58.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc2d1e640 1 --2- 172.21.15.44:0/550693369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccbc100610 0x7fccbc197fa0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009900 tx=0x7fccb0004360 comp rx=0 tx=0).stop 2024-04-21T23:16:58.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc2d1e640 1 -- 172.21.15.44:0/550693369 shutdown_connections 2024-04-21T23:16:58.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc2d1e640 1 --2- 172.21.15.44:0/550693369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccbc1066b0 0x7fccbc19c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc2d1e640 1 --2- 172.21.15.44:0/550693369 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fccac0a2d00 0x7fccac0a51c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc2d1e640 1 --2- 172.21.15.44:0/550693369 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fccbc100fd0 0x7fccbc1984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc2d1e640 1 --2- 172.21.15.44:0/550693369 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fccbc100610 0x7fccbc197fa0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:58.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.985+0000 7fccc2d1e640 1 -- 172.21.15.44:0/550693369 >> 172.21.15.44:0/550693369 conn(0x7fccbc0777f0 msgr2=0x7fccbc0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:58.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.986+0000 7fccc2d1e640 1 -- 172.21.15.44:0/550693369 shutdown_connections 2024-04-21T23:16:58.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:16:58.986+0000 7fccc2d1e640 1 -- 172.21.15.44:0/550693369 wait complete. 2024-04-21T23:16:58.996 INFO:tasks.cephfs.filesystem:scrub status for tag:96dae338-6355-4f4b-84bb-c3f9c701dae5 - {'path': '/', 'tag': '96dae338-6355-4f4b-84bb-c3f9c701dae5', 'options': 'recursive,force'} 2024-04-21T23:16:59.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:59 smithi044 ceph-mon[27934]: scrub queued for path: / 2024-04-21T23:16:59.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:59 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:16:59.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:59 smithi044 ceph-mon[27934]: scrub summary: active paths [/] 2024-04-21T23:16:59.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:16:59 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:16:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.481+0000 7faabe8fb640 1 -- 172.21.15.44:0/3514969206 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab810bd80 msgr2=0x7faab810e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.481+0000 7faabe8fb640 1 --2- 172.21.15.44:0/3514969206 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab810bd80 0x7faab810e1b0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7faab0009ca0 tx=0x7faab002c190 comp rx=0 tx=0).stop 2024-04-21T23:16:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.481+0000 7faabe8fb640 1 -- 172.21.15.44:0/3514969206 shutdown_connections 2024-04-21T23:16:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.481+0000 7faabe8fb640 1 --2- 172.21.15.44:0/3514969206 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab810bd80 0x7faab810e1b0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.481+0000 7faabe8fb640 1 --2- 172.21.15.44:0/3514969206 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faab81032d0 0x7faab810b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.481+0000 7faabe8fb640 1 --2- 172.21.15.44:0/3514969206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faab81029b0 0x7faab8102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.481+0000 7faabe8fb640 1 -- 172.21.15.44:0/3514969206 >> 172.21.15.44:0/3514969206 conn(0x7faab8077760 msgr2=0x7faab8077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.481+0000 7faabe8fb640 1 -- 172.21.15.44:0/3514969206 shutdown_connections 2024-04-21T23:16:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.482+0000 7faabe8fb640 1 -- 172.21.15.44:0/3514969206 wait complete. 2024-04-21T23:16:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.482+0000 7faabe8fb640 1 Processor -- start 2024-04-21T23:16:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.482+0000 7faabe8fb640 1 -- start start 2024-04-21T23:16:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.482+0000 7faabe8fb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faab81029b0 0x7faab8197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.482+0000 7faabe8fb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab81032d0 0x7faab8198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.483+0000 7faabe8fb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faab810bd80 0x7faab819c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.483+0000 7faabe8fb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faab8111b10 con 0x7faab81032d0 2024-04-21T23:16:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.483+0000 7faabe8fb640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7faab8111990 con 0x7faab810bd80 2024-04-21T23:16:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.483+0000 7faabe8fb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faab8111c90 con 0x7faab81029b0 2024-04-21T23:16:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.483+0000 7faab77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab81032d0 0x7faab8198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.483+0000 7faab77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab81032d0 0x7faab8198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59820/0 (socket says 172.21.15.44:59820) 2024-04-21T23:16:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.483+0000 7faab77fe640 1 -- 172.21.15.44:0/2348470027 learned_addr learned my addr 172.21.15.44:0/2348470027 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.483+0000 7faabce98640 1 --2- 172.21.15.44:0/2348470027 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faab810bd80 0x7faab819c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.483+0000 7faab7fff640 1 --2- 172.21.15.44:0/2348470027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faab81029b0 0x7faab8197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.483+0000 7faab77fe640 1 -- 172.21.15.44:0/2348470027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faab81029b0 msgr2=0x7faab8197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.483+0000 7faab77fe640 1 --2- 172.21.15.44:0/2348470027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faab81029b0 0x7faab8197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.483+0000 7faab77fe640 1 -- 172.21.15.44:0/2348470027 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faab810bd80 msgr2=0x7faab819c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.483+0000 7faab77fe640 1 --2- 172.21.15.44:0/2348470027 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faab810bd80 0x7faab819c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.483+0000 7faab77fe640 1 -- 172.21.15.44:0/2348470027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab819cfe0 con 0x7faab81032d0 2024-04-21T23:16:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.483+0000 7faab77fe640 1 --2- 172.21.15.44:0/2348470027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab81032d0 0x7faab8198530 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7faaa400e9f0 tx=0x7faaa400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.483+0000 7faab57fa640 1 -- 172.21.15.44:0/2348470027 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa400cd80 con 0x7faab81032d0 2024-04-21T23:16:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.483+0000 7faab57fa640 1 -- 172.21.15.44:0/2348470027 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaa4010910 con 0x7faab81032d0 2024-04-21T23:16:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.483+0000 7faabe8fb640 1 -- 172.21.15.44:0/2348470027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faab819d2d0 con 0x7faab81032d0 2024-04-21T23:16:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.483+0000 7faab57fa640 1 -- 172.21.15.44:0/2348470027 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa4018990 con 0x7faab81032d0 2024-04-21T23:16:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.483+0000 7faabe8fb640 1 -- 172.21.15.44:0/2348470027 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faab81a4b60 con 0x7faab81032d0 2024-04-21T23:16:59.492 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.485+0000 7faab57fa640 1 -- 172.21.15.44:0/2348470027 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7faaa4010ac0 con 0x7faab81032d0 2024-04-21T23:16:59.492 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.486+0000 7faac15be740 1 Processor -- start 2024-04-21T23:16:59.492 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.486+0000 7faac15be740 1 -- start start 2024-04-21T23:16:59.492 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.486+0000 7faac15be740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5560f06a9b20 0x5560f06a9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.486+0000 7faab57fa640 1 --2- 172.21.15.44:0/2348470027 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faaa80a2cf0 0x7faaa80a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.486+0000 7faac15be740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06aa440 0x5560f06b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.486+0000 7faac15be740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 0x5560f06b5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.486+0000 7faac15be740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5560f060db70 con 0x5560f06b3210 2024-04-21T23:16:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.486+0000 7faac15be740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5560f05b6470 con 0x5560f06aa440 2024-04-21T23:16:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.486+0000 7faac15be740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5560f0609b30 con 0x5560f06a9b20 2024-04-21T23:16:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.486+0000 7faab7fff640 1 --2- 172.21.15.44:0/2348470027 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faaa80a2cf0 0x7faaa80a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.486+0000 7faabce98640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 0x5560f06b5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.486+0000 7faabce98640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 0x5560f06b5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59828/0 (socket says 172.21.15.44:59828) 2024-04-21T23:16:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.486+0000 7faabce98640 1 -- 172.21.15.44:0/261099251 learned_addr learned my addr 172.21.15.44:0/261099251 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.486+0000 7faab77fe640 1 --2- 172.21.15.44:0/261099251 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06aa440 0x5560f06b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.486+0000 7faab57fa640 1 -- 172.21.15.44:0/2348470027 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7faaa40cfb30 con 0x7faab81032d0 2024-04-21T23:16:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.487+0000 7faabce98640 1 --2- 172.21.15.44:0/261099251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5560f06a9b20 0x5560f06a9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.487+0000 7faabce98640 1 -- 172.21.15.44:0/261099251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5560f06a9b20 msgr2=0x5560f06a9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.487+0000 7faabce98640 1 --2- 172.21.15.44:0/261099251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5560f06a9b20 0x5560f06a9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.487+0000 7faabce98640 1 -- 172.21.15.44:0/261099251 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06aa440 msgr2=0x5560f06b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.487+0000 7faabce98640 1 --2- 172.21.15.44:0/261099251 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06aa440 0x5560f06b2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.487+0000 7faabce98640 1 -- 172.21.15.44:0/261099251 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560f0322210 con 0x5560f06b3210 2024-04-21T23:16:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.487+0000 7faab77fe640 1 --2- 172.21.15.44:0/261099251 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06aa440 0x5560f06b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.487+0000 7faab7fff640 1 --2- 172.21.15.44:0/2348470027 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faaa80a2cf0 0x7faaa80a51b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7faaac005fd0 tx=0x7faaac005850 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.487+0000 7faabce98640 1 --2- 172.21.15.44:0/261099251 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 0x5560f06b5600 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7faab806a040 tx=0x7faab002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=682501c68a9cc9f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.487+0000 7faaa27fc640 1 -- 172.21.15.44:0/261099251 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab0039040 con 0x5560f06b3210 2024-04-21T23:16:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.487+0000 7faaa27fc640 1 -- 172.21.15.44:0/261099251 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faab0048070 con 0x5560f06b3210 2024-04-21T23:16:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.487+0000 7faaa27fc640 1 -- 172.21.15.44:0/261099251 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab0041a70 con 0x5560f06b3210 2024-04-21T23:16:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.488+0000 7faac15be740 1 -- 172.21.15.44:0/261099251 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 msgr2=0x5560f06b5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.488+0000 7faac15be740 1 --2- 172.21.15.44:0/261099251 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 0x5560f06b5600 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7faab806a040 tx=0x7faab002e040 comp rx=0 tx=0).stop 2024-04-21T23:16:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.488+0000 7faac15be740 1 -- 172.21.15.44:0/261099251 shutdown_connections 2024-04-21T23:16:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.488+0000 7faac15be740 1 --2- 172.21.15.44:0/261099251 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 0x5560f06b5600 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.488+0000 7faac15be740 1 --2- 172.21.15.44:0/261099251 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06aa440 0x5560f06b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.488+0000 7faac15be740 1 --2- 172.21.15.44:0/261099251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5560f06a9b20 0x5560f06a9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.488+0000 7faac15be740 1 -- 172.21.15.44:0/261099251 >> 172.21.15.44:0/261099251 conn(0x5560f06a4440 msgr2=0x5560f06a4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.488+0000 7faac15be740 1 -- 172.21.15.44:0/261099251 shutdown_connections 2024-04-21T23:16:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.488+0000 7faac15be740 1 -- 172.21.15.44:0/261099251 wait complete. 2024-04-21T23:16:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.488+0000 7faac15be740 1 -- start start 2024-04-21T23:16:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.488+0000 7faac15be740 1 Processor -- start 2024-04-21T23:16:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faac15be740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5560f06a9b20 0x5560f06b0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faac15be740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06aa440 0x5560f06b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faac15be740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 0x5560f06b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faac15be740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5560f06b5de0 con 0x5560f06b3210 2024-04-21T23:16:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faac15be740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5560f05c9a10 con 0x5560f06aa440 2024-04-21T23:16:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faac15be740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5560f06b5f60 con 0x5560f06a9b20 2024-04-21T23:16:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faac15be740 10 client.0 _ll_register_callbacks cb 0x5560f0604b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faabce98640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 0x5560f06b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faabce98640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 0x5560f06b0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59832/0 (socket says 172.21.15.44:59832) 2024-04-21T23:16:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faabce98640 1 -- 172.21.15.44:0/1929965548 learned_addr learned my addr 172.21.15.44:0/1929965548 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faab77fe640 1 --2- 172.21.15.44:0/1929965548 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06aa440 0x5560f06b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faab7fff640 1 --2- 172.21.15.44:0/1929965548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5560f06a9b20 0x5560f06b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faabce98640 1 -- 172.21.15.44:0/1929965548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5560f06a9b20 msgr2=0x5560f06b0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faabce98640 1 --2- 172.21.15.44:0/1929965548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5560f06a9b20 0x5560f06b0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faabce98640 1 -- 172.21.15.44:0/1929965548 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06aa440 msgr2=0x5560f06b07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faabce98640 1 --2- 172.21.15.44:0/1929965548 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06aa440 0x5560f06b07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faabce98640 1 -- 172.21.15.44:0/1929965548 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560f03ee620 con 0x5560f06b3210 2024-04-21T23:16:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faab77fe640 1 --2- 172.21.15.44:0/1929965548 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06aa440 0x5560f06b07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faabce98640 1 --2- 172.21.15.44:0/1929965548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 0x5560f06b0d00 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7faab002c6d0 tx=0x7faab0041cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df23b285c4b960c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faa977fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:16:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faa977fe640 1 -- 172.21.15.44:0/1929965548 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab0052020 con 0x5560f06b3210 2024-04-21T23:16:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faac15be740 1 -- 172.21.15.44:0/1929965548 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faab0048070 con 0x5560f06b3210 2024-04-21T23:16:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.489+0000 7faa977fe640 1 -- 172.21.15.44:0/1929965548 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7faab0032a80 con 0x5560f06b3210 2024-04-21T23:16:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.490+0000 7faa977fe640 1 -- 172.21.15.44:0/1929965548 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7faab005d050 con 0x5560f06b3210 2024-04-21T23:16:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.490+0000 7faac15be740 10 client.17520 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.490+0000 7faac15be740 1 -- 172.21.15.44:0/1929965548 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5560f06c20d0 con 0x5560f06b3210 2024-04-21T23:16:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.490+0000 7faa977fe640 1 -- 172.21.15.44:0/1929965548 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faab0048070 con 0x5560f06b3210 2024-04-21T23:16:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.490+0000 7faa977fe640 1 -- 172.21.15.44:0/1929965548 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7faab003f040 con 0x5560f06b3210 2024-04-21T23:16:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.490+0000 7faac15be740 10 client.17520 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.490+0000 7faac15be740 10 client.17520 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.490+0000 7faac15be740 20 client.17520 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.490+0000 7faac15be740 1 --2- 172.21.15.44:0/1929965548 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5560f06c3420 0x5560f06c5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.490+0000 7faac15be740 4 client.17520 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:16:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.490+0000 7faac15be740 1 -- 172.21.15.44:0/1929965548 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560f061b0b0 con 0x5560f06c3420 2024-04-21T23:16:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.490+0000 7faab7fff640 1 --2- 172.21.15.44:0/1929965548 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5560f06c3420 0x5560f06c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.491+0000 7faa977fe640 10 client.17520 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.491+0000 7faab7fff640 1 --2- 172.21.15.44:0/1929965548 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5560f06c3420 0x5560f06c5810 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba5548218092ca81 server_cookie=a0aca1f548c9e1d2 in_seq=0 out_seq=0 2024-04-21T23:16:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.491+0000 7faa977fe640 1 -- 172.21.15.44:0/1929965548 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5560f061b0b0 con 0x5560f06c3420 2024-04-21T23:16:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.491+0000 7faa977fe640 10 client.17520 handle_command_reply: tid=0 2024-04-21T23:16:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.491+0000 7faac15be740 1 client.17520 shutdown 2024-04-21T23:16:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.491+0000 7faac15be740 1 -- 172.21.15.44:0/1929965548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 msgr2=0x5560f06b0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.492+0000 7faac15be740 1 --2- 172.21.15.44:0/1929965548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 0x5560f06b0d00 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7faab002c6d0 tx=0x7faab0041cb0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.492+0000 7faac15be740 1 -- 172.21.15.44:0/1929965548 shutdown_connections 2024-04-21T23:16:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.492+0000 7faac15be740 1 --2- 172.21.15.44:0/1929965548 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5560f06c3420 0x5560f06c5810 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.492+0000 7faac15be740 1 --2- 172.21.15.44:0/1929965548 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 0x5560f06b0d00 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.492+0000 7faa977fe640 0 client.17520 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.492+0000 7faac15be740 1 --2- 172.21.15.44:0/1929965548 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06aa440 0x5560f06b07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.492+0000 7faac15be740 1 --2- 172.21.15.44:0/1929965548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5560f06a9b20 0x5560f06b0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.492+0000 7faac15be740 1 -- 172.21.15.44:0/1929965548 >> 172.21.15.44:0/1929965548 conn(0x5560f06a4440 msgr2=0x5560f06a6190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.492+0000 7faac15be740 1 -- 172.21.15.44:0/1929965548 shutdown_connections 2024-04-21T23:16:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.492+0000 7faac15be740 1 -- 172.21.15.44:0/1929965548 wait complete. 2024-04-21T23:16:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.492+0000 7faac15be740 20 client.17520 trim_cache size 0 max 16384 2024-04-21T23:16:59.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faac15be740 1 Processor -- start 2024-04-21T23:16:59.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faac15be740 1 -- start start 2024-04-21T23:16:59.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faac15be740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06a9b20 0x5560f06beca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:59.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faac15be740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5560f06aa440 0x5560f06b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:59.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faac15be740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 0x5560f06cade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:59.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faac15be740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faab81acb60 con 0x5560f06b3210 2024-04-21T23:16:59.508 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faac15be740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7faab81af3e0 con 0x5560f06a9b20 2024-04-21T23:16:59.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faac15be740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faab81ad1d0 con 0x5560f06aa440 2024-04-21T23:16:59.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faabce98640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 0x5560f06cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:59.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faabce98640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 0x5560f06cade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59840/0 (socket says 172.21.15.44:59840) 2024-04-21T23:16:59.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faabce98640 1 -- 172.21.15.44:0/977186448 learned_addr learned my addr 172.21.15.44:0/977186448 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:59.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faab7fff640 1 --2- 172.21.15.44:0/977186448 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06a9b20 0x5560f06beca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:59.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faabce98640 1 -- 172.21.15.44:0/977186448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5560f06aa440 msgr2=0x5560f06b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:59.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faabce98640 1 --2- 172.21.15.44:0/977186448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5560f06aa440 0x5560f06b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faabce98640 1 -- 172.21.15.44:0/977186448 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06a9b20 msgr2=0x5560f06beca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:59.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faabce98640 1 --2- 172.21.15.44:0/977186448 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06a9b20 0x5560f06beca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faabce98640 1 -- 172.21.15.44:0/977186448 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560f06af820 con 0x5560f06b3210 2024-04-21T23:16:59.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faabce98640 1 --2- 172.21.15.44:0/977186448 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 0x5560f06cade0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7faab004eaf0 tx=0x7faab0009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6628dd95f1358065 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:59.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faaa1ffb640 1 -- 172.21.15.44:0/977186448 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab0052040 con 0x5560f06b3210 2024-04-21T23:16:59.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faaa1ffb640 1 -- 172.21.15.44:0/977186448 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faab0048040 con 0x5560f06b3210 2024-04-21T23:16:59.509 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.508+0000 7faaa1ffb640 1 -- 172.21.15.44:0/977186448 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab0039040 con 0x5560f06b3210 2024-04-21T23:16:59.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.509+0000 7faac15be740 1 -- 172.21.15.44:0/977186448 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 msgr2=0x5560f06cade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:59.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.509+0000 7faac15be740 1 --2- 172.21.15.44:0/977186448 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 0x5560f06cade0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7faab004eaf0 tx=0x7faab0009540 comp rx=0 tx=0).stop 2024-04-21T23:16:59.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.509+0000 7faac15be740 1 -- 172.21.15.44:0/977186448 shutdown_connections 2024-04-21T23:16:59.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.509+0000 7faac15be740 1 --2- 172.21.15.44:0/977186448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5560f06aa440 0x5560f06b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.509+0000 7faac15be740 1 --2- 172.21.15.44:0/977186448 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06b3210 0x5560f06cade0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.509+0000 7faac15be740 1 --2- 172.21.15.44:0/977186448 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06a9b20 0x5560f06beca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.509+0000 7faac15be740 1 -- 172.21.15.44:0/977186448 >> 172.21.15.44:0/977186448 conn(0x5560f06a4440 msgr2=0x5560f06a2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:59.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.509+0000 7faac15be740 1 -- 172.21.15.44:0/977186448 shutdown_connections 2024-04-21T23:16:59.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.509+0000 7faac15be740 1 -- 172.21.15.44:0/977186448 wait complete. 2024-04-21T23:16:59.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.509+0000 7faac15be740 1 -- start start 2024-04-21T23:16:59.513 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.510+0000 7faac15be740 1 Processor -- start 2024-04-21T23:16:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.510+0000 7faac15be740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06a9b20 0x5560f06bb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.510+0000 7faac15be740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06aa440 0x5560f06acbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.510+0000 7faac15be740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5560f06b3210 0x5560f06ad110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.510+0000 7faac15be740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7faab81ad8a0 con 0x5560f06a9b20 2024-04-21T23:16:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.510+0000 7faac15be740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7faab81aed10 con 0x5560f06aa440 2024-04-21T23:16:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.510+0000 7faac15be740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faab81adf70 con 0x5560f06b3210 2024-04-21T23:16:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.510+0000 7faac15be740 10 client.0 _ll_register_callbacks cb 0x7faaa400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:16:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.510+0000 7faab7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06a9b20 0x5560f06bb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.510+0000 7faab7fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06a9b20 0x5560f06bb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:59848/0 (socket says 172.21.15.44:59848) 2024-04-21T23:16:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.510+0000 7faab77fe640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06aa440 0x5560f06acbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.510+0000 7faab7fff640 1 -- 172.21.15.44:0/804124048 learned_addr learned my addr 172.21.15.44:0/804124048 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:16:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.510+0000 7faabce98640 1 --2- 172.21.15.44:0/804124048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5560f06b3210 0x5560f06ad110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.510+0000 7faab7fff640 1 -- 172.21.15.44:0/804124048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5560f06b3210 msgr2=0x5560f06ad110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.510+0000 7faab7fff640 1 --2- 172.21.15.44:0/804124048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5560f06b3210 0x5560f06ad110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.510+0000 7faab7fff640 1 -- 172.21.15.44:0/804124048 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06aa440 msgr2=0x5560f06acbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:16:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.510+0000 7faab7fff640 1 --2- 172.21.15.44:0/804124048 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06aa440 0x5560f06acbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.510+0000 7faab7fff640 1 -- 172.21.15.44:0/804124048 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560f06ad940 con 0x5560f06a9b20 2024-04-21T23:16:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.510+0000 7faab77fe640 1 --2- 172.21.15.44:0/804124048 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06aa440 0x5560f06acbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:16:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.511+0000 7faab7fff640 1 --2- 172.21.15.44:0/804124048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06a9b20 0x5560f06bb4c0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x5560f06c5d80 tx=0x7faaac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d2bb4348bdafef6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:16:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.511+0000 7faa977fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:16:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.511+0000 7faa977fe640 1 -- 172.21.15.44:0/804124048 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaac011d60 con 0x5560f06a9b20 2024-04-21T23:16:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.511+0000 7faa977fe640 1 -- 172.21.15.44:0/804124048 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9445+0+0 (secure 0 0 0) 0x7faaac028300 con 0x5560f06a9b20 2024-04-21T23:16:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.511+0000 7faac15be740 1 -- 172.21.15.44:0/804124048 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faab0048040 con 0x5560f06a9b20 2024-04-21T23:16:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.511+0000 7faa977fe640 1 -- 172.21.15.44:0/804124048 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7faaac039050 con 0x5560f06a9b20 2024-04-21T23:16:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.511+0000 7faac15be740 10 client.17526 fetch_fsmap learned FSMap version 31 2024-04-21T23:16:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.511+0000 7faac15be740 1 -- 172.21.15.44:0/804124048 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5560f0729630 con 0x5560f06a9b20 2024-04-21T23:16:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.511+0000 7faa977fe640 1 -- 172.21.15.44:0/804124048 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faab0048040 con 0x5560f06a9b20 2024-04-21T23:16:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.511+0000 7faa977fe640 1 -- 172.21.15.44:0/804124048 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7faaac01fbc0 con 0x5560f06a9b20 2024-04-21T23:16:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.511+0000 7faac15be740 10 client.17526 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:16:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.511+0000 7faac15be740 10 client.17526 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T23:16:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.511+0000 7faac15be740 20 client.17526 populate_metadata read hostname 'smithi044' 2024-04-21T23:16:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.511+0000 7faac15be740 1 --2- 172.21.15.44:0/804124048 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5560f0729d90 0x5560f072c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:16:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.511+0000 7faac15be740 4 client.17526 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T23:16:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.511+0000 7faac15be740 1 -- 172.21.15.44:0/804124048 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5560f0615400 con 0x5560f0729d90 2024-04-21T23:16:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.512+0000 7faab77fe640 1 --2- 172.21.15.44:0/804124048 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5560f0729d90 0x5560f072c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:16:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.512+0000 7faab77fe640 1 --2- 172.21.15.44:0/804124048 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5560f0729d90 0x5560f072c1b0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2da1ad6ceca8bcc server_cookie=d18fa14e1312f1fe in_seq=0 out_seq=0 2024-04-21T23:16:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.512+0000 7faa977fe640 10 client.17526 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:59.709 INFO:tasks.workunit.client.0.smithi044.stdout: 1 945826 11.93 MB/sec execute 289 sec latency 110.331 ms 2024-04-21T23:16:59.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:59 smithi069 ceph-mon[31816]: scrub queued for path: / 2024-04-21T23:16:59.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:59 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:16:59.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:59 smithi069 ceph-mon[31816]: scrub summary: active paths [/] 2024-04-21T23:16:59.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:16:59 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:16:59.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:59 smithi055 ceph-mon[31117]: scrub queued for path: / 2024-04-21T23:16:59.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:59 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:16:59.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:59 smithi055 ceph-mon[31117]: scrub summary: active paths [/] 2024-04-21T23:16:59.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:16:59 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:16:59.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.961+0000 7faa977fe640 1 -- 172.21.15.44:0/804124048 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5560f0615400 con 0x5560f0729d90 2024-04-21T23:16:59.961 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.961+0000 7faa977fe640 10 client.17526 handle_command_reply: tid=0 2024-04-21T23:16:59.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.961+0000 7faac15be740 1 client.17526 shutdown 2024-04-21T23:16:59.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.961+0000 7faac15be740 1 -- 172.21.15.44:0/804124048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06a9b20 msgr2=0x5560f06bb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:59.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.961+0000 7faac15be740 1 --2- 172.21.15.44:0/804124048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06a9b20 0x5560f06bb4c0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x5560f06c5d80 tx=0x7faaac004730 comp rx=0 tx=0).stop 2024-04-21T23:16:59.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.961+0000 7faac15be740 1 -- 172.21.15.44:0/804124048 shutdown_connections 2024-04-21T23:16:59.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.961+0000 7faac15be740 1 --2- 172.21.15.44:0/804124048 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5560f0729d90 0x5560f072c1b0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.961+0000 7faac15be740 1 --2- 172.21.15.44:0/804124048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5560f06b3210 0x5560f06ad110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.961+0000 7faa977fe640 0 client.17526 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:16:59.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.961+0000 7faac15be740 1 --2- 172.21.15.44:0/804124048 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5560f06aa440 0x5560f06acbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.961+0000 7faac15be740 1 --2- 172.21.15.44:0/804124048 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5560f06a9b20 0x5560f06bb4c0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.961+0000 7faac15be740 1 -- 172.21.15.44:0/804124048 >> 172.21.15.44:0/804124048 conn(0x5560f06a4440 msgr2=0x5560f06af0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:59.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.961+0000 7faac15be740 1 -- 172.21.15.44:0/804124048 shutdown_connections 2024-04-21T23:16:59.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.961+0000 7faac15be740 1 -- 172.21.15.44:0/804124048 wait complete. 2024-04-21T23:16:59.962 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.962+0000 7faac15be740 20 client.17526 trim_cache size 0 max 16384 2024-04-21T23:16:59.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:{ 2024-04-21T23:16:59.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "message": "", 2024-04-21T23:16:59.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout: "return_code": 0 2024-04-21T23:16:59.963 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stdout:} 2024-04-21T23:16:59.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.962+0000 7faabe8fb640 1 -- 172.21.15.44:0/2348470027 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faaa80a2cf0 msgr2=0x7faaa80a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:59.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.962+0000 7faabe8fb640 1 --2- 172.21.15.44:0/2348470027 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faaa80a2cf0 0x7faaa80a51b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7faaac005fd0 tx=0x7faaac005850 comp rx=0 tx=0).stop 2024-04-21T23:16:59.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.962+0000 7faabe8fb640 1 -- 172.21.15.44:0/2348470027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab81032d0 msgr2=0x7faab8198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:16:59.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.962+0000 7faabe8fb640 1 --2- 172.21.15.44:0/2348470027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab81032d0 0x7faab8198530 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7faaa400e9f0 tx=0x7faaa400eeb0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.962+0000 7faabe8fb640 1 -- 172.21.15.44:0/2348470027 shutdown_connections 2024-04-21T23:16:59.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.962+0000 7faabe8fb640 1 --2- 172.21.15.44:0/2348470027 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7faaa80a2cf0 0x7faaa80a51b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.962+0000 7faabe8fb640 1 --2- 172.21.15.44:0/2348470027 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7faab810bd80 0x7faab819c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.962+0000 7faabe8fb640 1 --2- 172.21.15.44:0/2348470027 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7faab81032d0 0x7faab8198530 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.962+0000 7faabe8fb640 1 --2- 172.21.15.44:0/2348470027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faab81029b0 0x7faab8197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:16:59.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.962+0000 7faabe8fb640 1 -- 172.21.15.44:0/2348470027 >> 172.21.15.44:0/2348470027 conn(0x7faab8077760 msgr2=0x7faab80fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:16:59.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.962+0000 7faabe8fb640 1 -- 172.21.15.44:0/2348470027 shutdown_connections 2024-04-21T23:16:59.964 INFO:teuthology.task.background_exec.ubuntu@smithi044.front.sepia.ceph.com.smithi044.stderr:2024-04-21T23:16:59.962+0000 7faabe8fb640 1 -- 172.21.15.44:0/2348470027 wait complete. 2024-04-21T23:17:00.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:17:00.577 INFO:tasks.workunit.client.0.smithi044.stdout:[947103] write failed on handle 12151 (Resource temporarily unavailable) 2024-04-21T23:17:00.577 INFO:tasks.workunit.client.0.smithi044.stdout:Child failed with status 1 2024-04-21T23:17:00.577 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-04-21T23:17:00.577 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-21T23:17:00.599 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:00 smithi044 ceph-mon[27934]: pgmap v256: 129 pgs: 129 active+clean; 95 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 345 op/s 2024-04-21T23:17:00.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:00 smithi055 ceph-mon[31117]: pgmap v256: 129 pgs: 129 active+clean; 95 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 345 op/s 2024-04-21T23:17:00.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:00 smithi069 ceph-mon[31816]: pgmap v256: 129 pgs: 129 active+clean; 95 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.1 MiB/s wr, 345 op/s 2024-04-21T23:17:00.923 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi044 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=430e09df97c8fc7dc2b2ae424f68ed11366c540f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-21T23:17:01.160 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=63779f9470784c72a2273e77559e930d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_430e09df97c8fc7dc2b2ae424f68ed11366c540f/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi044 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=430e09df97c8fc7dc2b2ae424f68ed11366c540f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-21T23:17:01.162 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-21T23:17:01.173 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi044.front.sepia.ceph.com 2024-04-21T23:17:01.185 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-21T23:17:01.233 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-21T23:17:01.234 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T23:17:01.234 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi044.front.sepia.ceph.com 2024-04-21T23:17:01.234 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-21T23:17:01.262 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-21T23:17:02.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:02 smithi044 ceph-mon[27934]: pgmap v257: 129 pgs: 129 active+clean; 95 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 207 op/s 2024-04-21T23:17:02.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:02 smithi044 ceph-mon[27934]: scrub summary: idle 2024-04-21T23:17:02.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:02 smithi055 ceph-mon[31117]: pgmap v257: 129 pgs: 129 active+clean; 95 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 207 op/s 2024-04-21T23:17:02.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:02 smithi055 ceph-mon[31117]: scrub summary: idle 2024-04-21T23:17:02.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:02 smithi069 ceph-mon[31816]: pgmap v257: 129 pgs: 129 active+clean; 95 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 207 op/s 2024-04-21T23:17:02.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:02 smithi069 ceph-mon[31816]: scrub summary: idle 2024-04-21T23:17:03.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:03 smithi044 ceph-mon[27934]: osdmap e105: 12 total, 12 up, 12 in 2024-04-21T23:17:03.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:03 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:17:03.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:03 smithi055 ceph-mon[31117]: osdmap e105: 12 total, 12 up, 12 in 2024-04-21T23:17:03.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:03 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:17:03.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:03 smithi069 ceph-mon[31816]: osdmap e105: 12 total, 12 up, 12 in 2024-04-21T23:17:03.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:03 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:17:04.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:04 smithi044 ceph-mon[27934]: pgmap v259: 129 pgs: 129 active+clean; 80 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 451 op/s 2024-04-21T23:17:04.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:04 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:17:04.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:04 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:17:04.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:04 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:17:04.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:04 smithi055 ceph-mon[31117]: pgmap v259: 129 pgs: 129 active+clean; 80 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 451 op/s 2024-04-21T23:17:04.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:04 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:17:04.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:04 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:17:04.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:04 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:17:04.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:04 smithi069 ceph-mon[31816]: pgmap v259: 129 pgs: 129 active+clean; 80 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 451 op/s 2024-04-21T23:17:04.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:04 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:17:04.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:04 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:17:04.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:04 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:17:05.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:05 smithi044 ceph-mon[27934]: pgmap v260: 129 pgs: 129 active+clean; 80 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 451 op/s 2024-04-21T23:17:05.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:05 smithi069 ceph-mon[31816]: pgmap v260: 129 pgs: 129 active+clean; 80 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 451 op/s 2024-04-21T23:17:05.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:05 smithi055 ceph-mon[31117]: pgmap v260: 129 pgs: 129 active+clean; 80 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 451 op/s 2024-04-21T23:17:08.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:07 smithi044 ceph-mon[27934]: pgmap v261: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 76 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 2.9 MiB/s wr, 423 op/s 2024-04-21T23:17:08.213 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:07 smithi055 ceph-mon[31117]: pgmap v261: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 76 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 2.9 MiB/s wr, 423 op/s 2024-04-21T23:17:08.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:07 smithi069 ceph-mon[31816]: pgmap v261: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 76 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 2.9 MiB/s wr, 423 op/s 2024-04-21T23:17:10.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:09 smithi044 ceph-mon[27934]: pgmap v262: 129 pgs: 12 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 85 active+clean; 71 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 2.0 MiB/s wr, 375 op/s 2024-04-21T23:17:10.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:09 smithi055 ceph-mon[31117]: pgmap v262: 129 pgs: 12 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 85 active+clean; 71 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 2.0 MiB/s wr, 375 op/s 2024-04-21T23:17:10.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:09 smithi069 ceph-mon[31816]: pgmap v262: 129 pgs: 12 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 85 active+clean; 71 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 2.0 MiB/s wr, 375 op/s 2024-04-21T23:17:12.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:11 smithi044 ceph-mon[27934]: pgmap v263: 129 pgs: 12 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 85 active+clean; 71 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 2.0 MiB/s wr, 375 op/s 2024-04-21T23:17:12.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:11 smithi069 ceph-mon[31816]: pgmap v263: 129 pgs: 12 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 85 active+clean; 71 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 2.0 MiB/s wr, 375 op/s 2024-04-21T23:17:12.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:11 smithi055 ceph-mon[31117]: pgmap v263: 129 pgs: 12 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 85 active+clean; 71 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 2.0 MiB/s wr, 375 op/s 2024-04-21T23:17:14.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:14 smithi055 ceph-mon[31117]: pgmap v264: 129 pgs: 129 active+clean; 70 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 1.3 MiB/s wr, 92 op/s 2024-04-21T23:17:14.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:14 smithi055 ceph-mon[31117]: osdmap e106: 12 total, 12 up, 12 in 2024-04-21T23:17:14.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:14 smithi069 ceph-mon[31816]: pgmap v264: 129 pgs: 129 active+clean; 70 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 1.3 MiB/s wr, 92 op/s 2024-04-21T23:17:14.466 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:14 smithi069 ceph-mon[31816]: osdmap e106: 12 total, 12 up, 12 in 2024-04-21T23:17:14.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:14 smithi044 ceph-mon[27934]: pgmap v264: 129 pgs: 129 active+clean; 70 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 1.3 MiB/s wr, 92 op/s 2024-04-21T23:17:14.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:14 smithi044 ceph-mon[27934]: osdmap e106: 12 total, 12 up, 12 in 2024-04-21T23:17:16.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:16 smithi055 ceph-mon[31117]: pgmap v266: 129 pgs: 129 active+clean; 70 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 1.4 MiB/s wr, 96 op/s 2024-04-21T23:17:16.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:16 smithi069 ceph-mon[31816]: pgmap v266: 129 pgs: 129 active+clean; 70 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 1.4 MiB/s wr, 96 op/s 2024-04-21T23:17:16.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:16 smithi044 ceph-mon[27934]: pgmap v266: 129 pgs: 129 active+clean; 70 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 1.4 MiB/s wr, 96 op/s 2024-04-21T23:17:18.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:18 smithi055 ceph-mon[31117]: pgmap v267: 129 pgs: 129 active+clean; 89 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 3.8 MiB/s wr, 127 op/s 2024-04-21T23:17:18.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:18 smithi069 ceph-mon[31816]: pgmap v267: 129 pgs: 129 active+clean; 89 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 3.8 MiB/s wr, 127 op/s 2024-04-21T23:17:18.684 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:18 smithi044 ceph-mon[27934]: pgmap v267: 129 pgs: 129 active+clean; 89 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 3.8 MiB/s wr, 127 op/s 2024-04-21T23:17:20.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:20 smithi069 ceph-mon[31816]: pgmap v268: 129 pgs: 129 active+clean; 108 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 6.3 MiB/s wr, 165 op/s 2024-04-21T23:17:20.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:20 smithi055 ceph-mon[31117]: pgmap v268: 129 pgs: 129 active+clean; 108 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 6.3 MiB/s wr, 165 op/s 2024-04-21T23:17:20.487 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:20 smithi044 ceph-mon[27934]: pgmap v268: 129 pgs: 129 active+clean; 108 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 6.3 MiB/s wr, 165 op/s 2024-04-21T23:17:22.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:22 smithi055 ceph-mon[31117]: pgmap v269: 129 pgs: 129 active+clean; 108 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 6.3 MiB/s wr, 165 op/s 2024-04-21T23:17:22.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:22 smithi069 ceph-mon[31816]: pgmap v269: 129 pgs: 129 active+clean; 108 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 6.3 MiB/s wr, 165 op/s 2024-04-21T23:17:22.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:22 smithi044 ceph-mon[27934]: pgmap v269: 129 pgs: 129 active+clean; 108 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 6.3 MiB/s wr, 165 op/s 2024-04-21T23:17:24.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:24 smithi069 ceph-mon[31816]: pgmap v270: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 5.5 MiB/s wr, 209 op/s 2024-04-21T23:17:24.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:24 smithi055 ceph-mon[31117]: pgmap v270: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 5.5 MiB/s wr, 209 op/s 2024-04-21T23:17:24.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:24 smithi044 ceph-mon[27934]: pgmap v270: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 5.5 MiB/s wr, 209 op/s 2024-04-21T23:17:26.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:26 smithi069 ceph-mon[31816]: pgmap v271: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 4.8 MiB/s wr, 181 op/s 2024-04-21T23:17:26.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:26 smithi055 ceph-mon[31117]: pgmap v271: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 4.8 MiB/s wr, 181 op/s 2024-04-21T23:17:26.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:26 smithi044 ceph-mon[27934]: pgmap v271: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 4.8 MiB/s wr, 181 op/s 2024-04-21T23:17:28.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:28 smithi069 ceph-mon[31816]: pgmap v272: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 4.6 MiB/s wr, 175 op/s 2024-04-21T23:17:28.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:28 smithi055 ceph-mon[31117]: pgmap v272: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 4.6 MiB/s wr, 175 op/s 2024-04-21T23:17:28.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:28 smithi044 ceph-mon[27934]: pgmap v272: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 4.6 MiB/s wr, 175 op/s 2024-04-21T23:17:28.997 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:17:28.997 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:17:29.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.115+0000 7f3bdd0cd640 1 -- 172.21.15.44:0/601222094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bd8103bb0 msgr2=0x7f3bd810e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.115+0000 7f3bdd0cd640 1 --2- 172.21.15.44:0/601222094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bd8103bb0 0x7f3bd810e6e0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0009ca0 tx=0x7f3bd002c1d0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.115 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.115+0000 7f3bdd0cd640 1 -- 172.21.15.44:0/601222094 shutdown_connections 2024-04-21T23:17:29.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.115+0000 7f3bdd0cd640 1 --2- 172.21.15.44:0/601222094 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bd8103bb0 0x7f3bd810e6e0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.115+0000 7f3bdd0cd640 1 --2- 172.21.15.44:0/601222094 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bd8103210 0x7f3bd8103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.115+0000 7f3bdd0cd640 1 --2- 172.21.15.44:0/601222094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bd8101ad0 0x7f3bd8101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.115+0000 7f3bdd0cd640 1 -- 172.21.15.44:0/601222094 >> 172.21.15.44:0/601222094 conn(0x7f3bd80fd200 msgr2=0x7f3bd80ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:29.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.115+0000 7f3bdd0cd640 1 -- 172.21.15.44:0/601222094 shutdown_connections 2024-04-21T23:17:29.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.115+0000 7f3bdd0cd640 1 -- 172.21.15.44:0/601222094 wait complete. 2024-04-21T23:17:29.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.116+0000 7f3bdd0cd640 1 Processor -- start 2024-04-21T23:17:29.116 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.116+0000 7f3bdd0cd640 1 -- start start 2024-04-21T23:17:29.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.116+0000 7f3bdd0cd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bd8101ad0 0x7f3bd8198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.116+0000 7f3bdd0cd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bd8103210 0x7f3bd81985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.116+0000 7f3bdd0cd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bd8103bb0 0x7f3bd819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.116+0000 7f3bdd0cd640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd8111c20 con 0x7f3bd8101ad0 2024-04-21T23:17:29.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.116+0000 7f3bdd0cd640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd8111aa0 con 0x7f3bd8103210 2024-04-21T23:17:29.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.116+0000 7f3bdd0cd640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd8111da0 con 0x7f3bd8103bb0 2024-04-21T23:17:29.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.116+0000 7f3bd6d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bd8101ad0 0x7f3bd8198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.116+0000 7f3bd6d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bd8101ad0 0x7f3bd8198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49112/0 (socket says 172.21.15.44:49112) 2024-04-21T23:17:29.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.116+0000 7f3bd6d9d640 1 -- 172.21.15.44:0/918485915 learned_addr learned my addr 172.21.15.44:0/918485915 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:29.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.116+0000 7f3bd659c640 1 --2- 172.21.15.44:0/918485915 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bd8103210 0x7f3bd81985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.116+0000 7f3bd759e640 1 --2- 172.21.15.44:0/918485915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bd8103bb0 0x7f3bd819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.117+0000 7f3bd6d9d640 1 -- 172.21.15.44:0/918485915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bd8103bb0 msgr2=0x7f3bd819c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.117+0000 7f3bd6d9d640 1 --2- 172.21.15.44:0/918485915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bd8103bb0 0x7f3bd819c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.117+0000 7f3bd6d9d640 1 -- 172.21.15.44:0/918485915 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bd8103210 msgr2=0x7f3bd81985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.117+0000 7f3bd6d9d640 1 --2- 172.21.15.44:0/918485915 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bd8103210 0x7f3bd81985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.117+0000 7f3bd6d9d640 1 -- 172.21.15.44:0/918485915 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd819d000 con 0x7f3bd8101ad0 2024-04-21T23:17:29.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.117+0000 7f3bd6d9d640 1 --2- 172.21.15.44:0/918485915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bd8101ad0 0x7f3bd8198060 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800d770 tx=0x7f3bc800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:29.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.117+0000 7f3bbffff640 1 -- 172.21.15.44:0/918485915 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc8018070 con 0x7f3bd8101ad0 2024-04-21T23:17:29.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.117+0000 7f3bdd0cd640 1 -- 172.21.15.44:0/918485915 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bd819d2f0 con 0x7f3bd8101ad0 2024-04-21T23:17:29.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.117+0000 7f3bbffff640 1 -- 172.21.15.44:0/918485915 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bc8005500 con 0x7f3bd8101ad0 2024-04-21T23:17:29.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.117+0000 7f3bbffff640 1 -- 172.21.15.44:0/918485915 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc8004e00 con 0x7f3bd8101ad0 2024-04-21T23:17:29.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.117+0000 7f3bdd0cd640 1 -- 172.21.15.44:0/918485915 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bd8073970 con 0x7f3bd8101ad0 2024-04-21T23:17:29.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.119+0000 7f3bbffff640 1 -- 172.21.15.44:0/918485915 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3bc8004fa0 con 0x7f3bd8101ad0 2024-04-21T23:17:29.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.120+0000 7f3bdfd9a740 1 Processor -- start 2024-04-21T23:17:29.128 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.120+0000 7f3bdfd9a740 1 -- start start 2024-04-21T23:17:29.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.120+0000 7f3bdfd9a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a1f30 0x55f8560a2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.120+0000 7f3bdfd9a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8560a2850 0x55f8560aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.120+0000 7f3bdfd9a740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560ab430 0x55f8560ad820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.120+0000 7f3bdfd9a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f856006560 con 0x55f8560a1f30 2024-04-21T23:17:29.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.120+0000 7f3bdfd9a740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55f855fae730 con 0x55f8560ab430 2024-04-21T23:17:29.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.120+0000 7f3bdfd9a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f856002520 con 0x55f8560a2850 2024-04-21T23:17:29.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.120+0000 7f3bbffff640 1 --2- 172.21.15.44:0/918485915 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3bc00a2d00 0x7f3bc00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.120+0000 7f3bd759e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a1f30 0x55f8560a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.120+0000 7f3bd759e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a1f30 0x55f8560a2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49118/0 (socket says 172.21.15.44:49118) 2024-04-21T23:17:29.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.120+0000 7f3bd759e640 1 -- 172.21.15.44:0/617874592 learned_addr learned my addr 172.21.15.44:0/617874592 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:29.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.120+0000 7f3bd759e640 1 --2- 172.21.15.44:0/918485915 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3bc00a2d00 0x7f3bc00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.120+0000 7f3bbffff640 1 -- 172.21.15.44:0/918485915 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7f3bc80d3460 con 0x7f3bd8101ad0 2024-04-21T23:17:29.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.120+0000 7f3bd659c640 1 --2- 172.21.15.44:0/617874592 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560ab430 0x55f8560ad820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.120+0000 7f3bd6d9d640 1 --2- 172.21.15.44:0/617874592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8560a2850 0x55f8560aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.121+0000 7f3bd759e640 1 -- 172.21.15.44:0/617874592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8560a2850 msgr2=0x55f8560aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:29.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.121+0000 7f3bd759e640 1 --2- 172.21.15.44:0/617874592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8560a2850 0x55f8560aaef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.121+0000 7f3bd759e640 1 -- 172.21.15.44:0/617874592 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560ab430 msgr2=0x55f8560ad820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:29.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.121+0000 7f3bd759e640 1 --2- 172.21.15.44:0/617874592 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560ab430 0x55f8560ad820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.121+0000 7f3bd759e640 1 -- 172.21.15.44:0/617874592 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f855de47d0 con 0x55f8560a1f30 2024-04-21T23:17:29.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.121+0000 7f3bd759e640 1 --2- 172.21.15.44:0/617874592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a1f30 0x55f8560a2310 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0009430 tx=0x7f3bd0003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9210ce575274cd2a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:29.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.121+0000 7f3bd759e640 1 --2- 172.21.15.44:0/918485915 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3bc00a2d00 0x7f3bc00a51c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0009ca0 tx=0x7f3bd0047000 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:29.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.121+0000 7f3bbd7fa640 1 -- 172.21.15.44:0/617874592 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0039040 con 0x55f8560a1f30 2024-04-21T23:17:29.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.121+0000 7f3bbd7fa640 1 -- 172.21.15.44:0/617874592 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bd0051070 con 0x55f8560a1f30 2024-04-21T23:17:29.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.121+0000 7f3bbd7fa640 1 -- 172.21.15.44:0/617874592 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0032860 con 0x55f8560a1f30 2024-04-21T23:17:29.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.122+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/617874592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a1f30 msgr2=0x55f8560a2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.122+0000 7f3bdfd9a740 1 --2- 172.21.15.44:0/617874592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a1f30 0x55f8560a2310 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0009430 tx=0x7f3bd0003f50 comp rx=0 tx=0).stop 2024-04-21T23:17:29.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.122+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/617874592 shutdown_connections 2024-04-21T23:17:29.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.122+0000 7f3bdfd9a740 1 --2- 172.21.15.44:0/617874592 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8560a2850 0x55f8560aaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.122+0000 7f3bdfd9a740 1 --2- 172.21.15.44:0/617874592 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560ab430 0x55f8560ad820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.122+0000 7f3bdfd9a740 1 --2- 172.21.15.44:0/617874592 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a1f30 0x55f8560a2310 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.122+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/617874592 >> 172.21.15.44:0/617874592 conn(0x55f85609c660 msgr2=0x55f85609ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:29.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.122+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/617874592 shutdown_connections 2024-04-21T23:17:29.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.122+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/617874592 wait complete. 2024-04-21T23:17:29.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.122+0000 7f3bdfd9a740 1 -- start start 2024-04-21T23:17:29.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.122+0000 7f3bdfd9a740 1 Processor -- start 2024-04-21T23:17:29.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bdfd9a740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560a1f30 0x55f8560a8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bdfd9a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8560a2850 0x55f8560a8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bdfd9a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560ab430 0x55f8560a9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bdfd9a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f8560ae1c0 con 0x55f8560ab430 2024-04-21T23:17:29.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bdfd9a740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55f855fc1e00 con 0x55f8560a1f30 2024-04-21T23:17:29.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bdfd9a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f8560ae340 con 0x55f8560a2850 2024-04-21T23:17:29.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bdfd9a740 10 client.0 _ll_register_callbacks cb 0x55f855ffceb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:17:29.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bd759e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560ab430 0x55f8560a9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bd759e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560ab430 0x55f8560a9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49124/0 (socket says 172.21.15.44:49124) 2024-04-21T23:17:29.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bd759e640 1 -- 172.21.15.44:0/829482943 learned_addr learned my addr 172.21.15.44:0/829482943 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:29.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bd6d9d640 1 --2- 172.21.15.44:0/829482943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560a1f30 0x55f8560a8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bd659c640 1 --2- 172.21.15.44:0/829482943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8560a2850 0x55f8560a8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bd759e640 1 -- 172.21.15.44:0/829482943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8560a2850 msgr2=0x55f8560a8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:29.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bd759e640 1 --2- 172.21.15.44:0/829482943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8560a2850 0x55f8560a8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bd759e640 1 -- 172.21.15.44:0/829482943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560a1f30 msgr2=0x55f8560a8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:29.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bd759e640 1 --2- 172.21.15.44:0/829482943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560a1f30 0x55f8560a8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bd759e640 1 -- 172.21.15.44:0/829482943 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f855d1a3f0 con 0x55f8560ab430 2024-04-21T23:17:29.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bd759e640 1 --2- 172.21.15.44:0/829482943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560ab430 0x55f8560a9010 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0006010 tx=0x7f3bd00046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6f834a7a6b49d8f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:29.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bbaff5640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:17:29.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bbaff5640 1 -- 172.21.15.44:0/829482943 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0039050 con 0x55f8560ab430 2024-04-21T23:17:29.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bbaff5640 1 -- 172.21.15.44:0/829482943 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7f3bd0032800 con 0x55f8560ab430 2024-04-21T23:17:29.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.123+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/829482943 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bd0051070 con 0x55f8560ab430 2024-04-21T23:17:29.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.124+0000 7f3bbaff5640 1 -- 172.21.15.44:0/829482943 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3bd0063050 con 0x55f8560ab430 2024-04-21T23:17:29.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.124+0000 7f3bdfd9a740 10 client.17538 fetch_fsmap learned FSMap version 31 2024-04-21T23:17:29.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.124+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/829482943 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f855cda0f0 con 0x55f8560ab430 2024-04-21T23:17:29.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.124+0000 7f3bbaff5640 1 -- 172.21.15.44:0/829482943 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bd0051070 con 0x55f8560ab430 2024-04-21T23:17:29.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.124+0000 7f3bbaff5640 1 -- 172.21.15.44:0/829482943 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f3bd004c940 con 0x55f8560ab430 2024-04-21T23:17:29.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.124+0000 7f3bdfd9a740 10 client.17538 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:17:29.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.124+0000 7f3bdfd9a740 10 client.17538 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:17:29.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.124+0000 7f3bdfd9a740 20 client.17538 populate_metadata read hostname 'smithi044' 2024-04-21T23:17:29.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.124+0000 7f3bdfd9a740 1 --2- 172.21.15.44:0/829482943 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f8560bb710 0x55f8560bdb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.124+0000 7f3bdfd9a740 4 client.17538 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:17:29.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.124+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/829482943 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f856013aa0 con 0x55f8560bb710 2024-04-21T23:17:29.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.125+0000 7f3bd6d9d640 1 --2- 172.21.15.44:0/829482943 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f8560bb710 0x55f8560bdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.125+0000 7f3bd6d9d640 1 --2- 172.21.15.44:0/829482943 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f8560bb710 0x55f8560bdb00 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f30c7356a0a265f2 server_cookie=b3caf44153aae50c in_seq=0 out_seq=0 2024-04-21T23:17:29.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.125+0000 7f3bbaff5640 10 client.17538 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:17:29.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.126+0000 7f3bbaff5640 1 -- 172.21.15.44:0/829482943 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f856013aa0 con 0x55f8560bb710 2024-04-21T23:17:29.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.126+0000 7f3bbaff5640 10 client.17538 handle_command_reply: tid=0 2024-04-21T23:17:29.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.126+0000 7f3bdfd9a740 1 client.17538 shutdown 2024-04-21T23:17:29.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.126+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/829482943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560ab430 msgr2=0x55f8560a9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.126+0000 7f3bdfd9a740 1 --2- 172.21.15.44:0/829482943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560ab430 0x55f8560a9010 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0006010 tx=0x7f3bd00046d0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.126+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/829482943 shutdown_connections 2024-04-21T23:17:29.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.126+0000 7f3bdfd9a740 1 --2- 172.21.15.44:0/829482943 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f8560bb710 0x55f8560bdb00 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.126+0000 7f3bdfd9a740 1 --2- 172.21.15.44:0/829482943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8560a2850 0x55f8560a8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.126+0000 7f3bdfd9a740 1 --2- 172.21.15.44:0/829482943 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560ab430 0x55f8560a9010 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.126+0000 7f3bdfd9a740 1 --2- 172.21.15.44:0/829482943 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560a1f30 0x55f8560a8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.126+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/829482943 >> 172.21.15.44:0/829482943 conn(0x55f85609c660 msgr2=0x55f85609e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:29.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.126+0000 7f3bbaff5640 0 client.17538 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:17:29.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.126+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/829482943 shutdown_connections 2024-04-21T23:17:29.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.126+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/829482943 wait complete. 2024-04-21T23:17:29.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.126+0000 7f3bdfd9a740 20 client.17538 trim_cache size 0 max 16384 2024-04-21T23:17:29.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.143+0000 7f3bdfd9a740 1 Processor -- start 2024-04-21T23:17:29.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.144+0000 7f3bdfd9a740 1 -- start start 2024-04-21T23:17:29.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.144+0000 7f3bdfd9a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8560a1f30 0x55f8560b7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.144+0000 7f3bdfd9a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a2850 0x55f8560a74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.144+0000 7f3bdfd9a740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560ab430 0x55f8560a8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.144+0000 7f3bdfd9a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd81aa9e0 con 0x55f8560a2850 2024-04-21T23:17:29.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.144+0000 7f3bdfd9a740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd81ad240 con 0x55f8560ab430 2024-04-21T23:17:29.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.144+0000 7f3bdfd9a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd81ab030 con 0x55f8560a1f30 2024-04-21T23:17:29.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.144+0000 7f3bd659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a2850 0x55f8560a74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.144+0000 7f3bd659c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a2850 0x55f8560a74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49132/0 (socket says 172.21.15.44:49132) 2024-04-21T23:17:29.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.144+0000 7f3bd659c640 1 -- 172.21.15.44:0/1765929226 learned_addr learned my addr 172.21.15.44:0/1765929226 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:29.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.144+0000 7f3bd6d9d640 1 --2- 172.21.15.44:0/1765929226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8560a1f30 0x55f8560b7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.144+0000 7f3bd659c640 1 -- 172.21.15.44:0/1765929226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8560a1f30 msgr2=0x55f8560b7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:29.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.144+0000 7f3bd659c640 1 --2- 172.21.15.44:0/1765929226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8560a1f30 0x55f8560b7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.144+0000 7f3bd759e640 1 --2- 172.21.15.44:0/1765929226 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560ab430 0x55f8560a8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.144+0000 7f3bd659c640 1 -- 172.21.15.44:0/1765929226 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560ab430 msgr2=0x55f8560a8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:29.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.144+0000 7f3bd659c640 1 --2- 172.21.15.44:0/1765929226 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560ab430 0x55f8560a8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.144+0000 7f3bd659c640 1 -- 172.21.15.44:0/1765929226 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8560a6b00 con 0x55f8560a2850 2024-04-21T23:17:29.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.144+0000 7f3bd759e640 1 --2- 172.21.15.44:0/1765929226 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560ab430 0x55f8560a8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:29.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.144+0000 7f3bd659c640 1 --2- 172.21.15.44:0/1765929226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a2850 0x55f8560a74f0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc004540 tx=0x7f3bcc00bbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8904cacb8f2b196a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:29.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.144+0000 7f3bbcff9640 1 -- 172.21.15.44:0/1765929226 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc020070 con 0x55f8560a2850 2024-04-21T23:17:29.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.145+0000 7f3bbcff9640 1 -- 172.21.15.44:0/1765929226 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bcc00be40 con 0x55f8560a2850 2024-04-21T23:17:29.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.145+0000 7f3bbcff9640 1 -- 172.21.15.44:0/1765929226 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc01b650 con 0x55f8560a2850 2024-04-21T23:17:29.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.145+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/1765929226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a2850 msgr2=0x55f8560a74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.145+0000 7f3bdfd9a740 1 --2- 172.21.15.44:0/1765929226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a2850 0x55f8560a74f0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc004540 tx=0x7f3bcc00bbf0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.145+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/1765929226 shutdown_connections 2024-04-21T23:17:29.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.145+0000 7f3bdfd9a740 1 --2- 172.21.15.44:0/1765929226 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560ab430 0x55f8560a8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.145+0000 7f3bdfd9a740 1 --2- 172.21.15.44:0/1765929226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a2850 0x55f8560a74f0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.145+0000 7f3bdfd9a740 1 --2- 172.21.15.44:0/1765929226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8560a1f30 0x55f8560b7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.145+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/1765929226 >> 172.21.15.44:0/1765929226 conn(0x55f85609c660 msgr2=0x55f85609b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:29.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.145+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/1765929226 shutdown_connections 2024-04-21T23:17:29.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.145+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/1765929226 wait complete. 2024-04-21T23:17:29.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.145+0000 7f3bdfd9a740 1 -- start start 2024-04-21T23:17:29.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.145+0000 7f3bdfd9a740 1 Processor -- start 2024-04-21T23:17:29.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bdfd9a740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a1f30 0x55f8560b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bdfd9a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8560a2850 0x55f8560b3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bdfd9a740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560ab430 0x55f8560b0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bdfd9a740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd81ab700 con 0x55f8560a1f30 2024-04-21T23:17:29.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bdfd9a740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd81acb70 con 0x55f8560ab430 2024-04-21T23:17:29.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bdfd9a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd81abdd0 con 0x55f8560a2850 2024-04-21T23:17:29.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bdfd9a740 10 client.0 _ll_register_callbacks cb 0x7f3bcc00b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:17:29.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bd6d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a1f30 0x55f8560b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bd6d9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a1f30 0x55f8560b3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49134/0 (socket says 172.21.15.44:49134) 2024-04-21T23:17:29.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bd6d9d640 1 -- 172.21.15.44:0/1594912297 learned_addr learned my addr 172.21.15.44:0/1594912297 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:29.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bd659c640 1 --2- 172.21.15.44:0/1594912297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8560a2850 0x55f8560b3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bd759e640 1 --2- 172.21.15.44:0/1594912297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560ab430 0x55f8560b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bd6d9d640 1 -- 172.21.15.44:0/1594912297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8560a2850 msgr2=0x55f8560b3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:29.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bd6d9d640 1 --2- 172.21.15.44:0/1594912297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8560a2850 0x55f8560b3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bd6d9d640 1 -- 172.21.15.44:0/1594912297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560ab430 msgr2=0x55f8560b0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:29.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bd6d9d640 1 --2- 172.21.15.44:0/1594912297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560ab430 0x55f8560b0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bd6d9d640 1 -- 172.21.15.44:0/1594912297 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8560b0d70 con 0x55f8560a1f30 2024-04-21T23:17:29.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bd6d9d640 1 --2- 172.21.15.44:0/1594912297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a1f30 0x55f8560b3590 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f3bc809f630 tx=0x7f3bc806a220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7556351b881a2d0d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:29.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bbaff5640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:17:29.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bbaff5640 1 -- 172.21.15.44:0/1594912297 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc809de50 con 0x55f8560a1f30 2024-04-21T23:17:29.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bbaff5640 1 -- 172.21.15.44:0/1594912297 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7f3bc8009cc0 con 0x55f8560a1f30 2024-04-21T23:17:29.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.146+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/1594912297 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f85600ddf0 con 0x55f8560a1f30 2024-04-21T23:17:29.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.147+0000 7f3bbaff5640 1 -- 172.21.15.44:0/1594912297 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3bc809dc60 con 0x55f8560a1f30 2024-04-21T23:17:29.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.147+0000 7f3bdfd9a740 10 client.17544 fetch_fsmap learned FSMap version 31 2024-04-21T23:17:29.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.147+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/1594912297 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f8561223e0 con 0x55f8560a1f30 2024-04-21T23:17:29.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.147+0000 7f3bbaff5640 1 -- 172.21.15.44:0/1594912297 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f85600ddf0 con 0x55f8560a1f30 2024-04-21T23:17:29.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.147+0000 7f3bbaff5640 1 -- 172.21.15.44:0/1594912297 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f3bc80d0a40 con 0x55f8560a1f30 2024-04-21T23:17:29.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.147+0000 7f3bdfd9a740 10 client.17544 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:17:29.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.147+0000 7f3bdfd9a740 10 client.17544 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:17:29.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.147+0000 7f3bdfd9a740 20 client.17544 populate_metadata read hostname 'smithi044' 2024-04-21T23:17:29.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.147+0000 7f3bdfd9a740 1 --2- 172.21.15.44:0/1594912297 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f856122b00 0x55f856124ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.147+0000 7f3bdfd9a740 4 client.17544 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:17:29.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.147+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/1594912297 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f855f8b5a0 con 0x55f856122b00 2024-04-21T23:17:29.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.147+0000 7f3bd659c640 1 --2- 172.21.15.44:0/1594912297 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f856122b00 0x55f856124ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.148+0000 7f3bbaff5640 10 client.17544 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:17:29.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.148+0000 7f3bd659c640 1 --2- 172.21.15.44:0/1594912297 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f856122b00 0x55f856124ef0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=634d1a7e28d9d086 server_cookie=b8b0770baff0581b in_seq=0 out_seq=0 2024-04-21T23:17:29.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.148+0000 7f3bbaff5640 1 -- 172.21.15.44:0/1594912297 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f855f8b5a0 con 0x55f856122b00 2024-04-21T23:17:29.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.148+0000 7f3bbaff5640 10 client.17544 handle_command_reply: tid=0 2024-04-21T23:17:29.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.148+0000 7f3bdfd9a740 1 client.17544 shutdown 2024-04-21T23:17:29.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.148+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/1594912297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a1f30 msgr2=0x55f8560b3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.148+0000 7f3bdfd9a740 1 --2- 172.21.15.44:0/1594912297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a1f30 0x55f8560b3590 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f3bc809f630 tx=0x7f3bc806a220 comp rx=0 tx=0).stop 2024-04-21T23:17:29.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.148+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/1594912297 shutdown_connections 2024-04-21T23:17:29.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.148+0000 7f3bdfd9a740 1 --2- 172.21.15.44:0/1594912297 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f856122b00 0x55f856124ef0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.148+0000 7f3bdfd9a740 1 --2- 172.21.15.44:0/1594912297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8560a2850 0x55f8560b3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.148+0000 7f3bbaff5640 0 client.17544 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:17:29.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.148+0000 7f3bdfd9a740 1 --2- 172.21.15.44:0/1594912297 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f8560ab430 0x55f8560b0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.148+0000 7f3bdfd9a740 1 --2- 172.21.15.44:0/1594912297 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f8560a1f30 0x55f8560b3590 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.148+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/1594912297 >> 172.21.15.44:0/1594912297 conn(0x55f85609c660 msgr2=0x55f85609d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:29.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.148+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/1594912297 shutdown_connections 2024-04-21T23:17:29.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.148+0000 7f3bdfd9a740 1 -- 172.21.15.44:0/1594912297 wait complete. 2024-04-21T23:17:29.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.148+0000 7f3bdfd9a740 20 client.17544 trim_cache size 0 max 16384 2024-04-21T23:17:29.160 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:17:29.160 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-21T23:17:29.160 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-21T23:17:29.161 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:17:29.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.149+0000 7f3bdd0cd640 1 -- 172.21.15.44:0/918485915 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3bc00a2d00 msgr2=0x7f3bc00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.149+0000 7f3bdd0cd640 1 --2- 172.21.15.44:0/918485915 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3bc00a2d00 0x7f3bc00a51c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0009ca0 tx=0x7f3bd0047000 comp rx=0 tx=0).stop 2024-04-21T23:17:29.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.149+0000 7f3bdd0cd640 1 -- 172.21.15.44:0/918485915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bd8101ad0 msgr2=0x7f3bd8198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.149+0000 7f3bdd0cd640 1 --2- 172.21.15.44:0/918485915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bd8101ad0 0x7f3bd8198060 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800d770 tx=0x7f3bc800dc30 comp rx=0 tx=0).stop 2024-04-21T23:17:29.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.149+0000 7f3bdd0cd640 1 -- 172.21.15.44:0/918485915 shutdown_connections 2024-04-21T23:17:29.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.149+0000 7f3bdd0cd640 1 --2- 172.21.15.44:0/918485915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bd8103bb0 0x7f3bd819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.149+0000 7f3bdd0cd640 1 --2- 172.21.15.44:0/918485915 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3bc00a2d00 0x7f3bc00a51c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.149+0000 7f3bdd0cd640 1 --2- 172.21.15.44:0/918485915 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3bd8103210 0x7f3bd81985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.149+0000 7f3bdd0cd640 1 --2- 172.21.15.44:0/918485915 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3bd8101ad0 0x7f3bd8198060 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.149+0000 7f3bdd0cd640 1 -- 172.21.15.44:0/918485915 >> 172.21.15.44:0/918485915 conn(0x7f3bd80fd200 msgr2=0x7f3bd8107580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:29.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.149+0000 7f3bdd0cd640 1 -- 172.21.15.44:0/918485915 shutdown_connections 2024-04-21T23:17:29.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.149+0000 7f3bdd0cd640 1 -- 172.21.15.44:0/918485915 wait complete. 2024-04-21T23:17:29.164 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:17:29.164 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:17:29.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.268+0000 7f9b0680e640 1 -- 172.21.15.44:0/3469290941 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b00101230 msgr2=0x7f9b0010d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.268+0000 7f9b0680e640 1 --2- 172.21.15.44:0/3469290941 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b00101230 0x7f9b0010d460 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f9af8009ca0 tx=0x7f9af802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.268+0000 7f9b0680e640 1 -- 172.21.15.44:0/3469290941 shutdown_connections 2024-04-21T23:17:29.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.268+0000 7f9b0680e640 1 --2- 172.21.15.44:0/3469290941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b00100890 0x7f9b00100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.268+0000 7f9b0680e640 1 --2- 172.21.15.44:0/3469290941 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b00101230 0x7f9b0010d460 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.268+0000 7f9b0680e640 1 --2- 172.21.15.44:0/3469290941 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b001066c0 0x7f9b00106aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.268+0000 7f9b0680e640 1 -- 172.21.15.44:0/3469290941 >> 172.21.15.44:0/3469290941 conn(0x7f9b00077430 msgr2=0x7f9b00077840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:29.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.269+0000 7f9b0680e640 1 -- 172.21.15.44:0/3469290941 shutdown_connections 2024-04-21T23:17:29.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.269+0000 7f9b0680e640 1 -- 172.21.15.44:0/3469290941 wait complete. 2024-04-21T23:17:29.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.269+0000 7f9b0680e640 1 Processor -- start 2024-04-21T23:17:29.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.269+0000 7f9b0680e640 1 -- start start 2024-04-21T23:17:29.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9b0680e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b00100890 0x7f9b00197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9b0680e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b00101230 0x7f9b00198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9b0680e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b001066c0 0x7f9b0019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9b0680e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9b00110a40 con 0x7f9b001066c0 2024-04-21T23:17:29.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9b0680e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9b001108c0 con 0x7f9b00101230 2024-04-21T23:17:29.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9b0680e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9b00110bc0 con 0x7f9b00100890 2024-04-21T23:17:29.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9b04dab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b001066c0 0x7f9b0019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9b04dab640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b001066c0 0x7f9b0019c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49148/0 (socket says 172.21.15.44:49148) 2024-04-21T23:17:29.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9b04dab640 1 -- 172.21.15.44:0/2615146396 learned_addr learned my addr 172.21.15.44:0/2615146396 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:29.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9b04dab640 1 -- 172.21.15.44:0/2615146396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b00100890 msgr2=0x7f9b00197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9b04dab640 1 --2- 172.21.15.44:0/2615146396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b00100890 0x7f9b00197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9b04dab640 1 -- 172.21.15.44:0/2615146396 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b00101230 msgr2=0x7f9b00198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9aff7fe640 1 --2- 172.21.15.44:0/2615146396 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b00101230 0x7f9b00198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9b04dab640 1 --2- 172.21.15.44:0/2615146396 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b00101230 0x7f9b00198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9b04dab640 1 -- 172.21.15.44:0/2615146396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b0019cf90 con 0x7f9b001066c0 2024-04-21T23:17:29.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9b04dab640 1 --2- 172.21.15.44:0/2615146396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b001066c0 0x7f9b0019c860 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f9af802c700 tx=0x7f9af8032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:29.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9afd7fa640 1 -- 172.21.15.44:0/2615146396 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af8041020 con 0x7f9b001066c0 2024-04-21T23:17:29.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9afd7fa640 1 -- 172.21.15.44:0/2615146396 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9af8034cd0 con 0x7f9b001066c0 2024-04-21T23:17:29.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9b0680e640 1 -- 172.21.15.44:0/2615146396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b0019d220 con 0x7f9b001066c0 2024-04-21T23:17:29.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9afd7fa640 1 -- 172.21.15.44:0/2615146396 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af8039040 con 0x7f9b001066c0 2024-04-21T23:17:29.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.270+0000 7f9b0680e640 1 -- 172.21.15.44:0/2615146396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b001a4b10 con 0x7f9b001066c0 2024-04-21T23:17:29.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.271+0000 7f9b0680e640 1 -- 172.21.15.44:0/2615146396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b000685f0 con 0x7f9b001066c0 2024-04-21T23:17:29.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.272+0000 7f9afd7fa640 1 -- 172.21.15.44:0/2615146396 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f9af8044020 con 0x7f9b001066c0 2024-04-21T23:17:29.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.273+0000 7f9afd7fa640 1 --2- 172.21.15.44:0/2615146396 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9ae80a2d00 0x7f9ae80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.273+0000 7f9afffff640 1 --2- 172.21.15.44:0/2615146396 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9ae80a2d00 0x7f9ae80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.273+0000 7f9afd7fa640 1 -- 172.21.15.44:0/2615146396 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7f9af80f16b0 con 0x7f9b001066c0 2024-04-21T23:17:29.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.273+0000 7f9afffff640 1 --2- 172.21.15.44:0/2615146396 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9ae80a2d00 0x7f9ae80a51c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9af0009ad0 tx=0x7f9af0009590 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:29.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.277+0000 7f9afd7fa640 1 -- 172.21.15.44:0/2615146396 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f9af804b020 con 0x7f9b001066c0 2024-04-21T23:17:29.500 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.501+0000 7f9b0680e640 1 -- 172.21.15.44:0/2615146396 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b001a4ce0 con 0x7f9b001066c0 2024-04-21T23:17:29.501 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.502+0000 7f9afd7fa640 1 -- 172.21.15.44:0/2615146396 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11791 (secure 0 0 0) 0x7f9af80bdf90 con 0x7f9b001066c0 2024-04-21T23:17:29.502 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:17:29.502 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:16:20.187271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26863":{"gid":26863,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3179646768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3179646768},{"type":"v1","addr":"172.21.15.69:6839","nonce":3179646768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:17:29.502 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 31 2024-04-21T23:17:29.503 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.504+0000 7f9b0680e640 1 -- 172.21.15.44:0/2615146396 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9ae80a2d00 msgr2=0x7f9ae80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.504+0000 7f9b0680e640 1 --2- 172.21.15.44:0/2615146396 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9ae80a2d00 0x7f9ae80a51c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9af0009ad0 tx=0x7f9af0009590 comp rx=0 tx=0).stop 2024-04-21T23:17:29.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.504+0000 7f9b0680e640 1 -- 172.21.15.44:0/2615146396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b001066c0 msgr2=0x7f9b0019c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.504+0000 7f9b0680e640 1 --2- 172.21.15.44:0/2615146396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b001066c0 0x7f9b0019c860 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f9af802c700 tx=0x7f9af8032f20 comp rx=0 tx=0).stop 2024-04-21T23:17:29.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.504+0000 7f9b0680e640 1 -- 172.21.15.44:0/2615146396 shutdown_connections 2024-04-21T23:17:29.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.504+0000 7f9b0680e640 1 --2- 172.21.15.44:0/2615146396 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9ae80a2d00 0x7f9ae80a51c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.504+0000 7f9b0680e640 1 --2- 172.21.15.44:0/2615146396 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9b001066c0 0x7f9b0019c860 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.504+0000 7f9b0680e640 1 --2- 172.21.15.44:0/2615146396 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9b00101230 0x7f9b00198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.504+0000 7f9b0680e640 1 --2- 172.21.15.44:0/2615146396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b00100890 0x7f9b00197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.504 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.504+0000 7f9b0680e640 1 -- 172.21.15.44:0/2615146396 >> 172.21.15.44:0/2615146396 conn(0x7f9b00077430 msgr2=0x7f9b000ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:29.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.504+0000 7f9b0680e640 1 -- 172.21.15.44:0/2615146396 shutdown_connections 2024-04-21T23:17:29.505 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.504+0000 7f9b0680e640 1 -- 172.21.15.44:0/2615146396 wait complete. 2024-04-21T23:17:29.514 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:17:29.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.618+0000 7f3f0f8fb640 1 -- 172.21.15.44:0/141361503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f08102ac0 msgr2=0x7f3f0810d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.618+0000 7f3f0f8fb640 1 --2- 172.21.15.44:0/141361503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f08102ac0 0x7f3f0810d5f0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f3f00009ca0 tx=0x7f3f0002c1d0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.618 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.619+0000 7f3f0f8fb640 1 -- 172.21.15.44:0/141361503 shutdown_connections 2024-04-21T23:17:29.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.619+0000 7f3f0f8fb640 1 --2- 172.21.15.44:0/141361503 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f08102ac0 0x7f3f0810d5f0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.619+0000 7f3f0f8fb640 1 --2- 172.21.15.44:0/141361503 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f08102120 0x7f3f08102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.619+0000 7f3f0f8fb640 1 --2- 172.21.15.44:0/141361503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f081009e0 0x7f3f08100dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.619+0000 7f3f0f8fb640 1 -- 172.21.15.44:0/141361503 >> 172.21.15.44:0/141361503 conn(0x7f3f08077760 msgr2=0x7f3f08077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:29.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.619+0000 7f3f0f8fb640 1 -- 172.21.15.44:0/141361503 shutdown_connections 2024-04-21T23:17:29.619 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.619+0000 7f3f0f8fb640 1 -- 172.21.15.44:0/141361503 wait complete. 2024-04-21T23:17:29.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.619+0000 7f3f0f8fb640 1 Processor -- start 2024-04-21T23:17:29.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.619+0000 7f3f0f8fb640 1 -- start start 2024-04-21T23:17:29.621 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.619+0000 7f3f0f8fb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f081009e0 0x7f3f08198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.619+0000 7f3f0f8fb640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f08102120 0x7f3f081986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.619+0000 7f3f0f8fb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f08102ac0 0x7f3f0819ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.619+0000 7f3f0f8fb640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3f08111c60 con 0x7f3f081009e0 2024-04-21T23:17:29.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.619+0000 7f3f0f8fb640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3f08111ae0 con 0x7f3f08102120 2024-04-21T23:17:29.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.619+0000 7f3f0f8fb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3f08111de0 con 0x7f3f08102ac0 2024-04-21T23:17:29.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.620+0000 7f3f0d697640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f081009e0 0x7f3f08198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.620+0000 7f3f0d697640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f081009e0 0x7f3f08198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49164/0 (socket says 172.21.15.44:49164) 2024-04-21T23:17:29.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.620+0000 7f3f0d697640 1 -- 172.21.15.44:0/520254859 learned_addr learned my addr 172.21.15.44:0/520254859 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:29.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.620+0000 7f3f0ce96640 1 --2- 172.21.15.44:0/520254859 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f08102120 0x7f3f081986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.620+0000 7f3f0de98640 1 --2- 172.21.15.44:0/520254859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f08102ac0 0x7f3f0819ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.620+0000 7f3f0d697640 1 -- 172.21.15.44:0/520254859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f08102ac0 msgr2=0x7f3f0819ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.620+0000 7f3f0d697640 1 --2- 172.21.15.44:0/520254859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f08102ac0 0x7f3f0819ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.620+0000 7f3f0d697640 1 -- 172.21.15.44:0/520254859 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f08102120 msgr2=0x7f3f081986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.620+0000 7f3f0d697640 1 --2- 172.21.15.44:0/520254859 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f08102120 0x7f3f081986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.620+0000 7f3f0d697640 1 -- 172.21.15.44:0/520254859 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f0819d150 con 0x7f3f081009e0 2024-04-21T23:17:29.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.620+0000 7f3f0d697640 1 --2- 172.21.15.44:0/520254859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f081009e0 0x7f3f08198160 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f3f0400b4a0 tx=0x7f3f0400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:29.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.620+0000 7f3efe7fc640 1 -- 172.21.15.44:0/520254859 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f04014070 con 0x7f3f081009e0 2024-04-21T23:17:29.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.620+0000 7f3efe7fc640 1 -- 172.21.15.44:0/520254859 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f04010af0 con 0x7f3f081009e0 2024-04-21T23:17:29.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.620+0000 7f3f0f8fb640 1 -- 172.21.15.44:0/520254859 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f0819d440 con 0x7f3f081009e0 2024-04-21T23:17:29.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.620+0000 7f3efe7fc640 1 -- 172.21.15.44:0/520254859 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f04017d10 con 0x7f3f081009e0 2024-04-21T23:17:29.624 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.620+0000 7f3f0f8fb640 1 -- 172.21.15.44:0/520254859 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f081a4cd0 con 0x7f3f081009e0 2024-04-21T23:17:29.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.622+0000 7f3efe7fc640 1 -- 172.21.15.44:0/520254859 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3f04017070 con 0x7f3f081009e0 2024-04-21T23:17:29.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.623+0000 7f3f125c3740 1 Processor -- start 2024-04-21T23:17:29.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.623+0000 7f3f125c3740 1 -- start start 2024-04-21T23:17:29.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.623+0000 7f3f125c3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d4204020 0x5565d4204400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.631 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.623+0000 7f3f125c3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204940 0x5565d420cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.623+0000 7f3f125c3740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d420d520 0x5565d420f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.623+0000 7f3f125c3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5565d4168530 con 0x5565d4204940 2024-04-21T23:17:29.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.623+0000 7f3f125c3740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5565d4110830 con 0x5565d420d520 2024-04-21T23:17:29.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.623+0000 7f3f125c3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5565d41644f0 con 0x5565d4204020 2024-04-21T23:17:29.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.623+0000 7f3efe7fc640 1 --2- 172.21.15.44:0/520254859 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3ef40a2d00 0x7f3ef40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.624+0000 7f3f0d697640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204940 0x5565d420cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.624+0000 7f3f0d697640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204940 0x5565d420cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49172/0 (socket says 172.21.15.44:49172) 2024-04-21T23:17:29.632 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.624+0000 7f3f0d697640 1 -- 172.21.15.44:0/2116147304 learned_addr learned my addr 172.21.15.44:0/2116147304 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:29.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.624+0000 7f3f0de98640 1 --2- 172.21.15.44:0/520254859 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3ef40a2d00 0x7f3ef40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.624+0000 7f3efe7fc640 1 -- 172.21.15.44:0/520254859 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7f3f040ceaa0 con 0x7f3f081009e0 2024-04-21T23:17:29.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.624+0000 7f3f0ce96640 1 --2- 172.21.15.44:0/2116147304 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d420d520 0x5565d420f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.624+0000 7f3f0de98640 1 --2- 172.21.15.44:0/2116147304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d4204020 0x5565d4204400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.624+0000 7f3f0d697640 1 -- 172.21.15.44:0/2116147304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d4204020 msgr2=0x5565d4204400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:29.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.624+0000 7f3f0d697640 1 --2- 172.21.15.44:0/2116147304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d4204020 0x5565d4204400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.624+0000 7f3f0d697640 1 -- 172.21.15.44:0/2116147304 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d420d520 msgr2=0x5565d420f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:29.633 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.624+0000 7f3f0d697640 1 --2- 172.21.15.44:0/2116147304 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d420d520 0x5565d420f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.624+0000 7f3f0d697640 1 -- 172.21.15.44:0/2116147304 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565d41342c0 con 0x5565d4204940 2024-04-21T23:17:29.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.624+0000 7f3f0de98640 1 --2- 172.21.15.44:0/2116147304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d4204020 0x5565d4204400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:17:29.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.624+0000 7f3f0d697640 1 --2- 172.21.15.44:0/2116147304 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204940 0x5565d420cfe0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f3f04009420 tx=0x7f3f04017730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb0e2b09f221efaa server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:29.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.624+0000 7f3f0de98640 1 --2- 172.21.15.44:0/520254859 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3ef40a2d00 0x7f3ef40a51c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3f00005e30 tx=0x7f3f00005d80 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:29.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.624+0000 7f3ef37fe640 1 -- 172.21.15.44:0/2116147304 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0401e050 con 0x5565d4204940 2024-04-21T23:17:29.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.624+0000 7f3ef37fe640 1 -- 172.21.15.44:0/2116147304 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f0409b3e0 con 0x5565d4204940 2024-04-21T23:17:29.634 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.624+0000 7f3ef37fe640 1 -- 172.21.15.44:0/2116147304 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f040cc080 con 0x5565d4204940 2024-04-21T23:17:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.625+0000 7f3f125c3740 1 -- 172.21.15.44:0/2116147304 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204940 msgr2=0x5565d420cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.625+0000 7f3f125c3740 1 --2- 172.21.15.44:0/2116147304 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204940 0x5565d420cfe0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f3f04009420 tx=0x7f3f04017730 comp rx=0 tx=0).stop 2024-04-21T23:17:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.625+0000 7f3f125c3740 1 -- 172.21.15.44:0/2116147304 shutdown_connections 2024-04-21T23:17:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.625+0000 7f3f125c3740 1 --2- 172.21.15.44:0/2116147304 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d420d520 0x5565d420f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.625+0000 7f3f125c3740 1 --2- 172.21.15.44:0/2116147304 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204940 0x5565d420cfe0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.625+0000 7f3f125c3740 1 --2- 172.21.15.44:0/2116147304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d4204020 0x5565d4204400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.625+0000 7f3f125c3740 1 -- 172.21.15.44:0/2116147304 >> 172.21.15.44:0/2116147304 conn(0x5565d41fe750 msgr2=0x5565d41feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.625+0000 7f3f125c3740 1 -- 172.21.15.44:0/2116147304 shutdown_connections 2024-04-21T23:17:29.635 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.625+0000 7f3f125c3740 1 -- 172.21.15.44:0/2116147304 wait complete. 2024-04-21T23:17:29.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.625+0000 7f3f125c3740 1 -- start start 2024-04-21T23:17:29.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.626+0000 7f3f125c3740 1 Processor -- start 2024-04-21T23:17:29.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.626+0000 7f3f125c3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d4204020 0x5565d420cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.626+0000 7f3f125c3740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d4204940 0x5565d420a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.626+0000 7f3f125c3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d420d520 0x5565d420aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.626+0000 7f3f125c3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5565d4210250 con 0x5565d420d520 2024-04-21T23:17:29.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.626+0000 7f3f125c3740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5565d4123f10 con 0x5565d4204940 2024-04-21T23:17:29.636 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.626+0000 7f3f125c3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5565d42103d0 con 0x5565d4204020 2024-04-21T23:17:29.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.626+0000 7f3f125c3740 10 client.0 _ll_register_callbacks cb 0x5565d3e7bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:17:29.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.626+0000 7f3f0de98640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d420d520 0x5565d420aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.626+0000 7f3f0de98640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d420d520 0x5565d420aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49188/0 (socket says 172.21.15.44:49188) 2024-04-21T23:17:29.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.626+0000 7f3f0de98640 1 -- 172.21.15.44:0/815870218 learned_addr learned my addr 172.21.15.44:0/815870218 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:29.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.626+0000 7f3f0ce96640 1 --2- 172.21.15.44:0/815870218 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d4204940 0x5565d420a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.626+0000 7f3f0d697640 1 --2- 172.21.15.44:0/815870218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d4204020 0x5565d420cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.627+0000 7f3f0de98640 1 -- 172.21.15.44:0/815870218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d4204020 msgr2=0x5565d420cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:29.637 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.627+0000 7f3f0de98640 1 --2- 172.21.15.44:0/815870218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d4204020 0x5565d420cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.627+0000 7f3f0de98640 1 -- 172.21.15.44:0/815870218 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d4204940 msgr2=0x5565d420a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:29.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.627+0000 7f3f0de98640 1 --2- 172.21.15.44:0/815870218 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d4204940 0x5565d420a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.627+0000 7f3f0de98640 1 -- 172.21.15.44:0/815870218 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565d40f5c80 con 0x5565d420d520 2024-04-21T23:17:29.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.627+0000 7f3f0ce96640 1 --2- 172.21.15.44:0/815870218 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d4204940 0x5565d420a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:17:29.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.627+0000 7f3f0d697640 1 --2- 172.21.15.44:0/815870218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d4204020 0x5565d420cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:17:29.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.627+0000 7f3f0de98640 1 --2- 172.21.15.44:0/815870218 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d420d520 0x5565d420aab0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f3f00036000 tx=0x7f3f0002c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d57e7da770f3b01 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:29.638 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.627+0000 7f3ef0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:17:29.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.627+0000 7f3ef0ff9640 1 -- 172.21.15.44:0/815870218 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f000043c0 con 0x5565d420d520 2024-04-21T23:17:29.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.627+0000 7f3ef0ff9640 1 -- 172.21.15.44:0/815870218 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7f3f00004560 con 0x5565d420d520 2024-04-21T23:17:29.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.627+0000 7f3f125c3740 1 -- 172.21.15.44:0/815870218 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f0409b3e0 con 0x5565d420d520 2024-04-21T23:17:29.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.627+0000 7f3ef0ff9640 1 -- 172.21.15.44:0/815870218 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3f00004760 con 0x5565d420d520 2024-04-21T23:17:29.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.627+0000 7f3f125c3740 10 client.17562 fetch_fsmap learned FSMap version 31 2024-04-21T23:17:29.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.627+0000 7f3f125c3740 1 -- 172.21.15.44:0/815870218 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5565d3e7c3f0 con 0x5565d420d520 2024-04-21T23:17:29.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.627+0000 7f3ef0ff9640 1 -- 172.21.15.44:0/815870218 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f0409b3e0 con 0x5565d420d520 2024-04-21T23:17:29.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.628+0000 7f3ef0ff9640 1 -- 172.21.15.44:0/815870218 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f3f0004d070 con 0x5565d420d520 2024-04-21T23:17:29.639 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.628+0000 7f3f125c3740 10 client.17562 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:17:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.628+0000 7f3f125c3740 10 client.17562 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:17:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.628+0000 7f3f125c3740 20 client.17562 populate_metadata read hostname 'smithi044' 2024-04-21T23:17:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.628+0000 7f3f125c3740 1 --2- 172.21.15.44:0/815870218 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5565d421d690 0x5565d421fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.628+0000 7f3f125c3740 4 client.17562 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:17:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.628+0000 7f3f125c3740 1 -- 172.21.15.44:0/815870218 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565d4175a70 con 0x5565d421d690 2024-04-21T23:17:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.628+0000 7f3f0d697640 1 --2- 172.21.15.44:0/815870218 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5565d421d690 0x5565d421fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.628+0000 7f3ef0ff9640 10 client.17562 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:17:29.640 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.628+0000 7f3f0d697640 1 --2- 172.21.15.44:0/815870218 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5565d421d690 0x5565d421fb10 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e517947149ca78ca server_cookie=32efa013582ef7af in_seq=0 out_seq=0 2024-04-21T23:17:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.629+0000 7f3ef0ff9640 1 -- 172.21.15.44:0/815870218 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5565d4175a70 con 0x5565d421d690 2024-04-21T23:17:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.629+0000 7f3ef0ff9640 10 client.17562 handle_command_reply: tid=0 2024-04-21T23:17:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.629+0000 7f3f125c3740 1 client.17562 shutdown 2024-04-21T23:17:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.629+0000 7f3f125c3740 1 -- 172.21.15.44:0/815870218 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d420d520 msgr2=0x5565d420aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.629+0000 7f3f125c3740 1 --2- 172.21.15.44:0/815870218 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d420d520 0x5565d420aab0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f3f00036000 tx=0x7f3f0002c1d0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.629+0000 7f3f125c3740 1 -- 172.21.15.44:0/815870218 shutdown_connections 2024-04-21T23:17:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.629+0000 7f3f125c3740 1 --2- 172.21.15.44:0/815870218 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5565d421d690 0x5565d421fb10 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.641 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.629+0000 7f3f125c3740 1 --2- 172.21.15.44:0/815870218 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d420d520 0x5565d420aab0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.629+0000 7f3f125c3740 1 --2- 172.21.15.44:0/815870218 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d4204940 0x5565d420a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.629+0000 7f3ef0ff9640 0 client.17562 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:17:29.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.629+0000 7f3f125c3740 1 --2- 172.21.15.44:0/815870218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d4204020 0x5565d420cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.629+0000 7f3f125c3740 1 -- 172.21.15.44:0/815870218 >> 172.21.15.44:0/815870218 conn(0x5565d41fe750 msgr2=0x5565d42004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:29.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.629+0000 7f3f125c3740 1 -- 172.21.15.44:0/815870218 shutdown_connections 2024-04-21T23:17:29.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.629+0000 7f3f125c3740 1 -- 172.21.15.44:0/815870218 wait complete. 2024-04-21T23:17:29.642 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.629+0000 7f3f125c3740 20 client.17562 trim_cache size 0 max 16384 2024-04-21T23:17:29.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.643+0000 7f3f125c3740 1 Processor -- start 2024-04-21T23:17:29.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.643+0000 7f3f125c3740 1 -- start start 2024-04-21T23:17:29.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.643+0000 7f3f125c3740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d4204020 0x5565d4219110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.643+0000 7f3f125c3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204940 0x5565d420ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.643+0000 7f3f125c3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d420d520 0x5565d4209b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.643+0000 7f3f125c3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3f081ace50 con 0x5565d4204940 2024-04-21T23:17:29.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.643+0000 7f3f125c3740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3f081af6b0 con 0x5565d4204020 2024-04-21T23:17:29.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.643+0000 7f3f125c3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3f081ad4a0 con 0x5565d420d520 2024-04-21T23:17:29.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.643+0000 7f3f0ce96640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204940 0x5565d420ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.643+0000 7f3f0ce96640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204940 0x5565d420ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49190/0 (socket says 172.21.15.44:49190) 2024-04-21T23:17:29.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.643+0000 7f3f0ce96640 1 -- 172.21.15.44:0/1351977227 learned_addr learned my addr 172.21.15.44:0/1351977227 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:29.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.643+0000 7f3f0ce96640 1 -- 172.21.15.44:0/1351977227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d420d520 msgr2=0x5565d4209b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:29.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.643+0000 7f3f0ce96640 1 --2- 172.21.15.44:0/1351977227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d420d520 0x5565d4209b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.643+0000 7f3f0ce96640 1 -- 172.21.15.44:0/1351977227 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d4204020 msgr2=0x5565d4219110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:29.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.643+0000 7f3f0ce96640 1 --2- 172.21.15.44:0/1351977227 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d4204020 0x5565d4219110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.643+0000 7f3f0ce96640 1 -- 172.21.15.44:0/1351977227 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565d3f467d0 con 0x5565d4204940 2024-04-21T23:17:29.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.643+0000 7f3f0ce96640 1 --2- 172.21.15.44:0/1351977227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204940 0x5565d420ba90 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f3ef800b850 tx=0x7f3ef800bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee7806b65a251b00 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:29.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.644+0000 7f3ef2ffd640 1 -- 172.21.15.44:0/1351977227 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef8003d50 con 0x5565d4204940 2024-04-21T23:17:29.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.644+0000 7f3ef2ffd640 1 -- 172.21.15.44:0/1351977227 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ef8020070 con 0x5565d4204940 2024-04-21T23:17:29.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.644+0000 7f3ef2ffd640 1 -- 172.21.15.44:0/1351977227 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef801b770 con 0x5565d4204940 2024-04-21T23:17:29.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.644+0000 7f3f125c3740 1 -- 172.21.15.44:0/1351977227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204940 msgr2=0x5565d420ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.644+0000 7f3f125c3740 1 --2- 172.21.15.44:0/1351977227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204940 0x5565d420ba90 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f3ef800b850 tx=0x7f3ef800bd10 comp rx=0 tx=0).stop 2024-04-21T23:17:29.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.644+0000 7f3f125c3740 1 -- 172.21.15.44:0/1351977227 shutdown_connections 2024-04-21T23:17:29.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.644+0000 7f3f125c3740 1 --2- 172.21.15.44:0/1351977227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d420d520 0x5565d4209b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.644+0000 7f3f125c3740 1 --2- 172.21.15.44:0/1351977227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204940 0x5565d420ba90 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.644+0000 7f3f125c3740 1 --2- 172.21.15.44:0/1351977227 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d4204020 0x5565d4219110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.644+0000 7f3f125c3740 1 -- 172.21.15.44:0/1351977227 >> 172.21.15.44:0/1351977227 conn(0x5565d41fe750 msgr2=0x5565d41fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:29.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.644+0000 7f3f125c3740 1 -- 172.21.15.44:0/1351977227 shutdown_connections 2024-04-21T23:17:29.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.644+0000 7f3f125c3740 1 -- 172.21.15.44:0/1351977227 wait complete. 2024-04-21T23:17:29.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.645+0000 7f3f125c3740 1 -- start start 2024-04-21T23:17:29.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.645+0000 7f3f125c3740 1 Processor -- start 2024-04-21T23:17:29.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.645+0000 7f3f125c3740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204020 0x5565d4207440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.645+0000 7f3f125c3740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d4204940 0x5565d4207980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.645+0000 7f3f125c3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d420d520 0x5565d4282c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.645+0000 7f3f125c3740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3f081adb70 con 0x5565d4204020 2024-04-21T23:17:29.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.645+0000 7f3f125c3740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3f081aefe0 con 0x5565d4204940 2024-04-21T23:17:29.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.645+0000 7f3f125c3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3f081ae240 con 0x5565d420d520 2024-04-21T23:17:29.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.645+0000 7f3f125c3740 10 client.0 _ll_register_callbacks cb 0x7f3f00037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:17:29.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.645+0000 7f3f0d697640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204020 0x5565d4207440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.645+0000 7f3f0d697640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204020 0x5565d4207440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49206/0 (socket says 172.21.15.44:49206) 2024-04-21T23:17:29.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.645+0000 7f3f0d697640 1 -- 172.21.15.44:0/1447512734 learned_addr learned my addr 172.21.15.44:0/1447512734 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:29.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.645+0000 7f3f0ce96640 1 --2- 172.21.15.44:0/1447512734 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d4204940 0x5565d4207980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.645+0000 7f3f0de98640 1 --2- 172.21.15.44:0/1447512734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d420d520 0x5565d4282c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.645+0000 7f3f0d697640 1 -- 172.21.15.44:0/1447512734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d420d520 msgr2=0x5565d4282c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:29.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.645+0000 7f3f0d697640 1 --2- 172.21.15.44:0/1447512734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d420d520 0x5565d4282c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.645+0000 7f3f0d697640 1 -- 172.21.15.44:0/1447512734 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d4204940 msgr2=0x5565d4207980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:29.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.645+0000 7f3f0d697640 1 --2- 172.21.15.44:0/1447512734 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d4204940 0x5565d4207980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.645+0000 7f3f0d697640 1 -- 172.21.15.44:0/1447512734 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565d4283190 con 0x5565d4204020 2024-04-21T23:17:29.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.645+0000 7f3f0ce96640 1 --2- 172.21.15.44:0/1447512734 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d4204940 0x5565d4207980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:17:29.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.646+0000 7f3f0d697640 1 --2- 172.21.15.44:0/1447512734 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204020 0x5565d4207440 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f3f0409ac70 tx=0x7f3f040cee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c952b7e987ce26 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:29.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.646+0000 7f3ef0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:17:29.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.646+0000 7f3ef0ff9640 1 -- 172.21.15.44:0/1447512734 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0401e040 con 0x5565d4204020 2024-04-21T23:17:29.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.646+0000 7f3ef0ff9640 1 -- 172.21.15.44:0/1447512734 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7f3f04097350 con 0x5565d4204020 2024-04-21T23:17:29.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.646+0000 7f3f125c3740 1 -- 172.21.15.44:0/1447512734 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ef8020070 con 0x5565d4204020 2024-04-21T23:17:29.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.646+0000 7f3ef0ff9640 1 -- 172.21.15.44:0/1447512734 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3f0409b3e0 con 0x5565d4204020 2024-04-21T23:17:29.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.646+0000 7f3f125c3740 10 client.17568 fetch_fsmap learned FSMap version 31 2024-04-21T23:17:29.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.646+0000 7f3f125c3740 1 -- 172.21.15.44:0/1447512734 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5565d42843b0 con 0x5565d4204020 2024-04-21T23:17:29.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.646+0000 7f3ef0ff9640 1 -- 172.21.15.44:0/1447512734 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ef8020070 con 0x5565d4204020 2024-04-21T23:17:29.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.646+0000 7f3ef0ff9640 1 -- 172.21.15.44:0/1447512734 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f3f0409a130 con 0x5565d4204020 2024-04-21T23:17:29.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.646+0000 7f3f125c3740 10 client.17568 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:17:29.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.646+0000 7f3f125c3740 10 client.17568 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:17:29.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.646+0000 7f3f125c3740 20 client.17568 populate_metadata read hostname 'smithi044' 2024-04-21T23:17:29.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.646+0000 7f3f125c3740 1 --2- 172.21.15.44:0/1447512734 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5565d4284740 0x5565d4286b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:29.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.646+0000 7f3f125c3740 4 client.17568 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:17:29.657 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.646+0000 7f3f125c3740 1 -- 172.21.15.44:0/1447512734 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5565d416fdc0 con 0x5565d4284740 2024-04-21T23:17:29.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.647+0000 7f3f0ce96640 1 --2- 172.21.15.44:0/1447512734 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5565d4284740 0x5565d4286b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:29.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.647+0000 7f3f0ce96640 1 --2- 172.21.15.44:0/1447512734 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5565d4284740 0x5565d4286b30 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccd83c02dd3a447a server_cookie=b1d0c6c9bc1597fa in_seq=0 out_seq=0 2024-04-21T23:17:29.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.647+0000 7f3ef0ff9640 10 client.17568 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:17:29.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.647+0000 7f3ef0ff9640 1 -- 172.21.15.44:0/1447512734 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5565d416fdc0 con 0x5565d4284740 2024-04-21T23:17:29.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.647+0000 7f3ef0ff9640 10 client.17568 handle_command_reply: tid=0 2024-04-21T23:17:29.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.647+0000 7f3f125c3740 1 client.17568 shutdown 2024-04-21T23:17:29.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.647+0000 7f3f125c3740 1 -- 172.21.15.44:0/1447512734 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204020 msgr2=0x5565d4207440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.647+0000 7f3f125c3740 1 --2- 172.21.15.44:0/1447512734 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204020 0x5565d4207440 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f3f0409ac70 tx=0x7f3f040cee20 comp rx=0 tx=0).stop 2024-04-21T23:17:29.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.647+0000 7f3f125c3740 1 -- 172.21.15.44:0/1447512734 shutdown_connections 2024-04-21T23:17:29.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.647+0000 7f3f125c3740 1 --2- 172.21.15.44:0/1447512734 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5565d4284740 0x5565d4286b30 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.647+0000 7f3f125c3740 1 --2- 172.21.15.44:0/1447512734 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5565d420d520 0x5565d4282c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.647+0000 7f3f125c3740 1 --2- 172.21.15.44:0/1447512734 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5565d4204940 0x5565d4207980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.647+0000 7f3ef0ff9640 0 client.17568 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:17:29.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.647+0000 7f3f125c3740 1 --2- 172.21.15.44:0/1447512734 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5565d4204020 0x5565d4207440 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.647+0000 7f3f125c3740 1 -- 172.21.15.44:0/1447512734 >> 172.21.15.44:0/1447512734 conn(0x5565d41fe750 msgr2=0x5565d420b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:29.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.647+0000 7f3f125c3740 1 -- 172.21.15.44:0/1447512734 shutdown_connections 2024-04-21T23:17:29.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.648+0000 7f3f125c3740 1 -- 172.21.15.44:0/1447512734 wait complete. 2024-04-21T23:17:29.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.648+0000 7f3f125c3740 20 client.17568 trim_cache size 0 max 16384 2024-04-21T23:17:29.662 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-21T23:17:29.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.648+0000 7f3f0f8fb640 1 -- 172.21.15.44:0/520254859 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3ef40a2d00 msgr2=0x7f3ef40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.648+0000 7f3f0f8fb640 1 --2- 172.21.15.44:0/520254859 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3ef40a2d00 0x7f3ef40a51c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3f00005e30 tx=0x7f3f00005d80 comp rx=0 tx=0).stop 2024-04-21T23:17:29.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.648+0000 7f3f0f8fb640 1 -- 172.21.15.44:0/520254859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f081009e0 msgr2=0x7f3f08198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:29.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.648+0000 7f3f0f8fb640 1 --2- 172.21.15.44:0/520254859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f081009e0 0x7f3f08198160 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f3f0400b4a0 tx=0x7f3f0400b960 comp rx=0 tx=0).stop 2024-04-21T23:17:29.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.648+0000 7f3f0f8fb640 1 -- 172.21.15.44:0/520254859 shutdown_connections 2024-04-21T23:17:29.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.648+0000 7f3f0f8fb640 1 --2- 172.21.15.44:0/520254859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f08102ac0 0x7f3f0819ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.648+0000 7f3f0f8fb640 1 --2- 172.21.15.44:0/520254859 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3ef40a2d00 0x7f3ef40a51c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.648+0000 7f3f0f8fb640 1 --2- 172.21.15.44:0/520254859 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f08102120 0x7f3f081986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.648+0000 7f3f0f8fb640 1 --2- 172.21.15.44:0/520254859 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f081009e0 0x7f3f08198160 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:29.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.648+0000 7f3f0f8fb640 1 -- 172.21.15.44:0/520254859 >> 172.21.15.44:0/520254859 conn(0x7f3f08077760 msgr2=0x7f3f08105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:29.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.648+0000 7f3f0f8fb640 1 -- 172.21.15.44:0/520254859 shutdown_connections 2024-04-21T23:17:29.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:29.648+0000 7f3f0f8fb640 1 -- 172.21.15.44:0/520254859 wait complete. 2024-04-21T23:17:30.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:30 smithi069 ceph-mon[31816]: pgmap v273: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 2.5 MiB/s wr, 133 op/s 2024-04-21T23:17:30.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:30 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2615146396' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:17:30.666 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:17:30.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:17:30.667 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:17:30.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:30 smithi044 ceph-mon[27934]: pgmap v273: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 2.5 MiB/s wr, 133 op/s 2024-04-21T23:17:30.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:30 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2615146396' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:17:30.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:30 smithi055 ceph-mon[31117]: pgmap v273: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 2.5 MiB/s wr, 133 op/s 2024-04-21T23:17:30.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:30 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2615146396' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:17:30.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.788+0000 7f3f59b5c640 1 -- 172.21.15.44:0/793485423 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f541029b0 msgr2=0x7f3f54102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.788+0000 7f3f59b5c640 1 --2- 172.21.15.44:0/793485423 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f541029b0 0x7f3f54102d90 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009900 tx=0x7f3f4402bc30 comp rx=0 tx=0).stop 2024-04-21T23:17:30.788 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.788+0000 7f3f59b5c640 1 -- 172.21.15.44:0/793485423 shutdown_connections 2024-04-21T23:17:30.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.788+0000 7f3f59b5c640 1 --2- 172.21.15.44:0/793485423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f54103360 0x7f3f5410b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.788+0000 7f3f59b5c640 1 --2- 172.21.15.44:0/793485423 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f5410bec0 0x7f3f5410e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.788+0000 7f3f59b5c640 1 --2- 172.21.15.44:0/793485423 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f541029b0 0x7f3f54102d90 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.788+0000 7f3f59b5c640 1 -- 172.21.15.44:0/793485423 >> 172.21.15.44:0/793485423 conn(0x7f3f54077760 msgr2=0x7f3f54077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:30.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.788+0000 7f3f59b5c640 1 -- 172.21.15.44:0/793485423 shutdown_connections 2024-04-21T23:17:30.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.788+0000 7f3f59b5c640 1 -- 172.21.15.44:0/793485423 wait complete. 2024-04-21T23:17:30.789 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.789+0000 7f3f59b5c640 1 Processor -- start 2024-04-21T23:17:30.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.789+0000 7f3f59b5c640 1 -- start start 2024-04-21T23:17:30.791 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.789+0000 7f3f59b5c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f541029b0 0x7f3f541982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.789+0000 7f3f59b5c640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f5410bec0 0x7f3f54198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.789+0000 7f3f59b5c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f5419cb90 0x7f3f5419d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.789+0000 7f3f59b5c640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3f54111cd0 con 0x7f3f541029b0 2024-04-21T23:17:30.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.789+0000 7f3f59b5c640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3f54111b50 con 0x7f3f5410bec0 2024-04-21T23:17:30.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.789+0000 7f3f59b5c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3f54111e50 con 0x7f3f5419cb90 2024-04-21T23:17:30.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.789+0000 7f3f537fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f541029b0 0x7f3f541982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.789+0000 7f3f537fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f541029b0 0x7f3f541982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60138/0 (socket says 172.21.15.44:60138) 2024-04-21T23:17:30.792 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.789+0000 7f3f537fe640 1 -- 172.21.15.44:0/3311276109 learned_addr learned my addr 172.21.15.44:0/3311276109 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:30.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.790+0000 7f3f52ffd640 1 --2- 172.21.15.44:0/3311276109 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f5410bec0 0x7f3f54198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.790+0000 7f3f53fff640 1 --2- 172.21.15.44:0/3311276109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f5419cb90 0x7f3f5419d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.790+0000 7f3f537fe640 1 -- 172.21.15.44:0/3311276109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f5419cb90 msgr2=0x7f3f5419d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.790+0000 7f3f537fe640 1 --2- 172.21.15.44:0/3311276109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f5419cb90 0x7f3f5419d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.790+0000 7f3f537fe640 1 -- 172.21.15.44:0/3311276109 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f5410bec0 msgr2=0x7f3f54198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.790+0000 7f3f537fe640 1 --2- 172.21.15.44:0/3311276109 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f5410bec0 0x7f3f54198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.790+0000 7f3f537fe640 1 -- 172.21.15.44:0/3311276109 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f5419d680 con 0x7f3f541029b0 2024-04-21T23:17:30.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.790+0000 7f3f537fe640 1 --2- 172.21.15.44:0/3311276109 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f541029b0 0x7f3f541982d0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f3f44006010 tx=0x7f3f4402cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:30.793 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.790+0000 7f3f50ff9640 1 -- 172.21.15.44:0/3311276109 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f44041070 con 0x7f3f541029b0 2024-04-21T23:17:30.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.790+0000 7f3f50ff9640 1 -- 172.21.15.44:0/3311276109 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f44004450 con 0x7f3f541029b0 2024-04-21T23:17:30.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.790+0000 7f3f59b5c640 1 -- 172.21.15.44:0/3311276109 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f5419d910 con 0x7f3f541029b0 2024-04-21T23:17:30.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.790+0000 7f3f50ff9640 1 -- 172.21.15.44:0/3311276109 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f44034440 con 0x7f3f541029b0 2024-04-21T23:17:30.794 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.790+0000 7f3f59b5c640 1 -- 172.21.15.44:0/3311276109 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f541a50d0 con 0x7f3f541029b0 2024-04-21T23:17:30.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.792+0000 7f3f50ff9640 1 -- 172.21.15.44:0/3311276109 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3f44044070 con 0x7f3f541029b0 2024-04-21T23:17:30.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.793+0000 7f3f5c818740 1 Processor -- start 2024-04-21T23:17:30.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.793+0000 7f3f5c818740 1 -- start start 2024-04-21T23:17:30.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.793+0000 7f3f5c818740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a018720 0x55f84a017470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.793+0000 7f3f5c818740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a017a40 0x55f84a017ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.793+0000 7f3f5c818740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a020db0 0x55f84a0231a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.793+0000 7f3f5c818740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f849f7b170 con 0x55f84a020db0 2024-04-21T23:17:30.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.793+0000 7f3f5c818740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55f849f237c0 con 0x55f84a017a40 2024-04-21T23:17:30.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.793+0000 7f3f5c818740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f849f77130 con 0x55f84a018720 2024-04-21T23:17:30.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.793+0000 7f3f50ff9640 1 --2- 172.21.15.44:0/3311276109 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3f3c0a2d00 0x7f3f3c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.793+0000 7f3f52ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a020db0 0x55f84a0231a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.793+0000 7f3f52ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a020db0 0x55f84a0231a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60148/0 (socket says 172.21.15.44:60148) 2024-04-21T23:17:30.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.793+0000 7f3f52ffd640 1 -- 172.21.15.44:0/3196475113 learned_addr learned my addr 172.21.15.44:0/3196475113 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:30.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.793+0000 7f3f53fff640 1 --2- 172.21.15.44:0/3196475113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a018720 0x55f84a017470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.793+0000 7f3f53fff640 1 --2- 172.21.15.44:0/3311276109 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3f3c0a2d00 0x7f3f3c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.793+0000 7f3f50ff9640 1 -- 172.21.15.44:0/3311276109 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7f3f440f1340 con 0x7f3f541029b0 2024-04-21T23:17:30.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.793+0000 7f3f537fe640 1 --2- 172.21.15.44:0/3196475113 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a017a40 0x55f84a017ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.794+0000 7f3f52ffd640 1 -- 172.21.15.44:0/3196475113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a018720 msgr2=0x55f84a017470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:30.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.794+0000 7f3f52ffd640 1 --2- 172.21.15.44:0/3196475113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a018720 0x55f84a017470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.794+0000 7f3f52ffd640 1 -- 172.21.15.44:0/3196475113 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a017a40 msgr2=0x55f84a017ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:30.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.794+0000 7f3f52ffd640 1 --2- 172.21.15.44:0/3196475113 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a017a40 0x55f84a017ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.794+0000 7f3f52ffd640 1 -- 172.21.15.44:0/3196475113 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f849f34e60 con 0x55f84a020db0 2024-04-21T23:17:30.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.794+0000 7f3f53fff640 1 --2- 172.21.15.44:0/3196475113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a018720 0x55f84a017470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:17:30.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.794+0000 7f3f52ffd640 1 --2- 172.21.15.44:0/3196475113 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a020db0 0x55f84a0231a0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800ab70 tx=0x7f3f4800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8edeed816035380 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:30.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.794+0000 7f3f53fff640 1 --2- 172.21.15.44:0/3311276109 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3f3c0a2d00 0x7f3f3c0a51c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c009fd0 tx=0x7f3f4c009600 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:30.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.794+0000 7f3f39ffb640 1 -- 172.21.15.44:0/3196475113 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48004220 con 0x55f84a020db0 2024-04-21T23:17:30.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.794+0000 7f3f39ffb640 1 -- 172.21.15.44:0/3196475113 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f480043c0 con 0x55f84a020db0 2024-04-21T23:17:30.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.794+0000 7f3f39ffb640 1 -- 172.21.15.44:0/3196475113 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48018840 con 0x55f84a020db0 2024-04-21T23:17:30.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.795+0000 7f3f5c818740 1 -- 172.21.15.44:0/3196475113 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a020db0 msgr2=0x55f84a0231a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.795+0000 7f3f5c818740 1 --2- 172.21.15.44:0/3196475113 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a020db0 0x55f84a0231a0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800ab70 tx=0x7f3f4800f790 comp rx=0 tx=0).stop 2024-04-21T23:17:30.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.795+0000 7f3f5c818740 1 -- 172.21.15.44:0/3196475113 shutdown_connections 2024-04-21T23:17:30.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.795+0000 7f3f5c818740 1 --2- 172.21.15.44:0/3196475113 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a020db0 0x55f84a0231a0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.795+0000 7f3f5c818740 1 --2- 172.21.15.44:0/3196475113 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a017a40 0x55f84a017ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.795+0000 7f3f5c818740 1 --2- 172.21.15.44:0/3196475113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a018720 0x55f84a017470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.795+0000 7f3f5c818740 1 -- 172.21.15.44:0/3196475113 >> 172.21.15.44:0/3196475113 conn(0x55f849fde4f0 msgr2=0x55f84a012490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:30.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.795+0000 7f3f5c818740 1 -- 172.21.15.44:0/3196475113 shutdown_connections 2024-04-21T23:17:30.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.795+0000 7f3f5c818740 1 -- 172.21.15.44:0/3196475113 wait complete. 2024-04-21T23:17:30.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.795+0000 7f3f5c818740 1 -- start start 2024-04-21T23:17:30.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.795+0000 7f3f5c818740 1 Processor -- start 2024-04-21T23:17:30.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f5c818740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a017a40 0x55f84a01dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f5c818740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a018720 0x55f84a01e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f5c818740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a020db0 0x55f84a01e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f5c818740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55f84a023c20 con 0x55f84a020db0 2024-04-21T23:17:30.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f5c818740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55f849f36d40 con 0x55f84a017a40 2024-04-21T23:17:30.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f5c818740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f84a023da0 con 0x55f84a018720 2024-04-21T23:17:30.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f5c818740 10 client.0 _ll_register_callbacks cb 0x55f849f72820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:17:30.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f53fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a020db0 0x55f84a01e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f53fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a020db0 0x55f84a01e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60150/0 (socket says 172.21.15.44:60150) 2024-04-21T23:17:30.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f53fff640 1 -- 172.21.15.44:0/3440113480 learned_addr learned my addr 172.21.15.44:0/3440113480 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:30.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f537fe640 1 --2- 172.21.15.44:0/3440113480 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a017a40 0x55f84a01dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f52ffd640 1 --2- 172.21.15.44:0/3440113480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a018720 0x55f84a01e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f53fff640 1 -- 172.21.15.44:0/3440113480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a018720 msgr2=0x55f84a01e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:30.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f53fff640 1 --2- 172.21.15.44:0/3440113480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a018720 0x55f84a01e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f53fff640 1 -- 172.21.15.44:0/3440113480 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a017a40 msgr2=0x55f84a01dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:30.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f53fff640 1 --2- 172.21.15.44:0/3440113480 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a017a40 0x55f84a01dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f53fff640 1 -- 172.21.15.44:0/3440113480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f849d577e0 con 0x55f84a020db0 2024-04-21T23:17:30.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f52ffd640 1 --2- 172.21.15.44:0/3440113480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a018720 0x55f84a01e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:30.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f53fff640 1 --2- 172.21.15.44:0/3440113480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a020db0 0x55f84a01e7d0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00abf0 tx=0x7f3f4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d737ca19428ee51f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:30.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:17:30.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f32ffd640 1 -- 172.21.15.44:0/3440113480 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c024070 con 0x55f84a020db0 2024-04-21T23:17:30.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f32ffd640 1 -- 172.21.15.44:0/3440113480 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7f3f4c010d20 con 0x55f84a020db0 2024-04-21T23:17:30.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.796+0000 7f3f5c818740 1 -- 172.21.15.44:0/3440113480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f480043c0 con 0x55f84a020db0 2024-04-21T23:17:30.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.797+0000 7f3f32ffd640 1 -- 172.21.15.44:0/3440113480 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3f4c004b20 con 0x55f84a020db0 2024-04-21T23:17:30.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.797+0000 7f3f5c818740 10 client.17580 fetch_fsmap learned FSMap version 31 2024-04-21T23:17:30.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.797+0000 7f3f5c818740 1 -- 172.21.15.44:0/3440113480 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f849db7f70 con 0x55f84a020db0 2024-04-21T23:17:30.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.797+0000 7f3f32ffd640 1 -- 172.21.15.44:0/3440113480 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f480043c0 con 0x55f84a020db0 2024-04-21T23:17:30.808 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.797+0000 7f3f32ffd640 1 -- 172.21.15.44:0/3440113480 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f3f4c00f040 con 0x55f84a020db0 2024-04-21T23:17:30.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.797+0000 7f3f5c818740 10 client.17580 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:17:30.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.797+0000 7f3f5c818740 10 client.17580 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:17:30.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.797+0000 7f3f5c818740 20 client.17580 populate_metadata read hostname 'smithi044' 2024-04-21T23:17:30.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.797+0000 7f3f5c818740 1 --2- 172.21.15.44:0/3440113480 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f84a030f60 0x55f84a0333e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.797+0000 7f3f5c818740 4 client.17580 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:17:30.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.797+0000 7f3f5c818740 1 -- 172.21.15.44:0/3440113480 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f849f4f450 con 0x55f84a030f60 2024-04-21T23:17:30.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.797+0000 7f3f537fe640 1 --2- 172.21.15.44:0/3440113480 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f84a030f60 0x55f84a0333e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.798+0000 7f3f537fe640 1 --2- 172.21.15.44:0/3440113480 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f84a030f60 0x55f84a0333e0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edf9ca1b1ca4030b server_cookie=a328aa4e78ea1abe in_seq=0 out_seq=0 2024-04-21T23:17:30.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.798+0000 7f3f32ffd640 10 client.17580 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:17:30.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.799+0000 7f3f32ffd640 1 -- 172.21.15.44:0/3440113480 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f849f4f450 con 0x55f84a030f60 2024-04-21T23:17:30.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.799+0000 7f3f32ffd640 10 client.17580 handle_command_reply: tid=0 2024-04-21T23:17:30.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.799+0000 7f3f5c818740 1 client.17580 shutdown 2024-04-21T23:17:30.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.799+0000 7f3f5c818740 1 -- 172.21.15.44:0/3440113480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a020db0 msgr2=0x55f84a01e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.799+0000 7f3f5c818740 1 --2- 172.21.15.44:0/3440113480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a020db0 0x55f84a01e7d0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00abf0 tx=0x7f3f4c004a40 comp rx=0 tx=0).stop 2024-04-21T23:17:30.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.799+0000 7f3f5c818740 1 -- 172.21.15.44:0/3440113480 shutdown_connections 2024-04-21T23:17:30.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.799+0000 7f3f5c818740 1 --2- 172.21.15.44:0/3440113480 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f84a030f60 0x55f84a0333e0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.799+0000 7f3f5c818740 1 --2- 172.21.15.44:0/3440113480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a018720 0x55f84a01e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.799+0000 7f3f5c818740 1 --2- 172.21.15.44:0/3440113480 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a020db0 0x55f84a01e7d0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.799+0000 7f3f5c818740 1 --2- 172.21.15.44:0/3440113480 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a017a40 0x55f84a01dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.799+0000 7f3f32ffd640 0 client.17580 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:17:30.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.799+0000 7f3f5c818740 1 -- 172.21.15.44:0/3440113480 >> 172.21.15.44:0/3440113480 conn(0x55f849fde4f0 msgr2=0x55f84a0141e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:30.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.799+0000 7f3f5c818740 1 -- 172.21.15.44:0/3440113480 shutdown_connections 2024-04-21T23:17:30.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.799+0000 7f3f5c818740 1 -- 172.21.15.44:0/3440113480 wait complete. 2024-04-21T23:17:30.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.799+0000 7f3f5c818740 20 client.17580 trim_cache size 0 max 16384 2024-04-21T23:17:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.817+0000 7f3f5c818740 1 Processor -- start 2024-04-21T23:17:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.817+0000 7f3f5c818740 1 -- start start 2024-04-21T23:17:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.817+0000 7f3f5c818740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a017a40 0x55f84a020480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.817+0000 7f3f5c818740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a018720 0x55f84a02cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.817+0000 7f3f5c818740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a020db0 0x55f84a013ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.817+0000 7f3f5c818740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541accd0 con 0x55f84a018720 2024-04-21T23:17:30.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.817+0000 7f3f5c818740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541af530 con 0x55f84a020db0 2024-04-21T23:17:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.817+0000 7f3f5c818740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541ad320 con 0x55f84a017a40 2024-04-21T23:17:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.817+0000 7f3f52ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a018720 0x55f84a02cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.817+0000 7f3f52ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a018720 0x55f84a02cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60156/0 (socket says 172.21.15.44:60156) 2024-04-21T23:17:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.817+0000 7f3f52ffd640 1 -- 172.21.15.44:0/2381123781 learned_addr learned my addr 172.21.15.44:0/2381123781 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.817+0000 7f3f52ffd640 1 -- 172.21.15.44:0/2381123781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a017a40 msgr2=0x55f84a020480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.817+0000 7f3f52ffd640 1 --2- 172.21.15.44:0/2381123781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a017a40 0x55f84a020480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.817+0000 7f3f52ffd640 1 -- 172.21.15.44:0/2381123781 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a020db0 msgr2=0x55f84a013ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:30.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.817+0000 7f3f53fff640 1 --2- 172.21.15.44:0/2381123781 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a020db0 0x55f84a013ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.817+0000 7f3f52ffd640 1 --2- 172.21.15.44:0/2381123781 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a020db0 0x55f84a013ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.817+0000 7f3f52ffd640 1 -- 172.21.15.44:0/2381123781 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f84a01cd00 con 0x55f84a018720 2024-04-21T23:17:30.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.817+0000 7f3f53fff640 1 --2- 172.21.15.44:0/2381123781 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a020db0 0x55f84a013ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:30.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.817+0000 7f3f52ffd640 1 --2- 172.21.15.44:0/2381123781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a018720 0x55f84a02cb20 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800aaa0 tx=0x7f3f4800aad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7cc18e5d1f525f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:30.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.818+0000 7f3f397fa640 1 -- 172.21.15.44:0/2381123781 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4800a570 con 0x55f84a018720 2024-04-21T23:17:30.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.818+0000 7f3f397fa640 1 -- 172.21.15.44:0/2381123781 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f4800a710 con 0x55f84a018720 2024-04-21T23:17:30.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.818+0000 7f3f397fa640 1 -- 172.21.15.44:0/2381123781 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48004880 con 0x55f84a018720 2024-04-21T23:17:30.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.818+0000 7f3f5c818740 1 -- 172.21.15.44:0/2381123781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a018720 msgr2=0x55f84a02cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.818+0000 7f3f5c818740 1 --2- 172.21.15.44:0/2381123781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a018720 0x55f84a02cb20 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800aaa0 tx=0x7f3f4800aad0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.818+0000 7f3f5c818740 1 -- 172.21.15.44:0/2381123781 shutdown_connections 2024-04-21T23:17:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.818+0000 7f3f5c818740 1 --2- 172.21.15.44:0/2381123781 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a020db0 0x55f84a013ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.818+0000 7f3f5c818740 1 --2- 172.21.15.44:0/2381123781 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a018720 0x55f84a02cb20 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.818+0000 7f3f5c818740 1 --2- 172.21.15.44:0/2381123781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a017a40 0x55f84a020480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.818+0000 7f3f5c818740 1 -- 172.21.15.44:0/2381123781 >> 172.21.15.44:0/2381123781 conn(0x55f849fde4f0 msgr2=0x55f84a0138e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.818+0000 7f3f5c818740 1 -- 172.21.15.44:0/2381123781 shutdown_connections 2024-04-21T23:17:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.818+0000 7f3f5c818740 1 -- 172.21.15.44:0/2381123781 wait complete. 2024-04-21T23:17:30.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.818+0000 7f3f5c818740 1 -- start start 2024-04-21T23:17:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.819+0000 7f3f5c818740 1 Processor -- start 2024-04-21T23:17:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.819+0000 7f3f5c818740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a017a40 0x55f84a029400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.819+0000 7f3f5c818740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a018720 0x55f84a026000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.819+0000 7f3f5c818740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a020db0 0x55f84a026540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.819+0000 7f3f5c818740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541ad9f0 con 0x55f84a017a40 2024-04-21T23:17:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.819+0000 7f3f5c818740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541aee60 con 0x55f84a020db0 2024-04-21T23:17:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.819+0000 7f3f5c818740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541ae0c0 con 0x55f84a018720 2024-04-21T23:17:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.819+0000 7f3f5c818740 10 client.0 _ll_register_callbacks cb 0x7f3f4c00a840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:17:30.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.819+0000 7f3f537fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a017a40 0x55f84a029400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.819+0000 7f3f537fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a017a40 0x55f84a029400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60172/0 (socket says 172.21.15.44:60172) 2024-04-21T23:17:30.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.819+0000 7f3f537fe640 1 -- 172.21.15.44:0/370886579 learned_addr learned my addr 172.21.15.44:0/370886579 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:30.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.819+0000 7f3f52ffd640 1 --2- 172.21.15.44:0/370886579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a018720 0x55f84a026000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.819+0000 7f3f537fe640 1 -- 172.21.15.44:0/370886579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a018720 msgr2=0x55f84a026000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:30.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.819+0000 7f3f537fe640 1 --2- 172.21.15.44:0/370886579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a018720 0x55f84a026000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.819+0000 7f3f537fe640 1 -- 172.21.15.44:0/370886579 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a020db0 msgr2=0x55f84a026540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:30.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.819+0000 7f3f53fff640 1 --2- 172.21.15.44:0/370886579 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a020db0 0x55f84a026540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.819+0000 7f3f537fe640 1 --2- 172.21.15.44:0/370886579 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a020db0 0x55f84a026540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.819+0000 7f3f537fe640 1 -- 172.21.15.44:0/370886579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f849c4f170 con 0x55f84a017a40 2024-04-21T23:17:30.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.820+0000 7f3f537fe640 1 --2- 172.21.15.44:0/370886579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a017a40 0x55f84a029400 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f3f44005940 tx=0x7f3f4402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=720b77b8d59603b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:30.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.820+0000 7f3f32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:17:30.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.820+0000 7f3f32ffd640 1 -- 172.21.15.44:0/370886579 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4404b040 con 0x55f84a017a40 2024-04-21T23:17:30.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.820+0000 7f3f32ffd640 1 -- 172.21.15.44:0/370886579 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7f3f440b9c40 con 0x55f84a017a40 2024-04-21T23:17:30.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.820+0000 7f3f5c818740 1 -- 172.21.15.44:0/370886579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f4800a710 con 0x55f84a017a40 2024-04-21T23:17:30.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.820+0000 7f3f32ffd640 1 -- 172.21.15.44:0/370886579 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3f440f1120 con 0x55f84a017a40 2024-04-21T23:17:30.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.820+0000 7f3f5c818740 10 client.17586 fetch_fsmap learned FSMap version 31 2024-04-21T23:17:30.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.820+0000 7f3f5c818740 1 -- 172.21.15.44:0/370886579 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f84a098370 con 0x55f84a017a40 2024-04-21T23:17:30.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.820+0000 7f3f32ffd640 1 -- 172.21.15.44:0/370886579 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f4800a710 con 0x55f84a017a40 2024-04-21T23:17:30.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.820+0000 7f3f32ffd640 1 -- 172.21.15.44:0/370886579 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f3f440ee8c0 con 0x55f84a017a40 2024-04-21T23:17:30.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.820+0000 7f3f5c818740 10 client.17586 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:17:30.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.820+0000 7f3f5c818740 10 client.17586 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:17:30.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.820+0000 7f3f5c818740 20 client.17586 populate_metadata read hostname 'smithi044' 2024-04-21T23:17:30.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.820+0000 7f3f5c818740 1 --2- 172.21.15.44:0/370886579 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f84a098a90 0x55f84a09ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.820+0000 7f3f5c818740 4 client.17586 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:17:30.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.820+0000 7f3f5c818740 1 -- 172.21.15.44:0/370886579 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f849f886b0 con 0x55f84a098a90 2024-04-21T23:17:30.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.821+0000 7f3f52ffd640 1 --2- 172.21.15.44:0/370886579 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f84a098a90 0x55f84a09ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.821+0000 7f3f52ffd640 1 --2- 172.21.15.44:0/370886579 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f84a098a90 0x55f84a09ae80 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6242fcfd2ed4452b server_cookie=cc29ee1d78fb4ef0 in_seq=0 out_seq=0 2024-04-21T23:17:30.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.821+0000 7f3f32ffd640 10 client.17586 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:17:30.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.822+0000 7f3f32ffd640 1 -- 172.21.15.44:0/370886579 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f849f886b0 con 0x55f84a098a90 2024-04-21T23:17:30.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.822+0000 7f3f32ffd640 10 client.17586 handle_command_reply: tid=0 2024-04-21T23:17:30.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.822+0000 7f3f5c818740 1 client.17586 shutdown 2024-04-21T23:17:30.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.822+0000 7f3f5c818740 1 -- 172.21.15.44:0/370886579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a017a40 msgr2=0x55f84a029400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.822+0000 7f3f5c818740 1 --2- 172.21.15.44:0/370886579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a017a40 0x55f84a029400 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f3f44005940 tx=0x7f3f4402bc30 comp rx=0 tx=0).stop 2024-04-21T23:17:30.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.822+0000 7f3f5c818740 1 -- 172.21.15.44:0/370886579 shutdown_connections 2024-04-21T23:17:30.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.822+0000 7f3f5c818740 1 --2- 172.21.15.44:0/370886579 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55f84a098a90 0x55f84a09ae80 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.822+0000 7f3f5c818740 1 --2- 172.21.15.44:0/370886579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f84a018720 0x55f84a026000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.822+0000 7f3f5c818740 1 --2- 172.21.15.44:0/370886579 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55f84a020db0 0x55f84a026540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.822+0000 7f3f32ffd640 0 client.17586 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:17:30.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.822+0000 7f3f5c818740 1 --2- 172.21.15.44:0/370886579 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55f84a017a40 0x55f84a029400 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.822+0000 7f3f5c818740 1 -- 172.21.15.44:0/370886579 >> 172.21.15.44:0/370886579 conn(0x55f849fde4f0 msgr2=0x55f84a0114f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:30.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.823+0000 7f3f5c818740 1 -- 172.21.15.44:0/370886579 shutdown_connections 2024-04-21T23:17:30.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.823+0000 7f3f5c818740 1 -- 172.21.15.44:0/370886579 wait complete. 2024-04-21T23:17:30.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.823+0000 7f3f5c818740 20 client.17586 trim_cache size 0 max 16384 2024-04-21T23:17:30.832 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:17:30.832 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-21T23:17:30.832 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "813001d8-c3ad-4078-8ddf-b26d8e3cc826", 2024-04-21T23:17:30.832 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-21T23:17:30.832 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:17:30.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.823+0000 7f3f59b5c640 1 -- 172.21.15.44:0/3311276109 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3f3c0a2d00 msgr2=0x7f3f3c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.823+0000 7f3f59b5c640 1 --2- 172.21.15.44:0/3311276109 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3f3c0a2d00 0x7f3f3c0a51c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c009fd0 tx=0x7f3f4c009600 comp rx=0 tx=0).stop 2024-04-21T23:17:30.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.823+0000 7f3f59b5c640 1 -- 172.21.15.44:0/3311276109 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f541029b0 msgr2=0x7f3f541982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.823+0000 7f3f59b5c640 1 --2- 172.21.15.44:0/3311276109 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f541029b0 0x7f3f541982d0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f3f44006010 tx=0x7f3f4402cc70 comp rx=0 tx=0).stop 2024-04-21T23:17:30.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.823+0000 7f3f59b5c640 1 -- 172.21.15.44:0/3311276109 shutdown_connections 2024-04-21T23:17:30.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.823+0000 7f3f59b5c640 1 --2- 172.21.15.44:0/3311276109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f5419cb90 0x7f3f5419d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.823+0000 7f3f59b5c640 1 --2- 172.21.15.44:0/3311276109 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3f3c0a2d00 0x7f3f3c0a51c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.823+0000 7f3f59b5c640 1 --2- 172.21.15.44:0/3311276109 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f5410bec0 0x7f3f54198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.823+0000 7f3f59b5c640 1 --2- 172.21.15.44:0/3311276109 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f541029b0 0x7f3f541982d0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.823+0000 7f3f59b5c640 1 -- 172.21.15.44:0/3311276109 >> 172.21.15.44:0/3311276109 conn(0x7f3f54077760 msgr2=0x7f3f540feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:30.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.823+0000 7f3f59b5c640 1 -- 172.21.15.44:0/3311276109 shutdown_connections 2024-04-21T23:17:30.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.823+0000 7f3f59b5c640 1 -- 172.21.15.44:0/3311276109 wait complete. 2024-04-21T23:17:30.835 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:17:30.836 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:17:30.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.979+0000 7f60d7e5f640 1 -- 172.21.15.44:0/4229207954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60d0074400 msgr2=0x7f60d01121d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.979+0000 7f60d7e5f640 1 --2- 172.21.15.44:0/4229207954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60d0074400 0x7f60d01121d0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f60c0009900 tx=0x7f60c002bcd0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.979+0000 7f60d7e5f640 1 -- 172.21.15.44:0/4229207954 shutdown_connections 2024-04-21T23:17:30.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.979+0000 7f60d7e5f640 1 --2- 172.21.15.44:0/4229207954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60d0112710 0x7f60d0114b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.979+0000 7f60d7e5f640 1 --2- 172.21.15.44:0/4229207954 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60d0074400 0x7f60d01121d0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.979+0000 7f60d7e5f640 1 --2- 172.21.15.44:0/4229207954 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60d0073ae0 0x7f60d0073ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.979+0000 7f60d7e5f640 1 -- 172.21.15.44:0/4229207954 >> 172.21.15.44:0/4229207954 conn(0x7f60d00fd1f0 msgr2=0x7f60d00ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:30.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.979+0000 7f60d7e5f640 1 -- 172.21.15.44:0/4229207954 shutdown_connections 2024-04-21T23:17:30.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.979+0000 7f60d7e5f640 1 -- 172.21.15.44:0/4229207954 wait complete. 2024-04-21T23:17:30.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.979+0000 7f60d7e5f640 1 Processor -- start 2024-04-21T23:17:30.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.979+0000 7f60d7e5f640 1 -- start start 2024-04-21T23:17:30.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60d7e5f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60d0073ae0 0x7f60d019e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60d7e5f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60d0074400 0x7f60d019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60d7e5f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60d0112710 0x7f60d01a2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60d7e5f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f60d01180f0 con 0x7f60d0073ae0 2024-04-21T23:17:30.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60d7e5f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f60d0117f70 con 0x7f60d0074400 2024-04-21T23:17:30.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60d7e5f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f60d0118270 con 0x7f60d0112710 2024-04-21T23:17:30.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60d5bfb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60d0073ae0 0x7f60d019e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60d5bfb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60d0073ae0 0x7f60d019e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60190/0 (socket says 172.21.15.44:60190) 2024-04-21T23:17:30.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60d5bfb640 1 -- 172.21.15.44:0/3007140082 learned_addr learned my addr 172.21.15.44:0/3007140082 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:30.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60d63fc640 1 --2- 172.21.15.44:0/3007140082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60d0112710 0x7f60d01a2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60d53fa640 1 --2- 172.21.15.44:0/3007140082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60d0074400 0x7f60d019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60d5bfb640 1 -- 172.21.15.44:0/3007140082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60d0112710 msgr2=0x7f60d01a2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60d5bfb640 1 --2- 172.21.15.44:0/3007140082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60d0112710 0x7f60d01a2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60d5bfb640 1 -- 172.21.15.44:0/3007140082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60d0074400 msgr2=0x7f60d019eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60d5bfb640 1 --2- 172.21.15.44:0/3007140082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60d0074400 0x7f60d019eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60d5bfb640 1 -- 172.21.15.44:0/3007140082 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d01a3600 con 0x7f60d0073ae0 2024-04-21T23:17:30.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60d63fc640 1 --2- 172.21.15.44:0/3007140082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60d0112710 0x7f60d01a2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:17:30.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60d5bfb640 1 --2- 172.21.15.44:0/3007140082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60d0073ae0 0x7f60d019e610 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00cce0 tx=0x7f60cc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:30.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60c6ffd640 1 -- 172.21.15.44:0/3007140082 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc00f650 con 0x7f60d0073ae0 2024-04-21T23:17:30.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60c6ffd640 1 -- 172.21.15.44:0/3007140082 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60cc00fcd0 con 0x7f60d0073ae0 2024-04-21T23:17:30.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60d7e5f640 1 -- 172.21.15.44:0/3007140082 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60d01a38f0 con 0x7f60d0073ae0 2024-04-21T23:17:30.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60c6ffd640 1 -- 172.21.15.44:0/3007140082 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc018c70 con 0x7f60d0073ae0 2024-04-21T23:17:30.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.980+0000 7f60d7e5f640 1 -- 172.21.15.44:0/3007140082 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60d01ab180 con 0x7f60d0073ae0 2024-04-21T23:17:30.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.983+0000 7f60c6ffd640 1 -- 172.21.15.44:0/3007140082 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f60cc00f7f0 con 0x7f60d0073ae0 2024-04-21T23:17:30.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.983+0000 7f60daa42740 1 Processor -- start 2024-04-21T23:17:30.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60daa42740 1 -- start start 2024-04-21T23:17:30.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60daa42740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x560666542310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60daa42740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560666542850 0x56066654aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60daa42740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56066654b430 0x56066654d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60daa42740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5606664a6560 con 0x560666541f30 2024-04-21T23:17:30.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60daa42740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x56066644e730 con 0x56066654b430 2024-04-21T23:17:30.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60daa42740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5606664a2520 con 0x560666542850 2024-04-21T23:17:30.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60d63fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x560666542310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60d63fc640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x560666542310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60192/0 (socket says 172.21.15.44:60192) 2024-04-21T23:17:30.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60d63fc640 1 -- 172.21.15.44:0/517341364 learned_addr learned my addr 172.21.15.44:0/517341364 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:30.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60c6ffd640 1 --2- 172.21.15.44:0/3007140082 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f60bc0a2d00 0x7f60bc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60d53fa640 1 --2- 172.21.15.44:0/517341364 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56066654b430 0x56066654d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60d5bfb640 1 --2- 172.21.15.44:0/517341364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560666542850 0x56066654aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60d63fc640 1 -- 172.21.15.44:0/517341364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560666542850 msgr2=0x56066654aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:30.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60d63fc640 1 --2- 172.21.15.44:0/517341364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560666542850 0x56066654aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60d63fc640 1 -- 172.21.15.44:0/517341364 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56066654b430 msgr2=0x56066654d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:30.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60d63fc640 1 --2- 172.21.15.44:0/517341364 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56066654b430 0x56066654d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60d63fc640 1 -- 172.21.15.44:0/517341364 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606662847d0 con 0x560666541f30 2024-04-21T23:17:30.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60d53fa640 1 --2- 172.21.15.44:0/517341364 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56066654b430 0x56066654d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:17:30.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60d5bfb640 1 --2- 172.21.15.44:0/517341364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560666542850 0x56066654aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:30.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.984+0000 7f60c6ffd640 1 -- 172.21.15.44:0/3007140082 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7f60cc0cfd80 con 0x7f60d0073ae0 2024-04-21T23:17:30.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.985+0000 7f60d63fc640 1 --2- 172.21.15.44:0/3007140082 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f60bc0a2d00 0x7f60bc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.985+0000 7f60d63fc640 1 --2- 172.21.15.44:0/517341364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x560666542310 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f60c800bdc0 tx=0x7f60c800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5abeef9feffc9ce server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:30.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.985+0000 7f60bbfff640 1 -- 172.21.15.44:0/517341364 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c801c580 con 0x560666541f30 2024-04-21T23:17:30.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.985+0000 7f60bbfff640 1 -- 172.21.15.44:0/517341364 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60c801cc00 con 0x560666541f30 2024-04-21T23:17:30.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.985+0000 7f60bbfff640 1 -- 172.21.15.44:0/517341364 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c801bb50 con 0x560666541f30 2024-04-21T23:17:30.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.986+0000 7f60daa42740 1 -- 172.21.15.44:0/517341364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 msgr2=0x560666542310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:30.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.986+0000 7f60d63fc640 1 --2- 172.21.15.44:0/3007140082 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f60bc0a2d00 0x7f60bc0a51c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f60c8020000 tx=0x7f60c801d000 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:30.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.986+0000 7f60daa42740 1 --2- 172.21.15.44:0/517341364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x560666542310 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f60c800bdc0 tx=0x7f60c800f730 comp rx=0 tx=0).stop 2024-04-21T23:17:30.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.986+0000 7f60daa42740 1 -- 172.21.15.44:0/517341364 shutdown_connections 2024-04-21T23:17:30.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.986+0000 7f60daa42740 1 --2- 172.21.15.44:0/517341364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560666542850 0x56066654aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.986+0000 7f60daa42740 1 --2- 172.21.15.44:0/517341364 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56066654b430 0x56066654d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.986+0000 7f60daa42740 1 --2- 172.21.15.44:0/517341364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x560666542310 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.986+0000 7f60daa42740 1 -- 172.21.15.44:0/517341364 >> 172.21.15.44:0/517341364 conn(0x56066653c660 msgr2=0x56066653ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:30.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.986+0000 7f60daa42740 1 -- 172.21.15.44:0/517341364 shutdown_connections 2024-04-21T23:17:30.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.986+0000 7f60daa42740 1 -- 172.21.15.44:0/517341364 wait complete. 2024-04-21T23:17:30.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.986+0000 7f60daa42740 1 -- start start 2024-04-21T23:17:30.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.986+0000 7f60daa42740 1 Processor -- start 2024-04-21T23:17:30.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60daa42740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x560666548490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60daa42740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560666542850 0x5606665489d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60daa42740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56066654b430 0x560666548f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60daa42740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56066654e220 con 0x560666541f30 2024-04-21T23:17:30.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60daa42740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x560666461e00 con 0x56066654b430 2024-04-21T23:17:30.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60daa42740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56066654e3a0 con 0x560666542850 2024-04-21T23:17:30.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60daa42740 10 client.0 _ll_register_callbacks cb 0x56066649ceb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:17:30.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60d5bfb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x560666548490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60d5bfb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x560666548490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60206/0 (socket says 172.21.15.44:60206) 2024-04-21T23:17:30.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60d5bfb640 1 -- 172.21.15.44:0/969725132 learned_addr learned my addr 172.21.15.44:0/969725132 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:30.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60d53fa640 1 --2- 172.21.15.44:0/969725132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560666542850 0x5606665489d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60d63fc640 1 --2- 172.21.15.44:0/969725132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56066654b430 0x560666548f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:30.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60d5bfb640 1 -- 172.21.15.44:0/969725132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560666542850 msgr2=0x5606665489d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:30.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60d5bfb640 1 --2- 172.21.15.44:0/969725132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560666542850 0x5606665489d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60d5bfb640 1 -- 172.21.15.44:0/969725132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56066654b430 msgr2=0x560666548f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:30.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60d5bfb640 1 --2- 172.21.15.44:0/969725132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56066654b430 0x560666548f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:30.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60d5bfb640 1 -- 172.21.15.44:0/969725132 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606661ba3f0 con 0x560666541f30 2024-04-21T23:17:30.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60d53fa640 1 --2- 172.21.15.44:0/969725132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560666542850 0x5606665489d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:17:30.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60d5bfb640 1 --2- 172.21.15.44:0/969725132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x560666548490 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f60cc098310 tx=0x7f60cc06ade0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=122820174805a192 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:30.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:17:30.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60b97fa640 1 -- 172.21.15.44:0/969725132 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc09b810 con 0x560666541f30 2024-04-21T23:17:30.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60b97fa640 1 -- 172.21.15.44:0/969725132 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7f60cc0a4690 con 0x560666541f30 2024-04-21T23:17:30.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60daa42740 1 -- 172.21.15.44:0/969725132 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60c801cc00 con 0x560666541f30 2024-04-21T23:17:30.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60b97fa640 1 -- 172.21.15.44:0/969725132 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f60cc0671d0 con 0x560666541f30 2024-04-21T23:17:30.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60daa42740 10 client.17598 fetch_fsmap learned FSMap version 31 2024-04-21T23:17:30.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.987+0000 7f60daa42740 1 -- 172.21.15.44:0/969725132 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56066617a0f0 con 0x560666541f30 2024-04-21T23:17:30.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.988+0000 7f60b97fa640 1 -- 172.21.15.44:0/969725132 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60c801cc00 con 0x560666541f30 2024-04-21T23:17:30.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.988+0000 7f60b97fa640 1 -- 172.21.15.44:0/969725132 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f60cc09b300 con 0x560666541f30 2024-04-21T23:17:30.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.988+0000 7f60daa42740 10 client.17598 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:17:30.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.988+0000 7f60daa42740 10 client.17598 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:17:30.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.988+0000 7f60daa42740 20 client.17598 populate_metadata read hostname 'smithi044' 2024-04-21T23:17:30.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.988+0000 7f60daa42740 1 --2- 172.21.15.44:0/969725132 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56066655b7e0 0x56066655dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:30.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.988+0000 7f60daa42740 4 client.17598 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:17:30.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.988+0000 7f60daa42740 1 -- 172.21.15.44:0/969725132 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606664b3aa0 con 0x56066655b7e0 2024-04-21T23:17:30.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.988+0000 7f60d53fa640 1 --2- 172.21.15.44:0/969725132 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56066655b7e0 0x56066655dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:31.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.989+0000 7f60d53fa640 1 --2- 172.21.15.44:0/969725132 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56066655b7e0 0x56066655dbd0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=199f812d8735fb96 server_cookie=77f37e04461e8e in_seq=0 out_seq=0 2024-04-21T23:17:31.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.989+0000 7f60b97fa640 10 client.17598 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:17:31.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.989+0000 7f60b97fa640 1 -- 172.21.15.44:0/969725132 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5606664b3aa0 con 0x56066655b7e0 2024-04-21T23:17:31.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.989+0000 7f60b97fa640 10 client.17598 handle_command_reply: tid=0 2024-04-21T23:17:31.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.989+0000 7f60daa42740 1 client.17598 shutdown 2024-04-21T23:17:31.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.990+0000 7f60daa42740 1 -- 172.21.15.44:0/969725132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 msgr2=0x560666548490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:31.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.990+0000 7f60daa42740 1 --2- 172.21.15.44:0/969725132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x560666548490 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f60cc098310 tx=0x7f60cc06ade0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.990+0000 7f60daa42740 1 -- 172.21.15.44:0/969725132 shutdown_connections 2024-04-21T23:17:31.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.990+0000 7f60daa42740 1 --2- 172.21.15.44:0/969725132 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56066655b7e0 0x56066655dbd0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.990+0000 7f60daa42740 1 --2- 172.21.15.44:0/969725132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560666542850 0x5606665489d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.990+0000 7f60daa42740 1 --2- 172.21.15.44:0/969725132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56066654b430 0x560666548f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.990+0000 7f60daa42740 1 --2- 172.21.15.44:0/969725132 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x560666548490 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.990+0000 7f60b97fa640 0 client.17598 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:17:31.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.990+0000 7f60daa42740 1 -- 172.21.15.44:0/969725132 >> 172.21.15.44:0/969725132 conn(0x56066653c660 msgr2=0x56066653e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:31.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.990+0000 7f60daa42740 1 -- 172.21.15.44:0/969725132 shutdown_connections 2024-04-21T23:17:31.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.990+0000 7f60daa42740 1 -- 172.21.15.44:0/969725132 wait complete. 2024-04-21T23:17:31.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:30.990+0000 7f60daa42740 20 client.17598 trim_cache size 0 max 16384 2024-04-21T23:17:31.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.008+0000 7f60daa42740 1 Processor -- start 2024-04-21T23:17:31.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.008+0000 7f60daa42740 1 -- start start 2024-04-21T23:17:31.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.008+0000 7f60daa42740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x56066654abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:31.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.008+0000 7f60daa42740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560666542850 0x560666557110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:31.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.008+0000 7f60daa42740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56066654b430 0x56066653b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:31.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.008+0000 7f60daa42740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f60d01b33d0 con 0x560666541f30 2024-04-21T23:17:31.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.008+0000 7f60daa42740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f60d01b5c30 con 0x560666542850 2024-04-21T23:17:31.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.008+0000 7f60daa42740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f60d01b3a20 con 0x56066654b430 2024-04-21T23:17:31.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.008+0000 7f60d5bfb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x56066654abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:31.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.008+0000 7f60d5bfb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x56066654abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60208/0 (socket says 172.21.15.44:60208) 2024-04-21T23:17:31.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.008+0000 7f60d5bfb640 1 -- 172.21.15.44:0/2254591126 learned_addr learned my addr 172.21.15.44:0/2254591126 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:31.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.008+0000 7f60d63fc640 1 --2- 172.21.15.44:0/2254591126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56066654b430 0x56066653b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:31.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.008+0000 7f60d53fa640 1 --2- 172.21.15.44:0/2254591126 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560666542850 0x560666557110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:31.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.008+0000 7f60d5bfb640 1 -- 172.21.15.44:0/2254591126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56066654b430 msgr2=0x56066653b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:31.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.008+0000 7f60d5bfb640 1 --2- 172.21.15.44:0/2254591126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56066654b430 0x56066653b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.008+0000 7f60d5bfb640 1 -- 172.21.15.44:0/2254591126 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560666542850 msgr2=0x560666557110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:31.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.008+0000 7f60d5bfb640 1 --2- 172.21.15.44:0/2254591126 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560666542850 0x560666557110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.008+0000 7f60d5bfb640 1 -- 172.21.15.44:0/2254591126 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606665469e0 con 0x560666541f30 2024-04-21T23:17:31.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.008+0000 7f60d63fc640 1 --2- 172.21.15.44:0/2254591126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56066654b430 0x56066653b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:17:31.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.008+0000 7f60d5bfb640 1 --2- 172.21.15.44:0/2254591126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x56066654abc0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f60cc09a160 tx=0x7f60cc00fee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ebc2292e813fb24 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:31.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.009+0000 7f60bb7fe640 1 -- 172.21.15.44:0/2254591126 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc09b810 con 0x560666541f30 2024-04-21T23:17:31.015 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.009+0000 7f60bb7fe640 1 -- 172.21.15.44:0/2254591126 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60cc09c5b0 con 0x560666541f30 2024-04-21T23:17:31.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.009+0000 7f60bb7fe640 1 -- 172.21.15.44:0/2254591126 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc0a4690 con 0x560666541f30 2024-04-21T23:17:31.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.009+0000 7f60daa42740 1 -- 172.21.15.44:0/2254591126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 msgr2=0x56066654abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:31.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.009+0000 7f60daa42740 1 --2- 172.21.15.44:0/2254591126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x56066654abc0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f60cc09a160 tx=0x7f60cc00fee0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.009+0000 7f60daa42740 1 -- 172.21.15.44:0/2254591126 shutdown_connections 2024-04-21T23:17:31.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.009+0000 7f60daa42740 1 --2- 172.21.15.44:0/2254591126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56066654b430 0x56066653b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.009+0000 7f60daa42740 1 --2- 172.21.15.44:0/2254591126 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560666542850 0x560666557110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.009+0000 7f60daa42740 1 --2- 172.21.15.44:0/2254591126 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x56066654abc0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.016 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.009+0000 7f60daa42740 1 -- 172.21.15.44:0/2254591126 >> 172.21.15.44:0/2254591126 conn(0x56066653c660 msgr2=0x56066653ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:31.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.009+0000 7f60daa42740 1 -- 172.21.15.44:0/2254591126 shutdown_connections 2024-04-21T23:17:31.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.009+0000 7f60daa42740 1 -- 172.21.15.44:0/2254591126 wait complete. 2024-04-21T23:17:31.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.010+0000 7f60daa42740 1 -- start start 2024-04-21T23:17:31.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.010+0000 7f60daa42740 1 Processor -- start 2024-04-21T23:17:31.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.010+0000 7f60daa42740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x560666545310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:31.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.010+0000 7f60daa42740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560666542850 0x560666545850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:31.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.010+0000 7f60daa42740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56066654b430 0x560666545d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:31.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.010+0000 7f60daa42740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f60d01b40f0 con 0x560666541f30 2024-04-21T23:17:31.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.010+0000 7f60daa42740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f60d01b5560 con 0x560666542850 2024-04-21T23:17:31.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.010+0000 7f60daa42740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f60d01b47c0 con 0x56066654b430 2024-04-21T23:17:31.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.010+0000 7f60daa42740 10 client.0 _ll_register_callbacks cb 0x7f60cc00ac50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:17:31.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.010+0000 7f60d5bfb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x560666545310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:31.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.010+0000 7f60d5bfb640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x560666545310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:60218/0 (socket says 172.21.15.44:60218) 2024-04-21T23:17:31.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.010+0000 7f60d5bfb640 1 -- 172.21.15.44:0/1386546071 learned_addr learned my addr 172.21.15.44:0/1386546071 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:17:31.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.010+0000 7f60d53fa640 1 --2- 172.21.15.44:0/1386546071 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560666542850 0x560666545850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:31.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.010+0000 7f60d63fc640 1 --2- 172.21.15.44:0/1386546071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56066654b430 0x560666545d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:31.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.010+0000 7f60d5bfb640 1 -- 172.21.15.44:0/1386546071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56066654b430 msgr2=0x560666545d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:31.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.010+0000 7f60d5bfb640 1 --2- 172.21.15.44:0/1386546071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56066654b430 0x560666545d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.010+0000 7f60d5bfb640 1 -- 172.21.15.44:0/1386546071 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560666542850 msgr2=0x560666545850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:17:31.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.010+0000 7f60d5bfb640 1 --2- 172.21.15.44:0/1386546071 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560666542850 0x560666545850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.010+0000 7f60d5bfb640 1 -- 172.21.15.44:0/1386546071 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606665c0f10 con 0x560666541f30 2024-04-21T23:17:31.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.011+0000 7f60d5bfb640 1 --2- 172.21.15.44:0/1386546071 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x560666545310 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00ccb0 tx=0x7f60cc0a4b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a18f364cff62d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:17:31.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.011+0000 7f60b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:17:31.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.011+0000 7f60b97fa640 1 -- 172.21.15.44:0/1386546071 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc0cd570 con 0x560666541f30 2024-04-21T23:17:31.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.011+0000 7f60b97fa640 1 -- 172.21.15.44:0/1386546071 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7f60cc0a4890 con 0x560666541f30 2024-04-21T23:17:31.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.011+0000 7f60daa42740 1 -- 172.21.15.44:0/1386546071 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60cc09c5b0 con 0x560666541f30 2024-04-21T23:17:31.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.011+0000 7f60b97fa640 1 -- 172.21.15.44:0/1386546071 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f60cc00fae0 con 0x560666541f30 2024-04-21T23:17:31.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.011+0000 7f60daa42740 10 client.17604 fetch_fsmap learned FSMap version 31 2024-04-21T23:17:31.020 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.011+0000 7f60daa42740 1 -- 172.21.15.44:0/1386546071 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5606665c2150 con 0x560666541f30 2024-04-21T23:17:31.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.011+0000 7f60b97fa640 1 -- 172.21.15.44:0/1386546071 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60cc09c5b0 con 0x560666541f30 2024-04-21T23:17:31.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.011+0000 7f60b97fa640 1 -- 172.21.15.44:0/1386546071 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f60cc0ba020 con 0x560666541f30 2024-04-21T23:17:31.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.011+0000 7f60daa42740 10 client.17604 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:17:31.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.011+0000 7f60daa42740 10 client.17604 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:17:31.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.011+0000 7f60daa42740 20 client.17604 populate_metadata read hostname 'smithi044' 2024-04-21T23:17:31.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.011+0000 7f60daa42740 1 --2- 172.21.15.44:0/1386546071 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5606665c2620 0x5606665c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:17:31.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.011+0000 7f60daa42740 4 client.17604 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:17:31.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.011+0000 7f60daa42740 1 -- 172.21.15.44:0/1386546071 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5606664addf0 con 0x5606665c2620 2024-04-21T23:17:31.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.012+0000 7f60d53fa640 1 --2- 172.21.15.44:0/1386546071 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5606665c2620 0x5606665c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:17:31.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.012+0000 7f60b97fa640 10 client.17604 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:17:31.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.012+0000 7f60d53fa640 1 --2- 172.21.15.44:0/1386546071 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5606665c2620 0x5606665c4a10 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d4e5b76b174c586 server_cookie=7d0e86f59544ada2 in_seq=0 out_seq=0 2024-04-21T23:17:31.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.012+0000 7f60b97fa640 1 -- 172.21.15.44:0/1386546071 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5606664addf0 con 0x5606665c2620 2024-04-21T23:17:31.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.012+0000 7f60b97fa640 10 client.17604 handle_command_reply: tid=0 2024-04-21T23:17:31.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.012+0000 7f60daa42740 1 client.17604 shutdown 2024-04-21T23:17:31.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.012+0000 7f60daa42740 1 -- 172.21.15.44:0/1386546071 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 msgr2=0x560666545310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:31.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.012+0000 7f60daa42740 1 --2- 172.21.15.44:0/1386546071 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x560666545310 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00ccb0 tx=0x7f60cc0a4b20 comp rx=0 tx=0).stop 2024-04-21T23:17:31.022 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.012+0000 7f60daa42740 1 -- 172.21.15.44:0/1386546071 shutdown_connections 2024-04-21T23:17:31.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.012+0000 7f60daa42740 1 --2- 172.21.15.44:0/1386546071 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5606665c2620 0x5606665c4a10 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.012+0000 7f60daa42740 1 --2- 172.21.15.44:0/1386546071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56066654b430 0x560666545d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.012+0000 7f60daa42740 1 --2- 172.21.15.44:0/1386546071 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x560666542850 0x560666545850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.012+0000 7f60daa42740 1 --2- 172.21.15.44:0/1386546071 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x560666541f30 0x560666545310 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.012+0000 7f60daa42740 1 -- 172.21.15.44:0/1386546071 >> 172.21.15.44:0/1386546071 conn(0x56066653c660 msgr2=0x560666548ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:31.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.012+0000 7f60b97fa640 0 client.17604 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:17:31.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.012+0000 7f60daa42740 1 -- 172.21.15.44:0/1386546071 shutdown_connections 2024-04-21T23:17:31.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.012+0000 7f60daa42740 1 -- 172.21.15.44:0/1386546071 wait complete. 2024-04-21T23:17:31.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.012+0000 7f60daa42740 20 client.17604 trim_cache size 0 max 16384 2024-04-21T23:17:31.024 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:17:31.024 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:17:31.024 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-21T23:17:31.024 INFO:teuthology.orchestra.run.smithi044.stdout: "813001d8-c3ad-4078-8ddf-b26d8e3cc826": { 2024-04-21T23:17:31.024 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-21T23:17:31.024 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "813001d8-c3ad-4078-8ddf-b26d8e3cc826", 2024-04-21T23:17:31.024 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-04-21T23:17:31.025 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:17:31.025 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:17:31.025 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:17:31.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.013+0000 7f60d7e5f640 1 -- 172.21.15.44:0/3007140082 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f60bc0a2d00 msgr2=0x7f60bc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:31.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.013+0000 7f60d7e5f640 1 --2- 172.21.15.44:0/3007140082 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f60bc0a2d00 0x7f60bc0a51c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f60c8020000 tx=0x7f60c801d000 comp rx=0 tx=0).stop 2024-04-21T23:17:31.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.013+0000 7f60d7e5f640 1 -- 172.21.15.44:0/3007140082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60d0073ae0 msgr2=0x7f60d019e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:17:31.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.013+0000 7f60d7e5f640 1 --2- 172.21.15.44:0/3007140082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60d0073ae0 0x7f60d019e610 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00cce0 tx=0x7f60cc00a620 comp rx=0 tx=0).stop 2024-04-21T23:17:31.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.013+0000 7f60d7e5f640 1 -- 172.21.15.44:0/3007140082 shutdown_connections 2024-04-21T23:17:31.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.013+0000 7f60d7e5f640 1 --2- 172.21.15.44:0/3007140082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60d0112710 0x7f60d01a2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.013+0000 7f60d7e5f640 1 --2- 172.21.15.44:0/3007140082 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f60bc0a2d00 0x7f60bc0a51c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.013+0000 7f60d7e5f640 1 --2- 172.21.15.44:0/3007140082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f60d0074400 0x7f60d019eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.013+0000 7f60d7e5f640 1 --2- 172.21.15.44:0/3007140082 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f60d0073ae0 0x7f60d019e610 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:17:31.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.013+0000 7f60d7e5f640 1 -- 172.21.15.44:0/3007140082 >> 172.21.15.44:0/3007140082 conn(0x7f60d00fd1f0 msgr2=0x7f60d00fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:17:31.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.013+0000 7f60d7e5f640 1 -- 172.21.15.44:0/3007140082 shutdown_connections 2024-04-21T23:17:31.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:17:31.013+0000 7f60d7e5f640 1 -- 172.21.15.44:0/3007140082 wait complete. 2024-04-21T23:17:31.028 INFO:tasks.cephfs.filesystem:scrub status for tag:813001d8-c3ad-4078-8ddf-b26d8e3cc826 - {'path': '/', 'tag': '813001d8-c3ad-4078-8ddf-b26d8e3cc826', 'options': 'recursive,force'} 2024-04-21T23:17:31.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:31 smithi044 ceph-mon[27934]: scrub queued for path: / 2024-04-21T23:17:31.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:31 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:17:31.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:31 smithi044 ceph-mon[27934]: scrub summary: active paths [/] 2024-04-21T23:17:31.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:31 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:17:31.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:31 smithi055 ceph-mon[31117]: scrub queued for path: / 2024-04-21T23:17:31.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:31 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:17:31.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:31 smithi055 ceph-mon[31117]: scrub summary: active paths [/] 2024-04-21T23:17:31.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:31 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:17:31.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:31 smithi069 ceph-mon[31816]: scrub queued for path: / 2024-04-21T23:17:31.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:31 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:17:31.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:31 smithi069 ceph-mon[31816]: scrub summary: active paths [/] 2024-04-21T23:17:31.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:31 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:17:32.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:32 smithi044 ceph-mon[27934]: pgmap v274: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 190 KiB/s wr, 80 op/s 2024-04-21T23:17:32.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:32 smithi044 ceph-mon[27934]: scrub summary: idle 2024-04-21T23:17:32.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:32 smithi069 ceph-mon[31816]: pgmap v274: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 190 KiB/s wr, 80 op/s 2024-04-21T23:17:32.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:32 smithi069 ceph-mon[31816]: scrub summary: idle 2024-04-21T23:17:32.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:32 smithi055 ceph-mon[31117]: pgmap v274: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 190 KiB/s wr, 80 op/s 2024-04-21T23:17:32.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:32 smithi055 ceph-mon[31117]: scrub summary: idle 2024-04-21T23:17:34.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:34 smithi044 ceph-mon[27934]: pgmap v275: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 194 KiB/s wr, 88 op/s 2024-04-21T23:17:34.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:34 smithi055 ceph-mon[31117]: pgmap v275: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 194 KiB/s wr, 88 op/s 2024-04-21T23:17:34.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:34 smithi069 ceph-mon[31816]: pgmap v275: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 194 KiB/s wr, 88 op/s 2024-04-21T23:17:36.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:36 smithi044 ceph-mon[27934]: pgmap v276: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 26 KiB/s wr, 9 op/s 2024-04-21T23:17:36.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:36 smithi055 ceph-mon[31117]: pgmap v276: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 26 KiB/s wr, 9 op/s 2024-04-21T23:17:36.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:36 smithi069 ceph-mon[31816]: pgmap v276: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 26 KiB/s wr, 9 op/s 2024-04-21T23:17:38.516 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:38 smithi069 ceph-mon[31816]: pgmap v277: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 26 KiB/s wr, 10 op/s 2024-04-21T23:17:38.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:38 smithi044 ceph-mon[27934]: pgmap v277: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 26 KiB/s wr, 10 op/s 2024-04-21T23:17:38.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:38 smithi055 ceph-mon[31117]: pgmap v277: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 26 KiB/s wr, 10 op/s 2024-04-21T23:17:40.516 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:40 smithi055 ceph-mon[31117]: pgmap v278: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 26 KiB/s wr, 9 op/s 2024-04-21T23:17:40.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:40 smithi044 ceph-mon[27934]: pgmap v278: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 26 KiB/s wr, 9 op/s 2024-04-21T23:17:40.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:40 smithi069 ceph-mon[31816]: pgmap v278: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 26 KiB/s wr, 9 op/s 2024-04-21T23:17:42.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:42 smithi044 ceph-mon[27934]: pgmap v279: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.4 KiB/s wr, 8 op/s 2024-04-21T23:17:42.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:42 smithi055 ceph-mon[31117]: pgmap v279: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.4 KiB/s wr, 8 op/s 2024-04-21T23:17:42.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:42 smithi069 ceph-mon[31816]: pgmap v279: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.4 KiB/s wr, 8 op/s 2024-04-21T23:17:44.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:44 smithi044 ceph-mon[27934]: pgmap v280: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 3.4 KiB/s wr, 9 op/s 2024-04-21T23:17:44.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:44 smithi069 ceph-mon[31816]: pgmap v280: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 3.4 KiB/s wr, 9 op/s 2024-04-21T23:17:44.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:44 smithi055 ceph-mon[31117]: pgmap v280: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 3.4 KiB/s wr, 9 op/s 2024-04-21T23:17:46.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:46 smithi044 ceph-mon[27934]: pgmap v281: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:17:46.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:46 smithi055 ceph-mon[31117]: pgmap v281: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:17:46.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:46 smithi069 ceph-mon[31816]: pgmap v281: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:17:48.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:48 smithi044 ceph-mon[27934]: pgmap v282: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:17:48.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:48 smithi055 ceph-mon[31117]: pgmap v282: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:17:48.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:48 smithi069 ceph-mon[31816]: pgmap v282: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:17:50.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:50 smithi044 ceph-mon[27934]: pgmap v283: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-04-21T23:17:50.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:50 smithi069 ceph-mon[31816]: pgmap v283: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-04-21T23:17:50.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:50 smithi055 ceph-mon[31117]: pgmap v283: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-04-21T23:17:52.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:52 smithi044 ceph-mon[27934]: pgmap v284: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:17:52.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:52 smithi055 ceph-mon[31117]: pgmap v284: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:17:52.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:52 smithi069 ceph-mon[31816]: pgmap v284: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:17:54.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:54 smithi044 ceph-mon[27934]: pgmap v285: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:17:54.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:54 smithi055 ceph-mon[31117]: pgmap v285: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:17:54.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:54 smithi069 ceph-mon[31816]: pgmap v285: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:17:56.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:56 smithi044 ceph-mon[27934]: pgmap v286: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:17:56.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:56 smithi055 ceph-mon[31117]: pgmap v286: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:17:56.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:56 smithi069 ceph-mon[31816]: pgmap v286: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:17:58.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:17:58 smithi044 ceph-mon[27934]: pgmap v287: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:17:58.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:17:58 smithi055 ceph-mon[31117]: pgmap v287: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:17:58.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:17:58 smithi069 ceph-mon[31816]: pgmap v287: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:18:00.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:00 smithi044 ceph-mon[27934]: pgmap v288: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 2 op/s 2024-04-21T23:18:00.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:00 smithi069 ceph-mon[31816]: pgmap v288: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 2 op/s 2024-04-21T23:18:00.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:00 smithi055 ceph-mon[31117]: pgmap v288: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 2 op/s 2024-04-21T23:18:01.029 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:18:01.029 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:18:01.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.147+0000 7f2180d4b640 1 -- 172.21.15.44:0/1742555044 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f217c103210 msgr2=0x7f217c103670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.147+0000 7f2180d4b640 1 --2- 172.21.15.44:0/1742555044 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f217c103210 0x7f217c103670 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f2170009900 tx=0x7f217002bd10 comp rx=0 tx=0).stop 2024-04-21T23:18:01.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.147+0000 7f2180d4b640 1 -- 172.21.15.44:0/1742555044 shutdown_connections 2024-04-21T23:18:01.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.147+0000 7f2180d4b640 1 --2- 172.21.15.44:0/1742555044 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f217c103bb0 0x7f217c10e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.147+0000 7f2180d4b640 1 --2- 172.21.15.44:0/1742555044 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f217c103210 0x7f217c103670 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.147+0000 7f2180d4b640 1 --2- 172.21.15.44:0/1742555044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f217c101ad0 0x7f217c101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.147+0000 7f2180d4b640 1 -- 172.21.15.44:0/1742555044 >> 172.21.15.44:0/1742555044 conn(0x7f217c0fd200 msgr2=0x7f217c0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:01.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.147+0000 7f2180d4b640 1 -- 172.21.15.44:0/1742555044 shutdown_connections 2024-04-21T23:18:01.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.147+0000 7f2180d4b640 1 -- 172.21.15.44:0/1742555044 wait complete. 2024-04-21T23:18:01.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.148+0000 7f2180d4b640 1 Processor -- start 2024-04-21T23:18:01.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.148+0000 7f2180d4b640 1 -- start start 2024-04-21T23:18:01.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.148+0000 7f2180d4b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f217c101ad0 0x7f217c198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.148+0000 7f2180d4b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f217c103210 0x7f217c198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.148+0000 7f2180d4b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f217c103bb0 0x7f217c19c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.148+0000 7f2180d4b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f217c111c40 con 0x7f217c103bb0 2024-04-21T23:18:01.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.149+0000 7f2180d4b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f217c111ac0 con 0x7f217c103210 2024-04-21T23:18:01.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.149+0000 7f2180d4b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f217c111dc0 con 0x7f217c101ad0 2024-04-21T23:18:01.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.149+0000 7f217ad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f217c103bb0 0x7f217c19c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.149+0000 7f217ad9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f217c103bb0 0x7f217c19c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55164/0 (socket says 172.21.15.44:55164) 2024-04-21T23:18:01.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.149+0000 7f217ad9d640 1 -- 172.21.15.44:0/2046529714 learned_addr learned my addr 172.21.15.44:0/2046529714 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:01.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.149+0000 7f2179d9b640 1 --2- 172.21.15.44:0/2046529714 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f217c103210 0x7f217c198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.149+0000 7f217a59c640 1 --2- 172.21.15.44:0/2046529714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f217c101ad0 0x7f217c198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.149+0000 7f217ad9d640 1 -- 172.21.15.44:0/2046529714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f217c101ad0 msgr2=0x7f217c198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.149+0000 7f217ad9d640 1 --2- 172.21.15.44:0/2046529714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f217c101ad0 0x7f217c198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.149+0000 7f217ad9d640 1 -- 172.21.15.44:0/2046529714 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f217c103210 msgr2=0x7f217c198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.149+0000 7f217ad9d640 1 --2- 172.21.15.44:0/2046529714 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f217c103210 0x7f217c198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.149+0000 7f217ad9d640 1 -- 172.21.15.44:0/2046529714 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f217c19d0c0 con 0x7f217c103bb0 2024-04-21T23:18:01.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.149+0000 7f217a59c640 1 --2- 172.21.15.44:0/2046529714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f217c101ad0 0x7f217c198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:01.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.149+0000 7f217ad9d640 1 --2- 172.21.15.44:0/2046529714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f217c103bb0 0x7f217c19c990 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f217400b970 tx=0x7f217400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:01.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.149+0000 7f216b7fe640 1 -- 172.21.15.44:0/2046529714 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2174019070 con 0x7f217c103bb0 2024-04-21T23:18:01.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.149+0000 7f2180d4b640 1 -- 172.21.15.44:0/2046529714 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f217c19d3b0 con 0x7f217c103bb0 2024-04-21T23:18:01.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.149+0000 7f216b7fe640 1 -- 172.21.15.44:0/2046529714 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f217400cb40 con 0x7f217c103bb0 2024-04-21T23:18:01.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.149+0000 7f216b7fe640 1 -- 172.21.15.44:0/2046529714 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2174007650 con 0x7f217c103bb0 2024-04-21T23:18:01.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.149+0000 7f2180d4b640 1 -- 172.21.15.44:0/2046529714 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f217c1a4c40 con 0x7f217c103bb0 2024-04-21T23:18:01.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.152+0000 7f216b7fe640 1 -- 172.21.15.44:0/2046529714 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f217401c070 con 0x7f217c103bb0 2024-04-21T23:18:01.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.152+0000 7f2183a0b740 1 Processor -- start 2024-04-21T23:18:01.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.152+0000 7f2183a0b740 1 -- start start 2024-04-21T23:18:01.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.152+0000 7f2183a0b740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca131310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.152+0000 7f2183a0b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca131850 0x5581ca139ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.152+0000 7f216b7fe640 1 --2- 172.21.15.44:0/2046529714 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f21640a2d00 0x7f21640a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.152+0000 7f2183a0b740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca13a430 0x5581ca13c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.152+0000 7f2183a0b740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5581ca095560 con 0x5581ca130f30 2024-04-21T23:18:01.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.152+0000 7f2183a0b740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5581ca03d730 con 0x5581ca13a430 2024-04-21T23:18:01.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.152+0000 7f2183a0b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5581ca091520 con 0x5581ca131850 2024-04-21T23:18:01.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.153+0000 7f2179d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca131310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.153+0000 7f217ad9d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca13a430 0x5581ca13c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.153+0000 7f2179d9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca131310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55176/0 (socket says 172.21.15.44:55176) 2024-04-21T23:18:01.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.153+0000 7f2179d9b640 1 -- 172.21.15.44:0/1011134413 learned_addr learned my addr 172.21.15.44:0/1011134413 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:01.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.153+0000 7f217a59c640 1 --2- 172.21.15.44:0/1011134413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca131850 0x5581ca139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.153+0000 7f2179d9b640 1 --2- 172.21.15.44:0/2046529714 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f21640a2d00 0x7f21640a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.153+0000 7f216b7fe640 1 -- 172.21.15.44:0/2046529714 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7f21740ced70 con 0x7f217c103bb0 2024-04-21T23:18:01.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.153+0000 7f2179d9b640 1 -- 172.21.15.44:0/1011134413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca131850 msgr2=0x5581ca139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:01.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.153+0000 7f2179d9b640 1 --2- 172.21.15.44:0/1011134413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca131850 0x5581ca139ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.153+0000 7f2179d9b640 1 -- 172.21.15.44:0/1011134413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca13a430 msgr2=0x5581ca13c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:01.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.153+0000 7f2179d9b640 1 --2- 172.21.15.44:0/1011134413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca13a430 0x5581ca13c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.153+0000 7f2179d9b640 1 -- 172.21.15.44:0/1011134413 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581c9e737d0 con 0x5581ca130f30 2024-04-21T23:18:01.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.153+0000 7f217a59c640 1 --2- 172.21.15.44:0/1011134413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca131850 0x5581ca139ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:18:01.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.153+0000 7f2179d9b640 1 --2- 172.21.15.44:0/1011134413 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca131310 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f2170006010 tx=0x7f217002db00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cbaaa0bd2f9aa97 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:01.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.153+0000 7f2179d9b640 1 --2- 172.21.15.44:0/2046529714 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f21640a2d00 0x7f21640a51c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f21700096c0 tx=0x7f2170049000 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:01.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.154+0000 7f2168ff9640 1 -- 172.21.15.44:0/1011134413 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2170050070 con 0x5581ca130f30 2024-04-21T23:18:01.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.154+0000 7f2168ff9640 1 -- 172.21.15.44:0/1011134413 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2170004470 con 0x5581ca130f30 2024-04-21T23:18:01.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.154+0000 7f2168ff9640 1 -- 172.21.15.44:0/1011134413 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2170003770 con 0x5581ca130f30 2024-04-21T23:18:01.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.154+0000 7f2183a0b740 1 -- 172.21.15.44:0/1011134413 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 msgr2=0x5581ca131310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.154+0000 7f2183a0b740 1 --2- 172.21.15.44:0/1011134413 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca131310 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f2170006010 tx=0x7f217002db00 comp rx=0 tx=0).stop 2024-04-21T23:18:01.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.154+0000 7f2183a0b740 1 -- 172.21.15.44:0/1011134413 shutdown_connections 2024-04-21T23:18:01.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.154+0000 7f2183a0b740 1 --2- 172.21.15.44:0/1011134413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca131850 0x5581ca139ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.154+0000 7f2183a0b740 1 --2- 172.21.15.44:0/1011134413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca13a430 0x5581ca13c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.154+0000 7f2183a0b740 1 --2- 172.21.15.44:0/1011134413 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca131310 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.154+0000 7f2183a0b740 1 -- 172.21.15.44:0/1011134413 >> 172.21.15.44:0/1011134413 conn(0x5581ca12b660 msgr2=0x5581ca12ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:01.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.154+0000 7f2183a0b740 1 -- 172.21.15.44:0/1011134413 shutdown_connections 2024-04-21T23:18:01.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.154+0000 7f2183a0b740 1 -- 172.21.15.44:0/1011134413 wait complete. 2024-04-21T23:18:01.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.155+0000 7f2183a0b740 1 -- start start 2024-04-21T23:18:01.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.155+0000 7f2183a0b740 1 Processor -- start 2024-04-21T23:18:01.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.155+0000 7f2183a0b740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca1374a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.155+0000 7f2183a0b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca131850 0x5581ca1379e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.155+0000 7f2183a0b740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca13a430 0x5581ca137f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.155+0000 7f2183a0b740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5581ca13d1c0 con 0x5581ca130f30 2024-04-21T23:18:01.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.155+0000 7f2183a0b740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5581ca050e00 con 0x5581ca13a430 2024-04-21T23:18:01.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.155+0000 7f2183a0b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5581ca13d340 con 0x5581ca131850 2024-04-21T23:18:01.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.155+0000 7f217a59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca1374a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.155+0000 7f2183a0b740 10 client.0 _ll_register_callbacks cb 0x5581ca08beb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:01.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.155+0000 7f217a59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca1374a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55192/0 (socket says 172.21.15.44:55192) 2024-04-21T23:18:01.166 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.155+0000 7f217a59c640 1 -- 172.21.15.44:0/1397012909 learned_addr learned my addr 172.21.15.44:0/1397012909 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:01.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.155+0000 7f2179d9b640 1 --2- 172.21.15.44:0/1397012909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca131850 0x5581ca1379e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.155+0000 7f217ad9d640 1 --2- 172.21.15.44:0/1397012909 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca13a430 0x5581ca137f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.156+0000 7f217a59c640 1 -- 172.21.15.44:0/1397012909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca131850 msgr2=0x5581ca1379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:01.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.156+0000 7f217a59c640 1 --2- 172.21.15.44:0/1397012909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca131850 0x5581ca1379e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.156+0000 7f217a59c640 1 -- 172.21.15.44:0/1397012909 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca13a430 msgr2=0x5581ca137f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:01.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.156+0000 7f217a59c640 1 --2- 172.21.15.44:0/1397012909 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca13a430 0x5581ca137f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.156+0000 7f217a59c640 1 -- 172.21.15.44:0/1397012909 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581c9da93f0 con 0x5581ca130f30 2024-04-21T23:18:01.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.156+0000 7f217ad9d640 1 --2- 172.21.15.44:0/1397012909 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca13a430 0x5581ca137f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:01.167 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.156+0000 7f217a59c640 1 --2- 172.21.15.44:0/1397012909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca1374a0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f216c00ad50 tx=0x7f216c0086e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=536486328bbbfe3c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:01.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.156+0000 7f2161ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:18:01.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.156+0000 7f2161ffb640 1 -- 172.21.15.44:0/1397012909 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c005220 con 0x5581ca130f30 2024-04-21T23:18:01.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.156+0000 7f2161ffb640 1 -- 172.21.15.44:0/1397012909 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7f216c0053c0 con 0x5581ca130f30 2024-04-21T23:18:01.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.156+0000 7f2183a0b740 1 -- 172.21.15.44:0/1397012909 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2170004470 con 0x5581ca130f30 2024-04-21T23:18:01.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.156+0000 7f2161ffb640 1 -- 172.21.15.44:0/1397012909 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f216c027050 con 0x5581ca130f30 2024-04-21T23:18:01.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.156+0000 7f2183a0b740 10 client.17616 fetch_fsmap learned FSMap version 31 2024-04-21T23:18:01.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.156+0000 7f2183a0b740 1 -- 172.21.15.44:0/1397012909 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5581c9d690f0 con 0x5581ca130f30 2024-04-21T23:18:01.168 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.156+0000 7f2161ffb640 1 -- 172.21.15.44:0/1397012909 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2170004470 con 0x5581ca130f30 2024-04-21T23:18:01.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.157+0000 7f2161ffb640 1 -- 172.21.15.44:0/1397012909 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f216c01c680 con 0x5581ca130f30 2024-04-21T23:18:01.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.157+0000 7f2183a0b740 10 client.17616 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:18:01.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.157+0000 7f2183a0b740 10 client.17616 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:01.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.157+0000 7f2183a0b740 20 client.17616 populate_metadata read hostname 'smithi044' 2024-04-21T23:18:01.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.157+0000 7f2183a0b740 1 --2- 172.21.15.44:0/1397012909 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5581ca14a880 0x5581ca14cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.157+0000 7f2183a0b740 4 client.17616 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:18:01.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.157+0000 7f2183a0b740 1 -- 172.21.15.44:0/1397012909 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581ca0a2aa0 con 0x5581ca14a880 2024-04-21T23:18:01.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.157+0000 7f2179d9b640 1 --2- 172.21.15.44:0/1397012909 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5581ca14a880 0x5581ca14cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.169 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.157+0000 7f2179d9b640 1 --2- 172.21.15.44:0/1397012909 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5581ca14a880 0x5581ca14cd00 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4de7b52a16eaaa8 server_cookie=ce21377d9502a6b4 in_seq=0 out_seq=0 2024-04-21T23:18:01.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.157+0000 7f2161ffb640 10 client.17616 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:01.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.158+0000 7f2161ffb640 1 -- 172.21.15.44:0/1397012909 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5581ca0a2aa0 con 0x5581ca14a880 2024-04-21T23:18:01.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.158+0000 7f2161ffb640 10 client.17616 handle_command_reply: tid=0 2024-04-21T23:18:01.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.158+0000 7f2183a0b740 1 client.17616 shutdown 2024-04-21T23:18:01.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.158+0000 7f2183a0b740 1 -- 172.21.15.44:0/1397012909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 msgr2=0x5581ca1374a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.158+0000 7f2183a0b740 1 --2- 172.21.15.44:0/1397012909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca1374a0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f216c00ad50 tx=0x7f216c0086e0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.158+0000 7f2183a0b740 1 -- 172.21.15.44:0/1397012909 shutdown_connections 2024-04-21T23:18:01.170 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.158+0000 7f2183a0b740 1 --2- 172.21.15.44:0/1397012909 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5581ca14a880 0x5581ca14cd00 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.158+0000 7f2183a0b740 1 --2- 172.21.15.44:0/1397012909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca131850 0x5581ca1379e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.158+0000 7f2161ffb640 0 client.17616 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:01.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.158+0000 7f2183a0b740 1 --2- 172.21.15.44:0/1397012909 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca13a430 0x5581ca137f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.158+0000 7f2183a0b740 1 --2- 172.21.15.44:0/1397012909 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca1374a0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.158+0000 7f2183a0b740 1 -- 172.21.15.44:0/1397012909 >> 172.21.15.44:0/1397012909 conn(0x5581ca12b660 msgr2=0x5581ca12d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:01.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.158+0000 7f2183a0b740 1 -- 172.21.15.44:0/1397012909 shutdown_connections 2024-04-21T23:18:01.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.158+0000 7f2183a0b740 1 -- 172.21.15.44:0/1397012909 wait complete. 2024-04-21T23:18:01.171 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.158+0000 7f2183a0b740 20 client.17616 trim_cache size 0 max 16384 2024-04-21T23:18:01.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.176+0000 7f2183a0b740 1 Processor -- start 2024-04-21T23:18:01.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.176+0000 7f2183a0b740 1 -- start start 2024-04-21T23:18:01.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.176+0000 7f2183a0b740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca1460b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.176+0000 7f2183a0b740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca131850 0x5581ca1327b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.176+0000 7f2183a0b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca13a430 0x5581ca136d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.176+0000 7f2183a0b740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f217c1aaa50 con 0x5581ca130f30 2024-04-21T23:18:01.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.176+0000 7f2183a0b740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f217c1ad2b0 con 0x5581ca131850 2024-04-21T23:18:01.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.176+0000 7f2183a0b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f217c1ab0a0 con 0x5581ca13a430 2024-04-21T23:18:01.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.177+0000 7f217a59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca1460b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.177+0000 7f217a59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca1460b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55208/0 (socket says 172.21.15.44:55208) 2024-04-21T23:18:01.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.177+0000 7f217a59c640 1 -- 172.21.15.44:0/3611925133 learned_addr learned my addr 172.21.15.44:0/3611925133 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:01.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.177+0000 7f217ad9d640 1 --2- 172.21.15.44:0/3611925133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca13a430 0x5581ca136d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.177+0000 7f217a59c640 1 -- 172.21.15.44:0/3611925133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca13a430 msgr2=0x5581ca136d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:01.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.177+0000 7f217a59c640 1 --2- 172.21.15.44:0/3611925133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca13a430 0x5581ca136d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.177+0000 7f217a59c640 1 -- 172.21.15.44:0/3611925133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca131850 msgr2=0x5581ca1327b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:01.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.177+0000 7f217a59c640 1 --2- 172.21.15.44:0/3611925133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca131850 0x5581ca1327b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.177+0000 7f217a59c640 1 -- 172.21.15.44:0/3611925133 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581ca126cf0 con 0x5581ca130f30 2024-04-21T23:18:01.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.177+0000 7f217a59c640 1 --2- 172.21.15.44:0/3611925133 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca1460b0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f216c005d50 tx=0x7f216c00af60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=364f029e212e56ad server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:01.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.177+0000 7f2163fff640 1 -- 172.21.15.44:0/3611925133 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c02f070 con 0x5581ca130f30 2024-04-21T23:18:01.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.177+0000 7f2163fff640 1 -- 172.21.15.44:0/3611925133 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f216c010e50 con 0x5581ca130f30 2024-04-21T23:18:01.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.177+0000 7f2163fff640 1 -- 172.21.15.44:0/3611925133 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c0054e0 con 0x5581ca130f30 2024-04-21T23:18:01.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.178+0000 7f2183a0b740 1 -- 172.21.15.44:0/3611925133 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 msgr2=0x5581ca1460b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.178+0000 7f2183a0b740 1 --2- 172.21.15.44:0/3611925133 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca1460b0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f216c005d50 tx=0x7f216c00af60 comp rx=0 tx=0).stop 2024-04-21T23:18:01.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.178+0000 7f2183a0b740 1 -- 172.21.15.44:0/3611925133 shutdown_connections 2024-04-21T23:18:01.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.178+0000 7f2183a0b740 1 --2- 172.21.15.44:0/3611925133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca13a430 0x5581ca136d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.178+0000 7f2183a0b740 1 --2- 172.21.15.44:0/3611925133 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca131850 0x5581ca1327b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.178+0000 7f2183a0b740 1 --2- 172.21.15.44:0/3611925133 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca1460b0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.178+0000 7f2183a0b740 1 -- 172.21.15.44:0/3611925133 >> 172.21.15.44:0/3611925133 conn(0x5581ca12b660 msgr2=0x5581ca137460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:01.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.178+0000 7f2183a0b740 1 -- 172.21.15.44:0/3611925133 shutdown_connections 2024-04-21T23:18:01.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.178+0000 7f2183a0b740 1 -- 172.21.15.44:0/3611925133 wait complete. 2024-04-21T23:18:01.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.178+0000 7f2183a0b740 1 -- start start 2024-04-21T23:18:01.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.178+0000 7f2183a0b740 1 Processor -- start 2024-04-21T23:18:01.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.178+0000 7f2183a0b740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca142a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.178+0000 7f2183a0b740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca131850 0x5581ca13ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.178+0000 7f2183a0b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca13a430 0x5581ca13e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.178+0000 7f2183a0b740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f217c1ab770 con 0x5581ca130f30 2024-04-21T23:18:01.185 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.178+0000 7f2183a0b740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f217c1acbe0 con 0x5581ca131850 2024-04-21T23:18:01.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.178+0000 7f2183a0b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f217c1abe40 con 0x5581ca13a430 2024-04-21T23:18:01.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.179+0000 7f2183a0b740 10 client.0 _ll_register_callbacks cb 0x7f216c010b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:01.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.179+0000 7f217a59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca142a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.179+0000 7f217a59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca142a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55218/0 (socket says 172.21.15.44:55218) 2024-04-21T23:18:01.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.179+0000 7f217a59c640 1 -- 172.21.15.44:0/3287203491 learned_addr learned my addr 172.21.15.44:0/3287203491 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:01.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.179+0000 7f2179d9b640 1 --2- 172.21.15.44:0/3287203491 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca131850 0x5581ca13ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.179+0000 7f217ad9d640 1 --2- 172.21.15.44:0/3287203491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca13a430 0x5581ca13e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.186 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.179+0000 7f217a59c640 1 -- 172.21.15.44:0/3287203491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca13a430 msgr2=0x5581ca13e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:01.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.179+0000 7f217a59c640 1 --2- 172.21.15.44:0/3287203491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca13a430 0x5581ca13e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.179+0000 7f217a59c640 1 -- 172.21.15.44:0/3287203491 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca131850 msgr2=0x5581ca13ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:01.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.179+0000 7f217a59c640 1 --2- 172.21.15.44:0/3287203491 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca131850 0x5581ca13ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.179+0000 7f217a59c640 1 -- 172.21.15.44:0/3287203491 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581ca13eb80 con 0x5581ca130f30 2024-04-21T23:18:01.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.179+0000 7f2179d9b640 1 --2- 172.21.15.44:0/3287203491 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca131850 0x5581ca13ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:01.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.179+0000 7f217a59c640 1 --2- 172.21.15.44:0/3287203491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca142a40 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f216c005b90 tx=0x7f216c005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f593a34b094062b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:01.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.179+0000 7f2161ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:18:01.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.179+0000 7f2161ffb640 1 -- 172.21.15.44:0/3287203491 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c038060 con 0x5581ca130f30 2024-04-21T23:18:01.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.179+0000 7f2161ffb640 1 -- 172.21.15.44:0/3287203491 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7f216c0054f0 con 0x5581ca130f30 2024-04-21T23:18:01.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.179+0000 7f2183a0b740 1 -- 172.21.15.44:0/3287203491 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f216c010e50 con 0x5581ca130f30 2024-04-21T23:18:01.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.179+0000 7f2161ffb640 1 -- 172.21.15.44:0/3287203491 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f216c03f050 con 0x5581ca130f30 2024-04-21T23:18:01.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.179+0000 7f2183a0b740 10 client.17622 fetch_fsmap learned FSMap version 31 2024-04-21T23:18:01.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.179+0000 7f2183a0b740 1 -- 172.21.15.44:0/3287203491 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5581ca1b0f20 con 0x5581ca130f30 2024-04-21T23:18:01.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.180+0000 7f2161ffb640 1 -- 172.21.15.44:0/3287203491 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f216c010e50 con 0x5581ca130f30 2024-04-21T23:18:01.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.180+0000 7f2161ffb640 1 -- 172.21.15.44:0/3287203491 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f216c01cce0 con 0x5581ca130f30 2024-04-21T23:18:01.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.180+0000 7f2183a0b740 10 client.17622 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:18:01.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.180+0000 7f2183a0b740 10 client.17622 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:01.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.180+0000 7f2183a0b740 20 client.17622 populate_metadata read hostname 'smithi044' 2024-04-21T23:18:01.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.180+0000 7f2183a0b740 1 --2- 172.21.15.44:0/3287203491 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5581ca1b14d0 0x5581ca1b38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.180+0000 7f2183a0b740 4 client.17622 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:18:01.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.180+0000 7f2183a0b740 1 -- 172.21.15.44:0/3287203491 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5581ca09cdf0 con 0x5581ca1b14d0 2024-04-21T23:18:01.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.180+0000 7f2179d9b640 1 --2- 172.21.15.44:0/3287203491 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5581ca1b14d0 0x5581ca1b38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.181+0000 7f2161ffb640 10 client.17622 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:01.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.181+0000 7f2179d9b640 1 --2- 172.21.15.44:0/3287203491 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5581ca1b14d0 0x5581ca1b38c0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f063ead29e0f6110 server_cookie=6358af7c1970dbfe in_seq=0 out_seq=0 2024-04-21T23:18:01.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.181+0000 7f2161ffb640 1 -- 172.21.15.44:0/3287203491 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5581ca09cdf0 con 0x5581ca1b14d0 2024-04-21T23:18:01.189 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.181+0000 7f2161ffb640 10 client.17622 handle_command_reply: tid=0 2024-04-21T23:18:01.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.181+0000 7f2183a0b740 1 client.17622 shutdown 2024-04-21T23:18:01.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.181+0000 7f2183a0b740 1 -- 172.21.15.44:0/3287203491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 msgr2=0x5581ca142a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.181+0000 7f2183a0b740 1 --2- 172.21.15.44:0/3287203491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca142a40 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f216c005b90 tx=0x7f216c005da0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.181+0000 7f2183a0b740 1 -- 172.21.15.44:0/3287203491 shutdown_connections 2024-04-21T23:18:01.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.181+0000 7f2183a0b740 1 --2- 172.21.15.44:0/3287203491 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5581ca1b14d0 0x5581ca1b38c0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.181+0000 7f2183a0b740 1 --2- 172.21.15.44:0/3287203491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581ca13a430 0x5581ca13e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.181+0000 7f2183a0b740 1 --2- 172.21.15.44:0/3287203491 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5581ca131850 0x5581ca13ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.181+0000 7f2183a0b740 1 --2- 172.21.15.44:0/3287203491 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x5581ca130f30 0x5581ca142a40 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.190 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.181+0000 7f2183a0b740 1 -- 172.21.15.44:0/3287203491 >> 172.21.15.44:0/3287203491 conn(0x5581ca12b660 msgr2=0x5581ca12d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:01.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.181+0000 7f2161ffb640 0 client.17622 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:01.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.181+0000 7f2183a0b740 1 -- 172.21.15.44:0/3287203491 shutdown_connections 2024-04-21T23:18:01.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.181+0000 7f2183a0b740 1 -- 172.21.15.44:0/3287203491 wait complete. 2024-04-21T23:18:01.191 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.181+0000 7f2183a0b740 20 client.17622 trim_cache size 0 max 16384 2024-04-21T23:18:01.191 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:18:01.192 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-21T23:18:01.192 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-21T23:18:01.192 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:18:01.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.182+0000 7f2180d4b640 1 -- 172.21.15.44:0/2046529714 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f21640a2d00 msgr2=0x7f21640a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.182+0000 7f2180d4b640 1 --2- 172.21.15.44:0/2046529714 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f21640a2d00 0x7f21640a51c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f21700096c0 tx=0x7f2170049000 comp rx=0 tx=0).stop 2024-04-21T23:18:01.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.182+0000 7f2180d4b640 1 -- 172.21.15.44:0/2046529714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f217c103bb0 msgr2=0x7f217c19c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.193 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.182+0000 7f2180d4b640 1 --2- 172.21.15.44:0/2046529714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f217c103bb0 0x7f217c19c990 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f217400b970 tx=0x7f217400be30 comp rx=0 tx=0).stop 2024-04-21T23:18:01.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.182+0000 7f2180d4b640 1 -- 172.21.15.44:0/2046529714 shutdown_connections 2024-04-21T23:18:01.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.182+0000 7f2180d4b640 1 --2- 172.21.15.44:0/2046529714 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f21640a2d00 0x7f21640a51c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.182+0000 7f2180d4b640 1 --2- 172.21.15.44:0/2046529714 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f217c103bb0 0x7f217c19c990 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.182+0000 7f2180d4b640 1 --2- 172.21.15.44:0/2046529714 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f217c103210 0x7f217c198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.182+0000 7f2180d4b640 1 --2- 172.21.15.44:0/2046529714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f217c101ad0 0x7f217c198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.182+0000 7f2180d4b640 1 -- 172.21.15.44:0/2046529714 >> 172.21.15.44:0/2046529714 conn(0x7f217c0fd200 msgr2=0x7f217c107580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:01.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.182+0000 7f2180d4b640 1 -- 172.21.15.44:0/2046529714 shutdown_connections 2024-04-21T23:18:01.194 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.182+0000 7f2180d4b640 1 -- 172.21.15.44:0/2046529714 wait complete. 2024-04-21T23:18:01.195 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:18:01.195 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:18:01.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.300+0000 7fed34ea3640 1 -- 172.21.15.44:0/1174286610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed30101380 msgr2=0x7fed30101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.300+0000 7fed34ea3640 1 --2- 172.21.15.44:0/1174286610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed30101380 0x7fed30101760 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fed20009900 tx=0x7fed2002bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:01.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.300+0000 7fed34ea3640 1 -- 172.21.15.44:0/1174286610 shutdown_connections 2024-04-21T23:18:01.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.300+0000 7fed34ea3640 1 --2- 172.21.15.44:0/1174286610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed30105a00 0x7fed3010d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.300+0000 7fed34ea3640 1 --2- 172.21.15.44:0/1174286610 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed30101d30 0x7fed301052d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.300+0000 7fed34ea3640 1 --2- 172.21.15.44:0/1174286610 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed30101380 0x7fed30101760 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.300+0000 7fed34ea3640 1 -- 172.21.15.44:0/1174286610 >> 172.21.15.44:0/1174286610 conn(0x7fed300758f0 msgr2=0x7fed30075d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:01.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.300+0000 7fed34ea3640 1 -- 172.21.15.44:0/1174286610 shutdown_connections 2024-04-21T23:18:01.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.300+0000 7fed34ea3640 1 -- 172.21.15.44:0/1174286610 wait complete. 2024-04-21T23:18:01.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.300+0000 7fed34ea3640 1 Processor -- start 2024-04-21T23:18:01.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.301+0000 7fed34ea3640 1 -- start start 2024-04-21T23:18:01.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.301+0000 7fed34ea3640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed30101380 0x7fed30198de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.301+0000 7fed34ea3640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed30105a00 0x7fed30199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.301+0000 7fed34ea3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed301998b0 0x7fed301a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.301+0000 7fed34ea3640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fed30110a20 con 0x7fed30105a00 2024-04-21T23:18:01.303 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.301+0000 7fed34ea3640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fed301108a0 con 0x7fed30101380 2024-04-21T23:18:01.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.301+0000 7fed34ea3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fed30110ba0 con 0x7fed301998b0 2024-04-21T23:18:01.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.301+0000 7fed2dd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed30105a00 0x7fed30199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.301+0000 7fed2dd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed30105a00 0x7fed30199320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55244/0 (socket says 172.21.15.44:55244) 2024-04-21T23:18:01.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.301+0000 7fed2dd9b640 1 -- 172.21.15.44:0/3693433111 learned_addr learned my addr 172.21.15.44:0/3693433111 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:01.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.301+0000 7fed2ed9d640 1 --2- 172.21.15.44:0/3693433111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed301998b0 0x7fed301a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.301+0000 7fed2e59c640 1 --2- 172.21.15.44:0/3693433111 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed30101380 0x7fed30198de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.301+0000 7fed2dd9b640 1 -- 172.21.15.44:0/3693433111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed301998b0 msgr2=0x7fed301a3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.304 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.301+0000 7fed2dd9b640 1 --2- 172.21.15.44:0/3693433111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed301998b0 0x7fed301a3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.301+0000 7fed2dd9b640 1 -- 172.21.15.44:0/3693433111 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed30101380 msgr2=0x7fed30198de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.301+0000 7fed2dd9b640 1 --2- 172.21.15.44:0/3693433111 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed30101380 0x7fed30198de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.301+0000 7fed2dd9b640 1 -- 172.21.15.44:0/3693433111 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed301a4630 con 0x7fed30105a00 2024-04-21T23:18:01.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.301+0000 7fed2dd9b640 1 --2- 172.21.15.44:0/3693433111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed30105a00 0x7fed30199320 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fed1800e9f0 tx=0x7fed1800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:01.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.301+0000 7fed277fe640 1 -- 172.21.15.44:0/3693433111 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1800ccc0 con 0x7fed30105a00 2024-04-21T23:18:01.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.302+0000 7fed277fe640 1 -- 172.21.15.44:0/3693433111 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed18010910 con 0x7fed30105a00 2024-04-21T23:18:01.305 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.302+0000 7fed34ea3640 1 -- 172.21.15.44:0/3693433111 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed301a4920 con 0x7fed30105a00 2024-04-21T23:18:01.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.302+0000 7fed277fe640 1 -- 172.21.15.44:0/3693433111 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed18018920 con 0x7fed30105a00 2024-04-21T23:18:01.306 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.302+0000 7fed34ea3640 1 -- 172.21.15.44:0/3693433111 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed301a4e70 con 0x7fed30105a00 2024-04-21T23:18:01.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.302+0000 7fed34ea3640 1 -- 172.21.15.44:0/3693433111 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed30068a40 con 0x7fed30105a00 2024-04-21T23:18:01.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.304+0000 7fed277fe640 1 -- 172.21.15.44:0/3693433111 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fed18010ac0 con 0x7fed30105a00 2024-04-21T23:18:01.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.304+0000 7fed277fe640 1 --2- 172.21.15.44:0/3693433111 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fed1c0a2d00 0x7fed1c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.304+0000 7fed2e59c640 1 --2- 172.21.15.44:0/3693433111 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fed1c0a2d00 0x7fed1c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.304+0000 7fed277fe640 1 -- 172.21.15.44:0/3693433111 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7fed180cfc30 con 0x7fed30105a00 2024-04-21T23:18:01.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.305+0000 7fed2e59c640 1 --2- 172.21.15.44:0/3693433111 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fed1c0a2d00 0x7fed1c0a51c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fed20036040 tx=0x7fed20005fd0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:01.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.309+0000 7fed277fe640 1 -- 172.21.15.44:0/3693433111 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fed1809c460 con 0x7fed30105a00 2024-04-21T23:18:01.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.532+0000 7fed34ea3640 1 -- 172.21.15.44:0/3693433111 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed30101d30 con 0x7fed30105a00 2024-04-21T23:18:01.532 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.533+0000 7fed277fe640 1 -- 172.21.15.44:0/3693433111 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11791 (secure 0 0 0) 0x7fed1809bd60 con 0x7fed30105a00 2024-04-21T23:18:01.532 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:18:01.533 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:16:20.187271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26863":{"gid":26863,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3179646768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3179646768},{"type":"v1","addr":"172.21.15.69:6839","nonce":3179646768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:18:01.533 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 31 2024-04-21T23:18:01.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.535+0000 7fed34ea3640 1 -- 172.21.15.44:0/3693433111 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fed1c0a2d00 msgr2=0x7fed1c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.535+0000 7fed34ea3640 1 --2- 172.21.15.44:0/3693433111 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fed1c0a2d00 0x7fed1c0a51c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fed20036040 tx=0x7fed20005fd0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.535+0000 7fed34ea3640 1 -- 172.21.15.44:0/3693433111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed30105a00 msgr2=0x7fed30199320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.535+0000 7fed34ea3640 1 --2- 172.21.15.44:0/3693433111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed30105a00 0x7fed30199320 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fed1800e9f0 tx=0x7fed1800eeb0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.535+0000 7fed34ea3640 1 -- 172.21.15.44:0/3693433111 shutdown_connections 2024-04-21T23:18:01.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.535+0000 7fed34ea3640 1 --2- 172.21.15.44:0/3693433111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed301998b0 0x7fed301a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.535+0000 7fed34ea3640 1 --2- 172.21.15.44:0/3693433111 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fed1c0a2d00 0x7fed1c0a51c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.535+0000 7fed34ea3640 1 --2- 172.21.15.44:0/3693433111 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed30105a00 0x7fed30199320 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.535+0000 7fed34ea3640 1 --2- 172.21.15.44:0/3693433111 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed30101380 0x7fed30198de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.535+0000 7fed34ea3640 1 -- 172.21.15.44:0/3693433111 >> 172.21.15.44:0/3693433111 conn(0x7fed300758f0 msgr2=0x7fed300fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:01.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.535+0000 7fed34ea3640 1 -- 172.21.15.44:0/3693433111 shutdown_connections 2024-04-21T23:18:01.536 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.535+0000 7fed34ea3640 1 -- 172.21.15.44:0/3693433111 wait complete. 2024-04-21T23:18:01.545 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:18:01.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.651+0000 7fd94c3a0640 1 -- 172.21.15.44:0/423097847 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd944107710 msgr2=0x7fd944107af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.651+0000 7fd94c3a0640 1 --2- 172.21.15.44:0/423097847 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd944107710 0x7fd944107af0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fd940009930 tx=0x7fd94002bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:01.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.651+0000 7fd94c3a0640 1 -- 172.21.15.44:0/423097847 shutdown_connections 2024-04-21T23:18:01.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.651+0000 7fd94c3a0640 1 --2- 172.21.15.44:0/423097847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9441020c0 0x7fd94410e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.651+0000 7fd94c3a0640 1 --2- 172.21.15.44:0/423097847 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd944101700 0x7fd944101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.651+0000 7fd94c3a0640 1 --2- 172.21.15.44:0/423097847 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd944107710 0x7fd944107af0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.651+0000 7fd94c3a0640 1 -- 172.21.15.44:0/423097847 >> 172.21.15.44:0/423097847 conn(0x7fd9440fd1f0 msgr2=0x7fd9440ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:01.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.651+0000 7fd94c3a0640 1 -- 172.21.15.44:0/423097847 shutdown_connections 2024-04-21T23:18:01.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.651+0000 7fd94c3a0640 1 -- 172.21.15.44:0/423097847 wait complete. 2024-04-21T23:18:01.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.651+0000 7fd94c3a0640 1 Processor -- start 2024-04-21T23:18:01.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.651+0000 7fd94c3a0640 1 -- start start 2024-04-21T23:18:01.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd94c3a0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd944101700 0x7fd944198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd94c3a0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd9441020c0 0x7fd944198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd94c3a0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd944107710 0x7fd94419c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd94c3a0640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd944111b20 con 0x7fd944107710 2024-04-21T23:18:01.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd94c3a0640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441119a0 con 0x7fd9441020c0 2024-04-21T23:18:01.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd94c3a0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd944111ca0 con 0x7fd944101700 2024-04-21T23:18:01.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd94a93d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd944107710 0x7fd94419c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd94a93d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd944107710 0x7fd94419c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55266/0 (socket says 172.21.15.44:55266) 2024-04-21T23:18:01.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd94a93d640 1 -- 172.21.15.44:0/3630968215 learned_addr learned my addr 172.21.15.44:0/3630968215 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:01.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd94993b640 1 --2- 172.21.15.44:0/3630968215 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd9441020c0 0x7fd944198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd94a13c640 1 --2- 172.21.15.44:0/3630968215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd944101700 0x7fd944198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd94a93d640 1 -- 172.21.15.44:0/3630968215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd944101700 msgr2=0x7fd944198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd94a93d640 1 --2- 172.21.15.44:0/3630968215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd944101700 0x7fd944198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd94a93d640 1 -- 172.21.15.44:0/3630968215 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd9441020c0 msgr2=0x7fd944198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd94a93d640 1 --2- 172.21.15.44:0/3630968215 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd9441020c0 0x7fd944198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd94a93d640 1 -- 172.21.15.44:0/3630968215 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd94419d010 con 0x7fd944107710 2024-04-21T23:18:01.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd94a13c640 1 --2- 172.21.15.44:0/3630968215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd944101700 0x7fd944198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:01.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd94a93d640 1 --2- 172.21.15.44:0/3630968215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd944107710 0x7fd94419c8e0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fd93c00eaa0 tx=0x7fd93c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:01.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd93b7fe640 1 -- 172.21.15.44:0/3630968215 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93c00cdb0 con 0x7fd944107710 2024-04-21T23:18:01.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd93b7fe640 1 -- 172.21.15.44:0/3630968215 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd93c005020 con 0x7fd944107710 2024-04-21T23:18:01.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd94c3a0640 1 -- 172.21.15.44:0/3630968215 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd94419d300 con 0x7fd944107710 2024-04-21T23:18:01.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd93b7fe640 1 -- 172.21.15.44:0/3630968215 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93c0116b0 con 0x7fd944107710 2024-04-21T23:18:01.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.652+0000 7fd94c3a0640 1 -- 172.21.15.44:0/3630968215 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9441a4b90 con 0x7fd944107710 2024-04-21T23:18:01.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.655+0000 7fd93b7fe640 1 -- 172.21.15.44:0/3630968215 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd93c005560 con 0x7fd944107710 2024-04-21T23:18:01.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.655+0000 7fd94ef82740 1 Processor -- start 2024-04-21T23:18:01.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.655+0000 7fd94ef82740 1 -- start start 2024-04-21T23:18:01.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.655+0000 7fd94ef82740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b09020 0x55d397b09400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.655+0000 7fd94ef82740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d397b09940 0x55d397b11fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd94ef82740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 0x55d397b14910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd94ef82740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d397a6d530 con 0x55d397b12520 2024-04-21T23:18:01.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd94ef82740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55d397a15830 con 0x55d397b09940 2024-04-21T23:18:01.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd94ef82740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d397a694f0 con 0x55d397b09020 2024-04-21T23:18:01.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd93b7fe640 1 --2- 172.21.15.44:0/3630968215 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd9300a2d80 0x7fd9300a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd94993b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 0x55d397b14910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd94993b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 0x55d397b14910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55276/0 (socket says 172.21.15.44:55276) 2024-04-21T23:18:01.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd94993b640 1 -- 172.21.15.44:0/604741935 learned_addr learned my addr 172.21.15.44:0/604741935 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:01.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd94a13c640 1 --2- 172.21.15.44:0/604741935 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d397b09940 0x55d397b11fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd94a93d640 1 --2- 172.21.15.44:0/3630968215 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd9300a2d80 0x7fd9300a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd94993b640 1 --2- 172.21.15.44:0/604741935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b09020 0x55d397b09400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd93b7fe640 1 -- 172.21.15.44:0/3630968215 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7fd93c0cfa20 con 0x7fd944107710 2024-04-21T23:18:01.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd94993b640 1 -- 172.21.15.44:0/604741935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b09020 msgr2=0x55d397b09400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:01.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd94993b640 1 --2- 172.21.15.44:0/604741935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b09020 0x55d397b09400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd94993b640 1 -- 172.21.15.44:0/604741935 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d397b09940 msgr2=0x55d397b11fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:01.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd94993b640 1 --2- 172.21.15.44:0/604741935 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d397b09940 0x55d397b11fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd94993b640 1 -- 172.21.15.44:0/604741935 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d397a392c0 con 0x55d397b12520 2024-04-21T23:18:01.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd94993b640 1 --2- 172.21.15.44:0/604741935 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 0x55d397b14910 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fd9340124f0 tx=0x7fd93400e610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=727933eb1c9c0098 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:01.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd94a93d640 1 --2- 172.21.15.44:0/3630968215 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd9300a2d80 0x7fd9300a5240 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd93c0092f0 tx=0x7fd93c090fe0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:01.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd938bfe640 1 -- 172.21.15.44:0/604741935 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd934018070 con 0x55d397b12520 2024-04-21T23:18:01.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd938bfe640 1 -- 172.21.15.44:0/604741935 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9340047a0 con 0x55d397b12520 2024-04-21T23:18:01.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.656+0000 7fd938bfe640 1 -- 172.21.15.44:0/604741935 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd934010ae0 con 0x55d397b12520 2024-04-21T23:18:01.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.657+0000 7fd94ef82740 1 -- 172.21.15.44:0/604741935 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 msgr2=0x55d397b14910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.657+0000 7fd94ef82740 1 --2- 172.21.15.44:0/604741935 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 0x55d397b14910 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fd9340124f0 tx=0x7fd93400e610 comp rx=0 tx=0).stop 2024-04-21T23:18:01.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.657+0000 7fd94ef82740 1 -- 172.21.15.44:0/604741935 shutdown_connections 2024-04-21T23:18:01.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.657+0000 7fd94ef82740 1 --2- 172.21.15.44:0/604741935 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 0x55d397b14910 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.657+0000 7fd94ef82740 1 --2- 172.21.15.44:0/604741935 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d397b09940 0x55d397b11fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.657+0000 7fd94ef82740 1 --2- 172.21.15.44:0/604741935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b09020 0x55d397b09400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.657+0000 7fd94ef82740 1 -- 172.21.15.44:0/604741935 >> 172.21.15.44:0/604741935 conn(0x55d397b03750 msgr2=0x55d397b03b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:01.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.657+0000 7fd94ef82740 1 -- 172.21.15.44:0/604741935 shutdown_connections 2024-04-21T23:18:01.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.657+0000 7fd94ef82740 1 -- 172.21.15.44:0/604741935 wait complete. 2024-04-21T23:18:01.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.658+0000 7fd94ef82740 1 -- start start 2024-04-21T23:18:01.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.658+0000 7fd94ef82740 1 Processor -- start 2024-04-21T23:18:01.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.658+0000 7fd94ef82740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b09020 0x55d397b11ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.658+0000 7fd94ef82740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d397b09940 0x55d397b0f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.658+0000 7fd94ef82740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b12520 0x55d397b0f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.658+0000 7fd94ef82740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55d397b15310 con 0x55d397b09020 2024-04-21T23:18:01.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.658+0000 7fd94ef82740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55d397a28f10 con 0x55d397b09940 2024-04-21T23:18:01.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.658+0000 7fd94ef82740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d397b15490 con 0x55d397b12520 2024-04-21T23:18:01.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.658+0000 7fd94ef82740 10 client.0 _ll_register_callbacks cb 0x55d397780be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:01.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.658+0000 7fd94993b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b09020 0x55d397b11ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.658+0000 7fd94993b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b09020 0x55d397b11ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55288/0 (socket says 172.21.15.44:55288) 2024-04-21T23:18:01.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.658+0000 7fd94993b640 1 -- 172.21.15.44:0/2377030135 learned_addr learned my addr 172.21.15.44:0/2377030135 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:01.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.658+0000 7fd94993b640 1 --2- 172.21.15.44:0/2377030135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b12520 0x55d397b0f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.658+0000 7fd94993b640 1 -- 172.21.15.44:0/2377030135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b12520 msgr2=0x55d397b0f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:01.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.658+0000 7fd94993b640 1 --2- 172.21.15.44:0/2377030135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b12520 0x55d397b0f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.658+0000 7fd94993b640 1 -- 172.21.15.44:0/2377030135 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d397b09940 msgr2=0x55d397b0f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:01.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.658+0000 7fd94993b640 1 --2- 172.21.15.44:0/2377030135 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d397b09940 0x55d397b0f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.658+0000 7fd94993b640 1 -- 172.21.15.44:0/2377030135 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3979fac80 con 0x55d397b09020 2024-04-21T23:18:01.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.658+0000 7fd94993b640 1 --2- 172.21.15.44:0/2377030135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b09020 0x55d397b11ef0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fd934004fd0 tx=0x7fd934005ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f8a753e980593a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:01.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.658+0000 7fd92dffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:18:01.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.659+0000 7fd92dffb640 1 -- 172.21.15.44:0/2377030135 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd934018040 con 0x55d397b09020 2024-04-21T23:18:01.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.659+0000 7fd92dffb640 1 -- 172.21.15.44:0/2377030135 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7fd9340054d0 con 0x55d397b09020 2024-04-21T23:18:01.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.659+0000 7fd94ef82740 1 -- 172.21.15.44:0/2377030135 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9340047a0 con 0x55d397b09020 2024-04-21T23:18:01.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.659+0000 7fd92dffb640 1 -- 172.21.15.44:0/2377030135 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd934032050 con 0x55d397b09020 2024-04-21T23:18:01.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.659+0000 7fd94ef82740 10 client.17640 fetch_fsmap learned FSMap version 31 2024-04-21T23:18:01.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.659+0000 7fd94ef82740 1 -- 172.21.15.44:0/2377030135 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d3977813f0 con 0x55d397b09020 2024-04-21T23:18:01.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.659+0000 7fd92dffb640 1 -- 172.21.15.44:0/2377030135 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9340047a0 con 0x55d397b09020 2024-04-21T23:18:01.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.659+0000 7fd92dffb640 1 -- 172.21.15.44:0/2377030135 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7fd934029070 con 0x55d397b09020 2024-04-21T23:18:01.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.659+0000 7fd94ef82740 10 client.17640 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:18:01.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.659+0000 7fd94ef82740 10 client.17640 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:01.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.659+0000 7fd94ef82740 20 client.17640 populate_metadata read hostname 'smithi044' 2024-04-21T23:18:01.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.659+0000 7fd94ef82740 1 --2- 172.21.15.44:0/2377030135 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d397b22570 0x55d397b24960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.659+0000 7fd94ef82740 4 client.17640 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:18:01.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.659+0000 7fd94ef82740 1 -- 172.21.15.44:0/2377030135 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d397a7aa70 con 0x55d397b22570 2024-04-21T23:18:01.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.659+0000 7fd94a13c640 1 --2- 172.21.15.44:0/2377030135 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d397b22570 0x55d397b24960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.660+0000 7fd94a13c640 1 --2- 172.21.15.44:0/2377030135 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d397b22570 0x55d397b24960 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbe13efc125a3866 server_cookie=3a9452cad53e8f11 in_seq=0 out_seq=0 2024-04-21T23:18:01.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.660+0000 7fd92dffb640 10 client.17640 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:01.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.660+0000 7fd92dffb640 1 -- 172.21.15.44:0/2377030135 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d397a7aa70 con 0x55d397b22570 2024-04-21T23:18:01.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.660+0000 7fd92dffb640 10 client.17640 handle_command_reply: tid=0 2024-04-21T23:18:01.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.661+0000 7fd94ef82740 1 client.17640 shutdown 2024-04-21T23:18:01.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.661+0000 7fd94ef82740 1 -- 172.21.15.44:0/2377030135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b09020 msgr2=0x55d397b11ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.661+0000 7fd94ef82740 1 --2- 172.21.15.44:0/2377030135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b09020 0x55d397b11ef0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fd934004fd0 tx=0x7fd934005ce0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.661+0000 7fd94ef82740 1 -- 172.21.15.44:0/2377030135 shutdown_connections 2024-04-21T23:18:01.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.661+0000 7fd94ef82740 1 --2- 172.21.15.44:0/2377030135 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d397b22570 0x55d397b24960 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.661+0000 7fd94ef82740 1 --2- 172.21.15.44:0/2377030135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b12520 0x55d397b0f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.661+0000 7fd94ef82740 1 --2- 172.21.15.44:0/2377030135 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d397b09940 0x55d397b0f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.661+0000 7fd92dffb640 0 client.17640 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:01.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.661+0000 7fd94ef82740 1 --2- 172.21.15.44:0/2377030135 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b09020 0x55d397b11ef0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.661+0000 7fd94ef82740 1 -- 172.21.15.44:0/2377030135 >> 172.21.15.44:0/2377030135 conn(0x55d397b03750 msgr2=0x55d397b054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:01.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.661+0000 7fd94ef82740 1 -- 172.21.15.44:0/2377030135 shutdown_connections 2024-04-21T23:18:01.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.661+0000 7fd94ef82740 1 -- 172.21.15.44:0/2377030135 wait complete. 2024-04-21T23:18:01.674 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.661+0000 7fd94ef82740 20 client.17640 trim_cache size 0 max 16384 2024-04-21T23:18:01.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.674+0000 7fd94ef82740 1 Processor -- start 2024-04-21T23:18:01.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.674+0000 7fd94ef82740 1 -- start start 2024-04-21T23:18:01.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.674+0000 7fd94ef82740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b09020 0x55d397b02850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.674+0000 7fd94ef82740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d397b09940 0x55d397b051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.674+0000 7fd94ef82740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 0x55d397b119b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.674+0000 7fd94ef82740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441ace60 con 0x55d397b12520 2024-04-21T23:18:01.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.675+0000 7fd94ef82740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441af6c0 con 0x55d397b09940 2024-04-21T23:18:01.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.675+0000 7fd94ef82740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441ad4b0 con 0x55d397b09020 2024-04-21T23:18:01.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.675+0000 7fd94993b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 0x55d397b119b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.675+0000 7fd94993b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 0x55d397b119b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55302/0 (socket says 172.21.15.44:55302) 2024-04-21T23:18:01.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.675+0000 7fd94993b640 1 -- 172.21.15.44:0/3677178742 learned_addr learned my addr 172.21.15.44:0/3677178742 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:01.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.675+0000 7fd94993b640 1 --2- 172.21.15.44:0/3677178742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b09020 0x55d397b02850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.675+0000 7fd94a13c640 1 --2- 172.21.15.44:0/3677178742 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d397b09940 0x55d397b051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.675+0000 7fd94993b640 1 -- 172.21.15.44:0/3677178742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b09020 msgr2=0x55d397b02850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:01.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.675+0000 7fd94993b640 1 --2- 172.21.15.44:0/3677178742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b09020 0x55d397b02850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.675+0000 7fd94993b640 1 -- 172.21.15.44:0/3677178742 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d397b09940 msgr2=0x55d397b051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:01.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.675+0000 7fd94993b640 1 --2- 172.21.15.44:0/3677178742 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d397b09940 0x55d397b051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.675+0000 7fd94993b640 1 -- 172.21.15.44:0/3677178742 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d39784b7d0 con 0x55d397b12520 2024-04-21T23:18:01.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.675+0000 7fd94a13c640 1 --2- 172.21.15.44:0/3677178742 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d397b09940 0x55d397b051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:01.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.675+0000 7fd94993b640 1 --2- 172.21.15.44:0/3677178742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 0x55d397b119b0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fd93400e7a0 tx=0x7fd93401e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e982014706610e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:01.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.675+0000 7fd92ffff640 1 -- 172.21.15.44:0/3677178742 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93402f020 con 0x55d397b12520 2024-04-21T23:18:01.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.675+0000 7fd92ffff640 1 -- 172.21.15.44:0/3677178742 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd934020030 con 0x55d397b12520 2024-04-21T23:18:01.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.675+0000 7fd92ffff640 1 -- 172.21.15.44:0/3677178742 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd934018070 con 0x55d397b12520 2024-04-21T23:18:01.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.676+0000 7fd94ef82740 1 -- 172.21.15.44:0/3677178742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 msgr2=0x55d397b119b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.676+0000 7fd94ef82740 1 --2- 172.21.15.44:0/3677178742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 0x55d397b119b0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fd93400e7a0 tx=0x7fd93401e040 comp rx=0 tx=0).stop 2024-04-21T23:18:01.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.676+0000 7fd94ef82740 1 -- 172.21.15.44:0/3677178742 shutdown_connections 2024-04-21T23:18:01.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.676+0000 7fd94ef82740 1 --2- 172.21.15.44:0/3677178742 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 0x55d397b119b0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.676+0000 7fd94ef82740 1 --2- 172.21.15.44:0/3677178742 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d397b09940 0x55d397b051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.676+0000 7fd94ef82740 1 --2- 172.21.15.44:0/3677178742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b09020 0x55d397b02850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.676+0000 7fd94ef82740 1 -- 172.21.15.44:0/3677178742 >> 172.21.15.44:0/3677178742 conn(0x55d397b03750 msgr2=0x55d397b04ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:01.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.676+0000 7fd94ef82740 1 -- 172.21.15.44:0/3677178742 shutdown_connections 2024-04-21T23:18:01.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.676+0000 7fd94ef82740 1 -- 172.21.15.44:0/3677178742 wait complete. 2024-04-21T23:18:01.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.676+0000 7fd94ef82740 1 -- start start 2024-04-21T23:18:01.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.676+0000 7fd94ef82740 1 Processor -- start 2024-04-21T23:18:01.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.676+0000 7fd94ef82740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d397b09020 0x55d397b0c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.676+0000 7fd94ef82740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b09940 0x55d397b0c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd94ef82740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 0x55d397b0cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd94ef82740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441adb80 con 0x55d397b12520 2024-04-21T23:18:01.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd94ef82740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441aeff0 con 0x55d397b09020 2024-04-21T23:18:01.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd94ef82740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441ae250 con 0x55d397b09940 2024-04-21T23:18:01.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd94ef82740 10 client.0 _ll_register_callbacks cb 0x7fd940009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:01.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd94993b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 0x55d397b0cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd94993b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 0x55d397b0cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55304/0 (socket says 172.21.15.44:55304) 2024-04-21T23:18:01.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd94993b640 1 -- 172.21.15.44:0/3926458479 learned_addr learned my addr 172.21.15.44:0/3926458479 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:01.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd94993b640 1 --2- 172.21.15.44:0/3926458479 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d397b09020 0x55d397b0c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd94a13c640 1 --2- 172.21.15.44:0/3926458479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b09940 0x55d397b0c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd94993b640 1 -- 172.21.15.44:0/3926458479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b09940 msgr2=0x55d397b0c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:01.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd94993b640 1 --2- 172.21.15.44:0/3926458479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b09940 0x55d397b0c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd94993b640 1 -- 172.21.15.44:0/3926458479 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d397b09020 msgr2=0x55d397b0c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:01.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd94993b640 1 --2- 172.21.15.44:0/3926458479 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d397b09020 0x55d397b0c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd94993b640 1 -- 172.21.15.44:0/3926458479 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d397b87930 con 0x55d397b12520 2024-04-21T23:18:01.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd94a13c640 1 --2- 172.21.15.44:0/3926458479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b09940 0x55d397b0c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:01.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd94993b640 1 --2- 172.21.15.44:0/3926458479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 0x55d397b0cb50 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fd9340124f0 tx=0x7fd93400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=107fde3933adba0d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:01.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd92dffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:18:01.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd92dffb640 1 -- 172.21.15.44:0/3926458479 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93402f030 con 0x55d397b12520 2024-04-21T23:18:01.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd94ef82740 1 -- 172.21.15.44:0/3926458479 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd934020030 con 0x55d397b12520 2024-04-21T23:18:01.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.677+0000 7fd92dffb640 1 -- 172.21.15.44:0/3926458479 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7fd934004020 con 0x55d397b12520 2024-04-21T23:18:01.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.678+0000 7fd92dffb640 1 -- 172.21.15.44:0/3926458479 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd934018040 con 0x55d397b12520 2024-04-21T23:18:01.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.678+0000 7fd94ef82740 10 client.17646 fetch_fsmap learned FSMap version 31 2024-04-21T23:18:01.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.678+0000 7fd94ef82740 1 -- 172.21.15.44:0/3926458479 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d397b88b80 con 0x55d397b12520 2024-04-21T23:18:01.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.678+0000 7fd92dffb640 1 -- 172.21.15.44:0/3926458479 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd934020030 con 0x55d397b12520 2024-04-21T23:18:01.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.678+0000 7fd92dffb640 1 -- 172.21.15.44:0/3926458479 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7fd934029070 con 0x55d397b12520 2024-04-21T23:18:01.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.678+0000 7fd94ef82740 10 client.17646 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:18:01.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.678+0000 7fd94ef82740 10 client.17646 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:01.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.678+0000 7fd94ef82740 20 client.17646 populate_metadata read hostname 'smithi044' 2024-04-21T23:18:01.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.678+0000 7fd94ef82740 1 --2- 172.21.15.44:0/3926458479 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d397b89190 0x55d397b8b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:01.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.678+0000 7fd94ef82740 4 client.17646 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:18:01.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.678+0000 7fd94ef82740 1 -- 172.21.15.44:0/3926458479 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d397a74dc0 con 0x55d397b89190 2024-04-21T23:18:01.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.678+0000 7fd94a13c640 1 --2- 172.21.15.44:0/3926458479 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d397b89190 0x55d397b8b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:01.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.679+0000 7fd92dffb640 10 client.17646 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:01.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.679+0000 7fd94a13c640 1 --2- 172.21.15.44:0/3926458479 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d397b89190 0x55d397b8b580 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df94c664428a3c61 server_cookie=eec36c5084ee47d1 in_seq=0 out_seq=0 2024-04-21T23:18:01.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.679+0000 7fd92dffb640 1 -- 172.21.15.44:0/3926458479 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d397a74dc0 con 0x55d397b89190 2024-04-21T23:18:01.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.679+0000 7fd92dffb640 10 client.17646 handle_command_reply: tid=0 2024-04-21T23:18:01.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.679+0000 7fd94ef82740 1 client.17646 shutdown 2024-04-21T23:18:01.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.679+0000 7fd94ef82740 1 -- 172.21.15.44:0/3926458479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 msgr2=0x55d397b0cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.679+0000 7fd94ef82740 1 --2- 172.21.15.44:0/3926458479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 0x55d397b0cb50 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fd9340124f0 tx=0x7fd93400ed40 comp rx=0 tx=0).stop 2024-04-21T23:18:01.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.679+0000 7fd94ef82740 1 -- 172.21.15.44:0/3926458479 shutdown_connections 2024-04-21T23:18:01.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.679+0000 7fd94ef82740 1 --2- 172.21.15.44:0/3926458479 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55d397b89190 0x55d397b8b580 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.679+0000 7fd94ef82740 1 --2- 172.21.15.44:0/3926458479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d397b09940 0x55d397b0c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.679+0000 7fd94ef82740 1 --2- 172.21.15.44:0/3926458479 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55d397b12520 0x55d397b0cb50 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.679+0000 7fd92dffb640 0 client.17646 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:01.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.679+0000 7fd94ef82740 1 --2- 172.21.15.44:0/3926458479 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55d397b09020 0x55d397b0c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.679+0000 7fd94ef82740 1 -- 172.21.15.44:0/3926458479 >> 172.21.15.44:0/3926458479 conn(0x55d397b03750 msgr2=0x55d397b026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:01.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.679+0000 7fd94ef82740 1 -- 172.21.15.44:0/3926458479 shutdown_connections 2024-04-21T23:18:01.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.679+0000 7fd94ef82740 1 -- 172.21.15.44:0/3926458479 wait complete. 2024-04-21T23:18:01.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.679+0000 7fd94ef82740 20 client.17646 trim_cache size 0 max 16384 2024-04-21T23:18:01.690 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-21T23:18:01.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.680+0000 7fd94c3a0640 1 -- 172.21.15.44:0/3630968215 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd9300a2d80 msgr2=0x7fd9300a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.680+0000 7fd94c3a0640 1 --2- 172.21.15.44:0/3630968215 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd9300a2d80 0x7fd9300a5240 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd93c0092f0 tx=0x7fd93c090fe0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.680+0000 7fd94c3a0640 1 -- 172.21.15.44:0/3630968215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd944107710 msgr2=0x7fd94419c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:01.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.680+0000 7fd94c3a0640 1 --2- 172.21.15.44:0/3630968215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd944107710 0x7fd94419c8e0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fd93c00eaa0 tx=0x7fd93c00ef60 comp rx=0 tx=0).stop 2024-04-21T23:18:01.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.680+0000 7fd94c3a0640 1 -- 172.21.15.44:0/3630968215 shutdown_connections 2024-04-21T23:18:01.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.680+0000 7fd94c3a0640 1 --2- 172.21.15.44:0/3630968215 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd9300a2d80 0x7fd9300a5240 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.680+0000 7fd94c3a0640 1 --2- 172.21.15.44:0/3630968215 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd944107710 0x7fd94419c8e0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.680+0000 7fd94c3a0640 1 --2- 172.21.15.44:0/3630968215 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd9441020c0 0x7fd944198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.680+0000 7fd94c3a0640 1 --2- 172.21.15.44:0/3630968215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd944101700 0x7fd944198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:01.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.680+0000 7fd94c3a0640 1 -- 172.21.15.44:0/3630968215 >> 172.21.15.44:0/3630968215 conn(0x7fd9440fd1f0 msgr2=0x7fd9440fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:01.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.680+0000 7fd94c3a0640 1 -- 172.21.15.44:0/3630968215 shutdown_connections 2024-04-21T23:18:01.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:01.680+0000 7fd94c3a0640 1 -- 172.21.15.44:0/3630968215 wait complete. 2024-04-21T23:18:02.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:02 smithi044 ceph-mon[27934]: pgmap v289: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-04-21T23:18:02.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:02 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3693433111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:18:02.694 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:18:02.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:18:02.695 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:18:02.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:02 smithi069 ceph-mon[31816]: pgmap v289: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-04-21T23:18:02.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:02 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3693433111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:18:02.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:02 smithi055 ceph-mon[31117]: pgmap v289: 129 pgs: 129 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-04-21T23:18:02.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:02 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3693433111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:18:02.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.816+0000 7f9fd7fb8640 1 -- 172.21.15.44:0/3405338938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd010bec0 msgr2=0x7f9fd010e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:02.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.816+0000 7f9fd7fb8640 1 --2- 172.21.15.44:0/3405338938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd010bec0 0x7f9fd010e350 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8009ca0 tx=0x7f9fc802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.816+0000 7f9fd7fb8640 1 -- 172.21.15.44:0/3405338938 shutdown_connections 2024-04-21T23:18:02.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.816+0000 7f9fd7fb8640 1 --2- 172.21.15.44:0/3405338938 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd010bec0 0x7f9fd010e350 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.816+0000 7f9fd7fb8640 1 --2- 172.21.15.44:0/3405338938 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fd0103360 0x7f9fd010b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.816 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.816+0000 7f9fd7fb8640 1 --2- 172.21.15.44:0/3405338938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fd01029b0 0x7f9fd0102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.816+0000 7f9fd7fb8640 1 -- 172.21.15.44:0/3405338938 >> 172.21.15.44:0/3405338938 conn(0x7f9fd0077760 msgr2=0x7f9fd0077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:02.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.816+0000 7f9fd7fb8640 1 -- 172.21.15.44:0/3405338938 shutdown_connections 2024-04-21T23:18:02.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.816+0000 7f9fd7fb8640 1 -- 172.21.15.44:0/3405338938 wait complete. 2024-04-21T23:18:02.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.816+0000 7f9fd7fb8640 1 Processor -- start 2024-04-21T23:18:02.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.816+0000 7f9fd7fb8640 1 -- start start 2024-04-21T23:18:02.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fd7fb8640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd01029b0 0x7f9fd0198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fd7fb8640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fd010bec0 0x7f9fd01986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.819 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fd7fb8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fd019c9e0 0x7f9fd019ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fd7fb8640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd0111cb0 con 0x7f9fd01029b0 2024-04-21T23:18:02.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fd7fb8640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd0111b30 con 0x7f9fd010bec0 2024-04-21T23:18:02.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fd7fb8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd0111e30 con 0x7f9fd019c9e0 2024-04-21T23:18:02.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fd5d54640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd01029b0 0x7f9fd0198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fd5d54640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd01029b0 0x7f9fd0198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55322/0 (socket says 172.21.15.44:55322) 2024-04-21T23:18:02.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fd5d54640 1 -- 172.21.15.44:0/3977800643 learned_addr learned my addr 172.21.15.44:0/3977800643 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:02.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fd5553640 1 --2- 172.21.15.44:0/3977800643 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fd010bec0 0x7f9fd01986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.820 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fd6555640 1 --2- 172.21.15.44:0/3977800643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fd019c9e0 0x7f9fd019ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fd5d54640 1 -- 172.21.15.44:0/3977800643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fd019c9e0 msgr2=0x7f9fd019ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:02.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fd5d54640 1 --2- 172.21.15.44:0/3977800643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fd019c9e0 0x7f9fd019ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fd5d54640 1 -- 172.21.15.44:0/3977800643 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fd010bec0 msgr2=0x7f9fd01986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:02.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fd5d54640 1 --2- 172.21.15.44:0/3977800643 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fd010bec0 0x7f9fd01986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fd5d54640 1 -- 172.21.15.44:0/3977800643 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd019d4d0 con 0x7f9fd01029b0 2024-04-21T23:18:02.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fd5553640 1 --2- 172.21.15.44:0/3977800643 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fd010bec0 0x7f9fd01986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:02.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fd6555640 1 --2- 172.21.15.44:0/3977800643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fd019c9e0 0x7f9fd019ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:02.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fd5d54640 1 --2- 172.21.15.44:0/3977800643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd01029b0 0x7f9fd0198170 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc00d950 tx=0x7f9fcc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:02.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fc6ffd640 1 -- 172.21.15.44:0/3977800643 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc017070 con 0x7f9fd01029b0 2024-04-21T23:18:02.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fc6ffd640 1 -- 172.21.15.44:0/3977800643 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fcc009c30 con 0x7f9fd01029b0 2024-04-21T23:18:02.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fd7fb8640 1 -- 172.21.15.44:0/3977800643 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fd019d7c0 con 0x7f9fd01029b0 2024-04-21T23:18:02.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.817+0000 7f9fc6ffd640 1 -- 172.21.15.44:0/3977800643 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc00f6b0 con 0x7f9fd01029b0 2024-04-21T23:18:02.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.818+0000 7f9fd7fb8640 1 -- 172.21.15.44:0/3977800643 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fd01a4fb0 con 0x7f9fd01029b0 2024-04-21T23:18:02.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.820+0000 7f9fc6ffd640 1 -- 172.21.15.44:0/3977800643 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f9fcc0109d0 con 0x7f9fd01029b0 2024-04-21T23:18:02.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.820+0000 7f9fdab9d740 1 Processor -- start 2024-04-21T23:18:02.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.820+0000 7f9fdab9d740 1 -- start start 2024-04-21T23:18:02.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.821+0000 7f9fdab9d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 0x55955959a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.821+0000 7f9fdab9d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x55955959aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.821+0000 7f9fdab9d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 0x5595595a61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.821+0000 7f9fdab9d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5595594fe170 con 0x55955959aa40 2024-04-21T23:18:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.821+0000 7f9fdab9d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5595594a67c0 con 0x5595595a3db0 2024-04-21T23:18:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.821+0000 7f9fdab9d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5595594fa130 con 0x55955959b720 2024-04-21T23:18:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.821+0000 7f9fd5d54640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x55955959aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.821+0000 7f9fd5d54640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x55955959aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55328/0 (socket says 172.21.15.44:55328) 2024-04-21T23:18:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.821+0000 7f9fd5d54640 1 -- 172.21.15.44:0/3885132173 learned_addr learned my addr 172.21.15.44:0/3885132173 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.821+0000 7f9fc6ffd640 1 --2- 172.21.15.44:0/3977800643 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9fbc0a2d00 0x7f9fbc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.821+0000 7f9fd5553640 1 --2- 172.21.15.44:0/3885132173 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 0x5595595a61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.821+0000 7f9fd6555640 1 --2- 172.21.15.44:0/3885132173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 0x55955959a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.821+0000 7f9fd6555640 1 --2- 172.21.15.44:0/3977800643 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9fbc0a2d00 0x7f9fbc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.821+0000 7f9fd5d54640 1 -- 172.21.15.44:0/3885132173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 msgr2=0x55955959a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.821+0000 7f9fd5d54640 1 --2- 172.21.15.44:0/3885132173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 0x55955959a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.821+0000 7f9fd5d54640 1 -- 172.21.15.44:0/3885132173 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 msgr2=0x5595595a61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.821+0000 7f9fd5d54640 1 --2- 172.21.15.44:0/3885132173 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 0x5595595a61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.821+0000 7f9fc6ffd640 1 -- 172.21.15.44:0/3977800643 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7f9fcc0cf5e0 con 0x7f9fd01029b0 2024-04-21T23:18:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.821+0000 7f9fd5d54640 1 -- 172.21.15.44:0/3885132173 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595594b7e60 con 0x55955959aa40 2024-04-21T23:18:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.822+0000 7f9fd5d54640 1 --2- 172.21.15.44:0/3885132173 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x55955959aec0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc09b3b0 tx=0x7f9fcc065d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf91ed2e6cca527 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:02.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.822+0000 7f9fd6555640 1 --2- 172.21.15.44:0/3977800643 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9fbc0a2d00 0x7f9fbc0a51c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8037040 tx=0x7f9fc80095c0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:02.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.822+0000 7f9fbbfff640 1 -- 172.21.15.44:0/3885132173 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc020070 con 0x55955959aa40 2024-04-21T23:18:02.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.822+0000 7f9fbbfff640 1 -- 172.21.15.44:0/3885132173 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fcc097ea0 con 0x55955959aa40 2024-04-21T23:18:02.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.822+0000 7f9fbbfff640 1 -- 172.21.15.44:0/3885132173 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc099a40 con 0x55955959aa40 2024-04-21T23:18:02.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.822+0000 7f9fdab9d740 1 -- 172.21.15.44:0/3885132173 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 msgr2=0x55955959aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:02.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.822+0000 7f9fdab9d740 1 --2- 172.21.15.44:0/3885132173 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x55955959aec0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc09b3b0 tx=0x7f9fcc065d40 comp rx=0 tx=0).stop 2024-04-21T23:18:02.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.823+0000 7f9fdab9d740 1 -- 172.21.15.44:0/3885132173 shutdown_connections 2024-04-21T23:18:02.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.823+0000 7f9fdab9d740 1 --2- 172.21.15.44:0/3885132173 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 0x5595595a61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.823+0000 7f9fdab9d740 1 --2- 172.21.15.44:0/3885132173 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x55955959aec0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.823+0000 7f9fdab9d740 1 --2- 172.21.15.44:0/3885132173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 0x55955959a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.823+0000 7f9fdab9d740 1 -- 172.21.15.44:0/3885132173 >> 172.21.15.44:0/3885132173 conn(0x5595595614f0 msgr2=0x559559595490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:02.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.823+0000 7f9fdab9d740 1 -- 172.21.15.44:0/3885132173 shutdown_connections 2024-04-21T23:18:02.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.823+0000 7f9fdab9d740 1 -- 172.21.15.44:0/3885132173 wait complete. 2024-04-21T23:18:02.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.823+0000 7f9fdab9d740 1 -- start start 2024-04-21T23:18:02.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.823+0000 7f9fdab9d740 1 Processor -- start 2024-04-21T23:18:02.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fdab9d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x5595595a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fdab9d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 0x5595595a1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.833 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fdab9d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 0x5595595a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fdab9d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x5595595a6b60 con 0x55955959aa40 2024-04-21T23:18:02.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fdab9d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x5595594b9d40 con 0x5595595a3db0 2024-04-21T23:18:02.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fdab9d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5595595a6ce0 con 0x55955959b720 2024-04-21T23:18:02.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fdab9d740 10 client.0 _ll_register_callbacks cb 0x5595594f5820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:02.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fd5d54640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x5595595a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fd5d54640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x5595595a0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55344/0 (socket says 172.21.15.44:55344) 2024-04-21T23:18:02.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fd5d54640 1 -- 172.21.15.44:0/3128458741 learned_addr learned my addr 172.21.15.44:0/3128458741 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:02.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fd6555640 1 --2- 172.21.15.44:0/3128458741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 0x5595595a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fd5553640 1 --2- 172.21.15.44:0/3128458741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 0x5595595a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fd5d54640 1 -- 172.21.15.44:0/3128458741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 msgr2=0x5595595a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:02.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fd5d54640 1 --2- 172.21.15.44:0/3128458741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 0x5595595a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fd5d54640 1 -- 172.21.15.44:0/3128458741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 msgr2=0x5595595a17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:02.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fd5d54640 1 --2- 172.21.15.44:0/3128458741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 0x5595595a17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fd5d54640 1 -- 172.21.15.44:0/3128458741 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595592da7e0 con 0x55955959aa40 2024-04-21T23:18:02.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fd5d54640 1 --2- 172.21.15.44:0/3128458741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x5595595a0d40 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc0058c0 tx=0x7f9fcc09b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=516a8a2e0d82d568 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:02.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fb97fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:18:02.835 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fb97fa640 1 -- 172.21.15.44:0/3128458741 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc020070 con 0x55955959aa40 2024-04-21T23:18:02.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fb97fa640 1 -- 172.21.15.44:0/3128458741 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7f9fcc0cf170 con 0x55955959aa40 2024-04-21T23:18:02.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fdab9d740 1 -- 172.21.15.44:0/3128458741 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5595594d2450 con 0x55955959aa40 2024-04-21T23:18:02.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fb97fa640 1 -- 172.21.15.44:0/3128458741 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9fcc0ccd70 con 0x55955959aa40 2024-04-21T23:18:02.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fdab9d740 10 client.17658 fetch_fsmap learned FSMap version 31 2024-04-21T23:18:02.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.824+0000 7f9fdab9d740 1 -- 172.21.15.44:0/3128458741 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55955933af70 con 0x55955959aa40 2024-04-21T23:18:02.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.825+0000 7f9fb97fa640 1 -- 172.21.15.44:0/3128458741 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5595594d2450 con 0x55955959aa40 2024-04-21T23:18:02.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.825+0000 7f9fb97fa640 1 -- 172.21.15.44:0/3128458741 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f9fcc0bb070 con 0x55955959aa40 2024-04-21T23:18:02.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.825+0000 7f9fdab9d740 10 client.17658 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:18:02.836 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.825+0000 7f9fdab9d740 10 client.17658 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:02.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.825+0000 7f9fdab9d740 20 client.17658 populate_metadata read hostname 'smithi044' 2024-04-21T23:18:02.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.825+0000 7f9fdab9d740 1 --2- 172.21.15.44:0/3128458741 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5595595b3c40 0x5595595b6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.825+0000 7f9fdab9d740 4 client.17658 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:18:02.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.825+0000 7f9fdab9d740 1 -- 172.21.15.44:0/3128458741 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55955950b6b0 con 0x5595595b3c40 2024-04-21T23:18:02.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.825+0000 7f9fd5553640 1 --2- 172.21.15.44:0/3128458741 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5595595b3c40 0x5595595b6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.826+0000 7f9fd5553640 1 --2- 172.21.15.44:0/3128458741 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5595595b3c40 0x5595595b6030 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f32079a9e5a8f4aa server_cookie=9653cbf35598f4a1 in_seq=0 out_seq=0 2024-04-21T23:18:02.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.826+0000 7f9fb97fa640 10 client.17658 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:02.837 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.826+0000 7f9fb97fa640 1 -- 172.21.15.44:0/3128458741 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55955950b6b0 con 0x5595595b3c40 2024-04-21T23:18:02.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.826+0000 7f9fb97fa640 10 client.17658 handle_command_reply: tid=0 2024-04-21T23:18:02.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.826+0000 7f9fdab9d740 1 client.17658 shutdown 2024-04-21T23:18:02.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.826+0000 7f9fdab9d740 1 -- 172.21.15.44:0/3128458741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 msgr2=0x5595595a0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:02.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.826+0000 7f9fdab9d740 1 --2- 172.21.15.44:0/3128458741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x5595595a0d40 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc0058c0 tx=0x7f9fcc09b730 comp rx=0 tx=0).stop 2024-04-21T23:18:02.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.827+0000 7f9fdab9d740 1 -- 172.21.15.44:0/3128458741 shutdown_connections 2024-04-21T23:18:02.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.827+0000 7f9fdab9d740 1 --2- 172.21.15.44:0/3128458741 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x5595595b3c40 0x5595595b6030 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.827+0000 7f9fdab9d740 1 --2- 172.21.15.44:0/3128458741 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 0x5595595a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.838 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.827+0000 7f9fdab9d740 1 --2- 172.21.15.44:0/3128458741 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 0x5595595a17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.827+0000 7f9fb97fa640 0 client.17658 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:02.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.827+0000 7f9fdab9d740 1 --2- 172.21.15.44:0/3128458741 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x5595595a0d40 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.827+0000 7f9fdab9d740 1 -- 172.21.15.44:0/3128458741 >> 172.21.15.44:0/3128458741 conn(0x5595595614f0 msgr2=0x5595595971e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:02.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.827+0000 7f9fdab9d740 1 -- 172.21.15.44:0/3128458741 shutdown_connections 2024-04-21T23:18:02.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.827+0000 7f9fdab9d740 1 -- 172.21.15.44:0/3128458741 wait complete. 2024-04-21T23:18:02.839 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.827+0000 7f9fdab9d740 20 client.17658 trim_cache size 0 max 16384 2024-04-21T23:18:02.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.845+0000 7f9fdab9d740 1 Processor -- start 2024-04-21T23:18:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.845+0000 7f9fdab9d740 1 -- start start 2024-04-21T23:18:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.845+0000 7f9fdab9d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x5595595a3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.845+0000 7f9fdab9d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 0x5595595afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.845+0000 7f9fdab9d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 0x5595595bb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.845+0000 7f9fdab9d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd01acd20 con 0x55955959aa40 2024-04-21T23:18:02.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.845+0000 7f9fdab9d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd01af580 con 0x5595595a3db0 2024-04-21T23:18:02.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.845+0000 7f9fdab9d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd01ad370 con 0x55955959b720 2024-04-21T23:18:02.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.845+0000 7f9fd5d54640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x5595595a3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.845+0000 7f9fd5d54640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x5595595a3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55360/0 (socket says 172.21.15.44:55360) 2024-04-21T23:18:02.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.845+0000 7f9fd5d54640 1 -- 172.21.15.44:0/3603339597 learned_addr learned my addr 172.21.15.44:0/3603339597 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:02.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.846+0000 7f9fd5553640 1 --2- 172.21.15.44:0/3603339597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 0x5595595afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.846+0000 7f9fd6555640 1 --2- 172.21.15.44:0/3603339597 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 0x5595595bb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.846+0000 7f9fd5d54640 1 -- 172.21.15.44:0/3603339597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 msgr2=0x5595595afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:02.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.846+0000 7f9fd5d54640 1 --2- 172.21.15.44:0/3603339597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 0x5595595afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.846+0000 7f9fd5d54640 1 -- 172.21.15.44:0/3603339597 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 msgr2=0x5595595bb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:02.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.846+0000 7f9fd5d54640 1 --2- 172.21.15.44:0/3603339597 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 0x5595595bb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.846+0000 7f9fd5d54640 1 -- 172.21.15.44:0/3603339597 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595591d2170 con 0x55955959aa40 2024-04-21T23:18:02.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.846+0000 7f9fd5553640 1 --2- 172.21.15.44:0/3603339597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 0x5595595afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:02.851 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.846+0000 7f9fd6555640 1 --2- 172.21.15.44:0/3603339597 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 0x5595595bb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:02.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.846+0000 7f9fd5d54640 1 --2- 172.21.15.44:0/3603339597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x5595595a3470 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc09b640 tx=0x7f9fcc09fa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3dc7fdcb32d8082 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:02.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.846+0000 7f9fbb7fe640 1 -- 172.21.15.44:0/3603339597 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc020070 con 0x55955959aa40 2024-04-21T23:18:02.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.846+0000 7f9fbb7fe640 1 -- 172.21.15.44:0/3603339597 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fcc0a4320 con 0x55955959aa40 2024-04-21T23:18:02.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.846+0000 7f9fbb7fe640 1 -- 172.21.15.44:0/3603339597 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc09ad60 con 0x55955959aa40 2024-04-21T23:18:02.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.847+0000 7f9fdab9d740 1 -- 172.21.15.44:0/3603339597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 msgr2=0x5595595a3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:02.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.847+0000 7f9fdab9d740 1 --2- 172.21.15.44:0/3603339597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x5595595a3470 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc09b640 tx=0x7f9fcc09fa60 comp rx=0 tx=0).stop 2024-04-21T23:18:02.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.847+0000 7f9fdab9d740 1 -- 172.21.15.44:0/3603339597 shutdown_connections 2024-04-21T23:18:02.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.847+0000 7f9fdab9d740 1 --2- 172.21.15.44:0/3603339597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 0x5595595afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.847+0000 7f9fdab9d740 1 --2- 172.21.15.44:0/3603339597 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 0x5595595bb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.847+0000 7f9fdab9d740 1 --2- 172.21.15.44:0/3603339597 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x5595595a3470 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.847+0000 7f9fdab9d740 1 -- 172.21.15.44:0/3603339597 >> 172.21.15.44:0/3603339597 conn(0x5595595614f0 msgr2=0x5595595944f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:02.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.847+0000 7f9fdab9d740 1 -- 172.21.15.44:0/3603339597 shutdown_connections 2024-04-21T23:18:02.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.847+0000 7f9fdab9d740 1 -- 172.21.15.44:0/3603339597 wait complete. 2024-04-21T23:18:02.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.847+0000 7f9fdab9d740 1 -- start start 2024-04-21T23:18:02.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.847+0000 7f9fdab9d740 1 Processor -- start 2024-04-21T23:18:02.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fdab9d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x5595595abf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fdab9d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 0x5595595ac4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fdab9d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 0x5595595a8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fdab9d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd01ada40 con 0x55955959aa40 2024-04-21T23:18:02.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fdab9d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd01aeeb0 con 0x5595595a3db0 2024-04-21T23:18:02.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fdab9d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd01ae110 con 0x55955959b720 2024-04-21T23:18:02.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fdab9d740 10 client.0 _ll_register_callbacks cb 0x7f9fc0008310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:02.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fd5d54640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x5595595abf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fd5d54640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x5595595abf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55366/0 (socket says 172.21.15.44:55366) 2024-04-21T23:18:02.854 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fd5d54640 1 -- 172.21.15.44:0/1841784289 learned_addr learned my addr 172.21.15.44:0/1841784289 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:02.855 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fd6555640 1 --2- 172.21.15.44:0/1841784289 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 0x5595595a8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fd5553640 1 --2- 172.21.15.44:0/1841784289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 0x5595595ac4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fd5d54640 1 -- 172.21.15.44:0/1841784289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 msgr2=0x5595595ac4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:02.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fd5d54640 1 --2- 172.21.15.44:0/1841784289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 0x5595595ac4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fd5d54640 1 -- 172.21.15.44:0/1841784289 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 msgr2=0x5595595a8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:02.858 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fd5d54640 1 --2- 172.21.15.44:0/1841784289 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 0x5595595a8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fd5d54640 1 -- 172.21.15.44:0/1841784289 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595595a8d30 con 0x55955959aa40 2024-04-21T23:18:02.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fd6555640 1 --2- 172.21.15.44:0/1841784289 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 0x5595595a8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:02.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fd5d54640 1 --2- 172.21.15.44:0/1841784289 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x5595595abf60 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc098080 tx=0x7f9fcc0999b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9355a06d67dacad3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:02.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fb97fa640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:18:02.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fb97fa640 1 -- 172.21.15.44:0/1841784289 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc020040 con 0x55955959aa40 2024-04-21T23:18:02.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fb97fa640 1 -- 172.21.15.44:0/1841784289 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7f9fcc097400 con 0x55955959aa40 2024-04-21T23:18:02.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fdab9d740 1 -- 172.21.15.44:0/1841784289 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fcc0a4320 con 0x55955959aa40 2024-04-21T23:18:02.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fb97fa640 1 -- 172.21.15.44:0/1841784289 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9fcc009de0 con 0x55955959aa40 2024-04-21T23:18:02.859 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fdab9d740 10 client.17664 fetch_fsmap learned FSMap version 31 2024-04-21T23:18:02.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.848+0000 7f9fdab9d740 1 -- 172.21.15.44:0/1841784289 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5595595a8d30 con 0x55955959aa40 2024-04-21T23:18:02.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.849+0000 7f9fb97fa640 1 -- 172.21.15.44:0/1841784289 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fcc0a4320 con 0x55955959aa40 2024-04-21T23:18:02.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.849+0000 7f9fb97fa640 1 -- 172.21.15.44:0/1841784289 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f9fcc0bb040 con 0x55955959aa40 2024-04-21T23:18:02.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.849+0000 7f9fdab9d740 10 client.17664 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:18:02.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.849+0000 7f9fdab9d740 10 client.17664 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:02.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.849+0000 7f9fdab9d740 20 client.17664 populate_metadata read hostname 'smithi044' 2024-04-21T23:18:02.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.849+0000 7f9fdab9d740 1 --2- 172.21.15.44:0/1841784289 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55955961adc0 0x55955961d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.860 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.849+0000 7f9fdab9d740 4 client.17664 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:18:02.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.849+0000 7f9fdab9d740 1 -- 172.21.15.44:0/1841784289 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559559505a00 con 0x55955961adc0 2024-04-21T23:18:02.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.849+0000 7f9fd5553640 1 --2- 172.21.15.44:0/1841784289 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55955961adc0 0x55955961d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.849+0000 7f9fd5553640 1 --2- 172.21.15.44:0/1841784289 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55955961adc0 0x55955961d1b0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c72cbc3177566da server_cookie=193968252ae3a6de in_seq=0 out_seq=0 2024-04-21T23:18:02.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.849+0000 7f9fb97fa640 10 client.17664 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:02.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.850+0000 7f9fb97fa640 1 -- 172.21.15.44:0/1841784289 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559559505a00 con 0x55955961adc0 2024-04-21T23:18:02.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.850+0000 7f9fb97fa640 10 client.17664 handle_command_reply: tid=0 2024-04-21T23:18:02.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.850+0000 7f9fdab9d740 1 client.17664 shutdown 2024-04-21T23:18:02.861 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.851+0000 7f9fdab9d740 1 -- 172.21.15.44:0/1841784289 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 msgr2=0x5595595abf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:02.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.851+0000 7f9fdab9d740 1 --2- 172.21.15.44:0/1841784289 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x5595595abf60 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc098080 tx=0x7f9fcc0999b0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.851+0000 7f9fdab9d740 1 -- 172.21.15.44:0/1841784289 shutdown_connections 2024-04-21T23:18:02.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.851+0000 7f9fdab9d740 1 --2- 172.21.15.44:0/1841784289 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55955961adc0 0x55955961d1b0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.851+0000 7f9fdab9d740 1 --2- 172.21.15.44:0/1841784289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55955959b720 0x5595595ac4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.851+0000 7f9fdab9d740 1 --2- 172.21.15.44:0/1841784289 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x5595595a3db0 0x5595595a8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.851+0000 7f9fb97fa640 0 client.17664 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:02.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.851+0000 7f9fdab9d740 1 --2- 172.21.15.44:0/1841784289 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55955959aa40 0x5595595abf60 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.851+0000 7f9fdab9d740 1 -- 172.21.15.44:0/1841784289 >> 172.21.15.44:0/1841784289 conn(0x5595595614f0 msgr2=0x5595595a0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:02.862 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.851+0000 7f9fdab9d740 1 -- 172.21.15.44:0/1841784289 shutdown_connections 2024-04-21T23:18:02.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.851+0000 7f9fdab9d740 1 -- 172.21.15.44:0/1841784289 wait complete. 2024-04-21T23:18:02.863 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.851+0000 7f9fdab9d740 20 client.17664 trim_cache size 0 max 16384 2024-04-21T23:18:02.863 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:18:02.863 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-21T23:18:02.863 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "eaf8af63-1e96-4c6e-abc2-5e39143d25fe", 2024-04-21T23:18:02.863 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-21T23:18:02.864 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:18:02.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.851+0000 7f9fd7fb8640 1 -- 172.21.15.44:0/3977800643 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9fbc0a2d00 msgr2=0x7f9fbc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:02.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.851+0000 7f9fd7fb8640 1 --2- 172.21.15.44:0/3977800643 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9fbc0a2d00 0x7f9fbc0a51c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8037040 tx=0x7f9fc80095c0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.851+0000 7f9fd7fb8640 1 -- 172.21.15.44:0/3977800643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd01029b0 msgr2=0x7f9fd0198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:02.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.851+0000 7f9fd7fb8640 1 --2- 172.21.15.44:0/3977800643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd01029b0 0x7f9fd0198170 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc00d950 tx=0x7f9fcc00de10 comp rx=0 tx=0).stop 2024-04-21T23:18:02.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.852+0000 7f9fd7fb8640 1 -- 172.21.15.44:0/3977800643 shutdown_connections 2024-04-21T23:18:02.865 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.852+0000 7f9fd7fb8640 1 --2- 172.21.15.44:0/3977800643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fd019c9e0 0x7f9fd019ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.852+0000 7f9fd7fb8640 1 --2- 172.21.15.44:0/3977800643 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9fbc0a2d00 0x7f9fbc0a51c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.852+0000 7f9fd7fb8640 1 --2- 172.21.15.44:0/3977800643 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9fd010bec0 0x7f9fd01986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.852+0000 7f9fd7fb8640 1 --2- 172.21.15.44:0/3977800643 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9fd01029b0 0x7f9fd0198170 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.852+0000 7f9fd7fb8640 1 -- 172.21.15.44:0/3977800643 >> 172.21.15.44:0/3977800643 conn(0x7f9fd0077760 msgr2=0x7f9fd00fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:02.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.852+0000 7f9fd7fb8640 1 -- 172.21.15.44:0/3977800643 shutdown_connections 2024-04-21T23:18:02.866 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.852+0000 7f9fd7fb8640 1 -- 172.21.15.44:0/3977800643 wait complete. 2024-04-21T23:18:02.867 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:18:02.867 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:18:02.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.972+0000 7fcae4a40640 1 -- 172.21.15.44:0/2382690647 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcae0100610 msgr2=0x7fcae0100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.972+0000 7fcae4a40640 1 --2- 172.21.15.44:0/2382690647 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcae0100610 0x7fcae0100a90 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fcad4009930 tx=0x7fcad402bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.972+0000 7fcae4a40640 1 -- 172.21.15.44:0/2382690647 shutdown_connections 2024-04-21T23:18:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.972+0000 7fcae4a40640 1 --2- 172.21.15.44:0/2382690647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcae0100fd0 0x7fcae010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.972+0000 7fcae4a40640 1 --2- 172.21.15.44:0/2382690647 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcae0100610 0x7fcae0100a90 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.972+0000 7fcae4a40640 1 --2- 172.21.15.44:0/2382690647 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcae01066b0 0x7fcae0106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.972+0000 7fcae4a40640 1 -- 172.21.15.44:0/2382690647 >> 172.21.15.44:0/2382690647 conn(0x7fcae00777f0 msgr2=0x7fcae0077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:02.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.972+0000 7fcae4a40640 1 -- 172.21.15.44:0/2382690647 shutdown_connections 2024-04-21T23:18:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.972+0000 7fcae4a40640 1 -- 172.21.15.44:0/2382690647 wait complete. 2024-04-21T23:18:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.972+0000 7fcae4a40640 1 Processor -- start 2024-04-21T23:18:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.973+0000 7fcae4a40640 1 -- start start 2024-04-21T23:18:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.973+0000 7fcae4a40640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcae0100610 0x7fcae01980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.973+0000 7fcae4a40640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcae0100fd0 0x7fcae01985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.973+0000 7fcae4a40640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcae01066b0 0x7fcae019c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.973+0000 7fcae4a40640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcae0110a50 con 0x7fcae01066b0 2024-04-21T23:18:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.973+0000 7fcae4a40640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcae01108d0 con 0x7fcae0100610 2024-04-21T23:18:02.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.973+0000 7fcae4a40640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcae0110bd0 con 0x7fcae0100fd0 2024-04-21T23:18:02.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.973+0000 7fcaded9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcae01066b0 0x7fcae019c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.973+0000 7fcaded9d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcae01066b0 0x7fcae019c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55374/0 (socket says 172.21.15.44:55374) 2024-04-21T23:18:02.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.973+0000 7fcaded9d640 1 -- 172.21.15.44:0/3234790143 learned_addr learned my addr 172.21.15.44:0/3234790143 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:02.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.973+0000 7fcaddd9b640 1 --2- 172.21.15.44:0/3234790143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcae0100fd0 0x7fcae01985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.973+0000 7fcade59c640 1 --2- 172.21.15.44:0/3234790143 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcae0100610 0x7fcae01980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.973+0000 7fcaded9d640 1 -- 172.21.15.44:0/3234790143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcae0100fd0 msgr2=0x7fcae01985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:02.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.973+0000 7fcaded9d640 1 --2- 172.21.15.44:0/3234790143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcae0100fd0 0x7fcae01985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.973+0000 7fcaded9d640 1 -- 172.21.15.44:0/3234790143 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcae0100610 msgr2=0x7fcae01980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:02.976 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.973+0000 7fcaded9d640 1 --2- 172.21.15.44:0/3234790143 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcae0100610 0x7fcae01980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.973+0000 7fcaded9d640 1 -- 172.21.15.44:0/3234790143 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae019d040 con 0x7fcae01066b0 2024-04-21T23:18:02.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.973+0000 7fcade59c640 1 --2- 172.21.15.44:0/3234790143 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcae0100610 0x7fcae01980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:02.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.973+0000 7fcaded9d640 1 --2- 172.21.15.44:0/3234790143 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcae01066b0 0x7fcae019c910 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fcad000e960 tx=0x7fcad000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:02.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.973+0000 7fcacf7fe640 1 -- 172.21.15.44:0/3234790143 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad0004d20 con 0x7fcae01066b0 2024-04-21T23:18:02.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.974+0000 7fcacf7fe640 1 -- 172.21.15.44:0/3234790143 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcad0005020 con 0x7fcae01066b0 2024-04-21T23:18:02.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.974+0000 7fcae4a40640 1 -- 172.21.15.44:0/3234790143 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcae019d330 con 0x7fcae01066b0 2024-04-21T23:18:02.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.974+0000 7fcacf7fe640 1 -- 172.21.15.44:0/3234790143 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad00116d0 con 0x7fcae01066b0 2024-04-21T23:18:02.977 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.974+0000 7fcae4a40640 1 -- 172.21.15.44:0/3234790143 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcae01a4bc0 con 0x7fcae01066b0 2024-04-21T23:18:02.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.976+0000 7fcacf7fe640 1 -- 172.21.15.44:0/3234790143 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fcad0005560 con 0x7fcae01066b0 2024-04-21T23:18:02.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.976+0000 7fcae762d740 1 Processor -- start 2024-04-21T23:18:02.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.976+0000 7fcae762d740 1 -- start start 2024-04-21T23:18:02.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.976+0000 7fcae762d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56329a078f30 0x56329a079310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.976+0000 7fcae762d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56329a079850 0x56329a081ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.976+0000 7fcae762d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a082430 0x56329a084820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.976+0000 7fcae762d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x563299fdd560 con 0x56329a082430 2024-04-21T23:18:02.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.976+0000 7fcae762d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x563299f85730 con 0x56329a078f30 2024-04-21T23:18:02.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.976+0000 7fcae762d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563299fd9520 con 0x56329a079850 2024-04-21T23:18:02.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.977+0000 7fcaddd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a082430 0x56329a084820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.977+0000 7fcacf7fe640 1 --2- 172.21.15.44:0/3234790143 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcac80a2d00 0x7fcac80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.977+0000 7fcaddd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a082430 0x56329a084820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55380/0 (socket says 172.21.15.44:55380) 2024-04-21T23:18:02.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.977+0000 7fcaddd9b640 1 -- 172.21.15.44:0/1425390823 learned_addr learned my addr 172.21.15.44:0/1425390823 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:02.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.977+0000 7fcade59c640 1 --2- 172.21.15.44:0/1425390823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56329a079850 0x56329a081ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.977+0000 7fcaddd9b640 1 --2- 172.21.15.44:0/1425390823 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56329a078f30 0x56329a079310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.977+0000 7fcaded9d640 1 --2- 172.21.15.44:0/3234790143 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcac80a2d00 0x7fcac80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.977+0000 7fcacf7fe640 1 -- 172.21.15.44:0/3234790143 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7fcad00cf9d0 con 0x7fcae01066b0 2024-04-21T23:18:02.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.977+0000 7fcaddd9b640 1 -- 172.21.15.44:0/1425390823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56329a079850 msgr2=0x56329a081ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:02.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.977+0000 7fcaddd9b640 1 --2- 172.21.15.44:0/1425390823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56329a079850 0x56329a081ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.977+0000 7fcaddd9b640 1 -- 172.21.15.44:0/1425390823 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56329a078f30 msgr2=0x56329a079310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:02.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.977+0000 7fcaddd9b640 1 --2- 172.21.15.44:0/1425390823 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56329a078f30 0x56329a079310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.977+0000 7fcaddd9b640 1 -- 172.21.15.44:0/1425390823 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563299dbb7d0 con 0x56329a082430 2024-04-21T23:18:02.986 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.977+0000 7fcaddd9b640 1 --2- 172.21.15.44:0/1425390823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a082430 0x56329a084820 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fcad40096c0 tx=0x7fcad4004e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c538780867c22519 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:02.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.977+0000 7fcaded9d640 1 --2- 172.21.15.44:0/3234790143 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcac80a2d00 0x7fcac80a51c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcad009b800 tx=0x7fcad0066bd0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:02.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.977+0000 7fcaccff9640 1 -- 172.21.15.44:0/1425390823 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad4004230 con 0x56329a082430 2024-04-21T23:18:02.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.977+0000 7fcaccff9640 1 -- 172.21.15.44:0/1425390823 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcad402cce0 con 0x56329a082430 2024-04-21T23:18:02.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.977+0000 7fcaccff9640 1 -- 172.21.15.44:0/1425390823 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad4005500 con 0x56329a082430 2024-04-21T23:18:02.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.978+0000 7fcae762d740 1 -- 172.21.15.44:0/1425390823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a082430 msgr2=0x56329a084820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:02.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.978+0000 7fcae762d740 1 --2- 172.21.15.44:0/1425390823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a082430 0x56329a084820 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fcad40096c0 tx=0x7fcad4004e70 comp rx=0 tx=0).stop 2024-04-21T23:18:02.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.978+0000 7fcae762d740 1 -- 172.21.15.44:0/1425390823 shutdown_connections 2024-04-21T23:18:02.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.978+0000 7fcae762d740 1 --2- 172.21.15.44:0/1425390823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56329a079850 0x56329a081ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.987 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.978+0000 7fcae762d740 1 --2- 172.21.15.44:0/1425390823 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a082430 0x56329a084820 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.978+0000 7fcae762d740 1 --2- 172.21.15.44:0/1425390823 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56329a078f30 0x56329a079310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.978+0000 7fcae762d740 1 -- 172.21.15.44:0/1425390823 >> 172.21.15.44:0/1425390823 conn(0x56329a073660 msgr2=0x56329a073a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:02.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.978+0000 7fcae762d740 1 -- 172.21.15.44:0/1425390823 shutdown_connections 2024-04-21T23:18:02.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.978+0000 7fcae762d740 1 -- 172.21.15.44:0/1425390823 wait complete. 2024-04-21T23:18:02.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.978+0000 7fcae762d740 1 -- start start 2024-04-21T23:18:02.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.979+0000 7fcae762d740 1 Processor -- start 2024-04-21T23:18:02.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.979+0000 7fcae762d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a078f30 0x56329a07f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.988 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.979+0000 7fcae762d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56329a079850 0x56329a07f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.979+0000 7fcae762d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56329a082430 0x56329a07fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.979+0000 7fcae762d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56329a0851c0 con 0x56329a078f30 2024-04-21T23:18:02.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.979+0000 7fcae762d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x563299f98e00 con 0x56329a079850 2024-04-21T23:18:02.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.979+0000 7fcae762d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56329a085340 con 0x56329a082430 2024-04-21T23:18:02.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.979+0000 7fcae762d740 10 client.0 _ll_register_callbacks cb 0x563299fd3eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:02.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.979+0000 7fcaddd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a078f30 0x56329a07f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.989 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.979+0000 7fcaddd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a078f30 0x56329a07f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55394/0 (socket says 172.21.15.44:55394) 2024-04-21T23:18:02.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.979+0000 7fcaddd9b640 1 -- 172.21.15.44:0/1096731407 learned_addr learned my addr 172.21.15.44:0/1096731407 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:02.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.979+0000 7fcade59c640 1 --2- 172.21.15.44:0/1096731407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56329a079850 0x56329a07f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.979+0000 7fcaddd9b640 1 -- 172.21.15.44:0/1096731407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56329a082430 msgr2=0x56329a07fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:02.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.979+0000 7fcaddd9b640 1 --2- 172.21.15.44:0/1096731407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56329a082430 0x56329a07fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.979+0000 7fcaddd9b640 1 -- 172.21.15.44:0/1096731407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56329a079850 msgr2=0x56329a07f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:02.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.979+0000 7fcaddd9b640 1 --2- 172.21.15.44:0/1096731407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56329a079850 0x56329a07f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.979+0000 7fcaddd9b640 1 -- 172.21.15.44:0/1096731407 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563299cf13f0 con 0x56329a078f30 2024-04-21T23:18:02.990 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.979+0000 7fcaddd9b640 1 --2- 172.21.15.44:0/1096731407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a078f30 0x56329a07f450 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fcad4036040 tx=0x7fcad40058f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1c368f461353e4d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.979+0000 7fcac5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:18:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.980+0000 7fcac5ffb640 1 -- 172.21.15.44:0/1096731407 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad4004980 con 0x56329a078f30 2024-04-21T23:18:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.980+0000 7fcac5ffb640 1 -- 172.21.15.44:0/1096731407 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7fcad4004b20 con 0x56329a078f30 2024-04-21T23:18:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.980+0000 7fcae762d740 1 -- 172.21.15.44:0/1096731407 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563299feaaa0 con 0x56329a078f30 2024-04-21T23:18:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.980+0000 7fcac5ffb640 1 -- 172.21.15.44:0/1096731407 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcad4059050 con 0x56329a078f30 2024-04-21T23:18:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.980+0000 7fcae762d740 10 client.17676 fetch_fsmap learned FSMap version 31 2024-04-21T23:18:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.980+0000 7fcae762d740 1 -- 172.21.15.44:0/1096731407 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563299cb10f0 con 0x56329a078f30 2024-04-21T23:18:02.991 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.980+0000 7fcac5ffb640 1 -- 172.21.15.44:0/1096731407 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563299feaaa0 con 0x56329a078f30 2024-04-21T23:18:02.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.980+0000 7fcac5ffb640 1 -- 172.21.15.44:0/1096731407 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7fcad404b4b0 con 0x56329a078f30 2024-04-21T23:18:02.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.980+0000 7fcae762d740 10 client.17676 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:18:02.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.980+0000 7fcae762d740 10 client.17676 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:02.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.980+0000 7fcae762d740 20 client.17676 populate_metadata read hostname 'smithi044' 2024-04-21T23:18:02.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.980+0000 7fcae762d740 1 --2- 172.21.15.44:0/1096731407 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56329a0925f0 0x56329a0949e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:02.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.980+0000 7fcae762d740 4 client.17676 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:18:02.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.980+0000 7fcae762d740 1 -- 172.21.15.44:0/1096731407 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563299fe4df0 con 0x56329a0925f0 2024-04-21T23:18:02.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.980+0000 7fcade59c640 1 --2- 172.21.15.44:0/1096731407 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56329a0925f0 0x56329a0949e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:02.992 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.981+0000 7fcade59c640 1 --2- 172.21.15.44:0/1096731407 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56329a0925f0 0x56329a0949e0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60ead936b10b58e9 server_cookie=ad5058beb6a5e188 in_seq=0 out_seq=0 2024-04-21T23:18:02.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.981+0000 7fcac5ffb640 10 client.17676 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:02.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.982+0000 7fcac5ffb640 1 -- 172.21.15.44:0/1096731407 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563299fe4df0 con 0x56329a0925f0 2024-04-21T23:18:02.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.982+0000 7fcac5ffb640 10 client.17676 handle_command_reply: tid=0 2024-04-21T23:18:02.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.982+0000 7fcae762d740 1 client.17676 shutdown 2024-04-21T23:18:02.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.982+0000 7fcae762d740 1 -- 172.21.15.44:0/1096731407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a078f30 msgr2=0x56329a07f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:02.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.982+0000 7fcae762d740 1 --2- 172.21.15.44:0/1096731407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a078f30 0x56329a07f450 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fcad4036040 tx=0x7fcad40058f0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.982+0000 7fcae762d740 1 -- 172.21.15.44:0/1096731407 shutdown_connections 2024-04-21T23:18:02.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.982+0000 7fcae762d740 1 --2- 172.21.15.44:0/1096731407 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56329a0925f0 0x56329a0949e0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.993 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.982+0000 7fcae762d740 1 --2- 172.21.15.44:0/1096731407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56329a082430 0x56329a07fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.982+0000 7fcae762d740 1 --2- 172.21.15.44:0/1096731407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56329a079850 0x56329a07f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.982+0000 7fcae762d740 1 --2- 172.21.15.44:0/1096731407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a078f30 0x56329a07f450 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:02.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.982+0000 7fcac5ffb640 0 client.17676 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:02.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.982+0000 7fcae762d740 1 -- 172.21.15.44:0/1096731407 >> 172.21.15.44:0/1096731407 conn(0x56329a073660 msgr2=0x56329a0753b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:02.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.982+0000 7fcae762d740 1 -- 172.21.15.44:0/1096731407 shutdown_connections 2024-04-21T23:18:02.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.982+0000 7fcae762d740 1 -- 172.21.15.44:0/1096731407 wait complete. 2024-04-21T23:18:02.994 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.982+0000 7fcae762d740 20 client.17676 trim_cache size 0 max 16384 2024-04-21T23:18:03.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:02.999+0000 7fcae762d740 1 Processor -- start 2024-04-21T23:18:03.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.000+0000 7fcae762d740 1 -- start start 2024-04-21T23:18:03.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.000+0000 7fcae762d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56329a078f30 0x56329a081b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:03.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.000+0000 7fcae762d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56329a079850 0x56329a07a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:03.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.000+0000 7fcae762d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a082430 0x56329a07c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:03.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.000+0000 7fcae762d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcae01acda0 con 0x56329a082430 2024-04-21T23:18:03.005 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.000+0000 7fcae762d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcae01af600 con 0x56329a078f30 2024-04-21T23:18:03.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.000+0000 7fcae762d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcae01ad3f0 con 0x56329a079850 2024-04-21T23:18:03.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.000+0000 7fcaddd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a082430 0x56329a07c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:03.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.000+0000 7fcaddd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a082430 0x56329a07c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55406/0 (socket says 172.21.15.44:55406) 2024-04-21T23:18:03.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.000+0000 7fcaddd9b640 1 -- 172.21.15.44:0/1318587585 learned_addr learned my addr 172.21.15.44:0/1318587585 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:03.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.000+0000 7fcaddd9b640 1 --2- 172.21.15.44:0/1318587585 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56329a078f30 0x56329a081b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:03.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.000+0000 7fcade59c640 1 --2- 172.21.15.44:0/1318587585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56329a079850 0x56329a07a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:03.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.000+0000 7fcaddd9b640 1 -- 172.21.15.44:0/1318587585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56329a079850 msgr2=0x56329a07a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:03.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.000+0000 7fcaddd9b640 1 --2- 172.21.15.44:0/1318587585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56329a079850 0x56329a07a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.006 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.000+0000 7fcaddd9b640 1 -- 172.21.15.44:0/1318587585 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56329a078f30 msgr2=0x56329a081b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:03.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.000+0000 7fcaddd9b640 1 --2- 172.21.15.44:0/1318587585 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56329a078f30 0x56329a081b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.000+0000 7fcaddd9b640 1 -- 172.21.15.44:0/1318587585 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56329a074890 con 0x56329a082430 2024-04-21T23:18:03.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.000+0000 7fcaddd9b640 1 --2- 172.21.15.44:0/1318587585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a082430 0x56329a07c6b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fcad4009850 tx=0x7fcad4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e8e40690016b71f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:03.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.001+0000 7fcac7fff640 1 -- 172.21.15.44:0/1318587585 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad404baf0 con 0x56329a082430 2024-04-21T23:18:03.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.001+0000 7fcac7fff640 1 -- 172.21.15.44:0/1318587585 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcad4050070 con 0x56329a082430 2024-04-21T23:18:03.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.001+0000 7fcac7fff640 1 -- 172.21.15.44:0/1318587585 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad40346c0 con 0x56329a082430 2024-04-21T23:18:03.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.001+0000 7fcae762d740 1 -- 172.21.15.44:0/1318587585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a082430 msgr2=0x56329a07c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:03.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.001+0000 7fcae762d740 1 --2- 172.21.15.44:0/1318587585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a082430 0x56329a07c6b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fcad4009850 tx=0x7fcad4005fd0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.001+0000 7fcae762d740 1 -- 172.21.15.44:0/1318587585 shutdown_connections 2024-04-21T23:18:03.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.001+0000 7fcae762d740 1 --2- 172.21.15.44:0/1318587585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56329a079850 0x56329a07a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.001+0000 7fcae762d740 1 --2- 172.21.15.44:0/1318587585 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a082430 0x56329a07c6b0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.001+0000 7fcae762d740 1 --2- 172.21.15.44:0/1318587585 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56329a078f30 0x56329a081b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.001+0000 7fcae762d740 1 -- 172.21.15.44:0/1318587585 >> 172.21.15.44:0/1318587585 conn(0x56329a073660 msgr2=0x56329a075340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:03.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.001+0000 7fcae762d740 1 -- 172.21.15.44:0/1318587585 shutdown_connections 2024-04-21T23:18:03.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.001+0000 7fcae762d740 1 -- 172.21.15.44:0/1318587585 wait complete. 2024-04-21T23:18:03.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.001+0000 7fcae762d740 1 -- start start 2024-04-21T23:18:03.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.002+0000 7fcae762d740 1 Processor -- start 2024-04-21T23:18:03.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.002+0000 7fcae762d740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a078f30 0x56329a0976e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:03.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.002+0000 7fcae762d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56329a079850 0x56329a097c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:03.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.002+0000 7fcae762d740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56329a082430 0x56329a094c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:03.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.002+0000 7fcae762d740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcae01adac0 con 0x56329a078f30 2024-04-21T23:18:03.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.002+0000 7fcae762d740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcae01aef30 con 0x56329a082430 2024-04-21T23:18:03.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.002+0000 7fcae762d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcae01ae190 con 0x56329a079850 2024-04-21T23:18:03.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.002+0000 7fcae762d740 10 client.0 _ll_register_callbacks cb 0x7fcad402cc30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:03.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.002+0000 7fcaddd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a078f30 0x56329a0976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:03.009 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.002+0000 7fcaddd9b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a078f30 0x56329a0976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:55414/0 (socket says 172.21.15.44:55414) 2024-04-21T23:18:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.002+0000 7fcaddd9b640 1 -- 172.21.15.44:0/2541389228 learned_addr learned my addr 172.21.15.44:0/2541389228 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.002+0000 7fcade59c640 1 --2- 172.21.15.44:0/2541389228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56329a079850 0x56329a097c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.002+0000 7fcaddd9b640 1 -- 172.21.15.44:0/2541389228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56329a079850 msgr2=0x56329a097c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.002+0000 7fcaddd9b640 1 --2- 172.21.15.44:0/2541389228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56329a079850 0x56329a097c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.002+0000 7fcaddd9b640 1 -- 172.21.15.44:0/2541389228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56329a082430 msgr2=0x56329a094c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.002+0000 7fcaddd9b640 1 --2- 172.21.15.44:0/2541389228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56329a082430 0x56329a094c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.002+0000 7fcaddd9b640 1 -- 172.21.15.44:0/2541389228 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56329a0952f0 con 0x56329a078f30 2024-04-21T23:18:03.010 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.002+0000 7fcade59c640 1 --2- 172.21.15.44:0/2541389228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56329a079850 0x56329a097c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:03.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.002+0000 7fcaddd9b640 1 --2- 172.21.15.44:0/2541389228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a078f30 0x56329a0976e0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fcad404b900 tx=0x7fcad4005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c07a2f8029e3ae server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:03.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.002+0000 7fcac5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:18:03.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.003+0000 7fcac5ffb640 1 -- 172.21.15.44:0/2541389228 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad4063050 con 0x56329a078f30 2024-04-21T23:18:03.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.003+0000 7fcac5ffb640 1 -- 172.21.15.44:0/2541389228 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9533+0+0 (secure 0 0 0) 0x7fcad404bcf0 con 0x56329a078f30 2024-04-21T23:18:03.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.003+0000 7fcae762d740 1 -- 172.21.15.44:0/2541389228 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcad4050070 con 0x56329a078f30 2024-04-21T23:18:03.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.003+0000 7fcac5ffb640 1 -- 172.21.15.44:0/2541389228 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcad4004040 con 0x56329a078f30 2024-04-21T23:18:03.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.003+0000 7fcae762d740 10 client.17682 fetch_fsmap learned FSMap version 31 2024-04-21T23:18:03.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.003+0000 7fcae762d740 1 -- 172.21.15.44:0/2541389228 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56329a0f8260 con 0x56329a078f30 2024-04-21T23:18:03.011 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.003+0000 7fcac5ffb640 1 -- 172.21.15.44:0/2541389228 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcad4050070 con 0x56329a078f30 2024-04-21T23:18:03.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.003+0000 7fcac5ffb640 1 -- 172.21.15.44:0/2541389228 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7fcad4003680 con 0x56329a078f30 2024-04-21T23:18:03.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.003+0000 7fcae762d740 10 client.17682 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:18:03.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.003+0000 7fcae762d740 10 client.17682 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:03.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.003+0000 7fcae762d740 20 client.17682 populate_metadata read hostname 'smithi044' 2024-04-21T23:18:03.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.003+0000 7fcae762d740 1 --2- 172.21.15.44:0/2541389228 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56329a0f87c0 0x56329a0fabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:03.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.003+0000 7fcae762d740 4 client.17682 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:18:03.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.003+0000 7fcae762d740 1 -- 172.21.15.44:0/2541389228 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563299f625a0 con 0x56329a0f87c0 2024-04-21T23:18:03.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.003+0000 7fcade59c640 1 --2- 172.21.15.44:0/2541389228 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56329a0f87c0 0x56329a0fabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:03.012 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.004+0000 7fcade59c640 1 --2- 172.21.15.44:0/2541389228 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56329a0f87c0 0x56329a0fabe0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1fea2d9a0f312ba server_cookie=4abc60577fc1f30d in_seq=0 out_seq=0 2024-04-21T23:18:03.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.004+0000 7fcac5ffb640 10 client.17682 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:03.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.004+0000 7fcac5ffb640 1 -- 172.21.15.44:0/2541389228 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563299f625a0 con 0x56329a0f87c0 2024-04-21T23:18:03.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.004+0000 7fcac5ffb640 10 client.17682 handle_command_reply: tid=0 2024-04-21T23:18:03.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.004+0000 7fcae762d740 1 client.17682 shutdown 2024-04-21T23:18:03.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.004+0000 7fcae762d740 1 -- 172.21.15.44:0/2541389228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a078f30 msgr2=0x56329a0976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:03.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.004+0000 7fcae762d740 1 --2- 172.21.15.44:0/2541389228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a078f30 0x56329a0976e0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fcad404b900 tx=0x7fcad4005270 comp rx=0 tx=0).stop 2024-04-21T23:18:03.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.004+0000 7fcae762d740 1 -- 172.21.15.44:0/2541389228 shutdown_connections 2024-04-21T23:18:03.013 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.004+0000 7fcae762d740 1 --2- 172.21.15.44:0/2541389228 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56329a0f87c0 0x56329a0fabe0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.004+0000 7fcae762d740 1 --2- 172.21.15.44:0/2541389228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56329a079850 0x56329a097c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.004+0000 7fcac5ffb640 0 client.17682 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:03.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.004+0000 7fcae762d740 1 --2- 172.21.15.44:0/2541389228 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56329a082430 0x56329a094c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.004+0000 7fcae762d740 1 --2- 172.21.15.44:0/2541389228 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56329a078f30 0x56329a0976e0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.004+0000 7fcae762d740 1 -- 172.21.15.44:0/2541389228 >> 172.21.15.44:0/2541389228 conn(0x56329a073660 msgr2=0x56329a074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:03.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.004+0000 7fcae762d740 1 -- 172.21.15.44:0/2541389228 shutdown_connections 2024-04-21T23:18:03.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.004+0000 7fcae762d740 1 -- 172.21.15.44:0/2541389228 wait complete. 2024-04-21T23:18:03.014 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.004+0000 7fcae762d740 20 client.17682 trim_cache size 0 max 16384 2024-04-21T23:18:03.015 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:18:03.015 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:18:03.015 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-21T23:18:03.015 INFO:teuthology.orchestra.run.smithi044.stdout: "eaf8af63-1e96-4c6e-abc2-5e39143d25fe": { 2024-04-21T23:18:03.015 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-21T23:18:03.015 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "eaf8af63-1e96-4c6e-abc2-5e39143d25fe", 2024-04-21T23:18:03.015 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-04-21T23:18:03.016 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:18:03.016 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:18:03.016 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:18:03.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.005+0000 7fcae4a40640 1 -- 172.21.15.44:0/3234790143 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcac80a2d00 msgr2=0x7fcac80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:03.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.005+0000 7fcae4a40640 1 --2- 172.21.15.44:0/3234790143 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcac80a2d00 0x7fcac80a51c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcad009b800 tx=0x7fcad0066bd0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.017 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.005+0000 7fcae4a40640 1 -- 172.21.15.44:0/3234790143 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcae01066b0 msgr2=0x7fcae019c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:03.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.005+0000 7fcae4a40640 1 --2- 172.21.15.44:0/3234790143 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcae01066b0 0x7fcae019c910 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fcad000e960 tx=0x7fcad000ee20 comp rx=0 tx=0).stop 2024-04-21T23:18:03.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.005+0000 7fcae4a40640 1 -- 172.21.15.44:0/3234790143 shutdown_connections 2024-04-21T23:18:03.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.005+0000 7fcae4a40640 1 --2- 172.21.15.44:0/3234790143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcae0100fd0 0x7fcae01985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.005+0000 7fcae4a40640 1 --2- 172.21.15.44:0/3234790143 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcac80a2d00 0x7fcac80a51c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.005+0000 7fcae4a40640 1 --2- 172.21.15.44:0/3234790143 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcae01066b0 0x7fcae019c910 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.005+0000 7fcae4a40640 1 --2- 172.21.15.44:0/3234790143 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcae0100610 0x7fcae01980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:03.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.005+0000 7fcae4a40640 1 -- 172.21.15.44:0/3234790143 >> 172.21.15.44:0/3234790143 conn(0x7fcae00777f0 msgr2=0x7fcae00ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:03.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.005+0000 7fcae4a40640 1 -- 172.21.15.44:0/3234790143 shutdown_connections 2024-04-21T23:18:03.019 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:03.005+0000 7fcae4a40640 1 -- 172.21.15.44:0/3234790143 wait complete. 2024-04-21T23:18:03.019 INFO:tasks.cephfs.filesystem:scrub status for tag:eaf8af63-1e96-4c6e-abc2-5e39143d25fe - {'path': '/', 'tag': 'eaf8af63-1e96-4c6e-abc2-5e39143d25fe', 'options': 'recursive,force'} 2024-04-21T23:18:03.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:03 smithi044 ceph-mon[27934]: scrub queued for path: / 2024-04-21T23:18:03.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:03 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:18:03.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:03 smithi044 ceph-mon[27934]: scrub summary: active paths [/] 2024-04-21T23:18:03.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:03 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:18:03.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:03 smithi055 ceph-mon[31117]: scrub queued for path: / 2024-04-21T23:18:03.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:03 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:18:03.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:03 smithi055 ceph-mon[31117]: scrub summary: active paths [/] 2024-04-21T23:18:03.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:03 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:18:03.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:03 smithi069 ceph-mon[31816]: scrub queued for path: / 2024-04-21T23:18:03.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:03 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:18:03.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:03 smithi069 ceph-mon[31816]: scrub summary: active paths [/] 2024-04-21T23:18:03.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:03 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:18:04.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:04 smithi044 ceph-mon[27934]: pgmap v290: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 51 KiB/s wr, 19 op/s 2024-04-21T23:18:04.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:04 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:18:04.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:04 smithi069 ceph-mon[31816]: pgmap v290: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 51 KiB/s wr, 19 op/s 2024-04-21T23:18:04.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:04 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:18:04.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:04 smithi055 ceph-mon[31117]: pgmap v290: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 51 KiB/s wr, 19 op/s 2024-04-21T23:18:04.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:04 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:18:05.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:05 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:18:05.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:05 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:18:05.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:05 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:18:05.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:05 smithi044 ceph-mon[27934]: pgmap v291: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 51 KiB/s wr, 18 op/s 2024-04-21T23:18:05.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:05 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:18:05.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:05 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:18:05.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:05 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:18:05.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:05 smithi069 ceph-mon[31816]: pgmap v291: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 51 KiB/s wr, 18 op/s 2024-04-21T23:18:05.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:05 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:18:05.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:05 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:18:05.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:05 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:18:05.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:05 smithi055 ceph-mon[31117]: pgmap v291: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 51 KiB/s wr, 18 op/s 2024-04-21T23:18:08.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:07 smithi044 ceph-mon[27934]: pgmap v292: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 51 KiB/s wr, 22 op/s 2024-04-21T23:18:08.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:07 smithi044 ceph-mon[27934]: scrub summary: idle 2024-04-21T23:18:08.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:07 smithi069 ceph-mon[31816]: pgmap v292: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 51 KiB/s wr, 22 op/s 2024-04-21T23:18:08.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:07 smithi069 ceph-mon[31816]: scrub summary: idle 2024-04-21T23:18:08.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:07 smithi055 ceph-mon[31117]: pgmap v292: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 51 KiB/s wr, 22 op/s 2024-04-21T23:18:08.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:07 smithi055 ceph-mon[31117]: scrub summary: idle 2024-04-21T23:18:09.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:08 smithi044 ceph-mon[27934]: osdmap e107: 12 total, 12 up, 12 in 2024-04-21T23:18:09.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:08 smithi055 ceph-mon[31117]: osdmap e107: 12 total, 12 up, 12 in 2024-04-21T23:18:09.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:08 smithi069 ceph-mon[31816]: osdmap e107: 12 total, 12 up, 12 in 2024-04-21T23:18:10.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:10 smithi055 ceph-mon[31117]: pgmap v294: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 68 KiB/s wr, 31 op/s 2024-04-21T23:18:10.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:10 smithi069 ceph-mon[31816]: pgmap v294: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 68 KiB/s wr, 31 op/s 2024-04-21T23:18:10.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:10 smithi044 ceph-mon[27934]: pgmap v294: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 68 KiB/s wr, 31 op/s 2024-04-21T23:18:12.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:12 smithi055 ceph-mon[31117]: pgmap v295: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 68 KiB/s wr, 31 op/s 2024-04-21T23:18:12.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:12 smithi069 ceph-mon[31816]: pgmap v295: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 68 KiB/s wr, 31 op/s 2024-04-21T23:18:12.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:12 smithi044 ceph-mon[27934]: pgmap v295: 129 pgs: 129 active+clean; 81 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 68 KiB/s wr, 31 op/s 2024-04-21T23:18:14.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:14 smithi069 ceph-mon[31816]: pgmap v296: 129 pgs: 11 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 93 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-21T23:18:14.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:14 smithi055 ceph-mon[31117]: pgmap v296: 129 pgs: 11 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 93 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-21T23:18:14.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:14 smithi044 ceph-mon[27934]: pgmap v296: 129 pgs: 11 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 93 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-21T23:18:16.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:16 smithi055 ceph-mon[31117]: pgmap v297: 129 pgs: 11 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 93 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-21T23:18:16.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:16 smithi069 ceph-mon[31816]: pgmap v297: 129 pgs: 11 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 93 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-21T23:18:16.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:16 smithi044 ceph-mon[27934]: pgmap v297: 129 pgs: 11 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 93 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-21T23:18:18.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:18 smithi044 ceph-mon[27934]: pgmap v298: 129 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 110 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-04-21T23:18:18.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:18 smithi055 ceph-mon[31117]: pgmap v298: 129 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 110 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-04-21T23:18:18.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:18 smithi069 ceph-mon[31816]: pgmap v298: 129 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 110 active+clean; 80 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-04-21T23:18:20.486 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:20 smithi044 ceph-mon[27934]: pgmap v299: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-04-21T23:18:20.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:20 smithi069 ceph-mon[31816]: pgmap v299: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-04-21T23:18:20.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:20 smithi055 ceph-mon[31117]: pgmap v299: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-04-21T23:18:22.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:22 smithi044 ceph-mon[27934]: pgmap v300: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 4.5 KiB/s wr, 1 op/s 2024-04-21T23:18:22.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:22 smithi069 ceph-mon[31816]: pgmap v300: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 4.5 KiB/s wr, 1 op/s 2024-04-21T23:18:22.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:22 smithi055 ceph-mon[31117]: pgmap v300: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 4.5 KiB/s wr, 1 op/s 2024-04-21T23:18:24.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:24 smithi069 ceph-mon[31816]: pgmap v301: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 4.5 KiB/s wr, 2 op/s 2024-04-21T23:18:24.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:24 smithi069 ceph-mon[31816]: osdmap e108: 12 total, 12 up, 12 in 2024-04-21T23:18:24.465 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:24 smithi055 ceph-mon[31117]: pgmap v301: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 4.5 KiB/s wr, 2 op/s 2024-04-21T23:18:24.466 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:24 smithi055 ceph-mon[31117]: osdmap e108: 12 total, 12 up, 12 in 2024-04-21T23:18:24.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:24 smithi044 ceph-mon[27934]: pgmap v301: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 4.5 KiB/s wr, 2 op/s 2024-04-21T23:18:24.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:24 smithi044 ceph-mon[27934]: osdmap e108: 12 total, 12 up, 12 in 2024-04-21T23:18:26.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:26 smithi069 ceph-mon[31816]: pgmap v303: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-04-21T23:18:26.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:26 smithi055 ceph-mon[31117]: pgmap v303: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-04-21T23:18:26.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:26 smithi044 ceph-mon[27934]: pgmap v303: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-04-21T23:18:28.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:28 smithi055 ceph-mon[31117]: pgmap v304: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-04-21T23:18:28.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:28 smithi069 ceph-mon[31816]: pgmap v304: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-04-21T23:18:28.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:28 smithi044 ceph-mon[27934]: pgmap v304: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-04-21T23:18:30.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:30 smithi044 ceph-mon[27934]: pgmap v305: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:18:30.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:30 smithi055 ceph-mon[31117]: pgmap v305: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:18:30.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:30 smithi069 ceph-mon[31816]: pgmap v305: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:18:32.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:32 smithi044 ceph-mon[27934]: pgmap v306: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:18:32.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:32 smithi069 ceph-mon[31816]: pgmap v306: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:18:32.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:32 smithi055 ceph-mon[31117]: pgmap v306: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:18:33.020 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:18:33.020 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:18:33.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.139+0000 7f51a237a640 1 -- 172.21.15.44:0/2382923868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f519c103210 msgr2=0x7f519c103670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.139+0000 7f51a237a640 1 --2- 172.21.15.44:0/2382923868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f519c103210 0x7f519c103670 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f5190009900 tx=0x7f519002bd10 comp rx=0 tx=0).stop 2024-04-21T23:18:33.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.139+0000 7f51a237a640 1 -- 172.21.15.44:0/2382923868 shutdown_connections 2024-04-21T23:18:33.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.139+0000 7f51a237a640 1 --2- 172.21.15.44:0/2382923868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f519c103bb0 0x7f519c10e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.139+0000 7f51a237a640 1 --2- 172.21.15.44:0/2382923868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f519c103210 0x7f519c103670 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.139+0000 7f51a237a640 1 --2- 172.21.15.44:0/2382923868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f519c101ad0 0x7f519c101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.140 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.139+0000 7f51a237a640 1 -- 172.21.15.44:0/2382923868 >> 172.21.15.44:0/2382923868 conn(0x7f519c0fd200 msgr2=0x7f519c0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:33.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.139+0000 7f51a237a640 1 -- 172.21.15.44:0/2382923868 shutdown_connections 2024-04-21T23:18:33.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.139+0000 7f51a237a640 1 -- 172.21.15.44:0/2382923868 wait complete. 2024-04-21T23:18:33.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.140+0000 7f51a237a640 1 Processor -- start 2024-04-21T23:18:33.141 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.140+0000 7f51a237a640 1 -- start start 2024-04-21T23:18:33.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.140+0000 7f51a237a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f519c101ad0 0x7f519c073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.140+0000 7f51a237a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f519c103210 0x7f519c073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.140+0000 7f51a237a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f519c103bb0 0x7f519c0744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.140+0000 7f51a237a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f519c111c70 con 0x7f519c103210 2024-04-21T23:18:33.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.140+0000 7f51a237a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f519c111af0 con 0x7f519c103bb0 2024-04-21T23:18:33.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.140+0000 7f51a237a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f519c111df0 con 0x7f519c101ad0 2024-04-21T23:18:33.143 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.141+0000 7f519b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f519c103210 0x7f519c073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.141+0000 7f519b7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f519c103210 0x7f519c073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49322/0 (socket says 172.21.15.44:49322) 2024-04-21T23:18:33.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.141+0000 7f519b7fe640 1 -- 172.21.15.44:0/1577791216 learned_addr learned my addr 172.21.15.44:0/1577791216 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:33.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.141+0000 7f51a0917640 1 --2- 172.21.15.44:0/1577791216 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f519c103bb0 0x7f519c0744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.141+0000 7f519bfff640 1 --2- 172.21.15.44:0/1577791216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f519c101ad0 0x7f519c073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.144 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.141+0000 7f519b7fe640 1 -- 172.21.15.44:0/1577791216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f519c101ad0 msgr2=0x7f519c073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.145 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.141+0000 7f519b7fe640 1 --2- 172.21.15.44:0/1577791216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f519c101ad0 0x7f519c073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.141+0000 7f519b7fe640 1 -- 172.21.15.44:0/1577791216 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f519c103bb0 msgr2=0x7f519c0744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.141+0000 7f519b7fe640 1 --2- 172.21.15.44:0/1577791216 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f519c103bb0 0x7f519c0744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.141+0000 7f519b7fe640 1 -- 172.21.15.44:0/1577791216 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f519c1a6610 con 0x7f519c103210 2024-04-21T23:18:33.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.141+0000 7f519b7fe640 1 --2- 172.21.15.44:0/1577791216 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f519c103210 0x7f519c073f80 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f5190000c00 tx=0x7f5190004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:33.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.141+0000 7f51997fa640 1 -- 172.21.15.44:0/1577791216 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5190042020 con 0x7f519c103210 2024-04-21T23:18:33.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.141+0000 7f51a237a640 1 -- 172.21.15.44:0/1577791216 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f519c1a68a0 con 0x7f519c103210 2024-04-21T23:18:33.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.141+0000 7f51997fa640 1 -- 172.21.15.44:0/1577791216 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51900054e0 con 0x7f519c103210 2024-04-21T23:18:33.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.141+0000 7f51997fa640 1 -- 172.21.15.44:0/1577791216 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5190003c20 con 0x7f519c103210 2024-04-21T23:18:33.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.141+0000 7f51a237a640 1 -- 172.21.15.44:0/1577791216 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f519c1a6e40 con 0x7f519c103210 2024-04-21T23:18:33.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.144+0000 7f51997fa640 1 -- 172.21.15.44:0/1577791216 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f5190045020 con 0x7f519c103210 2024-04-21T23:18:33.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f51a4f64740 1 Processor -- start 2024-04-21T23:18:33.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f51a4f64740 1 -- start start 2024-04-21T23:18:33.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f51a4f64740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 0x558aeb863310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f51a4f64740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558aeb863850 0x558aeb86bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f51a4f64740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558aeb86c430 0x558aeb86e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f51a4f64740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558aeb7c7560 con 0x558aeb862f30 2024-04-21T23:18:33.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f51a4f64740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x558aeb76f730 con 0x558aeb863850 2024-04-21T23:18:33.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f51a4f64740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558aeb7c3520 con 0x558aeb86c430 2024-04-21T23:18:33.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f51a0917640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 0x558aeb863310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f51a0917640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 0x558aeb863310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49330/0 (socket says 172.21.15.44:49330) 2024-04-21T23:18:33.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f51a0917640 1 -- 172.21.15.44:0/2478880710 learned_addr learned my addr 172.21.15.44:0/2478880710 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:33.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f51997fa640 1 --2- 172.21.15.44:0/1577791216 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f51840a2d00 0x7f51840a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f519bfff640 1 --2- 172.21.15.44:0/2478880710 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558aeb863850 0x558aeb86bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f51a0917640 1 --2- 172.21.15.44:0/1577791216 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f51840a2d00 0x7f51840a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f51997fa640 1 -- 172.21.15.44:0/1577791216 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7f51900f1620 con 0x7f519c103210 2024-04-21T23:18:33.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f519b7fe640 1 --2- 172.21.15.44:0/2478880710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558aeb86c430 0x558aeb86e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f51a0917640 1 -- 172.21.15.44:0/2478880710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558aeb86c430 msgr2=0x558aeb86e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:33.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f51a0917640 1 --2- 172.21.15.44:0/2478880710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558aeb86c430 0x558aeb86e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f51a0917640 1 -- 172.21.15.44:0/2478880710 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558aeb863850 msgr2=0x558aeb86bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:33.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f51a0917640 1 --2- 172.21.15.44:0/2478880710 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558aeb863850 0x558aeb86bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.145+0000 7f51a0917640 1 -- 172.21.15.44:0/2478880710 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558aeb5a57d0 con 0x558aeb862f30 2024-04-21T23:18:33.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.146+0000 7f51a0917640 1 --2- 172.21.15.44:0/2478880710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 0x558aeb863310 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f5194014a60 tx=0x7f5194014f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37c152710b126427 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:33.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.146+0000 7f51827fc640 1 -- 172.21.15.44:0/2478880710 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519401ec50 con 0x558aeb862f30 2024-04-21T23:18:33.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.146+0000 7f51827fc640 1 -- 172.21.15.44:0/2478880710 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5194023070 con 0x558aeb862f30 2024-04-21T23:18:33.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.146+0000 7f51827fc640 1 -- 172.21.15.44:0/2478880710 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519401de00 con 0x558aeb862f30 2024-04-21T23:18:33.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.147+0000 7f51a4f64740 1 -- 172.21.15.44:0/2478880710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 msgr2=0x558aeb863310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.147+0000 7f51a0917640 1 --2- 172.21.15.44:0/1577791216 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f51840a2d00 0x7f51840a51c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5194022000 tx=0x7f519401e000 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:33.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.147+0000 7f51a4f64740 1 --2- 172.21.15.44:0/2478880710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 0x558aeb863310 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f5194014a60 tx=0x7f5194014f20 comp rx=0 tx=0).stop 2024-04-21T23:18:33.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.147+0000 7f51a4f64740 1 -- 172.21.15.44:0/2478880710 shutdown_connections 2024-04-21T23:18:33.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.147+0000 7f51a4f64740 1 --2- 172.21.15.44:0/2478880710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558aeb86c430 0x558aeb86e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.147+0000 7f51a4f64740 1 --2- 172.21.15.44:0/2478880710 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558aeb863850 0x558aeb86bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.147+0000 7f51a4f64740 1 --2- 172.21.15.44:0/2478880710 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 0x558aeb863310 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.147+0000 7f51a4f64740 1 -- 172.21.15.44:0/2478880710 >> 172.21.15.44:0/2478880710 conn(0x558aeb85d660 msgr2=0x558aeb85da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:33.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.147+0000 7f51a4f64740 1 -- 172.21.15.44:0/2478880710 shutdown_connections 2024-04-21T23:18:33.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.147+0000 7f51a4f64740 1 -- 172.21.15.44:0/2478880710 wait complete. 2024-04-21T23:18:33.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.147+0000 7f51a4f64740 1 -- start start 2024-04-21T23:18:33.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.147+0000 7f51a4f64740 1 Processor -- start 2024-04-21T23:18:33.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f51a4f64740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558aeb862f30 0x558aeb8694d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f51a4f64740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558aeb863850 0x558aeb869a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f51a4f64740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb86c430 0x558aeb869f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f51a4f64740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x558aeb86f220 con 0x558aeb86c430 2024-04-21T23:18:33.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f51a4f64740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x558aeb782e00 con 0x558aeb863850 2024-04-21T23:18:33.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f51a4f64740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558aeb86f3a0 con 0x558aeb862f30 2024-04-21T23:18:33.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f51a4f64740 10 client.0 _ll_register_callbacks cb 0x558aeb7bdeb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:33.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f51a0917640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb86c430 0x558aeb869f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f51a0917640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb86c430 0x558aeb869f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49346/0 (socket says 172.21.15.44:49346) 2024-04-21T23:18:33.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f51a0917640 1 -- 172.21.15.44:0/4038508336 learned_addr learned my addr 172.21.15.44:0/4038508336 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:33.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f519b7fe640 1 --2- 172.21.15.44:0/4038508336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558aeb863850 0x558aeb869a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f519bfff640 1 --2- 172.21.15.44:0/4038508336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558aeb862f30 0x558aeb8694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f51a0917640 1 -- 172.21.15.44:0/4038508336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558aeb862f30 msgr2=0x558aeb8694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:33.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f51a0917640 1 --2- 172.21.15.44:0/4038508336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558aeb862f30 0x558aeb8694d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f51a0917640 1 -- 172.21.15.44:0/4038508336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558aeb863850 msgr2=0x558aeb869a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:33.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f51a0917640 1 --2- 172.21.15.44:0/4038508336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558aeb863850 0x558aeb869a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f51a0917640 1 -- 172.21.15.44:0/4038508336 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558aeb4db3f0 con 0x558aeb86c430 2024-04-21T23:18:33.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f519bfff640 1 --2- 172.21.15.44:0/4038508336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558aeb862f30 0x558aeb8694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:33.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f51a0917640 1 --2- 172.21.15.44:0/4038508336 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb86c430 0x558aeb869f50 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f519401dad0 tx=0x7f51940144f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e21ec36e1ae981e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:33.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f517b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:18:33.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f517b7fe640 1 -- 172.21.15.44:0/4038508336 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5194037060 con 0x558aeb86c430 2024-04-21T23:18:33.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f51a4f64740 1 -- 172.21.15.44:0/4038508336 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5194023070 con 0x558aeb86c430 2024-04-21T23:18:33.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.148+0000 7f517b7fe640 1 -- 172.21.15.44:0/4038508336 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7f5194014590 con 0x558aeb86c430 2024-04-21T23:18:33.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.149+0000 7f517b7fe640 1 -- 172.21.15.44:0/4038508336 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f519403e050 con 0x558aeb86c430 2024-04-21T23:18:33.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.149+0000 7f51a4f64740 10 client.17694 fetch_fsmap learned FSMap version 31 2024-04-21T23:18:33.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.149+0000 7f51a4f64740 1 -- 172.21.15.44:0/4038508336 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558aeb49b0f0 con 0x558aeb86c430 2024-04-21T23:18:33.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.149+0000 7f517b7fe640 1 -- 172.21.15.44:0/4038508336 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5194023070 con 0x558aeb86c430 2024-04-21T23:18:33.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.149+0000 7f517b7fe640 1 -- 172.21.15.44:0/4038508336 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f519400f040 con 0x558aeb86c430 2024-04-21T23:18:33.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.149+0000 7f51a4f64740 10 client.17694 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:18:33.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.149+0000 7f51a4f64740 10 client.17694 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:33.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.149+0000 7f51a4f64740 20 client.17694 populate_metadata read hostname 'smithi044' 2024-04-21T23:18:33.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.149+0000 7f51a4f64740 1 --2- 172.21.15.44:0/4038508336 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558aeb87c860 0x558aeb87ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.149+0000 7f51a4f64740 4 client.17694 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:18:33.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.149+0000 7f51a4f64740 1 -- 172.21.15.44:0/4038508336 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558aeb7d4aa0 con 0x558aeb87c860 2024-04-21T23:18:33.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.149+0000 7f519bfff640 1 --2- 172.21.15.44:0/4038508336 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558aeb87c860 0x558aeb87ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.150+0000 7f517b7fe640 10 client.17694 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:33.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.150+0000 7f519bfff640 1 --2- 172.21.15.44:0/4038508336 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558aeb87c860 0x558aeb87ec50 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48191880aecc2732 server_cookie=bb7b825f656d5b9b in_seq=0 out_seq=0 2024-04-21T23:18:33.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.150+0000 7f517b7fe640 1 -- 172.21.15.44:0/4038508336 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558aeb7d4aa0 con 0x558aeb87c860 2024-04-21T23:18:33.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.150+0000 7f517b7fe640 10 client.17694 handle_command_reply: tid=0 2024-04-21T23:18:33.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.150+0000 7f51a4f64740 1 client.17694 shutdown 2024-04-21T23:18:33.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.151+0000 7f51a4f64740 1 -- 172.21.15.44:0/4038508336 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb86c430 msgr2=0x558aeb869f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.151+0000 7f51a4f64740 1 --2- 172.21.15.44:0/4038508336 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb86c430 0x558aeb869f50 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f519401dad0 tx=0x7f51940144f0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.151+0000 7f51a4f64740 1 -- 172.21.15.44:0/4038508336 shutdown_connections 2024-04-21T23:18:33.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.151+0000 7f51a4f64740 1 --2- 172.21.15.44:0/4038508336 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558aeb87c860 0x558aeb87ec50 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.151+0000 7f51a4f64740 1 --2- 172.21.15.44:0/4038508336 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb86c430 0x558aeb869f50 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.164 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.151+0000 7f51a4f64740 1 --2- 172.21.15.44:0/4038508336 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558aeb863850 0x558aeb869a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.151+0000 7f517b7fe640 0 client.17694 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:33.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.151+0000 7f51a4f64740 1 --2- 172.21.15.44:0/4038508336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558aeb862f30 0x558aeb8694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.151+0000 7f51a4f64740 1 -- 172.21.15.44:0/4038508336 >> 172.21.15.44:0/4038508336 conn(0x558aeb85d660 msgr2=0x558aeb85f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:33.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.151+0000 7f51a4f64740 1 -- 172.21.15.44:0/4038508336 shutdown_connections 2024-04-21T23:18:33.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.151+0000 7f51a4f64740 1 -- 172.21.15.44:0/4038508336 wait complete. 2024-04-21T23:18:33.165 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.151+0000 7f51a4f64740 20 client.17694 trim_cache size 0 max 16384 2024-04-21T23:18:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.168+0000 7f51a4f64740 1 Processor -- start 2024-04-21T23:18:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.168+0000 7f51a4f64740 1 -- start start 2024-04-21T23:18:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.169+0000 7f51a4f64740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 0x558aeb86bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.169+0000 7f51a4f64740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558aeb863850 0x558aeb864850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.169+0000 7f51a4f64740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558aeb86c430 0x558aeb8780e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.169+0000 7f51a4f64740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f519c1aee60 con 0x558aeb862f30 2024-04-21T23:18:33.175 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.169+0000 7f51a4f64740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f519c1b16c0 con 0x558aeb863850 2024-04-21T23:18:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.169+0000 7f51a4f64740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f519c1af4b0 con 0x558aeb86c430 2024-04-21T23:18:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.169+0000 7f519bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 0x558aeb86bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.169+0000 7f519bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 0x558aeb86bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49348/0 (socket says 172.21.15.44:49348) 2024-04-21T23:18:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.169+0000 7f519bfff640 1 -- 172.21.15.44:0/1883290715 learned_addr learned my addr 172.21.15.44:0/1883290715 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.169+0000 7f519b7fe640 1 --2- 172.21.15.44:0/1883290715 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558aeb863850 0x558aeb864850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.169+0000 7f519bfff640 1 -- 172.21.15.44:0/1883290715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558aeb86c430 msgr2=0x558aeb8780e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.169+0000 7f519bfff640 1 --2- 172.21.15.44:0/1883290715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558aeb86c430 0x558aeb8780e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.169+0000 7f519bfff640 1 -- 172.21.15.44:0/1883290715 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558aeb863850 msgr2=0x558aeb864850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:33.176 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.169+0000 7f519bfff640 1 --2- 172.21.15.44:0/1883290715 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558aeb863850 0x558aeb864850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.169+0000 7f519bfff640 1 -- 172.21.15.44:0/1883290715 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558aeb85f950 con 0x558aeb862f30 2024-04-21T23:18:33.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.169+0000 7f519bfff640 1 --2- 172.21.15.44:0/1883290715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 0x558aeb86bc00 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x558aeb87f250 tx=0x7f518c005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f7146f68e1050e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:33.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.169+0000 7f5181ffb640 1 -- 172.21.15.44:0/1883290715 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c014110 con 0x558aeb862f30 2024-04-21T23:18:33.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.169+0000 7f5181ffb640 1 -- 172.21.15.44:0/1883290715 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f518c0142b0 con 0x558aeb862f30 2024-04-21T23:18:33.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.169+0000 7f5181ffb640 1 -- 172.21.15.44:0/1883290715 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c01b330 con 0x558aeb862f30 2024-04-21T23:18:33.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.170+0000 7f51a4f64740 1 -- 172.21.15.44:0/1883290715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 msgr2=0x558aeb86bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.170+0000 7f51a4f64740 1 --2- 172.21.15.44:0/1883290715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 0x558aeb86bc00 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x558aeb87f250 tx=0x7f518c005f30 comp rx=0 tx=0).stop 2024-04-21T23:18:33.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.170+0000 7f51a4f64740 1 -- 172.21.15.44:0/1883290715 shutdown_connections 2024-04-21T23:18:33.177 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.170+0000 7f51a4f64740 1 --2- 172.21.15.44:0/1883290715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558aeb86c430 0x558aeb8780e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.170+0000 7f51a4f64740 1 --2- 172.21.15.44:0/1883290715 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558aeb863850 0x558aeb864850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.170+0000 7f51a4f64740 1 --2- 172.21.15.44:0/1883290715 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 0x558aeb86bc00 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.170+0000 7f51a4f64740 1 -- 172.21.15.44:0/1883290715 >> 172.21.15.44:0/1883290715 conn(0x558aeb85d660 msgr2=0x558aeb868500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.170+0000 7f51a4f64740 1 -- 172.21.15.44:0/1883290715 shutdown_connections 2024-04-21T23:18:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.170+0000 7f51a4f64740 1 -- 172.21.15.44:0/1883290715 wait complete. 2024-04-21T23:18:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.170+0000 7f51a4f64740 1 -- start start 2024-04-21T23:18:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.171+0000 7f51a4f64740 1 Processor -- start 2024-04-21T23:18:33.178 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.171+0000 7f51a4f64740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 0x558aeb874870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.171+0000 7f51a4f64740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558aeb863850 0x558aeb86fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.171+0000 7f51a4f64740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558aeb86c430 0x558aeb870170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.171+0000 7f51a4f64740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f519c1afb80 con 0x558aeb862f30 2024-04-21T23:18:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.171+0000 7f51a4f64740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f519c1b0ff0 con 0x558aeb863850 2024-04-21T23:18:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.171+0000 7f51a4f64740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f519c1b0250 con 0x558aeb86c430 2024-04-21T23:18:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.171+0000 7f51a4f64740 10 client.0 _ll_register_callbacks cb 0x7f5194014860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.171+0000 7f519bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 0x558aeb874870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.171+0000 7f519bfff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 0x558aeb874870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49354/0 (socket says 172.21.15.44:49354) 2024-04-21T23:18:33.179 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.171+0000 7f519bfff640 1 -- 172.21.15.44:0/4114610856 learned_addr learned my addr 172.21.15.44:0/4114610856 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:33.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.171+0000 7f519b7fe640 1 --2- 172.21.15.44:0/4114610856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558aeb863850 0x558aeb86fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.171+0000 7f51a0917640 1 --2- 172.21.15.44:0/4114610856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558aeb86c430 0x558aeb870170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.171+0000 7f519bfff640 1 -- 172.21.15.44:0/4114610856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558aeb86c430 msgr2=0x558aeb870170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:33.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.171+0000 7f519bfff640 1 --2- 172.21.15.44:0/4114610856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558aeb86c430 0x558aeb870170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.171+0000 7f519bfff640 1 -- 172.21.15.44:0/4114610856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558aeb863850 msgr2=0x558aeb86fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:33.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.171+0000 7f519bfff640 1 --2- 172.21.15.44:0/4114610856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558aeb863850 0x558aeb86fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.171+0000 7f519bfff640 1 -- 172.21.15.44:0/4114610856 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558aeb870810 con 0x558aeb862f30 2024-04-21T23:18:33.180 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.172+0000 7f519bfff640 1 --2- 172.21.15.44:0/4114610856 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 0x558aeb874870 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f518c005020 tx=0x7f518c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e7d938812cffa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:33.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.172+0000 7f517b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:18:33.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.172+0000 7f517b7fe640 1 -- 172.21.15.44:0/4114610856 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c029070 con 0x558aeb862f30 2024-04-21T23:18:33.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.172+0000 7f517b7fe640 1 -- 172.21.15.44:0/4114610856 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7f518c005380 con 0x558aeb862f30 2024-04-21T23:18:33.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.172+0000 7f51a4f64740 1 -- 172.21.15.44:0/4114610856 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558aeb7cedf0 con 0x558aeb862f30 2024-04-21T23:18:33.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.172+0000 7f517b7fe640 1 -- 172.21.15.44:0/4114610856 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f518c034050 con 0x558aeb862f30 2024-04-21T23:18:33.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.172+0000 7f51a4f64740 10 client.17700 fetch_fsmap learned FSMap version 31 2024-04-21T23:18:33.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.172+0000 7f51a4f64740 1 -- 172.21.15.44:0/4114610856 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558aeb8e29e0 con 0x558aeb862f30 2024-04-21T23:18:33.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.172+0000 7f517b7fe640 1 -- 172.21.15.44:0/4114610856 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558aeb7cedf0 con 0x558aeb862f30 2024-04-21T23:18:33.181 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.172+0000 7f517b7fe640 1 -- 172.21.15.44:0/4114610856 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f518c004a30 con 0x558aeb862f30 2024-04-21T23:18:33.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.172+0000 7f51a4f64740 10 client.17700 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:18:33.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.172+0000 7f51a4f64740 10 client.17700 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:33.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.172+0000 7f51a4f64740 20 client.17700 populate_metadata read hostname 'smithi044' 2024-04-21T23:18:33.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.172+0000 7f51a4f64740 1 --2- 172.21.15.44:0/4114610856 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558aeb8e3070 0x558aeb8e5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.172+0000 7f51a4f64740 4 client.17700 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:18:33.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.172+0000 7f51a4f64740 1 -- 172.21.15.44:0/4114610856 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558aeb74c5a0 con 0x558aeb8e3070 2024-04-21T23:18:33.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.173+0000 7f519b7fe640 1 --2- 172.21.15.44:0/4114610856 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558aeb8e3070 0x558aeb8e5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.182 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.173+0000 7f519b7fe640 1 --2- 172.21.15.44:0/4114610856 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558aeb8e3070 0x558aeb8e5490 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2316621118a7b334 server_cookie=3a1689e1d3134036 in_seq=0 out_seq=0 2024-04-21T23:18:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.173+0000 7f517b7fe640 10 client.17700 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.173+0000 7f517b7fe640 1 -- 172.21.15.44:0/4114610856 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558aeb74c5a0 con 0x558aeb8e3070 2024-04-21T23:18:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.173+0000 7f517b7fe640 10 client.17700 handle_command_reply: tid=0 2024-04-21T23:18:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.173+0000 7f51a4f64740 1 client.17700 shutdown 2024-04-21T23:18:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.173+0000 7f51a4f64740 1 -- 172.21.15.44:0/4114610856 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 msgr2=0x558aeb874870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.173+0000 7f51a4f64740 1 --2- 172.21.15.44:0/4114610856 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 0x558aeb874870 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f518c005020 tx=0x7f518c005050 comp rx=0 tx=0).stop 2024-04-21T23:18:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.173+0000 7f51a4f64740 1 -- 172.21.15.44:0/4114610856 shutdown_connections 2024-04-21T23:18:33.183 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.173+0000 7f51a4f64740 1 --2- 172.21.15.44:0/4114610856 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x558aeb8e3070 0x558aeb8e5490 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.173+0000 7f51a4f64740 1 --2- 172.21.15.44:0/4114610856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558aeb86c430 0x558aeb870170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.173+0000 7f51a4f64740 1 --2- 172.21.15.44:0/4114610856 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x558aeb863850 0x558aeb86fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.173+0000 7f517b7fe640 0 client.17700 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.173+0000 7f51a4f64740 1 --2- 172.21.15.44:0/4114610856 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x558aeb862f30 0x558aeb874870 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.173+0000 7f51a4f64740 1 -- 172.21.15.44:0/4114610856 >> 172.21.15.44:0/4114610856 conn(0x558aeb85d660 msgr2=0x558aeb867c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.173+0000 7f51a4f64740 1 -- 172.21.15.44:0/4114610856 shutdown_connections 2024-04-21T23:18:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.173+0000 7f51a4f64740 1 -- 172.21.15.44:0/4114610856 wait complete. 2024-04-21T23:18:33.184 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.174+0000 7f51a4f64740 20 client.17700 trim_cache size 0 max 16384 2024-04-21T23:18:33.185 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:18:33.185 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-21T23:18:33.185 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-21T23:18:33.185 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:18:33.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.174+0000 7f51a237a640 1 -- 172.21.15.44:0/1577791216 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f51840a2d00 msgr2=0x7f51840a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.174+0000 7f51a237a640 1 --2- 172.21.15.44:0/1577791216 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f51840a2d00 0x7f51840a51c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5194022000 tx=0x7f519401e000 comp rx=0 tx=0).stop 2024-04-21T23:18:33.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.174+0000 7f51a237a640 1 -- 172.21.15.44:0/1577791216 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f519c103210 msgr2=0x7f519c073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.174+0000 7f51a237a640 1 --2- 172.21.15.44:0/1577791216 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f519c103210 0x7f519c073f80 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f5190000c00 tx=0x7f5190004430 comp rx=0 tx=0).stop 2024-04-21T23:18:33.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.174+0000 7f51a237a640 1 -- 172.21.15.44:0/1577791216 shutdown_connections 2024-04-21T23:18:33.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.174+0000 7f51a237a640 1 --2- 172.21.15.44:0/1577791216 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f51840a2d00 0x7f51840a51c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.187 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.174+0000 7f51a237a640 1 --2- 172.21.15.44:0/1577791216 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f519c103bb0 0x7f519c0744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.174+0000 7f51a237a640 1 --2- 172.21.15.44:0/1577791216 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f519c103210 0x7f519c073f80 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.174+0000 7f51a237a640 1 --2- 172.21.15.44:0/1577791216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f519c101ad0 0x7f519c073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.174+0000 7f51a237a640 1 -- 172.21.15.44:0/1577791216 >> 172.21.15.44:0/1577791216 conn(0x7f519c0fd200 msgr2=0x7f519c1075e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:33.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.174+0000 7f51a237a640 1 -- 172.21.15.44:0/1577791216 shutdown_connections 2024-04-21T23:18:33.188 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.174+0000 7f51a237a640 1 -- 172.21.15.44:0/1577791216 wait complete. 2024-04-21T23:18:33.188 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:18:33.189 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:18:33.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.293+0000 7f2633fe0640 1 -- 172.21.15.44:0/4258584045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f262c1029b0 msgr2=0x7f262c102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.293+0000 7f2633fe0640 1 --2- 172.21.15.44:0/4258584045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f262c1029b0 0x7f262c102d90 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f2628009900 tx=0x7f262802bcd0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.293+0000 7f2633fe0640 1 -- 172.21.15.44:0/4258584045 shutdown_connections 2024-04-21T23:18:33.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.293+0000 7f2633fe0640 1 --2- 172.21.15.44:0/4258584045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f262c1032d0 0x7f262c10b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.293+0000 7f2633fe0640 1 --2- 172.21.15.44:0/4258584045 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f262c10bd80 0x7f262c10e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.293+0000 7f2633fe0640 1 --2- 172.21.15.44:0/4258584045 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f262c1029b0 0x7f262c102d90 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.293+0000 7f2633fe0640 1 -- 172.21.15.44:0/4258584045 >> 172.21.15.44:0/4258584045 conn(0x7f262c077740 msgr2=0x7f262c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:33.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.293+0000 7f2633fe0640 1 -- 172.21.15.44:0/4258584045 shutdown_connections 2024-04-21T23:18:33.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.293+0000 7f2633fe0640 1 -- 172.21.15.44:0/4258584045 wait complete. 2024-04-21T23:18:33.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.294+0000 7f2633fe0640 1 Processor -- start 2024-04-21T23:18:33.296 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.294+0000 7f2633fe0640 1 -- start start 2024-04-21T23:18:33.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.294+0000 7f2633fe0640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f262c1029b0 0x7f262c1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.294+0000 7f2633fe0640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f262c1032d0 0x7f262c198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.294+0000 7f2633fe0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f262c10bd80 0x7f262c19c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.294+0000 7f2633fe0640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f262c111b10 con 0x7f262c1032d0 2024-04-21T23:18:33.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.294+0000 7f2633fe0640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f262c111990 con 0x7f262c1029b0 2024-04-21T23:18:33.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.294+0000 7f2633fe0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f262c111c90 con 0x7f262c10bd80 2024-04-21T23:18:33.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.294+0000 7f263157b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f262c1032d0 0x7f262c198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.294+0000 7f263157b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f262c1032d0 0x7f262c198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49370/0 (socket says 172.21.15.44:49370) 2024-04-21T23:18:33.297 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.294+0000 7f263157b640 1 -- 172.21.15.44:0/3607886424 learned_addr learned my addr 172.21.15.44:0/3607886424 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:33.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.294+0000 7f2631d7c640 1 --2- 172.21.15.44:0/3607886424 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f262c1029b0 0x7f262c1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.294+0000 7f263257d640 1 --2- 172.21.15.44:0/3607886424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f262c10bd80 0x7f262c19c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.294+0000 7f263157b640 1 -- 172.21.15.44:0/3607886424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f262c10bd80 msgr2=0x7f262c19c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.294+0000 7f263157b640 1 --2- 172.21.15.44:0/3607886424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f262c10bd80 0x7f262c19c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.294+0000 7f263157b640 1 -- 172.21.15.44:0/3607886424 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f262c1029b0 msgr2=0x7f262c1980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.294+0000 7f263157b640 1 --2- 172.21.15.44:0/3607886424 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f262c1029b0 0x7f262c1980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.294+0000 7f263157b640 1 -- 172.21.15.44:0/3607886424 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f262c19d070 con 0x7f262c1032d0 2024-04-21T23:18:33.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.294+0000 7f2631d7c640 1 --2- 172.21.15.44:0/3607886424 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f262c1029b0 0x7f262c1980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:33.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.295+0000 7f263157b640 1 --2- 172.21.15.44:0/3607886424 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f262c1032d0 0x7f262c198610 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f261c00e9f0 tx=0x7f261c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:33.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.295+0000 7f2622ffd640 1 -- 172.21.15.44:0/3607886424 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c00cd80 con 0x7f262c1032d0 2024-04-21T23:18:33.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.295+0000 7f2633fe0640 1 -- 172.21.15.44:0/3607886424 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f262c19d360 con 0x7f262c1032d0 2024-04-21T23:18:33.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.295+0000 7f2622ffd640 1 -- 172.21.15.44:0/3607886424 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f261c005500 con 0x7f262c1032d0 2024-04-21T23:18:33.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.295+0000 7f2622ffd640 1 -- 172.21.15.44:0/3607886424 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c010690 con 0x7f262c1032d0 2024-04-21T23:18:33.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.295+0000 7f2633fe0640 1 -- 172.21.15.44:0/3607886424 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f262c1a4bf0 con 0x7f262c1032d0 2024-04-21T23:18:33.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.296+0000 7f2633fe0640 1 -- 172.21.15.44:0/3607886424 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f262c068a40 con 0x7f262c1032d0 2024-04-21T23:18:33.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.297+0000 7f2622ffd640 1 -- 172.21.15.44:0/3607886424 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f261c0046c0 con 0x7f262c1032d0 2024-04-21T23:18:33.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.298+0000 7f2622ffd640 1 --2- 172.21.15.44:0/3607886424 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f26180a2d00 0x7f26180a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.298+0000 7f2631d7c640 1 --2- 172.21.15.44:0/3607886424 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f26180a2d00 0x7f26180a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.300 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.298+0000 7f2622ffd640 1 -- 172.21.15.44:0/3607886424 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7f261c0cfd80 con 0x7f262c1032d0 2024-04-21T23:18:33.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.298+0000 7f2631d7c640 1 --2- 172.21.15.44:0/3607886424 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f26180a2d00 0x7f26180a51c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2628005ab0 tx=0x7f2628005a20 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:33.301 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.302+0000 7f2622ffd640 1 -- 172.21.15.44:0/3607886424 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f261c09c5b0 con 0x7f262c1032d0 2024-04-21T23:18:33.524 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.525+0000 7f2633fe0640 1 -- 172.21.15.44:0/3607886424 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f262c106ed0 con 0x7f262c1032d0 2024-04-21T23:18:33.525 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.525+0000 7f2622ffd640 1 -- 172.21.15.44:0/3607886424 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11791 (secure 0 0 0) 0x7f261c09beb0 con 0x7f262c1032d0 2024-04-21T23:18:33.525 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:18:33.525 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:16:20.187271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26863":{"gid":26863,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3179646768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3179646768},{"type":"v1","addr":"172.21.15.69:6839","nonce":3179646768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:18:33.526 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 31 2024-04-21T23:18:33.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.527+0000 7f2633fe0640 1 -- 172.21.15.44:0/3607886424 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f26180a2d00 msgr2=0x7f26180a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.527+0000 7f2633fe0640 1 --2- 172.21.15.44:0/3607886424 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f26180a2d00 0x7f26180a51c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2628005ab0 tx=0x7f2628005a20 comp rx=0 tx=0).stop 2024-04-21T23:18:33.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.527+0000 7f2633fe0640 1 -- 172.21.15.44:0/3607886424 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f262c1032d0 msgr2=0x7f262c198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.527 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.527+0000 7f2633fe0640 1 --2- 172.21.15.44:0/3607886424 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f262c1032d0 0x7f262c198610 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f261c00e9f0 tx=0x7f261c00edf0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.527+0000 7f2633fe0640 1 -- 172.21.15.44:0/3607886424 shutdown_connections 2024-04-21T23:18:33.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.527+0000 7f2633fe0640 1 --2- 172.21.15.44:0/3607886424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f262c10bd80 0x7f262c19c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.527+0000 7f2633fe0640 1 --2- 172.21.15.44:0/3607886424 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f26180a2d00 0x7f26180a51c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.527+0000 7f2633fe0640 1 --2- 172.21.15.44:0/3607886424 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f262c1032d0 0x7f262c198610 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.527+0000 7f2633fe0640 1 --2- 172.21.15.44:0/3607886424 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f262c1029b0 0x7f262c1980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.528+0000 7f2633fe0640 1 -- 172.21.15.44:0/3607886424 >> 172.21.15.44:0/3607886424 conn(0x7f262c077740 msgr2=0x7f262c0fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:33.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.528+0000 7f2633fe0640 1 -- 172.21.15.44:0/3607886424 shutdown_connections 2024-04-21T23:18:33.528 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.528+0000 7f2633fe0640 1 -- 172.21.15.44:0/3607886424 wait complete. 2024-04-21T23:18:33.538 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:18:33.644 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.643+0000 7f7bdd513640 1 -- 172.21.15.44:0/289709607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bd81032d0 msgr2=0x7f7bd810b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.643+0000 7f7bdd513640 1 --2- 172.21.15.44:0/289709607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bd81032d0 0x7f7bd810b7b0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0009900 tx=0x7f7bc002bcd0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.643+0000 7f7bdd513640 1 -- 172.21.15.44:0/289709607 shutdown_connections 2024-04-21T23:18:33.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.643+0000 7f7bdd513640 1 --2- 172.21.15.44:0/289709607 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7bd810bcf0 0x7f7bd810e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.643+0000 7f7bdd513640 1 --2- 172.21.15.44:0/289709607 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bd81032d0 0x7f7bd810b7b0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.643+0000 7f7bdd513640 1 --2- 172.21.15.44:0/289709607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bd81029b0 0x7f7bd8102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.643+0000 7f7bdd513640 1 -- 172.21.15.44:0/289709607 >> 172.21.15.44:0/289709607 conn(0x7f7bd80777f0 msgr2=0x7f7bd8077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:33.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.643+0000 7f7bdd513640 1 -- 172.21.15.44:0/289709607 shutdown_connections 2024-04-21T23:18:33.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.643+0000 7f7bdd513640 1 -- 172.21.15.44:0/289709607 wait complete. 2024-04-21T23:18:33.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.643+0000 7f7bdd513640 1 Processor -- start 2024-04-21T23:18:33.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.643+0000 7f7bdd513640 1 -- start start 2024-04-21T23:18:33.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bdd513640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7bd81029b0 0x7f7bd8198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.646 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bdd513640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bd81032d0 0x7f7bd8198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.647 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bdd513640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bd810bcf0 0x7f7bd819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bdd513640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd8111af0 con 0x7f7bd81032d0 2024-04-21T23:18:33.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bdd513640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd8111970 con 0x7f7bd81029b0 2024-04-21T23:18:33.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bdd513640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd8111c70 con 0x7f7bd810bcf0 2024-04-21T23:18:33.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bcffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bd81032d0 0x7f7bd8198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bcffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bd81032d0 0x7f7bd8198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49376/0 (socket says 172.21.15.44:49376) 2024-04-21T23:18:33.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bcffff640 1 -- 172.21.15.44:0/3093209695 learned_addr learned my addr 172.21.15.44:0/3093209695 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:33.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bd6ffd640 1 --2- 172.21.15.44:0/3093209695 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7bd81029b0 0x7f7bd8198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bd77fe640 1 --2- 172.21.15.44:0/3093209695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bd810bcf0 0x7f7bd819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.650 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bcffff640 1 -- 172.21.15.44:0/3093209695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bd810bcf0 msgr2=0x7f7bd819c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bcffff640 1 --2- 172.21.15.44:0/3093209695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bd810bcf0 0x7f7bd819c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bcffff640 1 -- 172.21.15.44:0/3093209695 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7bd81029b0 msgr2=0x7f7bd8198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bcffff640 1 --2- 172.21.15.44:0/3093209695 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7bd81029b0 0x7f7bd8198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bcffff640 1 -- 172.21.15.44:0/3093209695 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd819d000 con 0x7f7bd81032d0 2024-04-21T23:18:33.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bd77fe640 1 --2- 172.21.15.44:0/3093209695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bd810bcf0 0x7f7bd819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:33.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bcffff640 1 --2- 172.21.15.44:0/3093209695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bd81032d0 0x7f7bd8198550 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0005940 tx=0x7f7bc002cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:33.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bd4ff9640 1 -- 172.21.15.44:0/3093209695 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc0041070 con 0x7f7bd81032d0 2024-04-21T23:18:33.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bd4ff9640 1 -- 172.21.15.44:0/3093209695 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bc0004450 con 0x7f7bd81032d0 2024-04-21T23:18:33.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bdd513640 1 -- 172.21.15.44:0/3093209695 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bd819d290 con 0x7f7bd81032d0 2024-04-21T23:18:33.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.644+0000 7f7bd4ff9640 1 -- 172.21.15.44:0/3093209695 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc0034440 con 0x7f7bd81032d0 2024-04-21T23:18:33.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.645+0000 7f7bdd513640 1 -- 172.21.15.44:0/3093209695 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bd81a4b80 con 0x7f7bd81032d0 2024-04-21T23:18:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.647+0000 7f7bd4ff9640 1 -- 172.21.15.44:0/3093209695 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7bc0044070 con 0x7f7bd81032d0 2024-04-21T23:18:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.647+0000 7f7be01de740 1 Processor -- start 2024-04-21T23:18:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.647+0000 7f7be01de740 1 -- start start 2024-04-21T23:18:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.647+0000 7f7be01de740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 0x56469b592810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.658 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.647+0000 7f7be01de740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592d50 0x56469b59b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.647+0000 7f7be01de740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 0x56469b59dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.647+0000 7f7be01de740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56469b4f2530 con 0x56469b592430 2024-04-21T23:18:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.647+0000 7f7be01de740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x56469b49a830 con 0x56469b592d50 2024-04-21T23:18:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.647+0000 7f7be01de740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56469b4ee4f0 con 0x56469b59b930 2024-04-21T23:18:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.648+0000 7f7bd77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 0x56469b592810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.648+0000 7f7bd77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 0x56469b592810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49384/0 (socket says 172.21.15.44:49384) 2024-04-21T23:18:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.648+0000 7f7bd77fe640 1 -- 172.21.15.44:0/735871219 learned_addr learned my addr 172.21.15.44:0/735871219 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:33.659 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.648+0000 7f7bcffff640 1 --2- 172.21.15.44:0/735871219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 0x56469b59dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.648+0000 7f7bd4ff9640 1 --2- 172.21.15.44:0/3093209695 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7bc40a2d00 0x7f7bc40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.648+0000 7f7bd6ffd640 1 --2- 172.21.15.44:0/735871219 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592d50 0x56469b59b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.648+0000 7f7bd77fe640 1 -- 172.21.15.44:0/735871219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 msgr2=0x56469b59dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.648+0000 7f7bd77fe640 1 --2- 172.21.15.44:0/735871219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 0x56469b59dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.648+0000 7f7bd77fe640 1 -- 172.21.15.44:0/735871219 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592d50 msgr2=0x56469b59b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.648+0000 7f7bd77fe640 1 --2- 172.21.15.44:0/735871219 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592d50 0x56469b59b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.648+0000 7f7bd77fe640 1 -- 172.21.15.44:0/735871219 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56469b4be2c0 con 0x56469b592430 2024-04-21T23:18:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.648+0000 7f7bd6ffd640 1 --2- 172.21.15.44:0/735871219 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592d50 0x56469b59b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:33.660 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.648+0000 7f7bd4ff9640 1 -- 172.21.15.44:0/3093209695 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7f7bc00f13c0 con 0x7f7bd81032d0 2024-04-21T23:18:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.648+0000 7f7bd77fe640 1 --2- 172.21.15.44:0/3093209695 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7bc40a2d00 0x7f7bc40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.648+0000 7f7bd77fe640 1 --2- 172.21.15.44:0/735871219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 0x56469b592810 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f7bd000f360 tx=0x7f7bd000f820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46cdb1c4396dfde1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.648+0000 7f7bcd7fa640 1 -- 172.21.15.44:0/735871219 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd001a610 con 0x56469b592430 2024-04-21T23:18:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.648+0000 7f7bcd7fa640 1 -- 172.21.15.44:0/735871219 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bd001ac90 con 0x56469b592430 2024-04-21T23:18:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.648+0000 7f7bcd7fa640 1 -- 172.21.15.44:0/735871219 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd0010a60 con 0x56469b592430 2024-04-21T23:18:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.649+0000 7f7be01de740 1 -- 172.21.15.44:0/735871219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 msgr2=0x56469b592810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.649+0000 7f7bd77fe640 1 --2- 172.21.15.44:0/3093209695 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7bc40a2d00 0x7f7bc40a51c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0020000 tx=0x7f7bd001d000 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.649+0000 7f7be01de740 1 --2- 172.21.15.44:0/735871219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 0x56469b592810 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f7bd000f360 tx=0x7f7bd000f820 comp rx=0 tx=0).stop 2024-04-21T23:18:33.661 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.649+0000 7f7be01de740 1 -- 172.21.15.44:0/735871219 shutdown_connections 2024-04-21T23:18:33.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.649+0000 7f7be01de740 1 --2- 172.21.15.44:0/735871219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 0x56469b59dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.649+0000 7f7be01de740 1 --2- 172.21.15.44:0/735871219 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592d50 0x56469b59b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.649+0000 7f7be01de740 1 --2- 172.21.15.44:0/735871219 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 0x56469b592810 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.649+0000 7f7be01de740 1 -- 172.21.15.44:0/735871219 >> 172.21.15.44:0/735871219 conn(0x56469b58cb60 msgr2=0x56469b58cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:33.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.649+0000 7f7be01de740 1 -- 172.21.15.44:0/735871219 shutdown_connections 2024-04-21T23:18:33.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.649+0000 7f7be01de740 1 -- 172.21.15.44:0/735871219 wait complete. 2024-04-21T23:18:33.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7be01de740 1 -- start start 2024-04-21T23:18:33.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7be01de740 1 Processor -- start 2024-04-21T23:18:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7be01de740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 0x56469b59b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7be01de740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592d50 0x56469b598690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7be01de740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 0x56469b598ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7be01de740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x56469b59e720 con 0x56469b592430 2024-04-21T23:18:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7be01de740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x56469b4adf10 con 0x56469b592d50 2024-04-21T23:18:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7be01de740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56469b59e8a0 con 0x56469b59b930 2024-04-21T23:18:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7be01de740 10 client.0 _ll_register_callbacks cb 0x56469b205be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:33.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7bd6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 0x56469b59b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7bd6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 0x56469b59b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49386/0 (socket says 172.21.15.44:49386) 2024-04-21T23:18:33.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7bd6ffd640 1 -- 172.21.15.44:0/1655584340 learned_addr learned my addr 172.21.15.44:0/1655584340 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:33.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7bd77fe640 1 --2- 172.21.15.44:0/1655584340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 0x56469b598ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7bd6ffd640 1 -- 172.21.15.44:0/1655584340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 msgr2=0x56469b598ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:33.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7bd6ffd640 1 --2- 172.21.15.44:0/1655584340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 0x56469b598ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7bd6ffd640 1 -- 172.21.15.44:0/1655584340 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592d50 msgr2=0x56469b598690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:33.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7bd6ffd640 1 --2- 172.21.15.44:0/1655584340 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592d50 0x56469b598690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7bd6ffd640 1 -- 172.21.15.44:0/1655584340 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56469b47fc80 con 0x56469b592430 2024-04-21T23:18:33.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7bd77fe640 1 --2- 172.21.15.44:0/1655584340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 0x56469b598ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:33.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7bd6ffd640 1 --2- 172.21.15.44:0/1655584340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 0x56469b59b300 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800b9a0 tx=0x7f7bc800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca8fea1e326a66f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:33.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.650+0000 7f7bbe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:18:33.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.651+0000 7f7bbe7fc640 1 -- 172.21.15.44:0/1655584340 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8003ea0 con 0x56469b592430 2024-04-21T23:18:33.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.651+0000 7f7bbe7fc640 1 -- 172.21.15.44:0/1655584340 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7f7bc8005220 con 0x56469b592430 2024-04-21T23:18:33.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.651+0000 7f7be01de740 1 -- 172.21.15.44:0/1655584340 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bd001ac90 con 0x56469b592430 2024-04-21T23:18:33.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.651+0000 7f7bbe7fc640 1 -- 172.21.15.44:0/1655584340 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7bc8023050 con 0x56469b592430 2024-04-21T23:18:33.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.651+0000 7f7be01de740 10 client.17718 fetch_fsmap learned FSMap version 31 2024-04-21T23:18:33.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.651+0000 7f7be01de740 1 -- 172.21.15.44:0/1655584340 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56469b2063f0 con 0x56469b592430 2024-04-21T23:18:33.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.651+0000 7f7bbe7fc640 1 -- 172.21.15.44:0/1655584340 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bd001ac90 con 0x56469b592430 2024-04-21T23:18:33.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.651+0000 7f7bbe7fc640 1 -- 172.21.15.44:0/1655584340 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f7bc8019030 con 0x56469b592430 2024-04-21T23:18:33.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.651+0000 7f7be01de740 10 client.17718 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:18:33.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.651+0000 7f7be01de740 10 client.17718 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:33.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.651+0000 7f7be01de740 20 client.17718 populate_metadata read hostname 'smithi044' 2024-04-21T23:18:33.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.651+0000 7f7be01de740 1 --2- 172.21.15.44:0/1655584340 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56469b5aba40 0x56469b5adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.651+0000 7f7be01de740 4 client.17718 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:18:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.651+0000 7f7be01de740 1 -- 172.21.15.44:0/1655584340 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56469b4ffa70 con 0x56469b5aba40 2024-04-21T23:18:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.651+0000 7f7bcffff640 1 --2- 172.21.15.44:0/1655584340 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56469b5aba40 0x56469b5adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.652+0000 7f7bbe7fc640 10 client.17718 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.652+0000 7f7bcffff640 1 --2- 172.21.15.44:0/1655584340 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56469b5aba40 0x56469b5adeb0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8c6a4c8efc23e02 server_cookie=3ee8a77dbcf6a840 in_seq=0 out_seq=0 2024-04-21T23:18:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.652+0000 7f7bbe7fc640 1 -- 172.21.15.44:0/1655584340 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56469b4ffa70 con 0x56469b5aba40 2024-04-21T23:18:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.652+0000 7f7bbe7fc640 10 client.17718 handle_command_reply: tid=0 2024-04-21T23:18:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.653+0000 7f7be01de740 1 client.17718 shutdown 2024-04-21T23:18:33.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.653+0000 7f7be01de740 1 -- 172.21.15.44:0/1655584340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 msgr2=0x56469b59b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.653+0000 7f7be01de740 1 --2- 172.21.15.44:0/1655584340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 0x56469b59b300 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800b9a0 tx=0x7f7bc800bda0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.653+0000 7f7be01de740 1 -- 172.21.15.44:0/1655584340 shutdown_connections 2024-04-21T23:18:33.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.653+0000 7f7be01de740 1 --2- 172.21.15.44:0/1655584340 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56469b5aba40 0x56469b5adeb0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.653+0000 7f7be01de740 1 --2- 172.21.15.44:0/1655584340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 0x56469b598ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.653+0000 7f7bbe7fc640 0 client.17718 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:33.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.653+0000 7f7be01de740 1 --2- 172.21.15.44:0/1655584340 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592d50 0x56469b598690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.653+0000 7f7be01de740 1 --2- 172.21.15.44:0/1655584340 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 0x56469b59b300 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.653+0000 7f7be01de740 1 -- 172.21.15.44:0/1655584340 >> 172.21.15.44:0/1655584340 conn(0x56469b58cb60 msgr2=0x56469b58e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:33.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.653+0000 7f7be01de740 1 -- 172.21.15.44:0/1655584340 shutdown_connections 2024-04-21T23:18:33.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.653+0000 7f7be01de740 1 -- 172.21.15.44:0/1655584340 wait complete. 2024-04-21T23:18:33.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.653+0000 7f7be01de740 20 client.17718 trim_cache size 0 max 16384 2024-04-21T23:18:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.666+0000 7f7be01de740 1 Processor -- start 2024-04-21T23:18:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.666+0000 7f7be01de740 1 -- start start 2024-04-21T23:18:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.666+0000 7f7be01de740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592430 0x56469b5a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.666+0000 7f7be01de740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592d50 0x56469b59adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.667+0000 7f7be01de740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 0x56469b59b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.667+0000 7f7be01de740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd81aab40 con 0x56469b592d50 2024-04-21T23:18:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.667+0000 7f7be01de740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd81ad3a0 con 0x56469b592430 2024-04-21T23:18:33.675 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.667+0000 7f7be01de740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd81ab190 con 0x56469b59b930 2024-04-21T23:18:33.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.667+0000 7f7bcffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592d50 0x56469b59adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.667+0000 7f7bcffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592d50 0x56469b59adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49400/0 (socket says 172.21.15.44:49400) 2024-04-21T23:18:33.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.667+0000 7f7bcffff640 1 -- 172.21.15.44:0/2438555288 learned_addr learned my addr 172.21.15.44:0/2438555288 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:33.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.667+0000 7f7bd6ffd640 1 --2- 172.21.15.44:0/2438555288 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592430 0x56469b5a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.667+0000 7f7bd77fe640 1 --2- 172.21.15.44:0/2438555288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 0x56469b59b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.667+0000 7f7bcffff640 1 -- 172.21.15.44:0/2438555288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 msgr2=0x56469b59b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:33.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.667+0000 7f7bcffff640 1 --2- 172.21.15.44:0/2438555288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 0x56469b59b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.676 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.667+0000 7f7bcffff640 1 -- 172.21.15.44:0/2438555288 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592430 msgr2=0x56469b5a75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:33.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.667+0000 7f7bcffff640 1 --2- 172.21.15.44:0/2438555288 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592430 0x56469b5a75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.667+0000 7f7bcffff640 1 -- 172.21.15.44:0/2438555288 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56469b2d07d0 con 0x56469b592d50 2024-04-21T23:18:33.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.667+0000 7f7bd6ffd640 1 --2- 172.21.15.44:0/2438555288 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592430 0x56469b5a75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:33.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.667+0000 7f7bd77fe640 1 --2- 172.21.15.44:0/2438555288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 0x56469b59b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:33.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.667+0000 7f7bcffff640 1 --2- 172.21.15.44:0/2438555288 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592d50 0x56469b59adc0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0006010 tx=0x7f7bc002bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6957858c3ca4f2c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:33.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.667+0000 7f7bccff9640 1 -- 172.21.15.44:0/2438555288 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc004b040 con 0x56469b592d50 2024-04-21T23:18:33.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.667+0000 7f7bccff9640 1 -- 172.21.15.44:0/2438555288 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bc00b9040 con 0x56469b592d50 2024-04-21T23:18:33.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.667+0000 7f7bccff9640 1 -- 172.21.15.44:0/2438555288 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc00bd600 con 0x56469b592d50 2024-04-21T23:18:33.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.668+0000 7f7be01de740 1 -- 172.21.15.44:0/2438555288 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592d50 msgr2=0x56469b59adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.668+0000 7f7be01de740 1 --2- 172.21.15.44:0/2438555288 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592d50 0x56469b59adc0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0006010 tx=0x7f7bc002bcd0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.668+0000 7f7be01de740 1 -- 172.21.15.44:0/2438555288 shutdown_connections 2024-04-21T23:18:33.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.668+0000 7f7be01de740 1 --2- 172.21.15.44:0/2438555288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 0x56469b59b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.668+0000 7f7be01de740 1 --2- 172.21.15.44:0/2438555288 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592d50 0x56469b59adc0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.668+0000 7f7be01de740 1 --2- 172.21.15.44:0/2438555288 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592430 0x56469b5a75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.668+0000 7f7be01de740 1 -- 172.21.15.44:0/2438555288 >> 172.21.15.44:0/2438555288 conn(0x56469b58cb60 msgr2=0x56469b587850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:33.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.668+0000 7f7be01de740 1 -- 172.21.15.44:0/2438555288 shutdown_connections 2024-04-21T23:18:33.678 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.668+0000 7f7be01de740 1 -- 172.21.15.44:0/2438555288 wait complete. 2024-04-21T23:18:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.668+0000 7f7be01de740 1 -- start start 2024-04-21T23:18:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.668+0000 7f7be01de740 1 Processor -- start 2024-04-21T23:18:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.668+0000 7f7be01de740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 0x56469b5a4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7be01de740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592d50 0x56469b59f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7be01de740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 0x56469b59fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7be01de740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd81ab860 con 0x56469b592430 2024-04-21T23:18:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7be01de740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd81accd0 con 0x56469b592d50 2024-04-21T23:18:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7be01de740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd81abf30 con 0x56469b59b930 2024-04-21T23:18:33.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7be01de740 10 client.0 _ll_register_callbacks cb 0x7f7bc8004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7bd6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 0x56469b5a4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7bd6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 0x56469b5a4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49414/0 (socket says 172.21.15.44:49414) 2024-04-21T23:18:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7bd6ffd640 1 -- 172.21.15.44:0/1172547792 learned_addr learned my addr 172.21.15.44:0/1172547792 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7bcffff640 1 --2- 172.21.15.44:0/1172547792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592d50 0x56469b59f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7bd77fe640 1 --2- 172.21.15.44:0/1172547792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 0x56469b59fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7bd6ffd640 1 -- 172.21.15.44:0/1172547792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 msgr2=0x56469b59fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7bd6ffd640 1 --2- 172.21.15.44:0/1172547792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 0x56469b59fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7bd6ffd640 1 -- 172.21.15.44:0/1172547792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592d50 msgr2=0x56469b59f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:33.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7bd6ffd640 1 --2- 172.21.15.44:0/1172547792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592d50 0x56469b59f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7bd6ffd640 1 -- 172.21.15.44:0/1172547792 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56469b610ff0 con 0x56469b592430 2024-04-21T23:18:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7bd6ffd640 1 --2- 172.21.15.44:0/1172547792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 0x56469b5a4070 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800b9a0 tx=0x7f7bc8003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd0d882b12ce3c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7bbe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:18:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7bbe7fc640 1 -- 172.21.15.44:0/1172547792 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8011960 con 0x56469b592430 2024-04-21T23:18:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7bbe7fc640 1 -- 172.21.15.44:0/1172547792 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7f7bc8011b00 con 0x56469b592430 2024-04-21T23:18:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7be01de740 1 -- 172.21.15.44:0/1172547792 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bc00b9040 con 0x56469b592430 2024-04-21T23:18:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7bbe7fc640 1 -- 172.21.15.44:0/1172547792 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7bc8038050 con 0x56469b592430 2024-04-21T23:18:33.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7be01de740 10 client.17724 fetch_fsmap learned FSMap version 31 2024-04-21T23:18:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.669+0000 7f7be01de740 1 -- 172.21.15.44:0/1172547792 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56469b612180 con 0x56469b592430 2024-04-21T23:18:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.670+0000 7f7bbe7fc640 1 -- 172.21.15.44:0/1172547792 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bc00b9040 con 0x56469b592430 2024-04-21T23:18:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.670+0000 7f7bbe7fc640 1 -- 172.21.15.44:0/1172547792 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f7bc8019040 con 0x56469b592430 2024-04-21T23:18:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.670+0000 7f7be01de740 10 client.17724 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:18:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.670+0000 7f7be01de740 10 client.17724 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.670+0000 7f7be01de740 20 client.17724 populate_metadata read hostname 'smithi044' 2024-04-21T23:18:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.670+0000 7f7be01de740 1 --2- 172.21.15.44:0/1172547792 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56469b612900 0x56469b614d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:33.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.670+0000 7f7be01de740 4 client.17724 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:18:33.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.670+0000 7f7be01de740 1 -- 172.21.15.44:0/1172547792 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56469b4f9dc0 con 0x56469b612900 2024-04-21T23:18:33.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.670+0000 7f7bcffff640 1 --2- 172.21.15.44:0/1172547792 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56469b612900 0x56469b614d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:33.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.670+0000 7f7bcffff640 1 --2- 172.21.15.44:0/1172547792 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56469b612900 0x56469b614d20 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e28379ac4147ea33 server_cookie=ef1111d1d7116410 in_seq=0 out_seq=0 2024-04-21T23:18:33.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.670+0000 7f7bbe7fc640 10 client.17724 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:33.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7bbe7fc640 1 -- 172.21.15.44:0/1172547792 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56469b4f9dc0 con 0x56469b612900 2024-04-21T23:18:33.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7bbe7fc640 10 client.17724 handle_command_reply: tid=0 2024-04-21T23:18:33.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7be01de740 1 client.17724 shutdown 2024-04-21T23:18:33.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7be01de740 1 -- 172.21.15.44:0/1172547792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 msgr2=0x56469b5a4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7be01de740 1 --2- 172.21.15.44:0/1172547792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 0x56469b5a4070 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800b9a0 tx=0x7f7bc8003f90 comp rx=0 tx=0).stop 2024-04-21T23:18:33.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7be01de740 1 -- 172.21.15.44:0/1172547792 shutdown_connections 2024-04-21T23:18:33.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7be01de740 1 --2- 172.21.15.44:0/1172547792 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x56469b612900 0x56469b614d20 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7be01de740 1 --2- 172.21.15.44:0/1172547792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56469b59b930 0x56469b59fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7be01de740 1 --2- 172.21.15.44:0/1172547792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x56469b592d50 0x56469b59f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7bbe7fc640 0 client.17724 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:33.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7be01de740 1 --2- 172.21.15.44:0/1172547792 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x56469b592430 0x56469b5a4070 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7be01de740 1 -- 172.21.15.44:0/1172547792 >> 172.21.15.44:0/1172547792 conn(0x56469b58cb60 msgr2=0x56469b593270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:33.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7be01de740 1 -- 172.21.15.44:0/1172547792 shutdown_connections 2024-04-21T23:18:33.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7be01de740 1 -- 172.21.15.44:0/1172547792 wait complete. 2024-04-21T23:18:33.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7be01de740 20 client.17724 trim_cache size 0 max 16384 2024-04-21T23:18:33.685 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-21T23:18:33.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7bdd513640 1 -- 172.21.15.44:0/3093209695 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7bc40a2d00 msgr2=0x7f7bc40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7bdd513640 1 --2- 172.21.15.44:0/3093209695 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7bc40a2d00 0x7f7bc40a51c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0020000 tx=0x7f7bd001d000 comp rx=0 tx=0).stop 2024-04-21T23:18:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7bdd513640 1 -- 172.21.15.44:0/3093209695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bd81032d0 msgr2=0x7f7bd8198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7bdd513640 1 --2- 172.21.15.44:0/3093209695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bd81032d0 0x7f7bd8198550 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f7bc0005940 tx=0x7f7bc002cd10 comp rx=0 tx=0).stop 2024-04-21T23:18:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7bdd513640 1 -- 172.21.15.44:0/3093209695 shutdown_connections 2024-04-21T23:18:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7bdd513640 1 --2- 172.21.15.44:0/3093209695 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7bd810bcf0 0x7f7bd819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7bdd513640 1 --2- 172.21.15.44:0/3093209695 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7bc40a2d00 0x7f7bc40a51c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7bdd513640 1 --2- 172.21.15.44:0/3093209695 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7bd81032d0 0x7f7bd8198550 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7bdd513640 1 --2- 172.21.15.44:0/3093209695 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7bd81029b0 0x7f7bd8198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7bdd513640 1 -- 172.21.15.44:0/3093209695 >> 172.21.15.44:0/3093209695 conn(0x7f7bd80777f0 msgr2=0x7f7bd80fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7bdd513640 1 -- 172.21.15.44:0/3093209695 shutdown_connections 2024-04-21T23:18:33.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:33.671+0000 7f7bdd513640 1 -- 172.21.15.44:0/3093209695 wait complete. 2024-04-21T23:18:34.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:34 smithi044 ceph-mon[27934]: pgmap v307: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:18:34.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:34 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3607886424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:18:34.689 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T23:18:34.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:18:34.690 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T23:18:34.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:34 smithi069 ceph-mon[31816]: pgmap v307: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:18:34.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:34 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3607886424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:18:34.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:34 smithi055 ceph-mon[31117]: pgmap v307: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:18:34.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:34 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3607886424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:18:34.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.810+0000 7fa3cd6c9640 1 -- 172.21.15.44:0/3992929076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3c81029b0 msgr2=0x7fa3c8102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:34.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.810+0000 7fa3cd6c9640 1 --2- 172.21.15.44:0/3992929076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3c81029b0 0x7fa3c8102d90 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8009900 tx=0x7fa3b802bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:34.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.810+0000 7fa3cd6c9640 1 -- 172.21.15.44:0/3992929076 shutdown_connections 2024-04-21T23:18:34.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.810+0000 7fa3cd6c9640 1 --2- 172.21.15.44:0/3992929076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c810bec0 0x7fa3c810e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.810+0000 7fa3cd6c9640 1 --2- 172.21.15.44:0/3992929076 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3c8103360 0x7fa3c810b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.810+0000 7fa3cd6c9640 1 --2- 172.21.15.44:0/3992929076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3c81029b0 0x7fa3c8102d90 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.810+0000 7fa3cd6c9640 1 -- 172.21.15.44:0/3992929076 >> 172.21.15.44:0/3992929076 conn(0x7fa3c8077760 msgr2=0x7fa3c8077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:34.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.810+0000 7fa3cd6c9640 1 -- 172.21.15.44:0/3992929076 shutdown_connections 2024-04-21T23:18:34.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.810+0000 7fa3cd6c9640 1 -- 172.21.15.44:0/3992929076 wait complete. 2024-04-21T23:18:34.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.810+0000 7fa3cd6c9640 1 Processor -- start 2024-04-21T23:18:34.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.810+0000 7fa3cd6c9640 1 -- start start 2024-04-21T23:18:34.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3cd6c9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3c81029b0 0x7fa3c81981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3cd6c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c810bec0 0x7fa3c8198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3cd6c9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3c819ca60 0x7fa3c819cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3cd6c9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c8111cd0 con 0x7fa3c81029b0 2024-04-21T23:18:34.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3cd6c9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c8111b50 con 0x7fa3c819ca60 2024-04-21T23:18:34.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3cd6c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c8111e50 con 0x7fa3c810bec0 2024-04-21T23:18:34.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3c6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3c81029b0 0x7fa3c81981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3c6ffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3c81029b0 0x7fa3c81981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49432/0 (socket says 172.21.15.44:49432) 2024-04-21T23:18:34.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3c6ffd640 1 -- 172.21.15.44:0/306673263 learned_addr learned my addr 172.21.15.44:0/306673263 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:34.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3bffff640 1 --2- 172.21.15.44:0/306673263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c810bec0 0x7fa3c8198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3c77fe640 1 --2- 172.21.15.44:0/306673263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3c819ca60 0x7fa3c819cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3c6ffd640 1 -- 172.21.15.44:0/306673263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c810bec0 msgr2=0x7fa3c8198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:34.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3c6ffd640 1 --2- 172.21.15.44:0/306673263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c810bec0 0x7fa3c8198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3c6ffd640 1 -- 172.21.15.44:0/306673263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3c819ca60 msgr2=0x7fa3c819cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:34.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3c6ffd640 1 --2- 172.21.15.44:0/306673263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3c819ca60 0x7fa3c819cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3c6ffd640 1 -- 172.21.15.44:0/306673263 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3c819d550 con 0x7fa3c81029b0 2024-04-21T23:18:34.814 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3c77fe640 1 --2- 172.21.15.44:0/306673263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3c819ca60 0x7fa3c819cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:34.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3c6ffd640 1 --2- 172.21.15.44:0/306673263 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3c81029b0 0x7fa3c81981f0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8006010 tx=0x7fa3b802cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:34.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3c4ff9640 1 -- 172.21.15.44:0/306673263 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b8041070 con 0x7fa3c81029b0 2024-04-21T23:18:34.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3cd6c9640 1 -- 172.21.15.44:0/306673263 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3c819d7e0 con 0x7fa3c81029b0 2024-04-21T23:18:34.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3c4ff9640 1 -- 172.21.15.44:0/306673263 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3b8004450 con 0x7fa3c81029b0 2024-04-21T23:18:34.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.811+0000 7fa3c4ff9640 1 -- 172.21.15.44:0/306673263 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b8034440 con 0x7fa3c81029b0 2024-04-21T23:18:34.815 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.812+0000 7fa3cd6c9640 1 -- 172.21.15.44:0/306673263 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3c81a4fb0 con 0x7fa3c81029b0 2024-04-21T23:18:34.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.814+0000 7fa3c4ff9640 1 -- 172.21.15.44:0/306673263 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa3b8044070 con 0x7fa3c81029b0 2024-04-21T23:18:34.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.814+0000 7fa3d038c740 1 Processor -- start 2024-04-21T23:18:34.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.814+0000 7fa3d038c740 1 -- start start 2024-04-21T23:18:34.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.814+0000 7fa3d038c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0b5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.814+0000 7fa3d038c740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0b5a40 0x55b73f0b5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.814+0000 7fa3d038c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0bedb0 0x55b73f0c11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.814+0000 7fa3d038c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b73f019170 con 0x55b73f0b6720 2024-04-21T23:18:34.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.814+0000 7fa3d038c740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55b73efc17c0 con 0x55b73f0b5a40 2024-04-21T23:18:34.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.814+0000 7fa3d038c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b73f015130 con 0x55b73f0bedb0 2024-04-21T23:18:34.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.815+0000 7fa3c77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.815+0000 7fa3c77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0b5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49442/0 (socket says 172.21.15.44:49442) 2024-04-21T23:18:34.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.815+0000 7fa3c4ff9640 1 --2- 172.21.15.44:0/306673263 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fa3b40a2d00 0x7fa3b40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.815+0000 7fa3c77fe640 1 -- 172.21.15.44:0/2203035226 learned_addr learned my addr 172.21.15.44:0/2203035226 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:34.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.815+0000 7fa3c6ffd640 1 --2- 172.21.15.44:0/2203035226 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0b5a40 0x55b73f0b5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.815+0000 7fa3bffff640 1 --2- 172.21.15.44:0/2203035226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0bedb0 0x55b73f0c11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.815+0000 7fa3c77fe640 1 --2- 172.21.15.44:0/306673263 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fa3b40a2d00 0x7fa3b40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.815+0000 7fa3c4ff9640 1 -- 172.21.15.44:0/306673263 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7fa3b80f13e0 con 0x7fa3c81029b0 2024-04-21T23:18:34.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.815+0000 7fa3c77fe640 1 -- 172.21.15.44:0/2203035226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0bedb0 msgr2=0x55b73f0c11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:34.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.815+0000 7fa3c77fe640 1 --2- 172.21.15.44:0/2203035226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0bedb0 0x55b73f0c11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.815+0000 7fa3c77fe640 1 -- 172.21.15.44:0/2203035226 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0b5a40 msgr2=0x55b73f0b5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:34.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.815+0000 7fa3c77fe640 1 --2- 172.21.15.44:0/2203035226 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0b5a40 0x55b73f0b5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.815+0000 7fa3c77fe640 1 -- 172.21.15.44:0/2203035226 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b73efd2e60 con 0x55b73f0b6720 2024-04-21T23:18:34.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.815+0000 7fa3c77fe640 1 --2- 172.21.15.44:0/2203035226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0b5470 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0013910 tx=0x7fa3c0013dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10958a62510abaa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:34.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.815+0000 7fa3bd7fa640 1 -- 172.21.15.44:0/2203035226 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0004b20 con 0x55b73f0b6720 2024-04-21T23:18:34.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.815+0000 7fa3c77fe640 1 --2- 172.21.15.44:0/306673263 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fa3b40a2d00 0x7fa3b40a51c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0020000 tx=0x7fa3c001f000 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:34.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.815+0000 7fa3bd7fa640 1 -- 172.21.15.44:0/2203035226 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3c0004cc0 con 0x55b73f0b6720 2024-04-21T23:18:34.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.815+0000 7fa3bd7fa640 1 -- 172.21.15.44:0/2203035226 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c00059a0 con 0x55b73f0b6720 2024-04-21T23:18:34.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.816+0000 7fa3d038c740 1 -- 172.21.15.44:0/2203035226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 msgr2=0x55b73f0b5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:34.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.816+0000 7fa3d038c740 1 --2- 172.21.15.44:0/2203035226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0b5470 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0013910 tx=0x7fa3c0013dd0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.816+0000 7fa3d038c740 1 -- 172.21.15.44:0/2203035226 shutdown_connections 2024-04-21T23:18:34.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.816+0000 7fa3d038c740 1 --2- 172.21.15.44:0/2203035226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0bedb0 0x55b73f0c11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.816+0000 7fa3d038c740 1 --2- 172.21.15.44:0/2203035226 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0b5a40 0x55b73f0b5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.816+0000 7fa3d038c740 1 --2- 172.21.15.44:0/2203035226 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0b5470 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.816+0000 7fa3d038c740 1 -- 172.21.15.44:0/2203035226 >> 172.21.15.44:0/2203035226 conn(0x55b73f07c4f0 msgr2=0x55b73f0b0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:34.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.816+0000 7fa3d038c740 1 -- 172.21.15.44:0/2203035226 shutdown_connections 2024-04-21T23:18:34.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.816+0000 7fa3d038c740 1 -- 172.21.15.44:0/2203035226 wait complete. 2024-04-21T23:18:34.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.817+0000 7fa3d038c740 1 -- start start 2024-04-21T23:18:34.826 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.817+0000 7fa3d038c740 1 Processor -- start 2024-04-21T23:18:34.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.817+0000 7fa3d038c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0b5a40 0x55b73f0bbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.817+0000 7fa3d038c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0bc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.817+0000 7fa3d038c740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0bedb0 0x55b73f0bc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.817+0000 7fa3d038c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55b73f0c1c20 con 0x55b73f0b6720 2024-04-21T23:18:34.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.817+0000 7fa3d038c740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55b73efd4d40 con 0x55b73f0bedb0 2024-04-21T23:18:34.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.817+0000 7fa3d038c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b73f0c1da0 con 0x55b73f0b5a40 2024-04-21T23:18:34.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.817+0000 7fa3d038c740 10 client.0 _ll_register_callbacks cb 0x55b73f010820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:34.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.817+0000 7fa3bffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0bc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.817+0000 7fa3bffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0bc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49458/0 (socket says 172.21.15.44:49458) 2024-04-21T23:18:34.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.817+0000 7fa3bffff640 1 -- 172.21.15.44:0/986770009 learned_addr learned my addr 172.21.15.44:0/986770009 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:34.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.817+0000 7fa3c77fe640 1 --2- 172.21.15.44:0/986770009 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0bedb0 0x55b73f0bc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.817+0000 7fa3c6ffd640 1 --2- 172.21.15.44:0/986770009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0b5a40 0x55b73f0bbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.817+0000 7fa3bffff640 1 -- 172.21.15.44:0/986770009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0b5a40 msgr2=0x55b73f0bbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:34.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.817+0000 7fa3bffff640 1 --2- 172.21.15.44:0/986770009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0b5a40 0x55b73f0bbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.817+0000 7fa3bffff640 1 -- 172.21.15.44:0/986770009 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0bedb0 msgr2=0x55b73f0bc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:34.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.817+0000 7fa3bffff640 1 --2- 172.21.15.44:0/986770009 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0bedb0 0x55b73f0bc840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.817+0000 7fa3bffff640 1 -- 172.21.15.44:0/986770009 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b73edf57e0 con 0x55b73f0b6720 2024-04-21T23:18:34.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.818+0000 7fa3bffff640 1 --2- 172.21.15.44:0/986770009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0bc300 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fa3b000cec0 tx=0x7fa3b00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fabec5521a379f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:34.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.818+0000 7fa3ae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:18:34.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.818+0000 7fa3ae7fc640 1 -- 172.21.15.44:0/986770009 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0011a50 con 0x55b73f0b6720 2024-04-21T23:18:34.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.818+0000 7fa3ae7fc640 1 -- 172.21.15.44:0/986770009 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7fa3b0011bf0 con 0x55b73f0b6720 2024-04-21T23:18:34.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.818+0000 7fa3d038c740 1 -- 172.21.15.44:0/986770009 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3c0004cc0 con 0x55b73f0b6720 2024-04-21T23:18:34.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.818+0000 7fa3ae7fc640 1 -- 172.21.15.44:0/986770009 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa3b0011df0 con 0x55b73f0b6720 2024-04-21T23:18:34.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.818+0000 7fa3d038c740 10 client.17736 fetch_fsmap learned FSMap version 31 2024-04-21T23:18:34.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.818+0000 7fa3d038c740 1 -- 172.21.15.44:0/986770009 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b73ee55f70 con 0x55b73f0b6720 2024-04-21T23:18:34.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.818+0000 7fa3ae7fc640 1 -- 172.21.15.44:0/986770009 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3c0004cc0 con 0x55b73f0b6720 2024-04-21T23:18:34.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.818+0000 7fa3ae7fc640 1 -- 172.21.15.44:0/986770009 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7fa3b0018070 con 0x55b73f0b6720 2024-04-21T23:18:34.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.818+0000 7fa3d038c740 10 client.17736 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:18:34.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.818+0000 7fa3d038c740 10 client.17736 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:34.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.818+0000 7fa3d038c740 20 client.17736 populate_metadata read hostname 'smithi044' 2024-04-21T23:18:34.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.818+0000 7fa3d038c740 1 --2- 172.21.15.44:0/986770009 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b73f0cef20 0x55b73f0d13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.818+0000 7fa3d038c740 4 client.17736 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:18:34.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.818+0000 7fa3d038c740 1 -- 172.21.15.44:0/986770009 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b73efed450 con 0x55b73f0cef20 2024-04-21T23:18:34.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.819+0000 7fa3c6ffd640 1 --2- 172.21.15.44:0/986770009 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b73f0cef20 0x55b73f0d13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.819+0000 7fa3ae7fc640 10 client.17736 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:34.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.819+0000 7fa3c6ffd640 1 --2- 172.21.15.44:0/986770009 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b73f0cef20 0x55b73f0d13e0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dc4cefad0cfac56 server_cookie=e32544433ab7bf7f in_seq=0 out_seq=0 2024-04-21T23:18:34.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.820+0000 7fa3ae7fc640 1 -- 172.21.15.44:0/986770009 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b73efed450 con 0x55b73f0cef20 2024-04-21T23:18:34.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.820+0000 7fa3ae7fc640 10 client.17736 handle_command_reply: tid=0 2024-04-21T23:18:34.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.820+0000 7fa3d038c740 1 client.17736 shutdown 2024-04-21T23:18:34.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.820+0000 7fa3d038c740 1 -- 172.21.15.44:0/986770009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 msgr2=0x55b73f0bc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:34.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.820+0000 7fa3d038c740 1 --2- 172.21.15.44:0/986770009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0bc300 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fa3b000cec0 tx=0x7fa3b00039c0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.820+0000 7fa3d038c740 1 -- 172.21.15.44:0/986770009 shutdown_connections 2024-04-21T23:18:34.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.820+0000 7fa3d038c740 1 --2- 172.21.15.44:0/986770009 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b73f0cef20 0x55b73f0d13e0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.820+0000 7fa3d038c740 1 --2- 172.21.15.44:0/986770009 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0bedb0 0x55b73f0bc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.820+0000 7fa3d038c740 1 --2- 172.21.15.44:0/986770009 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0bc300 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.820+0000 7fa3d038c740 1 --2- 172.21.15.44:0/986770009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0b5a40 0x55b73f0bbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.820+0000 7fa3d038c740 1 -- 172.21.15.44:0/986770009 >> 172.21.15.44:0/986770009 conn(0x55b73f07c4f0 msgr2=0x55b73f0b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:34.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.820+0000 7fa3ae7fc640 0 client.17736 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:34.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.820+0000 7fa3d038c740 1 -- 172.21.15.44:0/986770009 shutdown_connections 2024-04-21T23:18:34.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.820+0000 7fa3d038c740 1 -- 172.21.15.44:0/986770009 wait complete. 2024-04-21T23:18:34.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.820+0000 7fa3d038c740 20 client.17736 trim_cache size 0 max 16384 2024-04-21T23:18:34.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.838+0000 7fa3d038c740 1 Processor -- start 2024-04-21T23:18:34.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.838+0000 7fa3d038c740 1 -- start start 2024-04-21T23:18:34.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.838+0000 7fa3d038c740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0b5a40 0x55b73f0be4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.842 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.838+0000 7fa3d038c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0cab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.838+0000 7fa3d038c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0bedb0 0x55b73f0d6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.838+0000 7fa3d038c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c81acdb0 con 0x55b73f0b6720 2024-04-21T23:18:34.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.838+0000 7fa3d038c740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c81af610 con 0x55b73f0b5a40 2024-04-21T23:18:34.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.838+0000 7fa3d038c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c81ad400 con 0x55b73f0bedb0 2024-04-21T23:18:34.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.838+0000 7fa3bffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.839+0000 7fa3bffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0cab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49464/0 (socket says 172.21.15.44:49464) 2024-04-21T23:18:34.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.839+0000 7fa3bffff640 1 -- 172.21.15.44:0/1517724185 learned_addr learned my addr 172.21.15.44:0/1517724185 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:34.843 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.839+0000 7fa3c77fe640 1 --2- 172.21.15.44:0/1517724185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0bedb0 0x55b73f0d6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.839+0000 7fa3c6ffd640 1 --2- 172.21.15.44:0/1517724185 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0b5a40 0x55b73f0be4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.839+0000 7fa3bffff640 1 -- 172.21.15.44:0/1517724185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0bedb0 msgr2=0x55b73f0d6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:34.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.839+0000 7fa3bffff640 1 --2- 172.21.15.44:0/1517724185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0bedb0 0x55b73f0d6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.839+0000 7fa3bffff640 1 -- 172.21.15.44:0/1517724185 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0b5a40 msgr2=0x55b73f0be4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:34.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.839+0000 7fa3bffff640 1 --2- 172.21.15.44:0/1517724185 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0b5a40 0x55b73f0be4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.839+0000 7fa3bffff640 1 -- 172.21.15.44:0/1517724185 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b73eced170 con 0x55b73f0b6720 2024-04-21T23:18:34.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.839+0000 7fa3c77fe640 1 --2- 172.21.15.44:0/1517724185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0bedb0 0x55b73f0d6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:34.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.839+0000 7fa3bffff640 1 --2- 172.21.15.44:0/1517724185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0cab20 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0019760 tx=0x7fa3b00195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d692e29db06fc843 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:34.844 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.839+0000 7fa3bcff9640 1 -- 172.21.15.44:0/1517724185 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b002c020 con 0x55b73f0b6720 2024-04-21T23:18:34.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.839+0000 7fa3bcff9640 1 -- 172.21.15.44:0/1517724185 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3b0019e30 con 0x55b73f0b6720 2024-04-21T23:18:34.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.839+0000 7fa3bcff9640 1 -- 172.21.15.44:0/1517724185 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b00055c0 con 0x55b73f0b6720 2024-04-21T23:18:34.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.840+0000 7fa3d038c740 1 -- 172.21.15.44:0/1517724185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 msgr2=0x55b73f0cab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:34.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.840+0000 7fa3d038c740 1 --2- 172.21.15.44:0/1517724185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0cab20 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0019760 tx=0x7fa3b00195f0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.840+0000 7fa3d038c740 1 -- 172.21.15.44:0/1517724185 shutdown_connections 2024-04-21T23:18:34.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.840+0000 7fa3d038c740 1 --2- 172.21.15.44:0/1517724185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0bedb0 0x55b73f0d6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.840+0000 7fa3d038c740 1 --2- 172.21.15.44:0/1517724185 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0cab20 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.845 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.840+0000 7fa3d038c740 1 --2- 172.21.15.44:0/1517724185 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0b5a40 0x55b73f0be4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.840+0000 7fa3d038c740 1 -- 172.21.15.44:0/1517724185 >> 172.21.15.44:0/1517724185 conn(0x55b73f07c4f0 msgr2=0x55b73f0af4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:34.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.840+0000 7fa3d038c740 1 -- 172.21.15.44:0/1517724185 shutdown_connections 2024-04-21T23:18:34.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.840+0000 7fa3d038c740 1 -- 172.21.15.44:0/1517724185 wait complete. 2024-04-21T23:18:34.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.840+0000 7fa3d038c740 1 -- start start 2024-04-21T23:18:34.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.840+0000 7fa3d038c740 1 Processor -- start 2024-04-21T23:18:34.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.840+0000 7fa3d038c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0b5a40 0x55b73f0c7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.840+0000 7fa3d038c740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0c35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3d038c740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0bedb0 0x55b73f0c3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.846 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3d038c740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c81adad0 con 0x55b73f0b6720 2024-04-21T23:18:34.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3d038c740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c81aef40 con 0x55b73f0bedb0 2024-04-21T23:18:34.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3d038c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c81ae1a0 con 0x55b73f0b5a40 2024-04-21T23:18:34.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3d038c740 10 client.0 _ll_register_callbacks cb 0x7fa3b0004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:34.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3bffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0c35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3bffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0c35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49472/0 (socket says 172.21.15.44:49472) 2024-04-21T23:18:34.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3bffff640 1 -- 172.21.15.44:0/4012387808 learned_addr learned my addr 172.21.15.44:0/4012387808 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:34.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3c77fe640 1 --2- 172.21.15.44:0/4012387808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0bedb0 0x55b73f0c3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.847 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3c6ffd640 1 --2- 172.21.15.44:0/4012387808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0b5a40 0x55b73f0c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3bffff640 1 -- 172.21.15.44:0/4012387808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0b5a40 msgr2=0x55b73f0c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:34.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3bffff640 1 --2- 172.21.15.44:0/4012387808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0b5a40 0x55b73f0c7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3bffff640 1 -- 172.21.15.44:0/4012387808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0bedb0 msgr2=0x55b73f0c3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:34.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3bffff640 1 --2- 172.21.15.44:0/4012387808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0bedb0 0x55b73f0c3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3bffff640 1 -- 172.21.15.44:0/4012387808 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b73f0c4160 con 0x55b73f0b6720 2024-04-21T23:18:34.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3c77fe640 1 --2- 172.21.15.44:0/4012387808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0bedb0 0x55b73f0c3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:34.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3c6ffd640 1 --2- 172.21.15.44:0/4012387808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0b5a40 0x55b73f0c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:34.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3bffff640 1 --2- 172.21.15.44:0/4012387808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0c35e0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0011fd0 tx=0x7fa3b0011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a8d7c8d73a3bc79 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:34.848 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3ae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:18:34.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3ae7fc640 1 -- 172.21.15.44:0/4012387808 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b002c070 con 0x55b73f0b6720 2024-04-21T23:18:34.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3ae7fc640 1 -- 172.21.15.44:0/4012387808 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7fa3b00198d0 con 0x55b73f0b6720 2024-04-21T23:18:34.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3d038c740 1 -- 172.21.15.44:0/4012387808 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3b0019e30 con 0x55b73f0b6720 2024-04-21T23:18:34.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3ae7fc640 1 -- 172.21.15.44:0/4012387808 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa3b0040050 con 0x55b73f0b6720 2024-04-21T23:18:34.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3d038c740 10 client.17742 fetch_fsmap learned FSMap version 31 2024-04-21T23:18:34.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.841+0000 7fa3d038c740 1 -- 172.21.15.44:0/4012387808 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b73f1363c0 con 0x55b73f0b6720 2024-04-21T23:18:34.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.842+0000 7fa3ae7fc640 1 -- 172.21.15.44:0/4012387808 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3b0019e30 con 0x55b73f0b6720 2024-04-21T23:18:34.849 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.842+0000 7fa3ae7fc640 1 -- 172.21.15.44:0/4012387808 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7fa3b0018070 con 0x55b73f0b6720 2024-04-21T23:18:34.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.842+0000 7fa3d038c740 10 client.17742 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:18:34.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.842+0000 7fa3d038c740 10 client.17742 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:34.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.842+0000 7fa3d038c740 20 client.17742 populate_metadata read hostname 'smithi044' 2024-04-21T23:18:34.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.842+0000 7fa3d038c740 1 --2- 172.21.15.44:0/4012387808 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b73f136a80 0x55b73f138ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.842+0000 7fa3d038c740 4 client.17742 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T23:18:34.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.842+0000 7fa3d038c740 1 -- 172.21.15.44:0/4012387808 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b73f0266b0 con 0x55b73f136a80 2024-04-21T23:18:34.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.842+0000 7fa3c6ffd640 1 --2- 172.21.15.44:0/4012387808 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b73f136a80 0x55b73f138ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.843+0000 7fa3ae7fc640 10 client.17742 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:34.850 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.843+0000 7fa3c6ffd640 1 --2- 172.21.15.44:0/4012387808 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b73f136a80 0x55b73f138ea0 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=149a99b21156da3d server_cookie=3fecfcad3fce9786 in_seq=0 out_seq=0 2024-04-21T23:18:34.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.844+0000 7fa3ae7fc640 1 -- 172.21.15.44:0/4012387808 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b73f0266b0 con 0x55b73f136a80 2024-04-21T23:18:34.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.844+0000 7fa3ae7fc640 10 client.17742 handle_command_reply: tid=0 2024-04-21T23:18:34.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.844+0000 7fa3d038c740 1 client.17742 shutdown 2024-04-21T23:18:34.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.844+0000 7fa3d038c740 1 -- 172.21.15.44:0/4012387808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 msgr2=0x55b73f0c35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:34.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.844+0000 7fa3d038c740 1 --2- 172.21.15.44:0/4012387808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0c35e0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0011fd0 tx=0x7fa3b0011b00 comp rx=0 tx=0).stop 2024-04-21T23:18:34.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.844+0000 7fa3d038c740 1 -- 172.21.15.44:0/4012387808 shutdown_connections 2024-04-21T23:18:34.852 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.844+0000 7fa3d038c740 1 --2- 172.21.15.44:0/4012387808 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55b73f136a80 0x55b73f138ea0 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.844+0000 7fa3d038c740 1 --2- 172.21.15.44:0/4012387808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55b73f0bedb0 0x55b73f0c3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.844+0000 7fa3d038c740 1 --2- 172.21.15.44:0/4012387808 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55b73f0b6720 0x55b73f0c35e0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.844+0000 7fa3ae7fc640 0 client.17742 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:34.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.844+0000 7fa3d038c740 1 --2- 172.21.15.44:0/4012387808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b73f0b5a40 0x55b73f0c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.844+0000 7fa3d038c740 1 -- 172.21.15.44:0/4012387808 >> 172.21.15.44:0/4012387808 conn(0x55b73f07c4f0 msgr2=0x55b73f0b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:34.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.844+0000 7fa3d038c740 1 -- 172.21.15.44:0/4012387808 shutdown_connections 2024-04-21T23:18:34.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.844+0000 7fa3d038c740 1 -- 172.21.15.44:0/4012387808 wait complete. 2024-04-21T23:18:34.853 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.844+0000 7fa3d038c740 20 client.17742 trim_cache size 0 max 16384 2024-04-21T23:18:34.854 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:18:34.854 INFO:teuthology.orchestra.run.smithi044.stdout: "return_code": 0, 2024-04-21T23:18:34.854 INFO:teuthology.orchestra.run.smithi044.stdout: "scrub_tag": "c8f75895-e9f0-4db8-a583-2b42ed79b48d", 2024-04-21T23:18:34.854 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": "asynchronous" 2024-04-21T23:18:34.854 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:18:34.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.845+0000 7fa3cd6c9640 1 -- 172.21.15.44:0/306673263 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fa3b40a2d00 msgr2=0x7fa3b40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:34.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.845+0000 7fa3cd6c9640 1 --2- 172.21.15.44:0/306673263 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fa3b40a2d00 0x7fa3b40a51c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0020000 tx=0x7fa3c001f000 comp rx=0 tx=0).stop 2024-04-21T23:18:34.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.845+0000 7fa3cd6c9640 1 -- 172.21.15.44:0/306673263 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3c81029b0 msgr2=0x7fa3c81981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:34.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.845+0000 7fa3cd6c9640 1 --2- 172.21.15.44:0/306673263 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3c81029b0 0x7fa3c81981f0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8006010 tx=0x7fa3b802cc70 comp rx=0 tx=0).stop 2024-04-21T23:18:34.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.845+0000 7fa3cd6c9640 1 -- 172.21.15.44:0/306673263 shutdown_connections 2024-04-21T23:18:34.856 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.845+0000 7fa3cd6c9640 1 --2- 172.21.15.44:0/306673263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3c810bec0 0x7fa3c8198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.845+0000 7fa3cd6c9640 1 --2- 172.21.15.44:0/306673263 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fa3b40a2d00 0x7fa3b40a51c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.845+0000 7fa3cd6c9640 1 --2- 172.21.15.44:0/306673263 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3c819ca60 0x7fa3c819cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.845+0000 7fa3cd6c9640 1 --2- 172.21.15.44:0/306673263 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fa3c81029b0 0x7fa3c81981f0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.845+0000 7fa3cd6c9640 1 -- 172.21.15.44:0/306673263 >> 172.21.15.44:0/306673263 conn(0x7fa3c8077760 msgr2=0x7fa3c80fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:34.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.845+0000 7fa3cd6c9640 1 -- 172.21.15.44:0/306673263 shutdown_connections 2024-04-21T23:18:34.857 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.845+0000 7fa3cd6c9640 1 -- 172.21.15.44:0/306673263 wait complete. 2024-04-21T23:18:34.858 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:18:34.858 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:18:34.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.962+0000 7f80e64c2640 1 -- 172.21.15.44:0/2167715961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80e0102ac0 msgr2=0x7f80e010d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:34.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.962+0000 7f80e64c2640 1 --2- 172.21.15.44:0/2167715961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80e0102ac0 0x7f80e010d5f0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f80d8009ca0 tx=0x7f80d802c1d0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.962+0000 7f80e64c2640 1 -- 172.21.15.44:0/2167715961 shutdown_connections 2024-04-21T23:18:34.962 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.962+0000 7f80e64c2640 1 --2- 172.21.15.44:0/2167715961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80e0102120 0x7f80e0102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.962+0000 7f80e64c2640 1 --2- 172.21.15.44:0/2167715961 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80e0102ac0 0x7f80e010d5f0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.962+0000 7f80e64c2640 1 --2- 172.21.15.44:0/2167715961 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80e01009e0 0x7f80e0100dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.962+0000 7f80e64c2640 1 -- 172.21.15.44:0/2167715961 >> 172.21.15.44:0/2167715961 conn(0x7f80e0077760 msgr2=0x7f80e0077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:34.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.962+0000 7f80e64c2640 1 -- 172.21.15.44:0/2167715961 shutdown_connections 2024-04-21T23:18:34.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.962+0000 7f80e64c2640 1 -- 172.21.15.44:0/2167715961 wait complete. 2024-04-21T23:18:34.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.963+0000 7f80e64c2640 1 Processor -- start 2024-04-21T23:18:34.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.963+0000 7f80e64c2640 1 -- start start 2024-04-21T23:18:34.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.963+0000 7f80e64c2640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80e01009e0 0x7f80e0196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.963+0000 7f80e64c2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80e0102120 0x7f80e0197490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.963+0000 7f80e64c2640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80e0102ac0 0x7f80e019b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.963+0000 7f80e64c2640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f80e0110b30 con 0x7f80e0102ac0 2024-04-21T23:18:34.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.963+0000 7f80e64c2640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f80e01109b0 con 0x7f80e01009e0 2024-04-21T23:18:34.968 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.963+0000 7f80e64c2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f80e0110cb0 con 0x7f80e0102120 2024-04-21T23:18:34.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.963+0000 7f80e4a5f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80e0102ac0 0x7f80e019b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.963+0000 7f80e4a5f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80e0102ac0 0x7f80e019b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49492/0 (socket says 172.21.15.44:49492) 2024-04-21T23:18:34.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.963+0000 7f80e4a5f640 1 -- 172.21.15.44:0/1803096281 learned_addr learned my addr 172.21.15.44:0/1803096281 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:34.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.963+0000 7f80dffff640 1 --2- 172.21.15.44:0/1803096281 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80e01009e0 0x7f80e0196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.963+0000 7f80df7fe640 1 --2- 172.21.15.44:0/1803096281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80e0102120 0x7f80e0197490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.963+0000 7f80e4a5f640 1 -- 172.21.15.44:0/1803096281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80e0102120 msgr2=0x7f80e0197490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:34.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.963+0000 7f80e4a5f640 1 --2- 172.21.15.44:0/1803096281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80e0102120 0x7f80e0197490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.969 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.963+0000 7f80e4a5f640 1 -- 172.21.15.44:0/1803096281 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80e01009e0 msgr2=0x7f80e0196f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:34.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.964+0000 7f80e4a5f640 1 --2- 172.21.15.44:0/1803096281 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80e01009e0 0x7f80e0196f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.964+0000 7f80e4a5f640 1 -- 172.21.15.44:0/1803096281 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e019bf40 con 0x7f80e0102ac0 2024-04-21T23:18:34.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.964+0000 7f80df7fe640 1 --2- 172.21.15.44:0/1803096281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80e0102120 0x7f80e0197490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:34.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.964+0000 7f80e4a5f640 1 --2- 172.21.15.44:0/1803096281 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80e0102ac0 0x7f80e019b810 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f80d802c7e0 tx=0x7f80d8003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:34.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.964+0000 7f80dd7fa640 1 -- 172.21.15.44:0/1803096281 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d8039050 con 0x7f80e0102ac0 2024-04-21T23:18:34.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.964+0000 7f80dd7fa640 1 -- 172.21.15.44:0/1803096281 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80d8045070 con 0x7f80e0102ac0 2024-04-21T23:18:34.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.964+0000 7f80e64c2640 1 -- 172.21.15.44:0/1803096281 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80e019c1d0 con 0x7f80e0102ac0 2024-04-21T23:18:34.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.964+0000 7f80dd7fa640 1 -- 172.21.15.44:0/1803096281 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d8032ce0 con 0x7f80e0102ac0 2024-04-21T23:18:34.970 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.964+0000 7f80e64c2640 1 -- 172.21.15.44:0/1803096281 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80e01a3ac0 con 0x7f80e0102ac0 2024-04-21T23:18:34.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.966+0000 7f80dd7fa640 1 -- 172.21.15.44:0/1803096281 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f80d8004450 con 0x7f80e0102ac0 2024-04-21T23:18:34.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.966+0000 7f80e9187740 1 Processor -- start 2024-04-21T23:18:34.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.966+0000 7f80e9187740 1 -- start start 2024-04-21T23:18:34.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80e9187740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55870b652c60 0x55870b653040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80e9187740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b653580 0x55870b65e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80e9187740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b65e640 0x55870b660a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80e9187740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55870b5b7560 con 0x55870b65e640 2024-04-21T23:18:34.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80e9187740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55870b55f730 con 0x55870b652c60 2024-04-21T23:18:34.971 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80e9187740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55870b5b3520 con 0x55870b653580 2024-04-21T23:18:34.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80df7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b65e640 0x55870b660a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80df7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b65e640 0x55870b660a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49504/0 (socket says 172.21.15.44:49504) 2024-04-21T23:18:34.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80df7fe640 1 -- 172.21.15.44:0/2329877857 learned_addr learned my addr 172.21.15.44:0/2329877857 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:34.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80dd7fa640 1 --2- 172.21.15.44:0/1803096281 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f80c80a2d00 0x7f80c80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80df7fe640 1 --2- 172.21.15.44:0/2329877857 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55870b652c60 0x55870b653040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80e4a5f640 1 --2- 172.21.15.44:0/1803096281 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f80c80a2d00 0x7f80c80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80dffff640 1 --2- 172.21.15.44:0/2329877857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b653580 0x55870b65e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.972 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80df7fe640 1 -- 172.21.15.44:0/2329877857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b653580 msgr2=0x55870b65e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:34.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80dd7fa640 1 -- 172.21.15.44:0/1803096281 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7f80d80f1ec0 con 0x7f80e0102ac0 2024-04-21T23:18:34.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80df7fe640 1 --2- 172.21.15.44:0/2329877857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b653580 0x55870b65e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80df7fe640 1 -- 172.21.15.44:0/2329877857 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55870b652c60 msgr2=0x55870b653040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:34.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80df7fe640 1 --2- 172.21.15.44:0/2329877857 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55870b652c60 0x55870b653040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80df7fe640 1 -- 172.21.15.44:0/2329877857 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55870b3957d0 con 0x55870b65e640 2024-04-21T23:18:34.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80dffff640 1 --2- 172.21.15.44:0/2329877857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b653580 0x55870b65e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:34.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80df7fe640 1 --2- 172.21.15.44:0/2329877857 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b65e640 0x55870b660a30 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f80d40124f0 tx=0x7f80d400b600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35172f3d35e0da09 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:34.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80e4a5f640 1 --2- 172.21.15.44:0/1803096281 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f80c80a2d00 0x7f80c80a51c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f80d80bde00 tx=0x7f80d8088c80 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:34.973 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.967+0000 7f80c67fc640 1 -- 172.21.15.44:0/2329877857 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d4018070 con 0x55870b65e640 2024-04-21T23:18:34.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.968+0000 7f80c67fc640 1 -- 172.21.15.44:0/2329877857 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80d401db40 con 0x55870b65e640 2024-04-21T23:18:34.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.968+0000 7f80c67fc640 1 -- 172.21.15.44:0/2329877857 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d4010b80 con 0x55870b65e640 2024-04-21T23:18:34.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.968+0000 7f80e9187740 1 -- 172.21.15.44:0/2329877857 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b65e640 msgr2=0x55870b660a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:34.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.968+0000 7f80e9187740 1 --2- 172.21.15.44:0/2329877857 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b65e640 0x55870b660a30 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f80d40124f0 tx=0x7f80d400b600 comp rx=0 tx=0).stop 2024-04-21T23:18:34.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.968+0000 7f80e9187740 1 -- 172.21.15.44:0/2329877857 shutdown_connections 2024-04-21T23:18:34.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.968+0000 7f80e9187740 1 --2- 172.21.15.44:0/2329877857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b653580 0x55870b65e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.968+0000 7f80e9187740 1 --2- 172.21.15.44:0/2329877857 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b65e640 0x55870b660a30 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.974 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.968+0000 7f80e9187740 1 --2- 172.21.15.44:0/2329877857 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55870b652c60 0x55870b653040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.968+0000 7f80e9187740 1 -- 172.21.15.44:0/2329877857 >> 172.21.15.44:0/2329877857 conn(0x55870b64d660 msgr2=0x55870b64da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:34.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.968+0000 7f80e9187740 1 -- 172.21.15.44:0/2329877857 shutdown_connections 2024-04-21T23:18:34.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.968+0000 7f80e9187740 1 -- 172.21.15.44:0/2329877857 wait complete. 2024-04-21T23:18:34.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.969+0000 7f80e9187740 1 -- start start 2024-04-21T23:18:34.975 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.969+0000 7f80e9187740 1 Processor -- start 2024-04-21T23:18:34.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.969+0000 7f80e9187740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55870b652c60 0x55870b659de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.969+0000 7f80e9187740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b653580 0x55870b65a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.969+0000 7f80e9187740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b65e640 0x55870b6621d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.969+0000 7f80e9187740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x55870b6540d0 con 0x55870b653580 2024-04-21T23:18:34.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.969+0000 7f80e9187740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x55870b572e00 con 0x55870b652c60 2024-04-21T23:18:34.979 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.969+0000 7f80e9187740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55870b654250 con 0x55870b65e640 2024-04-21T23:18:34.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.969+0000 7f80e9187740 10 client.0 _ll_register_callbacks cb 0x55870b5adeb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:34.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.969+0000 7f80dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b653580 0x55870b65a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.969+0000 7f80dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b653580 0x55870b65a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49520/0 (socket says 172.21.15.44:49520) 2024-04-21T23:18:34.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.969+0000 7f80dffff640 1 -- 172.21.15.44:0/3273455066 learned_addr learned my addr 172.21.15.44:0/3273455066 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:34.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.969+0000 7f80df7fe640 1 --2- 172.21.15.44:0/3273455066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b65e640 0x55870b6621d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.969+0000 7f80dffff640 1 -- 172.21.15.44:0/3273455066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b65e640 msgr2=0x55870b6621d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:34.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.969+0000 7f80dffff640 1 --2- 172.21.15.44:0/3273455066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b65e640 0x55870b6621d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.969+0000 7f80dffff640 1 -- 172.21.15.44:0/3273455066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55870b652c60 msgr2=0x55870b659de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:34.980 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.969+0000 7f80dffff640 1 --2- 172.21.15.44:0/3273455066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55870b652c60 0x55870b659de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.969+0000 7f80dffff640 1 -- 172.21.15.44:0/3273455066 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55870b2cb3f0 con 0x55870b653580 2024-04-21T23:18:34.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.969+0000 7f80df7fe640 1 --2- 172.21.15.44:0/3273455066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b65e640 0x55870b6621d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:34.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.970+0000 7f80dffff640 1 --2- 172.21.15.44:0/3273455066 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b653580 0x55870b65a320 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f80d000b8d0 tx=0x7f80d000bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee61af36e502f4c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:34.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.970+0000 7f80bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:18:34.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.970+0000 7f80bf7fe640 1 -- 172.21.15.44:0/3273455066 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d0020060 con 0x55870b653580 2024-04-21T23:18:34.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.970+0000 7f80bf7fe640 1 -- 172.21.15.44:0/3273455066 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7f80d000ac40 con 0x55870b653580 2024-04-21T23:18:34.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.970+0000 7f80e9187740 1 -- 172.21.15.44:0/3273455066 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80d401db40 con 0x55870b653580 2024-04-21T23:18:34.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.970+0000 7f80bf7fe640 1 -- 172.21.15.44:0/3273455066 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f80d000ae80 con 0x55870b653580 2024-04-21T23:18:34.981 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.970+0000 7f80e9187740 10 client.17754 fetch_fsmap learned FSMap version 31 2024-04-21T23:18:34.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.970+0000 7f80e9187740 1 -- 172.21.15.44:0/3273455066 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55870b28b0f0 con 0x55870b653580 2024-04-21T23:18:34.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.970+0000 7f80bf7fe640 1 -- 172.21.15.44:0/3273455066 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80d401db40 con 0x55870b653580 2024-04-21T23:18:34.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.970+0000 7f80bf7fe640 1 -- 172.21.15.44:0/3273455066 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f80d001a070 con 0x55870b653580 2024-04-21T23:18:34.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.970+0000 7f80e9187740 10 client.17754 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:18:34.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.970+0000 7f80e9187740 10 client.17754 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:34.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.970+0000 7f80e9187740 20 client.17754 populate_metadata read hostname 'smithi044' 2024-04-21T23:18:34.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.970+0000 7f80e9187740 1 --2- 172.21.15.44:0/3273455066 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55870b66c910 0x55870b66ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.982 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.970+0000 7f80e9187740 4 client.17754 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:18:34.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.970+0000 7f80e9187740 1 -- 172.21.15.44:0/3273455066 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55870b5c4aa0 con 0x55870b66c910 2024-04-21T23:18:34.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.971+0000 7f80df7fe640 1 --2- 172.21.15.44:0/3273455066 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55870b66c910 0x55870b66ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.971+0000 7f80df7fe640 1 --2- 172.21.15.44:0/3273455066 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55870b66c910 0x55870b66ed00 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2301c39205d1e09 server_cookie=2ab26690a20e465d in_seq=0 out_seq=0 2024-04-21T23:18:34.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.971+0000 7f80bf7fe640 10 client.17754 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:34.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.972+0000 7f80bf7fe640 1 -- 172.21.15.44:0/3273455066 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55870b5c4aa0 con 0x55870b66c910 2024-04-21T23:18:34.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.972+0000 7f80bf7fe640 10 client.17754 handle_command_reply: tid=0 2024-04-21T23:18:34.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.972+0000 7f80e9187740 1 client.17754 shutdown 2024-04-21T23:18:34.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.972+0000 7f80e9187740 1 -- 172.21.15.44:0/3273455066 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b653580 msgr2=0x55870b65a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:34.983 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.972+0000 7f80e9187740 1 --2- 172.21.15.44:0/3273455066 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b653580 0x55870b65a320 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f80d000b8d0 tx=0x7f80d000bd90 comp rx=0 tx=0).stop 2024-04-21T23:18:34.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.972+0000 7f80e9187740 1 -- 172.21.15.44:0/3273455066 shutdown_connections 2024-04-21T23:18:34.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.972+0000 7f80e9187740 1 --2- 172.21.15.44:0/3273455066 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55870b66c910 0x55870b66ed00 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.972+0000 7f80e9187740 1 --2- 172.21.15.44:0/3273455066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b65e640 0x55870b6621d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.972+0000 7f80e9187740 1 --2- 172.21.15.44:0/3273455066 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b653580 0x55870b65a320 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.972+0000 7f80bf7fe640 0 client.17754 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:34.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.972+0000 7f80e9187740 1 --2- 172.21.15.44:0/3273455066 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55870b652c60 0x55870b659de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.972+0000 7f80e9187740 1 -- 172.21.15.44:0/3273455066 >> 172.21.15.44:0/3273455066 conn(0x55870b64d660 msgr2=0x55870b64f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:34.984 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.972+0000 7f80e9187740 1 -- 172.21.15.44:0/3273455066 shutdown_connections 2024-04-21T23:18:34.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.972+0000 7f80e9187740 1 -- 172.21.15.44:0/3273455066 wait complete. 2024-04-21T23:18:34.985 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.972+0000 7f80e9187740 20 client.17754 trim_cache size 0 max 16384 2024-04-21T23:18:34.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.989+0000 7f80e9187740 1 Processor -- start 2024-04-21T23:18:34.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.990+0000 7f80e9187740 1 -- start start 2024-04-21T23:18:34.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.990+0000 7f80e9187740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b652c60 0x55870b658b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.990+0000 7f80e9187740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b653580 0x55870b6621d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.990+0000 7f80e9187740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55870b65e640 0x55870b6550a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.990+0000 7f80e9187740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f80e01aaa50 con 0x55870b653580 2024-04-21T23:18:34.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.990+0000 7f80e9187740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f80e01ad2b0 con 0x55870b65e640 2024-04-21T23:18:34.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.990+0000 7f80e9187740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f80e01ab0a0 con 0x55870b652c60 2024-04-21T23:18:34.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.990+0000 7f80dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b653580 0x55870b6621d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.990+0000 7f80dffff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b653580 0x55870b6621d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49524/0 (socket says 172.21.15.44:49524) 2024-04-21T23:18:34.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.990+0000 7f80dffff640 1 -- 172.21.15.44:0/4247760648 learned_addr learned my addr 172.21.15.44:0/4247760648 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:34.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.990+0000 7f80df7fe640 1 --2- 172.21.15.44:0/4247760648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b652c60 0x55870b658b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:34.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.990+0000 7f80dffff640 1 -- 172.21.15.44:0/4247760648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b652c60 msgr2=0x55870b658b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:34.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.990+0000 7f80dffff640 1 --2- 172.21.15.44:0/4247760648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b652c60 0x55870b658b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.990+0000 7f80dffff640 1 -- 172.21.15.44:0/4247760648 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55870b65e640 msgr2=0x55870b6550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:34.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.990+0000 7f80dffff640 1 --2- 172.21.15.44:0/4247760648 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55870b65e640 0x55870b6550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.990+0000 7f80dffff640 1 -- 172.21.15.44:0/4247760648 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55870b64d190 con 0x55870b653580 2024-04-21T23:18:34.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.990+0000 7f80df7fe640 1 --2- 172.21.15.44:0/4247760648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b652c60 0x55870b658b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:18:34.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.990+0000 7f80dffff640 1 --2- 172.21.15.44:0/4247760648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b653580 0x55870b6621d0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f80d001a870 tx=0x7f80d0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc47adbd9aa447d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:34.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.990+0000 7f80c5ffb640 1 -- 172.21.15.44:0/4247760648 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d0030070 con 0x55870b653580 2024-04-21T23:18:34.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.991+0000 7f80c5ffb640 1 -- 172.21.15.44:0/4247760648 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80d000aa40 con 0x55870b653580 2024-04-21T23:18:34.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.991+0000 7f80c5ffb640 1 -- 172.21.15.44:0/4247760648 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d0020070 con 0x55870b653580 2024-04-21T23:18:34.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.991+0000 7f80e9187740 1 -- 172.21.15.44:0/4247760648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b653580 msgr2=0x55870b6621d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:34.997 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.991+0000 7f80e9187740 1 --2- 172.21.15.44:0/4247760648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b653580 0x55870b6621d0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f80d001a870 tx=0x7f80d0000bc0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.991+0000 7f80e9187740 1 -- 172.21.15.44:0/4247760648 shutdown_connections 2024-04-21T23:18:34.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.991+0000 7f80e9187740 1 --2- 172.21.15.44:0/4247760648 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55870b65e640 0x55870b6550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.991+0000 7f80e9187740 1 --2- 172.21.15.44:0/4247760648 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b653580 0x55870b6621d0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.991+0000 7f80e9187740 1 --2- 172.21.15.44:0/4247760648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b652c60 0x55870b658b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:34.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.991+0000 7f80e9187740 1 -- 172.21.15.44:0/4247760648 >> 172.21.15.44:0/4247760648 conn(0x55870b64d660 msgr2=0x55870b64ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:34.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.991+0000 7f80e9187740 1 -- 172.21.15.44:0/4247760648 shutdown_connections 2024-04-21T23:18:34.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.991+0000 7f80e9187740 1 -- 172.21.15.44:0/4247760648 wait complete. 2024-04-21T23:18:34.998 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.991+0000 7f80e9187740 1 -- start start 2024-04-21T23:18:34.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.992+0000 7f80e9187740 1 Processor -- start 2024-04-21T23:18:34.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.992+0000 7f80e9187740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55870b652c60 0x55870b668bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.992+0000 7f80e9187740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b653580 0x55870b663fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.992+0000 7f80e9187740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b65e640 0x55870b6644f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:34.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.992+0000 7f80e9187740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f80e01ab770 con 0x55870b65e640 2024-04-21T23:18:34.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.992+0000 7f80e9187740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f80e01acbe0 con 0x55870b652c60 2024-04-21T23:18:34.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.992+0000 7f80e9187740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f80e01abe40 con 0x55870b653580 2024-04-21T23:18:34.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.992+0000 7f80e9187740 10 client.0 _ll_register_callbacks cb 0x7f80d00107c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:18:34.999 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.992+0000 7f80df7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b65e640 0x55870b6644f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:35.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.992+0000 7f80df7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b65e640 0x55870b6644f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:49532/0 (socket says 172.21.15.44:49532) 2024-04-21T23:18:35.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.992+0000 7f80df7fe640 1 -- 172.21.15.44:0/1659161236 learned_addr learned my addr 172.21.15.44:0/1659161236 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:35.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.992+0000 7f80dffff640 1 --2- 172.21.15.44:0/1659161236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b653580 0x55870b663fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:35.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.992+0000 7f80df7fe640 1 -- 172.21.15.44:0/1659161236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b653580 msgr2=0x55870b663fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:35.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.992+0000 7f80df7fe640 1 --2- 172.21.15.44:0/1659161236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b653580 0x55870b663fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:35.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.992+0000 7f80df7fe640 1 -- 172.21.15.44:0/1659161236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55870b652c60 msgr2=0x55870b668bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:18:35.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.992+0000 7f80df7fe640 1 --2- 172.21.15.44:0/1659161236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55870b652c60 0x55870b668bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:35.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.992+0000 7f80df7fe640 1 -- 172.21.15.44:0/1659161236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55870b664cf0 con 0x55870b65e640 2024-04-21T23:18:35.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.992+0000 7f80df7fe640 1 --2- 172.21.15.44:0/1659161236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b65e640 0x55870b6644f0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f80d401f010 tx=0x7f80d40058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a221c832a7386998 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:35.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.992+0000 7f80bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:18:35.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.993+0000 7f80bf7fe640 1 -- 172.21.15.44:0/1659161236 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d402e070 con 0x55870b65e640 2024-04-21T23:18:35.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.993+0000 7f80bf7fe640 1 -- 172.21.15.44:0/1659161236 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7f80d40104d0 con 0x55870b65e640 2024-04-21T23:18:35.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.993+0000 7f80e9187740 1 -- 172.21.15.44:0/1659161236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80d000aa40 con 0x55870b65e640 2024-04-21T23:18:35.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.993+0000 7f80bf7fe640 1 -- 172.21.15.44:0/1659161236 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f80d403a050 con 0x55870b65e640 2024-04-21T23:18:35.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.993+0000 7f80e9187740 10 client.17760 fetch_fsmap learned FSMap version 31 2024-04-21T23:18:35.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.993+0000 7f80e9187740 1 -- 172.21.15.44:0/1659161236 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55870b6d2dd0 con 0x55870b65e640 2024-04-21T23:18:35.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.993+0000 7f80bf7fe640 1 -- 172.21.15.44:0/1659161236 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80d000aa40 con 0x55870b65e640 2024-04-21T23:18:35.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.993+0000 7f80bf7fe640 1 -- 172.21.15.44:0/1659161236 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f80d400f040 con 0x55870b65e640 2024-04-21T23:18:35.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.993+0000 7f80e9187740 10 client.17760 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:18:35.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.993+0000 7f80e9187740 10 client.17760 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:18:35.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.993+0000 7f80e9187740 20 client.17760 populate_metadata read hostname 'smithi044' 2024-04-21T23:18:35.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.993+0000 7f80e9187740 1 --2- 172.21.15.44:0/1659161236 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55870b6d31d0 0x55870b6d55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:35.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.993+0000 7f80e9187740 4 client.17760 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:18:35.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.993+0000 7f80e9187740 1 -- 172.21.15.44:0/1659161236 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55870b5bedf0 con 0x55870b6d31d0 2024-04-21T23:18:35.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.993+0000 7f80dffff640 1 --2- 172.21.15.44:0/1659161236 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55870b6d31d0 0x55870b6d55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:35.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.994+0000 7f80dffff640 1 --2- 172.21.15.44:0/1659161236 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55870b6d31d0 0x55870b6d55f0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46e1ea2f98254f89 server_cookie=d482f45b70e8df6c in_seq=0 out_seq=0 2024-04-21T23:18:35.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.994+0000 7f80bf7fe640 10 client.17760 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:35.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.994+0000 7f80bf7fe640 1 -- 172.21.15.44:0/1659161236 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55870b5bedf0 con 0x55870b6d31d0 2024-04-21T23:18:35.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.994+0000 7f80bf7fe640 10 client.17760 handle_command_reply: tid=0 2024-04-21T23:18:35.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.994+0000 7f80e9187740 1 client.17760 shutdown 2024-04-21T23:18:35.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.994+0000 7f80e9187740 1 -- 172.21.15.44:0/1659161236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b65e640 msgr2=0x55870b6644f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:35.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.994+0000 7f80e9187740 1 --2- 172.21.15.44:0/1659161236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b65e640 0x55870b6644f0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f80d401f010 tx=0x7f80d40058e0 comp rx=0 tx=0).stop 2024-04-21T23:18:35.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.994+0000 7f80e9187740 1 -- 172.21.15.44:0/1659161236 shutdown_connections 2024-04-21T23:18:35.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.994+0000 7f80e9187740 1 --2- 172.21.15.44:0/1659161236 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x55870b6d31d0 0x55870b6d55f0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:35.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.994+0000 7f80e9187740 1 --2- 172.21.15.44:0/1659161236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55870b653580 0x55870b663fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:35.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.994+0000 7f80bf7fe640 0 client.17760 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:18:35.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.994+0000 7f80e9187740 1 --2- 172.21.15.44:0/1659161236 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x55870b65e640 0x55870b6644f0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:35.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.994+0000 7f80e9187740 1 --2- 172.21.15.44:0/1659161236 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x55870b652c60 0x55870b668bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:35.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.994+0000 7f80e9187740 1 -- 172.21.15.44:0/1659161236 >> 172.21.15.44:0/1659161236 conn(0x55870b64d660 msgr2=0x55870b65a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:35.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.994+0000 7f80e9187740 1 -- 172.21.15.44:0/1659161236 shutdown_connections 2024-04-21T23:18:35.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.994+0000 7f80e9187740 1 -- 172.21.15.44:0/1659161236 wait complete. 2024-04-21T23:18:35.004 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.994+0000 7f80e9187740 20 client.17760 trim_cache size 0 max 16384 2024-04-21T23:18:35.005 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:18:35.005 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T23:18:35.005 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": { 2024-04-21T23:18:35.005 INFO:teuthology.orchestra.run.smithi044.stdout: "c8f75895-e9f0-4db8-a583-2b42ed79b48d": { 2024-04-21T23:18:35.005 INFO:teuthology.orchestra.run.smithi044.stdout: "path": "/", 2024-04-21T23:18:35.005 INFO:teuthology.orchestra.run.smithi044.stdout: "tag": "c8f75895-e9f0-4db8-a583-2b42ed79b48d", 2024-04-21T23:18:35.005 INFO:teuthology.orchestra.run.smithi044.stdout: "options": "recursive,force" 2024-04-21T23:18:35.006 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:18:35.006 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:18:35.006 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:18:35.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.995+0000 7f80e64c2640 1 -- 172.21.15.44:0/1803096281 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f80c80a2d00 msgr2=0x7f80c80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:35.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.995+0000 7f80e64c2640 1 --2- 172.21.15.44:0/1803096281 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f80c80a2d00 0x7f80c80a51c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f80d80bde00 tx=0x7f80d8088c80 comp rx=0 tx=0).stop 2024-04-21T23:18:35.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.995+0000 7f80e64c2640 1 -- 172.21.15.44:0/1803096281 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80e0102ac0 msgr2=0x7f80e019b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:35.007 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.995+0000 7f80e64c2640 1 --2- 172.21.15.44:0/1803096281 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80e0102ac0 0x7f80e019b810 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f80d802c7e0 tx=0x7f80d8003c20 comp rx=0 tx=0).stop 2024-04-21T23:18:35.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.995+0000 7f80e64c2640 1 -- 172.21.15.44:0/1803096281 shutdown_connections 2024-04-21T23:18:35.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.995+0000 7f80e64c2640 1 --2- 172.21.15.44:0/1803096281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80e0102120 0x7f80e0197490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:35.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.995+0000 7f80e64c2640 1 --2- 172.21.15.44:0/1803096281 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f80c80a2d00 0x7f80c80a51c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:35.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.995+0000 7f80e64c2640 1 --2- 172.21.15.44:0/1803096281 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f80e0102ac0 0x7f80e019b810 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:35.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.995+0000 7f80e64c2640 1 --2- 172.21.15.44:0/1803096281 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f80e01009e0 0x7f80e0196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:35.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.995+0000 7f80e64c2640 1 -- 172.21.15.44:0/1803096281 >> 172.21.15.44:0/1803096281 conn(0x7f80e0077760 msgr2=0x7f80e01053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:35.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.995+0000 7f80e64c2640 1 -- 172.21.15.44:0/1803096281 shutdown_connections 2024-04-21T23:18:35.008 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:34.995+0000 7f80e64c2640 1 -- 172.21.15.44:0/1803096281 wait complete. 2024-04-21T23:18:35.009 INFO:tasks.cephfs.filesystem:scrub status for tag:c8f75895-e9f0-4db8-a583-2b42ed79b48d - {'path': '/', 'tag': 'c8f75895-e9f0-4db8-a583-2b42ed79b48d', 'options': 'recursive,force'} 2024-04-21T23:18:35.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:35 smithi044 ceph-mon[27934]: scrub queued for path: / 2024-04-21T23:18:35.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:35 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:18:35.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:35 smithi044 ceph-mon[27934]: scrub summary: active paths [/] 2024-04-21T23:18:35.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:35 smithi044 ceph-mon[27934]: scrub summary: idle+waiting paths [/] 2024-04-21T23:18:35.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:35 smithi069 ceph-mon[31816]: scrub queued for path: / 2024-04-21T23:18:35.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:35 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:18:35.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:35 smithi069 ceph-mon[31816]: scrub summary: active paths [/] 2024-04-21T23:18:35.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:35 smithi069 ceph-mon[31816]: scrub summary: idle+waiting paths [/] 2024-04-21T23:18:35.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:35 smithi055 ceph-mon[31117]: scrub queued for path: / 2024-04-21T23:18:35.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:35 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:18:35.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:35 smithi055 ceph-mon[31117]: scrub summary: active paths [/] 2024-04-21T23:18:35.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:35 smithi055 ceph-mon[31117]: scrub summary: idle+waiting paths [/] 2024-04-21T23:18:36.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:36 smithi044 ceph-mon[27934]: pgmap v308: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 886 B/s rd, 1 op/s 2024-04-21T23:18:36.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:36 smithi055 ceph-mon[31117]: pgmap v308: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 886 B/s rd, 1 op/s 2024-04-21T23:18:36.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:36 smithi069 ceph-mon[31816]: pgmap v308: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 886 B/s rd, 1 op/s 2024-04-21T23:18:37.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:37 smithi044 ceph-mon[27934]: scrub summary: idle 2024-04-21T23:18:37.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:37 smithi069 ceph-mon[31816]: scrub summary: idle 2024-04-21T23:18:37.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:37 smithi055 ceph-mon[31117]: scrub summary: idle 2024-04-21T23:18:38.615 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:38 smithi069 ceph-mon[31816]: pgmap v309: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 5 op/s 2024-04-21T23:18:38.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:38 smithi044 ceph-mon[27934]: pgmap v309: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 5 op/s 2024-04-21T23:18:38.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:38 smithi055 ceph-mon[31117]: pgmap v309: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 5 op/s 2024-04-21T23:18:40.516 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:40 smithi055 ceph-mon[31117]: pgmap v310: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 8 op/s 2024-04-21T23:18:40.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:40 smithi044 ceph-mon[27934]: pgmap v310: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 8 op/s 2024-04-21T23:18:40.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:40 smithi069 ceph-mon[31816]: pgmap v310: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 8 op/s 2024-04-21T23:18:42.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:42 smithi044 ceph-mon[27934]: pgmap v311: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 8 op/s 2024-04-21T23:18:42.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:42 smithi069 ceph-mon[31816]: pgmap v311: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 8 op/s 2024-04-21T23:18:42.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:42 smithi055 ceph-mon[31117]: pgmap v311: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 8 op/s 2024-04-21T23:18:44.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:44 smithi044 ceph-mon[27934]: pgmap v312: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 9 op/s 2024-04-21T23:18:44.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:44 smithi055 ceph-mon[31117]: pgmap v312: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 9 op/s 2024-04-21T23:18:44.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:44 smithi069 ceph-mon[31816]: pgmap v312: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 9 op/s 2024-04-21T23:18:46.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:46 smithi044 ceph-mon[27934]: pgmap v313: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 8 op/s 2024-04-21T23:18:46.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:46 smithi055 ceph-mon[31117]: pgmap v313: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 8 op/s 2024-04-21T23:18:46.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:46 smithi069 ceph-mon[31816]: pgmap v313: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 8 op/s 2024-04-21T23:18:48.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:48 smithi044 ceph-mon[27934]: pgmap v314: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 8 op/s 2024-04-21T23:18:48.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:48 smithi055 ceph-mon[31117]: pgmap v314: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 8 op/s 2024-04-21T23:18:48.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:48 smithi069 ceph-mon[31816]: pgmap v314: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 8 op/s 2024-04-21T23:18:50.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:50 smithi044 ceph-mon[27934]: pgmap v315: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 5 op/s 2024-04-21T23:18:50.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:50 smithi055 ceph-mon[31117]: pgmap v315: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 5 op/s 2024-04-21T23:18:50.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:50 smithi069 ceph-mon[31816]: pgmap v315: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 5 op/s 2024-04-21T23:18:52.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:52 smithi044 ceph-mon[27934]: pgmap v316: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:18:52.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:52 smithi055 ceph-mon[31117]: pgmap v316: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:18:52.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:52 smithi069 ceph-mon[31816]: pgmap v316: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:18:54.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:54 smithi044 ceph-mon[27934]: pgmap v317: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:18:54.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:54 smithi069 ceph-mon[31816]: pgmap v317: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:18:54.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:54 smithi055 ceph-mon[31117]: pgmap v317: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:18:56.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:56 smithi044 ceph-mon[27934]: pgmap v318: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:18:56.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:56 smithi069 ceph-mon[31816]: pgmap v318: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:18:56.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:56 smithi055 ceph-mon[31117]: pgmap v318: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:18:57.334 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-21T23:18:57.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.450+0000 7f1de487d640 1 -- 172.21.15.44:0/2283889868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1de010bd30 msgr2=0x7f1de010e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:57.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.450+0000 7f1de487d640 1 --2- 172.21.15.44:0/2283889868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1de010bd30 0x7f1de010e1d0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8009ca0 tx=0x7f1dd802c180 comp rx=0 tx=0).stop 2024-04-21T23:18:57.450 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.450+0000 7f1de487d640 1 -- 172.21.15.44:0/2283889868 shutdown_connections 2024-04-21T23:18:57.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.450+0000 7f1de487d640 1 --2- 172.21.15.44:0/2283889868 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1de010bd30 0x7f1de010e1d0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:57.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.450+0000 7f1de487d640 1 --2- 172.21.15.44:0/2283889868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1de0103310 0x7f1de010b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:57.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.450+0000 7f1de487d640 1 --2- 172.21.15.44:0/2283889868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1de01029f0 0x7f1de0102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:57.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.450+0000 7f1de487d640 1 -- 172.21.15.44:0/2283889868 >> 172.21.15.44:0/2283889868 conn(0x7f1de0077790 msgr2=0x7f1de0077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:57.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.450+0000 7f1de487d640 1 -- 172.21.15.44:0/2283889868 shutdown_connections 2024-04-21T23:18:57.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.450+0000 7f1de487d640 1 -- 172.21.15.44:0/2283889868 wait complete. 2024-04-21T23:18:57.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.451+0000 7f1de487d640 1 Processor -- start 2024-04-21T23:18:57.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.451+0000 7f1de487d640 1 -- start start 2024-04-21T23:18:57.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.451+0000 7f1de487d640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1de01029f0 0x7f1de0100640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:57.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.451+0000 7f1de487d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1de0103310 0x7f1de0100b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:57.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.451+0000 7f1de487d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1de010bd30 0x7f1de01010c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:57.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.451+0000 7f1de487d640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1de0111b30 con 0x7f1de01029f0 2024-04-21T23:18:57.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.451+0000 7f1de487d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1de01119b0 con 0x7f1de0103310 2024-04-21T23:18:57.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.451+0000 7f1de487d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1de0111cb0 con 0x7f1de010bd30 2024-04-21T23:18:57.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.452+0000 7f1dde59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1de01029f0 0x7f1de0100640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:57.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.452+0000 7f1dde59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1de01029f0 0x7f1de0100640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38732/0 (socket says 172.21.15.44:38732) 2024-04-21T23:18:57.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.452+0000 7f1dde59c640 1 -- 172.21.15.44:0/3140629024 learned_addr learned my addr 172.21.15.44:0/3140629024 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:57.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.452+0000 7f1dddd9b640 1 --2- 172.21.15.44:0/3140629024 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1de0103310 0x7f1de0100b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:57.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.452+0000 7f1dded9d640 1 --2- 172.21.15.44:0/3140629024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1de010bd30 0x7f1de01010c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:57.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.452+0000 7f1dde59c640 1 -- 172.21.15.44:0/3140629024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1de010bd30 msgr2=0x7f1de01010c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:57.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.452+0000 7f1dde59c640 1 --2- 172.21.15.44:0/3140629024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1de010bd30 0x7f1de01010c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:57.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.452+0000 7f1dde59c640 1 -- 172.21.15.44:0/3140629024 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1de0103310 msgr2=0x7f1de0100b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:57.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.452+0000 7f1dde59c640 1 --2- 172.21.15.44:0/3140629024 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1de0103310 0x7f1de0100b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:57.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.452+0000 7f1dde59c640 1 -- 172.21.15.44:0/3140629024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1de01a6470 con 0x7f1de01029f0 2024-04-21T23:18:57.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.452+0000 7f1dded9d640 1 --2- 172.21.15.44:0/3140629024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1de010bd30 0x7f1de01010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:18:57.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.452+0000 7f1dde59c640 1 --2- 172.21.15.44:0/3140629024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1de01029f0 0x7f1de0100640 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f1dd000d770 tx=0x7f1dd000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:57.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.452+0000 7f1dcf7fe640 1 -- 172.21.15.44:0/3140629024 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0017070 con 0x7f1de01029f0 2024-04-21T23:18:57.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.452+0000 7f1dcf7fe640 1 -- 172.21.15.44:0/3140629024 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dd0005500 con 0x7f1de01029f0 2024-04-21T23:18:57.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.452+0000 7f1de487d640 1 -- 172.21.15.44:0/3140629024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1de01a6760 con 0x7f1de01029f0 2024-04-21T23:18:57.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.452+0000 7f1dcf7fe640 1 -- 172.21.15.44:0/3140629024 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0004e00 con 0x7f1de01029f0 2024-04-21T23:18:57.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.452+0000 7f1de487d640 1 -- 172.21.15.44:0/3140629024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1de01a6cb0 con 0x7f1de01029f0 2024-04-21T23:18:57.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.454+0000 7f1de487d640 1 -- 172.21.15.44:0/3140629024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1de006a590 con 0x7f1de01029f0 2024-04-21T23:18:57.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.455+0000 7f1dcf7fe640 1 -- 172.21.15.44:0/3140629024 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1dd0004fa0 con 0x7f1de01029f0 2024-04-21T23:18:57.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.455+0000 7f1dcf7fe640 1 --2- 172.21.15.44:0/3140629024 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1dc80a2da0 0x7f1dc80a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:57.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.455+0000 7f1dddd9b640 1 --2- 172.21.15.44:0/3140629024 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1dc80a2da0 0x7f1dc80a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:57.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.455+0000 7f1dcf7fe640 1 -- 172.21.15.44:0/3140629024 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7f1dd00cf1f0 con 0x7f1de01029f0 2024-04-21T23:18:57.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.456+0000 7f1dddd9b640 1 --2- 172.21.15.44:0/3140629024 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1dc80a2da0 0x7f1dc80a5260 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1de0102080 tx=0x7f1dd400b390 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:57.459 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.460+0000 7f1dcf7fe640 1 -- 172.21.15.44:0/3140629024 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1dd009ba20 con 0x7f1de01029f0 2024-04-21T23:18:57.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.669+0000 7f1de487d640 1 -- 172.21.15.44:0/3140629024 --> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f1de0061b80 con 0x7f1dc80a2da0 2024-04-21T23:18:57.677 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.678+0000 7f1dcf7fe640 1 -- 172.21.15.44:0/3140629024 <== mgr.15396 v2:172.21.15.44:6800/3789150048 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f1de0061b80 con 0x7f1dc80a2da0 2024-04-21T23:18:57.678 INFO:teuthology.orchestra.run.smithi044.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-21T00:00:00", "created": "2024-04-21T23:08:56", "first": "2024-04-21T23:09:00", "last": "2024-04-21T23:18:00", "last_pruned": "2024-04-21T23:18:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-04-21T23:18:57.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.680+0000 7f1de487d640 1 -- 172.21.15.44:0/3140629024 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1dc80a2da0 msgr2=0x7f1dc80a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:57.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.680+0000 7f1de487d640 1 --2- 172.21.15.44:0/3140629024 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1dc80a2da0 0x7f1dc80a5260 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1de0102080 tx=0x7f1dd400b390 comp rx=0 tx=0).stop 2024-04-21T23:18:57.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.680+0000 7f1de487d640 1 -- 172.21.15.44:0/3140629024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1de01029f0 msgr2=0x7f1de0100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:57.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.680+0000 7f1de487d640 1 --2- 172.21.15.44:0/3140629024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1de01029f0 0x7f1de0100640 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f1dd000d770 tx=0x7f1dd000dc30 comp rx=0 tx=0).stop 2024-04-21T23:18:57.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.680+0000 7f1de487d640 1 -- 172.21.15.44:0/3140629024 shutdown_connections 2024-04-21T23:18:57.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.680+0000 7f1de487d640 1 --2- 172.21.15.44:0/3140629024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1de010bd30 0x7f1de01010c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:57.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.680+0000 7f1de487d640 1 --2- 172.21.15.44:0/3140629024 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1dc80a2da0 0x7f1dc80a5260 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:57.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.680+0000 7f1de487d640 1 --2- 172.21.15.44:0/3140629024 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1de0103310 0x7f1de0100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:57.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.680+0000 7f1de487d640 1 --2- 172.21.15.44:0/3140629024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1de01029f0 0x7f1de0100640 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:57.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.680+0000 7f1de487d640 1 -- 172.21.15.44:0/3140629024 >> 172.21.15.44:0/3140629024 conn(0x7f1de0077790 msgr2=0x7f1de00fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:57.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.680+0000 7f1de487d640 1 -- 172.21.15.44:0/3140629024 shutdown_connections 2024-04-21T23:18:57.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.680+0000 7f1de487d640 1 -- 172.21.15.44:0/3140629024 wait complete. 2024-04-21T23:18:57.690 DEBUG:teuthology.orchestra.run.smithi044:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-21T23:18:57.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.794+0000 7f4ebec96640 1 -- 172.21.15.44:0/2748047413 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4eb8100530 msgr2=0x7f4eb8100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:57.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.794+0000 7f4ebec96640 1 --2- 172.21.15.44:0/2748047413 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4eb8100530 0x7f4eb8100910 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8009900 tx=0x7f4ea802bc30 comp rx=0 tx=0).stop 2024-04-21T23:18:57.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.795+0000 7f4ebec96640 1 -- 172.21.15.44:0/2748047413 shutdown_connections 2024-04-21T23:18:57.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.795+0000 7f4ebec96640 1 --2- 172.21.15.44:0/2748047413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4eb810e0e0 0x7f4eb81104d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:57.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.795+0000 7f4ebec96640 1 --2- 172.21.15.44:0/2748047413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4eb8100e50 0x7f4eb810dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:57.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.795+0000 7f4ebec96640 1 --2- 172.21.15.44:0/2748047413 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4eb8100530 0x7f4eb8100910 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:57.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.795+0000 7f4ebec96640 1 -- 172.21.15.44:0/2748047413 >> 172.21.15.44:0/2748047413 conn(0x7f4eb8077760 msgr2=0x7f4eb8077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:57.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.795+0000 7f4ebec96640 1 -- 172.21.15.44:0/2748047413 shutdown_connections 2024-04-21T23:18:57.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.795+0000 7f4ebec96640 1 -- 172.21.15.44:0/2748047413 wait complete. 2024-04-21T23:18:57.799 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.795+0000 7f4ebec96640 1 Processor -- start 2024-04-21T23:18:57.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.795+0000 7f4ebec96640 1 -- start start 2024-04-21T23:18:57.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.795+0000 7f4ebec96640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4eb8100530 0x7f4eb819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:57.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.795+0000 7f4ebec96640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4eb8100e50 0x7f4eb819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:57.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.795+0000 7f4ebec96640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4eb810e0e0 0x7f4eb819eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:57.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.795+0000 7f4ebec96640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb8113d40 con 0x7f4eb810e0e0 2024-04-21T23:18:57.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.795+0000 7f4ebec96640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb8113bc0 con 0x7f4eb8100e50 2024-04-21T23:18:57.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.795+0000 7f4ebec96640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb8113ec0 con 0x7f4eb8100530 2024-04-21T23:18:57.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.796+0000 7f4ebd233640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4eb810e0e0 0x7f4eb819eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:57.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.796+0000 7f4ebd233640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4eb810e0e0 0x7f4eb819eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38764/0 (socket says 172.21.15.44:38764) 2024-04-21T23:18:57.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.796+0000 7f4ebd233640 1 -- 172.21.15.44:0/753851204 learned_addr learned my addr 172.21.15.44:0/753851204 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:18:57.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.796+0000 7f4ebca32640 1 --2- 172.21.15.44:0/753851204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4eb8100530 0x7f4eb819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:57.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.796+0000 7f4eb7fff640 1 --2- 172.21.15.44:0/753851204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4eb8100e50 0x7f4eb819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:57.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.796+0000 7f4ebd233640 1 -- 172.21.15.44:0/753851204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4eb8100530 msgr2=0x7f4eb819a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:57.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.796+0000 7f4ebd233640 1 --2- 172.21.15.44:0/753851204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4eb8100530 0x7f4eb819a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:57.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.796+0000 7f4ebd233640 1 -- 172.21.15.44:0/753851204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4eb8100e50 msgr2=0x7f4eb819a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:57.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.796+0000 7f4ebd233640 1 --2- 172.21.15.44:0/753851204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4eb8100e50 0x7f4eb819a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:57.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.796+0000 7f4ebd233640 1 -- 172.21.15.44:0/753851204 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eb819f230 con 0x7f4eb810e0e0 2024-04-21T23:18:57.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.796+0000 7f4ebd233640 1 --2- 172.21.15.44:0/753851204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4eb810e0e0 0x7f4eb819eb00 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0009fd0 tx=0x7f4eb000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:57.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.796+0000 7f4eb5ffb640 1 -- 172.21.15.44:0/753851204 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0004d20 con 0x7f4eb810e0e0 2024-04-21T23:18:57.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.796+0000 7f4eb5ffb640 1 -- 172.21.15.44:0/753851204 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4eb0005020 con 0x7f4eb810e0e0 2024-04-21T23:18:57.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.796+0000 7f4ebec96640 1 -- 172.21.15.44:0/753851204 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eb819f520 con 0x7f4eb810e0e0 2024-04-21T23:18:57.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.796+0000 7f4eb5ffb640 1 -- 172.21.15.44:0/753851204 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb00116d0 con 0x7f4eb810e0e0 2024-04-21T23:18:57.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.796+0000 7f4ebec96640 1 -- 172.21.15.44:0/753851204 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eb81a6db0 con 0x7f4eb810e0e0 2024-04-21T23:18:57.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.798+0000 7f4ebec96640 1 -- 172.21.15.44:0/753851204 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4eb804fd50 con 0x7f4eb810e0e0 2024-04-21T23:18:57.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.799+0000 7f4eb5ffb640 1 -- 172.21.15.44:0/753851204 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f4eb00051d0 con 0x7f4eb810e0e0 2024-04-21T23:18:57.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.799+0000 7f4eb5ffb640 1 --2- 172.21.15.44:0/753851204 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4ea00a2cf0 0x7f4ea00a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:18:57.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.799+0000 7f4ebca32640 1 --2- 172.21.15.44:0/753851204 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4ea00a2cf0 0x7f4ea00a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:18:57.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.799+0000 7f4eb5ffb640 1 -- 172.21.15.44:0/753851204 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7f4eb00cfdd0 con 0x7f4eb810e0e0 2024-04-21T23:18:57.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.800+0000 7f4ebca32640 1 --2- 172.21.15.44:0/753851204 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4ea00a2cf0 0x7f4ea00a51b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8006010 tx=0x7f4ea8000bc0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:18:57.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:57.804+0000 7f4eb5ffb640 1 -- 172.21.15.44:0/753851204 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f4eb0016030 con 0x7f4eb810e0e0 2024-04-21T23:18:58.018 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:58.019+0000 7f4ebec96640 1 -- 172.21.15.44:0/753851204 --> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f4eb8061b80 con 0x7f4ea00a2cf0 2024-04-21T23:18:58.025 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:58.026+0000 7f4eb5ffb640 1 -- 172.21.15.44:0/753851204 <== mgr.15396 v2:172.21.15.44:6800/3789150048 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f4eb8061b80 con 0x7f4ea00a2cf0 2024-04-21T23:18:58.026 INFO:teuthology.orchestra.run.smithi044.stdout:/ 1m 6m3h 2024-04-21T23:18:58.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:58.028+0000 7f4ebec96640 1 -- 172.21.15.44:0/753851204 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4ea00a2cf0 msgr2=0x7f4ea00a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:58.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:58.028+0000 7f4ebec96640 1 --2- 172.21.15.44:0/753851204 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4ea00a2cf0 0x7f4ea00a51b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8006010 tx=0x7f4ea8000bc0 comp rx=0 tx=0).stop 2024-04-21T23:18:58.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:58.028+0000 7f4ebec96640 1 -- 172.21.15.44:0/753851204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4eb810e0e0 msgr2=0x7f4eb819eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:18:58.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:58.028+0000 7f4ebec96640 1 --2- 172.21.15.44:0/753851204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4eb810e0e0 0x7f4eb819eb00 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0009fd0 tx=0x7f4eb000ee60 comp rx=0 tx=0).stop 2024-04-21T23:18:58.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:58.029+0000 7f4ebec96640 1 -- 172.21.15.44:0/753851204 shutdown_connections 2024-04-21T23:18:58.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:58.029+0000 7f4ebec96640 1 --2- 172.21.15.44:0/753851204 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f4ea00a2cf0 0x7f4ea00a51b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:58.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:58.029+0000 7f4ebec96640 1 --2- 172.21.15.44:0/753851204 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f4eb810e0e0 0x7f4eb819eb00 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:58.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:58.029+0000 7f4ebec96640 1 --2- 172.21.15.44:0/753851204 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4eb8100e50 0x7f4eb819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:58.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:58.029+0000 7f4ebec96640 1 --2- 172.21.15.44:0/753851204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4eb8100530 0x7f4eb819a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:18:58.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:58.029+0000 7f4ebec96640 1 -- 172.21.15.44:0/753851204 >> 172.21.15.44:0/753851204 conn(0x7f4eb8077760 msgr2=0x7f4eb810f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:18:58.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:58.029+0000 7f4ebec96640 1 -- 172.21.15.44:0/753851204 shutdown_connections 2024-04-21T23:18:58.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:18:58.029+0000 7f4ebec96640 1 -- 172.21.15.44:0/753851204 wait complete. 2024-04-21T23:18:58.039 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-21T23:18:58.050 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-21T23:18:58.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:58 smithi044 ceph-mon[27934]: pgmap v319: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:18:58.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:58 smithi069 ceph-mon[31816]: pgmap v319: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:18:58.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:58 smithi055 ceph-mon[31117]: pgmap v319: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:18:59.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:59 smithi044 ceph-mon[27934]: from='client.17766 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:59.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:18:59 smithi044 ceph-mon[27934]: from='client.17772 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:59.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:59 smithi055 ceph-mon[31117]: from='client.17766 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:59.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:18:59 smithi055 ceph-mon[31117]: from='client.17772 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:59.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:59 smithi069 ceph-mon[31816]: from='client.17766 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:18:59.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:18:59 smithi069 ceph-mon[31816]: from='client.17772 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T23:19:00.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:00 smithi044 ceph-mon[27934]: pgmap v320: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-04-21T23:19:00.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:00 smithi069 ceph-mon[31816]: pgmap v320: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-04-21T23:19:00.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:00 smithi055 ceph-mon[31117]: pgmap v320: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-04-21T23:19:02.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:02 smithi044 ceph-mon[27934]: pgmap v321: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:19:02.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:02 smithi055 ceph-mon[31117]: pgmap v321: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:19:02.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:02 smithi069 ceph-mon[31816]: pgmap v321: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:19:04.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:04 smithi044 ceph-mon[27934]: pgmap v322: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-21T23:19:04.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:04 smithi069 ceph-mon[31816]: pgmap v322: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-21T23:19:04.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:04 smithi055 ceph-mon[31117]: pgmap v322: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-21T23:19:05.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T23:19:05.010 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T23:19:05.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.116+0000 7f640373a640 1 -- 172.21.15.44:0/2855042896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63fc1066b0 msgr2=0x7f63fc106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.116+0000 7f640373a640 1 --2- 172.21.15.44:0/2855042896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63fc1066b0 0x7f63fc106a90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009cd0 tx=0x7f63f802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.118 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.117+0000 7f640373a640 1 -- 172.21.15.44:0/2855042896 shutdown_connections 2024-04-21T23:19:05.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.117+0000 7f640373a640 1 --2- 172.21.15.44:0/2855042896 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63fc100fd0 0x7f63fc10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.117+0000 7f640373a640 1 --2- 172.21.15.44:0/2855042896 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63fc100610 0x7f63fc100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.117+0000 7f640373a640 1 --2- 172.21.15.44:0/2855042896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63fc1066b0 0x7f63fc106a90 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.117+0000 7f640373a640 1 -- 172.21.15.44:0/2855042896 >> 172.21.15.44:0/2855042896 conn(0x7f63fc0777f0 msgr2=0x7f63fc077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:05.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.117+0000 7f640373a640 1 -- 172.21.15.44:0/2855042896 shutdown_connections 2024-04-21T23:19:05.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.117+0000 7f640373a640 1 -- 172.21.15.44:0/2855042896 wait complete. 2024-04-21T23:19:05.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.117+0000 7f640373a640 1 Processor -- start 2024-04-21T23:19:05.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.117+0000 7f640373a640 1 -- start start 2024-04-21T23:19:05.119 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.117+0000 7f640373a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63fc100610 0x7f63fc198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.117+0000 7f640373a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63fc100fd0 0x7f63fc199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.117+0000 7f640373a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63fc1066b0 0x7f63fc1a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.117+0000 7f640373a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc110a30 con 0x7f63fc100610 2024-04-21T23:19:05.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.117+0000 7f640373a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc1108b0 con 0x7f63fc1066b0 2024-04-21T23:19:05.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.118+0000 7f640373a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc110bb0 con 0x7f63fc100fd0 2024-04-21T23:19:05.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.118+0000 7f64014d6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63fc100610 0x7f63fc198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.118+0000 7f64014d6640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63fc100610 0x7f63fc198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38304/0 (socket says 172.21.15.44:38304) 2024-04-21T23:19:05.120 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.118+0000 7f64014d6640 1 -- 172.21.15.44:0/1624931140 learned_addr learned my addr 172.21.15.44:0/1624931140 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.118+0000 7f6401cd7640 1 --2- 172.21.15.44:0/1624931140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63fc1066b0 0x7f63fc1a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.118+0000 7f6400cd5640 1 --2- 172.21.15.44:0/1624931140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63fc100fd0 0x7f63fc199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.118+0000 7f64014d6640 1 -- 172.21.15.44:0/1624931140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63fc100fd0 msgr2=0x7f63fc199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.118+0000 7f64014d6640 1 --2- 172.21.15.44:0/1624931140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63fc100fd0 0x7f63fc199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.118+0000 7f64014d6640 1 -- 172.21.15.44:0/1624931140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63fc1066b0 msgr2=0x7f63fc1a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.118+0000 7f64014d6640 1 --2- 172.21.15.44:0/1624931140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63fc1066b0 0x7f63fc1a3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.118+0000 7f64014d6640 1 -- 172.21.15.44:0/1624931140 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63fc1a43b0 con 0x7f63fc100610 2024-04-21T23:19:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.118+0000 7f6401cd7640 1 --2- 172.21.15.44:0/1624931140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63fc1066b0 0x7f63fc1a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:05.121 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.118+0000 7f64014d6640 1 --2- 172.21.15.44:0/1624931140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63fc100610 0x7f63fc198df0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f63f8005cc0 tx=0x7f63f8032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:05.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.118+0000 7f63ee7fc640 1 -- 172.21.15.44:0/1624931140 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f8041020 con 0x7f63fc100610 2024-04-21T23:19:05.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.118+0000 7f640373a640 1 -- 172.21.15.44:0/1624931140 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63fc1a4580 con 0x7f63fc100610 2024-04-21T23:19:05.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.118+0000 7f63ee7fc640 1 -- 172.21.15.44:0/1624931140 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63f8034cd0 con 0x7f63fc100610 2024-04-21T23:19:05.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.118+0000 7f63ee7fc640 1 -- 172.21.15.44:0/1624931140 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f8039040 con 0x7f63fc100610 2024-04-21T23:19:05.122 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.118+0000 7f640373a640 1 -- 172.21.15.44:0/1624931140 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63fc1a49f0 con 0x7f63fc100610 2024-04-21T23:19:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.121+0000 7f63ee7fc640 1 -- 172.21.15.44:0/1624931140 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f63f8044020 con 0x7f63fc100610 2024-04-21T23:19:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.121+0000 7f64063ff740 1 Processor -- start 2024-04-21T23:19:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.121+0000 7f64063ff740 1 -- start start 2024-04-21T23:19:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.121+0000 7f63ee7fc640 1 --2- 172.21.15.44:0/1624931140 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f63e80a2d00 0x7f63e80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f64063ff740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173ddc60 0x556c173de040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f64063ff740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173de580 0x556c173eb300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.129 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f64063ff740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173eb840 0x556c173edc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f64063ff740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x556c17342560 con 0x556c173eb840 2024-04-21T23:19:05.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f64063ff740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x556c172ea730 con 0x556c173ddc60 2024-04-21T23:19:05.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f64063ff740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556c1733e520 con 0x556c173de580 2024-04-21T23:19:05.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f63ee7fc640 1 -- 172.21.15.44:0/1624931140 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7f63f80f13b0 con 0x7f63fc100610 2024-04-21T23:19:05.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f6400cd5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173de580 0x556c173eb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.130 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f6400cd5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173de580 0x556c173eb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:40204/0 (socket says 172.21.15.44:40204) 2024-04-21T23:19:05.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f6400cd5640 1 -- 172.21.15.44:0/3272255701 learned_addr learned my addr 172.21.15.44:0/3272255701 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:05.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f6401cd7640 1 --2- 172.21.15.44:0/3272255701 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173eb840 0x556c173edc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f64014d6640 1 --2- 172.21.15.44:0/3272255701 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173ddc60 0x556c173de040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f6401cd7640 1 --2- 172.21.15.44:0/1624931140 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f63e80a2d00 0x7f63e80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f6400cd5640 1 -- 172.21.15.44:0/3272255701 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173ddc60 msgr2=0x556c173de040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:05.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f6400cd5640 1 --2- 172.21.15.44:0/3272255701 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173ddc60 0x556c173de040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f6400cd5640 1 -- 172.21.15.44:0/3272255701 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173eb840 msgr2=0x556c173edc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:05.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f6400cd5640 1 --2- 172.21.15.44:0/3272255701 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173eb840 0x556c173edc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.131 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f6400cd5640 1 -- 172.21.15.44:0/3272255701 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c171207d0 con 0x556c173de580 2024-04-21T23:19:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f64014d6640 1 --2- 172.21.15.44:0/3272255701 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173ddc60 0x556c173de040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f6400cd5640 1 --2- 172.21.15.44:0/3272255701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173de580 0x556c173eb300 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f63f0009c90 tx=0x7f63f0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b07a300009e4a236 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f6401cd7640 1 --2- 172.21.15.44:0/1624931140 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f63e80a2d00 0x7f63e80a51c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f63f40058c0 tx=0x7f63f4005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.122+0000 7f63e77fe640 1 -- 172.21.15.44:0/3272255701 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f001e070 con 0x556c173de580 2024-04-21T23:19:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.123+0000 7f63e77fe640 1 -- 172.21.15.44:0/3272255701 <== mon.1 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63f0019b10 con 0x556c173de580 2024-04-21T23:19:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.123+0000 7f63e77fe640 1 -- 172.21.15.44:0/3272255701 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f000eaf0 con 0x556c173de580 2024-04-21T23:19:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.123+0000 7f64063ff740 1 -- 172.21.15.44:0/3272255701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173de580 msgr2=0x556c173eb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.123+0000 7f64063ff740 1 --2- 172.21.15.44:0/3272255701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173de580 0x556c173eb300 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f63f0009c90 tx=0x7f63f0011810 comp rx=0 tx=0).stop 2024-04-21T23:19:05.132 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.123+0000 7f64063ff740 1 -- 172.21.15.44:0/3272255701 shutdown_connections 2024-04-21T23:19:05.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.123+0000 7f64063ff740 1 --2- 172.21.15.44:0/3272255701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173de580 0x556c173eb300 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.123+0000 7f64063ff740 1 --2- 172.21.15.44:0/3272255701 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173eb840 0x556c173edc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.123+0000 7f64063ff740 1 --2- 172.21.15.44:0/3272255701 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173ddc60 0x556c173de040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.123+0000 7f64063ff740 1 -- 172.21.15.44:0/3272255701 >> 172.21.15.44:0/3272255701 conn(0x556c173d8660 msgr2=0x556c173d8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:05.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.123+0000 7f64063ff740 1 -- 172.21.15.44:0/3272255701 shutdown_connections 2024-04-21T23:19:05.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.123+0000 7f64063ff740 1 -- 172.21.15.44:0/3272255701 wait complete. 2024-04-21T23:19:05.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.124+0000 7f64063ff740 1 -- start start 2024-04-21T23:19:05.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.124+0000 7f64063ff740 1 Processor -- start 2024-04-21T23:19:05.133 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.124+0000 7f64063ff740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173ddc60 0x556c173e8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.124+0000 7f64063ff740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173de580 0x556c173e8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.124+0000 7f64063ff740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173eb840 0x556c173e9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.124+0000 7f64063ff740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x556c173ee600 con 0x556c173eb840 2024-04-21T23:19:05.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.124+0000 7f64063ff740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x556c172fde00 con 0x556c173ddc60 2024-04-21T23:19:05.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.124+0000 7f64063ff740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556c173ee780 con 0x556c173de580 2024-04-21T23:19:05.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.124+0000 7f64063ff740 10 client.0 _ll_register_callbacks cb 0x556c17338eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:05.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.124+0000 7f6401cd7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173eb840 0x556c173e9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.134 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.124+0000 7f6400cd5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173de580 0x556c173e8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.124+0000 7f6400cd5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173de580 0x556c173e8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:40216/0 (socket says 172.21.15.44:40216) 2024-04-21T23:19:05.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.124+0000 7f64014d6640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173ddc60 0x556c173e8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.124+0000 7f6400cd5640 1 -- 172.21.15.44:0/3752574489 learned_addr learned my addr 172.21.15.44:0/3752574489 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:05.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.125+0000 7f6400cd5640 1 -- 172.21.15.44:0/3752574489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173ddc60 msgr2=0x556c173e8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:05.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.125+0000 7f6400cd5640 1 --2- 172.21.15.44:0/3752574489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173ddc60 0x556c173e8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.125+0000 7f6400cd5640 1 -- 172.21.15.44:0/3752574489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173eb840 msgr2=0x556c173e9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:05.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.125+0000 7f6400cd5640 1 --2- 172.21.15.44:0/3752574489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173eb840 0x556c173e9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.135 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.125+0000 7f6400cd5640 1 -- 172.21.15.44:0/3752574489 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c170563f0 con 0x556c173de580 2024-04-21T23:19:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.125+0000 7f6400cd5640 1 --2- 172.21.15.44:0/3752574489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173de580 0x556c173e8e40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f63f0009890 tx=0x7f63f0019660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=631518fc6fe3dab8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.125+0000 7f63e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:19:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.125+0000 7f63e4ff9640 1 -- 172.21.15.44:0/3752574489 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f001e040 con 0x556c173de580 2024-04-21T23:19:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.125+0000 7f64063ff740 1 -- 172.21.15.44:0/3752574489 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63f0019b10 con 0x556c173de580 2024-04-21T23:19:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.125+0000 7f63e4ff9640 1 -- 172.21.15.44:0/3752574489 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7f63f0004030 con 0x556c173de580 2024-04-21T23:19:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.125+0000 7f63e4ff9640 1 -- 172.21.15.44:0/3752574489 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f63f0033050 con 0x556c173de580 2024-04-21T23:19:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.125+0000 7f64063ff740 10 client.27241 fetch_fsmap learned FSMap version 31 2024-04-21T23:19:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.125+0000 7f64063ff740 1 -- 172.21.15.44:0/3752574489 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556c170160f0 con 0x556c173de580 2024-04-21T23:19:05.136 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.125+0000 7f63e4ff9640 1 -- 172.21.15.44:0/3752574489 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63f0019b10 con 0x556c173de580 2024-04-21T23:19:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.126+0000 7f63e4ff9640 1 -- 172.21.15.44:0/3752574489 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f63f000e7f0 con 0x556c173de580 2024-04-21T23:19:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.126+0000 7f64063ff740 10 client.27241 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:19:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.126+0000 7f64063ff740 10 client.27241 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.126+0000 7f64063ff740 20 client.27241 populate_metadata read hostname 'smithi044' 2024-04-21T23:19:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.126+0000 7f64063ff740 1 --2- 172.21.15.44:0/3752574489 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556c173fbb70 0x556c173fdf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.126+0000 7f64063ff740 4 client.27241 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:19:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.126+0000 7f64063ff740 1 -- 172.21.15.44:0/3752574489 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c1734faa0 con 0x556c173fbb70 2024-04-21T23:19:05.137 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.126+0000 7f64014d6640 1 --2- 172.21.15.44:0/3752574489 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556c173fbb70 0x556c173fdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.126+0000 7f64014d6640 1 --2- 172.21.15.44:0/3752574489 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556c173fbb70 0x556c173fdf60 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66a51afea88bba99 server_cookie=e0c0a5408c437ed2 in_seq=0 out_seq=0 2024-04-21T23:19:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.126+0000 7f63e4ff9640 10 client.27241 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:19:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.127+0000 7f63e4ff9640 1 -- 172.21.15.44:0/3752574489 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556c1734faa0 con 0x556c173fbb70 2024-04-21T23:19:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.127+0000 7f63e4ff9640 10 client.27241 handle_command_reply: tid=0 2024-04-21T23:19:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.127+0000 7f64063ff740 1 client.27241 shutdown 2024-04-21T23:19:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.127+0000 7f64063ff740 1 -- 172.21.15.44:0/3752574489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173de580 msgr2=0x556c173e8e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.127+0000 7f64063ff740 1 --2- 172.21.15.44:0/3752574489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173de580 0x556c173e8e40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f63f0009890 tx=0x7f63f0019660 comp rx=0 tx=0).stop 2024-04-21T23:19:05.138 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.127+0000 7f64063ff740 1 -- 172.21.15.44:0/3752574489 shutdown_connections 2024-04-21T23:19:05.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.127+0000 7f64063ff740 1 --2- 172.21.15.44:0/3752574489 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556c173fbb70 0x556c173fdf60 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.127+0000 7f64063ff740 1 --2- 172.21.15.44:0/3752574489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173de580 0x556c173e8e40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.127+0000 7f64063ff740 1 --2- 172.21.15.44:0/3752574489 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173eb840 0x556c173e9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.127+0000 7f63e4ff9640 0 client.27241 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:19:05.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.127+0000 7f64063ff740 1 --2- 172.21.15.44:0/3752574489 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173ddc60 0x556c173e8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.127+0000 7f64063ff740 1 -- 172.21.15.44:0/3752574489 >> 172.21.15.44:0/3752574489 conn(0x556c173d8660 msgr2=0x556c173da3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:05.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.127+0000 7f64063ff740 1 -- 172.21.15.44:0/3752574489 shutdown_connections 2024-04-21T23:19:05.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.127+0000 7f64063ff740 1 -- 172.21.15.44:0/3752574489 wait complete. 2024-04-21T23:19:05.139 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.127+0000 7f64063ff740 20 client.27241 trim_cache size 0 max 16384 2024-04-21T23:19:05.146 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.145+0000 7f64063ff740 1 Processor -- start 2024-04-21T23:19:05.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.145+0000 7f64063ff740 1 -- start start 2024-04-21T23:19:05.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.145+0000 7f64063ff740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173ddc60 0x556c173eb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.145+0000 7f64063ff740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173de580 0x556c173f7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.145+0000 7f64063ff740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173eb840 0x556c173e7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.145+0000 7f64063ff740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc1aaaf0 con 0x556c173de580 2024-04-21T23:19:05.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.145+0000 7f64063ff740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc1ad350 con 0x556c173eb840 2024-04-21T23:19:05.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.145+0000 7f64063ff740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc1ab140 con 0x556c173ddc60 2024-04-21T23:19:05.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.145+0000 7f6400cd5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173de580 0x556c173f7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.147 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.145+0000 7f6400cd5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173de580 0x556c173f7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38324/0 (socket says 172.21.15.44:38324) 2024-04-21T23:19:05.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.145+0000 7f6400cd5640 1 -- 172.21.15.44:0/666786227 learned_addr learned my addr 172.21.15.44:0/666786227 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:05.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.145+0000 7f64014d6640 1 --2- 172.21.15.44:0/666786227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173ddc60 0x556c173eb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.145+0000 7f6401cd7640 1 --2- 172.21.15.44:0/666786227 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173eb840 0x556c173e7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.145+0000 7f6400cd5640 1 -- 172.21.15.44:0/666786227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173ddc60 msgr2=0x556c173eb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:05.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.145+0000 7f6400cd5640 1 --2- 172.21.15.44:0/666786227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173ddc60 0x556c173eb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.145+0000 7f6400cd5640 1 -- 172.21.15.44:0/666786227 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173eb840 msgr2=0x556c173e7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:05.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.145+0000 7f6400cd5640 1 --2- 172.21.15.44:0/666786227 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173eb840 0x556c173e7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.148 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.145+0000 7f6400cd5640 1 -- 172.21.15.44:0/666786227 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c173fe520 con 0x556c173de580 2024-04-21T23:19:05.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.146+0000 7f64014d6640 1 --2- 172.21.15.44:0/666786227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173ddc60 0x556c173eb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:05.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.146+0000 7f6400cd5640 1 --2- 172.21.15.44:0/666786227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173de580 0x556c173f7500 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f63f0000c00 tx=0x7f63f0009370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2abbb747e90d81f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:05.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.146+0000 7f63e6ffd640 1 -- 172.21.15.44:0/666786227 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f0011d10 con 0x556c173de580 2024-04-21T23:19:05.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.146+0000 7f63e6ffd640 1 -- 172.21.15.44:0/666786227 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63f000e3b0 con 0x556c173de580 2024-04-21T23:19:05.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.146+0000 7f63e6ffd640 1 -- 172.21.15.44:0/666786227 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f0004710 con 0x556c173de580 2024-04-21T23:19:05.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.147+0000 7f64063ff740 1 -- 172.21.15.44:0/666786227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173de580 msgr2=0x556c173f7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.147+0000 7f64063ff740 1 --2- 172.21.15.44:0/666786227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173de580 0x556c173f7500 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f63f0000c00 tx=0x7f63f0009370 comp rx=0 tx=0).stop 2024-04-21T23:19:05.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.147+0000 7f64063ff740 1 -- 172.21.15.44:0/666786227 shutdown_connections 2024-04-21T23:19:05.149 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.147+0000 7f64063ff740 1 --2- 172.21.15.44:0/666786227 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173eb840 0x556c173e7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.147+0000 7f64063ff740 1 --2- 172.21.15.44:0/666786227 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173de580 0x556c173f7500 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.147+0000 7f64063ff740 1 --2- 172.21.15.44:0/666786227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173ddc60 0x556c173eb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.147+0000 7f64063ff740 1 -- 172.21.15.44:0/666786227 >> 172.21.15.44:0/666786227 conn(0x556c173d8660 msgr2=0x556c173da340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:05.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.147+0000 7f64063ff740 1 -- 172.21.15.44:0/666786227 shutdown_connections 2024-04-21T23:19:05.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.147+0000 7f64063ff740 1 -- 172.21.15.44:0/666786227 wait complete. 2024-04-21T23:19:05.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.147+0000 7f64063ff740 1 -- start start 2024-04-21T23:19:05.150 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.148+0000 7f64063ff740 1 Processor -- start 2024-04-21T23:19:05.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.148+0000 7f64063ff740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173ddc60 0x556c173f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.148+0000 7f64063ff740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173de580 0x556c173e54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.148+0000 7f64063ff740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173eb840 0x556c173e5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.148+0000 7f64063ff740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc1ab810 con 0x556c173eb840 2024-04-21T23:19:05.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.148+0000 7f64063ff740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc1acc80 con 0x556c173de580 2024-04-21T23:19:05.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.148+0000 7f64063ff740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc1abee0 con 0x556c173ddc60 2024-04-21T23:19:05.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.148+0000 7f64063ff740 10 client.0 _ll_register_callbacks cb 0x7f63f001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:05.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.148+0000 7f6401cd7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173eb840 0x556c173e5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.148+0000 7f64014d6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173ddc60 0x556c173f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.148+0000 7f6400cd5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173de580 0x556c173e54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.148+0000 7f6401cd7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173eb840 0x556c173e5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38336/0 (socket says 172.21.15.44:38336) 2024-04-21T23:19:05.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.148+0000 7f64014d6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173ddc60 0x556c173f3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.44:40224/0 (socket says 172.21.15.44:40224) 2024-04-21T23:19:05.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.148+0000 7f6401cd7640 1 -- 172.21.15.44:0/2811208693 learned_addr learned my addr 172.21.15.44:0/2811208693 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:05.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.149+0000 7f64014d6640 1 -- 172.21.15.44:0/2811208693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173de580 msgr2=0x556c173e54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:05.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.149+0000 7f64014d6640 1 --2- 172.21.15.44:0/2811208693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173de580 0x556c173e54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.149+0000 7f64014d6640 1 -- 172.21.15.44:0/2811208693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173eb840 msgr2=0x556c173e5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:05.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.149+0000 7f64014d6640 1 --2- 172.21.15.44:0/2811208693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173eb840 0x556c173e5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.149+0000 7f64014d6640 1 -- 172.21.15.44:0/2811208693 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c17461000 con 0x556c173ddc60 2024-04-21T23:19:05.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.149+0000 7f6401cd7640 1 --2- 172.21.15.44:0/2811208693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173eb840 0x556c173e5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:05.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.149+0000 7f64014d6640 1 --2- 172.21.15.44:0/2811208693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173ddc60 0x556c173f3f90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f63f80ee820 tx=0x7f63f8032b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6483f3aa13d61c7d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:05.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.149+0000 7f63e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-21T23:19:05.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.149+0000 7f63e4ff9640 1 -- 172.21.15.44:0/2811208693 <== mon.1 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f80eee60 con 0x556c173ddc60 2024-04-21T23:19:05.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.149+0000 7f63e4ff9640 1 -- 172.21.15.44:0/2811208693 <== mon.1 v2:172.21.15.69:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7f63f80f10b0 con 0x556c173ddc60 2024-04-21T23:19:05.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.149+0000 7f64063ff740 1 -- 172.21.15.44:0/2811208693 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63f000e3b0 con 0x556c173ddc60 2024-04-21T23:19:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.149+0000 7f63e4ff9640 1 -- 172.21.15.44:0/2811208693 <== mon.1 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f63f80bcf50 con 0x556c173ddc60 2024-04-21T23:19:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.149+0000 7f64063ff740 10 client.27247 fetch_fsmap learned FSMap version 31 2024-04-21T23:19:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.149+0000 7f64063ff740 1 -- 172.21.15.44:0/2811208693 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556c17462230 con 0x556c173ddc60 2024-04-21T23:19:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.149+0000 7f63e4ff9640 1 -- 172.21.15.44:0/2811208693 <== mon.1 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63f000e3b0 con 0x556c173ddc60 2024-04-21T23:19:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.150+0000 7f63e4ff9640 1 -- 172.21.15.44:0/2811208693 <== mon.1 v2:172.21.15.69:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f63f803c030 con 0x556c173ddc60 2024-04-21T23:19:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.150+0000 7f64063ff740 10 client.27247 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:19:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.150+0000 7f64063ff740 10 client.27247 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.150+0000 7f64063ff740 20 client.27247 populate_metadata read hostname 'smithi044' 2024-04-21T23:19:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.150+0000 7f64063ff740 1 --2- 172.21.15.44:0/2811208693 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556c17462950 0x556c17464d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.150+0000 7f64063ff740 4 client.27247 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T23:19:05.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.150+0000 7f64063ff740 1 -- 172.21.15.44:0/2811208693 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556c17349df0 con 0x556c17462950 2024-04-21T23:19:05.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.150+0000 7f6400cd5640 1 --2- 172.21.15.44:0/2811208693 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556c17462950 0x556c17464d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.150+0000 7f6400cd5640 1 --2- 172.21.15.44:0/2811208693 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556c17462950 0x556c17464d70 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdccb4f0bdc6c29b server_cookie=81341f33850aaf76 in_seq=0 out_seq=0 2024-04-21T23:19:05.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.150+0000 7f63e4ff9640 10 client.27247 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:19:05.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.150+0000 7f63e4ff9640 1 -- 172.21.15.44:0/2811208693 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556c17349df0 con 0x556c17462950 2024-04-21T23:19:05.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.150+0000 7f63e4ff9640 10 client.27247 handle_command_reply: tid=0 2024-04-21T23:19:05.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.150+0000 7f64063ff740 1 client.27247 shutdown 2024-04-21T23:19:05.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f64063ff740 1 -- 172.21.15.44:0/2811208693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173ddc60 msgr2=0x556c173f3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f64063ff740 1 --2- 172.21.15.44:0/2811208693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173ddc60 0x556c173f3f90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f63f80ee820 tx=0x7f63f8032b00 comp rx=0 tx=0).stop 2024-04-21T23:19:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f64063ff740 1 -- 172.21.15.44:0/2811208693 shutdown_connections 2024-04-21T23:19:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f64063ff740 1 --2- 172.21.15.44:0/2811208693 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x556c17462950 0x556c17464d70 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f64063ff740 1 --2- 172.21.15.44:0/2811208693 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x556c173eb840 0x556c173e5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f64063ff740 1 --2- 172.21.15.44:0/2811208693 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x556c173de580 0x556c173e54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f64063ff740 1 --2- 172.21.15.44:0/2811208693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556c173ddc60 0x556c173f3f90 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f63e4ff9640 0 client.27247 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:19:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f64063ff740 1 -- 172.21.15.44:0/2811208693 >> 172.21.15.44:0/2811208693 conn(0x556c173d8660 msgr2=0x556c173d6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:05.159 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f64063ff740 1 -- 172.21.15.44:0/2811208693 shutdown_connections 2024-04-21T23:19:05.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f64063ff740 1 -- 172.21.15.44:0/2811208693 wait complete. 2024-04-21T23:19:05.160 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f64063ff740 20 client.27247 trim_cache size 0 max 16384 2024-04-21T23:19:05.160 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:19:05.160 INFO:teuthology.orchestra.run.smithi044.stdout: "status": "no active scrubs running", 2024-04-21T23:19:05.160 INFO:teuthology.orchestra.run.smithi044.stdout: "scrubs": {} 2024-04-21T23:19:05.160 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:19:05.161 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f640373a640 1 -- 172.21.15.44:0/1624931140 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f63e80a2d00 msgr2=0x7f63e80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f640373a640 1 --2- 172.21.15.44:0/1624931140 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f63e80a2d00 0x7f63e80a51c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f63f40058c0 tx=0x7f63f4005f90 comp rx=0 tx=0).stop 2024-04-21T23:19:05.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f640373a640 1 -- 172.21.15.44:0/1624931140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63fc100610 msgr2=0x7f63fc198df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f640373a640 1 --2- 172.21.15.44:0/1624931140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63fc100610 0x7f63fc198df0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f63f8005cc0 tx=0x7f63f8032c90 comp rx=0 tx=0).stop 2024-04-21T23:19:05.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f640373a640 1 -- 172.21.15.44:0/1624931140 shutdown_connections 2024-04-21T23:19:05.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f640373a640 1 --2- 172.21.15.44:0/1624931140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63fc100fd0 0x7f63fc199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f640373a640 1 --2- 172.21.15.44:0/1624931140 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f63e80a2d00 0x7f63e80a51c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.162 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f640373a640 1 --2- 172.21.15.44:0/1624931140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f63fc1066b0 0x7f63fc1a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f640373a640 1 --2- 172.21.15.44:0/1624931140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f63fc100610 0x7f63fc198df0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.151+0000 7f640373a640 1 -- 172.21.15.44:0/1624931140 >> 172.21.15.44:0/1624931140 conn(0x7f63fc0777f0 msgr2=0x7f63fc0ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.152+0000 7f640373a640 1 -- 172.21.15.44:0/1624931140 shutdown_connections 2024-04-21T23:19:05.163 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.152+0000 7f640373a640 1 -- 172.21.15.44:0/1624931140 wait complete. 2024-04-21T23:19:05.163 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T23:19:05.163 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:19:05.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.268+0000 7f5098a8f640 1 -- 172.21.15.44:0/3617537834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5094106630 msgr2=0x7f5094106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.268 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.268+0000 7f5098a8f640 1 --2- 172.21.15.44:0/3617537834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5094106630 0x7f5094106a10 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f5084009930 tx=0x7f508402bc30 comp rx=0 tx=0).stop 2024-04-21T23:19:05.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.268+0000 7f5098a8f640 1 -- 172.21.15.44:0/3617537834 shutdown_connections 2024-04-21T23:19:05.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.268+0000 7f5098a8f640 1 --2- 172.21.15.44:0/3617537834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5094100620 0x7f5094100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.268+0000 7f5098a8f640 1 --2- 172.21.15.44:0/3617537834 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5094100fe0 0x7f509410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.268+0000 7f5098a8f640 1 --2- 172.21.15.44:0/3617537834 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5094106630 0x7f5094106a10 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.268+0000 7f5098a8f640 1 -- 172.21.15.44:0/3617537834 >> 172.21.15.44:0/3617537834 conn(0x7f5094077820 msgr2=0x7f5094077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:05.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.268+0000 7f5098a8f640 1 -- 172.21.15.44:0/3617537834 shutdown_connections 2024-04-21T23:19:05.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.268+0000 7f5098a8f640 1 -- 172.21.15.44:0/3617537834 wait complete. 2024-04-21T23:19:05.269 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.269+0000 7f5098a8f640 1 Processor -- start 2024-04-21T23:19:05.270 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.269+0000 7f5098a8f640 1 -- start start 2024-04-21T23:19:05.271 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.269+0000 7f5098a8f640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5094100620 0x7f5094195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.269+0000 7f5098a8f640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5094100fe0 0x7f5094196350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.269+0000 7f5098a8f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5094106630 0x7f509419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.269+0000 7f5098a8f640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f509410f930 con 0x7f5094100620 2024-04-21T23:19:05.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.269+0000 7f5098a8f640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f509410f7b0 con 0x7f5094100fe0 2024-04-21T23:19:05.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.269+0000 7f5098a8f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f509410fab0 con 0x7f5094106630 2024-04-21T23:19:05.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.269+0000 7f509259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5094100620 0x7f5094195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.270+0000 7f5089d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5094100fe0 0x7f5094196350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.270+0000 7f509259c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5094100620 0x7f5094195e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38360/0 (socket says 172.21.15.44:38360) 2024-04-21T23:19:05.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.270+0000 7f509259c640 1 -- 172.21.15.44:0/693756140 learned_addr learned my addr 172.21.15.44:0/693756140 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:05.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.270+0000 7f5089d9b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5094100fe0 0x7f5094196350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:59856/0 (socket says 172.21.15.44:59856) 2024-04-21T23:19:05.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.270+0000 7f5092d9d640 1 --2- 172.21.15.44:0/693756140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5094106630 0x7f509419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.270+0000 7f5089d9b640 1 -- 172.21.15.44:0/693756140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5094106630 msgr2=0x7f509419a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.270+0000 7f5089d9b640 1 --2- 172.21.15.44:0/693756140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5094106630 0x7f509419a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.270+0000 7f5089d9b640 1 -- 172.21.15.44:0/693756140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5094100620 msgr2=0x7f5094195e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.270+0000 7f5089d9b640 1 --2- 172.21.15.44:0/693756140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5094100620 0x7f5094195e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.270+0000 7f5089d9b640 1 -- 172.21.15.44:0/693756140 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f509419ae00 con 0x7f5094100fe0 2024-04-21T23:19:05.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.270+0000 7f5092d9d640 1 --2- 172.21.15.44:0/693756140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5094106630 0x7f509419a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:05.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.270+0000 7f509259c640 1 --2- 172.21.15.44:0/693756140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5094100620 0x7f5094195e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:05.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.270+0000 7f5089d9b640 1 --2- 172.21.15.44:0/693756140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5094100fe0 0x7f5094196350 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f507c00e9f0 tx=0x7f507c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:05.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.270+0000 7f508bfff640 1 -- 172.21.15.44:0/693756140 <== mon.2 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c00ccc0 con 0x7f5094100fe0 2024-04-21T23:19:05.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.270+0000 7f5098a8f640 1 -- 172.21.15.44:0/693756140 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f509419b0f0 con 0x7f5094100fe0 2024-04-21T23:19:05.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.270+0000 7f508bfff640 1 -- 172.21.15.44:0/693756140 <== mon.2 v2:172.21.15.55:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f507c005020 con 0x7f5094100fe0 2024-04-21T23:19:05.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.270+0000 7f508bfff640 1 -- 172.21.15.44:0/693756140 <== mon.2 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c0049e0 con 0x7f5094100fe0 2024-04-21T23:19:05.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.270+0000 7f5098a8f640 1 -- 172.21.15.44:0/693756140 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50941a2980 con 0x7f5094100fe0 2024-04-21T23:19:05.274 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.271+0000 7f5098a8f640 1 -- 172.21.15.44:0/693756140 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5094101d60 con 0x7f5094100fe0 2024-04-21T23:19:05.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.273+0000 7f508bfff640 1 -- 172.21.15.44:0/693756140 <== mon.2 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f507c005560 con 0x7f5094100fe0 2024-04-21T23:19:05.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.273+0000 7f508bfff640 1 --2- 172.21.15.44:0/693756140 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f50800a4f10 0x7f50800a73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.273+0000 7f509259c640 1 --2- 172.21.15.44:0/693756140 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f50800a4f10 0x7f50800a73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.273+0000 7f508bfff640 1 -- 172.21.15.44:0/693756140 <== mon.2 v2:172.21.15.55:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7f507c0cf800 con 0x7f5094100fe0 2024-04-21T23:19:05.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.274+0000 7f509259c640 1 --2- 172.21.15.44:0/693756140 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f50800a4f10 0x7f50800a73d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5084000c00 tx=0x7f50840058d0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:05.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.282+0000 7f508bfff640 1 -- 172.21.15.44:0/693756140 <== mon.2 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f507c0051d0 con 0x7f5094100fe0 2024-04-21T23:19:05.530 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.531+0000 7f5098a8f640 1 -- 172.21.15.44:0/693756140 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5094068a40 con 0x7f5094100fe0 2024-04-21T23:19:05.531 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.532+0000 7f508bfff640 1 -- 172.21.15.44:0/693756140 <== mon.2 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11791 (secure 0 0 0) 0x7f5094068a40 con 0x7f5094100fe0 2024-04-21T23:19:05.531 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:19:05.532 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:16:20.187271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26863":{"gid":26863,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3179646768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3179646768},{"type":"v1","addr":"172.21.15.69:6839","nonce":3179646768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:19:05.532 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 31 2024-04-21T23:19:05.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.534+0000 7f5098a8f640 1 -- 172.21.15.44:0/693756140 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f50800a4f10 msgr2=0x7f50800a73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.534+0000 7f5098a8f640 1 --2- 172.21.15.44:0/693756140 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f50800a4f10 0x7f50800a73d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5084000c00 tx=0x7f50840058d0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.534+0000 7f5098a8f640 1 -- 172.21.15.44:0/693756140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5094100fe0 msgr2=0x7f5094196350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.534+0000 7f5098a8f640 1 --2- 172.21.15.44:0/693756140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5094100fe0 0x7f5094196350 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f507c00e9f0 tx=0x7f507c00eeb0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.534+0000 7f5098a8f640 1 -- 172.21.15.44:0/693756140 shutdown_connections 2024-04-21T23:19:05.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.534+0000 7f5098a8f640 1 --2- 172.21.15.44:0/693756140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5094106630 0x7f509419a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.534 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.534+0000 7f5098a8f640 1 --2- 172.21.15.44:0/693756140 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f50800a4f10 0x7f50800a73d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.534+0000 7f5098a8f640 1 --2- 172.21.15.44:0/693756140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5094100fe0 0x7f5094196350 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.534+0000 7f5098a8f640 1 --2- 172.21.15.44:0/693756140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f5094100620 0x7f5094195e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.534+0000 7f5098a8f640 1 -- 172.21.15.44:0/693756140 >> 172.21.15.44:0/693756140 conn(0x7f5094077820 msgr2=0x7f5094102ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:05.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.534+0000 7f5098a8f640 1 -- 172.21.15.44:0/693756140 shutdown_connections 2024-04-21T23:19:05.535 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.534+0000 7f5098a8f640 1 -- 172.21.15.44:0/693756140 wait complete. 2024-04-21T23:19:05.544 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T23:19:05.565 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:05 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:19:05.651 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.649+0000 7f3f3f666640 1 -- 172.21.15.44:0/1013506634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f38100610 msgr2=0x7f3f38100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.649+0000 7f3f3f666640 1 --2- 172.21.15.44:0/1013506634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f38100610 0x7f3f38100a90 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f3f28009930 tx=0x7f3f2802bc30 comp rx=0 tx=0).stop 2024-04-21T23:19:05.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.649+0000 7f3f3f666640 1 -- 172.21.15.44:0/1013506634 shutdown_connections 2024-04-21T23:19:05.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.649+0000 7f3f3f666640 1 --2- 172.21.15.44:0/1013506634 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f38100fd0 0x7f3f3810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.649+0000 7f3f3f666640 1 --2- 172.21.15.44:0/1013506634 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f38100610 0x7f3f38100a90 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.649+0000 7f3f3f666640 1 --2- 172.21.15.44:0/1013506634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f381066b0 0x7f3f38106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.649+0000 7f3f3f666640 1 -- 172.21.15.44:0/1013506634 >> 172.21.15.44:0/1013506634 conn(0x7f3f380777f0 msgr2=0x7f3f38077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:05.652 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.649+0000 7f3f3f666640 1 -- 172.21.15.44:0/1013506634 shutdown_connections 2024-04-21T23:19:05.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.649+0000 7f3f3f666640 1 -- 172.21.15.44:0/1013506634 wait complete. 2024-04-21T23:19:05.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.650+0000 7f3f3f666640 1 Processor -- start 2024-04-21T23:19:05.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.650+0000 7f3f3f666640 1 -- start start 2024-04-21T23:19:05.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.650+0000 7f3f3f666640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f38100610 0x7f3f38197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.650+0000 7f3f3f666640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f38100fd0 0x7f3f381984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.650+0000 7f3f3f666640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f381066b0 0x7f3f3819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.650+0000 7f3f3f666640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3f381109f0 con 0x7f3f38100fd0 2024-04-21T23:19:05.653 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.650+0000 7f3f3f666640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3f38110870 con 0x7f3f38100610 2024-04-21T23:19:05.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.650+0000 7f3f3f666640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3f38110b70 con 0x7f3f381066b0 2024-04-21T23:19:05.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.650+0000 7f3f3cc01640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f38100fd0 0x7f3f381984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.650+0000 7f3f3cc01640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f38100fd0 0x7f3f381984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38384/0 (socket says 172.21.15.44:38384) 2024-04-21T23:19:05.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.650+0000 7f3f3cc01640 1 -- 172.21.15.44:0/2473684505 learned_addr learned my addr 172.21.15.44:0/2473684505 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:05.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.650+0000 7f3f3d402640 1 --2- 172.21.15.44:0/2473684505 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f38100610 0x7f3f38197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.650+0000 7f3f3cc01640 1 -- 172.21.15.44:0/2473684505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f381066b0 msgr2=0x7f3f3819c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.650+0000 7f3f3cc01640 1 --2- 172.21.15.44:0/2473684505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f381066b0 0x7f3f3819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.654 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.650+0000 7f3f3cc01640 1 -- 172.21.15.44:0/2473684505 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f38100610 msgr2=0x7f3f38197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.650+0000 7f3f3cc01640 1 --2- 172.21.15.44:0/2473684505 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f38100610 0x7f3f38197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.650+0000 7f3f3cc01640 1 -- 172.21.15.44:0/2473684505 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3819cf80 con 0x7f3f38100fd0 2024-04-21T23:19:05.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.650+0000 7f3f3d402640 1 --2- 172.21.15.44:0/2473684505 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f38100610 0x7f3f38197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:05.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.650+0000 7f3f3cc01640 1 --2- 172.21.15.44:0/2473684505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f38100fd0 0x7f3f381984d0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f3f28009230 tx=0x7f3f2802cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:05.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.651+0000 7f3f2e7fc640 1 -- 172.21.15.44:0/2473684505 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f28041070 con 0x7f3f38100fd0 2024-04-21T23:19:05.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.651+0000 7f3f2e7fc640 1 -- 172.21.15.44:0/2473684505 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f28004450 con 0x7f3f38100fd0 2024-04-21T23:19:05.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.651+0000 7f3f2e7fc640 1 -- 172.21.15.44:0/2473684505 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f28003e50 con 0x7f3f38100fd0 2024-04-21T23:19:05.655 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.651+0000 7f3f3f666640 1 -- 172.21.15.44:0/2473684505 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f3819d210 con 0x7f3f38100fd0 2024-04-21T23:19:05.656 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.651+0000 7f3f3f666640 1 -- 172.21.15.44:0/2473684505 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f381a4b00 con 0x7f3f38100fd0 2024-04-21T23:19:05.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.653+0000 7f3f2e7fc640 1 -- 172.21.15.44:0/2473684505 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3f28044070 con 0x7f3f38100fd0 2024-04-21T23:19:05.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.653+0000 7f3f42246740 1 Processor -- start 2024-04-21T23:19:05.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.653+0000 7f3f42246740 1 -- start start 2024-04-21T23:19:05.662 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f42246740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 0x555a89f72130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f42246740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f72670 0x555a89f7f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f42246740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f7f930 0x555a89f81d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f42246740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x555a89ed6530 con 0x555a89f71d50 2024-04-21T23:19:05.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f42246740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x555a89e7e830 con 0x555a89f72670 2024-04-21T23:19:05.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f42246740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555a89ed24f0 con 0x555a89f7f930 2024-04-21T23:19:05.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f2e7fc640 1 --2- 172.21.15.44:0/2473684505 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3f240a2ea0 0x7f3f240a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f3dc03640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 0x555a89f72130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.663 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f3dc03640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 0x555a89f72130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38388/0 (socket says 172.21.15.44:38388) 2024-04-21T23:19:05.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f3dc03640 1 -- 172.21.15.44:0/3312098141 learned_addr learned my addr 172.21.15.44:0/3312098141 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:05.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f3dc03640 1 --2- 172.21.15.44:0/2473684505 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3f240a2ea0 0x7f3f240a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f3d402640 1 --2- 172.21.15.44:0/3312098141 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f72670 0x555a89f7f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f2e7fc640 1 -- 172.21.15.44:0/2473684505 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7f3f280f11d0 con 0x7f3f38100fd0 2024-04-21T23:19:05.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f3cc01640 1 --2- 172.21.15.44:0/3312098141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f7f930 0x555a89f81d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f3dc03640 1 -- 172.21.15.44:0/3312098141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f7f930 msgr2=0x555a89f81d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:05.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f3dc03640 1 --2- 172.21.15.44:0/3312098141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f7f930 0x555a89f81d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.664 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f3dc03640 1 -- 172.21.15.44:0/3312098141 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f72670 msgr2=0x555a89f7f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:05.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f3dc03640 1 --2- 172.21.15.44:0/3312098141 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f72670 0x555a89f7f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f3dc03640 1 -- 172.21.15.44:0/3312098141 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a89ea22c0 con 0x555a89f71d50 2024-04-21T23:19:05.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f3cc01640 1 --2- 172.21.15.44:0/3312098141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f7f930 0x555a89f81d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:19:05.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f3d402640 1 --2- 172.21.15.44:0/3312098141 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f72670 0x555a89f7f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:05.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f3dc03640 1 --2- 172.21.15.44:0/3312098141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 0x555a89f72130 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f3f30009270 tx=0x7f3f30010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2bd7c8ba1852df3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:05.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.654+0000 7f3f3dc03640 1 --2- 172.21.15.44:0/2473684505 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3f240a2ea0 0x7f3f240a5360 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3f30010c40 tx=0x7f3f30015000 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:05.665 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.655+0000 7f3f237fe640 1 -- 172.21.15.44:0/3312098141 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f300137c0 con 0x555a89f71d50 2024-04-21T23:19:05.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.655+0000 7f3f237fe640 1 -- 172.21.15.44:0/3312098141 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f30009e40 con 0x555a89f71d50 2024-04-21T23:19:05.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.655+0000 7f3f237fe640 1 -- 172.21.15.44:0/3312098141 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f30005280 con 0x555a89f71d50 2024-04-21T23:19:05.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.655+0000 7f3f42246740 1 -- 172.21.15.44:0/3312098141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 msgr2=0x555a89f72130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.655+0000 7f3f42246740 1 --2- 172.21.15.44:0/3312098141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 0x555a89f72130 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f3f30009270 tx=0x7f3f30010740 comp rx=0 tx=0).stop 2024-04-21T23:19:05.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.655+0000 7f3f42246740 1 -- 172.21.15.44:0/3312098141 shutdown_connections 2024-04-21T23:19:05.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.655+0000 7f3f42246740 1 --2- 172.21.15.44:0/3312098141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f7f930 0x555a89f81d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.655+0000 7f3f42246740 1 --2- 172.21.15.44:0/3312098141 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f72670 0x555a89f7f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.655+0000 7f3f42246740 1 --2- 172.21.15.44:0/3312098141 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 0x555a89f72130 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.666 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.655+0000 7f3f42246740 1 -- 172.21.15.44:0/3312098141 >> 172.21.15.44:0/3312098141 conn(0x555a89f6c750 msgr2=0x555a89f6cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:05.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.655+0000 7f3f42246740 1 -- 172.21.15.44:0/3312098141 shutdown_connections 2024-04-21T23:19:05.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.655+0000 7f3f42246740 1 -- 172.21.15.44:0/3312098141 wait complete. 2024-04-21T23:19:05.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.656+0000 7f3f42246740 1 -- start start 2024-04-21T23:19:05.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.656+0000 7f3f42246740 1 Processor -- start 2024-04-21T23:19:05.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.656+0000 7f3f42246740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 0x555a89f7f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.656+0000 7f3f42246740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f72670 0x555a89f7c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.656+0000 7f3f42246740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f7f930 0x555a89f7ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.656+0000 7f3f42246740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x555a89f82690 con 0x555a89f71d50 2024-04-21T23:19:05.667 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.656+0000 7f3f42246740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x555a89e91f10 con 0x555a89f72670 2024-04-21T23:19:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.656+0000 7f3f42246740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555a89f82810 con 0x555a89f7f930 2024-04-21T23:19:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.656+0000 7f3f42246740 10 client.0 _ll_register_callbacks cb 0x555a89be9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.656+0000 7f3f3d402640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 0x555a89f7f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.656+0000 7f3f3d402640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 0x555a89f7f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38404/0 (socket says 172.21.15.44:38404) 2024-04-21T23:19:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.656+0000 7f3f3d402640 1 -- 172.21.15.44:0/1942638893 learned_addr learned my addr 172.21.15.44:0/1942638893 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.656+0000 7f3f3cc01640 1 --2- 172.21.15.44:0/1942638893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f72670 0x555a89f7c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.656+0000 7f3f3dc03640 1 --2- 172.21.15.44:0/1942638893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f7f930 0x555a89f7ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.668 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.657+0000 7f3f3d402640 1 -- 172.21.15.44:0/1942638893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f7f930 msgr2=0x555a89f7ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:05.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.657+0000 7f3f3d402640 1 --2- 172.21.15.44:0/1942638893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f7f930 0x555a89f7ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.657+0000 7f3f3d402640 1 -- 172.21.15.44:0/1942638893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f72670 msgr2=0x555a89f7c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:05.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.657+0000 7f3f3d402640 1 --2- 172.21.15.44:0/1942638893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f72670 0x555a89f7c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.657+0000 7f3f3d402640 1 -- 172.21.15.44:0/1942638893 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a89e63c80 con 0x555a89f71d50 2024-04-21T23:19:05.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.657+0000 7f3f3d402640 1 --2- 172.21.15.44:0/1942638893 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 0x555a89f7f350 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f3f340136e0 tx=0x7f3f34013ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1575420134e080f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:05.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.657+0000 7f3f20ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:19:05.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.657+0000 7f3f20ff9640 1 -- 172.21.15.44:0/1942638893 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34020060 con 0x555a89f71d50 2024-04-21T23:19:05.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.657+0000 7f3f20ff9640 1 -- 172.21.15.44:0/1942638893 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7f3f34011810 con 0x555a89f71d50 2024-04-21T23:19:05.669 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.657+0000 7f3f42246740 1 -- 172.21.15.44:0/1942638893 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555a89ee3a70 con 0x555a89f71d50 2024-04-21T23:19:05.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.657+0000 7f3f20ff9640 1 -- 172.21.15.44:0/1942638893 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3f34027050 con 0x555a89f71d50 2024-04-21T23:19:05.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.657+0000 7f3f42246740 10 client.17808 fetch_fsmap learned FSMap version 31 2024-04-21T23:19:05.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.657+0000 7f3f42246740 1 -- 172.21.15.44:0/1942638893 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555a89bea3f0 con 0x555a89f71d50 2024-04-21T23:19:05.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.657+0000 7f3f20ff9640 1 -- 172.21.15.44:0/1942638893 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x555a89ee3a70 con 0x555a89f71d50 2024-04-21T23:19:05.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.657+0000 7f3f20ff9640 1 -- 172.21.15.44:0/1942638893 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f3f3400f040 con 0x555a89f71d50 2024-04-21T23:19:05.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.658+0000 7f3f42246740 10 client.17808 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:19:05.670 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.658+0000 7f3f42246740 10 client.17808 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:05.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.658+0000 7f3f42246740 20 client.17808 populate_metadata read hostname 'smithi044' 2024-04-21T23:19:05.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.658+0000 7f3f42246740 1 --2- 172.21.15.44:0/1942638893 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x555a89f8fa50 0x555a89f91eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.658+0000 7f3f42246740 4 client.17808 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T23:19:05.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.658+0000 7f3f42246740 1 -- 172.21.15.44:0/1942638893 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a89edddc0 con 0x555a89f8fa50 2024-04-21T23:19:05.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.658+0000 7f3f3cc01640 1 --2- 172.21.15.44:0/1942638893 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x555a89f8fa50 0x555a89f91eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.658+0000 7f3f3cc01640 1 --2- 172.21.15.44:0/1942638893 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x555a89f8fa50 0x555a89f91eb0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49b2612895492166 server_cookie=1c18e805ddee8235 in_seq=0 out_seq=0 2024-04-21T23:19:05.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.658+0000 7f3f20ff9640 10 client.17808 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:19:05.671 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.659+0000 7f3f20ff9640 1 -- 172.21.15.44:0/1942638893 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555a89edddc0 con 0x555a89f8fa50 2024-04-21T23:19:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.659+0000 7f3f20ff9640 10 client.17808 handle_command_reply: tid=0 2024-04-21T23:19:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.659+0000 7f3f42246740 1 client.17808 shutdown 2024-04-21T23:19:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.659+0000 7f3f42246740 1 -- 172.21.15.44:0/1942638893 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 msgr2=0x555a89f7f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.659+0000 7f3f42246740 1 --2- 172.21.15.44:0/1942638893 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 0x555a89f7f350 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f3f340136e0 tx=0x7f3f34013ae0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.659+0000 7f3f42246740 1 -- 172.21.15.44:0/1942638893 shutdown_connections 2024-04-21T23:19:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.659+0000 7f3f42246740 1 --2- 172.21.15.44:0/1942638893 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x555a89f8fa50 0x555a89f91eb0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.659+0000 7f3f42246740 1 --2- 172.21.15.44:0/1942638893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f7f930 0x555a89f7ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.659+0000 7f3f42246740 1 --2- 172.21.15.44:0/1942638893 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f72670 0x555a89f7c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.672 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.659+0000 7f3f20ff9640 0 client.17808 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:19:05.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.659+0000 7f3f42246740 1 --2- 172.21.15.44:0/1942638893 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 0x555a89f7f350 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.659+0000 7f3f42246740 1 -- 172.21.15.44:0/1942638893 >> 172.21.15.44:0/1942638893 conn(0x555a89f6c750 msgr2=0x555a89f6e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:05.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.659+0000 7f3f42246740 1 -- 172.21.15.44:0/1942638893 shutdown_connections 2024-04-21T23:19:05.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.659+0000 7f3f42246740 1 -- 172.21.15.44:0/1942638893 wait complete. 2024-04-21T23:19:05.673 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.659+0000 7f3f42246740 20 client.17808 trim_cache size 0 max 16384 2024-04-21T23:19:05.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.672+0000 7f3f42246740 1 Processor -- start 2024-04-21T23:19:05.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.672+0000 7f3f42246740 1 -- start start 2024-04-21T23:19:05.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f42246740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f71d50 0x555a89f8b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.681 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f42246740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f72670 0x555a89f7ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f42246740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f7f930 0x555a89f7f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f42246740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3f381aaaa0 con 0x555a89f7f930 2024-04-21T23:19:05.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f42246740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3f381ad300 con 0x555a89f71d50 2024-04-21T23:19:05.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f42246740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3f381ab0f0 con 0x555a89f72670 2024-04-21T23:19:05.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f3dc03640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f7f930 0x555a89f7f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f3dc03640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f7f930 0x555a89f7f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38408/0 (socket says 172.21.15.44:38408) 2024-04-21T23:19:05.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f3dc03640 1 -- 172.21.15.44:0/2930521529 learned_addr learned my addr 172.21.15.44:0/2930521529 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:05.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f3d402640 1 --2- 172.21.15.44:0/2930521529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f71d50 0x555a89f8b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f3dc03640 1 -- 172.21.15.44:0/2930521529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f72670 msgr2=0x555a89f7ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:05.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f3cc01640 1 --2- 172.21.15.44:0/2930521529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f72670 0x555a89f7ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f3dc03640 1 --2- 172.21.15.44:0/2930521529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f72670 0x555a89f7ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f3dc03640 1 -- 172.21.15.44:0/2930521529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f71d50 msgr2=0x555a89f8b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:05.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f3dc03640 1 --2- 172.21.15.44:0/2930521529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f71d50 0x555a89f8b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f3dc03640 1 -- 172.21.15.44:0/2930521529 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a89cb47d0 con 0x555a89f7f930 2024-04-21T23:19:05.684 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f3d402640 1 --2- 172.21.15.44:0/2930521529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f71d50 0x555a89f8b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:19:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f3dc03640 1 --2- 172.21.15.44:0/2930521529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f7f930 0x555a89f7f350 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f3f30006010 tx=0x7f3f3000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29fdb5ff777908ba server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f22ffd640 1 -- 172.21.15.44:0/2930521529 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f300137c0 con 0x555a89f7f930 2024-04-21T23:19:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f22ffd640 1 -- 172.21.15.44:0/2930521529 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f30013960 con 0x555a89f7f930 2024-04-21T23:19:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.673+0000 7f3f22ffd640 1 -- 172.21.15.44:0/2930521529 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f30004040 con 0x555a89f7f930 2024-04-21T23:19:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.674+0000 7f3f42246740 1 -- 172.21.15.44:0/2930521529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f7f930 msgr2=0x555a89f7f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.674+0000 7f3f42246740 1 --2- 172.21.15.44:0/2930521529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f7f930 0x555a89f7f350 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f3f30006010 tx=0x7f3f3000bd30 comp rx=0 tx=0).stop 2024-04-21T23:19:05.685 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.674+0000 7f3f42246740 1 -- 172.21.15.44:0/2930521529 shutdown_connections 2024-04-21T23:19:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.674+0000 7f3f42246740 1 --2- 172.21.15.44:0/2930521529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f72670 0x555a89f7ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.674+0000 7f3f42246740 1 --2- 172.21.15.44:0/2930521529 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f7f930 0x555a89f7f350 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.674+0000 7f3f42246740 1 --2- 172.21.15.44:0/2930521529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f71d50 0x555a89f8b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.674+0000 7f3f42246740 1 -- 172.21.15.44:0/2930521529 >> 172.21.15.44:0/2930521529 conn(0x555a89f6c750 msgr2=0x555a89f6e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.674+0000 7f3f42246740 1 -- 172.21.15.44:0/2930521529 shutdown_connections 2024-04-21T23:19:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.674+0000 7f3f42246740 1 -- 172.21.15.44:0/2930521529 wait complete. 2024-04-21T23:19:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.674+0000 7f3f42246740 1 -- start start 2024-04-21T23:19:05.686 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.674+0000 7f3f42246740 1 Processor -- start 2024-04-21T23:19:05.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f42246740 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 0x555a89f87d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f42246740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f72670 0x555a89f78fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f42246740 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f7f930 0x555a89f794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f42246740 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3f381ab7c0 con 0x555a89f71d50 2024-04-21T23:19:05.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f42246740 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3f381acc30 con 0x555a89f7f930 2024-04-21T23:19:05.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f42246740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3f381abe90 con 0x555a89f72670 2024-04-21T23:19:05.687 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f42246740 10 client.0 _ll_register_callbacks cb 0x7f3f34005870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T23:19:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f3d402640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 0x555a89f87d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f3d402640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 0x555a89f87d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38420/0 (socket says 172.21.15.44:38420) 2024-04-21T23:19:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f3d402640 1 -- 172.21.15.44:0/3882366687 learned_addr learned my addr 172.21.15.44:0/3882366687 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f3cc01640 1 --2- 172.21.15.44:0/3882366687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f72670 0x555a89f78fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f3dc03640 1 --2- 172.21.15.44:0/3882366687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f7f930 0x555a89f794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f3d402640 1 -- 172.21.15.44:0/3882366687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f72670 msgr2=0x555a89f78fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f3d402640 1 --2- 172.21.15.44:0/3882366687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f72670 0x555a89f78fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.688 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f3d402640 1 -- 172.21.15.44:0/3882366687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f7f930 msgr2=0x555a89f794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T23:19:05.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f3d402640 1 --2- 172.21.15.44:0/3882366687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f7f930 0x555a89f794f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f3d402640 1 -- 172.21.15.44:0/3882366687 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a89f79b60 con 0x555a89f71d50 2024-04-21T23:19:05.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f3d402640 1 --2- 172.21.15.44:0/3882366687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 0x555a89f87d00 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f3f34009cf0 tx=0x7f3f340103d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d914b3d85807353e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:05.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f20ff9640 10 client.0 ms_handle_connect on v2:172.21.15.44:3300/0 2024-04-21T23:19:05.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f20ff9640 1 -- 172.21.15.44:0/3882366687 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34005450 con 0x555a89f71d50 2024-04-21T23:19:05.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f20ff9640 1 -- 172.21.15.44:0/3882366687 <== mon.0 v2:172.21.15.44:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7f3f34011810 con 0x555a89f71d50 2024-04-21T23:19:05.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.675+0000 7f3f42246740 1 -- 172.21.15.44:0/3882366687 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f30013960 con 0x555a89f71d50 2024-04-21T23:19:05.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.676+0000 7f3f20ff9640 1 -- 172.21.15.44:0/3882366687 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3f34020020 con 0x555a89f71d50 2024-04-21T23:19:05.689 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.676+0000 7f3f42246740 10 client.17814 fetch_fsmap learned FSMap version 31 2024-04-21T23:19:05.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.676+0000 7f3f42246740 1 -- 172.21.15.44:0/3882366687 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555a89ff5ec0 con 0x555a89f71d50 2024-04-21T23:19:05.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.676+0000 7f3f20ff9640 1 -- 172.21.15.44:0/3882366687 <== mon.0 v2:172.21.15.44:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f30013960 con 0x555a89f71d50 2024-04-21T23:19:05.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.676+0000 7f3f20ff9640 1 -- 172.21.15.44:0/3882366687 <== mon.0 v2:172.21.15.44:3300/0 5 ==== fsmap(e 31) ==== 8330+0+0 (secure 0 0 0) 0x7f3f34033030 con 0x555a89f71d50 2024-04-21T23:19:05.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.676+0000 7f3f42246740 10 client.17814 fetch_fsmap finished waiting for FSMap version 31 2024-04-21T23:19:05.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.676+0000 7f3f42246740 10 client.17814 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T23:19:05.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.676+0000 7f3f42246740 20 client.17814 populate_metadata read hostname 'smithi044' 2024-04-21T23:19:05.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.676+0000 7f3f42246740 1 --2- 172.21.15.44:0/3882366687 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x555a89ff6690 0x555a89ff8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:05.690 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.676+0000 7f3f42246740 4 client.17814 mds_command: new command op to 24530 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T23:19:05.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.676+0000 7f3f42246740 1 -- 172.21.15.44:0/3882366687 --> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555a89e5b6e0 con 0x555a89ff6690 2024-04-21T23:19:05.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.676+0000 7f3f3cc01640 1 --2- 172.21.15.44:0/3882366687 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x555a89ff6690 0x555a89ff8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:05.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.677+0000 7f3f20ff9640 10 client.17814 ms_handle_connect on v2:172.21.15.55:6838/2066817543 2024-04-21T23:19:05.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.677+0000 7f3f3cc01640 1 --2- 172.21.15.44:0/3882366687 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x555a89ff6690 0x555a89ff8ab0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8731ed65a0eb62d server_cookie=44b91ddf17cd0c8d in_seq=0 out_seq=0 2024-04-21T23:19:05.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.677+0000 7f3f20ff9640 1 -- 172.21.15.44:0/3882366687 <== mds.0 v2:172.21.15.55:6838/2066817543 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555a89e5b6e0 con 0x555a89ff6690 2024-04-21T23:19:05.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.677+0000 7f3f20ff9640 10 client.17814 handle_command_reply: tid=0 2024-04-21T23:19:05.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.677+0000 7f3f42246740 1 client.17814 shutdown 2024-04-21T23:19:05.691 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.677+0000 7f3f42246740 1 -- 172.21.15.44:0/3882366687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 msgr2=0x555a89f87d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.677+0000 7f3f42246740 1 --2- 172.21.15.44:0/3882366687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 0x555a89f87d00 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f3f34009cf0 tx=0x7f3f340103d0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.677+0000 7f3f42246740 1 -- 172.21.15.44:0/3882366687 shutdown_connections 2024-04-21T23:19:05.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.677+0000 7f3f42246740 1 --2- 172.21.15.44:0/3882366687 >> [v2:172.21.15.55:6838/2066817543,v1:172.21.15.55:6839/2066817543] conn(0x555a89ff6690 0x555a89ff8ab0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.677+0000 7f3f42246740 1 --2- 172.21.15.44:0/3882366687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a89f72670 0x555a89f78fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.677+0000 7f3f20ff9640 0 client.17814 ms_handle_reset on v2:172.21.15.55:6838/2066817543 2024-04-21T23:19:05.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.677+0000 7f3f42246740 1 --2- 172.21.15.44:0/3882366687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x555a89f7f930 0x555a89f794f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.677+0000 7f3f42246740 1 --2- 172.21.15.44:0/3882366687 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x555a89f71d50 0x555a89f87d00 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.677+0000 7f3f42246740 1 -- 172.21.15.44:0/3882366687 >> 172.21.15.44:0/3882366687 conn(0x555a89f6c750 msgr2=0x555a89f6db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:05.692 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.677+0000 7f3f42246740 1 -- 172.21.15.44:0/3882366687 shutdown_connections 2024-04-21T23:19:05.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.677+0000 7f3f42246740 1 -- 172.21.15.44:0/3882366687 wait complete. 2024-04-21T23:19:05.693 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.677+0000 7f3f42246740 20 client.17814 trim_cache size 0 max 16384 2024-04-21T23:19:05.693 INFO:teuthology.orchestra.run.smithi044.stdout:[] 2024-04-21T23:19:05.694 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.678+0000 7f3f3f666640 1 -- 172.21.15.44:0/2473684505 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3f240a2ea0 msgr2=0x7f3f240a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.678+0000 7f3f3f666640 1 --2- 172.21.15.44:0/2473684505 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3f240a2ea0 0x7f3f240a5360 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3f30010c40 tx=0x7f3f30015000 comp rx=0 tx=0).stop 2024-04-21T23:19:05.697 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.678+0000 7f3f3f666640 1 -- 172.21.15.44:0/2473684505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f38100fd0 msgr2=0x7f3f381984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:05.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.678+0000 7f3f3f666640 1 --2- 172.21.15.44:0/2473684505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f38100fd0 0x7f3f381984d0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f3f28009230 tx=0x7f3f2802cb30 comp rx=0 tx=0).stop 2024-04-21T23:19:05.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.678+0000 7f3f3f666640 1 -- 172.21.15.44:0/2473684505 shutdown_connections 2024-04-21T23:19:05.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.678+0000 7f3f3f666640 1 --2- 172.21.15.44:0/2473684505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f381066b0 0x7f3f3819c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.678+0000 7f3f3f666640 1 --2- 172.21.15.44:0/2473684505 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f3f240a2ea0 0x7f3f240a5360 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.678+0000 7f3f3f666640 1 --2- 172.21.15.44:0/2473684505 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3f38100fd0 0x7f3f381984d0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.678+0000 7f3f3f666640 1 --2- 172.21.15.44:0/2473684505 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f38100610 0x7f3f38197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:05.698 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.678+0000 7f3f3f666640 1 -- 172.21.15.44:0/2473684505 >> 172.21.15.44:0/2473684505 conn(0x7f3f380777f0 msgr2=0x7f3f380ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:05.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.678+0000 7f3f3f666640 1 -- 172.21.15.44:0/2473684505 shutdown_connections 2024-04-21T23:19:05.699 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:05.678+0000 7f3f3f666640 1 -- 172.21.15.44:0/2473684505 wait complete. 2024-04-21T23:19:05.699 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:05 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:19:05.713 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:05 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T23:19:06.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:06 smithi055 ceph-mon[31117]: pgmap v323: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 21 op/s 2024-04-21T23:19:06.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:06 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/693756140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:19:06.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:06 smithi044 ceph-mon[27934]: pgmap v323: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 21 op/s 2024-04-21T23:19:06.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:06 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/693756140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:19:06.699 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-04-21T23:19:06.700 INFO:tasks.fwd_scrub:done joining 2024-04-21T23:19:06.700 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-21T23:19:06.718 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:19:06.719 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:06 smithi069 ceph-mon[31816]: pgmap v323: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 21 op/s 2024-04-21T23:19:06.719 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:06 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/693756140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:19:06.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.824+0000 7ffa52ff7640 1 -- 172.21.15.44:0/1591955076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa4c100890 msgr2=0x7ffa4c100cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:06.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.824+0000 7ffa52ff7640 1 --2- 172.21.15.44:0/1591955076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa4c100890 0x7ffa4c100cf0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7ffa34009900 tx=0x7ffa3402bc30 comp rx=0 tx=0).stop 2024-04-21T23:19:06.824 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.824+0000 7ffa52ff7640 1 -- 172.21.15.44:0/1591955076 shutdown_connections 2024-04-21T23:19:06.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.824+0000 7ffa52ff7640 1 --2- 172.21.15.44:0/1591955076 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffa4c101230 0x7ffa4c10d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:06.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.824+0000 7ffa52ff7640 1 --2- 172.21.15.44:0/1591955076 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa4c100890 0x7ffa4c100cf0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:06.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.824+0000 7ffa52ff7640 1 --2- 172.21.15.44:0/1591955076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa4c1066c0 0x7ffa4c106aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:06.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.824+0000 7ffa52ff7640 1 -- 172.21.15.44:0/1591955076 >> 172.21.15.44:0/1591955076 conn(0x7ffa4c077430 msgr2=0x7ffa4c077840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:06.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.824+0000 7ffa52ff7640 1 -- 172.21.15.44:0/1591955076 shutdown_connections 2024-04-21T23:19:06.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.824+0000 7ffa52ff7640 1 -- 172.21.15.44:0/1591955076 wait complete. 2024-04-21T23:19:06.827 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.825+0000 7ffa52ff7640 1 Processor -- start 2024-04-21T23:19:06.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.825+0000 7ffa52ff7640 1 -- start start 2024-04-21T23:19:06.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa52ff7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa4c100890 0x7ffa4c196f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:06.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa52ff7640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffa4c101230 0x7ffa4c197470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:06.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa52ff7640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa4c1066c0 0x7ffa4c19b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:06.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa52ff7640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4c110a90 con 0x7ffa4c1066c0 2024-04-21T23:19:06.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa52ff7640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4c110910 con 0x7ffa4c101230 2024-04-21T23:19:06.828 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa52ff7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4c110c10 con 0x7ffa4c100890 2024-04-21T23:19:06.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa51594640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa4c1066c0 0x7ffa4c19b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:06.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa51594640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa4c1066c0 0x7ffa4c19b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38444/0 (socket says 172.21.15.44:38444) 2024-04-21T23:19:06.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa51594640 1 -- 172.21.15.44:0/3981826913 learned_addr learned my addr 172.21.15.44:0/3981826913 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:06.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa4bfff640 1 --2- 172.21.15.44:0/3981826913 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffa4c101230 0x7ffa4c197470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:06.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa50d93640 1 --2- 172.21.15.44:0/3981826913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa4c100890 0x7ffa4c196f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:06.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa51594640 1 -- 172.21.15.44:0/3981826913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa4c100890 msgr2=0x7ffa4c196f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:06.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa51594640 1 --2- 172.21.15.44:0/3981826913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa4c100890 0x7ffa4c196f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:06.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa51594640 1 -- 172.21.15.44:0/3981826913 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffa4c101230 msgr2=0x7ffa4c197470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:06.829 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa51594640 1 --2- 172.21.15.44:0/3981826913 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffa4c101230 0x7ffa4c197470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:06.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa51594640 1 -- 172.21.15.44:0/3981826913 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4c19bf20 con 0x7ffa4c1066c0 2024-04-21T23:19:06.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa51594640 1 --2- 172.21.15.44:0/3981826913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa4c1066c0 0x7ffa4c19b7f0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7ffa4400eb10 tx=0x7ffa4400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:06.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa49ffb640 1 -- 172.21.15.44:0/3981826913 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa44004b20 con 0x7ffa4c1066c0 2024-04-21T23:19:06.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa49ffb640 1 -- 172.21.15.44:0/3981826913 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa44004cc0 con 0x7ffa4c1066c0 2024-04-21T23:19:06.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa52ff7640 1 -- 172.21.15.44:0/3981826913 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa4c19c210 con 0x7ffa4c1066c0 2024-04-21T23:19:06.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.826+0000 7ffa49ffb640 1 -- 172.21.15.44:0/3981826913 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa44010760 con 0x7ffa4c1066c0 2024-04-21T23:19:06.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.827+0000 7ffa52ff7640 1 -- 172.21.15.44:0/3981826913 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa4c1a3aa0 con 0x7ffa4c1066c0 2024-04-21T23:19:06.830 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.828+0000 7ffa52ff7640 1 -- 172.21.15.44:0/3981826913 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa4c0685f0 con 0x7ffa4c1066c0 2024-04-21T23:19:06.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.829+0000 7ffa49ffb640 1 -- 172.21.15.44:0/3981826913 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ffa44005020 con 0x7ffa4c1066c0 2024-04-21T23:19:06.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.830+0000 7ffa49ffb640 1 --2- 172.21.15.44:0/3981826913 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ffa380a2d00 0x7ffa380a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:06.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.830+0000 7ffa50d93640 1 --2- 172.21.15.44:0/3981826913 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ffa380a2d00 0x7ffa380a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:06.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.830+0000 7ffa49ffb640 1 -- 172.21.15.44:0/3981826913 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7ffa440cfdf0 con 0x7ffa4c1066c0 2024-04-21T23:19:06.832 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.830+0000 7ffa50d93640 1 --2- 172.21.15.44:0/3981826913 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ffa380a2d00 0x7ffa380a51c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c009b50 tx=0x7ffa3c005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:06.834 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:06.835+0000 7ffa49ffb640 1 -- 172.21.15.44:0/3981826913 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ffa4409c620 con 0x7ffa4c1066c0 2024-04-21T23:19:07.021 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.022+0000 7ffa52ff7640 1 -- 172.21.15.44:0/3981826913 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffa4c108570 con 0x7ffa4c1066c0 2024-04-21T23:19:07.023 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.024+0000 7ffa49ffb640 1 -- 172.21.15.44:0/3981826913 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) ==== 74+0+25278 (secure 0 0 0) 0x7ffa4409bf20 con 0x7ffa4c1066c0 2024-04-21T23:19:07.024 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:19:07.024 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":108,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:18:23.205015+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":9,"to":11},{"from":5,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/3134653118":"2024-04-22T23:08:50.585326+0000","172.21.15.44:0/101322314":"2024-04-22T23:08:50.585326+0000","172.21.15.44:0/792524018":"2024-04-22T23:08:50.585326+0000","172.21.15.44:0/2303525779":"2024-04-22T23:08:50.585326+0000","172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/1417504446":"2024-04-22T23:08:50.585326+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/1417504446":"2024-04-22T23:08:50.585326+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:19:07.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.026+0000 7ffa52ff7640 1 -- 172.21.15.44:0/3981826913 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ffa380a2d00 msgr2=0x7ffa380a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:07.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.026+0000 7ffa52ff7640 1 --2- 172.21.15.44:0/3981826913 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ffa380a2d00 0x7ffa380a51c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c009b50 tx=0x7ffa3c005f90 comp rx=0 tx=0).stop 2024-04-21T23:19:07.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.026+0000 7ffa52ff7640 1 -- 172.21.15.44:0/3981826913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa4c1066c0 msgr2=0x7ffa4c19b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:07.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.026+0000 7ffa52ff7640 1 --2- 172.21.15.44:0/3981826913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa4c1066c0 0x7ffa4c19b7f0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7ffa4400eb10 tx=0x7ffa4400c2d0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.026+0000 7ffa52ff7640 1 -- 172.21.15.44:0/3981826913 shutdown_connections 2024-04-21T23:19:07.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.026+0000 7ffa52ff7640 1 --2- 172.21.15.44:0/3981826913 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ffa380a2d00 0x7ffa380a51c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.026+0000 7ffa52ff7640 1 --2- 172.21.15.44:0/3981826913 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ffa4c1066c0 0x7ffa4c19b7f0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.026+0000 7ffa52ff7640 1 --2- 172.21.15.44:0/3981826913 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ffa4c101230 0x7ffa4c197470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.026+0000 7ffa52ff7640 1 --2- 172.21.15.44:0/3981826913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa4c100890 0x7ffa4c196f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.026+0000 7ffa52ff7640 1 -- 172.21.15.44:0/3981826913 >> 172.21.15.44:0/3981826913 conn(0x7ffa4c077430 msgr2=0x7ffa4c0ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:07.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.027+0000 7ffa52ff7640 1 -- 172.21.15.44:0/3981826913 shutdown_connections 2024-04-21T23:19:07.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.027+0000 7ffa52ff7640 1 -- 172.21.15.44:0/3981826913 wait complete. 2024-04-21T23:19:07.037 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:01:47.507503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:06:23.993760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:06:24.954661+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:06:28.677618+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:19:07.038 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:19:07.151 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.148+0000 7fb0d58ec640 1 -- 172.21.15.44:0/1359529364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0d010bcf0 msgr2=0x7fb0d010e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:07.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.148+0000 7fb0d58ec640 1 --2- 172.21.15.44:0/1359529364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0d010bcf0 0x7fb0d010e190 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8009ca0 tx=0x7fb0c802c180 comp rx=0 tx=0).stop 2024-04-21T23:19:07.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.149+0000 7fb0d58ec640 1 -- 172.21.15.44:0/1359529364 shutdown_connections 2024-04-21T23:19:07.152 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.149+0000 7fb0d58ec640 1 --2- 172.21.15.44:0/1359529364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0d01032d0 0x7fb0d010b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.149+0000 7fb0d58ec640 1 --2- 172.21.15.44:0/1359529364 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0d010bcf0 0x7fb0d010e190 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.149+0000 7fb0d58ec640 1 --2- 172.21.15.44:0/1359529364 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0d01029b0 0x7fb0d0102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.149+0000 7fb0d58ec640 1 -- 172.21.15.44:0/1359529364 >> 172.21.15.44:0/1359529364 conn(0x7fb0d00777f0 msgr2=0x7fb0d0077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:07.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.149+0000 7fb0d58ec640 1 -- 172.21.15.44:0/1359529364 shutdown_connections 2024-04-21T23:19:07.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.149+0000 7fb0d58ec640 1 -- 172.21.15.44:0/1359529364 wait complete. 2024-04-21T23:19:07.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.149+0000 7fb0d58ec640 1 Processor -- start 2024-04-21T23:19:07.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.149+0000 7fb0d58ec640 1 -- start start 2024-04-21T23:19:07.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.150+0000 7fb0d58ec640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0d01029b0 0x7fb0d0197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:07.153 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.150+0000 7fb0d58ec640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0d01032d0 0x7fb0d01984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:07.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.150+0000 7fb0d58ec640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0d010bcf0 0x7fb0d019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:07.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.150+0000 7fb0d58ec640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d0111af0 con 0x7fb0d01029b0 2024-04-21T23:19:07.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.150+0000 7fb0d58ec640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d0111970 con 0x7fb0d01032d0 2024-04-21T23:19:07.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.150+0000 7fb0d58ec640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d0111c70 con 0x7fb0d010bcf0 2024-04-21T23:19:07.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.150+0000 7fb0ceffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0d01029b0 0x7fb0d0197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:07.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.150+0000 7fb0ceffd640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0d01029b0 0x7fb0d0197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38458/0 (socket says 172.21.15.44:38458) 2024-04-21T23:19:07.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.150+0000 7fb0ceffd640 1 -- 172.21.15.44:0/1160401589 learned_addr learned my addr 172.21.15.44:0/1160401589 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:07.154 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.150+0000 7fb0ce7fc640 1 --2- 172.21.15.44:0/1160401589 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0d01032d0 0x7fb0d01984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:07.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.150+0000 7fb0cf7fe640 1 --2- 172.21.15.44:0/1160401589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0d010bcf0 0x7fb0d019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:07.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.150+0000 7fb0ceffd640 1 -- 172.21.15.44:0/1160401589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0d010bcf0 msgr2=0x7fb0d019c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:07.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.150+0000 7fb0ceffd640 1 --2- 172.21.15.44:0/1160401589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0d010bcf0 0x7fb0d019c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.150+0000 7fb0ceffd640 1 -- 172.21.15.44:0/1160401589 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0d01032d0 msgr2=0x7fb0d01984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:07.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.150+0000 7fb0ceffd640 1 --2- 172.21.15.44:0/1160401589 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0d01032d0 0x7fb0d01984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.150+0000 7fb0ceffd640 1 -- 172.21.15.44:0/1160401589 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d019cf50 con 0x7fb0d01029b0 2024-04-21T23:19:07.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.150+0000 7fb0ce7fc640 1 --2- 172.21.15.44:0/1160401589 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0d01032d0 0x7fb0d01984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:07.155 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.150+0000 7fb0ceffd640 1 --2- 172.21.15.44:0/1160401589 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0d01029b0 0x7fb0d0197fb0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000e9f0 tx=0x7fb0c000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:07.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.150+0000 7fb0d48ea640 1 -- 172.21.15.44:0/1160401589 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c000cd80 con 0x7fb0d01029b0 2024-04-21T23:19:07.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.151+0000 7fb0d48ea640 1 -- 172.21.15.44:0/1160401589 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0c0010910 con 0x7fb0d01029b0 2024-04-21T23:19:07.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.151+0000 7fb0d58ec640 1 -- 172.21.15.44:0/1160401589 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d019d240 con 0x7fb0d01029b0 2024-04-21T23:19:07.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.151+0000 7fb0d48ea640 1 -- 172.21.15.44:0/1160401589 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0018990 con 0x7fb0d01029b0 2024-04-21T23:19:07.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.151+0000 7fb0d58ec640 1 -- 172.21.15.44:0/1160401589 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0d01a4ad0 con 0x7fb0d01029b0 2024-04-21T23:19:07.156 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.152+0000 7fb0d58ec640 1 -- 172.21.15.44:0/1160401589 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0d0068a40 con 0x7fb0d01029b0 2024-04-21T23:19:07.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.153+0000 7fb0d48ea640 1 -- 172.21.15.44:0/1160401589 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb0c0010ac0 con 0x7fb0d01029b0 2024-04-21T23:19:07.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.153+0000 7fb0d48ea640 1 --2- 172.21.15.44:0/1160401589 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb0b80a2d80 0x7fb0b80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:07.157 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.154+0000 7fb0ce7fc640 1 --2- 172.21.15.44:0/1160401589 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb0b80a2d80 0x7fb0b80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:07.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.154+0000 7fb0d48ea640 1 -- 172.21.15.44:0/1160401589 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7fb0c00cfab0 con 0x7fb0d01029b0 2024-04-21T23:19:07.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.154+0000 7fb0ce7fc640 1 --2- 172.21.15.44:0/1160401589 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb0b80a2d80 0x7fb0b80a5240 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40095f0 tx=0x7fb0c4005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:07.158 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.158+0000 7fb0d48ea640 1 -- 172.21.15.44:0/1160401589 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb0c009bc90 con 0x7fb0d01029b0 2024-04-21T23:19:07.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.366+0000 7fb0d58ec640 1 -- 172.21.15.44:0/1160401589 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb0d0106ed0 con 0x7fb0d01029b0 2024-04-21T23:19:07.366 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.366+0000 7fb0d48ea640 1 -- 172.21.15.44:0/1160401589 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v108) ==== 93+0+10 (secure 0 0 0) 0x7fb0c009ff90 con 0x7fb0d01029b0 2024-04-21T23:19:07.366 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-21T23:19:07.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.368+0000 7fb0d58ec640 1 -- 172.21.15.44:0/1160401589 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb0b80a2d80 msgr2=0x7fb0b80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:07.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.368+0000 7fb0d58ec640 1 --2- 172.21.15.44:0/1160401589 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb0b80a2d80 0x7fb0b80a5240 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40095f0 tx=0x7fb0c4005f90 comp rx=0 tx=0).stop 2024-04-21T23:19:07.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.368+0000 7fb0d58ec640 1 -- 172.21.15.44:0/1160401589 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0d01029b0 msgr2=0x7fb0d0197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:07.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.368+0000 7fb0d58ec640 1 --2- 172.21.15.44:0/1160401589 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0d01029b0 0x7fb0d0197fb0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000e9f0 tx=0x7fb0c000eeb0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.368 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.368+0000 7fb0d58ec640 1 -- 172.21.15.44:0/1160401589 shutdown_connections 2024-04-21T23:19:07.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.368+0000 7fb0d58ec640 1 --2- 172.21.15.44:0/1160401589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0d010bcf0 0x7fb0d019c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.368+0000 7fb0d58ec640 1 --2- 172.21.15.44:0/1160401589 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb0b80a2d80 0x7fb0b80a5240 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.368+0000 7fb0d58ec640 1 --2- 172.21.15.44:0/1160401589 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0d01032d0 0x7fb0d01984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.368+0000 7fb0d58ec640 1 --2- 172.21.15.44:0/1160401589 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb0d01029b0 0x7fb0d0197fb0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.368+0000 7fb0d58ec640 1 -- 172.21.15.44:0/1160401589 >> 172.21.15.44:0/1160401589 conn(0x7fb0d00777f0 msgr2=0x7fb0d00fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:07.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.368+0000 7fb0d58ec640 1 -- 172.21.15.44:0/1160401589 shutdown_connections 2024-04-21T23:19:07.369 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.368+0000 7fb0d58ec640 1 -- 172.21.15.44:0/1160401589 wait complete. 2024-04-21T23:19:07.379 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:19:07.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.485+0000 7f290806b640 1 -- 172.21.15.44:0/3862175159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2900103bb0 msgr2=0x7f290010e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:07.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.485+0000 7f290806b640 1 --2- 172.21.15.44:0/3862175159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2900103bb0 0x7f290010e6e0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f28f8009ca0 tx=0x7f28f802c1d0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.485+0000 7f290806b640 1 -- 172.21.15.44:0/3862175159 shutdown_connections 2024-04-21T23:19:07.487 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.485+0000 7f290806b640 1 --2- 172.21.15.44:0/3862175159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2900103210 0x7f2900103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.485+0000 7f290806b640 1 --2- 172.21.15.44:0/3862175159 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2900103bb0 0x7f290010e6e0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.485+0000 7f290806b640 1 --2- 172.21.15.44:0/3862175159 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2900101ad0 0x7f2900101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.485+0000 7f290806b640 1 -- 172.21.15.44:0/3862175159 >> 172.21.15.44:0/3862175159 conn(0x7f29000fd200 msgr2=0x7f29000ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:07.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.485+0000 7f290806b640 1 -- 172.21.15.44:0/3862175159 shutdown_connections 2024-04-21T23:19:07.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.485+0000 7f290806b640 1 -- 172.21.15.44:0/3862175159 wait complete. 2024-04-21T23:19:07.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.485+0000 7f290806b640 1 Processor -- start 2024-04-21T23:19:07.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.485+0000 7f290806b640 1 -- start start 2024-04-21T23:19:07.488 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.486+0000 7f290806b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2900101ad0 0x7f29001980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:07.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.486+0000 7f290806b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2900103210 0x7f2900198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:07.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.486+0000 7f290806b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2900103bb0 0x7f290019c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:07.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.486+0000 7f290806b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f2900111c20 con 0x7f2900103bb0 2024-04-21T23:19:07.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.486+0000 7f290806b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f2900111aa0 con 0x7f2900103210 2024-04-21T23:19:07.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.486+0000 7f290806b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2900111da0 con 0x7f2900101ad0 2024-04-21T23:19:07.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.486+0000 7f2906608640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2900103bb0 0x7f290019c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:07.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.486+0000 7f2906608640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2900103bb0 0x7f290019c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38486/0 (socket says 172.21.15.44:38486) 2024-04-21T23:19:07.489 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.486+0000 7f2906608640 1 -- 172.21.15.44:0/740894983 learned_addr learned my addr 172.21.15.44:0/740894983 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:07.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.486+0000 7f2905606640 1 --2- 172.21.15.44:0/740894983 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2900103210 0x7f2900198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:07.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.486+0000 7f2905e07640 1 --2- 172.21.15.44:0/740894983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2900101ad0 0x7f29001980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:07.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.486+0000 7f2906608640 1 -- 172.21.15.44:0/740894983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2900101ad0 msgr2=0x7f29001980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:07.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.486+0000 7f2906608640 1 --2- 172.21.15.44:0/740894983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2900101ad0 0x7f29001980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.486+0000 7f2906608640 1 -- 172.21.15.44:0/740894983 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2900103210 msgr2=0x7f2900198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:07.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.486+0000 7f2906608640 1 --2- 172.21.15.44:0/740894983 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2900103210 0x7f2900198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.486+0000 7f2906608640 1 -- 172.21.15.44:0/740894983 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f290019d060 con 0x7f2900103bb0 2024-04-21T23:19:07.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.486+0000 7f2905e07640 1 --2- 172.21.15.44:0/740894983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2900101ad0 0x7f29001980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:07.490 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.486+0000 7f2906608640 1 --2- 172.21.15.44:0/740894983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2900103bb0 0x7f290019c930 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f28f802c7e0 tx=0x7f28f8034830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:07.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.486+0000 7f28f6ffd640 1 -- 172.21.15.44:0/740894983 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f8039050 con 0x7f2900103bb0 2024-04-21T23:19:07.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.487+0000 7f28f6ffd640 1 -- 172.21.15.44:0/740894983 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28f8045070 con 0x7f2900103bb0 2024-04-21T23:19:07.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.487+0000 7f290806b640 1 -- 172.21.15.44:0/740894983 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f290019d2f0 con 0x7f2900103bb0 2024-04-21T23:19:07.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.487+0000 7f28f6ffd640 1 -- 172.21.15.44:0/740894983 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f8032bf0 con 0x7f2900103bb0 2024-04-21T23:19:07.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.487+0000 7f290806b640 1 -- 172.21.15.44:0/740894983 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29001a4be0 con 0x7f2900103bb0 2024-04-21T23:19:07.491 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.487+0000 7f290806b640 1 -- 172.21.15.44:0/740894983 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2900068a40 con 0x7f2900103bb0 2024-04-21T23:19:07.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.489+0000 7f28f6ffd640 1 -- 172.21.15.44:0/740894983 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f28f80046b0 con 0x7f2900103bb0 2024-04-21T23:19:07.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.489+0000 7f28f6ffd640 1 --2- 172.21.15.44:0/740894983 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f28ec0a2d00 0x7f28ec0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:07.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.490+0000 7f2905e07640 1 --2- 172.21.15.44:0/740894983 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f28ec0a2d00 0x7f28ec0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:07.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.490+0000 7f28f6ffd640 1 -- 172.21.15.44:0/740894983 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7f28f80f2730 con 0x7f2900103bb0 2024-04-21T23:19:07.492 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.490+0000 7f2905e07640 1 --2- 172.21.15.44:0/740894983 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f28ec0a2d00 0x7f28ec0a51c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f28fc005d00 tx=0x7f28fc005ed0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:07.493 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.494+0000 7f28f6ffd640 1 -- 172.21.15.44:0/740894983 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f28f80bf070 con 0x7f2900103bb0 2024-04-21T23:19:07.679 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.680+0000 7f290806b640 1 -- 172.21.15.44:0/740894983 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2900109d30 con 0x7f2900103bb0 2024-04-21T23:19:07.680 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.680+0000 7f28f6ffd640 1 -- 172.21.15.44:0/740894983 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v108) ==== 104+0+11 (secure 0 0 0) 0x7f28f80be970 con 0x7f2900103bb0 2024-04-21T23:19:07.680 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:19:07.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.682+0000 7f290806b640 1 -- 172.21.15.44:0/740894983 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f28ec0a2d00 msgr2=0x7f28ec0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:07.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.682+0000 7f290806b640 1 --2- 172.21.15.44:0/740894983 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f28ec0a2d00 0x7f28ec0a51c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f28fc005d00 tx=0x7f28fc005ed0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.682+0000 7f290806b640 1 -- 172.21.15.44:0/740894983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2900103bb0 msgr2=0x7f290019c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:07.682 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.682+0000 7f290806b640 1 --2- 172.21.15.44:0/740894983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2900103bb0 0x7f290019c930 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f28f802c7e0 tx=0x7f28f8034830 comp rx=0 tx=0).stop 2024-04-21T23:19:07.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.682+0000 7f290806b640 1 -- 172.21.15.44:0/740894983 shutdown_connections 2024-04-21T23:19:07.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.682+0000 7f290806b640 1 --2- 172.21.15.44:0/740894983 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f28ec0a2d00 0x7f28ec0a51c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.682+0000 7f290806b640 1 --2- 172.21.15.44:0/740894983 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f2900103bb0 0x7f290019c930 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.682+0000 7f290806b640 1 --2- 172.21.15.44:0/740894983 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2900103210 0x7f2900198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.682+0000 7f290806b640 1 --2- 172.21.15.44:0/740894983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2900101ad0 0x7f29001980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.682+0000 7f290806b640 1 -- 172.21.15.44:0/740894983 >> 172.21.15.44:0/740894983 conn(0x7f29000fd200 msgr2=0x7f29001075f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:07.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.682+0000 7f290806b640 1 -- 172.21.15.44:0/740894983 shutdown_connections 2024-04-21T23:19:07.683 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.682+0000 7f290806b640 1 -- 172.21.15.44:0/740894983 wait complete. 2024-04-21T23:19:07.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:07 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:07.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:07 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:07.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:07 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:07.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:07 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:07.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:07 smithi044 ceph-mon[27934]: pgmap v324: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 21 op/s 2024-04-21T23:19:07.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:07 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:07.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:07 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:07.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:07 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3981826913' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:19:07.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:07 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:07 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:07 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:07 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:07 smithi044 ceph-mon[27934]: Adjusting osd_memory_target on smithi069 to 176.1M 2024-04-21T23:19:07.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:07 smithi044 ceph-mon[27934]: Unable to set osd_memory_target on smithi069 to 184727244: error parsing value: Value '184727244' is below minimum 939524096 2024-04-21T23:19:07.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:07 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:07 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:07 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:07 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:07 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1160401589' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:19:07.693 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:19:07.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.798+0000 7ff9862ae640 1 -- 172.21.15.44:0/2289950775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff980103210 msgr2=0x7ff980103670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:07.800 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.798+0000 7ff9862ae640 1 --2- 172.21.15.44:0/2289950775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff980103210 0x7ff980103670 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7ff974009900 tx=0x7ff97402bd10 comp rx=0 tx=0).stop 2024-04-21T23:19:07.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.798+0000 7ff9862ae640 1 -- 172.21.15.44:0/2289950775 shutdown_connections 2024-04-21T23:19:07.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.798+0000 7ff9862ae640 1 --2- 172.21.15.44:0/2289950775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff980103bb0 0x7ff98010e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.798+0000 7ff9862ae640 1 --2- 172.21.15.44:0/2289950775 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff980103210 0x7ff980103670 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.798+0000 7ff9862ae640 1 --2- 172.21.15.44:0/2289950775 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff980101ad0 0x7ff980101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.798+0000 7ff9862ae640 1 -- 172.21.15.44:0/2289950775 >> 172.21.15.44:0/2289950775 conn(0x7ff9800fd200 msgr2=0x7ff9800ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:07.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.799+0000 7ff9862ae640 1 -- 172.21.15.44:0/2289950775 shutdown_connections 2024-04-21T23:19:07.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.799+0000 7ff9862ae640 1 -- 172.21.15.44:0/2289950775 wait complete. 2024-04-21T23:19:07.801 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.799+0000 7ff9862ae640 1 Processor -- start 2024-04-21T23:19:07.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.799+0000 7ff9862ae640 1 -- start start 2024-04-21T23:19:07.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.799+0000 7ff9862ae640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff980101ad0 0x7ff980198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:07.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.799+0000 7ff9862ae640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff980103210 0x7ff980198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:07.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.799+0000 7ff9862ae640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff980103bb0 0x7ff98019c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:07.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.799+0000 7ff9862ae640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7ff980111c20 con 0x7ff980103210 2024-04-21T23:19:07.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.799+0000 7ff9862ae640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7ff980111aa0 con 0x7ff980101ad0 2024-04-21T23:19:07.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.799+0000 7ff9862ae640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff980111da0 con 0x7ff980103bb0 2024-04-21T23:19:07.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.800+0000 7ff97f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff980103210 0x7ff980198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:07.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.800+0000 7ff97f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff980103210 0x7ff980198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38506/0 (socket says 172.21.15.44:38506) 2024-04-21T23:19:07.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.800+0000 7ff97f7fe640 1 -- 172.21.15.44:0/1887495391 learned_addr learned my addr 172.21.15.44:0/1887495391 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:07.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.800+0000 7ff98484b640 1 --2- 172.21.15.44:0/1887495391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff980103bb0 0x7ff98019c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:07.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.800+0000 7ff97ffff640 1 --2- 172.21.15.44:0/1887495391 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff980101ad0 0x7ff980198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:07.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.800+0000 7ff97f7fe640 1 -- 172.21.15.44:0/1887495391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff980103bb0 msgr2=0x7ff98019c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:07.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.800+0000 7ff97f7fe640 1 --2- 172.21.15.44:0/1887495391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff980103bb0 0x7ff98019c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.800+0000 7ff97f7fe640 1 -- 172.21.15.44:0/1887495391 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff980101ad0 msgr2=0x7ff980198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:07.803 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.800+0000 7ff97f7fe640 1 --2- 172.21.15.44:0/1887495391 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff980101ad0 0x7ff980198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:07.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.800+0000 7ff97f7fe640 1 -- 172.21.15.44:0/1887495391 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98019cfe0 con 0x7ff980103210 2024-04-21T23:19:07.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.800+0000 7ff97ffff640 1 --2- 172.21.15.44:0/1887495391 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff980101ad0 0x7ff980198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:19:07.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.800+0000 7ff97f7fe640 1 --2- 172.21.15.44:0/1887495391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff980103210 0x7ff980198580 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7ff974006010 tx=0x7ff9740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:07.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.800+0000 7ff97d7fa640 1 -- 172.21.15.44:0/1887495391 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff974042020 con 0x7ff980103210 2024-04-21T23:19:07.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.800+0000 7ff9862ae640 1 -- 172.21.15.44:0/1887495391 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff98019d270 con 0x7ff980103210 2024-04-21T23:19:07.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.800+0000 7ff97d7fa640 1 -- 172.21.15.44:0/1887495391 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9740054e0 con 0x7ff980103210 2024-04-21T23:19:07.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.800+0000 7ff97d7fa640 1 -- 172.21.15.44:0/1887495391 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff974003ad0 con 0x7ff980103210 2024-04-21T23:19:07.805 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.800+0000 7ff9862ae640 1 -- 172.21.15.44:0/1887495391 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9801a4b60 con 0x7ff980103210 2024-04-21T23:19:07.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.801+0000 7ff9862ae640 1 -- 172.21.15.44:0/1887495391 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff980068a40 con 0x7ff980103210 2024-04-21T23:19:07.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.802+0000 7ff97d7fa640 1 -- 172.21.15.44:0/1887495391 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff974045020 con 0x7ff980103210 2024-04-21T23:19:07.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.803+0000 7ff97d7fa640 1 --2- 172.21.15.44:0/1887495391 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff9680a2d00 0x7ff9680a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:07.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.803+0000 7ff97d7fa640 1 -- 172.21.15.44:0/1887495391 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9533+0+0 (secure 0 0 0) 0x7ff9740f1430 con 0x7ff980103210 2024-04-21T23:19:07.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.803+0000 7ff97ffff640 1 --2- 172.21.15.44:0/1887495391 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff9680a2d00 0x7ff9680a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:07.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.803+0000 7ff97ffff640 1 --2- 172.21.15.44:0/1887495391 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff9680a2d00 0x7ff9680a51c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff97000abe0 tx=0x7ff97000a5c0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:07.807 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.807+0000 7ff97d7fa640 1 -- 172.21.15.44:0/1887495391 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7ff97404c020 con 0x7ff980103210 2024-04-21T23:19:07.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:07 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:07.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:07 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:07.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:07 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:07.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:07 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:07.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:07 smithi069 ceph-mon[31816]: pgmap v324: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 21 op/s 2024-04-21T23:19:07.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:07 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:07.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:07 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:07.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:07 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3981826913' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:19:07.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:07 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.965 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:07 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:07 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:07 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:07 smithi069 ceph-mon[31816]: Adjusting osd_memory_target on smithi069 to 176.1M 2024-04-21T23:19:07.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:07 smithi069 ceph-mon[31816]: Unable to set osd_memory_target on smithi069 to 184727244: error parsing value: Value '184727244' is below minimum 939524096 2024-04-21T23:19:07.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:07 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:07 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:07 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:07 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.967 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:07 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1160401589' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:19:07.967 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:07 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:07.967 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:07 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:07.967 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:07 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:07.967 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:07 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:07.967 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:07 smithi055 ceph-mon[31117]: pgmap v324: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 21 op/s 2024-04-21T23:19:07.967 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:07 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:07.967 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:07 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:07.968 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:07 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3981826913' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:19:07.968 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:07 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.968 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:07 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.968 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:07 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.968 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:07 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.968 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:07 smithi055 ceph-mon[31117]: Adjusting osd_memory_target on smithi069 to 176.1M 2024-04-21T23:19:07.968 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:07 smithi055 ceph-mon[31117]: Unable to set osd_memory_target on smithi069 to 184727244: error parsing value: Value '184727244' is below minimum 939524096 2024-04-21T23:19:07.968 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:07 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.969 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:07 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.969 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:07 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.969 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:07 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:07.969 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:07 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1160401589' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:19:07.995 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.995+0000 7ff9862ae640 1 -- 172.21.15.44:0/1887495391 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff980109d30 con 0x7ff980103210 2024-04-21T23:19:07.996 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:07.996+0000 7ff97d7fa640 1 -- 172.21.15.44:0/1887495391 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v109) ==== 100+0+11 (secure 0 0 0) 0x7ff9740bdd50 con 0x7ff980103210 2024-04-21T23:19:07.996 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-21T23:19:08.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.001+0000 7ff9862ae640 1 -- 172.21.15.44:0/1887495391 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff9680a2d00 msgr2=0x7ff9680a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:08.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.001+0000 7ff9862ae640 1 --2- 172.21.15.44:0/1887495391 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff9680a2d00 0x7ff9680a51c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff97000abe0 tx=0x7ff97000a5c0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.001+0000 7ff9862ae640 1 -- 172.21.15.44:0/1887495391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff980103210 msgr2=0x7ff980198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:08.000 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.001+0000 7ff9862ae640 1 --2- 172.21.15.44:0/1887495391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff980103210 0x7ff980198580 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7ff974006010 tx=0x7ff9740042f0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.002+0000 7ff9862ae640 1 -- 172.21.15.44:0/1887495391 shutdown_connections 2024-04-21T23:19:08.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.002+0000 7ff9862ae640 1 --2- 172.21.15.44:0/1887495391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff980103bb0 0x7ff98019c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.001 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.002+0000 7ff9862ae640 1 --2- 172.21.15.44:0/1887495391 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7ff9680a2d00 0x7ff9680a51c0 secure :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff97000abe0 tx=0x7ff97000a5c0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.002+0000 7ff9862ae640 1 --2- 172.21.15.44:0/1887495391 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7ff980103210 0x7ff980198580 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.002+0000 7ff9862ae640 1 --2- 172.21.15.44:0/1887495391 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff980101ad0 0x7ff980198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.002 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.002+0000 7ff9862ae640 1 -- 172.21.15.44:0/1887495391 >> 172.21.15.44:0/1887495391 conn(0x7ff9800fd200 msgr2=0x7ff980107580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:08.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.004+0000 7ff9862ae640 1 -- 172.21.15.44:0/1887495391 shutdown_connections 2024-04-21T23:19:08.003 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.004+0000 7ff9862ae640 1 -- 172.21.15.44:0/1887495391 wait complete. 2024-04-21T23:19:08.022 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:19:08.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.210+0000 7fd6134aa640 1 -- 172.21.15.44:0/650072802 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd60c10fe90 msgr2=0x7fd60c10da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:08.210 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.210+0000 7fd6134aa640 1 --2- 172.21.15.44:0/650072802 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd60c10fe90 0x7fd60c10da50 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fd60001a910 tx=0x7fd60003cca0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.214+0000 7fd6134aa640 1 -- 172.21.15.44:0/650072802 shutdown_connections 2024-04-21T23:19:08.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.214+0000 7fd6134aa640 1 --2- 172.21.15.44:0/650072802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd60c110300 0x7fd60c10df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.214 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.214+0000 7fd6134aa640 1 --2- 172.21.15.44:0/650072802 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd60c11af20 0x7fd60c11d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.214+0000 7fd6134aa640 1 --2- 172.21.15.44:0/650072802 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd60c10fe90 0x7fd60c10da50 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.214+0000 7fd6134aa640 1 -- 172.21.15.44:0/650072802 >> 172.21.15.44:0/650072802 conn(0x7fd60c06c7e0 msgr2=0x7fd60c06cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:08.215 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.215+0000 7fd6134aa640 1 -- 172.21.15.44:0/650072802 shutdown_connections 2024-04-21T23:19:08.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.218+0000 7fd6134aa640 1 -- 172.21.15.44:0/650072802 wait complete. 2024-04-21T23:19:08.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.218+0000 7fd6134aa640 1 Processor -- start 2024-04-21T23:19:08.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.220+0000 7fd6134aa640 1 -- start start 2024-04-21T23:19:08.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.220+0000 7fd6134aa640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd60c10fe90 0x7fd60c11a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:08.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.220+0000 7fd6134aa640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd60c110300 0x7fd60c1166f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:08.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.220+0000 7fd6134aa640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd60c11af20 0x7fd60c116c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:08.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.220+0000 7fd6134aa640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c113120 con 0x7fd60c110300 2024-04-21T23:19:08.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.220+0000 7fd6134aa640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c112fa0 con 0x7fd60c10fe90 2024-04-21T23:19:08.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.220+0000 7fd6134aa640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c1132a0 con 0x7fd60c11af20 2024-04-21T23:19:08.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.220+0000 7fd610a45640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd60c110300 0x7fd60c1166f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:08.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.220+0000 7fd610a45640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd60c110300 0x7fd60c1166f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38518/0 (socket says 172.21.15.44:38518) 2024-04-21T23:19:08.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.220+0000 7fd610a45640 1 -- 172.21.15.44:0/2539547277 learned_addr learned my addr 172.21.15.44:0/2539547277 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:08.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.220+0000 7fd611a47640 1 --2- 172.21.15.44:0/2539547277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd60c11af20 0x7fd60c116c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:08.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.221+0000 7fd611246640 1 --2- 172.21.15.44:0/2539547277 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd60c10fe90 0x7fd60c11a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:08.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.221+0000 7fd610a45640 1 -- 172.21.15.44:0/2539547277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd60c11af20 msgr2=0x7fd60c116c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:08.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.221+0000 7fd610a45640 1 --2- 172.21.15.44:0/2539547277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd60c11af20 0x7fd60c116c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.221+0000 7fd610a45640 1 -- 172.21.15.44:0/2539547277 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd60c10fe90 msgr2=0x7fd60c11a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:08.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.221+0000 7fd610a45640 1 --2- 172.21.15.44:0/2539547277 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd60c10fe90 0x7fd60c11a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.221+0000 7fd610a45640 1 -- 172.21.15.44:0/2539547277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd60c117360 con 0x7fd60c110300 2024-04-21T23:19:08.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.221+0000 7fd611246640 1 --2- 172.21.15.44:0/2539547277 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd60c10fe90 0x7fd60c11a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:08.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.221+0000 7fd610a45640 1 --2- 172.21.15.44:0/2539547277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd60c110300 0x7fd60c1166f0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fd60400eab0 tx=0x7fd60400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:08.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.222+0000 7fd5fe7fc640 1 -- 172.21.15.44:0/2539547277 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60400ce40 con 0x7fd60c110300 2024-04-21T23:19:08.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.222+0000 7fd6134aa640 1 -- 172.21.15.44:0/2539547277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd60c1aacc0 con 0x7fd60c110300 2024-04-21T23:19:08.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.222+0000 7fd6134aa640 1 -- 172.21.15.44:0/2539547277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd60c1ab120 con 0x7fd60c110300 2024-04-21T23:19:08.226 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.222+0000 7fd5fe7fc640 1 -- 172.21.15.44:0/2539547277 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd604011950 con 0x7fd60c110300 2024-04-21T23:19:08.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.222+0000 7fd5fe7fc640 1 -- 172.21.15.44:0/2539547277 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd604010ae0 con 0x7fd60c110300 2024-04-21T23:19:08.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.226+0000 7fd5fe7fc640 1 -- 172.21.15.44:0/2539547277 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd604010c80 con 0x7fd60c110300 2024-04-21T23:19:08.227 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.226+0000 7fd6134aa640 1 -- 172.21.15.44:0/2539547277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5f0005290 con 0x7fd60c110300 2024-04-21T23:19:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.226+0000 7fd5fe7fc640 1 --2- 172.21.15.44:0/2539547277 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd5f80a2f00 0x7fd5f80a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.227+0000 7fd5fe7fc640 1 -- 172.21.15.44:0/2539547277 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9621+0+0 (secure 0 0 0) 0x7fd6040cfc60 con 0x7fd60c110300 2024-04-21T23:19:08.228 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.227+0000 7fd611246640 1 --2- 172.21.15.44:0/2539547277 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd5f80a2f00 0x7fd5f80a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:08.229 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.227+0000 7fd611246640 1 --2- 172.21.15.44:0/2539547277 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd5f80a2f00 0x7fd5f80a53c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd600000c00 tx=0x7fd60003f040 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:08.236 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.236+0000 7fd5fe7fc640 1 -- 172.21.15.44:0/2539547277 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fd604016030 con 0x7fd60c110300 2024-04-21T23:19:08.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.594+0000 7fd6134aa640 1 -- 172.21.15.44:0/2539547277 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd5f0005850 con 0x7fd60c110300 2024-04-21T23:19:08.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.594+0000 7fd5fe7fc640 1 -- 172.21.15.44:0/2539547277 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v109) ==== 103+0+11 (secure 0 0 0) 0x7fd60409c3d0 con 0x7fd60c110300 2024-04-21T23:19:08.594 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:19:08.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.598+0000 7fd6134aa640 1 -- 172.21.15.44:0/2539547277 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd5f80a2f00 msgr2=0x7fd5f80a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:08.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.598+0000 7fd6134aa640 1 --2- 172.21.15.44:0/2539547277 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd5f80a2f00 0x7fd5f80a53c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd600000c00 tx=0x7fd60003f040 comp rx=0 tx=0).stop 2024-04-21T23:19:08.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.598+0000 7fd6134aa640 1 -- 172.21.15.44:0/2539547277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd60c110300 msgr2=0x7fd60c1166f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:08.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.598+0000 7fd6134aa640 1 --2- 172.21.15.44:0/2539547277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd60c110300 0x7fd60c1166f0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fd60400eab0 tx=0x7fd60400ef70 comp rx=0 tx=0).stop 2024-04-21T23:19:08.599 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.599+0000 7fd6134aa640 1 -- 172.21.15.44:0/2539547277 shutdown_connections 2024-04-21T23:19:08.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.599+0000 7fd6134aa640 1 --2- 172.21.15.44:0/2539547277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd60c11af20 0x7fd60c116c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.599+0000 7fd6134aa640 1 --2- 172.21.15.44:0/2539547277 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fd5f80a2f00 0x7fd5f80a53c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.599+0000 7fd6134aa640 1 --2- 172.21.15.44:0/2539547277 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fd60c110300 0x7fd60c1166f0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.599+0000 7fd6134aa640 1 --2- 172.21.15.44:0/2539547277 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd60c10fe90 0x7fd60c11a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.600 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.599+0000 7fd6134aa640 1 -- 172.21.15.44:0/2539547277 >> 172.21.15.44:0/2539547277 conn(0x7fd60c06c7e0 msgr2=0x7fd60c11bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:08.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.599+0000 7fd6134aa640 1 -- 172.21.15.44:0/2539547277 shutdown_connections 2024-04-21T23:19:08.601 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.600+0000 7fd6134aa640 1 -- 172.21.15.44:0/2539547277 wait complete. 2024-04-21T23:19:08.619 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-21T23:19:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.799+0000 7fed31001640 1 -- 172.21.15.44:0/3325155043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed2c100f00 msgr2=0x7fed2c068a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:08.798 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.799+0000 7fed31001640 1 --2- 172.21.15.44:0/3325155043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed2c100f00 0x7fed2c068a40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fed1c009900 tx=0x7fed1c02bc60 comp rx=0 tx=0).stop 2024-04-21T23:19:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.802+0000 7fed31001640 1 -- 172.21.15.44:0/3325155043 shutdown_connections 2024-04-21T23:19:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.802+0000 7fed31001640 1 --2- 172.21.15.44:0/3325155043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed2c100540 0x7fed2c1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.802+0000 7fed31001640 1 --2- 172.21.15.44:0/3325155043 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed2c100f00 0x7fed2c068a40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.802+0000 7fed31001640 1 --2- 172.21.15.44:0/3325155043 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed2c106600 0x7fed2c1069e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.802 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.802+0000 7fed31001640 1 -- 172.21.15.44:0/3325155043 >> 172.21.15.44:0/3325155043 conn(0x7fed2c0fc030 msgr2=0x7fed2c0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:08.804 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.805+0000 7fed31001640 1 -- 172.21.15.44:0/3325155043 shutdown_connections 2024-04-21T23:19:08.806 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.807+0000 7fed31001640 1 -- 172.21.15.44:0/3325155043 wait complete. 2024-04-21T23:19:08.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.810+0000 7fed31001640 1 Processor -- start 2024-04-21T23:19:08.809 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.810+0000 7fed31001640 1 -- start start 2024-04-21T23:19:08.810 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.811+0000 7fed31001640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed2c100540 0x7fed2c072e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.811+0000 7fed31001640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed2c100f00 0x7fed2c073360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.811+0000 7fed31001640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed2c106600 0x7fed2c06c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.811+0000 7fed31001640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fed2c111ab0 con 0x7fed2c106600 2024-04-21T23:19:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.811+0000 7fed2b59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed2c106600 0x7fed2c06c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.811+0000 7fed2b59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed2c106600 0x7fed2c06c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:38534/0 (socket says 172.21.15.44:38534) 2024-04-21T23:19:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.811+0000 7fed2b59e640 1 -- 172.21.15.44:0/1175083084 learned_addr learned my addr 172.21.15.44:0/1175083084 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.811+0000 7fed31001640 1 -- 172.21.15.44:0/1175083084 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fed2c111930 con 0x7fed2c100540 2024-04-21T23:19:08.811 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.811+0000 7fed31001640 1 -- 172.21.15.44:0/1175083084 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fed2c111c30 con 0x7fed2c100f00 2024-04-21T23:19:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.811+0000 7fed2a59c640 1 --2- 172.21.15.44:0/1175083084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed2c100f00 0x7fed2c073360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.813+0000 7fed2b59e640 1 -- 172.21.15.44:0/1175083084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed2c100f00 msgr2=0x7fed2c073360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.813+0000 7fed2b59e640 1 --2- 172.21.15.44:0/1175083084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed2c100f00 0x7fed2c073360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.812 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.813+0000 7fed2b59e640 1 -- 172.21.15.44:0/1175083084 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed2c100540 msgr2=0x7fed2c072e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T23:19:08.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.813+0000 7fed2b59e640 1 --2- 172.21.15.44:0/1175083084 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed2c100540 0x7fed2c072e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:08.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.813+0000 7fed2b59e640 1 -- 172.21.15.44:0/1175083084 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed2c06c8d0 con 0x7fed2c106600 2024-04-21T23:19:08.813 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.813+0000 7fed2b59e640 1 --2- 172.21.15.44:0/1175083084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed2c106600 0x7fed2c06c010 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fed1c005ab0 tx=0x7fed1c02c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:08.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.817+0000 7fed13fff640 1 -- 172.21.15.44:0/1175083084 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1c041070 con 0x7fed2c106600 2024-04-21T23:19:08.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.817+0000 7fed13fff640 1 -- 172.21.15.44:0/1175083084 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed1c0044c0 con 0x7fed2c106600 2024-04-21T23:19:08.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.817+0000 7fed13fff640 1 -- 172.21.15.44:0/1175083084 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1c003d80 con 0x7fed2c106600 2024-04-21T23:19:08.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.818+0000 7fed31001640 1 -- 172.21.15.44:0/1175083084 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed2c06cb60 con 0x7fed2c106600 2024-04-21T23:19:08.817 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.818+0000 7fed31001640 1 -- 172.21.15.44:0/1175083084 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed2c070b20 con 0x7fed2c106600 2024-04-21T23:19:08.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.821+0000 7fed31001640 1 -- 172.21.15.44:0/1175083084 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed18005290 con 0x7fed2c106600 2024-04-21T23:19:08.821 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.822+0000 7fed13fff640 1 -- 172.21.15.44:0/1175083084 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fed1c044070 con 0x7fed2c106600 2024-04-21T23:19:08.822 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.823+0000 7fed13fff640 1 --2- 172.21.15.44:0/1175083084 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fed140a2e40 0x7fed140a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:08.823 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.823+0000 7fed13fff640 1 -- 172.21.15.44:0/1175083084 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9621+0+0 (secure 0 0 0) 0x7fed1c0f1150 con 0x7fed2c106600 2024-04-21T23:19:08.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.825+0000 7fed2ad9d640 1 --2- 172.21.15.44:0/1175083084 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fed140a2e40 0x7fed140a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:08.825 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.826+0000 7fed2ad9d640 1 --2- 172.21.15.44:0/1175083084 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fed140a2e40 0x7fed140a5300 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fed24005fd0 tx=0x7fed24009500 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:08.831 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:08.832+0000 7fed13fff640 1 -- 172.21.15.44:0/1175083084 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fed1c0bd9d0 con 0x7fed2c106600 2024-04-21T23:19:08.865 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:08 smithi069 ceph-mon[31816]: Adjusting osd_memory_target on smithi044 to 177.2M 2024-04-21T23:19:08.865 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:08 smithi069 ceph-mon[31816]: Unable to set osd_memory_target on smithi044 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-21T23:19:08.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:08 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:08.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:08 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:08.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:08 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:08.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:08 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:08.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:08 smithi069 ceph-mon[31816]: Adjusting osd_memory_target on smithi055 to 177.2M 2024-04-21T23:19:08.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:08 smithi069 ceph-mon[31816]: Unable to set osd_memory_target on smithi055 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-21T23:19:08.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:08 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:19:08.867 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:08 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:19:08.867 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:08 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:08.867 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:08 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/740894983' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:19:08.867 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:08 smithi069 ceph-mon[31816]: osdmap e109: 12 total, 12 up, 12 in 2024-04-21T23:19:08.867 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:08 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1887495391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:19:08.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:08 smithi044 ceph-mon[27934]: Adjusting osd_memory_target on smithi044 to 177.2M 2024-04-21T23:19:08.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:08 smithi044 ceph-mon[27934]: Unable to set osd_memory_target on smithi044 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-21T23:19:08.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:08 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:08.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:08 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:08.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:08 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:08.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:08 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:08.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:08 smithi044 ceph-mon[27934]: Adjusting osd_memory_target on smithi055 to 177.2M 2024-04-21T23:19:08.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:08 smithi044 ceph-mon[27934]: Unable to set osd_memory_target on smithi055 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-21T23:19:08.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:08 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:19:08.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:08 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:19:08.936 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:08 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:08.937 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:08 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/740894983' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:19:08.937 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:08 smithi044 ceph-mon[27934]: osdmap e109: 12 total, 12 up, 12 in 2024-04-21T23:19:08.937 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:08 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1887495391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:19:08.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:08 smithi055 ceph-mon[31117]: Adjusting osd_memory_target on smithi044 to 177.2M 2024-04-21T23:19:08.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:08 smithi055 ceph-mon[31117]: Unable to set osd_memory_target on smithi044 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-21T23:19:08.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:08 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:08.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:08 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:08.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:08 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:08.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:08 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T23:19:08.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:08 smithi055 ceph-mon[31117]: Adjusting osd_memory_target on smithi055 to 177.2M 2024-04-21T23:19:08.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:08 smithi055 ceph-mon[31117]: Unable to set osd_memory_target on smithi055 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-21T23:19:08.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:08 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T23:19:08.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:08 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T23:19:08.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:08 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' 2024-04-21T23:19:08.965 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:08 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/740894983' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:19:08.966 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:08 smithi055 ceph-mon[31117]: osdmap e109: 12 total, 12 up, 12 in 2024-04-21T23:19:08.966 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:08 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1887495391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:19:09.295 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:09.296+0000 7fed31001640 1 -- 172.21.15.44:0/1175083084 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fed18005850 con 0x7fed2c106600 2024-04-21T23:19:09.309 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:09.309+0000 7fed13fff640 1 -- 172.21.15.44:0/1175083084 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) ==== 81+0+740902 (secure 0 0 0) 0x7fed1c0bd2d0 con 0x7fed2c106600 2024-04-21T23:19:09.310 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:19:09.320 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:19:09.320 INFO:teuthology.orchestra.run.smithi044.stdout: "epoch": 25, 2024-04-21T23:19:09.320 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.321 INFO:teuthology.orchestra.run.smithi044.stdout: "active_gid": 15396, 2024-04-21T23:19:09.321 INFO:teuthology.orchestra.run.smithi044.stdout: "active_name": "x", 2024-04-21T23:19:09.321 INFO:teuthology.orchestra.run.smithi044.stdout: "active_addrs": { 2024-04-21T23:19:09.321 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2024-04-21T23:19:09.321 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.321 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2024-04-21T23:19:09.321 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:6800", 2024-04-21T23:19:09.321 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 3789150048 2024-04-21T23:19:09.322 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.322 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.322 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v1", 2024-04-21T23:19:09.322 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:6801", 2024-04-21T23:19:09.322 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 3789150048 2024-04-21T23:19:09.322 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.322 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-21T23:19:09.322 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.323 INFO:teuthology.orchestra.run.smithi044.stdout: "active_addr": "172.21.15.44:6801/3789150048", 2024-04-21T23:19:09.323 INFO:teuthology.orchestra.run.smithi044.stdout: "active_change": "2024-04-21T23:08:50.585684+0000", 2024-04-21T23:19:09.323 INFO:teuthology.orchestra.run.smithi044.stdout: "active_mgr_features": 4540701547738038271, 2024-04-21T23:19:09.323 INFO:teuthology.orchestra.run.smithi044.stdout: "available": true, 2024-04-21T23:19:09.323 INFO:teuthology.orchestra.run.smithi044.stdout: "standbys": [ 2024-04-21T23:19:09.323 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.323 INFO:teuthology.orchestra.run.smithi044.stdout: "gid": 25049, 2024-04-21T23:19:09.323 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "y", 2024-04-21T23:19:09.323 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr_features": 4540701547738038271, 2024-04-21T23:19:09.324 INFO:teuthology.orchestra.run.smithi044.stdout: "available_modules": [ 2024-04-21T23:19:09.324 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.324 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "alerts", 2024-04-21T23:19:09.324 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.324 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.324 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.324 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-04-21T23:19:09.324 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-04-21T23:19:09.325 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:09.325 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.325 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.325 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-04-21T23:19:09.325 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.325 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.325 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.325 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How frequently to reexamine health status", 2024-04-21T23:19:09.325 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.326 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.326 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.326 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.326 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.326 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.326 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.326 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.326 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.326 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.327 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.327 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.327 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.327 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.327 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.327 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.327 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.327 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.328 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.328 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.328 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.328 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.328 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.328 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.328 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.328 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.329 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.329 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.329 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.329 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.329 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.329 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.329 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.329 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.329 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.330 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.330 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.330 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.330 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.330 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.330 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.330 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.330 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.330 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.331 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.331 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.331 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.331 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.331 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.331 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.331 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.331 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.332 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.332 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.332 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.332 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.332 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.332 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.332 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.332 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.333 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.333 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.333 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.333 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.333 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.333 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.333 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.333 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.334 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.334 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.334 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.334 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.334 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.334 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.334 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_destination": { 2024-04-21T23:19:09.334 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_destination", 2024-04-21T23:19:09.335 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.335 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.335 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.335 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.335 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.335 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.335 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.335 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Email address to send alerts to", 2024-04-21T23:19:09.336 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.336 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.336 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.336 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.336 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_from_name": { 2024-04-21T23:19:09.336 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_from_name", 2024-04-21T23:19:09.336 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.336 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.336 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.337 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "Ceph", 2024-04-21T23:19:09.337 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.337 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.337 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.337 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Email From: name", 2024-04-21T23:19:09.337 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.337 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.337 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.337 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.338 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_host": { 2024-04-21T23:19:09.338 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_host", 2024-04-21T23:19:09.338 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.338 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.338 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.338 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.338 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.338 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.339 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.339 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP server", 2024-04-21T23:19:09.339 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.339 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.339 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.339 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.339 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_password": { 2024-04-21T23:19:09.339 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_password", 2024-04-21T23:19:09.339 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.340 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.340 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.340 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.340 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.340 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.340 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.340 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Password to authenticate with", 2024-04-21T23:19:09.340 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.341 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.341 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.341 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.341 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_port": { 2024-04-21T23:19:09.341 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_port", 2024-04-21T23:19:09.341 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.341 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.341 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.341 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "465", 2024-04-21T23:19:09.342 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.342 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.342 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.342 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP port", 2024-04-21T23:19:09.342 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.342 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.342 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.342 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.342 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_sender": { 2024-04-21T23:19:09.343 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_sender", 2024-04-21T23:19:09.343 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.343 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.343 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.343 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.343 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.343 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.343 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.344 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP envelope sender", 2024-04-21T23:19:09.344 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.344 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.344 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.344 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.344 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_ssl": { 2024-04-21T23:19:09.344 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_ssl", 2024-04-21T23:19:09.344 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.345 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.345 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.345 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.345 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.345 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.345 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.345 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-21T23:19:09.345 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.345 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.346 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.346 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.346 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_user": { 2024-04-21T23:19:09.346 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_user", 2024-04-21T23:19:09.346 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.346 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.346 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.346 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.347 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.347 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.347 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.347 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "User to authenticate as", 2024-04-21T23:19:09.347 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.347 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.347 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.347 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.347 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.348 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.348 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.348 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "balancer", 2024-04-21T23:19:09.348 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.348 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.348 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.348 INFO:teuthology.orchestra.run.smithi044.stdout: "active": { 2024-04-21T23:19:09.348 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "active", 2024-04-21T23:19:09.349 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.349 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.349 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.349 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.349 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.349 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.349 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.349 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically balance PGs across cluster", 2024-04-21T23:19:09.349 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.350 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.350 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.350 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.350 INFO:teuthology.orchestra.run.smithi044.stdout: "begin_time": { 2024-04-21T23:19:09.350 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "begin_time", 2024-04-21T23:19:09.350 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.350 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.350 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.350 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0000", 2024-04-21T23:19:09.351 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.351 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.351 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.351 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "beginning time of day to automatically balance", 2024-04-21T23:19:09.351 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T23:19:09.351 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.351 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.351 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.352 INFO:teuthology.orchestra.run.smithi044.stdout: "begin_weekday": { 2024-04-21T23:19:09.352 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "begin_weekday", 2024-04-21T23:19:09.352 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-04-21T23:19:09.352 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.352 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.352 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-04-21T23:19:09.352 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0", 2024-04-21T23:19:09.352 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "6", 2024-04-21T23:19:09.352 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.353 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-21T23:19:09.353 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T23:19:09.353 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.353 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.353 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.353 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_max_iterations": { 2024-04-21T23:19:09.353 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_max_iterations", 2024-04-21T23:19:09.353 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-04-21T23:19:09.354 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.354 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.354 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "25", 2024-04-21T23:19:09.354 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-04-21T23:19:09.354 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "250", 2024-04-21T23:19:09.354 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.354 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-21T23:19:09.354 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.355 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.355 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.355 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.355 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_metrics": { 2024-04-21T23:19:09.355 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_metrics", 2024-04-21T23:19:09.355 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.355 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.355 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.356 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "pgs,objects,bytes", 2024-04-21T23:19:09.356 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.356 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.356 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.356 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-21T23:19:09.356 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-21T23:19:09.356 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.356 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.356 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.357 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_step": { 2024-04-21T23:19:09.357 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_step", 2024-04-21T23:19:09.357 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:09.357 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.357 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.357 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0.5", 2024-04-21T23:19:09.357 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0.001", 2024-04-21T23:19:09.357 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "0.999", 2024-04-21T23:19:09.358 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.358 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "aggressiveness of optimization", 2024-04-21T23:19:09.358 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-21T23:19:09.358 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.358 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.358 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.358 INFO:teuthology.orchestra.run.smithi044.stdout: "end_time": { 2024-04-21T23:19:09.358 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "end_time", 2024-04-21T23:19:09.358 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.359 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.359 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.359 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "2359", 2024-04-21T23:19:09.359 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.359 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.359 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.359 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "ending time of day to automatically balance", 2024-04-21T23:19:09.359 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T23:19:09.360 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.360 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.360 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.360 INFO:teuthology.orchestra.run.smithi044.stdout: "end_weekday": { 2024-04-21T23:19:09.360 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "end_weekday", 2024-04-21T23:19:09.360 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-04-21T23:19:09.360 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.360 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.360 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-04-21T23:19:09.361 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0", 2024-04-21T23:19:09.361 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "6", 2024-04-21T23:19:09.361 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.361 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-21T23:19:09.361 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T23:19:09.361 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.361 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.361 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.361 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.362 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.362 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.362 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.362 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.362 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.362 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.362 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.362 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.363 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.363 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.363 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.363 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.363 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.363 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.363 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.363 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.364 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.364 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.364 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.364 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.364 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.364 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.364 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.364 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.364 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.365 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.365 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.365 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.365 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.365 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.365 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.365 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.365 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.366 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.366 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.366 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.366 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.366 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.366 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.366 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.367 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.367 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.367 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.367 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.367 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.367 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.367 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.367 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.367 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.368 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.368 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.368 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.368 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.368 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.368 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.368 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.368 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.369 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.369 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.369 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.369 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.369 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.369 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.369 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.369 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.369 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.370 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.370 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.370 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.370 INFO:teuthology.orchestra.run.smithi044.stdout: "min_score": { 2024-04-21T23:19:09.370 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "min_score", 2024-04-21T23:19:09.370 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:09.370 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.370 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.371 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-04-21T23:19:09.371 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.371 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.371 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.371 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-21T23:19:09.371 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.371 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.371 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.372 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.372 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": { 2024-04-21T23:19:09.372 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mode", 2024-04-21T23:19:09.372 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.372 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.372 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.372 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "upmap", 2024-04-21T23:19:09.372 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.372 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.373 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.373 INFO:teuthology.orchestra.run.smithi044.stdout: "crush-compat", 2024-04-21T23:19:09.373 INFO:teuthology.orchestra.run.smithi044.stdout: "none", 2024-04-21T23:19:09.373 INFO:teuthology.orchestra.run.smithi044.stdout: "read", 2024-04-21T23:19:09.373 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap", 2024-04-21T23:19:09.373 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap-read" 2024-04-21T23:19:09.373 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.373 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Balancer mode", 2024-04-21T23:19:09.373 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.374 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.374 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.374 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.374 INFO:teuthology.orchestra.run.smithi044.stdout: "pool_ids": { 2024-04-21T23:19:09.374 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pool_ids", 2024-04-21T23:19:09.374 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.374 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.374 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.375 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.375 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.375 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.375 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.375 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-21T23:19:09.375 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.375 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.375 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.377 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.377 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-04-21T23:19:09.377 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-04-21T23:19:09.378 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:09.378 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.378 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.378 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-04-21T23:19:09.378 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.378 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.378 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.378 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-21T23:19:09.379 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.379 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.379 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.379 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.379 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap_max_deviation": { 2024-04-21T23:19:09.379 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "upmap_max_deviation", 2024-04-21T23:19:09.379 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.379 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.380 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.380 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-04-21T23:19:09.380 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-04-21T23:19:09.380 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.380 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.380 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-21T23:19:09.380 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-21T23:19:09.380 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.380 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.381 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.381 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap_max_optimizations": { 2024-04-21T23:19:09.381 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "upmap_max_optimizations", 2024-04-21T23:19:09.381 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-04-21T23:19:09.381 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.381 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.381 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:09.381 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.382 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.382 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.382 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-21T23:19:09.382 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.382 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.382 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.382 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.382 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.382 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.383 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.383 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cephadm", 2024-04-21T23:19:09.383 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.383 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.383 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.383 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_down_multiplier": { 2024-04-21T23:19:09.383 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_down_multiplier", 2024-04-21T23:19:09.383 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:09.384 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.384 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.384 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3.0", 2024-04-21T23:19:09.384 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.384 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.384 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.384 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-21T23:19:09.384 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.384 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.385 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.385 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.385 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_refresh_rate": { 2024-04-21T23:19:09.385 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_refresh_rate", 2024-04-21T23:19:09.385 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:09.385 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.385 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.385 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "20", 2024-04-21T23:19:09.385 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.386 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.386 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.386 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-21T23:19:09.386 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.386 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.386 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.386 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.386 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_starting_port": { 2024-04-21T23:19:09.387 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_starting_port", 2024-04-21T23:19:09.387 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.387 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.387 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.387 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "4721", 2024-04-21T23:19:09.387 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.388 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.388 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.388 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-21T23:19:09.388 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.388 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.389 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.389 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.389 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_ptrace": { 2024-04-21T23:19:09.389 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_ptrace", 2024-04-21T23:19:09.389 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.389 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.389 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.389 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.389 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.390 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.390 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.390 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-21T23:19:09.390 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-21T23:19:09.390 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.390 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.390 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.390 INFO:teuthology.orchestra.run.smithi044.stdout: "autotune_interval": { 2024-04-21T23:19:09.391 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "autotune_interval", 2024-04-21T23:19:09.391 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:09.391 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.391 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.391 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-04-21T23:19:09.391 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.391 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.392 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.392 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-21T23:19:09.392 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.392 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.392 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.392 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.392 INFO:teuthology.orchestra.run.smithi044.stdout: "autotune_memory_target_ratio": { 2024-04-21T23:19:09.392 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "autotune_memory_target_ratio", 2024-04-21T23:19:09.393 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:09.393 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.393 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.393 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0.7", 2024-04-21T23:19:09.393 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.393 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.393 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.393 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-21T23:19:09.394 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.394 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.394 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.394 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.394 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm_log_destination": { 2024-04-21T23:19:09.394 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cephadm_log_destination", 2024-04-21T23:19:09.394 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.394 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.395 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.395 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.395 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.395 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.395 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.395 INFO:teuthology.orchestra.run.smithi044.stdout: "file", 2024-04-21T23:19:09.395 INFO:teuthology.orchestra.run.smithi044.stdout: "file,syslog", 2024-04-21T23:19:09.395 INFO:teuthology.orchestra.run.smithi044.stdout: "syslog" 2024-04-21T23:19:09.396 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.396 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-21T23:19:09.396 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.396 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.396 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.396 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.396 INFO:teuthology.orchestra.run.smithi044.stdout: "cgroups_split": { 2024-04-21T23:19:09.396 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cgroups_split", 2024-04-21T23:19:09.396 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.397 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.397 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.397 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.397 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.397 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.397 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.397 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-21T23:19:09.397 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.398 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.398 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.398 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.398 INFO:teuthology.orchestra.run.smithi044.stdout: "config_checks_enabled": { 2024-04-21T23:19:09.398 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_checks_enabled", 2024-04-21T23:19:09.398 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.398 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.398 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.399 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.399 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.399 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.399 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.399 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-21T23:19:09.399 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.399 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.399 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.400 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.400 INFO:teuthology.orchestra.run.smithi044.stdout: "config_dashboard": { 2024-04-21T23:19:09.400 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_dashboard", 2024-04-21T23:19:09.400 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.400 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.400 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.400 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.400 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.401 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.401 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.401 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-21T23:19:09.401 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.401 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.401 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.401 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.401 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_alertmanager": { 2024-04-21T23:19:09.402 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_alertmanager", 2024-04-21T23:19:09.402 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.402 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.402 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.402 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-21T23:19:09.402 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.402 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.402 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.403 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-04-21T23:19:09.403 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.403 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.403 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.403 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.403 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_base": { 2024-04-21T23:19:09.403 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_base", 2024-04-21T23:19:09.403 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.403 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.404 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.404 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-21T23:19:09.404 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.404 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.404 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.404 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Container image name, without the tag", 2024-04-21T23:19:09.404 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.404 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.405 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.405 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.405 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_elasticsearch": { 2024-04-21T23:19:09.405 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_elasticsearch", 2024-04-21T23:19:09.405 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.405 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.405 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.405 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-21T23:19:09.405 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.406 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.406 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.406 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "elasticsearch container image", 2024-04-21T23:19:09.406 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.406 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.406 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.406 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.406 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_grafana": { 2024-04-21T23:19:09.407 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_grafana", 2024-04-21T23:19:09.407 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.407 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.407 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.407 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-21T23:19:09.407 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.407 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.407 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.408 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-04-21T23:19:09.408 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.408 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.408 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.408 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.408 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_haproxy": { 2024-04-21T23:19:09.408 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_haproxy", 2024-04-21T23:19:09.408 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.408 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.409 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.409 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-21T23:19:09.409 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.409 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.409 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.409 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "HAproxy container image", 2024-04-21T23:19:09.409 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.409 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.410 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.410 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.410 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_agent": { 2024-04-21T23:19:09.410 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_agent", 2024-04-21T23:19:09.410 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.410 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.410 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.410 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-21T23:19:09.411 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.411 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.411 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.411 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger agent container image", 2024-04-21T23:19:09.411 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.411 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.411 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.411 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.412 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_collector": { 2024-04-21T23:19:09.412 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_collector", 2024-04-21T23:19:09.412 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.412 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.412 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.412 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-21T23:19:09.412 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.412 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.412 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.413 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger collector container image", 2024-04-21T23:19:09.413 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.413 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.413 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.413 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.413 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_query": { 2024-04-21T23:19:09.413 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_query", 2024-04-21T23:19:09.413 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.414 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.414 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.414 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-21T23:19:09.414 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.414 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.414 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.414 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger query container image", 2024-04-21T23:19:09.414 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.414 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.415 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.415 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.415 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_keepalived": { 2024-04-21T23:19:09.415 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_keepalived", 2024-04-21T23:19:09.415 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.415 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.415 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.415 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-21T23:19:09.416 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.416 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.416 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.416 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Keepalived container image", 2024-04-21T23:19:09.416 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.416 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.416 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.416 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.416 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_loki": { 2024-04-21T23:19:09.417 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_loki", 2024-04-21T23:19:09.417 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.417 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.417 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.417 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-21T23:19:09.417 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.417 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.417 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.418 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Loki container image", 2024-04-21T23:19:09.418 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.418 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.418 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.418 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.418 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_node_exporter": { 2024-04-21T23:19:09.418 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_node_exporter", 2024-04-21T23:19:09.418 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.418 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.419 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.419 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-21T23:19:09.419 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.419 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.419 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.419 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-04-21T23:19:09.419 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.419 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.419 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.420 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.420 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_nvmeof": { 2024-04-21T23:19:09.420 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_nvmeof", 2024-04-21T23:19:09.420 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.420 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.420 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.420 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-21T23:19:09.420 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.421 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.421 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.421 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Nvme-of container image", 2024-04-21T23:19:09.421 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.421 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.421 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.421 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.421 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_prometheus": { 2024-04-21T23:19:09.422 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_prometheus", 2024-04-21T23:19:09.422 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.422 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.422 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.422 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-21T23:19:09.422 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.422 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.422 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.423 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-04-21T23:19:09.423 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.423 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.423 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.423 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.423 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_promtail": { 2024-04-21T23:19:09.423 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_promtail", 2024-04-21T23:19:09.423 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.424 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.424 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.424 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-21T23:19:09.424 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.424 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.424 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.424 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Promtail container image", 2024-04-21T23:19:09.424 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.424 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.425 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.425 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.425 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_samba": { 2024-04-21T23:19:09.425 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_samba", 2024-04-21T23:19:09.425 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.425 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.425 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.425 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-21T23:19:09.426 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.426 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.426 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.426 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Samba/SMB container image", 2024-04-21T23:19:09.426 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.426 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.426 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.426 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.427 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_snmp_gateway": { 2024-04-21T23:19:09.427 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_snmp_gateway", 2024-04-21T23:19:09.427 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.427 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.427 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.427 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-21T23:19:09.427 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.427 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.428 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.428 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SNMP Gateway container image", 2024-04-21T23:19:09.428 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.428 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.428 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.428 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.428 INFO:teuthology.orchestra.run.smithi044.stdout: "container_init": { 2024-04-21T23:19:09.428 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_init", 2024-04-21T23:19:09.428 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.429 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.429 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.429 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.429 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.429 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.429 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.429 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Run podman/docker with `--init`", 2024-04-21T23:19:09.430 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.430 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.430 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.430 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.430 INFO:teuthology.orchestra.run.smithi044.stdout: "daemon_cache_timeout": { 2024-04-21T23:19:09.430 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "daemon_cache_timeout", 2024-04-21T23:19:09.430 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:09.430 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.430 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.431 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-04-21T23:19:09.431 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.431 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.431 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.431 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-21T23:19:09.431 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.431 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.431 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.431 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.432 INFO:teuthology.orchestra.run.smithi044.stdout: "default_cephadm_command_timeout": { 2024-04-21T23:19:09.432 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default_cephadm_command_timeout", 2024-04-21T23:19:09.432 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.432 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.432 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.432 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "900", 2024-04-21T23:19:09.432 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.432 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.433 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.433 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-21T23:19:09.433 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.433 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.433 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.433 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.433 INFO:teuthology.orchestra.run.smithi044.stdout: "default_registry": { 2024-04-21T23:19:09.433 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default_registry", 2024-04-21T23:19:09.433 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.434 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.434 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.434 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io", 2024-04-21T23:19:09.434 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.434 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.434 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.434 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-21T23:19:09.434 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.435 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.435 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.435 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.435 INFO:teuthology.orchestra.run.smithi044.stdout: "device_cache_timeout": { 2024-04-21T23:19:09.435 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_cache_timeout", 2024-04-21T23:19:09.435 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:09.435 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.435 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.436 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1800", 2024-04-21T23:19:09.436 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.436 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.436 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.436 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache device inventory", 2024-04-21T23:19:09.436 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.436 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.436 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.437 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.437 INFO:teuthology.orchestra.run.smithi044.stdout: "device_enhanced_scan": { 2024-04-21T23:19:09.437 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_enhanced_scan", 2024-04-21T23:19:09.437 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.437 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.437 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.437 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.437 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.437 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.438 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.438 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-21T23:19:09.438 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.438 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.438 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.438 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.438 INFO:teuthology.orchestra.run.smithi044.stdout: "facts_cache_timeout": { 2024-04-21T23:19:09.438 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "facts_cache_timeout", 2024-04-21T23:19:09.439 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:09.439 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.439 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.439 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-04-21T23:19:09.439 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.439 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.439 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.439 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache host facts data", 2024-04-21T23:19:09.439 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.440 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.440 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.440 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.440 INFO:teuthology.orchestra.run.smithi044.stdout: "grafana_dashboards_path": { 2024-04-21T23:19:09.440 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "grafana_dashboards_path", 2024-04-21T23:19:09.440 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.440 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.441 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.441 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-21T23:19:09.441 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.441 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.441 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.441 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-21T23:19:09.442 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.442 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.442 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.442 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.442 INFO:teuthology.orchestra.run.smithi044.stdout: "host_check_interval": { 2024-04-21T23:19:09.442 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "host_check_interval", 2024-04-21T23:19:09.443 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:09.443 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.443 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.443 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-04-21T23:19:09.443 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.443 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.443 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.444 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to perform a host check", 2024-04-21T23:19:09.444 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.444 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.444 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.444 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.444 INFO:teuthology.orchestra.run.smithi044.stdout: "hw_monitoring": { 2024-04-21T23:19:09.444 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "hw_monitoring", 2024-04-21T23:19:09.445 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.445 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.445 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.445 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.445 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.445 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.445 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.446 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-21T23:19:09.446 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.446 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.446 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.446 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.446 INFO:teuthology.orchestra.run.smithi044.stdout: "inventory_list_all": { 2024-04-21T23:19:09.447 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "inventory_list_all", 2024-04-21T23:19:09.447 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.447 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.447 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.447 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.447 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.447 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.448 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.448 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-21T23:19:09.448 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.448 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.448 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.448 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.448 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.449 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.449 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.449 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.449 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.449 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.449 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.449 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.450 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.450 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.450 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.450 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.450 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.450 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.450 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.451 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.451 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.451 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.451 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.451 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.451 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.451 INFO:teuthology.orchestra.run.smithi044.stdout: "log_refresh_metadata": { 2024-04-21T23:19:09.452 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_refresh_metadata", 2024-04-21T23:19:09.452 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.452 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.452 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.452 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.453 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.453 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.453 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.453 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-21T23:19:09.453 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.453 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.453 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.454 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.454 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.454 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.454 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.454 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.454 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.454 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.455 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.455 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.455 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.455 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-21T23:19:09.455 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.455 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.455 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.456 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.456 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.456 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.456 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.456 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.456 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.457 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.457 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.457 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.457 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.457 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.457 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.457 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.458 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.458 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.458 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.458 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.458 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.458 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.459 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.459 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.459 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.459 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.459 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.459 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.459 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.459 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.460 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.460 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.460 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.460 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.460 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.460 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.460 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.461 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.461 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.461 INFO:teuthology.orchestra.run.smithi044.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-21T23:19:09.461 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-21T23:19:09.461 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.461 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.462 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.462 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.462 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.462 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.462 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.462 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-21T23:19:09.462 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.463 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.463 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.463 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.463 INFO:teuthology.orchestra.run.smithi044.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-21T23:19:09.463 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-21T23:19:09.463 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.463 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.464 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.464 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "*", 2024-04-21T23:19:09.464 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.464 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.464 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.464 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-21T23:19:09.465 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.465 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.465 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.465 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.465 INFO:teuthology.orchestra.run.smithi044.stdout: "max_count_per_host": { 2024-04-21T23:19:09.465 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_count_per_host", 2024-04-21T23:19:09.465 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.466 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.466 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.466 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:09.466 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.466 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.466 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.467 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "max number of daemons per service per host", 2024-04-21T23:19:09.467 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.467 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.467 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.467 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.467 INFO:teuthology.orchestra.run.smithi044.stdout: "max_osd_draining_count": { 2024-04-21T23:19:09.468 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_osd_draining_count", 2024-04-21T23:19:09.468 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.468 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.468 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.468 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:09.468 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.468 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.469 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.469 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-21T23:19:09.469 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.469 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.469 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.469 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.469 INFO:teuthology.orchestra.run.smithi044.stdout: "migration_current": { 2024-04-21T23:19:09.470 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "migration_current", 2024-04-21T23:19:09.470 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.470 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.470 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.470 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.470 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.470 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.471 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.471 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "internal - do not modify", 2024-04-21T23:19:09.471 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.471 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.471 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.471 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.472 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": { 2024-04-21T23:19:09.472 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mode", 2024-04-21T23:19:09.472 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.472 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.472 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.472 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "root", 2024-04-21T23:19:09.472 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.473 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.473 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.473 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm-package", 2024-04-21T23:19:09.473 INFO:teuthology.orchestra.run.smithi044.stdout: "root" 2024-04-21T23:19:09.473 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.473 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "mode for remote execution of cephadm", 2024-04-21T23:19:09.473 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.473 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.474 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.474 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.474 INFO:teuthology.orchestra.run.smithi044.stdout: "oob_default_addr": { 2024-04-21T23:19:09.474 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "oob_default_addr", 2024-04-21T23:19:09.474 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.474 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.474 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.474 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "169.254.1.1", 2024-04-21T23:19:09.475 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.475 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.475 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.475 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-21T23:19:09.475 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.475 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.476 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.476 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.476 INFO:teuthology.orchestra.run.smithi044.stdout: "prometheus_alerts_path": { 2024-04-21T23:19:09.476 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus_alerts_path", 2024-04-21T23:19:09.476 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.476 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.477 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.477 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-21T23:19:09.477 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.477 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.477 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.478 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-21T23:19:09.478 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.478 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.478 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.478 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.478 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_insecure": { 2024-04-21T23:19:09.479 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_insecure", 2024-04-21T23:19:09.479 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.479 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.479 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.479 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.479 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.479 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.480 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.480 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-21T23:19:09.480 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.480 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.480 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.480 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.480 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_password": { 2024-04-21T23:19:09.480 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_password", 2024-04-21T23:19:09.481 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.481 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.481 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.481 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.481 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.481 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.481 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.481 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-21T23:19:09.482 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.482 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.482 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.482 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.482 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_url": { 2024-04-21T23:19:09.482 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_url", 2024-04-21T23:19:09.482 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.482 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.483 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.483 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.483 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.483 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.483 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.483 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-21T23:19:09.483 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.483 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.484 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.484 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.484 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_username": { 2024-04-21T23:19:09.484 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_username", 2024-04-21T23:19:09.484 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.484 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.484 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.484 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.485 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.485 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.485 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.485 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-21T23:19:09.485 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.485 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.485 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.485 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.486 INFO:teuthology.orchestra.run.smithi044.stdout: "secure_monitoring_stack": { 2024-04-21T23:19:09.486 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "secure_monitoring_stack", 2024-04-21T23:19:09.486 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.486 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.486 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.486 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.486 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.486 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.487 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.487 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-21T23:19:09.487 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.487 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.487 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.487 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.487 INFO:teuthology.orchestra.run.smithi044.stdout: "service_discovery_port": { 2024-04-21T23:19:09.487 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "service_discovery_port", 2024-04-21T23:19:09.488 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.488 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.488 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.488 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8765", 2024-04-21T23:19:09.488 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.488 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.488 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.488 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "cephadm service discovery port", 2024-04-21T23:19:09.489 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.489 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.489 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.489 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.489 INFO:teuthology.orchestra.run.smithi044.stdout: "ssh_config_file": { 2024-04-21T23:19:09.489 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssh_config_file", 2024-04-21T23:19:09.489 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.489 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.490 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.490 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.490 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.490 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.490 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.490 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-21T23:19:09.490 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.491 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.491 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.491 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.491 INFO:teuthology.orchestra.run.smithi044.stdout: "use_agent": { 2024-04-21T23:19:09.491 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "use_agent", 2024-04-21T23:19:09.491 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.491 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.491 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.492 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.492 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.492 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.492 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.492 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-21T23:19:09.492 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.492 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.492 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.493 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.493 INFO:teuthology.orchestra.run.smithi044.stdout: "use_repo_digest": { 2024-04-21T23:19:09.493 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "use_repo_digest", 2024-04-21T23:19:09.493 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.493 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.493 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.493 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.493 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.494 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.494 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.494 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-21T23:19:09.494 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.494 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.494 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.494 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.494 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_failed_host_check": { 2024-04-21T23:19:09.494 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_failed_host_check", 2024-04-21T23:19:09.495 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.495 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.495 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.495 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.495 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.495 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.495 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.495 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if the host check fails", 2024-04-21T23:19:09.496 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.496 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.496 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.496 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.496 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_stray_daemons": { 2024-04-21T23:19:09.496 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_stray_daemons", 2024-04-21T23:19:09.496 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.496 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.497 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.497 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.497 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.497 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.497 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.497 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-21T23:19:09.497 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.497 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.498 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.498 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.498 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_stray_hosts": { 2024-04-21T23:19:09.498 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_stray_hosts", 2024-04-21T23:19:09.498 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.498 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.498 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.498 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.499 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.499 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.499 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.499 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-21T23:19:09.499 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.499 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.499 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.499 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.499 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.500 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.500 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.500 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crash", 2024-04-21T23:19:09.500 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.500 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.500 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.500 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.500 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.501 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.501 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.501 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.501 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.501 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.501 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.501 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.502 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.502 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.502 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.502 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.502 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.502 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.502 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.502 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.503 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.503 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.503 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.503 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.503 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.503 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.503 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.503 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.504 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.504 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.504 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.504 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.504 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.504 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.504 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.504 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.505 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.505 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.505 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.505 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.505 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.505 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.506 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.506 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.506 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.506 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.506 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.506 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.506 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.507 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.507 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.507 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.507 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.507 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.507 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.507 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.508 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.508 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.508 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.508 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.508 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.508 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.508 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.509 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.509 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.509 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.509 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.509 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.509 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.510 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.510 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.510 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.510 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.510 INFO:teuthology.orchestra.run.smithi044.stdout: "retain_interval": { 2024-04-21T23:19:09.510 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "retain_interval", 2024-04-21T23:19:09.510 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:09.511 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.511 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.511 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "31536000", 2024-04-21T23:19:09.511 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.511 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.511 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.511 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-21T23:19:09.512 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.512 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.512 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.512 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.512 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_recent_interval": { 2024-04-21T23:19:09.512 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_recent_interval", 2024-04-21T23:19:09.513 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:09.513 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.513 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.513 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1209600", 2024-04-21T23:19:09.513 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.513 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.514 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.514 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-21T23:19:09.514 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.514 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.514 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.515 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.515 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.515 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.515 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.515 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "dashboard", 2024-04-21T23:19:09.515 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.515 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.516 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.516 INFO:teuthology.orchestra.run.smithi044.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-21T23:19:09.516 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-21T23:19:09.516 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.516 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.516 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.516 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:09.516 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.517 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.517 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.517 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.517 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.517 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.517 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.517 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.517 INFO:teuthology.orchestra.run.smithi044.stdout: "ALERTMANAGER_API_HOST": { 2024-04-21T23:19:09.518 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-21T23:19:09.518 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.518 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.518 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.518 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.518 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.518 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.518 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.519 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.519 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.519 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.519 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.519 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.519 INFO:teuthology.orchestra.run.smithi044.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-21T23:19:09.519 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-21T23:19:09.519 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.520 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.520 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.520 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.520 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.520 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.520 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.520 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.520 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.521 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.521 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.521 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.521 INFO:teuthology.orchestra.run.smithi044.stdout: "AUDIT_API_ENABLED": { 2024-04-21T23:19:09.521 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "AUDIT_API_ENABLED", 2024-04-21T23:19:09.521 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.521 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.522 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.522 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.522 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.522 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.522 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.522 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.522 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.522 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.523 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.523 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.523 INFO:teuthology.orchestra.run.smithi044.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-21T23:19:09.523 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-21T23:19:09.523 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.523 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.523 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.523 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.524 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.524 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.524 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.524 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.524 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.524 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.524 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.525 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.525 INFO:teuthology.orchestra.run.smithi044.stdout: "ENABLE_BROWSABLE_API": { 2024-04-21T23:19:09.525 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-21T23:19:09.525 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.525 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.525 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.525 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.525 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.526 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.526 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.526 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.526 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.526 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.526 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.527 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.527 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-21T23:19:09.527 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-21T23:19:09.527 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.527 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.527 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.527 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.527 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.528 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.528 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.528 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.528 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.528 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.528 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.528 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.528 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-21T23:19:09.529 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-21T23:19:09.529 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.529 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.529 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.529 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.529 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.529 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.529 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.529 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.530 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.530 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.530 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.530 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.530 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-21T23:19:09.530 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-21T23:19:09.530 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.530 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.531 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.531 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.531 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.531 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.531 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.531 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.531 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.531 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.532 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.532 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.532 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-21T23:19:09.532 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-21T23:19:09.532 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.532 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.532 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.532 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.533 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.533 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.533 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.533 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.533 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.533 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.533 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.533 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.534 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-21T23:19:09.534 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-21T23:19:09.534 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.534 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.534 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.534 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.534 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.534 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.535 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.535 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.535 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.535 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.535 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.535 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.535 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-21T23:19:09.535 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-21T23:19:09.535 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.536 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.536 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.536 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.536 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.536 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.536 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.536 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.536 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.536 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.537 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.537 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.537 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-21T23:19:09.537 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-21T23:19:09.537 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.537 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.537 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.537 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.538 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.538 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.538 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.538 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.538 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.538 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.538 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.538 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.539 INFO:teuthology.orchestra.run.smithi044.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-21T23:19:09.539 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-21T23:19:09.539 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.539 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.539 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.539 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.539 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.540 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.540 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.540 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.540 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.540 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.540 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.541 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.541 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_PASSWORD": { 2024-04-21T23:19:09.541 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-21T23:19:09.541 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.541 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.541 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.541 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2024-04-21T23:19:09.542 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.542 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.542 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.542 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.542 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.542 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.542 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.543 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.543 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-21T23:19:09.543 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-21T23:19:09.543 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.543 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.543 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.544 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.544 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.544 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.544 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.544 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.544 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.544 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.544 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.545 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.545 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_URL": { 2024-04-21T23:19:09.545 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_URL", 2024-04-21T23:19:09.545 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.545 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.545 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.546 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.546 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.546 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.546 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.546 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.546 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.546 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.547 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.547 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.547 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_USERNAME": { 2024-04-21T23:19:09.547 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-21T23:19:09.547 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.547 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.547 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.548 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2024-04-21T23:19:09.548 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.548 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.548 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.548 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.548 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.548 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.549 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.549 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.549 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-21T23:19:09.549 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-21T23:19:09.549 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.549 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.550 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.550 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.550 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.550 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.550 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.550 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.550 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.551 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.551 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.551 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.551 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-21T23:19:09.552 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-21T23:19:09.552 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.552 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.552 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.552 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.553 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.553 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.553 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.553 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.553 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.553 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.553 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.554 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.554 INFO:teuthology.orchestra.run.smithi044.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-21T23:19:09.554 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-21T23:19:09.554 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.554 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.554 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.554 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.554 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.555 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.555 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.555 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.555 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.555 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.555 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.555 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.555 INFO:teuthology.orchestra.run.smithi044.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-21T23:19:09.556 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-21T23:19:09.556 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.556 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.556 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.556 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.556 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.556 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.556 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.557 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.557 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.557 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.557 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.557 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.557 INFO:teuthology.orchestra.run.smithi044.stdout: "PROMETHEUS_API_HOST": { 2024-04-21T23:19:09.557 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-21T23:19:09.557 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.557 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.558 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.558 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.558 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.558 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.558 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.558 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.558 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.558 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.559 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.559 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.559 INFO:teuthology.orchestra.run.smithi044.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-21T23:19:09.559 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-21T23:19:09.559 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.559 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.559 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.559 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.560 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.560 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.560 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.560 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.560 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.560 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.560 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.560 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.560 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-21T23:19:09.561 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-21T23:19:09.561 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.561 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.561 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.561 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.561 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.561 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.561 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.562 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.562 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.562 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.562 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.562 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.562 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-21T23:19:09.562 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-21T23:19:09.562 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.563 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.563 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.563 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.563 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.563 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.563 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.563 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.563 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.563 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.564 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.564 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.564 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-21T23:19:09.564 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-21T23:19:09.564 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.564 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.564 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.564 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.564 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.565 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.565 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.565 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.565 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.565 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.565 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.565 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.565 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-21T23:19:09.566 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-21T23:19:09.566 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.566 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.566 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.566 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.566 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.566 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.566 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.566 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.567 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.567 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.567 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.567 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.567 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-21T23:19:09.567 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-21T23:19:09.567 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.567 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.568 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.568 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.568 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.568 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.568 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.568 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.568 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.568 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.568 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.569 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.569 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-21T23:19:09.569 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-21T23:19:09.569 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.569 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.569 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.569 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.569 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.570 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.570 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.570 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.570 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.570 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.570 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.570 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.570 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-21T23:19:09.571 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-21T23:19:09.571 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.571 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.571 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.571 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.571 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.571 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.571 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.571 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.572 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.572 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.572 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.572 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.572 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_ENABLED": { 2024-04-21T23:19:09.572 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-21T23:19:09.572 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.572 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.573 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.573 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.573 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.573 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.573 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.573 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.573 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.573 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.573 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.574 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.574 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-21T23:19:09.574 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-21T23:19:09.574 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.574 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.574 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.574 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-21T23:19:09.575 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.575 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.575 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.575 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.575 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.575 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.575 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.576 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.576 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-21T23:19:09.576 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-21T23:19:09.576 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.576 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.576 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.576 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:09.577 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.577 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.577 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.577 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.577 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.577 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.577 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.577 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.578 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-21T23:19:09.578 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-21T23:19:09.578 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.578 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.578 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.579 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8", 2024-04-21T23:19:09.579 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.579 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.579 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.579 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.579 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.579 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.579 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.580 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.580 INFO:teuthology.orchestra.run.smithi044.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-21T23:19:09.580 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-21T23:19:09.580 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.580 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.580 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.581 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "45", 2024-04-21T23:19:09.581 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.581 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.581 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.581 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.581 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.581 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.582 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.582 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.582 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_ACCESS_KEY": { 2024-04-21T23:19:09.582 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-21T23:19:09.582 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.582 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.582 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.583 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.583 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.583 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.583 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.583 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.583 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.583 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.584 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.584 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.584 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-21T23:19:09.584 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-21T23:19:09.584 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.584 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.584 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.584 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2024-04-21T23:19:09.585 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.585 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.585 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.585 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.585 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.585 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.586 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.586 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.586 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_SECRET_KEY": { 2024-04-21T23:19:09.586 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-21T23:19:09.586 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.586 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.587 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.587 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.587 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.587 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.587 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.587 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.588 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.588 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.588 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.588 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.588 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_SSL_VERIFY": { 2024-04-21T23:19:09.588 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-21T23:19:09.588 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.589 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.589 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.589 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.589 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.589 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.589 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.589 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.589 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.589 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.590 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.590 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.590 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-21T23:19:09.590 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-21T23:19:09.590 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.590 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.590 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.590 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-04-21T23:19:09.591 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.591 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.591 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.591 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.591 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.591 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.591 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.591 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.591 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-21T23:19:09.592 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-21T23:19:09.592 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.592 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.592 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.592 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:09.592 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.592 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.592 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.593 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.593 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.593 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.593 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.593 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.593 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-21T23:19:09.593 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-21T23:19:09.593 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.594 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.594 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.594 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-04-21T23:19:09.594 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.594 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.594 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.594 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.594 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.595 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.595 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.595 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.595 INFO:teuthology.orchestra.run.smithi044.stdout: "cross_origin_url": { 2024-04-21T23:19:09.595 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cross_origin_url", 2024-04-21T23:19:09.595 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.595 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.595 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.595 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.596 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.596 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.596 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.596 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.596 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.596 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.596 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.596 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.597 INFO:teuthology.orchestra.run.smithi044.stdout: "crt_file": { 2024-04-21T23:19:09.597 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crt_file", 2024-04-21T23:19:09.597 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.597 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.597 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.597 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.597 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.597 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.598 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.598 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.598 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.598 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.598 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.598 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.598 INFO:teuthology.orchestra.run.smithi044.stdout: "debug": { 2024-04-21T23:19:09.598 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "debug", 2024-04-21T23:19:09.599 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.599 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.599 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.599 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.599 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.599 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.599 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.600 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable/disable debug options", 2024-04-21T23:19:09.600 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.600 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.600 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.600 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.600 INFO:teuthology.orchestra.run.smithi044.stdout: "jwt_token_ttl": { 2024-04-21T23:19:09.600 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "jwt_token_ttl", 2024-04-21T23:19:09.600 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.600 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.601 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.601 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "28800", 2024-04-21T23:19:09.601 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.601 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.601 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.601 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.601 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.601 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.601 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.602 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.602 INFO:teuthology.orchestra.run.smithi044.stdout: "key_file": { 2024-04-21T23:19:09.602 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "key_file", 2024-04-21T23:19:09.602 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.602 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.602 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.602 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.602 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.603 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.603 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.603 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.603 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.603 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.603 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.603 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.603 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.604 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.604 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.604 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.604 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.604 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.604 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.604 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.604 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.604 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.605 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.605 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.605 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.605 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.605 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.605 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.605 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.605 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.606 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.606 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.606 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.606 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.606 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.606 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.606 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.606 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.607 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.607 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.607 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.607 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.607 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.607 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.607 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.607 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.607 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.608 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.608 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.608 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.608 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.608 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.608 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.608 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.608 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.609 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.609 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.609 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.609 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.609 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.609 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.609 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.609 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.609 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.610 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.610 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.610 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.610 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.610 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.610 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.610 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.610 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.611 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.611 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.611 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.611 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.611 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.611 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.611 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.611 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.611 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.612 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.612 INFO:teuthology.orchestra.run.smithi044.stdout: "motd": { 2024-04-21T23:19:09.612 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "motd", 2024-04-21T23:19:09.612 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.612 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.612 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.612 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.612 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.612 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.613 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.613 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "The message of the day", 2024-04-21T23:19:09.613 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.613 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.613 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.613 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.613 INFO:teuthology.orchestra.run.smithi044.stdout: "redirect_resolve_ip_addr": { 2024-04-21T23:19:09.613 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "redirect_resolve_ip_addr", 2024-04-21T23:19:09.614 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.614 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.614 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.614 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.614 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.614 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.614 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.614 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.614 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.615 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.615 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.615 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.615 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2024-04-21T23:19:09.615 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2024-04-21T23:19:09.616 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.616 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.616 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.616 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "::", 2024-04-21T23:19:09.616 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.616 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.616 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.616 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.616 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.617 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.617 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.617 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.617 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2024-04-21T23:19:09.617 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2024-04-21T23:19:09.617 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.617 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.618 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.618 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8080", 2024-04-21T23:19:09.618 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.618 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.618 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.618 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.619 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.619 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.619 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.619 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.619 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl": { 2024-04-21T23:19:09.619 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl", 2024-04-21T23:19:09.619 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.619 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.620 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.620 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.620 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.620 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.620 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.620 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.621 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.621 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.621 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.621 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.621 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl_server_port": { 2024-04-21T23:19:09.621 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl_server_port", 2024-04-21T23:19:09.621 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.622 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.622 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.622 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8443", 2024-04-21T23:19:09.622 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.622 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.622 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.622 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.623 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.623 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.623 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.623 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.623 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_behaviour": { 2024-04-21T23:19:09.624 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_behaviour", 2024-04-21T23:19:09.624 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.624 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.624 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.624 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "redirect", 2024-04-21T23:19:09.624 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.624 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.624 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.625 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.625 INFO:teuthology.orchestra.run.smithi044.stdout: "redirect" 2024-04-21T23:19:09.625 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.625 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.625 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.626 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.626 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.626 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.626 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_error_status_code": { 2024-04-21T23:19:09.626 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_error_status_code", 2024-04-21T23:19:09.626 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.626 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.627 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.627 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "500", 2024-04-21T23:19:09.627 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "400", 2024-04-21T23:19:09.627 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "599", 2024-04-21T23:19:09.627 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.627 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.627 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.627 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.628 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.628 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.628 INFO:teuthology.orchestra.run.smithi044.stdout: "url_prefix": { 2024-04-21T23:19:09.628 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "url_prefix", 2024-04-21T23:19:09.628 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.628 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.628 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.628 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.628 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.629 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.629 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.629 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.629 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.629 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.629 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.629 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.630 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.630 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.630 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.630 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "devicehealth", 2024-04-21T23:19:09.630 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.631 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.631 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.631 INFO:teuthology.orchestra.run.smithi044.stdout: "enable_monitoring": { 2024-04-21T23:19:09.631 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enable_monitoring", 2024-04-21T23:19:09.631 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.631 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.631 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.632 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.632 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.632 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.632 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.632 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "monitor device health metrics", 2024-04-21T23:19:09.632 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.632 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.632 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.632 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.633 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.633 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.633 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.633 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.633 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.633 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.633 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.633 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.634 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.634 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.634 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.634 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.634 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.634 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.634 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.634 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.634 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.635 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.635 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.635 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.635 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.635 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.635 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.635 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.635 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.636 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.636 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.636 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.636 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.636 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.636 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.636 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.636 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.636 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.637 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.637 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.637 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.637 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.637 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.637 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.637 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.637 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.638 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.638 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.638 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.638 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.638 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.638 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.638 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.638 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.638 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.639 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.639 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.639 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.639 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.639 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.639 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.639 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.640 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.640 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.640 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.640 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.640 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.640 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.640 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.640 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.641 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.641 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.641 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.641 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.641 INFO:teuthology.orchestra.run.smithi044.stdout: "mark_out_threshold": { 2024-04-21T23:19:09.641 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mark_out_threshold", 2024-04-21T23:19:09.641 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:09.641 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.641 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.642 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "2419200", 2024-04-21T23:19:09.642 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.642 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.642 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.642 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-21T23:19:09.642 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.642 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.642 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.643 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.643 INFO:teuthology.orchestra.run.smithi044.stdout: "pool_name": { 2024-04-21T23:19:09.643 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pool_name", 2024-04-21T23:19:09.643 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.643 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.643 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.643 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "device_health_metrics", 2024-04-21T23:19:09.643 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.644 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.644 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.644 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-21T23:19:09.644 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.644 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.644 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.644 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.644 INFO:teuthology.orchestra.run.smithi044.stdout: "retention_period": { 2024-04-21T23:19:09.645 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "retention_period", 2024-04-21T23:19:09.645 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:09.645 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.645 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.645 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15552000", 2024-04-21T23:19:09.645 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.645 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.645 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.646 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long to retain device health metrics", 2024-04-21T23:19:09.646 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.646 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.646 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.646 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.646 INFO:teuthology.orchestra.run.smithi044.stdout: "scrape_frequency": { 2024-04-21T23:19:09.646 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "scrape_frequency", 2024-04-21T23:19:09.646 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:09.647 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.647 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.647 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "86400", 2024-04-21T23:19:09.647 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.647 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.647 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.647 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-21T23:19:09.648 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.648 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.648 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.648 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.648 INFO:teuthology.orchestra.run.smithi044.stdout: "self_heal": { 2024-04-21T23:19:09.648 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "self_heal", 2024-04-21T23:19:09.648 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.648 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.649 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.649 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.649 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.649 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.649 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.649 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-21T23:19:09.649 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.649 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.650 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.650 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.650 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-04-21T23:19:09.650 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-04-21T23:19:09.650 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:09.650 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.650 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.650 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-04-21T23:19:09.651 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.651 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.651 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.651 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to wake up and check device health", 2024-04-21T23:19:09.651 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.651 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.651 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.651 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.652 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_threshold": { 2024-04-21T23:19:09.652 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_threshold", 2024-04-21T23:19:09.652 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:09.652 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.652 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.652 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "7257600", 2024-04-21T23:19:09.652 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.652 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.653 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.653 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-21T23:19:09.653 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.653 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.653 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.653 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.653 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.653 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.654 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.654 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "diskprediction_local", 2024-04-21T23:19:09.654 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.654 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.654 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.654 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.654 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.654 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.655 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.655 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.655 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.655 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.655 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.655 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.656 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.656 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.656 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.656 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.656 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.656 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.656 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.656 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.657 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.657 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.657 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.657 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.657 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.657 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.658 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.658 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.658 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.658 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.658 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.658 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.658 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.658 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.659 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.659 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.659 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.659 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.659 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.659 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.660 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.660 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.660 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.660 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.660 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.660 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.660 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.661 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.661 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.661 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.661 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.661 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.661 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.661 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.662 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.662 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.662 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.662 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.662 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.662 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.663 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.663 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.663 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.663 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.663 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.663 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.664 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.664 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.664 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.664 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.664 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.664 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.665 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.665 INFO:teuthology.orchestra.run.smithi044.stdout: "predict_interval": { 2024-04-21T23:19:09.665 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "predict_interval", 2024-04-21T23:19:09.665 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.665 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.665 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.666 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "86400", 2024-04-21T23:19:09.666 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.666 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.666 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.666 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.666 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.666 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.666 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.667 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.667 INFO:teuthology.orchestra.run.smithi044.stdout: "predictor_model": { 2024-04-21T23:19:09.667 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "predictor_model", 2024-04-21T23:19:09.667 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.667 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.667 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.667 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "prophetstor", 2024-04-21T23:19:09.667 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.668 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.668 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.668 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.668 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.668 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.668 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.668 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.668 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-04-21T23:19:09.669 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-04-21T23:19:09.669 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.669 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.669 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.669 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-04-21T23:19:09.669 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.669 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.669 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.670 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.670 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.670 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.670 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.670 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.670 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.670 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.670 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.671 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "influx", 2024-04-21T23:19:09.671 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": false, 2024-04-21T23:19:09.671 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "influxdb python module not found", 2024-04-21T23:19:09.671 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.671 INFO:teuthology.orchestra.run.smithi044.stdout: "batch_size": { 2024-04-21T23:19:09.671 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "batch_size", 2024-04-21T23:19:09.671 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.671 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.672 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.672 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5000", 2024-04-21T23:19:09.672 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.672 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.672 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.672 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-21T23:19:09.672 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.672 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.672 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.673 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.673 INFO:teuthology.orchestra.run.smithi044.stdout: "database": { 2024-04-21T23:19:09.673 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "database", 2024-04-21T23:19:09.673 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.673 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.673 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.673 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "ceph", 2024-04-21T23:19:09.674 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.674 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.674 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.674 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-21T23:19:09.674 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.674 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.674 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.674 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.675 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": { 2024-04-21T23:19:09.675 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "hostname", 2024-04-21T23:19:09.675 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.675 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.675 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.675 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.675 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.675 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.675 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.676 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB server hostname", 2024-04-21T23:19:09.676 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.676 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.676 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.676 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.676 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-04-21T23:19:09.676 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-04-21T23:19:09.676 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:09.676 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.677 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.677 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "30", 2024-04-21T23:19:09.677 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "5", 2024-04-21T23:19:09.677 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.677 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.677 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-21T23:19:09.677 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.677 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.678 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.678 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.678 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.678 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.678 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.678 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.678 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.678 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.679 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.679 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.679 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.679 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.679 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.679 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.679 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.679 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.679 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.680 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.680 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.680 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.680 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.680 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.680 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.680 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.680 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.681 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.681 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.681 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.681 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.681 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.681 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.681 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.681 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.681 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.682 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.682 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.682 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.682 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.682 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.682 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.682 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.682 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.683 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.683 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.683 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.683 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.683 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.683 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.683 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.683 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.684 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.684 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.684 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.684 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.684 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.684 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.684 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.684 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.684 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.685 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.685 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.685 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.685 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.685 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.685 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.685 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.685 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.686 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.686 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.686 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.686 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.686 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.686 INFO:teuthology.orchestra.run.smithi044.stdout: "password": { 2024-04-21T23:19:09.686 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "password", 2024-04-21T23:19:09.686 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.687 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.687 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.687 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.687 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.687 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.687 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.688 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "password of InfluxDB server user", 2024-04-21T23:19:09.688 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.688 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.688 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.688 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.688 INFO:teuthology.orchestra.run.smithi044.stdout: "port": { 2024-04-21T23:19:09.688 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "port", 2024-04-21T23:19:09.688 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.688 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.689 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.689 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8086", 2024-04-21T23:19:09.689 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.689 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.689 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.689 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB server port", 2024-04-21T23:19:09.690 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.690 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.690 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.690 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.690 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl": { 2024-04-21T23:19:09.690 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl", 2024-04-21T23:19:09.690 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.690 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.691 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.691 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "false", 2024-04-21T23:19:09.691 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.691 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.691 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.691 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T23:19:09.691 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.692 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.692 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.692 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.692 INFO:teuthology.orchestra.run.smithi044.stdout: "threads": { 2024-04-21T23:19:09.692 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "threads", 2024-04-21T23:19:09.692 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.693 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.693 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.693 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-04-21T23:19:09.693 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-04-21T23:19:09.693 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "32", 2024-04-21T23:19:09.693 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.693 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-21T23:19:09.694 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.694 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.694 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.694 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.694 INFO:teuthology.orchestra.run.smithi044.stdout: "username": { 2024-04-21T23:19:09.694 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "username", 2024-04-21T23:19:09.694 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.695 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.695 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.695 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.695 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.695 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.695 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.696 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "username of InfluxDB server user", 2024-04-21T23:19:09.696 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.696 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.696 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.696 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.696 INFO:teuthology.orchestra.run.smithi044.stdout: "verify_ssl": { 2024-04-21T23:19:09.697 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "verify_ssl", 2024-04-21T23:19:09.697 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.697 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.697 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.697 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "true", 2024-04-21T23:19:09.697 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.698 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.698 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.698 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T23:19:09.698 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.698 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.699 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.699 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.699 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.699 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.699 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.700 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "insights", 2024-04-21T23:19:09.700 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.700 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.700 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.700 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.700 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.700 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.701 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.701 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.701 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.701 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.701 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.701 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.701 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.701 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.701 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.702 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.702 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.702 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.702 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.702 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.702 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.702 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.702 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.703 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.703 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.703 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.703 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.703 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.703 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.703 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.703 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.704 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.704 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.704 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.704 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.704 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.704 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.704 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.704 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.704 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.705 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.705 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.705 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.705 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.705 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.705 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.705 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.705 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.706 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.706 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.706 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.706 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.706 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.706 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.706 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.706 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.706 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.707 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.707 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.707 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.707 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.707 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.707 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.707 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.707 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.708 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.708 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.708 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.708 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.708 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.708 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.708 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.708 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.708 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.709 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.709 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.709 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "iostat", 2024-04-21T23:19:09.709 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.709 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.709 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.709 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.710 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.710 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.710 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.710 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.710 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.710 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.710 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.710 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.710 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.711 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.711 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.711 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.711 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.711 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.711 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.711 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.711 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.712 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.712 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.712 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.712 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.712 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.712 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.712 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.712 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.712 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.713 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.713 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.713 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.713 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.713 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.713 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.713 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.713 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.713 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.714 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.714 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.714 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.714 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.714 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.714 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.714 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.714 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.715 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.715 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.715 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.715 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.715 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.715 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.715 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.715 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.716 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.716 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.716 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.716 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.716 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.716 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.716 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.716 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.716 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.717 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.717 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.717 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.717 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.717 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.717 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.717 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.717 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.718 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.718 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.718 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.718 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.718 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "k8sevents", 2024-04-21T23:19:09.718 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.718 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.718 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.718 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_event_retention_days": { 2024-04-21T23:19:09.719 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ceph_event_retention_days", 2024-04-21T23:19:09.719 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.719 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.719 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.719 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "7", 2024-04-21T23:19:09.719 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.719 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.719 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.720 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-21T23:19:09.720 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.720 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.720 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.720 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.720 INFO:teuthology.orchestra.run.smithi044.stdout: "config_check_secs": { 2024-04-21T23:19:09.720 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_check_secs", 2024-04-21T23:19:09.720 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.720 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.721 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.721 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:09.721 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "10", 2024-04-21T23:19:09.721 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.721 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.721 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-21T23:19:09.721 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.722 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.722 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.722 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.722 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.722 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.722 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.722 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.722 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.722 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.723 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.723 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.723 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.723 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.723 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.723 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.723 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.723 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.723 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.724 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.724 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.724 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.724 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.724 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.724 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.725 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.725 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.725 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.725 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.725 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.725 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.725 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.726 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.726 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.726 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.726 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.726 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.726 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.726 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.726 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.727 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.727 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.727 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.727 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.727 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.727 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.728 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.728 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.728 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.728 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.728 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.728 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.728 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.728 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.729 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.729 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.729 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.729 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.729 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.729 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.730 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.730 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.730 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.730 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.730 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.730 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.730 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.731 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.731 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.731 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.731 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.731 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.731 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.732 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.732 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.732 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.732 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.732 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "localpool", 2024-04-21T23:19:09.732 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.732 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.733 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.733 INFO:teuthology.orchestra.run.smithi044.stdout: "failure_domain": { 2024-04-21T23:19:09.733 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "failure_domain", 2024-04-21T23:19:09.733 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.733 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.733 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.733 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "host", 2024-04-21T23:19:09.734 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.734 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.734 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.734 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "failure domain for any created local pool", 2024-04-21T23:19:09.734 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-21T23:19:09.734 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.735 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.735 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.735 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.735 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.735 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.735 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.736 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.736 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.736 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.736 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.736 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.736 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.737 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.737 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.737 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.737 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.737 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.737 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.738 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.738 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.738 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.738 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.738 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.738 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.738 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.738 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.739 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.739 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.739 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.739 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.739 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.739 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.739 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.739 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.740 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.740 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.740 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.740 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.740 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.740 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.740 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.740 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.741 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.741 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.741 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.741 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.741 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.741 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.741 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.741 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.741 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.742 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.742 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.742 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.742 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.742 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.742 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.742 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.742 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.743 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.743 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.743 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.743 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.743 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.743 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.743 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.743 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.744 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.744 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.744 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.744 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.744 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.744 INFO:teuthology.orchestra.run.smithi044.stdout: "min_size": { 2024-04-21T23:19:09.744 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "min_size", 2024-04-21T23:19:09.744 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.745 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.745 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.745 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.745 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.745 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.745 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.745 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default min_size for any created local pool", 2024-04-21T23:19:09.745 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-21T23:19:09.745 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.746 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.746 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.746 INFO:teuthology.orchestra.run.smithi044.stdout: "num_rep": { 2024-04-21T23:19:09.746 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "num_rep", 2024-04-21T23:19:09.746 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.746 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.747 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.747 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3", 2024-04-21T23:19:09.747 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.747 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.747 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.748 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default replica count for any created local pool", 2024-04-21T23:19:09.748 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.748 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.748 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.748 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.748 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_num": { 2024-04-21T23:19:09.748 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pg_num", 2024-04-21T23:19:09.748 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.749 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.749 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.749 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "128", 2024-04-21T23:19:09.749 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.749 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.749 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.749 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default pg_num for any created local pool", 2024-04-21T23:19:09.750 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.750 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.750 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.750 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.750 INFO:teuthology.orchestra.run.smithi044.stdout: "prefix": { 2024-04-21T23:19:09.750 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prefix", 2024-04-21T23:19:09.750 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.750 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.751 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.751 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.751 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.751 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.751 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.751 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "name prefix for any created local pool", 2024-04-21T23:19:09.751 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.752 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.752 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.752 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.752 INFO:teuthology.orchestra.run.smithi044.stdout: "subtree": { 2024-04-21T23:19:09.752 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "subtree", 2024-04-21T23:19:09.752 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.752 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.752 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.752 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "rack", 2024-04-21T23:19:09.753 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.753 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.753 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.753 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-21T23:19:09.753 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-21T23:19:09.753 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.753 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.753 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.754 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.754 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.754 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.754 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mds_autoscaler", 2024-04-21T23:19:09.754 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.754 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.754 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.754 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.754 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.755 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.755 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.755 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.755 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.755 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.755 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.755 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.755 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.756 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.756 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.756 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.756 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.756 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.756 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.756 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.756 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.757 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.757 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.757 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.757 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.757 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.757 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.757 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.757 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.757 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.758 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.758 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.758 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.758 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.758 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.758 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.758 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.758 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.759 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.759 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.759 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.759 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.759 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.759 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.759 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.759 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.760 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.760 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.760 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.760 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.760 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.760 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.760 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.760 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.760 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.761 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.761 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.761 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.761 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.761 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.761 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.762 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.762 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.762 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.762 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.762 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.762 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.762 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.762 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.763 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.763 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.763 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.763 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.763 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.763 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.764 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.764 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mirroring", 2024-04-21T23:19:09.764 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.764 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.764 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.764 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.764 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.765 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.765 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.765 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.765 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.765 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.765 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.765 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.766 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.766 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.766 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.766 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.766 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.766 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.766 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.766 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.767 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.767 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.767 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.767 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.767 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.768 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.768 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.768 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.768 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.768 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.768 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.769 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.769 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.769 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.769 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.769 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.769 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.769 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.770 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.770 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.770 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.770 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.770 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.770 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.771 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.771 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.771 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.771 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.771 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.771 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.772 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.772 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.772 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.772 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.772 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.772 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.772 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.772 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.773 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.773 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.773 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.773 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.773 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.773 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.773 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.774 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.774 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.774 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.774 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.774 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.774 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.774 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.774 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.774 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.775 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.775 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.775 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "nfs", 2024-04-21T23:19:09.775 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.775 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.775 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.775 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.775 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.776 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.776 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.776 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.777 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.777 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.777 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.778 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.778 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.778 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.778 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.778 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.778 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.778 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.778 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.779 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.779 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.779 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.779 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.779 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.779 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.779 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.779 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.780 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.780 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.780 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.780 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.780 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.780 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.780 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.780 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.781 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.781 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.781 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.781 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.781 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.781 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.781 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.781 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.781 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.782 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.782 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.782 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.782 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.782 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.782 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.782 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.782 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.783 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.783 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.783 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.783 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.783 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.783 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.783 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.783 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.783 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.784 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.784 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.784 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.784 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.784 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.784 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.784 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.784 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.785 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.785 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.785 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.785 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.785 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.785 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.785 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.785 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "orchestrator", 2024-04-21T23:19:09.786 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.786 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.786 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.786 INFO:teuthology.orchestra.run.smithi044.stdout: "fail_fs": { 2024-04-21T23:19:09.786 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "fail_fs", 2024-04-21T23:19:09.786 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.786 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.786 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.787 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.787 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.787 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.787 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.787 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-21T23:19:09.787 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.787 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.787 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.787 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.788 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.788 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.788 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.788 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.788 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.788 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.788 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.788 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.788 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.789 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.789 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.789 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.789 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.789 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.789 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.789 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.789 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.790 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.790 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.790 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.790 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.790 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.790 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.790 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.790 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.791 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.791 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.791 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.791 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.791 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.791 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.791 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.791 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.791 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.792 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.792 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.792 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.792 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.792 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.792 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.792 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.792 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.793 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.793 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.793 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.793 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.793 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.793 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.793 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.793 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.793 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.794 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.794 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.794 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.794 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.794 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.794 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.794 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.794 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.795 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.795 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.795 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.795 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.795 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.795 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.795 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.795 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.795 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.796 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.796 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.796 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator": { 2024-04-21T23:19:09.796 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "orchestrator", 2024-04-21T23:19:09.796 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.796 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.796 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.797 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.797 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.797 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.797 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.797 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm", 2024-04-21T23:19:09.797 INFO:teuthology.orchestra.run.smithi044.stdout: "rook", 2024-04-21T23:19:09.797 INFO:teuthology.orchestra.run.smithi044.stdout: "test_orchestrator" 2024-04-21T23:19:09.797 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.798 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Orchestrator backend", 2024-04-21T23:19:09.798 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.798 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.798 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.798 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.798 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.798 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.799 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.799 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "osd_perf_query", 2024-04-21T23:19:09.799 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.799 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.799 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.799 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.799 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.799 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.800 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.800 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.800 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.800 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.800 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.800 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.800 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.800 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.800 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.801 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.801 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.801 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.801 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.801 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.801 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.801 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.801 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.802 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.802 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.802 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.802 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.802 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.802 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.802 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.802 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.803 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.803 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.803 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.803 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.803 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.803 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.803 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.803 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.804 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.804 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.804 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.804 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.804 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.804 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.804 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.804 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.804 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.805 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.805 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.805 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.805 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.805 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.805 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.805 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.805 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.806 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.806 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.806 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.806 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.806 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.806 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.806 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.806 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.807 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.807 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.807 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.807 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.807 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.807 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.807 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.807 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.807 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.808 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.808 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.808 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.808 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "osd_support", 2024-04-21T23:19:09.808 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.808 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.808 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.808 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.808 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.809 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.809 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.809 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.809 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.809 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.809 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.810 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.810 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.810 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.810 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.810 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.810 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.810 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.811 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.811 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.811 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.811 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.811 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.811 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.811 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.811 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.811 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.812 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.812 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.812 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.812 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.812 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.812 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.812 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.813 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.813 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.813 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.813 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.813 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.813 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.813 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.814 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.814 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.814 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.814 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.814 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.814 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.815 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.815 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.815 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.815 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.815 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.815 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.815 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.815 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.816 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.816 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.816 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.816 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.816 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.817 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.817 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.817 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.817 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.817 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.817 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.817 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.818 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.818 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.818 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.818 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.818 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.818 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.818 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.819 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.819 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.819 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pg_autoscaler", 2024-04-21T23:19:09.819 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.819 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.819 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.820 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.820 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.820 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.820 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.820 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.820 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.820 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.821 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.821 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.821 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.821 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.821 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.821 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.821 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.821 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.821 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.822 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.822 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.822 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.822 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.822 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.822 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.822 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.822 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.822 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.823 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.823 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.823 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.823 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.823 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.823 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.823 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.823 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.824 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.824 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.824 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.824 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.824 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.824 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.824 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.825 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.825 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.825 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.825 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.825 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.825 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.825 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.825 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.825 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.826 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.826 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.826 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.826 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.826 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.826 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.826 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.826 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.827 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.827 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.827 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.827 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.827 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.827 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.827 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.827 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.828 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.828 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.828 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.828 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.828 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.828 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-04-21T23:19:09.828 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-04-21T23:19:09.828 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:09.829 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.829 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.829 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-04-21T23:19:09.829 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.829 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.829 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.829 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.829 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.829 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.830 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.830 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.830 INFO:teuthology.orchestra.run.smithi044.stdout: "threshold": { 2024-04-21T23:19:09.830 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "threshold", 2024-04-21T23:19:09.830 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:09.830 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.830 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.830 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3.0", 2024-04-21T23:19:09.831 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1.0", 2024-04-21T23:19:09.831 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.831 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.831 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "scaling threshold", 2024-04-21T23:19:09.831 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-21T23:19:09.831 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.831 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.831 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.832 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.832 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.832 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.832 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "progress", 2024-04-21T23:19:09.832 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.832 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.832 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.832 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_pg_recovery_event": { 2024-04-21T23:19:09.833 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_pg_recovery_event", 2024-04-21T23:19:09.833 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.833 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.833 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.833 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.833 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.833 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.833 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.834 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-21T23:19:09.834 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.834 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.834 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.834 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.834 INFO:teuthology.orchestra.run.smithi044.stdout: "enabled": { 2024-04-21T23:19:09.834 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enabled", 2024-04-21T23:19:09.835 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.835 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.835 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.835 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.835 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.835 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.835 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.835 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.835 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.836 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.836 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.836 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.836 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.836 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.836 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.836 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.836 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.836 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.837 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.837 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.837 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.837 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.837 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.837 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.837 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.837 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.838 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.838 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.838 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.838 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.838 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.838 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.838 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.838 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.838 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.839 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.839 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.839 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.839 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.839 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.839 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.839 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.839 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.840 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.840 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.840 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.840 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.840 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.840 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.840 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.840 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.841 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.841 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.841 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.841 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.841 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.841 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.841 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.841 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.842 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.842 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.842 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.842 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.842 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.842 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.842 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.842 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.843 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.843 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.843 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.843 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.843 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.843 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.843 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.843 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.843 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.844 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.844 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.844 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.844 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.844 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.844 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.844 INFO:teuthology.orchestra.run.smithi044.stdout: "max_completed_events": { 2024-04-21T23:19:09.844 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_completed_events", 2024-04-21T23:19:09.844 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.845 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.845 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.845 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "50", 2024-04-21T23:19:09.845 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.845 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.845 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.845 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "number of past completed events to remember", 2024-04-21T23:19:09.845 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.846 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.846 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.846 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.846 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-04-21T23:19:09.846 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-04-21T23:19:09.846 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:09.846 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.846 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.847 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-04-21T23:19:09.847 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.847 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.847 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.847 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long the module is going to sleep", 2024-04-21T23:19:09.847 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.847 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.847 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.848 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.848 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.848 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.848 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.848 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus", 2024-04-21T23:19:09.848 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.848 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.848 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.848 INFO:teuthology.orchestra.run.smithi044.stdout: "cache": { 2024-04-21T23:19:09.849 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cache", 2024-04-21T23:19:09.849 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.849 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.849 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.849 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.849 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.849 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.849 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.850 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.850 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.850 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.850 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.850 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.850 INFO:teuthology.orchestra.run.smithi044.stdout: "exclude_perf_counters": { 2024-04-21T23:19:09.850 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "exclude_perf_counters", 2024-04-21T23:19:09.850 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.851 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.851 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.851 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.851 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.851 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.851 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.851 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-21T23:19:09.851 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-21T23:19:09.852 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.852 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.852 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.852 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.852 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.852 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.852 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.852 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.852 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.853 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.853 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.853 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.853 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.853 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.853 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.853 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.854 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.854 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.854 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.854 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.854 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.854 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.855 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.855 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.855 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.855 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.855 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.855 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.855 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.855 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.856 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.857 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.858 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.858 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.858 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.858 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.858 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.858 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.859 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.859 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.859 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.859 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.859 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.859 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.859 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.860 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.861 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.861 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.861 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.861 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.861 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.861 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.861 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.862 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.862 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.862 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_stats_pools": { 2024-04-21T23:19:09.862 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_stats_pools", 2024-04-21T23:19:09.862 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.863 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.864 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-21T23:19:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-21T23:19:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "300", 2024-04-21T23:19:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.865 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: "scrape_interval": { 2024-04-21T23:19:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "scrape_interval", 2024-04-21T23:19:09.866 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:09.867 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.867 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.867 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15.0", 2024-04-21T23:19:09.867 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.867 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.867 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.867 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.867 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.867 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.868 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.868 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.868 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2024-04-21T23:19:09.868 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2024-04-21T23:19:09.868 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.868 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.868 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.868 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "::", 2024-04-21T23:19:09.869 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.869 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.869 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.869 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-21T23:19:09.869 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.869 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.869 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.869 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.869 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2024-04-21T23:19:09.870 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2024-04-21T23:19:09.870 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.870 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.870 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.870 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "9283", 2024-04-21T23:19:09.870 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.870 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.870 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.871 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-21T23:19:09.871 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.871 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.871 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.871 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.871 INFO:teuthology.orchestra.run.smithi044.stdout: "stale_cache_strategy": { 2024-04-21T23:19:09.871 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "stale_cache_strategy", 2024-04-21T23:19:09.872 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.872 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.872 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.872 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "log", 2024-04-21T23:19:09.872 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.872 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.872 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.872 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.873 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.873 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.873 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.873 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.873 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_behaviour": { 2024-04-21T23:19:09.873 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_behaviour", 2024-04-21T23:19:09.873 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.873 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.873 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.874 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "default", 2024-04-21T23:19:09.874 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.874 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.874 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.874 INFO:teuthology.orchestra.run.smithi044.stdout: "default", 2024-04-21T23:19:09.874 INFO:teuthology.orchestra.run.smithi044.stdout: "error" 2024-04-21T23:19:09.874 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.874 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.875 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.875 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.875 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.875 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.875 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_error_status_code": { 2024-04-21T23:19:09.875 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_error_status_code", 2024-04-21T23:19:09.875 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.875 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.876 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.876 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "500", 2024-04-21T23:19:09.876 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "400", 2024-04-21T23:19:09.876 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "599", 2024-04-21T23:19:09.876 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.876 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.876 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.876 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.877 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.877 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.877 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.877 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.877 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.877 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_support", 2024-04-21T23:19:09.877 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.877 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.877 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.878 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.878 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.878 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.878 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.878 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.878 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.878 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.878 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.879 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.879 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.879 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.879 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.879 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.879 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.879 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.879 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.880 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.880 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.880 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.880 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.880 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.880 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.880 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.880 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.881 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.881 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.881 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.881 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.881 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.881 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.881 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.881 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.881 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.882 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.882 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.882 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.882 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.882 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.882 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.882 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.882 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.883 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.883 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.883 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.883 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.883 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.883 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.883 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.883 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.884 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.884 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.884 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.884 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.884 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.884 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.884 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.884 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.885 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.885 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.885 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.885 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.885 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.885 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.885 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.886 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.886 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.886 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.886 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.886 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.886 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.886 INFO:teuthology.orchestra.run.smithi044.stdout: "max_concurrent_snap_create": { 2024-04-21T23:19:09.887 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_concurrent_snap_create", 2024-04-21T23:19:09.887 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.887 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.887 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.887 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:09.887 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.887 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.887 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.888 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.888 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.888 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.888 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.888 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.888 INFO:teuthology.orchestra.run.smithi044.stdout: "mirror_snapshot_schedule": { 2024-04-21T23:19:09.889 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mirror_snapshot_schedule", 2024-04-21T23:19:09.889 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.889 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.889 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.889 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.889 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.889 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.889 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.890 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.890 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.890 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.890 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.890 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.891 INFO:teuthology.orchestra.run.smithi044.stdout: "trash_purge_schedule": { 2024-04-21T23:19:09.891 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "trash_purge_schedule", 2024-04-21T23:19:09.891 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.891 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.891 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.891 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.891 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.891 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.891 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.892 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.892 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.892 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.892 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.892 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.892 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.892 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.892 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.893 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "restful", 2024-04-21T23:19:09.893 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.893 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.893 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.893 INFO:teuthology.orchestra.run.smithi044.stdout: "enable_auth": { 2024-04-21T23:19:09.893 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enable_auth", 2024-04-21T23:19:09.893 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.893 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.894 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.894 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.894 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.894 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.894 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.894 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.894 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.895 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.895 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.895 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.895 INFO:teuthology.orchestra.run.smithi044.stdout: "key_file": { 2024-04-21T23:19:09.895 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "key_file", 2024-04-21T23:19:09.895 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.896 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.896 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.896 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.896 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.896 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.896 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.896 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.896 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.897 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.897 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.897 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.898 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.898 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.898 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.898 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.898 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.898 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.899 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.899 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.899 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.899 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.899 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.899 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.899 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.900 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.900 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.900 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.900 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.900 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.901 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.901 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.901 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.901 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.901 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.901 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.902 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.902 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.902 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.902 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.902 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.902 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.902 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.902 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.903 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.903 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.903 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.903 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.903 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.903 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.903 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.903 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.904 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.905 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.905 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.905 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.905 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.905 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.905 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.905 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.905 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.906 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.906 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.906 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.906 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.906 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.906 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.906 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.906 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.907 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.907 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.907 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.907 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.907 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.907 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.907 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2024-04-21T23:19:09.907 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2024-04-21T23:19:09.907 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.907 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.908 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.908 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.908 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.908 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.908 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.908 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.908 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.908 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.909 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.909 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.909 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2024-04-21T23:19:09.909 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2024-04-21T23:19:09.909 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.909 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.909 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.909 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.910 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.910 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.910 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.910 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.910 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.910 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.910 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.910 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.911 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.911 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.911 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.911 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rgw", 2024-04-21T23:19:09.911 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.911 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.911 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.911 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.912 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.912 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.912 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.912 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.912 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.912 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.912 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.912 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.913 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.913 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.913 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.913 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.913 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.913 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.913 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.913 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.914 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.914 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.914 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.914 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.914 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.914 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.914 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.914 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.915 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.915 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.915 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.915 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.915 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.915 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.915 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.915 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.916 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.916 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.916 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.916 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.916 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.916 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.916 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.916 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.916 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.917 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.917 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.917 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.917 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.917 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.917 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.917 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.917 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.918 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.918 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.918 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.918 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.918 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.918 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.918 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.919 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.919 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.919 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.919 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.919 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.919 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.919 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.919 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.919 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.920 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.920 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.920 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.920 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.920 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.920 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.920 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.920 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rook", 2024-04-21T23:19:09.920 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.921 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.921 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.921 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.921 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.921 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.921 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.921 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.921 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.922 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.922 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.922 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.922 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.922 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.922 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.922 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.923 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.923 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.923 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.923 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.923 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.923 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.924 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.924 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.924 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.924 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.924 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.924 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.924 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.924 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.924 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.925 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.925 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.925 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.925 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.925 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.925 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.926 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.926 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.926 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.926 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.926 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.926 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.926 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.927 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.927 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.927 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.927 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.927 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.927 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.927 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.928 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.928 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.928 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.928 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.928 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.928 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.928 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.928 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.929 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.929 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.929 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.929 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.929 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.929 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.930 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.930 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.930 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.930 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.930 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.930 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.931 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.931 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.931 INFO:teuthology.orchestra.run.smithi044.stdout: "storage_class": { 2024-04-21T23:19:09.931 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "storage_class", 2024-04-21T23:19:09.931 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.931 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.931 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.932 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "local", 2024-04-21T23:19:09.932 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.932 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.932 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.932 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-21T23:19:09.932 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.933 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.933 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.933 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.933 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.933 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.933 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.934 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "selftest", 2024-04-21T23:19:09.934 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.934 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.934 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.934 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.935 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.935 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.935 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.935 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.935 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.935 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.935 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.935 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.936 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.936 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.936 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.936 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.936 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.936 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.936 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.936 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.937 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.937 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.937 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.937 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.937 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.937 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.937 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.937 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.937 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.938 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.938 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.938 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.938 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.938 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.938 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.938 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.938 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.939 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.939 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.939 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.939 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.939 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.939 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.939 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.939 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.940 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.940 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.940 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.940 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.940 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.940 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.940 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.940 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.941 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.941 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.941 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.941 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.941 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.941 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.941 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.941 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.942 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.942 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.942 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.942 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.942 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.942 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.942 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.942 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.942 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.943 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.943 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.943 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.943 INFO:teuthology.orchestra.run.smithi044.stdout: "roption1": { 2024-04-21T23:19:09.943 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "roption1", 2024-04-21T23:19:09.943 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.943 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.943 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.944 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.944 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.944 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.944 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.944 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.944 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.944 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.944 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.945 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.945 INFO:teuthology.orchestra.run.smithi044.stdout: "roption2": { 2024-04-21T23:19:09.945 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "roption2", 2024-04-21T23:19:09.945 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.945 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.945 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.945 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "xyz", 2024-04-21T23:19:09.945 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.945 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.946 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.946 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.946 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.946 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.946 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.946 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.946 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption1": { 2024-04-21T23:19:09.946 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption1", 2024-04-21T23:19:09.947 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.947 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.947 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.947 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.947 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.947 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.947 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.947 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.948 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.948 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.948 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.948 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.948 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption2": { 2024-04-21T23:19:09.948 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption2", 2024-04-21T23:19:09.948 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.948 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.949 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.949 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.949 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.949 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.949 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.949 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.949 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.949 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.949 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.950 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.950 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption3": { 2024-04-21T23:19:09.950 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption3", 2024-04-21T23:19:09.950 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:09.950 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.950 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.950 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.950 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.951 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.951 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.951 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.951 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.951 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.951 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.951 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.951 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption4": { 2024-04-21T23:19:09.952 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption4", 2024-04-21T23:19:09.952 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.952 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.952 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.952 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.952 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.952 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.952 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.953 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.953 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.953 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.953 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.953 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.953 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption5": { 2024-04-21T23:19:09.953 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption5", 2024-04-21T23:19:09.953 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.953 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.954 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.954 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.954 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.954 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.954 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.954 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.954 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.954 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.955 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.955 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.955 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption6": { 2024-04-21T23:19:09.955 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption6", 2024-04-21T23:19:09.955 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.955 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.955 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.955 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:09.956 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.956 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.956 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.956 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.956 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.956 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.956 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.957 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.957 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption7": { 2024-04-21T23:19:09.957 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption7", 2024-04-21T23:19:09.957 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:09.957 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.957 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.957 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.958 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-04-21T23:19:09.958 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "42", 2024-04-21T23:19:09.958 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.958 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.958 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.958 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.958 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.959 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.959 INFO:teuthology.orchestra.run.smithi044.stdout: "testkey": { 2024-04-21T23:19:09.959 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testkey", 2024-04-21T23:19:09.959 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.959 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.959 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.959 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.960 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.960 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.960 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.960 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.960 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.960 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.960 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.961 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.961 INFO:teuthology.orchestra.run.smithi044.stdout: "testlkey": { 2024-04-21T23:19:09.961 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testlkey", 2024-04-21T23:19:09.961 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.961 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.961 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.962 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.962 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.962 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.962 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.962 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.962 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.962 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.962 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.963 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.963 INFO:teuthology.orchestra.run.smithi044.stdout: "testnewline": { 2024-04-21T23:19:09.963 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testnewline", 2024-04-21T23:19:09.963 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.963 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.963 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:09.963 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.964 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.964 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.964 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.964 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.964 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.964 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.964 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.964 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.964 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.965 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.965 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.965 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snap_schedule", 2024-04-21T23:19:09.965 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.965 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.965 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.965 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_m_granularity": { 2024-04-21T23:19:09.965 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_m_granularity", 2024-04-21T23:19:09.966 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.966 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.966 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.966 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.966 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.966 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.966 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.967 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow minute scheduled snapshots", 2024-04-21T23:19:09.967 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.967 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.967 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.967 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.967 INFO:teuthology.orchestra.run.smithi044.stdout: "dump_on_update": { 2024-04-21T23:19:09.968 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "dump_on_update", 2024-04-21T23:19:09.968 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.968 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.968 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.968 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.968 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.968 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.969 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.969 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "dump database to debug log on update", 2024-04-21T23:19:09.969 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.969 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.969 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.970 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.970 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.970 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.970 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.970 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.970 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.971 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.971 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.971 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.971 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.971 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.972 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.972 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.972 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.972 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.972 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.972 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.973 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.973 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.973 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.973 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.973 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.973 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.973 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.973 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.974 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.974 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.974 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.974 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.974 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.974 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.974 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.974 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.975 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.975 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.975 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.975 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.975 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.975 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.975 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.975 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.976 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.976 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.976 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.976 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.976 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.976 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.976 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.976 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.977 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.977 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.977 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.977 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.977 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.977 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.977 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.977 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.978 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.978 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.978 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.978 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.978 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.978 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.978 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.978 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.979 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.979 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.979 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.979 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.979 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.979 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.979 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.979 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.979 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.980 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "stats", 2024-04-21T23:19:09.980 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.980 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.980 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.980 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.980 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.980 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.980 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.980 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.981 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.981 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.981 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.981 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.981 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.981 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.981 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.981 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.982 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.982 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.982 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.982 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.982 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.982 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.982 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.982 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.983 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.983 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.983 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.983 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.983 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.983 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.983 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.983 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.983 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.984 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.984 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.984 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.984 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.984 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.984 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.984 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.984 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.985 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.985 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.985 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.985 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.985 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.985 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.985 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.985 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.985 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.986 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.986 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.986 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.986 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.986 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.986 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.986 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.986 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.987 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.987 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.987 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.987 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.987 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.987 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.987 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.987 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.988 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.988 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.988 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.988 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.988 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.988 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.988 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.988 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.988 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.989 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.989 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "status", 2024-04-21T23:19:09.989 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.989 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.989 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.989 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:09.989 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:09.989 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.990 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.990 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.990 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:09.990 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.990 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.990 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.990 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.990 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.991 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.991 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.991 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.991 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.991 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.991 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.991 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.991 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.991 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.992 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.992 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:09.992 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:09.992 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.992 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.992 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.992 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.992 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.993 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.993 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.993 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.993 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.993 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.993 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.993 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.993 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:09.994 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:09.994 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:09.994 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.994 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.994 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:09.994 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.994 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.994 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:09.995 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:09.995 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:09.995 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:09.995 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:09.995 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:09.995 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:09.995 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:09.996 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.996 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.996 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.996 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.996 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.996 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:09.996 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:09.996 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:09.997 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:09.997 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:09.997 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:09.997 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:09.997 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:09.997 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:09.997 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:09.998 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:09.998 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:09.998 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:09.998 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.998 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:09.998 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:09.998 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:09.999 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "telegraf", 2024-04-21T23:19:09.999 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:09.999 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:09.999 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:09.999 INFO:teuthology.orchestra.run.smithi044.stdout: "address": { 2024-04-21T23:19:09.999 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "address", 2024-04-21T23:19:09.999 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.000 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.000 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.000 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-21T23:19:10.000 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.000 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.000 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.000 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.001 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.001 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.001 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.001 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.001 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-04-21T23:19:10.001 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-04-21T23:19:10.002 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.002 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.002 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.002 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15", 2024-04-21T23:19:10.002 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.002 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.003 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.003 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.003 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.003 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.003 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.003 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.004 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.004 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.004 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.004 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.004 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.004 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.005 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.005 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.005 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.005 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.005 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.005 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.006 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.006 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.006 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.006 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.006 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.006 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.006 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.006 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.006 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.007 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.007 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.007 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.007 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.007 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.007 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.007 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.007 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.008 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.008 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.008 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.008 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.008 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.008 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.008 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.008 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.009 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.009 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.009 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.009 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.009 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.009 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.009 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.009 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.009 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.010 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.010 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.010 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.010 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.010 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.010 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.010 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.010 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.010 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.011 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.011 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.011 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.011 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.011 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.011 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.011 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.011 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.012 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.012 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.012 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.012 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.012 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.012 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.012 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.012 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.012 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.013 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.013 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "telemetry", 2024-04-21T23:19:10.013 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.013 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.013 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.013 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_basic": { 2024-04-21T23:19:10.013 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_basic", 2024-04-21T23:19:10.013 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.013 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.014 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.014 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.014 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.014 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.014 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.014 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-21T23:19:10.014 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.015 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.015 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.015 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.015 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_crash": { 2024-04-21T23:19:10.015 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_crash", 2024-04-21T23:19:10.015 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.015 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.015 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.015 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.016 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.016 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.016 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.016 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-21T23:19:10.016 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.016 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.016 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.016 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.017 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_device": { 2024-04-21T23:19:10.017 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_device", 2024-04-21T23:19:10.017 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.017 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.017 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.017 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.017 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.017 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.017 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.018 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-21T23:19:10.018 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.018 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.018 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.018 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.018 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_ident": { 2024-04-21T23:19:10.018 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_ident", 2024-04-21T23:19:10.018 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.019 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.019 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.019 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.019 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.019 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.019 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.019 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-21T23:19:10.019 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.019 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.020 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.020 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.020 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_perf": { 2024-04-21T23:19:10.020 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_perf", 2024-04-21T23:19:10.020 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.020 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.020 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.020 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.021 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.021 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.021 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.021 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-21T23:19:10.021 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.021 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.021 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.021 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.021 INFO:teuthology.orchestra.run.smithi044.stdout: "contact": { 2024-04-21T23:19:10.022 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "contact", 2024-04-21T23:19:10.022 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.022 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.022 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.022 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.022 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.022 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.022 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.023 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.023 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.023 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.023 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.023 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.023 INFO:teuthology.orchestra.run.smithi044.stdout: "description": { 2024-04-21T23:19:10.023 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "description", 2024-04-21T23:19:10.023 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.024 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.024 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.024 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.024 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.024 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.024 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.024 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.024 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.025 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.025 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.025 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.025 INFO:teuthology.orchestra.run.smithi044.stdout: "device_url": { 2024-04-21T23:19:10.025 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_url", 2024-04-21T23:19:10.025 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.025 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.025 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.026 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-21T23:19:10.026 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.026 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.026 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.026 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.026 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.026 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.027 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.027 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.027 INFO:teuthology.orchestra.run.smithi044.stdout: "enabled": { 2024-04-21T23:19:10.027 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enabled", 2024-04-21T23:19:10.027 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.027 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.027 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.028 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.028 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.028 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.028 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.028 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.028 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.028 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.028 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.028 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.029 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-04-21T23:19:10.029 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-04-21T23:19:10.029 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.029 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.029 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.029 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "24", 2024-04-21T23:19:10.030 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "8", 2024-04-21T23:19:10.030 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.030 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.030 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.030 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.030 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.030 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.030 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.031 INFO:teuthology.orchestra.run.smithi044.stdout: "last_opt_revision": { 2024-04-21T23:19:10.031 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "last_opt_revision", 2024-04-21T23:19:10.031 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.031 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.031 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.032 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1", 2024-04-21T23:19:10.032 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.032 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.032 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.032 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.032 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.032 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.032 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.033 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.033 INFO:teuthology.orchestra.run.smithi044.stdout: "leaderboard": { 2024-04-21T23:19:10.033 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "leaderboard", 2024-04-21T23:19:10.033 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.033 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.033 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.034 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.034 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.034 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.034 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.034 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.034 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.034 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.035 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.035 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.035 INFO:teuthology.orchestra.run.smithi044.stdout: "leaderboard_description": { 2024-04-21T23:19:10.035 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "leaderboard_description", 2024-04-21T23:19:10.035 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.035 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.036 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.036 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.036 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.036 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.036 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.036 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.036 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.037 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.037 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.037 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.037 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.037 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.037 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.037 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.037 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.037 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.038 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.038 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.038 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.038 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.038 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.038 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.038 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.038 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.038 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.039 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.039 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.039 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.039 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.039 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.039 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.039 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.040 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.040 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.040 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.040 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.040 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.041 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.041 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.041 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.041 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.041 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.042 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.042 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.042 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.042 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.042 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.042 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.042 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.042 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.043 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.043 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.043 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.043 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.043 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.043 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.043 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.043 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.044 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.044 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.044 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.044 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.044 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.044 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.044 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.044 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.045 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.045 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.045 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.045 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.045 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.045 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.045 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.045 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.046 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.046 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.046 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.046 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.046 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.046 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.046 INFO:teuthology.orchestra.run.smithi044.stdout: "organization": { 2024-04-21T23:19:10.046 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "organization", 2024-04-21T23:19:10.047 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.047 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.047 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.047 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.047 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.047 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.047 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.047 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.048 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.048 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.048 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.048 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.048 INFO:teuthology.orchestra.run.smithi044.stdout: "proxy": { 2024-04-21T23:19:10.048 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "proxy", 2024-04-21T23:19:10.048 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.048 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.049 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.049 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.049 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.049 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.049 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.049 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.049 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.049 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.049 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.050 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.050 INFO:teuthology.orchestra.run.smithi044.stdout: "url": { 2024-04-21T23:19:10.050 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "url", 2024-04-21T23:19:10.050 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.050 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.050 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.050 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-21T23:19:10.051 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.051 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.051 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.051 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.051 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.051 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.051 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.051 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.051 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.052 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.052 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.052 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "test_orchestrator", 2024-04-21T23:19:10.052 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.052 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.052 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.052 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.052 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.052 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.053 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.053 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.053 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.053 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.053 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.053 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.053 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.053 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.053 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.054 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.054 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.054 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.054 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.054 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.054 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.054 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.054 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.055 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.055 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.055 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.055 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.055 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.055 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.055 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.055 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.055 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.056 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.056 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.056 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.056 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.056 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.056 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.056 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.056 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.057 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.057 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.057 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.057 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.057 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.057 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.057 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.057 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.057 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.058 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.058 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.058 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.058 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.058 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.058 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.058 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.058 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.059 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.059 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.059 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.059 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.059 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.059 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.059 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.059 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.059 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.060 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.060 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.060 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.060 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.060 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.060 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.060 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.060 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.061 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.061 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.061 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "volumes", 2024-04-21T23:19:10.061 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.061 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.061 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.061 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.061 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.061 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.062 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.062 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.062 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.062 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.062 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.062 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.062 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.062 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.063 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.063 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.063 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.063 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.063 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.063 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.063 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.063 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.063 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.064 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.064 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.064 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.064 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.064 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.064 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.064 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.064 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.065 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.065 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.065 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.065 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.065 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.065 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.066 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.066 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.066 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.066 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.066 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.066 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.066 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.066 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.067 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.067 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.067 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.067 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.067 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.067 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.068 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.068 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.068 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.068 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.068 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.068 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.068 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.069 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.069 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.069 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.069 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.069 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.069 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.069 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.069 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.070 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.070 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.070 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.070 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.070 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.070 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.071 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.071 INFO:teuthology.orchestra.run.smithi044.stdout: "max_concurrent_clones": { 2024-04-21T23:19:10.071 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_concurrent_clones", 2024-04-21T23:19:10.071 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.071 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.071 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.071 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "4", 2024-04-21T23:19:10.072 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.072 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.072 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.072 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-21T23:19:10.072 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.072 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.073 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.073 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.073 INFO:teuthology.orchestra.run.smithi044.stdout: "periodic_async_work": { 2024-04-21T23:19:10.073 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "periodic_async_work", 2024-04-21T23:19:10.073 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.073 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.074 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.074 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.074 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.074 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.074 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.074 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Periodically check for async work", 2024-04-21T23:19:10.075 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.075 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.075 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.075 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.075 INFO:teuthology.orchestra.run.smithi044.stdout: "snapshot_clone_delay": { 2024-04-21T23:19:10.075 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snapshot_clone_delay", 2024-04-21T23:19:10.075 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.076 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.076 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.076 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-04-21T23:19:10.076 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.076 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.076 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.076 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-21T23:19:10.076 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.076 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.077 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.077 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.077 INFO:teuthology.orchestra.run.smithi044.stdout: "snapshot_clone_no_wait": { 2024-04-21T23:19:10.077 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snapshot_clone_no_wait", 2024-04-21T23:19:10.077 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.077 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.077 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.077 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.078 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.078 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.078 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.078 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-21T23:19:10.078 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.078 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.078 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.078 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.079 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.079 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.079 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.079 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix", 2024-04-21T23:19:10.079 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.079 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.079 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.079 INFO:teuthology.orchestra.run.smithi044.stdout: "discovery_interval": { 2024-04-21T23:19:10.079 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "discovery_interval", 2024-04-21T23:19:10.080 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-04-21T23:19:10.080 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.080 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.080 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "100", 2024-04-21T23:19:10.080 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.080 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.080 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.080 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.080 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.081 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.081 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.081 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.081 INFO:teuthology.orchestra.run.smithi044.stdout: "identifier": { 2024-04-21T23:19:10.081 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "identifier", 2024-04-21T23:19:10.081 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.081 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.081 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.082 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.082 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.082 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.082 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.082 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.082 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.082 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.082 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.083 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.083 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-04-21T23:19:10.083 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-04-21T23:19:10.083 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.083 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.083 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.083 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-04-21T23:19:10.083 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.084 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.084 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.084 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.084 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.084 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.084 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.084 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.084 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.084 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.085 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.085 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.085 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.085 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.085 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.085 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.085 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.085 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.086 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.086 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.086 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.086 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.086 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.086 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.086 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.086 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.086 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.087 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.087 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.087 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.087 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.087 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.087 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.087 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.087 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.088 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.088 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.088 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.088 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.088 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.088 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.088 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.088 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.088 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.089 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.089 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.089 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.089 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.089 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.089 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.089 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.089 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.089 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.090 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.090 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.090 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.090 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.090 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.090 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.090 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.090 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.091 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.091 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.091 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.091 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.091 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.091 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.091 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.091 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.091 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.092 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.092 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.092 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.092 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.092 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.092 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.092 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.092 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.092 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_host": { 2024-04-21T23:19:10.093 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_host", 2024-04-21T23:19:10.093 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.093 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.093 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.093 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.093 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.093 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.093 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.094 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.094 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.094 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.094 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.094 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.094 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_port": { 2024-04-21T23:19:10.094 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_port", 2024-04-21T23:19:10.094 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.094 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.095 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.095 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10051", 2024-04-21T23:19:10.095 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.095 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.095 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.095 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.095 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.095 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.096 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.096 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.096 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_sender": { 2024-04-21T23:19:10.096 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_sender", 2024-04-21T23:19:10.096 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.096 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.096 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.096 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-21T23:19:10.097 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.097 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.097 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.097 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.097 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.097 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.097 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.097 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.098 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.098 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.098 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-21T23:19:10.098 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.098 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.098 INFO:teuthology.orchestra.run.smithi044.stdout: "gid": 25093, 2024-04-21T23:19:10.099 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "z", 2024-04-21T23:19:10.099 INFO:teuthology.orchestra.run.smithi044.stdout: "mgr_features": 4540701547738038271, 2024-04-21T23:19:10.099 INFO:teuthology.orchestra.run.smithi044.stdout: "available_modules": [ 2024-04-21T23:19:10.099 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.099 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "alerts", 2024-04-21T23:19:10.099 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.099 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.099 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.100 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-04-21T23:19:10.100 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-04-21T23:19:10.100 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.100 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.100 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.100 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-04-21T23:19:10.101 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.101 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.101 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.101 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How frequently to reexamine health status", 2024-04-21T23:19:10.101 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.101 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.101 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.101 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.102 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.102 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.102 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.102 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.102 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.102 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.103 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.103 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.103 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.103 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.103 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.103 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.103 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.103 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.104 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.104 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.104 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.104 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.104 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.104 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.105 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.105 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.105 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.105 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.105 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.105 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.106 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.106 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.106 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.106 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.106 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.106 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.107 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.107 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.107 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.107 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.107 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.107 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.107 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.107 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.107 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.108 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.108 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.108 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.108 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.108 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.108 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.108 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.108 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.109 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.109 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.109 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.109 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.109 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.109 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.109 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.110 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.110 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.110 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.110 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.110 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.111 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.111 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.111 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.111 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.111 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.111 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.111 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.112 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.112 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.112 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_destination": { 2024-04-21T23:19:10.112 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_destination", 2024-04-21T23:19:10.112 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.112 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.112 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.112 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.113 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.113 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.113 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.113 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Email address to send alerts to", 2024-04-21T23:19:10.113 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.113 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.113 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.113 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.113 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_from_name": { 2024-04-21T23:19:10.114 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_from_name", 2024-04-21T23:19:10.114 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.114 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.114 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.114 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "Ceph", 2024-04-21T23:19:10.114 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.114 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.114 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.115 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Email From: name", 2024-04-21T23:19:10.115 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.115 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.115 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.115 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.115 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_host": { 2024-04-21T23:19:10.115 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_host", 2024-04-21T23:19:10.115 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.116 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.116 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.116 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.116 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.116 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.116 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.116 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP server", 2024-04-21T23:19:10.116 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.116 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.117 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.117 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.117 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_password": { 2024-04-21T23:19:10.117 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_password", 2024-04-21T23:19:10.117 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.117 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.117 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.117 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.117 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.118 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.118 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.118 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Password to authenticate with", 2024-04-21T23:19:10.118 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.118 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.118 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.118 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.118 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_port": { 2024-04-21T23:19:10.119 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_port", 2024-04-21T23:19:10.119 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.119 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.119 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.119 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "465", 2024-04-21T23:19:10.119 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.119 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.119 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.120 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP port", 2024-04-21T23:19:10.120 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.120 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.120 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.120 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.120 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_sender": { 2024-04-21T23:19:10.120 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_sender", 2024-04-21T23:19:10.120 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.120 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.121 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.121 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.121 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.121 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.121 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.121 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP envelope sender", 2024-04-21T23:19:10.121 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.121 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.122 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.122 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.122 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_ssl": { 2024-04-21T23:19:10.122 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_ssl", 2024-04-21T23:19:10.122 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.122 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.122 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.122 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.123 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.123 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.123 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.123 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-21T23:19:10.123 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.123 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.123 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.123 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.124 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_user": { 2024-04-21T23:19:10.124 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_user", 2024-04-21T23:19:10.124 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.124 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.124 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.124 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.124 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.124 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.125 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.125 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "User to authenticate as", 2024-04-21T23:19:10.125 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.125 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.125 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.125 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.125 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.125 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.126 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.126 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "balancer", 2024-04-21T23:19:10.126 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.126 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.126 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.126 INFO:teuthology.orchestra.run.smithi044.stdout: "active": { 2024-04-21T23:19:10.126 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "active", 2024-04-21T23:19:10.126 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.127 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.127 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.127 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.127 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.127 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.127 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.127 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically balance PGs across cluster", 2024-04-21T23:19:10.127 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.127 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.128 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.128 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.128 INFO:teuthology.orchestra.run.smithi044.stdout: "begin_time": { 2024-04-21T23:19:10.128 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "begin_time", 2024-04-21T23:19:10.128 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.128 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.128 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.128 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0000", 2024-04-21T23:19:10.129 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.129 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.129 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.129 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "beginning time of day to automatically balance", 2024-04-21T23:19:10.129 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T23:19:10.129 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.129 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.129 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.130 INFO:teuthology.orchestra.run.smithi044.stdout: "begin_weekday": { 2024-04-21T23:19:10.130 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "begin_weekday", 2024-04-21T23:19:10.130 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-04-21T23:19:10.130 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.130 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.130 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-04-21T23:19:10.130 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0", 2024-04-21T23:19:10.130 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "6", 2024-04-21T23:19:10.131 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.131 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-21T23:19:10.131 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T23:19:10.131 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.131 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.131 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.131 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_max_iterations": { 2024-04-21T23:19:10.131 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_max_iterations", 2024-04-21T23:19:10.132 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-04-21T23:19:10.132 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.132 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.132 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "25", 2024-04-21T23:19:10.132 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-04-21T23:19:10.132 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "250", 2024-04-21T23:19:10.132 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.133 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-21T23:19:10.133 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.133 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.133 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.133 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.133 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_metrics": { 2024-04-21T23:19:10.133 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_metrics", 2024-04-21T23:19:10.133 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.133 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.134 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.134 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "pgs,objects,bytes", 2024-04-21T23:19:10.134 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.134 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.134 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.135 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-21T23:19:10.135 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-21T23:19:10.135 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.135 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.135 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.135 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_step": { 2024-04-21T23:19:10.135 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_step", 2024-04-21T23:19:10.135 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:10.135 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.136 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.136 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0.5", 2024-04-21T23:19:10.136 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0.001", 2024-04-21T23:19:10.136 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "0.999", 2024-04-21T23:19:10.136 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.136 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "aggressiveness of optimization", 2024-04-21T23:19:10.137 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-21T23:19:10.137 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.137 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.137 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.137 INFO:teuthology.orchestra.run.smithi044.stdout: "end_time": { 2024-04-21T23:19:10.137 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "end_time", 2024-04-21T23:19:10.137 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.137 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.138 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.138 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "2359", 2024-04-21T23:19:10.138 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.138 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.138 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.138 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "ending time of day to automatically balance", 2024-04-21T23:19:10.139 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T23:19:10.139 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.139 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.139 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.139 INFO:teuthology.orchestra.run.smithi044.stdout: "end_weekday": { 2024-04-21T23:19:10.139 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "end_weekday", 2024-04-21T23:19:10.139 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-04-21T23:19:10.140 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.140 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.140 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-04-21T23:19:10.140 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0", 2024-04-21T23:19:10.140 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "6", 2024-04-21T23:19:10.140 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.140 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-21T23:19:10.141 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T23:19:10.141 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.141 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.141 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.141 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.141 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.141 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.142 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.142 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.142 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.142 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.142 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.142 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.143 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.143 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.143 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.143 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.143 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.143 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.143 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.144 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.144 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.144 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.144 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.144 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.144 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.144 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.145 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.145 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.145 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.145 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.145 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.145 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.145 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.145 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.146 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.146 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.146 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.146 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.146 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.146 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.146 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.146 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.147 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.147 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.147 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.147 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.147 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.147 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.147 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.147 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.147 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.148 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.148 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.148 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.148 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.148 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.148 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.148 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.148 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.149 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.149 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.149 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.149 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.149 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.149 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.149 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.149 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.149 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.150 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.150 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.150 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.150 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.150 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.150 INFO:teuthology.orchestra.run.smithi044.stdout: "min_score": { 2024-04-21T23:19:10.150 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "min_score", 2024-04-21T23:19:10.150 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:10.151 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.151 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.151 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-04-21T23:19:10.151 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.151 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.151 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.151 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-21T23:19:10.151 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.151 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.152 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.152 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.152 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": { 2024-04-21T23:19:10.152 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mode", 2024-04-21T23:19:10.152 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.152 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.152 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.152 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "upmap", 2024-04-21T23:19:10.152 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.153 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.153 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.153 INFO:teuthology.orchestra.run.smithi044.stdout: "crush-compat", 2024-04-21T23:19:10.153 INFO:teuthology.orchestra.run.smithi044.stdout: "none", 2024-04-21T23:19:10.153 INFO:teuthology.orchestra.run.smithi044.stdout: "read", 2024-04-21T23:19:10.153 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap", 2024-04-21T23:19:10.153 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap-read" 2024-04-21T23:19:10.153 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.154 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Balancer mode", 2024-04-21T23:19:10.154 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.154 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.154 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.154 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.154 INFO:teuthology.orchestra.run.smithi044.stdout: "pool_ids": { 2024-04-21T23:19:10.154 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pool_ids", 2024-04-21T23:19:10.154 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.155 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.155 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.155 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.155 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.155 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.155 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.155 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-21T23:19:10.155 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.155 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.156 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.156 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.156 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-04-21T23:19:10.156 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-04-21T23:19:10.156 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.156 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.156 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.156 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-04-21T23:19:10.157 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.157 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.157 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.157 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-21T23:19:10.157 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.157 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.157 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.157 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.157 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap_max_deviation": { 2024-04-21T23:19:10.158 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "upmap_max_deviation", 2024-04-21T23:19:10.158 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.158 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.158 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.158 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-04-21T23:19:10.158 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-04-21T23:19:10.158 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.158 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.159 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-21T23:19:10.159 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-21T23:19:10.159 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.159 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.159 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.159 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap_max_optimizations": { 2024-04-21T23:19:10.159 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "upmap_max_optimizations", 2024-04-21T23:19:10.159 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-04-21T23:19:10.159 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.160 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.160 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:10.160 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.160 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.160 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.160 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-21T23:19:10.160 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.160 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.161 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.161 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.161 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.161 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.161 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.161 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cephadm", 2024-04-21T23:19:10.161 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.161 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.161 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.162 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_down_multiplier": { 2024-04-21T23:19:10.162 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_down_multiplier", 2024-04-21T23:19:10.162 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:10.162 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.162 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.162 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3.0", 2024-04-21T23:19:10.162 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.162 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.162 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.163 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-21T23:19:10.163 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.163 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.163 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.163 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.163 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_refresh_rate": { 2024-04-21T23:19:10.163 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_refresh_rate", 2024-04-21T23:19:10.163 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.164 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.164 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.164 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "20", 2024-04-21T23:19:10.164 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.164 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.164 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.164 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-21T23:19:10.164 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.165 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.165 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.165 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.165 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_starting_port": { 2024-04-21T23:19:10.165 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_starting_port", 2024-04-21T23:19:10.165 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.166 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.166 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.166 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "4721", 2024-04-21T23:19:10.166 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.166 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.166 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.166 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-21T23:19:10.166 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.166 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.167 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.167 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.167 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_ptrace": { 2024-04-21T23:19:10.167 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_ptrace", 2024-04-21T23:19:10.167 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.167 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.168 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.168 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.168 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.168 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.168 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.168 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-21T23:19:10.168 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-21T23:19:10.169 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.169 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.169 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.169 INFO:teuthology.orchestra.run.smithi044.stdout: "autotune_interval": { 2024-04-21T23:19:10.169 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "autotune_interval", 2024-04-21T23:19:10.169 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.170 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.170 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.170 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-04-21T23:19:10.170 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.170 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.170 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.170 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-21T23:19:10.170 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.171 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.171 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.171 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.171 INFO:teuthology.orchestra.run.smithi044.stdout: "autotune_memory_target_ratio": { 2024-04-21T23:19:10.171 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "autotune_memory_target_ratio", 2024-04-21T23:19:10.171 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:10.172 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.172 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.172 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0.7", 2024-04-21T23:19:10.172 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.172 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.172 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.172 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-21T23:19:10.173 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.173 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.173 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.173 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.173 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm_log_destination": { 2024-04-21T23:19:10.173 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cephadm_log_destination", 2024-04-21T23:19:10.173 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.174 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.174 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.174 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.174 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.174 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.174 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.175 INFO:teuthology.orchestra.run.smithi044.stdout: "file", 2024-04-21T23:19:10.175 INFO:teuthology.orchestra.run.smithi044.stdout: "file,syslog", 2024-04-21T23:19:10.175 INFO:teuthology.orchestra.run.smithi044.stdout: "syslog" 2024-04-21T23:19:10.175 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.175 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-21T23:19:10.175 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.176 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.176 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.176 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.176 INFO:teuthology.orchestra.run.smithi044.stdout: "cgroups_split": { 2024-04-21T23:19:10.176 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cgroups_split", 2024-04-21T23:19:10.176 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.177 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.177 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.177 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.177 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.177 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.177 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.178 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-21T23:19:10.178 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.178 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.178 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.178 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.178 INFO:teuthology.orchestra.run.smithi044.stdout: "config_checks_enabled": { 2024-04-21T23:19:10.178 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_checks_enabled", 2024-04-21T23:19:10.178 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.179 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.179 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.179 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.179 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.179 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.179 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.179 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-21T23:19:10.179 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.180 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.180 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.180 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.180 INFO:teuthology.orchestra.run.smithi044.stdout: "config_dashboard": { 2024-04-21T23:19:10.180 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_dashboard", 2024-04-21T23:19:10.180 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.180 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.180 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.181 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.181 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.181 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.181 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.181 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-21T23:19:10.181 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.181 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.181 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.181 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.182 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_alertmanager": { 2024-04-21T23:19:10.182 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_alertmanager", 2024-04-21T23:19:10.182 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.182 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.182 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.182 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-21T23:19:10.182 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.182 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.182 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.183 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-04-21T23:19:10.183 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.183 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.183 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.183 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.183 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_base": { 2024-04-21T23:19:10.183 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_base", 2024-04-21T23:19:10.183 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.184 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.184 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.184 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-21T23:19:10.186 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.186 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.186 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.186 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Container image name, without the tag", 2024-04-21T23:19:10.186 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.186 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.186 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.186 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.187 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_elasticsearch": { 2024-04-21T23:19:10.187 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_elasticsearch", 2024-04-21T23:19:10.187 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.187 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.187 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.187 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-21T23:19:10.187 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.187 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.187 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.188 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "elasticsearch container image", 2024-04-21T23:19:10.188 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.188 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.188 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.188 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.188 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_grafana": { 2024-04-21T23:19:10.188 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_grafana", 2024-04-21T23:19:10.188 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.189 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.189 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.189 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-21T23:19:10.189 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.189 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.189 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.189 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-04-21T23:19:10.189 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.189 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.190 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.190 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.190 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_haproxy": { 2024-04-21T23:19:10.190 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_haproxy", 2024-04-21T23:19:10.190 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.190 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.190 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.190 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-21T23:19:10.191 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.191 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.191 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.191 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "HAproxy container image", 2024-04-21T23:19:10.191 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.191 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.191 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.191 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.191 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_agent": { 2024-04-21T23:19:10.192 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_agent", 2024-04-21T23:19:10.192 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.192 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.192 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.192 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-21T23:19:10.192 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.192 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.193 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.193 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger agent container image", 2024-04-21T23:19:10.193 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.193 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.193 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.193 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.193 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_collector": { 2024-04-21T23:19:10.193 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_collector", 2024-04-21T23:19:10.193 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.194 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.194 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.194 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-21T23:19:10.194 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.194 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.194 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.194 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger collector container image", 2024-04-21T23:19:10.194 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.195 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.195 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.195 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.195 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_query": { 2024-04-21T23:19:10.195 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_query", 2024-04-21T23:19:10.195 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.195 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.195 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.195 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-21T23:19:10.196 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.196 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.196 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.196 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger query container image", 2024-04-21T23:19:10.196 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.196 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.196 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.196 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.197 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_keepalived": { 2024-04-21T23:19:10.197 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_keepalived", 2024-04-21T23:19:10.197 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.197 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.197 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.197 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-21T23:19:10.197 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.197 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.197 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.198 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Keepalived container image", 2024-04-21T23:19:10.198 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.198 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.198 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.198 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.198 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_loki": { 2024-04-21T23:19:10.198 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_loki", 2024-04-21T23:19:10.198 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.199 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.199 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.199 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-21T23:19:10.199 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.199 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.199 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.199 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Loki container image", 2024-04-21T23:19:10.199 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.199 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.200 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.200 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.200 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_node_exporter": { 2024-04-21T23:19:10.200 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_node_exporter", 2024-04-21T23:19:10.200 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.200 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.200 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.200 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-21T23:19:10.201 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.201 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.201 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.201 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-04-21T23:19:10.201 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.201 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.201 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.201 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.201 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_nvmeof": { 2024-04-21T23:19:10.202 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_nvmeof", 2024-04-21T23:19:10.202 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.202 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.202 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.202 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-21T23:19:10.202 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.202 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.202 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.202 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Nvme-of container image", 2024-04-21T23:19:10.203 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.203 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.203 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.203 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.203 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_prometheus": { 2024-04-21T23:19:10.203 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_prometheus", 2024-04-21T23:19:10.203 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.203 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.204 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.204 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-21T23:19:10.204 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.204 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.204 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.204 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-04-21T23:19:10.204 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.204 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.205 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.205 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.205 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_promtail": { 2024-04-21T23:19:10.205 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_promtail", 2024-04-21T23:19:10.205 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.205 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.206 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.206 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-21T23:19:10.206 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.206 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.206 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.206 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Promtail container image", 2024-04-21T23:19:10.206 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.206 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.206 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.207 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.207 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_samba": { 2024-04-21T23:19:10.207 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_samba", 2024-04-21T23:19:10.207 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.207 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.207 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.208 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-21T23:19:10.208 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.208 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.208 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.208 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Samba/SMB container image", 2024-04-21T23:19:10.208 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.208 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.208 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.209 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.209 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_snmp_gateway": { 2024-04-21T23:19:10.209 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_snmp_gateway", 2024-04-21T23:19:10.209 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.209 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.209 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.210 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-21T23:19:10.210 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.210 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.210 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.210 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SNMP Gateway container image", 2024-04-21T23:19:10.210 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.210 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.210 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.211 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.211 INFO:teuthology.orchestra.run.smithi044.stdout: "container_init": { 2024-04-21T23:19:10.211 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_init", 2024-04-21T23:19:10.211 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.211 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.211 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.211 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.211 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.212 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.212 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.212 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Run podman/docker with `--init`", 2024-04-21T23:19:10.212 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.212 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.212 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.212 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.212 INFO:teuthology.orchestra.run.smithi044.stdout: "daemon_cache_timeout": { 2024-04-21T23:19:10.213 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "daemon_cache_timeout", 2024-04-21T23:19:10.213 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.213 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.213 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.213 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-04-21T23:19:10.213 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.213 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.214 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.214 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-21T23:19:10.214 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.214 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.214 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.214 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.214 INFO:teuthology.orchestra.run.smithi044.stdout: "default_cephadm_command_timeout": { 2024-04-21T23:19:10.215 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default_cephadm_command_timeout", 2024-04-21T23:19:10.215 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.215 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.215 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.215 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "900", 2024-04-21T23:19:10.215 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.215 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.215 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.216 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-21T23:19:10.216 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.216 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.216 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.216 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.216 INFO:teuthology.orchestra.run.smithi044.stdout: "default_registry": { 2024-04-21T23:19:10.216 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default_registry", 2024-04-21T23:19:10.216 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.217 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.217 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.217 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io", 2024-04-21T23:19:10.217 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.217 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.217 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.217 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-21T23:19:10.217 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.218 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.218 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.218 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.218 INFO:teuthology.orchestra.run.smithi044.stdout: "device_cache_timeout": { 2024-04-21T23:19:10.218 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_cache_timeout", 2024-04-21T23:19:10.218 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.218 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.219 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.219 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1800", 2024-04-21T23:19:10.219 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.219 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.219 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.219 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache device inventory", 2024-04-21T23:19:10.219 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.220 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.220 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.220 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.220 INFO:teuthology.orchestra.run.smithi044.stdout: "device_enhanced_scan": { 2024-04-21T23:19:10.220 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_enhanced_scan", 2024-04-21T23:19:10.220 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.221 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.221 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.221 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.221 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.221 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.221 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.221 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-21T23:19:10.222 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.222 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.222 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.222 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.222 INFO:teuthology.orchestra.run.smithi044.stdout: "facts_cache_timeout": { 2024-04-21T23:19:10.222 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "facts_cache_timeout", 2024-04-21T23:19:10.222 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.223 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.223 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.223 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-04-21T23:19:10.223 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.223 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.223 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.223 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache host facts data", 2024-04-21T23:19:10.224 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.224 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.224 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.224 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.224 INFO:teuthology.orchestra.run.smithi044.stdout: "grafana_dashboards_path": { 2024-04-21T23:19:10.224 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "grafana_dashboards_path", 2024-04-21T23:19:10.224 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.225 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.225 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.225 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-21T23:19:10.225 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.225 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.225 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.226 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-21T23:19:10.226 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.226 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.226 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.226 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.226 INFO:teuthology.orchestra.run.smithi044.stdout: "host_check_interval": { 2024-04-21T23:19:10.227 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "host_check_interval", 2024-04-21T23:19:10.227 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.227 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.227 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.227 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-04-21T23:19:10.227 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.227 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.228 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.228 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to perform a host check", 2024-04-21T23:19:10.228 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.228 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.228 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.228 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.228 INFO:teuthology.orchestra.run.smithi044.stdout: "hw_monitoring": { 2024-04-21T23:19:10.229 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "hw_monitoring", 2024-04-21T23:19:10.229 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.229 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.229 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.229 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.229 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.230 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.230 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.230 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-21T23:19:10.230 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.230 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.230 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.231 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.231 INFO:teuthology.orchestra.run.smithi044.stdout: "inventory_list_all": { 2024-04-21T23:19:10.231 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "inventory_list_all", 2024-04-21T23:19:10.231 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.231 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.231 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.231 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.232 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.232 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.232 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.232 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-21T23:19:10.232 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.232 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.232 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.233 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.233 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.233 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.233 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.233 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.233 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.233 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.234 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.234 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.234 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.234 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.234 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.234 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.235 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.235 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.235 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.235 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.235 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.235 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.235 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.236 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.236 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.236 INFO:teuthology.orchestra.run.smithi044.stdout: "log_refresh_metadata": { 2024-04-21T23:19:10.236 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_refresh_metadata", 2024-04-21T23:19:10.236 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.236 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.236 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.236 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.237 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.237 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.237 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.237 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-21T23:19:10.237 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.237 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.238 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.238 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.238 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.238 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.238 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.238 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.239 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.239 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.239 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.239 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.239 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.239 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-21T23:19:10.240 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.240 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.240 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.240 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.240 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.240 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.240 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.241 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.241 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.241 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.241 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.241 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.241 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.242 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.242 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.242 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.242 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.242 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.242 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.242 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.243 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.243 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.243 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.243 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.243 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.243 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.244 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.244 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.244 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.244 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.244 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.244 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.244 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.245 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.245 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.245 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.245 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.245 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.245 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.246 INFO:teuthology.orchestra.run.smithi044.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-21T23:19:10.246 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-21T23:19:10.246 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.246 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.246 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.246 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.246 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.247 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.247 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.247 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-21T23:19:10.247 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.247 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.247 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.247 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.248 INFO:teuthology.orchestra.run.smithi044.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-21T23:19:10.248 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-21T23:19:10.248 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.248 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.248 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.248 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "*", 2024-04-21T23:19:10.249 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.249 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.249 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.249 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-21T23:19:10.249 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.249 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.250 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.250 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.250 INFO:teuthology.orchestra.run.smithi044.stdout: "max_count_per_host": { 2024-04-21T23:19:10.250 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_count_per_host", 2024-04-21T23:19:10.250 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.250 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.251 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.251 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:10.251 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.251 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.251 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.251 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "max number of daemons per service per host", 2024-04-21T23:19:10.252 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.252 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.252 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.252 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.252 INFO:teuthology.orchestra.run.smithi044.stdout: "max_osd_draining_count": { 2024-04-21T23:19:10.252 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_osd_draining_count", 2024-04-21T23:19:10.253 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.253 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.253 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.253 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:10.253 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.253 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.253 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.254 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-21T23:19:10.254 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.254 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.254 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.254 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.254 INFO:teuthology.orchestra.run.smithi044.stdout: "migration_current": { 2024-04-21T23:19:10.255 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "migration_current", 2024-04-21T23:19:10.255 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.255 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.255 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.255 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.255 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.255 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.256 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.256 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "internal - do not modify", 2024-04-21T23:19:10.256 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.256 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.256 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.256 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.257 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": { 2024-04-21T23:19:10.257 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mode", 2024-04-21T23:19:10.257 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.257 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.257 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.257 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "root", 2024-04-21T23:19:10.257 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.258 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.258 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.258 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm-package", 2024-04-21T23:19:10.258 INFO:teuthology.orchestra.run.smithi044.stdout: "root" 2024-04-21T23:19:10.258 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.258 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "mode for remote execution of cephadm", 2024-04-21T23:19:10.259 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.259 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.259 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.259 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.259 INFO:teuthology.orchestra.run.smithi044.stdout: "oob_default_addr": { 2024-04-21T23:19:10.259 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "oob_default_addr", 2024-04-21T23:19:10.260 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.260 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.260 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.260 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "169.254.1.1", 2024-04-21T23:19:10.260 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.260 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.260 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.261 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-21T23:19:10.261 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.261 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.261 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.261 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.261 INFO:teuthology.orchestra.run.smithi044.stdout: "prometheus_alerts_path": { 2024-04-21T23:19:10.262 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus_alerts_path", 2024-04-21T23:19:10.262 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.262 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.262 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.262 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-21T23:19:10.262 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.263 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.263 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.263 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-21T23:19:10.263 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.263 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.263 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.263 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.264 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_insecure": { 2024-04-21T23:19:10.264 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_insecure", 2024-04-21T23:19:10.264 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.264 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.264 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.264 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.265 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.265 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.265 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.265 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-21T23:19:10.265 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.265 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.266 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.266 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.266 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_password": { 2024-04-21T23:19:10.266 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_password", 2024-04-21T23:19:10.266 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.266 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.266 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.267 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.267 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.267 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.267 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.267 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-21T23:19:10.267 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.268 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.268 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.268 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.268 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_url": { 2024-04-21T23:19:10.268 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_url", 2024-04-21T23:19:10.268 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.268 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.269 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.269 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.269 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.269 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.269 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.269 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-21T23:19:10.270 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.270 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.270 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.270 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.270 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_username": { 2024-04-21T23:19:10.270 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_username", 2024-04-21T23:19:10.270 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.271 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.271 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.271 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.271 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.271 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.271 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.272 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-21T23:19:10.272 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.272 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.272 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.272 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.272 INFO:teuthology.orchestra.run.smithi044.stdout: "secure_monitoring_stack": { 2024-04-21T23:19:10.272 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "secure_monitoring_stack", 2024-04-21T23:19:10.273 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.273 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.273 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.273 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.273 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.273 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.274 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.274 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-21T23:19:10.274 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.274 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.274 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.274 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.274 INFO:teuthology.orchestra.run.smithi044.stdout: "service_discovery_port": { 2024-04-21T23:19:10.275 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "service_discovery_port", 2024-04-21T23:19:10.275 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.275 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.275 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.275 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8765", 2024-04-21T23:19:10.275 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.276 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.276 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.276 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "cephadm service discovery port", 2024-04-21T23:19:10.276 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.276 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.276 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.276 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.277 INFO:teuthology.orchestra.run.smithi044.stdout: "ssh_config_file": { 2024-04-21T23:19:10.277 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssh_config_file", 2024-04-21T23:19:10.277 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.277 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.277 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.277 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.277 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.278 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.278 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.278 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-21T23:19:10.278 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.278 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.278 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.279 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.279 INFO:teuthology.orchestra.run.smithi044.stdout: "use_agent": { 2024-04-21T23:19:10.279 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "use_agent", 2024-04-21T23:19:10.279 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.279 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.279 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.279 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.280 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.280 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.280 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.280 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-21T23:19:10.280 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.280 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.281 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.281 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.281 INFO:teuthology.orchestra.run.smithi044.stdout: "use_repo_digest": { 2024-04-21T23:19:10.281 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "use_repo_digest", 2024-04-21T23:19:10.281 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.281 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.281 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.282 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.282 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.282 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.282 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.282 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-21T23:19:10.282 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.282 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.283 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.283 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.283 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_failed_host_check": { 2024-04-21T23:19:10.283 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_failed_host_check", 2024-04-21T23:19:10.283 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.283 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.283 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.284 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.284 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.284 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.284 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.284 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if the host check fails", 2024-04-21T23:19:10.284 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.284 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.285 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.285 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.285 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_stray_daemons": { 2024-04-21T23:19:10.285 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_stray_daemons", 2024-04-21T23:19:10.285 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.285 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.286 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.286 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.286 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.286 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.286 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.286 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-21T23:19:10.286 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.287 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.287 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.287 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.287 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_stray_hosts": { 2024-04-21T23:19:10.287 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_stray_hosts", 2024-04-21T23:19:10.287 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.288 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.288 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.288 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.288 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.288 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.288 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.288 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-21T23:19:10.289 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.289 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.289 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.289 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.289 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.289 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.289 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.290 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crash", 2024-04-21T23:19:10.290 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.290 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.290 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.290 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.290 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.291 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.291 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.291 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.291 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.291 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.291 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.291 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.292 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.292 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.292 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.292 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.292 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.292 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.293 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.293 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.293 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.293 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.293 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.293 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.293 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.293 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.294 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.294 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.294 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.294 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.294 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.294 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.294 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.295 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.295 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.295 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.295 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.295 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.295 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.295 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.296 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.296 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.296 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.296 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.296 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.296 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.296 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.296 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.296 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.297 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.297 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.297 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.297 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.297 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.297 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.297 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.297 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.298 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.298 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.298 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.298 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.298 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.298 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.298 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.298 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.298 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.299 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.299 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.299 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.299 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.299 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.299 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.299 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.299 INFO:teuthology.orchestra.run.smithi044.stdout: "retain_interval": { 2024-04-21T23:19:10.300 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "retain_interval", 2024-04-21T23:19:10.300 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.300 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.300 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.300 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "31536000", 2024-04-21T23:19:10.300 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.300 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.300 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.300 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-21T23:19:10.301 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.301 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.301 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.301 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.301 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_recent_interval": { 2024-04-21T23:19:10.301 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_recent_interval", 2024-04-21T23:19:10.301 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.301 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.301 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.302 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1209600", 2024-04-21T23:19:10.302 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.302 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.302 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.302 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-21T23:19:10.302 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.302 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.302 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.303 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.303 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.303 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.303 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.303 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "dashboard", 2024-04-21T23:19:10.303 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.303 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.303 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.303 INFO:teuthology.orchestra.run.smithi044.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-21T23:19:10.304 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-21T23:19:10.304 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.304 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.304 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.304 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:10.304 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.304 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.304 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.305 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.305 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.305 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.305 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.305 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.305 INFO:teuthology.orchestra.run.smithi044.stdout: "ALERTMANAGER_API_HOST": { 2024-04-21T23:19:10.305 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-21T23:19:10.305 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.305 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.306 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.306 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.306 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.306 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.306 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.306 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.306 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.306 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.306 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.307 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.307 INFO:teuthology.orchestra.run.smithi044.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-21T23:19:10.307 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-21T23:19:10.307 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.307 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.307 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.307 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.307 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.308 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.308 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.308 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.308 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.308 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.308 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.308 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.308 INFO:teuthology.orchestra.run.smithi044.stdout: "AUDIT_API_ENABLED": { 2024-04-21T23:19:10.308 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "AUDIT_API_ENABLED", 2024-04-21T23:19:10.309 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.309 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.309 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.309 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.309 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.309 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.309 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.309 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.309 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.310 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.310 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.310 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.310 INFO:teuthology.orchestra.run.smithi044.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-21T23:19:10.310 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-21T23:19:10.310 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.310 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.311 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.311 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.311 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.311 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.311 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.311 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.311 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.311 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.312 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.312 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.312 INFO:teuthology.orchestra.run.smithi044.stdout: "ENABLE_BROWSABLE_API": { 2024-04-21T23:19:10.312 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-21T23:19:10.312 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.312 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.312 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.312 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.313 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.313 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.313 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.313 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.313 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.313 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.313 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.313 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.314 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-21T23:19:10.314 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-21T23:19:10.314 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.314 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.314 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.314 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.314 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.314 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.315 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.315 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.315 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.315 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.315 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.315 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.315 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-21T23:19:10.315 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-21T23:19:10.316 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.316 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.316 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.316 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.316 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.316 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.316 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.316 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.317 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.317 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.317 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.317 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.317 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-21T23:19:10.317 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-21T23:19:10.317 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.317 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.318 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.318 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.318 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.318 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.318 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.318 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.318 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.319 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.319 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.319 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.319 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-21T23:19:10.319 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-21T23:19:10.319 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.319 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.319 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.320 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.320 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.320 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.320 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.320 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.320 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.320 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.321 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.321 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.321 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-21T23:19:10.321 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-21T23:19:10.321 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.321 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.321 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.321 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.321 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.322 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.322 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.322 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.322 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.322 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.322 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.322 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.322 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-21T23:19:10.323 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-21T23:19:10.323 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.323 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.323 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.323 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.323 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.323 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.323 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.324 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.324 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.324 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.324 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.324 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.324 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-21T23:19:10.324 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-21T23:19:10.324 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.325 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.325 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.325 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.325 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.325 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.325 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.325 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.325 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.326 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.326 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.326 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.326 INFO:teuthology.orchestra.run.smithi044.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-21T23:19:10.326 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-21T23:19:10.326 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.326 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.326 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.327 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.327 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.327 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.327 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.327 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.327 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.327 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.327 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.327 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.328 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_PASSWORD": { 2024-04-21T23:19:10.328 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-21T23:19:10.328 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.328 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.328 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.328 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2024-04-21T23:19:10.328 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.329 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.329 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.329 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.329 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.329 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.329 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.329 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.329 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-21T23:19:10.329 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-21T23:19:10.330 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.330 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.330 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.330 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.330 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.330 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.330 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.330 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.331 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.331 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.331 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.331 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.331 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_URL": { 2024-04-21T23:19:10.331 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_URL", 2024-04-21T23:19:10.331 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.331 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.331 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.332 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.332 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.332 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.332 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.332 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.332 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.332 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.332 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.333 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.333 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_USERNAME": { 2024-04-21T23:19:10.333 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-21T23:19:10.333 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.333 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.333 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.333 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2024-04-21T23:19:10.334 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.334 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.334 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.334 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.334 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.334 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.334 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.334 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.335 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-21T23:19:10.335 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-21T23:19:10.335 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.335 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.335 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.335 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.335 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.335 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.335 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.336 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.336 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.336 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.336 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.336 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.336 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-21T23:19:10.336 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-21T23:19:10.336 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.337 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.337 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.337 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.337 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.337 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.337 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.337 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.337 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.338 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.338 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.338 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.338 INFO:teuthology.orchestra.run.smithi044.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-21T23:19:10.338 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-21T23:19:10.338 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.338 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.339 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.339 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.339 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.339 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.339 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.339 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.339 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.339 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.339 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.340 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.340 INFO:teuthology.orchestra.run.smithi044.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-21T23:19:10.340 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-21T23:19:10.340 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.340 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.340 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.340 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.340 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.341 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.341 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.341 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.341 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.341 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.341 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.341 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.341 INFO:teuthology.orchestra.run.smithi044.stdout: "PROMETHEUS_API_HOST": { 2024-04-21T23:19:10.342 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-21T23:19:10.342 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.342 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.342 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.342 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.342 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.342 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.342 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.343 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.343 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.343 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.343 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.343 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.343 INFO:teuthology.orchestra.run.smithi044.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-21T23:19:10.343 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-21T23:19:10.343 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.343 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.344 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.344 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.344 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.344 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.344 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.344 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.344 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.344 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.345 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.345 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.345 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-21T23:19:10.345 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-21T23:19:10.345 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.345 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.345 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.345 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.346 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.346 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.346 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.346 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.346 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.346 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.346 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.346 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.346 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-21T23:19:10.347 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-21T23:19:10.347 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.347 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.347 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.347 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.347 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.347 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.347 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.348 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.348 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.348 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.348 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.348 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.348 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-21T23:19:10.348 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-21T23:19:10.348 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.349 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.349 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.349 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.349 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.349 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.349 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.349 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.349 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.350 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.350 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.350 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.350 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-21T23:19:10.350 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-21T23:19:10.350 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.350 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.350 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.350 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.351 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.351 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.351 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.351 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.351 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.351 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.351 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.351 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.352 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-21T23:19:10.352 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-21T23:19:10.352 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.352 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.352 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.352 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.352 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.352 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.353 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.353 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.353 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.353 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.353 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.353 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.353 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-21T23:19:10.353 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-21T23:19:10.354 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.354 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.354 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.354 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.354 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.354 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.354 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.354 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.354 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.355 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.355 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.355 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.355 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-21T23:19:10.355 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-21T23:19:10.355 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.355 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.355 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.355 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.356 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.356 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.356 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.356 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.356 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.356 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.356 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.356 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.357 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_ENABLED": { 2024-04-21T23:19:10.357 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-21T23:19:10.357 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.357 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.357 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.357 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.357 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.357 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.357 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.358 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.358 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.358 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.358 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.358 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.358 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-21T23:19:10.358 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-21T23:19:10.358 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.359 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.359 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.359 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-21T23:19:10.359 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.359 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.359 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.359 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.359 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.360 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.360 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.360 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.360 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-21T23:19:10.360 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-21T23:19:10.360 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.360 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.360 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.361 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:10.361 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.361 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.361 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.361 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.361 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.361 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.361 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.362 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.362 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-21T23:19:10.362 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-21T23:19:10.362 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.362 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.362 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.362 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8", 2024-04-21T23:19:10.362 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.362 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.363 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.363 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.363 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.363 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.363 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.363 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.363 INFO:teuthology.orchestra.run.smithi044.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-21T23:19:10.363 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-21T23:19:10.364 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.364 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.364 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.364 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "45", 2024-04-21T23:19:10.364 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.364 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.364 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.364 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.364 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.365 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.365 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.365 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.365 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_ACCESS_KEY": { 2024-04-21T23:19:10.365 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-21T23:19:10.365 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.365 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.365 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.365 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.366 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.366 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.366 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.366 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.366 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.366 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.366 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.366 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.367 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-21T23:19:10.367 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-21T23:19:10.367 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.367 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.367 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.367 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2024-04-21T23:19:10.367 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.367 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.368 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.368 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.368 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.368 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.368 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.368 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.368 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_SECRET_KEY": { 2024-04-21T23:19:10.368 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-21T23:19:10.368 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.369 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.369 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.369 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.369 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.369 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.369 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.369 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.369 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.370 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.370 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.370 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.370 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_SSL_VERIFY": { 2024-04-21T23:19:10.370 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-21T23:19:10.370 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.370 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.370 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.370 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.371 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.371 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.371 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.371 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.371 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.371 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.371 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.371 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.372 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-21T23:19:10.372 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-21T23:19:10.372 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.372 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.372 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.372 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-04-21T23:19:10.372 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.372 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.372 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.373 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.373 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.373 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.373 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.373 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.373 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-21T23:19:10.373 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-21T23:19:10.373 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.373 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.374 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.374 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:10.374 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.374 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.374 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.374 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.374 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.374 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.375 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.375 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.375 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-21T23:19:10.375 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-21T23:19:10.375 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.375 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.375 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.375 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-04-21T23:19:10.376 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.376 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.376 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.376 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.376 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.376 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.376 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.376 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.376 INFO:teuthology.orchestra.run.smithi044.stdout: "cross_origin_url": { 2024-04-21T23:19:10.377 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cross_origin_url", 2024-04-21T23:19:10.377 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.377 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.377 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.377 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.377 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.377 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.377 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.377 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.378 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.378 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.378 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.378 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.378 INFO:teuthology.orchestra.run.smithi044.stdout: "crt_file": { 2024-04-21T23:19:10.378 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crt_file", 2024-04-21T23:19:10.378 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.378 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.379 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.379 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.379 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.379 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.379 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.379 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.379 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.379 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.379 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.380 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.380 INFO:teuthology.orchestra.run.smithi044.stdout: "debug": { 2024-04-21T23:19:10.380 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "debug", 2024-04-21T23:19:10.380 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.380 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.380 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.380 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.380 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.381 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.381 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.381 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable/disable debug options", 2024-04-21T23:19:10.381 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.381 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.381 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.381 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.381 INFO:teuthology.orchestra.run.smithi044.stdout: "jwt_token_ttl": { 2024-04-21T23:19:10.381 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "jwt_token_ttl", 2024-04-21T23:19:10.382 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.382 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.382 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.382 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "28800", 2024-04-21T23:19:10.382 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.382 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.382 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.382 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.383 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.383 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.383 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.383 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.383 INFO:teuthology.orchestra.run.smithi044.stdout: "key_file": { 2024-04-21T23:19:10.383 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "key_file", 2024-04-21T23:19:10.383 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.383 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.384 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.384 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.384 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.384 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.384 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.384 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.384 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.384 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.384 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.385 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.385 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.385 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.385 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.385 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.385 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.385 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.385 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.386 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.386 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.386 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.386 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.386 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.386 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.386 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.386 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.387 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.387 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.387 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.387 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.387 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.387 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.387 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.387 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.387 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.388 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.388 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.391 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.391 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.391 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.391 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.391 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.391 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.391 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.391 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.392 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.392 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.392 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.392 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.392 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.392 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.392 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.392 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.392 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.393 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.393 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.393 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.393 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.393 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.393 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.393 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.393 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.394 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.394 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.394 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.394 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.394 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.394 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.394 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.394 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.395 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.395 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.395 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.395 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.395 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.395 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.395 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.395 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.395 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.396 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.396 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.396 INFO:teuthology.orchestra.run.smithi044.stdout: "motd": { 2024-04-21T23:19:10.396 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "motd", 2024-04-21T23:19:10.396 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.396 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.396 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.396 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.397 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.397 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.397 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.397 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "The message of the day", 2024-04-21T23:19:10.397 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.397 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.397 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.397 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.398 INFO:teuthology.orchestra.run.smithi044.stdout: "redirect_resolve_ip_addr": { 2024-04-21T23:19:10.398 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "redirect_resolve_ip_addr", 2024-04-21T23:19:10.398 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.398 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.398 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.398 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.398 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.398 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.399 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.399 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.399 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.399 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.399 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.399 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.399 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2024-04-21T23:19:10.399 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2024-04-21T23:19:10.400 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.400 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.400 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.400 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "::", 2024-04-21T23:19:10.400 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.400 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.400 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.400 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.400 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.401 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.401 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.401 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.401 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2024-04-21T23:19:10.401 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2024-04-21T23:19:10.401 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.401 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.401 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.401 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8080", 2024-04-21T23:19:10.402 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.402 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.402 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.402 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.402 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.402 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.402 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.402 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.403 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl": { 2024-04-21T23:19:10.403 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl", 2024-04-21T23:19:10.403 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.403 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.403 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.403 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.403 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.403 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.403 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.404 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.404 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.404 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.404 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.404 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.404 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl_server_port": { 2024-04-21T23:19:10.404 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl_server_port", 2024-04-21T23:19:10.404 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.405 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.405 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.405 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8443", 2024-04-21T23:19:10.405 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.405 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.405 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.405 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.405 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.406 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.406 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.406 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.406 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_behaviour": { 2024-04-21T23:19:10.406 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_behaviour", 2024-04-21T23:19:10.406 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.406 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.406 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.406 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "redirect", 2024-04-21T23:19:10.407 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.407 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.407 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.407 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.407 INFO:teuthology.orchestra.run.smithi044.stdout: "redirect" 2024-04-21T23:19:10.407 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.407 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.407 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.408 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.408 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.408 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.408 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_error_status_code": { 2024-04-21T23:19:10.408 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_error_status_code", 2024-04-21T23:19:10.408 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.408 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.408 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.409 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "500", 2024-04-21T23:19:10.409 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "400", 2024-04-21T23:19:10.409 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "599", 2024-04-21T23:19:10.409 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.409 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.409 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.409 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.409 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.409 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.410 INFO:teuthology.orchestra.run.smithi044.stdout: "url_prefix": { 2024-04-21T23:19:10.410 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "url_prefix", 2024-04-21T23:19:10.410 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.410 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.410 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.410 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.410 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.410 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.411 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.411 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.411 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.411 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.411 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.411 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.411 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.411 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.412 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.412 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "devicehealth", 2024-04-21T23:19:10.412 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.412 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.412 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.412 INFO:teuthology.orchestra.run.smithi044.stdout: "enable_monitoring": { 2024-04-21T23:19:10.412 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enable_monitoring", 2024-04-21T23:19:10.412 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.412 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.413 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.413 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.413 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.413 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.413 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.413 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "monitor device health metrics", 2024-04-21T23:19:10.413 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.413 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.414 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.414 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.414 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.414 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.414 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.414 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.414 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.414 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.414 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.415 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.415 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.415 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.415 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.415 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.415 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.415 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.415 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.416 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.416 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.416 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.416 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.416 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.416 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.416 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.416 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.416 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.417 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.417 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.417 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.417 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.417 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.417 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.417 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.417 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.417 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.418 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.418 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.418 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.418 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.418 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.418 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.418 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.418 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.419 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.419 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.419 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.419 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.419 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.419 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.419 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.419 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.419 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.420 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.420 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.420 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.420 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.420 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.420 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.420 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.420 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.420 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.421 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.421 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.421 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.421 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.421 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.421 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.421 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.421 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.422 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.422 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.422 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.422 INFO:teuthology.orchestra.run.smithi044.stdout: "mark_out_threshold": { 2024-04-21T23:19:10.422 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mark_out_threshold", 2024-04-21T23:19:10.422 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.422 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.422 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.422 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "2419200", 2024-04-21T23:19:10.423 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.423 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.423 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.423 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-21T23:19:10.423 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.423 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.423 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.423 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.424 INFO:teuthology.orchestra.run.smithi044.stdout: "pool_name": { 2024-04-21T23:19:10.424 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pool_name", 2024-04-21T23:19:10.424 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.424 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.424 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.424 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "device_health_metrics", 2024-04-21T23:19:10.424 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.424 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.425 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.425 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-21T23:19:10.425 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.425 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.425 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.425 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.425 INFO:teuthology.orchestra.run.smithi044.stdout: "retention_period": { 2024-04-21T23:19:10.425 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "retention_period", 2024-04-21T23:19:10.425 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.426 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.426 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.426 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15552000", 2024-04-21T23:19:10.426 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.426 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.426 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.426 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long to retain device health metrics", 2024-04-21T23:19:10.426 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.427 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.427 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.427 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.427 INFO:teuthology.orchestra.run.smithi044.stdout: "scrape_frequency": { 2024-04-21T23:19:10.427 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "scrape_frequency", 2024-04-21T23:19:10.427 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.427 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.427 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.428 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "86400", 2024-04-21T23:19:10.428 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.428 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.428 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.428 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-21T23:19:10.428 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.428 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.428 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.428 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.429 INFO:teuthology.orchestra.run.smithi044.stdout: "self_heal": { 2024-04-21T23:19:10.429 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "self_heal", 2024-04-21T23:19:10.429 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.429 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.429 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.429 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.429 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.429 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.430 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.430 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-21T23:19:10.430 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.430 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.430 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.430 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.430 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-04-21T23:19:10.430 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-04-21T23:19:10.431 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.431 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.431 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.431 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-04-21T23:19:10.431 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.431 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.431 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.431 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to wake up and check device health", 2024-04-21T23:19:10.431 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.432 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.432 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.432 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.432 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_threshold": { 2024-04-21T23:19:10.432 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_threshold", 2024-04-21T23:19:10.432 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.432 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.432 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.433 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "7257600", 2024-04-21T23:19:10.433 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.433 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.433 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.433 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-21T23:19:10.433 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.433 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.433 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.433 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.434 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.434 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.434 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.434 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "diskprediction_local", 2024-04-21T23:19:10.434 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.434 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.434 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.434 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.435 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.435 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.435 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.435 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.435 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.435 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.435 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.435 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.436 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.436 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.436 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.436 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.436 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.436 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.436 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.436 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.437 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.437 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.437 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.437 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.437 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.437 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.437 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.437 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.437 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.438 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.438 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.438 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.438 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.438 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.438 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.438 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.438 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.439 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.439 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.439 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.439 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.439 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.439 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.439 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.439 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.439 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.440 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.440 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.440 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.440 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.440 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.440 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.440 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.440 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.441 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.441 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.441 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.441 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.441 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.441 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.441 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.441 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.441 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.442 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.442 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.442 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.442 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.442 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.442 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.442 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.442 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.443 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.443 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.443 INFO:teuthology.orchestra.run.smithi044.stdout: "predict_interval": { 2024-04-21T23:19:10.443 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "predict_interval", 2024-04-21T23:19:10.443 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.443 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.443 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.443 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "86400", 2024-04-21T23:19:10.443 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.444 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.444 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.444 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.444 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.444 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.444 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.444 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.444 INFO:teuthology.orchestra.run.smithi044.stdout: "predictor_model": { 2024-04-21T23:19:10.444 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "predictor_model", 2024-04-21T23:19:10.445 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.445 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.445 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.445 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "prophetstor", 2024-04-21T23:19:10.445 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.445 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.445 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.445 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.446 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.446 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.446 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.446 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.446 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-04-21T23:19:10.446 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-04-21T23:19:10.446 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.446 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.446 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.447 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-04-21T23:19:10.447 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.447 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.447 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.447 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.447 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.447 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.447 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.448 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.448 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.448 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.448 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.448 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "influx", 2024-04-21T23:19:10.448 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": false, 2024-04-21T23:19:10.448 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "influxdb python module not found", 2024-04-21T23:19:10.448 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.449 INFO:teuthology.orchestra.run.smithi044.stdout: "batch_size": { 2024-04-21T23:19:10.449 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "batch_size", 2024-04-21T23:19:10.449 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.449 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.449 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.449 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5000", 2024-04-21T23:19:10.449 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.449 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.449 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.450 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-21T23:19:10.450 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.450 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.450 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.450 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.450 INFO:teuthology.orchestra.run.smithi044.stdout: "database": { 2024-04-21T23:19:10.450 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "database", 2024-04-21T23:19:10.450 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.451 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.451 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.451 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "ceph", 2024-04-21T23:19:10.451 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.451 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.451 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.451 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-21T23:19:10.451 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.452 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.452 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.452 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.452 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": { 2024-04-21T23:19:10.452 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "hostname", 2024-04-21T23:19:10.452 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.452 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.452 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.452 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.453 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.453 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.453 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.453 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB server hostname", 2024-04-21T23:19:10.453 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.453 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.453 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.453 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.454 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-04-21T23:19:10.454 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-04-21T23:19:10.454 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.454 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.454 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.454 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "30", 2024-04-21T23:19:10.454 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "5", 2024-04-21T23:19:10.454 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.454 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.455 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-21T23:19:10.455 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.455 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.455 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.455 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.455 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.455 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.455 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.456 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.456 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.456 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.456 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.456 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.456 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.456 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.456 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.456 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.457 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.457 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.457 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.457 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.457 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.457 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.457 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.457 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.457 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.458 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.458 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.458 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.458 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.458 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.458 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.458 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.458 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.459 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.459 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.459 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.459 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.459 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.459 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.459 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.459 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.459 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.460 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.460 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.460 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.460 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.460 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.460 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.460 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.460 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.460 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.461 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.461 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.461 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.461 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.461 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.461 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.461 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.461 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.462 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.462 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.462 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.462 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.462 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.462 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.462 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.462 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.463 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.463 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.463 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.463 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.463 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.463 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.463 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.463 INFO:teuthology.orchestra.run.smithi044.stdout: "password": { 2024-04-21T23:19:10.463 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "password", 2024-04-21T23:19:10.464 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.464 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.464 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.464 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.464 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.464 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.464 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.464 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "password of InfluxDB server user", 2024-04-21T23:19:10.464 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.465 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.465 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.465 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.465 INFO:teuthology.orchestra.run.smithi044.stdout: "port": { 2024-04-21T23:19:10.465 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "port", 2024-04-21T23:19:10.465 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.465 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.465 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.466 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8086", 2024-04-21T23:19:10.466 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.466 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.466 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.466 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB server port", 2024-04-21T23:19:10.466 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.466 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.466 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.466 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.467 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl": { 2024-04-21T23:19:10.467 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl", 2024-04-21T23:19:10.467 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.467 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.467 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.467 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "false", 2024-04-21T23:19:10.467 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.467 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.468 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.468 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T23:19:10.468 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.468 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.468 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.468 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.468 INFO:teuthology.orchestra.run.smithi044.stdout: "threads": { 2024-04-21T23:19:10.468 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "threads", 2024-04-21T23:19:10.468 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.469 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.469 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.469 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-04-21T23:19:10.469 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-04-21T23:19:10.469 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "32", 2024-04-21T23:19:10.469 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.469 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-21T23:19:10.469 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.470 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.470 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.470 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.470 INFO:teuthology.orchestra.run.smithi044.stdout: "username": { 2024-04-21T23:19:10.470 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "username", 2024-04-21T23:19:10.470 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.470 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.470 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.470 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.471 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.471 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.471 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.471 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "username of InfluxDB server user", 2024-04-21T23:19:10.471 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.471 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.471 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.471 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.472 INFO:teuthology.orchestra.run.smithi044.stdout: "verify_ssl": { 2024-04-21T23:19:10.472 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "verify_ssl", 2024-04-21T23:19:10.472 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.472 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.472 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.472 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "true", 2024-04-21T23:19:10.472 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.472 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.473 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.473 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T23:19:10.473 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.473 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.473 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.473 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.473 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.473 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.474 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.474 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "insights", 2024-04-21T23:19:10.474 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.474 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.474 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.474 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.474 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.474 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.475 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.475 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.475 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.475 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.475 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.475 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.475 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.475 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.476 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.476 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.476 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.476 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.476 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.476 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.476 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.476 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.476 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.477 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.477 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.477 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.477 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.477 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.477 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.477 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.477 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.477 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.478 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.478 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.478 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.478 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.478 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.478 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.478 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.478 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.479 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.479 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.479 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.479 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.479 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.479 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.479 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.479 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.480 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.480 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.480 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.480 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.480 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.480 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.480 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.480 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.480 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.481 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.481 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.481 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.481 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.481 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.481 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.481 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.481 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.481 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.482 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.482 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.482 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.482 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.482 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.482 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.482 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.482 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.483 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.483 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.483 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "iostat", 2024-04-21T23:19:10.483 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.483 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.483 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.483 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.483 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.483 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.484 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.484 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.484 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.484 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.484 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.484 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.484 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.484 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.485 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.485 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.485 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.485 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.485 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.485 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.485 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.485 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.485 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.486 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.486 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.486 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.486 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.486 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.486 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.486 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.486 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.487 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.487 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.487 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.487 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.487 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.487 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.487 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.487 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.488 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.488 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.488 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.488 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.488 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.488 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.488 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.488 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.488 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.489 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.489 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.489 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.489 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.489 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.489 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.489 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.489 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.489 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.490 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.490 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.490 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.490 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.490 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.490 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.490 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.490 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.491 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.491 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.491 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.491 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.491 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.491 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.491 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.491 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.491 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.492 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.492 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.492 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "k8sevents", 2024-04-21T23:19:10.492 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.492 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.492 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.492 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_event_retention_days": { 2024-04-21T23:19:10.492 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ceph_event_retention_days", 2024-04-21T23:19:10.492 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.493 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.493 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.493 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "7", 2024-04-21T23:19:10.493 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.493 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.493 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.493 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-21T23:19:10.493 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.493 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.494 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.494 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.494 INFO:teuthology.orchestra.run.smithi044.stdout: "config_check_secs": { 2024-04-21T23:19:10.494 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_check_secs", 2024-04-21T23:19:10.494 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.494 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.494 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.494 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:10.495 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "10", 2024-04-21T23:19:10.495 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.495 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.495 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-21T23:19:10.495 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.495 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.495 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.495 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.496 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.496 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.496 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.496 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.496 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.496 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.496 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.496 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.497 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.497 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.497 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.497 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.497 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.497 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.497 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.497 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.497 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.498 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.498 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.498 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.498 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.498 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.498 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.498 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.498 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.499 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.499 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.499 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.499 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.499 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.499 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.499 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.499 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.499 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.500 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.500 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.500 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.500 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.500 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.500 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.500 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.500 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.501 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.501 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.501 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.501 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.501 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.501 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.501 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.501 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.501 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.502 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.502 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.502 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.502 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.502 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.502 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.502 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.502 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.503 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.503 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.503 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.503 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.503 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.503 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.503 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.503 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.503 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.504 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.504 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.504 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.504 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.504 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.504 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "localpool", 2024-04-21T23:19:10.504 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.504 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.505 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.505 INFO:teuthology.orchestra.run.smithi044.stdout: "failure_domain": { 2024-04-21T23:19:10.505 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "failure_domain", 2024-04-21T23:19:10.505 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.505 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.505 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.505 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "host", 2024-04-21T23:19:10.505 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.505 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.506 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.506 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "failure domain for any created local pool", 2024-04-21T23:19:10.506 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-21T23:19:10.506 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.506 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.506 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.506 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.506 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.507 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.507 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.507 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.507 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.507 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.507 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.507 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.507 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.507 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.508 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.508 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.508 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.508 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.508 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.508 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.508 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.508 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.509 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.509 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.509 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.509 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.509 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.509 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.509 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.509 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.510 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.510 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.510 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.510 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.510 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.510 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.510 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.510 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.510 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.511 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.511 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.511 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.511 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.511 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.511 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.511 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.511 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.512 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.512 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.512 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.512 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.512 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.512 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.512 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.512 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.512 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.513 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.513 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.513 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.513 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.513 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.513 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.513 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.513 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.514 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.514 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.514 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.514 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.514 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.514 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.514 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.514 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.515 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.515 INFO:teuthology.orchestra.run.smithi044.stdout: "min_size": { 2024-04-21T23:19:10.515 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "min_size", 2024-04-21T23:19:10.515 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.515 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.515 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.515 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.515 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.515 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.516 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.516 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default min_size for any created local pool", 2024-04-21T23:19:10.516 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-21T23:19:10.516 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.516 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.516 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.516 INFO:teuthology.orchestra.run.smithi044.stdout: "num_rep": { 2024-04-21T23:19:10.516 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "num_rep", 2024-04-21T23:19:10.516 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.517 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.517 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.517 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3", 2024-04-21T23:19:10.517 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.517 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.517 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.517 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default replica count for any created local pool", 2024-04-21T23:19:10.517 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.518 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.518 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.518 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.518 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_num": { 2024-04-21T23:19:10.518 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pg_num", 2024-04-21T23:19:10.518 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.518 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.518 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.519 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "128", 2024-04-21T23:19:10.519 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.519 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.519 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.519 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default pg_num for any created local pool", 2024-04-21T23:19:10.519 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.519 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.520 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.520 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.520 INFO:teuthology.orchestra.run.smithi044.stdout: "prefix": { 2024-04-21T23:19:10.520 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prefix", 2024-04-21T23:19:10.520 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.520 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.520 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.520 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.520 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.521 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.521 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.521 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "name prefix for any created local pool", 2024-04-21T23:19:10.521 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.521 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.521 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.521 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.521 INFO:teuthology.orchestra.run.smithi044.stdout: "subtree": { 2024-04-21T23:19:10.522 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "subtree", 2024-04-21T23:19:10.522 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.522 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.522 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.522 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "rack", 2024-04-21T23:19:10.522 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.522 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.522 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.522 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-21T23:19:10.523 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-21T23:19:10.523 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.523 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.523 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.523 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.523 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.523 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.524 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mds_autoscaler", 2024-04-21T23:19:10.524 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.524 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.524 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.524 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.524 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.524 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.524 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.524 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.525 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.525 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.525 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.525 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.525 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.525 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.525 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.525 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.526 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.526 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.526 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.526 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.526 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.526 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.526 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.526 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.526 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.527 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.527 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.527 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.527 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.527 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.527 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.527 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.527 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.528 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.528 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.528 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.528 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.528 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.528 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.528 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.528 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.528 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.529 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.529 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.529 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.529 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.529 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.529 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.529 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.529 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.529 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.530 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.530 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.530 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.530 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.530 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.530 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.530 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.530 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.531 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.531 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.531 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.531 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.531 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.531 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.531 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.531 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.531 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.532 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.532 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.532 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.532 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.532 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.532 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.532 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.532 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.533 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mirroring", 2024-04-21T23:19:10.533 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.533 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.533 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.533 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.533 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.533 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.533 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.533 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.534 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.534 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.534 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.534 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.534 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.534 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.534 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.534 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.535 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.535 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.535 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.535 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.535 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.535 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.535 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.535 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.535 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.536 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.536 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.536 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.536 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.536 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.536 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.536 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.536 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.537 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.537 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.537 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.537 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.537 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.537 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.537 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.537 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.538 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.538 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.538 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.538 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.538 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.538 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.538 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.539 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.539 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.539 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.539 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.539 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.539 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.539 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.539 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.540 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.540 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.540 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.540 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.540 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.540 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.540 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.540 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.541 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.541 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.541 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.541 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.541 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.541 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.541 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.541 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.541 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.542 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.542 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.542 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.542 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "nfs", 2024-04-21T23:19:10.542 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.542 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.543 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.543 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.543 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.543 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.543 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.543 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.543 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.543 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.544 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.544 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.544 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.544 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.544 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.544 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.544 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.545 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.545 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.545 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.545 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.545 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.545 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.545 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.545 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.546 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.546 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.546 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.546 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.546 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.546 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.546 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.546 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.547 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.547 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.547 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.547 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.547 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.547 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.547 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.547 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.548 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.548 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.548 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.548 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.548 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.548 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.548 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.548 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.549 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.549 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.549 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.549 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.549 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.549 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.549 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.549 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.550 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.550 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.550 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.550 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.550 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.550 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.550 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.550 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.551 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.551 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.551 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.551 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.551 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.551 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.551 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.551 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.551 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.552 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.552 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.552 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "orchestrator", 2024-04-21T23:19:10.552 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.552 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.552 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.552 INFO:teuthology.orchestra.run.smithi044.stdout: "fail_fs": { 2024-04-21T23:19:10.552 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "fail_fs", 2024-04-21T23:19:10.553 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.553 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.553 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.553 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.553 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.553 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.553 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.553 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-21T23:19:10.554 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.554 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.554 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.554 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.554 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.554 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.554 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.554 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.554 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.555 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.555 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.555 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.555 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.555 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.555 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.555 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.555 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.556 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.556 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.556 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.556 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.556 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.556 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.556 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.556 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.557 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.557 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.557 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.557 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.557 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.557 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.557 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.557 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.557 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.558 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.558 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.558 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.558 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.558 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.558 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.558 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.558 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.559 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.559 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.559 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.559 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.559 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.559 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.559 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.559 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.559 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.560 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.560 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.560 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.560 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.560 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.560 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.560 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.560 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.561 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.561 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.561 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.561 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.561 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.561 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.561 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.561 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.562 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.562 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.562 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.562 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.562 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.562 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.562 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.562 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator": { 2024-04-21T23:19:10.562 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "orchestrator", 2024-04-21T23:19:10.563 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.563 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.563 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.563 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.563 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.563 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.564 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.564 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm", 2024-04-21T23:19:10.564 INFO:teuthology.orchestra.run.smithi044.stdout: "rook", 2024-04-21T23:19:10.564 INFO:teuthology.orchestra.run.smithi044.stdout: "test_orchestrator" 2024-04-21T23:19:10.564 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.564 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Orchestrator backend", 2024-04-21T23:19:10.564 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.564 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.565 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.565 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.565 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.565 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.565 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.565 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "osd_perf_query", 2024-04-21T23:19:10.565 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.565 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.565 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.566 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.566 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.566 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.566 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.566 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.566 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.566 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.567 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.567 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.567 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.567 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.567 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.567 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.567 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.567 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.567 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.568 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.568 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.568 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.568 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.568 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.568 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.568 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.568 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.568 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.569 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.569 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.569 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.569 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.569 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.569 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.569 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.569 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.570 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.570 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.570 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.570 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.570 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.570 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.570 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.570 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.570 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.571 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.571 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.571 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.571 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.571 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.571 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.571 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.571 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.572 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.572 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.572 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.572 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.572 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.572 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.572 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.572 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.572 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.573 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.573 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.573 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.573 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.573 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.573 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.573 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.573 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.573 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.574 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.574 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.574 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.574 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.574 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.574 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "osd_support", 2024-04-21T23:19:10.574 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.574 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.575 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.575 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.575 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.575 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.575 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.575 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.575 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.575 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.575 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.576 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.576 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.576 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.576 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.576 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.576 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.576 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.576 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.577 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.577 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.577 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.577 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.577 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.577 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.577 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.577 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.577 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.578 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.578 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.578 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.578 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.578 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.578 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.578 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.578 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.579 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.579 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.579 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.579 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.579 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.579 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.579 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.579 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.579 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.580 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.580 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.580 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.580 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.580 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.580 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.580 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.580 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.580 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.581 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.581 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.581 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.581 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.581 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.581 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.581 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.581 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.582 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.582 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.582 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.582 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.582 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.582 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.582 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.582 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.582 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.583 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.583 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.583 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.583 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.583 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.583 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pg_autoscaler", 2024-04-21T23:19:10.583 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.583 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.584 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.584 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.584 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.584 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.584 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.584 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.584 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.584 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.584 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.585 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.585 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.585 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.585 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.585 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.585 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.585 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.585 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.585 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.586 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.586 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.586 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.586 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.586 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.586 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.586 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.586 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.587 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.587 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.587 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.587 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.587 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.587 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.587 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.587 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.587 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.588 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.588 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.588 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.588 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.588 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.588 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.588 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.588 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.589 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.589 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.589 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.589 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.589 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.589 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.589 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.589 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.590 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.590 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.590 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.590 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.590 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.590 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.590 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.590 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.591 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.591 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.591 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.595 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.595 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.595 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.595 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.595 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.596 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.596 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.596 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.596 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.596 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-04-21T23:19:10.596 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-04-21T23:19:10.596 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.596 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.597 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.597 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-04-21T23:19:10.597 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.597 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.597 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.597 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.597 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.597 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.597 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.598 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.598 INFO:teuthology.orchestra.run.smithi044.stdout: "threshold": { 2024-04-21T23:19:10.598 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "threshold", 2024-04-21T23:19:10.598 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:10.598 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.598 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.598 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3.0", 2024-04-21T23:19:10.598 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1.0", 2024-04-21T23:19:10.599 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.599 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.599 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "scaling threshold", 2024-04-21T23:19:10.599 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-21T23:19:10.599 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.599 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.599 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.600 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.600 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.600 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.600 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "progress", 2024-04-21T23:19:10.600 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.600 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.600 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.600 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_pg_recovery_event": { 2024-04-21T23:19:10.601 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_pg_recovery_event", 2024-04-21T23:19:10.601 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.601 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.601 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.601 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.601 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.601 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.601 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.601 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-21T23:19:10.602 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.602 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.602 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.602 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.602 INFO:teuthology.orchestra.run.smithi044.stdout: "enabled": { 2024-04-21T23:19:10.602 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enabled", 2024-04-21T23:19:10.602 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.602 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.603 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.603 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.603 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.603 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.603 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.603 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.603 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.603 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.604 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.604 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.604 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.604 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.604 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.604 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.604 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.604 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.604 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.605 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.605 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.605 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.605 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.605 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.605 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.605 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.605 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.606 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.606 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.606 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.606 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.606 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.606 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.606 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.606 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.606 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.607 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.607 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.607 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.607 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.607 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.607 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.607 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.607 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.608 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.608 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.608 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.608 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.608 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.608 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.608 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.608 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.608 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.609 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.609 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.609 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.609 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.609 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.609 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.609 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.610 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.610 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.610 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.610 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.610 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.610 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.610 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.610 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.611 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.611 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.611 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.611 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.611 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.611 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.611 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.611 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.612 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.612 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.612 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.612 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.612 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.612 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.612 INFO:teuthology.orchestra.run.smithi044.stdout: "max_completed_events": { 2024-04-21T23:19:10.612 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_completed_events", 2024-04-21T23:19:10.612 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.613 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.613 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.613 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "50", 2024-04-21T23:19:10.613 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.613 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.613 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.613 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "number of past completed events to remember", 2024-04-21T23:19:10.613 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.614 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.614 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.614 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.614 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-04-21T23:19:10.614 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-04-21T23:19:10.614 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.614 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.614 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.614 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-04-21T23:19:10.615 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.615 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.615 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.615 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long the module is going to sleep", 2024-04-21T23:19:10.615 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.615 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.615 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.615 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.616 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.616 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.616 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.616 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus", 2024-04-21T23:19:10.616 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.616 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.616 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.616 INFO:teuthology.orchestra.run.smithi044.stdout: "cache": { 2024-04-21T23:19:10.616 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cache", 2024-04-21T23:19:10.617 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.617 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.617 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.617 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.617 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.617 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.618 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.618 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.618 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.618 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.618 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.618 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.618 INFO:teuthology.orchestra.run.smithi044.stdout: "exclude_perf_counters": { 2024-04-21T23:19:10.618 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "exclude_perf_counters", 2024-04-21T23:19:10.619 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.619 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.619 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.619 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.619 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.619 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.619 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.619 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-21T23:19:10.619 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-21T23:19:10.620 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.620 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.620 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.620 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.620 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.620 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.620 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.621 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.621 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.621 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.621 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.621 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.621 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.621 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.621 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.622 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.622 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.622 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.622 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.622 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.622 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.622 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.622 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.623 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.623 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.623 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.623 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.623 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.623 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.623 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.623 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.624 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.624 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.624 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.624 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.624 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.624 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.624 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.624 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.624 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.625 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.625 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.625 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.625 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.625 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.625 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.625 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.625 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.626 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.626 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.626 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.626 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.626 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.626 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.626 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.626 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.626 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.627 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.627 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.627 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.627 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.627 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.627 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.665 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.666 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.666 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.666 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.666 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.667 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.667 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.667 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.667 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.667 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.667 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_stats_pools": { 2024-04-21T23:19:10.668 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_stats_pools", 2024-04-21T23:19:10.668 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.668 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.668 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.669 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.669 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.669 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.669 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.670 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.670 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.670 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.670 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.671 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.671 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-21T23:19:10.671 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-21T23:19:10.671 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.671 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.671 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.671 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "300", 2024-04-21T23:19:10.671 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.672 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.672 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.672 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.672 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.673 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.673 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.673 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.673 INFO:teuthology.orchestra.run.smithi044.stdout: "scrape_interval": { 2024-04-21T23:19:10.673 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "scrape_interval", 2024-04-21T23:19:10.673 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:10.673 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.674 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.674 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15.0", 2024-04-21T23:19:10.674 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.674 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.674 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.674 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.674 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.674 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.674 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.675 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.675 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2024-04-21T23:19:10.675 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2024-04-21T23:19:10.675 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.675 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.675 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.675 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "::", 2024-04-21T23:19:10.675 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.676 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.676 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.676 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-21T23:19:10.676 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.677 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.677 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.677 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.677 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2024-04-21T23:19:10.677 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2024-04-21T23:19:10.677 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.678 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.678 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.678 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "9283", 2024-04-21T23:19:10.678 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.678 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.678 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.679 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-21T23:19:10.679 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.679 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.679 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.679 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.679 INFO:teuthology.orchestra.run.smithi044.stdout: "stale_cache_strategy": { 2024-04-21T23:19:10.679 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "stale_cache_strategy", 2024-04-21T23:19:10.680 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.680 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.680 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.680 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "log", 2024-04-21T23:19:10.680 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.680 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.680 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.681 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.681 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.681 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.681 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.681 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.681 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_behaviour": { 2024-04-21T23:19:10.681 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_behaviour", 2024-04-21T23:19:10.681 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.682 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.682 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.682 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "default", 2024-04-21T23:19:10.682 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.682 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.682 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.683 INFO:teuthology.orchestra.run.smithi044.stdout: "default", 2024-04-21T23:19:10.683 INFO:teuthology.orchestra.run.smithi044.stdout: "error" 2024-04-21T23:19:10.683 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.683 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.683 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.683 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.684 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.684 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.684 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_error_status_code": { 2024-04-21T23:19:10.684 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_error_status_code", 2024-04-21T23:19:10.684 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.685 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.685 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.685 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "500", 2024-04-21T23:19:10.685 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "400", 2024-04-21T23:19:10.685 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "599", 2024-04-21T23:19:10.685 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.685 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.685 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.686 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.686 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.686 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.686 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.686 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.686 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.687 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_support", 2024-04-21T23:19:10.687 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.687 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.687 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.687 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.687 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.688 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.688 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.688 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.688 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.689 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.689 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.689 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.689 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.689 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.690 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.690 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.690 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.690 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.691 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.691 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.691 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.691 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.691 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.691 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.691 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.692 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.692 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.692 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.692 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.692 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.693 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.693 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.693 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.693 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.693 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.693 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.693 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.694 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.694 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.694 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.694 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.694 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.694 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.695 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.695 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.695 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.695 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.695 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.695 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.696 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.696 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.696 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.696 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.696 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.697 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.697 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.697 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.697 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.697 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.697 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.697 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.698 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.698 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.698 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.698 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.698 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.698 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.699 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.699 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.699 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.699 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.699 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.699 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.699 INFO:teuthology.orchestra.run.smithi044.stdout: "max_concurrent_snap_create": { 2024-04-21T23:19:10.699 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_concurrent_snap_create", 2024-04-21T23:19:10.700 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.700 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.700 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.700 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:10.701 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.701 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.701 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.701 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.701 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.701 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.701 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.702 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.702 INFO:teuthology.orchestra.run.smithi044.stdout: "mirror_snapshot_schedule": { 2024-04-21T23:19:10.702 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mirror_snapshot_schedule", 2024-04-21T23:19:10.702 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.702 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.702 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.702 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.703 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.703 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.703 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.703 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.703 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.703 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.703 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.703 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.704 INFO:teuthology.orchestra.run.smithi044.stdout: "trash_purge_schedule": { 2024-04-21T23:19:10.704 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "trash_purge_schedule", 2024-04-21T23:19:10.704 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.704 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.705 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.705 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.705 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.705 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.706 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.706 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.706 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.706 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.707 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.707 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.707 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.708 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.708 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.708 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "restful", 2024-04-21T23:19:10.708 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.709 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.709 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.709 INFO:teuthology.orchestra.run.smithi044.stdout: "enable_auth": { 2024-04-21T23:19:10.709 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enable_auth", 2024-04-21T23:19:10.710 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.710 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.710 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.711 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.711 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.711 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.711 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.712 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.712 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.712 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.712 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.713 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.713 INFO:teuthology.orchestra.run.smithi044.stdout: "key_file": { 2024-04-21T23:19:10.713 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "key_file", 2024-04-21T23:19:10.713 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.714 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.714 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.714 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.715 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.715 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.715 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.716 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.716 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.716 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.716 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.717 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.717 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.717 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.717 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.718 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.718 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.718 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.719 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.719 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.719 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.719 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.720 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.720 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.720 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.721 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.721 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.721 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.721 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.722 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.722 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.722 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.722 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.722 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.722 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.723 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.723 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.723 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.723 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.723 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.724 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.724 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.724 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.724 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.724 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.724 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.725 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.725 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.725 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.725 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.725 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.725 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.726 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.726 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.726 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.726 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.726 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.726 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.726 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.727 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.727 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.727 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.727 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.727 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.727 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.727 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.727 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.727 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.728 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.728 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.728 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.728 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.728 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.728 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.728 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.728 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.728 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.729 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.729 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.729 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.729 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.729 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.729 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2024-04-21T23:19:10.729 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2024-04-21T23:19:10.730 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.730 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.730 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.730 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.730 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.731 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.731 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.731 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.731 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.731 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.731 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.731 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.732 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2024-04-21T23:19:10.732 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2024-04-21T23:19:10.732 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.732 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.732 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.732 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.733 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.733 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.733 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.733 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.734 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.734 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.734 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.735 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.735 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.735 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.735 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.736 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rgw", 2024-04-21T23:19:10.736 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.736 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.736 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.736 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.736 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.736 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.736 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.737 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.737 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.737 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.737 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.737 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.737 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.737 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.737 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.737 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.738 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.738 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.738 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.738 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.738 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.738 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.738 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.738 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.739 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.739 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.739 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.739 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.739 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.739 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.739 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.739 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.739 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.740 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.740 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.740 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.740 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.740 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.740 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.740 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.740 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.741 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.741 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.741 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.741 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.741 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.741 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.741 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.741 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.742 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.742 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.742 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.742 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.742 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.742 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.742 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.742 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.743 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.743 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.743 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.743 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.743 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.743 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.743 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.743 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.743 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.744 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.744 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.744 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.744 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.744 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.745 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.745 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.745 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.745 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.746 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.746 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rook", 2024-04-21T23:19:10.746 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.746 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.746 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.747 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.747 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.747 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.748 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.748 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.748 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.748 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.749 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.749 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.749 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.749 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.750 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.750 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.750 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.750 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.751 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.751 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.751 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.751 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.751 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.751 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.751 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.751 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.752 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.752 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.752 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.752 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.752 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.752 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.753 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.753 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.753 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.753 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.753 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.753 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.753 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.753 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.753 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.754 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.754 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.754 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.754 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.754 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.754 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.754 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.755 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.755 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.755 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.755 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.755 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.756 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.756 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.756 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.756 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.756 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.756 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.756 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.756 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.757 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.757 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.757 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.757 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.757 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.757 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.758 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.758 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.758 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.758 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.759 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.759 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.759 INFO:teuthology.orchestra.run.smithi044.stdout: "storage_class": { 2024-04-21T23:19:10.759 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "storage_class", 2024-04-21T23:19:10.759 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.759 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.760 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.760 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "local", 2024-04-21T23:19:10.760 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.760 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.760 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.760 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-21T23:19:10.760 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.760 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.761 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.761 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.761 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.761 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.762 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.762 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "selftest", 2024-04-21T23:19:10.762 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.762 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.762 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.762 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.763 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.763 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.763 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.763 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.763 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.764 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.764 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.764 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.765 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.765 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.766 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.766 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.767 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.767 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.767 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.767 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.767 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.768 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.768 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.768 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.768 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.769 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.769 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.769 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.769 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.769 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.769 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.769 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.769 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.770 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.770 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.770 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.770 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.770 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.770 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.770 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.770 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.770 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.771 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.771 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.771 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.771 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.771 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.771 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.771 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.771 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.772 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.772 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.772 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.772 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.772 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.772 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.772 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.773 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.773 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.773 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.773 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.773 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.773 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.773 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.773 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.774 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.774 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.774 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.774 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.774 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.774 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.774 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.774 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.775 INFO:teuthology.orchestra.run.smithi044.stdout: "roption1": { 2024-04-21T23:19:10.775 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "roption1", 2024-04-21T23:19:10.775 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.775 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.775 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.775 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.775 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.775 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.775 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.776 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.776 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.776 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.776 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.776 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.776 INFO:teuthology.orchestra.run.smithi044.stdout: "roption2": { 2024-04-21T23:19:10.776 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "roption2", 2024-04-21T23:19:10.776 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.777 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.777 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.777 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "xyz", 2024-04-21T23:19:10.777 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.777 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.777 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.777 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.778 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.778 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.778 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.778 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.778 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption1": { 2024-04-21T23:19:10.778 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption1", 2024-04-21T23:19:10.778 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.778 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.779 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.779 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.779 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.779 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.779 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.779 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.780 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.780 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.780 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.780 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.780 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption2": { 2024-04-21T23:19:10.780 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption2", 2024-04-21T23:19:10.780 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.780 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.780 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.781 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.781 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.781 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.781 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.781 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.781 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.781 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.781 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.781 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.782 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption3": { 2024-04-21T23:19:10.782 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption3", 2024-04-21T23:19:10.782 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:10.782 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.782 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.782 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.782 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.783 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.783 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.783 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.783 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.783 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.784 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.784 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.784 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption4": { 2024-04-21T23:19:10.784 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption4", 2024-04-21T23:19:10.784 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.784 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.784 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.784 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.785 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.785 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.785 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.785 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.785 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.785 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.785 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.786 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.786 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption5": { 2024-04-21T23:19:10.786 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption5", 2024-04-21T23:19:10.786 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.786 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.786 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.786 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.787 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.787 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.787 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.787 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.787 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.787 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.787 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.788 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.788 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption6": { 2024-04-21T23:19:10.788 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption6", 2024-04-21T23:19:10.788 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.788 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.788 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.788 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.789 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.789 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.789 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.789 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.789 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.789 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.790 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.790 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.790 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption7": { 2024-04-21T23:19:10.790 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption7", 2024-04-21T23:19:10.790 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.790 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.791 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.791 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.791 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-04-21T23:19:10.791 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "42", 2024-04-21T23:19:10.791 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.791 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.791 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.791 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.792 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.792 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.792 INFO:teuthology.orchestra.run.smithi044.stdout: "testkey": { 2024-04-21T23:19:10.792 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testkey", 2024-04-21T23:19:10.792 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.792 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.792 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.793 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.793 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.793 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.793 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.794 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.794 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.794 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.794 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.795 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.795 INFO:teuthology.orchestra.run.smithi044.stdout: "testlkey": { 2024-04-21T23:19:10.799 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testlkey", 2024-04-21T23:19:10.799 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.799 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.800 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.800 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.800 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.800 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.801 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.801 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.801 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.801 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.802 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.802 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.802 INFO:teuthology.orchestra.run.smithi044.stdout: "testnewline": { 2024-04-21T23:19:10.803 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testnewline", 2024-04-21T23:19:10.803 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.804 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.804 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.804 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.805 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.805 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.805 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.805 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.806 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.806 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.806 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.807 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.807 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.807 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.807 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.808 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snap_schedule", 2024-04-21T23:19:10.808 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.808 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.808 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.808 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_m_granularity": { 2024-04-21T23:19:10.809 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_m_granularity", 2024-04-21T23:19:10.809 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.809 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.809 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.810 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.810 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.810 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.810 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.810 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow minute scheduled snapshots", 2024-04-21T23:19:10.811 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.811 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.811 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.811 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.812 INFO:teuthology.orchestra.run.smithi044.stdout: "dump_on_update": { 2024-04-21T23:19:10.812 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "dump_on_update", 2024-04-21T23:19:10.812 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.812 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.813 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.813 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.813 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.813 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.813 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.814 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "dump database to debug log on update", 2024-04-21T23:19:10.814 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.814 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.815 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.815 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.815 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.815 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.816 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.816 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.816 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.816 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.816 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.817 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.817 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.817 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.817 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.817 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.818 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.818 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.818 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.818 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.819 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.819 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.819 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.819 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.819 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.820 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.820 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.820 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.820 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.821 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.821 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.821 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.821 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.821 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.822 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.822 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.822 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.822 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.822 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.823 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.823 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.823 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.823 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.823 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.824 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.824 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.824 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.824 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.824 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.825 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.825 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.825 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.825 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.826 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.826 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.826 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.827 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.827 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.827 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.827 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.828 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.828 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.828 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.828 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.828 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.828 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.828 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.828 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.828 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.829 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.829 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.829 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.829 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.829 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.830 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.830 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.830 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.830 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "stats", 2024-04-21T23:19:10.830 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.830 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.830 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.831 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.831 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.831 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.831 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.831 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.831 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.832 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.832 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.832 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.832 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.832 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.832 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.832 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.832 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.833 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.833 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.833 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.833 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.833 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.833 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.833 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.834 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.834 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.834 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.834 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.834 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.834 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.835 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.835 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.835 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.835 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.835 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.835 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.836 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.836 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.836 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.836 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.837 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.837 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.837 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.837 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.837 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.837 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.838 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.838 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.838 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.838 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.838 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.838 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.839 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.839 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.839 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.839 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.839 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.839 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.840 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.840 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.840 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.840 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.840 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.840 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.840 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.840 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.841 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.841 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.841 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.841 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.841 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.841 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.841 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.841 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.842 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.842 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.842 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "status", 2024-04-21T23:19:10.842 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.842 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.842 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.842 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.842 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.843 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.843 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.843 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.843 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.843 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.843 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.843 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.843 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.843 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.844 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.844 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.844 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.844 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.844 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.844 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.844 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.844 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.844 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.845 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.845 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.845 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.845 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.845 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.845 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.845 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.845 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.846 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.846 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.846 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.846 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.846 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.846 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.846 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.846 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.847 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.847 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.847 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.847 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.847 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.847 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.847 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.847 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.847 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.848 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.848 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.848 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.848 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.848 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.848 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.848 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.848 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.849 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.849 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.849 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.849 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.849 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.849 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.849 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.849 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.850 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.850 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.850 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.850 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.850 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.850 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.850 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.850 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.851 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.851 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.851 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.851 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.851 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "telegraf", 2024-04-21T23:19:10.851 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.851 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.851 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.852 INFO:teuthology.orchestra.run.smithi044.stdout: "address": { 2024-04-21T23:19:10.852 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "address", 2024-04-21T23:19:10.852 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.852 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.852 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.852 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-21T23:19:10.852 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.852 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.853 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.853 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.853 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.853 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.853 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.853 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.853 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-04-21T23:19:10.853 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-04-21T23:19:10.853 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.854 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.854 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.854 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15", 2024-04-21T23:19:10.854 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.854 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.854 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.854 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.854 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.855 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.855 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.855 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.855 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.855 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.855 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.855 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.855 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.856 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.856 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.856 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.856 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.856 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.856 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.856 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.856 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.856 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.857 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.857 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.857 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.857 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.857 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.857 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.857 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.857 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.857 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.858 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.858 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.858 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.858 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.858 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.858 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.858 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.858 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.859 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.859 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.859 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.859 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.859 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.859 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.859 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.860 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.860 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.860 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.860 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.860 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.860 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.860 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.860 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.861 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.861 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.861 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.861 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.861 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.861 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.861 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.861 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.861 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.862 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.862 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.862 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.862 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.862 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.863 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.863 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.863 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.863 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.863 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.863 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.863 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.863 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.863 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.864 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.864 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.864 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.864 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.864 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "telemetry", 2024-04-21T23:19:10.864 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.865 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.865 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.865 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_basic": { 2024-04-21T23:19:10.865 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_basic", 2024-04-21T23:19:10.865 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.865 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.865 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.865 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.866 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.866 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.866 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.866 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-21T23:19:10.866 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.867 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.867 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.867 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.867 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_crash": { 2024-04-21T23:19:10.867 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_crash", 2024-04-21T23:19:10.867 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.867 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.867 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.868 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.868 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.868 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.868 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.868 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-21T23:19:10.868 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.869 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.869 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.869 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.869 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_device": { 2024-04-21T23:19:10.869 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_device", 2024-04-21T23:19:10.869 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.869 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.870 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.870 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.870 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.870 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.870 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.870 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-21T23:19:10.871 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.871 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.871 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.871 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.871 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_ident": { 2024-04-21T23:19:10.871 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_ident", 2024-04-21T23:19:10.871 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.871 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.871 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.872 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.872 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.872 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.872 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.872 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-21T23:19:10.872 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.872 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.873 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.873 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.873 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_perf": { 2024-04-21T23:19:10.873 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_perf", 2024-04-21T23:19:10.873 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.874 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.874 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.874 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.874 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.874 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.874 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.874 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-21T23:19:10.875 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.875 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.875 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.875 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.875 INFO:teuthology.orchestra.run.smithi044.stdout: "contact": { 2024-04-21T23:19:10.875 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "contact", 2024-04-21T23:19:10.875 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.875 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.876 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.876 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.876 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.876 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.876 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.876 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.876 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.876 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.876 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.877 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.877 INFO:teuthology.orchestra.run.smithi044.stdout: "description": { 2024-04-21T23:19:10.877 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "description", 2024-04-21T23:19:10.877 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.877 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.877 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.877 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.877 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.877 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.878 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.878 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.878 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.878 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.878 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.878 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.878 INFO:teuthology.orchestra.run.smithi044.stdout: "device_url": { 2024-04-21T23:19:10.878 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_url", 2024-04-21T23:19:10.879 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.879 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.879 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.879 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-21T23:19:10.879 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.879 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.879 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.879 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.880 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.880 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.880 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.880 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.880 INFO:teuthology.orchestra.run.smithi044.stdout: "enabled": { 2024-04-21T23:19:10.880 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enabled", 2024-04-21T23:19:10.880 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.880 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.880 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.881 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.881 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.881 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.881 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.881 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.881 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.881 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.881 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.881 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.882 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-04-21T23:19:10.882 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-04-21T23:19:10.882 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.882 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.882 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.882 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "24", 2024-04-21T23:19:10.882 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "8", 2024-04-21T23:19:10.882 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.883 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.883 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.883 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.883 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.883 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.883 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.883 INFO:teuthology.orchestra.run.smithi044.stdout: "last_opt_revision": { 2024-04-21T23:19:10.884 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "last_opt_revision", 2024-04-21T23:19:10.884 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.884 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.884 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.884 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1", 2024-04-21T23:19:10.884 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.884 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.884 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.885 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.885 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.885 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.885 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.885 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.885 INFO:teuthology.orchestra.run.smithi044.stdout: "leaderboard": { 2024-04-21T23:19:10.885 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "leaderboard", 2024-04-21T23:19:10.885 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.885 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.886 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.886 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.886 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.886 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.886 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.886 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.886 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.886 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.887 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.887 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.887 INFO:teuthology.orchestra.run.smithi044.stdout: "leaderboard_description": { 2024-04-21T23:19:10.887 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "leaderboard_description", 2024-04-21T23:19:10.887 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.887 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.887 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.887 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.887 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.888 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.888 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.888 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.888 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.888 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.888 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.888 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.888 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.888 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.889 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.889 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.889 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.889 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.889 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.889 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.889 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.889 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.889 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.890 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.890 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.890 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.890 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.890 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.890 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.890 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.891 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.891 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.891 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.891 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.891 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.891 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.891 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.891 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.891 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.892 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.892 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.892 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.892 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.892 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.892 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.892 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.892 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.892 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.893 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.893 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.893 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.893 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.893 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.893 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.893 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.893 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.893 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.894 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.894 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.894 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.894 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.894 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.894 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.894 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.894 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.895 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.895 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.895 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.895 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.895 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.895 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.896 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.896 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.896 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.896 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.896 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.896 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.896 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.896 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.897 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.897 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.897 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.897 INFO:teuthology.orchestra.run.smithi044.stdout: "organization": { 2024-04-21T23:19:10.897 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "organization", 2024-04-21T23:19:10.897 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.897 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.897 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.898 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.898 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.898 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.898 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.898 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.898 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.899 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.899 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.899 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.899 INFO:teuthology.orchestra.run.smithi044.stdout: "proxy": { 2024-04-21T23:19:10.899 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "proxy", 2024-04-21T23:19:10.899 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.899 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.899 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.900 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.900 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.900 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.900 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.900 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.900 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.900 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.901 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.901 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.901 INFO:teuthology.orchestra.run.smithi044.stdout: "url": { 2024-04-21T23:19:10.901 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "url", 2024-04-21T23:19:10.901 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.901 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.902 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.902 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-21T23:19:10.902 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.902 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.902 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.902 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.902 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.903 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.903 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.903 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.903 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.903 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.903 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.904 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "test_orchestrator", 2024-04-21T23:19:10.904 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.904 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.904 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.904 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.904 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.904 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.904 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.905 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.905 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.905 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.905 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.905 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.905 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.906 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.906 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.906 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.906 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.906 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.906 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.906 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.907 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.907 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.907 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.907 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.907 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.907 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.908 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.908 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.908 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.908 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.908 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.908 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.908 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.908 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.908 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.909 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.909 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.909 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.909 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.909 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.909 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.909 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.909 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.909 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.910 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.910 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.910 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.910 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.910 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.910 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.910 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.910 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.911 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.911 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.911 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.911 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.911 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.911 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.911 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.911 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.911 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.912 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.912 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.912 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.912 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.912 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.912 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.912 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.912 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.912 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.913 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.913 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.913 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.913 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.913 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.913 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.913 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "volumes", 2024-04-21T23:19:10.913 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.913 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.914 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.914 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.914 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.914 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.914 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.914 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.914 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.914 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.915 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.915 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.915 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.915 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.915 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.915 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.915 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.915 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.915 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.916 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.916 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.916 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.916 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.916 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.916 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.916 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.916 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.916 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.917 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.917 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.917 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.917 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.917 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.917 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.917 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.917 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.917 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.918 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.918 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.918 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.918 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.918 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.918 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.918 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.918 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.919 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.919 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.919 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.919 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.919 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.919 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.919 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.919 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.919 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.920 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.920 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.920 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.920 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.920 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.920 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.920 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.920 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.921 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.921 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.921 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.921 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.921 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.921 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.921 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.921 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.921 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.922 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.922 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.922 INFO:teuthology.orchestra.run.smithi044.stdout: "max_concurrent_clones": { 2024-04-21T23:19:10.922 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_concurrent_clones", 2024-04-21T23:19:10.922 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.922 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.922 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.922 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "4", 2024-04-21T23:19:10.923 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.923 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.923 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.923 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-21T23:19:10.923 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.923 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.923 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.923 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.923 INFO:teuthology.orchestra.run.smithi044.stdout: "periodic_async_work": { 2024-04-21T23:19:10.924 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "periodic_async_work", 2024-04-21T23:19:10.924 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.924 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.924 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.924 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.924 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.924 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.924 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.924 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Periodically check for async work", 2024-04-21T23:19:10.925 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.925 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.925 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.925 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.925 INFO:teuthology.orchestra.run.smithi044.stdout: "snapshot_clone_delay": { 2024-04-21T23:19:10.925 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snapshot_clone_delay", 2024-04-21T23:19:10.925 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.925 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.926 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.926 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-04-21T23:19:10.926 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.926 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.926 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.926 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-21T23:19:10.926 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.926 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.927 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.927 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.927 INFO:teuthology.orchestra.run.smithi044.stdout: "snapshot_clone_no_wait": { 2024-04-21T23:19:10.927 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snapshot_clone_no_wait", 2024-04-21T23:19:10.927 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.927 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.927 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.927 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.927 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.928 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.928 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.928 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-21T23:19:10.928 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.928 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.928 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.928 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.929 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.929 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.929 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.929 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix", 2024-04-21T23:19:10.929 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.929 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.929 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.930 INFO:teuthology.orchestra.run.smithi044.stdout: "discovery_interval": { 2024-04-21T23:19:10.930 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "discovery_interval", 2024-04-21T23:19:10.930 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-04-21T23:19:10.930 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.930 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.930 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "100", 2024-04-21T23:19:10.930 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.930 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.931 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.931 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.931 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.931 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.931 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.931 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.931 INFO:teuthology.orchestra.run.smithi044.stdout: "identifier": { 2024-04-21T23:19:10.932 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "identifier", 2024-04-21T23:19:10.932 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.932 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.932 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.932 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.932 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.932 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.932 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.933 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.933 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.933 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.933 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.933 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.933 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-04-21T23:19:10.933 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-04-21T23:19:10.934 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.934 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.934 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.934 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-04-21T23:19:10.934 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.934 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.934 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.935 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.935 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.935 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.935 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.935 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.935 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.935 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.935 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.936 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.936 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.936 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.936 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.936 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.936 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.937 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.937 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.937 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.937 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.937 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.937 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.937 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.938 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.938 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.938 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.938 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.938 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.939 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.939 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.939 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.939 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.939 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.939 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.939 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.940 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.940 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.940 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.940 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.940 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.940 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.940 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.941 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.941 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.941 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.941 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.941 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.941 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.942 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.942 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.942 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.942 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.942 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.942 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.942 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.943 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.943 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.943 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.943 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.943 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.943 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.943 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.943 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.944 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.944 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.944 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.944 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.944 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.944 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.944 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.944 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.944 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.945 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.945 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.945 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.945 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.945 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.945 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_host": { 2024-04-21T23:19:10.945 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_host", 2024-04-21T23:19:10.945 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.945 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.946 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.946 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.946 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.946 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.946 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.946 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.946 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.946 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.947 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.947 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.947 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_port": { 2024-04-21T23:19:10.947 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_port", 2024-04-21T23:19:10.947 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.947 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.947 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.947 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10051", 2024-04-21T23:19:10.947 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.948 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.948 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.948 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.948 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.948 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.948 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.948 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.948 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_sender": { 2024-04-21T23:19:10.948 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_sender", 2024-04-21T23:19:10.949 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.949 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.949 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:10.949 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-21T23:19:10.949 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.949 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.949 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.949 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.949 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.950 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.950 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.950 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.950 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.950 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.950 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-21T23:19:10.950 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.950 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.951 INFO:teuthology.orchestra.run.smithi044.stdout: "modules": [ 2024-04-21T23:19:10.951 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm", 2024-04-21T23:19:10.951 INFO:teuthology.orchestra.run.smithi044.stdout: "dashboard", 2024-04-21T23:19:10.951 INFO:teuthology.orchestra.run.smithi044.stdout: "iostat", 2024-04-21T23:19:10.951 INFO:teuthology.orchestra.run.smithi044.stdout: "nfs", 2024-04-21T23:19:10.951 INFO:teuthology.orchestra.run.smithi044.stdout: "restful", 2024-04-21T23:19:10.951 INFO:teuthology.orchestra.run.smithi044.stdout: "snap_schedule" 2024-04-21T23:19:10.951 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.951 INFO:teuthology.orchestra.run.smithi044.stdout: "available_modules": [ 2024-04-21T23:19:10.952 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.952 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "alerts", 2024-04-21T23:19:10.952 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.952 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.952 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.952 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-04-21T23:19:10.952 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-04-21T23:19:10.952 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:10.952 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.953 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.953 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-04-21T23:19:10.953 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.953 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.953 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.953 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How frequently to reexamine health status", 2024-04-21T23:19:10.953 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.953 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.954 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.954 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.954 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.954 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.954 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.954 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.954 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.954 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.954 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.955 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.955 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.955 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.955 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.955 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.955 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.955 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.955 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.955 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.956 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.956 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.956 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.956 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.956 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.956 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.956 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.956 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.956 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.957 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.957 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.957 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.957 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.957 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.957 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.957 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.957 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.957 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.958 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.958 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.958 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.958 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.958 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.958 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.958 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.958 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.959 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.959 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.959 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.959 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.959 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.959 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.959 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.959 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.959 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.960 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.960 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.960 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.960 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.960 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.960 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.960 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.960 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.960 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.961 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.961 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.961 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.961 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.961 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.961 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.961 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.961 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.962 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.962 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.962 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_destination": { 2024-04-21T23:19:10.962 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_destination", 2024-04-21T23:19:10.962 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.962 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.962 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.962 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.963 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.963 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.963 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.963 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Email address to send alerts to", 2024-04-21T23:19:10.963 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.963 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.963 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.963 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.964 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_from_name": { 2024-04-21T23:19:10.964 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_from_name", 2024-04-21T23:19:10.964 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.964 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.964 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.964 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "Ceph", 2024-04-21T23:19:10.965 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.965 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.965 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.965 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Email From: name", 2024-04-21T23:19:10.965 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.965 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.965 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.965 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.965 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_host": { 2024-04-21T23:19:10.966 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_host", 2024-04-21T23:19:10.966 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.966 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.966 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.966 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.967 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.967 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.967 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.967 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP server", 2024-04-21T23:19:10.967 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.967 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.967 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.967 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.967 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_password": { 2024-04-21T23:19:10.968 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_password", 2024-04-21T23:19:10.968 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.968 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.968 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.968 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.968 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.969 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.969 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.969 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Password to authenticate with", 2024-04-21T23:19:10.969 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.969 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.969 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.969 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.969 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_port": { 2024-04-21T23:19:10.969 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_port", 2024-04-21T23:19:10.970 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:10.970 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.970 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.970 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "465", 2024-04-21T23:19:10.970 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.970 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.971 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.971 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP port", 2024-04-21T23:19:10.971 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.971 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.971 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.971 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.971 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_sender": { 2024-04-21T23:19:10.971 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_sender", 2024-04-21T23:19:10.972 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.972 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.972 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.972 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.972 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.972 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.972 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.973 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SMTP envelope sender", 2024-04-21T23:19:10.973 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.973 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.973 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.973 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.973 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_ssl": { 2024-04-21T23:19:10.973 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_ssl", 2024-04-21T23:19:10.974 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.974 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.974 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.974 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.974 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.974 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.975 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.975 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-21T23:19:10.975 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.975 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.975 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.975 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.976 INFO:teuthology.orchestra.run.smithi044.stdout: "smtp_user": { 2024-04-21T23:19:10.976 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "smtp_user", 2024-04-21T23:19:10.976 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.976 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.976 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.976 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.976 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.976 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.977 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.977 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "User to authenticate as", 2024-04-21T23:19:10.977 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.977 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.977 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.977 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.977 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:10.977 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.977 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:10.978 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "balancer", 2024-04-21T23:19:10.978 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:10.978 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:10.978 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:10.978 INFO:teuthology.orchestra.run.smithi044.stdout: "active": { 2024-04-21T23:19:10.978 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "active", 2024-04-21T23:19:10.978 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.978 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.978 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.979 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:10.979 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.979 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.979 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.979 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically balance PGs across cluster", 2024-04-21T23:19:10.979 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.979 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.979 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.979 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.980 INFO:teuthology.orchestra.run.smithi044.stdout: "begin_time": { 2024-04-21T23:19:10.980 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "begin_time", 2024-04-21T23:19:10.980 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.980 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.980 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.980 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0000", 2024-04-21T23:19:10.980 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.980 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.981 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.981 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "beginning time of day to automatically balance", 2024-04-21T23:19:10.981 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T23:19:10.981 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.981 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.981 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.981 INFO:teuthology.orchestra.run.smithi044.stdout: "begin_weekday": { 2024-04-21T23:19:10.981 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "begin_weekday", 2024-04-21T23:19:10.981 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-04-21T23:19:10.982 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.982 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.982 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-04-21T23:19:10.982 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0", 2024-04-21T23:19:10.982 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "6", 2024-04-21T23:19:10.982 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.982 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-21T23:19:10.982 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T23:19:10.983 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.983 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.983 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.983 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_max_iterations": { 2024-04-21T23:19:10.983 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_max_iterations", 2024-04-21T23:19:10.983 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-04-21T23:19:10.983 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.983 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.984 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "25", 2024-04-21T23:19:10.984 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-04-21T23:19:10.984 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "250", 2024-04-21T23:19:10.984 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.984 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-21T23:19:10.984 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.984 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.985 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.985 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.985 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_metrics": { 2024-04-21T23:19:10.985 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_metrics", 2024-04-21T23:19:10.985 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.985 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.985 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.985 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "pgs,objects,bytes", 2024-04-21T23:19:10.985 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.986 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.986 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.986 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-21T23:19:10.986 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-21T23:19:10.986 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.986 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.986 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.986 INFO:teuthology.orchestra.run.smithi044.stdout: "crush_compat_step": { 2024-04-21T23:19:10.987 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crush_compat_step", 2024-04-21T23:19:10.987 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:10.987 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.987 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.987 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0.5", 2024-04-21T23:19:10.987 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0.001", 2024-04-21T23:19:10.987 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "0.999", 2024-04-21T23:19:10.987 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.987 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "aggressiveness of optimization", 2024-04-21T23:19:10.988 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-21T23:19:10.988 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.988 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.988 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.988 INFO:teuthology.orchestra.run.smithi044.stdout: "end_time": { 2024-04-21T23:19:10.988 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "end_time", 2024-04-21T23:19:10.988 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.988 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.988 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.989 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "2359", 2024-04-21T23:19:10.989 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.989 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.989 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.989 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "ending time of day to automatically balance", 2024-04-21T23:19:10.989 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T23:19:10.989 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.989 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.990 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.990 INFO:teuthology.orchestra.run.smithi044.stdout: "end_weekday": { 2024-04-21T23:19:10.990 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "end_weekday", 2024-04-21T23:19:10.990 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-04-21T23:19:10.990 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.990 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.990 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-04-21T23:19:10.990 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "0", 2024-04-21T23:19:10.990 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "6", 2024-04-21T23:19:10.991 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.991 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-21T23:19:10.991 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T23:19:10.991 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.991 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.991 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.991 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:10.991 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:10.991 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.992 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.992 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.992 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:10.992 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.992 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.992 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.992 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.992 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.993 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.993 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.993 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.993 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.993 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.993 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.993 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.993 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.993 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.994 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.994 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:10.994 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:10.994 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.994 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.994 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.994 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.994 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.994 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.995 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.995 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.995 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.995 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.995 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.995 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.995 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:10.995 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:10.996 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:10.996 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.996 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.996 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:10.996 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.996 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.996 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:10.996 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:10.997 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:10.997 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:10.997 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:10.997 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:10.997 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:10.997 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:10.997 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.997 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.997 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:10.998 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:10.998 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:10.998 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:10.998 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:10.998 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:10.998 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:10.998 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:10.998 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:10.999 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:10.999 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:10.999 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:10.999 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:10.999 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:10.999 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.000 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.000 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.000 INFO:teuthology.orchestra.run.smithi044.stdout: "min_score": { 2024-04-21T23:19:11.000 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "min_score", 2024-04-21T23:19:11.000 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:11.000 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.000 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.000 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-04-21T23:19:11.000 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.001 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.001 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.001 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-21T23:19:11.001 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.001 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.001 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.002 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.002 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": { 2024-04-21T23:19:11.002 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mode", 2024-04-21T23:19:11.002 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.002 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.002 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.002 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "upmap", 2024-04-21T23:19:11.002 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.003 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.003 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.003 INFO:teuthology.orchestra.run.smithi044.stdout: "crush-compat", 2024-04-21T23:19:11.003 INFO:teuthology.orchestra.run.smithi044.stdout: "none", 2024-04-21T23:19:11.003 INFO:teuthology.orchestra.run.smithi044.stdout: "read", 2024-04-21T23:19:11.003 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap", 2024-04-21T23:19:11.004 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap-read" 2024-04-21T23:19:11.004 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.004 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Balancer mode", 2024-04-21T23:19:11.004 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.004 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.004 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.004 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.005 INFO:teuthology.orchestra.run.smithi044.stdout: "pool_ids": { 2024-04-21T23:19:11.005 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pool_ids", 2024-04-21T23:19:11.005 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.005 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.005 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.005 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.005 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.006 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.006 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.006 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-21T23:19:11.006 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.006 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.006 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.006 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.007 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-04-21T23:19:11.007 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-04-21T23:19:11.007 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:11.007 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.007 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.007 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-04-21T23:19:11.008 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.008 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.008 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.008 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-21T23:19:11.008 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.008 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.009 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.009 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.009 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap_max_deviation": { 2024-04-21T23:19:11.009 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "upmap_max_deviation", 2024-04-21T23:19:11.009 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.009 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.009 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.010 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-04-21T23:19:11.010 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-04-21T23:19:11.010 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.010 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.010 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-21T23:19:11.010 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-21T23:19:11.010 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.010 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.010 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.011 INFO:teuthology.orchestra.run.smithi044.stdout: "upmap_max_optimizations": { 2024-04-21T23:19:11.011 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "upmap_max_optimizations", 2024-04-21T23:19:11.011 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-04-21T23:19:11.011 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.011 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.011 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:11.011 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.011 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.012 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.012 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-21T23:19:11.012 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.012 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.012 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.012 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.012 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.012 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.012 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.013 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cephadm", 2024-04-21T23:19:11.013 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.013 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.013 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.013 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_down_multiplier": { 2024-04-21T23:19:11.013 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_down_multiplier", 2024-04-21T23:19:11.013 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:11.013 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.013 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.014 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3.0", 2024-04-21T23:19:11.014 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.014 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.014 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.014 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-21T23:19:11.014 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.014 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.014 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.015 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.015 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_refresh_rate": { 2024-04-21T23:19:11.015 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_refresh_rate", 2024-04-21T23:19:11.015 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:11.015 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.015 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.015 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "20", 2024-04-21T23:19:11.015 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.015 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.016 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.016 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-21T23:19:11.016 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.016 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.016 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.016 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.016 INFO:teuthology.orchestra.run.smithi044.stdout: "agent_starting_port": { 2024-04-21T23:19:11.016 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "agent_starting_port", 2024-04-21T23:19:11.017 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.017 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.017 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.017 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "4721", 2024-04-21T23:19:11.017 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.017 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.017 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.017 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-21T23:19:11.018 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.018 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.018 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.018 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.018 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_ptrace": { 2024-04-21T23:19:11.018 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_ptrace", 2024-04-21T23:19:11.018 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.018 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.018 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.019 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.019 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.019 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.019 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.019 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-21T23:19:11.019 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-21T23:19:11.019 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.019 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.020 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.020 INFO:teuthology.orchestra.run.smithi044.stdout: "autotune_interval": { 2024-04-21T23:19:11.020 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "autotune_interval", 2024-04-21T23:19:11.020 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:11.020 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.020 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.020 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-04-21T23:19:11.020 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.020 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.021 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.021 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-21T23:19:11.021 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.021 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.021 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.021 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.021 INFO:teuthology.orchestra.run.smithi044.stdout: "autotune_memory_target_ratio": { 2024-04-21T23:19:11.021 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "autotune_memory_target_ratio", 2024-04-21T23:19:11.022 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:11.022 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.022 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.022 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0.7", 2024-04-21T23:19:11.022 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.022 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.022 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.022 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-21T23:19:11.022 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.023 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.023 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.023 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.023 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm_log_destination": { 2024-04-21T23:19:11.023 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cephadm_log_destination", 2024-04-21T23:19:11.023 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.023 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.023 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.024 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.024 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.024 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.024 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.024 INFO:teuthology.orchestra.run.smithi044.stdout: "file", 2024-04-21T23:19:11.024 INFO:teuthology.orchestra.run.smithi044.stdout: "file,syslog", 2024-04-21T23:19:11.024 INFO:teuthology.orchestra.run.smithi044.stdout: "syslog" 2024-04-21T23:19:11.024 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.025 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-21T23:19:11.025 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.025 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.025 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.025 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.025 INFO:teuthology.orchestra.run.smithi044.stdout: "cgroups_split": { 2024-04-21T23:19:11.025 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cgroups_split", 2024-04-21T23:19:11.025 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.026 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.026 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.026 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.026 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.026 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.026 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.026 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-21T23:19:11.027 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.027 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.027 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.027 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.027 INFO:teuthology.orchestra.run.smithi044.stdout: "config_checks_enabled": { 2024-04-21T23:19:11.027 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_checks_enabled", 2024-04-21T23:19:11.027 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.027 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.028 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.028 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.028 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.028 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.028 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.028 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-21T23:19:11.028 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.028 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.029 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.029 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.029 INFO:teuthology.orchestra.run.smithi044.stdout: "config_dashboard": { 2024-04-21T23:19:11.029 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_dashboard", 2024-04-21T23:19:11.029 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.029 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.029 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.029 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.030 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.030 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.030 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.030 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-21T23:19:11.030 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.030 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.030 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.030 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.031 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_alertmanager": { 2024-04-21T23:19:11.031 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_alertmanager", 2024-04-21T23:19:11.031 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.031 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.031 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.031 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-21T23:19:11.032 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.032 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.032 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.032 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-04-21T23:19:11.032 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.032 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.032 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.032 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.033 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_base": { 2024-04-21T23:19:11.033 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_base", 2024-04-21T23:19:11.033 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.033 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.033 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.033 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-21T23:19:11.033 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.033 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.033 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.034 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Container image name, without the tag", 2024-04-21T23:19:11.034 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.034 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.034 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.034 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.034 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_elasticsearch": { 2024-04-21T23:19:11.034 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_elasticsearch", 2024-04-21T23:19:11.034 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.035 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.035 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.035 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-21T23:19:11.035 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.035 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.035 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.035 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "elasticsearch container image", 2024-04-21T23:19:11.035 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.035 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.036 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.036 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.036 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_grafana": { 2024-04-21T23:19:11.036 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_grafana", 2024-04-21T23:19:11.036 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.036 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.037 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.037 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-21T23:19:11.037 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.037 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.037 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.037 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-04-21T23:19:11.037 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.037 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.038 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.038 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.038 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_haproxy": { 2024-04-21T23:19:11.038 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_haproxy", 2024-04-21T23:19:11.038 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.038 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.039 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.039 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-21T23:19:11.039 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.039 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.039 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.039 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "HAproxy container image", 2024-04-21T23:19:11.039 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.039 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.040 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.040 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.040 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_agent": { 2024-04-21T23:19:11.040 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_agent", 2024-04-21T23:19:11.040 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.040 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.041 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.041 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-21T23:19:11.041 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.041 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.041 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.041 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger agent container image", 2024-04-21T23:19:11.041 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.042 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.042 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.042 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.042 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_collector": { 2024-04-21T23:19:11.042 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_collector", 2024-04-21T23:19:11.042 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.042 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.043 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.043 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-21T23:19:11.043 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.043 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.043 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.043 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger collector container image", 2024-04-21T23:19:11.044 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.044 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.044 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.044 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.044 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_jaeger_query": { 2024-04-21T23:19:11.044 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_jaeger_query", 2024-04-21T23:19:11.045 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.045 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.045 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.045 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-21T23:19:11.045 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.045 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.046 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.046 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Jaeger query container image", 2024-04-21T23:19:11.046 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.046 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.046 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.046 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.046 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_keepalived": { 2024-04-21T23:19:11.046 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_keepalived", 2024-04-21T23:19:11.047 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.047 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.047 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.047 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-21T23:19:11.047 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.047 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.047 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.047 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Keepalived container image", 2024-04-21T23:19:11.047 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.048 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.048 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.048 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.048 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_loki": { 2024-04-21T23:19:11.048 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_loki", 2024-04-21T23:19:11.048 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.048 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.048 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.048 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-21T23:19:11.049 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.049 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.049 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.049 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Loki container image", 2024-04-21T23:19:11.049 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.049 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.049 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.049 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.049 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_node_exporter": { 2024-04-21T23:19:11.050 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_node_exporter", 2024-04-21T23:19:11.050 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.050 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.050 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.050 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-21T23:19:11.050 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.050 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.050 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.051 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-04-21T23:19:11.051 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.051 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.051 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.051 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.051 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_nvmeof": { 2024-04-21T23:19:11.051 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_nvmeof", 2024-04-21T23:19:11.051 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.051 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.052 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.052 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-21T23:19:11.052 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.052 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.052 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.052 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Nvme-of container image", 2024-04-21T23:19:11.052 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.052 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.053 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.053 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.053 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_prometheus": { 2024-04-21T23:19:11.053 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_prometheus", 2024-04-21T23:19:11.053 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.053 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.053 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.053 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-21T23:19:11.054 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.054 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.054 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.054 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Prometheus container image", 2024-04-21T23:19:11.054 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.054 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.054 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.054 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.055 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_promtail": { 2024-04-21T23:19:11.055 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_promtail", 2024-04-21T23:19:11.055 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.055 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.055 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.055 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-21T23:19:11.055 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.055 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.055 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.056 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Promtail container image", 2024-04-21T23:19:11.056 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.056 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.056 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.056 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.056 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_samba": { 2024-04-21T23:19:11.056 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_samba", 2024-04-21T23:19:11.056 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.057 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.057 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.057 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-21T23:19:11.057 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.057 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.057 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.057 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Samba/SMB container image", 2024-04-21T23:19:11.057 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.058 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.058 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.058 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.058 INFO:teuthology.orchestra.run.smithi044.stdout: "container_image_snmp_gateway": { 2024-04-21T23:19:11.058 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_image_snmp_gateway", 2024-04-21T23:19:11.058 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.058 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.058 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.059 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-21T23:19:11.059 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.059 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.059 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.059 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "SNMP Gateway container image", 2024-04-21T23:19:11.059 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.059 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.060 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.060 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.060 INFO:teuthology.orchestra.run.smithi044.stdout: "container_init": { 2024-04-21T23:19:11.060 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "container_init", 2024-04-21T23:19:11.060 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.060 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.060 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.060 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.061 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.061 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.061 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.061 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Run podman/docker with `--init`", 2024-04-21T23:19:11.061 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.061 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.061 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.061 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.061 INFO:teuthology.orchestra.run.smithi044.stdout: "daemon_cache_timeout": { 2024-04-21T23:19:11.062 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "daemon_cache_timeout", 2024-04-21T23:19:11.062 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:11.062 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.062 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.062 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-04-21T23:19:11.062 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.062 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.063 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.063 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-21T23:19:11.063 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.063 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.063 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.063 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.063 INFO:teuthology.orchestra.run.smithi044.stdout: "default_cephadm_command_timeout": { 2024-04-21T23:19:11.063 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default_cephadm_command_timeout", 2024-04-21T23:19:11.063 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.064 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.064 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.064 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "900", 2024-04-21T23:19:11.064 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.064 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.064 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.064 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-21T23:19:11.064 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.065 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.065 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.065 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.065 INFO:teuthology.orchestra.run.smithi044.stdout: "default_registry": { 2024-04-21T23:19:11.065 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "default_registry", 2024-04-21T23:19:11.065 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.065 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.065 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.066 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "docker.io", 2024-04-21T23:19:11.066 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.066 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.066 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.066 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-21T23:19:11.066 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.066 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.066 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.066 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.067 INFO:teuthology.orchestra.run.smithi044.stdout: "device_cache_timeout": { 2024-04-21T23:19:11.067 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_cache_timeout", 2024-04-21T23:19:11.067 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:11.067 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.067 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.067 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1800", 2024-04-21T23:19:11.068 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.068 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.068 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.068 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache device inventory", 2024-04-21T23:19:11.068 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.068 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.068 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.068 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.069 INFO:teuthology.orchestra.run.smithi044.stdout: "device_enhanced_scan": { 2024-04-21T23:19:11.069 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_enhanced_scan", 2024-04-21T23:19:11.069 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.069 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.069 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.069 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.070 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.070 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.070 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.070 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-21T23:19:11.070 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.070 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.071 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.071 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.071 INFO:teuthology.orchestra.run.smithi044.stdout: "facts_cache_timeout": { 2024-04-21T23:19:11.071 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "facts_cache_timeout", 2024-04-21T23:19:11.071 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:11.071 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.071 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.072 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-04-21T23:19:11.072 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.072 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.072 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.072 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "seconds to cache host facts data", 2024-04-21T23:19:11.072 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.072 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.073 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.073 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.073 INFO:teuthology.orchestra.run.smithi044.stdout: "grafana_dashboards_path": { 2024-04-21T23:19:11.073 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "grafana_dashboards_path", 2024-04-21T23:19:11.073 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.073 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.074 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.074 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-21T23:19:11.074 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.074 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.074 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.074 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-21T23:19:11.074 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.075 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.075 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.075 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.075 INFO:teuthology.orchestra.run.smithi044.stdout: "host_check_interval": { 2024-04-21T23:19:11.075 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "host_check_interval", 2024-04-21T23:19:11.075 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:11.076 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.076 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.076 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-04-21T23:19:11.076 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.076 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.077 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.077 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to perform a host check", 2024-04-21T23:19:11.077 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.077 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.077 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.077 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.078 INFO:teuthology.orchestra.run.smithi044.stdout: "hw_monitoring": { 2024-04-21T23:19:11.078 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "hw_monitoring", 2024-04-21T23:19:11.078 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.078 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.078 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.078 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.079 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.079 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.079 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.079 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-21T23:19:11.079 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.079 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.079 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.080 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.080 INFO:teuthology.orchestra.run.smithi044.stdout: "inventory_list_all": { 2024-04-21T23:19:11.080 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "inventory_list_all", 2024-04-21T23:19:11.080 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.080 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.080 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.080 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.081 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.081 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.081 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.081 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-21T23:19:11.081 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.081 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.081 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.082 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.082 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.082 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.082 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.082 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.082 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.082 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.082 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.083 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.083 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.083 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.083 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.083 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.083 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.083 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.083 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.083 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.084 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.084 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.084 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.084 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.084 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.084 INFO:teuthology.orchestra.run.smithi044.stdout: "log_refresh_metadata": { 2024-04-21T23:19:11.084 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_refresh_metadata", 2024-04-21T23:19:11.084 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.085 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.085 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.085 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.085 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.085 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.085 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.085 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-21T23:19:11.085 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.086 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.086 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.086 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.086 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.086 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.086 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.086 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.086 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.087 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.087 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.087 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.087 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.087 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-21T23:19:11.087 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.087 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.087 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.087 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.088 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.088 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.088 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.088 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.088 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.088 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.088 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.088 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.088 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.089 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.089 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.089 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.089 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.089 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.089 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.089 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.089 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.090 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.090 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.090 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.090 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.090 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.090 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.090 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.090 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.091 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.091 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.091 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.091 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.091 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.091 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.092 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.092 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.092 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.092 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.092 INFO:teuthology.orchestra.run.smithi044.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-21T23:19:11.092 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-21T23:19:11.092 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.092 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.093 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.093 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.093 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.093 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.093 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.093 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-21T23:19:11.094 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.094 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.094 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.094 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.094 INFO:teuthology.orchestra.run.smithi044.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-21T23:19:11.094 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-21T23:19:11.094 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.095 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.095 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.095 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "*", 2024-04-21T23:19:11.095 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.095 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.095 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.095 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-21T23:19:11.096 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.096 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.096 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.096 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.096 INFO:teuthology.orchestra.run.smithi044.stdout: "max_count_per_host": { 2024-04-21T23:19:11.096 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_count_per_host", 2024-04-21T23:19:11.096 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.096 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.097 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.097 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:11.097 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.097 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.097 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.097 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "max number of daemons per service per host", 2024-04-21T23:19:11.098 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.098 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.098 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.098 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.098 INFO:teuthology.orchestra.run.smithi044.stdout: "max_osd_draining_count": { 2024-04-21T23:19:11.098 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_osd_draining_count", 2024-04-21T23:19:11.098 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.098 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.099 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.099 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:11.099 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.099 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.099 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.099 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-21T23:19:11.099 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.099 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.100 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.100 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.100 INFO:teuthology.orchestra.run.smithi044.stdout: "migration_current": { 2024-04-21T23:19:11.100 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "migration_current", 2024-04-21T23:19:11.100 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.100 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.100 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.101 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.101 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.101 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.101 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.101 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "internal - do not modify", 2024-04-21T23:19:11.101 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.101 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.101 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.102 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.102 INFO:teuthology.orchestra.run.smithi044.stdout: "mode": { 2024-04-21T23:19:11.102 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mode", 2024-04-21T23:19:11.102 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.102 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.102 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.102 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "root", 2024-04-21T23:19:11.103 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.103 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.103 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.103 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm-package", 2024-04-21T23:19:11.103 INFO:teuthology.orchestra.run.smithi044.stdout: "root" 2024-04-21T23:19:11.103 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.103 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "mode for remote execution of cephadm", 2024-04-21T23:19:11.104 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.104 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.104 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.104 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.104 INFO:teuthology.orchestra.run.smithi044.stdout: "oob_default_addr": { 2024-04-21T23:19:11.104 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "oob_default_addr", 2024-04-21T23:19:11.104 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.104 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.104 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.105 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "169.254.1.1", 2024-04-21T23:19:11.105 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.105 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.105 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.105 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-21T23:19:11.105 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.105 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.105 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.105 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.106 INFO:teuthology.orchestra.run.smithi044.stdout: "prometheus_alerts_path": { 2024-04-21T23:19:11.106 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus_alerts_path", 2024-04-21T23:19:11.106 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.106 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.106 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.106 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-21T23:19:11.106 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.106 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.107 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.107 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-21T23:19:11.107 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.107 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.107 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.107 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.107 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_insecure": { 2024-04-21T23:19:11.108 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_insecure", 2024-04-21T23:19:11.108 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.108 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.108 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.108 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.108 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.108 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.109 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.109 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-21T23:19:11.109 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.109 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.109 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.109 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.109 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_password": { 2024-04-21T23:19:11.110 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_password", 2024-04-21T23:19:11.110 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.110 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.110 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.110 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.110 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.110 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.111 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.111 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-21T23:19:11.111 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.111 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.111 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.111 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.112 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_url": { 2024-04-21T23:19:11.112 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_url", 2024-04-21T23:19:11.112 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.112 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.112 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.112 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.113 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.113 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.113 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.113 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-21T23:19:11.113 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.114 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.114 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.114 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.114 INFO:teuthology.orchestra.run.smithi044.stdout: "registry_username": { 2024-04-21T23:19:11.114 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "registry_username", 2024-04-21T23:19:11.115 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.115 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.115 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.115 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.116 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.116 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.116 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.116 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-21T23:19:11.116 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.116 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.116 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.116 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.116 INFO:teuthology.orchestra.run.smithi044.stdout: "secure_monitoring_stack": { 2024-04-21T23:19:11.117 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "secure_monitoring_stack", 2024-04-21T23:19:11.117 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.117 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.117 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.117 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.117 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.117 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.117 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.118 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-21T23:19:11.118 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.118 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.118 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.118 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.118 INFO:teuthology.orchestra.run.smithi044.stdout: "service_discovery_port": { 2024-04-21T23:19:11.118 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "service_discovery_port", 2024-04-21T23:19:11.119 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.119 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.119 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.119 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8765", 2024-04-21T23:19:11.119 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.119 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.119 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.119 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "cephadm service discovery port", 2024-04-21T23:19:11.119 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.120 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.120 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.120 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.120 INFO:teuthology.orchestra.run.smithi044.stdout: "ssh_config_file": { 2024-04-21T23:19:11.120 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssh_config_file", 2024-04-21T23:19:11.120 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.120 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.120 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.120 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.121 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.121 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.121 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.121 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-21T23:19:11.121 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.121 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.121 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.121 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.122 INFO:teuthology.orchestra.run.smithi044.stdout: "use_agent": { 2024-04-21T23:19:11.122 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "use_agent", 2024-04-21T23:19:11.122 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.122 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.122 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.122 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.122 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.122 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.123 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.123 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-21T23:19:11.123 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.123 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.123 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.123 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.123 INFO:teuthology.orchestra.run.smithi044.stdout: "use_repo_digest": { 2024-04-21T23:19:11.123 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "use_repo_digest", 2024-04-21T23:19:11.124 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.124 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.124 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.124 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.124 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.124 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.124 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.124 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-21T23:19:11.124 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.125 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.125 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.125 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.125 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_failed_host_check": { 2024-04-21T23:19:11.125 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_failed_host_check", 2024-04-21T23:19:11.125 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.125 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.125 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.125 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.126 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.126 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.126 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.126 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if the host check fails", 2024-04-21T23:19:11.126 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.126 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.127 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.127 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.127 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_stray_daemons": { 2024-04-21T23:19:11.127 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_stray_daemons", 2024-04-21T23:19:11.127 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.127 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.127 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.127 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.128 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.128 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.128 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.128 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-21T23:19:11.128 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.128 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.128 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.128 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.128 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_on_stray_hosts": { 2024-04-21T23:19:11.129 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_on_stray_hosts", 2024-04-21T23:19:11.129 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.129 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.129 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.129 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.129 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.129 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.129 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.129 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-21T23:19:11.130 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.130 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.130 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.130 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.130 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.130 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.130 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.130 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crash", 2024-04-21T23:19:11.131 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.131 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.131 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.131 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.131 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.131 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.131 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.131 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.131 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.132 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.132 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.132 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.132 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.132 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.132 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.132 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.132 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.132 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.133 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.133 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.133 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.133 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.133 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.133 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.133 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.133 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.133 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.134 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.134 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.134 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.134 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.134 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.134 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.134 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.134 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.135 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.135 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.135 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.135 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.135 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.135 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.135 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.135 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.135 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.136 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.136 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.136 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.136 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.136 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.136 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.136 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.136 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.136 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.137 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.137 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.137 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.137 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.137 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.137 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.138 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.138 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.138 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.138 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.138 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.138 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.138 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.138 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.139 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.139 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.139 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.139 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.139 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.139 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.140 INFO:teuthology.orchestra.run.smithi044.stdout: "retain_interval": { 2024-04-21T23:19:11.140 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "retain_interval", 2024-04-21T23:19:11.140 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:11.140 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.140 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.140 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "31536000", 2024-04-21T23:19:11.140 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.140 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.141 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.141 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-21T23:19:11.141 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.141 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.141 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.141 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.141 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_recent_interval": { 2024-04-21T23:19:11.142 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_recent_interval", 2024-04-21T23:19:11.142 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:11.142 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.142 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.142 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1209600", 2024-04-21T23:19:11.142 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.142 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.142 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.143 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-21T23:19:11.143 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.143 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.143 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.143 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.143 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.144 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.144 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.144 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "dashboard", 2024-04-21T23:19:11.144 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.144 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.144 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.144 INFO:teuthology.orchestra.run.smithi044.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-21T23:19:11.144 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-21T23:19:11.145 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.145 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.145 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.145 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:11.145 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.145 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.146 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.146 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.146 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.146 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.146 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.146 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.147 INFO:teuthology.orchestra.run.smithi044.stdout: "ALERTMANAGER_API_HOST": { 2024-04-21T23:19:11.147 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-21T23:19:11.147 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.147 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.147 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.147 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.148 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.148 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.148 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.148 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.148 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.148 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.149 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.149 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.149 INFO:teuthology.orchestra.run.smithi044.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-21T23:19:11.149 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-21T23:19:11.149 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.150 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.150 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.150 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.150 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.150 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.150 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.150 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.150 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.151 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.151 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.151 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.151 INFO:teuthology.orchestra.run.smithi044.stdout: "AUDIT_API_ENABLED": { 2024-04-21T23:19:11.151 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "AUDIT_API_ENABLED", 2024-04-21T23:19:11.151 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.151 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.151 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.151 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.152 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.152 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.152 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.152 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.152 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.152 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.152 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.152 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.152 INFO:teuthology.orchestra.run.smithi044.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-21T23:19:11.153 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-21T23:19:11.153 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.153 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.153 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.153 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.153 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.153 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.153 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.153 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.154 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.154 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.154 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.154 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.154 INFO:teuthology.orchestra.run.smithi044.stdout: "ENABLE_BROWSABLE_API": { 2024-04-21T23:19:11.154 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-21T23:19:11.154 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.154 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.154 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.155 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.155 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.155 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.155 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.155 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.155 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.155 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.155 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.155 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.156 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-21T23:19:11.156 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-21T23:19:11.156 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.156 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.156 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.156 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.156 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.156 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.157 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.157 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.157 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.157 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.157 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.157 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.157 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-21T23:19:11.157 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-21T23:19:11.157 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.158 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.158 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.158 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.158 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.158 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.158 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.158 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.158 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.159 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.159 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.159 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.159 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-21T23:19:11.159 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-21T23:19:11.159 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.159 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.159 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.160 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.160 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.160 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.160 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.160 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.160 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.160 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.160 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.161 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.161 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-21T23:19:11.161 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-21T23:19:11.161 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.161 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.161 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.161 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.161 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.162 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.162 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.162 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.162 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.162 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.162 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.162 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.163 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-21T23:19:11.163 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-21T23:19:11.163 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.163 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.163 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.163 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.163 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.163 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.164 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.164 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.164 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.164 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.164 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.164 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.164 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-21T23:19:11.164 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-21T23:19:11.165 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.165 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.165 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.165 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.165 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.165 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.165 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.165 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.166 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.166 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.166 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.166 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.166 INFO:teuthology.orchestra.run.smithi044.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-21T23:19:11.166 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-21T23:19:11.166 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.166 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.167 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.167 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.167 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.167 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.167 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.167 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.167 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.167 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.167 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.168 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.168 INFO:teuthology.orchestra.run.smithi044.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-21T23:19:11.168 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-21T23:19:11.168 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.168 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.168 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.168 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.168 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.168 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.169 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.169 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.169 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.169 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.169 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.169 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.169 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_PASSWORD": { 2024-04-21T23:19:11.169 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-21T23:19:11.170 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.170 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.170 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.170 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2024-04-21T23:19:11.170 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.170 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.170 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.171 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.171 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.171 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.171 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.171 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.171 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-21T23:19:11.171 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-21T23:19:11.172 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.172 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.172 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.172 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.172 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.172 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.172 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.172 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.172 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.173 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.173 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.173 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.173 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_URL": { 2024-04-21T23:19:11.173 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_URL", 2024-04-21T23:19:11.174 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.174 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.174 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.174 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.174 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.174 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.174 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.174 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.175 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.175 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.175 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.175 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.175 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_API_USERNAME": { 2024-04-21T23:19:11.175 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-21T23:19:11.176 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.176 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.176 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.176 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2024-04-21T23:19:11.176 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.176 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.176 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.176 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.177 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.177 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.177 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.177 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.177 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-21T23:19:11.177 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-21T23:19:11.178 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.178 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.178 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.178 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.178 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.178 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.178 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.178 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.179 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.179 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.179 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.179 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.179 INFO:teuthology.orchestra.run.smithi044.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-21T23:19:11.179 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-21T23:19:11.179 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.179 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.179 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.180 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.180 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.180 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.180 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.180 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.180 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.180 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.181 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.181 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.181 INFO:teuthology.orchestra.run.smithi044.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-21T23:19:11.181 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-21T23:19:11.181 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.181 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.181 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.182 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.182 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.182 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.182 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.182 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.182 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.183 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.183 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.183 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.183 INFO:teuthology.orchestra.run.smithi044.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-21T23:19:11.183 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-21T23:19:11.184 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.184 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.184 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.184 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.184 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.185 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.185 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.185 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.185 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.185 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.185 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.186 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.186 INFO:teuthology.orchestra.run.smithi044.stdout: "PROMETHEUS_API_HOST": { 2024-04-21T23:19:11.186 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-21T23:19:11.186 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.186 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.186 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.186 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.186 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.187 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.187 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.187 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.187 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.187 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.187 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.187 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.187 INFO:teuthology.orchestra.run.smithi044.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-21T23:19:11.187 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-21T23:19:11.188 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.188 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.188 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.188 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.188 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.188 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.188 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.188 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.189 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.189 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.189 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.189 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.190 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-21T23:19:11.190 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-21T23:19:11.190 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.190 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.190 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.190 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.190 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.191 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.191 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.191 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.191 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.191 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.191 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.192 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.192 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-21T23:19:11.192 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-21T23:19:11.192 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.192 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.192 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.192 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.193 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.193 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.193 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.193 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.193 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.193 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.193 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.193 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.193 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-21T23:19:11.194 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-21T23:19:11.194 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.194 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.194 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.194 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.194 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.194 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.194 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.195 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.195 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.195 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.195 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.195 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.195 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-21T23:19:11.195 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-21T23:19:11.195 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.196 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.196 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.196 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.196 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.196 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.196 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.196 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.196 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.197 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.197 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.197 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.197 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-21T23:19:11.197 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-21T23:19:11.197 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.197 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.197 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.197 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.198 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.198 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.198 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.198 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.198 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.198 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.198 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.198 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.199 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-21T23:19:11.199 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-21T23:19:11.199 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.199 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.199 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.199 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.199 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.199 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.200 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.200 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.200 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.200 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.200 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.200 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.200 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-21T23:19:11.200 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-21T23:19:11.200 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.201 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.201 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.201 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.201 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.201 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.201 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.201 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.201 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.202 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.202 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.202 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.202 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_ENABLED": { 2024-04-21T23:19:11.202 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-21T23:19:11.202 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.202 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.202 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.202 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.203 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.203 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.203 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.203 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.203 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.203 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.203 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.203 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.203 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-21T23:19:11.204 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-21T23:19:11.204 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.204 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.204 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.204 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-21T23:19:11.204 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.204 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.205 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.205 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.205 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.205 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.205 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.205 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.205 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-21T23:19:11.205 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-21T23:19:11.205 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.206 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.206 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.206 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:11.206 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.206 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.206 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.206 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.207 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.207 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.207 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.207 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.207 INFO:teuthology.orchestra.run.smithi044.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-21T23:19:11.207 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-21T23:19:11.207 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.208 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.208 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.208 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8", 2024-04-21T23:19:11.208 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.208 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.208 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.208 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.208 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.208 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.209 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.209 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.209 INFO:teuthology.orchestra.run.smithi044.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-21T23:19:11.209 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-21T23:19:11.209 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.209 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.209 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.209 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "45", 2024-04-21T23:19:11.209 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.210 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.210 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.210 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.210 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.210 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.210 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.210 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.210 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_ACCESS_KEY": { 2024-04-21T23:19:11.211 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-21T23:19:11.211 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.211 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.211 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.211 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.211 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.211 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.211 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.211 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.212 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.212 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.212 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.212 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.212 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-21T23:19:11.212 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-21T23:19:11.212 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.212 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.212 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.213 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "admin", 2024-04-21T23:19:11.213 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.213 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.213 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.213 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.213 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.213 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.213 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.213 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.214 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_SECRET_KEY": { 2024-04-21T23:19:11.214 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-21T23:19:11.214 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.214 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.214 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.214 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.214 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.214 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.215 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.215 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.215 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.215 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.215 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.215 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.215 INFO:teuthology.orchestra.run.smithi044.stdout: "RGW_API_SSL_VERIFY": { 2024-04-21T23:19:11.215 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-21T23:19:11.215 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.216 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.216 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.216 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.216 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.216 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.216 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.216 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.216 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.216 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.217 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.217 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.217 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-21T23:19:11.217 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-21T23:19:11.217 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.217 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.217 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.217 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-04-21T23:19:11.217 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.218 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.218 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.218 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.218 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.218 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.218 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.218 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.218 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-21T23:19:11.219 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-21T23:19:11.219 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.219 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.219 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.219 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:11.219 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.219 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.219 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.219 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.220 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.220 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.220 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.220 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.220 INFO:teuthology.orchestra.run.smithi044.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-21T23:19:11.220 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-21T23:19:11.220 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.220 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.220 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.221 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-04-21T23:19:11.221 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.221 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.221 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.221 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.221 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.221 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.221 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.222 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.222 INFO:teuthology.orchestra.run.smithi044.stdout: "cross_origin_url": { 2024-04-21T23:19:11.222 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cross_origin_url", 2024-04-21T23:19:11.222 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.222 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.222 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.222 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.222 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.222 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.223 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.223 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.223 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.223 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.223 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.223 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.223 INFO:teuthology.orchestra.run.smithi044.stdout: "crt_file": { 2024-04-21T23:19:11.224 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "crt_file", 2024-04-21T23:19:11.224 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.224 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.224 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.224 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.224 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.224 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.225 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.225 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.225 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.225 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.225 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.225 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.225 INFO:teuthology.orchestra.run.smithi044.stdout: "debug": { 2024-04-21T23:19:11.226 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "debug", 2024-04-21T23:19:11.226 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.226 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.226 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.226 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.226 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.226 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.227 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.227 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Enable/disable debug options", 2024-04-21T23:19:11.227 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.227 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.227 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.227 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.228 INFO:teuthology.orchestra.run.smithi044.stdout: "jwt_token_ttl": { 2024-04-21T23:19:11.228 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "jwt_token_ttl", 2024-04-21T23:19:11.228 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.228 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.228 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.228 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "28800", 2024-04-21T23:19:11.228 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.229 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.229 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.229 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.229 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.229 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.229 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.229 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.229 INFO:teuthology.orchestra.run.smithi044.stdout: "key_file": { 2024-04-21T23:19:11.230 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "key_file", 2024-04-21T23:19:11.230 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.230 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.230 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.230 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.230 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.231 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.231 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.231 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.231 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.231 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.231 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.231 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.232 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.232 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.232 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.232 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.232 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.232 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.232 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.233 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.233 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.233 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.233 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.233 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.233 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.233 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.234 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.234 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.234 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.234 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.234 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.234 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.234 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.235 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.235 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.235 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.235 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.235 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.235 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.235 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.236 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.236 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.236 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.236 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.236 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.236 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.236 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.237 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.237 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.237 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.237 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.237 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.237 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.238 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.238 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.238 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.238 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.238 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.238 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.238 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.239 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.239 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.239 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.239 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.239 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.239 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.240 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.240 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.240 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.240 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.240 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.240 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.240 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.240 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.241 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.241 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.241 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.241 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.241 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.241 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.241 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.242 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.242 INFO:teuthology.orchestra.run.smithi044.stdout: "motd": { 2024-04-21T23:19:11.242 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "motd", 2024-04-21T23:19:11.242 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.242 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.243 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.243 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.243 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.243 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.243 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.244 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "The message of the day", 2024-04-21T23:19:11.244 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.244 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.244 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.244 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.245 INFO:teuthology.orchestra.run.smithi044.stdout: "redirect_resolve_ip_addr": { 2024-04-21T23:19:11.245 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "redirect_resolve_ip_addr", 2024-04-21T23:19:11.245 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.245 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.245 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.245 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.245 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.245 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.246 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.246 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.246 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.246 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.246 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.246 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.247 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2024-04-21T23:19:11.247 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2024-04-21T23:19:11.247 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.247 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.247 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.247 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "::", 2024-04-21T23:19:11.247 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.248 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.248 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.248 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.248 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.248 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.249 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.249 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.249 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2024-04-21T23:19:11.249 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2024-04-21T23:19:11.249 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.249 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.249 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.250 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8080", 2024-04-21T23:19:11.250 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.250 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.250 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.250 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.250 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.250 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.250 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.251 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.251 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl": { 2024-04-21T23:19:11.251 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl", 2024-04-21T23:19:11.251 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.251 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.251 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.251 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.252 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.252 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.252 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.252 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.252 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.252 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.252 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.253 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.253 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl_server_port": { 2024-04-21T23:19:11.253 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl_server_port", 2024-04-21T23:19:11.253 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.253 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.253 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.253 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8443", 2024-04-21T23:19:11.253 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.254 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.254 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.254 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.254 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.254 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.254 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.254 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.255 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_behaviour": { 2024-04-21T23:19:11.255 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_behaviour", 2024-04-21T23:19:11.255 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.255 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.255 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.255 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "redirect", 2024-04-21T23:19:11.255 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.256 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.256 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.256 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.256 INFO:teuthology.orchestra.run.smithi044.stdout: "redirect" 2024-04-21T23:19:11.256 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.256 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.256 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.257 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.257 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.257 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.257 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_error_status_code": { 2024-04-21T23:19:11.257 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_error_status_code", 2024-04-21T23:19:11.257 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.257 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.258 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.258 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "500", 2024-04-21T23:19:11.258 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "400", 2024-04-21T23:19:11.258 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "599", 2024-04-21T23:19:11.258 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.258 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.258 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.259 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.259 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.259 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.259 INFO:teuthology.orchestra.run.smithi044.stdout: "url_prefix": { 2024-04-21T23:19:11.259 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "url_prefix", 2024-04-21T23:19:11.259 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.259 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.260 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.260 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.260 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.260 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.260 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.260 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.260 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.260 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.261 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.261 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.261 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.261 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.261 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.261 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "devicehealth", 2024-04-21T23:19:11.262 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.262 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.262 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.262 INFO:teuthology.orchestra.run.smithi044.stdout: "enable_monitoring": { 2024-04-21T23:19:11.262 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enable_monitoring", 2024-04-21T23:19:11.262 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.262 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.263 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.263 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.263 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.263 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.263 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.263 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "monitor device health metrics", 2024-04-21T23:19:11.263 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.264 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.264 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.264 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.264 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.264 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.264 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.264 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.265 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.265 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.265 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.265 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.265 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.265 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.265 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.266 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.266 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.266 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.266 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.266 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.266 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.266 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.267 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.267 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.267 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.267 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.267 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.267 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.268 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.268 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.268 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.268 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.268 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.268 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.268 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.269 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.269 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.269 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.269 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.269 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.269 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.269 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.270 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.270 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.270 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.270 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.270 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.270 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.270 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.271 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.271 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.271 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.271 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.271 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.271 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.271 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.272 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.272 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.272 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.272 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.272 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.272 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.273 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.273 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.273 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.273 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.273 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.273 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.273 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.274 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.274 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.274 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.274 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.274 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.274 INFO:teuthology.orchestra.run.smithi044.stdout: "mark_out_threshold": { 2024-04-21T23:19:11.275 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mark_out_threshold", 2024-04-21T23:19:11.275 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:11.275 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.275 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.275 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "2419200", 2024-04-21T23:19:11.275 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.275 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.276 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.276 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-21T23:19:11.276 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.276 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.276 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.276 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.276 INFO:teuthology.orchestra.run.smithi044.stdout: "pool_name": { 2024-04-21T23:19:11.276 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pool_name", 2024-04-21T23:19:11.277 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.277 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.277 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.277 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "device_health_metrics", 2024-04-21T23:19:11.277 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.277 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.278 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.278 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-21T23:19:11.278 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.278 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.278 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.278 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.278 INFO:teuthology.orchestra.run.smithi044.stdout: "retention_period": { 2024-04-21T23:19:11.279 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "retention_period", 2024-04-21T23:19:11.279 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:11.279 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.279 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.279 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15552000", 2024-04-21T23:19:11.279 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.279 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.280 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.280 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long to retain device health metrics", 2024-04-21T23:19:11.280 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.280 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.280 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.280 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.280 INFO:teuthology.orchestra.run.smithi044.stdout: "scrape_frequency": { 2024-04-21T23:19:11.281 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "scrape_frequency", 2024-04-21T23:19:11.281 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:11.281 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.281 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.281 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "86400", 2024-04-21T23:19:11.281 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.282 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.282 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.282 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-21T23:19:11.282 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.282 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.282 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.282 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.283 INFO:teuthology.orchestra.run.smithi044.stdout: "self_heal": { 2024-04-21T23:19:11.283 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "self_heal", 2024-04-21T23:19:11.283 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.283 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.283 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.283 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.283 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.284 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.284 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.284 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-21T23:19:11.284 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.284 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.284 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.285 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.285 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-04-21T23:19:11.285 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-04-21T23:19:11.285 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:11.285 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.285 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.285 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-04-21T23:19:11.286 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.286 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.286 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.286 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how frequently to wake up and check device health", 2024-04-21T23:19:11.286 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.286 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.287 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.287 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.287 INFO:teuthology.orchestra.run.smithi044.stdout: "warn_threshold": { 2024-04-21T23:19:11.287 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "warn_threshold", 2024-04-21T23:19:11.287 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:11.287 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.287 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.288 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "7257600", 2024-04-21T23:19:11.288 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.288 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.288 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.288 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-21T23:19:11.288 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.289 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.289 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.289 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.289 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.289 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.289 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.289 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "diskprediction_local", 2024-04-21T23:19:11.290 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.290 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.290 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.290 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.290 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.290 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.291 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.291 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.291 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.291 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.291 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.291 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.291 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.292 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.292 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.292 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.292 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.292 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.292 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.292 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.293 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.293 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.293 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.293 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.293 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.293 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.293 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.294 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.294 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.294 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.294 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.294 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.294 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.295 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.295 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.295 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.295 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.295 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.295 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.295 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.295 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.296 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.296 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.296 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.296 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.296 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.296 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.297 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.297 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.297 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.297 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.297 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.297 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.297 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.298 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.298 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.298 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.298 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.298 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.298 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.298 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.299 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.299 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.299 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.299 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.299 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.299 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.300 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.300 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.300 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.300 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.300 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.300 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.300 INFO:teuthology.orchestra.run.smithi044.stdout: "predict_interval": { 2024-04-21T23:19:11.301 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "predict_interval", 2024-04-21T23:19:11.301 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.301 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.301 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.301 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "86400", 2024-04-21T23:19:11.302 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.302 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.302 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.302 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.302 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.303 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.303 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.303 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.303 INFO:teuthology.orchestra.run.smithi044.stdout: "predictor_model": { 2024-04-21T23:19:11.303 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "predictor_model", 2024-04-21T23:19:11.303 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.303 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.304 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.304 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "prophetstor", 2024-04-21T23:19:11.304 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.304 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.304 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.304 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.305 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.305 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.305 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.305 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.305 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-04-21T23:19:11.305 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-04-21T23:19:11.306 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.306 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.306 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.306 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "600", 2024-04-21T23:19:11.306 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.306 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.306 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.307 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.307 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.307 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.307 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.307 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.307 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.307 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.307 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.308 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "influx", 2024-04-21T23:19:11.308 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": false, 2024-04-21T23:19:11.308 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "influxdb python module not found", 2024-04-21T23:19:11.308 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.308 INFO:teuthology.orchestra.run.smithi044.stdout: "batch_size": { 2024-04-21T23:19:11.308 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "batch_size", 2024-04-21T23:19:11.309 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.309 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.309 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.309 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5000", 2024-04-21T23:19:11.309 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.309 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.309 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.310 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-21T23:19:11.310 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.310 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.310 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.310 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.310 INFO:teuthology.orchestra.run.smithi044.stdout: "database": { 2024-04-21T23:19:11.311 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "database", 2024-04-21T23:19:11.311 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.311 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.311 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.311 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "ceph", 2024-04-21T23:19:11.311 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.311 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.312 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.312 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-21T23:19:11.312 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.312 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.312 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.312 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.313 INFO:teuthology.orchestra.run.smithi044.stdout: "hostname": { 2024-04-21T23:19:11.313 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "hostname", 2024-04-21T23:19:11.313 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.313 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.313 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.313 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.313 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.313 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.314 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.314 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB server hostname", 2024-04-21T23:19:11.314 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.314 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.314 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.314 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.314 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-04-21T23:19:11.315 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-04-21T23:19:11.315 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:11.315 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.315 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.315 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "30", 2024-04-21T23:19:11.315 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "5", 2024-04-21T23:19:11.315 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.316 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.316 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-21T23:19:11.316 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.316 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.316 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.316 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.317 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.317 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.317 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.317 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.317 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.317 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.317 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.317 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.318 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.318 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.318 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.318 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.318 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.318 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.319 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.319 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.319 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.319 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.319 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.319 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.319 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.320 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.320 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.320 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.320 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.320 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.320 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.320 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.321 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.321 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.321 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.321 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.321 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.321 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.321 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.322 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.322 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.322 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.322 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.322 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.322 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.322 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.323 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.323 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.323 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.323 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.323 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.323 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.323 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.324 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.324 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.324 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.324 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.324 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.324 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.325 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.325 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.325 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.325 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.325 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.325 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.325 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.326 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.326 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.326 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.326 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.326 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.326 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.327 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.327 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.327 INFO:teuthology.orchestra.run.smithi044.stdout: "password": { 2024-04-21T23:19:11.327 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "password", 2024-04-21T23:19:11.327 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.327 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.327 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.327 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.328 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.328 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.328 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.328 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "password of InfluxDB server user", 2024-04-21T23:19:11.328 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.328 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.328 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.329 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.329 INFO:teuthology.orchestra.run.smithi044.stdout: "port": { 2024-04-21T23:19:11.329 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "port", 2024-04-21T23:19:11.329 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.329 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.329 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.330 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "8086", 2024-04-21T23:19:11.330 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.330 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.330 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.330 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "InfluxDB server port", 2024-04-21T23:19:11.330 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.331 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.331 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.331 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.331 INFO:teuthology.orchestra.run.smithi044.stdout: "ssl": { 2024-04-21T23:19:11.331 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ssl", 2024-04-21T23:19:11.331 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.331 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.332 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.332 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "false", 2024-04-21T23:19:11.332 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.332 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.332 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.332 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T23:19:11.332 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.332 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.333 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.333 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.333 INFO:teuthology.orchestra.run.smithi044.stdout: "threads": { 2024-04-21T23:19:11.333 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "threads", 2024-04-21T23:19:11.333 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.333 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.333 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.334 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-04-21T23:19:11.334 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-04-21T23:19:11.334 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "32", 2024-04-21T23:19:11.334 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.334 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-21T23:19:11.334 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.335 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.335 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.335 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.335 INFO:teuthology.orchestra.run.smithi044.stdout: "username": { 2024-04-21T23:19:11.335 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "username", 2024-04-21T23:19:11.335 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.336 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.336 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.336 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.336 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.336 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.336 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.336 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "username of InfluxDB server user", 2024-04-21T23:19:11.337 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.337 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.337 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.337 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.337 INFO:teuthology.orchestra.run.smithi044.stdout: "verify_ssl": { 2024-04-21T23:19:11.337 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "verify_ssl", 2024-04-21T23:19:11.338 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.338 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.338 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.338 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "true", 2024-04-21T23:19:11.338 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.338 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.339 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.339 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T23:19:11.339 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.339 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.339 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.339 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.339 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.339 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.340 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.340 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "insights", 2024-04-21T23:19:11.340 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.340 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.340 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.340 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.340 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.341 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.341 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.341 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.341 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.341 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.341 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.342 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.342 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.342 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.342 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.342 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.342 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.342 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.343 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.343 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.343 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.343 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.343 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.343 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.343 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.344 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.344 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.344 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.344 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.344 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.344 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.344 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.345 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.345 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.345 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.345 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.345 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.345 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.345 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.346 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.346 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.346 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.346 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.346 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.346 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.347 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.347 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.347 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.347 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.347 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.347 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.347 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.348 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.348 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.348 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.348 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.348 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.348 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.349 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.349 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.349 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.349 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.349 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.349 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.349 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.349 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.350 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.350 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.350 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.350 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.350 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.350 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.350 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.351 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.351 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.351 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.351 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "iostat", 2024-04-21T23:19:11.351 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.351 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.351 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.352 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.352 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.352 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.352 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.352 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.352 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.352 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.353 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.353 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.353 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.353 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.353 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.353 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.353 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.353 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.354 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.354 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.354 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.354 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.354 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.354 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.355 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.355 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.355 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.355 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.355 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.355 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.356 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.356 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.356 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.356 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.356 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.356 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.357 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.357 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.357 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.357 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.357 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.357 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.358 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.358 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.358 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.358 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.358 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.358 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.358 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.359 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.359 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.359 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.359 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.359 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.359 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.359 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.360 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.360 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.360 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.360 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.360 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.360 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.361 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.361 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.361 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.361 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.361 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.361 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.361 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.362 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.362 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.362 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.362 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.362 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.362 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.362 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.363 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "k8sevents", 2024-04-21T23:19:11.363 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.363 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.363 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.363 INFO:teuthology.orchestra.run.smithi044.stdout: "ceph_event_retention_days": { 2024-04-21T23:19:11.363 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "ceph_event_retention_days", 2024-04-21T23:19:11.363 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.364 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.364 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.364 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "7", 2024-04-21T23:19:11.364 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.364 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.364 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.364 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-21T23:19:11.365 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.365 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.365 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.365 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.365 INFO:teuthology.orchestra.run.smithi044.stdout: "config_check_secs": { 2024-04-21T23:19:11.365 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "config_check_secs", 2024-04-21T23:19:11.366 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.366 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.366 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.366 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:11.366 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "10", 2024-04-21T23:19:11.366 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.366 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.367 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-21T23:19:11.367 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.367 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.367 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.367 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.367 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.367 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.368 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.368 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.368 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.368 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.368 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.368 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.368 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.369 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.369 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.369 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.369 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.369 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.369 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.369 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.370 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.370 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.370 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.370 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.370 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.370 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.371 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.371 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.371 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.371 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.371 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.371 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.371 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.372 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.372 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.372 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.372 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.372 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.372 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.372 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.373 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.373 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.373 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.373 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.373 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.373 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.374 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.374 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.374 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.374 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.374 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.374 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.374 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.375 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.375 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.375 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.375 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.375 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.375 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.375 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.376 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.376 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.376 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.376 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.376 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.376 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.376 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.377 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.377 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.377 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.377 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.377 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.377 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.377 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.378 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.378 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.378 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.378 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "localpool", 2024-04-21T23:19:11.378 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.378 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.378 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.379 INFO:teuthology.orchestra.run.smithi044.stdout: "failure_domain": { 2024-04-21T23:19:11.379 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "failure_domain", 2024-04-21T23:19:11.379 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.379 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.379 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.379 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "host", 2024-04-21T23:19:11.379 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.380 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.380 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.380 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "failure domain for any created local pool", 2024-04-21T23:19:11.380 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-21T23:19:11.380 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.380 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.380 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.381 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.381 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.381 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.381 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.381 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.381 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.381 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.381 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.382 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.382 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.382 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.382 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.382 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.382 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.383 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.383 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.383 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.383 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.383 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.383 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.383 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.384 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.384 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.384 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.384 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.384 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.384 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.384 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.385 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.385 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.385 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.385 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.385 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.385 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.385 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.386 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.386 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.386 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.386 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.386 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.386 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.386 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.387 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.387 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.387 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.387 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.387 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.387 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.387 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.387 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.388 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.388 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.388 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.388 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.388 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.388 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.389 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.389 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.389 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.389 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.389 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.389 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.389 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.390 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.390 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.390 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.390 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.390 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.390 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.390 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.391 INFO:teuthology.orchestra.run.smithi044.stdout: "min_size": { 2024-04-21T23:19:11.391 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "min_size", 2024-04-21T23:19:11.391 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.391 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.391 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.391 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.391 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.392 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.392 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.392 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default min_size for any created local pool", 2024-04-21T23:19:11.392 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-21T23:19:11.392 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.392 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.392 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.393 INFO:teuthology.orchestra.run.smithi044.stdout: "num_rep": { 2024-04-21T23:19:11.393 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "num_rep", 2024-04-21T23:19:11.393 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.393 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.393 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.393 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3", 2024-04-21T23:19:11.393 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.394 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.394 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.394 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default replica count for any created local pool", 2024-04-21T23:19:11.394 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.394 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.394 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.394 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.395 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_num": { 2024-04-21T23:19:11.395 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pg_num", 2024-04-21T23:19:11.395 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.395 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.395 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.395 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "128", 2024-04-21T23:19:11.395 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.396 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.396 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.396 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "default pg_num for any created local pool", 2024-04-21T23:19:11.396 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.396 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.396 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.396 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.397 INFO:teuthology.orchestra.run.smithi044.stdout: "prefix": { 2024-04-21T23:19:11.397 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prefix", 2024-04-21T23:19:11.397 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.397 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.397 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.397 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.397 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.398 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.398 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.398 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "name prefix for any created local pool", 2024-04-21T23:19:11.398 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.398 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.398 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.398 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.399 INFO:teuthology.orchestra.run.smithi044.stdout: "subtree": { 2024-04-21T23:19:11.399 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "subtree", 2024-04-21T23:19:11.399 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.399 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.399 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.399 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "rack", 2024-04-21T23:19:11.399 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.400 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.400 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.400 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-21T23:19:11.400 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-21T23:19:11.400 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.400 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.400 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.401 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.401 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.401 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.401 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mds_autoscaler", 2024-04-21T23:19:11.401 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.401 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.401 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.402 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.402 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.402 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.402 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.402 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.402 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.403 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.403 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.403 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.403 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.403 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.403 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.403 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.404 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.404 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.404 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.404 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.404 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.404 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.405 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.405 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.405 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.405 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.405 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.405 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.405 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.406 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.406 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.406 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.406 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.406 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.406 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.406 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.407 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.407 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.407 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.407 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.407 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.407 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.408 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.408 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.408 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.408 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.408 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.408 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.408 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.409 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.409 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.409 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.409 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.409 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.409 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.410 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.410 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.410 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.410 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.410 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.410 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.411 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.411 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.411 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.411 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.411 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.411 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.411 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.411 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.412 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.412 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.412 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.412 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.412 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.412 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.412 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.412 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mirroring", 2024-04-21T23:19:11.412 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.413 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.413 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.413 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.413 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.413 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.413 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.413 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.413 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.413 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.414 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.414 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.414 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.414 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.414 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.414 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.414 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.414 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.415 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.415 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.415 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.415 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.415 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.415 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.415 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.415 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.415 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.416 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.416 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.416 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.416 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.416 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.416 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.416 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.416 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.417 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.417 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.417 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.417 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.417 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.417 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.417 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.417 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.418 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.418 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.418 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.418 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.418 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.418 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.418 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.418 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.418 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.419 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.419 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.419 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.419 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.419 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.419 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.419 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.419 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.420 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.420 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.420 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.420 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.420 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.420 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.420 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.420 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.420 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.421 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.421 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.421 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.421 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.421 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.421 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.421 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.421 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "nfs", 2024-04-21T23:19:11.422 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.422 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.422 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.422 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.422 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.422 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.422 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.422 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.422 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.423 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.423 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.423 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.423 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.423 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.423 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.423 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.423 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.424 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.424 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.424 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.424 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.424 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.424 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.424 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.424 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.424 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.425 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.425 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.425 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.425 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.425 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.425 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.425 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.425 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.425 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.426 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.426 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.426 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.426 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.426 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.426 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.426 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.426 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.427 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.427 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.427 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.427 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.427 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.427 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.427 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.427 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.427 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.428 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.428 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.428 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.428 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.428 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.428 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.428 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.428 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.429 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.429 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.429 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.429 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.429 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.429 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.429 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.429 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.429 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.430 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.430 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.430 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.430 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.430 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.430 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.430 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.430 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "orchestrator", 2024-04-21T23:19:11.431 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.431 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.431 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.431 INFO:teuthology.orchestra.run.smithi044.stdout: "fail_fs": { 2024-04-21T23:19:11.431 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "fail_fs", 2024-04-21T23:19:11.431 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.431 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.431 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.432 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.432 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.432 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.432 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.432 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-21T23:19:11.432 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.432 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.432 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.432 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.433 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.433 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.433 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.433 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.433 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.433 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.433 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.433 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.434 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.434 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.434 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.434 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.434 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.434 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.434 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.435 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.435 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.435 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.435 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.435 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.435 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.435 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.436 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.436 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.436 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.436 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.436 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.436 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.436 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.436 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.437 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.437 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.437 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.437 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.437 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.437 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.437 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.438 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.438 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.438 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.438 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.438 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.438 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.438 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.439 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.439 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.439 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.439 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.439 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.439 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.439 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.440 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.440 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.440 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.440 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.440 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.440 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.440 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.441 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.441 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.441 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.441 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.441 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.441 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.441 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.442 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.442 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.442 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.442 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.442 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.443 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator": { 2024-04-21T23:19:11.443 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "orchestrator", 2024-04-21T23:19:11.443 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.443 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.443 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.443 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.443 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.444 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.444 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.444 INFO:teuthology.orchestra.run.smithi044.stdout: "cephadm", 2024-04-21T23:19:11.444 INFO:teuthology.orchestra.run.smithi044.stdout: "rook", 2024-04-21T23:19:11.444 INFO:teuthology.orchestra.run.smithi044.stdout: "test_orchestrator" 2024-04-21T23:19:11.444 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.445 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Orchestrator backend", 2024-04-21T23:19:11.445 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.445 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.445 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.445 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.445 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.445 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.446 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.446 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "osd_perf_query", 2024-04-21T23:19:11.446 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.446 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.446 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.446 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.446 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.446 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.447 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.447 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.447 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.447 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.447 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.447 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.447 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.447 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.447 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.448 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.448 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.448 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.448 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.448 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.448 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.448 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.448 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.448 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.449 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.449 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.449 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.449 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.449 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.449 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.449 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.449 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.449 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.450 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.450 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.450 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.450 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.450 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.450 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.450 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.450 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.451 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.451 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.451 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.451 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.451 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.451 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.451 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.451 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.451 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.452 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.452 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.452 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.452 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.452 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.452 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.452 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.452 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.452 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.453 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.453 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.453 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.453 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.453 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.453 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.453 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.453 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.453 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.454 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.454 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.454 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.454 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.454 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.454 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.454 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.454 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.455 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "osd_support", 2024-04-21T23:19:11.455 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.455 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.455 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.455 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.455 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.455 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.455 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.455 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.456 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.456 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.456 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.456 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.456 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.456 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.456 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.456 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.456 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.457 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.457 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.457 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.457 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.457 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.457 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.457 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.457 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.458 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.458 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.458 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.458 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.458 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.458 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.458 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.458 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.458 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.459 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.459 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.459 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.459 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.459 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.459 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.459 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.459 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.459 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.460 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.460 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.460 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.460 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.460 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.460 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.460 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.460 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.461 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.461 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.461 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.461 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.461 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.461 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.461 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.461 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.461 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.462 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.462 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.462 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.462 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.462 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.462 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.462 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.462 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.462 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.463 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.463 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.463 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.463 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.463 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.463 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.463 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.463 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "pg_autoscaler", 2024-04-21T23:19:11.463 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.464 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.464 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.464 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.464 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.464 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.464 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.464 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.464 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.465 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.465 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.465 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.465 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.465 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.465 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.465 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.465 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.466 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.466 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.466 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.466 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.466 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.466 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.467 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.467 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.467 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.467 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.467 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.467 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.467 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.467 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.467 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.468 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.468 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.468 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.468 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.468 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.469 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.469 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.469 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.469 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.469 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.469 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.469 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.469 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.469 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.470 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.470 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.470 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.470 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.470 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.470 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.471 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.471 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.471 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.471 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.471 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.471 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.471 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.471 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.472 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.472 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.472 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.472 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.472 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.472 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.472 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.473 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.473 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.473 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.473 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.473 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.473 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.473 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-04-21T23:19:11.474 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-04-21T23:19:11.474 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:11.474 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.474 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.474 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-04-21T23:19:11.474 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.474 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.475 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.475 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.475 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.475 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.475 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.475 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.476 INFO:teuthology.orchestra.run.smithi044.stdout: "threshold": { 2024-04-21T23:19:11.476 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "threshold", 2024-04-21T23:19:11.476 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:11.476 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.476 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.476 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "3.0", 2024-04-21T23:19:11.476 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1.0", 2024-04-21T23:19:11.477 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.477 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.477 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "scaling threshold", 2024-04-21T23:19:11.477 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-21T23:19:11.477 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.477 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.477 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.477 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.478 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.478 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.478 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "progress", 2024-04-21T23:19:11.478 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.478 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.478 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.479 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_pg_recovery_event": { 2024-04-21T23:19:11.479 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_pg_recovery_event", 2024-04-21T23:19:11.479 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.479 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.479 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.479 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.479 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.480 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.480 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.480 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-21T23:19:11.480 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.480 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.481 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.481 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.481 INFO:teuthology.orchestra.run.smithi044.stdout: "enabled": { 2024-04-21T23:19:11.481 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enabled", 2024-04-21T23:19:11.481 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.481 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.482 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.482 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.482 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.482 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.482 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.482 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.482 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.482 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.483 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.483 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.483 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.483 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.483 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.483 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.483 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.484 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.484 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.484 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.484 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.484 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.484 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.484 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.484 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.484 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.485 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.485 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.485 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.485 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.485 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.485 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.486 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.486 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.486 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.486 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.487 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.487 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.487 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.487 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.487 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.488 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.488 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.488 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.489 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.489 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.489 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.490 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.490 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.490 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.491 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.491 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.491 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.491 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.491 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.491 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.492 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.492 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.492 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.492 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.492 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.492 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.492 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.492 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.493 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.493 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.493 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.493 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.493 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.493 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.493 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.493 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.494 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.494 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.494 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.494 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.494 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.494 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.494 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.494 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.495 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.495 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.495 INFO:teuthology.orchestra.run.smithi044.stdout: "max_completed_events": { 2024-04-21T23:19:11.495 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_completed_events", 2024-04-21T23:19:11.495 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.495 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.495 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.495 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "50", 2024-04-21T23:19:11.495 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.496 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.496 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.496 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "number of past completed events to remember", 2024-04-21T23:19:11.496 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.496 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.496 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.496 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.496 INFO:teuthology.orchestra.run.smithi044.stdout: "sleep_interval": { 2024-04-21T23:19:11.496 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "sleep_interval", 2024-04-21T23:19:11.497 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:11.497 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.497 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.497 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "5", 2024-04-21T23:19:11.497 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.497 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.497 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.497 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "how long the module is going to sleep", 2024-04-21T23:19:11.498 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.498 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.498 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.498 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.498 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.498 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.498 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.499 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "prometheus", 2024-04-21T23:19:11.499 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.499 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.499 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.499 INFO:teuthology.orchestra.run.smithi044.stdout: "cache": { 2024-04-21T23:19:11.499 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "cache", 2024-04-21T23:19:11.499 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.499 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.499 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.500 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.500 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.500 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.500 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.500 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.500 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.500 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.500 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.501 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.501 INFO:teuthology.orchestra.run.smithi044.stdout: "exclude_perf_counters": { 2024-04-21T23:19:11.501 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "exclude_perf_counters", 2024-04-21T23:19:11.501 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.501 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.501 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.501 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.501 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.502 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.502 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.502 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-21T23:19:11.502 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-21T23:19:11.502 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.502 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.502 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.502 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.502 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.503 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.503 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.503 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.503 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.503 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.503 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.503 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.503 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.504 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.504 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.504 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.504 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.504 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.504 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.504 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.504 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.504 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.505 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.505 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.505 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.505 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.505 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.505 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.505 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.505 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.505 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.506 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.506 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.506 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.506 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.506 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.506 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.506 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.506 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.507 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.507 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.507 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.507 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.507 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.507 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.507 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.507 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.507 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.508 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.508 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.508 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.508 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.508 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.508 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.508 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.508 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.508 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.509 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.509 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.509 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.509 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.509 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.509 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.509 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.509 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.510 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.510 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.510 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.510 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.510 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.510 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.510 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.510 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.511 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_stats_pools": { 2024-04-21T23:19:11.511 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_stats_pools", 2024-04-21T23:19:11.511 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.511 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.511 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.511 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.511 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.511 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.512 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.512 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.512 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.512 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.512 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.512 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.512 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-21T23:19:11.512 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-21T23:19:11.513 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.513 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.513 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.513 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "300", 2024-04-21T23:19:11.513 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.513 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.513 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.513 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.514 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.514 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.514 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.514 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.514 INFO:teuthology.orchestra.run.smithi044.stdout: "scrape_interval": { 2024-04-21T23:19:11.514 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "scrape_interval", 2024-04-21T23:19:11.514 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:11.514 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.514 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.515 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15.0", 2024-04-21T23:19:11.515 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.515 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.515 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.515 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.515 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.515 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.515 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.515 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.516 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2024-04-21T23:19:11.516 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2024-04-21T23:19:11.516 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.516 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.516 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.516 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "::", 2024-04-21T23:19:11.516 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.516 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.517 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.517 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-21T23:19:11.517 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.517 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.517 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.517 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.517 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2024-04-21T23:19:11.517 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2024-04-21T23:19:11.517 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.518 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.518 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.518 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "9283", 2024-04-21T23:19:11.518 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.518 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.518 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.518 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-21T23:19:11.518 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.519 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.519 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.519 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.519 INFO:teuthology.orchestra.run.smithi044.stdout: "stale_cache_strategy": { 2024-04-21T23:19:11.519 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "stale_cache_strategy", 2024-04-21T23:19:11.519 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.519 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.519 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.519 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "log", 2024-04-21T23:19:11.520 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.520 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.520 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.520 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.520 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.520 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.520 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.520 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.520 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_behaviour": { 2024-04-21T23:19:11.521 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_behaviour", 2024-04-21T23:19:11.521 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.521 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.521 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.521 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "default", 2024-04-21T23:19:11.521 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.521 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.521 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.521 INFO:teuthology.orchestra.run.smithi044.stdout: "default", 2024-04-21T23:19:11.522 INFO:teuthology.orchestra.run.smithi044.stdout: "error" 2024-04-21T23:19:11.522 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.522 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.522 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.522 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.522 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.522 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.522 INFO:teuthology.orchestra.run.smithi044.stdout: "standby_error_status_code": { 2024-04-21T23:19:11.522 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "standby_error_status_code", 2024-04-21T23:19:11.523 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.523 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.523 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.523 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "500", 2024-04-21T23:19:11.523 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "400", 2024-04-21T23:19:11.523 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "599", 2024-04-21T23:19:11.523 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.523 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.524 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.524 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.524 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.524 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.524 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.524 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.524 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.524 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_support", 2024-04-21T23:19:11.524 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.525 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.525 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.525 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.525 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.525 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.525 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.525 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.525 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.525 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.526 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.526 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.526 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.526 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.526 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.526 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.526 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.527 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.527 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.527 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.527 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.527 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.527 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.527 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.527 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.527 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.528 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.528 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.528 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.528 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.528 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.528 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.528 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.528 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.529 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.529 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.529 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.529 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.529 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.529 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.529 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.529 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.529 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.530 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.530 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.530 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.530 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.530 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.530 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.530 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.530 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.530 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.531 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.531 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.531 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.531 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.531 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.531 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.531 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.531 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.532 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.532 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.532 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.532 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.532 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.532 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.532 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.532 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.532 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.533 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.533 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.533 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.533 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.533 INFO:teuthology.orchestra.run.smithi044.stdout: "max_concurrent_snap_create": { 2024-04-21T23:19:11.533 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_concurrent_snap_create", 2024-04-21T23:19:11.533 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.533 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.534 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.534 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10", 2024-04-21T23:19:11.534 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.534 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.534 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.534 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.534 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.534 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.534 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.535 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.535 INFO:teuthology.orchestra.run.smithi044.stdout: "mirror_snapshot_schedule": { 2024-04-21T23:19:11.535 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "mirror_snapshot_schedule", 2024-04-21T23:19:11.535 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.535 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.535 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.535 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.535 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.536 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.536 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.536 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.536 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.536 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.536 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.536 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.536 INFO:teuthology.orchestra.run.smithi044.stdout: "trash_purge_schedule": { 2024-04-21T23:19:11.536 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "trash_purge_schedule", 2024-04-21T23:19:11.537 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.537 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.537 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.537 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.537 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.537 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.537 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.537 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.538 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.538 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.538 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.538 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.538 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.538 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.538 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.538 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "restful", 2024-04-21T23:19:11.539 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.539 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.539 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.539 INFO:teuthology.orchestra.run.smithi044.stdout: "enable_auth": { 2024-04-21T23:19:11.539 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enable_auth", 2024-04-21T23:19:11.539 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.539 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.539 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.539 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.540 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.540 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.540 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.540 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.540 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.540 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.540 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.540 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.541 INFO:teuthology.orchestra.run.smithi044.stdout: "key_file": { 2024-04-21T23:19:11.541 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "key_file", 2024-04-21T23:19:11.541 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.541 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.541 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.541 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.541 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.541 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.541 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.542 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.542 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.542 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.542 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.542 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.542 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.542 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.542 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.543 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.543 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.543 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.543 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.543 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.543 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.543 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.543 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.544 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.544 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.544 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.544 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.544 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.544 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.544 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.544 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.545 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.545 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.545 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.545 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.545 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.545 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.545 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.545 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.545 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.546 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.546 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.546 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.546 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.546 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.546 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.546 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.546 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.547 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.547 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.547 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.547 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.547 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.547 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.547 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.547 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.547 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.548 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.548 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.548 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.548 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.548 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.548 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.548 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.548 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.548 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.549 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.549 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.549 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.549 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.549 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.549 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.549 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.549 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.549 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.550 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.550 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.550 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.550 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.550 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.550 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.550 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.550 INFO:teuthology.orchestra.run.smithi044.stdout: "server_addr": { 2024-04-21T23:19:11.551 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_addr", 2024-04-21T23:19:11.551 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.551 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.551 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.551 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.551 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.551 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.551 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.551 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.552 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.552 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.552 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.552 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.552 INFO:teuthology.orchestra.run.smithi044.stdout: "server_port": { 2024-04-21T23:19:11.552 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "server_port", 2024-04-21T23:19:11.552 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.552 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.552 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.553 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.553 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.553 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.553 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.553 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.553 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.553 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.553 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.554 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.554 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.554 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.554 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.554 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rgw", 2024-04-21T23:19:11.554 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.554 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.554 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.555 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.555 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.555 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.555 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.555 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.555 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.555 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.555 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.555 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.556 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.556 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.556 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.556 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.556 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.556 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.556 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.556 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.556 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.557 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.557 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.557 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.557 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.557 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.557 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.557 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.557 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.558 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.558 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.558 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.558 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.558 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.558 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.558 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.558 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.559 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.559 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.559 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.559 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.559 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.559 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.559 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.559 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.559 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.560 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.560 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.560 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.560 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.560 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.560 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.560 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.560 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.560 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.561 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.561 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.561 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.561 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.561 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.561 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.561 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.561 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.561 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.562 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.562 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.562 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.562 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.562 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.562 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.562 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.562 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.563 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.563 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.563 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.563 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.563 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rook", 2024-04-21T23:19:11.563 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.563 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.563 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.563 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.564 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.564 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.564 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.564 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.564 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.564 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.564 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.564 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.564 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.565 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.565 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.565 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.565 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.565 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.565 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.565 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.565 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.565 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.566 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.566 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.566 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.566 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.566 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.566 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.566 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.566 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.567 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.567 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.567 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.567 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.567 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.567 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.567 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.567 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.567 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.568 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.568 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.568 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.568 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.568 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.568 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.568 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.568 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.569 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.569 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.569 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.569 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.569 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.569 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.569 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.569 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.569 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.570 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.570 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.570 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.570 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.570 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.570 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.570 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.570 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.570 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.571 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.571 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.571 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.571 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.571 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.571 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.571 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.571 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.571 INFO:teuthology.orchestra.run.smithi044.stdout: "storage_class": { 2024-04-21T23:19:11.572 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "storage_class", 2024-04-21T23:19:11.572 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.572 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.572 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.572 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "local", 2024-04-21T23:19:11.572 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.572 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.572 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.573 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-21T23:19:11.573 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.573 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.573 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.573 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.573 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.573 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.573 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.573 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "selftest", 2024-04-21T23:19:11.574 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.574 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.574 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.574 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.574 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.574 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.574 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.574 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.574 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.575 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.575 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.575 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.575 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.575 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.575 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.575 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.575 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.576 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.576 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.576 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.576 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.576 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.576 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.576 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.576 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.576 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.577 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.577 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.577 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.577 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.577 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.577 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.577 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.577 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.577 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.578 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.578 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.578 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.578 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.578 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.578 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.578 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.578 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.579 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.579 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.579 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.579 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.579 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.579 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.579 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.579 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.579 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.580 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.580 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.580 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.580 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.580 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.580 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.580 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.580 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.581 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.581 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.581 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.581 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.581 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.581 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.581 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.581 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.582 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.582 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.582 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.582 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.582 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.582 INFO:teuthology.orchestra.run.smithi044.stdout: "roption1": { 2024-04-21T23:19:11.582 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "roption1", 2024-04-21T23:19:11.582 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.583 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.583 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.583 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.583 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.583 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.583 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.583 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.583 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.584 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.584 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.584 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.584 INFO:teuthology.orchestra.run.smithi044.stdout: "roption2": { 2024-04-21T23:19:11.584 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "roption2", 2024-04-21T23:19:11.584 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.584 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.584 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.585 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "xyz", 2024-04-21T23:19:11.585 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.585 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.585 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.585 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.585 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.585 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.585 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.585 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.586 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption1": { 2024-04-21T23:19:11.586 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption1", 2024-04-21T23:19:11.586 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.586 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.586 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.586 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.586 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.586 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.587 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.587 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.587 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.587 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.587 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.587 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.587 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption2": { 2024-04-21T23:19:11.587 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption2", 2024-04-21T23:19:11.588 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.588 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.588 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.588 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.588 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.588 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.588 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.588 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.589 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.589 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.589 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.589 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.589 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption3": { 2024-04-21T23:19:11.589 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption3", 2024-04-21T23:19:11.589 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "float", 2024-04-21T23:19:11.589 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.589 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.590 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.590 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.590 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.590 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.590 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.590 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.590 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.590 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.591 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.591 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption4": { 2024-04-21T23:19:11.591 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption4", 2024-04-21T23:19:11.591 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.591 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.591 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.591 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.591 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.591 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.592 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.592 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.592 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.592 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.592 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.592 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.592 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption5": { 2024-04-21T23:19:11.592 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption5", 2024-04-21T23:19:11.592 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.593 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.593 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.593 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.593 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.593 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.593 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.593 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.593 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.594 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.594 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.594 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.594 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption6": { 2024-04-21T23:19:11.594 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption6", 2024-04-21T23:19:11.594 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.594 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.594 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.594 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.595 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.595 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.595 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.595 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.595 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.595 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.595 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.595 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.595 INFO:teuthology.orchestra.run.smithi044.stdout: "rwoption7": { 2024-04-21T23:19:11.596 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rwoption7", 2024-04-21T23:19:11.596 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.596 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.596 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.596 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.596 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "1", 2024-04-21T23:19:11.596 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "42", 2024-04-21T23:19:11.596 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.596 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.597 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.597 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.597 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.597 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.597 INFO:teuthology.orchestra.run.smithi044.stdout: "testkey": { 2024-04-21T23:19:11.597 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testkey", 2024-04-21T23:19:11.597 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.597 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.598 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.598 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.598 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.598 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.598 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.598 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.598 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.598 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.598 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.599 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.599 INFO:teuthology.orchestra.run.smithi044.stdout: "testlkey": { 2024-04-21T23:19:11.599 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testlkey", 2024-04-21T23:19:11.599 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.599 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.599 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.599 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.599 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.599 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.600 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.600 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.600 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.600 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.600 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.600 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.600 INFO:teuthology.orchestra.run.smithi044.stdout: "testnewline": { 2024-04-21T23:19:11.600 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "testnewline", 2024-04-21T23:19:11.601 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.601 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.601 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.601 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.601 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.601 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.601 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.601 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.601 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.602 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.602 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.602 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.602 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.602 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.602 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.602 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snap_schedule", 2024-04-21T23:19:11.602 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.603 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.603 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.603 INFO:teuthology.orchestra.run.smithi044.stdout: "allow_m_granularity": { 2024-04-21T23:19:11.603 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "allow_m_granularity", 2024-04-21T23:19:11.603 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.603 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.603 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.603 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.603 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.604 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.604 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.604 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "allow minute scheduled snapshots", 2024-04-21T23:19:11.604 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.604 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.604 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.604 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.604 INFO:teuthology.orchestra.run.smithi044.stdout: "dump_on_update": { 2024-04-21T23:19:11.605 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "dump_on_update", 2024-04-21T23:19:11.605 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.605 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.605 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.605 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.605 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.605 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.605 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.605 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "dump database to debug log on update", 2024-04-21T23:19:11.606 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.606 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.606 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.606 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.606 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.606 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.606 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.606 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.606 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.607 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.607 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.607 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.607 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.607 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.607 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.607 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.607 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.607 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.608 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.608 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.608 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.608 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.608 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.608 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.608 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.608 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.608 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.609 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.609 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.609 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.609 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.609 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.609 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.609 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.609 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.610 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.610 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.610 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.610 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.610 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.610 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.610 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.610 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.610 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.611 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.611 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.611 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.611 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.611 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.611 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.611 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.612 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.612 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.612 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.612 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.612 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.612 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.612 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.613 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.613 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.613 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.613 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.613 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.613 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.613 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.613 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.614 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.614 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.614 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.614 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.614 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.614 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.614 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.614 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.615 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.615 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.615 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.615 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "stats", 2024-04-21T23:19:11.615 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.615 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.615 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.615 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.616 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.616 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.616 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.616 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.616 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.616 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.616 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.617 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.617 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.617 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.617 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.617 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.617 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.617 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.617 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.618 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.618 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.618 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.618 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.618 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.618 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.618 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.618 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.619 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.619 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.619 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.619 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.619 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.619 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.619 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.619 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.620 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.620 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.620 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.620 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.620 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.620 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.620 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.620 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.620 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.621 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.621 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.621 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.621 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.621 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.621 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.621 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.621 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.622 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.622 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.622 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.622 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.622 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.622 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.622 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.622 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.623 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.623 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.623 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.623 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.623 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.623 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.623 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.623 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.624 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.624 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.624 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.624 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.624 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.624 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.624 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.624 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.624 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "status", 2024-04-21T23:19:11.625 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.625 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.625 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.625 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.625 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.625 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.625 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.626 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.626 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.626 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.626 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.626 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.626 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.626 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.626 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.626 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.627 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.627 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.627 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.627 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.627 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.627 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.627 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.627 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.628 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.628 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.628 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.628 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.628 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.628 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.628 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.628 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.629 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.629 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.629 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.629 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.629 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.629 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.629 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.629 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.629 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.630 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.630 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.630 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.630 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.630 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.630 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.630 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.631 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.631 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.631 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.631 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.631 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.631 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.631 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.631 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.631 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.632 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.632 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.632 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.632 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.632 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.632 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.632 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.633 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.633 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.633 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.633 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.633 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.633 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.633 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.633 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.633 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.634 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.634 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.634 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.634 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "telegraf", 2024-04-21T23:19:11.634 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.634 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.634 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.634 INFO:teuthology.orchestra.run.smithi044.stdout: "address": { 2024-04-21T23:19:11.635 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "address", 2024-04-21T23:19:11.635 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.635 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.635 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.635 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-21T23:19:11.635 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.635 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.635 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.635 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.636 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.636 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.636 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.636 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.636 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-04-21T23:19:11.636 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-04-21T23:19:11.636 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:11.636 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.637 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.637 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "15", 2024-04-21T23:19:11.637 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.637 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.637 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.637 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.637 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.637 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.637 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.638 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.638 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.638 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.638 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.638 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.638 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.638 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.638 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.639 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.639 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.639 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.639 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.639 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.639 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.639 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.639 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.639 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.640 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.640 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.640 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.640 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.640 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.640 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.640 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.640 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.641 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.641 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.641 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.641 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.641 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.641 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.641 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.641 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.641 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.642 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.642 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.642 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.642 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.642 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.642 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.642 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.642 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.643 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.643 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.643 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.643 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.643 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.643 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.643 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.643 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.643 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.644 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.644 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.644 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.644 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.644 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.644 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.644 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.644 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.644 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.645 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.645 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.645 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.645 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.645 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.645 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.645 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.645 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.646 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.646 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.646 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.646 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.646 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.646 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.646 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "telemetry", 2024-04-21T23:19:11.646 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.647 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.647 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.647 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_basic": { 2024-04-21T23:19:11.647 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_basic", 2024-04-21T23:19:11.647 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.647 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.647 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.647 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.647 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.648 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.648 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.648 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-21T23:19:11.648 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.648 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.648 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.648 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.648 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_crash": { 2024-04-21T23:19:11.649 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_crash", 2024-04-21T23:19:11.649 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.649 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.649 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.649 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.649 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.649 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.649 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.649 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-21T23:19:11.650 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.650 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.650 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.650 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.650 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_device": { 2024-04-21T23:19:11.650 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_device", 2024-04-21T23:19:11.650 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.650 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.651 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.651 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.651 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.651 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.651 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.651 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-21T23:19:11.651 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.651 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.651 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.652 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.652 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_ident": { 2024-04-21T23:19:11.652 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_ident", 2024-04-21T23:19:11.652 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.652 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.652 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.652 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.652 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.653 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.653 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.653 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-21T23:19:11.653 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.653 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.653 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.653 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.653 INFO:teuthology.orchestra.run.smithi044.stdout: "channel_perf": { 2024-04-21T23:19:11.654 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "channel_perf", 2024-04-21T23:19:11.654 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.654 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.654 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.654 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.654 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.654 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.654 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.654 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-21T23:19:11.655 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.655 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.655 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.655 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.655 INFO:teuthology.orchestra.run.smithi044.stdout: "contact": { 2024-04-21T23:19:11.655 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "contact", 2024-04-21T23:19:11.655 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.655 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.656 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.656 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.656 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.656 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.656 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.656 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.656 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.656 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.657 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.657 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.657 INFO:teuthology.orchestra.run.smithi044.stdout: "description": { 2024-04-21T23:19:11.657 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "description", 2024-04-21T23:19:11.657 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.657 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.657 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.657 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.657 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.658 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.658 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.658 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.658 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.658 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.658 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.658 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.658 INFO:teuthology.orchestra.run.smithi044.stdout: "device_url": { 2024-04-21T23:19:11.659 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "device_url", 2024-04-21T23:19:11.659 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.659 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.659 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.659 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-21T23:19:11.659 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.659 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.659 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.660 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.660 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.660 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.660 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.660 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.660 INFO:teuthology.orchestra.run.smithi044.stdout: "enabled": { 2024-04-21T23:19:11.660 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "enabled", 2024-04-21T23:19:11.660 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.660 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.661 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.661 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.661 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.661 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.661 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.661 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.661 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.661 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.661 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.662 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.662 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-04-21T23:19:11.662 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-04-21T23:19:11.662 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.662 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.662 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.662 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "24", 2024-04-21T23:19:11.662 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "8", 2024-04-21T23:19:11.663 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.663 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.663 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.663 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.663 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.663 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.663 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.663 INFO:teuthology.orchestra.run.smithi044.stdout: "last_opt_revision": { 2024-04-21T23:19:11.663 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "last_opt_revision", 2024-04-21T23:19:11.664 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.664 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.664 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.664 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "1", 2024-04-21T23:19:11.664 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.664 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.664 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.664 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.665 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.665 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.665 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.665 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.665 INFO:teuthology.orchestra.run.smithi044.stdout: "leaderboard": { 2024-04-21T23:19:11.665 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "leaderboard", 2024-04-21T23:19:11.665 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.665 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.665 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.666 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.666 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.666 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.666 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.666 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.666 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.666 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.666 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.667 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.667 INFO:teuthology.orchestra.run.smithi044.stdout: "leaderboard_description": { 2024-04-21T23:19:11.667 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "leaderboard_description", 2024-04-21T23:19:11.667 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.667 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.667 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.667 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.667 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.667 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.668 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.668 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.668 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.668 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.668 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.668 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.668 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.668 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.668 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.669 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.669 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.669 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.669 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.669 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.669 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.669 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.669 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.670 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.670 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.670 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.670 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.670 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.670 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.670 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.670 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.670 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.671 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.671 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.671 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.671 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.671 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.671 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.671 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.671 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.671 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.672 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.672 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.672 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.672 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.672 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.672 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.672 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.672 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.673 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.673 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.673 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.673 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.673 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.673 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.673 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.673 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.673 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.674 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.674 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.674 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.674 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.674 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.674 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.674 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.674 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.674 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.675 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.675 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.675 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.675 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.675 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.675 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.675 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.675 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.676 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.676 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.676 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.676 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.676 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.676 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.676 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.676 INFO:teuthology.orchestra.run.smithi044.stdout: "organization": { 2024-04-21T23:19:11.676 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "organization", 2024-04-21T23:19:11.677 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.677 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.677 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.677 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.677 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.677 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.677 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.677 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.677 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.678 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.678 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.678 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.678 INFO:teuthology.orchestra.run.smithi044.stdout: "proxy": { 2024-04-21T23:19:11.678 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "proxy", 2024-04-21T23:19:11.678 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.678 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.678 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.679 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.679 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.679 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.679 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.679 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.679 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.679 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.679 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.680 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.680 INFO:teuthology.orchestra.run.smithi044.stdout: "url": { 2024-04-21T23:19:11.680 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "url", 2024-04-21T23:19:11.680 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.680 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.680 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.680 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-21T23:19:11.680 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.681 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.681 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.681 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.681 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.681 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.681 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.681 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.681 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.682 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.682 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.682 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "test_orchestrator", 2024-04-21T23:19:11.682 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.682 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.682 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.682 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.682 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.682 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.683 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.683 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.683 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.683 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.683 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.683 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.683 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.683 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.684 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.684 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.684 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.684 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.684 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.684 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.684 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.684 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.684 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.685 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.685 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.685 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.685 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.685 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.685 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.685 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.685 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.685 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.686 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.686 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.686 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.686 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.686 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.686 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.686 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.686 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.687 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.687 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.687 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.687 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.687 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.687 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.687 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.687 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.687 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.688 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.688 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.688 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.688 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.688 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.688 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.688 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.688 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.688 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.689 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.689 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.689 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.689 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.689 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.689 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.689 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.689 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.689 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.690 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.690 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.690 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.690 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.690 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.690 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.690 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.690 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.691 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.691 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "volumes", 2024-04-21T23:19:11.691 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.691 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.691 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.691 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.691 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.691 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.692 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.692 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.692 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.692 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.692 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.692 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.692 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.692 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.692 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.693 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.693 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.693 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.693 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.693 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.693 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.693 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.693 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.694 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.694 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.694 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.694 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.694 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.694 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.694 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.694 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.695 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.695 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.695 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.695 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.695 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.695 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.695 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.695 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.695 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.696 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.696 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.696 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.696 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.696 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.696 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.696 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.696 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.697 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.697 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.697 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.697 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.697 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.697 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.697 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.697 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.697 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.698 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.698 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.698 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.698 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.698 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.698 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.698 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.698 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.699 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.699 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.699 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.699 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.699 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.699 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.699 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.699 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.700 INFO:teuthology.orchestra.run.smithi044.stdout: "max_concurrent_clones": { 2024-04-21T23:19:11.700 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "max_concurrent_clones", 2024-04-21T23:19:11.700 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.700 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.700 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.700 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "4", 2024-04-21T23:19:11.700 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.700 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.701 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.701 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-21T23:19:11.701 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.701 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.701 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.701 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.701 INFO:teuthology.orchestra.run.smithi044.stdout: "periodic_async_work": { 2024-04-21T23:19:11.701 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "periodic_async_work", 2024-04-21T23:19:11.701 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.702 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.702 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.702 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.702 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.702 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.702 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.702 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Periodically check for async work", 2024-04-21T23:19:11.703 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.703 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.703 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.703 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.703 INFO:teuthology.orchestra.run.smithi044.stdout: "snapshot_clone_delay": { 2024-04-21T23:19:11.703 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snapshot_clone_delay", 2024-04-21T23:19:11.703 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.703 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.704 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.704 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "0", 2024-04-21T23:19:11.704 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.704 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.704 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.704 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-21T23:19:11.704 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.704 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.704 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.705 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.705 INFO:teuthology.orchestra.run.smithi044.stdout: "snapshot_clone_no_wait": { 2024-04-21T23:19:11.705 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snapshot_clone_no_wait", 2024-04-21T23:19:11.705 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.705 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.705 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.705 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "True", 2024-04-21T23:19:11.705 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.705 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.706 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.706 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-21T23:19:11.706 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.706 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.706 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.706 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.706 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.706 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.707 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.707 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix", 2024-04-21T23:19:11.707 INFO:teuthology.orchestra.run.smithi044.stdout: "can_run": true, 2024-04-21T23:19:11.707 INFO:teuthology.orchestra.run.smithi044.stdout: "error_string": "", 2024-04-21T23:19:11.707 INFO:teuthology.orchestra.run.smithi044.stdout: "module_options": { 2024-04-21T23:19:11.707 INFO:teuthology.orchestra.run.smithi044.stdout: "discovery_interval": { 2024-04-21T23:19:11.707 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "discovery_interval", 2024-04-21T23:19:11.707 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "uint", 2024-04-21T23:19:11.707 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.708 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.708 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "100", 2024-04-21T23:19:11.708 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.708 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.708 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.708 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.708 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.708 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.708 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.709 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.709 INFO:teuthology.orchestra.run.smithi044.stdout: "identifier": { 2024-04-21T23:19:11.709 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "identifier", 2024-04-21T23:19:11.709 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.709 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.709 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.709 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.709 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.710 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.710 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.710 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.710 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.710 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.710 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.710 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.710 INFO:teuthology.orchestra.run.smithi044.stdout: "interval": { 2024-04-21T23:19:11.710 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "interval", 2024-04-21T23:19:11.711 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "secs", 2024-04-21T23:19:11.711 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.711 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.711 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "60", 2024-04-21T23:19:11.711 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.711 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.711 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.711 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.711 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.712 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.712 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.712 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.712 INFO:teuthology.orchestra.run.smithi044.stdout: "log_level": { 2024-04-21T23:19:11.712 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_level", 2024-04-21T23:19:11.712 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.712 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.712 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.713 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.713 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.713 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.713 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.713 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.713 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.713 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.713 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.713 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.714 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.714 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.714 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.714 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.714 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.714 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.714 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.714 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster": { 2024-04-21T23:19:11.714 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster", 2024-04-21T23:19:11.715 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.715 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.715 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.715 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.715 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.715 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.715 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.715 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.715 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.716 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.716 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.716 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.716 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_cluster_level": { 2024-04-21T23:19:11.716 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_cluster_level", 2024-04-21T23:19:11.716 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.716 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.716 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.717 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "info", 2024-04-21T23:19:11.717 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.717 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.717 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [ 2024-04-21T23:19:11.717 INFO:teuthology.orchestra.run.smithi044.stdout: "", 2024-04-21T23:19:11.717 INFO:teuthology.orchestra.run.smithi044.stdout: "critical", 2024-04-21T23:19:11.717 INFO:teuthology.orchestra.run.smithi044.stdout: "debug", 2024-04-21T23:19:11.717 INFO:teuthology.orchestra.run.smithi044.stdout: "error", 2024-04-21T23:19:11.717 INFO:teuthology.orchestra.run.smithi044.stdout: "info", 2024-04-21T23:19:11.718 INFO:teuthology.orchestra.run.smithi044.stdout: "warning" 2024-04-21T23:19:11.718 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.718 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.718 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.718 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.718 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.718 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.719 INFO:teuthology.orchestra.run.smithi044.stdout: "log_to_file": { 2024-04-21T23:19:11.719 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "log_to_file", 2024-04-21T23:19:11.719 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "bool", 2024-04-21T23:19:11.719 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.719 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 1, 2024-04-21T23:19:11.719 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "False", 2024-04-21T23:19:11.719 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.719 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.719 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.720 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.720 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.720 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.720 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.720 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.720 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_host": { 2024-04-21T23:19:11.720 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_host", 2024-04-21T23:19:11.721 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.721 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.721 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.721 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "", 2024-04-21T23:19:11.721 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.721 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.721 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.721 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.721 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.722 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.722 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.722 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.722 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_port": { 2024-04-21T23:19:11.722 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_port", 2024-04-21T23:19:11.722 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "int", 2024-04-21T23:19:11.722 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.723 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.723 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "10051", 2024-04-21T23:19:11.723 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.723 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.723 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.723 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.723 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.723 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.724 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.724 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.724 INFO:teuthology.orchestra.run.smithi044.stdout: "zabbix_sender": { 2024-04-21T23:19:11.724 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "zabbix_sender", 2024-04-21T23:19:11.724 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "str", 2024-04-21T23:19:11.724 INFO:teuthology.orchestra.run.smithi044.stdout: "level": "advanced", 2024-04-21T23:19:11.724 INFO:teuthology.orchestra.run.smithi044.stdout: "flags": 0, 2024-04-21T23:19:11.724 INFO:teuthology.orchestra.run.smithi044.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-21T23:19:11.724 INFO:teuthology.orchestra.run.smithi044.stdout: "min": "", 2024-04-21T23:19:11.725 INFO:teuthology.orchestra.run.smithi044.stdout: "max": "", 2024-04-21T23:19:11.725 INFO:teuthology.orchestra.run.smithi044.stdout: "enum_allowed": [], 2024-04-21T23:19:11.725 INFO:teuthology.orchestra.run.smithi044.stdout: "desc": "", 2024-04-21T23:19:11.725 INFO:teuthology.orchestra.run.smithi044.stdout: "long_desc": "", 2024-04-21T23:19:11.725 INFO:teuthology.orchestra.run.smithi044.stdout: "tags": [], 2024-04-21T23:19:11.725 INFO:teuthology.orchestra.run.smithi044.stdout: "see_also": [] 2024-04-21T23:19:11.725 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.725 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.725 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.726 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.726 INFO:teuthology.orchestra.run.smithi044.stdout: "services": { 2024-04-21T23:19:11.726 INFO:teuthology.orchestra.run.smithi044.stdout: "dashboard": "https://172.21.15.44:8443/" 2024-04-21T23:19:11.726 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.726 INFO:teuthology.orchestra.run.smithi044.stdout: "always_on_modules": { 2024-04-21T23:19:11.726 INFO:teuthology.orchestra.run.smithi044.stdout: "octopus": [ 2024-04-21T23:19:11.726 INFO:teuthology.orchestra.run.smithi044.stdout: "balancer", 2024-04-21T23:19:11.726 INFO:teuthology.orchestra.run.smithi044.stdout: "crash", 2024-04-21T23:19:11.727 INFO:teuthology.orchestra.run.smithi044.stdout: "devicehealth", 2024-04-21T23:19:11.727 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator", 2024-04-21T23:19:11.727 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_autoscaler", 2024-04-21T23:19:11.727 INFO:teuthology.orchestra.run.smithi044.stdout: "progress", 2024-04-21T23:19:11.727 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_support", 2024-04-21T23:19:11.727 INFO:teuthology.orchestra.run.smithi044.stdout: "status", 2024-04-21T23:19:11.727 INFO:teuthology.orchestra.run.smithi044.stdout: "telemetry", 2024-04-21T23:19:11.727 INFO:teuthology.orchestra.run.smithi044.stdout: "volumes" 2024-04-21T23:19:11.727 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.728 INFO:teuthology.orchestra.run.smithi044.stdout: "pacific": [ 2024-04-21T23:19:11.728 INFO:teuthology.orchestra.run.smithi044.stdout: "balancer", 2024-04-21T23:19:11.728 INFO:teuthology.orchestra.run.smithi044.stdout: "crash", 2024-04-21T23:19:11.728 INFO:teuthology.orchestra.run.smithi044.stdout: "devicehealth", 2024-04-21T23:19:11.728 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator", 2024-04-21T23:19:11.728 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_autoscaler", 2024-04-21T23:19:11.728 INFO:teuthology.orchestra.run.smithi044.stdout: "progress", 2024-04-21T23:19:11.728 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_support", 2024-04-21T23:19:11.728 INFO:teuthology.orchestra.run.smithi044.stdout: "status", 2024-04-21T23:19:11.729 INFO:teuthology.orchestra.run.smithi044.stdout: "telemetry", 2024-04-21T23:19:11.729 INFO:teuthology.orchestra.run.smithi044.stdout: "volumes" 2024-04-21T23:19:11.729 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.729 INFO:teuthology.orchestra.run.smithi044.stdout: "quincy": [ 2024-04-21T23:19:11.729 INFO:teuthology.orchestra.run.smithi044.stdout: "balancer", 2024-04-21T23:19:11.729 INFO:teuthology.orchestra.run.smithi044.stdout: "crash", 2024-04-21T23:19:11.730 INFO:teuthology.orchestra.run.smithi044.stdout: "devicehealth", 2024-04-21T23:19:11.730 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator", 2024-04-21T23:19:11.730 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_autoscaler", 2024-04-21T23:19:11.730 INFO:teuthology.orchestra.run.smithi044.stdout: "progress", 2024-04-21T23:19:11.730 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_support", 2024-04-21T23:19:11.730 INFO:teuthology.orchestra.run.smithi044.stdout: "status", 2024-04-21T23:19:11.730 INFO:teuthology.orchestra.run.smithi044.stdout: "telemetry", 2024-04-21T23:19:11.730 INFO:teuthology.orchestra.run.smithi044.stdout: "volumes" 2024-04-21T23:19:11.731 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.731 INFO:teuthology.orchestra.run.smithi044.stdout: "reef": [ 2024-04-21T23:19:11.731 INFO:teuthology.orchestra.run.smithi044.stdout: "balancer", 2024-04-21T23:19:11.731 INFO:teuthology.orchestra.run.smithi044.stdout: "crash", 2024-04-21T23:19:11.731 INFO:teuthology.orchestra.run.smithi044.stdout: "devicehealth", 2024-04-21T23:19:11.731 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator", 2024-04-21T23:19:11.731 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_autoscaler", 2024-04-21T23:19:11.731 INFO:teuthology.orchestra.run.smithi044.stdout: "progress", 2024-04-21T23:19:11.731 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_support", 2024-04-21T23:19:11.732 INFO:teuthology.orchestra.run.smithi044.stdout: "status", 2024-04-21T23:19:11.732 INFO:teuthology.orchestra.run.smithi044.stdout: "telemetry", 2024-04-21T23:19:11.732 INFO:teuthology.orchestra.run.smithi044.stdout: "volumes" 2024-04-21T23:19:11.732 INFO:teuthology.orchestra.run.smithi044.stdout: ], 2024-04-21T23:19:11.732 INFO:teuthology.orchestra.run.smithi044.stdout: "squid": [ 2024-04-21T23:19:11.732 INFO:teuthology.orchestra.run.smithi044.stdout: "balancer", 2024-04-21T23:19:11.732 INFO:teuthology.orchestra.run.smithi044.stdout: "crash", 2024-04-21T23:19:11.732 INFO:teuthology.orchestra.run.smithi044.stdout: "devicehealth", 2024-04-21T23:19:11.732 INFO:teuthology.orchestra.run.smithi044.stdout: "orchestrator", 2024-04-21T23:19:11.733 INFO:teuthology.orchestra.run.smithi044.stdout: "pg_autoscaler", 2024-04-21T23:19:11.733 INFO:teuthology.orchestra.run.smithi044.stdout: "progress", 2024-04-21T23:19:11.733 INFO:teuthology.orchestra.run.smithi044.stdout: "rbd_support", 2024-04-21T23:19:11.733 INFO:teuthology.orchestra.run.smithi044.stdout: "status", 2024-04-21T23:19:11.733 INFO:teuthology.orchestra.run.smithi044.stdout: "telemetry", 2024-04-21T23:19:11.733 INFO:teuthology.orchestra.run.smithi044.stdout: "volumes" 2024-04-21T23:19:11.733 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-21T23:19:11.733 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.734 INFO:teuthology.orchestra.run.smithi044.stdout: "last_failure_osd_epoch": 95, 2024-04-21T23:19:11.734 INFO:teuthology.orchestra.run.smithi044.stdout: "active_clients": [ 2024-04-21T23:19:11.734 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.734 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "devicehealth", 2024-04-21T23:19:11.734 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2024-04-21T23:19:11.734 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.734 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2024-04-21T23:19:11.734 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:0", 2024-04-21T23:19:11.734 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2931747234 2024-04-21T23:19:11.735 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.735 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-21T23:19:11.735 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.735 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.735 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "libcephsqlite", 2024-04-21T23:19:11.735 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2024-04-21T23:19:11.735 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.735 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2024-04-21T23:19:11.735 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:0", 2024-04-21T23:19:11.736 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2309855766 2024-04-21T23:19:11.736 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.736 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-21T23:19:11.736 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.736 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.736 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "rbd_support", 2024-04-21T23:19:11.736 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2024-04-21T23:19:11.736 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.737 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2024-04-21T23:19:11.737 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:0", 2024-04-21T23:19:11.737 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2703194203 2024-04-21T23:19:11.737 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.737 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-21T23:19:11.737 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.737 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.737 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snap_schedule", 2024-04-21T23:19:11.737 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2024-04-21T23:19:11.738 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.738 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2024-04-21T23:19:11.738 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:0", 2024-04-21T23:19:11.738 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 1647419826 2024-04-21T23:19:11.738 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.738 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-21T23:19:11.738 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.738 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.739 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "snap_schedule", 2024-04-21T23:19:11.739 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2024-04-21T23:19:11.739 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.739 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2024-04-21T23:19:11.739 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:0", 2024-04-21T23:19:11.739 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 458048788 2024-04-21T23:19:11.739 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.739 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-21T23:19:11.739 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:11.740 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.740 INFO:teuthology.orchestra.run.smithi044.stdout: "name": "volumes", 2024-04-21T23:19:11.740 INFO:teuthology.orchestra.run.smithi044.stdout: "addrvec": [ 2024-04-21T23:19:11.740 INFO:teuthology.orchestra.run.smithi044.stdout: { 2024-04-21T23:19:11.740 INFO:teuthology.orchestra.run.smithi044.stdout: "type": "v2", 2024-04-21T23:19:11.740 INFO:teuthology.orchestra.run.smithi044.stdout: "addr": "172.21.15.44:0", 2024-04-21T23:19:11.741 INFO:teuthology.orchestra.run.smithi044.stdout: "nonce": 2585153298 2024-04-21T23:19:11.741 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.741 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-21T23:19:11.741 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:11.741 INFO:teuthology.orchestra.run.smithi044.stdout: ] 2024-04-21T23:19:11.741 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:19:11.743 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:09 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2539547277' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:19:11.743 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:09 smithi069 ceph-mon[31816]: pgmap v326: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 25 op/s 2024-04-21T23:19:11.743 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:09 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1175083084' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-21T23:19:11.743 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:09 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2539547277' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:19:11.743 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:09 smithi055 ceph-mon[31117]: pgmap v326: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 25 op/s 2024-04-21T23:19:11.743 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:09 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1175083084' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-21T23:19:11.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:09.319+0000 7fed31001640 1 -- 172.21.15.44:0/1175083084 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fed140a2e40 msgr2=0x7fed140a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:11.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:09.319+0000 7fed31001640 1 --2- 172.21.15.44:0/1175083084 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fed140a2e40 0x7fed140a5300 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fed24005fd0 tx=0x7fed24009500 comp rx=0 tx=0).stop 2024-04-21T23:19:11.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:09.319+0000 7fed31001640 1 -- 172.21.15.44:0/1175083084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed2c106600 msgr2=0x7fed2c06c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:11.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:09.319+0000 7fed31001640 1 --2- 172.21.15.44:0/1175083084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed2c106600 0x7fed2c06c010 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fed1c005ab0 tx=0x7fed1c02c810 comp rx=0 tx=0).stop 2024-04-21T23:19:11.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:09.319+0000 7fed31001640 1 -- 172.21.15.44:0/1175083084 shutdown_connections 2024-04-21T23:19:11.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:09.319+0000 7fed31001640 1 --2- 172.21.15.44:0/1175083084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed2c100f00 0x7fed2c073360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:11.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:09.319+0000 7fed31001640 1 --2- 172.21.15.44:0/1175083084 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fed140a2e40 0x7fed140a5300 secure :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fed24005fd0 tx=0x7fed24009500 comp rx=0 tx=0).stop 2024-04-21T23:19:11.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:09.319+0000 7fed31001640 1 --2- 172.21.15.44:0/1175083084 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fed2c106600 0x7fed2c06c010 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:11.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:09.319+0000 7fed31001640 1 --2- 172.21.15.44:0/1175083084 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fed2c100540 0x7fed2c072e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:11.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:09.319+0000 7fed31001640 1 -- 172.21.15.44:0/1175083084 >> 172.21.15.44:0/1175083084 conn(0x7fed2c0fc030 msgr2=0x7fed2c1040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:11.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:09.319+0000 7fed31001640 1 -- 172.21.15.44:0/1175083084 shutdown_connections 2024-04-21T23:19:11.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:09.319+0000 7fed31001640 1 -- 172.21.15.44:0/1175083084 wait complete. 2024-04-21T23:19:11.753 DEBUG:tasks.check_counter:Getting stats from a 2024-04-21T23:19:11.753 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-21T23:19:11.753 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:09 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2539547277' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:19:11.754 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:09 smithi044 ceph-mon[27934]: pgmap v326: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 25 op/s 2024-04-21T23:19:11.754 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:09 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1175083084' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-21T23:19:12.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:11 smithi044 ceph-mon[27934]: pgmap v327: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 25 op/s 2024-04-21T23:19:12.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:11 smithi069 ceph-mon[31816]: pgmap v327: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 25 op/s 2024-04-21T23:19:12.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:11 smithi055 ceph-mon[31117]: pgmap v327: 129 pgs: 129 active+clean; 80 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 25 op/s 2024-04-21T23:19:12.215 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:19:13.961 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:13 smithi044 ceph-mon[27934]: pgmap v328: 129 pgs: 10 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 94 active+clean; 62 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-21T23:19:14.093 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:19:14.093 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:19:14.094 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 0, 2024-04-21T23:19:14.094 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 2, 2024-04-21T23:19:14.094 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 0, 2024-04-21T23:19:14.094 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 0, 2024-04-21T23:19:14.094 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 4, 2024-04-21T23:19:14.094 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 0, 2024-04-21T23:19:14.094 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.001451409, 2024-04-21T23:19:14.094 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T23:19:14.095 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.000297043, 2024-04-21T23:19:14.095 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:19:14.095 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:14.095 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-04-21T23:19:14.095 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-04-21T23:19:14.095 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-04-21T23:19:14.095 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:14.095 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:14.096 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-04-21T23:19:14.096 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-04-21T23:19:14.096 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-04-21T23:19:14.096 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:14.096 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T23:19:14.096 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T23:19:14.096 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:14.096 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:19:14.097 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 2, 2024-04-21T23:19:14.097 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 159, 2024-04-21T23:19:14.097 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 320, 2024-04-21T23:19:14.097 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 183264, 2024-04-21T23:19:14.097 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 5, 2024-04-21T23:19:14.097 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 3, 2024-04-21T23:19:14.097 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.011941203, 2024-04-21T23:19:14.097 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.007778304, 2024-04-21T23:19:14.097 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.001962815, 2024-04-21T23:19:14.098 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:19:14.098 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:14.098 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 156, 2024-04-21T23:19:14.098 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.018896650, 2024-04-21T23:19:14.098 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000121132 2024-04-21T23:19:14.098 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:14.098 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:14.098 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-04-21T23:19:14.099 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-04-21T23:19:14.099 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-04-21T23:19:14.099 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:14.099 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T23:19:14.099 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 183264 2024-04-21T23:19:14.099 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:14.100 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:19:14.100 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 223, 2024-04-21T23:19:14.100 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 211, 2024-04-21T23:19:14.100 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 1445280, 2024-04-21T23:19:14.100 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 128512, 2024-04-21T23:19:14.100 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 2, 2024-04-21T23:19:14.100 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 1, 2024-04-21T23:19:14.100 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.040766044, 2024-04-21T23:19:14.101 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.012207286, 2024-04-21T23:19:14.101 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.032720124, 2024-04-21T23:19:14.101 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.004053499, 2024-04-21T23:19:14.101 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:14.101 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 211, 2024-04-21T23:19:14.101 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.010139746, 2024-04-21T23:19:14.101 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000048055 2024-04-21T23:19:14.101 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:14.102 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:14.102 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-04-21T23:19:14.102 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-04-21T23:19:14.102 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-04-21T23:19:14.102 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:14.102 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 1445280, 2024-04-21T23:19:14.102 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 128512 2024-04-21T23:19:14.102 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:14.103 INFO:teuthology.orchestra.run.smithi044.stdout: "cct": { 2024-04-21T23:19:14.103 INFO:teuthology.orchestra.run.smithi044.stdout: "total_workers": 0, 2024-04-21T23:19:14.103 INFO:teuthology.orchestra.run.smithi044.stdout: "unhealthy_workers": 0 2024-04-21T23:19:14.103 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:14.103 INFO:teuthology.orchestra.run.smithi044.stdout: "mempool": { 2024-04-21T23:19:14.103 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:19:14.103 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_items": 0, 2024-04-21T23:19:14.103 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:19:14.104 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:19:14.104 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:19:14.104 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:19:14.104 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:19:14.104 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:19:14.104 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:19:14.104 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:19:14.104 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:19:14.104 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:19:14.105 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:19:14.105 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:19:14.105 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:19:14.105 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_items": 0, 2024-04-21T23:19:14.105 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:19:14.105 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_items": 0, 2024-04-21T23:19:14.105 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:19:14.105 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:19:14.106 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:19:14.106 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:19:14.106 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:19:14.106 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:19:14.106 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:19:14.106 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_items": 0, 2024-04-21T23:19:14.107 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:19:14.107 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:19:14.107 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:19:14.107 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_items": 0, 2024-04-21T23:19:14.107 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_bytes": 0, 2024-04-21T23:19:14.107 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_items": 0, 2024-04-21T23:19:14.107 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:19:14.107 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:19:14.107 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:19:14.108 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:19:14.108 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_bytes": 149057, 2024-04-21T23:19:14.108 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_items": 32, 2024-04-21T23:19:14.108 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_bytes": 0, 2024-04-21T23:19:14.108 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_items": 0, 2024-04-21T23:19:14.108 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_bytes": 0, 2024-04-21T23:19:14.108 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_items": 0, 2024-04-21T23:19:14.108 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:19:14.108 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_items": 0, 2024-04-21T23:19:14.109 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:19:14.109 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_items": 0, 2024-04-21T23:19:14.109 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_bytes": 0, 2024-04-21T23:19:14.109 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_items": 0, 2024-04-21T23:19:14.109 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:19:14.109 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:19:14.109 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_bytes": 0, 2024-04-21T23:19:14.109 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_items": 0, 2024-04-21T23:19:14.109 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_bytes": 928, 2024-04-21T23:19:14.110 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_items": 2, 2024-04-21T23:19:14.110 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_bytes": 0, 2024-04-21T23:19:14.110 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_items": 0, 2024-04-21T23:19:14.110 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_bytes": 0, 2024-04-21T23:19:14.110 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_items": 0 2024-04-21T23:19:14.110 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:14.110 INFO:teuthology.orchestra.run.smithi044.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:19:14.110 INFO:teuthology.orchestra.run.smithi044.stdout: "val": 0, 2024-04-21T23:19:14.111 INFO:teuthology.orchestra.run.smithi044.stdout: "max": 104857600, 2024-04-21T23:19:14.111 INFO:teuthology.orchestra.run.smithi044.stdout: "get_started": 0, 2024-04-21T23:19:14.111 INFO:teuthology.orchestra.run.smithi044.stdout: "get": 221, 2024-04-21T23:19:14.111 INFO:teuthology.orchestra.run.smithi044.stdout: "get_sum": 1411410, 2024-04-21T23:19:14.111 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_fail": 0, 2024-04-21T23:19:14.111 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_success": 221, 2024-04-21T23:19:14.111 INFO:teuthology.orchestra.run.smithi044.stdout: "take": 0, 2024-04-21T23:19:14.111 INFO:teuthology.orchestra.run.smithi044.stdout: "take_sum": 0, 2024-04-21T23:19:14.111 INFO:teuthology.orchestra.run.smithi044.stdout: "put": 221, 2024-04-21T23:19:14.112 INFO:teuthology.orchestra.run.smithi044.stdout: "put_sum": 1411410, 2024-04-21T23:19:14.112 INFO:teuthology.orchestra.run.smithi044.stdout: "wait": { 2024-04-21T23:19:14.112 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-04-21T23:19:14.112 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-04-21T23:19:14.112 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-04-21T23:19:14.112 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:14.112 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:14.112 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:19:14.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:13 smithi055 ceph-mon[31117]: pgmap v328: 129 pgs: 10 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 94 active+clean; 62 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-21T23:19:14.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:13 smithi069 ceph-mon[31816]: pgmap v328: 129 pgs: 10 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 94 active+clean; 62 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-21T23:19:14.836 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-21T23:19:14.836 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-21T23:19:14.836 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-21T23:19:14.837 DEBUG:tasks.check_counter:Getting stats from d 2024-04-21T23:19:14.837 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-21T23:19:15.086 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:19:16.185 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:15 smithi044 ceph-mon[27934]: pgmap v329: 129 pgs: 10 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 94 active+clean; 62 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-21T23:19:16.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:15 smithi055 ceph-mon[31117]: pgmap v329: 129 pgs: 10 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 94 active+clean; 62 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-21T23:19:16.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:15 smithi069 ceph-mon[31816]: pgmap v329: 129 pgs: 10 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 94 active+clean; 62 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-21T23:19:16.786 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:19:16.787 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:19:16.787 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 0, 2024-04-21T23:19:16.787 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 2, 2024-04-21T23:19:16.787 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 0, 2024-04-21T23:19:16.787 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 0, 2024-04-21T23:19:16.787 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 4, 2024-04-21T23:19:16.788 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 0, 2024-04-21T23:19:16.788 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.001307986, 2024-04-21T23:19:16.788 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T23:19:16.788 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.000265087, 2024-04-21T23:19:16.788 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:19:16.788 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:16.788 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-04-21T23:19:16.788 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-04-21T23:19:16.788 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-04-21T23:19:16.789 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:16.789 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:16.789 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-04-21T23:19:16.789 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-04-21T23:19:16.789 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-04-21T23:19:16.789 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:16.789 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T23:19:16.789 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T23:19:16.790 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:16.790 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:19:16.790 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 229, 2024-04-21T23:19:16.790 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 217, 2024-04-21T23:19:16.790 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 1446912, 2024-04-21T23:19:16.790 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 132352, 2024-04-21T23:19:16.790 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 2, 2024-04-21T23:19:16.790 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 1, 2024-04-21T23:19:16.790 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.042656948, 2024-04-21T23:19:16.791 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.012263044, 2024-04-21T23:19:16.791 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.034846338, 2024-04-21T23:19:16.791 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.004129639, 2024-04-21T23:19:16.791 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:16.791 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 217, 2024-04-21T23:19:16.791 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.010042042, 2024-04-21T23:19:16.791 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000046276 2024-04-21T23:19:16.791 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:16.791 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:16.792 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-04-21T23:19:16.792 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-04-21T23:19:16.792 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-04-21T23:19:16.792 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:16.792 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 1446912, 2024-04-21T23:19:16.792 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 132352 2024-04-21T23:19:16.792 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:16.792 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:19:16.792 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 2, 2024-04-21T23:19:16.793 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 164, 2024-04-21T23:19:16.793 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 320, 2024-04-21T23:19:16.793 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 184144, 2024-04-21T23:19:16.793 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 5, 2024-04-21T23:19:16.793 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 3, 2024-04-21T23:19:16.793 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.012241136, 2024-04-21T23:19:16.793 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.007954643, 2024-04-21T23:19:16.793 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.002050215, 2024-04-21T23:19:16.793 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:19:16.794 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:16.794 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 161, 2024-04-21T23:19:16.794 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.016824699, 2024-04-21T23:19:16.794 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000104501 2024-04-21T23:19:16.794 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:16.794 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:16.794 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-04-21T23:19:16.794 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-04-21T23:19:16.795 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-04-21T23:19:16.795 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:16.795 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T23:19:16.795 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 184144 2024-04-21T23:19:16.795 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:16.795 INFO:teuthology.orchestra.run.smithi044.stdout: "cct": { 2024-04-21T23:19:16.795 INFO:teuthology.orchestra.run.smithi044.stdout: "total_workers": 0, 2024-04-21T23:19:16.795 INFO:teuthology.orchestra.run.smithi044.stdout: "unhealthy_workers": 0 2024-04-21T23:19:16.795 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:16.796 INFO:teuthology.orchestra.run.smithi044.stdout: "mempool": { 2024-04-21T23:19:16.796 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:19:16.796 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_items": 0, 2024-04-21T23:19:16.796 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:19:16.796 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:19:16.796 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:19:16.796 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:19:16.796 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:19:16.796 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:19:16.797 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:19:16.797 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:19:16.797 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:19:16.797 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:19:16.797 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:19:16.797 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:19:16.797 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:19:16.797 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_items": 0, 2024-04-21T23:19:16.798 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:19:16.798 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_items": 0, 2024-04-21T23:19:16.798 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:19:16.798 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:19:16.798 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:19:16.798 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:19:16.798 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:19:16.798 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:19:16.798 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:19:16.799 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_items": 0, 2024-04-21T23:19:16.799 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:19:16.799 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:19:16.799 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:19:16.799 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_items": 0, 2024-04-21T23:19:16.799 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_bytes": 0, 2024-04-21T23:19:16.799 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_items": 0, 2024-04-21T23:19:16.799 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:19:16.799 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:19:16.800 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:19:16.800 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:19:16.800 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_bytes": 149057, 2024-04-21T23:19:16.800 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_items": 32, 2024-04-21T23:19:16.800 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_bytes": 0, 2024-04-21T23:19:16.800 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_items": 0, 2024-04-21T23:19:16.800 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_bytes": 0, 2024-04-21T23:19:16.800 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_items": 0, 2024-04-21T23:19:16.800 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:19:16.801 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_items": 0, 2024-04-21T23:19:16.801 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:19:16.801 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_items": 0, 2024-04-21T23:19:16.801 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_bytes": 0, 2024-04-21T23:19:16.801 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_items": 0, 2024-04-21T23:19:16.801 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:19:16.801 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:19:16.801 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_bytes": 0, 2024-04-21T23:19:16.802 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_items": 0, 2024-04-21T23:19:16.802 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_bytes": 928, 2024-04-21T23:19:16.802 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_items": 2, 2024-04-21T23:19:16.802 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_bytes": 0, 2024-04-21T23:19:16.802 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_items": 0, 2024-04-21T23:19:16.802 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_bytes": 0, 2024-04-21T23:19:16.802 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_items": 0 2024-04-21T23:19:16.802 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:16.802 INFO:teuthology.orchestra.run.smithi044.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:19:16.803 INFO:teuthology.orchestra.run.smithi044.stdout: "val": 0, 2024-04-21T23:19:16.803 INFO:teuthology.orchestra.run.smithi044.stdout: "max": 104857600, 2024-04-21T23:19:16.803 INFO:teuthology.orchestra.run.smithi044.stdout: "get_started": 0, 2024-04-21T23:19:16.803 INFO:teuthology.orchestra.run.smithi044.stdout: "get": 227, 2024-04-21T23:19:16.803 INFO:teuthology.orchestra.run.smithi044.stdout: "get_sum": 1412190, 2024-04-21T23:19:16.803 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_fail": 0, 2024-04-21T23:19:16.803 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_success": 227, 2024-04-21T23:19:16.803 INFO:teuthology.orchestra.run.smithi044.stdout: "take": 0, 2024-04-21T23:19:16.803 INFO:teuthology.orchestra.run.smithi044.stdout: "take_sum": 0, 2024-04-21T23:19:16.804 INFO:teuthology.orchestra.run.smithi044.stdout: "put": 227, 2024-04-21T23:19:16.804 INFO:teuthology.orchestra.run.smithi044.stdout: "put_sum": 1412190, 2024-04-21T23:19:16.804 INFO:teuthology.orchestra.run.smithi044.stdout: "wait": { 2024-04-21T23:19:16.804 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-04-21T23:19:16.804 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-04-21T23:19:16.804 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-04-21T23:19:16.804 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:16.804 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:16.804 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:19:17.512 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-21T23:19:17.512 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-21T23:19:17.512 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-21T23:19:17.512 DEBUG:tasks.check_counter:Getting stats from g 2024-04-21T23:19:17.512 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-21T23:19:17.768 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:19:18.155 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:17 smithi044 ceph-mon[27934]: pgmap v330: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 60 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-21T23:19:18.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:17 smithi055 ceph-mon[31117]: pgmap v330: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 60 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-21T23:19:18.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:17 smithi069 ceph-mon[31816]: pgmap v330: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 60 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-21T23:19:19.487 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:19:19.487 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:19:19.487 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 221, 2024-04-21T23:19:19.487 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 211, 2024-04-21T23:19:19.487 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 1442832, 2024-04-21T23:19:19.487 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 129344, 2024-04-21T23:19:19.488 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 4, 2024-04-21T23:19:19.488 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 0, 2024-04-21T23:19:19.488 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.038688418, 2024-04-21T23:19:19.488 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.011448258, 2024-04-21T23:19:19.488 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.031157730, 2024-04-21T23:19:19.488 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.004015917, 2024-04-21T23:19:19.489 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:19.489 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 210, 2024-04-21T23:19:19.489 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.009341443, 2024-04-21T23:19:19.489 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000044483 2024-04-21T23:19:19.489 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:19.489 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:19.489 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-04-21T23:19:19.489 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-04-21T23:19:19.490 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-04-21T23:19:19.490 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:19.490 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 1442832, 2024-04-21T23:19:19.490 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 129344 2024-04-21T23:19:19.490 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:19.490 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:19:19.490 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 2, 2024-04-21T23:19:19.490 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 161, 2024-04-21T23:19:19.491 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 320, 2024-04-21T23:19:19.491 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 183616, 2024-04-21T23:19:19.491 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 5, 2024-04-21T23:19:19.491 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 3, 2024-04-21T23:19:19.491 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.012652009, 2024-04-21T23:19:19.491 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.008057837, 2024-04-21T23:19:19.491 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.002579130, 2024-04-21T23:19:19.491 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:19:19.491 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:19.492 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 158, 2024-04-21T23:19:19.492 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.013507101, 2024-04-21T23:19:19.492 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000085487 2024-04-21T23:19:19.492 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:19.492 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:19.492 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-04-21T23:19:19.492 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-04-21T23:19:19.492 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-04-21T23:19:19.493 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:19.493 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T23:19:19.493 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 183616 2024-04-21T23:19:19.493 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:19.493 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:19:19.493 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 4, 2024-04-21T23:19:19.493 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 4, 2024-04-21T23:19:19.493 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 2992, 2024-04-21T23:19:19.493 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 448, 2024-04-21T23:19:19.494 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 2, 2024-04-21T23:19:19.494 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 1, 2024-04-21T23:19:19.494 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.000735486, 2024-04-21T23:19:19.494 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.000045845, 2024-04-21T23:19:19.494 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.000663969, 2024-04-21T23:19:19.494 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:19:19.495 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:19.495 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 3, 2024-04-21T23:19:19.495 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000891775, 2024-04-21T23:19:19.495 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000297258 2024-04-21T23:19:19.495 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:19.495 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:19.495 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-04-21T23:19:19.495 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-04-21T23:19:19.496 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-04-21T23:19:19.496 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:19.496 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-21T23:19:19.496 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T23:19:19.496 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:19.496 INFO:teuthology.orchestra.run.smithi044.stdout: "cct": { 2024-04-21T23:19:19.496 INFO:teuthology.orchestra.run.smithi044.stdout: "total_workers": 0, 2024-04-21T23:19:19.496 INFO:teuthology.orchestra.run.smithi044.stdout: "unhealthy_workers": 0 2024-04-21T23:19:19.496 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:19.497 INFO:teuthology.orchestra.run.smithi044.stdout: "mempool": { 2024-04-21T23:19:19.497 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:19:19.497 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_items": 0, 2024-04-21T23:19:19.497 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:19:19.497 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:19:19.497 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:19:19.497 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:19:19.497 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:19:19.498 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:19:19.498 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:19:19.498 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:19:19.498 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:19:19.498 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:19:19.498 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:19:19.498 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:19:19.498 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:19:19.499 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_items": 0, 2024-04-21T23:19:19.499 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:19:19.499 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_items": 0, 2024-04-21T23:19:19.499 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:19:19.499 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:19:19.499 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:19:19.499 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:19:19.499 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:19:19.500 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:19:19.500 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:19:19.500 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_items": 0, 2024-04-21T23:19:19.500 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:19:19.500 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:19:19.500 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:19:19.500 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_items": 0, 2024-04-21T23:19:19.500 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_bytes": 0, 2024-04-21T23:19:19.500 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_items": 0, 2024-04-21T23:19:19.501 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:19:19.501 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:19:19.501 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:19:19.501 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:19:19.501 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_bytes": 148977, 2024-04-21T23:19:19.501 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_items": 30, 2024-04-21T23:19:19.502 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_bytes": 0, 2024-04-21T23:19:19.502 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_items": 0, 2024-04-21T23:19:19.502 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_bytes": 0, 2024-04-21T23:19:19.502 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_items": 0, 2024-04-21T23:19:19.502 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:19:19.502 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_items": 0, 2024-04-21T23:19:19.502 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:19:19.502 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_items": 0, 2024-04-21T23:19:19.503 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_bytes": 0, 2024-04-21T23:19:19.503 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_items": 0, 2024-04-21T23:19:19.503 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:19:19.503 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:19:19.503 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_bytes": 0, 2024-04-21T23:19:19.503 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_items": 0, 2024-04-21T23:19:19.503 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_bytes": 928, 2024-04-21T23:19:19.503 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_items": 2, 2024-04-21T23:19:19.504 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_bytes": 0, 2024-04-21T23:19:19.504 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_items": 0, 2024-04-21T23:19:19.504 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_bytes": 0, 2024-04-21T23:19:19.504 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_items": 0 2024-04-21T23:19:19.504 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:19.504 INFO:teuthology.orchestra.run.smithi044.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:19:19.504 INFO:teuthology.orchestra.run.smithi044.stdout: "val": 0, 2024-04-21T23:19:19.504 INFO:teuthology.orchestra.run.smithi044.stdout: "max": 104857600, 2024-04-21T23:19:19.505 INFO:teuthology.orchestra.run.smithi044.stdout: "get_started": 0, 2024-04-21T23:19:19.505 INFO:teuthology.orchestra.run.smithi044.stdout: "get": 223, 2024-04-21T23:19:19.505 INFO:teuthology.orchestra.run.smithi044.stdout: "get_sum": 1411670, 2024-04-21T23:19:19.505 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_fail": 0, 2024-04-21T23:19:19.505 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_success": 223, 2024-04-21T23:19:19.505 INFO:teuthology.orchestra.run.smithi044.stdout: "take": 0, 2024-04-21T23:19:19.505 INFO:teuthology.orchestra.run.smithi044.stdout: "take_sum": 0, 2024-04-21T23:19:19.505 INFO:teuthology.orchestra.run.smithi044.stdout: "put": 223, 2024-04-21T23:19:19.506 INFO:teuthology.orchestra.run.smithi044.stdout: "put_sum": 1411670, 2024-04-21T23:19:19.506 INFO:teuthology.orchestra.run.smithi044.stdout: "wait": { 2024-04-21T23:19:19.506 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-04-21T23:19:19.506 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-04-21T23:19:19.506 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-04-21T23:19:19.506 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:19.506 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:19.506 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:19:20.182 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:19 smithi044 ceph-mon[27934]: pgmap v331: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 190 B/s wr, 2 op/s 2024-04-21T23:19:20.213 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-21T23:19:20.213 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-21T23:19:20.213 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-21T23:19:20.213 DEBUG:tasks.check_counter:Getting stats from j 2024-04-21T23:19:20.214 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-21T23:19:20.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:19 smithi069 ceph-mon[31816]: pgmap v331: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 190 B/s wr, 2 op/s 2024-04-21T23:19:20.215 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:19 smithi055 ceph-mon[31117]: pgmap v331: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 190 B/s wr, 2 op/s 2024-04-21T23:19:20.468 INFO:teuthology.orchestra.run.smithi044.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.a/config 2024-04-21T23:19:21.935 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:21 smithi044 ceph-mon[27934]: pgmap v332: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-04-21T23:19:22.214 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:21 smithi055 ceph-mon[31117]: pgmap v332: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-04-21T23:19:22.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:21 smithi069 ceph-mon[31816]: pgmap v332: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-04-21T23:19:22.518 INFO:teuthology.orchestra.run.smithi044.stdout:{ 2024-04-21T23:19:22.519 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:19:22.519 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 227, 2024-04-21T23:19:22.519 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 217, 2024-04-21T23:19:22.519 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 1444464, 2024-04-21T23:19:22.519 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 133184, 2024-04-21T23:19:22.519 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 4, 2024-04-21T23:19:22.519 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 0, 2024-04-21T23:19:22.520 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.042262908, 2024-04-21T23:19:22.520 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.011584612, 2024-04-21T23:19:22.520 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.033948563, 2024-04-21T23:19:22.520 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.004169989, 2024-04-21T23:19:22.520 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:22.520 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 216, 2024-04-21T23:19:22.520 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.008579814, 2024-04-21T23:19:22.520 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000039721 2024-04-21T23:19:22.521 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:22.521 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:22.521 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-04-21T23:19:22.521 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-04-21T23:19:22.521 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-04-21T23:19:22.521 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:22.521 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 1444464, 2024-04-21T23:19:22.521 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 133184 2024-04-21T23:19:22.521 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:22.522 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:19:22.522 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 2, 2024-04-21T23:19:22.522 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 166, 2024-04-21T23:19:22.522 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 320, 2024-04-21T23:19:22.522 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 184496, 2024-04-21T23:19:22.522 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 5, 2024-04-21T23:19:22.522 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 3, 2024-04-21T23:19:22.522 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.013377100, 2024-04-21T23:19:22.523 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.009027569, 2024-04-21T23:19:22.523 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.002132065, 2024-04-21T23:19:22.523 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:19:22.523 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:22.523 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 163, 2024-04-21T23:19:22.523 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.014183203, 2024-04-21T23:19:22.523 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000087013 2024-04-21T23:19:22.523 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:22.524 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:22.524 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-04-21T23:19:22.524 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-04-21T23:19:22.524 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-04-21T23:19:22.524 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:22.524 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T23:19:22.524 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 184496 2024-04-21T23:19:22.525 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:22.525 INFO:teuthology.orchestra.run.smithi044.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:19:22.525 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_messages": 4, 2024-04-21T23:19:22.525 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages": 4, 2024-04-21T23:19:22.525 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_bytes": 2992, 2024-04-21T23:19:22.525 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_bytes": 448, 2024-04-21T23:19:22.525 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_created_connections": 2, 2024-04-21T23:19:22.525 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_active_connections": 1, 2024-04-21T23:19:22.526 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_total_time": 0.000884693, 2024-04-21T23:19:22.526 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_send_time": 0.000049873, 2024-04-21T23:19:22.526 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_recv_time": 0.000829514, 2024-04-21T23:19:22.526 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:19:22.526 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:22.526 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 3, 2024-04-21T23:19:22.526 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000866680, 2024-04-21T23:19:22.526 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000288893 2024-04-21T23:19:22.527 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:22.527 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:22.527 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-04-21T23:19:22.527 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-04-21T23:19:22.527 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-04-21T23:19:22.527 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:22.527 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-21T23:19:22.527 INFO:teuthology.orchestra.run.smithi044.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T23:19:22.527 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:22.528 INFO:teuthology.orchestra.run.smithi044.stdout: "cct": { 2024-04-21T23:19:22.528 INFO:teuthology.orchestra.run.smithi044.stdout: "total_workers": 0, 2024-04-21T23:19:22.528 INFO:teuthology.orchestra.run.smithi044.stdout: "unhealthy_workers": 0 2024-04-21T23:19:22.528 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:22.528 INFO:teuthology.orchestra.run.smithi044.stdout: "mempool": { 2024-04-21T23:19:22.528 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:19:22.528 INFO:teuthology.orchestra.run.smithi044.stdout: "bloom_filter_items": 0, 2024-04-21T23:19:22.528 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:19:22.529 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:19:22.529 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:19:22.529 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:19:22.529 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:19:22.529 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:19:22.529 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:19:22.529 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:19:22.529 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:19:22.529 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:19:22.530 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:19:22.530 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:19:22.530 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:19:22.530 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_extent_items": 0, 2024-04-21T23:19:22.530 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:19:22.530 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_blob_items": 0, 2024-04-21T23:19:22.530 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:19:22.530 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:19:22.531 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:19:22.531 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:19:22.531 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:19:22.531 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:19:22.531 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:19:22.531 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_txc_items": 0, 2024-04-21T23:19:22.531 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:19:22.531 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:19:22.532 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:19:22.532 INFO:teuthology.orchestra.run.smithi044.stdout: "bluestore_writing_items": 0, 2024-04-21T23:19:22.532 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_bytes": 0, 2024-04-21T23:19:22.532 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_items": 0, 2024-04-21T23:19:22.532 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:19:22.532 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:19:22.532 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:19:22.533 INFO:teuthology.orchestra.run.smithi044.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:19:22.533 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_bytes": 149011, 2024-04-21T23:19:22.533 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_anon_items": 32, 2024-04-21T23:19:22.533 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_bytes": 0, 2024-04-21T23:19:22.533 INFO:teuthology.orchestra.run.smithi044.stdout: "buffer_meta_items": 0, 2024-04-21T23:19:22.533 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_bytes": 0, 2024-04-21T23:19:22.533 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_items": 0, 2024-04-21T23:19:22.533 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:19:22.533 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_mapbl_items": 0, 2024-04-21T23:19:22.534 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:19:22.534 INFO:teuthology.orchestra.run.smithi044.stdout: "osd_pglog_items": 0, 2024-04-21T23:19:22.534 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_bytes": 0, 2024-04-21T23:19:22.534 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_items": 0, 2024-04-21T23:19:22.534 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:19:22.534 INFO:teuthology.orchestra.run.smithi044.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:19:22.534 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_bytes": 0, 2024-04-21T23:19:22.535 INFO:teuthology.orchestra.run.smithi044.stdout: "pgmap_items": 0, 2024-04-21T23:19:22.535 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_bytes": 928, 2024-04-21T23:19:22.535 INFO:teuthology.orchestra.run.smithi044.stdout: "mds_co_items": 2, 2024-04-21T23:19:22.535 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_bytes": 0, 2024-04-21T23:19:22.535 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_1_items": 0, 2024-04-21T23:19:22.535 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_bytes": 0, 2024-04-21T23:19:22.535 INFO:teuthology.orchestra.run.smithi044.stdout: "unittest_2_items": 0 2024-04-21T23:19:22.535 INFO:teuthology.orchestra.run.smithi044.stdout: }, 2024-04-21T23:19:22.536 INFO:teuthology.orchestra.run.smithi044.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:19:22.536 INFO:teuthology.orchestra.run.smithi044.stdout: "val": 0, 2024-04-21T23:19:22.536 INFO:teuthology.orchestra.run.smithi044.stdout: "max": 104857600, 2024-04-21T23:19:22.536 INFO:teuthology.orchestra.run.smithi044.stdout: "get_started": 0, 2024-04-21T23:19:22.536 INFO:teuthology.orchestra.run.smithi044.stdout: "get": 229, 2024-04-21T23:19:22.536 INFO:teuthology.orchestra.run.smithi044.stdout: "get_sum": 1412450, 2024-04-21T23:19:22.536 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_fail": 0, 2024-04-21T23:19:22.536 INFO:teuthology.orchestra.run.smithi044.stdout: "get_or_fail_success": 229, 2024-04-21T23:19:22.537 INFO:teuthology.orchestra.run.smithi044.stdout: "take": 0, 2024-04-21T23:19:22.537 INFO:teuthology.orchestra.run.smithi044.stdout: "take_sum": 0, 2024-04-21T23:19:22.537 INFO:teuthology.orchestra.run.smithi044.stdout: "put": 229, 2024-04-21T23:19:22.537 INFO:teuthology.orchestra.run.smithi044.stdout: "put_sum": 1412450, 2024-04-21T23:19:22.537 INFO:teuthology.orchestra.run.smithi044.stdout: "wait": { 2024-04-21T23:19:22.537 INFO:teuthology.orchestra.run.smithi044.stdout: "avgcount": 0, 2024-04-21T23:19:22.537 INFO:teuthology.orchestra.run.smithi044.stdout: "sum": 0.000000000, 2024-04-21T23:19:22.537 INFO:teuthology.orchestra.run.smithi044.stdout: "avgtime": 0.000000000 2024-04-21T23:19:22.538 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:22.538 INFO:teuthology.orchestra.run.smithi044.stdout: } 2024-04-21T23:19:22.538 INFO:teuthology.orchestra.run.smithi044.stdout:} 2024-04-21T23:19:23.248 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-21T23:19:23.248 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-21T23:19:23.248 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-21T23:19:23.248 DEBUG:tasks.check_counter:Getting stats from b 2024-04-21T23:19:23.248 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-21T23:19:23.528 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:19:24.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:24 smithi044 ceph-mon[27934]: pgmap v333: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-04-21T23:19:24.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:24 smithi044 ceph-mon[27934]: osdmap e110: 12 total, 12 up, 12 in 2024-04-21T23:19:24.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:24 smithi055 ceph-mon[31117]: pgmap v333: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-04-21T23:19:24.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:24 smithi055 ceph-mon[31117]: osdmap e110: 12 total, 12 up, 12 in 2024-04-21T23:19:24.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:24 smithi069 ceph-mon[31816]: pgmap v333: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-04-21T23:19:24.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:24 smithi069 ceph-mon[31816]: osdmap e110: 12 total, 12 up, 12 in 2024-04-21T23:19:25.844 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-04-21T23:19:25.844 INFO:teuthology.orchestra.run.smithi055.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:19:25.845 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_messages": 67221, 2024-04-21T23:19:25.845 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages": 66628, 2024-04-21T23:19:25.845 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_bytes": 18084192, 2024-04-21T23:19:25.845 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_bytes": 274920401, 2024-04-21T23:19:25.845 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_created_connections": 16, 2024-04-21T23:19:25.845 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_active_connections": 5, 2024-04-21T23:19:25.846 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_total_time": 5.442879207, 2024-04-21T23:19:25.846 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_send_time": 2.162378698, 2024-04-21T23:19:25.846 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_recv_time": 1.023468652, 2024-04-21T23:19:25.846 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_fast_dispatch_time": 1.838441398, 2024-04-21T23:19:25.846 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:25.846 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 66627, 2024-04-21T23:19:25.846 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 54.181988971, 2024-04-21T23:19:25.846 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000813213 2024-04-21T23:19:25.847 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.847 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:25.847 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 913, 2024-04-21T23:19:25.847 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000139353, 2024-04-21T23:19:25.847 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000152 2024-04-21T23:19:25.847 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.847 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_encrypted_bytes": 333936, 2024-04-21T23:19:25.847 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_encrypted_bytes": 99680 2024-04-21T23:19:25.847 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.848 INFO:teuthology.orchestra.run.smithi055.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:19:25.848 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_messages": 54455, 2024-04-21T23:19:25.848 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages": 54349, 2024-04-21T23:19:25.848 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_bytes": 19398284, 2024-04-21T23:19:25.848 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_bytes": 229673681, 2024-04-21T23:19:25.848 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_created_connections": 233, 2024-04-21T23:19:25.849 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_active_connections": 8, 2024-04-21T23:19:25.849 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_total_time": 4.397142119, 2024-04-21T23:19:25.849 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_send_time": 1.684384022, 2024-04-21T23:19:25.849 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_recv_time": 0.944447643, 2024-04-21T23:19:25.849 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_fast_dispatch_time": 1.438104131, 2024-04-21T23:19:25.849 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:25.849 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 54348, 2024-04-21T23:19:25.850 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 32.690700713, 2024-04-21T23:19:25.850 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000601506 2024-04-21T23:19:25.850 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.851 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:25.851 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 331, 2024-04-21T23:19:25.851 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000038852, 2024-04-21T23:19:25.851 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000117 2024-04-21T23:19:25.851 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.851 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_encrypted_bytes": 1554352, 2024-04-21T23:19:25.851 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_encrypted_bytes": 168768 2024-04-21T23:19:25.851 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.852 INFO:teuthology.orchestra.run.smithi055.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:19:25.852 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_messages": 321543, 2024-04-21T23:19:25.852 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages": 504396, 2024-04-21T23:19:25.852 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_bytes": 96457246, 2024-04-21T23:19:25.852 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_bytes": 458642586, 2024-04-21T23:19:25.852 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_created_connections": 16, 2024-04-21T23:19:25.853 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_active_connections": 5, 2024-04-21T23:19:25.853 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_total_time": 18.128354729, 2024-04-21T23:19:25.853 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_send_time": 7.813542474, 2024-04-21T23:19:25.853 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_recv_time": 10.444483166, 2024-04-21T23:19:25.853 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_fast_dispatch_time": 1.251845625, 2024-04-21T23:19:25.853 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:25.853 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 504393, 2024-04-21T23:19:25.853 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 42.190180933, 2024-04-21T23:19:25.854 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000083645 2024-04-21T23:19:25.854 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.854 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:25.854 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 704503, 2024-04-21T23:19:25.854 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.106307026, 2024-04-21T23:19:25.854 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000150 2024-04-21T23:19:25.854 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.854 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_encrypted_bytes": 102432, 2024-04-21T23:19:25.855 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_encrypted_bytes": 489552 2024-04-21T23:19:25.855 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.855 INFO:teuthology.orchestra.run.smithi055.stdout: "cct": { 2024-04-21T23:19:25.855 INFO:teuthology.orchestra.run.smithi055.stdout: "total_workers": 1, 2024-04-21T23:19:25.855 INFO:teuthology.orchestra.run.smithi055.stdout: "unhealthy_workers": 0 2024-04-21T23:19:25.855 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.855 INFO:teuthology.orchestra.run.smithi055.stdout: "finisher-MDSRank": { 2024-04-21T23:19:25.856 INFO:teuthology.orchestra.run.smithi055.stdout: "queue_len": 0, 2024-04-21T23:19:25.856 INFO:teuthology.orchestra.run.smithi055.stdout: "complete_latency": { 2024-04-21T23:19:25.856 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 105364, 2024-04-21T23:19:25.856 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 102.812282212, 2024-04-21T23:19:25.856 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000975781 2024-04-21T23:19:25.856 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-04-21T23:19:25.856 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.856 INFO:teuthology.orchestra.run.smithi055.stdout: "finisher-PurgeQueue": { 2024-04-21T23:19:25.857 INFO:teuthology.orchestra.run.smithi055.stdout: "queue_len": 0, 2024-04-21T23:19:25.857 INFO:teuthology.orchestra.run.smithi055.stdout: "complete_latency": { 2024-04-21T23:19:25.857 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 39031, 2024-04-21T23:19:25.857 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 8.429965088, 2024-04-21T23:19:25.858 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000215981 2024-04-21T23:19:25.858 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-04-21T23:19:25.858 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.858 INFO:teuthology.orchestra.run.smithi055.stdout: "mds": { 2024-04-21T23:19:25.858 INFO:teuthology.orchestra.run.smithi055.stdout: "request": 205729, 2024-04-21T23:19:25.858 INFO:teuthology.orchestra.run.smithi055.stdout: "reply": 205729, 2024-04-21T23:19:25.859 INFO:teuthology.orchestra.run.smithi055.stdout: "reply_latency": { 2024-04-21T23:19:25.859 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 205729, 2024-04-21T23:19:25.859 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 290.429061008, 2024-04-21T23:19:25.859 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.001411706 2024-04-21T23:19:25.859 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.860 INFO:teuthology.orchestra.run.smithi055.stdout: "slow_reply": 0, 2024-04-21T23:19:25.860 INFO:teuthology.orchestra.run.smithi055.stdout: "forward": 0, 2024-04-21T23:19:25.860 INFO:teuthology.orchestra.run.smithi055.stdout: "dir_fetch_complete": 0, 2024-04-21T23:19:25.860 INFO:teuthology.orchestra.run.smithi055.stdout: "dir_fetch_keys": 0, 2024-04-21T23:19:25.860 INFO:teuthology.orchestra.run.smithi055.stdout: "dir_commit": 2241, 2024-04-21T23:19:25.860 INFO:teuthology.orchestra.run.smithi055.stdout: "dir_split": 17, 2024-04-21T23:19:25.860 INFO:teuthology.orchestra.run.smithi055.stdout: "dir_merge": 19, 2024-04-21T23:19:25.860 INFO:teuthology.orchestra.run.smithi055.stdout: "inodes": 2350, 2024-04-21T23:19:25.860 INFO:teuthology.orchestra.run.smithi055.stdout: "inodes_top": 806, 2024-04-21T23:19:25.861 INFO:teuthology.orchestra.run.smithi055.stdout: "inodes_bottom": 1544, 2024-04-21T23:19:25.861 INFO:teuthology.orchestra.run.smithi055.stdout: "inodes_pin_tail": 0, 2024-04-21T23:19:25.861 INFO:teuthology.orchestra.run.smithi055.stdout: "inodes_pinned": 1199, 2024-04-21T23:19:25.861 INFO:teuthology.orchestra.run.smithi055.stdout: "inodes_expired": 0, 2024-04-21T23:19:25.861 INFO:teuthology.orchestra.run.smithi055.stdout: "inodes_with_caps": 53, 2024-04-21T23:19:25.861 INFO:teuthology.orchestra.run.smithi055.stdout: "caps": 54, 2024-04-21T23:19:25.861 INFO:teuthology.orchestra.run.smithi055.stdout: "subtrees": 2, 2024-04-21T23:19:25.861 INFO:teuthology.orchestra.run.smithi055.stdout: "traverse": 284362, 2024-04-21T23:19:25.861 INFO:teuthology.orchestra.run.smithi055.stdout: "traverse_hit": 173001, 2024-04-21T23:19:25.862 INFO:teuthology.orchestra.run.smithi055.stdout: "traverse_forward": 0, 2024-04-21T23:19:25.862 INFO:teuthology.orchestra.run.smithi055.stdout: "traverse_discover": 0, 2024-04-21T23:19:25.862 INFO:teuthology.orchestra.run.smithi055.stdout: "traverse_dir_fetch": 0, 2024-04-21T23:19:25.862 INFO:teuthology.orchestra.run.smithi055.stdout: "traverse_remote_ino": 0, 2024-04-21T23:19:25.862 INFO:teuthology.orchestra.run.smithi055.stdout: "traverse_lock": 2462, 2024-04-21T23:19:25.862 INFO:teuthology.orchestra.run.smithi055.stdout: "load_cent": 0, 2024-04-21T23:19:25.862 INFO:teuthology.orchestra.run.smithi055.stdout: "q": 0, 2024-04-21T23:19:25.863 INFO:teuthology.orchestra.run.smithi055.stdout: "exported": 0, 2024-04-21T23:19:25.863 INFO:teuthology.orchestra.run.smithi055.stdout: "exported_inodes": 0, 2024-04-21T23:19:25.863 INFO:teuthology.orchestra.run.smithi055.stdout: "imported": 0, 2024-04-21T23:19:25.863 INFO:teuthology.orchestra.run.smithi055.stdout: "imported_inodes": 0, 2024-04-21T23:19:25.863 INFO:teuthology.orchestra.run.smithi055.stdout: "openino_dir_fetch": 0, 2024-04-21T23:19:25.863 INFO:teuthology.orchestra.run.smithi055.stdout: "openino_backtrace_fetch": 1668, 2024-04-21T23:19:25.863 INFO:teuthology.orchestra.run.smithi055.stdout: "openino_peer_discover": 0, 2024-04-21T23:19:25.863 INFO:teuthology.orchestra.run.smithi055.stdout: "root_rfiles": 68, 2024-04-21T23:19:25.864 INFO:teuthology.orchestra.run.smithi055.stdout: "root_rbytes": 21196213, 2024-04-21T23:19:25.864 INFO:teuthology.orchestra.run.smithi055.stdout: "root_rsnaps": 6, 2024-04-21T23:19:25.864 INFO:teuthology.orchestra.run.smithi055.stdout: "scrub_backtrace_fetch": 1668, 2024-04-21T23:19:25.864 INFO:teuthology.orchestra.run.smithi055.stdout: "scrub_set_tag": 0, 2024-04-21T23:19:25.864 INFO:teuthology.orchestra.run.smithi055.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T23:19:25.864 INFO:teuthology.orchestra.run.smithi055.stdout: "scrub_inotable_repaired": 0, 2024-04-21T23:19:25.864 INFO:teuthology.orchestra.run.smithi055.stdout: "scrub_dir_inodes": 380, 2024-04-21T23:19:25.865 INFO:teuthology.orchestra.run.smithi055.stdout: "scrub_dir_base_inodes": 20, 2024-04-21T23:19:25.865 INFO:teuthology.orchestra.run.smithi055.stdout: "scrub_dirfrag_rstats": 360, 2024-04-21T23:19:25.865 INFO:teuthology.orchestra.run.smithi055.stdout: "scrub_file_inodes": 1288, 2024-04-21T23:19:25.865 INFO:teuthology.orchestra.run.smithi055.stdout: "handle_inode_file_caps": 0, 2024-04-21T23:19:25.865 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_cap_op_revoke": 16919, 2024-04-21T23:19:25.865 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_cap_op_grant": 146722, 2024-04-21T23:19:25.865 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_cap_op_trunc": 0, 2024-04-21T23:19:25.866 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T23:19:25.866 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T23:19:25.866 INFO:teuthology.orchestra.run.smithi055.stdout: "handle_client_caps": 62138, 2024-04-21T23:19:25.866 INFO:teuthology.orchestra.run.smithi055.stdout: "handle_client_caps_dirty": 25969, 2024-04-21T23:19:25.866 INFO:teuthology.orchestra.run.smithi055.stdout: "handle_client_cap_release": 12795, 2024-04-21T23:19:25.866 INFO:teuthology.orchestra.run.smithi055.stdout: "process_request_cap_release": 131502 2024-04-21T23:19:25.866 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.866 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_cache": { 2024-04-21T23:19:25.867 INFO:teuthology.orchestra.run.smithi055.stdout: "dir_update": 0, 2024-04-21T23:19:25.867 INFO:teuthology.orchestra.run.smithi055.stdout: "dir_update_receipt": 0, 2024-04-21T23:19:25.867 INFO:teuthology.orchestra.run.smithi055.stdout: "dir_try_discover": 0, 2024-04-21T23:19:25.867 INFO:teuthology.orchestra.run.smithi055.stdout: "dir_send_discover": 0, 2024-04-21T23:19:25.867 INFO:teuthology.orchestra.run.smithi055.stdout: "dir_handle_discover": 0, 2024-04-21T23:19:25.867 INFO:teuthology.orchestra.run.smithi055.stdout: "num_strays": 0, 2024-04-21T23:19:25.867 INFO:teuthology.orchestra.run.smithi055.stdout: "num_strays_delayed": 0, 2024-04-21T23:19:25.867 INFO:teuthology.orchestra.run.smithi055.stdout: "num_strays_enqueuing": 0, 2024-04-21T23:19:25.868 INFO:teuthology.orchestra.run.smithi055.stdout: "strays_created": 33133, 2024-04-21T23:19:25.868 INFO:teuthology.orchestra.run.smithi055.stdout: "strays_enqueued": 33133, 2024-04-21T23:19:25.868 INFO:teuthology.orchestra.run.smithi055.stdout: "strays_reintegrated": 0, 2024-04-21T23:19:25.868 INFO:teuthology.orchestra.run.smithi055.stdout: "strays_migrated": 0, 2024-04-21T23:19:25.868 INFO:teuthology.orchestra.run.smithi055.stdout: "num_recovering_processing": 0, 2024-04-21T23:19:25.868 INFO:teuthology.orchestra.run.smithi055.stdout: "num_recovering_enqueued": 0, 2024-04-21T23:19:25.868 INFO:teuthology.orchestra.run.smithi055.stdout: "num_recovering_prioritized": 0, 2024-04-21T23:19:25.868 INFO:teuthology.orchestra.run.smithi055.stdout: "recovery_started": 0, 2024-04-21T23:19:25.868 INFO:teuthology.orchestra.run.smithi055.stdout: "recovery_completed": 0, 2024-04-21T23:19:25.869 INFO:teuthology.orchestra.run.smithi055.stdout: "ireq_quiesce_path": 0, 2024-04-21T23:19:25.869 INFO:teuthology.orchestra.run.smithi055.stdout: "ireq_quiesce_inode": 0, 2024-04-21T23:19:25.869 INFO:teuthology.orchestra.run.smithi055.stdout: "ireq_enqueue_scrub": 20, 2024-04-21T23:19:25.869 INFO:teuthology.orchestra.run.smithi055.stdout: "ireq_exportdir": 0, 2024-04-21T23:19:25.869 INFO:teuthology.orchestra.run.smithi055.stdout: "ireq_flush": 0, 2024-04-21T23:19:25.869 INFO:teuthology.orchestra.run.smithi055.stdout: "ireq_fragmentdir": 36, 2024-04-21T23:19:25.869 INFO:teuthology.orchestra.run.smithi055.stdout: "ireq_fragstats": 0, 2024-04-21T23:19:25.869 INFO:teuthology.orchestra.run.smithi055.stdout: "ireq_inodestats": 0 2024-04-21T23:19:25.869 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.870 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_log": { 2024-04-21T23:19:25.870 INFO:teuthology.orchestra.run.smithi055.stdout: "evlrg": 0, 2024-04-21T23:19:25.870 INFO:teuthology.orchestra.run.smithi055.stdout: "evadd": 150841, 2024-04-21T23:19:25.870 INFO:teuthology.orchestra.run.smithi055.stdout: "evex": 148403, 2024-04-21T23:19:25.870 INFO:teuthology.orchestra.run.smithi055.stdout: "evtrm": 148403, 2024-04-21T23:19:25.870 INFO:teuthology.orchestra.run.smithi055.stdout: "ev": 2438, 2024-04-21T23:19:25.870 INFO:teuthology.orchestra.run.smithi055.stdout: "evexg": 0, 2024-04-21T23:19:25.870 INFO:teuthology.orchestra.run.smithi055.stdout: "evexd": 4390, 2024-04-21T23:19:25.871 INFO:teuthology.orchestra.run.smithi055.stdout: "segadd": 236, 2024-04-21T23:19:25.871 INFO:teuthology.orchestra.run.smithi055.stdout: "segex": 233, 2024-04-21T23:19:25.871 INFO:teuthology.orchestra.run.smithi055.stdout: "segtrm": 233, 2024-04-21T23:19:25.871 INFO:teuthology.orchestra.run.smithi055.stdout: "seg": 3, 2024-04-21T23:19:25.871 INFO:teuthology.orchestra.run.smithi055.stdout: "segmjr": 1, 2024-04-21T23:19:25.871 INFO:teuthology.orchestra.run.smithi055.stdout: "segexg": 0, 2024-04-21T23:19:25.871 INFO:teuthology.orchestra.run.smithi055.stdout: "segexd": 7, 2024-04-21T23:19:25.871 INFO:teuthology.orchestra.run.smithi055.stdout: "expos": 484446288, 2024-04-21T23:19:25.872 INFO:teuthology.orchestra.run.smithi055.stdout: "wrpos": 489967567, 2024-04-21T23:19:25.872 INFO:teuthology.orchestra.run.smithi055.stdout: "rdpos": 0, 2024-04-21T23:19:25.872 INFO:teuthology.orchestra.run.smithi055.stdout: "jlat": { 2024-04-21T23:19:25.872 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 52032, 2024-04-21T23:19:25.872 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 145.169445928, 2024-04-21T23:19:25.872 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.002790003 2024-04-21T23:19:25.872 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.872 INFO:teuthology.orchestra.run.smithi055.stdout: "replayed": 0 2024-04-21T23:19:25.872 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.873 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_mem": { 2024-04-21T23:19:25.873 INFO:teuthology.orchestra.run.smithi055.stdout: "ino": 515, 2024-04-21T23:19:25.873 INFO:teuthology.orchestra.run.smithi055.stdout: "ino+": 33668, 2024-04-21T23:19:25.873 INFO:teuthology.orchestra.run.smithi055.stdout: "ino-": 33153, 2024-04-21T23:19:25.873 INFO:teuthology.orchestra.run.smithi055.stdout: "dir": 46, 2024-04-21T23:19:25.873 INFO:teuthology.orchestra.run.smithi055.stdout: "dir+": 187, 2024-04-21T23:19:25.873 INFO:teuthology.orchestra.run.smithi055.stdout: "dir-": 141, 2024-04-21T23:19:25.873 INFO:teuthology.orchestra.run.smithi055.stdout: "dn": 2350, 2024-04-21T23:19:25.873 INFO:teuthology.orchestra.run.smithi055.stdout: "dn+": 71307, 2024-04-21T23:19:25.874 INFO:teuthology.orchestra.run.smithi055.stdout: "dn-": 68957, 2024-04-21T23:19:25.874 INFO:teuthology.orchestra.run.smithi055.stdout: "cap": 54, 2024-04-21T23:19:25.874 INFO:teuthology.orchestra.run.smithi055.stdout: "cap+": 34982, 2024-04-21T23:19:25.874 INFO:teuthology.orchestra.run.smithi055.stdout: "cap-": 34928, 2024-04-21T23:19:25.874 INFO:teuthology.orchestra.run.smithi055.stdout: "rss": 94156, 2024-04-21T23:19:25.874 INFO:teuthology.orchestra.run.smithi055.stdout: "heap": 231804 2024-04-21T23:19:25.874 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.874 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_server": { 2024-04-21T23:19:25.875 INFO:teuthology.orchestra.run.smithi055.stdout: "dispatch_client_request": 258580, 2024-04-21T23:19:25.875 INFO:teuthology.orchestra.run.smithi055.stdout: "dispatch_server_request": 0, 2024-04-21T23:19:25.875 INFO:teuthology.orchestra.run.smithi055.stdout: "handle_client_request": 205729, 2024-04-21T23:19:25.875 INFO:teuthology.orchestra.run.smithi055.stdout: "handle_client_session": 4567, 2024-04-21T23:19:25.875 INFO:teuthology.orchestra.run.smithi055.stdout: "handle_peer_request": 0, 2024-04-21T23:19:25.875 INFO:teuthology.orchestra.run.smithi055.stdout: "req_create_latency": { 2024-04-21T23:19:25.875 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 33201, 2024-04-21T23:19:25.875 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 39.319093391, 2024-04-21T23:19:25.875 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.001184274 2024-04-21T23:19:25.876 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.876 INFO:teuthology.orchestra.run.smithi055.stdout: "req_getattr_latency": { 2024-04-21T23:19:25.876 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 401, 2024-04-21T23:19:25.876 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.830141392, 2024-04-21T23:19:25.876 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.002070178 2024-04-21T23:19:25.876 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.876 INFO:teuthology.orchestra.run.smithi055.stdout: "req_getfilelock_latency": { 2024-04-21T23:19:25.876 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:25.876 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:25.877 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:25.877 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.877 INFO:teuthology.orchestra.run.smithi055.stdout: "req_link_latency": { 2024-04-21T23:19:25.877 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:25.877 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:25.877 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:25.877 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.877 INFO:teuthology.orchestra.run.smithi055.stdout: "req_lookup_latency": { 2024-04-21T23:19:25.877 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 68878, 2024-04-21T23:19:25.878 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 40.474345744, 2024-04-21T23:19:25.878 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000587623 2024-04-21T23:19:25.878 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.878 INFO:teuthology.orchestra.run.smithi055.stdout: "req_lookuphash_latency": { 2024-04-21T23:19:25.878 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:25.878 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:25.878 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:25.879 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.879 INFO:teuthology.orchestra.run.smithi055.stdout: "req_lookupino_latency": { 2024-04-21T23:19:25.879 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 2, 2024-04-21T23:19:25.879 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000559918, 2024-04-21T23:19:25.879 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000279959 2024-04-21T23:19:25.879 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.879 INFO:teuthology.orchestra.run.smithi055.stdout: "req_lookupname_latency": { 2024-04-21T23:19:25.880 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:25.880 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:25.880 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:25.880 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.880 INFO:teuthology.orchestra.run.smithi055.stdout: "req_lookupparent_latency": { 2024-04-21T23:19:25.880 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:25.880 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:25.880 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:25.880 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.881 INFO:teuthology.orchestra.run.smithi055.stdout: "req_lookupsnap_latency": { 2024-04-21T23:19:25.881 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:25.881 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:25.882 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:25.882 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.882 INFO:teuthology.orchestra.run.smithi055.stdout: "req_lssnap_latency": { 2024-04-21T23:19:25.882 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 11, 2024-04-21T23:19:25.882 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.004284444, 2024-04-21T23:19:25.882 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000389494 2024-04-21T23:19:25.882 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.882 INFO:teuthology.orchestra.run.smithi055.stdout: "req_mkdir_latency": { 2024-04-21T23:19:25.883 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 20, 2024-04-21T23:19:25.883 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.021271714, 2024-04-21T23:19:25.883 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.001063585 2024-04-21T23:19:25.883 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.883 INFO:teuthology.orchestra.run.smithi055.stdout: "req_mknod_latency": { 2024-04-21T23:19:25.884 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:25.884 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:25.884 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:25.884 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.884 INFO:teuthology.orchestra.run.smithi055.stdout: "req_mksnap_latency": { 2024-04-21T23:19:25.884 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 11, 2024-04-21T23:19:25.884 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.069411202, 2024-04-21T23:19:25.884 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.006310109 2024-04-21T23:19:25.884 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.885 INFO:teuthology.orchestra.run.smithi055.stdout: "req_open_latency": { 2024-04-21T23:19:25.885 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 1, 2024-04-21T23:19:25.885 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000286954, 2024-04-21T23:19:25.885 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000286954 2024-04-21T23:19:25.885 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.885 INFO:teuthology.orchestra.run.smithi055.stdout: "req_readdir_latency": { 2024-04-21T23:19:25.885 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 62057, 2024-04-21T23:19:25.885 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 152.579508788, 2024-04-21T23:19:25.886 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.002458699 2024-04-21T23:19:25.886 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.886 INFO:teuthology.orchestra.run.smithi055.stdout: "req_rename_latency": { 2024-04-21T23:19:25.886 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 6931, 2024-04-21T23:19:25.886 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 10.801798642, 2024-04-21T23:19:25.886 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.001558476 2024-04-21T23:19:25.886 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.886 INFO:teuthology.orchestra.run.smithi055.stdout: "req_renamesnap_latency": { 2024-04-21T23:19:25.887 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:25.887 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:25.887 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:25.887 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.887 INFO:teuthology.orchestra.run.smithi055.stdout: "req_snapdiff_latency": { 2024-04-21T23:19:25.887 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:25.887 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:25.887 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:25.887 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.888 INFO:teuthology.orchestra.run.smithi055.stdout: "req_rmdir_latency": { 2024-04-21T23:19:25.888 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:25.888 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:25.888 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:25.888 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.888 INFO:teuthology.orchestra.run.smithi055.stdout: "req_rmsnap_latency": { 2024-04-21T23:19:25.888 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 5, 2024-04-21T23:19:25.888 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.067192884, 2024-04-21T23:19:25.889 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.013438576 2024-04-21T23:19:25.889 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.889 INFO:teuthology.orchestra.run.smithi055.stdout: "req_rmxattr_latency": { 2024-04-21T23:19:25.889 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:25.889 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:25.889 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:25.889 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.889 INFO:teuthology.orchestra.run.smithi055.stdout: "req_setattr_latency": { 2024-04-21T23:19:25.889 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 6, 2024-04-21T23:19:25.890 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.044348247, 2024-04-21T23:19:25.890 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.007391374 2024-04-21T23:19:25.890 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.890 INFO:teuthology.orchestra.run.smithi055.stdout: "req_setdirlayout_latency": { 2024-04-21T23:19:25.891 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:25.891 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:25.891 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:25.891 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.892 INFO:teuthology.orchestra.run.smithi055.stdout: "req_setfilelock_latency": { 2024-04-21T23:19:25.892 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 1064, 2024-04-21T23:19:25.892 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.640011071, 2024-04-21T23:19:25.892 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000601514 2024-04-21T23:19:25.893 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.893 INFO:teuthology.orchestra.run.smithi055.stdout: "req_setlayout_latency": { 2024-04-21T23:19:25.893 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:25.893 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:25.893 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:25.894 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.894 INFO:teuthology.orchestra.run.smithi055.stdout: "req_setxattr_latency": { 2024-04-21T23:19:25.894 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 8, 2024-04-21T23:19:25.894 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.022384438, 2024-04-21T23:19:25.895 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.002798054 2024-04-21T23:19:25.895 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.895 INFO:teuthology.orchestra.run.smithi055.stdout: "req_symlink_latency": { 2024-04-21T23:19:25.895 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:25.896 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:25.896 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:25.896 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.896 INFO:teuthology.orchestra.run.smithi055.stdout: "req_unlink_latency": { 2024-04-21T23:19:25.896 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 33133, 2024-04-21T23:19:25.897 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 45.554422179, 2024-04-21T23:19:25.897 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.001374895 2024-04-21T23:19:25.897 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.897 INFO:teuthology.orchestra.run.smithi055.stdout: "cap_revoke_eviction": 0, 2024-04-21T23:19:25.898 INFO:teuthology.orchestra.run.smithi055.stdout: "cap_acquisition_throttle": 0, 2024-04-21T23:19:25.898 INFO:teuthology.orchestra.run.smithi055.stdout: "req_getvxattr_latency": { 2024-04-21T23:19:25.898 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:25.898 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:25.899 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:25.899 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-04-21T23:19:25.899 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.899 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_sessions": { 2024-04-21T23:19:25.900 INFO:teuthology.orchestra.run.smithi055.stdout: "session_count": 2, 2024-04-21T23:19:25.900 INFO:teuthology.orchestra.run.smithi055.stdout: "session_add": 6, 2024-04-21T23:19:25.900 INFO:teuthology.orchestra.run.smithi055.stdout: "session_remove": 4, 2024-04-21T23:19:25.900 INFO:teuthology.orchestra.run.smithi055.stdout: "sessions_open": 2, 2024-04-21T23:19:25.901 INFO:teuthology.orchestra.run.smithi055.stdout: "sessions_stale": 0, 2024-04-21T23:19:25.901 INFO:teuthology.orchestra.run.smithi055.stdout: "total_load": 10546, 2024-04-21T23:19:25.901 INFO:teuthology.orchestra.run.smithi055.stdout: "average_load": 5273, 2024-04-21T23:19:25.901 INFO:teuthology.orchestra.run.smithi055.stdout: "avg_session_uptime": 627, 2024-04-21T23:19:25.901 INFO:teuthology.orchestra.run.smithi055.stdout: "mdthresh_evicted": 0 2024-04-21T23:19:25.902 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.902 INFO:teuthology.orchestra.run.smithi055.stdout: "mempool": { 2024-04-21T23:19:25.902 INFO:teuthology.orchestra.run.smithi055.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:19:25.902 INFO:teuthology.orchestra.run.smithi055.stdout: "bloom_filter_items": 0, 2024-04-21T23:19:25.903 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:19:25.903 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:19:25.903 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:19:25.903 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:19:25.904 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:19:25.904 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:19:25.904 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:19:25.904 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:19:25.905 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:19:25.905 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:19:25.905 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:19:25.905 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:19:25.906 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:19:25.906 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_extent_items": 0, 2024-04-21T23:19:25.906 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:19:25.906 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_blob_items": 0, 2024-04-21T23:19:25.906 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:19:25.906 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:19:25.907 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:19:25.907 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:19:25.907 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:19:25.907 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:19:25.908 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:19:25.908 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_txc_items": 0, 2024-04-21T23:19:25.908 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:19:25.908 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:19:25.908 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:19:25.908 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_writing_items": 0, 2024-04-21T23:19:25.908 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_bytes": 0, 2024-04-21T23:19:25.909 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_items": 0, 2024-04-21T23:19:25.909 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:19:25.909 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:19:25.909 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:19:25.909 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:19:25.909 INFO:teuthology.orchestra.run.smithi055.stdout: "buffer_anon_bytes": 313676, 2024-04-21T23:19:25.909 INFO:teuthology.orchestra.run.smithi055.stdout: "buffer_anon_items": 413, 2024-04-21T23:19:25.909 INFO:teuthology.orchestra.run.smithi055.stdout: "buffer_meta_bytes": 352, 2024-04-21T23:19:25.910 INFO:teuthology.orchestra.run.smithi055.stdout: "buffer_meta_items": 4, 2024-04-21T23:19:25.910 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_bytes": 0, 2024-04-21T23:19:25.910 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_items": 0, 2024-04-21T23:19:25.911 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:19:25.911 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mapbl_items": 0, 2024-04-21T23:19:25.911 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:19:25.911 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pglog_items": 0, 2024-04-21T23:19:25.911 INFO:teuthology.orchestra.run.smithi055.stdout: "osdmap_bytes": 9856, 2024-04-21T23:19:25.911 INFO:teuthology.orchestra.run.smithi055.stdout: "osdmap_items": 191, 2024-04-21T23:19:25.911 INFO:teuthology.orchestra.run.smithi055.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:19:25.911 INFO:teuthology.orchestra.run.smithi055.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:19:25.912 INFO:teuthology.orchestra.run.smithi055.stdout: "pgmap_bytes": 0, 2024-04-21T23:19:25.912 INFO:teuthology.orchestra.run.smithi055.stdout: "pgmap_items": 0, 2024-04-21T23:19:25.912 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_co_bytes": 3128846, 2024-04-21T23:19:25.912 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_co_items": 32219, 2024-04-21T23:19:25.913 INFO:teuthology.orchestra.run.smithi055.stdout: "unittest_1_bytes": 0, 2024-04-21T23:19:25.913 INFO:teuthology.orchestra.run.smithi055.stdout: "unittest_1_items": 0, 2024-04-21T23:19:25.913 INFO:teuthology.orchestra.run.smithi055.stdout: "unittest_2_bytes": 0, 2024-04-21T23:19:25.913 INFO:teuthology.orchestra.run.smithi055.stdout: "unittest_2_items": 0 2024-04-21T23:19:25.913 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.913 INFO:teuthology.orchestra.run.smithi055.stdout: "objecter": { 2024-04-21T23:19:25.914 INFO:teuthology.orchestra.run.smithi055.stdout: "op_active": 0, 2024-04-21T23:19:25.914 INFO:teuthology.orchestra.run.smithi055.stdout: "op_laggy": 0, 2024-04-21T23:19:25.914 INFO:teuthology.orchestra.run.smithi055.stdout: "op_send": 155912, 2024-04-21T23:19:25.914 INFO:teuthology.orchestra.run.smithi055.stdout: "op_send_bytes": 512440547, 2024-04-21T23:19:25.914 INFO:teuthology.orchestra.run.smithi055.stdout: "op_resend": 37, 2024-04-21T23:19:25.914 INFO:teuthology.orchestra.run.smithi055.stdout: "op_reply": 155912, 2024-04-21T23:19:25.914 INFO:teuthology.orchestra.run.smithi055.stdout: "op_latency": { 2024-04-21T23:19:25.915 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 155912, 2024-04-21T23:19:25.915 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 5222.321136759, 2024-04-21T23:19:25.915 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.033495312 2024-04-21T23:19:25.915 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.915 INFO:teuthology.orchestra.run.smithi055.stdout: "op_inflight": 0, 2024-04-21T23:19:25.915 INFO:teuthology.orchestra.run.smithi055.stdout: "oplen_avg": { 2024-04-21T23:19:25.915 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 155912, 2024-04-21T23:19:25.915 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 207354 2024-04-21T23:19:25.916 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.916 INFO:teuthology.orchestra.run.smithi055.stdout: "op": 155912, 2024-04-21T23:19:25.916 INFO:teuthology.orchestra.run.smithi055.stdout: "op_r": 2565, 2024-04-21T23:19:25.916 INFO:teuthology.orchestra.run.smithi055.stdout: "op_w": 153347, 2024-04-21T23:19:25.916 INFO:teuthology.orchestra.run.smithi055.stdout: "op_rmw": 0, 2024-04-21T23:19:25.916 INFO:teuthology.orchestra.run.smithi055.stdout: "op_pg": 0, 2024-04-21T23:19:25.916 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_stat": 2054, 2024-04-21T23:19:25.916 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_create": 30038, 2024-04-21T23:19:25.916 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_read": 877, 2024-04-21T23:19:25.917 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_write": 52889, 2024-04-21T23:19:25.917 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_writefull": 530, 2024-04-21T23:19:25.917 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_writesame": 0, 2024-04-21T23:19:25.917 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_append": 0, 2024-04-21T23:19:25.917 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_zero": 0, 2024-04-21T23:19:25.917 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_truncate": 17, 2024-04-21T23:19:25.917 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_delete": 67131, 2024-04-21T23:19:25.917 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_mapext": 0, 2024-04-21T23:19:25.918 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_sparse_read": 0, 2024-04-21T23:19:25.918 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_clonerange": 0, 2024-04-21T23:19:25.918 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_getxattr": 1688, 2024-04-21T23:19:25.918 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_setxattr": 45252, 2024-04-21T23:19:25.918 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_cmpxattr": 0, 2024-04-21T23:19:25.918 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_rmxattr": 0, 2024-04-21T23:19:25.918 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_resetxattrs": 0, 2024-04-21T23:19:25.918 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_call": 0, 2024-04-21T23:19:25.919 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_watch": 0, 2024-04-21T23:19:25.919 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_notify": 0, 2024-04-21T23:19:25.919 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T23:19:25.919 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_pgls": 0, 2024-04-21T23:19:25.919 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_pgls_filter": 0, 2024-04-21T23:19:25.919 INFO:teuthology.orchestra.run.smithi055.stdout: "osdop_other": 0, 2024-04-21T23:19:25.919 INFO:teuthology.orchestra.run.smithi055.stdout: "linger_active": 0, 2024-04-21T23:19:25.919 INFO:teuthology.orchestra.run.smithi055.stdout: "linger_send": 0, 2024-04-21T23:19:25.920 INFO:teuthology.orchestra.run.smithi055.stdout: "linger_resend": 0, 2024-04-21T23:19:25.920 INFO:teuthology.orchestra.run.smithi055.stdout: "linger_ping": 0, 2024-04-21T23:19:25.920 INFO:teuthology.orchestra.run.smithi055.stdout: "poolop_active": 0, 2024-04-21T23:19:25.920 INFO:teuthology.orchestra.run.smithi055.stdout: "poolop_send": 0, 2024-04-21T23:19:25.920 INFO:teuthology.orchestra.run.smithi055.stdout: "poolop_resend": 0, 2024-04-21T23:19:25.921 INFO:teuthology.orchestra.run.smithi055.stdout: "poolstat_active": 0, 2024-04-21T23:19:25.921 INFO:teuthology.orchestra.run.smithi055.stdout: "poolstat_send": 0, 2024-04-21T23:19:25.921 INFO:teuthology.orchestra.run.smithi055.stdout: "poolstat_resend": 0, 2024-04-21T23:19:25.921 INFO:teuthology.orchestra.run.smithi055.stdout: "statfs_active": 0, 2024-04-21T23:19:25.921 INFO:teuthology.orchestra.run.smithi055.stdout: "statfs_send": 0, 2024-04-21T23:19:25.921 INFO:teuthology.orchestra.run.smithi055.stdout: "statfs_resend": 0, 2024-04-21T23:19:25.921 INFO:teuthology.orchestra.run.smithi055.stdout: "command_active": 0, 2024-04-21T23:19:25.922 INFO:teuthology.orchestra.run.smithi055.stdout: "command_send": 0, 2024-04-21T23:19:25.922 INFO:teuthology.orchestra.run.smithi055.stdout: "command_resend": 0, 2024-04-21T23:19:25.922 INFO:teuthology.orchestra.run.smithi055.stdout: "map_epoch": 110, 2024-04-21T23:19:25.922 INFO:teuthology.orchestra.run.smithi055.stdout: "map_full": 0, 2024-04-21T23:19:25.922 INFO:teuthology.orchestra.run.smithi055.stdout: "map_inc": 25, 2024-04-21T23:19:25.922 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_sessions": 12, 2024-04-21T23:19:25.922 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_session_open": 12, 2024-04-21T23:19:25.923 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_session_close": 0, 2024-04-21T23:19:25.923 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_laggy": 0, 2024-04-21T23:19:25.923 INFO:teuthology.orchestra.run.smithi055.stdout: "omap_wr": 5002, 2024-04-21T23:19:25.923 INFO:teuthology.orchestra.run.smithi055.stdout: "omap_rd": 0, 2024-04-21T23:19:25.923 INFO:teuthology.orchestra.run.smithi055.stdout: "omap_del": 1876 2024-04-21T23:19:25.923 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.924 INFO:teuthology.orchestra.run.smithi055.stdout: "oft": { 2024-04-21T23:19:25.924 INFO:teuthology.orchestra.run.smithi055.stdout: "omap_total_objs": 1, 2024-04-21T23:19:25.924 INFO:teuthology.orchestra.run.smithi055.stdout: "omap_total_kv_pairs": 0, 2024-04-21T23:19:25.924 INFO:teuthology.orchestra.run.smithi055.stdout: "omap_total_updates": 0, 2024-04-21T23:19:25.924 INFO:teuthology.orchestra.run.smithi055.stdout: "omap_total_removes": 0 2024-04-21T23:19:25.924 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.924 INFO:teuthology.orchestra.run.smithi055.stdout: "osd-slow-ops": { 2024-04-21T23:19:25.925 INFO:teuthology.orchestra.run.smithi055.stdout: "slow_ops_count": 0 2024-04-21T23:19:25.925 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.925 INFO:teuthology.orchestra.run.smithi055.stdout: "purge_queue": { 2024-04-21T23:19:25.925 INFO:teuthology.orchestra.run.smithi055.stdout: "pq_executing_ops": 0, 2024-04-21T23:19:25.925 INFO:teuthology.orchestra.run.smithi055.stdout: "pq_executing_ops_high_water": 48, 2024-04-21T23:19:25.925 INFO:teuthology.orchestra.run.smithi055.stdout: "pq_executing": 0, 2024-04-21T23:19:25.925 INFO:teuthology.orchestra.run.smithi055.stdout: "pq_executing_high_water": 24, 2024-04-21T23:19:25.925 INFO:teuthology.orchestra.run.smithi055.stdout: "pq_executed_ops": 66266, 2024-04-21T23:19:25.926 INFO:teuthology.orchestra.run.smithi055.stdout: "pq_executed": 33133, 2024-04-21T23:19:25.926 INFO:teuthology.orchestra.run.smithi055.stdout: "pq_item_in_journal": 0 2024-04-21T23:19:25.926 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.926 INFO:teuthology.orchestra.run.smithi055.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:19:25.926 INFO:teuthology.orchestra.run.smithi055.stdout: "val": 0, 2024-04-21T23:19:25.926 INFO:teuthology.orchestra.run.smithi055.stdout: "max": 104857600, 2024-04-21T23:19:25.926 INFO:teuthology.orchestra.run.smithi055.stdout: "get_started": 0, 2024-04-21T23:19:25.926 INFO:teuthology.orchestra.run.smithi055.stdout: "get": 443215, 2024-04-21T23:19:25.927 INFO:teuthology.orchestra.run.smithi055.stdout: "get_sum": 93980471, 2024-04-21T23:19:25.927 INFO:teuthology.orchestra.run.smithi055.stdout: "get_or_fail_fail": 0, 2024-04-21T23:19:25.927 INFO:teuthology.orchestra.run.smithi055.stdout: "get_or_fail_success": 443215, 2024-04-21T23:19:25.927 INFO:teuthology.orchestra.run.smithi055.stdout: "take": 0, 2024-04-21T23:19:25.927 INFO:teuthology.orchestra.run.smithi055.stdout: "take_sum": 0, 2024-04-21T23:19:25.927 INFO:teuthology.orchestra.run.smithi055.stdout: "put": 443215, 2024-04-21T23:19:25.927 INFO:teuthology.orchestra.run.smithi055.stdout: "put_sum": 93980471, 2024-04-21T23:19:25.927 INFO:teuthology.orchestra.run.smithi055.stdout: "wait": { 2024-04-21T23:19:25.927 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:25.928 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:25.928 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:25.928 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-04-21T23:19:25.928 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.928 INFO:teuthology.orchestra.run.smithi055.stdout: "throttle-objecter_bytes": { 2024-04-21T23:19:25.928 INFO:teuthology.orchestra.run.smithi055.stdout: "val": 0, 2024-04-21T23:19:25.928 INFO:teuthology.orchestra.run.smithi055.stdout: "max": 104857600, 2024-04-21T23:19:25.929 INFO:teuthology.orchestra.run.smithi055.stdout: "get_started": 0, 2024-04-21T23:19:25.929 INFO:teuthology.orchestra.run.smithi055.stdout: "get": 0, 2024-04-21T23:19:25.929 INFO:teuthology.orchestra.run.smithi055.stdout: "get_sum": 0, 2024-04-21T23:19:25.929 INFO:teuthology.orchestra.run.smithi055.stdout: "get_or_fail_fail": 0, 2024-04-21T23:19:25.929 INFO:teuthology.orchestra.run.smithi055.stdout: "get_or_fail_success": 0, 2024-04-21T23:19:25.929 INFO:teuthology.orchestra.run.smithi055.stdout: "take": 155912, 2024-04-21T23:19:25.929 INFO:teuthology.orchestra.run.smithi055.stdout: "take_sum": 516055532, 2024-04-21T23:19:25.929 INFO:teuthology.orchestra.run.smithi055.stdout: "put": 155912, 2024-04-21T23:19:25.929 INFO:teuthology.orchestra.run.smithi055.stdout: "put_sum": 516055532, 2024-04-21T23:19:25.930 INFO:teuthology.orchestra.run.smithi055.stdout: "wait": { 2024-04-21T23:19:25.930 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:25.930 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:25.930 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:25.930 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-04-21T23:19:25.930 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.930 INFO:teuthology.orchestra.run.smithi055.stdout: "throttle-objecter_ops": { 2024-04-21T23:19:25.930 INFO:teuthology.orchestra.run.smithi055.stdout: "val": 0, 2024-04-21T23:19:25.931 INFO:teuthology.orchestra.run.smithi055.stdout: "max": 1024, 2024-04-21T23:19:25.931 INFO:teuthology.orchestra.run.smithi055.stdout: "get_started": 0, 2024-04-21T23:19:25.931 INFO:teuthology.orchestra.run.smithi055.stdout: "get": 0, 2024-04-21T23:19:25.931 INFO:teuthology.orchestra.run.smithi055.stdout: "get_sum": 0, 2024-04-21T23:19:25.931 INFO:teuthology.orchestra.run.smithi055.stdout: "get_or_fail_fail": 0, 2024-04-21T23:19:25.931 INFO:teuthology.orchestra.run.smithi055.stdout: "get_or_fail_success": 0, 2024-04-21T23:19:25.931 INFO:teuthology.orchestra.run.smithi055.stdout: "take": 155912, 2024-04-21T23:19:25.931 INFO:teuthology.orchestra.run.smithi055.stdout: "take_sum": 155912, 2024-04-21T23:19:25.932 INFO:teuthology.orchestra.run.smithi055.stdout: "put": 155912, 2024-04-21T23:19:25.932 INFO:teuthology.orchestra.run.smithi055.stdout: "put_sum": 155912, 2024-04-21T23:19:25.932 INFO:teuthology.orchestra.run.smithi055.stdout: "wait": { 2024-04-21T23:19:25.932 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:25.932 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:25.932 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:25.932 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-04-21T23:19:25.933 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.933 INFO:teuthology.orchestra.run.smithi055.stdout: "throttle-write_buf_throttle": { 2024-04-21T23:19:25.933 INFO:teuthology.orchestra.run.smithi055.stdout: "val": 0, 2024-04-21T23:19:25.933 INFO:teuthology.orchestra.run.smithi055.stdout: "max": 3758096384, 2024-04-21T23:19:25.933 INFO:teuthology.orchestra.run.smithi055.stdout: "get_started": 0, 2024-04-21T23:19:25.933 INFO:teuthology.orchestra.run.smithi055.stdout: "get": 33133, 2024-04-21T23:19:25.933 INFO:teuthology.orchestra.run.smithi055.stdout: "get_sum": 3614985, 2024-04-21T23:19:25.933 INFO:teuthology.orchestra.run.smithi055.stdout: "get_or_fail_fail": 0, 2024-04-21T23:19:25.933 INFO:teuthology.orchestra.run.smithi055.stdout: "get_or_fail_success": 33133, 2024-04-21T23:19:25.934 INFO:teuthology.orchestra.run.smithi055.stdout: "take": 0, 2024-04-21T23:19:25.934 INFO:teuthology.orchestra.run.smithi055.stdout: "take_sum": 0, 2024-04-21T23:19:25.934 INFO:teuthology.orchestra.run.smithi055.stdout: "put": 857, 2024-04-21T23:19:25.934 INFO:teuthology.orchestra.run.smithi055.stdout: "put_sum": 3614985, 2024-04-21T23:19:25.934 INFO:teuthology.orchestra.run.smithi055.stdout: "wait": { 2024-04-21T23:19:25.934 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:25.934 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:25.934 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:25.935 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-04-21T23:19:25.935 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:25.935 INFO:teuthology.orchestra.run.smithi055.stdout: "throttle-write_buf_throttle-0x56041e774460": { 2024-04-21T23:19:25.935 INFO:teuthology.orchestra.run.smithi055.stdout: "val": 0, 2024-04-21T23:19:25.935 INFO:teuthology.orchestra.run.smithi055.stdout: "max": 3758096384, 2024-04-21T23:19:25.935 INFO:teuthology.orchestra.run.smithi055.stdout: "get_started": 0, 2024-04-21T23:19:25.935 INFO:teuthology.orchestra.run.smithi055.stdout: "get": 150841, 2024-04-21T23:19:25.935 INFO:teuthology.orchestra.run.smithi055.stdout: "get_sum": 485773263, 2024-04-21T23:19:25.935 INFO:teuthology.orchestra.run.smithi055.stdout: "get_or_fail_fail": 0, 2024-04-21T23:19:25.936 INFO:teuthology.orchestra.run.smithi055.stdout: "get_or_fail_success": 150841, 2024-04-21T23:19:25.936 INFO:teuthology.orchestra.run.smithi055.stdout: "take": 0, 2024-04-21T23:19:25.936 INFO:teuthology.orchestra.run.smithi055.stdout: "take_sum": 0, 2024-04-21T23:19:25.936 INFO:teuthology.orchestra.run.smithi055.stdout: "put": 52032, 2024-04-21T23:19:25.936 INFO:teuthology.orchestra.run.smithi055.stdout: "put_sum": 485773263, 2024-04-21T23:19:25.936 INFO:teuthology.orchestra.run.smithi055.stdout: "wait": { 2024-04-21T23:19:25.936 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:25.937 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:25.937 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:25.937 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-04-21T23:19:25.937 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-04-21T23:19:25.937 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-04-21T23:19:26.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:26 smithi055 ceph-mon[31117]: pgmap v335: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-04-21T23:19:26.564 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-21T23:19:26.564 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=11 2024-04-21T23:19:26.564 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=5 2024-04-21T23:19:26.564 DEBUG:tasks.check_counter:Getting stats from e 2024-04-21T23:19:26.565 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-21T23:19:26.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:26 smithi044 ceph-mon[27934]: pgmap v335: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-04-21T23:19:26.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:26 smithi069 ceph-mon[31816]: pgmap v335: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-04-21T23:19:26.817 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:19:28.331 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:28 smithi055 ceph-mon[31117]: pgmap v336: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-04-21T23:19:28.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:28 smithi044 ceph-mon[27934]: pgmap v336: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-04-21T23:19:28.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:28 smithi069 ceph-mon[31816]: pgmap v336: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-04-21T23:19:28.725 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-04-21T23:19:28.725 INFO:teuthology.orchestra.run.smithi055.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:19:28.725 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_messages": 231, 2024-04-21T23:19:28.725 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages": 219, 2024-04-21T23:19:28.725 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_bytes": 1447456, 2024-04-21T23:19:28.726 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_bytes": 133632, 2024-04-21T23:19:28.726 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_created_connections": 4, 2024-04-21T23:19:28.726 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_active_connections": 0, 2024-04-21T23:19:28.726 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_total_time": 0.043417298, 2024-04-21T23:19:28.726 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_send_time": 0.011648415, 2024-04-21T23:19:28.726 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_recv_time": 0.034798214, 2024-04-21T23:19:28.726 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_fast_dispatch_time": 0.005025757, 2024-04-21T23:19:28.726 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:28.727 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 219, 2024-04-21T23:19:28.727 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.009311367, 2024-04-21T23:19:28.727 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000042517 2024-04-21T23:19:28.727 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:28.727 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:28.727 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:28.727 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:28.727 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:28.727 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:28.728 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_encrypted_bytes": 1447456, 2024-04-21T23:19:28.728 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_encrypted_bytes": 133632 2024-04-21T23:19:28.728 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:28.728 INFO:teuthology.orchestra.run.smithi055.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:19:28.728 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_messages": 2, 2024-04-21T23:19:28.728 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages": 166, 2024-04-21T23:19:28.728 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_bytes": 320, 2024-04-21T23:19:28.728 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_bytes": 184496, 2024-04-21T23:19:28.728 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_created_connections": 5, 2024-04-21T23:19:28.729 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_active_connections": 3, 2024-04-21T23:19:28.729 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_total_time": 0.009300600, 2024-04-21T23:19:28.729 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_send_time": 0.005800058, 2024-04-21T23:19:28.729 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_recv_time": 0.001594193, 2024-04-21T23:19:28.729 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:19:28.729 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:28.729 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 163, 2024-04-21T23:19:28.729 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.018289221, 2024-04-21T23:19:28.729 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000112203 2024-04-21T23:19:28.730 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:28.730 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:28.730 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:28.730 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:28.730 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:28.730 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:28.730 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T23:19:28.730 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_encrypted_bytes": 184496 2024-04-21T23:19:28.730 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:28.731 INFO:teuthology.orchestra.run.smithi055.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:19:28.731 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_messages": 0, 2024-04-21T23:19:28.731 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages": 2, 2024-04-21T23:19:28.731 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_bytes": 0, 2024-04-21T23:19:28.731 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_bytes": 0, 2024-04-21T23:19:28.731 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_created_connections": 2, 2024-04-21T23:19:28.731 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_active_connections": 1, 2024-04-21T23:19:28.731 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_total_time": 0.000308216, 2024-04-21T23:19:28.731 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T23:19:28.732 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_recv_time": 0.000238244, 2024-04-21T23:19:28.732 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:19:28.732 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:28.732 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:28.732 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:28.732 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:28.732 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:28.732 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:28.733 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:28.733 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:28.733 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:28.733 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:28.733 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T23:19:28.733 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T23:19:28.733 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:28.733 INFO:teuthology.orchestra.run.smithi055.stdout: "cct": { 2024-04-21T23:19:28.733 INFO:teuthology.orchestra.run.smithi055.stdout: "total_workers": 0, 2024-04-21T23:19:28.734 INFO:teuthology.orchestra.run.smithi055.stdout: "unhealthy_workers": 0 2024-04-21T23:19:28.734 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:28.734 INFO:teuthology.orchestra.run.smithi055.stdout: "mempool": { 2024-04-21T23:19:28.734 INFO:teuthology.orchestra.run.smithi055.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:19:28.734 INFO:teuthology.orchestra.run.smithi055.stdout: "bloom_filter_items": 0, 2024-04-21T23:19:28.734 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:19:28.734 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:19:28.734 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:19:28.734 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:19:28.735 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:19:28.735 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:19:28.735 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:19:28.735 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:19:28.735 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:19:28.735 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:19:28.735 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:19:28.735 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:19:28.735 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:19:28.736 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_extent_items": 0, 2024-04-21T23:19:28.736 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:19:28.736 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_blob_items": 0, 2024-04-21T23:19:28.736 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:19:28.736 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:19:28.736 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:19:28.736 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:19:28.736 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:19:28.737 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:19:28.737 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:19:28.737 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_txc_items": 0, 2024-04-21T23:19:28.737 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:19:28.737 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:19:28.737 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:19:28.737 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_writing_items": 0, 2024-04-21T23:19:28.737 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_bytes": 0, 2024-04-21T23:19:28.737 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_items": 0, 2024-04-21T23:19:28.738 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:19:28.738 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:19:28.738 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:19:28.738 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:19:28.738 INFO:teuthology.orchestra.run.smithi055.stdout: "buffer_anon_bytes": 149017, 2024-04-21T23:19:28.738 INFO:teuthology.orchestra.run.smithi055.stdout: "buffer_anon_items": 31, 2024-04-21T23:19:28.738 INFO:teuthology.orchestra.run.smithi055.stdout: "buffer_meta_bytes": 0, 2024-04-21T23:19:28.738 INFO:teuthology.orchestra.run.smithi055.stdout: "buffer_meta_items": 0, 2024-04-21T23:19:28.739 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_bytes": 0, 2024-04-21T23:19:28.739 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_items": 0, 2024-04-21T23:19:28.739 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:19:28.739 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mapbl_items": 0, 2024-04-21T23:19:28.739 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:19:28.739 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pglog_items": 0, 2024-04-21T23:19:28.739 INFO:teuthology.orchestra.run.smithi055.stdout: "osdmap_bytes": 0, 2024-04-21T23:19:28.739 INFO:teuthology.orchestra.run.smithi055.stdout: "osdmap_items": 0, 2024-04-21T23:19:28.739 INFO:teuthology.orchestra.run.smithi055.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:19:28.740 INFO:teuthology.orchestra.run.smithi055.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:19:28.740 INFO:teuthology.orchestra.run.smithi055.stdout: "pgmap_bytes": 0, 2024-04-21T23:19:28.740 INFO:teuthology.orchestra.run.smithi055.stdout: "pgmap_items": 0, 2024-04-21T23:19:28.740 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_co_bytes": 928, 2024-04-21T23:19:28.740 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_co_items": 2, 2024-04-21T23:19:28.740 INFO:teuthology.orchestra.run.smithi055.stdout: "unittest_1_bytes": 0, 2024-04-21T23:19:28.740 INFO:teuthology.orchestra.run.smithi055.stdout: "unittest_1_items": 0, 2024-04-21T23:19:28.741 INFO:teuthology.orchestra.run.smithi055.stdout: "unittest_2_bytes": 0, 2024-04-21T23:19:28.741 INFO:teuthology.orchestra.run.smithi055.stdout: "unittest_2_items": 0 2024-04-21T23:19:28.741 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:28.741 INFO:teuthology.orchestra.run.smithi055.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:19:28.741 INFO:teuthology.orchestra.run.smithi055.stdout: "val": 0, 2024-04-21T23:19:28.741 INFO:teuthology.orchestra.run.smithi055.stdout: "max": 104857600, 2024-04-21T23:19:28.741 INFO:teuthology.orchestra.run.smithi055.stdout: "get_started": 0, 2024-04-21T23:19:28.741 INFO:teuthology.orchestra.run.smithi055.stdout: "get": 229, 2024-04-21T23:19:28.741 INFO:teuthology.orchestra.run.smithi055.stdout: "get_sum": 1412450, 2024-04-21T23:19:28.742 INFO:teuthology.orchestra.run.smithi055.stdout: "get_or_fail_fail": 0, 2024-04-21T23:19:28.742 INFO:teuthology.orchestra.run.smithi055.stdout: "get_or_fail_success": 229, 2024-04-21T23:19:28.742 INFO:teuthology.orchestra.run.smithi055.stdout: "take": 0, 2024-04-21T23:19:28.742 INFO:teuthology.orchestra.run.smithi055.stdout: "take_sum": 0, 2024-04-21T23:19:28.742 INFO:teuthology.orchestra.run.smithi055.stdout: "put": 229, 2024-04-21T23:19:28.742 INFO:teuthology.orchestra.run.smithi055.stdout: "put_sum": 1412450, 2024-04-21T23:19:28.742 INFO:teuthology.orchestra.run.smithi055.stdout: "wait": { 2024-04-21T23:19:28.742 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:28.743 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:28.743 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:28.743 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-04-21T23:19:28.743 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-04-21T23:19:28.743 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-04-21T23:19:29.418 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2024-04-21T23:19:29.419 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2024-04-21T23:19:29.419 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2024-04-21T23:19:29.419 DEBUG:tasks.check_counter:Getting stats from h 2024-04-21T23:19:29.419 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-21T23:19:29.674 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:19:30.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:30 smithi055 ceph-mon[31117]: pgmap v337: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:19:30.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:30 smithi044 ceph-mon[27934]: pgmap v337: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:19:30.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:30 smithi069 ceph-mon[31816]: pgmap v337: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:19:31.480 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-04-21T23:19:31.480 INFO:teuthology.orchestra.run.smithi055.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:19:31.481 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_messages": 226, 2024-04-21T23:19:31.481 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages": 216, 2024-04-21T23:19:31.481 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_bytes": 1444192, 2024-04-21T23:19:31.481 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_bytes": 132544, 2024-04-21T23:19:31.481 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_created_connections": 4, 2024-04-21T23:19:31.481 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_active_connections": 0, 2024-04-21T23:19:31.481 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_total_time": 0.039727330, 2024-04-21T23:19:31.481 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_send_time": 0.011219793, 2024-04-21T23:19:31.481 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_recv_time": 0.030782837, 2024-04-21T23:19:31.482 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_fast_dispatch_time": 0.004784917, 2024-04-21T23:19:31.482 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:31.482 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 215, 2024-04-21T23:19:31.482 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.009027383, 2024-04-21T23:19:31.482 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000041987 2024-04-21T23:19:31.482 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:31.482 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:31.482 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:31.483 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:31.483 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:31.483 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:31.483 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_encrypted_bytes": 1444192, 2024-04-21T23:19:31.483 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_encrypted_bytes": 132544 2024-04-21T23:19:31.483 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:31.483 INFO:teuthology.orchestra.run.smithi055.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:19:31.483 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_messages": 6, 2024-04-21T23:19:31.483 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages": 166, 2024-04-21T23:19:31.484 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_bytes": 3312, 2024-04-21T23:19:31.484 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_bytes": 184592, 2024-04-21T23:19:31.484 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_created_connections": 5, 2024-04-21T23:19:31.484 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_active_connections": 3, 2024-04-21T23:19:31.484 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_total_time": 0.009951657, 2024-04-21T23:19:31.484 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_send_time": 0.006231256, 2024-04-21T23:19:31.484 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_recv_time": 0.001808180, 2024-04-21T23:19:31.484 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:19:31.484 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:31.485 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 164, 2024-04-21T23:19:31.485 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.019451130, 2024-04-21T23:19:31.485 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000118604 2024-04-21T23:19:31.485 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:31.485 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:31.485 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:31.485 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:31.485 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:31.485 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:31.486 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-21T23:19:31.486 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_encrypted_bytes": 184592 2024-04-21T23:19:31.486 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:31.486 INFO:teuthology.orchestra.run.smithi055.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:19:31.486 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_messages": 0, 2024-04-21T23:19:31.486 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages": 2, 2024-04-21T23:19:31.486 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_bytes": 0, 2024-04-21T23:19:31.486 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_bytes": 0, 2024-04-21T23:19:31.487 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_created_connections": 2, 2024-04-21T23:19:31.487 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_active_connections": 1, 2024-04-21T23:19:31.487 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_total_time": 0.000692842, 2024-04-21T23:19:31.487 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T23:19:31.487 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_recv_time": 0.000610960, 2024-04-21T23:19:31.487 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:19:31.487 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:31.487 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:31.487 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:31.488 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:31.488 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:31.488 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:31.488 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:31.488 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:31.488 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:31.488 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:31.488 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T23:19:31.488 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T23:19:31.489 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:31.489 INFO:teuthology.orchestra.run.smithi055.stdout: "cct": { 2024-04-21T23:19:31.489 INFO:teuthology.orchestra.run.smithi055.stdout: "total_workers": 0, 2024-04-21T23:19:31.489 INFO:teuthology.orchestra.run.smithi055.stdout: "unhealthy_workers": 0 2024-04-21T23:19:31.489 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:31.489 INFO:teuthology.orchestra.run.smithi055.stdout: "mempool": { 2024-04-21T23:19:31.489 INFO:teuthology.orchestra.run.smithi055.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:19:31.489 INFO:teuthology.orchestra.run.smithi055.stdout: "bloom_filter_items": 0, 2024-04-21T23:19:31.489 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:19:31.490 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:19:31.490 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:19:31.490 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:19:31.490 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:19:31.490 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:19:31.490 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:19:31.490 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:19:31.490 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:19:31.490 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:19:31.491 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:19:31.491 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:19:31.491 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:19:31.491 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_extent_items": 0, 2024-04-21T23:19:31.491 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:19:31.491 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_blob_items": 0, 2024-04-21T23:19:31.491 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:19:31.491 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:19:31.491 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:19:31.492 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:19:31.492 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:19:31.492 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:19:31.492 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:19:31.492 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_txc_items": 0, 2024-04-21T23:19:31.492 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:19:31.492 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:19:31.492 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:19:31.492 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_writing_items": 0, 2024-04-21T23:19:31.493 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_bytes": 0, 2024-04-21T23:19:31.493 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_items": 0, 2024-04-21T23:19:31.493 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:19:31.493 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:19:31.493 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:19:31.493 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:19:31.493 INFO:teuthology.orchestra.run.smithi055.stdout: "buffer_anon_bytes": 150008, 2024-04-21T23:19:31.493 INFO:teuthology.orchestra.run.smithi055.stdout: "buffer_anon_items": 31, 2024-04-21T23:19:31.493 INFO:teuthology.orchestra.run.smithi055.stdout: "buffer_meta_bytes": 0, 2024-04-21T23:19:31.494 INFO:teuthology.orchestra.run.smithi055.stdout: "buffer_meta_items": 0, 2024-04-21T23:19:31.494 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_bytes": 0, 2024-04-21T23:19:31.494 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_items": 0, 2024-04-21T23:19:31.494 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:19:31.494 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mapbl_items": 0, 2024-04-21T23:19:31.494 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:19:31.494 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pglog_items": 0, 2024-04-21T23:19:31.494 INFO:teuthology.orchestra.run.smithi055.stdout: "osdmap_bytes": 0, 2024-04-21T23:19:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: "osdmap_items": 0, 2024-04-21T23:19:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:19:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:19:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: "pgmap_bytes": 0, 2024-04-21T23:19:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: "pgmap_items": 0, 2024-04-21T23:19:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_co_bytes": 928, 2024-04-21T23:19:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_co_items": 2, 2024-04-21T23:19:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: "unittest_1_bytes": 0, 2024-04-21T23:19:31.495 INFO:teuthology.orchestra.run.smithi055.stdout: "unittest_1_items": 0, 2024-04-21T23:19:31.496 INFO:teuthology.orchestra.run.smithi055.stdout: "unittest_2_bytes": 0, 2024-04-21T23:19:31.496 INFO:teuthology.orchestra.run.smithi055.stdout: "unittest_2_items": 0 2024-04-21T23:19:31.496 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:31.496 INFO:teuthology.orchestra.run.smithi055.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:19:31.496 INFO:teuthology.orchestra.run.smithi055.stdout: "val": 0, 2024-04-21T23:19:31.496 INFO:teuthology.orchestra.run.smithi055.stdout: "max": 104857600, 2024-04-21T23:19:31.496 INFO:teuthology.orchestra.run.smithi055.stdout: "get_started": 0, 2024-04-21T23:19:31.497 INFO:teuthology.orchestra.run.smithi055.stdout: "get": 228, 2024-04-21T23:19:31.497 INFO:teuthology.orchestra.run.smithi055.stdout: "get_sum": 1412320, 2024-04-21T23:19:31.497 INFO:teuthology.orchestra.run.smithi055.stdout: "get_or_fail_fail": 0, 2024-04-21T23:19:31.497 INFO:teuthology.orchestra.run.smithi055.stdout: "get_or_fail_success": 228, 2024-04-21T23:19:31.497 INFO:teuthology.orchestra.run.smithi055.stdout: "take": 0, 2024-04-21T23:19:31.497 INFO:teuthology.orchestra.run.smithi055.stdout: "take_sum": 0, 2024-04-21T23:19:31.497 INFO:teuthology.orchestra.run.smithi055.stdout: "put": 228, 2024-04-21T23:19:31.497 INFO:teuthology.orchestra.run.smithi055.stdout: "put_sum": 1412320, 2024-04-21T23:19:31.498 INFO:teuthology.orchestra.run.smithi055.stdout: "wait": { 2024-04-21T23:19:31.498 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:31.498 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:31.498 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:31.498 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-04-21T23:19:31.498 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-04-21T23:19:31.498 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-04-21T23:19:32.161 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.h 2024-04-21T23:19:32.161 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.h 2024-04-21T23:19:32.162 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.h 2024-04-21T23:19:32.162 DEBUG:tasks.check_counter:Getting stats from k 2024-04-21T23:19:32.162 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-21T23:19:32.413 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.b/config 2024-04-21T23:19:32.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:32 smithi055 ceph-mon[31117]: pgmap v338: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:19:32.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:32 smithi044 ceph-mon[27934]: pgmap v338: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:19:32.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:32 smithi069 ceph-mon[31816]: pgmap v338: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:19:34.118 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2024-04-21T23:19:34.118 INFO:teuthology.orchestra.run.smithi055.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:19:34.118 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_messages": 0, 2024-04-21T23:19:34.118 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages": 2, 2024-04-21T23:19:34.118 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_bytes": 0, 2024-04-21T23:19:34.119 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_bytes": 0, 2024-04-21T23:19:34.119 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_created_connections": 4, 2024-04-21T23:19:34.119 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_active_connections": 0, 2024-04-21T23:19:34.119 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_total_time": 0.001208565, 2024-04-21T23:19:34.119 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T23:19:34.119 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_recv_time": 0.000263958, 2024-04-21T23:19:34.119 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:19:34.119 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:34.120 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:34.120 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:34.120 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:34.120 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:34.120 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:34.120 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:34.120 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:34.120 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:34.121 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:34.121 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T23:19:34.121 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T23:19:34.121 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:34.121 INFO:teuthology.orchestra.run.smithi055.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:19:34.121 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_messages": 2, 2024-04-21T23:19:34.121 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages": 168, 2024-04-21T23:19:34.121 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_bytes": 320, 2024-04-21T23:19:34.122 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_bytes": 184848, 2024-04-21T23:19:34.122 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_created_connections": 5, 2024-04-21T23:19:34.122 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_active_connections": 3, 2024-04-21T23:19:34.122 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_total_time": 0.010394255, 2024-04-21T23:19:34.122 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_send_time": 0.006500857, 2024-04-21T23:19:34.122 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_recv_time": 0.001879673, 2024-04-21T23:19:34.122 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:19:34.122 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:34.123 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 165, 2024-04-21T23:19:34.123 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.019547027, 2024-04-21T23:19:34.123 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000118466 2024-04-21T23:19:34.123 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:34.123 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:34.123 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:34.123 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:34.123 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:34.123 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:34.124 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T23:19:34.124 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_encrypted_bytes": 184848 2024-04-21T23:19:34.124 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:34.124 INFO:teuthology.orchestra.run.smithi055.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:19:34.124 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_messages": 235, 2024-04-21T23:19:34.124 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages": 223, 2024-04-21T23:19:34.124 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_bytes": 1448544, 2024-04-21T23:19:34.125 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_bytes": 136192, 2024-04-21T23:19:34.125 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_created_connections": 2, 2024-04-21T23:19:34.125 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_active_connections": 1, 2024-04-21T23:19:34.125 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_total_time": 0.041009791, 2024-04-21T23:19:34.125 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_send_time": 0.012064607, 2024-04-21T23:19:34.125 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_recv_time": 0.032746478, 2024-04-21T23:19:34.125 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_running_fast_dispatch_time": 0.004988643, 2024-04-21T23:19:34.125 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:34.126 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 223, 2024-04-21T23:19:34.126 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.009884115, 2024-04-21T23:19:34.126 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000044323 2024-04-21T23:19:34.126 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:34.126 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:34.126 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:34.126 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:34.126 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:34.126 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:34.127 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_recv_encrypted_bytes": 1448544, 2024-04-21T23:19:34.127 INFO:teuthology.orchestra.run.smithi055.stdout: "msgr_send_encrypted_bytes": 136192 2024-04-21T23:19:34.127 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:34.127 INFO:teuthology.orchestra.run.smithi055.stdout: "cct": { 2024-04-21T23:19:34.127 INFO:teuthology.orchestra.run.smithi055.stdout: "total_workers": 0, 2024-04-21T23:19:34.127 INFO:teuthology.orchestra.run.smithi055.stdout: "unhealthy_workers": 0 2024-04-21T23:19:34.127 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:34.127 INFO:teuthology.orchestra.run.smithi055.stdout: "mempool": { 2024-04-21T23:19:34.127 INFO:teuthology.orchestra.run.smithi055.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:19:34.128 INFO:teuthology.orchestra.run.smithi055.stdout: "bloom_filter_items": 0, 2024-04-21T23:19:34.128 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:19:34.128 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:19:34.128 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:19:34.128 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:19:34.128 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:19:34.128 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:19:34.128 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:19:34.128 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:19:34.129 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:19:34.129 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:19:34.129 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:19:34.129 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:19:34.129 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:19:34.129 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_extent_items": 0, 2024-04-21T23:19:34.129 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:19:34.129 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_blob_items": 0, 2024-04-21T23:19:34.129 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:19:34.130 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:19:34.130 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:19:34.130 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:19:34.130 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:19:34.130 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:19:34.130 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:19:34.130 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_txc_items": 0, 2024-04-21T23:19:34.130 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:19:34.130 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:19:34.131 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:19:34.131 INFO:teuthology.orchestra.run.smithi055.stdout: "bluestore_writing_items": 0, 2024-04-21T23:19:34.131 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_bytes": 0, 2024-04-21T23:19:34.131 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_items": 0, 2024-04-21T23:19:34.131 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:19:34.131 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:19:34.131 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:19:34.131 INFO:teuthology.orchestra.run.smithi055.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:19:34.131 INFO:teuthology.orchestra.run.smithi055.stdout: "buffer_anon_bytes": 149057, 2024-04-21T23:19:34.132 INFO:teuthology.orchestra.run.smithi055.stdout: "buffer_anon_items": 32, 2024-04-21T23:19:34.132 INFO:teuthology.orchestra.run.smithi055.stdout: "buffer_meta_bytes": 0, 2024-04-21T23:19:34.132 INFO:teuthology.orchestra.run.smithi055.stdout: "buffer_meta_items": 0, 2024-04-21T23:19:34.132 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_bytes": 0, 2024-04-21T23:19:34.132 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_items": 0, 2024-04-21T23:19:34.132 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:19:34.132 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_mapbl_items": 0, 2024-04-21T23:19:34.132 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:19:34.132 INFO:teuthology.orchestra.run.smithi055.stdout: "osd_pglog_items": 0, 2024-04-21T23:19:34.133 INFO:teuthology.orchestra.run.smithi055.stdout: "osdmap_bytes": 0, 2024-04-21T23:19:34.133 INFO:teuthology.orchestra.run.smithi055.stdout: "osdmap_items": 0, 2024-04-21T23:19:34.133 INFO:teuthology.orchestra.run.smithi055.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:19:34.133 INFO:teuthology.orchestra.run.smithi055.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:19:34.133 INFO:teuthology.orchestra.run.smithi055.stdout: "pgmap_bytes": 0, 2024-04-21T23:19:34.133 INFO:teuthology.orchestra.run.smithi055.stdout: "pgmap_items": 0, 2024-04-21T23:19:34.133 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_co_bytes": 928, 2024-04-21T23:19:34.133 INFO:teuthology.orchestra.run.smithi055.stdout: "mds_co_items": 2, 2024-04-21T23:19:34.133 INFO:teuthology.orchestra.run.smithi055.stdout: "unittest_1_bytes": 0, 2024-04-21T23:19:34.134 INFO:teuthology.orchestra.run.smithi055.stdout: "unittest_1_items": 0, 2024-04-21T23:19:34.134 INFO:teuthology.orchestra.run.smithi055.stdout: "unittest_2_bytes": 0, 2024-04-21T23:19:34.134 INFO:teuthology.orchestra.run.smithi055.stdout: "unittest_2_items": 0 2024-04-21T23:19:34.134 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2024-04-21T23:19:34.134 INFO:teuthology.orchestra.run.smithi055.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:19:34.134 INFO:teuthology.orchestra.run.smithi055.stdout: "val": 0, 2024-04-21T23:19:34.134 INFO:teuthology.orchestra.run.smithi055.stdout: "max": 104857600, 2024-04-21T23:19:34.134 INFO:teuthology.orchestra.run.smithi055.stdout: "get_started": 0, 2024-04-21T23:19:34.135 INFO:teuthology.orchestra.run.smithi055.stdout: "get": 233, 2024-04-21T23:19:34.135 INFO:teuthology.orchestra.run.smithi055.stdout: "get_sum": 1412970, 2024-04-21T23:19:34.135 INFO:teuthology.orchestra.run.smithi055.stdout: "get_or_fail_fail": 0, 2024-04-21T23:19:34.135 INFO:teuthology.orchestra.run.smithi055.stdout: "get_or_fail_success": 233, 2024-04-21T23:19:34.135 INFO:teuthology.orchestra.run.smithi055.stdout: "take": 0, 2024-04-21T23:19:34.135 INFO:teuthology.orchestra.run.smithi055.stdout: "take_sum": 0, 2024-04-21T23:19:34.135 INFO:teuthology.orchestra.run.smithi055.stdout: "put": 233, 2024-04-21T23:19:34.135 INFO:teuthology.orchestra.run.smithi055.stdout: "put_sum": 1412970, 2024-04-21T23:19:34.135 INFO:teuthology.orchestra.run.smithi055.stdout: "wait": { 2024-04-21T23:19:34.136 INFO:teuthology.orchestra.run.smithi055.stdout: "avgcount": 0, 2024-04-21T23:19:34.136 INFO:teuthology.orchestra.run.smithi055.stdout: "sum": 0.000000000, 2024-04-21T23:19:34.136 INFO:teuthology.orchestra.run.smithi055.stdout: "avgtime": 0.000000000 2024-04-21T23:19:34.136 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-04-21T23:19:34.136 INFO:teuthology.orchestra.run.smithi055.stdout: } 2024-04-21T23:19:34.136 INFO:teuthology.orchestra.run.smithi055.stdout:} 2024-04-21T23:19:34.464 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:34 smithi055 ceph-mon[31117]: pgmap v339: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:19:34.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:34 smithi044 ceph-mon[27934]: pgmap v339: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:19:34.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:34 smithi069 ceph-mon[31816]: pgmap v339: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:19:34.846 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-21T23:19:34.846 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-21T23:19:34.846 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-21T23:19:34.846 DEBUG:tasks.check_counter:Getting stats from c 2024-04-21T23:19:34.846 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-21T23:19:35.093 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:19:36.500 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:36 smithi069 ceph-mon[31816]: pgmap v340: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 886 B/s rd, 1 op/s 2024-04-21T23:19:36.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:36 smithi044 ceph-mon[27934]: pgmap v340: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 886 B/s rd, 1 op/s 2024-04-21T23:19:36.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:36 smithi055 ceph-mon[31117]: pgmap v340: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 886 B/s rd, 1 op/s 2024-04-21T23:19:36.908 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-21T23:19:36.908 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:19:36.909 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 4, 2024-04-21T23:19:36.909 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 4, 2024-04-21T23:19:36.909 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 2992, 2024-04-21T23:19:36.909 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 448, 2024-04-21T23:19:36.909 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 4, 2024-04-21T23:19:36.909 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 0, 2024-04-21T23:19:36.909 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.001418192, 2024-04-21T23:19:36.910 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.000032834, 2024-04-21T23:19:36.910 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.000594798, 2024-04-21T23:19:36.910 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:19:36.910 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:36.910 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 3, 2024-04-21T23:19:36.910 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000803201, 2024-04-21T23:19:36.910 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000267733 2024-04-21T23:19:36.910 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:36.911 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:36.911 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:36.911 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:36.911 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:36.911 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:36.911 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-21T23:19:36.911 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T23:19:36.912 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:36.912 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:19:36.912 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 2, 2024-04-21T23:19:36.912 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 167, 2024-04-21T23:19:36.912 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 320, 2024-04-21T23:19:36.912 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 184672, 2024-04-21T23:19:36.912 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 5, 2024-04-21T23:19:36.912 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 3, 2024-04-21T23:19:36.913 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.010045914, 2024-04-21T23:19:36.913 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.006626481, 2024-04-21T23:19:36.913 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.001424308, 2024-04-21T23:19:36.913 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:19:36.913 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:36.913 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 164, 2024-04-21T23:19:36.913 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.013074646, 2024-04-21T23:19:36.913 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000079723 2024-04-21T23:19:36.914 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:36.914 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:36.914 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:36.914 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:36.914 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:36.914 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:36.914 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T23:19:36.915 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 184672 2024-04-21T23:19:36.915 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:36.915 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:19:36.915 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 228, 2024-04-21T23:19:36.915 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 218, 2024-04-21T23:19:36.915 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 1444736, 2024-04-21T23:19:36.915 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 133824, 2024-04-21T23:19:36.915 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 2, 2024-04-21T23:19:36.916 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 1, 2024-04-21T23:19:36.916 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.040485957, 2024-04-21T23:19:36.916 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.012435375, 2024-04-21T23:19:36.916 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.031235531, 2024-04-21T23:19:36.916 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.004816050, 2024-04-21T23:19:36.916 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:36.916 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 217, 2024-04-21T23:19:36.917 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.009321907, 2024-04-21T23:19:36.917 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000042958 2024-04-21T23:19:36.917 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:36.917 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:36.917 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:36.917 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:36.918 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:36.918 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:36.918 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 1444736, 2024-04-21T23:19:36.918 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 133824 2024-04-21T23:19:36.918 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:36.918 INFO:teuthology.orchestra.run.smithi069.stdout: "cct": { 2024-04-21T23:19:36.918 INFO:teuthology.orchestra.run.smithi069.stdout: "total_workers": 0, 2024-04-21T23:19:36.919 INFO:teuthology.orchestra.run.smithi069.stdout: "unhealthy_workers": 0 2024-04-21T23:19:36.919 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:36.919 INFO:teuthology.orchestra.run.smithi069.stdout: "mempool": { 2024-04-21T23:19:36.919 INFO:teuthology.orchestra.run.smithi069.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:19:36.919 INFO:teuthology.orchestra.run.smithi069.stdout: "bloom_filter_items": 0, 2024-04-21T23:19:36.919 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:19:36.919 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:19:36.919 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:19:36.920 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:19:36.920 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:19:36.920 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:19:36.920 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:19:36.920 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:19:36.920 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:19:36.920 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:19:36.920 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:19:36.921 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:19:36.921 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:19:36.921 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_items": 0, 2024-04-21T23:19:36.921 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:19:36.921 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_blob_items": 0, 2024-04-21T23:19:36.921 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:19:36.921 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:19:36.921 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:19:36.922 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:19:36.922 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:19:36.922 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:19:36.922 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:19:36.922 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_txc_items": 0, 2024-04-21T23:19:36.922 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:19:36.922 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:19:36.923 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:19:36.923 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_items": 0, 2024-04-21T23:19:36.923 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_bytes": 0, 2024-04-21T23:19:36.923 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_items": 0, 2024-04-21T23:19:36.923 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:19:36.923 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:19:36.923 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:19:36.923 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:19:36.924 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_anon_bytes": 149017, 2024-04-21T23:19:36.924 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_anon_items": 31, 2024-04-21T23:19:36.924 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_meta_bytes": 0, 2024-04-21T23:19:36.924 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_meta_items": 0, 2024-04-21T23:19:36.924 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_bytes": 0, 2024-04-21T23:19:36.924 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_items": 0, 2024-04-21T23:19:36.924 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:19:36.924 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mapbl_items": 0, 2024-04-21T23:19:36.925 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:19:36.925 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pglog_items": 0, 2024-04-21T23:19:36.925 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_bytes": 0, 2024-04-21T23:19:36.925 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_items": 0, 2024-04-21T23:19:36.925 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:19:36.925 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:19:36.925 INFO:teuthology.orchestra.run.smithi069.stdout: "pgmap_bytes": 0, 2024-04-21T23:19:36.925 INFO:teuthology.orchestra.run.smithi069.stdout: "pgmap_items": 0, 2024-04-21T23:19:36.926 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_co_bytes": 928, 2024-04-21T23:19:36.926 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_co_items": 2, 2024-04-21T23:19:36.926 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_1_bytes": 0, 2024-04-21T23:19:36.926 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_1_items": 0, 2024-04-21T23:19:36.926 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_2_bytes": 0, 2024-04-21T23:19:36.926 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_2_items": 0 2024-04-21T23:19:36.926 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:36.926 INFO:teuthology.orchestra.run.smithi069.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:19:36.927 INFO:teuthology.orchestra.run.smithi069.stdout: "val": 0, 2024-04-21T23:19:36.927 INFO:teuthology.orchestra.run.smithi069.stdout: "max": 104857600, 2024-04-21T23:19:36.927 INFO:teuthology.orchestra.run.smithi069.stdout: "get_started": 0, 2024-04-21T23:19:36.927 INFO:teuthology.orchestra.run.smithi069.stdout: "get": 230, 2024-04-21T23:19:36.927 INFO:teuthology.orchestra.run.smithi069.stdout: "get_sum": 1412580, 2024-04-21T23:19:36.927 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_fail": 0, 2024-04-21T23:19:36.927 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_success": 230, 2024-04-21T23:19:36.927 INFO:teuthology.orchestra.run.smithi069.stdout: "take": 0, 2024-04-21T23:19:36.928 INFO:teuthology.orchestra.run.smithi069.stdout: "take_sum": 0, 2024-04-21T23:19:36.928 INFO:teuthology.orchestra.run.smithi069.stdout: "put": 230, 2024-04-21T23:19:36.928 INFO:teuthology.orchestra.run.smithi069.stdout: "put_sum": 1412580, 2024-04-21T23:19:36.928 INFO:teuthology.orchestra.run.smithi069.stdout: "wait": { 2024-04-21T23:19:36.928 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:36.928 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:36.928 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:36.928 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-21T23:19:36.929 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-21T23:19:36.929 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-21T23:19:37.604 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.c 2024-04-21T23:19:37.604 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.c 2024-04-21T23:19:37.604 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.c 2024-04-21T23:19:37.604 DEBUG:tasks.check_counter:Getting stats from f 2024-04-21T23:19:37.605 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-21T23:19:37.851 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:19:38.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:38 smithi069 ceph-mon[31816]: pgmap v341: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:19:38.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:38 smithi044 ceph-mon[27934]: pgmap v341: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:19:38.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:38 smithi055 ceph-mon[31117]: pgmap v341: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:19:39.588 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-21T23:19:39.588 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:19:39.588 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 4, 2024-04-21T23:19:39.588 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 4, 2024-04-21T23:19:39.589 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 2992, 2024-04-21T23:19:39.589 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 448, 2024-04-21T23:19:39.589 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 4, 2024-04-21T23:19:39.589 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 0, 2024-04-21T23:19:39.589 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.001468131, 2024-04-21T23:19:39.589 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.000034649, 2024-04-21T23:19:39.589 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.000677435, 2024-04-21T23:19:39.589 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:19:39.590 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:39.590 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 3, 2024-04-21T23:19:39.590 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000736357, 2024-04-21T23:19:39.590 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000245452 2024-04-21T23:19:39.590 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:39.590 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:39.590 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:39.590 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:39.591 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:39.591 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:39.591 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-21T23:19:39.591 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T23:19:39.591 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:39.591 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:19:39.591 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 229, 2024-04-21T23:19:39.591 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 219, 2024-04-21T23:19:39.591 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 1445008, 2024-04-21T23:19:39.591 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 134464, 2024-04-21T23:19:39.592 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 2, 2024-04-21T23:19:39.592 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 1, 2024-04-21T23:19:39.592 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.039552045, 2024-04-21T23:19:39.592 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.012150076, 2024-04-21T23:19:39.592 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.030423327, 2024-04-21T23:19:39.592 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.004874922, 2024-04-21T23:19:39.592 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:39.592 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 218, 2024-04-21T23:19:39.593 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.010315296, 2024-04-21T23:19:39.593 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000047317 2024-04-21T23:19:39.593 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:39.593 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:39.593 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:39.593 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:39.593 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:39.593 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:39.593 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 1445008, 2024-04-21T23:19:39.593 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 134464 2024-04-21T23:19:39.594 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:39.594 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:19:39.594 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 2, 2024-04-21T23:19:39.594 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 167, 2024-04-21T23:19:39.594 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 320, 2024-04-21T23:19:39.594 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 184672, 2024-04-21T23:19:39.594 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 5, 2024-04-21T23:19:39.594 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 3, 2024-04-21T23:19:39.595 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.009523863, 2024-04-21T23:19:39.595 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.006128652, 2024-04-21T23:19:39.595 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.001570729, 2024-04-21T23:19:39.595 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:19:39.595 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:39.595 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 164, 2024-04-21T23:19:39.595 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.014596241, 2024-04-21T23:19:39.595 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000089001 2024-04-21T23:19:39.595 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:39.596 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:39.596 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:39.596 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:39.596 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:39.596 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:39.596 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T23:19:39.596 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 184672 2024-04-21T23:19:39.596 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:39.596 INFO:teuthology.orchestra.run.smithi069.stdout: "cct": { 2024-04-21T23:19:39.597 INFO:teuthology.orchestra.run.smithi069.stdout: "total_workers": 0, 2024-04-21T23:19:39.597 INFO:teuthology.orchestra.run.smithi069.stdout: "unhealthy_workers": 0 2024-04-21T23:19:39.597 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:39.597 INFO:teuthology.orchestra.run.smithi069.stdout: "mempool": { 2024-04-21T23:19:39.597 INFO:teuthology.orchestra.run.smithi069.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:19:39.597 INFO:teuthology.orchestra.run.smithi069.stdout: "bloom_filter_items": 0, 2024-04-21T23:19:39.597 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:19:39.597 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:19:39.597 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:19:39.597 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:19:39.598 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:19:39.598 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:19:39.598 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:19:39.598 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:19:39.598 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:19:39.598 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:19:39.598 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:19:39.598 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:19:39.599 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:19:39.599 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_items": 0, 2024-04-21T23:19:39.599 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:19:39.599 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_blob_items": 0, 2024-04-21T23:19:39.599 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:19:39.599 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:19:39.599 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:19:39.599 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:19:39.600 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:19:39.600 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:19:39.600 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:19:39.600 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_txc_items": 0, 2024-04-21T23:19:39.600 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:19:39.600 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:19:39.600 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:19:39.600 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_items": 0, 2024-04-21T23:19:39.601 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_bytes": 0, 2024-04-21T23:19:39.601 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_items": 0, 2024-04-21T23:19:39.601 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:19:39.601 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:19:39.601 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:19:39.601 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:19:39.601 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_anon_bytes": 149017, 2024-04-21T23:19:39.601 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_anon_items": 31, 2024-04-21T23:19:39.601 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_meta_bytes": 0, 2024-04-21T23:19:39.602 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_meta_items": 0, 2024-04-21T23:19:39.602 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_bytes": 0, 2024-04-21T23:19:39.602 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_items": 0, 2024-04-21T23:19:39.602 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:19:39.602 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mapbl_items": 0, 2024-04-21T23:19:39.602 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:19:39.602 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pglog_items": 0, 2024-04-21T23:19:39.602 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_bytes": 0, 2024-04-21T23:19:39.602 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_items": 0, 2024-04-21T23:19:39.603 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:19:39.603 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:19:39.603 INFO:teuthology.orchestra.run.smithi069.stdout: "pgmap_bytes": 0, 2024-04-21T23:19:39.603 INFO:teuthology.orchestra.run.smithi069.stdout: "pgmap_items": 0, 2024-04-21T23:19:39.603 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_co_bytes": 928, 2024-04-21T23:19:39.603 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_co_items": 2, 2024-04-21T23:19:39.603 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_1_bytes": 0, 2024-04-21T23:19:39.603 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_1_items": 0, 2024-04-21T23:19:39.603 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_2_bytes": 0, 2024-04-21T23:19:39.604 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_2_items": 0 2024-04-21T23:19:39.604 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:39.604 INFO:teuthology.orchestra.run.smithi069.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:19:39.604 INFO:teuthology.orchestra.run.smithi069.stdout: "val": 0, 2024-04-21T23:19:39.604 INFO:teuthology.orchestra.run.smithi069.stdout: "max": 104857600, 2024-04-21T23:19:39.604 INFO:teuthology.orchestra.run.smithi069.stdout: "get_started": 0, 2024-04-21T23:19:39.604 INFO:teuthology.orchestra.run.smithi069.stdout: "get": 231, 2024-04-21T23:19:39.604 INFO:teuthology.orchestra.run.smithi069.stdout: "get_sum": 1412710, 2024-04-21T23:19:39.604 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_fail": 0, 2024-04-21T23:19:39.605 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_success": 231, 2024-04-21T23:19:39.605 INFO:teuthology.orchestra.run.smithi069.stdout: "take": 0, 2024-04-21T23:19:39.605 INFO:teuthology.orchestra.run.smithi069.stdout: "take_sum": 0, 2024-04-21T23:19:39.605 INFO:teuthology.orchestra.run.smithi069.stdout: "put": 231, 2024-04-21T23:19:39.605 INFO:teuthology.orchestra.run.smithi069.stdout: "put_sum": 1412710, 2024-04-21T23:19:39.605 INFO:teuthology.orchestra.run.smithi069.stdout: "wait": { 2024-04-21T23:19:39.605 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:39.605 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:39.605 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:39.605 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-21T23:19:39.606 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-21T23:19:39.606 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-21T23:19:40.211 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.f 2024-04-21T23:19:40.211 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.f 2024-04-21T23:19:40.211 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.f 2024-04-21T23:19:40.211 DEBUG:tasks.check_counter:Getting stats from i 2024-04-21T23:19:40.212 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-21T23:19:40.449 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:19:40.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:40 smithi069 ceph-mon[31816]: pgmap v342: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-04-21T23:19:40.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:40 smithi044 ceph-mon[27934]: pgmap v342: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-04-21T23:19:40.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:40 smithi055 ceph-mon[31117]: pgmap v342: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-04-21T23:19:42.122 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-21T23:19:42.122 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:19:42.122 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 276, 2024-04-21T23:19:42.122 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 276, 2024-04-21T23:19:42.123 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 16873116, 2024-04-21T23:19:42.123 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 86963, 2024-04-21T23:19:42.123 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 8, 2024-04-21T23:19:42.123 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 4, 2024-04-21T23:19:42.123 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.038387898, 2024-04-21T23:19:42.123 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.009059929, 2024-04-21T23:19:42.123 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.015918503, 2024-04-21T23:19:42.123 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.009884391, 2024-04-21T23:19:42.124 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:42.124 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 275, 2024-04-21T23:19:42.124 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.014368538, 2024-04-21T23:19:42.124 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000052249 2024-04-21T23:19:42.124 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.124 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:42.124 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.124 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.125 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.125 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.125 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-21T23:19:42.125 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T23:19:42.125 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.125 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:19:42.125 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 217, 2024-04-21T23:19:42.125 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 260, 2024-04-21T23:19:42.125 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 16477161, 2024-04-21T23:19:42.126 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 227961, 2024-04-21T23:19:42.126 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 6, 2024-04-21T23:19:42.126 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 5, 2024-04-21T23:19:42.126 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.028965991, 2024-04-21T23:19:42.126 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.007637900, 2024-04-21T23:19:42.126 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.012556564, 2024-04-21T23:19:42.126 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.006478605, 2024-04-21T23:19:42.126 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:42.127 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 258, 2024-04-21T23:19:42.127 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.014054405, 2024-04-21T23:19:42.127 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000054474 2024-04-21T23:19:42.127 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.127 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:42.127 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.127 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.127 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.128 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.128 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-21T23:19:42.128 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 159216 2024-04-21T23:19:42.128 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.128 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:19:42.128 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 318, 2024-04-21T23:19:42.128 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 317, 2024-04-21T23:19:42.128 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 28833484, 2024-04-21T23:19:42.128 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 115029, 2024-04-21T23:19:42.129 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 5, 2024-04-21T23:19:42.129 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 4, 2024-04-21T23:19:42.129 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.042253614, 2024-04-21T23:19:42.129 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.011471221, 2024-04-21T23:19:42.129 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.021716915, 2024-04-21T23:19:42.129 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.007266620, 2024-04-21T23:19:42.129 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:42.129 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 316, 2024-04-21T23:19:42.130 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.014255620, 2024-04-21T23:19:42.130 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000045112 2024-04-21T23:19:42.130 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.130 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:42.130 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.130 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.130 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.130 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.131 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 175184, 2024-04-21T23:19:42.131 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 36800 2024-04-21T23:19:42.131 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.131 INFO:teuthology.orchestra.run.smithi069.stdout: "cct": { 2024-04-21T23:19:42.131 INFO:teuthology.orchestra.run.smithi069.stdout: "total_workers": 1, 2024-04-21T23:19:42.131 INFO:teuthology.orchestra.run.smithi069.stdout: "unhealthy_workers": 0 2024-04-21T23:19:42.131 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.131 INFO:teuthology.orchestra.run.smithi069.stdout: "finisher-MDSRank": { 2024-04-21T23:19:42.132 INFO:teuthology.orchestra.run.smithi069.stdout: "queue_len": 0, 2024-04-21T23:19:42.132 INFO:teuthology.orchestra.run.smithi069.stdout: "complete_latency": { 2024-04-21T23:19:42.132 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 1168, 2024-04-21T23:19:42.132 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.047602234, 2024-04-21T23:19:42.132 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000040755 2024-04-21T23:19:42.132 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-21T23:19:42.132 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.132 INFO:teuthology.orchestra.run.smithi069.stdout: "finisher-PurgeQueue": { 2024-04-21T23:19:42.132 INFO:teuthology.orchestra.run.smithi069.stdout: "queue_len": 0, 2024-04-21T23:19:42.133 INFO:teuthology.orchestra.run.smithi069.stdout: "complete_latency": { 2024-04-21T23:19:42.133 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.133 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.133 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.133 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-21T23:19:42.133 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.133 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": { 2024-04-21T23:19:42.133 INFO:teuthology.orchestra.run.smithi069.stdout: "request": 0, 2024-04-21T23:19:42.134 INFO:teuthology.orchestra.run.smithi069.stdout: "reply": 0, 2024-04-21T23:19:42.134 INFO:teuthology.orchestra.run.smithi069.stdout: "reply_latency": { 2024-04-21T23:19:42.134 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.134 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.134 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.134 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.134 INFO:teuthology.orchestra.run.smithi069.stdout: "slow_reply": 0, 2024-04-21T23:19:42.134 INFO:teuthology.orchestra.run.smithi069.stdout: "forward": 0, 2024-04-21T23:19:42.135 INFO:teuthology.orchestra.run.smithi069.stdout: "dir_fetch_complete": 0, 2024-04-21T23:19:42.135 INFO:teuthology.orchestra.run.smithi069.stdout: "dir_fetch_keys": 0, 2024-04-21T23:19:42.135 INFO:teuthology.orchestra.run.smithi069.stdout: "dir_commit": 0, 2024-04-21T23:19:42.135 INFO:teuthology.orchestra.run.smithi069.stdout: "dir_split": 0, 2024-04-21T23:19:42.135 INFO:teuthology.orchestra.run.smithi069.stdout: "dir_merge": 0, 2024-04-21T23:19:42.135 INFO:teuthology.orchestra.run.smithi069.stdout: "inodes": 3578, 2024-04-21T23:19:42.135 INFO:teuthology.orchestra.run.smithi069.stdout: "inodes_top": 1588, 2024-04-21T23:19:42.135 INFO:teuthology.orchestra.run.smithi069.stdout: "inodes_bottom": 1990, 2024-04-21T23:19:42.135 INFO:teuthology.orchestra.run.smithi069.stdout: "inodes_pin_tail": 0, 2024-04-21T23:19:42.136 INFO:teuthology.orchestra.run.smithi069.stdout: "inodes_pinned": 1310, 2024-04-21T23:19:42.136 INFO:teuthology.orchestra.run.smithi069.stdout: "inodes_expired": 0, 2024-04-21T23:19:42.136 INFO:teuthology.orchestra.run.smithi069.stdout: "inodes_with_caps": 0, 2024-04-21T23:19:42.136 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": 0, 2024-04-21T23:19:42.136 INFO:teuthology.orchestra.run.smithi069.stdout: "subtrees": 2, 2024-04-21T23:19:42.136 INFO:teuthology.orchestra.run.smithi069.stdout: "traverse": 0, 2024-04-21T23:19:42.136 INFO:teuthology.orchestra.run.smithi069.stdout: "traverse_hit": 0, 2024-04-21T23:19:42.136 INFO:teuthology.orchestra.run.smithi069.stdout: "traverse_forward": 0, 2024-04-21T23:19:42.137 INFO:teuthology.orchestra.run.smithi069.stdout: "traverse_discover": 0, 2024-04-21T23:19:42.137 INFO:teuthology.orchestra.run.smithi069.stdout: "traverse_dir_fetch": 0, 2024-04-21T23:19:42.137 INFO:teuthology.orchestra.run.smithi069.stdout: "traverse_remote_ino": 0, 2024-04-21T23:19:42.137 INFO:teuthology.orchestra.run.smithi069.stdout: "traverse_lock": 0, 2024-04-21T23:19:42.137 INFO:teuthology.orchestra.run.smithi069.stdout: "load_cent": 0, 2024-04-21T23:19:42.137 INFO:teuthology.orchestra.run.smithi069.stdout: "q": 0, 2024-04-21T23:19:42.137 INFO:teuthology.orchestra.run.smithi069.stdout: "exported": 0, 2024-04-21T23:19:42.137 INFO:teuthology.orchestra.run.smithi069.stdout: "exported_inodes": 0, 2024-04-21T23:19:42.137 INFO:teuthology.orchestra.run.smithi069.stdout: "imported": 0, 2024-04-21T23:19:42.138 INFO:teuthology.orchestra.run.smithi069.stdout: "imported_inodes": 0, 2024-04-21T23:19:42.138 INFO:teuthology.orchestra.run.smithi069.stdout: "openino_dir_fetch": 0, 2024-04-21T23:19:42.138 INFO:teuthology.orchestra.run.smithi069.stdout: "openino_backtrace_fetch": 0, 2024-04-21T23:19:42.138 INFO:teuthology.orchestra.run.smithi069.stdout: "openino_peer_discover": 0, 2024-04-21T23:19:42.138 INFO:teuthology.orchestra.run.smithi069.stdout: "root_rfiles": 68, 2024-04-21T23:19:42.138 INFO:teuthology.orchestra.run.smithi069.stdout: "root_rbytes": 21196213, 2024-04-21T23:19:42.138 INFO:teuthology.orchestra.run.smithi069.stdout: "root_rsnaps": 6, 2024-04-21T23:19:42.138 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_backtrace_fetch": 0, 2024-04-21T23:19:42.139 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_set_tag": 0, 2024-04-21T23:19:42.139 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T23:19:42.139 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_inotable_repaired": 0, 2024-04-21T23:19:42.139 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_dir_inodes": 0, 2024-04-21T23:19:42.139 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_dir_base_inodes": 0, 2024-04-21T23:19:42.139 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_dirfrag_rstats": 0, 2024-04-21T23:19:42.139 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_file_inodes": 0, 2024-04-21T23:19:42.139 INFO:teuthology.orchestra.run.smithi069.stdout: "handle_inode_file_caps": 0, 2024-04-21T23:19:42.139 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_cap_op_revoke": 0, 2024-04-21T23:19:42.140 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_cap_op_grant": 0, 2024-04-21T23:19:42.140 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_cap_op_trunc": 0, 2024-04-21T23:19:42.140 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T23:19:42.140 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T23:19:42.140 INFO:teuthology.orchestra.run.smithi069.stdout: "handle_client_caps": 0, 2024-04-21T23:19:42.140 INFO:teuthology.orchestra.run.smithi069.stdout: "handle_client_caps_dirty": 0, 2024-04-21T23:19:42.140 INFO:teuthology.orchestra.run.smithi069.stdout: "handle_client_cap_release": 0, 2024-04-21T23:19:42.140 INFO:teuthology.orchestra.run.smithi069.stdout: "process_request_cap_release": 0 2024-04-21T23:19:42.141 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.141 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_cache": { 2024-04-21T23:19:42.141 INFO:teuthology.orchestra.run.smithi069.stdout: "dir_update": 0, 2024-04-21T23:19:42.141 INFO:teuthology.orchestra.run.smithi069.stdout: "dir_update_receipt": 0, 2024-04-21T23:19:42.141 INFO:teuthology.orchestra.run.smithi069.stdout: "dir_try_discover": 0, 2024-04-21T23:19:42.141 INFO:teuthology.orchestra.run.smithi069.stdout: "dir_send_discover": 0, 2024-04-21T23:19:42.141 INFO:teuthology.orchestra.run.smithi069.stdout: "dir_handle_discover": 0, 2024-04-21T23:19:42.141 INFO:teuthology.orchestra.run.smithi069.stdout: "num_strays": 0, 2024-04-21T23:19:42.142 INFO:teuthology.orchestra.run.smithi069.stdout: "num_strays_delayed": 0, 2024-04-21T23:19:42.142 INFO:teuthology.orchestra.run.smithi069.stdout: "num_strays_enqueuing": 0, 2024-04-21T23:19:42.142 INFO:teuthology.orchestra.run.smithi069.stdout: "strays_created": 0, 2024-04-21T23:19:42.142 INFO:teuthology.orchestra.run.smithi069.stdout: "strays_enqueued": 0, 2024-04-21T23:19:42.142 INFO:teuthology.orchestra.run.smithi069.stdout: "strays_reintegrated": 0, 2024-04-21T23:19:42.142 INFO:teuthology.orchestra.run.smithi069.stdout: "strays_migrated": 0, 2024-04-21T23:19:42.142 INFO:teuthology.orchestra.run.smithi069.stdout: "num_recovering_processing": 0, 2024-04-21T23:19:42.142 INFO:teuthology.orchestra.run.smithi069.stdout: "num_recovering_enqueued": 0, 2024-04-21T23:19:42.143 INFO:teuthology.orchestra.run.smithi069.stdout: "num_recovering_prioritized": 0, 2024-04-21T23:19:42.143 INFO:teuthology.orchestra.run.smithi069.stdout: "recovery_started": 0, 2024-04-21T23:19:42.143 INFO:teuthology.orchestra.run.smithi069.stdout: "recovery_completed": 0, 2024-04-21T23:19:42.143 INFO:teuthology.orchestra.run.smithi069.stdout: "ireq_quiesce_path": 0, 2024-04-21T23:19:42.143 INFO:teuthology.orchestra.run.smithi069.stdout: "ireq_quiesce_inode": 0, 2024-04-21T23:19:42.143 INFO:teuthology.orchestra.run.smithi069.stdout: "ireq_enqueue_scrub": 0, 2024-04-21T23:19:42.143 INFO:teuthology.orchestra.run.smithi069.stdout: "ireq_exportdir": 0, 2024-04-21T23:19:42.143 INFO:teuthology.orchestra.run.smithi069.stdout: "ireq_flush": 0, 2024-04-21T23:19:42.143 INFO:teuthology.orchestra.run.smithi069.stdout: "ireq_fragmentdir": 0, 2024-04-21T23:19:42.144 INFO:teuthology.orchestra.run.smithi069.stdout: "ireq_fragstats": 0, 2024-04-21T23:19:42.144 INFO:teuthology.orchestra.run.smithi069.stdout: "ireq_inodestats": 0 2024-04-21T23:19:42.144 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.144 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_log": { 2024-04-21T23:19:42.144 INFO:teuthology.orchestra.run.smithi069.stdout: "evlrg": 0, 2024-04-21T23:19:42.144 INFO:teuthology.orchestra.run.smithi069.stdout: "evadd": 0, 2024-04-21T23:19:42.144 INFO:teuthology.orchestra.run.smithi069.stdout: "evex": 0, 2024-04-21T23:19:42.144 INFO:teuthology.orchestra.run.smithi069.stdout: "evtrm": 0, 2024-04-21T23:19:42.145 INFO:teuthology.orchestra.run.smithi069.stdout: "ev": 14423, 2024-04-21T23:19:42.145 INFO:teuthology.orchestra.run.smithi069.stdout: "evexg": 0, 2024-04-21T23:19:42.145 INFO:teuthology.orchestra.run.smithi069.stdout: "evexd": 0, 2024-04-21T23:19:42.145 INFO:teuthology.orchestra.run.smithi069.stdout: "segadd": 0, 2024-04-21T23:19:42.145 INFO:teuthology.orchestra.run.smithi069.stdout: "segex": 0, 2024-04-21T23:19:42.145 INFO:teuthology.orchestra.run.smithi069.stdout: "segtrm": 0, 2024-04-21T23:19:42.145 INFO:teuthology.orchestra.run.smithi069.stdout: "seg": 3, 2024-04-21T23:19:42.145 INFO:teuthology.orchestra.run.smithi069.stdout: "segmjr": 4, 2024-04-21T23:19:42.145 INFO:teuthology.orchestra.run.smithi069.stdout: "segexg": 0, 2024-04-21T23:19:42.146 INFO:teuthology.orchestra.run.smithi069.stdout: "segexd": 0, 2024-04-21T23:19:42.146 INFO:teuthology.orchestra.run.smithi069.stdout: "expos": 484446288, 2024-04-21T23:19:42.146 INFO:teuthology.orchestra.run.smithi069.stdout: "wrpos": 489967567, 2024-04-21T23:19:42.146 INFO:teuthology.orchestra.run.smithi069.stdout: "rdpos": 489967431, 2024-04-21T23:19:42.146 INFO:teuthology.orchestra.run.smithi069.stdout: "jlat": { 2024-04-21T23:19:42.146 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.146 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.146 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.147 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.147 INFO:teuthology.orchestra.run.smithi069.stdout: "replayed": 14423 2024-04-21T23:19:42.147 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.147 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_mem": { 2024-04-21T23:19:42.147 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 198, 2024-04-21T23:19:42.147 INFO:teuthology.orchestra.run.smithi069.stdout: "ino+": 3455, 2024-04-21T23:19:42.147 INFO:teuthology.orchestra.run.smithi069.stdout: "ino-": 3257, 2024-04-21T23:19:42.147 INFO:teuthology.orchestra.run.smithi069.stdout: "dir": 44, 2024-04-21T23:19:42.148 INFO:teuthology.orchestra.run.smithi069.stdout: "dir+": 55, 2024-04-21T23:19:42.148 INFO:teuthology.orchestra.run.smithi069.stdout: "dir-": 11, 2024-04-21T23:19:42.148 INFO:teuthology.orchestra.run.smithi069.stdout: "dn": 3578, 2024-04-21T23:19:42.148 INFO:teuthology.orchestra.run.smithi069.stdout: "dn+": 3578, 2024-04-21T23:19:42.148 INFO:teuthology.orchestra.run.smithi069.stdout: "dn-": 0, 2024-04-21T23:19:42.148 INFO:teuthology.orchestra.run.smithi069.stdout: "cap": 0, 2024-04-21T23:19:42.148 INFO:teuthology.orchestra.run.smithi069.stdout: "cap+": 0, 2024-04-21T23:19:42.148 INFO:teuthology.orchestra.run.smithi069.stdout: "cap-": 0, 2024-04-21T23:19:42.148 INFO:teuthology.orchestra.run.smithi069.stdout: "rss": 56012, 2024-04-21T23:19:42.149 INFO:teuthology.orchestra.run.smithi069.stdout: "heap": 215396 2024-04-21T23:19:42.149 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.149 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_server": { 2024-04-21T23:19:42.149 INFO:teuthology.orchestra.run.smithi069.stdout: "dispatch_client_request": 0, 2024-04-21T23:19:42.149 INFO:teuthology.orchestra.run.smithi069.stdout: "dispatch_server_request": 0, 2024-04-21T23:19:42.149 INFO:teuthology.orchestra.run.smithi069.stdout: "handle_client_request": 0, 2024-04-21T23:19:42.149 INFO:teuthology.orchestra.run.smithi069.stdout: "handle_client_session": 0, 2024-04-21T23:19:42.149 INFO:teuthology.orchestra.run.smithi069.stdout: "handle_peer_request": 0, 2024-04-21T23:19:42.150 INFO:teuthology.orchestra.run.smithi069.stdout: "req_create_latency": { 2024-04-21T23:19:42.150 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.150 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.150 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.150 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.150 INFO:teuthology.orchestra.run.smithi069.stdout: "req_getattr_latency": { 2024-04-21T23:19:42.150 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.150 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.151 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.151 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.151 INFO:teuthology.orchestra.run.smithi069.stdout: "req_getfilelock_latency": { 2024-04-21T23:19:42.151 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.151 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.151 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.151 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.151 INFO:teuthology.orchestra.run.smithi069.stdout: "req_link_latency": { 2024-04-21T23:19:42.151 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.152 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.152 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.152 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.152 INFO:teuthology.orchestra.run.smithi069.stdout: "req_lookup_latency": { 2024-04-21T23:19:42.152 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.152 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.152 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.152 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.153 INFO:teuthology.orchestra.run.smithi069.stdout: "req_lookuphash_latency": { 2024-04-21T23:19:42.153 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.153 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.153 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.153 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.153 INFO:teuthology.orchestra.run.smithi069.stdout: "req_lookupino_latency": { 2024-04-21T23:19:42.153 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.153 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.153 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.154 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.154 INFO:teuthology.orchestra.run.smithi069.stdout: "req_lookupname_latency": { 2024-04-21T23:19:42.154 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.154 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.154 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.154 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.154 INFO:teuthology.orchestra.run.smithi069.stdout: "req_lookupparent_latency": { 2024-04-21T23:19:42.154 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.155 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.155 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.155 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.155 INFO:teuthology.orchestra.run.smithi069.stdout: "req_lookupsnap_latency": { 2024-04-21T23:19:42.155 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.155 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.155 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.155 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.156 INFO:teuthology.orchestra.run.smithi069.stdout: "req_lssnap_latency": { 2024-04-21T23:19:42.156 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.156 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.156 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.156 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.157 INFO:teuthology.orchestra.run.smithi069.stdout: "req_mkdir_latency": { 2024-04-21T23:19:42.157 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.157 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.157 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.157 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.157 INFO:teuthology.orchestra.run.smithi069.stdout: "req_mknod_latency": { 2024-04-21T23:19:42.158 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.158 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.158 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.158 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.158 INFO:teuthology.orchestra.run.smithi069.stdout: "req_mksnap_latency": { 2024-04-21T23:19:42.158 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.158 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.158 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.159 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.159 INFO:teuthology.orchestra.run.smithi069.stdout: "req_open_latency": { 2024-04-21T23:19:42.159 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.159 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.159 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.159 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.159 INFO:teuthology.orchestra.run.smithi069.stdout: "req_readdir_latency": { 2024-04-21T23:19:42.159 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.160 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.160 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.160 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.160 INFO:teuthology.orchestra.run.smithi069.stdout: "req_rename_latency": { 2024-04-21T23:19:42.160 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.160 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.160 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.160 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.160 INFO:teuthology.orchestra.run.smithi069.stdout: "req_renamesnap_latency": { 2024-04-21T23:19:42.161 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.161 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.161 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.161 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.161 INFO:teuthology.orchestra.run.smithi069.stdout: "req_snapdiff_latency": { 2024-04-21T23:19:42.161 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.161 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.161 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.161 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.162 INFO:teuthology.orchestra.run.smithi069.stdout: "req_rmdir_latency": { 2024-04-21T23:19:42.162 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.162 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.162 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.162 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.162 INFO:teuthology.orchestra.run.smithi069.stdout: "req_rmsnap_latency": { 2024-04-21T23:19:42.162 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.162 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.163 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.163 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.163 INFO:teuthology.orchestra.run.smithi069.stdout: "req_rmxattr_latency": { 2024-04-21T23:19:42.163 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.163 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.163 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.163 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.163 INFO:teuthology.orchestra.run.smithi069.stdout: "req_setattr_latency": { 2024-04-21T23:19:42.164 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.164 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.164 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.164 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.164 INFO:teuthology.orchestra.run.smithi069.stdout: "req_setdirlayout_latency": { 2024-04-21T23:19:42.164 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.164 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.164 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.165 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.165 INFO:teuthology.orchestra.run.smithi069.stdout: "req_setfilelock_latency": { 2024-04-21T23:19:42.165 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.165 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.165 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.165 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.165 INFO:teuthology.orchestra.run.smithi069.stdout: "req_setlayout_latency": { 2024-04-21T23:19:42.165 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.166 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.166 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.166 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.166 INFO:teuthology.orchestra.run.smithi069.stdout: "req_setxattr_latency": { 2024-04-21T23:19:42.166 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.166 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.166 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.166 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.167 INFO:teuthology.orchestra.run.smithi069.stdout: "req_symlink_latency": { 2024-04-21T23:19:42.167 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.167 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.167 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.167 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.167 INFO:teuthology.orchestra.run.smithi069.stdout: "req_unlink_latency": { 2024-04-21T23:19:42.167 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.167 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.168 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.168 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.168 INFO:teuthology.orchestra.run.smithi069.stdout: "cap_revoke_eviction": 0, 2024-04-21T23:19:42.168 INFO:teuthology.orchestra.run.smithi069.stdout: "cap_acquisition_throttle": 0, 2024-04-21T23:19:42.168 INFO:teuthology.orchestra.run.smithi069.stdout: "req_getvxattr_latency": { 2024-04-21T23:19:42.168 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.168 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.168 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.168 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-21T23:19:42.169 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.169 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_sessions": { 2024-04-21T23:19:42.169 INFO:teuthology.orchestra.run.smithi069.stdout: "session_count": 2, 2024-04-21T23:19:42.169 INFO:teuthology.orchestra.run.smithi069.stdout: "session_add": 2, 2024-04-21T23:19:42.169 INFO:teuthology.orchestra.run.smithi069.stdout: "session_remove": 0, 2024-04-21T23:19:42.169 INFO:teuthology.orchestra.run.smithi069.stdout: "sessions_open": 0, 2024-04-21T23:19:42.169 INFO:teuthology.orchestra.run.smithi069.stdout: "sessions_stale": 0, 2024-04-21T23:19:42.169 INFO:teuthology.orchestra.run.smithi069.stdout: "total_load": 0, 2024-04-21T23:19:42.170 INFO:teuthology.orchestra.run.smithi069.stdout: "average_load": 0, 2024-04-21T23:19:42.170 INFO:teuthology.orchestra.run.smithi069.stdout: "avg_session_uptime": 1823, 2024-04-21T23:19:42.170 INFO:teuthology.orchestra.run.smithi069.stdout: "mdthresh_evicted": 0 2024-04-21T23:19:42.170 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.170 INFO:teuthology.orchestra.run.smithi069.stdout: "mempool": { 2024-04-21T23:19:42.170 INFO:teuthology.orchestra.run.smithi069.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:19:42.170 INFO:teuthology.orchestra.run.smithi069.stdout: "bloom_filter_items": 0, 2024-04-21T23:19:42.170 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:19:42.171 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:19:42.171 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:19:42.171 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:19:42.171 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:19:42.171 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:19:42.171 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:19:42.171 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:19:42.171 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:19:42.172 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:19:42.172 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:19:42.172 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:19:42.172 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:19:42.172 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_items": 0, 2024-04-21T23:19:42.172 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:19:42.172 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_blob_items": 0, 2024-04-21T23:19:42.172 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:19:42.173 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:19:42.173 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:19:42.173 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:19:42.173 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:19:42.173 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:19:42.173 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:19:42.173 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_txc_items": 0, 2024-04-21T23:19:42.173 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:19:42.173 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:19:42.174 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:19:42.174 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_items": 0, 2024-04-21T23:19:42.174 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_bytes": 0, 2024-04-21T23:19:42.174 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_items": 0, 2024-04-21T23:19:42.174 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:19:42.174 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:19:42.174 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:19:42.174 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:19:42.175 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_anon_bytes": 241585, 2024-04-21T23:19:42.175 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_anon_items": 364, 2024-04-21T23:19:42.177 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_meta_bytes": 0, 2024-04-21T23:19:42.177 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_meta_items": 0, 2024-04-21T23:19:42.178 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_bytes": 0, 2024-04-21T23:19:42.178 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_items": 0, 2024-04-21T23:19:42.178 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:19:42.178 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mapbl_items": 0, 2024-04-21T23:19:42.178 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:19:42.178 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pglog_items": 0, 2024-04-21T23:19:42.178 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_bytes": 9616, 2024-04-21T23:19:42.179 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_items": 185, 2024-04-21T23:19:42.179 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:19:42.179 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:19:42.179 INFO:teuthology.orchestra.run.smithi069.stdout: "pgmap_bytes": 0, 2024-04-21T23:19:42.179 INFO:teuthology.orchestra.run.smithi069.stdout: "pgmap_items": 0, 2024-04-21T23:19:42.179 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_co_bytes": 3179412, 2024-04-21T23:19:42.179 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_co_items": 21161, 2024-04-21T23:19:42.179 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_1_bytes": 0, 2024-04-21T23:19:42.179 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_1_items": 0, 2024-04-21T23:19:42.180 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_2_bytes": 0, 2024-04-21T23:19:42.180 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_2_items": 0 2024-04-21T23:19:42.180 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.180 INFO:teuthology.orchestra.run.smithi069.stdout: "objecter": { 2024-04-21T23:19:42.180 INFO:teuthology.orchestra.run.smithi069.stdout: "op_active": 0, 2024-04-21T23:19:42.180 INFO:teuthology.orchestra.run.smithi069.stdout: "op_laggy": 0, 2024-04-21T23:19:42.180 INFO:teuthology.orchestra.run.smithi069.stdout: "op_send": 734, 2024-04-21T23:19:42.180 INFO:teuthology.orchestra.run.smithi069.stdout: "op_send_bytes": 26, 2024-04-21T23:19:42.181 INFO:teuthology.orchestra.run.smithi069.stdout: "op_resend": 4, 2024-04-21T23:19:42.181 INFO:teuthology.orchestra.run.smithi069.stdout: "op_reply": 734, 2024-04-21T23:19:42.181 INFO:teuthology.orchestra.run.smithi069.stdout: "op_latency": { 2024-04-21T23:19:42.181 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 734, 2024-04-21T23:19:42.181 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.848981128, 2024-04-21T23:19:42.181 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.001156650 2024-04-21T23:19:42.181 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.181 INFO:teuthology.orchestra.run.smithi069.stdout: "op_inflight": 0, 2024-04-21T23:19:42.182 INFO:teuthology.orchestra.run.smithi069.stdout: "oplen_avg": { 2024-04-21T23:19:42.182 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 734, 2024-04-21T23:19:42.182 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 735 2024-04-21T23:19:42.182 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.182 INFO:teuthology.orchestra.run.smithi069.stdout: "op": 734, 2024-04-21T23:19:42.182 INFO:teuthology.orchestra.run.smithi069.stdout: "op_r": 734, 2024-04-21T23:19:42.182 INFO:teuthology.orchestra.run.smithi069.stdout: "op_w": 0, 2024-04-21T23:19:42.182 INFO:teuthology.orchestra.run.smithi069.stdout: "op_rmw": 0, 2024-04-21T23:19:42.182 INFO:teuthology.orchestra.run.smithi069.stdout: "op_pg": 0, 2024-04-21T23:19:42.183 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_stat": 452, 2024-04-21T23:19:42.183 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_create": 0, 2024-04-21T23:19:42.190 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_read": 279, 2024-04-21T23:19:42.190 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_write": 0, 2024-04-21T23:19:42.190 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_writefull": 0, 2024-04-21T23:19:42.190 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_writesame": 0, 2024-04-21T23:19:42.191 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_append": 0, 2024-04-21T23:19:42.191 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_zero": 0, 2024-04-21T23:19:42.191 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_truncate": 0, 2024-04-21T23:19:42.191 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_delete": 0, 2024-04-21T23:19:42.191 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_mapext": 0, 2024-04-21T23:19:42.191 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_sparse_read": 0, 2024-04-21T23:19:42.191 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_clonerange": 0, 2024-04-21T23:19:42.191 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_getxattr": 2, 2024-04-21T23:19:42.192 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_setxattr": 0, 2024-04-21T23:19:42.192 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_cmpxattr": 0, 2024-04-21T23:19:42.192 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_rmxattr": 0, 2024-04-21T23:19:42.192 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_resetxattrs": 0, 2024-04-21T23:19:42.192 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_call": 0, 2024-04-21T23:19:42.192 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_watch": 0, 2024-04-21T23:19:42.192 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_notify": 0, 2024-04-21T23:19:42.192 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T23:19:42.192 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_pgls": 0, 2024-04-21T23:19:42.193 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_pgls_filter": 0, 2024-04-21T23:19:42.193 INFO:teuthology.orchestra.run.smithi069.stdout: "osdop_other": 0, 2024-04-21T23:19:42.193 INFO:teuthology.orchestra.run.smithi069.stdout: "linger_active": 0, 2024-04-21T23:19:42.193 INFO:teuthology.orchestra.run.smithi069.stdout: "linger_send": 0, 2024-04-21T23:19:42.193 INFO:teuthology.orchestra.run.smithi069.stdout: "linger_resend": 0, 2024-04-21T23:19:42.193 INFO:teuthology.orchestra.run.smithi069.stdout: "linger_ping": 0, 2024-04-21T23:19:42.193 INFO:teuthology.orchestra.run.smithi069.stdout: "poolop_active": 0, 2024-04-21T23:19:42.193 INFO:teuthology.orchestra.run.smithi069.stdout: "poolop_send": 0, 2024-04-21T23:19:42.194 INFO:teuthology.orchestra.run.smithi069.stdout: "poolop_resend": 0, 2024-04-21T23:19:42.194 INFO:teuthology.orchestra.run.smithi069.stdout: "poolstat_active": 0, 2024-04-21T23:19:42.194 INFO:teuthology.orchestra.run.smithi069.stdout: "poolstat_send": 0, 2024-04-21T23:19:42.194 INFO:teuthology.orchestra.run.smithi069.stdout: "poolstat_resend": 0, 2024-04-21T23:19:42.194 INFO:teuthology.orchestra.run.smithi069.stdout: "statfs_active": 0, 2024-04-21T23:19:42.194 INFO:teuthology.orchestra.run.smithi069.stdout: "statfs_send": 0, 2024-04-21T23:19:42.194 INFO:teuthology.orchestra.run.smithi069.stdout: "statfs_resend": 0, 2024-04-21T23:19:42.194 INFO:teuthology.orchestra.run.smithi069.stdout: "command_active": 0, 2024-04-21T23:19:42.195 INFO:teuthology.orchestra.run.smithi069.stdout: "command_send": 0, 2024-04-21T23:19:42.195 INFO:teuthology.orchestra.run.smithi069.stdout: "command_resend": 0, 2024-04-21T23:19:42.195 INFO:teuthology.orchestra.run.smithi069.stdout: "map_epoch": 110, 2024-04-21T23:19:42.195 INFO:teuthology.orchestra.run.smithi069.stdout: "map_full": 0, 2024-04-21T23:19:42.195 INFO:teuthology.orchestra.run.smithi069.stdout: "map_inc": 6, 2024-04-21T23:19:42.195 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_sessions": 10, 2024-04-21T23:19:42.195 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_session_open": 10, 2024-04-21T23:19:42.195 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_session_close": 0, 2024-04-21T23:19:42.196 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_laggy": 0, 2024-04-21T23:19:42.196 INFO:teuthology.orchestra.run.smithi069.stdout: "omap_wr": 0, 2024-04-21T23:19:42.196 INFO:teuthology.orchestra.run.smithi069.stdout: "omap_rd": 2, 2024-04-21T23:19:42.202 INFO:teuthology.orchestra.run.smithi069.stdout: "omap_del": 0 2024-04-21T23:19:42.202 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.202 INFO:teuthology.orchestra.run.smithi069.stdout: "oft": { 2024-04-21T23:19:42.203 INFO:teuthology.orchestra.run.smithi069.stdout: "omap_total_objs": 0, 2024-04-21T23:19:42.203 INFO:teuthology.orchestra.run.smithi069.stdout: "omap_total_kv_pairs": 0, 2024-04-21T23:19:42.203 INFO:teuthology.orchestra.run.smithi069.stdout: "omap_total_updates": 0, 2024-04-21T23:19:42.203 INFO:teuthology.orchestra.run.smithi069.stdout: "omap_total_removes": 0 2024-04-21T23:19:42.203 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.203 INFO:teuthology.orchestra.run.smithi069.stdout: "osd-slow-ops": { 2024-04-21T23:19:42.203 INFO:teuthology.orchestra.run.smithi069.stdout: "slow_ops_count": 0 2024-04-21T23:19:42.204 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.204 INFO:teuthology.orchestra.run.smithi069.stdout: "purge_queue": { 2024-04-21T23:19:42.204 INFO:teuthology.orchestra.run.smithi069.stdout: "pq_executing_ops": 0, 2024-04-21T23:19:42.204 INFO:teuthology.orchestra.run.smithi069.stdout: "pq_executing_ops_high_water": 0, 2024-04-21T23:19:42.204 INFO:teuthology.orchestra.run.smithi069.stdout: "pq_executing": 0, 2024-04-21T23:19:42.204 INFO:teuthology.orchestra.run.smithi069.stdout: "pq_executing_high_water": 0, 2024-04-21T23:19:42.204 INFO:teuthology.orchestra.run.smithi069.stdout: "pq_executed_ops": 0, 2024-04-21T23:19:42.204 INFO:teuthology.orchestra.run.smithi069.stdout: "pq_executed": 0, 2024-04-21T23:19:42.205 INFO:teuthology.orchestra.run.smithi069.stdout: "pq_item_in_journal": 0 2024-04-21T23:19:42.205 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.205 INFO:teuthology.orchestra.run.smithi069.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:19:42.205 INFO:teuthology.orchestra.run.smithi069.stdout: "val": 0, 2024-04-21T23:19:42.205 INFO:teuthology.orchestra.run.smithi069.stdout: "max": 104857600, 2024-04-21T23:19:42.205 INFO:teuthology.orchestra.run.smithi069.stdout: "get_started": 0, 2024-04-21T23:19:42.205 INFO:teuthology.orchestra.run.smithi069.stdout: "get": 807, 2024-04-21T23:19:42.205 INFO:teuthology.orchestra.run.smithi069.stdout: "get_sum": 62104465, 2024-04-21T23:19:42.206 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_fail": 0, 2024-04-21T23:19:42.206 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_success": 807, 2024-04-21T23:19:42.206 INFO:teuthology.orchestra.run.smithi069.stdout: "take": 0, 2024-04-21T23:19:42.206 INFO:teuthology.orchestra.run.smithi069.stdout: "take_sum": 0, 2024-04-21T23:19:42.206 INFO:teuthology.orchestra.run.smithi069.stdout: "put": 807, 2024-04-21T23:19:42.206 INFO:teuthology.orchestra.run.smithi069.stdout: "put_sum": 62104465, 2024-04-21T23:19:42.206 INFO:teuthology.orchestra.run.smithi069.stdout: "wait": { 2024-04-21T23:19:42.206 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.207 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.207 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.207 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-21T23:19:42.207 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.207 INFO:teuthology.orchestra.run.smithi069.stdout: "throttle-objecter_bytes": { 2024-04-21T23:19:42.207 INFO:teuthology.orchestra.run.smithi069.stdout: "val": 0, 2024-04-21T23:19:42.207 INFO:teuthology.orchestra.run.smithi069.stdout: "max": 104857600, 2024-04-21T23:19:42.207 INFO:teuthology.orchestra.run.smithi069.stdout: "get_started": 0, 2024-04-21T23:19:42.208 INFO:teuthology.orchestra.run.smithi069.stdout: "get": 0, 2024-04-21T23:19:42.208 INFO:teuthology.orchestra.run.smithi069.stdout: "get_sum": 0, 2024-04-21T23:19:42.208 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_fail": 0, 2024-04-21T23:19:42.208 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_success": 0, 2024-04-21T23:19:42.208 INFO:teuthology.orchestra.run.smithi069.stdout: "take": 734, 2024-04-21T23:19:42.208 INFO:teuthology.orchestra.run.smithi069.stdout: "take_sum": 61796298, 2024-04-21T23:19:42.208 INFO:teuthology.orchestra.run.smithi069.stdout: "put": 734, 2024-04-21T23:19:42.208 INFO:teuthology.orchestra.run.smithi069.stdout: "put_sum": 61796298, 2024-04-21T23:19:42.209 INFO:teuthology.orchestra.run.smithi069.stdout: "wait": { 2024-04-21T23:19:42.209 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.209 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.209 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.209 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-21T23:19:42.209 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.209 INFO:teuthology.orchestra.run.smithi069.stdout: "throttle-objecter_ops": { 2024-04-21T23:19:42.209 INFO:teuthology.orchestra.run.smithi069.stdout: "val": 0, 2024-04-21T23:19:42.210 INFO:teuthology.orchestra.run.smithi069.stdout: "max": 1024, 2024-04-21T23:19:42.210 INFO:teuthology.orchestra.run.smithi069.stdout: "get_started": 0, 2024-04-21T23:19:42.210 INFO:teuthology.orchestra.run.smithi069.stdout: "get": 0, 2024-04-21T23:19:42.210 INFO:teuthology.orchestra.run.smithi069.stdout: "get_sum": 0, 2024-04-21T23:19:42.210 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_fail": 0, 2024-04-21T23:19:42.210 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_success": 0, 2024-04-21T23:19:42.210 INFO:teuthology.orchestra.run.smithi069.stdout: "take": 734, 2024-04-21T23:19:42.211 INFO:teuthology.orchestra.run.smithi069.stdout: "take_sum": 734, 2024-04-21T23:19:42.211 INFO:teuthology.orchestra.run.smithi069.stdout: "put": 734, 2024-04-21T23:19:42.211 INFO:teuthology.orchestra.run.smithi069.stdout: "put_sum": 734, 2024-04-21T23:19:42.211 INFO:teuthology.orchestra.run.smithi069.stdout: "wait": { 2024-04-21T23:19:42.211 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.211 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.211 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.211 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-21T23:19:42.212 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.212 INFO:teuthology.orchestra.run.smithi069.stdout: "throttle-write_buf_throttle": { 2024-04-21T23:19:42.212 INFO:teuthology.orchestra.run.smithi069.stdout: "val": 0, 2024-04-21T23:19:42.212 INFO:teuthology.orchestra.run.smithi069.stdout: "max": 3758096384, 2024-04-21T23:19:42.212 INFO:teuthology.orchestra.run.smithi069.stdout: "get_started": 0, 2024-04-21T23:19:42.212 INFO:teuthology.orchestra.run.smithi069.stdout: "get": 0, 2024-04-21T23:19:42.212 INFO:teuthology.orchestra.run.smithi069.stdout: "get_sum": 0, 2024-04-21T23:19:42.212 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_fail": 0, 2024-04-21T23:19:42.212 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_success": 0, 2024-04-21T23:19:42.213 INFO:teuthology.orchestra.run.smithi069.stdout: "take": 0, 2024-04-21T23:19:42.213 INFO:teuthology.orchestra.run.smithi069.stdout: "take_sum": 0, 2024-04-21T23:19:42.213 INFO:teuthology.orchestra.run.smithi069.stdout: "put": 0, 2024-04-21T23:19:42.213 INFO:teuthology.orchestra.run.smithi069.stdout: "put_sum": 0, 2024-04-21T23:19:42.213 INFO:teuthology.orchestra.run.smithi069.stdout: "wait": { 2024-04-21T23:19:42.213 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.213 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.213 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.214 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-21T23:19:42.214 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:42.214 INFO:teuthology.orchestra.run.smithi069.stdout: "throttle-write_buf_throttle-0x5648d18fe6e0": { 2024-04-21T23:19:42.214 INFO:teuthology.orchestra.run.smithi069.stdout: "val": 0, 2024-04-21T23:19:42.214 INFO:teuthology.orchestra.run.smithi069.stdout: "max": 3758096384, 2024-04-21T23:19:42.214 INFO:teuthology.orchestra.run.smithi069.stdout: "get_started": 0, 2024-04-21T23:19:42.214 INFO:teuthology.orchestra.run.smithi069.stdout: "get": 0, 2024-04-21T23:19:42.214 INFO:teuthology.orchestra.run.smithi069.stdout: "get_sum": 0, 2024-04-21T23:19:42.215 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_fail": 0, 2024-04-21T23:19:42.215 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_success": 0, 2024-04-21T23:19:42.215 INFO:teuthology.orchestra.run.smithi069.stdout: "take": 0, 2024-04-21T23:19:42.215 INFO:teuthology.orchestra.run.smithi069.stdout: "take_sum": 0, 2024-04-21T23:19:42.215 INFO:teuthology.orchestra.run.smithi069.stdout: "put": 0, 2024-04-21T23:19:42.215 INFO:teuthology.orchestra.run.smithi069.stdout: "put_sum": 0, 2024-04-21T23:19:42.215 INFO:teuthology.orchestra.run.smithi069.stdout: "wait": { 2024-04-21T23:19:42.215 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:42.216 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:42.216 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:42.216 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-21T23:19:42.216 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-21T23:19:42.216 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-21T23:19:42.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:42 smithi069 ceph-mon[31816]: pgmap v343: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:19:42.516 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:42 smithi055 ceph-mon[31117]: pgmap v343: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:19:42.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:42 smithi044 ceph-mon[27934]: pgmap v343: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:19:42.897 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-21T23:19:42.898 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-21T23:19:42.898 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-21T23:19:42.898 DEBUG:tasks.check_counter:Getting stats from l 2024-04-21T23:19:42.898 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b shell --fsid 80909b94-0032-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-21T23:19:43.136 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/mon.c/config 2024-04-21T23:19:44.565 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:44 smithi069 ceph-mon[31816]: pgmap v344: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:19:44.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:44 smithi044 ceph-mon[27934]: pgmap v344: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:19:44.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:44 smithi055 ceph-mon[31117]: pgmap v344: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:19:44.799 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-21T23:19:44.800 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T23:19:44.800 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 4, 2024-04-21T23:19:44.800 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 4, 2024-04-21T23:19:44.800 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 2992, 2024-04-21T23:19:44.800 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 448, 2024-04-21T23:19:44.800 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 4, 2024-04-21T23:19:44.800 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 0, 2024-04-21T23:19:44.800 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.001304373, 2024-04-21T23:19:44.801 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.000039234, 2024-04-21T23:19:44.801 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.000542608, 2024-04-21T23:19:44.801 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:19:44.801 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:44.801 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 3, 2024-04-21T23:19:44.801 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000716180, 2024-04-21T23:19:44.801 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000238726 2024-04-21T23:19:44.801 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:44.802 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:44.802 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:44.802 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:44.802 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:44.802 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:44.802 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-21T23:19:44.802 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T23:19:44.803 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:44.803 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T23:19:44.803 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 234, 2024-04-21T23:19:44.803 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 224, 2024-04-21T23:19:44.803 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 1446368, 2024-04-21T23:19:44.803 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 137648, 2024-04-21T23:19:44.803 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 2, 2024-04-21T23:19:44.803 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 1, 2024-04-21T23:19:44.803 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.043225246, 2024-04-21T23:19:44.804 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.012557356, 2024-04-21T23:19:44.804 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.033357332, 2024-04-21T23:19:44.804 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.005139463, 2024-04-21T23:19:44.804 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:44.805 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 223, 2024-04-21T23:19:44.805 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.010414007, 2024-04-21T23:19:44.805 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000046699 2024-04-21T23:19:44.805 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:44.805 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:44.805 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:44.805 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:44.805 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:44.806 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:44.806 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 1446368, 2024-04-21T23:19:44.806 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 137648 2024-04-21T23:19:44.806 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:44.806 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T23:19:44.806 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 2, 2024-04-21T23:19:44.806 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 171, 2024-04-21T23:19:44.806 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 320, 2024-04-21T23:19:44.807 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 185376, 2024-04-21T23:19:44.807 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 5, 2024-04-21T23:19:44.807 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 3, 2024-04-21T23:19:44.807 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.010729302, 2024-04-21T23:19:44.807 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.007005539, 2024-04-21T23:19:44.807 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.001581217, 2024-04-21T23:19:44.807 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T23:19:44.807 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T23:19:44.807 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 168, 2024-04-21T23:19:44.808 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.011277490, 2024-04-21T23:19:44.808 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000067127 2024-04-21T23:19:44.808 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:44.808 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-04-21T23:19:44.808 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:44.808 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:44.808 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:44.808 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:44.808 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T23:19:44.809 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 185376 2024-04-21T23:19:44.809 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:44.809 INFO:teuthology.orchestra.run.smithi069.stdout: "cct": { 2024-04-21T23:19:44.809 INFO:teuthology.orchestra.run.smithi069.stdout: "total_workers": 0, 2024-04-21T23:19:44.809 INFO:teuthology.orchestra.run.smithi069.stdout: "unhealthy_workers": 0 2024-04-21T23:19:44.809 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:44.809 INFO:teuthology.orchestra.run.smithi069.stdout: "mempool": { 2024-04-21T23:19:44.809 INFO:teuthology.orchestra.run.smithi069.stdout: "bloom_filter_bytes": 0, 2024-04-21T23:19:44.809 INFO:teuthology.orchestra.run.smithi069.stdout: "bloom_filter_items": 0, 2024-04-21T23:19:44.810 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T23:19:44.810 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_alloc_items": 0, 2024-04-21T23:19:44.810 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T23:19:44.810 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_data_items": 0, 2024-04-21T23:19:44.810 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T23:19:44.810 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T23:19:44.810 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T23:19:44.810 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T23:19:44.811 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T23:19:44.811 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_other_items": 0, 2024-04-21T23:19:44.811 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T23:19:44.811 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T23:19:44.811 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_bytes": 0, 2024-04-21T23:19:44.811 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_items": 0, 2024-04-21T23:19:44.811 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_blob_bytes": 0, 2024-04-21T23:19:44.811 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_blob_items": 0, 2024-04-21T23:19:44.812 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T23:19:44.812 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T23:19:44.812 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T23:19:44.812 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T23:19:44.812 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T23:19:44.812 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_items": 0, 2024-04-21T23:19:44.812 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_txc_bytes": 0, 2024-04-21T23:19:44.812 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_txc_items": 0, 2024-04-21T23:19:44.812 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T23:19:44.812 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T23:19:44.813 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_bytes": 0, 2024-04-21T23:19:44.813 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_items": 0, 2024-04-21T23:19:44.813 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_bytes": 0, 2024-04-21T23:19:44.813 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_items": 0, 2024-04-21T23:19:44.813 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T23:19:44.813 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_reader_items": 0, 2024-04-21T23:19:44.813 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T23:19:44.813 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_writer_items": 0, 2024-04-21T23:19:44.814 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_anon_bytes": 148731, 2024-04-21T23:19:44.814 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_anon_items": 31, 2024-04-21T23:19:44.814 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_meta_bytes": 0, 2024-04-21T23:19:44.814 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_meta_items": 0, 2024-04-21T23:19:44.814 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_bytes": 0, 2024-04-21T23:19:44.814 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_items": 0, 2024-04-21T23:19:44.815 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mapbl_bytes": 0, 2024-04-21T23:19:44.815 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mapbl_items": 0, 2024-04-21T23:19:44.815 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pglog_bytes": 0, 2024-04-21T23:19:44.815 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pglog_items": 0, 2024-04-21T23:19:44.815 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_bytes": 0, 2024-04-21T23:19:44.815 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_items": 0, 2024-04-21T23:19:44.815 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T23:19:44.815 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_mapping_items": 0, 2024-04-21T23:19:44.816 INFO:teuthology.orchestra.run.smithi069.stdout: "pgmap_bytes": 0, 2024-04-21T23:19:44.816 INFO:teuthology.orchestra.run.smithi069.stdout: "pgmap_items": 0, 2024-04-21T23:19:44.816 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_co_bytes": 928, 2024-04-21T23:19:44.816 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_co_items": 2, 2024-04-21T23:19:44.816 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_1_bytes": 0, 2024-04-21T23:19:44.816 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_1_items": 0, 2024-04-21T23:19:44.817 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_2_bytes": 0, 2024-04-21T23:19:44.817 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_2_items": 0 2024-04-21T23:19:44.817 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-21T23:19:44.817 INFO:teuthology.orchestra.run.smithi069.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T23:19:44.817 INFO:teuthology.orchestra.run.smithi069.stdout: "val": 0, 2024-04-21T23:19:44.817 INFO:teuthology.orchestra.run.smithi069.stdout: "max": 104857600, 2024-04-21T23:19:44.817 INFO:teuthology.orchestra.run.smithi069.stdout: "get_started": 0, 2024-04-21T23:19:44.817 INFO:teuthology.orchestra.run.smithi069.stdout: "get": 236, 2024-04-21T23:19:44.817 INFO:teuthology.orchestra.run.smithi069.stdout: "get_sum": 1413360, 2024-04-21T23:19:44.818 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_fail": 0, 2024-04-21T23:19:44.818 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_success": 236, 2024-04-21T23:19:44.818 INFO:teuthology.orchestra.run.smithi069.stdout: "take": 0, 2024-04-21T23:19:44.818 INFO:teuthology.orchestra.run.smithi069.stdout: "take_sum": 0, 2024-04-21T23:19:44.818 INFO:teuthology.orchestra.run.smithi069.stdout: "put": 236, 2024-04-21T23:19:44.818 INFO:teuthology.orchestra.run.smithi069.stdout: "put_sum": 1413360, 2024-04-21T23:19:44.818 INFO:teuthology.orchestra.run.smithi069.stdout: "wait": { 2024-04-21T23:19:44.818 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-04-21T23:19:44.819 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-04-21T23:19:44.819 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-04-21T23:19:44.819 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-21T23:19:44.819 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-21T23:19:44.819 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-21T23:19:45.514 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-21T23:19:45.514 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-21T23:19:45.514 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-21T23:19:45.515 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-21T23:19:45.527 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-21T23:19:45.527 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T23:19:45.527 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T23:19:45.548 DEBUG:teuthology.orchestra.run.smithi044:> set -ex 2024-04-21T23:19:45.548 DEBUG:teuthology.orchestra.run.smithi044:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T23:19:45.609 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:19:45.764 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.766+0000 7f3039e0e640 1 -- 172.21.15.44:0/2509254720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3034107730 msgr2=0x7f3034107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:45.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.766+0000 7f3039e0e640 1 --2- 172.21.15.44:0/2509254720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3034107730 0x7f3034107b10 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f3024009930 tx=0x7f302402bc20 comp rx=0 tx=0).stop 2024-04-21T23:19:45.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.766+0000 7f3039e0e640 1 -- 172.21.15.44:0/2509254720 shutdown_connections 2024-04-21T23:19:45.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.766+0000 7f3039e0e640 1 --2- 172.21.15.44:0/2509254720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3034101720 0x7f3034101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:45.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.766+0000 7f3039e0e640 1 --2- 172.21.15.44:0/2509254720 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f30341020e0 0x7f303410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:45.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.766+0000 7f3039e0e640 1 --2- 172.21.15.44:0/2509254720 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3034107730 0x7f3034107b10 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:45.765 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.766+0000 7f3039e0e640 1 -- 172.21.15.44:0/2509254720 >> 172.21.15.44:0/2509254720 conn(0x7f30340fd230 msgr2=0x7f30340ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:45.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.766+0000 7f3039e0e640 1 -- 172.21.15.44:0/2509254720 shutdown_connections 2024-04-21T23:19:45.766 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.766+0000 7f3039e0e640 1 -- 172.21.15.44:0/2509254720 wait complete. 2024-04-21T23:19:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.767+0000 7f3039e0e640 1 Processor -- start 2024-04-21T23:19:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.767+0000 7f3039e0e640 1 -- start start 2024-04-21T23:19:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.767+0000 7f3039e0e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3034101720 0x7f3034073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.767+0000 7f3039e0e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30341020e0 0x7f3034073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.767+0000 7f3039e0e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3034107730 0x7f30340744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:45.768 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.767+0000 7f3039e0e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f3034111b00 con 0x7f3034107730 2024-04-21T23:19:45.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.767+0000 7f3039e0e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f3034111980 con 0x7f3034101720 2024-04-21T23:19:45.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.767+0000 7f3039e0e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3034111c80 con 0x7f30341020e0 2024-04-21T23:19:45.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.767+0000 7f3033fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3034107730 0x7f30340744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:45.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.767+0000 7f3033fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3034107730 0x7f30340744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47896/0 (socket says 172.21.15.44:47896) 2024-04-21T23:19:45.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.767+0000 7f3033fff640 1 -- 172.21.15.44:0/1329195024 learned_addr learned my addr 172.21.15.44:0/1329195024 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:45.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.767+0000 7f3032ffd640 1 --2- 172.21.15.44:0/1329195024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30341020e0 0x7f3034073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:45.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.768+0000 7f30337fe640 1 --2- 172.21.15.44:0/1329195024 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3034101720 0x7f3034073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:45.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.768+0000 7f3033fff640 1 -- 172.21.15.44:0/1329195024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30341020e0 msgr2=0x7f3034073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:45.769 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.768+0000 7f3033fff640 1 --2- 172.21.15.44:0/1329195024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30341020e0 0x7f3034073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:45.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.768+0000 7f3033fff640 1 -- 172.21.15.44:0/1329195024 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3034101720 msgr2=0x7f3034073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:45.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.768+0000 7f3033fff640 1 --2- 172.21.15.44:0/1329195024 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3034101720 0x7f3034073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:45.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.768+0000 7f3033fff640 1 -- 172.21.15.44:0/1329195024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30341a6520 con 0x7f3034107730 2024-04-21T23:19:45.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.768+0000 7f3032ffd640 1 --2- 172.21.15.44:0/1329195024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30341020e0 0x7f3034073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:19:45.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.768+0000 7f3033fff640 1 --2- 172.21.15.44:0/1329195024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3034107730 0x7f30340744c0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f302c00eaa0 tx=0x7f302c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:45.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.768+0000 7f3030ff9640 1 -- 172.21.15.44:0/1329195024 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c00cdb0 con 0x7f3034107730 2024-04-21T23:19:45.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.768+0000 7f3030ff9640 1 -- 172.21.15.44:0/1329195024 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f302c005020 con 0x7f3034107730 2024-04-21T23:19:45.770 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.768+0000 7f3039e0e640 1 -- 172.21.15.44:0/1329195024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30341a6810 con 0x7f3034107730 2024-04-21T23:19:45.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.768+0000 7f3030ff9640 1 -- 172.21.15.44:0/1329195024 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c0116b0 con 0x7f3034107730 2024-04-21T23:19:45.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.768+0000 7f3039e0e640 1 -- 172.21.15.44:0/1329195024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30341a6d10 con 0x7f3034107730 2024-04-21T23:19:45.771 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.769+0000 7f3039e0e640 1 -- 172.21.15.44:0/1329195024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3034068a40 con 0x7f3034107730 2024-04-21T23:19:45.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.770+0000 7f3030ff9640 1 -- 172.21.15.44:0/1329195024 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f302c005560 con 0x7f3034107730 2024-04-21T23:19:45.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.771+0000 7f3030ff9640 1 --2- 172.21.15.44:0/1329195024 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f301c0a2d30 0x7f301c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:45.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.771+0000 7f30337fe640 1 --2- 172.21.15.44:0/1329195024 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f301c0a2d30 0x7f301c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:45.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.771+0000 7f3030ff9640 1 -- 172.21.15.44:0/1329195024 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9533+0+0 (secure 0 0 0) 0x7f302c0cfab0 con 0x7f3034107730 2024-04-21T23:19:45.772 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.771+0000 7f30337fe640 1 --2- 172.21.15.44:0/1329195024 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f301c0a2d30 0x7f301c0a51f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3024006010 tx=0x7f3024000bc0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:45.773 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.775+0000 7f3030ff9640 1 -- 172.21.15.44:0/1329195024 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f302c016030 con 0x7f3034107730 2024-04-21T23:19:45.959 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.960+0000 7f3039e0e640 1 -- 172.21.15.44:0/1329195024 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f303402d170 con 0x7f3034107730 2024-04-21T23:19:45.960 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.962+0000 7f3030ff9640 1 -- 172.21.15.44:0/1329195024 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+25278 (secure 0 0 0) 0x7f302c09c2e0 con 0x7f3034107730 2024-04-21T23:19:45.961 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:19:45.961 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":110,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:19:23.219564+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":109,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":109,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":9,"to":11},{"from":5,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/3134653118":"2024-04-22T23:08:50.585326+0000","172.21.15.44:0/101322314":"2024-04-22T23:08:50.585326+0000","172.21.15.44:0/792524018":"2024-04-22T23:08:50.585326+0000","172.21.15.44:0/2303525779":"2024-04-22T23:08:50.585326+0000","172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/1417504446":"2024-04-22T23:08:50.585326+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/1417504446":"2024-04-22T23:08:50.585326+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:19:45.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.964+0000 7f3039e0e640 1 -- 172.21.15.44:0/1329195024 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f301c0a2d30 msgr2=0x7f301c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:45.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.964+0000 7f3039e0e640 1 --2- 172.21.15.44:0/1329195024 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f301c0a2d30 0x7f301c0a51f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3024006010 tx=0x7f3024000bc0 comp rx=0 tx=0).stop 2024-04-21T23:19:45.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.964+0000 7f3039e0e640 1 -- 172.21.15.44:0/1329195024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3034107730 msgr2=0x7f30340744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:45.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.964+0000 7f3039e0e640 1 --2- 172.21.15.44:0/1329195024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3034107730 0x7f30340744c0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f302c00eaa0 tx=0x7f302c00ef60 comp rx=0 tx=0).stop 2024-04-21T23:19:45.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.964+0000 7f3039e0e640 1 -- 172.21.15.44:0/1329195024 shutdown_connections 2024-04-21T23:19:45.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.964+0000 7f3039e0e640 1 --2- 172.21.15.44:0/1329195024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30341020e0 0x7f3034073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:45.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.964+0000 7f3039e0e640 1 --2- 172.21.15.44:0/1329195024 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f301c0a2d30 0x7f301c0a51f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:45.963 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.964+0000 7f3039e0e640 1 --2- 172.21.15.44:0/1329195024 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f3034107730 0x7f30340744c0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.964+0000 7f3039e0e640 1 --2- 172.21.15.44:0/1329195024 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3034101720 0x7f3034073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.964+0000 7f3039e0e640 1 -- 172.21.15.44:0/1329195024 >> 172.21.15.44:0/1329195024 conn(0x7f30340fd230 msgr2=0x7f30340fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.964+0000 7f3039e0e640 1 -- 172.21.15.44:0/1329195024 shutdown_connections 2024-04-21T23:19:45.964 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:45.964+0000 7f3039e0e640 1 -- 172.21.15.44:0/1329195024 wait complete. 2024-04-21T23:19:45.975 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T23:01:47.507503+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T23:06:23.993760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T23:06:24.954661+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 109, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T23:06:28.677618+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 109, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T23:19:45.976 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T23:19:46.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.085+0000 7fb193ab4640 1 -- 172.21.15.44:0/115322470 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb18c100070 msgr2=0x7fb18c1004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:46.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.085+0000 7fb193ab4640 1 --2- 172.21.15.44:0/115322470 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb18c100070 0x7fb18c1004f0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fb180009900 tx=0x7fb18002bc30 comp rx=0 tx=0).stop 2024-04-21T23:19:46.086 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.085+0000 7fb193ab4640 1 -- 172.21.15.44:0/115322470 shutdown_connections 2024-04-21T23:19:46.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.085+0000 7fb193ab4640 1 --2- 172.21.15.44:0/115322470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb18c100a30 0x7fb18c10e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.085+0000 7fb193ab4640 1 --2- 172.21.15.44:0/115322470 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb18c100070 0x7fb18c1004f0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.085+0000 7fb193ab4640 1 --2- 172.21.15.44:0/115322470 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb18c106870 0x7fb18c106c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.085+0000 7fb193ab4640 1 -- 172.21.15.44:0/115322470 >> 172.21.15.44:0/115322470 conn(0x7fb18c0758c0 msgr2=0x7fb18c075cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:46.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.085+0000 7fb193ab4640 1 -- 172.21.15.44:0/115322470 shutdown_connections 2024-04-21T23:19:46.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.085+0000 7fb193ab4640 1 -- 172.21.15.44:0/115322470 wait complete. 2024-04-21T23:19:46.087 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.086+0000 7fb193ab4640 1 Processor -- start 2024-04-21T23:19:46.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.086+0000 7fb193ab4640 1 -- start start 2024-04-21T23:19:46.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.086+0000 7fb193ab4640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb18c100070 0x7fb18c101250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:46.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.086+0000 7fb193ab4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb18c100a30 0x7fb18c101790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:46.088 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.086+0000 7fb193ab4640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb18c106870 0x7fb18c101cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:46.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.086+0000 7fb193ab4640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c111c80 con 0x7fb18c106870 2024-04-21T23:19:46.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.086+0000 7fb193ab4640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c111b00 con 0x7fb18c100070 2024-04-21T23:19:46.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.086+0000 7fb193ab4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c111e00 con 0x7fb18c100a30 2024-04-21T23:19:46.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.086+0000 7fb192051640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb18c106870 0x7fb18c101cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:46.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.086+0000 7fb19104f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb18c100a30 0x7fb18c101790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:46.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.086+0000 7fb192051640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb18c106870 0x7fb18c101cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47908/0 (socket says 172.21.15.44:47908) 2024-04-21T23:19:46.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.086+0000 7fb192051640 1 -- 172.21.15.44:0/2679989498 learned_addr learned my addr 172.21.15.44:0/2679989498 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:46.089 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.086+0000 7fb191850640 1 --2- 172.21.15.44:0/2679989498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb18c100070 0x7fb18c101250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:46.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.087+0000 7fb192051640 1 -- 172.21.15.44:0/2679989498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb18c100a30 msgr2=0x7fb18c101790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:46.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.087+0000 7fb192051640 1 --2- 172.21.15.44:0/2679989498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb18c100a30 0x7fb18c101790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.087+0000 7fb192051640 1 -- 172.21.15.44:0/2679989498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb18c100070 msgr2=0x7fb18c101250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:46.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.087+0000 7fb192051640 1 --2- 172.21.15.44:0/2679989498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb18c100070 0x7fb18c101250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.087+0000 7fb192051640 1 -- 172.21.15.44:0/2679989498 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb18c1a6540 con 0x7fb18c106870 2024-04-21T23:19:46.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.087+0000 7fb191850640 1 --2- 172.21.15.44:0/2679989498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb18c100070 0x7fb18c101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:19:46.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.087+0000 7fb19104f640 1 --2- 172.21.15.44:0/2679989498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb18c100a30 0x7fb18c101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:46.090 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.087+0000 7fb192051640 1 --2- 172.21.15.44:0/2679989498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb18c106870 0x7fb18c101cd0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fb18400eb10 tx=0x7fb18400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.087+0000 7fb18affd640 1 -- 172.21.15.44:0/2679989498 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb184004b20 con 0x7fb18c106870 2024-04-21T23:19:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.087+0000 7fb193ab4640 1 -- 172.21.15.44:0/2679989498 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb18c1a6830 con 0x7fb18c106870 2024-04-21T23:19:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.087+0000 7fb18affd640 1 -- 172.21.15.44:0/2679989498 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb184004cc0 con 0x7fb18c106870 2024-04-21T23:19:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.087+0000 7fb18affd640 1 -- 172.21.15.44:0/2679989498 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb184010760 con 0x7fb18c106870 2024-04-21T23:19:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.087+0000 7fb193ab4640 1 -- 172.21.15.44:0/2679989498 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb18c1a6d50 con 0x7fb18c106870 2024-04-21T23:19:46.091 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.088+0000 7fb193ab4640 1 -- 172.21.15.44:0/2679989498 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb18c068a40 con 0x7fb18c106870 2024-04-21T23:19:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.089+0000 7fb18affd640 1 -- 172.21.15.44:0/2679989498 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb184005020 con 0x7fb18c106870 2024-04-21T23:19:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.090+0000 7fb18affd640 1 --2- 172.21.15.44:0/2679989498 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb1740a2d30 0x7fb1740a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:46.092 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.090+0000 7fb191850640 1 --2- 172.21.15.44:0/2679989498 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb1740a2d30 0x7fb1740a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:46.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.090+0000 7fb18affd640 1 -- 172.21.15.44:0/2679989498 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9533+0+0 (secure 0 0 0) 0x7fb1840cfe60 con 0x7fb18c106870 2024-04-21T23:19:46.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.090+0000 7fb191850640 1 --2- 172.21.15.44:0/2679989498 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb1740a2d30 0x7fb1740a51f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb17c005d90 tx=0x7fb17c005d20 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:46.093 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.094+0000 7fb18affd640 1 -- 172.21.15.44:0/2679989498 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fb18409c630 con 0x7fb18c106870 2024-04-21T23:19:46.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.280+0000 7fb193ab4640 1 -- 172.21.15.44:0/2679989498 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb18c109d60 con 0x7fb18c106870 2024-04-21T23:19:46.279 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.281+0000 7fb18affd640 1 -- 172.21.15.44:0/2679989498 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) ==== 93+0+10 (secure 0 0 0) 0x7fb18409bf30 con 0x7fb18c106870 2024-04-21T23:19:46.279 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 1 2024-04-21T23:19:46.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.283+0000 7fb193ab4640 1 -- 172.21.15.44:0/2679989498 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb1740a2d30 msgr2=0x7fb1740a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:46.281 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.283+0000 7fb193ab4640 1 --2- 172.21.15.44:0/2679989498 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb1740a2d30 0x7fb1740a51f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb17c005d90 tx=0x7fb17c005d20 comp rx=0 tx=0).stop 2024-04-21T23:19:46.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.283+0000 7fb193ab4640 1 -- 172.21.15.44:0/2679989498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb18c106870 msgr2=0x7fb18c101cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:46.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.283+0000 7fb193ab4640 1 --2- 172.21.15.44:0/2679989498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb18c106870 0x7fb18c101cd0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fb18400eb10 tx=0x7fb18400c2d0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.283+0000 7fb193ab4640 1 -- 172.21.15.44:0/2679989498 shutdown_connections 2024-04-21T23:19:46.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.283+0000 7fb193ab4640 1 --2- 172.21.15.44:0/2679989498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb18c100a30 0x7fb18c101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.283+0000 7fb193ab4640 1 --2- 172.21.15.44:0/2679989498 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fb1740a2d30 0x7fb1740a51f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.283+0000 7fb193ab4640 1 --2- 172.21.15.44:0/2679989498 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fb18c106870 0x7fb18c101cd0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.283+0000 7fb193ab4640 1 --2- 172.21.15.44:0/2679989498 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb18c100070 0x7fb18c101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.282 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.283+0000 7fb193ab4640 1 -- 172.21.15.44:0/2679989498 >> 172.21.15.44:0/2679989498 conn(0x7fb18c0758c0 msgr2=0x7fb18c104680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:46.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.283+0000 7fb193ab4640 1 -- 172.21.15.44:0/2679989498 shutdown_connections 2024-04-21T23:19:46.283 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.283+0000 7fb193ab4640 1 -- 172.21.15.44:0/2679989498 wait complete. 2024-04-21T23:19:46.292 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T23:19:46.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.398+0000 7f9ab6e88640 1 -- 172.21.15.44:0/738435473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ab01029b0 msgr2=0x7f9ab0102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:46.399 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.398+0000 7f9ab6e88640 1 --2- 172.21.15.44:0/738435473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ab01029b0 0x7f9ab0102d90 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009900 tx=0x7f9aac02bcd0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.398+0000 7f9ab6e88640 1 -- 172.21.15.44:0/738435473 shutdown_connections 2024-04-21T23:19:46.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.398+0000 7f9ab6e88640 1 --2- 172.21.15.44:0/738435473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ab01032d0 0x7f9ab010b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.398+0000 7f9ab6e88640 1 --2- 172.21.15.44:0/738435473 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9ab010bcf0 0x7f9ab010e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.398+0000 7f9ab6e88640 1 --2- 172.21.15.44:0/738435473 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ab01029b0 0x7f9ab0102d90 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.399+0000 7f9ab6e88640 1 -- 172.21.15.44:0/738435473 >> 172.21.15.44:0/738435473 conn(0x7f9ab00777f0 msgr2=0x7f9ab0077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:46.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.399+0000 7f9ab6e88640 1 -- 172.21.15.44:0/738435473 shutdown_connections 2024-04-21T23:19:46.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.399+0000 7f9ab6e88640 1 -- 172.21.15.44:0/738435473 wait complete. 2024-04-21T23:19:46.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.399+0000 7f9ab6e88640 1 Processor -- start 2024-04-21T23:19:46.400 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.399+0000 7f9ab6e88640 1 -- start start 2024-04-21T23:19:46.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.399+0000 7f9ab6e88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ab01029b0 0x7f9ab0100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:46.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.399+0000 7f9ab6e88640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9ab01032d0 0x7f9ab0100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:46.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.399+0000 7f9ab6e88640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ab010bcf0 0x7f9ab0101080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:46.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.399+0000 7f9ab6e88640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab0111b30 con 0x7f9ab010bcf0 2024-04-21T23:19:46.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.399+0000 7f9ab6e88640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab01119b0 con 0x7f9ab01032d0 2024-04-21T23:19:46.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.399+0000 7f9ab6e88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab0111cb0 con 0x7f9ab01029b0 2024-04-21T23:19:46.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.400+0000 7f9ab5425640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ab010bcf0 0x7f9ab0101080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:46.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.400+0000 7f9ab5425640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ab010bcf0 0x7f9ab0101080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47924/0 (socket says 172.21.15.44:47924) 2024-04-21T23:19:46.401 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.400+0000 7f9ab5425640 1 -- 172.21.15.44:0/4263176209 learned_addr learned my addr 172.21.15.44:0/4263176209 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:46.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.400+0000 7f9aa7fff640 1 --2- 172.21.15.44:0/4263176209 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9ab01032d0 0x7f9ab0100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:46.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.400+0000 7f9ab4c24640 1 --2- 172.21.15.44:0/4263176209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ab01029b0 0x7f9ab0100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:46.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.400+0000 7f9ab5425640 1 -- 172.21.15.44:0/4263176209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ab01029b0 msgr2=0x7f9ab0100600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:46.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.400+0000 7f9ab5425640 1 --2- 172.21.15.44:0/4263176209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ab01029b0 0x7f9ab0100600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.400+0000 7f9ab5425640 1 -- 172.21.15.44:0/4263176209 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9ab01032d0 msgr2=0x7f9ab0100b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:46.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.400+0000 7f9ab5425640 1 --2- 172.21.15.44:0/4263176209 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9ab01032d0 0x7f9ab0100b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.400+0000 7f9ab5425640 1 -- 172.21.15.44:0/4263176209 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab01a6570 con 0x7f9ab010bcf0 2024-04-21T23:19:46.402 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.400+0000 7f9ab5425640 1 --2- 172.21.15.44:0/4263176209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ab010bcf0 0x7f9ab0101080 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f9aa800cab0 tx=0x7f9aa800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:46.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.400+0000 7f9aa5ffb640 1 -- 172.21.15.44:0/4263176209 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa800adc0 con 0x7f9ab010bcf0 2024-04-21T23:19:46.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.400+0000 7f9aa5ffb640 1 -- 172.21.15.44:0/4263176209 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9aa800f930 con 0x7f9ab010bcf0 2024-04-21T23:19:46.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.400+0000 7f9ab6e88640 1 -- 172.21.15.44:0/4263176209 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ab01a6860 con 0x7f9ab010bcf0 2024-04-21T23:19:46.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.400+0000 7f9aa5ffb640 1 -- 172.21.15.44:0/4263176209 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8018ae0 con 0x7f9ab010bcf0 2024-04-21T23:19:46.403 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.400+0000 7f9ab6e88640 1 -- 172.21.15.44:0/4263176209 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ab01a6d60 con 0x7f9ab010bcf0 2024-04-21T23:19:46.404 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.401+0000 7f9ab6e88640 1 -- 172.21.15.44:0/4263176209 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ab00685f0 con 0x7f9ab010bcf0 2024-04-21T23:19:46.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.402+0000 7f9aa5ffb640 1 -- 172.21.15.44:0/4263176209 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f9aa800fae0 con 0x7f9ab010bcf0 2024-04-21T23:19:46.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.403+0000 7f9aa5ffb640 1 --2- 172.21.15.44:0/4263176209 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9a980a2d00 0x7f9a980a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:46.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.403+0000 7f9ab4c24640 1 --2- 172.21.15.44:0/4263176209 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9a980a2d00 0x7f9a980a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:46.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.403+0000 7f9aa5ffb640 1 -- 172.21.15.44:0/4263176209 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9533+0+0 (secure 0 0 0) 0x7f9aa80cffd0 con 0x7f9ab010bcf0 2024-04-21T23:19:46.405 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.404+0000 7f9ab4c24640 1 --2- 172.21.15.44:0/4263176209 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9a980a2d00 0x7f9a980a51c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9aac000c00 tx=0x7f9aac02cf90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:46.406 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.408+0000 7f9aa5ffb640 1 -- 172.21.15.44:0/4263176209 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f9aa8014030 con 0x7f9ab010bcf0 2024-04-21T23:19:46.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.591+0000 7f9ab6e88640 1 -- 172.21.15.44:0/4263176209 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9ab002d170 con 0x7f9ab010bcf0 2024-04-21T23:19:46.590 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.592+0000 7f9aa5ffb640 1 -- 172.21.15.44:0/4263176209 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v110) ==== 104+0+11 (secure 0 0 0) 0x7f9aa809c800 con 0x7f9ab010bcf0 2024-04-21T23:19:46.590 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:19:46.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.594+0000 7f9ab6e88640 1 -- 172.21.15.44:0/4263176209 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9a980a2d00 msgr2=0x7f9a980a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:46.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.594+0000 7f9ab6e88640 1 --2- 172.21.15.44:0/4263176209 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9a980a2d00 0x7f9a980a51c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9aac000c00 tx=0x7f9aac02cf90 comp rx=0 tx=0).stop 2024-04-21T23:19:46.593 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.594+0000 7f9ab6e88640 1 -- 172.21.15.44:0/4263176209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ab010bcf0 msgr2=0x7f9ab0101080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:46.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.594+0000 7f9ab6e88640 1 --2- 172.21.15.44:0/4263176209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ab010bcf0 0x7f9ab0101080 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f9aa800cab0 tx=0x7f9aa800cf70 comp rx=0 tx=0).stop 2024-04-21T23:19:46.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.594+0000 7f9ab6e88640 1 -- 172.21.15.44:0/4263176209 shutdown_connections 2024-04-21T23:19:46.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.594+0000 7f9ab6e88640 1 --2- 172.21.15.44:0/4263176209 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f9a980a2d00 0x7f9a980a51c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.594+0000 7f9ab6e88640 1 --2- 172.21.15.44:0/4263176209 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9ab010bcf0 0x7f9ab0101080 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.594+0000 7f9ab6e88640 1 --2- 172.21.15.44:0/4263176209 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9ab01032d0 0x7f9ab0100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.594 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.594+0000 7f9ab6e88640 1 --2- 172.21.15.44:0/4263176209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ab01029b0 0x7f9ab0100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.594+0000 7f9ab6e88640 1 -- 172.21.15.44:0/4263176209 >> 172.21.15.44:0/4263176209 conn(0x7f9ab00777f0 msgr2=0x7f9ab00fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:46.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.594+0000 7f9ab6e88640 1 -- 172.21.15.44:0/4263176209 shutdown_connections 2024-04-21T23:19:46.595 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.594+0000 7f9ab6e88640 1 -- 172.21.15.44:0/4263176209 wait complete. 2024-04-21T23:19:46.602 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:46 smithi044 ceph-mon[27934]: pgmap v345: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:19:46.603 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:46 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1329195024' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:19:46.604 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T23:19:46.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.711+0000 7f7a9b59e640 1 -- 172.21.15.44:0/3742512407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a9c100df0 msgr2=0x7f7a9c10d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:46.713 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.711+0000 7f7a9b59e640 1 --2- 172.21.15.44:0/3742512407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a9c100df0 0x7f7a9c10d300 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009cd0 tx=0x7f7a9402c0f0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.712+0000 7f7a9b59e640 1 -- 172.21.15.44:0/3742512407 shutdown_connections 2024-04-21T23:19:46.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.712+0000 7f7a9b59e640 1 --2- 172.21.15.44:0/3742512407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a9c100430 0x7f7a9c1008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.712+0000 7f7a9b59e640 1 --2- 172.21.15.44:0/3742512407 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a9c100df0 0x7f7a9c10d300 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.712+0000 7f7a9b59e640 1 --2- 172.21.15.44:0/3742512407 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a9c1064d0 0x7f7a9c1068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.712+0000 7f7a9b59e640 1 -- 172.21.15.44:0/3742512407 >> 172.21.15.44:0/3742512407 conn(0x7f7a9c0771f0 msgr2=0x7f7a9c0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:46.714 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.712+0000 7f7a9b59e640 1 -- 172.21.15.44:0/3742512407 shutdown_connections 2024-04-21T23:19:46.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.712+0000 7f7a9b59e640 1 -- 172.21.15.44:0/3742512407 wait complete. 2024-04-21T23:19:46.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.712+0000 7f7a9b59e640 1 Processor -- start 2024-04-21T23:19:46.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.712+0000 7f7a9b59e640 1 -- start start 2024-04-21T23:19:46.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a9b59e640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a9c100430 0x7f7a9c0fc670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:46.715 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a9b59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a9c100df0 0x7f7a9c0fcbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:46.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a9b59e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a9c1064d0 0x7f7a9c0fd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:46.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a9b59e640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c110850 con 0x7f7a9c100df0 2024-04-21T23:19:46.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a9b59e640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c1106d0 con 0x7f7a9c100430 2024-04-21T23:19:46.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a9b59e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c1109d0 con 0x7f7a9c1064d0 2024-04-21T23:19:46.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a93fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a9c100df0 0x7f7a9c0fcbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:46.716 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a93fff640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a9c100df0 0x7f7a9c0fcbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47930/0 (socket says 172.21.15.44:47930) 2024-04-21T23:19:46.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a93fff640 1 -- 172.21.15.44:0/1238050939 learned_addr learned my addr 172.21.15.44:0/1238050939 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:46.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a9a59c640 1 --2- 172.21.15.44:0/1238050939 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a9c100430 0x7f7a9c0fc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:46.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a93fff640 1 -- 172.21.15.44:0/1238050939 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a9c1064d0 msgr2=0x7f7a9c0fd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:46.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a93fff640 1 --2- 172.21.15.44:0/1238050939 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a9c1064d0 0x7f7a9c0fd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a93fff640 1 -- 172.21.15.44:0/1238050939 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a9c100430 msgr2=0x7f7a9c0fc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:46.717 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a93fff640 1 --2- 172.21.15.44:0/1238050939 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a9c100430 0x7f7a9c0fc670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a93fff640 1 -- 172.21.15.44:0/1238050939 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a9c1a6260 con 0x7f7a9c100df0 2024-04-21T23:19:46.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a9a59c640 1 --2- 172.21.15.44:0/1238050939 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a9c100430 0x7f7a9c0fc670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:46.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a93fff640 1 --2- 172.21.15.44:0/1238050939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a9c100df0 0x7f7a9c0fcbb0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400c920 tx=0x7f7a8400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:46.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a937fe640 1 -- 172.21.15.44:0/1238050939 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a84007c70 con 0x7f7a9c100df0 2024-04-21T23:19:46.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a9b59e640 1 -- 172.21.15.44:0/1238050939 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a9c1a6550 con 0x7f7a9c100df0 2024-04-21T23:19:46.718 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a937fe640 1 -- 172.21.15.44:0/1238050939 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a84007e10 con 0x7f7a9c100df0 2024-04-21T23:19:46.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a937fe640 1 -- 172.21.15.44:0/1238050939 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a84018890 con 0x7f7a9c100df0 2024-04-21T23:19:46.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.713+0000 7f7a9b59e640 1 -- 172.21.15.44:0/1238050939 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a9c1a6af0 con 0x7f7a9c100df0 2024-04-21T23:19:46.719 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.714+0000 7f7a9b59e640 1 -- 172.21.15.44:0/1238050939 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a9c04f500 con 0x7f7a9c100df0 2024-04-21T23:19:46.720 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:46 smithi055 ceph-mon[31117]: pgmap v345: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:19:46.720 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:46 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1329195024' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:19:46.721 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:46 smithi069 ceph-mon[31816]: pgmap v345: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T23:19:46.721 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:46 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1329195024' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:19:46.722 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.715+0000 7f7a937fe640 1 -- 172.21.15.44:0/1238050939 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7a84018a30 con 0x7f7a9c100df0 2024-04-21T23:19:46.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.716+0000 7f7a937fe640 1 --2- 172.21.15.44:0/1238050939 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7a880a2d30 0x7f7a880a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:46.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.716+0000 7f7a9a59c640 1 --2- 172.21.15.44:0/1238050939 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7a880a2d30 0x7f7a880a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:46.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.716+0000 7f7a937fe640 1 -- 172.21.15.44:0/1238050939 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9533+0+0 (secure 0 0 0) 0x7f7a840cfa60 con 0x7f7a9c100df0 2024-04-21T23:19:46.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.717+0000 7f7a9a59c640 1 --2- 172.21.15.44:0/1238050939 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7a880a2d30 0x7f7a880a51f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c006fd0 tx=0x7f7a8c008040 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:46.723 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.720+0000 7f7a937fe640 1 -- 172.21.15.44:0/1238050939 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f7a84014030 con 0x7f7a9c100df0 2024-04-21T23:19:46.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.905+0000 7f7a9b59e640 1 -- 172.21.15.44:0/1238050939 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7a9c02d170 con 0x7f7a9c100df0 2024-04-21T23:19:46.904 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.906+0000 7f7a937fe640 1 -- 172.21.15.44:0/1238050939 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v110) ==== 100+0+11 (secure 0 0 0) 0x7f7a8401f300 con 0x7f7a9c100df0 2024-04-21T23:19:46.905 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 64 2024-04-21T23:19:46.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.908+0000 7f7a9b59e640 1 -- 172.21.15.44:0/1238050939 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7a880a2d30 msgr2=0x7f7a880a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:46.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.908+0000 7f7a9b59e640 1 --2- 172.21.15.44:0/1238050939 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7a880a2d30 0x7f7a880a51f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c006fd0 tx=0x7f7a8c008040 comp rx=0 tx=0).stop 2024-04-21T23:19:46.907 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.908+0000 7f7a9b59e640 1 -- 172.21.15.44:0/1238050939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a9c100df0 msgr2=0x7f7a9c0fcbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:46.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.908+0000 7f7a9b59e640 1 --2- 172.21.15.44:0/1238050939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a9c100df0 0x7f7a9c0fcbb0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400c920 tx=0x7f7a8400cde0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.908+0000 7f7a9b59e640 1 -- 172.21.15.44:0/1238050939 shutdown_connections 2024-04-21T23:19:46.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.908+0000 7f7a9b59e640 1 --2- 172.21.15.44:0/1238050939 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a9c1064d0 0x7f7a9c0fd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.908+0000 7f7a9b59e640 1 --2- 172.21.15.44:0/1238050939 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f7a880a2d30 0x7f7a880a51f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.908+0000 7f7a9b59e640 1 --2- 172.21.15.44:0/1238050939 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f7a9c100df0 0x7f7a9c0fcbb0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.908+0000 7f7a9b59e640 1 --2- 172.21.15.44:0/1238050939 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7a9c100430 0x7f7a9c0fc670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:46.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.908+0000 7f7a9b59e640 1 -- 172.21.15.44:0/1238050939 >> 172.21.15.44:0/1238050939 conn(0x7f7a9c0771f0 msgr2=0x7f7a9c068ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:46.908 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.908+0000 7f7a9b59e640 1 -- 172.21.15.44:0/1238050939 shutdown_connections 2024-04-21T23:19:46.909 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:46.908+0000 7f7a9b59e640 1 -- 172.21.15.44:0/1238050939 wait complete. 2024-04-21T23:19:46.918 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T23:19:47.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.025+0000 7feb5c3c9640 1 -- 172.21.15.44:0/1049661912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb54103bb0 msgr2=0x7feb5410e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:47.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.025+0000 7feb5c3c9640 1 --2- 172.21.15.44:0/1049661912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb54103bb0 0x7feb5410e6e0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7feb4c009900 tx=0x7feb4c02bd10 comp rx=0 tx=0).stop 2024-04-21T23:19:47.026 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.025+0000 7feb5c3c9640 1 -- 172.21.15.44:0/1049661912 shutdown_connections 2024-04-21T23:19:47.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.025+0000 7feb5c3c9640 1 --2- 172.21.15.44:0/1049661912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb54103210 0x7feb54103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:47.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.025+0000 7feb5c3c9640 1 --2- 172.21.15.44:0/1049661912 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb54103bb0 0x7feb5410e6e0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:47.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.025+0000 7feb5c3c9640 1 --2- 172.21.15.44:0/1049661912 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb54101ad0 0x7feb54101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:47.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.025+0000 7feb5c3c9640 1 -- 172.21.15.44:0/1049661912 >> 172.21.15.44:0/1049661912 conn(0x7feb540fd200 msgr2=0x7feb540ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:47.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.025+0000 7feb5c3c9640 1 -- 172.21.15.44:0/1049661912 shutdown_connections 2024-04-21T23:19:47.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.025+0000 7feb5c3c9640 1 -- 172.21.15.44:0/1049661912 wait complete. 2024-04-21T23:19:47.027 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.025+0000 7feb5c3c9640 1 Processor -- start 2024-04-21T23:19:47.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.026+0000 7feb5c3c9640 1 -- start start 2024-04-21T23:19:47.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.026+0000 7feb5c3c9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb54101ad0 0x7feb54073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:47.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.026+0000 7feb5c3c9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb54103210 0x7feb54073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:47.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.026+0000 7feb5c3c9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb54103bb0 0x7feb540744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:47.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.026+0000 7feb5c3c9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7feb54111c20 con 0x7feb54103bb0 2024-04-21T23:19:47.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.026+0000 7feb5c3c9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7feb54111aa0 con 0x7feb54103210 2024-04-21T23:19:47.028 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.026+0000 7feb5c3c9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7feb54111da0 con 0x7feb54101ad0 2024-04-21T23:19:47.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.026+0000 7feb5a966640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb54103bb0 0x7feb540744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:47.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.026+0000 7feb5a966640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb54103bb0 0x7feb540744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47944/0 (socket says 172.21.15.44:47944) 2024-04-21T23:19:47.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.026+0000 7feb5a966640 1 -- 172.21.15.44:0/2031087140 learned_addr learned my addr 172.21.15.44:0/2031087140 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:47.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.026+0000 7feb59964640 1 --2- 172.21.15.44:0/2031087140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb54103210 0x7feb54073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:47.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.026+0000 7feb5a165640 1 --2- 172.21.15.44:0/2031087140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb54101ad0 0x7feb54073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:47.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.026+0000 7feb5a966640 1 -- 172.21.15.44:0/2031087140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb54101ad0 msgr2=0x7feb54073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:47.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.026+0000 7feb5a966640 1 --2- 172.21.15.44:0/2031087140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb54101ad0 0x7feb54073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:47.029 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.026+0000 7feb5a966640 1 -- 172.21.15.44:0/2031087140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb54103210 msgr2=0x7feb54073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:47.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.026+0000 7feb5a966640 1 --2- 172.21.15.44:0/2031087140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb54103210 0x7feb54073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:47.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.026+0000 7feb5a966640 1 -- 172.21.15.44:0/2031087140 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb541a6540 con 0x7feb54103bb0 2024-04-21T23:19:47.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.026+0000 7feb59964640 1 --2- 172.21.15.44:0/2031087140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb54103210 0x7feb54073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:19:47.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.026+0000 7feb5a165640 1 --2- 172.21.15.44:0/2031087140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb54101ad0 0x7feb54073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:47.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.026+0000 7feb5a966640 1 --2- 172.21.15.44:0/2031087140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb54103bb0 0x7feb540744c0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7feb4c009230 tx=0x7feb4c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:47.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.027+0000 7feb537fe640 1 -- 172.21.15.44:0/2031087140 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4c042020 con 0x7feb54103bb0 2024-04-21T23:19:47.030 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.027+0000 7feb537fe640 1 -- 172.21.15.44:0/2031087140 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb4c0054e0 con 0x7feb54103bb0 2024-04-21T23:19:47.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.027+0000 7feb537fe640 1 -- 172.21.15.44:0/2031087140 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4c003a50 con 0x7feb54103bb0 2024-04-21T23:19:47.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.027+0000 7feb5c3c9640 1 -- 172.21.15.44:0/2031087140 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb541a67d0 con 0x7feb54103bb0 2024-04-21T23:19:47.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.027+0000 7feb5c3c9640 1 -- 172.21.15.44:0/2031087140 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb541a6cd0 con 0x7feb54103bb0 2024-04-21T23:19:47.031 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.028+0000 7feb5c3c9640 1 -- 172.21.15.44:0/2031087140 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb54068a40 con 0x7feb54103bb0 2024-04-21T23:19:47.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.029+0000 7feb537fe640 1 -- 172.21.15.44:0/2031087140 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7feb4c045020 con 0x7feb54103bb0 2024-04-21T23:19:47.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.029+0000 7feb537fe640 1 --2- 172.21.15.44:0/2031087140 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7feb3c0a2d00 0x7feb3c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:47.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.030+0000 7feb5a165640 1 --2- 172.21.15.44:0/2031087140 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7feb3c0a2d00 0x7feb3c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:47.032 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.030+0000 7feb537fe640 1 -- 172.21.15.44:0/2031087140 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9533+0+0 (secure 0 0 0) 0x7feb4c0f18f0 con 0x7feb54103bb0 2024-04-21T23:19:47.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.030+0000 7feb5a165640 1 --2- 172.21.15.44:0/2031087140 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7feb3c0a2d00 0x7feb3c0a51c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feb44005fd0 tx=0x7feb44009500 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:47.033 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.033+0000 7feb537fe640 1 -- 172.21.15.44:0/2031087140 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7feb4c04c020 con 0x7feb54103bb0 2024-04-21T23:19:47.219 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.221+0000 7feb5c3c9640 1 -- 172.21.15.44:0/2031087140 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7feb54109d30 con 0x7feb54103bb0 2024-04-21T23:19:47.220 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.221+0000 7feb537fe640 1 -- 172.21.15.44:0/2031087140 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v110) ==== 103+0+11 (secure 0 0 0) 0x7feb4c0be230 con 0x7feb54103bb0 2024-04-21T23:19:47.220 INFO:teuthology.orchestra.run.smithi044.stdout:pg_num: 32 2024-04-21T23:19:47.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.223+0000 7feb5c3c9640 1 -- 172.21.15.44:0/2031087140 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7feb3c0a2d00 msgr2=0x7feb3c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:47.222 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.223+0000 7feb5c3c9640 1 --2- 172.21.15.44:0/2031087140 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7feb3c0a2d00 0x7feb3c0a51c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feb44005fd0 tx=0x7feb44009500 comp rx=0 tx=0).stop 2024-04-21T23:19:47.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.223+0000 7feb5c3c9640 1 -- 172.21.15.44:0/2031087140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb54103bb0 msgr2=0x7feb540744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:47.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.223+0000 7feb5c3c9640 1 --2- 172.21.15.44:0/2031087140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb54103bb0 0x7feb540744c0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7feb4c009230 tx=0x7feb4c0042f0 comp rx=0 tx=0).stop 2024-04-21T23:19:47.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.223+0000 7feb5c3c9640 1 -- 172.21.15.44:0/2031087140 shutdown_connections 2024-04-21T23:19:47.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.223+0000 7feb5c3c9640 1 --2- 172.21.15.44:0/2031087140 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7feb3c0a2d00 0x7feb3c0a51c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:47.223 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.223+0000 7feb5c3c9640 1 --2- 172.21.15.44:0/2031087140 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb54103bb0 0x7feb540744c0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:47.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.223+0000 7feb5c3c9640 1 --2- 172.21.15.44:0/2031087140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb54103210 0x7feb54073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:47.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.223+0000 7feb5c3c9640 1 --2- 172.21.15.44:0/2031087140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb54101ad0 0x7feb54073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:47.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.223+0000 7feb5c3c9640 1 -- 172.21.15.44:0/2031087140 >> 172.21.15.44:0/2031087140 conn(0x7feb540fd200 msgr2=0x7feb541075b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:47.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.223+0000 7feb5c3c9640 1 -- 172.21.15.44:0/2031087140 shutdown_connections 2024-04-21T23:19:47.224 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.223+0000 7feb5c3c9640 1 -- 172.21.15.44:0/2031087140 wait complete. 2024-04-21T23:19:47.233 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-21T23:19:47.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.339+0000 7f1b5c3e5640 1 -- 172.21.15.44:0/1571256337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b54101ad0 msgr2=0x7f1b54101eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:47.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.339+0000 7f1b5c3e5640 1 --2- 172.21.15.44:0/1571256337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b54101ad0 0x7f1b54101eb0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f1b50009900 tx=0x7f1b5002bd10 comp rx=0 tx=0).stop 2024-04-21T23:19:47.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.339+0000 7f1b5c3e5640 1 -- 172.21.15.44:0/1571256337 shutdown_connections 2024-04-21T23:19:47.340 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.339+0000 7f1b5c3e5640 1 --2- 172.21.15.44:0/1571256337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b54103210 0x7f1b54103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:47.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.339+0000 7f1b5c3e5640 1 --2- 172.21.15.44:0/1571256337 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b54103bb0 0x7f1b5410e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:47.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.339+0000 7f1b5c3e5640 1 --2- 172.21.15.44:0/1571256337 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b54101ad0 0x7f1b54101eb0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:47.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.339+0000 7f1b5c3e5640 1 -- 172.21.15.44:0/1571256337 >> 172.21.15.44:0/1571256337 conn(0x7f1b540fd200 msgr2=0x7f1b540ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:47.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.339+0000 7f1b5c3e5640 1 -- 172.21.15.44:0/1571256337 shutdown_connections 2024-04-21T23:19:47.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.339+0000 7f1b5c3e5640 1 -- 172.21.15.44:0/1571256337 wait complete. 2024-04-21T23:19:47.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.340+0000 7f1b5c3e5640 1 Processor -- start 2024-04-21T23:19:47.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.340+0000 7f1b5c3e5640 1 -- start start 2024-04-21T23:19:47.341 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.340+0000 7f1b5c3e5640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b54101ad0 0x7f1b541980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:47.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.340+0000 7f1b5c3e5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b54103210 0x7f1b54198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:47.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.340+0000 7f1b5c3e5640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b54103bb0 0x7f1b5419c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:47.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.340+0000 7f1b5c3e5640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f1b54111c20 con 0x7f1b54101ad0 2024-04-21T23:19:47.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.340+0000 7f1b5c3e5640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f1b54111aa0 con 0x7f1b54103bb0 2024-04-21T23:19:47.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.340+0000 7f1b5c3e5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1b54111da0 con 0x7f1b54103210 2024-04-21T23:19:47.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.340+0000 7f1b5a181640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b54101ad0 0x7f1b541980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:47.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.340+0000 7f1b5a181640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b54101ad0 0x7f1b541980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47958/0 (socket says 172.21.15.44:47958) 2024-04-21T23:19:47.342 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.340+0000 7f1b5a181640 1 -- 172.21.15.44:0/2654806851 learned_addr learned my addr 172.21.15.44:0/2654806851 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:47.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.340+0000 7f1b5a982640 1 --2- 172.21.15.44:0/2654806851 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b54103bb0 0x7f1b5419c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:47.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.340+0000 7f1b59980640 1 --2- 172.21.15.44:0/2654806851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b54103210 0x7f1b54198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:47.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.340+0000 7f1b5a181640 1 -- 172.21.15.44:0/2654806851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b54103210 msgr2=0x7f1b54198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:47.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.340+0000 7f1b5a181640 1 --2- 172.21.15.44:0/2654806851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b54103210 0x7f1b54198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:47.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.340+0000 7f1b5a181640 1 -- 172.21.15.44:0/2654806851 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b54103bb0 msgr2=0x7f1b5419c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:47.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.340+0000 7f1b5a181640 1 --2- 172.21.15.44:0/2654806851 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b54103bb0 0x7f1b5419c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:47.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.340+0000 7f1b5a181640 1 -- 172.21.15.44:0/2654806851 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b5419d080 con 0x7f1b54101ad0 2024-04-21T23:19:47.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.340+0000 7f1b5a181640 1 --2- 172.21.15.44:0/2654806851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b54101ad0 0x7f1b541980e0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f1b50006010 tx=0x7f1b500043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:47.343 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.341+0000 7f1b4b7fe640 1 -- 172.21.15.44:0/2654806851 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b50042020 con 0x7f1b54101ad0 2024-04-21T23:19:47.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.341+0000 7f1b4b7fe640 1 -- 172.21.15.44:0/2654806851 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b500054e0 con 0x7f1b54101ad0 2024-04-21T23:19:47.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.341+0000 7f1b5c3e5640 1 -- 172.21.15.44:0/2654806851 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b5419d310 con 0x7f1b54101ad0 2024-04-21T23:19:47.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.341+0000 7f1b4b7fe640 1 -- 172.21.15.44:0/2654806851 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b50003bc0 con 0x7f1b54101ad0 2024-04-21T23:19:47.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.341+0000 7f1b5c3e5640 1 -- 172.21.15.44:0/2654806851 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b541a4c00 con 0x7f1b54101ad0 2024-04-21T23:19:47.344 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.342+0000 7f1b5c3e5640 1 -- 172.21.15.44:0/2654806851 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b54068a40 con 0x7f1b54101ad0 2024-04-21T23:19:47.345 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.343+0000 7f1b4b7fe640 1 -- 172.21.15.44:0/2654806851 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1b50045020 con 0x7f1b54101ad0 2024-04-21T23:19:47.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.343+0000 7f1b4b7fe640 1 --2- 172.21.15.44:0/2654806851 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1b400a2d00 0x7f1b400a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:47.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.344+0000 7f1b4b7fe640 1 -- 172.21.15.44:0/2654806851 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9533+0+0 (secure 0 0 0) 0x7f1b500f1c30 con 0x7f1b54101ad0 2024-04-21T23:19:47.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.344+0000 7f1b59980640 1 --2- 172.21.15.44:0/2654806851 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1b400a2d00 0x7f1b400a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:47.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.344+0000 7f1b59980640 1 --2- 172.21.15.44:0/2654806851 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1b400a2d00 0x7f1b400a51c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1b44005fd0 tx=0x7f1b44005850 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:47.346 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.347+0000 7f1b4b7fe640 1 -- 172.21.15.44:0/2654806851 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f1b5004c020 con 0x7f1b54101ad0 2024-04-21T23:19:47.544 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.546+0000 7f1b5c3e5640 1 -- 172.21.15.44:0/2654806851 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1b54109d30 con 0x7f1b54101ad0 2024-04-21T23:19:47.545 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.547+0000 7f1b4b7fe640 1 -- 172.21.15.44:0/2654806851 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v110) ==== 81+0+942 (secure 0 0 0) 0x7f1b500be570 con 0x7f1b54101ad0 2024-04-21T23:19:47.545 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3134653118 2024-04-22T23:08:50.585326+0000 2024-04-21T23:19:47.546 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/101322314 2024-04-22T23:08:50.585326+0000 2024-04-21T23:19:47.546 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/792524018 2024-04-22T23:08:50.585326+0000 2024-04-21T23:19:47.546 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2303525779 2024-04-22T23:08:50.585326+0000 2024-04-21T23:19:47.546 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2729363939 2024-04-22T22:59:22.969731+0000 2024-04-21T23:19:47.546 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/3462209685 2024-04-22T22:59:22.969731+0000 2024-04-21T23:19:47.546 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/4238737803 2024-04-22T22:59:22.969731+0000 2024-04-21T23:19:47.546 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/1417504446 2024-04-22T23:08:50.585326+0000 2024-04-21T23:19:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/4238737803 2024-04-22T22:59:22.969731+0000 2024-04-21T23:19:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/1417504446 2024-04-22T23:08:50.585326+0000 2024-04-21T23:19:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6800/2900331072 2024-04-22T22:58:56.511296+0000 2024-04-21T23:19:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/908290973 2024-04-22T22:58:56.511296+0000 2024-04-21T23:19:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/2994566338 2024-04-22T22:58:56.511296+0000 2024-04-21T23:19:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:6801/2900331072 2024-04-22T22:58:56.511296+0000 2024-04-21T23:19:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/693229699 2024-04-22T22:58:56.511296+0000 2024-04-21T23:19:47.547 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44:0/1931984406 2024-04-22T22:59:22.969731+0000 2024-04-21T23:19:47.548 INFO:teuthology.orchestra.run.smithi044.stderr:listed 16 entries 2024-04-21T23:19:47.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.548+0000 7f1b5c3e5640 1 -- 172.21.15.44:0/2654806851 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1b400a2d00 msgr2=0x7f1b400a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:47.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.548+0000 7f1b5c3e5640 1 --2- 172.21.15.44:0/2654806851 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1b400a2d00 0x7f1b400a51c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1b44005fd0 tx=0x7f1b44005850 comp rx=0 tx=0).stop 2024-04-21T23:19:47.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.548+0000 7f1b5c3e5640 1 -- 172.21.15.44:0/2654806851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b54101ad0 msgr2=0x7f1b541980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:47.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.548+0000 7f1b5c3e5640 1 --2- 172.21.15.44:0/2654806851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b54101ad0 0x7f1b541980e0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f1b50006010 tx=0x7f1b500043b0 comp rx=0 tx=0).stop 2024-04-21T23:19:47.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.548+0000 7f1b5c3e5640 1 -- 172.21.15.44:0/2654806851 shutdown_connections 2024-04-21T23:19:47.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.548+0000 7f1b5c3e5640 1 --2- 172.21.15.44:0/2654806851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b54103210 0x7f1b54198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:47.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.548+0000 7f1b5c3e5640 1 --2- 172.21.15.44:0/2654806851 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f1b400a2d00 0x7f1b400a51c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:47.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.548+0000 7f1b5c3e5640 1 --2- 172.21.15.44:0/2654806851 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b54103bb0 0x7f1b5419c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:47.549 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.548+0000 7f1b5c3e5640 1 --2- 172.21.15.44:0/2654806851 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f1b54101ad0 0x7f1b541980e0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:47.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.548+0000 7f1b5c3e5640 1 -- 172.21.15.44:0/2654806851 >> 172.21.15.44:0/2654806851 conn(0x7f1b540fd200 msgr2=0x7f1b541075f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:47.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.549+0000 7f1b5c3e5640 1 -- 172.21.15.44:0/2654806851 shutdown_connections 2024-04-21T23:19:47.550 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:47.549+0000 7f1b5c3e5640 1 -- 172.21.15.44:0/2654806851 wait complete. 2024-04-21T23:19:47.557 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:47 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2679989498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:19:47.557 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:47 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/4263176209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:19:47.557 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:47 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1238050939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:19:47.557 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:47 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2031087140' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:19:47.558 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-21T23:19:47.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:19:47.558 DEBUG:teuthology.orchestra.run.smithi044:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-21T23:19:47.603 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi044.front.sepia.ceph.com 2024-04-21T23:19:47.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:19:47.604 DEBUG:teuthology.orchestra.run.smithi044:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-21T23:19:47.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:19:47.630 DEBUG:teuthology.orchestra.run.smithi044:> ip netns list 2024-04-21T23:19:47.685 INFO:teuthology.orchestra.run.smithi044.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-21T23:19:47.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:19:47.686 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-21T23:19:47.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:47 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2679989498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:19:47.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:47 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/4263176209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:19:47.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:47 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1238050939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:19:47.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:47 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2031087140' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:19:47.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:47 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2679989498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T23:19:47.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:47 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/4263176209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T23:19:47.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:47 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1238050939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T23:19:47.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:47 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2031087140' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T23:19:47.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T23:19:47.750 DEBUG:teuthology.orchestra.run.smithi044:> sudo ip link delete ceph-brx 2024-04-21T23:19:47.844 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-21T23:19:47.936 INFO:tasks.cephadm:Teardown begin 2024-04-21T23:19:47.937 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-21T23:19:47.937 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:19:48.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.043+0000 7f64811af640 1 -- 172.21.15.44:0/1907103696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f647c1029b0 msgr2=0x7f647c102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:48.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.043+0000 7f64811af640 1 --2- 172.21.15.44:0/1907103696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f647c1029b0 0x7f647c102d90 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f646c009900 tx=0x7f646c02bcd0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.043+0000 7f64811af640 1 -- 172.21.15.44:0/1907103696 shutdown_connections 2024-04-21T23:19:48.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.043+0000 7f64811af640 1 --2- 172.21.15.44:0/1907103696 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f647c10bd80 0x7f647c10e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.043+0000 7f64811af640 1 --2- 172.21.15.44:0/1907103696 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f647c1032d0 0x7f647c10b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.042 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.043+0000 7f64811af640 1 --2- 172.21.15.44:0/1907103696 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f647c1029b0 0x7f647c102d90 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.043+0000 7f64811af640 1 -- 172.21.15.44:0/1907103696 >> 172.21.15.44:0/1907103696 conn(0x7f647c077740 msgr2=0x7f647c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:48.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.043+0000 7f64811af640 1 -- 172.21.15.44:0/1907103696 shutdown_connections 2024-04-21T23:19:48.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.043+0000 7f64811af640 1 -- 172.21.15.44:0/1907103696 wait complete. 2024-04-21T23:19:48.043 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.044+0000 7f64811af640 1 Processor -- start 2024-04-21T23:19:48.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.044+0000 7f64811af640 1 -- start start 2024-04-21T23:19:48.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.044+0000 7f64811af640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f647c1029b0 0x7f647c100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:48.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.044+0000 7f64811af640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f647c1032d0 0x7f647c100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:48.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.044+0000 7f64811af640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f647c10bd80 0x7f647c101080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:48.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.044+0000 7f64811af640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f647c111b10 con 0x7f647c10bd80 2024-04-21T23:19:48.045 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.044+0000 7f64811af640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f647c111990 con 0x7f647c1032d0 2024-04-21T23:19:48.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.044+0000 7f64811af640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f647c111c90 con 0x7f647c1029b0 2024-04-21T23:19:48.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.044+0000 7f647b59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f647c10bd80 0x7f647c101080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:48.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.044+0000 7f647b59e640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f647c10bd80 0x7f647c101080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47968/0 (socket says 172.21.15.44:47968) 2024-04-21T23:19:48.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.044+0000 7f647b59e640 1 -- 172.21.15.44:0/3024951347 learned_addr learned my addr 172.21.15.44:0/3024951347 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:48.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.044+0000 7f647ad9d640 1 --2- 172.21.15.44:0/3024951347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f647c1029b0 0x7f647c100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:48.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.044+0000 7f647a59c640 1 --2- 172.21.15.44:0/3024951347 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f647c1032d0 0x7f647c100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:48.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.045+0000 7f647b59e640 1 -- 172.21.15.44:0/3024951347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f647c1029b0 msgr2=0x7f647c100600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:48.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.045+0000 7f647b59e640 1 --2- 172.21.15.44:0/3024951347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f647c1029b0 0x7f647c100600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.046 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.045+0000 7f647b59e640 1 -- 172.21.15.44:0/3024951347 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f647c1032d0 msgr2=0x7f647c100b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:48.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.045+0000 7f647b59e640 1 --2- 172.21.15.44:0/3024951347 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f647c1032d0 0x7f647c100b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.045+0000 7f647b59e640 1 -- 172.21.15.44:0/3024951347 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f647c1a6510 con 0x7f647c10bd80 2024-04-21T23:19:48.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.045+0000 7f647ad9d640 1 --2- 172.21.15.44:0/3024951347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f647c1029b0 0x7f647c100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T23:19:48.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.045+0000 7f647b59e640 1 --2- 172.21.15.44:0/3024951347 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f647c10bd80 0x7f647c101080 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f647400cab0 tx=0x7f647400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:48.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.045+0000 7f6463fff640 1 -- 172.21.15.44:0/3024951347 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647400ae20 con 0x7f647c10bd80 2024-04-21T23:19:48.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.045+0000 7f64811af640 1 -- 172.21.15.44:0/3024951347 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f647c1a6800 con 0x7f647c10bd80 2024-04-21T23:19:48.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.045+0000 7f6463fff640 1 -- 172.21.15.44:0/3024951347 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f647400f930 con 0x7f647c10bd80 2024-04-21T23:19:48.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.045+0000 7f6463fff640 1 -- 172.21.15.44:0/3024951347 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6474018b80 con 0x7f647c10bd80 2024-04-21T23:19:48.047 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.045+0000 7f64811af640 1 -- 172.21.15.44:0/3024951347 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f647c1a6d50 con 0x7f647c10bd80 2024-04-21T23:19:48.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.046+0000 7f64811af640 1 -- 172.21.15.44:0/3024951347 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f647c068a40 con 0x7f647c10bd80 2024-04-21T23:19:48.048 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.047+0000 7f6463fff640 1 -- 172.21.15.44:0/3024951347 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f647400fae0 con 0x7f647c10bd80 2024-04-21T23:19:48.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.048+0000 7f6463fff640 1 --2- 172.21.15.44:0/3024951347 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f64640a2d00 0x7f64640a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:48.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.048+0000 7f6463fff640 1 -- 172.21.15.44:0/3024951347 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9533+0+0 (secure 0 0 0) 0x7f64740d00e0 con 0x7f647c10bd80 2024-04-21T23:19:48.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.048+0000 7f647ad9d640 1 --2- 172.21.15.44:0/3024951347 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f64640a2d00 0x7f64640a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:48.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.048+0000 7f647ad9d640 1 --2- 172.21.15.44:0/3024951347 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f64640a2d00 0x7f64640a51c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f646c006010 tx=0x7f646c02cf90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:48.049 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.051+0000 7f6463fff640 1 -- 172.21.15.44:0/3024951347 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6474014030 con 0x7f647c10bd80 2024-04-21T23:19:48.272 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.274+0000 7f64811af640 1 -- 172.21.15.44:0/3024951347 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f647c02d170 con 0x7f647c10bd80 2024-04-21T23:19:48.273 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.275+0000 7f6463fff640 1 -- 172.21.15.44:0/3024951347 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11791 (secure 0 0 0) 0x7f647409c910 con 0x7f647c10bd80 2024-04-21T23:19:48.273 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:19:48.273 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:16:20.187271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26863":{"gid":26863,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3179646768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3179646768},{"type":"v1","addr":"172.21.15.69:6839","nonce":3179646768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:19:48.274 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 31 2024-04-21T23:19:48.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.277+0000 7f64811af640 1 -- 172.21.15.44:0/3024951347 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f64640a2d00 msgr2=0x7f64640a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:48.275 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.277+0000 7f64811af640 1 --2- 172.21.15.44:0/3024951347 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f64640a2d00 0x7f64640a51c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f646c006010 tx=0x7f646c02cf90 comp rx=0 tx=0).stop 2024-04-21T23:19:48.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.277+0000 7f64811af640 1 -- 172.21.15.44:0/3024951347 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f647c10bd80 msgr2=0x7f647c101080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:48.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.277+0000 7f64811af640 1 --2- 172.21.15.44:0/3024951347 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f647c10bd80 0x7f647c101080 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f647400cab0 tx=0x7f647400cf70 comp rx=0 tx=0).stop 2024-04-21T23:19:48.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.277+0000 7f64811af640 1 -- 172.21.15.44:0/3024951347 shutdown_connections 2024-04-21T23:19:48.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.277+0000 7f64811af640 1 --2- 172.21.15.44:0/3024951347 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f64640a2d00 0x7f64640a51c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.277+0000 7f64811af640 1 --2- 172.21.15.44:0/3024951347 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f647c10bd80 0x7f647c101080 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.277+0000 7f64811af640 1 --2- 172.21.15.44:0/3024951347 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f647c1032d0 0x7f647c100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.277+0000 7f64811af640 1 --2- 172.21.15.44:0/3024951347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f647c1029b0 0x7f647c100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.276 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.277+0000 7f64811af640 1 -- 172.21.15.44:0/3024951347 >> 172.21.15.44:0/3024951347 conn(0x7f647c077740 msgr2=0x7f647c0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:48.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.277+0000 7f64811af640 1 -- 172.21.15.44:0/3024951347 shutdown_connections 2024-04-21T23:19:48.277 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.277+0000 7f64811af640 1 -- 172.21.15.44:0/3024951347 wait complete. 2024-04-21T23:19:48.286 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T23:19:48.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.389+0000 7fcf9675b640 1 -- 172.21.15.44:0/3995081361 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf90107730 msgr2=0x7fcf90107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:48.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.389+0000 7fcf9675b640 1 --2- 172.21.15.44:0/3995081361 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf90107730 0x7fcf90107b10 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fcf80009930 tx=0x7fcf8002bc30 comp rx=0 tx=0).stop 2024-04-21T23:19:48.390 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.389+0000 7fcf9675b640 1 -- 172.21.15.44:0/3995081361 shutdown_connections 2024-04-21T23:19:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.389+0000 7fcf9675b640 1 --2- 172.21.15.44:0/3995081361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf901020e0 0x7fcf9010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.389+0000 7fcf9675b640 1 --2- 172.21.15.44:0/3995081361 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf90101720 0x7fcf90101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.389+0000 7fcf9675b640 1 --2- 172.21.15.44:0/3995081361 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf90107730 0x7fcf90107b10 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.389+0000 7fcf9675b640 1 -- 172.21.15.44:0/3995081361 >> 172.21.15.44:0/3995081361 conn(0x7fcf900fd230 msgr2=0x7fcf900ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.390+0000 7fcf9675b640 1 -- 172.21.15.44:0/3995081361 shutdown_connections 2024-04-21T23:19:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.390+0000 7fcf9675b640 1 -- 172.21.15.44:0/3995081361 wait complete. 2024-04-21T23:19:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.390+0000 7fcf9675b640 1 Processor -- start 2024-04-21T23:19:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.390+0000 7fcf9675b640 1 -- start start 2024-04-21T23:19:48.391 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.390+0000 7fcf9675b640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf90101720 0x7fcf90073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:48.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.390+0000 7fcf9675b640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf901020e0 0x7fcf90073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:48.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.390+0000 7fcf9675b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf90107730 0x7fcf900744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:48.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.390+0000 7fcf9675b640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7fcf90111b60 con 0x7fcf901020e0 2024-04-21T23:19:48.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.391+0000 7fcf9675b640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7fcf901119e0 con 0x7fcf90101720 2024-04-21T23:19:48.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.391+0000 7fcf9675b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcf90111ce0 con 0x7fcf90107730 2024-04-21T23:19:48.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.391+0000 7fcf8f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf901020e0 0x7fcf90073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:48.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.391+0000 7fcf8ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf90101720 0x7fcf90073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:48.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.391+0000 7fcf8ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf90101720 0x7fcf90073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.44:59480/0 (socket says 172.21.15.44:59480) 2024-04-21T23:19:48.392 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.391+0000 7fcf8ffff640 1 -- 172.21.15.44:0/1373738895 learned_addr learned my addr 172.21.15.44:0/1373738895 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:48.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.391+0000 7fcf8f7fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf901020e0 0x7fcf90073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:47986/0 (socket says 172.21.15.44:47986) 2024-04-21T23:19:48.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.391+0000 7fcf94cf8640 1 --2- 172.21.15.44:0/1373738895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf90107730 0x7fcf900744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:48.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.391+0000 7fcf8f7fe640 1 -- 172.21.15.44:0/1373738895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf90107730 msgr2=0x7fcf900744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:48.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.391+0000 7fcf8f7fe640 1 --2- 172.21.15.44:0/1373738895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf90107730 0x7fcf900744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.391+0000 7fcf8f7fe640 1 -- 172.21.15.44:0/1373738895 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf90101720 msgr2=0x7fcf90073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:48.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.391+0000 7fcf8f7fe640 1 --2- 172.21.15.44:0/1373738895 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf90101720 0x7fcf90073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.393 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.391+0000 7fcf8f7fe640 1 -- 172.21.15.44:0/1373738895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf901a65e0 con 0x7fcf901020e0 2024-04-21T23:19:48.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.391+0000 7fcf8ffff640 1 --2- 172.21.15.44:0/1373738895 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf90101720 0x7fcf90073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T23:19:48.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.391+0000 7fcf8f7fe640 1 --2- 172.21.15.44:0/1373738895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf901020e0 0x7fcf90073f80 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fcf7800b3f0 tx=0x7fcf7800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:48.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.391+0000 7fcf8d7fa640 1 -- 172.21.15.44:0/1373738895 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf78015020 con 0x7fcf901020e0 2024-04-21T23:19:48.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.391+0000 7fcf8d7fa640 1 -- 172.21.15.44:0/1373738895 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf78011d10 con 0x7fcf901020e0 2024-04-21T23:19:48.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.391+0000 7fcf9675b640 1 -- 172.21.15.44:0/1373738895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf901a68d0 con 0x7fcf901020e0 2024-04-21T23:19:48.394 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.391+0000 7fcf8d7fa640 1 -- 172.21.15.44:0/1373738895 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf7800f2e0 con 0x7fcf901020e0 2024-04-21T23:19:48.395 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.391+0000 7fcf9675b640 1 -- 172.21.15.44:0/1373738895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf901a6e20 con 0x7fcf901020e0 2024-04-21T23:19:48.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.393+0000 7fcf9675b640 1 -- 172.21.15.44:0/1373738895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf90068a40 con 0x7fcf901020e0 2024-04-21T23:19:48.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.394+0000 7fcf8d7fa640 1 -- 172.21.15.44:0/1373738895 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fcf78018020 con 0x7fcf901020e0 2024-04-21T23:19:48.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.394+0000 7fcf8d7fa640 1 --2- 172.21.15.44:0/1373738895 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcf7c0a2d00 0x7fcf7c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:48.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.394+0000 7fcf8ffff640 1 --2- 172.21.15.44:0/1373738895 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcf7c0a2d00 0x7fcf7c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:48.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.394+0000 7fcf8d7fa640 1 -- 172.21.15.44:0/1373738895 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9533+0+0 (secure 0 0 0) 0x7fcf780cef90 con 0x7fcf901020e0 2024-04-21T23:19:48.396 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.395+0000 7fcf8ffff640 1 --2- 172.21.15.44:0/1373738895 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcf7c0a2d00 0x7fcf7c0a51c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcf80000c00 tx=0x7fcf800058d0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:48.397 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.399+0000 7fcf8d7fa640 1 -- 172.21.15.44:0/1373738895 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7fcf7801f020 con 0x7fcf901020e0 2024-04-21T23:19:48.622 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.624+0000 7fcf9675b640 1 -- 172.21.15.44:0/1373738895 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf90109c90 con 0x7fcf901020e0 2024-04-21T23:19:48.623 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.624+0000 7fcf8d7fa640 1 -- 172.21.15.44:0/1373738895 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11791 (secure 0 0 0) 0x7fcf7809b8d0 con 0x7fcf901020e0 2024-04-21T23:19:48.623 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:19:48.623 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6835/634418901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6834","nonce":634418901},{"type":"v1","addr":"172.21.15.44:6835","nonce":634418901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6837/3848365642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6836","nonce":3848365642},{"type":"v1","addr":"172.21.15.44:6837","nonce":3848365642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6839/1544956047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6838","nonce":1544956047},{"type":"v1","addr":"172.21.15.44:6839","nonce":1544956047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.44:6841/1863599778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6840","nonce":1863599778},{"type":"v1","addr":"172.21.15.44:6841","nonce":1863599778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6833/1288947368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6832","nonce":1288947368},{"type":"v1","addr":"172.21.15.55:6833","nonce":1288947368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/43117883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":43117883},{"type":"v1","addr":"172.21.15.69:6833","nonce":43117883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6835/1607421809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6834","nonce":1607421809},{"type":"v1","addr":"172.21.15.55:6835","nonce":1607421809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2637621987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2637621987},{"type":"v1","addr":"172.21.15.69:6835","nonce":2637621987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.55:6837/1372428654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6836","nonce":1372428654},{"type":"v1","addr":"172.21.15.55:6837","nonce":1372428654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/3371833935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":3371833935},{"type":"v1","addr":"172.21.15.69:6837","nonce":3371833935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T23:06:25.562805+0000","modified":"2024-04-21T23:16:20.187271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.55:6839/2066817543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6838","nonce":2066817543},{"type":"v1","addr":"172.21.15.55:6839","nonce":2066817543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26863":{"gid":26863,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.69:6839/3179646768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":3179646768},{"type":"v1","addr":"172.21.15.69:6839","nonce":3179646768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24530,"qdb_cluster":[24530]},"id":1}]} 2024-04-21T23:19:48.623 INFO:teuthology.orchestra.run.smithi044.stderr:dumped fsmap epoch 31 2024-04-21T23:19:48.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.627+0000 7fcf9675b640 1 -- 172.21.15.44:0/1373738895 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcf7c0a2d00 msgr2=0x7fcf7c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:48.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.627+0000 7fcf9675b640 1 --2- 172.21.15.44:0/1373738895 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcf7c0a2d00 0x7fcf7c0a51c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcf80000c00 tx=0x7fcf800058d0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.627+0000 7fcf9675b640 1 -- 172.21.15.44:0/1373738895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf901020e0 msgr2=0x7fcf90073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:48.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.627+0000 7fcf9675b640 1 --2- 172.21.15.44:0/1373738895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf901020e0 0x7fcf90073f80 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fcf7800b3f0 tx=0x7fcf7800b8b0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.627+0000 7fcf9675b640 1 -- 172.21.15.44:0/1373738895 shutdown_connections 2024-04-21T23:19:48.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.627+0000 7fcf9675b640 1 --2- 172.21.15.44:0/1373738895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf90107730 0x7fcf900744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.627+0000 7fcf9675b640 1 --2- 172.21.15.44:0/1373738895 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7fcf7c0a2d00 0x7fcf7c0a51c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.627+0000 7fcf9675b640 1 --2- 172.21.15.44:0/1373738895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7fcf901020e0 0x7fcf90073f80 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.627+0000 7fcf9675b640 1 --2- 172.21.15.44:0/1373738895 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcf90101720 0x7fcf90073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.626 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.627+0000 7fcf9675b640 1 -- 172.21.15.44:0/1373738895 >> 172.21.15.44:0/1373738895 conn(0x7fcf900fd230 msgr2=0x7fcf900fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:48.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.627+0000 7fcf9675b640 1 -- 172.21.15.44:0/1373738895 shutdown_connections 2024-04-21T23:19:48.627 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.627+0000 7fcf9675b640 1 -- 172.21.15.44:0/1373738895 wait complete. 2024-04-21T23:19:48.635 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:48 smithi044 ceph-mon[27934]: pgmap v346: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:19:48.635 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:48 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2654806851' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:19:48.636 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T23:19:48.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:48 smithi055 ceph-mon[31117]: pgmap v346: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:19:48.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:48 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2654806851' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:19:48.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:48 smithi069 ceph-mon[31816]: pgmap v346: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T23:19:48.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:48 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2654806851' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T23:19:48.740 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.740+0000 7f6efe377640 1 -- 172.21.15.44:0/2763535732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ef8101720 msgr2=0x7f6ef8101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:48.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.740+0000 7f6efe377640 1 --2- 172.21.15.44:0/2763535732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ef8101720 0x7f6ef8101ba0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f6eec009930 tx=0x7f6eec02bc30 comp rx=0 tx=0).stop 2024-04-21T23:19:48.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.740+0000 7f6efe377640 1 -- 172.21.15.44:0/2763535732 shutdown_connections 2024-04-21T23:19:48.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.740+0000 7f6efe377640 1 --2- 172.21.15.44:0/2763535732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef81020e0 0x7f6ef810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.740+0000 7f6efe377640 1 --2- 172.21.15.44:0/2763535732 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ef8101720 0x7f6ef8101ba0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.740+0000 7f6efe377640 1 --2- 172.21.15.44:0/2763535732 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6ef8107730 0x7f6ef8107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.740+0000 7f6efe377640 1 -- 172.21.15.44:0/2763535732 >> 172.21.15.44:0/2763535732 conn(0x7f6ef80fd230 msgr2=0x7f6ef80ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:48.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.740+0000 7f6efe377640 1 -- 172.21.15.44:0/2763535732 shutdown_connections 2024-04-21T23:19:48.741 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.740+0000 7f6efe377640 1 -- 172.21.15.44:0/2763535732 wait complete. 2024-04-21T23:19:48.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.741+0000 7f6efe377640 1 Processor -- start 2024-04-21T23:19:48.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.741+0000 7f6efe377640 1 -- start start 2024-04-21T23:19:48.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.741+0000 7f6efe377640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef8101720 0x7f6ef8198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:48.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.741+0000 7f6efe377640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ef81020e0 0x7f6ef81985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:48.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.741+0000 7f6efe377640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6ef8107730 0x7f6ef819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:48.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.741+0000 7f6efe377640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef8111b40 con 0x7f6ef81020e0 2024-04-21T23:19:48.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.741+0000 7f6efe377640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef81119c0 con 0x7f6ef8107730 2024-04-21T23:19:48.742 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.741+0000 7f6efe377640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef8111cc0 con 0x7f6ef8101720 2024-04-21T23:19:48.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.741+0000 7f6ef77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ef81020e0 0x7f6ef81985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:48.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.741+0000 7f6ef77fe640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ef81020e0 0x7f6ef81985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48012/0 (socket says 172.21.15.44:48012) 2024-04-21T23:19:48.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.741+0000 7f6ef77fe640 1 -- 172.21.15.44:0/4098539191 learned_addr learned my addr 172.21.15.44:0/4098539191 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:48.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.741+0000 7f6ef7fff640 1 --2- 172.21.15.44:0/4098539191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef8101720 0x7f6ef8198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:48.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.741+0000 7f6ef77fe640 1 -- 172.21.15.44:0/4098539191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef8101720 msgr2=0x7f6ef8198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:48.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.741+0000 7f6ef77fe640 1 --2- 172.21.15.44:0/4098539191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef8101720 0x7f6ef8198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.743 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.741+0000 7f6ef77fe640 1 -- 172.21.15.44:0/4098539191 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6ef8107730 msgr2=0x7f6ef819c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:48.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.741+0000 7f6ef77fe640 1 --2- 172.21.15.44:0/4098539191 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6ef8107730 0x7f6ef819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.741+0000 7f6ef77fe640 1 -- 172.21.15.44:0/4098539191 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef819d050 con 0x7f6ef81020e0 2024-04-21T23:19:48.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.742+0000 7f6ef77fe640 1 --2- 172.21.15.44:0/4098539191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ef81020e0 0x7f6ef81985a0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f6eec009230 tx=0x7f6eec02cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:48.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.742+0000 7f6ef57fa640 1 -- 172.21.15.44:0/4098539191 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eec041070 con 0x7f6ef81020e0 2024-04-21T23:19:48.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.742+0000 7f6efe377640 1 -- 172.21.15.44:0/4098539191 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ef819d2e0 con 0x7f6ef81020e0 2024-04-21T23:19:48.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.742+0000 7f6ef57fa640 1 -- 172.21.15.44:0/4098539191 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6eec004450 con 0x7f6ef81020e0 2024-04-21T23:19:48.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.742+0000 7f6ef57fa640 1 -- 172.21.15.44:0/4098539191 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eec003e50 con 0x7f6ef81020e0 2024-04-21T23:19:48.744 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.742+0000 7f6efe377640 1 -- 172.21.15.44:0/4098539191 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ef81a4bd0 con 0x7f6ef81020e0 2024-04-21T23:19:48.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.743+0000 7f6efe377640 1 -- 172.21.15.44:0/4098539191 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ef8068a40 con 0x7f6ef81020e0 2024-04-21T23:19:48.745 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.744+0000 7f6ef57fa640 1 -- 172.21.15.44:0/4098539191 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6eec044070 con 0x7f6ef81020e0 2024-04-21T23:19:48.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.744+0000 7f6ef57fa640 1 --2- 172.21.15.44:0/4098539191 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6ee00a2ea0 0x7f6ee00a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:48.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.744+0000 7f6ef7fff640 1 --2- 172.21.15.44:0/4098539191 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6ee00a2ea0 0x7f6ee00a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:48.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.745+0000 7f6ef57fa640 1 -- 172.21.15.44:0/4098539191 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9533+0+0 (secure 0 0 0) 0x7f6eec0f1100 con 0x7f6ef81020e0 2024-04-21T23:19:48.746 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.745+0000 7f6ef7fff640 1 --2- 172.21.15.44:0/4098539191 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6ee00a2ea0 0x7f6ee00a5360 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8009ae0 tx=0x7f6ee8005f90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:48.747 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.749+0000 7f6ef57fa640 1 -- 172.21.15.44:0/4098539191 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f6eec0bda40 con 0x7f6ef81020e0 2024-04-21T23:19:48.934 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.936+0000 7f6efe377640 1 -- 172.21.15.44:0/4098539191 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6ef802d170 con 0x7f6ef81020e0 2024-04-21T23:19:48.936 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.938+0000 7f6ef57fa640 1 -- 172.21.15.44:0/4098539191 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+25278 (secure 0 0 0) 0x7f6eec0bd340 con 0x7f6ef81020e0 2024-04-21T23:19:48.936 INFO:teuthology.orchestra.run.smithi044.stdout: 2024-04-21T23:19:48.937 INFO:teuthology.orchestra.run.smithi044.stdout:{"epoch":110,"fsid":"80909b94-0032-11ef-bc93-c7b262605968","created":"2024-04-21T22:58:23.196166+0000","modified":"2024-04-21T23:19:23.219564+0000","last_up_change":"2024-04-21T23:05:34.874838+0000","last_in_change":"2024-04-21T23:05:14.249579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T23:01:47.507503+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T23:06:23.993760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T23:06:24.954661+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":109,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T23:06:28.677618+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":109,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"aaaab74d-dfe1-4669-9839-36b2664ac353","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6803","nonce":4094542096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6805","nonce":4094542096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6809","nonce":4094542096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":4094542096},{"type":"v1","addr":"172.21.15.44:6807","nonce":4094542096}]},"public_addr":"172.21.15.44:6803/4094542096","cluster_addr":"172.21.15.44:6805/4094542096","heartbeat_back_addr":"172.21.15.44:6809/4094542096","heartbeat_front_addr":"172.21.15.44:6807/4094542096","state":["exists","up"]},{"osd":1,"uuid":"7950725c-5dea-4a3b-a06d-4e50b621110b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6801","nonce":1988017680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6803","nonce":1988017680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6807","nonce":1988017680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":1988017680},{"type":"v1","addr":"172.21.15.55:6805","nonce":1988017680}]},"public_addr":"172.21.15.55:6801/1988017680","cluster_addr":"172.21.15.55:6803/1988017680","heartbeat_back_addr":"172.21.15.55:6807/1988017680","heartbeat_front_addr":"172.21.15.55:6805/1988017680","state":["exists","up"]},{"osd":2,"uuid":"e99888f4-9a3e-4856-89a8-1a35817893b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6801","nonce":3485429011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6803","nonce":3485429011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6807","nonce":3485429011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":3485429011},{"type":"v1","addr":"172.21.15.69:6805","nonce":3485429011}]},"public_addr":"172.21.15.69:6801/3485429011","cluster_addr":"172.21.15.69:6803/3485429011","heartbeat_back_addr":"172.21.15.69:6807/3485429011","heartbeat_front_addr":"172.21.15.69:6805/3485429011","state":["exists","up"]},{"osd":3,"uuid":"508c528e-8e90-408b-97e3-5eded769bf1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6810","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6811","nonce":746259227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6812","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6813","nonce":746259227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6816","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6817","nonce":746259227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6814","nonce":746259227},{"type":"v1","addr":"172.21.15.44:6815","nonce":746259227}]},"public_addr":"172.21.15.44:6811/746259227","cluster_addr":"172.21.15.44:6813/746259227","heartbeat_back_addr":"172.21.15.44:6817/746259227","heartbeat_front_addr":"172.21.15.44:6815/746259227","state":["exists","up"]},{"osd":4,"uuid":"44a9925a-84ae-474a-8d5c-bec46baaa91c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6809","nonce":988958724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6811","nonce":988958724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6815","nonce":988958724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":988958724},{"type":"v1","addr":"172.21.15.55:6813","nonce":988958724}]},"public_addr":"172.21.15.55:6809/988958724","cluster_addr":"172.21.15.55:6811/988958724","heartbeat_back_addr":"172.21.15.55:6815/988958724","heartbeat_front_addr":"172.21.15.55:6813/988958724","state":["exists","up"]},{"osd":5,"uuid":"f219c5c9-af53-4cf0-a77a-56595b2db93b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6809","nonce":2597202863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6811","nonce":2597202863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6815","nonce":2597202863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":2597202863},{"type":"v1","addr":"172.21.15.69:6813","nonce":2597202863}]},"public_addr":"172.21.15.69:6809/2597202863","cluster_addr":"172.21.15.69:6811/2597202863","heartbeat_back_addr":"172.21.15.69:6815/2597202863","heartbeat_front_addr":"172.21.15.69:6813/2597202863","state":["exists","up"]},{"osd":6,"uuid":"6c9b8cd5-e20b-49eb-af02-87709b05e2e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6818","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6819","nonce":1221340748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6820","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6821","nonce":1221340748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6824","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6825","nonce":1221340748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6822","nonce":1221340748},{"type":"v1","addr":"172.21.15.44:6823","nonce":1221340748}]},"public_addr":"172.21.15.44:6819/1221340748","cluster_addr":"172.21.15.44:6821/1221340748","heartbeat_back_addr":"172.21.15.44:6825/1221340748","heartbeat_front_addr":"172.21.15.44:6823/1221340748","state":["exists","up"]},{"osd":7,"uuid":"12c98dc9-81df-4a59-9e6f-e480f5162390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6817","nonce":3062702978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6819","nonce":3062702978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6823","nonce":3062702978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3062702978},{"type":"v1","addr":"172.21.15.55:6821","nonce":3062702978}]},"public_addr":"172.21.15.55:6817/3062702978","cluster_addr":"172.21.15.55:6819/3062702978","heartbeat_back_addr":"172.21.15.55:6823/3062702978","heartbeat_front_addr":"172.21.15.55:6821/3062702978","state":["exists","up"]},{"osd":8,"uuid":"d1633bc7-35b7-491c-be5c-4375af06fd67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6817","nonce":635270546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6819","nonce":635270546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6823","nonce":635270546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":635270546},{"type":"v1","addr":"172.21.15.69:6821","nonce":635270546}]},"public_addr":"172.21.15.69:6817/635270546","cluster_addr":"172.21.15.69:6819/635270546","heartbeat_back_addr":"172.21.15.69:6823/635270546","heartbeat_front_addr":"172.21.15.69:6821/635270546","state":["exists","up"]},{"osd":9,"uuid":"fc0ed998-2d10-4bc6-9a3a-d6afe9135386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6826","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6827","nonce":1043222908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6828","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6829","nonce":1043222908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6832","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6833","nonce":1043222908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6830","nonce":1043222908},{"type":"v1","addr":"172.21.15.44:6831","nonce":1043222908}]},"public_addr":"172.21.15.44:6827/1043222908","cluster_addr":"172.21.15.44:6829/1043222908","heartbeat_back_addr":"172.21.15.44:6833/1043222908","heartbeat_front_addr":"172.21.15.44:6831/1043222908","state":["exists","up"]},{"osd":10,"uuid":"c8dddb89-8c25-4df2-8993-3f61e52c53c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6825","nonce":3066586400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6827","nonce":3066586400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6830","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6831","nonce":3066586400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":3066586400},{"type":"v1","addr":"172.21.15.55:6829","nonce":3066586400}]},"public_addr":"172.21.15.55:6825/3066586400","cluster_addr":"172.21.15.55:6827/3066586400","heartbeat_back_addr":"172.21.15.55:6831/3066586400","heartbeat_front_addr":"172.21.15.55:6829/3066586400","state":["exists","up"]},{"osd":11,"uuid":"1623764d-b8ca-4222-a1e4-2714f17d64e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6825","nonce":1593504610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6827","nonce":1593504610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6831","nonce":1593504610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1593504610},{"type":"v1","addr":"172.21.15.69:6829","nonce":1593504610}]},"public_addr":"172.21.15.69:6825/1593504610","cluster_addr":"172.21.15.69:6827/1593504610","heartbeat_back_addr":"172.21.15.69:6831/1593504610","heartbeat_front_addr":"172.21.15.69:6829/1593504610","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:00:56.740313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:20.780696+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:01:44.581487+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:10.312968+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:34.999366+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:02:59.777620+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:24.645936+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:03:49.930294+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:14.356095+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:04:40.647956+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:06.767845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T23:05:32.215283+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.e","mappings":[{"from":9,"to":11},{"from":5,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.44:0/3134653118":"2024-04-22T23:08:50.585326+0000","172.21.15.44:0/101322314":"2024-04-22T23:08:50.585326+0000","172.21.15.44:0/792524018":"2024-04-22T23:08:50.585326+0000","172.21.15.44:0/2303525779":"2024-04-22T23:08:50.585326+0000","172.21.15.44:0/2729363939":"2024-04-22T22:59:22.969731+0000","172.21.15.44:0/3462209685":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6800/1417504446":"2024-04-22T23:08:50.585326+0000","172.21.15.44:6800/4238737803":"2024-04-22T22:59:22.969731+0000","172.21.15.44:6801/1417504446":"2024-04-22T23:08:50.585326+0000","172.21.15.44:6800/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/908290973":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/2994566338":"2024-04-22T22:58:56.511296+0000","172.21.15.44:6801/2900331072":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/693229699":"2024-04-22T22:58:56.511296+0000","172.21.15.44:0/1931984406":"2024-04-22T22:59:22.969731+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T23:19:48.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.940+0000 7f6efe377640 1 -- 172.21.15.44:0/4098539191 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6ee00a2ea0 msgr2=0x7f6ee00a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:48.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.940+0000 7f6efe377640 1 --2- 172.21.15.44:0/4098539191 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6ee00a2ea0 0x7f6ee00a5360 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8009ae0 tx=0x7f6ee8005f90 comp rx=0 tx=0).stop 2024-04-21T23:19:48.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.940+0000 7f6efe377640 1 -- 172.21.15.44:0/4098539191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ef81020e0 msgr2=0x7f6ef81985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:48.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.940+0000 7f6efe377640 1 --2- 172.21.15.44:0/4098539191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ef81020e0 0x7f6ef81985a0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f6eec009230 tx=0x7f6eec02cb30 comp rx=0 tx=0).stop 2024-04-21T23:19:48.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.940+0000 7f6efe377640 1 -- 172.21.15.44:0/4098539191 shutdown_connections 2024-04-21T23:19:48.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.940+0000 7f6efe377640 1 --2- 172.21.15.44:0/4098539191 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f6ee00a2ea0 0x7f6ee00a5360 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.939 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.940+0000 7f6efe377640 1 --2- 172.21.15.44:0/4098539191 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6ef8107730 0x7f6ef819c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.940+0000 7f6efe377640 1 --2- 172.21.15.44:0/4098539191 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f6ef81020e0 0x7f6ef81985a0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.940+0000 7f6efe377640 1 --2- 172.21.15.44:0/4098539191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef8101720 0x7f6ef8198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:48.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.940+0000 7f6efe377640 1 -- 172.21.15.44:0/4098539191 >> 172.21.15.44:0/4098539191 conn(0x7f6ef80fd230 msgr2=0x7f6ef80fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:48.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.940+0000 7f6efe377640 1 -- 172.21.15.44:0/4098539191 shutdown_connections 2024-04-21T23:19:48.940 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:48.940+0000 7f6efe377640 1 -- 172.21.15.44:0/4098539191 wait complete. 2024-04-21T23:19:48.949 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-21T23:19:49.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.053+0000 7efcb43de640 1 -- 172.21.15.44:0/880400786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efcac100530 msgr2=0x7efcac100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:49.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.053+0000 7efcb43de640 1 --2- 172.21.15.44:0/880400786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efcac100530 0x7efcac100910 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7efca4009900 tx=0x7efca402bc30 comp rx=0 tx=0).stop 2024-04-21T23:19:49.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.054+0000 7efcb43de640 1 -- 172.21.15.44:0/880400786 shutdown_connections 2024-04-21T23:19:49.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.054+0000 7efcb43de640 1 --2- 172.21.15.44:0/880400786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcac100e50 0x7efcac10dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:49.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.054+0000 7efcb43de640 1 --2- 172.21.15.44:0/880400786 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efcac10e0e0 0x7efcac1104d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:49.054 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.054+0000 7efcb43de640 1 --2- 172.21.15.44:0/880400786 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efcac100530 0x7efcac100910 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:49.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.054+0000 7efcb43de640 1 -- 172.21.15.44:0/880400786 >> 172.21.15.44:0/880400786 conn(0x7efcac0777f0 msgr2=0x7efcac077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:49.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.054+0000 7efcb43de640 1 -- 172.21.15.44:0/880400786 shutdown_connections 2024-04-21T23:19:49.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.054+0000 7efcb43de640 1 -- 172.21.15.44:0/880400786 wait complete. 2024-04-21T23:19:49.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.054+0000 7efcb43de640 1 Processor -- start 2024-04-21T23:19:49.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.054+0000 7efcb43de640 1 -- start start 2024-04-21T23:19:49.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.054+0000 7efcb43de640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcac100530 0x7efcac19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:49.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.054+0000 7efcb43de640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efcac100e50 0x7efcac19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:49.055 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.054+0000 7efcb43de640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efcac10e0e0 0x7efcac19eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:49.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.054+0000 7efcb43de640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7efcac113d10 con 0x7efcac100e50 2024-04-21T23:19:49.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.054+0000 7efcb43de640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7efcac113b90 con 0x7efcac10e0e0 2024-04-21T23:19:49.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.054+0000 7efcb43de640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efcac113e90 con 0x7efcac100530 2024-04-21T23:19:49.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.055+0000 7efcb1979640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efcac100e50 0x7efcac19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:49.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.055+0000 7efcb1979640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efcac100e50 0x7efcac19a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48022/0 (socket says 172.21.15.44:48022) 2024-04-21T23:19:49.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.055+0000 7efcb1979640 1 -- 172.21.15.44:0/118283096 learned_addr learned my addr 172.21.15.44:0/118283096 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:49.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.055+0000 7efcb217a640 1 --2- 172.21.15.44:0/118283096 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcac100530 0x7efcac19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:49.056 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.055+0000 7efcb297b640 1 --2- 172.21.15.44:0/118283096 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efcac10e0e0 0x7efcac19eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:49.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.055+0000 7efcb1979640 1 -- 172.21.15.44:0/118283096 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcac100530 msgr2=0x7efcac19a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:49.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.055+0000 7efcb1979640 1 --2- 172.21.15.44:0/118283096 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcac100530 0x7efcac19a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:49.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.055+0000 7efcb1979640 1 -- 172.21.15.44:0/118283096 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efcac10e0e0 msgr2=0x7efcac19eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:49.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.055+0000 7efcb1979640 1 --2- 172.21.15.44:0/118283096 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efcac10e0e0 0x7efcac19eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:49.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.055+0000 7efcb1979640 1 -- 172.21.15.44:0/118283096 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcac19f220 con 0x7efcac100e50 2024-04-21T23:19:49.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.055+0000 7efcb1979640 1 --2- 172.21.15.44:0/118283096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efcac100e50 0x7efcac19a770 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00d770 tx=0x7efc9c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:49.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.055+0000 7efca37fe640 1 -- 172.21.15.44:0/118283096 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c017070 con 0x7efcac100e50 2024-04-21T23:19:49.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.055+0000 7efca37fe640 1 -- 172.21.15.44:0/118283096 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc9c005500 con 0x7efcac100e50 2024-04-21T23:19:49.057 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.055+0000 7efcb43de640 1 -- 172.21.15.44:0/118283096 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcac19f510 con 0x7efcac100e50 2024-04-21T23:19:49.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.055+0000 7efca37fe640 1 -- 172.21.15.44:0/118283096 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c004e00 con 0x7efcac100e50 2024-04-21T23:19:49.058 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.055+0000 7efcb43de640 1 -- 172.21.15.44:0/118283096 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcac1a6da0 con 0x7efcac100e50 2024-04-21T23:19:49.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.056+0000 7efcb43de640 1 -- 172.21.15.44:0/118283096 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcac0685f0 con 0x7efcac100e50 2024-04-21T23:19:49.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.057+0000 7efca37fe640 1 -- 172.21.15.44:0/118283096 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7efc9c004fa0 con 0x7efcac100e50 2024-04-21T23:19:49.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.058+0000 7efca37fe640 1 --2- 172.21.15.44:0/118283096 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7efc980a2d00 0x7efc980a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:49.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.058+0000 7efcb217a640 1 --2- 172.21.15.44:0/118283096 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7efc980a2d00 0x7efc980a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:49.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.058+0000 7efca37fe640 1 -- 172.21.15.44:0/118283096 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9533+0+0 (secure 0 0 0) 0x7efc9c0cf2e0 con 0x7efcac100e50 2024-04-21T23:19:49.059 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.058+0000 7efcb217a640 1 --2- 172.21.15.44:0/118283096 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7efc980a2d00 0x7efc980a51c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efca402efd0 tx=0x7efca402cf90 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:49.061 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.062+0000 7efca37fe640 1 -- 172.21.15.44:0/118283096 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7efc9c09bab0 con 0x7efcac100e50 2024-04-21T23:19:49.285 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.287+0000 7efcb43de640 1 -- 172.21.15.44:0/118283096 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7efcac109270 con 0x7efcac100e50 2024-04-21T23:19:49.288 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.289+0000 7efca37fe640 1 -- 172.21.15.44:0/118283096 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v32) ==== 161+0+0 (secure 0 0 0) 0x7efc9c09b3b0 con 0x7efcac100e50 2024-04-21T23:19:49.288 INFO:teuthology.orchestra.run.smithi044.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-21T23:19:49.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.292+0000 7efcb43de640 1 -- 172.21.15.44:0/118283096 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7efc980a2d00 msgr2=0x7efc980a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:49.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.292+0000 7efcb43de640 1 --2- 172.21.15.44:0/118283096 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7efc980a2d00 0x7efc980a51c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efca402efd0 tx=0x7efca402cf90 comp rx=0 tx=0).stop 2024-04-21T23:19:49.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.292+0000 7efcb43de640 1 -- 172.21.15.44:0/118283096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efcac100e50 msgr2=0x7efcac19a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:49.291 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.292+0000 7efcb43de640 1 --2- 172.21.15.44:0/118283096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efcac100e50 0x7efcac19a770 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00d770 tx=0x7efc9c00dc30 comp rx=0 tx=0).stop 2024-04-21T23:19:49.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.293+0000 7efcb43de640 1 -- 172.21.15.44:0/118283096 shutdown_connections 2024-04-21T23:19:49.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.293+0000 7efcb43de640 1 --2- 172.21.15.44:0/118283096 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7efc980a2d00 0x7efc980a51c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:49.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.293+0000 7efcb43de640 1 --2- 172.21.15.44:0/118283096 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efcac10e0e0 0x7efcac19eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:49.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.293+0000 7efcb43de640 1 --2- 172.21.15.44:0/118283096 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7efcac100e50 0x7efcac19a770 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:49.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.293+0000 7efcb43de640 1 --2- 172.21.15.44:0/118283096 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcac100530 0x7efcac19a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:49.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.293+0000 7efcb43de640 1 -- 172.21.15.44:0/118283096 >> 172.21.15.44:0/118283096 conn(0x7efcac0777f0 msgr2=0x7efcac10f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:49.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.293+0000 7efcb43de640 1 -- 172.21.15.44:0/118283096 shutdown_connections 2024-04-21T23:19:49.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.293+0000 7efcb43de640 1 -- 172.21.15.44:0/118283096 wait complete. 2024-04-21T23:19:49.302 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-21T23:19:49.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.410+0000 7f0965215640 1 -- 172.21.15.44:0/743607895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0960100f90 msgr2=0x7f096010d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:49.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.410+0000 7f0965215640 1 --2- 172.21.15.44:0/743607895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0960100f90 0x7f096010d4a0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f0958009cd0 tx=0x7f095802c0f0 comp rx=0 tx=0).stop 2024-04-21T23:19:49.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.410+0000 7f0965215640 1 -- 172.21.15.44:0/743607895 shutdown_connections 2024-04-21T23:19:49.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.410+0000 7f0965215640 1 --2- 172.21.15.44:0/743607895 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0960100f90 0x7f096010d4a0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:49.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.410+0000 7f0965215640 1 --2- 172.21.15.44:0/743607895 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09601005d0 0x7f0960100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:49.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.410+0000 7f0965215640 1 --2- 172.21.15.44:0/743607895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0960106670 0x7f0960106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:49.409 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.410+0000 7f0965215640 1 -- 172.21.15.44:0/743607895 >> 172.21.15.44:0/743607895 conn(0x7f0960077790 msgr2=0x7f0960077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:49.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.410+0000 7f0965215640 1 -- 172.21.15.44:0/743607895 shutdown_connections 2024-04-21T23:19:49.410 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.410+0000 7f0965215640 1 -- 172.21.15.44:0/743607895 wait complete. 2024-04-21T23:19:49.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.411+0000 7f0965215640 1 Processor -- start 2024-04-21T23:19:49.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.411+0000 7f0965215640 1 -- start start 2024-04-21T23:19:49.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.411+0000 7f0965215640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09601005d0 0x7f0960197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:49.412 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.411+0000 7f0965215640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0960100f90 0x7f0960198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:49.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.411+0000 7f0965215640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0960106670 0x7f096019c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:49.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.411+0000 7f0965215640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f09601109b0 con 0x7f0960100f90 2024-04-21T23:19:49.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.411+0000 7f0965215640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f0960110830 con 0x7f0960106670 2024-04-21T23:19:49.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.411+0000 7f0965215640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0960110b30 con 0x7f09601005d0 2024-04-21T23:19:49.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.411+0000 7f095e59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0960100f90 0x7f0960198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:49.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.411+0000 7f095e59c640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0960100f90 0x7f0960198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48048/0 (socket says 172.21.15.44:48048) 2024-04-21T23:19:49.413 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.411+0000 7f095e59c640 1 -- 172.21.15.44:0/569891164 learned_addr learned my addr 172.21.15.44:0/569891164 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:49.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.411+0000 7f095ed9d640 1 --2- 172.21.15.44:0/569891164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09601005d0 0x7f0960197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:49.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.412+0000 7f095f59e640 1 --2- 172.21.15.44:0/569891164 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0960106670 0x7f096019c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:49.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.412+0000 7f095e59c640 1 -- 172.21.15.44:0/569891164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09601005d0 msgr2=0x7f0960197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:49.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.412+0000 7f095e59c640 1 --2- 172.21.15.44:0/569891164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09601005d0 0x7f0960197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:49.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.412+0000 7f095e59c640 1 -- 172.21.15.44:0/569891164 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0960106670 msgr2=0x7f096019c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:49.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.412+0000 7f095e59c640 1 --2- 172.21.15.44:0/569891164 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0960106670 0x7f096019c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:49.414 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.412+0000 7f095e59c640 1 -- 172.21.15.44:0/569891164 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f096019cef0 con 0x7f0960100f90 2024-04-21T23:19:49.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.412+0000 7f095e59c640 1 --2- 172.21.15.44:0/569891164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0960100f90 0x7f0960198440 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f095400d950 tx=0x7f095400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:49.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.412+0000 7f0947fff640 1 -- 172.21.15.44:0/569891164 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0954017030 con 0x7f0960100f90 2024-04-21T23:19:49.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.412+0000 7f0947fff640 1 -- 172.21.15.44:0/569891164 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0954004020 con 0x7f0960100f90 2024-04-21T23:19:49.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.412+0000 7f0965215640 1 -- 172.21.15.44:0/569891164 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f096019d1e0 con 0x7f0960100f90 2024-04-21T23:19:49.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.412+0000 7f0947fff640 1 -- 172.21.15.44:0/569891164 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09540047a0 con 0x7f0960100f90 2024-04-21T23:19:49.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.412+0000 7f0965215640 1 -- 172.21.15.44:0/569891164 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09601a4a70 con 0x7f0960100f90 2024-04-21T23:19:49.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.413+0000 7f0965215640 1 -- 172.21.15.44:0/569891164 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0960101d10 con 0x7f0960100f90 2024-04-21T23:19:49.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.414+0000 7f0947fff640 1 -- 172.21.15.44:0/569891164 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f09540041d0 con 0x7f0960100f90 2024-04-21T23:19:49.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.415+0000 7f0947fff640 1 --2- 172.21.15.44:0/569891164 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f09480a2d00 0x7f09480a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:49.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.415+0000 7f095ed9d640 1 --2- 172.21.15.44:0/569891164 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f09480a2d00 0x7f09480a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:49.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.415+0000 7f0947fff640 1 -- 172.21.15.44:0/569891164 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9531+0+0 (secure 0 0 0) 0x7f09540cf8c0 con 0x7f0960100f90 2024-04-21T23:19:49.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.416+0000 7f095ed9d640 1 --2- 172.21.15.44:0/569891164 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f09480a2d00 0x7f09480a51c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f09500058c0 tx=0x7f095000a2d0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:49.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.419+0000 7f0947fff640 1 -- 172.21.15.44:0/569891164 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f095409c0f0 con 0x7f0960100f90 2024-04-21T23:19:49.577 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:49 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/3024951347' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:19:49.577 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:49 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/1373738895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:19:49.577 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:49 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/4098539191' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:19:49.578 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:49 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-a[27930]: 2024-04-21T23:19:49.288+0000 7f5e04e39700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T23:19:49.645 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:49.647+0000 7f0965215640 1 -- 172.21.15.44:0/569891164 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f09601a4d50 con 0x7f0960100f90 2024-04-21T23:19:49.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:49 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/3024951347' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:19:49.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:49 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/1373738895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:19:49.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:49 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/4098539191' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:19:49.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:49 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/3024951347' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:19:49.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:49 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/1373738895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T23:19:49.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:49 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/4098539191' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T23:19:49.716 INFO:journalctl@ceph.mds.b.smithi055.stdout:Apr 21 23:19:49 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-b[61560]: ignoring --setuser ceph since I am not root 2024-04-21T23:19:49.716 INFO:journalctl@ceph.mds.b.smithi055.stdout:Apr 21 23:19:49 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-b[61560]: ignoring --setgroup ceph since I am not root 2024-04-21T23:19:49.716 INFO:journalctl@ceph.mds.b.smithi055.stdout:Apr 21 23:19:49 smithi055 ceph-mds[61564]: -- 172.21.15.55:0/3558535995 <== mon.2 v2:172.21.15.55:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561cc2eb9a40 con 0x561cc2e8cc00 2024-04-21T23:19:49.716 INFO:journalctl@ceph.mds.b.smithi055.stdout:Apr 21 23:19:49 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-b[61560]: starting mds.b at 2024-04-21T23:19:49.717 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:19:49 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: ignoring --setuser ceph since I am not root 2024-04-21T23:19:49.717 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:19:49 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: ignoring --setgroup ceph since I am not root 2024-04-21T23:19:49.717 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:19:49 smithi069 ceph-mds[63204]: -- 172.21.15.69:0/3483746193 <== mon.1 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5557428d5a40 con 0x5557428a9000 2024-04-21T23:19:49.717 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:19:49 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: starting mds.i at 2024-04-21T23:19:50.289 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.291+0000 7f0947fff640 1 -- 172.21.15.44:0/569891164 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v33) ==== 104+0+0 (secure 0 0 0) 0x7f095409b9f0 con 0x7f0960100f90 2024-04-21T23:19:50.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.293+0000 7f0965215640 1 -- 172.21.15.44:0/569891164 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f09480a2d00 msgr2=0x7f09480a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:50.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.293+0000 7f0965215640 1 --2- 172.21.15.44:0/569891164 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f09480a2d00 0x7f09480a51c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f09500058c0 tx=0x7f095000a2d0 comp rx=0 tx=0).stop 2024-04-21T23:19:50.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.293+0000 7f0965215640 1 -- 172.21.15.44:0/569891164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0960100f90 msgr2=0x7f0960198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:50.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.293+0000 7f0965215640 1 --2- 172.21.15.44:0/569891164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0960100f90 0x7f0960198440 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f095400d950 tx=0x7f095400de10 comp rx=0 tx=0).stop 2024-04-21T23:19:50.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.293+0000 7f0965215640 1 -- 172.21.15.44:0/569891164 shutdown_connections 2024-04-21T23:19:50.292 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.293+0000 7f0965215640 1 --2- 172.21.15.44:0/569891164 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f09480a2d00 0x7f09480a51c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:50.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.293+0000 7f0965215640 1 --2- 172.21.15.44:0/569891164 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0960106670 0x7f096019c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:50.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.293+0000 7f0965215640 1 --2- 172.21.15.44:0/569891164 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f0960100f90 0x7f0960198440 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:50.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.293+0000 7f0965215640 1 --2- 172.21.15.44:0/569891164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09601005d0 0x7f0960197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:50.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.293+0000 7f0965215640 1 -- 172.21.15.44:0/569891164 >> 172.21.15.44:0/569891164 conn(0x7f0960077790 msgr2=0x7f09600ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:50.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.293+0000 7f0965215640 1 -- 172.21.15.44:0/569891164 shutdown_connections 2024-04-21T23:19:50.293 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.293+0000 7f0965215640 1 -- 172.21.15.44:0/569891164 wait complete. 2024-04-21T23:19:50.304 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-21T23:19:50.415 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.414+0000 7f968edc9640 1 -- 172.21.15.44:0/1627997041 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96881066b0 msgr2=0x7f9688106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:50.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.414+0000 7f968edc9640 1 --2- 172.21.15.44:0/1627997041 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96881066b0 0x7f9688106a90 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f967c009930 tx=0x7f967c02bc30 comp rx=0 tx=0).stop 2024-04-21T23:19:50.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.415+0000 7f968edc9640 1 -- 172.21.15.44:0/1627997041 shutdown_connections 2024-04-21T23:19:50.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.415+0000 7f968edc9640 1 --2- 172.21.15.44:0/1627997041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9688100610 0x7f9688100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:50.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.415+0000 7f968edc9640 1 --2- 172.21.15.44:0/1627997041 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9688100fd0 0x7f968810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:50.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.415+0000 7f968edc9640 1 --2- 172.21.15.44:0/1627997041 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f96881066b0 0x7f9688106a90 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:50.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.415+0000 7f968edc9640 1 -- 172.21.15.44:0/1627997041 >> 172.21.15.44:0/1627997041 conn(0x7f96880777f0 msgr2=0x7f9688077c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:50.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.415+0000 7f968edc9640 1 -- 172.21.15.44:0/1627997041 shutdown_connections 2024-04-21T23:19:50.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.415+0000 7f968edc9640 1 -- 172.21.15.44:0/1627997041 wait complete. 2024-04-21T23:19:50.416 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.415+0000 7f968edc9640 1 Processor -- start 2024-04-21T23:19:50.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.415+0000 7f968edc9640 1 -- start start 2024-04-21T23:19:50.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.415+0000 7f968edc9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9688100610 0x7f9688198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:50.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.415+0000 7f968edc9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9688100fd0 0x7f96881985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:50.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.415+0000 7f968edc9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96881066b0 0x7f968819c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:50.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.415+0000 7f968edc9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f9688110a30 con 0x7f9688100610 2024-04-21T23:19:50.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.415+0000 7f968edc9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f96881108b0 con 0x7f96881066b0 2024-04-21T23:19:50.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.415+0000 7f968edc9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9688110bb0 con 0x7f9688100fd0 2024-04-21T23:19:50.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.416+0000 7f968cb65640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9688100610 0x7f9688198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:50.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.416+0000 7f968cb65640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9688100610 0x7f9688198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:48070/0 (socket says 172.21.15.44:48070) 2024-04-21T23:19:50.417 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.416+0000 7f968cb65640 1 -- 172.21.15.44:0/2450949926 learned_addr learned my addr 172.21.15.44:0/2450949926 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:50.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.416+0000 7f9687fff640 1 --2- 172.21.15.44:0/2450949926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9688100fd0 0x7f96881985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:50.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.416+0000 7f968d366640 1 --2- 172.21.15.44:0/2450949926 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96881066b0 0x7f968819c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:50.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.416+0000 7f968cb65640 1 -- 172.21.15.44:0/2450949926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9688100fd0 msgr2=0x7f96881985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:50.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.416+0000 7f968cb65640 1 --2- 172.21.15.44:0/2450949926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9688100fd0 0x7f96881985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:50.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.416+0000 7f968cb65640 1 -- 172.21.15.44:0/2450949926 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96881066b0 msgr2=0x7f968819c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:50.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.416+0000 7f968cb65640 1 --2- 172.21.15.44:0/2450949926 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96881066b0 0x7f968819c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:50.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.416+0000 7f968cb65640 1 -- 172.21.15.44:0/2450949926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f968819d080 con 0x7f9688100610 2024-04-21T23:19:50.418 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.416+0000 7f968d366640 1 --2- 172.21.15.44:0/2450949926 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96881066b0 0x7f968819c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T23:19:50.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.416+0000 7f968cb65640 1 --2- 172.21.15.44:0/2450949926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9688100610 0x7f9688198090 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f967c009230 tx=0x7f967c02ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:50.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.416+0000 7f9685ffb640 1 -- 172.21.15.44:0/2450949926 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f967c041070 con 0x7f9688100610 2024-04-21T23:19:50.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.416+0000 7f9685ffb640 1 -- 172.21.15.44:0/2450949926 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f967c0054e0 con 0x7f9688100610 2024-04-21T23:19:50.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.416+0000 7f968edc9640 1 -- 172.21.15.44:0/2450949926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f968819d310 con 0x7f9688100610 2024-04-21T23:19:50.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.416+0000 7f9685ffb640 1 -- 172.21.15.44:0/2450949926 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f967c034440 con 0x7f9688100610 2024-04-21T23:19:50.419 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.416+0000 7f968edc9640 1 -- 172.21.15.44:0/2450949926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96881a4c00 con 0x7f9688100610 2024-04-21T23:19:50.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.417+0000 7f968edc9640 1 -- 172.21.15.44:0/2450949926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96880685f0 con 0x7f9688100610 2024-04-21T23:19:50.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.419+0000 7f9685ffb640 1 -- 172.21.15.44:0/2450949926 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f967c044070 con 0x7f9688100610 2024-04-21T23:19:50.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.419+0000 7f9685ffb640 1 --2- 172.21.15.44:0/2450949926 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f96780a2d00 0x7f96780a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:50.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.419+0000 7f9685ffb640 1 -- 172.21.15.44:0/2450949926 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9531+0+0 (secure 0 0 0) 0x7f967c0f1760 con 0x7f9688100610 2024-04-21T23:19:50.420 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.420+0000 7f9687fff640 1 --2- 172.21.15.44:0/2450949926 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f96780a2d00 0x7f96780a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:50.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.420+0000 7f9687fff640 1 --2- 172.21.15.44:0/2450949926 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f96780a2d00 0x7f96780a51c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9688077df0 tx=0x7f9674008040 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:50.421 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.423+0000 7f9685ffb640 1 -- 172.21.15.44:0/2450949926 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f967c0be130 con 0x7f9688100610 2024-04-21T23:19:50.552 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:50 smithi044 ceph-mon[27934]: pgmap v347: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-04-21T23:19:50.552 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:50 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/118283096' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T23:19:50.552 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:50 smithi044 ceph-mon[27934]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-21T23:19:50.553 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:50 smithi044 ceph-mon[27934]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T23:19:50.553 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:50 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/118283096' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-21T23:19:50.553 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:50 smithi044 ceph-mon[27934]: osdmap e111: 12 total, 12 up, 12 in 2024-04-21T23:19:50.553 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:50 smithi044 ceph-mon[27934]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-04-21T23:19:50.553 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:50 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/569891164' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:19:50.609 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:50.611+0000 7f968edc9640 1 -- 172.21.15.44:0/2450949926 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f96881a4dd0 con 0x7f9688100610 2024-04-21T23:19:50.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:50 smithi069 ceph-mon[31816]: pgmap v347: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-04-21T23:19:50.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:50 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/118283096' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T23:19:50.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:50 smithi069 ceph-mon[31816]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-21T23:19:50.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:50 smithi069 ceph-mon[31816]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T23:19:50.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:50 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/118283096' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-21T23:19:50.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:50 smithi069 ceph-mon[31816]: osdmap e111: 12 total, 12 up, 12 in 2024-04-21T23:19:50.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:50 smithi069 ceph-mon[31816]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-04-21T23:19:50.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:50 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/569891164' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:19:50.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:50 smithi055 ceph-mon[31117]: pgmap v347: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-04-21T23:19:50.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:50 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/118283096' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T23:19:50.716 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:50 smithi055 ceph-mon[31117]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-21T23:19:50.716 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:50 smithi055 ceph-mon[31117]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T23:19:50.716 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:50 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/118283096' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-21T23:19:50.716 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:50 smithi055 ceph-mon[31117]: osdmap e111: 12 total, 12 up, 12 in 2024-04-21T23:19:50.716 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:50 smithi055 ceph-mon[31117]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-04-21T23:19:50.716 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:50 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/569891164' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T23:19:51.294 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.295+0000 7f9685ffb640 1 -- 172.21.15.44:0/2450949926 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v112) ==== 181+0+0 (secure 0 0 0) 0x7f967c0bda30 con 0x7f9688100610 2024-04-21T23:19:51.295 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_metadata' removed 2024-04-21T23:19:51.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.299+0000 7f968edc9640 1 -- 172.21.15.44:0/2450949926 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f96780a2d00 msgr2=0x7f96780a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:51.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.299+0000 7f968edc9640 1 --2- 172.21.15.44:0/2450949926 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f96780a2d00 0x7f96780a51c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9688077df0 tx=0x7f9674008040 comp rx=0 tx=0).stop 2024-04-21T23:19:51.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.299+0000 7f968edc9640 1 -- 172.21.15.44:0/2450949926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9688100610 msgr2=0x7f9688198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:51.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.299+0000 7f968edc9640 1 --2- 172.21.15.44:0/2450949926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9688100610 0x7f9688198090 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f967c009230 tx=0x7f967c02ce20 comp rx=0 tx=0).stop 2024-04-21T23:19:51.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.299+0000 7f968edc9640 1 -- 172.21.15.44:0/2450949926 shutdown_connections 2024-04-21T23:19:51.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.299+0000 7f968edc9640 1 --2- 172.21.15.44:0/2450949926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9688100fd0 0x7f96881985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:51.298 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.299+0000 7f968edc9640 1 --2- 172.21.15.44:0/2450949926 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f96780a2d00 0x7f96780a51c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:51.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.299+0000 7f968edc9640 1 --2- 172.21.15.44:0/2450949926 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f96881066b0 0x7f968819c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:51.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.299+0000 7f968edc9640 1 --2- 172.21.15.44:0/2450949926 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f9688100610 0x7f9688198090 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:51.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.299+0000 7f968edc9640 1 -- 172.21.15.44:0/2450949926 >> 172.21.15.44:0/2450949926 conn(0x7f96880777f0 msgr2=0x7f96880ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:51.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.299+0000 7f968edc9640 1 -- 172.21.15.44:0/2450949926 shutdown_connections 2024-04-21T23:19:51.299 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.299+0000 7f968edc9640 1 -- 172.21.15.44:0/2450949926 wait complete. 2024-04-21T23:19:51.313 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-21T23:19:51.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.427+0000 7feb2edc9640 1 -- 172.21.15.44:0/1324214334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb28103360 msgr2=0x7feb2810b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:51.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.427+0000 7feb2edc9640 1 --2- 172.21.15.44:0/1324214334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb28103360 0x7feb2810b7b0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7feb18009900 tx=0x7feb1802bc30 comp rx=0 tx=0).stop 2024-04-21T23:19:51.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.427+0000 7feb2edc9640 1 -- 172.21.15.44:0/1324214334 shutdown_connections 2024-04-21T23:19:51.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.427+0000 7feb2edc9640 1 --2- 172.21.15.44:0/1324214334 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb2810bec0 0x7feb2810e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:51.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.427+0000 7feb2edc9640 1 --2- 172.21.15.44:0/1324214334 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb28103360 0x7feb2810b7b0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:51.428 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.427+0000 7feb2edc9640 1 --2- 172.21.15.44:0/1324214334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb281029b0 0x7feb28102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.427+0000 7feb2edc9640 1 -- 172.21.15.44:0/1324214334 >> 172.21.15.44:0/1324214334 conn(0x7feb28077760 msgr2=0x7feb28077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.427+0000 7feb2edc9640 1 -- 172.21.15.44:0/1324214334 shutdown_connections 2024-04-21T23:19:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.427+0000 7feb2edc9640 1 -- 172.21.15.44:0/1324214334 wait complete. 2024-04-21T23:19:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.428+0000 7feb2edc9640 1 Processor -- start 2024-04-21T23:19:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.428+0000 7feb2edc9640 1 -- start start 2024-04-21T23:19:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.428+0000 7feb2edc9640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb281029b0 0x7feb28198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.428+0000 7feb2edc9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb2810bec0 0x7feb28198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:51.429 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.428+0000 7feb2edc9640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb2819cb10 0x7feb2819cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:51.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.428+0000 7feb2edc9640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7feb28111d10 con 0x7feb2819cb10 2024-04-21T23:19:51.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.428+0000 7feb2edc9640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7feb28111b90 con 0x7feb281029b0 2024-04-21T23:19:51.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.428+0000 7feb2edc9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7feb28111e90 con 0x7feb2810bec0 2024-04-21T23:19:51.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.428+0000 7feb2d366640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb2819cb10 0x7feb2819cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:51.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.428+0000 7feb2d366640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb2819cb10 0x7feb2819cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45380/0 (socket says 172.21.15.44:45380) 2024-04-21T23:19:51.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.428+0000 7feb2d366640 1 -- 172.21.15.44:0/2446205440 learned_addr learned my addr 172.21.15.44:0/2446205440 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:51.430 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.428+0000 7feb1ffff640 1 --2- 172.21.15.44:0/2446205440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb2810bec0 0x7feb28198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:51.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.428+0000 7feb2cb65640 1 --2- 172.21.15.44:0/2446205440 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb281029b0 0x7feb28198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:51.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.428+0000 7feb2d366640 1 -- 172.21.15.44:0/2446205440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb2810bec0 msgr2=0x7feb28198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:51.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.428+0000 7feb2d366640 1 --2- 172.21.15.44:0/2446205440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb2810bec0 0x7feb28198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:51.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.428+0000 7feb2d366640 1 -- 172.21.15.44:0/2446205440 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb281029b0 msgr2=0x7feb28198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:51.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.428+0000 7feb2d366640 1 --2- 172.21.15.44:0/2446205440 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb281029b0 0x7feb28198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:51.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.428+0000 7feb2d366640 1 -- 172.21.15.44:0/2446205440 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb2819d600 con 0x7feb2819cb10 2024-04-21T23:19:51.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.429+0000 7feb2d366640 1 --2- 172.21.15.44:0/2446205440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb2819cb10 0x7feb2819cfc0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7feb2000b970 tx=0x7feb2000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:51.431 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.429+0000 7feb1dffb640 1 -- 172.21.15.44:0/2446205440 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb20019070 con 0x7feb2819cb10 2024-04-21T23:19:51.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.429+0000 7feb1dffb640 1 -- 172.21.15.44:0/2446205440 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb2000cb40 con 0x7feb2819cb10 2024-04-21T23:19:51.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.429+0000 7feb2edc9640 1 -- 172.21.15.44:0/2446205440 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb2819d8f0 con 0x7feb2819cb10 2024-04-21T23:19:51.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.429+0000 7feb1dffb640 1 -- 172.21.15.44:0/2446205440 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb20007650 con 0x7feb2819cb10 2024-04-21T23:19:51.432 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.429+0000 7feb2edc9640 1 -- 172.21.15.44:0/2446205440 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb281a50b0 con 0x7feb2819cb10 2024-04-21T23:19:51.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.430+0000 7feb2edc9640 1 -- 172.21.15.44:0/2446205440 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb28068a40 con 0x7feb2819cb10 2024-04-21T23:19:51.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.431+0000 7feb1dffb640 1 -- 172.21.15.44:0/2446205440 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7feb2001c070 con 0x7feb2819cb10 2024-04-21T23:19:51.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.432+0000 7feb1dffb640 1 --2- 172.21.15.44:0/2446205440 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7feb100a2d80 0x7feb100a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:51.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.432+0000 7feb2cb65640 1 --2- 172.21.15.44:0/2446205440 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7feb100a2d80 0x7feb100a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:51.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.432+0000 7feb1dffb640 1 -- 172.21.15.44:0/2446205440 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8977+0+0 (secure 0 0 0) 0x7feb200ceac0 con 0x7feb2819cb10 2024-04-21T23:19:51.433 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.432+0000 7feb2cb65640 1 --2- 172.21.15.44:0/2446205440 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7feb100a2d80 0x7feb100a5240 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7feb24005fd0 tx=0x7feb24005850 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:51.434 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.436+0000 7feb1dffb640 1 -- 172.21.15.44:0/2446205440 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7feb2009af70 con 0x7feb2819cb10 2024-04-21T23:19:51.625 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:51.627+0000 7feb2edc9640 1 -- 172.21.15.44:0/2446205440 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7feb28106e70 con 0x7feb2819cb10 2024-04-21T23:19:51.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:51 smithi044 ceph-mon[27934]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-21T23:19:51.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:51 smithi044 ceph-mon[27934]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-21T23:19:51.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:51 smithi044 ceph-mon[27934]: Cluster is now healthy 2024-04-21T23:19:51.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:51 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/569891164' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-21T23:19:51.686 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:51 smithi044 ceph-mon[27934]: mds.? [v2:172.21.15.69:6838/3355749937,v1:172.21.15.69:6839/3355749937] up:boot 2024-04-21T23:19:51.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:51 smithi044 ceph-mon[27934]: mds.? [v2:172.21.15.55:6838/2098809400,v1:172.21.15.55:6839/2098809400] up:boot 2024-04-21T23:19:51.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:51 smithi044 ceph-mon[27934]: fsmap 12 up:standby 2024-04-21T23:19:51.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:51 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:19:51.687 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:51 smithi044 ceph-mon[27934]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:19:51.688 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:51 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2450949926' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T23:19:51.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:51 smithi055 ceph-mon[31117]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-21T23:19:51.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:51 smithi055 ceph-mon[31117]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-21T23:19:51.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:51 smithi055 ceph-mon[31117]: Cluster is now healthy 2024-04-21T23:19:51.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:51 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/569891164' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-21T23:19:51.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:51 smithi055 ceph-mon[31117]: mds.? [v2:172.21.15.69:6838/3355749937,v1:172.21.15.69:6839/3355749937] up:boot 2024-04-21T23:19:51.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:51 smithi055 ceph-mon[31117]: mds.? [v2:172.21.15.55:6838/2098809400,v1:172.21.15.55:6839/2098809400] up:boot 2024-04-21T23:19:51.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:51 smithi055 ceph-mon[31117]: fsmap 12 up:standby 2024-04-21T23:19:51.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:51 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:19:51.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:51 smithi055 ceph-mon[31117]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:19:51.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:51 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2450949926' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T23:19:51.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:51 smithi069 ceph-mon[31816]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-21T23:19:51.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:51 smithi069 ceph-mon[31816]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-21T23:19:51.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:51 smithi069 ceph-mon[31816]: Cluster is now healthy 2024-04-21T23:19:51.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:51 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/569891164' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-21T23:19:51.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:51 smithi069 ceph-mon[31816]: mds.? [v2:172.21.15.69:6838/3355749937,v1:172.21.15.69:6839/3355749937] up:boot 2024-04-21T23:19:51.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:51 smithi069 ceph-mon[31816]: mds.? [v2:172.21.15.55:6838/2098809400,v1:172.21.15.55:6839/2098809400] up:boot 2024-04-21T23:19:51.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:51 smithi069 ceph-mon[31816]: fsmap 12 up:standby 2024-04-21T23:19:51.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:51 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T23:19:51.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:51 smithi069 ceph-mon[31816]: from='mgr.15396 172.21.15.44:0/1312371081' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T23:19:51.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:51 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2450949926' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T23:19:52.302 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.304+0000 7feb1dffb640 1 -- 172.21.15.44:0/2446205440 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v113) ==== 169+0+0 (secure 0 0 0) 0x7feb2009f270 con 0x7feb2819cb10 2024-04-21T23:19:52.303 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data' removed 2024-04-21T23:19:52.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.308+0000 7feb2edc9640 1 -- 172.21.15.44:0/2446205440 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7feb100a2d80 msgr2=0x7feb100a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:52.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.308+0000 7feb2edc9640 1 --2- 172.21.15.44:0/2446205440 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7feb100a2d80 0x7feb100a5240 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7feb24005fd0 tx=0x7feb24005850 comp rx=0 tx=0).stop 2024-04-21T23:19:52.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.308+0000 7feb2edc9640 1 -- 172.21.15.44:0/2446205440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb2819cb10 msgr2=0x7feb2819cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:52.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.308+0000 7feb2edc9640 1 --2- 172.21.15.44:0/2446205440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb2819cb10 0x7feb2819cfc0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7feb2000b970 tx=0x7feb2000be30 comp rx=0 tx=0).stop 2024-04-21T23:19:52.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.308+0000 7feb2edc9640 1 -- 172.21.15.44:0/2446205440 shutdown_connections 2024-04-21T23:19:52.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.308+0000 7feb2edc9640 1 --2- 172.21.15.44:0/2446205440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb2810bec0 0x7feb28198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.308+0000 7feb2edc9640 1 --2- 172.21.15.44:0/2446205440 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7feb100a2d80 0x7feb100a5240 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.307 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.308+0000 7feb2edc9640 1 --2- 172.21.15.44:0/2446205440 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7feb2819cb10 0x7feb2819cfc0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.308+0000 7feb2edc9640 1 --2- 172.21.15.44:0/2446205440 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7feb281029b0 0x7feb28198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.308+0000 7feb2edc9640 1 -- 172.21.15.44:0/2446205440 >> 172.21.15.44:0/2446205440 conn(0x7feb28077760 msgr2=0x7feb280fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:52.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.308+0000 7feb2edc9640 1 -- 172.21.15.44:0/2446205440 shutdown_connections 2024-04-21T23:19:52.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.308+0000 7feb2edc9640 1 -- 172.21.15.44:0/2446205440 wait complete. 2024-04-21T23:19:52.326 DEBUG:teuthology.orchestra.run.smithi044:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-21T23:19:52.451 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.451+0000 7f82a078a640 1 -- 172.21.15.44:0/3434779932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f82981029b0 msgr2=0x7f8298102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:52.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.451+0000 7f82a078a640 1 --2- 172.21.15.44:0/3434779932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f82981029b0 0x7f8298102d90 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f8294009900 tx=0x7f829402bc30 comp rx=0 tx=0).stop 2024-04-21T23:19:52.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.451+0000 7f82a078a640 1 -- 172.21.15.44:0/3434779932 shutdown_connections 2024-04-21T23:19:52.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.451+0000 7f82a078a640 1 --2- 172.21.15.44:0/3434779932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8298103360 0x7f829810b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.451+0000 7f82a078a640 1 --2- 172.21.15.44:0/3434779932 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f829810bec0 0x7f829810e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.451+0000 7f82a078a640 1 --2- 172.21.15.44:0/3434779932 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f82981029b0 0x7f8298102d90 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.451+0000 7f82a078a640 1 -- 172.21.15.44:0/3434779932 >> 172.21.15.44:0/3434779932 conn(0x7f8298077760 msgr2=0x7f8298077b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:52.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.451+0000 7f82a078a640 1 -- 172.21.15.44:0/3434779932 shutdown_connections 2024-04-21T23:19:52.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.451+0000 7f82a078a640 1 -- 172.21.15.44:0/3434779932 wait complete. 2024-04-21T23:19:52.452 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.452+0000 7f82a078a640 1 Processor -- start 2024-04-21T23:19:52.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.452+0000 7f82a078a640 1 -- start start 2024-04-21T23:19:52.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.452+0000 7f82a078a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82981029b0 0x7f8298198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.452+0000 7f82a078a640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f829810bec0 0x7f8298198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.452+0000 7f82a078a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f829819cb10 0x7f829819cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.452+0000 7f82a078a640 1 -- --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_getmap magic: 0 -- 0x7f8298111d30 con 0x7f829810bec0 2024-04-21T23:19:52.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.452+0000 7f82a078a640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 -- 0x7f8298111bb0 con 0x7f82981029b0 2024-04-21T23:19:52.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.452+0000 7f82a078a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8298111eb0 con 0x7f829819cb10 2024-04-21T23:19:52.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.452+0000 7f829dd25640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f829810bec0 0x7f8298198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.453 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.452+0000 7f829dd25640 1 --2- >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f829810bec0 0x7f8298198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.44:3300/0 says I am v2:172.21.15.44:45398/0 (socket says 172.21.15.44:45398) 2024-04-21T23:19:52.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.452+0000 7f829dd25640 1 -- 172.21.15.44:0/4222216257 learned_addr learned my addr 172.21.15.44:0/4222216257 (peer_addr_for_me v2:172.21.15.44:0/0) 2024-04-21T23:19:52.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.452+0000 7f829ed27640 1 --2- 172.21.15.44:0/4222216257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f829819cb10 0x7f829819cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.452+0000 7f829e526640 1 --2- 172.21.15.44:0/4222216257 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82981029b0 0x7f8298198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.452+0000 7f829dd25640 1 -- 172.21.15.44:0/4222216257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f829819cb10 msgr2=0x7f829819cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:52.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.452+0000 7f829dd25640 1 --2- 172.21.15.44:0/4222216257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f829819cb10 0x7f829819cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.452+0000 7f829dd25640 1 -- 172.21.15.44:0/4222216257 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82981029b0 msgr2=0x7f8298198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:52.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.452+0000 7f829dd25640 1 --2- 172.21.15.44:0/4222216257 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82981029b0 0x7f8298198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:52.454 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.452+0000 7f829dd25640 1 -- 172.21.15.44:0/4222216257 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829819d600 con 0x7f829810bec0 2024-04-21T23:19:52.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.452+0000 7f829dd25640 1 --2- 172.21.15.44:0/4222216257 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f829810bec0 0x7f8298198790 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f828800d740 tx=0x7f828800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:52.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.453+0000 7f828f7fe640 1 -- 172.21.15.44:0/4222216257 <== mon.0 v2:172.21.15.44:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8288017070 con 0x7f829810bec0 2024-04-21T23:19:52.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.453+0000 7f82a078a640 1 -- 172.21.15.44:0/4222216257 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f829819d8f0 con 0x7f829810bec0 2024-04-21T23:19:52.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.453+0000 7f828f7fe640 1 -- 172.21.15.44:0/4222216257 <== mon.0 v2:172.21.15.44:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f828800de50 con 0x7f829810bec0 2024-04-21T23:19:52.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.453+0000 7f828f7fe640 1 -- 172.21.15.44:0/4222216257 <== mon.0 v2:172.21.15.44:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828800f6b0 con 0x7f829810bec0 2024-04-21T23:19:52.455 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.453+0000 7f82a078a640 1 -- 172.21.15.44:0/4222216257 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82981a5130 con 0x7f829810bec0 2024-04-21T23:19:52.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.454+0000 7f82a078a640 1 -- 172.21.15.44:0/4222216257 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f829806a630 con 0x7f829810bec0 2024-04-21T23:19:52.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.455+0000 7f828f7fe640 1 -- 172.21.15.44:0/4222216257 <== mon.0 v2:172.21.15.44:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f828801a070 con 0x7f829810bec0 2024-04-21T23:19:52.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.455+0000 7f828f7fe640 1 --2- 172.21.15.44:0/4222216257 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f82840a2d00 0x7f82840a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T23:19:52.456 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.456+0000 7f829e526640 1 --2- 172.21.15.44:0/4222216257 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f82840a2d00 0x7f82840a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T23:19:52.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.456+0000 7f828f7fe640 1 -- 172.21.15.44:0/4222216257 <== mon.0 v2:172.21.15.44:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8389+0+0 (secure 0 0 0) 0x7f82880ceed0 con 0x7f829810bec0 2024-04-21T23:19:52.457 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.456+0000 7f829e526640 1 --2- 172.21.15.44:0/4222216257 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f82840a2d00 0x7f82840a51c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8294006010 tx=0x7f8294000bc0 comp rx=0 tx=0).ready entity=mgr.15396 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T23:19:52.458 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.460+0000 7f828f7fe640 1 -- 172.21.15.44:0/4222216257 <== mon.0 v2:172.21.15.44:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186074 (secure 0 0 0) 0x7f828809bb70 con 0x7f829810bec0 2024-04-21T23:19:52.649 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:52.651+0000 7f82a078a640 1 -- 172.21.15.44:0/4222216257 --> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8298106e70 con 0x7f829810bec0 2024-04-21T23:19:52.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:52 smithi044 ceph-mon[27934]: pgmap v349: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:19:52.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:52 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2450949926' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T23:19:52.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:52 smithi044 ceph-mon[27934]: osdmap e112: 12 total, 12 up, 12 in 2024-04-21T23:19:52.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:52 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2446205440' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T23:19:52.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:52 smithi055 ceph-mon[31117]: pgmap v349: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:19:52.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:52 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2450949926' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T23:19:52.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:52 smithi055 ceph-mon[31117]: osdmap e112: 12 total, 12 up, 12 in 2024-04-21T23:19:52.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:52 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2446205440' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T23:19:52.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:52 smithi069 ceph-mon[31816]: pgmap v349: 129 pgs: 129 active+clean; 59 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T23:19:52.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:52 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2450949926' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T23:19:52.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:52 smithi069 ceph-mon[31816]: osdmap e112: 12 total, 12 up, 12 in 2024-04-21T23:19:52.715 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:52 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2446205440' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T23:19:53.308 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:53.310+0000 7f828f7fe640 1 -- 172.21.15.44:0/4222216257 <== mon.0 v2:172.21.15.44:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v114) ==== 178+0+0 (secure 0 0 0) 0x7f828809b470 con 0x7f829810bec0 2024-04-21T23:19:53.309 INFO:teuthology.orchestra.run.smithi044.stderr:pool 'cephfs_data_ec' removed 2024-04-21T23:19:53.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:53.314+0000 7f82a078a640 1 -- 172.21.15.44:0/4222216257 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f82840a2d00 msgr2=0x7f82840a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:53.314 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:53.314+0000 7f82a078a640 1 --2- 172.21.15.44:0/4222216257 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f82840a2d00 0x7f82840a51c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8294006010 tx=0x7f8294000bc0 comp rx=0 tx=0).stop 2024-04-21T23:19:53.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:53.314+0000 7f82a078a640 1 -- 172.21.15.44:0/4222216257 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f829810bec0 msgr2=0x7f8298198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T23:19:53.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:53.314+0000 7f82a078a640 1 --2- 172.21.15.44:0/4222216257 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f829810bec0 0x7f8298198790 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f828800d740 tx=0x7f828800dc00 comp rx=0 tx=0).stop 2024-04-21T23:19:53.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:53.314+0000 7f82a078a640 1 -- 172.21.15.44:0/4222216257 shutdown_connections 2024-04-21T23:19:53.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:53.314+0000 7f82a078a640 1 --2- 172.21.15.44:0/4222216257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f829819cb10 0x7f829819cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:53.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:53.314+0000 7f82a078a640 1 --2- 172.21.15.44:0/4222216257 >> [v2:172.21.15.44:6800/3789150048,v1:172.21.15.44:6801/3789150048] conn(0x7f82840a2d00 0x7f82840a51c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:53.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:53.314+0000 7f82a078a640 1 --2- 172.21.15.44:0/4222216257 >> [v2:172.21.15.44:3300/0,v1:172.21.15.44:6789/0] conn(0x7f829810bec0 0x7f8298198790 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:53.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:53.314+0000 7f82a078a640 1 --2- 172.21.15.44:0/4222216257 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82981029b0 0x7f8298198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T23:19:53.315 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:53.314+0000 7f82a078a640 1 -- 172.21.15.44:0/4222216257 >> 172.21.15.44:0/4222216257 conn(0x7f8298077760 msgr2=0x7f82980fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T23:19:53.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:53.314+0000 7f82a078a640 1 -- 172.21.15.44:0/4222216257 shutdown_connections 2024-04-21T23:19:53.316 INFO:teuthology.orchestra.run.smithi044.stderr:2024-04-21T23:19:53.314+0000 7f82a078a640 1 -- 172.21.15.44:0/4222216257 wait complete. 2024-04-21T23:19:53.333 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T23:19:53.360 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T23:19:53.410 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T23:19:53.445 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-21T23:19:53.445 DEBUG:teuthology.orchestra.run.smithi044:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-21T23:19:53.469 DEBUG:teuthology.orchestra.run.smithi055:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-21T23:19:53.485 DEBUG:teuthology.orchestra.run.smithi069:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-21T23:19:53.500 INFO:tasks.cephadm:Stopping all daemons... 2024-04-21T23:19:53.500 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-21T23:19:53.501 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@mon.a 2024-04-21T23:19:53.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:53 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/2446205440' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T23:19:53.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:53 smithi044 ceph-mon[27934]: osdmap e113: 12 total, 12 up, 12 in 2024-04-21T23:19:53.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:53 smithi044 ceph-mon[27934]: from='client.? 172.21.15.44:0/4222216257' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T23:19:53.685 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:53 smithi044 systemd[1]: Stopping Ceph mon.a for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:19:53.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:53 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/2446205440' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T23:19:53.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:53 smithi069 ceph-mon[31816]: osdmap e113: 12 total, 12 up, 12 in 2024-04-21T23:19:53.714 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:53 smithi069 ceph-mon[31816]: from='client.? 172.21.15.44:0/4222216257' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T23:19:53.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:53 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/2446205440' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T23:19:53.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:53 smithi055 ceph-mon[31117]: osdmap e113: 12 total, 12 up, 12 in 2024-04-21T23:19:53.715 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:53 smithi055 ceph-mon[31117]: from='client.? 172.21.15.44:0/4222216257' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T23:19:54.107 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:53 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-a[27930]: 2024-04-21T23:19:53.816+0000 7f5e0a644700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:19:54.108 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:53 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-a[27930]: 2024-04-21T23:19:53.816+0000 7f5e0a644700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-21T23:19:54.108 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:53 smithi044 podman[93098]: 2024-04-21 23:19:53.846980113 +0000 UTC m=+0.146601551 container died 50bff5b2e0ac0929af5754e15288de6003b952f9fe5ba91c6344ad3d1779c457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-21T23:19:54.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:54 smithi044 podman[93098]: 2024-04-21 23:19:54.208170617 +0000 UTC m=+0.507792053 container cleanup 50bff5b2e0ac0929af5754e15288de6003b952f9fe5ba91c6344ad3d1779c457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:19:54.435 INFO:journalctl@ceph.mon.a.smithi044.stdout:Apr 21 23:19:54 smithi044 bash[93098]: ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-a 2024-04-21T23:19:54.633 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mon.a.service' 2024-04-21T23:19:54.675 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:19:54.675 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-21T23:19:54.676 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-21T23:19:54.676 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@mon.b 2024-04-21T23:19:54.964 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:54 smithi055 systemd[1]: Stopping Ceph mon.b for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:19:55.421 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:55 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-b[31113]: 2024-04-21T23:19:55.003+0000 7fcbff282700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:19:55.421 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:55 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-b[31113]: 2024-04-21T23:19:55.003+0000 7fcbff282700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-21T23:19:55.421 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:55 smithi055 podman[66839]: 2024-04-21 23:19:55.127148407 +0000 UTC m=+0.245852454 container died 64497f39ca46e5aa21c3be74ca775bbc2c582391f8631d5330f423de239a6779 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:19:55.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:55 smithi055 podman[66839]: 2024-04-21 23:19:55.513934191 +0000 UTC m=+0.632638246 container cleanup 64497f39ca46e5aa21c3be74ca775bbc2c582391f8631d5330f423de239a6779 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-21T23:19:55.714 INFO:journalctl@ceph.mon.b.smithi055.stdout:Apr 21 23:19:55 smithi055 bash[66839]: ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-b 2024-04-21T23:19:55.856 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mon.b.service' 2024-04-21T23:19:55.903 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:19:55.904 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-21T23:19:55.904 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-21T23:19:55.904 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@mon.c 2024-04-21T23:19:56.214 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:55 smithi069 systemd[1]: Stopping Ceph mon.c for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:19:56.526 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:56 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-c[31790]: 2024-04-21T23:19:56.217+0000 7ff17ddff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:19:56.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:56 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-c[31790]: 2024-04-21T23:19:56.217+0000 7ff17ddff700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-21T23:19:56.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:56 smithi069 podman[69136]: 2024-04-21 23:19:56.269377181 +0000 UTC m=+0.169038027 container died 3b503e5b60964bfba24296f677d3b7e41d53d17e716dc11ac66b4f81a35ad7db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-c, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:19:56.927 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:56 smithi069 podman[69136]: 2024-04-21 23:19:56.62793438 +0000 UTC m=+0.527595220 container cleanup 3b503e5b60964bfba24296f677d3b7e41d53d17e716dc11ac66b4f81a35ad7db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-c, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:19:56.928 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 21 23:19:56 smithi069 bash[69136]: ceph-80909b94-0032-11ef-bc93-c7b262605968-mon-c 2024-04-21T23:19:57.093 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mon.c.service' 2024-04-21T23:19:57.139 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:19:57.140 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-21T23:19:57.140 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-21T23:19:57.140 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@mgr.x 2024-04-21T23:19:57.435 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:19:57 smithi044 systemd[1]: Stopping Ceph mgr.x for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:19:57.830 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:19:57 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x[28164]: 2024-04-21T23:19:57.472+0000 7feff59f6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:19:57.831 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:19:57 smithi044 podman[93220]: 2024-04-21 23:19:57.502915501 +0000 UTC m=+0.154238099 container died 4543c9b91d4a44095f76e2eef0a4cc748039ca7ff6988fed59f02a205f68c34b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-21T23:19:58.148 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:19:57 smithi044 podman[93220]: 2024-04-21 23:19:57.931120542 +0000 UTC m=+0.582443185 container cleanup 4543c9b91d4a44095f76e2eef0a4cc748039ca7ff6988fed59f02a205f68c34b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-820a2f8) 2024-04-21T23:19:58.148 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:19:57 smithi044 bash[93220]: ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x 2024-04-21T23:19:58.375 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-21T23:19:58.399 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:19:58 smithi044 podman[93234]: 2024-04-21 23:19:58.149628397 +0000 UTC m=+0.643656328 container remove 4543c9b91d4a44095f76e2eef0a4cc748039ca7ff6988fed59f02a205f68c34b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:820a2f8ced66152ed3d765c31444b8076ea28c6b, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-x, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:19:58.399 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:19:58 smithi044 systemd[1]: ceph-80909b94-0032-11ef-bc93-c7b262605968@mgr.x.service: Deactivated successfully. 2024-04-21T23:19:58.399 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:19:58 smithi044 systemd[1]: Stopped Ceph mgr.x for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:19:58.400 INFO:journalctl@ceph.mgr.x.smithi044.stdout:Apr 21 23:19:58 smithi044 systemd[1]: ceph-80909b94-0032-11ef-bc93-c7b262605968@mgr.x.service: Consumed 1min 23.613s CPU time. 2024-04-21T23:19:58.410 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:19:58.410 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-21T23:19:58.410 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-21T23:19:58.411 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@mgr.y 2024-04-21T23:19:58.714 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:19:58 smithi055 systemd[1]: Stopping Ceph mgr.y for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:19:59.036 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:19:58 smithi055 podman[66962]: 2024-04-21 23:19:58.740189001 +0000 UTC m=+0.132248411 container died 43fd150ed11af7a594d896e713d9606aa6e1d862306326e27ae0f235ed146823 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-21T23:19:59.395 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:19:59 smithi055 podman[66962]: 2024-04-21 23:19:59.120183555 +0000 UTC m=+0.512242964 container cleanup 43fd150ed11af7a594d896e713d9606aa6e1d862306326e27ae0f235ed146823 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-21T23:19:59.395 INFO:journalctl@ceph.mgr.y.smithi055.stdout:Apr 21 23:19:59 smithi055 bash[66962]: ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-y 2024-04-21T23:19:59.553 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-21T23:19:59.587 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:19:59.587 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-21T23:19:59.587 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-21T23:19:59.587 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@mgr.z 2024-04-21T23:19:59.918 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:19:59 smithi069 systemd[1]: Stopping Ceph mgr.z for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:20:00.190 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:19:59 smithi069 podman[69257]: 2024-04-21 23:19:59.919589901 +0000 UTC m=+0.138262470 container died 2fc9ae974ee6ba6994ac8963a4ea5d7c658d2a3030bc3a0c301ed88069060511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-21T23:20:00.464 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:20:00 smithi069 podman[69257]: 2024-04-21 23:20:00.274610165 +0000 UTC m=+0.493282732 container cleanup 2fc9ae974ee6ba6994ac8963a4ea5d7c658d2a3030bc3a0c301ed88069060511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-21T23:20:00.464 INFO:journalctl@ceph.mgr.z.smithi069.stdout:Apr 21 23:20:00 smithi069 bash[69257]: ceph-80909b94-0032-11ef-bc93-c7b262605968-mgr-z 2024-04-21T23:20:00.845 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-21T23:20:00.879 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:20:00.880 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-21T23:20:00.880 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-21T23:20:00.880 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.0 2024-04-21T23:20:01.185 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:20:00 smithi044 systemd[1]: Stopping Ceph osd.0 for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:20:01.685 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:20:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-0[40172]: 2024-04-21T23:20:01.193+0000 7fe7f205c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:20:01.685 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:20:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-0[40172]: 2024-04-21T23:20:01.193+0000 7fe7f205c700 -1 osd.0 114 *** Got signal Terminated *** 2024-04-21T23:20:01.685 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:20:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-0[40172]: 2024-04-21T23:20:01.193+0000 7fe7f205c700 -1 osd.0 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:20:07.036 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:20:06 smithi044 podman[93345]: 2024-04-21 23:20:06.734545025 +0000 UTC m=+5.656112063 container died 77e17c2104babccb26c2ef3b0acec71bcaf31b69e27def4d45a27e958c33a9af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-21T23:20:07.362 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:20:07 smithi044 podman[93345]: 2024-04-21 23:20:07.145325514 +0000 UTC m=+6.066892565 container cleanup 77e17c2104babccb26c2ef3b0acec71bcaf31b69e27def4d45a27e958c33a9af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-04-21T23:20:07.362 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:20:07 smithi044 bash[93345]: ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-0 2024-04-21T23:20:07.685 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:20:07 smithi044 podman[93362]: 2024-04-21 23:20:07.363620192 +0000 UTC m=+0.627441485 container remove 77e17c2104babccb26c2ef3b0acec71bcaf31b69e27def4d45a27e958c33a9af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416) 2024-04-21T23:20:07.937 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:20:07 smithi044 podman[93450]: 2024-04-21 23:20:07.688062528 +0000 UTC m=+0.018865292 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:20:07.937 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:20:07 smithi044 podman[93450]: 2024-04-21 23:20:07.814738191 +0000 UTC m=+0.145540938 container create dce827a00a87238a5f03d387b7cd9bbd5a10ea1d35a1370dc6e15b4febcb4d10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:20:08.356 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:20:08 smithi044 podman[93450]: 2024-04-21 23:20:08.050718865 +0000 UTC m=+0.381521621 container init dce827a00a87238a5f03d387b7cd9bbd5a10ea1d35a1370dc6e15b4febcb4d10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-21T23:20:08.356 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:20:08 smithi044 podman[93450]: 2024-04-21 23:20:08.053891671 +0000 UTC m=+0.384694417 container start dce827a00a87238a5f03d387b7cd9bbd5a10ea1d35a1370dc6e15b4febcb4d10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-0-deactivate, ceph=True, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-21T23:20:08.357 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:20:08 smithi044 podman[93450]: 2024-04-21 23:20:08.142433779 +0000 UTC m=+0.473236526 container attach dce827a00a87238a5f03d387b7cd9bbd5a10ea1d35a1370dc6e15b4febcb4d10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux ) 2024-04-21T23:20:08.685 INFO:journalctl@ceph.osd.0.smithi044.stdout:Apr 21 23:20:08 smithi044 podman[93450]: 2024-04-21 23:20:08.358563776 +0000 UTC m=+0.689366529 container died dce827a00a87238a5f03d387b7cd9bbd5a10ea1d35a1370dc6e15b4febcb4d10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-21T23:20:09.005 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.0.service' 2024-04-21T23:20:09.045 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:20:09.045 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-21T23:20:09.046 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-21T23:20:09.046 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.1 2024-04-21T23:20:09.357 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:20:09 smithi055 systemd[1]: Stopping Ceph osd.1 for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:20:09.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:20:09 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-1[35754]: 2024-04-21T23:20:09.358+0000 7f0cbe0f2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:20:09.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:20:09 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-1[35754]: 2024-04-21T23:20:09.358+0000 7f0cbe0f2700 -1 osd.1 114 *** Got signal Terminated *** 2024-04-21T23:20:09.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:20:09 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-1[35754]: 2024-04-21T23:20:09.358+0000 7f0cbe0f2700 -1 osd.1 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:20:15.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:20:14 smithi055 podman[67086]: 2024-04-21 23:20:14.91299815 +0000 UTC m=+5.669235689 container died 2fe887551d2862f0768c917c5972b5a3f32603dd2dcb02caa822ed80d99823ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8) 2024-04-21T23:20:15.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:20:15 smithi055 podman[67086]: 2024-04-21 23:20:15.443294797 +0000 UTC m=+6.199532344 container cleanup 2fe887551d2862f0768c917c5972b5a3f32603dd2dcb02caa822ed80d99823ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-21T23:20:15.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:20:15 smithi055 bash[67086]: ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-1 2024-04-21T23:20:15.964 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:20:15 smithi055 podman[67103]: 2024-04-21 23:20:15.644956234 +0000 UTC m=+0.718618101 container remove 2fe887551d2862f0768c917c5972b5a3f32603dd2dcb02caa822ed80d99823ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-21T23:20:16.364 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:20:16 smithi055 podman[67204]: 2024-04-21 23:20:15.986794097 +0000 UTC m=+0.018986821 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:20:16.364 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:20:16 smithi055 podman[67204]: 2024-04-21 23:20:16.104278518 +0000 UTC m=+0.136471224 container create 126c39038252bfc0d69640ade1090f94c114fac4e3ef560524d09abe5e2ed63f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-21T23:20:16.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:20:16 smithi055 podman[67204]: 2024-04-21 23:20:16.365993757 +0000 UTC m=+0.398186457 container init 126c39038252bfc0d69640ade1090f94c114fac4e3ef560524d09abe5e2ed63f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-21T23:20:16.715 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:20:16 smithi055 podman[67204]: 2024-04-21 23:20:16.369504762 +0000 UTC m=+0.401697471 container start 126c39038252bfc0d69640ade1090f94c114fac4e3ef560524d09abe5e2ed63f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-21T23:20:16.715 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:20:16 smithi055 podman[67204]: 2024-04-21 23:20:16.466532814 +0000 UTC m=+0.498725525 container attach 126c39038252bfc0d69640ade1090f94c114fac4e3ef560524d09abe5e2ed63f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8) 2024-04-21T23:20:17.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Apr 21 23:20:16 smithi055 podman[67204]: 2024-04-21 23:20:16.777908668 +0000 UTC m=+0.810101365 container died 126c39038252bfc0d69640ade1090f94c114fac4e3ef560524d09abe5e2ed63f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-21T23:20:17.335 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.1.service' 2024-04-21T23:20:17.376 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:20:17.377 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-21T23:20:17.377 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-21T23:20:17.377 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.2 2024-04-21T23:20:17.712 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:20:17 smithi069 systemd[1]: Stopping Ceph osd.2 for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:20:17.964 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:20:17 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-2[37031]: 2024-04-21T23:20:17.712+0000 7ff41130a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:20:17.964 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:20:17 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-2[37031]: 2024-04-21T23:20:17.712+0000 7ff41130a700 -1 osd.2 114 *** Got signal Terminated *** 2024-04-21T23:20:17.964 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:20:17 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-2[37031]: 2024-04-21T23:20:17.712+0000 7ff41130a700 -1 osd.2 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:20:23.562 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:20:23 smithi069 podman[69397]: 2024-04-21 23:20:23.29257891 +0000 UTC m=+5.719440027 container died d62a10db90423ff6759355c5c852ff0bad27054f3f5043039cdd120ffc80ebed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-2, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-21T23:20:23.955 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:20:23 smithi069 podman[69397]: 2024-04-21 23:20:23.655003082 +0000 UTC m=+6.081864190 container cleanup d62a10db90423ff6759355c5c852ff0bad27054f3f5043039cdd120ffc80ebed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-21T23:20:23.955 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:20:23 smithi069 bash[69397]: ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-2 2024-04-21T23:20:24.214 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:20:23 smithi069 podman[69417]: 2024-04-21 23:20:23.956654816 +0000 UTC m=+0.658864015 container remove d62a10db90423ff6759355c5c852ff0bad27054f3f5043039cdd120ffc80ebed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-2, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-21T23:20:24.592 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:20:24 smithi069 podman[69505]: 2024-04-21 23:20:24.221070668 +0000 UTC m=+0.017863945 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:20:24.593 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:20:24 smithi069 podman[69505]: 2024-04-21 23:20:24.341119131 +0000 UTC m=+0.137912391 container create 67066a765871f42aad986bd92d6917048a6c1515cf04486a768f3e0a39cc5762 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-04-21T23:20:24.888 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:20:24 smithi069 podman[69505]: 2024-04-21 23:20:24.59400502 +0000 UTC m=+0.390798284 container init 67066a765871f42aad986bd92d6917048a6c1515cf04486a768f3e0a39cc5762 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-21T23:20:24.888 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:20:24 smithi069 podman[69505]: 2024-04-21 23:20:24.597479921 +0000 UTC m=+0.394273181 container start 67066a765871f42aad986bd92d6917048a6c1515cf04486a768f3e0a39cc5762 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:20:24.888 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:20:24 smithi069 podman[69505]: 2024-04-21 23:20:24.685745744 +0000 UTC m=+0.482539013 container attach 67066a765871f42aad986bd92d6917048a6c1515cf04486a768f3e0a39cc5762 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-21T23:20:25.214 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 21 23:20:24 smithi069 podman[69505]: 2024-04-21 23:20:24.889826752 +0000 UTC m=+0.686620007 container died 67066a765871f42aad986bd92d6917048a6c1515cf04486a768f3e0a39cc5762 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:20:25.536 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.2.service' 2024-04-21T23:20:25.574 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:20:25.575 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-21T23:20:25.575 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-21T23:20:25.575 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.3 2024-04-21T23:20:25.932 INFO:journalctl@ceph.osd.3.smithi044.stdout:Apr 21 23:20:25 smithi044 systemd[1]: Stopping Ceph osd.3 for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:20:26.185 INFO:journalctl@ceph.osd.3.smithi044.stdout:Apr 21 23:20:25 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-3[46996]: 2024-04-21T23:20:25.932+0000 7f6005ecc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:20:26.185 INFO:journalctl@ceph.osd.3.smithi044.stdout:Apr 21 23:20:25 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-3[46996]: 2024-04-21T23:20:25.932+0000 7f6005ecc700 -1 osd.3 114 *** Got signal Terminated *** 2024-04-21T23:20:26.185 INFO:journalctl@ceph.osd.3.smithi044.stdout:Apr 21 23:20:25 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-3[46996]: 2024-04-21T23:20:25.932+0000 7f6005ecc700 -1 osd.3 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:20:27.964 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:27 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:27.692+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:02.160241+0000 front 2024-04-21T23:20:02.160356+0000 (oldest deadline 2024-04-21T23:20:26.860204+0000) 2024-04-21T23:20:28.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:28 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:28.353+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:28.958 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:28 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:28.702+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:02.160241+0000 front 2024-04-21T23:20:02.160356+0000 (oldest deadline 2024-04-21T23:20:26.860204+0000) 2024-04-21T23:20:29.214 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:28 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:28.959+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:29.684 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:29 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:29.392+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:30.001 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:29 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:29.738+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:02.160241+0000 front 2024-04-21T23:20:02.160356+0000 (oldest deadline 2024-04-21T23:20:26.860204+0000) 2024-04-21T23:20:30.464 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:30 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:30.003+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:30.685 INFO:journalctl@ceph.osd.3.smithi044.stdout:Apr 21 23:20:30 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-3[46996]: 2024-04-21T23:20:30.574+0000 7f60024c0700 -1 osd.3 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.628613+0000 front 2024-04-21T23:20:05.628485+0000 (oldest deadline 2024-04-21T23:20:29.728630+0000) 2024-04-21T23:20:30.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:30 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:30.396+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:30.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:30 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:30.411+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:31.214 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:30 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:30.749+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:02.160241+0000 front 2024-04-21T23:20:02.160356+0000 (oldest deadline 2024-04-21T23:20:26.860204+0000) 2024-04-21T23:20:31.214 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:30 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:30.979+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:31.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:31 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:31.429+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:31.858 INFO:journalctl@ceph.osd.3.smithi044.stdout:Apr 21 23:20:31 smithi044 podman[93559]: 2024-04-21 23:20:31.481609566 +0000 UTC m=+5.673034328 container died 1897a033407686cd39c3c120b10b059ecf956b7a15668aa8db185069e489e099 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-21T23:20:31.859 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:31 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:31.445+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:32.185 INFO:journalctl@ceph.osd.3.smithi044.stdout:Apr 21 23:20:31 smithi044 podman[93559]: 2024-04-21 23:20:31.963823595 +0000 UTC m=+6.155248345 container cleanup 1897a033407686cd39c3c120b10b059ecf956b7a15668aa8db185069e489e099 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:20:32.185 INFO:journalctl@ceph.osd.3.smithi044.stdout:Apr 21 23:20:31 smithi044 bash[93559]: ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-3 2024-04-21T23:20:32.214 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:31 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:31.780+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:02.160241+0000 front 2024-04-21T23:20:02.160356+0000 (oldest deadline 2024-04-21T23:20:26.860204+0000) 2024-04-21T23:20:32.214 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:32 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:32.011+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:32.544 INFO:journalctl@ceph.osd.3.smithi044.stdout:Apr 21 23:20:32 smithi044 podman[93589]: 2024-04-21 23:20:32.223457714 +0000 UTC m=+0.736890918 container remove 1897a033407686cd39c3c120b10b059ecf956b7a15668aa8db185069e489e099 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-3, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-21T23:20:32.545 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:32 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:32.437+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:32.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:32 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:32.437+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:32.862 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:32 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:32.580+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:32.863 INFO:journalctl@ceph.osd.3.smithi044.stdout:Apr 21 23:20:32 smithi044 podman[93681]: 2024-04-21 23:20:32.448822061 +0000 UTC m=+0.018636278 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:20:32.863 INFO:journalctl@ceph.osd.3.smithi044.stdout:Apr 21 23:20:32 smithi044 podman[93681]: 2024-04-21 23:20:32.5613356 +0000 UTC m=+0.131149857 container create cf99c7e6aa4537d5f172f2b105550736e2edb40f3aff616b276a23b5ff17f195 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-21T23:20:32.863 INFO:journalctl@ceph.osd.3.smithi044.stdout:Apr 21 23:20:32 smithi044 podman[93681]: 2024-04-21 23:20:32.772522584 +0000 UTC m=+0.342336808 container init cf99c7e6aa4537d5f172f2b105550736e2edb40f3aff616b276a23b5ff17f195 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-21T23:20:32.863 INFO:journalctl@ceph.osd.3.smithi044.stdout:Apr 21 23:20:32 smithi044 podman[93681]: 2024-04-21 23:20:32.776255347 +0000 UTC m=+0.346069568 container start cf99c7e6aa4537d5f172f2b105550736e2edb40f3aff616b276a23b5ff17f195 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-21T23:20:33.185 INFO:journalctl@ceph.osd.3.smithi044.stdout:Apr 21 23:20:32 smithi044 podman[93681]: 2024-04-21 23:20:32.863912996 +0000 UTC m=+0.433727213 container attach cf99c7e6aa4537d5f172f2b105550736e2edb40f3aff616b276a23b5ff17f195 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux ) 2024-04-21T23:20:33.185 INFO:journalctl@ceph.osd.3.smithi044.stdout:Apr 21 23:20:33 smithi044 podman[93681]: 2024-04-21 23:20:33.081653102 +0000 UTC m=+0.651467308 container died cf99c7e6aa4537d5f172f2b105550736e2edb40f3aff616b276a23b5ff17f195 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:20:33.214 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:32 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:32.780+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:02.160241+0000 front 2024-04-21T23:20:02.160356+0000 (oldest deadline 2024-04-21T23:20:26.860204+0000) 2024-04-21T23:20:33.214 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:33 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:33.010+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:33.658 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:33 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:33.453+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:33.659 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:33 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:33.557+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:33.680 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.3.service' 2024-04-21T23:20:33.724 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:20:33.724 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-21T23:20:33.724 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-21T23:20:33.724 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.4 2024-04-21T23:20:33.751 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:33 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:33.466+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:34.034 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:33 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:33.762+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:02.160241+0000 front 2024-04-21T23:20:02.160356+0000 (oldest deadline 2024-04-21T23:20:26.860204+0000) 2024-04-21T23:20:34.081 INFO:journalctl@ceph.osd.4.smithi055.stdout:Apr 21 23:20:33 smithi055 systemd[1]: Stopping Ceph osd.4 for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:20:34.457 INFO:journalctl@ceph.osd.4.smithi055.stdout:Apr 21 23:20:34 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-4[42691]: 2024-04-21T23:20:34.083+0000 7fd34ca34700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:20:34.458 INFO:journalctl@ceph.osd.4.smithi055.stdout:Apr 21 23:20:34 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-4[42691]: 2024-04-21T23:20:34.083+0000 7fd34ca34700 -1 osd.4 114 *** Got signal Terminated *** 2024-04-21T23:20:34.458 INFO:journalctl@ceph.osd.4.smithi055.stdout:Apr 21 23:20:34 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-4[42691]: 2024-04-21T23:20:34.083+0000 7fd34ca34700 -1 osd.4 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:20:34.463 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:34 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:34.035+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:34.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:34 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:34.407+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:34.685 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:34 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:34.595+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:34.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:34 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:34.456+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:35.078 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:34 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:34.773+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:02.160241+0000 front 2024-04-21T23:20:02.160356+0000 (oldest deadline 2024-04-21T23:20:26.860204+0000) 2024-04-21T23:20:35.464 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:35 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:35.079+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:35.637 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:35 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:35.361+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:35.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:35 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:35.638+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:35.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:35 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:35.478+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:35.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:35 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:35.582+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:36.040 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:35 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:35.761+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:02.160241+0000 front 2024-04-21T23:20:02.160356+0000 (oldest deadline 2024-04-21T23:20:26.860204+0000) 2024-04-21T23:20:36.040 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:35 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:35.761+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:13.862098+0000 front 2024-04-21T23:20:13.862298+0000 (oldest deadline 2024-04-21T23:20:34.962077+0000) 2024-04-21T23:20:36.464 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:36 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:36.041+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:36.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:36 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:36.400+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:36.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:36 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:36.400+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:10.815926+0000 front 2024-04-21T23:20:10.816021+0000 (oldest deadline 2024-04-21T23:20:36.115934+0000) 2024-04-21T23:20:36.685 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:36 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:36.625+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:36.686 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:36 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:36.625+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:36.786 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:36 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:36.489+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:36.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:36 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:36.612+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:36.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:36 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:36.498+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:37.214 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:36 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:36.808+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:02.160241+0000 front 2024-04-21T23:20:02.160356+0000 (oldest deadline 2024-04-21T23:20:26.860204+0000) 2024-04-21T23:20:37.214 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:36 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:36.808+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:13.862098+0000 front 2024-04-21T23:20:13.862298+0000 (oldest deadline 2024-04-21T23:20:34.962077+0000) 2024-04-21T23:20:37.215 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:36 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:36.997+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:37.215 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:36 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:36.997+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:37.463 INFO:journalctl@ceph.osd.4.smithi055.stdout:Apr 21 23:20:37 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-4[42691]: 2024-04-21T23:20:37.002+0000 7fd349028700 -1 osd.4 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:13.832515+0000 front 2024-04-21T23:20:13.832476+0000 (oldest deadline 2024-04-21T23:20:36.132592+0000) 2024-04-21T23:20:37.659 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:37 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:37.377+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:37.659 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:37 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:37.377+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:10.815926+0000 front 2024-04-21T23:20:10.816021+0000 (oldest deadline 2024-04-21T23:20:36.115934+0000) 2024-04-21T23:20:37.714 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:37 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:37.593+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:37.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:37 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:37.463+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:37.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:37 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:37.463+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:37.839 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:37 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:37.535+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:37.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:37 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:37.659+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:38.028 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:37 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:37.659+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:38.214 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:37 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:37.839+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:02.160241+0000 front 2024-04-21T23:20:02.160356+0000 (oldest deadline 2024-04-21T23:20:26.860204+0000) 2024-04-21T23:20:38.214 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:37 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:37.839+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:13.862098+0000 front 2024-04-21T23:20:13.862298+0000 (oldest deadline 2024-04-21T23:20:34.962077+0000) 2024-04-21T23:20:38.215 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:38 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:38.026+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:38.215 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:38 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:38.026+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:38.418 INFO:journalctl@ceph.osd.4.smithi055.stdout:Apr 21 23:20:38 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-4[42691]: 2024-04-21T23:20:38.037+0000 7fd349028700 -1 osd.4 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:13.832515+0000 front 2024-04-21T23:20:13.832476+0000 (oldest deadline 2024-04-21T23:20:36.132592+0000) 2024-04-21T23:20:38.632 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:38 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:38.373+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:38.633 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:38 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:38.373+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:10.815926+0000 front 2024-04-21T23:20:10.816021+0000 (oldest deadline 2024-04-21T23:20:36.115934+0000) 2024-04-21T23:20:38.714 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:38 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:38.607+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:38.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:38 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:38.418+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:38.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:38 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:38.418+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:38.846 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:38 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:38.540+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:38.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:38 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:38.633+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:38.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:38 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:38.633+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:39.214 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:39 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:39.025+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:39.214 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:39 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:39.025+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:39.214 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:38 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:38.847+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:02.160241+0000 front 2024-04-21T23:20:02.160356+0000 (oldest deadline 2024-04-21T23:20:26.860204+0000) 2024-04-21T23:20:39.215 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:38 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:38.847+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:13.862098+0000 front 2024-04-21T23:20:13.862298+0000 (oldest deadline 2024-04-21T23:20:34.962077+0000) 2024-04-21T23:20:39.411 INFO:journalctl@ceph.osd.4.smithi055.stdout:Apr 21 23:20:39 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-4[42691]: 2024-04-21T23:20:39.040+0000 7fd349028700 -1 osd.4 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:13.832515+0000 front 2024-04-21T23:20:13.832476+0000 (oldest deadline 2024-04-21T23:20:36.132592+0000) 2024-04-21T23:20:39.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:39 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:39.340+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:39.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:39 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:39.340+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:10.815926+0000 front 2024-04-21T23:20:10.816021+0000 (oldest deadline 2024-04-21T23:20:36.115934+0000) 2024-04-21T23:20:39.685 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:39 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:39.586+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:39.685 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:39 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:39.586+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:39.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:39 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:39.500+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:39.714 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:39 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:39.573+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:39.714 INFO:journalctl@ceph.osd.4.smithi055.stdout:Apr 21 23:20:39 smithi055 podman[67308]: 2024-04-21 23:20:39.529249534 +0000 UTC m=+5.604703246 container died 372c604fcc943949a70684597f49f14241b7d41644367fa74b9c6f598f10209d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-4, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-04-21T23:20:39.715 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:39 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:39.412+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:39.715 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:39 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:39.412+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:40.185 INFO:journalctl@ceph.osd.4.smithi055.stdout:Apr 21 23:20:39 smithi055 podman[67308]: 2024-04-21 23:20:39.882942766 +0000 UTC m=+5.958396485 container cleanup 372c604fcc943949a70684597f49f14241b7d41644367fa74b9c6f598f10209d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-4, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8) 2024-04-21T23:20:40.185 INFO:journalctl@ceph.osd.4.smithi055.stdout:Apr 21 23:20:39 smithi055 bash[67308]: ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-4 2024-04-21T23:20:40.214 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:40 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:40.057+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:40.214 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:40 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:40.057+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:40.214 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:39 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:39.876+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:02.160241+0000 front 2024-04-21T23:20:02.160356+0000 (oldest deadline 2024-04-21T23:20:26.860204+0000) 2024-04-21T23:20:40.214 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:39 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:39.876+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:13.862098+0000 front 2024-04-21T23:20:13.862298+0000 (oldest deadline 2024-04-21T23:20:34.962077+0000) 2024-04-21T23:20:40.464 INFO:journalctl@ceph.osd.4.smithi055.stdout:Apr 21 23:20:40 smithi055 podman[67325]: 2024-04-21 23:20:40.184497138 +0000 UTC m=+0.649272689 container remove 372c604fcc943949a70684597f49f14241b7d41644367fa74b9c6f598f10209d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-4, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:20:40.464 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:40 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:40.388+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:40.465 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:40 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:40.389+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:40.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:40 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:40.347+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:40.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:40 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:40.347+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:10.815926+0000 front 2024-04-21T23:20:10.816021+0000 (oldest deadline 2024-04-21T23:20:36.115934+0000) 2024-04-21T23:20:40.685 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:40 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:40.586+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:40.685 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:40 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:40.586+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:40.787 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:40 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:40.605+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:40.787 INFO:journalctl@ceph.osd.4.smithi055.stdout:Apr 21 23:20:40 smithi055 podman[67414]: 2024-04-21 23:20:40.418866311 +0000 UTC m=+0.018264981 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:20:40.787 INFO:journalctl@ceph.osd.4.smithi055.stdout:Apr 21 23:20:40 smithi055 podman[67414]: 2024-04-21 23:20:40.535990543 +0000 UTC m=+0.135389189 container create c86fb347121811aed4eea90aa62afafd8b77a3f94f3eb1ea891e4915906faeb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:20:40.788 INFO:journalctl@ceph.osd.4.smithi055.stdout:Apr 21 23:20:40 smithi055 podman[67414]: 2024-04-21 23:20:40.788761203 +0000 UTC m=+0.388159849 container init c86fb347121811aed4eea90aa62afafd8b77a3f94f3eb1ea891e4915906faeb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:20:40.827 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:40 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:40.537+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:41.103 INFO:journalctl@ceph.osd.4.smithi055.stdout:Apr 21 23:20:40 smithi055 podman[67414]: 2024-04-21 23:20:40.792075215 +0000 UTC m=+0.391473861 container start c86fb347121811aed4eea90aa62afafd8b77a3f94f3eb1ea891e4915906faeb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:20:41.103 INFO:journalctl@ceph.osd.4.smithi055.stdout:Apr 21 23:20:40 smithi055 podman[67414]: 2024-04-21 23:20:40.88057713 +0000 UTC m=+0.479975775 container attach c86fb347121811aed4eea90aa62afafd8b77a3f94f3eb1ea891e4915906faeb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-21T23:20:41.214 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:40 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:40.828+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:02.160241+0000 front 2024-04-21T23:20:02.160356+0000 (oldest deadline 2024-04-21T23:20:26.860204+0000) 2024-04-21T23:20:41.214 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:40 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:40.828+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:13.862098+0000 front 2024-04-21T23:20:13.862298+0000 (oldest deadline 2024-04-21T23:20:34.962077+0000) 2024-04-21T23:20:41.214 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:41 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:41.032+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:41.215 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:41 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:41.032+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:41.391 INFO:journalctl@ceph.osd.4.smithi055.stdout:Apr 21 23:20:41 smithi055 podman[67414]: 2024-04-21 23:20:41.104549064 +0000 UTC m=+0.703947701 container died c86fb347121811aed4eea90aa62afafd8b77a3f94f3eb1ea891e4915906faeb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-21T23:20:41.610 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:41 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:41.323+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:41.610 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:41 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:41.323+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:10.815926+0000 front 2024-04-21T23:20:10.816021+0000 (oldest deadline 2024-04-21T23:20:36.115934+0000) 2024-04-21T23:20:41.714 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:41 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:41.633+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:41.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:41 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:41.392+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:41.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:41 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:41.392+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:41.772 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.4.service' 2024-04-21T23:20:41.858 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:20:41.859 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-21T23:20:41.859 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-21T23:20:41.859 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.5 2024-04-21T23:20:41.887 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:41 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:41.807+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:02.160241+0000 front 2024-04-21T23:20:02.160356+0000 (oldest deadline 2024-04-21T23:20:26.860204+0000) 2024-04-21T23:20:41.887 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:41 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:41.807+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:13.862098+0000 front 2024-04-21T23:20:13.862298+0000 (oldest deadline 2024-04-21T23:20:34.962077+0000) 2024-04-21T23:20:41.887 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:41 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:41.587+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:41.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:41 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:41.611+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:41.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:41 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:41.611+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:42.150 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:41 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:41.992+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:42.150 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:41 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:41.992+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:42.150 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:41 smithi069 systemd[1]: Stopping Ceph osd.5 for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:20:42.150 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:42 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:42.150+0000 7f00fac15700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:20:42.151 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:42 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:42.150+0000 7f00fac15700 -1 osd.5 114 *** Got signal Terminated *** 2024-04-21T23:20:42.151 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:42 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:42.150+0000 7f00fac15700 -1 osd.5 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:20:42.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:42 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:42.331+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:42.686 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:42 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:42.331+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:10.815926+0000 front 2024-04-21T23:20:10.816021+0000 (oldest deadline 2024-04-21T23:20:36.115934+0000) 2024-04-21T23:20:42.686 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:42 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:42.580+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:42.686 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:42 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:42.580+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:42.714 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:42 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:42.656+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:42.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:42 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:42.411+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:42.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:42 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:42.411+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:42.964 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:42 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:42.549+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:42.964 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:42 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:42.780+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:02.160241+0000 front 2024-04-21T23:20:02.160356+0000 (oldest deadline 2024-04-21T23:20:26.860204+0000) 2024-04-21T23:20:42.964 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:42 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:42.780+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:13.862098+0000 front 2024-04-21T23:20:13.862298+0000 (oldest deadline 2024-04-21T23:20:34.962077+0000) 2024-04-21T23:20:43.435 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:43 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:43.307+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:43.435 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:43 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:43.307+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:10.815926+0000 front 2024-04-21T23:20:10.816021+0000 (oldest deadline 2024-04-21T23:20:36.115934+0000) 2024-04-21T23:20:43.464 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:43 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:43.036+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:43.464 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:43 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:43.036+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:43.714 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:43 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:43.622+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:43.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:43 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:43.453+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:43.715 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:43 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:43.453+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:43.794 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:43 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:43.503+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:43.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:43 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:43.589+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:43.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:43 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:43.589+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:44.214 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:43 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:43.795+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:02.160241+0000 front 2024-04-21T23:20:02.160356+0000 (oldest deadline 2024-04-21T23:20:26.860204+0000) 2024-04-21T23:20:44.214 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:43 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:43.795+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:13.862098+0000 front 2024-04-21T23:20:13.862298+0000 (oldest deadline 2024-04-21T23:20:34.962077+0000) 2024-04-21T23:20:44.214 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:44 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:44.033+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:44.215 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:44 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:44.033+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:44.612 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:44 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:44.288+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:44.612 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:44 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:44.288+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:10.815926+0000 front 2024-04-21T23:20:10.816021+0000 (oldest deadline 2024-04-21T23:20:36.115934+0000) 2024-04-21T23:20:44.785 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:44 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:44.487+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:44.785 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:44 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:44.487+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:20:44.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:44 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:44.613+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:44.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:44 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:44.613+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:44.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:44 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:44.628+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:44.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:44 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:44.628+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.533381+0000 front 2024-04-21T23:20:22.533232+0000 (oldest deadline 2024-04-21T23:20:43.633212+0000) 2024-04-21T23:20:44.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:44 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:44.495+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:44.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:44 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:44.495+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:45.081 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:44 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:44.785+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:02.160241+0000 front 2024-04-21T23:20:02.160356+0000 (oldest deadline 2024-04-21T23:20:26.860204+0000) 2024-04-21T23:20:45.082 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:44 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:44.785+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:13.862098+0000 front 2024-04-21T23:20:13.862298+0000 (oldest deadline 2024-04-21T23:20:34.962077+0000) 2024-04-21T23:20:45.464 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:45 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:45.081+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:45.464 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:45 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:45.081+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:45.645 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:45 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:45.335+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:45.646 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:45 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:45.335+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:10.815926+0000 front 2024-04-21T23:20:10.816021+0000 (oldest deadline 2024-04-21T23:20:36.115934+0000) 2024-04-21T23:20:45.646 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:45 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:45.335+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.518301+0000 front 2024-04-21T23:20:22.518321+0000 (oldest deadline 2024-04-21T23:20:44.818306+0000) 2024-04-21T23:20:45.782 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:45 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:45.501+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:45.782 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:45 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:45.501+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:20:45.865 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:45 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:45.543+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:45.866 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:45 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:45.543+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:45.866 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:45 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:45.626+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:45.866 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:45 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:45.626+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.533381+0000 front 2024-04-21T23:20:22.533232+0000 (oldest deadline 2024-04-21T23:20:43.633212+0000) 2024-04-21T23:20:45.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:45 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:45.647+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:45.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:45 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:45.647+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:45.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:45 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:45.647+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:20:46.125 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:45 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:45.782+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:02.160241+0000 front 2024-04-21T23:20:02.160356+0000 (oldest deadline 2024-04-21T23:20:26.860204+0000) 2024-04-21T23:20:46.126 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:45 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:45.782+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:13.862098+0000 front 2024-04-21T23:20:13.862298+0000 (oldest deadline 2024-04-21T23:20:34.962077+0000) 2024-04-21T23:20:46.126 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:45 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:45.782+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.363642+0000 front 2024-04-21T23:20:21.363546+0000 (oldest deadline 2024-04-21T23:20:45.463719+0000) 2024-04-21T23:20:46.464 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:46 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:46.125+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:46.464 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:46 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:46.125+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:46.659 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:46 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:46.330+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:46.659 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:46 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:46.330+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:10.815926+0000 front 2024-04-21T23:20:10.816021+0000 (oldest deadline 2024-04-21T23:20:36.115934+0000) 2024-04-21T23:20:46.659 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:46 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:46.330+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.518301+0000 front 2024-04-21T23:20:22.518321+0000 (oldest deadline 2024-04-21T23:20:44.818306+0000) 2024-04-21T23:20:46.750 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:46 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:46.493+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:46.750 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:46 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:46.493+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:20:46.882 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:46 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:46.555+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:46.883 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:46 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:46.555+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:46.883 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:46 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:46.555+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:20:46.883 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:46 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:46.594+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:46.883 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:46 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:46.594+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.533381+0000 front 2024-04-21T23:20:22.533232+0000 (oldest deadline 2024-04-21T23:20:43.633212+0000) 2024-04-21T23:20:46.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:46 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:46.660+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:46.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:46 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:46.660+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:46.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:46 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:46.660+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:20:47.169 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:46 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:46.750+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:02.160241+0000 front 2024-04-21T23:20:02.160356+0000 (oldest deadline 2024-04-21T23:20:26.860204+0000) 2024-04-21T23:20:47.169 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:46 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:46.750+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:13.862098+0000 front 2024-04-21T23:20:13.862298+0000 (oldest deadline 2024-04-21T23:20:34.962077+0000) 2024-04-21T23:20:47.170 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:46 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5[43574]: 2024-04-21T23:20:46.750+0000 7f00f7209700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.363642+0000 front 2024-04-21T23:20:21.363546+0000 (oldest deadline 2024-04-21T23:20:45.463719+0000) 2024-04-21T23:20:47.464 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:47 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:47.169+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:47.464 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:47 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:47.169+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:47.677 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:47 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:47.377+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:47.677 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:47 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:47.377+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:10.815926+0000 front 2024-04-21T23:20:10.816021+0000 (oldest deadline 2024-04-21T23:20:36.115934+0000) 2024-04-21T23:20:47.678 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:47 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:47.377+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.518301+0000 front 2024-04-21T23:20:22.518321+0000 (oldest deadline 2024-04-21T23:20:44.818306+0000) 2024-04-21T23:20:47.875 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:47 smithi069 podman[69637]: 2024-04-21 23:20:47.610546029 +0000 UTC m=+5.557355226 container died 02df57ff13591d6d46d3e423c1a4c3aac055e0b8fe6573f1b91a690ed1444813 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-21T23:20:47.876 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:47 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:47.528+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:47.876 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:47 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:47.528+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:20:47.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:47 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:47.679+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:47.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:47 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:47.679+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:47.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:47 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:47.679+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:20:47.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:47 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:47.635+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:47.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:47 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:47.635+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.533381+0000 front 2024-04-21T23:20:22.533232+0000 (oldest deadline 2024-04-21T23:20:43.633212+0000) 2024-04-21T23:20:47.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:47 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:47.538+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:47.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:47 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:47.538+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:47.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:47 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:47.538+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:20:48.193 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:47 smithi069 podman[69637]: 2024-04-21 23:20:47.976083838 +0000 UTC m=+5.922893033 container cleanup 02df57ff13591d6d46d3e423c1a4c3aac055e0b8fe6573f1b91a690ed1444813 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-21T23:20:48.193 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:47 smithi069 bash[69637]: ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5 2024-04-21T23:20:48.464 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:48 smithi069 podman[69654]: 2024-04-21 23:20:48.235825613 +0000 UTC m=+0.626517086 container remove 02df57ff13591d6d46d3e423c1a4c3aac055e0b8fe6573f1b91a690ed1444813 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-820a2f8, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-21T23:20:48.464 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:48 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:48.194+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:48.464 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:48 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:48.194+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:48.676 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:48 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:48.367+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:48.676 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:48 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:48.367+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:10.815926+0000 front 2024-04-21T23:20:10.816021+0000 (oldest deadline 2024-04-21T23:20:36.115934+0000) 2024-04-21T23:20:48.676 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:48 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:48.367+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.518301+0000 front 2024-04-21T23:20:22.518321+0000 (oldest deadline 2024-04-21T23:20:44.818306+0000) 2024-04-21T23:20:48.829 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:48 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:48.515+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:48.830 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:48 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:48.515+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:20:48.830 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:48 smithi069 podman[69746]: 2024-04-21 23:20:48.431990073 +0000 UTC m=+0.017758980 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:20:48.830 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:48 smithi069 podman[69746]: 2024-04-21 23:20:48.55339735 +0000 UTC m=+0.139166302 container create 35ec05fad6ad5a65d5215e5fe1e24fbcb166ab0f02e88717e72f58751ce3122b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-21T23:20:48.830 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:48 smithi069 podman[69746]: 2024-04-21 23:20:48.831063757 +0000 UTC m=+0.416832673 container init 35ec05fad6ad5a65d5215e5fe1e24fbcb166ab0f02e88717e72f58751ce3122b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8) 2024-04-21T23:20:48.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:48 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:48.677+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:48.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:48 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:48.677+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:48.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:48 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:48.677+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:20:48.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:48 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:48.660+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:48.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:48 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:48.660+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.533381+0000 front 2024-04-21T23:20:22.533232+0000 (oldest deadline 2024-04-21T23:20:43.633212+0000) 2024-04-21T23:20:48.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:48 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:48.572+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:48.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:48 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:48.572+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:48.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:48 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:48.572+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:20:49.122 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:48 smithi069 podman[69746]: 2024-04-21 23:20:48.834488798 +0000 UTC m=+0.420257711 container start 35ec05fad6ad5a65d5215e5fe1e24fbcb166ab0f02e88717e72f58751ce3122b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-820a2f8, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-21T23:20:49.122 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:48 smithi069 podman[69746]: 2024-04-21 23:20:48.93098562 +0000 UTC m=+0.516754535 container attach 35ec05fad6ad5a65d5215e5fe1e24fbcb166ab0f02e88717e72f58751ce3122b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-21T23:20:49.424 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:49 smithi069 podman[69746]: 2024-04-21 23:20:49.123999825 +0000 UTC m=+0.709768738 container died 35ec05fad6ad5a65d5215e5fe1e24fbcb166ab0f02e88717e72f58751ce3122b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-21T23:20:49.425 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:49 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:49.208+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:49.425 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:49 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:49.208+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:49.425 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:49 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:49.208+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:20:49.667 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.5.service' 2024-04-21T23:20:49.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:49 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:49.386+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:49.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:49 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:49.386+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:10.815926+0000 front 2024-04-21T23:20:10.816021+0000 (oldest deadline 2024-04-21T23:20:36.115934+0000) 2024-04-21T23:20:49.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:49 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:49.386+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.518301+0000 front 2024-04-21T23:20:22.518321+0000 (oldest deadline 2024-04-21T23:20:44.818306+0000) 2024-04-21T23:20:49.714 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:49 smithi069 podman[69773]: 2024-04-21 23:20:49.651239996 +0000 UTC m=+0.522092783 container remove 35ec05fad6ad5a65d5215e5fe1e24fbcb166ab0f02e88717e72f58751ce3122b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:20:49.714 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:49 smithi069 systemd[1]: ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.5.service: Deactivated successfully. 2024-04-21T23:20:49.714 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:49 smithi069 systemd[1]: Stopped Ceph osd.5 for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:20:49.714 INFO:journalctl@ceph.osd.5.smithi069.stdout:Apr 21 23:20:49 smithi069 systemd[1]: ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.5.service: Consumed 3min 17.929s CPU time. 2024-04-21T23:20:49.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:49 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:49.489+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:49.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:49 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:49.489+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:20:49.744 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:20:49.744 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-21T23:20:49.745 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-21T23:20:49.745 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.6 2024-04-21T23:20:49.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:49 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:49.663+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:49.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:49 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:49.663+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.533381+0000 front 2024-04-21T23:20:22.533232+0000 (oldest deadline 2024-04-21T23:20:43.633212+0000) 2024-04-21T23:20:49.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:49 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:49.532+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:49.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:49 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:49.532+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:49.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:49 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:49.532+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:20:50.070 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:49 smithi044 systemd[1]: Stopping Ceph osd.6 for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:20:50.070 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:49 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:49.718+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:50.070 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:49 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:49.718+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:50.071 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:49 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:49.718+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:20:50.408 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:50.071+0000 7fee2cef7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:20:50.409 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:50.071+0000 7fee2cef7700 -1 osd.6 114 *** Got signal Terminated *** 2024-04-21T23:20:50.409 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:50.071+0000 7fee2cef7700 -1 osd.6 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:20:50.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:50.410+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:50.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:50.410+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:10.815926+0000 front 2024-04-21T23:20:10.816021+0000 (oldest deadline 2024-04-21T23:20:36.115934+0000) 2024-04-21T23:20:50.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:50.410+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.518301+0000 front 2024-04-21T23:20:22.518321+0000 (oldest deadline 2024-04-21T23:20:44.818306+0000) 2024-04-21T23:20:50.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:50 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:50.466+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:50.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:50 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:50.466+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:20:50.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:50 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:50.230+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:50.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:50 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:50.230+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:50.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:50 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:50.231+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:20:50.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:50 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:50.641+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:50.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:50 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:50.641+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.533381+0000 front 2024-04-21T23:20:22.533232+0000 (oldest deadline 2024-04-21T23:20:43.633212+0000) 2024-04-21T23:20:50.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:50 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:50.541+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:50.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:50 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:50.541+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:50.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:50 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:50.541+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:20:51.185 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:50.728+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:51.185 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:50.728+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:51.185 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:50 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:50.728+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:20:51.464 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:51 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:51.432+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:51.464 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:51 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:51.432+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:20:51.464 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:51 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:51.205+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:51.464 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:51 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:51.205+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:51.465 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:51 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:51.205+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:20:51.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:51 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:51.373+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:51.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:51 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:51.373+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:10.815926+0000 front 2024-04-21T23:20:10.816021+0000 (oldest deadline 2024-04-21T23:20:36.115934+0000) 2024-04-21T23:20:51.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:51 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:51.373+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.518301+0000 front 2024-04-21T23:20:22.518321+0000 (oldest deadline 2024-04-21T23:20:44.818306+0000) 2024-04-21T23:20:51.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:51 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:51.633+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:51.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:51 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:51.633+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.533381+0000 front 2024-04-21T23:20:22.533232+0000 (oldest deadline 2024-04-21T23:20:43.633212+0000) 2024-04-21T23:20:51.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:51 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:51.523+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:51.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:51 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:51.523+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:51.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:51 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:51.523+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:20:52.185 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:51 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:51.723+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:52.185 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:51 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:51.723+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:52.185 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:51 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:51.723+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:20:52.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:52 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:52.366+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:52.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:52 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:52.366+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:10.815926+0000 front 2024-04-21T23:20:10.816021+0000 (oldest deadline 2024-04-21T23:20:36.115934+0000) 2024-04-21T23:20:52.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:52 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:52.366+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.518301+0000 front 2024-04-21T23:20:22.518321+0000 (oldest deadline 2024-04-21T23:20:44.818306+0000) 2024-04-21T23:20:52.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:52 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:52.366+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:27.019754+0000 front 2024-04-21T23:20:27.019806+0000 (oldest deadline 2024-04-21T23:20:52.319900+0000) 2024-04-21T23:20:52.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:52 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:52.461+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:52.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:52 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:52.461+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:20:52.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:52 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:52.241+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:52.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:52 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:52.241+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:52.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:52 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:52.242+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:20:52.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:52 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:52.663+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:52.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:52 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:52.663+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.533381+0000 front 2024-04-21T23:20:22.533232+0000 (oldest deadline 2024-04-21T23:20:43.633212+0000) 2024-04-21T23:20:52.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:52 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:52.522+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:52.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:52 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:52.522+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:52.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:52 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:52.522+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:20:53.185 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:52 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:52.753+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:53.185 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:52 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:52.753+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:53.185 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:52 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:52.753+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:20:53.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:53 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:53.341+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:53.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:53 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:53.341+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:10.815926+0000 front 2024-04-21T23:20:10.816021+0000 (oldest deadline 2024-04-21T23:20:36.115934+0000) 2024-04-21T23:20:53.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:53 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:53.341+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.518301+0000 front 2024-04-21T23:20:22.518321+0000 (oldest deadline 2024-04-21T23:20:44.818306+0000) 2024-04-21T23:20:53.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:53 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:53.341+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:27.019754+0000 front 2024-04-21T23:20:27.019806+0000 (oldest deadline 2024-04-21T23:20:52.319900+0000) 2024-04-21T23:20:53.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:53 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:53.483+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:53.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:53 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:53.483+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:20:53.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:53 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:53.255+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:53.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:53 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:53.255+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:53.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:53 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:53.255+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:20:53.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:53 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:53.513+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:53.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:53 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:53.513+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:53.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:53 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:53.513+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:20:53.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:53 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:53.614+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:53.965 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:53 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:53.614+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.533381+0000 front 2024-04-21T23:20:22.533232+0000 (oldest deadline 2024-04-21T23:20:43.633212+0000) 2024-04-21T23:20:53.965 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:53 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:53.614+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.935487+0000 front 2024-04-21T23:20:29.935525+0000 (oldest deadline 2024-04-21T23:20:52.835639+0000) 2024-04-21T23:20:54.185 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:53 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:53.726+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:54.185 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:53 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:53.726+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:54.185 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:53 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:53.726+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:20:54.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:54 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:54.326+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.014482+0000 front 2024-04-21T23:20:05.014534+0000 (oldest deadline 2024-04-21T23:20:27.914723+0000) 2024-04-21T23:20:54.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:54 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:54.326+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:10.815926+0000 front 2024-04-21T23:20:10.816021+0000 (oldest deadline 2024-04-21T23:20:36.115934+0000) 2024-04-21T23:20:54.685 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:54 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:54.326+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.518301+0000 front 2024-04-21T23:20:22.518321+0000 (oldest deadline 2024-04-21T23:20:44.818306+0000) 2024-04-21T23:20:54.686 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:54 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6[53545]: 2024-04-21T23:20:54.326+0000 7fee294eb700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:27.019754+0000 front 2024-04-21T23:20:27.019806+0000 (oldest deadline 2024-04-21T23:20:52.319900+0000) 2024-04-21T23:20:54.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:54 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:54.506+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:54.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:54 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:54.506+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:20:54.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:54 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:54.506+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.642659+0000 front 2024-04-21T23:20:29.642537+0000 (oldest deadline 2024-04-21T23:20:54.243095+0000) 2024-04-21T23:20:54.716 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:54 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:54.279+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:54.716 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:54 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:54.279+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:54.716 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:54 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:54.279+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:20:54.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:54 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:54.577+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:54.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:54 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:54.577+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.533381+0000 front 2024-04-21T23:20:22.533232+0000 (oldest deadline 2024-04-21T23:20:43.633212+0000) 2024-04-21T23:20:54.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:54 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:54.577+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.935487+0000 front 2024-04-21T23:20:29.935525+0000 (oldest deadline 2024-04-21T23:20:52.835639+0000) 2024-04-21T23:20:54.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:54 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:54.476+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:54.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:54 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:54.476+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:54.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:54 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:54.476+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:20:54.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:54 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:54.476+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:20:55.185 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:54 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:54.720+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:55.185 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:54 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:54.721+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:55.185 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:54 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:54.721+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:20:55.185 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:54 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:54.721+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:20:55.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:55 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:55.245+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:55.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:55 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:55.245+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:55.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:55 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:55.245+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:20:55.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:55 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:55.492+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:55.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:55 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:55.492+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:20:55.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:55 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:55.492+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.642659+0000 front 2024-04-21T23:20:29.642537+0000 (oldest deadline 2024-04-21T23:20:54.243095+0000) 2024-04-21T23:20:55.935 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:55 smithi044 podman[93786]: 2024-04-21 23:20:55.650937462 +0000 UTC m=+5.697893657 container died 13bded7a8ab409119560bd3018c3e19b12674f4c65540d5dbe9093807f492c3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=) 2024-04-21T23:20:55.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:55 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:55.674+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:55.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:55 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:55.674+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:55.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:55 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:55.674+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:20:55.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:55 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:55.674+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:20:55.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:55 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:55.616+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:55.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:55 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:55.616+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.533381+0000 front 2024-04-21T23:20:22.533232+0000 (oldest deadline 2024-04-21T23:20:43.633212+0000) 2024-04-21T23:20:55.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:55 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:55.616+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.935487+0000 front 2024-04-21T23:20:29.935525+0000 (oldest deadline 2024-04-21T23:20:52.835639+0000) 2024-04-21T23:20:55.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:55 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:55.489+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:55.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:55 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:55.489+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:55.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:55 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:55.489+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:20:55.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:55 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:55.489+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:20:56.304 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:56 smithi044 podman[93786]: 2024-04-21 23:20:56.062834275 +0000 UTC m=+6.109790478 container cleanup 13bded7a8ab409119560bd3018c3e19b12674f4c65540d5dbe9093807f492c3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8) 2024-04-21T23:20:56.305 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:56 smithi044 bash[93786]: ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6 2024-04-21T23:20:56.627 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:56 smithi044 podman[93812]: 2024-04-21 23:20:56.306192739 +0000 UTC m=+0.653689893 container remove 13bded7a8ab409119560bd3018c3e19b12674f4c65540d5dbe9093807f492c3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, ceph=True) 2024-04-21T23:20:56.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:56 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:56.455+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:56.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:56 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:56.455+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:56.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:56 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:56.455+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:20:56.715 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:56 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:56.455+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:20:56.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:56 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:56.456+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:56.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:56 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:56.456+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:20:56.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:56 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:56.456+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.642659+0000 front 2024-04-21T23:20:29.642537+0000 (oldest deadline 2024-04-21T23:20:54.243095+0000) 2024-04-21T23:20:56.715 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:56 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:56.602+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:56.716 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:56 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:56.602+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.533381+0000 front 2024-04-21T23:20:22.533232+0000 (oldest deadline 2024-04-21T23:20:43.633212+0000) 2024-04-21T23:20:56.716 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:56 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:56.602+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.935487+0000 front 2024-04-21T23:20:29.935525+0000 (oldest deadline 2024-04-21T23:20:52.835639+0000) 2024-04-21T23:20:56.716 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:56 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:56.276+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:56.716 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:56 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:56.276+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:56.716 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:56 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:56.276+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:20:56.880 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:56 smithi044 podman[93905]: 2024-04-21 23:20:56.621889982 +0000 UTC m=+0.018675127 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:20:56.881 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:56 smithi044 podman[93905]: 2024-04-21 23:20:56.748970911 +0000 UTC m=+0.145756098 container create 5d70e9fc817d81799931c97415284c4203feda5be5a85ace718982793fc7d5db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-21T23:20:56.881 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:56 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:56.628+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:56.881 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:56 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:56.628+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:56.881 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:56 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:56.628+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:20:56.882 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:56 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:56.628+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:20:57.185 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:57 smithi044 podman[93905]: 2024-04-21 23:20:57.018244032 +0000 UTC m=+0.415029177 container init 5d70e9fc817d81799931c97415284c4203feda5be5a85ace718982793fc7d5db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-21T23:20:57.185 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:57 smithi044 podman[93905]: 2024-04-21 23:20:57.021566213 +0000 UTC m=+0.418351359 container start 5d70e9fc817d81799931c97415284c4203feda5be5a85ace718982793fc7d5db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, io.buildah.version=1.33.5) 2024-04-21T23:20:57.185 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:57 smithi044 podman[93905]: 2024-04-21 23:20:57.09309912 +0000 UTC m=+0.489884275 container attach 5d70e9fc817d81799931c97415284c4203feda5be5a85ace718982793fc7d5db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-21T23:20:57.603 INFO:journalctl@ceph.osd.6.smithi044.stdout:Apr 21 23:20:57 smithi044 podman[93905]: 2024-04-21 23:20:57.322366196 +0000 UTC m=+0.719151341 container died 5d70e9fc817d81799931c97415284c4203feda5be5a85ace718982793fc7d5db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:20:57.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:57 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:57.282+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:57.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:57 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:57.282+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:57.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:57 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:57.282+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:20:57.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:57 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:57.423+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:57.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:57 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:57.423+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:20:57.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:57 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:57.423+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.642659+0000 front 2024-04-21T23:20:29.642537+0000 (oldest deadline 2024-04-21T23:20:54.243095+0000) 2024-04-21T23:20:57.903 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:57 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:57.611+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:57.904 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:57 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:57.611+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:57.904 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:57 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:57.611+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:20:57.904 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:57 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:57.611+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:20:57.923 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.6.service' 2024-04-21T23:20:57.964 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:20:57.964 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-21T23:20:57.965 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-21T23:20:57.965 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.7 2024-04-21T23:20:57.966 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:57 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:57.559+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:57.966 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:57 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:57.559+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.533381+0000 front 2024-04-21T23:20:22.533232+0000 (oldest deadline 2024-04-21T23:20:43.633212+0000) 2024-04-21T23:20:57.966 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:57 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:57.559+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.935487+0000 front 2024-04-21T23:20:29.935525+0000 (oldest deadline 2024-04-21T23:20:52.835639+0000) 2024-04-21T23:20:57.966 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:57 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:57.470+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:57.967 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:57 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:57.470+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:57.967 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:57 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:57.470+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:20:57.967 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:57 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:57.470+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:20:58.304 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:58 smithi055 systemd[1]: Stopping Ceph osd.7 for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:20:58.304 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:58 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:58.304+0000 7f1ba3d9f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:20:58.304 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:58 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:58.304+0000 7f1ba3d9f700 -1 osd.7 114 *** Got signal Terminated *** 2024-04-21T23:20:58.304 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:58 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:58.304+0000 7f1ba3d9f700 -1 osd.7 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:20:58.581 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:58 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:58.421+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:58.582 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:58 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:58.421+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:58.582 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:58 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:58.421+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:20:58.582 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:58 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:58.421+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:20:58.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:58 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:58.288+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:58.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:58 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:58.288+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:58.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:58 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:58.288+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:20:58.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:58 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:58.402+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:58.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:58 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:58.402+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:20:58.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:58 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:58.402+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.642659+0000 front 2024-04-21T23:20:29.642537+0000 (oldest deadline 2024-04-21T23:20:54.243095+0000) 2024-04-21T23:20:58.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:58 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:58.569+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:58.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:58 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:58.569+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:58.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:58 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:58.569+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:20:58.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:58 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:58.569+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:20:58.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:58 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:58.582+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:58.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:58 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:58.582+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.533381+0000 front 2024-04-21T23:20:22.533232+0000 (oldest deadline 2024-04-21T23:20:43.633212+0000) 2024-04-21T23:20:58.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:58 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:58.582+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.935487+0000 front 2024-04-21T23:20:29.935525+0000 (oldest deadline 2024-04-21T23:20:52.835639+0000) 2024-04-21T23:20:59.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:59 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:59.295+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:20:59.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:59 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:59.295+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:20:59.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:20:59 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:20:59.295+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:20:59.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:59 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:59.423+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:20:59.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:59 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:59.423+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:20:59.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:20:59 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:20:59.423+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.642659+0000 front 2024-04-21T23:20:29.642537+0000 (oldest deadline 2024-04-21T23:20:54.243095+0000) 2024-04-21T23:20:59.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:59 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:59.612+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:20:59.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:59 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:59.612+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:20:59.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:59 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:59.612+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:20:59.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:20:59 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:20:59.612+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:20:59.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:59 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:59.533+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:20:59.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:59 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:59.533+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.533381+0000 front 2024-04-21T23:20:22.533232+0000 (oldest deadline 2024-04-21T23:20:43.633212+0000) 2024-04-21T23:20:59.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:20:59 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:20:59.533+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.935487+0000 front 2024-04-21T23:20:29.935525+0000 (oldest deadline 2024-04-21T23:20:52.835639+0000) 2024-04-21T23:20:59.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:59 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:59.467+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:20:59.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:59 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:59.467+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:20:59.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:59 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:59.467+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:20:59.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:20:59 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:20:59.467+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:00.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:00 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:00.315+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:00.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:00 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:00.315+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:00.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:00 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:00.315+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:00.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:00 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:00.419+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:21:00.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:00 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:00.419+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:21:00.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:00 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:00.419+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.642659+0000 front 2024-04-21T23:20:29.642537+0000 (oldest deadline 2024-04-21T23:20:54.243095+0000) 2024-04-21T23:21:00.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:00 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:00.419+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:34.243934+0000 front 2024-04-21T23:20:34.243867+0000 (oldest deadline 2024-04-21T23:21:00.143727+0000) 2024-04-21T23:21:00.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:00 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:00.587+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:21:00.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:00 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:00.587+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:21:00.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:00 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:00.587+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:21:00.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:00 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:00.587+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:21:00.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:00 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:00.587+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.935138+0000 front 2024-04-21T23:20:37.935372+0000 (oldest deadline 2024-04-21T23:20:59.635092+0000) 2024-04-21T23:21:00.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:00 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:00.503+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:00.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:00 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:00.503+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:00.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:00 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:00.503+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:00.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:00 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:00.503+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:00.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:00 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:21:00.560+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:21:00.965 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:00 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:21:00.560+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.533381+0000 front 2024-04-21T23:20:22.533232+0000 (oldest deadline 2024-04-21T23:20:43.633212+0000) 2024-04-21T23:21:00.965 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:00 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:21:00.560+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.935487+0000 front 2024-04-21T23:20:29.935525+0000 (oldest deadline 2024-04-21T23:20:52.835639+0000) 2024-04-21T23:21:01.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:01 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:01.330+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:01.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:01 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:01.330+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:01.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:01 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:01.330+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:01.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:01 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:01.393+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:21:01.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:01 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:01.393+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:21:01.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:01 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:01.393+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.642659+0000 front 2024-04-21T23:20:29.642537+0000 (oldest deadline 2024-04-21T23:20:54.243095+0000) 2024-04-21T23:21:01.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:01 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:01.393+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:34.243934+0000 front 2024-04-21T23:20:34.243867+0000 (oldest deadline 2024-04-21T23:21:00.143727+0000) 2024-04-21T23:21:01.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:01.571+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:21:01.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:01.571+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:21:01.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:01.571+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:21:01.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:01.571+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:21:01.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:01 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:01.571+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.935138+0000 front 2024-04-21T23:20:37.935372+0000 (oldest deadline 2024-04-21T23:20:59.635092+0000) 2024-04-21T23:21:01.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:01 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:01.504+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:01.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:01 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:01.504+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:01.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:01 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:01.504+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:01.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:01 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:01.504+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:01.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:01 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:01.504+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:01.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:01 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:21:01.582+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:21:01.965 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:01 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:21:01.582+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.533381+0000 front 2024-04-21T23:20:22.533232+0000 (oldest deadline 2024-04-21T23:20:43.633212+0000) 2024-04-21T23:21:01.965 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:01 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:21:01.582+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.935487+0000 front 2024-04-21T23:20:29.935525+0000 (oldest deadline 2024-04-21T23:20:52.835639+0000) 2024-04-21T23:21:01.965 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:01 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:21:01.582+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:35.136812+0000 front 2024-04-21T23:20:35.136865+0000 (oldest deadline 2024-04-21T23:21:01.036948+0000) 2024-04-21T23:21:02.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:02 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:02.389+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:21:02.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:02 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:02.389+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:21:02.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:02 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:02.389+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.642659+0000 front 2024-04-21T23:20:29.642537+0000 (oldest deadline 2024-04-21T23:20:54.243095+0000) 2024-04-21T23:21:02.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:02 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:02.390+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:34.243934+0000 front 2024-04-21T23:20:34.243867+0000 (oldest deadline 2024-04-21T23:21:00.143727+0000) 2024-04-21T23:21:02.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:02 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:02.370+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:02.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:02 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:02.370+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:02.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:02 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:02.370+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:02.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:02 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:02.370+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:02.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:02 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:02.553+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:21:02.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:02 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:02.553+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:21:02.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:02 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:02.553+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:21:02.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:02 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:02.553+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:21:02.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:02 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:02.553+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.935138+0000 front 2024-04-21T23:20:37.935372+0000 (oldest deadline 2024-04-21T23:20:59.635092+0000) 2024-04-21T23:21:02.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:02 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:21:02.554+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:12.631490+0000 front 2024-04-21T23:20:12.631460+0000 (oldest deadline 2024-04-21T23:20:35.531594+0000) 2024-04-21T23:21:02.964 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:02 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:21:02.554+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.533381+0000 front 2024-04-21T23:20:22.533232+0000 (oldest deadline 2024-04-21T23:20:43.633212+0000) 2024-04-21T23:21:02.965 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:02 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:21:02.554+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.935487+0000 front 2024-04-21T23:20:29.935525+0000 (oldest deadline 2024-04-21T23:20:52.835639+0000) 2024-04-21T23:21:02.965 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:02 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7[49425]: 2024-04-21T23:21:02.554+0000 7f1ba0393700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:35.136812+0000 front 2024-04-21T23:20:35.136865+0000 (oldest deadline 2024-04-21T23:21:01.036948+0000) 2024-04-21T23:21:02.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:02 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:02.547+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:02.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:02 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:02.547+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:02.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:02 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:02.547+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:02.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:02 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:02.547+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:02.966 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:02 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:02.547+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:03.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:03 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:03.379+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:03.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:03 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:03.379+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:03.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:03 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:03.379+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:03.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:03 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:03.379+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:03.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:03 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:03.382+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:21:03.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:03 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:03.382+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:21:03.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:03 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:03.382+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.642659+0000 front 2024-04-21T23:20:29.642537+0000 (oldest deadline 2024-04-21T23:20:54.243095+0000) 2024-04-21T23:21:03.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:03 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:03.382+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:34.243934+0000 front 2024-04-21T23:20:34.243867+0000 (oldest deadline 2024-04-21T23:21:00.143727+0000) 2024-04-21T23:21:03.806 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:03 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:03.535+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:03.806 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:03 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:03.535+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:03.806 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:03 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:03.535+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:03.807 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:03 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:03.535+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:03.807 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:03 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:03.535+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:03.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:03 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:03.583+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:21:03.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:03 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:03.583+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:21:03.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:03 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:03.583+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:21:03.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:03 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:03.583+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:21:03.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:03 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:03.583+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.935138+0000 front 2024-04-21T23:20:37.935372+0000 (oldest deadline 2024-04-21T23:20:59.635092+0000) 2024-04-21T23:21:04.063 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:03 smithi055 podman[67532]: 2024-04-21 23:21:03.807664127 +0000 UTC m=+5.625703463 container died a6eb6a3d68650025d282a88cb96d44c6ed7214eb245256c748d7958aa36584a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:21:04.464 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:04 smithi055 podman[67532]: 2024-04-21 23:21:04.147044348 +0000 UTC m=+5.965083685 container cleanup a6eb6a3d68650025d282a88cb96d44c6ed7214eb245256c748d7958aa36584a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-04-21T23:21:04.464 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:04 smithi055 bash[67532]: ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7 2024-04-21T23:21:04.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:04 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:04.396+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:21:04.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:04 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:04.396+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:21:04.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:04 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:04.396+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.642659+0000 front 2024-04-21T23:20:29.642537+0000 (oldest deadline 2024-04-21T23:20:54.243095+0000) 2024-04-21T23:21:04.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:04 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:04.396+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:34.243934+0000 front 2024-04-21T23:20:34.243867+0000 (oldest deadline 2024-04-21T23:21:00.143727+0000) 2024-04-21T23:21:04.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:04 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:04.395+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:04.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:04 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:04.395+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:04.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:04 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:04.395+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:04.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:04 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:04.395+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:04.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:04 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:04.559+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:21:04.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:04 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:04.559+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:21:04.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:04 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:04.559+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:21:04.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:04 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:04.559+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:21:04.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:04 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:04.559+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.935138+0000 front 2024-04-21T23:20:37.935372+0000 (oldest deadline 2024-04-21T23:20:59.635092+0000) 2024-04-21T23:21:04.956 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:04 smithi055 podman[67550]: 2024-04-21 23:21:04.507116803 +0000 UTC m=+0.696459029 container remove a6eb6a3d68650025d282a88cb96d44c6ed7214eb245256c748d7958aa36584a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:21:04.956 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:04 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:04.496+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:04.957 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:04 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:04.496+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:04.957 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:04 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:04.496+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:04.957 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:04 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:04.496+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:04.957 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:04 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:04.496+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:05.214 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:04 smithi055 podman[67643]: 2024-04-21 23:21:04.958318838 +0000 UTC m=+0.113831052 container create fab26ce71287a236a2bca514a1bd70839f10510f43511a42fb541bcc4807dd84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=) 2024-04-21T23:21:05.214 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:04 smithi055 podman[67643]: 2024-04-21 23:21:04.862970189 +0000 UTC m=+0.018482420 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:21:05.215 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:05 smithi055 podman[67643]: 2024-04-21 23:21:05.160775531 +0000 UTC m=+0.316287746 container init fab26ce71287a236a2bca514a1bd70839f10510f43511a42fb541bcc4807dd84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-820a2f8) 2024-04-21T23:21:05.215 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:05 smithi055 podman[67643]: 2024-04-21 23:21:05.163882118 +0000 UTC m=+0.319394336 container start fab26ce71287a236a2bca514a1bd70839f10510f43511a42fb541bcc4807dd84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:21:05.464 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:05 smithi055 podman[67643]: 2024-04-21 23:21:05.246925253 +0000 UTC m=+0.402437490 container attach fab26ce71287a236a2bca514a1bd70839f10510f43511a42fb541bcc4807dd84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-21T23:21:05.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:05 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:05.378+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:21:05.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:05 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:05.378+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:21:05.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:05 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:05.378+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.642659+0000 front 2024-04-21T23:20:29.642537+0000 (oldest deadline 2024-04-21T23:20:54.243095+0000) 2024-04-21T23:21:05.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:05 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:05.378+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:34.243934+0000 front 2024-04-21T23:20:34.243867+0000 (oldest deadline 2024-04-21T23:21:00.143727+0000) 2024-04-21T23:21:05.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:05 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:05.440+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:05.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:05 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:05.440+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:05.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:05 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:05.440+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:05.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:05 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:05.440+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:05.748 INFO:journalctl@ceph.osd.7.smithi055.stdout:Apr 21 23:21:05 smithi055 podman[67643]: 2024-04-21 23:21:05.466429213 +0000 UTC m=+0.621941429 container died fab26ce71287a236a2bca514a1bd70839f10510f43511a42fb541bcc4807dd84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-21T23:21:05.748 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:05 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:05.517+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:05.749 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:05 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:05.517+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:05.749 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:05 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:05.517+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:05.749 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:05 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:05.517+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:05.749 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:05 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:05.517+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:05.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:05 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:05.528+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:21:05.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:05 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:05.528+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:21:05.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:05 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:05.528+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:21:05.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:05 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:05.528+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:21:05.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:05 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:05.528+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.935138+0000 front 2024-04-21T23:20:37.935372+0000 (oldest deadline 2024-04-21T23:20:59.635092+0000) 2024-04-21T23:21:06.075 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.7.service' 2024-04-21T23:21:06.113 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:21:06.113 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-21T23:21:06.114 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-21T23:21:06.114 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.8 2024-04-21T23:21:06.425 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:06 smithi069 systemd[1]: Stopping Ceph osd.8 for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:21:06.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:06 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:06.469+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:06.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:06 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:06.469+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:06.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:06 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:06.469+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:06.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:06 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:06.469+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:06.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:06 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:06.425+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:21:06.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:06 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:06.425+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:21:06.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:06 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:06.425+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.642659+0000 front 2024-04-21T23:20:29.642537+0000 (oldest deadline 2024-04-21T23:20:54.243095+0000) 2024-04-21T23:21:06.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:06 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:06.425+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:34.243934+0000 front 2024-04-21T23:20:34.243867+0000 (oldest deadline 2024-04-21T23:21:00.143727+0000) 2024-04-21T23:21:06.716 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:06 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:06.428+0000 7f810a9ec700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:21:06.716 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:06 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:06.428+0000 7f810a9ec700 -1 osd.8 114 *** Got signal Terminated *** 2024-04-21T23:21:06.716 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:06 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:06.428+0000 7f810a9ec700 -1 osd.8 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:21:06.732 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:06 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:06.474+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:06.732 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:06 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:06.474+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:06.732 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:06 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:06.474+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:06.732 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:06 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:06.474+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:06.732 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:06 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:06.474+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:06.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:06 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:06.513+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:21:06.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:06 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:06.513+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:21:06.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:06 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:06.513+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:21:06.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:06 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:06.514+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:21:06.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:06 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:06.514+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.935138+0000 front 2024-04-21T23:20:37.935372+0000 (oldest deadline 2024-04-21T23:20:59.635092+0000) 2024-04-21T23:21:07.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:07 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:07.453+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:07.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:07 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:07.453+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:07.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:07 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:07.453+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:07.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:07 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:07.453+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:07.715 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:07 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:07.453+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:07.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:07 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:07.527+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:21:07.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:07 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:07.527+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:21:07.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:07 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:07.527+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:21:07.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:07 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:07.527+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:21:07.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:07 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:07.527+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.935138+0000 front 2024-04-21T23:20:37.935372+0000 (oldest deadline 2024-04-21T23:20:59.635092+0000) 2024-04-21T23:21:07.964 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:07 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:07.473+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:07.964 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:07 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:07.473+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:07.964 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:07 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:07.473+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:07.964 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:07 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:07.473+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:07.964 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:07 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:07.472+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:21:07.965 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:07 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:07.472+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:21:07.965 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:07 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:07.472+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.642659+0000 front 2024-04-21T23:20:29.642537+0000 (oldest deadline 2024-04-21T23:20:54.243095+0000) 2024-04-21T23:21:07.965 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:07 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:07.472+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:34.243934+0000 front 2024-04-21T23:20:34.243867+0000 (oldest deadline 2024-04-21T23:21:00.143727+0000) 2024-04-21T23:21:08.865 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:08 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:08.490+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:21:08.866 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:08 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:08.490+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:21:08.866 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:08 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:08.490+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.642659+0000 front 2024-04-21T23:20:29.642537+0000 (oldest deadline 2024-04-21T23:20:54.243095+0000) 2024-04-21T23:21:08.866 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:08 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:08.490+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:34.243934+0000 front 2024-04-21T23:20:34.243867+0000 (oldest deadline 2024-04-21T23:21:00.143727+0000) 2024-04-21T23:21:08.866 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:08 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:08.490+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:45.945124+0000 front 2024-04-21T23:20:45.945099+0000 (oldest deadline 2024-04-21T23:21:07.645263+0000) 2024-04-21T23:21:08.866 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:08 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:08.475+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:08.867 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:08 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:08.475+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:08.867 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:08 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:08.475+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:08.867 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:08 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:08.475+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:08.867 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:08 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:08.475+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:08.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:08 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:08.551+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:21:08.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:08 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:08.551+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:21:08.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:08 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:08.551+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:21:08.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:08 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:08.551+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:21:08.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:08 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:08.551+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.935138+0000 front 2024-04-21T23:20:37.935372+0000 (oldest deadline 2024-04-21T23:20:59.635092+0000) 2024-04-21T23:21:08.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:08 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:08.486+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:08.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:08 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:08.486+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:08.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:08 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:08.486+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:08.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:08 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:08.486+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:08.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:08 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:08.486+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:09.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:09 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:09.454+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:09.715 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:09 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:09.454+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:09.715 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:09 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:09.454+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:09.715 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:09 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:09.454+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:09.715 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:09 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:09.454+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:09.715 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:09 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:09.454+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:43.958266+0000 front 2024-04-21T23:20:43.958140+0000 (oldest deadline 2024-04-21T23:21:08.657993+0000) 2024-04-21T23:21:09.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:09 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:09.430+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:09.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:09 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:09.430+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:09.716 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:09 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:09.430+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:09.716 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:09 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:09.430+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:09.716 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:09 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:09.430+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:09.716 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:09 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:09.484+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:21:09.716 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:09 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:09.484+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:21:09.716 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:09 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:09.484+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.642659+0000 front 2024-04-21T23:20:29.642537+0000 (oldest deadline 2024-04-21T23:20:54.243095+0000) 2024-04-21T23:21:09.716 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:09 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:09.484+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:34.243934+0000 front 2024-04-21T23:20:34.243867+0000 (oldest deadline 2024-04-21T23:21:00.143727+0000) 2024-04-21T23:21:09.717 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:09 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:09.484+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:45.945124+0000 front 2024-04-21T23:20:45.945099+0000 (oldest deadline 2024-04-21T23:21:07.645263+0000) 2024-04-21T23:21:09.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:09 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:09.594+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:21:09.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:09 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:09.594+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:21:09.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:09 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:09.594+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:21:09.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:09 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:09.594+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:21:09.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:09 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:09.594+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.935138+0000 front 2024-04-21T23:20:37.935372+0000 (oldest deadline 2024-04-21T23:20:59.635092+0000) 2024-04-21T23:21:09.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:09 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:09.594+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:46.537361+0000 front 2024-04-21T23:20:46.537218+0000 (oldest deadline 2024-04-21T23:21:08.837126+0000) 2024-04-21T23:21:10.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:10 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:10.452+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.039387+0000 front 2024-04-21T23:20:11.039462+0000 (oldest deadline 2024-04-21T23:20:36.339370+0000) 2024-04-21T23:21:10.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:10 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:10.452+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.641449+0000 front 2024-04-21T23:20:22.641517+0000 (oldest deadline 2024-04-21T23:20:44.341529+0000) 2024-04-21T23:21:10.714 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:10 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:10.452+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:29.642659+0000 front 2024-04-21T23:20:29.642537+0000 (oldest deadline 2024-04-21T23:20:54.243095+0000) 2024-04-21T23:21:10.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:10 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:10.452+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:34.243934+0000 front 2024-04-21T23:20:34.243867+0000 (oldest deadline 2024-04-21T23:21:00.143727+0000) 2024-04-21T23:21:10.715 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:10 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8[50530]: 2024-04-21T23:21:10.452+0000 7f8102fd8700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:45.945124+0000 front 2024-04-21T23:20:45.945099+0000 (oldest deadline 2024-04-21T23:21:07.645263+0000) 2024-04-21T23:21:10.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:10 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:10.387+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:10.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:10 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:10.387+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:10.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:10 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:10.387+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:10.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:10 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:10.387+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:10.716 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:10 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:10.387+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:10.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:10 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:10.550+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:21:10.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:10 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:10.550+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:21:10.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:10 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:10.550+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:21:10.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:10 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:10.550+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:21:10.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:10 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:10.550+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.935138+0000 front 2024-04-21T23:20:37.935372+0000 (oldest deadline 2024-04-21T23:20:59.635092+0000) 2024-04-21T23:21:10.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:10 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:10.550+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:46.537361+0000 front 2024-04-21T23:20:46.537218+0000 (oldest deadline 2024-04-21T23:21:08.837126+0000) 2024-04-21T23:21:10.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:10 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:10.496+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:10.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:10 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:10.496+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:10.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:10 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:10.496+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:10.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:10 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:10.496+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:10.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:10 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:10.496+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:10.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:10 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:10.496+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:43.958266+0000 front 2024-04-21T23:20:43.958140+0000 (oldest deadline 2024-04-21T23:21:08.657993+0000) 2024-04-21T23:21:11.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:11 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:11.351+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:11.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:11 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:11.351+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:11.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:11 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:11.351+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:11.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:11 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:11.351+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:11.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:11 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:11.351+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:11.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:11 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:11.534+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:21:11.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:11 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:11.534+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:21:11.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:11 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:11.534+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:21:11.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:11 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:11.534+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:21:11.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:11 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:11.534+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.935138+0000 front 2024-04-21T23:20:37.935372+0000 (oldest deadline 2024-04-21T23:20:59.635092+0000) 2024-04-21T23:21:11.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:11 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:11.534+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:46.537361+0000 front 2024-04-21T23:20:46.537218+0000 (oldest deadline 2024-04-21T23:21:08.837126+0000) 2024-04-21T23:21:11.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:11 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:11.506+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:11.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:11 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:11.506+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:11.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:11 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:11.506+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:11.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:11 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:11.506+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:11.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:11 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:11.506+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:11.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:11 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:11.506+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:43.958266+0000 front 2024-04-21T23:20:43.958140+0000 (oldest deadline 2024-04-21T23:21:08.657993+0000) 2024-04-21T23:21:12.143 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:11 smithi069 podman[69851]: 2024-04-21 23:21:11.845683401 +0000 UTC m=+5.540104499 container died 16976d762e553d86fa07164b8573422983f4068e31cfd143c0356fa29e5ff179 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-21T23:21:12.143 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:12 smithi069 podman[69851]: 2024-04-21 23:21:12.145038927 +0000 UTC m=+5.839460030 container cleanup 16976d762e553d86fa07164b8573422983f4068e31cfd143c0356fa29e5ff179 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:21:12.411 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:12 smithi069 bash[69851]: ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8 2024-04-21T23:21:12.411 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:12 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:12.348+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:12.411 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:12 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:12.348+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:12.411 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:12 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:12.348+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:12.412 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:12 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:12.348+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:12.412 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:12 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:12.348+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:12.705 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:12 smithi069 podman[69882]: 2024-04-21 23:21:12.41281597 +0000 UTC m=+0.568226654 container remove 16976d762e553d86fa07164b8573422983f4068e31cfd143c0356fa29e5ff179 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-21T23:21:12.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:12 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:12.561+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:21:12.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:12 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:12.561+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:21:12.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:12 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:12.561+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:21:12.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:12 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:12.561+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:21:12.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:12 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:12.561+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.935138+0000 front 2024-04-21T23:20:37.935372+0000 (oldest deadline 2024-04-21T23:20:59.635092+0000) 2024-04-21T23:21:12.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:12 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:12.561+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:46.537361+0000 front 2024-04-21T23:20:46.537218+0000 (oldest deadline 2024-04-21T23:21:08.837126+0000) 2024-04-21T23:21:12.956 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:12 smithi069 podman[69973]: 2024-04-21 23:21:12.609799422 +0000 UTC m=+0.018119760 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:21:12.956 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:12 smithi069 podman[69973]: 2024-04-21 23:21:12.722002295 +0000 UTC m=+0.130322621 container create 4ed060fe0a6beb8998a008f5e8854370f908c5c934a7e870e671a19c083d1963 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:21:12.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:12 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:12.549+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:12.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:12 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:12.549+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:12.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:12 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:12.549+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:12.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:12 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:12.549+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:12.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:12 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:12.549+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:12.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:12 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:12.549+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:43.958266+0000 front 2024-04-21T23:20:43.958140+0000 (oldest deadline 2024-04-21T23:21:08.657993+0000) 2024-04-21T23:21:13.214 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:12 smithi069 podman[69973]: 2024-04-21 23:21:12.958216604 +0000 UTC m=+0.366536944 container init 4ed060fe0a6beb8998a008f5e8854370f908c5c934a7e870e671a19c083d1963 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-21T23:21:13.214 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:12 smithi069 podman[69973]: 2024-04-21 23:21:12.961893027 +0000 UTC m=+0.370213354 container start 4ed060fe0a6beb8998a008f5e8854370f908c5c934a7e870e671a19c083d1963 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=squid-820a2f8) 2024-04-21T23:21:13.215 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:13 smithi069 podman[69973]: 2024-04-21 23:21:13.057978088 +0000 UTC m=+0.466298440 container attach 4ed060fe0a6beb8998a008f5e8854370f908c5c934a7e870e671a19c083d1963 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:21:13.559 INFO:journalctl@ceph.osd.8.smithi069.stdout:Apr 21 23:21:13 smithi069 podman[69973]: 2024-04-21 23:21:13.253749851 +0000 UTC m=+0.662070182 container died 4ed060fe0a6beb8998a008f5e8854370f908c5c934a7e870e671a19c083d1963 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, ceph=True, io.buildah.version=1.33.5) 2024-04-21T23:21:13.560 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:13 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:13.370+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:13.560 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:13 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:13.370+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:13.560 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:13 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:13.370+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:13.560 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:13 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:13.370+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:13.560 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:13 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:13.370+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:13.879 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.8.service' 2024-04-21T23:21:13.917 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:21:13.917 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-21T23:21:13.918 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-21T23:21:13.918 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.9 2024-04-21T23:21:13.934 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:13 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:13.545+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:21:13.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:13 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:13.545+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:21:13.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:13 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:13.545+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:21:13.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:13 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:13.545+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:21:13.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:13 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:13.545+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.935138+0000 front 2024-04-21T23:20:37.935372+0000 (oldest deadline 2024-04-21T23:20:59.635092+0000) 2024-04-21T23:21:13.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:13 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:13.545+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:46.537361+0000 front 2024-04-21T23:20:46.537218+0000 (oldest deadline 2024-04-21T23:21:08.837126+0000) 2024-04-21T23:21:13.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:13 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:13.595+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:13.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:13 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:13.595+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:13.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:13 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:13.595+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:13.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:13 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:13.595+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:13.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:13 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:13.595+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:13.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:13 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:13.595+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:43.958266+0000 front 2024-04-21T23:20:43.958140+0000 (oldest deadline 2024-04-21T23:21:08.657993+0000) 2024-04-21T23:21:14.242 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:13 smithi044 systemd[1]: Stopping Ceph osd.9 for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:21:14.560 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:14 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:14.243+0000 7f88f1e0e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:21:14.560 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:14 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:14.243+0000 7f88f1e0e700 -1 osd.9 114 *** Got signal Terminated *** 2024-04-21T23:21:14.560 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:14 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:14.243+0000 7f88f1e0e700 -1 osd.9 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:21:14.560 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:14 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:14.560+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:21:14.560 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:14 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:14.560+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:21:14.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:14 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:14.345+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:14.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:14 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:14.345+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:14.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:14 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:14.345+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:14.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:14 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:14.345+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:14.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:14 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:14.345+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:14.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:14 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:14.560+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:21:14.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:14 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:14.560+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:21:14.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:14 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:14.560+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.935138+0000 front 2024-04-21T23:20:37.935372+0000 (oldest deadline 2024-04-21T23:20:59.635092+0000) 2024-04-21T23:21:14.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:14 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:14.560+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:46.537361+0000 front 2024-04-21T23:20:46.537218+0000 (oldest deadline 2024-04-21T23:21:08.837126+0000) 2024-04-21T23:21:14.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:14 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:14.614+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:14.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:14 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:14.614+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:14.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:14 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:14.614+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:14.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:14 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:14.614+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:14.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:14 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:14.614+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:14.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:14 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:14.614+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:43.958266+0000 front 2024-04-21T23:20:43.958140+0000 (oldest deadline 2024-04-21T23:21:08.657993+0000) 2024-04-21T23:21:15.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:15 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:15.320+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:15.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:15 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:15.320+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:15.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:15 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:15.320+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:15.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:15 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:15.320+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:15.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:15 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:15.320+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:15.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:15 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:15.521+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:21:15.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:15 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:15.521+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:21:15.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:15 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:15.521+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:21:15.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:15 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:15.521+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:21:15.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:15 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:15.521+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.935138+0000 front 2024-04-21T23:20:37.935372+0000 (oldest deadline 2024-04-21T23:20:59.635092+0000) 2024-04-21T23:21:15.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:15 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:15.521+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:46.537361+0000 front 2024-04-21T23:20:46.537218+0000 (oldest deadline 2024-04-21T23:21:08.837126+0000) 2024-04-21T23:21:15.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:15 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:15.636+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:15.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:15 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:15.636+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:15.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:15 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:15.636+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:15.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:15 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:15.636+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:15.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:15 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:15.636+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:15.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:15 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:15.636+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:43.958266+0000 front 2024-04-21T23:20:43.958140+0000 (oldest deadline 2024-04-21T23:21:08.657993+0000) 2024-04-21T23:21:16.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:16 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:16.330+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:16.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:16 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:16.330+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:16.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:16 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:16.330+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:16.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:16 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:16.330+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:16.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:16 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:16.330+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:16.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:16 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:16.330+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:54.195151+0000 front 2024-04-21T23:20:54.195068+0000 (oldest deadline 2024-04-21T23:21:15.895091+0000) 2024-04-21T23:21:16.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:16 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:16.538+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:21:16.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:16 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:16.538+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:21:16.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:16 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:16.538+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:21:16.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:16 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:16.538+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:21:16.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:16 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:16.538+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.935138+0000 front 2024-04-21T23:20:37.935372+0000 (oldest deadline 2024-04-21T23:20:59.635092+0000) 2024-04-21T23:21:16.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:16 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:16.538+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:46.537361+0000 front 2024-04-21T23:20:46.537218+0000 (oldest deadline 2024-04-21T23:21:08.837126+0000) 2024-04-21T23:21:16.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:16 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:16.677+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:16.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:16 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:16.677+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:16.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:16 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:16.677+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:16.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:16 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:16.677+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:16.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:16 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:16.677+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:16.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:16 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:16.677+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:43.958266+0000 front 2024-04-21T23:20:43.958140+0000 (oldest deadline 2024-04-21T23:21:08.657993+0000) 2024-04-21T23:21:17.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:17 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:17.314+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:17.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:17 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:17.314+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:17.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:17 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:17.314+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:17.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:17 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:17.314+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:17.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:17 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:17.314+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:17.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:17 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:17.314+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:54.195151+0000 front 2024-04-21T23:20:54.195068+0000 (oldest deadline 2024-04-21T23:21:15.895091+0000) 2024-04-21T23:21:17.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:17 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:17.496+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:21:17.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:17 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:17.496+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:21:17.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:17 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:17.496+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:21:17.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:17 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:17.496+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:21:17.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:17 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:17.496+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.935138+0000 front 2024-04-21T23:20:37.935372+0000 (oldest deadline 2024-04-21T23:20:59.635092+0000) 2024-04-21T23:21:17.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:17 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:17.496+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:46.537361+0000 front 2024-04-21T23:20:46.537218+0000 (oldest deadline 2024-04-21T23:21:08.837126+0000) 2024-04-21T23:21:17.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:17 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:17.651+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:17.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:17 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:17.651+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:17.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:17 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:17.651+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:17.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:17 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:17.651+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:17.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:17 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:17.651+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:17.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:17 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:17.651+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:43.958266+0000 front 2024-04-21T23:20:43.958140+0000 (oldest deadline 2024-04-21T23:21:08.657993+0000) 2024-04-21T23:21:17.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:17 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:17.651+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:53.859738+0000 front 2024-04-21T23:20:53.859716+0000 (oldest deadline 2024-04-21T23:21:16.759719+0000) 2024-04-21T23:21:18.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:18 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:18.280+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:18.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:18 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:18.280+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:18.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:18 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:18.280+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:18.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:18 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:18.280+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:18.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:18 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:18.280+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:18.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:18 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:18.280+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:54.195151+0000 front 2024-04-21T23:20:54.195068+0000 (oldest deadline 2024-04-21T23:21:15.895091+0000) 2024-04-21T23:21:18.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:18 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:18.505+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:01.129360+0000 front 2024-04-21T23:20:01.129445+0000 (oldest deadline 2024-04-21T23:20:31.729996+0000) 2024-04-21T23:21:18.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:18 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:18.505+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.730523+0000 front 2024-04-21T23:20:11.730771+0000 (oldest deadline 2024-04-21T23:20:35.830569+0000) 2024-04-21T23:21:18.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:18 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:18.505+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:21.131848+0000 front 2024-04-21T23:20:21.131932+0000 (oldest deadline 2024-04-21T23:20:45.231931+0000) 2024-04-21T23:21:18.935 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:18 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:18.505+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.333898+0000 front 2024-04-21T23:20:30.333934+0000 (oldest deadline 2024-04-21T23:20:53.834027+0000) 2024-04-21T23:21:18.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:18 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:18.505+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.935138+0000 front 2024-04-21T23:20:37.935372+0000 (oldest deadline 2024-04-21T23:20:59.635092+0000) 2024-04-21T23:21:18.936 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:18 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9[60560]: 2024-04-21T23:21:18.505+0000 7f88ee402700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:46.537361+0000 front 2024-04-21T23:20:46.537218+0000 (oldest deadline 2024-04-21T23:21:08.837126+0000) 2024-04-21T23:21:18.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:18 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:18.613+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:18.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:18 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:18.613+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:18.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:18 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:18.613+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:18.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:18 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:18.613+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:18.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:18 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:18.613+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:18.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:18 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:18.613+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:43.958266+0000 front 2024-04-21T23:20:43.958140+0000 (oldest deadline 2024-04-21T23:21:08.657993+0000) 2024-04-21T23:21:18.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:18 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:18.613+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:53.859738+0000 front 2024-04-21T23:20:53.859716+0000 (oldest deadline 2024-04-21T23:21:16.759719+0000) 2024-04-21T23:21:19.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:19 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:19.304+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:19.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:19 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:19.304+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:19.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:19 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:19.304+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:19.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:19 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:19.304+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:19.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:19 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:19.304+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:19.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:19 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:19.304+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:54.195151+0000 front 2024-04-21T23:20:54.195068+0000 (oldest deadline 2024-04-21T23:21:15.895091+0000) 2024-04-21T23:21:19.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:19 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:19.659+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:19.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:19 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:19.659+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:19.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:19 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:19.659+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:19.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:19 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:19.659+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:19.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:19 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:19.659+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:19.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:19 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:19.659+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:43.958266+0000 front 2024-04-21T23:20:43.958140+0000 (oldest deadline 2024-04-21T23:21:08.657993+0000) 2024-04-21T23:21:19.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:19 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:19.659+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:53.859738+0000 front 2024-04-21T23:20:53.859716+0000 (oldest deadline 2024-04-21T23:21:16.759719+0000) 2024-04-21T23:21:19.984 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:19 smithi044 podman[94012]: 2024-04-21 23:21:19.721259835 +0000 UTC m=+5.601156353 container died 6c5d54bdf816b4807687196aceab8762fc68793fbe3a76285f59ecd032ea559e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, CEPH_POINT_RELEASE=) 2024-04-21T23:21:20.360 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:20 smithi044 podman[94012]: 2024-04-21 23:21:20.059888843 +0000 UTC m=+5.939785368 container cleanup 6c5d54bdf816b4807687196aceab8762fc68793fbe3a76285f59ecd032ea559e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-820a2f8, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-21T23:21:20.360 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:20 smithi044 bash[94012]: ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9 2024-04-21T23:21:20.684 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:20 smithi044 podman[94029]: 2024-04-21 23:21:20.361125082 +0000 UTC m=+0.636189559 container remove 6c5d54bdf816b4807687196aceab8762fc68793fbe3a76285f59ecd032ea559e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8) 2024-04-21T23:21:20.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:20 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:20.347+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:20.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:20 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:20.347+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:20.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:20 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:20.347+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:20.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:20 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:20.347+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:20.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:20 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:20.347+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:20.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:20 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:20.347+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:54.195151+0000 front 2024-04-21T23:20:54.195068+0000 (oldest deadline 2024-04-21T23:21:15.895091+0000) 2024-04-21T23:21:20.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:20 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:20.642+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:20.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:20 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:20.642+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:20.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:20 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:20.642+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:20.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:20 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:20.642+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:20.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:20 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:20.642+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:20.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:20 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:20.642+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:43.958266+0000 front 2024-04-21T23:20:43.958140+0000 (oldest deadline 2024-04-21T23:21:08.657993+0000) 2024-04-21T23:21:20.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:20 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:20.642+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:53.859738+0000 front 2024-04-21T23:20:53.859716+0000 (oldest deadline 2024-04-21T23:21:16.759719+0000) 2024-04-21T23:21:21.185 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:20 smithi044 podman[94117]: 2024-04-21 23:21:20.749043686 +0000 UTC m=+0.018242533 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:21:21.185 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:20 smithi044 podman[94117]: 2024-04-21 23:21:20.853780453 +0000 UTC m=+0.122979330 container create 809d3ef6059540905614dfa6a8cba05622e32452d740e19b386391706e5d7713 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-21T23:21:21.462 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:21 smithi044 podman[94117]: 2024-04-21 23:21:21.189582955 +0000 UTC m=+0.458781786 container init 809d3ef6059540905614dfa6a8cba05622e32452d740e19b386391706e5d7713 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:21:21.462 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:21 smithi044 podman[94117]: 2024-04-21 23:21:21.193013697 +0000 UTC m=+0.462212533 container start 809d3ef6059540905614dfa6a8cba05622e32452d740e19b386391706e5d7713 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:21:21.463 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:21 smithi044 podman[94117]: 2024-04-21 23:21:21.264625668 +0000 UTC m=+0.533824499 container attach 809d3ef6059540905614dfa6a8cba05622e32452d740e19b386391706e5d7713 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=squid-820a2f8, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-21T23:21:21.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:21 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:21.330+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:21.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:21 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:21.330+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:21.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:21 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:21.330+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:21.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:21 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:21.330+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:21.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:21 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:21.330+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:21.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:21 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:21.330+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:54.195151+0000 front 2024-04-21T23:20:54.195068+0000 (oldest deadline 2024-04-21T23:21:15.895091+0000) 2024-04-21T23:21:21.758 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:21 smithi044 podman[94143]: 2024-04-21 23:21:21.543413705 +0000 UTC m=+0.019874273 container died 809d3ef6059540905614dfa6a8cba05622e32452d740e19b386391706e5d7713 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True) 2024-04-21T23:21:21.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:21 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:21.678+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:21.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:21 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:21.678+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:21.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:21 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:21.678+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:21.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:21 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:21.678+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:21.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:21 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:21.678+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:21.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:21 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:21.678+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:43.958266+0000 front 2024-04-21T23:20:43.958140+0000 (oldest deadline 2024-04-21T23:21:08.657993+0000) 2024-04-21T23:21:21.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:21 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:21.678+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:53.859738+0000 front 2024-04-21T23:20:53.859716+0000 (oldest deadline 2024-04-21T23:21:16.759719+0000) 2024-04-21T23:21:21.986 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.9.service' 2024-04-21T23:21:22.048 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:21 smithi044 podman[94143]: 2024-04-21 23:21:21.967945043 +0000 UTC m=+0.444405613 container remove 809d3ef6059540905614dfa6a8cba05622e32452d740e19b386391706e5d7713 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-820a2f8) 2024-04-21T23:21:22.049 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:21 smithi044 systemd[1]: ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.9.service: Deactivated successfully. 2024-04-21T23:21:22.049 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:21 smithi044 systemd[1]: Stopped Ceph osd.9 for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:21:22.049 INFO:journalctl@ceph.osd.9.smithi044.stdout:Apr 21 23:21:21 smithi044 systemd[1]: ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.9.service: Consumed 2min 54.745s CPU time. 2024-04-21T23:21:22.064 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:21:22.065 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-21T23:21:22.065 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-21T23:21:22.065 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.10 2024-04-21T23:21:22.378 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:22 smithi055 systemd[1]: Stopping Ceph osd.10 for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:21:22.378 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:22 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:22.378+0000 7f277e741700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:21:22.661 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:22 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:22.378+0000 7f277e741700 -1 osd.10 114 *** Got signal Terminated *** 2024-04-21T23:21:22.662 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:22 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:22.378+0000 7f277e741700 -1 osd.10 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:21:22.662 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:22 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:22.663+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:22.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:22 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:22.361+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:22.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:22 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:22.361+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:22.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:22 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:22.361+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:22.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:22 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:22.361+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:22.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:22 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:22.361+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:22.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:22 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:22.361+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:54.195151+0000 front 2024-04-21T23:20:54.195068+0000 (oldest deadline 2024-04-21T23:21:15.895091+0000) 2024-04-21T23:21:22.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:22 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:22.663+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:22.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:22 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:22.663+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:22.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:22 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:22.663+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:22.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:22 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:22.663+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:22.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:22 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:22.663+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:43.958266+0000 front 2024-04-21T23:20:43.958140+0000 (oldest deadline 2024-04-21T23:21:08.657993+0000) 2024-04-21T23:21:22.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:22 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:22.663+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:53.859738+0000 front 2024-04-21T23:20:53.859716+0000 (oldest deadline 2024-04-21T23:21:16.759719+0000) 2024-04-21T23:21:23.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:23 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:23.399+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:23.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:23 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:23.399+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:23.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:23 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:23.399+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:23.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:23 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:23.399+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:23.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:23 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:23.399+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:23.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:23 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:23.399+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:54.195151+0000 front 2024-04-21T23:20:54.195068+0000 (oldest deadline 2024-04-21T23:21:15.895091+0000) 2024-04-21T23:21:23.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:23 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:23.624+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:23.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:23 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:23.624+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:23.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:23 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:23.624+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:23.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:23 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:23.624+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:23.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:23 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:23.624+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:23.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:23 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:23.624+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:43.958266+0000 front 2024-04-21T23:20:43.958140+0000 (oldest deadline 2024-04-21T23:21:08.657993+0000) 2024-04-21T23:21:23.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:23 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:23.624+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:53.859738+0000 front 2024-04-21T23:20:53.859716+0000 (oldest deadline 2024-04-21T23:21:16.759719+0000) 2024-04-21T23:21:24.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:24 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:24.377+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:24.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:24 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:24.377+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:24.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:24 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:24.377+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:24.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:24 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:24.377+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:24.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:24 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:24.377+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:24.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:24 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:24.377+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:54.195151+0000 front 2024-04-21T23:20:54.195068+0000 (oldest deadline 2024-04-21T23:21:15.895091+0000) 2024-04-21T23:21:24.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:24 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:24.628+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:24.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:24 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:24.628+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:24.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:24 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:24.628+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:24.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:24 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:24.628+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:24.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:24 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:24.628+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:24.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:24 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:24.628+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:43.958266+0000 front 2024-04-21T23:20:43.958140+0000 (oldest deadline 2024-04-21T23:21:08.657993+0000) 2024-04-21T23:21:24.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:24 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:24.628+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:53.859738+0000 front 2024-04-21T23:20:53.859716+0000 (oldest deadline 2024-04-21T23:21:16.759719+0000) 2024-04-21T23:21:25.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:25 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:25.379+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:25.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:25 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:25.379+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:25.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:25 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:25.379+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:25.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:25 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:25.379+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:25.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:25 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:25.379+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:25.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:25 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:25.379+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:54.195151+0000 front 2024-04-21T23:20:54.195068+0000 (oldest deadline 2024-04-21T23:21:15.895091+0000) 2024-04-21T23:21:25.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:25 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:25.651+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:25.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:25 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:25.651+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:25.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:25 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:25.651+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:25.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:25 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:25.651+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:25.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:25 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:25.651+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:25.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:25 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:25.651+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:43.958266+0000 front 2024-04-21T23:20:43.958140+0000 (oldest deadline 2024-04-21T23:21:08.657993+0000) 2024-04-21T23:21:25.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:25 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:25.651+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:53.859738+0000 front 2024-04-21T23:20:53.859716+0000 (oldest deadline 2024-04-21T23:21:16.759719+0000) 2024-04-21T23:21:26.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:26 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:26.412+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:26.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:26 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:26.412+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:26.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:26 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:26.412+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:26.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:26 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:26.412+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:26.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:26 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:26.412+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:26.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:26 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:26.412+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:54.195151+0000 front 2024-04-21T23:20:54.195068+0000 (oldest deadline 2024-04-21T23:21:15.895091+0000) 2024-04-21T23:21:26.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:26 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:26.606+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:04.149278+0000 front 2024-04-21T23:20:04.149246+0000 (oldest deadline 2024-04-21T23:20:30.049120+0000) 2024-04-21T23:21:26.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:26 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:26.606+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:14.050956+0000 front 2024-04-21T23:20:14.050991+0000 (oldest deadline 2024-04-21T23:20:36.851779+0000) 2024-04-21T23:21:26.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:26 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:26.606+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:20.853264+0000 front 2024-04-21T23:20:20.853285+0000 (oldest deadline 2024-04-21T23:20:45.553257+0000) 2024-04-21T23:21:26.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:26 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:26.606+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6814 osd.3 since back 2024-04-21T23:20:30.754756+0000 front 2024-04-21T23:20:30.754851+0000 (oldest deadline 2024-04-21T23:20:53.654891+0000) 2024-04-21T23:21:26.964 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:26 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:26.606+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:37.155955+0000 front 2024-04-21T23:20:37.155908+0000 (oldest deadline 2024-04-21T23:21:00.656191+0000) 2024-04-21T23:21:26.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:26 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:26.606+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:43.958266+0000 front 2024-04-21T23:20:43.958140+0000 (oldest deadline 2024-04-21T23:21:08.657993+0000) 2024-04-21T23:21:26.965 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:26 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10[56306]: 2024-04-21T23:21:26.606+0000 7f2776d2d700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:53.859738+0000 front 2024-04-21T23:20:53.859716+0000 (oldest deadline 2024-04-21T23:21:16.759719+0000) 2024-04-21T23:21:27.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:27 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:27.417+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:27.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:27 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:27.417+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:27.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:27 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:27.417+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:27.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:27 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:27.417+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:27.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:27 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:27.417+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:27.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:27 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:27.417+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:54.195151+0000 front 2024-04-21T23:20:54.195068+0000 (oldest deadline 2024-04-21T23:21:15.895091+0000) 2024-04-21T23:21:27.716 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:27 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:27.417+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6820 osd.7 since back 2024-04-21T23:21:01.096791+0000 front 2024-04-21T23:21:01.096914+0000 (oldest deadline 2024-04-21T23:21:26.897337+0000) 2024-04-21T23:21:28.020 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:27 smithi055 podman[67764]: 2024-04-21 23:21:27.76044855 +0000 UTC m=+5.491774760 container died 78a7d8836a7ceca598c8fac5d6453136bdf9fac1a5d89db5108c120f820a475d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8) 2024-04-21T23:21:28.395 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:28 smithi055 podman[67764]: 2024-04-21 23:21:28.112012884 +0000 UTC m=+5.843339103 container cleanup 78a7d8836a7ceca598c8fac5d6453136bdf9fac1a5d89db5108c120f820a475d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:21:28.395 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:28 smithi055 bash[67764]: ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10 2024-04-21T23:21:28.714 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:28 smithi055 podman[67782]: 2024-04-21 23:21:28.396568762 +0000 UTC m=+0.633006020 container remove 78a7d8836a7ceca598c8fac5d6453136bdf9fac1a5d89db5108c120f820a475d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-21T23:21:28.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:28 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:28.420+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:28.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:28 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:28.420+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:28.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:28 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:28.420+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:28.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:28 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:28.420+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:28.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:28 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:28.420+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:28.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:28 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:28.420+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:54.195151+0000 front 2024-04-21T23:20:54.195068+0000 (oldest deadline 2024-04-21T23:21:15.895091+0000) 2024-04-21T23:21:28.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:28 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:28.420+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6820 osd.7 since back 2024-04-21T23:21:01.096791+0000 front 2024-04-21T23:21:01.096914+0000 (oldest deadline 2024-04-21T23:21:26.897337+0000) 2024-04-21T23:21:29.007 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:28 smithi055 podman[67871]: 2024-04-21 23:21:28.656366523 +0000 UTC m=+0.018301770 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:21:29.008 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:28 smithi055 podman[67871]: 2024-04-21 23:21:28.897794092 +0000 UTC m=+0.259729365 container create 4b39ab3ff356f01fe075e597924afcbdd02a5baaf1a240458030b0f1914e4f76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:21:29.403 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:29 smithi055 podman[67871]: 2024-04-21 23:21:29.100494844 +0000 UTC m=+0.462430074 container init 4b39ab3ff356f01fe075e597924afcbdd02a5baaf1a240458030b0f1914e4f76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-21T23:21:29.403 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:29 smithi055 podman[67871]: 2024-04-21 23:21:29.103840349 +0000 UTC m=+0.465775577 container start 4b39ab3ff356f01fe075e597924afcbdd02a5baaf1a240458030b0f1914e4f76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-21T23:21:29.403 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:29 smithi055 podman[67871]: 2024-04-21 23:21:29.17566403 +0000 UTC m=+0.537599267 container attach 4b39ab3ff356f01fe075e597924afcbdd02a5baaf1a240458030b0f1914e4f76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:21:29.685 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:29 smithi055 podman[67871]: 2024-04-21 23:21:29.405045144 +0000 UTC m=+0.766980373 container died 4b39ab3ff356f01fe075e597924afcbdd02a5baaf1a240458030b0f1914e4f76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, ceph=True, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:21:29.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:29 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:29.452+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:29.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:29 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:29.452+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:29.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:29 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:29.452+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:29.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:29 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:29.452+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:29.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:29 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:29.452+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:29.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:29 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:29.452+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:54.195151+0000 front 2024-04-21T23:20:54.195068+0000 (oldest deadline 2024-04-21T23:21:15.895091+0000) 2024-04-21T23:21:29.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:29 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:29.452+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6820 osd.7 since back 2024-04-21T23:21:01.096791+0000 front 2024-04-21T23:21:01.096914+0000 (oldest deadline 2024-04-21T23:21:26.897337+0000) 2024-04-21T23:21:29.942 INFO:journalctl@ceph.osd.10.smithi055.stdout:Apr 21 23:21:29 smithi055 podman[67897]: 2024-04-21 23:21:29.929299929 +0000 UTC m=+0.518576719 container remove 4b39ab3ff356f01fe075e597924afcbdd02a5baaf1a240458030b0f1914e4f76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:21:29.948 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.10.service' 2024-04-21T23:21:29.986 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:21:29.987 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-21T23:21:29.987 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-21T23:21:29.987 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.11 2024-04-21T23:21:30.345 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:30 smithi069 systemd[1]: Stopping Ceph osd.11 for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:21:30.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:30 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:30.345+0000 7f2db16e1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:21:30.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:30 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:30.345+0000 7f2db16e1700 -1 osd.11 114 *** Got signal Terminated *** 2024-04-21T23:21:30.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:30 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:30.345+0000 7f2db16e1700 -1 osd.11 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T23:21:30.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:30 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:30.445+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:30.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:30 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:30.445+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:30.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:30 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:30.445+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:30.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:30 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:30.445+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:30.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:30 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:30.445+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:30.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:30 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:30.445+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:54.195151+0000 front 2024-04-21T23:20:54.195068+0000 (oldest deadline 2024-04-21T23:21:15.895091+0000) 2024-04-21T23:21:30.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:30 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:30.445+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6820 osd.7 since back 2024-04-21T23:21:01.096791+0000 front 2024-04-21T23:21:01.096914+0000 (oldest deadline 2024-04-21T23:21:26.897337+0000) 2024-04-21T23:21:31.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:31 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:31.405+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:31.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:31 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:31.405+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:31.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:31 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:31.405+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:31.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:31 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:31.405+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:31.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:31 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:31.405+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:31.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:31 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:31.405+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:54.195151+0000 front 2024-04-21T23:20:54.195068+0000 (oldest deadline 2024-04-21T23:21:15.895091+0000) 2024-04-21T23:21:31.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:31 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:31.405+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6820 osd.7 since back 2024-04-21T23:21:01.096791+0000 front 2024-04-21T23:21:01.096914+0000 (oldest deadline 2024-04-21T23:21:26.897337+0000) 2024-04-21T23:21:32.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:32 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:32.417+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:32.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:32 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:32.417+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:32.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:32 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:32.417+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:32.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:32 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:32.417+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:32.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:32 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:32.417+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:32.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:32 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:32.417+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:54.195151+0000 front 2024-04-21T23:20:54.195068+0000 (oldest deadline 2024-04-21T23:21:15.895091+0000) 2024-04-21T23:21:32.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:32 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:32.417+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6820 osd.7 since back 2024-04-21T23:21:01.096791+0000 front 2024-04-21T23:21:01.096914+0000 (oldest deadline 2024-04-21T23:21:26.897337+0000) 2024-04-21T23:21:33.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:33 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:33.451+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:33.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:33 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:33.451+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:33.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:33 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:33.451+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:33.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:33 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:33.451+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:33.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:33 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:33.451+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:33.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:33 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:33.451+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:54.195151+0000 front 2024-04-21T23:20:54.195068+0000 (oldest deadline 2024-04-21T23:21:15.895091+0000) 2024-04-21T23:21:33.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:33 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:33.451+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6820 osd.7 since back 2024-04-21T23:21:01.096791+0000 front 2024-04-21T23:21:01.096914+0000 (oldest deadline 2024-04-21T23:21:26.897337+0000) 2024-04-21T23:21:34.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:34 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:34.428+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6806 osd.0 since back 2024-04-21T23:20:05.386688+0000 front 2024-04-21T23:20:05.386395+0000 (oldest deadline 2024-04-21T23:20:28.187119+0000) 2024-04-21T23:21:34.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:34 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:34.428+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6804 osd.1 since back 2024-04-21T23:20:11.588691+0000 front 2024-04-21T23:20:11.589013+0000 (oldest deadline 2024-04-21T23:20:36.888707+0000) 2024-04-21T23:21:34.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:34 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:34.428+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6804 osd.2 since back 2024-04-21T23:20:22.090413+0000 front 2024-04-21T23:20:22.090592+0000 (oldest deadline 2024-04-21T23:20:48.490964+0000) 2024-04-21T23:21:34.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:34 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:34.428+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6812 osd.4 since back 2024-04-21T23:20:38.393037+0000 front 2024-04-21T23:20:38.392926+0000 (oldest deadline 2024-04-21T23:21:01.892806+0000) 2024-04-21T23:21:34.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:34 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:34.428+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.69:6812 osd.5 since back 2024-04-21T23:20:41.893288+0000 front 2024-04-21T23:20:41.893260+0000 (oldest deadline 2024-04-21T23:21:07.793239+0000) 2024-04-21T23:21:34.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:34 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:34.428+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.44:6822 osd.6 since back 2024-04-21T23:20:54.195151+0000 front 2024-04-21T23:20:54.195068+0000 (oldest deadline 2024-04-21T23:21:15.895091+0000) 2024-04-21T23:21:34.715 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:34 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11[57536]: 2024-04-21T23:21:34.428+0000 7f2da9ccd700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.55:6820 osd.7 since back 2024-04-21T23:21:01.096791+0000 front 2024-04-21T23:21:01.096914+0000 (oldest deadline 2024-04-21T23:21:26.897337+0000) 2024-04-21T23:21:36.214 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:35 smithi069 podman[70079]: 2024-04-21 23:21:35.906956068 +0000 UTC m=+5.728247733 container died 7f59ec119856afe7b8595d8b509ceeece18543d070b504055862fe3560f15410 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-820a2f8, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-21T23:21:36.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:36 smithi069 podman[70079]: 2024-04-21 23:21:36.238211613 +0000 UTC m=+6.059503284 container cleanup 7f59ec119856afe7b8595d8b509ceeece18543d070b504055862fe3560f15410 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-21T23:21:36.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:36 smithi069 bash[70079]: ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11 2024-04-21T23:21:36.714 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:36 smithi069 podman[70099]: 2024-04-21 23:21:36.43046895 +0000 UTC m=+0.520745763 container remove 7f59ec119856afe7b8595d8b509ceeece18543d070b504055862fe3560f15410 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD) 2024-04-21T23:21:37.081 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:36 smithi069 podman[70189]: 2024-04-21 23:21:36.831222843 +0000 UTC m=+0.094672390 container create 1fa85a5c099d333cbe7df16ee75059f6a5346d75de66d43fd99f5888c4089969 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11-deactivate, ceph=True, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-21T23:21:37.082 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:36 smithi069 podman[70189]: 2024-04-21 23:21:36.755126365 +0000 UTC m=+0.018575938 image pull 6702148920adeca800d3e709b657e413bd3a63e83b20c5b975b68dabf7989f32 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9 2024-04-21T23:21:37.082 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:37 smithi069 podman[70189]: 2024-04-21 23:21:37.025543662 +0000 UTC m=+0.288993222 container init 1fa85a5c099d333cbe7df16ee75059f6a5346d75de66d43fd99f5888c4089969 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8) 2024-04-21T23:21:37.082 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:37 smithi069 podman[70189]: 2024-04-21 23:21:37.028885812 +0000 UTC m=+0.292335365 container start 1fa85a5c099d333cbe7df16ee75059f6a5346d75de66d43fd99f5888c4089969 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:21:37.082 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:37 smithi069 podman[70189]: 2024-04-21 23:21:37.083362507 +0000 UTC m=+0.346812068 container attach 1fa85a5c099d333cbe7df16ee75059f6a5346d75de66d43fd99f5888c4089969 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:21:37.464 INFO:journalctl@ceph.osd.11.smithi069.stdout:Apr 21 23:21:37 smithi069 podman[70189]: 2024-04-21 23:21:37.326459075 +0000 UTC m=+0.589908635 container died 1fa85a5c099d333cbe7df16ee75059f6a5346d75de66d43fd99f5888c4089969 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T23:21:37.889 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@osd.11.service' 2024-04-21T23:21:37.926 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:21:37.926 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-21T23:21:37.926 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-21T23:21:37.927 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.a 2024-04-21T23:21:38.253 INFO:journalctl@ceph.mds.a.smithi044.stdout:Apr 21 23:21:37 smithi044 systemd[1]: Stopping Ceph mds.a for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:21:38.685 INFO:journalctl@ceph.mds.a.smithi044.stdout:Apr 21 23:21:38 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-a[67112]: 2024-04-21T23:21:38.253+0000 7fa05adbb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:21:38.685 INFO:journalctl@ceph.mds.a.smithi044.stdout:Apr 21 23:21:38 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-a[67112]: 2024-04-21T23:21:38.253+0000 7fa05adbb700 -1 mds.a *** got signal Terminated *** 2024-04-21T23:21:42.670 INFO:journalctl@ceph.mds.a.smithi044.stdout:Apr 21 23:21:42 smithi044 podman[94238]: 2024-04-21 23:21:42.259215429 +0000 UTC m=+4.123775205 container died 8d3075c20f536433232d7aba0d67f3399ecade9e9dff0b0404fcd14bb89cd97d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-a, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-820a2f8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-21T23:21:42.935 INFO:journalctl@ceph.mds.a.smithi044.stdout:Apr 21 23:21:42 smithi044 podman[94238]: 2024-04-21 23:21:42.745632085 +0000 UTC m=+4.610191858 container cleanup 8d3075c20f536433232d7aba0d67f3399ecade9e9dff0b0404fcd14bb89cd97d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8) 2024-04-21T23:21:42.935 INFO:journalctl@ceph.mds.a.smithi044.stdout:Apr 21 23:21:42 smithi044 bash[94238]: ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-a 2024-04-21T23:21:43.179 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.a.service' 2024-04-21T23:21:43.214 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:21:43.215 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-21T23:21:43.215 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-21T23:21:43.215 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.d 2024-04-21T23:21:43.685 INFO:journalctl@ceph.mds.d.smithi044.stdout:Apr 21 23:21:43 smithi044 systemd[1]: Stopping Ceph mds.d for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:21:43.685 INFO:journalctl@ceph.mds.d.smithi044.stdout:Apr 21 23:21:43 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-d[66153]: 2024-04-21T23:21:43.515+0000 7fde9badf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:21:43.685 INFO:journalctl@ceph.mds.d.smithi044.stdout:Apr 21 23:21:43 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-d[66153]: 2024-04-21T23:21:43.515+0000 7fde9badf700 -1 mds.d *** got signal Terminated *** 2024-04-21T23:21:47.557 INFO:journalctl@ceph.mds.d.smithi044.stdout:Apr 21 23:21:47 smithi044 podman[94358]: 2024-04-21 23:21:47.286574509 +0000 UTC m=+3.851439012 container died 126f059275fe00e55915d9306d286b9b9862e9b6a0ccdf6aa0fdea108ad01431 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:21:47.934 INFO:journalctl@ceph.mds.d.smithi044.stdout:Apr 21 23:21:47 smithi044 podman[94358]: 2024-04-21 23:21:47.632928983 +0000 UTC m=+4.197793485 container cleanup 126f059275fe00e55915d9306d286b9b9862e9b6a0ccdf6aa0fdea108ad01431 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-21T23:21:47.935 INFO:journalctl@ceph.mds.d.smithi044.stdout:Apr 21 23:21:47 smithi044 bash[94358]: ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-d 2024-04-21T23:21:47.935 INFO:journalctl@ceph.mds.d.smithi044.stdout:Apr 21 23:21:47 smithi044 podman[94372]: 2024-04-21 23:21:47.784121634 +0000 UTC m=+0.487582186 container remove 126f059275fe00e55915d9306d286b9b9862e9b6a0ccdf6aa0fdea108ad01431 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-21T23:21:48.019 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.d.service' 2024-04-21T23:21:48.051 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:21:48.051 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-21T23:21:48.051 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-21T23:21:48.051 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.g 2024-04-21T23:21:48.427 INFO:journalctl@ceph.mds.g.smithi044.stdout:Apr 21 23:21:48 smithi044 systemd[1]: Stopping Ceph mds.g for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:21:48.684 INFO:journalctl@ceph.mds.g.smithi044.stdout:Apr 21 23:21:48 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-g[66765]: 2024-04-21T23:21:48.428+0000 7fea92d19700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:21:48.685 INFO:journalctl@ceph.mds.g.smithi044.stdout:Apr 21 23:21:48 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-g[66765]: 2024-04-21T23:21:48.428+0000 7fea92d19700 -1 mds.g *** got signal Terminated *** 2024-04-21T23:21:52.435 INFO:journalctl@ceph.mds.g.smithi044.stdout:Apr 21 23:21:52 smithi044 podman[94477]: 2024-04-21 23:21:52.143447714 +0000 UTC m=+3.860884094 container died 0bc183c485bb907862c5a1f46ea168313a615e7967aa2a3f43dd4d3a6f751810 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-g, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-21T23:21:52.745 INFO:journalctl@ceph.mds.g.smithi044.stdout:Apr 21 23:21:52 smithi044 podman[94477]: 2024-04-21 23:21:52.536774724 +0000 UTC m=+4.254211102 container cleanup 0bc183c485bb907862c5a1f46ea168313a615e7967aa2a3f43dd4d3a6f751810 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-21T23:21:52.745 INFO:journalctl@ceph.mds.g.smithi044.stdout:Apr 21 23:21:52 smithi044 bash[94477]: ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-g 2024-04-21T23:21:52.860 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.g.service' 2024-04-21T23:21:52.892 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:21:52.892 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-21T23:21:52.892 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-21T23:21:52.893 DEBUG:teuthology.orchestra.run.smithi044:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.j 2024-04-21T23:21:53.185 INFO:journalctl@ceph.mds.j.smithi044.stdout:Apr 21 23:21:52 smithi044 systemd[1]: Stopping Ceph mds.j for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:21:53.684 INFO:journalctl@ceph.mds.j.smithi044.stdout:Apr 21 23:21:53 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-j[65801]: 2024-04-21T23:21:53.210+0000 7f79d1ed9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:21:53.685 INFO:journalctl@ceph.mds.j.smithi044.stdout:Apr 21 23:21:53 smithi044 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-j[65801]: 2024-04-21T23:21:53.210+0000 7f79d1ed9700 -1 mds.j *** got signal Terminated *** 2024-04-21T23:21:56.548 INFO:journalctl@ceph.mds.j.smithi044.stdout:Apr 21 23:21:56 smithi044 podman[94600]: 2024-04-21 23:21:56.291255805 +0000 UTC m=+3.181253238 container died a058353c7bf20855a3fe8914bc84d667efda97a0592989028146f7861faef1ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-21T23:21:56.807 INFO:journalctl@ceph.mds.j.smithi044.stdout:Apr 21 23:21:56 smithi044 podman[94600]: 2024-04-21 23:21:56.623928237 +0000 UTC m=+3.513925663 container cleanup a058353c7bf20855a3fe8914bc84d667efda97a0592989028146f7861faef1ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-820a2f8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-04-21T23:21:56.807 INFO:journalctl@ceph.mds.j.smithi044.stdout:Apr 21 23:21:56 smithi044 bash[94600]: ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-j 2024-04-21T23:21:57.057 DEBUG:teuthology.orchestra.run.smithi044:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.j.service' 2024-04-21T23:21:57.077 INFO:journalctl@ceph.mds.j.smithi044.stdout:Apr 21 23:21:56 smithi044 podman[94614]: 2024-04-21 23:21:56.808895124 +0000 UTC m=+0.517260084 container remove a058353c7bf20855a3fe8914bc84d667efda97a0592989028146f7861faef1ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS) 2024-04-21T23:21:57.077 INFO:journalctl@ceph.mds.j.smithi044.stdout:Apr 21 23:21:57 smithi044 systemd[1]: ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.j.service: Deactivated successfully. 2024-04-21T23:21:57.078 INFO:journalctl@ceph.mds.j.smithi044.stdout:Apr 21 23:21:57 smithi044 systemd[1]: Stopped Ceph mds.j for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:21:57.078 INFO:journalctl@ceph.mds.j.smithi044.stdout:Apr 21 23:21:57 smithi044 systemd[1]: ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.j.service: Consumed 1.071s CPU time. 2024-04-21T23:21:57.088 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:21:57.088 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-21T23:21:57.089 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-21T23:21:57.089 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.b 2024-04-21T23:21:57.404 INFO:journalctl@ceph.mds.b.smithi055.stdout:Apr 21 23:21:57 smithi055 systemd[1]: Stopping Ceph mds.b for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:21:57.714 INFO:journalctl@ceph.mds.b.smithi055.stdout:Apr 21 23:21:57 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-b[61560]: 2024-04-21T23:21:57.405+0000 7f0b9c852700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:21:57.714 INFO:journalctl@ceph.mds.b.smithi055.stdout:Apr 21 23:21:57 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-b[61560]: 2024-04-21T23:21:57.405+0000 7f0b9c852700 -1 mds.b *** got signal Terminated *** 2024-04-21T23:22:01.631 INFO:journalctl@ceph.mds.b.smithi055.stdout:Apr 21 23:22:01 smithi055 podman[67977]: 2024-04-21 23:22:01.355460558 +0000 UTC m=+4.059420594 container died c87ce9581cca077152dacb68c6ca328247ae81f1ef015fb0371aeb85cf16ac13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, GIT_BRANCH=HEAD) 2024-04-21T23:22:01.939 INFO:journalctl@ceph.mds.b.smithi055.stdout:Apr 21 23:22:01 smithi055 podman[67977]: 2024-04-21 23:22:01.723100172 +0000 UTC m=+4.427060213 container cleanup c87ce9581cca077152dacb68c6ca328247ae81f1ef015fb0371aeb85cf16ac13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-21T23:22:01.940 INFO:journalctl@ceph.mds.b.smithi055.stdout:Apr 21 23:22:01 smithi055 bash[67977]: ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-b 2024-04-21T23:22:02.081 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.b.service' 2024-04-21T23:22:02.118 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:22:02.118 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-21T23:22:02.118 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-21T23:22:02.118 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.e 2024-04-21T23:22:02.463 INFO:journalctl@ceph.mds.e.smithi055.stdout:Apr 21 23:22:02 smithi055 systemd[1]: Stopping Ceph mds.e for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:22:02.464 INFO:journalctl@ceph.mds.e.smithi055.stdout:Apr 21 23:22:02 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-e[60930]: 2024-04-21T23:22:02.418+0000 7f9251c19700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:22:02.464 INFO:journalctl@ceph.mds.e.smithi055.stdout:Apr 21 23:22:02 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-e[60930]: 2024-04-21T23:22:02.418+0000 7f9251c19700 -1 mds.e *** got signal Terminated *** 2024-04-21T23:22:06.217 INFO:journalctl@ceph.mds.e.smithi055.stdout:Apr 21 23:22:06 smithi055 podman[68111]: 2024-04-21 23:22:06.217676451 +0000 UTC m=+3.880727644 container died b2cad9b74fe43640e56397bf03c733fa9e2b56af61c8ed5e8db38892686ace04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0) 2024-04-21T23:22:06.754 INFO:journalctl@ceph.mds.e.smithi055.stdout:Apr 21 23:22:06 smithi055 podman[68111]: 2024-04-21 23:22:06.554930336 +0000 UTC m=+4.217981527 container cleanup b2cad9b74fe43640e56397bf03c733fa9e2b56af61c8ed5e8db38892686ace04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416) 2024-04-21T23:22:06.755 INFO:journalctl@ceph.mds.e.smithi055.stdout:Apr 21 23:22:06 smithi055 bash[68111]: ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-e 2024-04-21T23:22:06.989 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.e.service' 2024-04-21T23:22:07.011 INFO:journalctl@ceph.mds.e.smithi055.stdout:Apr 21 23:22:06 smithi055 podman[68125]: 2024-04-21 23:22:06.756076265 +0000 UTC m=+0.536146396 container remove b2cad9b74fe43640e56397bf03c733fa9e2b56af61c8ed5e8db38892686ace04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-21T23:22:07.011 INFO:journalctl@ceph.mds.e.smithi055.stdout:Apr 21 23:22:06 smithi055 systemd[1]: ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.e.service: Deactivated successfully. 2024-04-21T23:22:07.011 INFO:journalctl@ceph.mds.e.smithi055.stdout:Apr 21 23:22:06 smithi055 systemd[1]: Stopped Ceph mds.e for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:22:07.011 INFO:journalctl@ceph.mds.e.smithi055.stdout:Apr 21 23:22:06 smithi055 systemd[1]: ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.e.service: Consumed 1.081s CPU time. 2024-04-21T23:22:07.022 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:22:07.023 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-21T23:22:07.023 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-21T23:22:07.023 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.h 2024-04-21T23:22:07.340 INFO:journalctl@ceph.mds.h.smithi055.stdout:Apr 21 23:22:07 smithi055 systemd[1]: Stopping Ceph mds.h for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:22:07.713 INFO:journalctl@ceph.mds.h.smithi055.stdout:Apr 21 23:22:07 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-h[61239]: 2024-04-21T23:22:07.341+0000 7fca37608700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:22:07.714 INFO:journalctl@ceph.mds.h.smithi055.stdout:Apr 21 23:22:07 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-h[61239]: 2024-04-21T23:22:07.341+0000 7fca37608700 -1 mds.h *** got signal Terminated *** 2024-04-21T23:22:10.749 INFO:journalctl@ceph.mds.h.smithi055.stdout:Apr 21 23:22:10 smithi055 podman[68232]: 2024-04-21 23:22:10.498743462 +0000 UTC m=+3.250633908 container died 4ecdcab93f9b8f822121d19431bee23217e89a0a1e5ecbf2e77c39b6ef36ebf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, RELEASE=squid-820a2f8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:22:10.999 INFO:journalctl@ceph.mds.h.smithi055.stdout:Apr 21 23:22:10 smithi055 podman[68232]: 2024-04-21 23:22:10.841854135 +0000 UTC m=+3.593744587 container cleanup 4ecdcab93f9b8f822121d19431bee23217e89a0a1e5ecbf2e77c39b6ef36ebf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-h, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:22:10.999 INFO:journalctl@ceph.mds.h.smithi055.stdout:Apr 21 23:22:10 smithi055 bash[68232]: ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-h 2024-04-21T23:22:11.216 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.h.service' 2024-04-21T23:22:11.248 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:22:11.248 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-21T23:22:11.248 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-21T23:22:11.248 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.k 2024-04-21T23:22:11.577 INFO:journalctl@ceph.mds.k.smithi055.stdout:Apr 21 23:22:11 smithi055 systemd[1]: Stopping Ceph mds.k for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:22:11.964 INFO:journalctl@ceph.mds.k.smithi055.stdout:Apr 21 23:22:11 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-k[60610]: 2024-04-21T23:22:11.578+0000 7f0245849700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:22:11.964 INFO:journalctl@ceph.mds.k.smithi055.stdout:Apr 21 23:22:11 smithi055 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-k[60610]: 2024-04-21T23:22:11.578+0000 7f0245849700 -1 mds.k *** got signal Terminated *** 2024-04-21T23:22:13.662 INFO:journalctl@ceph.mds.k.smithi055.stdout:Apr 21 23:22:13 smithi055 podman[68353]: 2024-04-21 23:22:13.386026431 +0000 UTC m=+1.918786871 container died 3800890d2e1efc0e0f4102cb6fb7e676caf462c103554086d1a24f80fe003c8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-k, GIT_CLEAN=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0) 2024-04-21T23:22:13.964 INFO:journalctl@ceph.mds.k.smithi055.stdout:Apr 21 23:22:13 smithi055 podman[68353]: 2024-04-21 23:22:13.7626796 +0000 UTC m=+2.295440052 container cleanup 3800890d2e1efc0e0f4102cb6fb7e676caf462c103554086d1a24f80fe003c8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T23:22:13.964 INFO:journalctl@ceph.mds.k.smithi055.stdout:Apr 21 23:22:13 smithi055 bash[68353]: ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-k 2024-04-21T23:22:14.200 DEBUG:teuthology.orchestra.run.smithi055:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.k.service' 2024-04-21T23:22:14.231 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:22:14.231 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-21T23:22:14.231 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-21T23:22:14.232 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.c 2024-04-21T23:22:14.593 INFO:journalctl@ceph.mds.c.smithi069.stdout:Apr 21 23:22:14 smithi069 systemd[1]: Stopping Ceph mds.c for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:22:14.964 INFO:journalctl@ceph.mds.c.smithi069.stdout:Apr 21 23:22:14 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-c[62885]: 2024-04-21T23:22:14.593+0000 7f38be1e2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:22:14.964 INFO:journalctl@ceph.mds.c.smithi069.stdout:Apr 21 23:22:14 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-c[62885]: 2024-04-21T23:22:14.593+0000 7f38be1e2700 -1 mds.c *** got signal Terminated *** 2024-04-21T23:22:15.935 INFO:journalctl@ceph.mds.c.smithi069.stdout:Apr 21 23:22:15 smithi069 podman[70305]: 2024-04-21 23:22:15.677749451 +0000 UTC m=+1.255510216 container died ce10f093697d202b2a4f899a0e6b2eb704a8b696ddeebed27d47d37464804689 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-21T23:22:16.214 INFO:journalctl@ceph.mds.c.smithi069.stdout:Apr 21 23:22:16 smithi069 podman[70305]: 2024-04-21 23:22:16.010604757 +0000 UTC m=+1.588365528 container cleanup ce10f093697d202b2a4f899a0e6b2eb704a8b696ddeebed27d47d37464804689 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-820a2f8, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-21T23:22:16.214 INFO:journalctl@ceph.mds.c.smithi069.stdout:Apr 21 23:22:16 smithi069 bash[70305]: ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-c 2024-04-21T23:22:16.434 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.c.service' 2024-04-21T23:22:16.468 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:22:16.469 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-21T23:22:16.469 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-21T23:22:16.469 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.f 2024-04-21T23:22:16.779 INFO:journalctl@ceph.mds.f.smithi069.stdout:Apr 21 23:22:16 smithi069 systemd[1]: Stopping Ceph mds.f for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:22:17.214 INFO:journalctl@ceph.mds.f.smithi069.stdout:Apr 21 23:22:16 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-f[62569]: 2024-04-21T23:22:16.779+0000 7f3bfffc1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:22:17.214 INFO:journalctl@ceph.mds.f.smithi069.stdout:Apr 21 23:22:16 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-f[62569]: 2024-04-21T23:22:16.779+0000 7f3bfffc1700 -1 mds.f *** got signal Terminated *** 2024-04-21T23:22:21.214 INFO:journalctl@ceph.mds.f.smithi069.stdout:Apr 21 23:22:20 smithi069 podman[70426]: 2024-04-21 23:22:20.78545985 +0000 UTC m=+4.099336833 container died 81754f9122c7465848eb9e454aa7b4a255acba0530f845cbe697d087a32df721 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-820a2f8) 2024-04-21T23:22:21.663 INFO:journalctl@ceph.mds.f.smithi069.stdout:Apr 21 23:22:21 smithi069 podman[70426]: 2024-04-21 23:22:21.355359022 +0000 UTC m=+4.669235992 container cleanup 81754f9122c7465848eb9e454aa7b4a255acba0530f845cbe697d087a32df721 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-f, RELEASE=squid-820a2f8, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-21T23:22:21.663 INFO:journalctl@ceph.mds.f.smithi069.stdout:Apr 21 23:22:21 smithi069 bash[70426]: ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-f 2024-04-21T23:22:21.839 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.f.service' 2024-04-21T23:22:21.870 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:22:21.870 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-21T23:22:21.870 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-21T23:22:21.870 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.i 2024-04-21T23:22:22.192 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:22:21 smithi069 systemd[1]: Stopping Ceph mds.i for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:22:22.464 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:22:22 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: 2024-04-21T23:22:22.192+0000 7fa8e0451700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:22:22.464 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:22:22 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i[63200]: 2024-04-21T23:22:22.192+0000 7fa8e0451700 -1 mds.i *** got signal Terminated *** 2024-04-21T23:22:25.700 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:22:25 smithi069 podman[70547]: 2024-04-21 23:22:25.347731061 +0000 UTC m=+3.260650335 container died fa6d4b0214a33607ad61e9bcbd42c69147945602695b3896aaf4739d44caef61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i, RELEASE=squid-820a2f8, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-21T23:22:25.964 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:22:25 smithi069 podman[70547]: 2024-04-21 23:22:25.793035386 +0000 UTC m=+3.705954648 container cleanup fa6d4b0214a33607ad61e9bcbd42c69147945602695b3896aaf4739d44caef61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-21T23:22:25.964 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:22:25 smithi069 bash[70547]: ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i 2024-04-21T23:22:26.210 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.i.service' 2024-04-21T23:22:26.230 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:22:25 smithi069 podman[70561]: 2024-04-21 23:22:25.976975391 +0000 UTC m=+0.626165989 container remove fa6d4b0214a33607ad61e9bcbd42c69147945602695b3896aaf4739d44caef61 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-820a2f8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-21T23:22:26.230 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:22:26 smithi069 systemd[1]: ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.i.service: Deactivated successfully. 2024-04-21T23:22:26.230 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:22:26 smithi069 systemd[1]: Stopped Ceph mds.i for 80909b94-0032-11ef-bc93-c7b262605968. 2024-04-21T23:22:26.230 INFO:journalctl@ceph.mds.i.smithi069.stdout:Apr 21 23:22:26 smithi069 systemd[1]: ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.i.service: Consumed 28.576s CPU time. 2024-04-21T23:22:26.240 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:22:26.241 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-21T23:22:26.241 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-21T23:22:26.241 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.l 2024-04-21T23:22:26.561 INFO:journalctl@ceph.mds.l.smithi069.stdout:Apr 21 23:22:26 smithi069 systemd[1]: Stopping Ceph mds.l for 80909b94-0032-11ef-bc93-c7b262605968... 2024-04-21T23:22:26.964 INFO:journalctl@ceph.mds.l.smithi069.stdout:Apr 21 23:22:26 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-l[62263]: 2024-04-21T23:22:26.562+0000 7ff944a66700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T23:22:26.964 INFO:journalctl@ceph.mds.l.smithi069.stdout:Apr 21 23:22:26 smithi069 ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-l[62263]: 2024-04-21T23:22:26.562+0000 7ff944a66700 -1 mds.l *** got signal Terminated *** 2024-04-21T23:22:30.770 INFO:journalctl@ceph.mds.l.smithi069.stdout:Apr 21 23:22:30 smithi069 podman[70670]: 2024-04-21 23:22:30.485557102 +0000 UTC m=+4.026337738 container died 4512bfa6f430ef16036785807c87e7ce38128a41f2c8e90473903b94d33c8c5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T23:22:31.095 INFO:journalctl@ceph.mds.l.smithi069.stdout:Apr 21 23:22:30 smithi069 podman[70670]: 2024-04-21 23:22:30.837569658 +0000 UTC m=+4.378350344 container cleanup 4512bfa6f430ef16036785807c87e7ce38128a41f2c8e90473903b94d33c8c5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d4c08cf2bab46b9f2f2d301f0b4aed7a04ebf590fcda86c56afbf3304d624cb9, name=ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-820a2f8, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-21T23:22:31.095 INFO:journalctl@ceph.mds.l.smithi069.stdout:Apr 21 23:22:30 smithi069 bash[70670]: ceph-80909b94-0032-11ef-bc93-c7b262605968-mds-l 2024-04-21T23:22:31.268 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80909b94-0032-11ef-bc93-c7b262605968@mds.l.service' 2024-04-21T23:22:31.299 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T23:22:31.299 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-21T23:22:31.299 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 80909b94-0032-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-21T23:22:31.500 INFO:teuthology.orchestra.run.smithi044.stdout:Deleting cluster with fsid: 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T23:22:34.010 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 80909b94-0032-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-21T23:22:34.208 INFO:teuthology.orchestra.run.smithi055.stdout:Deleting cluster with fsid: 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T23:22:36.773 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 80909b94-0032-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-21T23:22:36.960 INFO:teuthology.orchestra.run.smithi069.stdout:Deleting cluster with fsid: 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T23:22:39.361 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T23:22:39.395 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T23:22:39.427 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T23:22:39.452 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-21T23:22:39.455 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667584/remote/smithi044/crash 2024-04-21T23:22:39.456 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/crash -- . 2024-04-21T23:22:39.497 INFO:teuthology.orchestra.run.smithi044.stderr:tar: /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-21T23:22:39.498 INFO:teuthology.orchestra.run.smithi044.stderr:tar: Error is not recoverable: exiting now 2024-04-21T23:22:39.500 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667584/remote/smithi055/crash 2024-04-21T23:22:39.501 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/crash -- . 2024-04-21T23:22:39.537 INFO:teuthology.orchestra.run.smithi055.stderr:tar: /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-21T23:22:39.537 INFO:teuthology.orchestra.run.smithi055.stderr:tar: Error is not recoverable: exiting now 2024-04-21T23:22:39.539 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667584/remote/smithi069/crash 2024-04-21T23:22:39.540 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/crash -- . 2024-04-21T23:22:39.572 INFO:teuthology.orchestra.run.smithi069.stderr:tar: /var/lib/ceph/80909b94-0032-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-21T23:22:39.572 INFO:teuthology.orchestra.run.smithi069.stderr:tar: Error is not recoverable: exiting now 2024-04-21T23:22:39.573 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-21T23:22:39.573 DEBUG:teuthology.orchestra.run.smithi044:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-21T23:22:39.608 INFO:tasks.cephadm:Compressing logs... 2024-04-21T23:22:39.608 DEBUG:teuthology.orchestra.run.smithi044:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-21T23:22:39.650 DEBUG:teuthology.orchestra.run.smithi055:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-21T23:22:39.653 DEBUG:teuthology.orchestra.run.smithi069:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-21T23:22:39.674 INFO:teuthology.orchestra.run.smithi044.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.log 2024-04-21T23:22:39.674 INFO:teuthology.orchestra.run.smithi044.stderr:: No such file or directory 2024-04-21T23:22:39.674 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-21T23:22:39.675 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-21T23:22:39.676 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-21T23:22:39.676 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-21T23:22:39.676 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-21T23:22:39.679 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.j.log: /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-21T23:22:39.679 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-21T23:22:39.679 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-21T23:22:39.679 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mon.a.log: 86.7% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-21T23:22:39.679 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-21T23:22:39.680 INFO:teuthology.orchestra.run.smithi044.stderr: 90.0% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-21T23:22:39.690 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.6.log: 94.6% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-21T23:22:39.690 INFO:teuthology.orchestra.run.smithi044.stderr: 94.6% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-21T23:22:39.690 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-21T23:22:39.690 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-21T23:22:39.690 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.a.log: 83.8% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-21T23:22:39.691 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-21T23:22:39.691 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-volume.log: --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-21T23:22:39.691 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.d.log: 94.6% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-21T23:22:39.691 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-21T23:22:39.691 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.9.log: 94.5% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-21T23:22:39.691 INFO:teuthology.orchestra.run.smithi044.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-21T23:22:39.693 INFO:teuthology.orchestra.run.smithi069.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.log 2024-04-21T23:22:39.693 INFO:teuthology.orchestra.run.smithi069.stderr:: No such file or directory 2024-04-21T23:22:39.693 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-21T23:22:39.694 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-21T23:22:39.694 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-21T23:22:39.694 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-21T23:22:39.694 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-21T23:22:39.694 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-21T23:22:39.694 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-21T23:22:39.694 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-21T23:22:39.695 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mgr.z.log: 86.7% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-21T23:22:39.695 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-21T23:22:39.695 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.i.log: 90.1% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-21T23:22:39.695 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-21T23:22:39.695 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.6% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-21T23:22:39.695 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 94.5% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-21T23:22:39.695 INFO:teuthology.orchestra.run.smithi069.stderr: --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-21T23:22:39.695 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-21T23:22:39.696 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.8.log: 94.6% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-21T23:22:39.696 INFO:teuthology.orchestra.run.smithi069.stderr: 91.9% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-21T23:22:39.696 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-21T23:22:39.696 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-21T23:22:39.699 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/cephadm.log: 91.1% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-21T23:22:39.699 INFO:teuthology.orchestra.run.smithi044.stderr: 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-21T23:22:39.699 INFO:teuthology.orchestra.run.smithi044.stderr:/var/log/ceph/ceph-client.admin.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-21T23:22:39.700 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.l.log: 94.5% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-21T23:22:39.700 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/cephadm.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-21T23:22:39.702 INFO:teuthology.orchestra.run.smithi055.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-21T23:22:39.702 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.log 2024-04-21T23:22:39.702 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-21T23:22:39.702 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-21T23:22:39.703 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-21T23:22:39.703 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-21T23:22:39.703 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-21T23:22:39.703 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-21T23:22:39.703 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-21T23:22:39.703 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-21T23:22:39.703 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-21T23:22:39.703 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-21T23:22:39.703 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-21T23:22:39.703 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-21T23:22:39.704 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-21T23:22:39.704 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-21T23:22:39.704 INFO:teuthology.orchestra.run.smithi069.stderr: 91.0% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-21T23:22:39.807 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 91.9% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-21T23:22:39.876 INFO:teuthology.orchestra.run.smithi055.stderr: 94.6% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-21T23:22:39.910 INFO:teuthology.orchestra.run.smithi055.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-21T23:22:39.911 INFO:teuthology.orchestra.run.smithi055.stderr: 91.1% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-21T23:22:40.024 INFO:teuthology.orchestra.run.smithi055.stderr: 83.7% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-21T23:22:40.025 INFO:teuthology.orchestra.run.smithi055.stderr: 94.5% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-21T23:22:40.049 INFO:teuthology.orchestra.run.smithi044.stderr: 90.3% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-21T23:22:40.067 INFO:teuthology.orchestra.run.smithi055.stderr: 90.1% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-21T23:22:40.068 INFO:teuthology.orchestra.run.smithi055.stderr: 86.7% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-21T23:22:40.084 INFO:teuthology.orchestra.run.smithi055.stderr: 94.5% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-21T23:22:40.645 INFO:teuthology.orchestra.run.smithi069.stderr: 92.9% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-21T23:22:40.959 INFO:teuthology.orchestra.run.smithi044.stderr: 90.5% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-21T23:22:41.026 INFO:teuthology.orchestra.run.smithi055.stderr: 91.8% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-21T23:22:50.716 INFO:teuthology.orchestra.run.smithi069.stderr: 95.4% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-21T23:22:57.339 INFO:teuthology.orchestra.run.smithi069.stderr: 91.7% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-21T23:22:58.679 INFO:teuthology.orchestra.run.smithi044.stderr: 91.8% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-21T23:22:58.720 INFO:teuthology.orchestra.run.smithi069.stderr: 91.6% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-21T23:22:59.678 INFO:teuthology.orchestra.run.smithi055.stderr: 91.8% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-21T23:23:01.834 INFO:teuthology.orchestra.run.smithi044.stderr: 91.5% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-21T23:23:02.388 INFO:teuthology.orchestra.run.smithi044.stderr: 91.6% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-21T23:23:02.472 INFO:teuthology.orchestra.run.smithi069.stderr: 91.6% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-21T23:23:04.435 INFO:teuthology.orchestra.run.smithi044.stderr: 91.5% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-21T23:23:04.437 INFO:teuthology.orchestra.run.smithi044.stderr: 2024-04-21T23:23:04.437 INFO:teuthology.orchestra.run.smithi044.stderr:real 0m24.773s 2024-04-21T23:23:04.438 INFO:teuthology.orchestra.run.smithi044.stderr:user 1m27.653s 2024-04-21T23:23:04.438 INFO:teuthology.orchestra.run.smithi044.stderr:sys 0m2.544s 2024-04-21T23:23:11.275 INFO:teuthology.orchestra.run.smithi055.stderr: 91.9% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-21T23:23:12.580 INFO:teuthology.orchestra.run.smithi069.stderr: 91.3% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-21T23:23:12.582 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-21T23:23:12.582 INFO:teuthology.orchestra.run.smithi069.stderr:real 0m32.915s 2024-04-21T23:23:12.582 INFO:teuthology.orchestra.run.smithi069.stderr:user 1m41.081s 2024-04-21T23:23:12.582 INFO:teuthology.orchestra.run.smithi069.stderr:sys 0m3.158s 2024-04-21T23:23:24.546 INFO:teuthology.orchestra.run.smithi055.stderr: 91.5% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-21T23:23:28.213 INFO:teuthology.orchestra.run.smithi055.stderr: 91.7% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-21T23:25:34.744 INFO:teuthology.orchestra.run.smithi055.stderr: 93.6% -- replaced with /var/log/ceph/80909b94-0032-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-21T23:25:34.745 INFO:teuthology.orchestra.run.smithi055.stderr: 2024-04-21T23:25:34.746 INFO:teuthology.orchestra.run.smithi055.stderr:real 2m55.077s 2024-04-21T23:25:34.746 INFO:teuthology.orchestra.run.smithi055.stderr:user 3m57.941s 2024-04-21T23:25:34.746 INFO:teuthology.orchestra.run.smithi055.stderr:sys 0m7.241s 2024-04-21T23:25:34.747 INFO:tasks.cephadm:Archiving logs... 2024-04-21T23:25:34.747 DEBUG:teuthology.misc:Transferring archived files from smithi044:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667584/remote/smithi044/log 2024-04-21T23:25:34.748 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-21T23:25:49.540 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667584/remote/smithi055/log 2024-04-21T23:25:49.541 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-21T23:26:26.712 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667584/remote/smithi069/log 2024-04-21T23:26:26.714 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-21T23:26:43.974 INFO:tasks.cephadm:Removing cluster... 2024-04-21T23:26:43.974 DEBUG:teuthology.orchestra.run.smithi044:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 80909b94-0032-11ef-bc93-c7b262605968 --force 2024-04-21T23:26:44.175 INFO:teuthology.orchestra.run.smithi044.stdout:Deleting cluster with fsid: 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T23:26:44.595 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 80909b94-0032-11ef-bc93-c7b262605968 --force 2024-04-21T23:26:44.795 INFO:teuthology.orchestra.run.smithi055.stdout:Deleting cluster with fsid: 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T23:26:45.330 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 80909b94-0032-11ef-bc93-c7b262605968 --force 2024-04-21T23:26:45.519 INFO:teuthology.orchestra.run.smithi069.stdout:Deleting cluster with fsid: 80909b94-0032-11ef-bc93-c7b262605968 2024-04-21T23:26:45.909 INFO:tasks.cephadm:Removing cephadm ... 2024-04-21T23:26:45.909 DEBUG:teuthology.orchestra.run.smithi044:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-21T23:26:45.926 DEBUG:teuthology.orchestra.run.smithi055:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-21T23:26:45.943 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-21T23:26:45.958 INFO:tasks.cephadm:Teardown complete 2024-04-21T23:26:45.958 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-21T23:26:45.971 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-21T23:26:45.971 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-21T23:26:45.974 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-21T23:26:45.985 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-21T23:26:46.086 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-21T23:26:46.097 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-21T23:26:46.098 DEBUG:teuthology.orchestra.run.smithi044:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T23:26:46.101 DEBUG:teuthology.orchestra.run.smithi055:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T23:26:46.103 DEBUG:teuthology.orchestra.run.smithi069:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T23:26:46.113 INFO:teuthology.orchestra.run.smithi044.stderr:bash: line 1: ntpq: command not found 2024-04-21T23:26:46.117 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-04-21T23:26:46.117 INFO:teuthology.orchestra.run.smithi055.stderr:bash: line 1: ntpq: command not found 2024-04-21T23:26:46.118 INFO:teuthology.orchestra.run.smithi044.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T23:26:46.118 INFO:teuthology.orchestra.run.smithi044.stdout:=============================================================================== 2024-04-21T23:26:46.119 INFO:teuthology.orchestra.run.smithi044.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 63 -593us[ -593us] +/- 71ms 2024-04-21T23:26:46.119 INFO:teuthology.orchestra.run.smithi044.stdout:^* hv02.front.sepia.ceph.com 3 6 377 64 +294us[ +309us] +/- 65ms 2024-04-21T23:26:46.119 INFO:teuthology.orchestra.run.smithi044.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 +292us[ +292us] +/- 76ms 2024-04-21T23:26:46.119 INFO:teuthology.orchestra.run.smithi044.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:26:46.121 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T23:26:46.121 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-04-21T23:26:46.122 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 65 -598us[ -592us] +/- 71ms 2024-04-21T23:26:46.122 INFO:teuthology.orchestra.run.smithi069.stdout:^* hv02.front.sepia.ceph.com 3 6 377 4 +302us[ +309us] +/- 66ms 2024-04-21T23:26:46.122 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 +275us[ +275us] +/- 76ms 2024-04-21T23:26:46.122 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:26:46.127 INFO:teuthology.orchestra.run.smithi055.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T23:26:46.127 INFO:teuthology.orchestra.run.smithi055.stdout:=============================================================================== 2024-04-21T23:26:46.127 INFO:teuthology.orchestra.run.smithi055.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 64 -605us[ -600us] +/- 71ms 2024-04-21T23:26:46.127 INFO:teuthology.orchestra.run.smithi055.stdout:^* hv02.front.sepia.ceph.com 3 6 377 1 +291us[ +297us] +/- 66ms 2024-04-21T23:26:46.127 INFO:teuthology.orchestra.run.smithi055.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 +284us[ +289us] +/- 75ms 2024-04-21T23:26:46.127 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T23:26:46.128 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-21T23:26:46.140 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-21T23:26:46.140 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-21T23:26:46.182 DEBUG:teuthology.orchestra.run.smithi044:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-21T23:26:46.261 DEBUG:teuthology.orchestra.run.smithi055:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-21T23:26:46.355 DEBUG:teuthology.orchestra.run.smithi069:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-21T23:26:46.418 DEBUG:teuthology.orchestra.run.smithi044:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T23:26:46.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:26:46.447 DEBUG:teuthology.orchestra.run.smithi055:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T23:26:46.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:26:46.478 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T23:26:46.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:26:46.504 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-21T23:26:46.516 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-21T23:26:46.605 INFO:teuthology.task.internal:Duration was 2130.591264 seconds 2024-04-21T23:26:46.606 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-21T23:26:46.654 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-21T23:26:46.654 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-21T23:26:46.657 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-21T23:26:46.659 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-21T23:26:46.694 INFO:teuthology.orchestra.run.smithi044.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T23:26:46.695 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T23:26:46.709 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T23:26:47.017 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-21T23:26:47.017 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi044.front.sepia.ceph.com 2024-04-21T23:26:47.017 DEBUG:teuthology.orchestra.run.smithi044:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-21T23:26:47.039 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi055.front.sepia.ceph.com 2024-04-21T23:26:47.039 DEBUG:teuthology.orchestra.run.smithi055:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-21T23:26:47.066 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi069.front.sepia.ceph.com 2024-04-21T23:26:47.067 DEBUG:teuthology.orchestra.run.smithi069:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-21T23:26:47.092 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-21T23:26:47.092 DEBUG:teuthology.orchestra.run.smithi044:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-21T23:26:47.095 DEBUG:teuthology.orchestra.run.smithi055:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-21T23:26:47.109 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-21T23:26:47.157 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-21T23:26:47.158 DEBUG:teuthology.orchestra.run.smithi044:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-21T23:26:47.161 DEBUG:teuthology.orchestra.run.smithi055:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-21T23:26:47.179 DEBUG:teuthology.orchestra.run.smithi069:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-21T23:26:47.550 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-21T23:26:47.562 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-21T23:26:47.563 DEBUG:teuthology.orchestra.run.smithi044:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-21T23:26:47.587 DEBUG:teuthology.orchestra.run.smithi055:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-21T23:26:47.619 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-21T23:26:47.643 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-21T23:26:47.654 DEBUG:teuthology.orchestra.run.smithi044:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-21T23:26:47.657 DEBUG:teuthology.orchestra.run.smithi055:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-21T23:26:47.661 DEBUG:teuthology.orchestra.run.smithi069:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-21T23:26:47.677 INFO:teuthology.orchestra.run.smithi044.stdout:kernel.core_pattern = core 2024-04-21T23:26:47.688 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = core 2024-04-21T23:26:47.704 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = core 2024-04-21T23:26:47.721 DEBUG:teuthology.orchestra.run.smithi044:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-21T23:26:47.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:26:47.744 DEBUG:teuthology.orchestra.run.smithi055:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-21T23:26:47.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:26:47.764 DEBUG:teuthology.orchestra.run.smithi069:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-21T23:26:47.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T23:26:47.778 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-21T23:26:47.790 INFO:teuthology.task.internal:Transferring archived files... 2024-04-21T23:26:47.792 DEBUG:teuthology.misc:Transferring archived files from smithi044:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667584/remote/smithi044 2024-04-21T23:26:47.792 DEBUG:teuthology.orchestra.run.smithi044:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-21T23:26:47.836 DEBUG:teuthology.misc:Transferring archived files from smithi055:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667584/remote/smithi055 2024-04-21T23:26:47.836 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-21T23:26:47.872 DEBUG:teuthology.misc:Transferring archived files from smithi069:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667584/remote/smithi069 2024-04-21T23:26:47.872 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-21T23:26:47.906 INFO:teuthology.task.internal:Removing archive directory... 2024-04-21T23:26:47.906 DEBUG:teuthology.orchestra.run.smithi044:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-21T23:26:47.909 DEBUG:teuthology.orchestra.run.smithi055:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-21T23:26:47.911 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-21T23:26:47.955 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-21T23:26:47.969 INFO:teuthology.task.internal:Not uploading archives. 2024-04-21T23:26:47.970 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-21T23:26:48.001 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-21T23:26:48.001 DEBUG:teuthology.orchestra.run.smithi044:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-21T23:26:48.004 DEBUG:teuthology.orchestra.run.smithi055:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-21T23:26:48.006 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-21T23:26:48.016 INFO:teuthology.orchestra.run.smithi044.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 21 23:26 /home/ubuntu/cephtest 2024-04-21T23:26:48.019 INFO:teuthology.orchestra.run.smithi055.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 21 23:26 /home/ubuntu/cephtest 2024-04-21T23:26:48.021 INFO:teuthology.orchestra.run.smithi069.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 21 23:26 /home/ubuntu/cephtest 2024-04-21T23:26:48.022 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-21T23:26:48.034 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-21T23:26:48.151 INFO:teuthology.nuke:Checking targets against current locks 2024-04-21T23:26:48.179 DEBUG:teuthology.nuke:shortname: smithi044 2024-04-21T23:26:48.179 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T23:26:48.205 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi044.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667584', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 22:45:08.998255', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZOvhnCoO7RuFOartC1IN/OgpYX71LoFas7Yf+L8iyezIJEJJHq9lkcX4y7PFhh+Cl+y29sQRuC/d2gEfYhkcc='} 2024-04-21T23:26:48.206 DEBUG:teuthology.nuke:shortname: smithi055 2024-04-21T23:26:48.206 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T23:26:48.230 DEBUG:teuthology.nuke:shortname: smithi069 2024-04-21T23:26:48.231 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T23:26:48.238 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667584', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 22:45:09.000616', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKSVAq56VARQO+y2yeuZ/vYCWSxTekgJKkLSDcXTkdmzYvXXVZJGLVvqpiR/TTZlKTDEVRNMCAkJv/SD0t7xD5c='} 2024-04-21T23:26:48.274 INFO:teuthology.orchestra.console.smithi044:Power off 2024-04-21T23:26:48.274 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-21T23:26:48.283 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-21_21:24:02-fs-squid-distro-default-smithi/7667584', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 22:45:08.999772', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNZ2ARyEPbkEtkRZnOsqMYXK/A5Ks0sUyozuovJ0cl0Nx8nokcMxbrshyyLbO8SudTx87tx1rfIYFL2CIQupYJQ='} 2024-04-21T23:26:48.298 DEBUG:teuthology.orchestra.console.smithi044:power off output: Chassis Power Control: Down/Off 2024-04-21T23:26:48.298 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:26:48.316 INFO:teuthology.orchestra.console.smithi055:Power off 2024-04-21T23:26:48.316 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-21T23:26:48.330 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on 2024-04-21T23:26:48.340 DEBUG:teuthology.orchestra.console.smithi055:power off output: Chassis Power Control: Down/Off 2024-04-21T23:26:48.341 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:26:48.359 INFO:teuthology.orchestra.console.smithi069:Power off 2024-04-21T23:26:48.359 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-21T23:26:48.371 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-04-21T23:26:48.384 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2024-04-21T23:26:48.384 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:26:48.408 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-04-21T23:26:52.331 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:26:52.372 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:26:52.409 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:26:52.444 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is on 2024-04-21T23:26:52.485 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-04-21T23:26:52.520 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-04-21T23:26:56.446 DEBUG:teuthology.orchestra.console.smithi044:pexpect command: ipmitool -H smithi044.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:26:56.486 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:26:56.521 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:26:56.558 DEBUG:teuthology.orchestra.console.smithi044:check power output: Chassis Power is off 2024-04-21T23:26:56.598 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-04-21T23:26:56.633 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2024-04-21T23:26:56.659 INFO:teuthology.orchestra.console.smithi044:Power off completed 2024-04-21T23:26:56.734 INFO:teuthology.orchestra.console.smithi069:Power off completed 2024-04-21T23:27:00.603 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T23:27:00.713 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-04-21T23:27:00.815 INFO:teuthology.orchestra.console.smithi055:Power off completed 2024-04-21T23:27:00.921 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} duration: 2130.591263771057 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi044 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=430e09df97c8fc7dc2b2ae424f68ed11366c540f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=63779f9470784c72a2273e77559e930d status: fail success: false 2024-04-21T23:27:00.922 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-21T23:27:00.996 INFO:teuthology.run:FAIL